Develop and maintain ETL pipelines using PySpark / Spark SQL in Azure Databricks.
Process and transform large-scale financial data for various business and compliance applications.
Implement data pipelines ensuring data quality, governance, and lineage tracking.
Integrate data from different systems such as core banking, payments, and trading platforms.
Optimize Spark jobs for high performance and cost efficiency.
Collaborate with business teams and compliance departments to align on project requirements.
Apply error handling, logging, and monitoring techniques during pipeline development.
Perform advanced data transformations and develop user-defined functions (UDFs) where necessary.
Use Unix scripting, scheduling tools, and maintain generic process development and performance tuning.
Support audit requirements and ensure compliance with regulatory standards
