Founded in 1976, CGI is among the largest independent IT and business consulting services firms in the world. With 94,000 consultants and professionals across the globe, CGI delivers an end-to-end portfolio of capabilities, from strategic IT and business consulting to systems integration, managed IT and business process services and intellectual property solutions. CGI works with clients through a local relationship model complemented by a global delivery network that helps clients digitally transform their organizations and accelerate results. CGI Fiscal 2024 reported revenue is CA$14.68 billion and CGI shares are listed on the TSX (GIB.A) and the NYSE (GIB). Learn more at cgi.com.
Job Title: ETL Developer
Position: Software Engineer
Experience: 4- 7 Years
Category: Software Development/ Engineering
Shift: 1-10PM (Hybrid)
Main location: Bangalore / Chennai
Position ID: J0726-1184
Employment Type: Full Time
Education Qualification: Bachelor's degree in Computer Science or related field or higher with minimum 3 years of relevant experience.
Position Description: We are looking for an experienced .Net Developer to join our team. The ideal candidate should be passionate about coding and developing scalable and high-performance applications. The successful candidate will be responsible for designing, developing, optimizing, and supporting enterprise-grade ETL and ELT pipelines primarily using Azure Databricks, PySpark, SQL, and related Microsoft Azure services. This is a hands-on engineering role focused on building scalable, reliable, and high-performing data pipelines across development, testing, and production environments. The engineer will work closely with data architects, business analysts, application teams, quality assurance teams, and production support teams to deliver secure and well-governed data solutions in a regulated financial-services environment.
#LI-MP14
Your future duties and responsibilities
- Design, develop, test, deploy, and support ETL and ELT pipelines using Azure Databricks.
- Build scalable data transformation workflows using PySpark, Spark SQL, and Databricks notebooks.
- Develop reusable Databricks frameworks, utilities, and common pipeline components.
- Ingest and process data from relational databases, flat files, APIs, cloud storage, and enterprise applications.
- Design and implement batch and near-real-time data pipelines.
- Build and manage Databricks jobs, workflows, schedules, dependencies, retries, and alerts.
- Develop data cleansing, validation, enrichment, reconciliation, and transformation logic.
- Implement data quality controls, audit checks, exception handling, and restart capabilities.
- Optimize Spark workloads, SQL queries, cluster configuration, partitioning, caching, and file formats.
- Work with Delta Lake to support reliable, scalable, and transactional data processing.
- Design and maintain bronze, silver, and gold data layers within a lakehouse architecture.
- Support source-to-target mapping, data profiling, data lineage, and technical design activities.
- Develop and maintain SQL queries, stored procedures, views, and data reconciliation scripts.
- Integrate Databricks pipelines with Azure Data Lake Storage and other Azure services.
- Build and maintain automated deployment processes using Azure DevOps CI/CD pipelines.
- Promote code, notebooks, jobs, and configuration across development, testing, and production environments.
- Monitor pipeline performance and troubleshoot failures, data issues, and production incidents.
- Perform root-cause analysis and implement permanent fixes for recurring issues.
- Create and maintain technical documentation, data mappings, deployment procedures, and support runbooks.
- Ensure solutions comply with CIBC security, data governance, risk management, and regulatory requirements.
Required qualifications to be successful in this role
Must-Have Skills:
- Bachelor’s degree or diploma in Computer Science, Software Engineering, Data Engineering, Information Technology, or a related discipline.
- Strong professional experience developing ETL or ELT solutions using Azure Databricks.
- Strong hands-on experience with:
o Databricks notebooks
o Databricks Jobs and Workflows
o Apache Spark
o PySpark
o Spark SQL
o Delta Lake
o Cluster configuration and management
o Performance tuning and optimization
- Strong experience designing and developing complex ETL pipelines in Databricks.
- Experience building reusable and parameterized data pipelines.
- Strong SQL development skills, including:
o Complex queries
o Stored procedures
o Views and functions
o Data reconciliation
o Query optimization
o Performance troubleshooting
- Experience with Azure Data Lake Storage and cloud-based data ingestion.
- Experience with structured, semi-structured, and unstructured data formats, including CSV, JSON, Parquet, and Delta.
- Strong understanding of ETL and ELT patterns, data integration principles, and data processing frameworks.
- Experience implementing data quality checks, logging, monitoring, alerting, and exception handling.
- Experience with job orchestration, scheduling, dependency management, and restartability.
- Experience with source control and CI/CD deployment practices using Azure DevOps.
- Experience supporting Databricks pipelines in production environments.
- Strong analytical, troubleshooting, and problem-solving skills.
- Strong written and verbal communication skills.
Good-to-Have Skills:
- Experience with Databricks Unity Catalog, including access controls, governance, and data lineage.
- Experience with Databricks Asset Bundles or other Databricks deployment frameworks.
- Experience with Databricks SQL Warehouses.
- Knowledge of medallion and lakehouse architecture patterns.
- Experience with Auto Loader and incremental data ingestion.
- Experience with streaming pipelines using Spark Structured Streaming.
- Experience with Delta Live Tables or Lakeflow Declarative Pipelines.
- Experience with Python software engineering practices, including modular development and unit testing.
- Experience integrating Databricks with Azure Data Factory.
- Experience with Azure API Management and REST API integrations.
- Experience with PowerShell or other scripting languages.
- Familiarity with data modelling techniques, including dimensional and relational models.
- Experience migrating legacy ETL workloads from Informatica, SSIS, or similar platforms to Databricks.
- Previous experience working within banking, capital markets, commercial banking, corporate banking, or another regulated financial-services environment.
- Knowledge of corporate banking, lending, credit risk, client onboarding, financial transactions, or regulatory reporting data.
CGI is an equal opportunity employer. In addition, CGI is committed to providing accommodation for people with disabilities in accordance with provincial legislation. Please let us know if you require reasonable accommodation due to a disability during any aspect of the recruitment process and we will work with you to address your needs.
Together, as owners, let’s turn meaningful insights into action.
Life at CGI is rooted in ownership, teamwork, respect and belonging. Here, you’ll reach your full potential because…
You are invited to be an owner from day 1 as we work together to bring our Dream to life. That’s why we call ourselves CGI Partners rather than employees. We benefit from our collective success and actively shape our company’s strategy and direction.
Your work creates value. You’ll develop innovative solutions and build relationships with teammates and clients while accessing global capabilities to scale your ideas, embrace new opportunities, and benefit from expansive industry and technology expertise.
You’ll shape your career by joining a company built to grow and last. You’ll be supported by leaders who care about your health and well-being and provide you with opportunities to deepen your skills and broaden your horizons.
That same commitment to fairness extends to how we use technology. To support our recruitment team, AI tools may be used to help assess applications though they never replace human judgement. All hiring decisions remain entirely in the hands of our recruitment professionals.
Come join our team—one of the largest IT and business consulting services firms in the world.
