Project Role : Performance Engineer
Project Role Description : Diagnose issues that an in-house performance testing team has been unable to. There are five aspects to Performance Engineering: software development lifecycle and architecture, performance testing and validation, capacity planning, application performance management and problem detection and resolution.
Must have skills : Data Engineering
Good to have skills : Python (Programming Language), GitHub
Minimum
3 year(s) of experience is required
Educational Qualification : 15 years full time education
Summary:
As a Performance Engineer, a typical day involves investigating and resolving complex performance issues that have not been identified by the internal performance testing team. The role encompasses a broad spectrum of activities including analyzing software development processes and architecture, validating performance through rigorous testing, planning for capacity needs, managing application performance, and swiftly detecting and addressing problems to ensure optimal system functionality. This position requires a proactive approach to identifying bottlenecks and collaborating with various teams to enhance overall system efficiency and reliability.
KEY RESPONSIBILITIES
Set up and configure Databricks workspaces, including cluster management, access controls, and integration with cloud identity and access management services
Design and implement Medallion architecture (Bronze, Silver, and Gold layers) on cloud object storage using Delta Lake format, ensuring data quality and traceability at each layer
Build and maintain ETL and ELT data pipelines using PySpark and Spark SQL within Databricks, covering ingestion, transformation, deduplication, normalisation, and standardisation
Ingest data from multiple source systems including relational databases (Oracle, MySQL, SQL Server), flat files, and SaaS platforms such as Salesforce, into the Lakehouse
Execute data migration from Oracle and other legacy systems, storing transformed outputs in the cloud storage Gold Layer using Databricks batch processing
Orchestrate and automate data workflows using pipeline orchestration tools such as Apache Airflow, AWS Glue, or Azure Data Factory, ensuring timely, dependency-aware, and reliable data delivery
Create and manage Linked Services, Datasets, and connection configurations for source and sink systems
Monitor pipeline execution end-to-end, troubleshoot failures, implement corrective actions, and perform root cause analysis for recurring issues
Optimise Spark jobs for performance, scalability, and cost efficiency — including partitioning strategy, caching, and resource configuration
Ensure all data is stored in Delta format with appropriate schema enforcement, ACID compliance, and incremental load patterns
Manage sensitive credentials and connection strings securely using a secrets management service (e.g., AWS Secrets Manager, Azure Key Vault, or HashiCorp Vault)
Maintain data quality standards by implementing validation and reconciliation logic within pipeline workflows
Collaborate with data analysts, data scientists, and product teams across global delivery environments to understand data needs and deliver trusted datasets
Maintain thorough documentation of pipeline designs, data flows, architecture decisions, and operational runbooks
Required Skills & Experience
3–4 years of professional experience in data engineering, with a significant and demonstrable portion of that time working on Databricks in a production environment
Strong hands-on proficiency in PySpark and Python, with the ability to write clean, efficient, and maintainable transformation code
Proven experience designing and implementing Medallion architecture (Bronze / Silver / Gold) on cloud object storage, with Delta Lake as the storage layer
Solid understanding of Delta Lake capabilities: ACID transactions, schema evolution, time travel, and incremental ingestion patterns
Hands-on experience migrating and extracting data from Oracle and other relational databases (MySQL, SQL Server) into a cloud-based Lakehouse
Experience with batch data ingestion from SaaS platforms such as Salesforce
Working knowledge of at least one pipeline orchestration tool — Apache Airflow, AWS Glue, Azure Data Factory, or similar — for scheduling and dependency management
Familiarity with cloud-hosted relational databases and MySQL for structured data storage and maintenance
Sound understanding of ETL and ELT design patterns and the trade-offs between them
Proficiency in Git and GitHub for version control, branching, and collaborative code review
Experience working within Agile or Waterfall delivery frameworks with structured sprint or milestone-based planning
Strong analytical and troubleshooting skills ability to diagnose and resolve pipeline failures methodically
Effective communication and collaboration skills with the ability to work across cross-functional and geographically distributed teams
GOOD TO HAVE
Experience with Databricks Unity Catalog for data governance, lineage tracking, and fine-grained access control
Exposure to cloud secrets management integration (e.g., AWS Secrets Manager, Azure Key Vault, or HashiCorp Vault) within Databricks secret scopes
Background in legacy big data tooling — Hadoop, Hive, or Sqoop — particularly in the context of migration or modernisation projects
Familiarity with advanced Spark optimisation techniques including broadcast joins, adaptive query execution, and partition tuning
Basic understanding of data modelling concepts and dimensional or star schema design
Exposure to DevOps practices for data pipelines, including CI/CD, automated testing frameworks, or infrastructure-as-code
Databricks Certified Associate Developer for Apache Spark, or a relevant cloud data certification (AWS Certified Data Analytics, Microsoft Certified: Azure Data Engineer Associate, or equivalent)
EDUCATION
Bachelor s degree (B.Tech / B.E. or equivalent) in Computer Science, Information Technology, Data Science, or a related technical discipline is strongly preferred
A Master s degree in Computer Science, Data Engineering, or a related field is an advantage but not mandatory
Candidates with degrees in other engineering disciplines who can demonstrate strong, production-grade data engineering experience will also be considered
Databricks Certified Associate Developer for Apache Spark, or a relevant cloud data certification (AWS Certified Data Analytics, Microsoft Certified: Azure Data Engineer Associate, or equivalent) are a distinct advantage
Additional Information:
- The candidate should have minimum 3 years of experience in Data Engineering.
- This position is based at our Mumbai office.
- A 15 years full time education is required.
