Job Title: Databricks Engineer (7+ Years Experience) — MetaDesignSolutions
Location: Gurugram, Haryana, India (On-site) — Employment Type: Full-Time
MetaDesignSolutions is a technology consulting and digital transformation firm focused on delivering cloud-native data engineering, analytics, and AI-enabled solutions to enterprise clients across industries. We partner with customers to modernize data platforms, accelerate time-to-insight, and enable governed, scalable data architectures. The Databricks Engineer will be a senior contributor responsible for designing, implementing, and optimizing large-scale data solutions on Databricks and associated cloud services to support the company’s analytics and cloud migration engagements.
- Design, develop, and maintain scalable, resilient data pipelines and ETL/ELT workflows using Databricks and Apache Spark to ingest, transform, and curate large volumes of structured and semi-structured data.
- Migrate legacy SSIS packages and SQL-based ETL processes to cloud-native Databricks and managed data services (Azure/AWS), ensuring functional parity, performance improvements, and maintainability.
- Develop and maintain Databricks notebooks, jobs, Delta Lake tables, and orchestration workflows to support data lakes, data warehouse ingestion, and downstream analytics.
- Integrate data from multiple sources including relational databases, cloud object stores, REST APIs, message queues, and flat files; implement robust error handling and retry logic.
- Perform performance tuning and cost optimization of Spark jobs, cluster configurations, and SQL queries; implement partitioning, caching, and resource controls for efficient processing.
- Implement data quality frameworks, monitoring, alerting, and observability for pipelines; establish automated validation, reconciliation, and lineage tracking in collaboration with data governance teams.
- Work independently on end-to-end delivery of data engineering projects, collaborate with architects, business analysts, and stakeholders to translate requirements into pragmatic technical solutions, and participate in design reviews.
- Participate in release management, CI/CD for data artifacts, and provide production support, troubleshooting, and operational runbooks to ensure high availability and SLAs.
- Minimum 7 years of experience in Data Engineering, ETL development, or related roles with proven delivery on enterprise data programs.
- Strong hands-on experience with Databricks and Apache Spark; demonstrated ability to build production-grade Spark jobs and optimize job performance.
- Proficiency in SQL and Python (PySpark) for data transformation, analysis, and scripting.
- Experience with SSIS packages and proven experience migrating SSIS/legacy ETL to cloud-native platforms.
- Solid understanding of data warehousing concepts, dimensional modeling, and best practices for data lakes and lakehouse architectures.
- Hands-on experience with cloud data services such as Azure Data Factory, Azure Databricks, AWS Glue, S3/ADLS, and related security/configuration patterns.
- Familiarity with Delta Lake, Unity Catalog, ACID transactions, and principles of data governance and access control.
- Demonstrated skills in performance tuning Spark jobs, cluster sizing, and query optimization; experience with monitoring and profiling tools.
- Bachelor’s or Master’s degree in Computer Science, Information Technology, Engineering, or a related field.
- Strong analytical, problem-solving abilities and the capacity to work independently with minimal supervision while communicating effectively with technical and non-technical stakeholders.
- Experience implementing CI/CD pipelines for data engineering artifacts using tools such as Azure DevOps, GitHub Actions, Jenkins, or Bitbucket Pipelines.
- Exposure to streaming and real-time processing frameworks such as Kafka, Structured Streaming, or Kinesis.
- Familiarity with BI tools such as Power BI or Tableau and an understanding of downstream reporting/consumption patterns.
- Certifications such as Databricks Certified Associate/Professional, Microsoft Azure Data Engineer, or AWS Big Data/Analytics certifications.
- Experience with data cataloging, metadata management, and data lineage tools.
- Previous consulting or client-facing experience delivering cloud migration and data modernization engagements.
- Competitive salary commensurate with experience and market standards; comprehensive compensation package.
- Full-time, on-site role based in Gurugram with opportunities to work on strategic, high-impact data modernization projects across industries.
- Health insurance and employee benefits aligned with company policy; paid time off and statutory benefits.
- Professional development support including training, certification sponsorship, and mentorship to advance technical and leadership skills.
- Exposure to modern data architectures, large-scale datasets, and best-in-class tools and platforms to accelerate career growth.
- Collaborative and inclusive culture with emphasis on knowledge sharing, technical excellence, and client-focused delivery.
- Opportunities to contribute to architecture design, influence technical direction, and mentor junior engineers.
