Job Title: Senior Databricks Data Engineer / Data Architect
Experience: 8–12+ Years
Location: Remote
Employment Type: C2C
Job Summary
We are seeking an experienced Databricks Data Engineer / Data Architect to design, develop, and optimize enterprise-scale data platforms and analytics solutions. The ideal candidate will have strong expertise in Databricks, PySpark, Apache Spark, SQL, and cloud technologies (Azure, AWS, or GCP). The role involves building scalable data pipelines, implementing modern data architectures, and delivering high-performance data solutions.
Key Responsibilities
- Design, develop, and deploy end-to-end data engineering solutions using Databricks.
- Build and optimize large-scale ETL/ELT pipelines using PySpark, Spark, and SQL.
- Develop and maintain data models, data lakes, and modern Lakehouse architectures.
- Collaborate with business and technical stakeholders to translate requirements into scalable data solutions.
- Implement data ingestion, transformation, and processing workflows from multiple data sources.
- Build APIs and data services supporting analytics, reporting, and application integration.
- Ensure data quality, governance, security, and compliance standards.
- Optimize performance, scalability, and reliability of data platforms.
- Support cloud migration and modernization initiatives.
- Work with open-source technologies including Apache Spark, Delta Lake, and MLflow.
- Contribute to architecture discussions, technical strategy, and best practices.
Required Skills
- 8–12+ years of Data Engineering experience.
- 5+ years of hands-on Databricks experience.
- Strong expertise in Python, PySpark, SQL, and Apache Spark.
- Deep understanding of Spark Core, Spark SQL, DataFrames, Spark Streaming, and distributed computing.
- Experience with Delta Lake, Unity Catalog, and Databricks Workflows.
- Strong knowledge of Data Lakes, Data Warehousing, ETL/ELT, and Data Modeling.
- Experience working with structured and unstructured data formats such as JSON, Parquet, and Avro.
- Hands-on experience with Azure Databricks, AWS, or GCP.
- Experience implementing cloud-native data processing and migration solutions.
- Familiarity with CI/CD, Git, Azure DevOps, and DevOps practices.
Preferred Skills
- Experience with Kafka and real-time streaming solutions.
- Exposure to MLflow and machine learning data pipelines.
- Manufacturing, Energy, or similar industry experience.
- Databricks, AWS, Azure, or GCP certifications.
- Experience presenting technical solutions and architecture to stakeholders.
Qualifications
- Bachelor's or Master's degree in Computer Science, Engineering, Applied Mathematics, or a related field.
- Excellent communication, analytical, and problem-solving skills.
Experience:
- Data Engineer: 10 years (Preferred)
- Databricks: 5 years (Preferred)
- AWS/Azure/GCP: 5 years (Preferred)
Work Location: Remote
