Data Engineer (PySpark, SQL) – Databricks Certified
Experience: 8+ Years
Work Mode: Remote
Employment Type: Contract / Full-Time
Job Summary:
We are looking for an experienced Data Engineer with strong expertise in PySpark, SQL, and Databricks to design, build, and optimize scalable data pipelines and modern data platforms. The ideal candidate should have hands-on experience working with large-scale datasets, distributed data processing, cloud-based analytics platforms, and data warehousing solutions.
Databricks Certification is Mandatory.
Key Responsibilities:
- Design, develop, and maintain scalable data pipelines using PySpark and SQL.
- Build and optimize ETL/ELT workflows for large-scale data processing.
- Develop and manage data solutions on Databricks platforms.
- Work with structured and unstructured datasets to support analytics and business intelligence initiatives.
- Implement data quality checks, validation processes, and monitoring mechanisms.
- Optimize data processing jobs for performance, scalability, and cost efficiency.
- Collaborate with Data Architects, Data Analysts, and business stakeholders to understand requirements.
- Develop reusable data engineering frameworks and best practices.
- Support data migration, transformation, and integration projects.
- Ensure data security, governance, and compliance standards are followed.
Required Skills:
- 8+ years of experience in Data Engineering.
- Strong expertise in:
- PySpark
- SQL
- Databricks
- Hands-on experience with data pipeline development and optimization.
- Strong understanding of Data Warehousing concepts and dimensional modeling.
- Experience working with large-scale distributed data processing environments.
- Proficiency in performance tuning and query optimization.
- Experience with cloud platforms such as Azure, AWS, or GCP.
- Strong understanding of ETL/ELT frameworks and data integration patterns.
- Experience with Git, CI/CD pipelines, and Agile methodologies.
Mandatory Requirement:
- Databricks Certified Professional/Associate Certification
Preferred Skills:
- Experience with Delta Lake and Lakehouse Architecture.
- Knowledge of Azure Data Factory, Snowflake, or Microsoft Fabric.
- Exposure to streaming technologies such as Kafka or Event Hub.
- Experience with Power BI, Tableau, or other reporting platforms.
- Knowledge of DevOps and Infrastructure as Code practices.
Key Competencies:
- Strong analytical and problem-solving skills
- Excellent communication and stakeholder management
- Ability to work independently in a remote environment
- Ownership mindset and attention to detail
Pay: ₹80,000.00 - ₹100,000.00 per month
Work Location: Remote
