Job Title: Cloud Platform Engineer
Experience: 7+ Years
Employment Type: Full-Time
Work Mode: Remote
Job Summary
We are seeking a highly skilled Cloud Platform Engineer / Cloud Data Engineer to join our team for a Cloud Data Modernization initiative.
The ideal candidate will have strong hands-on experience with Databricks, Azure Cloud data services, Apache Spark, PySpark, and Snowflake, with a proven background in migrating and modernizing on-premises ETL workloads to cloud-based data platforms.
Databricks is the primary data technology, with Snowflake as a secondary data platform. Candidates should also have experience with Azure and/or AWS cloud infrastructure and data services.
The role will involve designing and implementing modern cloud data solutions, developing scalable data pipelines, supporting ETL migrations, implementing DevOps and CI/CD practices, and ensuring data quality, security, observability, and performance.
Key Responsibilities
- Contribute to the migration and modernization of on-premises ETL workloads to Azure Cloud, Databricks, and Snowflake.
- Design, develop, and maintain scalable data engineering solutions using Databricks, Apache Spark, PySpark, and Python.
- Develop data pipelines using Azure Data Factory (ADF), Self-hosted Integration Runtime (SHIR), Logic Apps, ADLS Gen2, Blob Storage, and Snowflake.
- Review, analyze, and document existing on-premises ETL processes and data workflows.
- Translate legacy ETL processes into scalable and maintainable cloud-native solutions.
- Develop and optimize data ingestion, transformation, integration, and processing pipelines.
- Implement DevOps practices and CI/CD pipelines using GitHub Actions.
- Follow Git branching strategies, pull requests, code reviews, and software engineering best practices.
- Collaborate with architects, data engineers, application teams, QA, business stakeholders, and other cross-functional teams.
- Tune and optimize Databricks, Spark, PySpark, and Snowflake workloads for performance and scalability.
- Implement data quality validation, monitoring, reconciliation, and observability processes.
- Support automated data validation and reconciliation frameworks.
- Implement appropriate logging, monitoring, alerting, and operational processes for data pipelines.
- Ensure data solutions comply with organizational security, privacy, governance, and regulatory standards.
- Participate in Agile ceremonies, sprint planning, technical discussions, and delivery activities.
- Troubleshoot production issues and proactively identify performance, reliability, and data-quality problems.
- Leverage AI-assisted development tools to improve engineering productivity, code quality, testing, and documentation.
- Work independently, manage ambiguity, prioritize tasks, and deliver high-quality solutions within established timelines.
Primary Skills
- Databricks
- Azure Data Engineering
- Azure Data Factory (ADF)
- Self-hosted Integration Runtime (SHIR)
- Azure Data Lake Storage Gen2 (ADLS Gen2)
- Azure Blob Storage
- Logic Apps
- Apache Spark
- PySpark
- Python
- Snowflake
- ETL / ELT Development
- Cloud Data Migration
- Data Quality & Validation
- GitHub Actions
- CI/CD
Secondary Skills
- Delta Lake
- Lakehouse Architecture
- Medallion Architecture – Bronze / Silver / Gold
- Snowflake Data Engineering
- Data Modeling
- Database Design
- Airflow
- Data Governance
- Data Observability
- Performance Optimization
- AWS Cloud
- Azure Cloud
- AI/ML for Data Engineering
- AI-Assisted Development Tools
- Healthcare Payer Data
Required Qualifications
- Bachelor's degree in Computer Science, Engineering, Information Technology, or a related field.
- Minimum 3+ years of experience as a Cloud Data Engineer, Cloud Platform Engineer, or similar role.
- Strong hands-on experience with Databricks in enterprise data engineering environments.
- Experience with Azure data services, particularly ADF, ADLS Gen2, Blob Storage, Logic Apps, and SHIR.
- Strong experience with Apache Spark and PySpark.
- Hands-on experience with Snowflake for data engineering or analytics workloads.
- Experience developing and supporting data migration and cloud modernization initiatives.
- Strong experience with ETL development using on-premises databases and ETL technologies.
- Strong SQL development skills, including:
- Complex queries
- Stored procedures
- Views
- Joins
- Data transformations
- Performance optimization
- Experience with Python or other scripting languages for data processing and automation.
- Hands-on experience with GitHub, including branching, pull requests, and code reviews.
- Experience implementing GitHub Actions and CI/CD pipelines.
- Experience supporting automated testing and data quality validation processes.
- Strong understanding of Agile/Scrum delivery methodologies.
- Ability to learn new technologies quickly and adapt to changing project requirements.
- Strong analytical, troubleshooting, communication, and collaboration skills.
- Ability to work independently and effectively manage ambiguity and competing priorities.
- Ability to work under tight deadlines and proactively escalate critical technical or delivery issues.
- Willingness to travel when required based on business needs and applicable travel conditions.
Preferred Qualifications
- Experience with Azure and/or AWS cloud platforms.
- Experience implementing Bronze/Silver/Gold (Medallion) architecture.
- Strong understanding of Delta Lake and Lakehouse architecture.
- Experience with Delta Lake optimization techniques.
- Experience with Databricks performance tuning and optimization.
- Experience with Snowflake performance optimization.
- Knowledge of enterprise data governance and data quality best practices.
- Experience with data modeling and database architecture.
- Experience with Airflow or other data orchestration frameworks.
- Experience developing Snowflake-based data engineering and analytics solutions.
- Experience leveraging AI/ML solutions for data engineering automation and workflow optimization.
- Familiarity with AI-assisted development tools such as GitHub Copilot, Databricks Assistant, ChatGPT, or Claude.
- Experience working with healthcare payer data, including:
- Claims
- Membership
- Enrollment
- Provider
- Clinical
- Financial data
- Azure or Databricks certification is a plus.
DevOps & Engineering Practices
- GitHub
- GitHub Actions
- CI/CD
- Git branching and pull requests
- Code reviews
- Automated testing
- Infrastructure and deployment automation
- Monitoring and observability
- Agile/Scrum
Key Competencies
- Cloud Data Engineering
- Databricks Engineering
- ETL Modernization
- Cloud Migration
- Azure Data Platform
- Spark & PySpark
- Snowflake
- Data Pipeline Development
- Data Quality & Reconciliation
- DevOps & CI/CD
- Performance Optimization
- Data Governance
- Problem Solving
- Technical Collaboration
Work Location: Remote
