-
Design, develop, and maintain scalable data pipelines using Databricks and PySpark
-
Build robust ETL/ELT workflows for batch and near real‑time data processing
-
Write optimized and efficient SQL queries for data transformation, validation, and analysis
-
Work with large-scale distributed datasets using Spark-based processing frameworks
-
Perform data ingestion from multiple sources (APIs, files, databases) into data lake/lakehouse architecture
-
Ensure data quality, integrity, and consistency through validations and monitoring
-
Optimize pipeline performance and cost (partitioning, caching, query tuning, etc.)
-
Collaborate with cross-functional teams (Data Ops, Business, Analytics) to deliver data solutions
-
Troubleshoot pipeline failures and implement proactive monitoring and alerting
-
Maintain proper documentation for data models, transformations, and workflows
-
Strong hands-on experience in:
-
Databricks (Lakehouse platform, notebooks, jobs, clusters)
-
PySpark / Apache Spark
-
Advanced SQL (joins, window functions, performance tuning)
-
Solid understanding of ETL concepts, data warehousing, and data modeling
-
Experience building scalable data pipelines for structured and semi-structured data
-
Familiarity with data lake / delta lake architecture
-
Good understanding of performance optimization techniques in Spark and SQL
-
Experience with version control (Git) and deployment practices
-
Strong problem-solving and analytical skills
-
Experience with cloud platforms (AWS preferably)
-
Exposure to CI/CD pipelines (Jenkins)
-
Knowledge of Delta Lake, Unity Catalog, or similar governance tools
-
Understanding of data quality frameworks / monitoring tools
-
Experience working in Agile / Scrum teams
This position can be based in any of the following locations:
Chennai
Current Guardian Colleagues: Please apply through the internal Jobs Hub in Workday
