We at Innovecture are hiring for a "Senior Data Engineer" to expand our team, this is a Remote position in India. You will work across various Innovecture and client teams and apply your technical expertise to some of the most complex and challenging technology problems.
About Innovecture:
Founded in 2007 under the leadership of CEO Shreyas Kamat, Innovecture LLC, began as a U.S.-based Information Technology and Management Consulting Company focusing on technology consulting and services. With international development centers located in Salt Lake City, USA, and Pune, India, Innovecture leverages its Global Agile Delivery Model to effectively deliver client projects within budget scope and project deadline. The primary focus of Innovecture is to provide a unique wealth of expertise and experience to the IT and Management Consulting realm by utilizing various technologies across multiple industry domains. Innovecture uses best-in-class design processes and top-quality talent to ensure the highest quality deliverables. With innovation embedded in its consulting and services approach, Innovecture will continue to deliver outstanding results for its Fortune 500 clients and employees.
Role and Responsibilities
We are looking for a highly skilled Senior Data Engineer with deep expertise in Databricks and modern data engineering practices to join our Data COE.
Your Experience
As a Senior Data Engineer, you will be responsible for developing Data solutions to address business problems. This role will be responsible for designing, building, and optimizing scalable data pipelines and data platforms that support advanced analytics and business intelligence initiatives. The ideal candidate is a problem-solver with hands-on expertise in Databricks, Spark, Delta Lake, cloud platforms (AWS), and data pipeline orchestration, with a strong focus on performance, reliability, and scalability. This is a hands-on role that requires the candidate to work collaboratively in a squad following a Scaled Agile development methodology. You must be a self-starter, delivery-focused, and possess a broad set of technology skills.
Things you will do:
- Ensure that solution requirements are gathered accurately, understood, and that all stakeholders have transparency on impacts.
- Design, develop, and maintain scalable ETL/ELT pipelines using Databricks (PySpark/Scala/Spark SQL)
- Build and optimize batch and streaming pipelines for large-scale data processing.
- Implement data ingestion frameworks for structured and unstructured data.
- Develop solutions leveraging Databricks Lakehouse architecture (Delta Lake)
- Implement data models, schema design, partitioning, and performance tuning.
- Ensure ACID compliance, data versioning, and time travel capabilities in Delta.
- Work with cloud platforms (preferably AWS – S3, Glue)
- Integrate data pipelines with data warehouses, APIs, and downstream applications.
- Implement CI/CD pipelines for data workflows.
- Enforce data quality, validation, and monitoring frameworks.
- Build reports using Power BI.
- Implement data lineage, cataloguing (Unity Catalog preferred), and governance standards.
- Ensure compliance with security, privacy, and regulatory requirements.
- Optimize Spark jobs for cost and performance.
- Monitor cluster utilization and improve efficiency in Databricks.
- Implement caching, partition pruning, and query optimization techniques.
- Collaborate with data scientists, analysts, and business stakeholders.
- Mentor junior engineers and promote best practices within the team.
- Contribute to the data engineering roadmap, standards, and reusable frameworks.
- Monitoring performance and advising any necessary infrastructure changes
- Selecting and integrating any Big Data tools and frameworks required to provide requested capabilities.
- Champion good agile practices that are foundational to product delivery
- Build strong relationships with product owners, solutions architects, analysts, and pod leadership.
- Work with state-of-the-art technologies to solve genuine, real-world problems.
- Troubleshoot and resolved development issues.
- Provide technical support to clients regarding existing problems.
- Recommend and execute code improvements based on current solutions.
What will you bring
- For the Senior Data Engineer role, we are looking for a candidate with at least 8+ years of extensive experience in Data Engineering.
- Proven record of successfully delivering software with a broad mix of languages, technologies, and platforms.
- Proven experience in Big Data Engineering.
- Experience working with Agile at Scale, Lean and Continuous Delivery approaches such as Continuous Integration, Test-Driven Development and Infrastructure as Code
Technical Skills Required:
Must have skills.
- Strong hands-on experience with Databricks (must-have)
- Expertise in Apache Spark (PySpark/Scala)
- Strong proficiency in SQL and data modelling
- Strong hands-on experience in Power BI
- Experience with Delta Lake and Lakehouse architecture
- Hands-on experience with cloud platforms (AWS preferred)
- Experience with orchestration tools
- Knowledge of streaming frameworks (Kafka)
- Proficiency in Python (mandatory)
- Experience with Git, CI/CD pipelines
