Role Summary & Role Description:
Technical Lead with specific Oracle, PLSQL and design, develop, and optimize data workflows on the Databricks platform. The ideal candidate will have deep expertise in Apache Spark, Pyspark, Python, job orchestration, and CI/CD integration to support scalable data engineering and analytics solutions. Analyzes, designs, develops and maintains software applications to support business units.
Expected to spend 100 % of the time on hands-on development, design and architecture. Capital Markets Projects experience preferred Provides advanced technical expertise in analyzing, designing, estimating, and developing software applications to project schedule. Oversees systems design and implementation of most complex design components. Creates project plans and deliverables and monitors task deadlines. Oversees, maintains and supports existing software applications. Provides subject matter expertise in reviewing, analyzing, and resolving complex issues. Designs and executes end to end system tests of new installations and/or software prior to release to minimize failures and impact to business and end users. Responsible for resolution, communication, and escalation of critical technical issues. Prepares user and systems documentation as needed. Identifies and recommends Industry best practices. Serves as a mentor to junior staff.
-
Acts as a technical lead/mentor for developers in day to day and overall project areas.
-
Ability to lead a team of agile developers.
-
Worked in a complex deadline driven projects with minimal supervision. Ability to architect/design/develop with minimum requirements by effectively coordinating activities between business analysts, scrum leads, developers and managers.
-
Ability to provide agile status notes on day to day project tasks.
Key Responsibilities:
-
Data Lakehouse Development:
-
Design and implement scalable data Lakehouse solutions using Databricks and Delta Lake.
-
Develop and maintain Delta Lake tables with ACID transactions and schema evolution.
-
Data Ingestion & Autoloaders:
-
Build ingestion pipelines using Databricks Autoloader for real-time and batch data.
-
Integrate data from various sources including cloud storage (e.g., ADLS, S3), databases, APIs, and streaming platforms.
-
ETL & Data Enrichment:
-
Develop ETL workflows to cleanse, transform, and enrich raw data.
-
Implement business logic and data quality checks to ensure reliability and accuracy.
-
Performance Optimization:
-
Optimize Spark jobs for performance and cost-efficiency.
-
Monitor and troubleshoot data pipelines using Databricks tools and logging frameworks.
-
Data Access & Governance:
-
Enable secure and governed access to data using Unity Catalog or other access control mechanisms.
-
Collaborate with data analysts and scientists to ensure data availability and usability.
-
Collaboration & Documentation:
-
Work closely with cross-functional teams including data architects, analysts, and business stakeholders.
-
Document data models, pipeline designs, and operational procedures.
Technical Skills:
-
Design and implement robust ETL pipelines using Databricks notebooks and workflows.
-
Strong proficiency in Apache Spark, PySpark, and Databricks.
-
Hands-on experience with Delta Lake, Autoloader, and Structured Streaming.
-
Proficiency in SQL, Python, and cloud platforms (Azure, AWS, or GCP).
-
Experience with ETL frameworks, data modeling, and data warehousing concepts.
-
Familiarity with CI/CD, Git, and DevOps practices in data engineering.
-
Knowledge of data governance, security, and compliance standards.
-
Optimize Spark jobs for performance and cost-efficiency.
-
Develop and manage job orchestration strategies using Databricks Jobs and Workflows.
-
Monitor and troubleshoot production jobs, ensuring reliability and data quality.
-
Implement security and governance best practices including access control and encryption.
-
Strong Practical experience using Scrum, Agile modelling and adaptive software development.
-
Ability to understand and grasp the big picture of system components.
-
Experience building environment and architecture and design guides and architecture and application blueprints.
-
Strong understanding of data modeling, warehousing, and performance tuning.
-
Excellent problem-solving and communication skills.
Core/Must have skills:
Python, Scala, Spark, Databricks, Oracle, SQL, PLSQL, Python, Scala, Apache Spark, Spark Streaming, CI CD pipeline,
AWS cloud experience
Exposure to real-time data processing and event-driven architectures.
Work Schedule: 12 PM IST to 9 PM (IST)
Why this role is important to us:
Our technology function, Global Technology Services (GTS), is vital to State Street and is the key enabler for our business to deliver data and insights to our clients. We’re driving the company’s digital transformation and expanding business capabilities using industry best practices and advanced technologies such as cloud, artificial intelligence and robotics process automation.
We offer a collaborative environment where technology skills and innovation are valued in a global organization. We’re looking for top technical talent to join our team and deliver creative technology solutions that help us become an end-to-end, next-generation financial services company.
Join us if you want to grow your technical skills, solve real problems and make your mark on our industry.
Across the globe, institutional investors rely on us to help them manage risk, respond to challenges, and drive performance and profitability. We keep our clients at the heart of everything we do, and smart, engaged employees are essential to our continued success.
We are committed to fostering an environment where every employee feels valued and empowered to reach their full potential. As an essential partner in our shared success, you’ll benefit from inclusive development opportunities, flexible work-life support, paid volunteer days, and vibrant employee networks that keep you connected to what matters most. Join us in shaping the future.
As an Equal Opportunity Employer, we consider all qualified applicants for all positions without regard to race, creed, color, religion, national origin, ancestry, ethnicity, age, disability, genetic information, sex, sexual orientation, gender identity or expression, citizenship, marital status, domestic partnership or civil union status, familial status, military and veteran status, and other characteristics protected by applicable law.
Discover more information on jobs at StateStreet.com/careers
Read our CEO Statement
