- Assist the team with development and operation of cloud-based data platforms and pipelines along with ETL/ELT functions, cloud database migrations and management, support downstream analytics and BI use cases, as those skills are established and team needs identified.
- Assist the team on call representative and address issues as needed.
- Proactively communicate with business users, stakeholders, technical peers, and project teams.
- Learn and apply industry trends, concepts, and techniques to meet organizational requirements
- Design and implement real-time streaming pipelines using Flink or ksqlDB, backed by Kafka/Confluent Cloud.
- Develop integration/services using C#/.NET Core (REST APIs, JSON, Azure Functions, console apps, services, etc.).
- Build scalable ETL/ELT pipelines on modern tooling (Databricks, Snowflake, Fabric, etc.)
- Orchestrate batch pipelines with Databricks Jobs/Workflows, SSIS, Azure Data Factory, or equivalent tools.
- Partner with stakeholders to translate requirements into robust data solutions; provide documentation and guidance.
- Participate in agile ceremonies and contribute to continuous improvement.
- Experience with Delta Lake, Apache Iceberg, or Parquet as part of modern Lakehouse architecture.
- Experience in T-SQL programming and general knowledge of SQL Server.
- Familiarity with AI-assisted development tools and their application in data engineering workflows.
- Experience in Source Control (GIT, Evolve/Flyway), desired.
- Experience with Azure cloud services (Data Factory, Key Vault, Storage Accounts, Azure DevOps, Event Hubs, Functions, Container Apps), desired.
- Experience with SSIS or Azure Data Factory for legacy ETL systems, desired.
- Perform reviews of development team data transformations, models and queries.
- Define and manage topics, partitions, schemas, retention, and consumer group strategies, leveraging Schema Registry and data contracts.
- Integrate data and automation systems using C#/.NET Core, including REST APIs, JSON processing, Azure Functions, console apps, and services.
- Contribute to catalogs, lineage, and metadata (e.g., Unity Catalog / Fabric governance / Snowflake governance).
- Implement robust data modeling (e.g., Medallion/Lakehouse patterns) for analytics and operational use cases.
- Optimize jobs for cost, reliability, and performance.
- Design BigQuery data objects including tables, views, scheduled queries, etc.
- Work with Azure DevOps Pipelines to assist in the teams reviewing and promoting of work.
- Implement backup plans, develop and test disaster recovery scenarios, and manage security.
- Build/operate Kafka Connect source/sink connectors where applicable.
- Work with Operations and DevOps teams to ensure a healthy and stable environment.
- Design, implement, and manage cloud infrastructure using Terraform, ensuring reusable, version‑controlled, and secure IaC modules.
- Implement configuration, secrets, and key management (Key Vault), and automate routine operations using C#/.NET Core and/or PowerShell.
- On call duties, as needed.
This is a summary of the primary accountabilities and requirements for this position. The company reserves the right to modify or amend accountabilities and requirements at any time at its sole discretion based on business needs. Any part of this job description is subject to possible modification to reasonably accommodate individuals with disabilities.
