JOB DESCRIPTION
Role TitleData Engineer
Employee Interviewed
Reports ToChetan Pant
Company DMI
Function/ DepartmentConsumer Retail / Data Engineering
1. JOB PURPOSE
(Summarize in one statement why the job exists; and how it contributes to the overall objective of the company)
This role is integral to support the Strategic data initiatives related to DMI Consumer Retail LOB .
This role aids development of basic building blocks / modules / Components to support the Enterprise-wide Data Engineering/BI Initiatives.
2.PRINCIPAL ACCOUNTABILITIES
(Accountabilities associated with the job)
Design and Development of Data Engineering Code Modules:
Design And Develop Data Engineering Code Modules and Data Pipelines keeping in view Modularity and Reusability on Cloud Data Platforms like Snowflake and AWS.
Ability to write Python Constructs.
Extract, Transform and Load data from Enterprise Source Systems like Salesforce, DBMS, Flat Files.
Data Analysis and Reporting:
Write complex SQL queries to extract, transform, and load (ETL) data efficiently.
Work with big data technologies such as Apache Spark(Py Spark), Hadoop, EMR, Snowflake, AWS Glue.
Automate data workflows and implement data validation processes:
Design Workflow Orchestration based on the load order plans(using , understanding upstream and downstream dependencies).
Data Modeling:
Ability to model and design database objects based on Entity Relationship Modeling and Dimensional Data Modeling for Data Warehousing.
Data Governance, Integrity And Security:
Designs and Develops constructs adhering to Data Governance, Data Integrity and Security.
3.MAJOR CHALLENGES
(Challenges faced on an on-going basis in carrying out the job)
Quick Time to Resolution of Data Pipelines Debugging and rendering a complete RCA for the same.
Willingness to take on innovative tasks, with keen attention to details.
Ensuring Data Quality, Governance and Integrity to the maximum.
4. DECISIONS
(Key decisions which the role needs to escalate to superior)
Keep the leads apprised of any requests for code enhancements, bug fixing and technical issues.
5. DIMENSIONS
(Key numerical data which will reflect the scope and scale of activities concerning this job)
NA
Financial Dimensions (FY 20)
(These should be quantifiable numerical amounts)
NA
Other Dimensions (FY 19)
(Significant volume dimensions associated with the job)
Total Team Size:
Number of Direct Reports: NA
Number of Outsourced employees: NA
< Other volume dimensions e.g. No. of plants overseen / No. of vendors managed / No. of clients etc. >
7. SKILLS AND KNOWLEDGE
a. Programming And Scripting:
Excellent Skills on SQL/PL SQL (Writing Stored Procedures, Triggers , UDFs)
Worked on SQL Constructs like Joins, CTEs, Window functions, Indexing.
b. Strong hands on experience on RDBMS including but not limited to MySQL, PostgreSQL.
c. Data Processing Frameworks: PySpark for ETL Processing, Apache Airflow for Orchestrating Workflows, Strong hands on experience building ETL Pipelines on any Cloud Platform (Preferable AWS)
d. Databases and Data Warehousing:
Hands on experience on SQL Dbs. (PostgreSQL, MySQL), Good to have exposure/understanding of No SQL Dbs. (MongoDB/Document DB)
Hands on experience on designing and developing code for building Data Warehouses on Redshift/Snowflake.
e. Cloud Platforms:
AWS (S3, Glue, Athena, Redshift, Lambda, EMR)
f. Data Pipelines & ETL Tools:
Apache Airflow (Scheduling & workflow automation)
AWS Glue (ETL jobs in AWS)
g. Data Modeling & Optimization:
Star & Snowflake Schema
Partitioning & Clustering
Indexing & Query Optimization
h. CI/CD & DevOps for Data Engineering:
Git/GitHub/GitLab (Version control)- Good to have
Educational Qualifications
a)Qualifications
Minimum Qualification required: Bachelors of Technology (BTech), Masters of Technology (MTech )
Qualifications preferred: BTech / MTech in Computer Science, Information Technology, Electronics & Communications.
b)Work Experience
Minimum 1-3 years of experience in the Data Engineering Domain, with excellent programming skills in SQL PL SQL and Python.
Good team player, having an open mindset to accept new challenges and innovate.
Quick Learner, Responsive with keen attention to details.
Information Security expectation: (Please select one of them)
oUser Level
