Job Description: • Databricks, PySpark and Python project development work experience is a must • Develop, design, tune and maintain PySpark scripts using Databricks Notebook. • Expertise in Bigdata Eco systems like HDFS and Spark • Experience in GitHuB repository • Data Warehouse/Data Marts/Data Modelling/Analytics experience is a must • Able to convert the SQL stored procedures to Python code in Pyspark frame work using Dataframes. • Implementing data ingestion pipelines from multiple data sources using Azure Data Factory, Azure Databricks and other ETL tools. • Developing Big Data and non-Big Data cloud-based enterprise solutions in PySpark and SparkSQL and related frameworks/libraries. • Developing scalable and re-usable, self-service frameworks for data ingestion and processing. • Integrating end to end data pipelines to take data from data source to target data repositories ensuring the quality and consistency of data. • Processing performance analysis and optimization. • Collaborate with business users, support team members, and other developers throughout the organization to help everyone understand issues that affect the data warehouse • Good experience on customer interaction is required. • Possesses good interpersonal and communication skills.
