The Associate Solution Architect will be responsible for providing technical leadership, architecture, governance, and oversight of our next generation of data processing software using a modern cloud-native technology stack with Python on AWS cloud infrastructure. This is a rare opportunity to make a big impact on both the team and the organization by being part of the design and development of application frameworks. The ideal candidate has a passion for solving business problems with technology and can effectively communicate business and technical needs to stakeholders.
Please send in your resumes to [email protected]
Key Responsibilities:
- Requirement Analysis: Collaborate with stakeholders to understand business requirements and data sources, and define the architecture and design of data pipelines to meet these requirements.
- Architecture Design: Design scalable, reliable, and efficient data pipelines to ingest, process, transform, and store large volumes of data from diverse sources, ensuring data quality and consistency.
- Technology Selection: Evaluate using POCs and recommend appropriate technologies, frameworks, and tools for building and managing data pipelines, considering factors like performance, scalability, and cost-effectiveness.
- Data Processing: Develop and implement data processing logic, including data cleansing, transformation, and aggregation, using technologies like AWS Glue, Batch, Lambda
- Data Storage: Design and implement data storage solutions, including data lakes, data warehouses, and NoSQL databases, to store and manage data for analysis and reporting.
- Data Integration: Integrate data pipelines with external systems, APIs, and services to automate data workflows and ensure seamless data exchange between systems.
- Monitoring and Logging: Implement monitoring and logging solutions for data pipelines using tools like AWS CloudWatch to ensure data pipeline health and performance.
- Security and Compliance: Ensure data pipeline architecture complies with security best practices and regulatory requirements, implementing encryption, access controls, and data masking as needed.
- Documentation: Create and maintain documentation for data pipeline architecture, design, and implementation, including diagrams, data flow descriptions, and operational procedures.
- Collaboration: Collaborate with cross-functional teams, including data engineers, data scientists, and business analysts, to understand their requirements and integrate data pipelines into their workflows.
- Continuous Improvement: Stay updated with the latest trends, tools, and technologies in data engineering and data pipeline architecture, and continuously improve data pipeline processes and methodologies.
Desired Skills & Requirement
> Data Engineering: Understanding of data engineering concepts and principles, including data ingestion, processing, transformation, and storage, using tools and technologies such as AWS Glue, Batch, Lambda.
> Data Modeling: Knowledge of data modeling techniques and best practices for designing efficient and scalable data pipelines, including dimensional modeling, star schemas, and snowflake schemas.
> Programming Languages: Proficiency in programming languages commonly used in data engineering, such as Python, Java, or Scala, as well as experience with SQL for data querying and manipulation.
> Cloud Computing: Proficiency in using AWS cloud services for data pipeline architecture and implementation.
> Data Integration: Understanding of data integration techniques and tools for integrating data from various sources, including batch and real-time data integration, and experience with ETL (Extract, Transform, Load) processes.
> Database Systems: Knowledge of database systems, including relational databases (e.g., MySQL, PostgreSQL) and NoSQL databases (e.g., MongoDB, Cassandra), and experience in designing and managing database schemas.
> Data Governance: Awareness of data governance principles and practices, including data quality, data lineage, and data privacy, and ability to ensure compliance with relevant regulations and standards.
> Problem-solving Skills: Excellent problem-solving skills, with the ability to analyze complex data pipeline requirements and design solutions that meet business needs.
> Communication and Collaboration: Strong communication and collaboration skills, with the ability to effectively communicate technical concepts to non-technical stakeholders and work effectively in a team environment.
> Learning Agility: A commitment to continuous learning and staying updated with the latest trends, tools, and technologies in data engineering and data pipeline architecture.
Academic Qualification and Industry Experience
> Bachelor’s Degree: A bachelor’s degree in Computer Science, Software Engineering, or a related field is often preferred, although equivalent experience and certifications can also be valuable. Experience with Financial domain knowledge is a plus.
> At least 7 years of experience in solutions architecture, software development, or any other related field.
