Required Skills & Qualifications:
1. Experience: 5+ years in data engineering with significant hands-on experience in Go (Golang).
2. Apache Beam & Dataflow: Strong experience with Apache Beam SDK (Go preferred) and deploying jobs on Google Cloud Dataflow.
3. GCP Services: Proficiency with BigQuery, Pub/Sub, and Cloud Storage.
4. Data Technologies: Experience with real-time streaming (Kafka or Pub/Sub) and strong SQL skills.
5. DevOps & Automation: Familiarity with Docker, Kubernetes, Terraform, or Ansible is preferred.
6. Methodologies: Experience working in Agile or Scrum environments.
Key Responsibilities:
1. Design and deploy data processing pipelines using Apache Beam Go SDK on Google Dataflow.
2. Integrate GCP services including BigQuery, Pub/Sub, Cloud Storage, and BigTable.
3. Develop and optimize batch and real-time streaming pipelines.
4. Monitor and troubleshoot pipelines using Cloud Monitoring (Stackdriver) to ensure cost-effectiveness and reliability.
5. Maintain code quality and implement testing best practices for Beam pipelines.
6.
