Python Developer/ Web Scraper

HIR INFOTECH -
India

Apply Now

Job details

Qualifications

BCS
CI/CD
Power BI
Web analytics
Azure
Computer Science
Kubernetes
B.Sc
NumPy
NoSQL
Selenium
Git
English
MongoDB
Tableau
Master's degree
Databases
SQL
MCA
Pandas
AWS
Docker
Bachelor's degree
Machine learning
PostgreSQL
Neo4j
Continuous integration
REST
Natural language processing
Data analytics
GitHub
APIs
B.E.
S3
Apache
Kafka
Data science
GraphQL
Flask
AI
Jenkins
Graph databases
Python
Data Science

Full job description

Web Scraping & Data Intelligence

01

Available Positions

2–3 years in web scraping, data analytics, and data-driven insights

Experience

BCA, BE.IT/CS, MCA, BSc.IT/CS or equivalent

Qualification

Development

Department

Key Responsibilities

Web Scraping & Data Acquisition

Design, develop, and maintain robust web scraping pipelines using Python frameworks such as Scrapy, Selenium, Playwright, and Beautiful Soup. Implement headless browser automation and proxy rotation to extract structured and unstructured data at scale.

Data Processing & Analysis

Build ETL/ELT workflows with pandas, NumPy, and Dask for efficient data cleansing, transformation, and normalization. Integrate API-based data sources (REST, GraphQL) and schedule batch or real-time ingestion via Apache Airflow or Prefect.

Insight Generation & Visualization

Leverage Python libraries (Matplotlib, Seaborn, Plotly) and BI tools (Tableau, Power BI) to convert raw data into actionable dashboards. Produce executive-ready reports highlighting key trends, anomalies, and opportunities.

Data Intelligence & Machine Learning

Collaborate with data science teams to deploy simple ML models (scikit-learn, XGBoost) for classification, clustering, and predictive analytics. Implement NLP pipelines (spaCy, NLTK) to extract semantics from text-based web content.

AI-Driven Automation

Explore AI-powered scraping and data enrichment services (Diffbot, Import.io with AI modules) and leverage OpenAI APIs or LangChain for intelligent data parsing, summarization, and anomaly detection.

Performance & Reliability

Ensure scraper resiliency by handling anti-bot measures, CAPTCHAs, and dynamic content. Optimize code for concurrency (Asyncio, multiprocessing) and implement unit tests (pytest) and CI/CD pipelines (GitHub Actions, Jenkins).

Proof of Impact

Demonstrate successful delivery of high-volume scraping projects—processing 5M+ records monthly with 99% data accuracy—and generation of insights that led to a 25% improvement in decision-making speed.

Required Qualifications

Bachelor’s degree in Computer Science, Data Science, or related field with fluency in English communication
Proficiency in Python, web frameworks (Flask, FastAPI), and SQL/noSQL databases (PostgreSQL, MongoDB)
Familiarity with cloud environments (AWS Lambda, EC2, S3; Azure Functions) and containerization (Docker, Kubernetes)
Strong problem-solving skills, attention to detail, and ability to translate complex data challenges into scalable solutions

Preferred: Experience with graph databases (Neo4j), vector databases (Pinecone), and real-time streaming (Kafka, AWS Kinesis).

Apply Now