Experience ( Relevant) : 10+ Years
Shift Timings : 2pm to 11pm IST
Hybrid : 3 days a week
Start date : ASAP
Top 3 skills which is mandatory :
Please focus on the Databricks AI/ML skills and we can be flexible on the cost for the right resource.
" Target individuals with Databricks Machine Learning or Databricks Generative AI Associate certification "
Position Overview
NTT DATA Americas is seeking a highly skilled Senior Data Engineer to support a Teradata Utilization Analysis engagement, which focuses on analyzing Teradata platform utilization to identify cost-reduction opportunities and deliver a repeatable, log-driven analysis solution.
The ideal candidate combines deep Teradata DBA expertise with hands-on Databricks engineering capability and applied AI/ML skills. The analysis will be performed primarily in Databricks, leveraging DBQL logs, system metadata, and external scheduling/orchestration data sources.
Key Responsibilities
Ingest and validate 18+ months of Teradata DBQL logs including SQL text, object usage, timestamps, user/application IDs, row counts, and steps.
Integrate metadata from Autosys (scheduling), DataStage (orchestration), and MagicWand (observability) to supplement DBQL analysis.
Build end-to-end, log-driven analysis pipelines in Databricks to identify unused datasets, read-only (non-updating) datasets, and unused partitions within active datasets.
Capture and analyze CPU/IO resource usage and workload statistics (ResUsage) to quantify cost-reduction opportunities.
Classify data into cold, warm, and hot tiers; generate heatmaps of date/partition access patterns.
Develop a prioritized recommendation backlog with expected savings, risk levels, and required changes.
Apply AI/ML models or LLM-assisted analysis to detect access pattern anomalies, predict cold data candidates, and automate classification.
Produce and present deliverables: Observation Report, Workshop Notes & Action Log, and Final Readout for customer stakeholders.
Required Skills & Experience
Teradata DBA Skills
Skill / Technology
Min. Experience
Relevance to Work Order
Teradata DBA / Administration
7+ years
Primary platform under analysis; deep knowledge of system internals required
Teradata DBQL (Database Query Logging)
5+ years
Core data source — SQL text, object usage, timestamps, user/app IDs, row counts, steps
Teradata System Views & Space Metadata
5+ years
Object inventory, space analysis, last read/write tracking
Teradata SQL & Performance Tuning
7+ years
Writing complex analytical queries across DBQL and system catalog tables
Teradata ResUsage / Workload Statistics
3+ years
Quantifying CPU/IO cost impact for dataset removal/archival recommendations
Data Classification (hot/warm/cold)
3+ years
Categorizing datasets for tiered storage management recommendations
ETL Pipeline Assessment (DataStage, Autosys)
4+ years
Identifying and recommending decommission of stale ingestion pipelines
Teradata BTEQ / FastExport / TPT
3+ years
Data extraction, log export, and metadata collection utilities
Databricks Engineering Skills
Skill / Technology
Min. Experience
Relevance to Work Order
Databricks Platform (Workspaces, Clusters, Jobs)
4+ years
Core analysis and solution delivery platform for the engagement
Apache Spark (PySpark / Spark SQL)
5+ years
Large-scale log ingestion, transformation, and analytical processing
Delta Lake
3+ years
Building reliable, ACID-compliant data pipelines for log-driven analysis
Databricks Notebooks & Workflows
3+ years
Developing repeatable, shareable analytical notebooks for customer handoff
Python (pandas, numpy, matplotlib)
5+ years
Data wrangling, statistical analysis, and visualization of usage heatmaps
SQL (Spark SQL / ANSI SQL)
7+ years
Analytical querying across large DBQL datasets in Databricks
Unity Catalog / Data Governance in Databricks
2+ years
Organizing deliverable assets and maintaining data lineage within the solution
Dashboarding (Databricks SQL / Power BI)
3+ years
Usage heatmaps, partition access patterns, and recommendation visualizations
AI / ML Skills
Skill / Technology
Min. Experience
Relevance to Work Order
Machine Learning (scikit-learn, MLflow)
3+ years
Anomaly detection on access patterns, predictive cold-data classification
LLM / Generative AI Integration
2+ years
LLM-assisted log interpretation, automated recommendation narrative generation
Feature Engineering on Time-Series Log Data
3+ years
Extracting access frequency, recency, and seasonality signals from DBQL logs
Clustering & Classification Algorithms
3+ years
Unsupervised grouping of datasets by usage behavior for cold/warm/hot tiers
MLflow / Experiment Tracking
2+ years
Tracking model runs for reproducible analysis and customer handoff artifacts
Natural Language Processing (NLP)
2+ years
Parsing and classifying SQL text from DBQL for workload pattern analysis
Data Visualization (Plotly, Matplotlib, Seaborn)
3+ years
Generating usage heatmaps and partition access charts for stakeholder readouts
Preferred Qualifications
Familiarity with DataStage orchestration log parsing
Prior work on data platform cost optimization or cloud migration readiness assessments
Experience working in healthcare payer data environments (HIPAA awareness)
Strong written communication skills for advisory deliverable authoring (Observation Reports, recommendation backlogs)
Pay: Up to ₹170,000.00 per month
Work Location: Hybrid remote in Bangalore Corporation Building, Bengaluru, Karnataka
