Lead Data Platform Engineer - Enterprise, Data & AI

Job description

Job responsibilities

• Scaling self-healing data pipelines that automatically handle schema evolution, detect anomalies, execute circuit breakers and recover from failures without manual intervention.
• Define and enforce company-wide data governance, access, automated data quality testing, schema evolution policies and metadata management to maintain high-fidelity data assets.
• Implement intelligent storage lifecycle strategies, resource throttling and query optimization to minimize compute overhead while delivering high-throughput, low-latency data access for analytics and AI agents.
• Build automated CI/CD deployment templates, environment isolation, testing frameworks and version control standards, enabling rapid, reliable and zero-downtime deployments

Job requirements

• 10+ years of hands-on experience in Data Platform Engineering or Software Engineering with a proven track record of architecting and scaling production-grade data foundations.
• Deep expertise in scaling and optimization, query tuning, compute resource allocation and implementing efficient compute-storage lifecycle policies to minimize infrastructure costs.
• Experience implementing enterprise security standards, Role-Based Access Control (RBAC), Active Directory/IAM roles and fine-grained data masking, with strong familiarity supporting enterprise compliance requirements…
• Track record of establishing enterprise data governance frameworks, automated schema evolution controls, data quality audits and near real-time observability/alerting.
• Hands-on experience building automated CI/CD pipelines, environment isolation (branch testing, rollback mechanisms), version control and automated deployment testing for data assets.
• Experience incorporating LLMs or GenAI directly into pipeline operations (e.g., code generation, triage and root-cause analysis, automated data reconciliation, backfills or anomaly detection).
• Experience with infrastructure-as-code (Terraform) and containerization/orchestration (EKS) for elastic compute management. GPU/CPU performance optimization & utilization.
• Previous experience in the autonomous vehicle, robotics or high-tech manufacturing sectors.