Software Engineer, II - Autonomy Data

Job description

Job responsibilities

• Data Lake and Ingestion Pipeline Contribute to the design and organization of the program’s data lake, including schema definitions, partitioning strategy and metadata indexing.
• Build and maintain end-to-end pipelines that ingest high-bandwidth sensor logs from vehicles into cloud storage with high reliability and tolerant of ad-hoc and intermittent connectivity mechanisms.
• Implement data validation and integrity checks that can detect corrupted information, missing sensors, and inconsistent calibration prior to the data being processed by downstream systems.
• Implement retention, tiering and lifecycle policies for data to balance storage costs with development value.
• Dataset Curation and Labeling Infrastructure Build tooling to query raw logs to produce curated training and evaluation datasets.
• Build automation to run cost-effective pseudo-labeling workflows at the scale of data ingest.
• Implement data quality and model performance metrics that are used to direct labeling effort toward the highest-value examples.
• Autonomy Data Visualization Deploy and maintain data visualization tooling to support log review, annotation QA, and autonomy debugging workflows.

Job requirements

• Bachelor’s degree in Computer Science, Computer Engineering, Software Engineering, Electrical Engineering or a related field with 4+ years of data engineering experience or a Master’s with 2+ years.
• Strong proficiency in Python and SQL, with demonstrated ability to build production-quality data pipelines
• Experience with cloud data infrastructure (AWS preferred: S3, Glue Athena, redshift, or equivalent) and infrastructure-as-code tools (Terraform, Cloud Formation).
• Solid understanding of data partitioning strategies and columnar storage formats (Parquet, Orc, etc.)
• Experience building and operating data pipelines that process time-series and binary data.
• Proven ability to evaluate and integrate open-source tooling when appropriate versus building from scratch.
• Good instincts for delivering data quality through first-class implementations of monitoring, validation and lineage tracking.
• A competitive compensation package that includes a bonus component and stock options