Senior Data Engineer

Job description

Job responsibilities

• Lead the design and roadmap for our data platform
• Partner with operations, product, and engineering to advocate best practices and build supporting systems and infrastructure for the various data needs
• Own the ingest and egress frameworks for data pipelines that stitch together various data sources in order to produce valuable data products that drive the business
• Palantir Foundry is our central hub for data driven applications, visualizations and large scale data analysis across the Anduril org. We manage a large user base and provide true data self service at scale.
• We use SQLMesh for data transformations, giving our engineers the ability to work with SQL while enjoying software engineering best practices like version control, testing, and CI/CD for data pipelines.
• For querying data, we utilize Athena, an aws serverless solution that allows our team to run analytics directly on our data lake without managing infrastructure.
• We've implemented Apache Iceberg as our table format, providing advanced data management capabilities like schema evolution and time travel queries.
• We use Flyte for orchestration. It is a highly scalable, kubernetes native modern orchestration platform similar to Airflow or Dagster.