SWE, Data Ingestion

Job description

Job responsibilities

• Debug and resolve failing or blocked ingestion pipelines.
• Investigate issues caused by corrupt, malformed or unexpected data.
• Design and implement more resilient pipelines so individual bad data segments do not block wider workflows.
• Improve how we handle varied data formats from partners, suppliers and third-party sources.
• Support orchestration across multi-step ingestion workflows, including dependencies, retries and queue management.
• Optimise Spark jobs and data-processing pipelines for throughput, compute efficiency and reliability.
• Reduce operational toil around failed jobs, stalled pipelines and manual interventions.
• Work on high-volume batch-processing systems where throughput, reliability and cost all matter.