AI Engineer, Vision & Foundation Models, Optimus

Job description

Job responsibilities

• Design and build scalable data pipelines, utilizing classical computer vision and off-the-shelf models to auto label massive video datasets
• Develop and train state-of-the-art foundational vision and multimodal networks, focusing on efficient tokenization, compression, and fusion of vision, language, audio, and tactile data
• Iterate on neural network architectures to improve perception, 3D understanding, and world modeling capabilities
• Optimize models for deployment on edge compute, utilizing quantization, pruning, and distillation to ensure real-time performance on the robot
• Collaborate with hardware and controls engineers to close the loop between perception and actuation in the real world

Job requirements

• Strong software engineering skills in Python with an ability to write production-quality code
• Experience with PyTorch or another major deep learning framework (e.g., JAX, TensorFlow)
• Experience training large-scale models on distributed GPU clusters
• Strong mathematical fundamentals, including linear algebra, computational geometry, probability theory, and numerical optimization, are a plus
• Familiarity with classical computer vision (camera models, calibration, sensor fusion) is a plus
• Strong mathematical fundamentals, including linear algebra, computational geometry, probability theory, and numerical optimization, are a plus
• Familiarity with classical computer vision (camera models, calibration, sensor fusion) is a plus
• Domain expertise in one or more of the following areas:
• Foundational Vision: Vision encoder pretraining, dense prediction, segmentation
• 3D & Geometry: Multiview geometry, NeRFs, 3D Gaussian Splatting, geometric vision models
• Multimodal Models: World models, video generation, vision-language-action models
• Human/Object Reconstruction: Mesh reconstruction, hand pose estimation, human-object interaction, object affordance