Software Engineer, ML Training Optimization

Job description

Job responsibilities

• Design, implement, and operate cutting-edge ML training performance optimization techniques — distributed training (data/tensor/pipeline parallelism), mixed-precision, gradient checkpointing, and GPU kernels — to scale…
• Work on RL framework and RL performance optimization for driving foundation model
• Profile and debug training throughput bottlenecks across the compute, compiler, framework, and cluster-networking stack.
• Collaborate closely with ML researchers, data engineers, and hardware engineers to define requirements and align on architectural decisions for large-scale training infrastructure.
• Implement custom kernels and build benchmarks to track and validate training-performance optimizations across the stack.

Job requirements

• 4+ years of total experience, including 2+ years working on large-scale model training platforms.
• Experience with distributed training frameworks (Megatron-LM, TorchTitan), efficiently utilizing GPU/accelerator clusters at scale.
• Experience with RL framework such as Verl/MILES and RL algorithms(GRPO, PPO)
• Experience with collective communication libraries (e.g., NCCL) and optimizing communication/compute overlap in multi-node training.
• Experience using profiling tools (NVIDIA Nsight, PyTorch Profiler, or XLA/JAX profiling) to identify and resolve training bottlenecks.
• Proficient in Python or C++.
• Experience with ML compilers, performance tooling, or hardware/software co-design (e.g., GPU/TPU/accelerator architectures).