Inference Systems Engineer

Role & Responsibilities

• Design and deploy low-latency inference pipelines for LLMs, diffusion models, and vision transformers across CPU, GPU, and NPU architectures.
• Optimize model serving stacks using Triton Inference Server, vLLM, or TGI—tuning batch sizes, quantization, and memory layout for peak performance.
• Containerize and orchestrate inference services via Docker and Kubernetes, ensuring high availability and auto-scaling under fluctuating workloads.
• Implement model monitoring, health checks, and A/B testing frameworks to validate performance and drift in production.
• Collaborate with ML Engineers to convert trained models into production-ready formats (ONNX, TensorRT, GGUF) with minimal accuracy loss.
• Build observability dashboards (Prometheus/Grafana) and alerting logic to detect and mitigate inference bottlenecks in real time.

Must-Have

• Python
• Docker
• Kubernetes
• Triton Inference Server
• ONNX Runtime
• PyTorch
• TensorRT
• Prometheus
• Grafana
• CI/CD (GitHub Actions, GitLab CI)

• Experience with vLLM or TGI
• Knowledge of Model Quantization (AWQ, GPTQ, GGUF)
• Familiarity with NVIDIA Triton model ensemble pipelines

Skills: cuda,architecture,building,foundation,ml,management,models,distributed systems,infrastructure,inference