Inference Engineer — Production AI Systems

The engineer on duty
when frontier models
meet production.

At Together AI I'm an inference engineer on dedicated endpoints — embedded with the platform's highest-stakes customers, owning everything from model bring-up on new GPU fleets to the 3 a.m. root cause inside the serving engine. Before this: six years of solutions engineering and data platforms — pre-sales, POCs, and GenAI architecture at SingleStore and Confluent. I work best forward-deployed: where the customer, the infrastructure, and the source code meet.

1B+
requests / quarter on the fleets I support
3 / 10
top-revenue accounts owned day-to-day
PR→
root causes landed in the inference engine itself
6+ yrs
data & AI systems before this
01

Currently

Nov 2025 — present · remote
Together AI — Inference Engineer

Post-sales engineering for dedicated inference endpoints.

Together AI · dedicated endpoints & CX engineering · B200 / GB300 / H200 / H100 fleets

My customers run open-weight frontier models — Kimi K2.6 / K3, MiniMax M2.7 / M3, and other large MoE, long-context, and multimodal models — as single-tenant deployments where an outage is their product going down. I own that surface end to end, working hands-on across TensorRT-LLM, vLLM, and SGLang, down to engine source when needed.

Ownership

The revenue-critical edge

Primary engineering owner for 3 of the platform's top-10 revenue accounts ($8M+ quarterly GMV): a frontier AI lab, one of the largest AI-coding products, and enterprise conversational-AI customers.

Depth

Root causes, not ticket ping-pong

When a flagship deployment kept failing in ways restarts couldn't fix, I read the serving-engine source, isolated the failure, found the upstream open-source fix, and authored the PR.

Outcomes

Ship the fix and the story

Caught an API parameter that was accepted but silently ignored across service layers; wrote the engineering handoff that unblocked a customer's production launch.

Serving stacks

Disaggregated serving, at full tilt

Support NVIDIA Dynamo deployments in production for frontier MoEs (Kimi K2.6, K3) — prefill/decode disaggregation and KV-aware routing that pull full capacity out of B200-class fleets.

Automation

Days to minutes, literally

Built AI-powered triage skills — a diagnostic toolkit that walks the full request path from edge to engine — cutting self-serve customer resolution time from days to minutes.

Observability

See the problem before the page

Ship Grafana dashboards and alerts as deployments need them, plus a generic alerting framework — and new diagnostic methods that pinpoint what's wrong with a model or deployment in minutes, feeding findings back into product fixes.

02

Selected engineering work

problem → root cause → outcome
i.

The crash that restarts couldn't fix

LLM serving engines · speculative decoding · guided decoding · CUDA graphs
The symptom

A flagship production deployment kept paging with recurring worker crashes — always the same serving stage, in waves, and weeks of restarts and redeploys hadn't made it go away.

The root cause

Instead of another restart, I went into the serving engine's source and isolated the failure to an interaction between two engine features that only triggers on this deployment's exact shape — explainable from code alone.

The outcome

Matched it to an upstream open-source fix, backported the patch onto the release branch in use, and authored the PR — handing the engine team a merge-ready fix instead of a bug report.

ii.

The flag that went nowhere

OpenAI-compatible APIs · request routing · protocol adaptation · agent frameworks
The symptom

A customer running a production agent framework set a standard API parameter — the platform returned 200 and then behaved as if it was never sent, causing duplicated side effects in their live product.

The root cause

Traced the request across every service layer it passes through and proved where the parameter was being dropped — accepted at the front door, gone before it ever reached the model. Verified at each hop, not guessed.

The outcome

Wrote the complete fix handoff — root cause, the expected API contract, streaming edge cases, and a local reproduction guide — so engineering could ship the fix without touching a live endpoint.

iii.

The limit nobody documented

Batch APIs · Go services · log forensics · docs fix
The symptom

A wave of batch-processing failures across multiple enterprise customers, all dying at job creation with the same opaque low-level error.

The root cause

Read the service source and found an undocumented payload size limit buried in a standard-library default — silently rejecting oversized inputs with the real reason never surfaced to the caller.

The outcome

Proved it with a log query, unblocked every affected customer the same day, and drove the product fix: a structured validation error at upload time, with the limit documented.

03

Systems I live in

daily toolkit
NVIDIA DynamoTRT-LLMvLLMSGLangSMG (model gateway) Disaggregated serving (prefill/decode)KV-aware routingMoE servingCUDA graphsSpeculative decoding Tool-call parsingKubernetesB200 / GB300 / H200 / H100 fleets GrafanaLokiPrometheus MCPAgentic AIRAG KafkaFlinkSingleStore vectors PythonGoTypeScript
04

Open source & building

github.com/vishwajeetdabholkar
SMG — model gateway, from source to prod Rust · gRPC · cache-aware routing I run the open-source Shepherd Model Gateway in production daily — and go deeper: a local lab running my own fork (Docker / cargo) against toy and live Kubernetes engines, plus a from-zero mastery guide written off a full walk of the source. eGet-Crawler-for-ai ★ 57 · Python AI-first web scraper: clean markdown / JSON / table extraction with dynamic-content handling and an MCP plug for instant Q&A and RAG ingestion. MCP servers & clients Protocol · Playwright · Claude · Ollama Build and customize both sides of Model Context Protocol — browser-controlling agents, doc-site crawlers, and tool integrations, with public walkthroughs and an MCP-listing entry. Real-time RAG systems SingleStore vectors · Hybrid search Retrieval that goes beyond "search + LLM" — semantic + SQL in one query, hybrid search, and streaming ingestion, with deep-dive posts and live code. resume_parser_api ★ 19 · OCR + LLM OCR + LLM-driven parser generating ATS-ready structured JSON, with public docs and integration walkthroughs. Druid tools for agents · agno · SingleStore notebooks Contributor · Demos Agent-safe Druid introspection with query guardrails, plus contributions to the agno agent framework and SingleStore's public notebooks — demos and code drops on LinkedIn.
05

Previously

the road to inference
Nov 2024 — Nov 20251 yr 1 mo
Confluent
Solutions Engineer, Bengaluru (remote)
  • Designed and implemented real-time data-streaming architectures on Apache Kafka and Confluent Cloud for enterprise clients.
  • Brought GenAI to the streaming stack — built agentic-AI and MCP demos and real-time RAG architectures on live data streams, showing customers what "AI on data in motion" actually looks like.
  • Shipped the open-source work that backs it: MCP servers/clients, browser-automation agents, and agent-safe database tooling — with public demos and walkthroughs.
Mar 2023 — Sep 20241 yr 7 mos
SingleStore
Enterprise Solutions Engineer, Pune (remote)
  • The GenAI-era database SE: guided enterprise customers on vector search, embeddings, and hybrid (semantic + SQL) retrieval — architecting RAG applications on SingleStore when the vector wave was just breaking.
  • Drove the pre-sales engine: customer meetings, technical demonstrations, and proof-of-concepts across industries, then led implementations in customer environments end to end.
  • Acted as technical evangelist — training sessions, workshops, blogs, and public presentations on GenAI and real-time analytics use cases.
Apr 2022 — Feb 202311 mos
Fresh Gravity
Analyst — Data Engineering, Maharashtra
  • Built a product's data-ingestion platform with Databricks and PySpark — extracting from S3, Azure Blob, GCS, Redshift, BigQuery, and Snowflake into profiling and quality-check flows.
Mar 2021 — Mar 2022Lumiq + freelance
Lumiq
Big Data Engineer, Pune · then freelance Spark work for a fintech
  • Built a data lake from scratch for an insurance company on AWS — PySpark and SparkSQL ETL, Glue ingestion/transformation jobs, API ingestion via Lambda and API Gateway.
Sep 2019 — Mar 20211 yr 7 mos
Tata Consultancy Services
Data Engineer, Mumbai
  • Built optimized ETL and migration pipelines with PySpark and AWS — moving data through the layers of a production data lake, with continuous automation of existing processes.
profile.md — copy it anywhere