Together AI released the continuous compilation and inference system TKCC
The story so far
Together AI released Together Compiler (TKCC), a continuous compilation system driven by high-throughput coding agents: the agents rewrite model forward computation and hot-load the results into the resident inference engine, allowing repeated optimization of inference with production traffic changes. Together AI claims that TKCC achieved up to a 2.6x TPS improvement on cutting-edge open-source models such as Kimi K3, GLM 5.3, MiniMax M3.1-Flash, and DeepSeek v4.1-Flash within less than 24 hours; on the SemiAnalysis InferenceX low-latency benchmark, MiniMax M3.1-Flash reached 1111 TPS, and DeepSeek v4.1-Flash reached 1205 TPS, claiming to be 1.7x faster than existing open-source megakernels in Llama 1B with batch size 1 settings. According to MarkTechPost, these results were achieved on AgentX load, with each configuration completed within 24 hours on no more than 2 nodes. TKCC has already been running on some of Together's serverless endpoints and is currently being provided to Together customers in a limited beta form; Together AI claims that customers will later be able to optimize their dedicated endpoints with TKCC with a single click. In a subsequent article, Together AI stated that it achieved 98% of megakernel performance on CUDA Graphs and PDL on the default Torch Inductor core, and used 'multi-core' plus agent loops to increase Llama 1B inference on H100 by 1.7x to 1100 TPS. The above performance figures are claims by Together AI and related reports and have not been independently verified.
AI-written from the reports · Updated 2 hours ago
Developments
3 developments
- 16 Nov, 08:00 · 1 reportsTogether AI built megakernels last year and deleted them this year: the path to Together CompilerTogether AI Blog:Together AI built megakernels last year and deleted them this year: the path to Together Compiler
- 4 Nov, 08:00 · 3 reportsTogether Compiler: Continuous Compilation for AI InferenceTogether AI Blog:Together AI has released Together Compiler, continuously compiling and optimizing AI inference.
- 4 Nov, 08:00 · 1 reportsPolymorphic Inference: self-tuning inference for open modelsTogether AI Blog:Together AI launches Polymorphic Inference: Adaptive open-source model inference technology
Coverage timeline
Follow the coverage to see every side of the story.
- Together AI BlogTogether AI built megakernels last year and deleted them this year: the path to Together Compiler
Together AI achieved 98% megakernel performance on the default Torch Inductor core using CUDA Graphs and PDL, and used 'multi-core' and agent loops to increase the inference speed of Llama 1B on H100 by 1.7 times to 1100 TPS.
- MarkTechPostFeaturedTogether AI has released Together Compiler, with agents enabling a 2.6x speedup in inference.
Together AI has released Together Compiler (TKCC), a continuous compilation system driven by coding agents, which rewrites model forward computation and hot loads it into the resident inference engine, achieving up to 2.6 times the TPS of the publicly disclosed best configuration on SemiAnalysis InferenceX's AgentX workload, with each configuration completed within 2 nodes and 24 hours.
- Together AI BlogTogether Compiler: A continuous compiler for AI inference
Together Compiler (TKCC) achieves continuous optimization of AI inference workloads through high-throughput agents, reducing the latency of cutting-edge open-source models (Kimi K3, GLM 5.3) within 24 hours.
- Together AI BlogFeaturedTogether AI has released Together Compiler, continuously compiling and optimizing AI inference.
Together AI has released Together Compiler (TKCC), which uses high-throughput intelligent agents to continuously generate optimization solutions for inference engines, achieving 1.3-3x TPS improvements on cutting-edge open-source models like Kimi K3, GLM 5.3, MiniMax M3.1-Flash, and DeepSeek v4.1-Flash within 24 hours.
- Together AI BlogTogether AI launches Polymorphic Inference: Adaptive open-source model inference technology
Together AI Launches Polymorphic Inference Technology, Continuously Compiling and Optimizing Open Source Model Inference Performance Using AI Agents to Address Challenges Such as Model Release Acceleration, Architectural Diversity, and Rapid Changes in Workloads and Hardware Platforms. This Technology Enables Inference Systems to Adapt Quickly to Dynamic Environments, for Example, Achieving Near-Hardware-Limit Performance When 80% of OpenRouter Traffic Comes from Models Released in the Past Three Months.
Heat over time
Not enough data yet to draw a trend.