Skip to content
Live
Trending storyPast story

Inference Performance Index released

1 reports1 outlets10 days agoUpdate

The story so far

AI overview

Epoch AI has released the Inference Performance Index (IPI), used to measure the capability of chips in intelligent agent inference tasks. Nvidia GB300's performance improvement over H100 is 25 times greater, far exceeding the 2.5-fold increase in 8-bit FLOP/s. The IPI measures inference capability by comparing the number of requests a chip can handle simultaneously under various output speed requirements, with GB300 set as the baseline value of 1. This index combines the results of four representative models, using GB300 as the reference benchmark. Epoch AI claims that the IPI can more accurately reflect inference performance because comparing FLOP/s in isolation would underestimate GB300's capability relative to H100. The IPI index uses Nvidia's AISimulate to estimate inference performance, based on the operations required by the model and hardware input. The index expresses scores as GB300 equivalent (GB300e), with GB300 defined as 1 point. Epoch AI estimates that on simulated intelligent agent inference workloads, the inference capability of GB300 per chip is about 25 times that of H100, despite its 8-bit FLOP being only 2.5 times that of the latter.

AI-written from the reports · Updated 8 days ago

Coverage timeline

Follow the coverage to see every side of the story.

9 Nov
  1. Epoch AI
    Inference Performance Index: changing the way we measure inference capacity

    Nvidia GB300 offers a 25-fold improvement in capability compared to H100 for intelligent agent inference tasks, far exceeding the 2.5-fold increase in 8-bit FLOP/s. The IPI index measures inference capability by comparing the number of requests a chip can handle simultaneously under various output speed requirements. GB300 is set as the baseline value of 1, and chips scoring 1 can serve twice as many users under the same conditions. IPI version 1 combines results from four representative models, using GB300 as the reference benchmark.