Turbopack benchmark numbers questioned by Vite creator, Evan You

by
Tags: , , , , ,
Category: ,

In the most recent release of Next.js, an alpha of a rust-based build tool was made available in alpha as a preview. This tool, Turbopack, is written by a team that includes the Webpack founder, Tobias Koppers who recently joined Vercel.

Turbopack build performance claims

During a presentation by Turbopack engineer Maia Teegarden benchmarks were displayed comparing the performance in various build frameworks for hot module reloading (HMR), the process of hot-patching a running React application after saving changes to source code during development.

One of those benchmark results compared the application server startup of a large (3K files) application in development mode for components using HMR. In this benchmark, Turbopack was shown to speed up Next.js startup times from 16.515 seconds in Next.js 11, to 6.331 seconds in Next.js 12, to 1.877 seconds in Next.js 13 with the Turbopack build enabled. That shows steady progress in starting up in development mode.

That same graphic claims that Vite (no version given) had an 11.448 second application server startup time for HMR development mode.

The next page claimed that Turbopack is 700x faster than Webpack in HMR reload activities and 10x faster than Vite.

Pushback on claims by Vite creator

Evan You, the creator of Vite (and Vue.js), took time to both create and discuss his own benchmark, as well as run the Vercel benchmark for Next.js 13 / Turbopack.

He responds that there were some issues with the way the information was delivered, and that if building performance were critical, that the SWC compiler should have been enabled on Vite (by default it uses Babel), and that the defaults should have included server-rendered components since Vercel recommends they be used by default in the Next.js 13 components to improve performance. It turns out that Turbopack’s performance for HMR suffers in that case, bringing the base comparisons down to more reasonable differences.

Evan also points out that the benchmark numbers seem flawed and put together his own benchmarks to show what he considers to be a fairer comparison.

Review for yourself

The research that Evan has done is excellent. Both the original benchmarks by Vercel’s Turbopack team and Evan’s responses are well worth your reading, to see how data-based discussion of issues with profiling/benchmarking can be handled.

You can see the entire Vercel benchmark on the Turbo.build website which was published after the events of the past week.

Evan’s response and his continued updates to further well-informed discussion and push back on errors in the original claims are posted in this great discussion thread on GitHub.

Spirited debate is healthy

While I do think that Turbopack is potentially an important new build tool, I hope to see the work done in the open, benefitting not just Next.js first but the entire build community. Webpack was used by just about every production JS build stack, and to replace it, Turbopack needs to be just as open.

I think that the way the teams are discussing these benchmarks is a healthy process, and let’s hope we keep hype out of the discussion and focus directly on the data and improvements as Evan and the Vercel team have been doing.