dotnet/BenchmarkDotNet

> The de facto standard for .NET microbenchmarking — turns annotated methods into statistically rigorous, process-isolated performance experiments.

GitHub repo · Official website · License: MIT

Overview

BenchmarkDotNet is a .NET library for measuring the performance of individual methods. You mark a method with [Benchmark], run a host program, and the library handles the parts that people get wrong by hand: pilot runs to pick invocation counts, warm-up to reach steady state, overhead subtraction, dead-code-elimination guards, outlier detection, and statistical summarization. It was started by Andrey Akinshin, who remains the project lead, and now lives under the dotnet organization as a .NET Foundation project¹.

The defining design choice is that a benchmark is not run in-process. For each [Benchmark] × job combination, BenchmarkDotNet generates a small standalone project, compiles it in Release, and launches it as a separate process, then measures inside that isolated host². This is what lets a single benchmark class target .NET Framework, several .NET Core/.NET versions, Mono, and NativeAOT in one run — each target gets its own generated, compiled, and executed host. It is also the root cause of the tool's biggest practical cost: benchmark runs are slow, because compilation and multi-iteration execution happen per job.

The tension to understand up front: BenchmarkDotNet optimizes for correct numbers, not fast numbers. It will spend seconds to minutes per method and actively refuses to run configurations it considers unreliable (DEBUG builds, for instance). For microbenchmarks — comparing two implementations of a hot method — this is exactly right. For anything macro (end-to-end request latency, load, throughput of a whole service) it is the wrong tool, and the docs say so.

Getting Started

dotnet new console -n MyBenchmarks
cd MyBenchmarks
dotnet add package BenchmarkDotNet

Benchmarks must run from a Release build; the recommended entry point is BenchmarkRunner:

using BenchmarkDotNet.Attributes;
using BenchmarkDotNet.Running;
using System.Security.Cryptography;

BenchmarkRunner.Run<HashBenchmarks>();

[MemoryDiagnoser]                       // also report allocations / GC
public class HashBenchmarks
{
    private readonly byte[] _data = new byte[10_000];
    private readonly SHA256 _sha256 = SHA256.Create();
    private readonly MD5 _md5 = MD5.Create();

    public HashBenchmarks() => new Random(42).NextBytes(_data);

    [Benchmark(Baseline = true)]
    public byte[] Sha256() => _sha256.ComputeHash(_data);

    [Benchmark]
    public byte[] Md5() => _md5.ComputeHash(_data);
}

dotnet run -c Release

Returning a value from the benchmark method is the idiomatic way to defeat dead-code elimination; for void work, take a Consumer or assign to a field the JIT cannot prove is dead.

Architecture / How It Works

The pipeline for a single run:

1. Discovery — reflection finds [Benchmark] methods and reads attributes ([Params], [Arguments], [GlobalSetup], jobs, diagnosers) into an immutable configuration. 2. Code generation — for each benchmark × job, BenchmarkDotNet emits a BenchmarkDotNet.Autogenerated project that wraps the target method in an unrolled loop and an idle-overhead counterpart. 3. Build — each generated project is compiled in Release via the SDK/MSBuild for the job's target framework (RuntimeMoniker.Net80, Mono, NativeAot80, etc.). 4. Execution — the host process runs stages in order: Pilot (find the invocation count that yields a target iteration time), Warmup (reach steady state — JIT tiering, caches), Actual iterations, plus separate Overhead (idle) iterations that are subtracted from the workload. 5. Analysis — the statistics engine (perfolizer, more recently pragmastat, both by the same author) computes mean/median/StdDev, detects multimodality and outliers, and applies analysers that emit warnings³. 6. Export — results render as a console table and optionally Markdown, HTML, CSV, JSON, XML, and R plots via [RPlotExporter].

Jobs are the multi-environment mechanism. A job pins runtime, JIT, platform, GC mode (workstation/server, concurrent), environment variables, and iteration counts. [SimpleJob(RuntimeMoniker.Net80)] plus [SimpleJob(RuntimeMoniker.Mono)] on the same class runs the full matrix. Everything an attribute expresses is also available fluently via ManualConfig / Job.

Diagnosers are the pluggable measurement backends. [MemoryDiagnoser] reports allocations and GC collections; there are also disassembly, ETW/EventPipe, native-memory, and threading diagnosers. Some are platform-specific (ETW is Windows-only).

The coupling story: BenchmarkDotNet is tightly bound to the .NET SDK/MSBuild toolchain because it shells out to build generated projects. This means the build environment (SDK version, TargetFramework, NuGet restore) directly affects whether a run succeeds, and it is why only console applications are supported as hosts.

Production Notes

Runs are slow, and that is by design. A class with a handful of benchmarks across two or three runtimes can take several minutes because each combination is compiled and then executed for pilot + warmup + actual iterations. In CI, use [ShortRunJob] or a custom job with reduced warmup/iteration counts to trade precision for wall-clock time — but treat those numbers as indicative, not publication-grade.

It refuses unreliable configurations. Running from a DEBUG build, under an attached debugger, or with an optimization-disabled assembly triggers errors or warnings; this trips up people who hit "Run" in their IDE instead of dotnet run -c Release. Virtualization (Hyper-V, VMware, VirtualBox) and running on battery also produce warnings because they perturb timing.

The generated-project build is a common failure point. Errors like "the benchmark project failed to build" usually mean an SDK/TargetFramework mismatch, a missing runtime pack for a requested moniker (e.g. asking for Net472 on Linux, or a NativeAOT job without the toolchain installed), or InternalsVisibleTo/access issues in the code the generator wraps. The generated sources land in bin/.../BenchmarkDotNet.Artifacts/ and are worth reading when a run fails.

Numbers are only comparable within one run on one machine. Absolute nanoseconds are not portable across hardware, OS, SDK version, or BenchmarkDotNet version. Compare ratios against a Baseline = true benchmark rather than raw means, and re-baseline whenever the environment changes.

Allocation numbers can be subtle. [MemoryDiagnoser] measures managed allocations well, but for very small or very fast methods GC accounting granularity and tiered JIT warm-up can move results; give allocation-sensitive benchmarks enough iterations, and be aware that displayGenColumns and server GC change what the columns mean.

Version cadence is slow and pre-1.0. The library has shipped 0.x for its entire history; releases are infrequent but individually significant, and moniker/API surface can shift between minor versions (new RuntimeMoniker values, diagnoser changes). Pin the package version in CI so a new release does not silently change your measured baselines.

When to Use / When Not

Use when:

• You are comparing implementations of a hot method and need defensible, statistically analyzed numbers.
• You want one benchmark class to measure across multiple runtimes/JITs/GC modes.
• You need allocation and GC data alongside timing ([MemoryDiagnoser]).
• You are producing performance results others will scrutinize — the guardrails and warnings matter.

Avoid when:

• You need macro/end-to-end or load benchmarks (whole-service latency, throughput under concurrency) — use a load/macro harness instead.
• You want fast, throwaway "is this faster?" checks in a tight loop — the per-run compile-and-iterate cost is high.
• You are benchmarking something that only exists inside a web/host process that cannot be reduced to a console app.
• You need cross-machine absolute numbers as a stable metric — results are environment-bound.

Alternatives

• dotnet/crank — use when you need macro/load benchmarking of ASP.NET-scale services rather than method-level microbenchmarks.
• petabridge/NBench — use when you want assertion-based performance tests that pass/fail in CI as regression gates.
• google/benchmark — use when your code is C++ and you want the analogous microbenchmark library.
• openjdk/jmh — use on the JVM; same generate-isolate-measure philosophy for Java/Kotlin.
• bheisler/criterion.rs — use in Rust for statistics-driven microbenchmarks with a comparable analysis focus.

History

References

1. ^ BenchmarkDotNet project — dotnet organization / .NET Foundation. https://github.com/dotnet/BenchmarkDotNet
2. ^ "How it works" — BenchmarkDotNet documentation (generation, build, and process isolation per job). https://benchmarkdotnet.org/articles/guides/how-it-works.html
3. ^ perfolizer statistical engine by Andrey Akinshin. https://github.com/AndreyAkinshin/perfolizer
4. ^ BenchmarkDotNet README, Md5VsSha256 sample and summary output. https://github.com/dotnet/BenchmarkDotNet

Tags

dotnet, csharp, benchmarking, performance, microbenchmark, profiling, statistics, testing, library, netcore