How to compare the performance of different versions of the same library

In this note I’ll show how to use BenchmarkDotNet to compare the performance of different versions of the same library.

First of all - these libraries must have the same API (at least that we need to benchmark). Consider the library Lib with SomethingUsefull method. For simplicity, this method only freezes the calling thread for some time:

namespace Lib;

public class Class
{
    public void SomethingUsefull()
    {
        Thread.Sleep(1000); // emulates long operation
    }
}

We need to configure the project with NuGet properties:

<PropertyGroup>
  <Copyright>© 2024, Albert Akhmetov</Copyright>
  <Authors>Albert Akhmetov</Authors>
  <Product>Lib</Product>
  <PackageId>Lib</PackageId>
  <Version>0.1.0-alpha1</Version>
</PropertyGroup>

To perform the benchmark, create a console app and add references to BenchmarkDotNet and the baseline version of our library:

<ItemGroup>
  <PackageReference Include="BenchmarkDotNet" Version="0.14.0" />
  <PackageReference Include="Lib" Version="0.1.0-alpha1"/>
</ItemGroup>

For testing purposes we can use the local NuGet repository. For example, let’s use .target directory for NuGet packages. To use this directory in our console test project add the following NuGet.config:

<?xml version="1.0" encoding="utf-8"?>
<configuration>
    <packageSources>
        <clear />
        <add key="local" value="../.target/" />
        <add key="nuget.org" value="https://api.nuget.org/v3/index.json" protocolVersion="3" />
    </packageSources>
</configuration>

Add the following Bench class to the test project:

using BenchmarkDotNet.Attributes;
using BenchmarkDotNet.Configs;
using BenchmarkDotNet.Diagnosers;
using BenchmarkDotNet.Engines;
using BenchmarkDotNet.Jobs;
using BenchmarkDotNet.Reports;

[Config(typeof(Config))]
public class Bench
{
    private class Config : ManualConfig
    {
        public Config()
        {
            var job = Job.Default
                .WithStrategy(RunStrategy.Monitoring) // only monitoring
                .WithWarmupCount(3) // light warm-up
                .WithIterationCount(10) // only 10 interations
                .WithLaunchCount(1); // launch a test only once

            // add the jobs for all versions:
            AddJob(
                job.WithNuGet("Lib", "0.1.0-alpha1").WithId("alpha1").WithBaseline(true),
                job.WithNuGet("Lib", "0.1.0-alpha2").WithId("alpha2"),
                job.WithNuGet("Lib", "0.1.0-alpha3").WithId("alpha3"),
                job.WithNuGet("Lib", "0.1.0-beta").WithId("beta")
            );

            // add the percentage column
            SummaryStyle =
                SummaryStyle.Default
                .WithRatioStyle(BenchmarkDotNet.Columns.RatioStyle.Percentage);
        }
    }

    [Benchmark]
    public void SomethingUsefull()
    {
        new Lib.Class().SomethingUsefull();
    }
}

To run a benchmark add the following code to Program.cs:

using BenchmarkDotNet.Running;

BenchmarkRunner.Run<Bench>();

We need to put all the required NuGet packages into .target folder. The easy way to do it is to use the following command:

dotnet pack -c:Release -o "../.target"

Our versions differ only for timeout for Sleep method:

Version	Timeout
0.1.0-alpha1	1000
0.1.0-alpha2	800
0.1.0-alpha3	200
0.1.0-beta	1500

Change timeout and a version number, then execute dotnet pack command. As a result .target directory will contain the following packages:

Lib.0.1.0-alpha1.nupkg
Lib.0.1.0-alpha2.nupkg
Lib.0.1.0-alpha3.nupkg
Lib.0.1.0-beta.nupkg

At this time we’ve got all we need to run our benchmark. To run the benchmark use the following command for the test app:

dotnet run -c:Release

After the benchmark is passed you’ll see the following table:

Method	Job	NuGetReferences	Mean	Error	StdDev	Ratio	RatioSD
SomethingUsefull	alpha1	Lib 0.1.0-alpha1	1,007.7 ms	6.30 ms	4.17 ms	baseline
SomethingUsefull	alpha2	Lib 0.1.0-alpha2	807.2 ms	7.83 ms	5.18 ms	-20%	0.7%
SomethingUsefull	alpha3	Lib 0.1.0-alpha3	110.5 ms	7.83 ms	5.18 ms	-89%	4.5%
SomethingUsefull	beta	Lib 0.1.0-beta	1,508.5 ms	9.43 ms	6.24 ms	+50%	0.6%

Sample code is available here.