The xUnit test framework is a .NET staple, so why reinvent that wheel with the Fixie test framework? I wanted to see what would happen if we reimagined .NET testing through the lense of developer ergonomics. We got a taste for this emphasis back in Meaningful test coverage by enhancing the developer’s ability to quickly spot the cause of a test failure. Today, we’ll explore Fixie’s ergonomics beyond test code: from installation to command line execution to running under Azure DevOps. At each step, we’ll contrast the ergonomics with that of xUnit to highlight what has been cut, smoothed over, rejoined, and reimagined with Fixie to provide an excellent development experience.
When setting up xUnit, you might study the documentation and ultimately decide to install three of the 13 listed packages:
Once you’ve got it working, your project file would look like so:
<Project Sdk="Microsoft.NET.Sdk"> <PropertyGroup> <TargetFramework>netcoreapp3.0</TargetFramework> </PropertyGroup> <ItemGroup> <PackageReference Include="Microsoft.NET.Test.Sdk" Version="15.9.0" /> <PackageReference Include="xunit" Version="2.4.1" /> <PackageReference Include="xunit.runner.visualstudio" Version="2.4.1" /> </ItemGroup> </Project>
When setting up Fixie, you could merely install the “Fixie” package and your assertion library of choice. I like to use Shouldly for readability:
<Project Sdk="Microsoft.NET.Sdk"> <PropertyGroup> <TargetFramework>netcoreapp3.0</TargetFramework> </PropertyGroup> <ItemGroup> <PackageReference Include="Fixie" Version="2.2.0" /> <PackageReference Include="Shouldly" Version="3.0.2" /> </ItemGroup> </Project>
The “Fixie” package alone accomplishes many things. Your test project’s assembly is the test runner executable. If it runs under TeamCity, AppVeyor, or Azure DevOps, you’ll get specialized output for that environment. Test Explorer and dotnet test work right away as well: no need for a separate Test Explorer plugin package, and no need to include the SDK yourself. No Googling for error messages. No wondering why your tests aren’t showing up. Install “Fixie” and go.
Command Line Experience
For a rough benchmark, we have a solution with 25,000 empty test methods cross-targeting .NET Core 2.2 and 3.0, runnable with either xUnit or Fixie. Since the tests are empty, we’re comparing the overhead of the test frameworks.
With xUnit, here’s the developer experience under dotnet test:
Test run for C:\dev\fixie.benchmark\src\xUnit.Tests\bin\Release\netcoreapp2.2\xUnit.Tests.dll(.NETCoreApp,Version=v2.2) Microsoft (R) Test Execution Command Line Tool Version 16.3.0 Copyright (c) Microsoft Corporation. All rights reserved. Starting test execution, please wait... A total of 1 test files matched the specified pattern. Test Run Successful. Total tests: 25000 Passed: 25000 Total time: 10.3963 Seconds Test run for C:\dev\fixie.benchmark\src\xUnit.Tests\bin\Release\netcoreapp3.0\xUnit.Tests.dll(.NETCoreApp,Version=v3.0) Microsoft (R) Test Execution Command Line Tool Version 16.3.0 Copyright (c) Microsoft Corporation. All rights reserved. Starting test execution, please wait... A total of 1 test files matched the specified pattern. Test Run Successful. Total tests: 25000 Passed: 25000 Total time: 8.8996 Seconds
This is not ergonomic. Most of this output is not information. It’s hard to spot at first where the first target framework run ends and the second begins. We should be suspicious about how long this took to run: the tests are empty but took 19 seconds to run.
As a more ergonomic alternative to dotnet test, the “Fixie.Console” command line extension eases test running further:
<DotNetCliToolReference Include="Fixie.Console" Version="2.2.0" />
dotnet fixie does for developer ergonomics what Marie Kondo did for living spaces. Because it directly executes your test projects immediately, performs no specialized assembly loading, and skips the Visual Studio plugin overhead entirely, it gets right down to business. 50,000 tests in just over 1 second. The console output includes everything of value hiding in dotnet test’s output, ruthlessly omitting everything else:
Running Fixie.Tests (netcoreapp2.2) 25000 passed, took 0.61 seconds Running Fixie.Tests (netcoreapp3.0) 25000 passed, took 0.48 seconds
We’re optimizing for the developer’s own feedback loop. If my CI build fails, I want to be able to quickly scan the log for meaningful information without first having to parse that information out of the noise. When I’m running these tests locally throughout development 200 times a day, I’m going to care that I don’t even have enough time to get distracted while I wait, and I’m going to be more likely to actually run my tests with every code change. These simple removals and optimizations allow me to maintain my train of thought and focus on meaningful information.
.NET test frameworks integrate with Visual Studio’s Test Explorer, which has a few integration quirks.
Bitness: With other test frameworks, you have to select whether to run in 32 or 64 bit mode, each of which can only run compatible projects. You might have wondered why your tests aren’t showing up for you, while they do for your teammates, only to find that your bitness setting was flipped. Worse yet, if your solution has a mix of bitness (the result of opinionated dependencies), you may have to keep on remembering to toggle the setting to run all your tests (spoiler alert: you will forget). Fixie’s design allows us to ignore the setting entirely: your solution can have a mix of test projects of either bitness, and Test Explorer will discover and run them all side by side, regardless of the bitness setting.
UI Stability: Under Test Explorer, parameterized xUnit tests are unstable: unless your parameters are literal constant values like numbers or strings, some tests in the tree will not run when asked, and Visual Studio will log errors. Unit testing is all about establishing confidence levels, and instability of test running does not inspire confidence. By accurately mapping Fixie’s test/result model onto that of Test Explorer, the experience remains stable: all of your tests run on command, with no obstacles, UI artifacts, or logged error messages.
Stack Overflows: You simply aren’t allowed to catch a StackOverflowException. If your system under test causes one, most test frameworks can only crash out leaving Test Explorer holding the bag, giving you no understanding of what happened or even which test revealed your bug. Fixie’s design allows it to catch the uncatchable, reporting accurately which test killed the run and for which reason.
Since xUnit’s Azure DevOps support works through its integration with the framework-agnostic dotnet test command, the experience is a bit challenging. By combining their documentation, a linked github issue, and this Stack Overlow answer to navigate the subtleties of newline characters in YAML blocks, you can carefully assemble a “Visual Studio Test” task, using several wildcards to hopefully find your test assemblies while omitting unintentional matches of the /obj/ folder and of xUnit itself. Hopefully your task finds the right assemblies and all of the right assemblies, or your build will pass while you introduce bugs. The documentation also suggests that you should ignore the occasional AppDomainUnloadedException; hopefully your application doesn’t have a bug of its own that emits that exception, or you will have trained yourself to ignore it.
With Fixie, you first run the same dotnet fixie command in your CI build as you do locally. This alone will run your tests and report results to the captured console. More importantly, it will also detect that you’re running under Azure DevOps, and will suggest the exact simple line you need to add to your Azure DevOps pipeline in order to light up the full reporting experience, even linking to Microsoft’s own documentation on the subject for context.
Minimalism isn’t about getting rid of useful things, but about eliminating the unimportant to enable greater focus on the important. Fixie’s design eliminates much of the complexity found in other .NET test frameworks, enabling its laser focus on smooth integrations. By placing developer ergonomics as the top priority, the end user in turn can concentrate on the task at hand.