If you enjoy the subject of human cognitive biases, you should check out the curse of knowledge. When dealing with others, we tend to assume they know what we know. And we do this when no justification for the assumption exists.
Do you fancy a more concrete example? Take a new job and count how many people bombard you with company jargon and acronyms, knowing full well you just started a few hours ago. This happens because these folks cannot imagine not knowing these things without expending considerable mental effort.
Why do I lead with this in a post about unit test frameworks?
Well, it seems entirely appropriate to me. I earn my living as an IT management and strategy consultant, causing me to spend time with many companies helping them improve software development practice. Because of this, I have occasion to see an awful lot of introductions to unit testing.
And these introductions usually subconsciously assume knowledge of unit testing.
“It’s easy! Just pick a unit test runner and a coverage tool, and get those setup. Oh, you’ll also probably want to pick a mocking framework, and here are some helpful Nuget packages. Anyway, now just write a test. We’ll start with a calculator class…”
Today, I will do my best to spare you that. I have some practice with this, since I write a lot, publish courses and train developers. So let’s take a look at test frameworks.
Thought you’d caught me there, didn’t you? Don’t worry. I won’t just assume you know these things.
Let’s start with unit testing in its most basic form, leaving all other subjects aside. You want to focus on a piece of functionality in your code and test it in isolation. For example, let’s say that we had the aforementioned Calculator class and that it contained an Add(int, int) method.
Let’s say that you want to write some code to test that method.
public class CalculatorTester
{
public void TestAdd()
{
var calculator = new Calculator();
if (calculator.Add(2, 2) == 4)
Console.WriteLine("Success");
else
Console.WriteLine("Failure");
}
}
No magic there. I just create a test called “CalculatorTester” and then write a method that instantiates and exercises Calculator.Add(). You could write this knowing nothing about unit testing practice at all. And, if someone had told you to automate the testing of Calculator.Add(), you may have done this exact thing.
Congratulations. You have written a unit test. I say this because it focuses on a method and tests it in isolation.
Well, as you can imagine, having this sort of unit testing across your entire codebase could prove cumbersome. You’d write lots of classes just like this one and then… what? You’d look at all of the console output for failures, only to discover that failure proved pretty inscrutable.
Most likely you would then have the clever idea to include the name of the test in the output so that you could see what had failed. From there, you might start to add some sort of GUI-like feedback to the results, to present them in a more readable fashion. From there, who knows? Maybe you write a Visual Studio plugin or maybe you find a way to incorporate this test suite into your build?
Well, it turns out that if you did all that stuff, you would have built yourself a unit testing framework. Let’s take a look now at what some code written for an actual unit test framework (MS Test) looks like.
[TestClass]
public class CalculatorTests
{
[TestMethod]
public void TestMethod1()
{
var calculator = new Calculator();
Assert.AreEqual(4, calculator.Add(2, 2));
}
}
Unlike the last snippet, we have a bit of magic here. Notice the attributes, TestClass and TestMethod. Those exist simply to tell the unit test framework to pay attention to them when executing the unit test suite.
When you want to get results, you invoke the unit test runner, and it executes all methods decorated like this, compiling the results into a visually pleasing report that you can view.
So, with that background established, let’s take a look at your top 3 unit test framework options for C#.
When it comes to unit testing frameworks for C#/.NET, there are three ones you must know about:
You can use any of these to get the job done. However, each one has its list of pros and cons and a unique style to it. So that’s why now we’ll cover each one at a time, leading with MSTest.
MSTest has undergone a lot of change over the years. As is the case with many .NET-related tools, MSTest used to be tightly integrated with Visual Studio and supported exclusively by the Windows-only version of the .NET framework.
That is no longer the case. Nowadays, MSTest is an open-source, cross-platform framework that supports not only Windows but also Linux and OSX. Let’s take a brief history tour to understand what happened.
MSTest was actually the name of a command line tool for executing tests. The actual framework was something called the “Visual Studio Unit Testing Framework.” But veterans would colloquially call it MSTest, so that’s the name that stuck.
MSTest started shipping with some versions of Visual Studio around 2005. The lack of friction to getting started was arguably its killer feature at the time. The preferred .NET pattern for unit tests is to have a test project for each production (regular) project in your codebase.
With MSTest, getting that setup was as easy as File->New Project. Then, when you wrote a test, you could right-click on it and execute, having your result displayed in the IDE.
The fact that I’m using the past tense in the sentences above doesn’t mean that all of that changed and MSTest became super hard to use all of a sudden. On the contrary, it’s possibly way easier now than before.
But a lot of things changed in 2016, when Microsoft released .NET Core 1.0.
.NET Core was essentially an open-source, cross-platform version of the .NET framework. Miguel de Icaza described it then as “a redesigned version of .NET that is based on the simplified version of the class libraries.” In 2020, the “Core” branding was dropped, and since then, the framework’s name is simply “.NET”, in order to distinguish it from the Windows-specific product, .NET Framework.
.NET (the former “Core” version) ships with a CLI you can use to scaffold projects based on templates, build and run said projects, and run unit tests. As such, Visual Studio ceased to be an utmost necessity when working with .NET (unless you’re creating Windows Desktop apps). If you’re working with project types such as APIs, micro-services, or console applications, you could use any text editor you like—Visual Studio Code is a super popular choice nowadays.
This deemphasizing of Windows and Visual Studio meant that many tools had to change, and MSTest was one of those. Nowadays, you can easily use it outside of Visual Studio. Using the .NET CLI, creating a MSTest unit test project is easy as running the following command:
dotnet new mstest
So, what are the main pros and cons of MSTest? Let’s start with the good:
Now, for the cons:
Next up, I’ll talk about NUnit. Unit test frameworks have a history dating back almost 30 years, so they long predated .NET. NUnit started out as a port from Java’s JUnit, but the authors eventually redid it to be more C# idiomatic. So the tool has the rich history of unit testing behind it, but with an appropriately C# flavor.
Because of its independent history, NUnit also has the distinction of interoperating nicely with other tools, such as non-Microsoft build platforms and custom test runners. On top of that, NUnit also has a reputation for fast testing running, and it has some nice additional features as well, including test annotations allowing easy specification of multiple inputs to a given test.
In the past, the main downside is that it didn’t integrate into Visual Studio the way that MSTest did. Using it meant doing extra work, and installing extra tools, regardless of how easy those tools’ authors made the process. Nowadays, NUnit’s also a first-class citizen of .NET. With the CLI, creating a new NUnit test project is as easy as running dotnet new nunit.
Does NUnit have any cons? Sure, but not that many, to be honest. One is the potential lack of innovation. Pieces of technology that reach the degree of maturity that NUnit has reached often don’t add new features as aggressively as newer tools—that still need to prove themselves and gain a user base—do.
Another factor that draws criticism upon NUnit is its extensibility—or lack thereof. Although NUnit was intended to be extensible, it seems to offer fewer possibilities for extensibility when compared to, for instance, xUnit.NET.
The last option that I’ll cover is xUnit.NET. This one has a confusing name. Here’s why: the set of all unit test frameworks based on the structure, functionality, and terminology of SUnit—created for Smalltalk—is called xUnit. So, we can say that NUnit, JUnit (Java), PHPUnit (PHP), and MSTest all belong to the xUnit family.
As such, it’s not surprising that naming a specific tool after the whole category it belongs to is bound to generate some confusion.
From now on, I’ll stop mentioning the xUnit family of tools. In other words, I’ll simply use xUnit and xUnit.NET interchangeably.
One of the creators of the idiomatic version of NUnit went on to create xUnit. xUnit has a relatively innovative way for users to reason about their tests, dividing tests into “facts” and “theories” to distinguish between “always true” and “true for the right data,” respectively.
xUnit earns points for creating extremely intuitive terminology and ways to reason about the language of tests. On top of that, the tool has a reputation for excellent extensibility.
Another awesome feature of xUnit is actually not a feature of the software, but a feature of the authors. They have a reputation for commitment, responsiveness, and evangelism.
On the con side, some users seem to wish the tool had more documentation. Beyond that, the community seems pretty enthusiastic and it’s hard to find a lot of detractors.
Which framework should you pick? This might sound like a super anti-climatic answer, but at the end of the day, it doesn’t matter all that much. You see, nowadays, the differences between the frameworks aren’t as pronounced as they used to be years ago.
When it looked like NUnit would reign supreme, xUnit.NET appeared on the scene, shaking things a lot—in a great way. It introduced innovative features and new terminology that made developers think about unit tests in a fresh way. Since then, however, NUnit has bridged that gap to the point of offering features that didn’t become available in the competitor for quite a while, such as the Assert.Multiple assertion method.
A similar thing happened to MSTest. What was once a tool tightly coupled to Windows and Visual Studio is now a cross-platform, open-source tool that can easily be leveraged through the .NET CLI.
If I may suggest a course of action, I’d say: go with xUnit.NET. It seems to be the most popular right now if we go by NuGet download numbers (332 million for xUnit.NET vs. 262 million for NUnit for their latest stable releases.) It’s a powerful and flexible framework, and with the help of documentation and other resources, it isn’t hard to get started.
Trust me: you’ll probably love xUnit.NET. If you don’t, you can always try NUnit or MSTest. But regardless of the framework you pick, start testing ASAP, and you’ll see the health of your codebases improve like magic.
Stackify's APM tools are used by thousands of .NET, Java, PHP, Node.js, Python, & Ruby developers all over the world.
Explore Retrace's product features to learn more.
If you would like to be a guest contributor to the Stackify blog please reach out to [email protected]
