NUnit is an open-source, very flexible, and widely adopted unit testing framework for .NET. Regression tests verify that software previously developed still performs correctly even after it or its environment is changed. The goal is to discover any defects introduced or caused by updates and modifications. Regression tests are run regularly, from once a sprint to every night or, in some situations, even after each change.

This post is a guide to using the NUnit framework to support a regression test strategy.

Before We Get Started

We will use the BDD Framework as the entry point to OutSystems code, where the actual test is coded using Gherkin-like language. It is available in the Forge, and it provides a set of tools for producing BDD test scenarios. It is a prerequisite for the examples we will present.

As very well explained by Martin Fowler, the test pyramid is a way of thinking about different kinds of automated tests that should be used to create a balanced portfolio. The essential point is that you should have many more low-level unit tests than high-level UI tests.

Unit tests have several benefits, like faster execution speed, faster feedback, and lower maintenance costs.

The Test Pyramid

With all that in mind, you can follow these steps to use NUnit for regression testing.

Create the Test Project

We will use the Visual Studio 2017 Community to guide you through the creation and execution of an NUnit test. The source code is available in Github.

  1. Open Visual Studio and create a new project.
  2. The following screen appears.
  3. This screen will create a .NET Framework Unit Test (MSTest) project; that is a good start.
  4. Add a name for the project and a name for the solution.
  5. Press OK.
    • For demonstration purposes, the “Cases_Tests” application available in the Forge will be used.
    • The support for the Microsoft .NET Unit Testing Framework, MSTest, may be removed.
  6. To add NUnit support, select the Manage NuGet Packages option from the Project menu.
  7. The following screen appears:
  8. Select the Browse option at the top and search for “nunit.”
  9. Select NUnit from the list.
  10. For the version (in the right window), select 2.6.4, and press Install. (We currently recommend version 2.6.4 because it has a GUI that may be used for manual test execution; version 3 is having the GUI rewritten).
  11. The following window will appear; press OK.
  12. The installation steps are logged in the Output window:
  13. Our example requires an additional component: System.Web.Extensions. To add it, select the Add Reference option from the Project menu.
  14. The following window appears; select Framework under Assemblies on the left and put a check mark next to System.Web.Extensions. Press OK, and the support for System.Web.Script.Serialization is added.

Add an NUnit Test

Here’s how to add an NUnit test:

  1. When the test project is created, an initial MSTest class is also created: UnitTest1.
  2. Our goal is to create an NUnit test that invokes BDD Framework test scenarios, where the actual test is to make changes to the class until it has the following code:
    • See this testing guide to understand how to use BDD Framework for unit testing
    • In this example, the test eSpace is named Cases_UnitTests and is installed in the testenv environment; and the invoked test is named Case_CRUD:
  3. Build the project.
  4. Your NUnit test is ready.
  5. You may add other test classes (TestFixture) or other test methods (Test).
  6. You may create a utility library with the BDDTestSuiteResult class and the GetOutputMessage static method so no code is duplicated.

Execute Tests with the NUnit GUI

This how you can execute tests with the NUnit GUI:

  1. You may manually execute your NUnit tests from the NUnit GUI
  2. Go to www.nunit.org, and click Download (at the top of the page). Under Latest NUnit 2 Releases on the Download page, click NUnit 2.6.4.
  3. Download the NUnit-2.6.4.msi file (hosted in GitHub).
  4. Install NUnit 2.6.4 (that will install the NUnit GUI).
  5. Start NUnit GUI from the start menu.
  6. This will open the NUnit GUI:
  7. Select the New Project option from the File menu; the following window appears:
  8. Choose an appropriate name for the project and press Save.
  9. To add an NUnit test, select the Add Assembly option from the Project menu, as illustrated in this image:
  10. When the following window appears, navigate to the folder where the NUnit DLL built with Visual Studio is located. Select the correct DLL and press Open.
  11. The NUnit tests are extracted from the DLL and presented in the tree view like this:
  12. You may execute the test by pressing the Run button. This is an example of the execution:
  13. It the test fails, BDD Framework outputs the report with the scenario and step that failed (the following image shows that the Then step of the Close Case scenario failed because “Could not close Case”).

Execute Tests From Jenkins

You may trigger the regression tests from your Jenkins pipeline (or any other CI pipeline). We will illustrate this with Jenkins 2.107.2. The Jenkins NUnit Plugin must be installed (as well as its dependencies):

  1. Create a “Free Style Project”.
  2. Add the following Build step: Execute Windows batch command.
  3. Add the command “"C:\Program Files (x86)\NUnit 2.6.4\bin\nunit-console.exe" "<full path to your dll>" /xml=nunit-result.xml” (as illustrated in the following image).
  4. In the Post-build Actions area, add Publish NUnit test result report (also shown in the following image below).
  5. Press Save. The project will look like the following screen:
    jenkins regression tests
  6. To trigger this Jenkin project manually, press Build Now. (Jenkins Projects may also be triggered as a consequence of an external action like a successful commit of code, artifact promotion, or anything else.)
  7. After the build is executed, select the build number to see the results:
  8. If you press Console Output, you will see the execution of the build; it will look similar to this:

Debug NUnit Tests Inside Visual Studio

You may debug your NUnit tests inside Visual Studio as follows:

  1. To configure the debug, choose the <Project> properties option from the Project menu, and the following window appears:
  2. Choose Debug in the left nav; the following screen is displayed:
  3. In the Start External program field, enter the full path to the nunit-console.exe that was installed from the NUnit-2.6.4.msi file.
  4. In the Command Line Arguments field, enter the name of the DLL to test, as shown in the image.
  5. Press the Save button from the toolbar.
  6. You may debug the project by pressing ▶ Start from the toolbar.
  7. You may place a breakpoint in the code; if the execution does not stop at a breakpoint, then the nunit-console.exe.config file that is located in the same folder as the nunit-console.exe file has to be updated to support .Net 4.x, as shown here:

Final Considerations

This article introduces a method for regression testing using NUnit for .NET and BDD Framework. You could achieve similar outcomes with different technology, like Python and Cucumber, and the concepts would still be valid.

Want to Go Deeper?

If you would like additional information about testing OutSystems applications, we’ve got you covered. Check out these resources:

  • Here are some testing principles for OutSystems applications.
  • Here is guidance for unit testing and API testing.
  • Here is guidance for architecture best practices (that make testing easier).
  • Here is guidance for creating a build pipeline with Jenkins.
  • Here is the guided path for becoming a tester with OutSystems.
  • Here is a guide on how to use Selenium to do UI testing.