This section details features and limitations specific to the BDDFramework client-side tool. For general information about BDD testing with OutSystems, see the Overview tab.

We'll further detail these topics, which are specific to the BDDFramework client-side tool:

• Tagging
• API v2
• FAQs
• Limitations

Tagging

New functionality allows you to create tags, and then during development you can associate one or more tags with test scenarios. These user-defined tags allow you more granular control of your test portfolio. You can use tags to identify test characteristics such as:

• Priority or criticality
• Complexity and speed
• Purpose (business rules, API, DB consistency, or error handling)
• Test mapping to a specific part of your application

Creating Tags

Tags are a new block in the BDDFramework. When using the BDDFramework application template, a tag template is included in your initial module. Tags should be generic and reusable. Define each tag in its own block and then add it to relevant tests. Additionally, create tags in a separate UI Flow (for example in TestTags in the following example).

We recommend creating a new tag from the template. To do so:

1. Copy the Template_Tag block into your new TestTags flow.
2. Rename the copied block. For example, if this tag is to identify critical tests, call it CriticalTag.
3. Select the TagLabel local variable inside the block and provide a name. The name defines your tag and is used to filter tests to execute. In the previous example, the value could be Critical.

Adding Tags to Scenarios

The BDD Scenario screen has a Tags placeholder, to which you can drag tags for a test scenario. The Tags placeholder is between the scenario Description and Setup placeholder files. You can add one or more tags to the Tags placeholder.

Adding tags to scenarios is easy. You simply search for the tag in the TestTags flow and drag it to the Tag placeholder in your scenario block. And you're done!

Using Tags during Test Execution

You can use tags to specify tests to execute or skip. The new API accepts two new optional parameters in the header:

• SkipTags--A comma-separated list of tags that specifies tests to skip. This parameter supersedes ExecuteTags (That is, the same tag in both SkipTags and ExecuteTags is skipped and not executed).
• ExecuteTags--A comma-separated list of tags that specfity tests to execute. With no tags specified, all tests execute, except those included in SkipTags.

Tests execute based on the following logic.

• A test executes when:

  • It has no tags in the SkipTags parameter
  • AND it has at least one tag in the ExecuteTags parameter OR the ExecuteTags parameter is empty

• A test is skipped when:

  • It has at least one tag in the SkipTags parameter
  • OR it does not have any tags in the ExecuteTags parameter AND the ExecuteTags parameter is not empty

API v2

Based on server-side tool feedback, we've added a new version of the execution API that provides more information about test execution. The existing API (v1) is still compatible as well. Let's take a look at what's new with API v2.

Specifying Tags

The API v2 supports these two new header parameters, which are described earlier in this article:

• SkipTags
• ExecuteTags

API output changes

The API v2 output includes the new output attributes defined here:

{
  "SuiteScreen": "string",
  "IsSuccess": true,
  "SuccessfulScenarios": 0,
  "FailedScenarios": 0,
  "SkippedScenarios": 0,         // number of tests skipped, as specified in the SkipTags parameter
  "TestScenarioResults": [       // list of tests that exist inside the test screen (executed or not)
    {
      "ScenarioId": "string",    // scenario ID, as specified in the "Scenario Identifier" placeholder
      "Description": "string",   // scenario description, as specified in the "Scenario Description" placeholder
      "IsSuccess": true,         // indicates if this scenario was successful or not
      "IsSkipped": true,         // indicates if the scenario was skipped, as specified in the SkipTags parameter
      "FailureReport": "string", // given-when-then detail, which appears only if a scenario fails
      "Tags": [                  // list of all the tags associated with the scenario
        "string"
      ]
    }
  ],
  "ErrorMessage": "string"
} 

Enabling Test executions through the API

The client-side tool requires that you enable the API by installing a chromium instance on your OutSystems server. This post-installation step is not required for the server-side tool.

After installing your BDDFramework client-side tool, install chromium:

1. Go to https://<your_server_name>/BDDFrameworkMobile/Welcome, where <your_server_name> is the name of your OutSystems server

2. Confirm whether chromium is installed. Initially, it should not be.

3. Optionally, change the folder where chromium is installed. You do this in the ChromiumInstallPath site property in the TestRunner_API module.

4. Click Install Chromium.

Why do I need Chromium?

Chromium is required on the server to execute tests through the API because the execution flow of the BDDFramework client-side tool is javascript-based. It requires a javascript engine on which to execute.

When the BDDFramework client-side API is called, internally it uses Puppeteer to interact with the browser, execute tests, and then read test results. Puppeteer, which requires chromium, enables test execution through APIs. In the client-side BDDFramework tool, we use the .NET port of the Puppeteer node library, Puppeteer Sharp (.NET port used by the tool).

Though technically possible, we opted not to embed a chromium image as a DLL in an OutSystems extension because the chromium image size (~140Mb) would make the BDDFramework tool too large.

FAQs

Why can’t I have one BDDFramework tool for both server-side and client-side testing?

Initially, our goal was to use the new tool to cover both server-side and client-side testing. However, we found significantly slower performance (up to 19x slower) for server-side testing with this new tool. Performance for client-side testing, however, is comparable with other javascript testing platforms. Use this tool for clent-side testing, and continue to use the separate server-side tool for server-side testing.

Performance degradation for server-side testing occurred for these reasons:

• The BDDFramework client-side tool runs in a browser's javascript engine; javascript executes more slowly than compiled .Net code
• Server-side calls add latency
• Tests executed through the API repeatedly launch chromium instances, which adds overhead

Can I use this tool to test client-side logic in mobile and Reactive Web Applications?

Yes. Internally, BDDFramework for client-side testing is built as a mobile application. Through inheritance, your test applications are also mobile applications. In OutSystems, mobile applications can reference client-side actions and entities from both mobile and Reactive Web Apps. Consequently, you can test both mobile and Reactive Web client-side logic.

Limitations

The client-side testing tool has a few limitations compared with the server-side tool, as noted in this section.

Execute one step multiple times inside a scenario

Though an edge case, note that iterating a BDDStep block with a ListRecords widget inside the BDDScenario block is not supported. Unpredictable results, including interrupting test execution, may occur. Here’s an example of the unsupported structure:

| BDDScenario
| - BDDStep 
| - BDDStep 
| - | ListRecords <-- Not Supported 
| - | - BDDStep 
| - BDDStep

This scenario differs from a data-driven one, where you iterate one BDDScenario block with a ListRecords widget. This type of data-driven scenario is supported.

Mixing data-driven and simple scenarios in a single screen is not supported

You can’t have one test screen with both simple and data-driven scenarios. If you create a test with a data-driven scenario, it should have its own screen. We recommend the same approach with server-side testing, though technically the server-side can support mixing these two scenarios.

Execute through the API in a personal environment

In a personal environment, executing tests through the API is not supported. This limitation exists because personal environments do not include necessary permissions to install chromium on the server.