BDD Testing Framework

  

Hi everyone,


I've just published a new component to the forge called the BDDFramework, which provides a set of tools for easily creating BDD / TDD style tests for your OutSystems Applications.

The main focus is testing the logic of your modules, by exercising the critical actions that support your app's use cases.


You can check this demo video where we show how to use the framework to create a test for a piece of logic of the eCommerce app.


Main highlights of the component:


  • Create test scenarios and steps that are conformant to the principles of Behavior Driven Development.
  • Enable Test Driven Development - you can define scenarios and steps early in the app design stage and write the test code when the features have been implemented.
  • Easily add scenarios and steps by dragging and dropping web blocks (Scenario + Given / When / Then clauses).
  • Each step is implemented in its own action, making sure that steps are sequential and occur in the same request.
  • Enhance Test Maintenance - test outputs are identical to what is designed in Service Studio, making it easy to pinpoint step failures and evaluate the code behind them.
  • Supports multiple tests in the same page and evaluating final stats (number of successful / failed tests).
  • Open-source component which you can improve and change to your own needs.


Hope you find it useful!


Cheers!


 Hi  Joao,

This is indeed a great tool, I'm a new Outsystems user and I was able to create a small test. 
I think this is a good easy to use tool for unit test level for Outsystems applications that give the user the above mentioned benefits. So thank you for this!

Question: BDD tools supports also specification by example. I mean her support of parametrization and data in table format, similar with scenario outline in SpecFlow and Cucumber. 

 see below

Scenario Outline: eating
  Given there are <start> cucumbers
  When I eat <eat> cucumbers
  Then I should have <left> cucumbers

  Examples:
    | start | eat | left |
    |  12   |  5  |  7   |
    |  20   |  5  |  15  |

Do BDDFramework support this feature?
any plans for this?
Hi Rachid!

Thank you for your feedback!

I had not thought of the scenario you mentioned, but I believe you can achieve something quite compatible in the OutSystems Platform in a easy way through a web block.

If you take your example: you could create a new web block (let's call it "Eating") in the eSpace where you are building your tests and add three input parameters - Start, Eat, Left.
You can then drag a BDDScenario block (from the framework) inside the Eating block and add the Given, When, Then clauses you mentioned, with the corresponding actions for each step.

You'll get an "Eating" block that you can reuse several times inside a test screen - for each instance of the "Eating" block you set different values to the Start, Eat and Left parameters to test different values.

Hope this helps.

Cheers!
Hi João,

Thanks for your reaction. With your solution based on web block, we can indeed realise my proposed feature. However it is not completly the idea that i want to achieve.

My idea is to extend your tool with a component that can be re-used in such a way that table and the data becomes visible in the scenario outline as mentioned in my previous post. my idea is to achive a readbale specification by example.
With the solution you proposed, the data will not be visible in the scenario's but will be hiden in the code.

One principle of BDD and specfification by example is that the test cases (in Given-When-Then + table outline) becomes a commen readable specification. and can be used as an ubiquitous language, a (semi-)formal language that is shared by all members of a software development team — both software developers and non-technical personnel. The language in question is both used and developed by all team members as a common means of discussing the domain of the software in question. In this way BDD becomes a vehicle for communication between all the different roles in a software project.

Kind regards

Rachid
 

Hi Rachid,

I see what you mean with the "data" for the test scenario and how it helps in terms of specification by example. 
It is indeed an interesting concept that we didn't consider when developing the BDD Framework, but if you end up extending the component in such a way, please do share your experience with us!

Cheers!

The component is a good starting point for automating tests, a feature that is long overdue. However, Rachid's idea would be the cherry on the cake, or the minimum to put the component in a real and robust use case!

Hi everyone,


Here's an update: a new version of the BDD Framework has just been published that includes a REST API for running tests.

You'll still develop tests the same way you've done up until now, but you also have the option to run them through this new API, enabling a more direct integration to other tools you may already use to manage your tests.


Once you publish the new 1.2.1 version of the component, check out the REST API documentation at 

https://<YOUR_ENVIRONMENT_HOST>/BDDFramework/rest/v1/#!/v1/


You'll see there's a single API method called BDDTestRunner you can call, passing the name of the eSpace that contains the tests and the corresponding test suite screen (through "rest/v1/BDDTestRunner/{TestESpace}/{TestSuiteScreen}").


Let's say you use the API to successfully run a single test contained in a test screen. The output you should obtain is a JSON such as the following:

{  

   "SuiteSuccess":true,

   "SuccessfulScenarios":1,

   "FailedScenarios":0,

   "FailureReports":[  ],

   "ErrorMessage":""

}


However, if you run another screen that has 2 BDDFramework tests, in which one is successful and the other fails, you'll obtain something like this:

{  

   "SuiteSuccess":false,

   "SuccessfulScenarios":1,

   "FailedScenarios":1,

   "FailureReports":[  

      "------------------------------------\r\nScenario\r\nWhen a user adds a product to the cart, it should be correctly updated\r\n------------------------------------\r\nGiven\r\nGiven that I have a cart [P] \r\nAnd there is a product called \"Prosecco Armani DOC\" [P] \r\n\r\nWhen\r\nWhen I add the product to the cart [P] \r\n\r\nThen\r\nThen the operation should be successful [P] [F] [P] \r\n    [F] Expected \"Message: 1 Barca Velha added to your shoppping cart\" but obtained \"Message: 1 Prosecco Armani DOC added to your shoppping cart\"\r\nAnd the cart should have been correctly updated [Skipped]\r\n------------------------------------\r\n\r\n"

   ],

   "ErrorMessage":""

}


The data includes a failure report with the stripped down BDD output from the test that failed. Once you unescape that report:

------------------------------------
Scenario
When a user adds a product to the cart, it should be correctly updated
------------------------------------
Given
Given that I have a cart [P] 
And there is a product called "Prosecco Armani DOC" [P] 

When
When I add the product to the cart [P] 

Then
Then the operation should be successful [P] [F] [P] 
    [F] Expected "Message: 1 Barca Velha added to your shoppping cart" but obtained "Message: 1 Prosecco Armani DOC added to your shoppping cart"
And the cart should have been correctly updated [Skipped]
------------------------------------


You'll see that the information is the same you would see by running the test in the screen itself through a browser:

We have found this to be useful when integrating with other tools, since you can include the full test output for each test run report, thus accelerating the analysis of test failures.


Happy Testing!


João Proença wrote:

Hi everyone,


Here's an update: a new version of the BDD Framework has just been published that includes a REST API for running tests.

You'll still develop tests the same way you've done up until now, but you also have the option to run them through this new API, enabling a more direct integration to other tools you may already use to manage your tests.


Once you publish the new 1.2.1 version of the component, check out the REST API documentation at 

https://<YOUR_ENVIRONMENT_HOST>/BDDFramework/rest/v1/#!/v1/


You'll see there's a single API method called BDDTestRunner you can call, passing the name of the eSpace that contains the tests and the corresponding test suite screen (through "rest/v1/BDDTestRunner/{TestESpace}/{TestSuiteScreen}").


Let's say you use the API to successfully run a single test contained in a test screen. The output you should obtain is a JSON such as the following:

{  

   "SuiteSuccess":true,

   "SuccessfulScenarios":1,

   "FailedScenarios":0,

   "FailureReports":[  ],

   "ErrorMessage":""

}


However, if you run another screen that has 2 BDDFramework tests, in which one is successful and the other fails, you'll obtain something like this:

{  

   "SuiteSuccess":false,

   "SuccessfulScenarios":1,

   "FailedScenarios":1,

   "FailureReports":[  

      "------------------------------------\r\nScenario\r\nWhen a user adds a product to the cart, it should be correctly updated\r\n------------------------------------\r\nGiven\r\nGiven that I have a cart [P] \r\nAnd there is a product called \"Prosecco Armani DOC\" [P] \r\n\r\nWhen\r\nWhen I add the product to the cart [P] \r\n\r\nThen\r\nThen the operation should be successful [P] [F] [P] \r\n    [F] Expected \"Message: 1 Barca Velha added to your shoppping cart\" but obtained \"Message: 1 Prosecco Armani DOC added to your shoppping cart\"\r\nAnd the cart should have been correctly updated [Skipped]\r\n------------------------------------\r\n\r\n"

   ],

   "ErrorMessage":""

}


The data includes a failure report with the stripped down BDD output from the test that failed. Once you unescape that report:

------------------------------------
Scenario
When a user adds a product to the cart, it should be correctly updated
------------------------------------
Given
Given that I have a cart [P] 
And there is a product called "Prosecco Armani DOC" [P] 

When
When I add the product to the cart [P] 

Then
Then the operation should be successful [P] [F] [P] 
    [F] Expected "Message: 1 Barca Velha added to your shoppping cart" but obtained "Message: 1 Prosecco Armani DOC added to your shoppping cart"
And the cart should have been correctly updated [Skipped]
------------------------------------


You'll see that the information is the same you would see by running the test in the screen itself through a browser:

We have found this to be useful when integrating with other tools, since you can include the full test output for each test run report, thus accelerating the analysis of test failures.


Happy Testing!



This looks very interesting, Joao. Amazing discussion here. Some of us in Atlanta actually had a session/training with Grorge Dinwiddie on BDD and are excited to put that to use.

I am interested in synching up with you. Please look forward to an invite on your calendar!

Regards,

Sameer

1.What can i do for mobile app testing?.

2. i need to test in real device.?

3. how to test explain?