Integrating a Rules Engine

  
I am looking for one or more rules engines that can be integrated into the OutSystems platform.
 
The requirements are for a rules engine that can be programmed using Boolean logic for forward and backwards chaining.
 
Consider the following: 

A+B=C 

C+D=E 

In a RETE based engine these rules are sitting in the 'rules engine' fully instantiated in the CPU -- that is, the rules engine is 'state full'. 

When A changes, the rules engine immediately recalculates C. Now, because C has changed, it immediately recalculates E. This is forward chaining. 

Backward chaining is the inverse. I want to know what E is, so I need to walk backwards and calculate C+D; but this means I want to know what C is, so I need to walk backwards and calculate A+B. 

The rules engines are 'state full’, ie discrete data sitting in an engine designed to quickly interpret them each time an event is raised (either by forward or backward chaining, depending on the nature of the task).  
 
So forward chaining is useful for network monitoring and workflow tasks; backward chaining for dynamically calculating a desired result by walking backwards to its inputs. 

Has anyone integreated a rules engine into the OutSystems platform?
Hello Peter,

I've used a rules engine in the past and the integration was trough web services. Others have different ways of integrating... I think that the most important knowledge that the community can pass along is the ways that you have to integrate with external systems. The most important options you have here are:

- REST Webservice consumption;
- SOAP Webservice consumption;
- Integration by using a DLL/JAR that communicates with the rules engine.

Does your rules engine exposes any of this, or other?

Cheers
Pedro Cardoso

That is fine -- but for me it takes away one of the major reasons why I want to use the OutSystems platform -- the applications I want to build are very rules intensive -- I am looking for a way to simplify and reduce the effort in developing the systems -- hence, having the rules engine on the 'inside' would be a real advantage.

Regards
Peter
What do you mean in the inside? Like having the Rules Engine accessible through "an orage ball"?


Never heard of an "an orage ball" -- is that some strange Americanism?  Google has never heard of it, either.  Please explain.
Pedro is talking about an Outsystems Action.  Actions in Outsystems have an orange ball as its default icon.
Thanks -- I was puzzled -- but mainly by the misspelling.

What I am looking for is a rules-engine that can be built into OutSystems and available for all applications running on the platform.

It appears that bobody has found a rules-engine that can be applied in this way.

There are many rules-engines available, eg: JBoss Rules (Drools), InRule, XpertRule, ExsysCORVID, OpenRules -- so I am wondering why there are no instances of them being integrated into the OutSystems platform.
Hello Peter,

sorry for the misspelling...

There are projects that integrate Rules Engine into the platform. And they become available to all applications in your environment, just like any other system that you integrate with.

I've worked in a customer's project that integrates with a Rules Engine through Web Services, and it's pretty much just calling webservices and getting replies back... no big issue.... After that you can even encapsulate the calls in some module and make it really simple to other apps to use it...abstract complexity, simplify calls... all of that!

Do you have a specific one you want to use?

Cheers,
Pedro Cardoso


Hi Peter,

The key here is that your rules engine is stateful while the web requests on the web paradigm are stateless (and they are also in OutSystems), so there is no way for you to keep a single rules engine instance available across requests. And I can imagine that keeping the rules on the database and load them on every single call, and also dump the rules to the database after execute each call is not a best solution performance-wise.

The solution I'm thinking of would be to launch the rules engine as a service on the front end machine and implement a socket based API (to be as lightweight as possible) to handle the calls done by the platform. Then use integration studio to implement an extension that would interact with such API. If you have more than one front-end you would make this service available on a single node to be called by the several nodes.

Hope this helps.

Cheers,
Bruno Gonçalves
That is making a lot of sense -- but, I remember seeing several requests in the OutSystems Community for a rules-engine to be included in the OutSystems platform itself to reduce the complexity of this approach of using a 'service' from an external rules-engine --- in cases that are heavily rules intensive (with, thousands of rules) there is a significant problem of degradation of performance by using a services arrangement -- there is going to be a lot of data going back and forth -- i'm looking for something that will maximise efficiency.

Should I be looking at this the other way around -- as a rules-engine problem that calls the procedures on the outsystems platform when necessary??

The pros and cons of using the OutSystems platform then depend on what the platform can do in terms of automation for the rules-engine platform.

My main interest in the OutSystems platform is in its capacity to automate and efficiently manage and coordinate a set of highly specialised scientific statistical applications that operate externally to the OutSystems platform (I have not at this stage seen how each of these can be programmed in the OutSystems platform to any advantage), and to support them by some bespoke procesess developed on the OutSystem platform itself. 

My preference is to integrate what are currently external systems (including the rules-engine, and a set of specialist statistical modelling procedures) into the OutSystems platform --- has anyone used the OutSystems platform to build a system like this?? 

Any ideas?

Cheers
Peter
Peter,

Can you elaborate on this: "My main interest in the OutSystems platform is in its capacity to automate and efficiently manage and coordinate a set of highly specialised scientific statistical applications that operate externally to the OutSystems platform"

About this sentence "I have not at this stage seen how each of these can be programmed in the OutSystems platform to any advantage" are you thinking that those systems could be built with the Platform? What do they do?


The application will manage a set of external processes through the implementation of a specific set of rules -- a simple rules system can be configured as a workflow, but that becomes unweildly once the number of rules to be selected from at each stage is more than a handful.


The following is a quick overview of the functionality that will be provided by the application.

1.     A rules-driven ordinary-language algorithm queries the user for input to determine the user's requirements.
2.     Templates are selected to specify the data to be extracted from a series of DBs.
3.     Data is extracted from multiple sources (external to the immediate operating environment) and stored in a working DB.
4.     A specialist external statistical modelling and statistical testing application is triggered to undertake the analysis using pre-defined templates (written in the statistical language used by the statistical analysis engine) – with the output saved to an external SQL DB.
5.     The Manager Application will poll/watch the DB and when the output data is available, it will trigger a backwards-chaining AI algorithm to generate an interpretation of the data, and save the interpretation to the external DB.
6.     Once the interpretation is saved, the Manager Application will trigger another external application to produce a series of interactive reports tailored to the needs of the individual members of the target audience.
7.     Finally, the Manager Application will notify the user that their report is ready for use.

 

Most of the above steps will involve a series of procedural rules that select key resources (e.g. the templates) and trigger the next step in the process.

The following provides an overview of the components that will incorporate a rules-engine.

User Interface 

1.     The UI will engage the user in a dialogue to work out the nature of the question they want answered by the analysis of specific data.
•  The user will be presented with a sequence of questions, with each following question being determined by the combination of responses to the preceding questions.
•  The rules engine will compare the sequence of user responses with a set of pre-specified models of the types of analysis that can be undertaken — when the rules-engine considers that the user’s input matches a pre-specified model (or possibly multiple models with a high degree of certainty) the application will present the Analytical Model (as a diagram with causal/directional links between the key concepts in the model) to the user and ask them to confirm that it has interpreted their requirements correctly.  If the user responds in the negative, the application will ask further questions, until it has received confirmation from the user that it has interpreted their requirements correctly.

The above could be programmed as a business process workflow in the OutSystems platform -- tedious, but doable, and would probably require a substantial amount of input to draw up the few hundred pathways that would be required.

Data Selection 
2.     Following the determination of the model that fits the user’s question(s) the rules-engine will then specify the data required to analyse the model.
•  The first time that a specific data element is required, the user will be asked to point the application to the data — the data my be held anywhere inside or outside the user’s network.  
•  Once the data path has been identified, it will be stored for future use.

The data selection process can be managed in the OutSystems platform as a series of workflow processes.

Data Preparation 
3.     Each type of data will be passed through a specific set of processes to test and ensure that it is of the validity and reliability required — this typically involves using data from several sources to test validity, and where necessary, imputing more valid values for specific data cases.
•  The rules-engine will select a set of pre-specified sub-templates (specialised data processing and statistical procedures) to specify the required set of processes that will form the Data Template.
•  The Data Template will be processed by the R-Statistical Package — the rules-engine will interpret the messages and output produced and trigger actions that are required to undertake the data processing to the standard required. 

The rules for selecting the data processing sub-templates need to be handled by the OutSyatems application — however, the work will be undertaken by the R package.

Statistical Modelling and Hypothesis Testing 
4.     Each Analytical Model specified in Step 1 will have a parallel Statistical Model for analysing the data that is processed through the Data Template.
•  The data produced from the Data Template will be fed to the Statistical Model as it is submitted to the R-Statistical Package — the rules-engine will interpret the messages and output produced and trigger actions that are required to undertake the statistical modelling to the standard required. A series of iterations may be required to achieve the quality of results that are required.
•  The rules for the Data Template will be generated by a small team of R-Statistics application specialists, based on the initial applications and continually built as required.

The rules for selecting the Statistical Modelling templates will need to be handled within the OutSystems application, as they derive from the definition of the Analytical Model.

The OutSystems application will monitor and manage changes to the definition of the selected Statistical Model in response to signals and the output from the model — each model is typically run 500 times to obtain the optimal estimates. If the distribution of the estimates includes a high level of 'inadmissable' solutions, rules will be invoked to implement changes parallel to those a professional statistician would normall make manually, before submitting the model for reprocessing. 

Interpretation of the Results of the Statistical Modelling
5.     The output from the Statistical Modelling will be saved to a DB, following which the rules-engine will ‘interpret’ the output and ‘translate’ it into 'plain English statements’.  
•  These statements will be backed up by statistical tables and graphs that provide the user with access to the evidence providing the ‘chain of reasoning’ that sits behind the 'plain English statements’.
•  The specialist rules for interpreting the output from the Statistical Modelling will be generated by a small team of R-Statistics application specialists, based on the initial applications and continually built as required.

This is probably the most significant challenge in terms of the rules-engine, as it will require a backward-chaining approach that evaluates the pattern of estimates for the statistical paremeters in the model. Each parameter will have two properties -- its numeric value, and the test of statistical significance associated with a specific hypothesis (eg. is the parameter value for one set of data statistically different to that for another set of data). The number of parameters in the pattern to be evaluated will comprise a matrix with a dimension of up to 50 x 50. Given this, it is not feasible to threat this rules-problem as one that can be built from a multiplicity of workflow systems -- the solution would not be computationally efficient, which is a key to why i am looking to integrate the rules-engine into the base code of the application.

The above suggests that we should probably start by positioning the rules-engine and the statistical-engine at the core of the application, and then consider which parts of the workflow process can be built in the OutSystems platform to coordinate the services provided to the core.

In this scenario, the evaluation of whether OutSystems provides value-for-money comes down to the question of how much of the coordination of the services for the core can be managed by OutSystems workflows, plus the effort required to build the interface that wil be required between OutSystems and the rules- and statistical-engines.
Pedro Cardoso wrote:
Hello Peter,

I've used a rules engine in the past and the integration was trough web services. Others have different ways of integrating... I think that the most important knowledge that the community can pass along is the ways that you have to integrate with external systems. The most important options you have here are:

- REST Webservice consumption;
- SOAP Webservice consumption;
- Integration by using a DLL/JAR that communicates with the rules engine.

Does your rules engine exposes any of this, or other?

Cheers
Pedro Cardoso


Hi Pedro

You will see from my most recent posting that I am coming around to the view that you expressed earlier (see above) -- however, placing the rules/statistical engines at the core of the application substantially reduces the value proposition of taking up OutSystems as it would then be handling only a series of very simple workflow processes.

We have not determined which rules engine to use -- the three primary considerations are: (1) price, (2) programmability by a business analyst, and (3) whether it provides both forward and backwards chaining.

You indicated that you are aware of projects involving rules-engines -- which do you suggest would be a good fit given the criteria I has indicated above.

Regards
Peter 
 
Hi Peter!  We have built a financial "modeling" tool (no external services) that provides several outputs based on a number of different inputs and logical scenarios provided by a user; When the outputs are displayed to the user, they have an opportunity to work backwards to see the detail of the contributing factors to the output (e.g. a forecasted cost amount is driven x% by a labor category and y% by deprecated value of an asset) and modify the input factors on the fly to visualize the impact to the outputs - then of course commit the changes if they choose. 

We can, but haven't yet, automated the use cases to run say, 100 at a time with fixed variations to the inputs but that most certainly can be done.

Not sure if that makes any sense or if that's the type of thing you're trying to accomplish...if it is let me know and I'd be happy to help!