63
Views
4
Comments
Stream Outsystems platform logs to ELK

Hello everyone,

We have a centralized logging ELK stack where we want to store OutSystems platform logs. Please let me know how can I stream log data from Outysystems DB to ELK stack?

Thanks,


mvp_badge
MVP
Rank: #69

Hello Hemant.

You asked for different things.


Store OS logs in ELK: 

https://success.outsystems.com/Support/Enterprise_Customers/Upgrading/Keep_OutSystems_log_data_in_a_separate_database



Stream from OS to ELK:

Logs in OS are stored in different tables depending on the date.

https://success.outsystems.com/Documentation/11/Managing_the_Applications_Lifecycle/Monitor_and_Troubleshoot/Logging_database_and_architecture/The_log_tables_and_views

I would advise you to take the table when it is not being written on and copy all the data.


For streaming you probably want something more constant. I would do a SQL with NOLOCK to copy the data. Why in SQL? Imagine your log copying mechanism is done in OS and somehow fails That will write in the log. That means there is one more entry to send. That will cause another error... and loop.



Rank: #109

Hi Hemant.

OutSystems 11 enables you to store the log data in a separate database. Each OutSystems 11 application writes its own log entries using an internal logging API. This logging API collects log information produced by the application and sends it to the log database. Its "Asynchronous Logging API" provides few actions to perform on Database. Possibly you can create an API to trigger it with every event of LogError and LogRecord to post it to external service which is responsible to expose to centralized logging system. Checkout  the Log data reference


You may would like to read the OutSystems manual for Keep OutSystems log data in a separate database

Regards,

Swatantra

Staff
Rank: #5400

Hi everyone,


To current available ways(1) of getting OutSystems monitoring logs and performance events to Elastic stack(2) are:

  1. Direct Access to the DB and by using SQL Queries and Logstash fetch the logs and performance events and ship it to the Elasticsearch.
    1. If you want some artifacts to help on this, please check the one the OutSystems Professional Services DevOps team release in OutSystems Github. Available here:
      1. https://github.com/OutSystems/outsystems-elastic-integration
  2. Expose logs leveraging OutSystems System components then consume it or directly with Elastic or with Logstash if you want to "ETL" the data
    1. Here there also an artifact released by the same team spoken before. Check out: https://www.outsystems.com/forge/component-overview/4559/monitorprobe


Let us know if it works. We have been using in some customers with "GREAT SUCCESS" ;)


(1) Off course there are others, we are talking of IT ;) But this ones are the easiest ones in my perspective;

(2) What was called ELK.