What could be the best architecture pattern I can choose?
I need to Build a dashboard in OutSystems (refer screenshot), which should show real time data (keep it live as much as possible), Show pie chart based on the filter criteria.The dashboard data will be updated every 5 minutes to keep it live, wherein we are fetching the data using external rest API's.
Note: the external data base is having large number of records.
Also, any insights into any potential limitations or constraints we should be aware of when utilizing OutSystems in this case of huge data?
When building a real-time dashboard in OutSystems that involves displaying data from an external REST API, especially with a large number of records, you should consider an architecture pattern that efficiently handles data fetching, processing, and display. Here are a few recommendations for the architecture pattern and considerations regarding potential limitations:
Client-Server with Caching Mechanism:
Client-Side: The OutSystems application will be responsible for displaying the data on the dashboard. It will send periodic requests to the server to fetch the latest data.
Server-Side: Create a backend service (which could be an OutSystems server action or an external service) that interacts with the external REST API. This service will fetch and process the data.
Caching: Implement a caching mechanism on the server side to store the processed data. Each time the external API is called, the data is processed and cached. The client-side application then fetches this cached data, reducing the load on the external API and improving response time.
Data Processing and Aggregation:
Consider processing and aggregating data on the server side before it’s sent to the client. This reduces the amount of data transferred and speeds up the rendering of the dashboard.
Performance with Large Data Sets:
OutSystems can handle large datasets, but performance might be a concern, especially if data processing is complex. Consider optimizing your data fetching and processing logic to handle large datasets efficiently.
API Rate Limiting:
External APIs often have rate limits. Ensure that your application respects these limits. Implementing caching as suggested above can help mitigate this issue.
Data Refresh Rate:
Although you plan to refresh data every 5 minutes, consider the impact on the server and client performance. Test different refresh intervals to find a balance between real-time data representation and system performance.
Scalability:
As your application grows, both in terms of data and user base, ensure your architecture is scalable. OutSystems supports scalability, but it requires proper planning and implementation.
Error Handling:
Implement robust error handling, especially for scenarios where the external API fails or is slow to respond.
Security Considerations:
Ensure secure data transmission, especially if sensitive data is involved. Implement appropriate authentication and authorization mechanisms.