Hello Community,
I'm going through the Integration patterns, can anyone explain what are the steps involved in implementing the Queued real time sync pattern.
will data inserted to IS from the external system via API?
and to which database event does BPT from CS module react and process the data?
Thanks in advance,
Sai.
The Queued Real-Time Sync pattern is a common integration pattern used to synchronize data between systems in a near real-time manner while maintaining reliability and scalability. In this pattern, the data is synchronized through a message queue, which acts as an intermediary between the source system and the target system. Here are the steps involved in implementing the Queued Real-Time Sync pattern:
Message Queue Setup: Set up a message queue system (e.g., RabbitMQ, Apache Kafka) that acts as a buffer between the source system (IS - Integration System) and the target system (CS - Consumer System). Both the IS and CS should be integrated with the message queue system.
Data Insertion from External System: Data is inserted into the Integration System (IS) from the external system using an API or any other method of communication.
Enqueuing Data into the Message Queue: When data is inserted into the Integration System (IS), instead of processing it immediately, the IS enqueues the data as a message into the message queue.
Asynchronous Processing by CS: The Consumer System (CS) continuously listens to the message queue for incoming messages. When a message is available, the CS dequeues the message and processes it asynchronously.
Business Process Technology (BPT) Handling: The CS module, when processing the message, may involve the use of Business Process Technology (BPT) to handle the data. BPT allows you to define workflows and business logic to manipulate and process the data according to your requirements.
Database Event Trigger (Optional): In some cases, the CS module may not directly react to a database event. Instead, the message queue acts as an event trigger. However, if required, the CS module may use a database event trigger to capture data changes from the database and enqueue those changes into the message queue for processing.
To clarify, in the Queued Real-Time Sync pattern, data is inserted into the Integration System (IS) from the external system. The data is not directly inserted into the Consumer System (CS) database. Instead, it is enqueued into the message queue. The CS module listens to the message queue, dequeues the messages, and then processes the data, which may involve using Business Process Technology (BPT) or other logic.
By using this pattern, you decouple the integration process between the IS and CS, making it more scalable, resilient, and easier to manage, especially in scenarios where the CS may experience intermittent downtime or require some time to process the data. The message queue acts as a reliable intermediary, ensuring that data is not lost and the integration process remains efficient and robust.