[MQTT Web Client] How can I assure that I can handle multiple messages being received at the same time?

Forge Component
(7)
Published on 2019-06-05 by Barduino
7 votes
Published on 2019-06-05 by Barduino

Hello, 

I have created a Traditional Web App and I am using the MQTT Web Client. 

How can I assure that if I receive multiple messages at the same time, of the same topic, that I can hangle them (eg. write them in the database)?, Because what I noticed is that I lost some messages when this happened, which makes sense because screen actions are not multi threading, so, it can only handle one message at the same time.


Thank you,

João Lopes

Very interesting question Joao...

In theory it should raise an event for each message it receives.

I wonder if the message is actually being lost in the sense it doesn't reach the client or if OutSystems skips the event for each message.

How can we troubleshoot this? Do you have a reproducible situation?


If the message is not reaching the client, it could be a matter of fine-tuning the quality of service (QOS) parameter of the MQTT broker:

QOS 0 - Fire and forget, the client may or may not receive the message, (this is the default)

QOS 1 - The client will receive the message at least once, but multiple reception of the same message is possible

QOS 2 - The client will receive the message exactly once


Do remember that for any QOS setting other than 0 you must specify it both at the send and subscription.

On the connection you also need to set the use clean session parameter to false, so that the client receives all pending messages while it was disconnected.


Hopefully you are not using the public mosquito server (mqtt.mosquitto.org), it often plays tricks on us ;)


Cheers.

(these screen shots are from a mobile app but still apply)




Thank you for your reply.

I didn't know that about the clean session parameter. That's useful.

No, the client is receiving all of the messages (it is logging all of the messages on the javascript console).

The test I did was the following:

I had a for loop ranging i from 0 to 500. For each i, I would send a message with the payload i (this is running on an Arduino).

Next, imagine that I received the message with the payload "1". On the MessageReceivedEventHandler, I had a breakpoint (and was on debug). If I stayed for a couple of seconds on the breakpoint, and then clicked continue, the next message on the EventHandler would not have the payload 2, but the payload 300, for example. Does this test makes sense? What I thought was, if the MessageReceivedEventHandler has some logic that takes some time, wouldn't it lose messages. Also, if two messages arrive at the exact same time, wouldn't the same happen (messages being lost)?

Maybe this is not a problem and I am just confused. 


I am using the CloudMQTT broker.


Thank you, 

João Lopes

Hi Joao,

Multiple interesting concepts here, let's tackle one by one.

So I'm expecting this to have some kind of limit, the platform should make an AJAX request for each message (since this is web), so you'll find a limit for the number of AJAX request per second it can do, The platform should make those request in sequence though. 

So that is problem one. 

The behavior may also be different while you are debugging, I'm expecting to see multiple requests on the debugger but it will eventually fail.

I'll try to run some tests, lets say create 500 entries on a database and see how it behaves on these 2 scenarios.

However your use case is interesting.

Arduino, MQTT, database, sounds like your collecting data from sensors, some home automation project?

Any way collecting data using the javascript MQTT client might not be the best way, not only because of the eventual limit but also because you'll need a browser constantly working.

Usually on these use cases you need something in between, for example having the Arduino make a normal http call to a REST interface on the platform or have a nodeJS reading from Arduino and making the REST call if your Arduino only has serial.

This will raise the limit of messages per second but not enough for IOT scale. For this you'll probably need some time series database and then use the platform to consume its data.

So it all depends on the scale of your use case.

I was doing a home automation project a while ago and choose the nodeJS option, the Arduino would send the sensors data via serial and the nodeJS would make both the REST call to update the data and an MQTT message to signal the app something new had arrived or even some small confirmation data. On the app side it would send via MQTT user commands back to the sensors and actuators.

Either way the scale was never that big...

I never tested how many request per second this component could handle, I'll do some and in the mean time care to elaborate on your use case?


João Lopes wrote:

Thank you for your reply.

...

I am using the CloudMQTT broker.


Thank you, 

João Lopes


Hi Joao,

The test results could not have been worst.

I've created a couple of modules. One sends 500 messages with a serial from 000 to 500 and some random string and the other once it receives the message it creates an entry on a table with the payload.

All 500 messages get to the client and all 500 entries on the database are executed.

However the data gets all messes up, meaning I get message serial 1, then 6 repeated 5 times then serial 250 repeated and most of them with serial 500.

If I debug I get most of them with serial 500.

So although the javascript is fast enough to receive the messages and serialize the event, but something happens between the javascript and the platform event that the variable always has the last value available even if it still has pending events. This explains why most of the entries are the last message.


As I suspected this not a good use case for this component.

My suggestion is that you send the data via some REST interface and use MQTT to signal the app that something new has arrived.

Cheers