How we ditched HTTP and transitioned to MQTT!

In the field of location tracking, there needs to be lot of back-and-forth communication between devices and the backend. Device transmits location, activity and health data. Then the backend processes this information, applies business logic on top and sends configuration commands back to the devices in order to orchestrate tracking. These configuration commands determine when to start/stop tracking, frequency at which to collect GPS data (time and distance), frequency at which to transmit GPS data and so on.

In a world with patchy mobile networks, making all this communication robust is quite a task. It is important to choose the right network protocol and design the communication semantics to get maximum benefit of the protocol’s capabilities. We recently switched a large part of our device-backend communication from HTTP to MQTT. This blog is about how we achieved it and our learning from it so far.

First of all here’s why we decided to move to MQTT:

• MQTT is much more battery-efficient (than HTTP) in maintaining long running connections and transmitting data periodically
• With MQTT, we now have a real 2-way connection where server can push commands to the clients whenever required

With these goals in mind, we setup a Mosquitto broker on an EC2 instance. We combined header and body of our regular HTTP request into a single MQTT message that devices send to the Mosquitto broker. We made a simple MQTT topic scheme that maps 1:1 with our API endpoints. So far so good.

Now the challenge was processing these large number of incoming messages and POST’ing them from the Mosquitto broker to our API server. We thought of writing a subscriber to Mosquitto which would parse these incoming messages and make API calls for each message. But when this subscriber is deployed onto multiple hosts, each of those hosts would get a copy of EVERY message. We would then have to make sure that only one subscriber is making the POST call in response to a given message. To achieve this, we would have needed to maintain state across subscribers and so on.

This is when we stumbled across AWS IoT and it’s powerful rule engine. At the core of it, IoT has its own MQTT broker and you can define custom rules to process each incoming message. E.g. You can store the incoming message into S3/DynamoDB push a notification on SNS/SQS or write a lambda function that will be invoked in response to each incoming message. We created a bridge from our Mosquitto broker to AWS IoT so that messages on Mosquitto are forwarded to the IoT. We defined rules so that our own lambda function is invoked in response to each incoming message on a topic. The lambda function simply parses the message, takes out header and body, and makes a usual HTTP POST call to the API server. This bridging approach has given us simplicity of Mosquitto and flexibility of IoT’s rule engine. Here is a diagram of our current architecture.

Here are some of our learnings so far:

• Quality of Service (QoS) selection: MQTT provides three QoS levels (0, 1, 2). This is super useful because the protocol handles re-transmission and guarantees the delivery of the message even when the underlying network is unreliable. It is important to choose appropriate QoS levels for different messages in your application. E.g. we chose QoS 0 (fire and forget) for transmitting health because losing the occasional health message does not affect us. But we use QoS 1 (delivery  guaranteed at least once) while sending configuration commands from server since we cannot afford to lose the commands and our clients can handle duplication.
• Retain messages: Another neat feature of MQTT is that the broker can retain last message on each topic. In a rightly designed topic scheme, retained messages can act as a status information for the client. E.g. Our topic scheme for sending configuration commands is in the form of ‘Push/SDKControls/<driver_id>’ so whenever a driver comes online he receives the last command sent by our API server as the last message for him is retained at the broker. This makes the protocol asynchronous by relaxing the requirement for client to be connected to network when message is sent from server. We can simply fire the message from our API server and just rely on the MQTT broker to forward the message when clients connect back to network.
• Processing MQTT messages: While it has been super easy to setup the Mosquitto broker and get the messages flowing into it, we did not find any libraries which allow rule engine type processing of these messages. Maybe there is a tooling gap in the MQTT ecosystem. Hence we chose to forward the messages onto AWS IoT and use the rule engine they provide for processing these messages. The initial bridge setup is little tricky because IoT broker does not implement full MQTT protocol specification but once we got it working it has been reliable.
• Logging: It is useful to run the Mosquitto broker in verbose mode and capture all logging in a searchable store. We are using Papertrail and have setup remote syslog on the broker host to transmist logs to Papetrail.

We are using these MQTT client libraries for Android, iOS and Python. We really liked the MQTT essentials series by HiveMQ and highly recommend reading through it if you are planning to use the protocol in your production services.

We are yet to benchmark the battery gains from switching to MQTT those might come in another blog. We have achieved the goal of having a real 2-way communication where we can reliably push commands down to our SDK clients. Overall, we have been happy with the setup so far and thanks to Mosquitto-IoT bridge the effort required to switch have been minimal.

Have questions? Suggestions? Join the discussion on Slack.

Like what we are doing? Sign up to use HyperTrack and build movement-aware applications!