Exploring Golang with MQTT: File Transfers

While I've dabbled in writing scripts using Perl, Bash, and Python, I wouldn't necessarily label myself as a developer. However, I do have a penchant for automation, organizing logic, and embracing challenges. It was this innate curiosity that led me to explore Golang recently and experiment with its integration with MQTT.  The harmonious combination between MQTT's lightweight messaging protocol and Go's concurrency model presents a compelling case for utilizing MQTT with Golang.

MQTT (Message Queuing Telemetry Transport) is a lightweight publish-subscribe messaging protocol designed for efficient communication between devices with limited bandwidth and processing capabilities. It excels in scenarios where real-time data exchange is crucial, such as IoT (Internet of Things) applications, telemetry systems, and messaging platforms.

Golang, or Go, is a modern and efficient programming language known for its simplicity, concurrency support, and performance. It provides built-in support for concurrent programming through goroutines and channels, making it ideal for building highly concurrent and scalable systems.

When you combine MQTT with Go, you leverage the strengths of both technologies to create robust, scalable, and real-time communication systems. Here's why this combination is so compelling:

Efficiency: Both MQTT and Go are designed for efficiency. MQTT's lightweight protocol minimizes bandwidth and processing overhead, making it suitable for resource-constrained environments. Go's efficient runtime and concurrency model allow you to handle a large number of concurrent connections and process messages concurrently with minimal overhead.

Concurrency: Go's built-in support for concurrency with goroutines and channels aligns perfectly with MQTT's asynchronous messaging paradigm. You can easily handle thousands of concurrent MQTT connections and process incoming messages concurrently, leveraging the power of parallelism to scale your applications.

Simplicity: MQTT and Go are both known for their simplicity and ease of use. With the paho.mqtt.golang library, integrating MQTT into your Go applications is straightforward and intuitive. You can quickly connect to MQTT brokers, publish and subscribe to topics, and handle messages with minimal boilerplate code.

Scalability: The combination of MQTT and Go enables you to build highly scalable systems that can handle massive workloads with ease. Whether you're building IoT platforms with millions of devices or real-time messaging systems with high throughput requirements, MQTT and Go provide the scalability you need to meet your application's demands.

Fast forward to my little Go project which came out of some of my research around transferring files via MQTT.  While maybe not the most practical I had read of others doing it with Python and even just using the Mosquito publisher and subscriber tools.   But again the goal here was to learn a little about Go and tie it into something that motivated me to figure it out.  Hence my own file transfer publisher and subscriber written in Go.

The publisher code, which can be found here, does the following:

• Establishes a connection to the broker server with the topic of transfer
• Watches the following directory on where it is run: /root/outbound
• Any files that are dropped into the directory are then published to the broker on the topic of transfer

Note: MQTT has a 256MB limit of data it can transfer.  Further it will chunk the data up into segment messages which creates another challenge.

The subscriber code, which can be found here, does the following:

• Establishes a connection to the broker server with the topic of transfer
• Listens for any published messages on the topic transfer
• When a message is published, in this case a file, the subscriber pulls it down and places it into the /root/inbound directory
• The subscriber will also try to determine the file type by using the mimetype library and looking at the first 512 bytes of the file.  If it cannot be determined it defaults to a .txt extension.

The go.mod file used for the project can be found here.

For the experiment I simply created the two inbound/outbound directories on my system.   In separate terminals I went ahead and ran each of the Go programs.   In a third terminal I set up a watch on the directory listing for inbound.  Then in a fourth terminal I went ahead and used the copy command to place some files into the outbound directory which the publisher code was watching.  The demo is below:

Now for a first pass at this experiment I was fairly pleased but there is definitely room for improvement in the following items:

• I want to be able to pass as arguments my MQTT server, the topic and what directory to watch.
• I need to figure out a better way to handle file names on the subscriber side.  Currently all the filenames end up having the name file with a Unix timestamp and then maybe if identified right the correct extension.   One thought I have for this is to actually bundle up the file on the publisher side into a JSON payload where we have the file name, the extension, the size and then the actual data blob.  Then on the subscriber side we would get that file and process it on receipt to obtain the real file name, extension and data of the file.
• We need to handle large files better since they are chunked up so on the subscriber side we need to be able to take the chunks and assemble them back together and then process them.

Using MQTT with Golang allows you to leverage the lightweight, efficient messaging protocol of MQTT and the concurrency and scalability of Go to build robust, scalable, and real-time communication systems. Whether you're building IoT applications, telemetry systems, or messaging platforms, this combination provides the performance, efficiency, and simplicity you need to succeed.