To learn RocketMQ, you need to understand two key aspects: communication and storage. I spent some time creating a Wireshark Lua plugin specifically for RocketMQ communication. The process was quite interesting, and I’ve documented it here.
Through reading this article, you will gain the following knowledge.
From Wireshark’s about page, you can see the Lua version it currently supports. Below is the corresponding page for my v3.0.6 version of Wireshark.
As you can see, the currently supported Lua version is 5.2.4. Now, let’s look at a skeleton code.
Find the Wireshark plugin directory; on my computer, this path is, modify the init.lua file therein.
Add a line to load the Lua file above with a dofile call.
The effect before and after execution is as follows.
The communication protocol of RocketMQ is relatively simple; the overall protocol format is shown below.
The RocketMQ communication protocol consists of four parts:
Take an actual packet as an example:
The first four bytes indicate the length of the entire packet 411(0x019b). The next four bytes indicate the Header Length, which is 212(0xD4) here. The following 212 bytes represent the contents of the Header, which can be seen as a json string. The final 195 (411-4-212) bytes represent the actual content of the Body. The specific message format will be discussed later.
Next, let’s write the parsing program.
The parsing logic is carried out in the proto.dissector method. Its signature is shown below.
The meanings of these parameters are as follows:
Next, we display the four parts of RocketMQ communication in Wireshark. Modify the proto.dissector function code as follows.
Reload the Lua script, and you can see the various parts of the RocketMQ protocol already displayed in Wireshark.
To distinguish whether it is a communication Request or Response, we can differentiate through the destination port number and add a new method.
Add the distinction between requests and responses in proto.dissector, with more readable descriptions.
The effect is as follows.
Next, what we need to do is parse the json to make the display more visually appealing. First, look at the header and body when the format is json for requests and responses. Add a recursive method to uniformly handle JSON data formats.
In the proto.dissector method, add Header parsing as shown below.
Reload and run the above code; the effect is as follows.
At the same time, we can also find more readable string representations corresponding to the request and response codes in RocketMQ’s source code.
If the Body is a JSON string, you can also handle it this way, as shown below.
However, in some cases, the Body is not represented as a JSON string. For example, when PULL messages, if the server returns consumable messages, the Body does not contain strings but RocketMQ’s custom message format, as shown below.
Writing this parsing is a physical effort; I referred to RocketMQ’s Java source code to implement a Lua version. The complete code is shown below.
The running effect is as follows.
I put the complete code on GitHub: https://github.com/arthur-zhang/rocketmq-wireshark-lua-plugin. Interested peers can take a look. Besides the functions mentioned before, it also implements extracting useful information like topics to the Info column for easier communication process viewing.
Tinkering with these things is quite interesting. In backend development, Lua, aside from its significant uses in OpenResty and Redis, has many interesting applications waiting for us to explore.
By writing this plugin, my understanding of the details of RocketMQ communication has become clearer. I might write more about RocketMQ communication details in the future.