This article covers the key concepts of MGR, Monitoring, and Wireshark, with a particular focus on MGR MySQL.
In an MGR cluster, how does MySQL manage when a node exits unexpectedly? When will the abnormal node be expelled from the cluster?
Before we begin the experiment, let me share something with you.
We first use three virtual machines to create an MGR cluster. The version of MySQL is 5.7.20 (the reason for using a lower version of MySQL is because no one can clearly explain the behavior of this older version of MGR; we can only observe without speculation).
This time, we will skip this step and focus only on the created cluster:
Let’s check who is the Primary:
Now we need to choose an observation tool. We know MGR communicates over the network to reach consensus on the status of nodes within the cluster. Analyzing MGR’s behavior via packet capture seems like a good idea.
However, Wireshark does not support analysis of MGR’s network protocol; we have to create our own Wireshark analysis plugin (dissector).
Here we skip the plugin creation process and directly use it.
We visit:
Download the Wireshark installation package and install it:
We capture packets on the MGR Primary node (test-mgr-1):
Then we take down a node (test-mgr-3):
We use the recently installed Wireshark to open the captured packet file test-mgr-1.cap and use the MGR protocol to analyze the capture:
The effect is as shown:
We now add a column to Wireshark’s list to display MGR information types:
The formula for this column is: mgr.app_data.body.cargo_type
In this way, we can see the MGR information types:
We now use a filter formula to find specific types of MGR information:
We can see that the MGR Primary (10.186.61.45, which is test-mgr-1) sent three messages to another active node (10.186.61.71, which is test-mgr-2): view_msg, remove_node_type, and view_msg. Let’s take a closer look at the details of these three packets:
The first packet is the view information sent by the Primary (view is the state of the various nodes in MGR), showing that the first node is online, the second node is online, and the third node is offline.
The second packet is the notification to remove the node, where the Primary informs other nodes to remove the offline third node.
The third packet, sent a second later, is the Primary notifying other nodes of the new view: now there are only two nodes in the view.
Through packet capture, we observed the communication between MGR nodes to understand the scheduling behavior between nodes.
After a node crashes, the Primary node quickly updates everyone about the offline status of a certain node. It then expels it from the cluster and informs everyone of the decision.
We hope this Wireshark plugin we created will help everyone understand MGR’s scheduling behavior, and we welcome any requests or reports of deficiencies.