Understanding Metadata Transfer in BitTorrent: Protocol Analysis and uTP Data Transmission

0. Review

Previous articles:

– A Brief Analysis of the Bittorrent Protocol (Part 1) Metadata File https://cloud.tencent.com/developer/article/2332701

– A Brief Analysis of the Bittorrent Protocol (Part 2) Tracker and Peers https://cloud.tencent.com/developer/article/2333043

– A Brief Analysis of the Bittorrent Protocol (Part 3) Peer Data Transmission Example https://cloud.tencent.com/developer/article/2333677

– A Brief Analysis of the Bittorrent Protocol (Part 4) Distributed Hashing https://cloud.tencent.com/developer/article/2334440

– A Brief Analysis of the Bittorrent Protocol (Part 5) Extension Protocol and Metadata Transmission Extension https://cloud.tencent.com/developer/article/2334776

• A Brief Analysis of the Bittorrent Protocol (Part 6) Peer Exchange, Local Service Discovery, Multi Tracker, and Private Torrents https://cloud.tencent.com/developer/article/2334906
• A Brief Analysis of the Bittorrent Protocol (Part 7) uTorrent Transport, Hole Punching Extension, and UDP Tracker https://cloud.tencent.com/developer/article/2337388

Recap of previous content:

BitTorrent is a protocol for distributing files. Metadata files are encoded using bencode, hashed with SHA-1, and compared for verification. The structure of metadata files is introduced, and Tracker exchanges node information via HTTP or UDP requests. Nodes establish direct TCP connections or UDP-based uTP communications. Hole punching extension assists nodes behind NAT to connect.

The metadata transfer extension in the extension protocol allows nodes to transfer metadata. PEX extension facilitates node information exchange, DHT retrieves nodes based on the hash with KRPC, and local service discovery utilizes multicast. These features should be disabled in private torrents.

So far, the discussed contents have been primarily based on TCP implementations of BitTorrent. In certain network environments, establishing TCP connections has limitations, as excessive TCP connections unfairly consume network resources. UDP-based uTorrent and hole punching extensions address these issues, providing connectivity for downloaders behind NAT or firewalls.

uTP Packet Analysis

In the previous articles, we simulated Bittorrent requests over TCP using the Sockit tool, but uTP is a UDP-based protocol. Constructing requests for analysis is challenging, so we use Wireshark to capture connections for analysis directly.

Using the Ubuntu system image mentioned earlier, connections are captured via Wireshark to analyze uTP handshakes and data transfers.

Note: My device has Docker Desktop installed, which modifies hosts to point host.docker.internal to the network exit IP address.

First, begin capturing with Wireshark, and stop after sufficient data is collected. As stated, uTP transmits data over UDP, without distinctive identification features, so data isn’t categorized in Wireshark. Manually filter and determine using port numbers and transmitted content. Right-click the compliant Wireshark data and choose “Decode As,” setting it as BT-uTP for analysis. See the image:

WireShark Decode Method

After setting, click OK to view the packet analysis. Below is a stream and handshake packet from one connection:

Handshake Packet

For convenient analysis, here’s the packet structure presented in Part 7:

0       4       8               16              24              32
+-------+-------+---------------+---------------+---------------+
| type  | ver   | extension     | connection_id                 |
+-------+-------+---------------+---------------+---------------+
| timestamp_microseconds                                        |
+---------------+---------------+---------------+---------------+
| timestamp_difference_microseconds                             |
+---------------+---------------+---------------+---------------+
| wnd_size                                                      |
+---------------+---------------+---------------+---------------+
| seq_nr                        | ack_nr                        |
+---------------+---------------+---------------+---------------+

Now, let’s examine the first packet of this stream:

uTP Handshake

This packet is the handshake packet for the uTP connection, initiated from the local port.

UDP data content:

21 02 45 f9 22 30 32 fc 00 00 00 00 00 38 00 00 91 1a 3f 0e 00 08 00 00 00 00 00 00 00 00

Included contents:

• type: 4 (ST_SYN), Connection SYN
• ver, version: 1
• extension: 0x00, no extension used
• connection_id: 0x45F9
• timestamp_microseconds: 0x033F7B72 Current timestamp
• timestamp_difference_microseconds: 0x00000000 First packet fixed at 0
• wnd_size, window size: 0x40000
• seq_nr, sequence number: 0x3f0e
• ack_nr, ACK number: 0

The second packet:

uTP Status 1

Upon receiving the handshake packet, the connection request recipient sends a status packet with contents:

21 02 45 f9 22 30 32 fc 00 00 00 00 00 38 00 00 91 1a 3f 0e 00 08 00 00 00 00 00 00 00 00

• type: 4 (ST_SYN), Connection SYN
• ver, version: 1
• extension: 0x02, extension flag, 2-bit extension
• connection_id: 0x45F9, consistent with request
• timestamp_microseconds: 0x0223032FC Current timestamp
• timestamp_difference_microseconds: 0x00000000
• wnd_size, window size: 0x380000
• seq_nr, sequence number: 0x911A
• ack_nr, ACK number: 0x3F0E, consistent with handshake request’s seq_nre
• Extension content, 0x00

Data can then be transmitted.

During testing, some pitfalls were encountered. Prior tests set Transmission downloader’s encryption to “Prefer Encryption,” resulting in encrypted data that hindered analysis. Changing encryption to “Allow Encryption” avoids this, showing BitTorrent handshake details after uTP handshake:

uTP BitTorrent Handshake

Encrypted transmission is intriguing but complex. My current understanding is superficial, as it isn’t illustrated in any BEP or documentation. It can only be understood through open-source implementation analysis. Interested individuals can focus on:

• BitTorrent protocol encryption
• LibTorrent – Github

In libtorrent, encryption is mainly implemented in bt_peer_connection.cpp and pe_crypto.cpp. Focus on these three functions and related implementations: write_pe1_2_dhkey() initializes encrypted handshake, generates Diffie-Hellman key pair, and sends local public key; write_pe3_sync() performs synchronous handshake, generates shared key, sets RC4 encryption key, and sends synchronization request; write_pe4_sync() processes sync response.

This section reviews the uTP protocol by analyzing an actual uTP data transmission. Though it could end here, let’s add more content, such as super seeds that many notice but don’t fully understand.

Super Seed

Super seeding mode is proposed in BEP drafts, subject to changes. Refer to BEP16 for reading.

When using a BitTorrent downloader, super seed/super seeding mode is often an option. So, what is it? Before reading this part, I strongly suggest reading and understanding A Brief Analysis of the Bittorrent Protocol (Part 3) Peer Data Transmission Example to get familiar with data content during node transmissions.

When a downloader operates in “super seeding mode,” it masquerades as a regular client with no data. When other nodes connect, it sends a have message indicating possession of an unsent piece. This prompts nodes to download only that piece. After a client completes this download, the seed node doesn’t signal possession of other pieces until the previously sent piece appears at another node. This is super seeding mode.

Super seed mode reduces redundant data transmission and limits nodes that download without contributing to the swarm.

This mode and option shouldn’t generally be used, except for initial seeders. Other nodes shouldn’t utilize this mode.

uTP Packet Analysis, Super Seed Section End

During the creation of these articles, it increasingly becomes evident that many seemingly simple concepts may encounter complexities in real-world implementations. Additionally, not everyone or every program complies strictly with stipulated protocols. Some contents appear to be customary conventions, understood generally without documented formats.