Troubleshooting SSL Handshake Failure: Understanding Cipher Suite Mismatches and Configuration Solutions

In previous articles, I discussed how to use Zabbix’s auto-discovery feature to automatically monitor the expiration dates of all server domain SSL certificates. If you’re interested, you can read more in these articles: [Simple Script Monitoring SSL Certificate Expiration Dates](https://mp.weixin.qq.com/s/IGJHZXv-XGunjj9Fle7IHQ) and [Configuring Zabbix to Automatically Discover and Monitor SSL Certificate Expiration for All Server Domains](https://mp.weixin.qq.com/s/Lp0mlhyIuXLTJPrB7HLdLA). Also, while setting up SSL monitoring, you may encounter an SSL handshake failure, which indicates an issue in the SSL/TLS negotiation process.

Then a colleague mentioned that during use, the script to obtain certificate expiration reported an error.

>Script error obtaining certificate information

As shown, the error is quite apparent, it’s an SSL handshake failure, and the protocol used is SSLv3.

>Obtaining certificate script

In the script to obtain SSL certificate information, a context object is constructed with SSL.Context, specifying the use of TLSv1, but the error states SSLv3.

So, I tested it locally; everything was fine on my end. I added an output to the script to display the link protocol version.

Output the established connection TLS version

The output protocol indeed used TLSv1; here, get_protocol_version returns an int, where 769 represents TLSv1.

Run script to get TLS version

So, I asked my colleague to send me the domain. I executed the script and encountered the same error.

During do_handshake execution, the handshake failed, hence no protocol version for the established connection was output. With no other choice, I had to packet capture for analysis.

Packet capture analysis

By capturing the packets and analyzing them with Wireshark, it was evident that after the client’s “hello” message, the negotiation was unsuccessful. Regarding SSL handshakes, I’ve written a previous article Wireshark Packet Capture Helps Clarify HTTPS Request Process. Feel free to check it out if interested.

Here, we analyze the client’s “client hello” packet and examine the cipher suites.

Packet capture examining client cipher suites

We also requested the nginx-configured cipher suites from the colleague.

Server cipher suites

It is evident that the client and server cipher suites do not match, which explains why the handshake was unsuccessful. The colleague’s cipher settings were too strict, so I suggested a reconfiguration of the cipher suites, which resolved the issue, allowing a normal connection to be established.

Aside from packet capturing, you can also use the openssl tool to establish a connection and view the entire connection process, for example by using openssl s_client -connect xxx.com:443 (where the port must be included, or use -h to view usage instructions).

Openssl tool establishing connection process

Additionally, regarding the configuration of cipher suites in nginx, they are specified using the ssl_ciphers directive. The cipher suite format is typically separated by ‘:’ and written on one line. What does a cipher suite include?

Take the one I suggested to the colleague as an example: ECDHE-RSA-AES128-GCM-SHA256

ECDHE: Key exchange algorithm

RSA: Signature algorithm

AES128: Symmetric encryption algorithm

GCM-SHA256: Signature algorithm

A typical cipher suite includes these parts.

The latter part, HIGH:!aNULL:!MD5:!RC4:!DHE

This part is a macro definition of the cipher suite, representing a type of cipher suite. Openssl’s ciphers can be used to check cipher suites, let’s look at HIGH.

Review cipher suites

For neatness, I used column to align it. You can see that HIGH represents a type of cipher suite, containing detailed version numbers and separate algorithm components.

Hence, HIGH represents advanced cipher suites, i.e., those with key lengths greater than 128 bits. In openssl’s ciphers, there are also MEDIUM and LOW, but LOW has been disabled since openssl 1.0.2g. More information about ciphers can be found in the official openssl documentation https://www.openssl.org/docs/man1.0.2/man1/ciphers.html.

In summary, when configuring nginx, for regular services, do not configure cipher suites that are overly strict to prevent mismatches and incompatibility with older client versions.