Axway TSIM incoming TLS connection errors

Summary

Incoming OFTP2 TLS connections from an Axway TSIM system are not successful. OS4X logs this in several ways, such as "Unsuccessful connect try from IP ..." and others. Outgoing connections from OS4X to affected partners work fine.

Technical background

Incoming TLS connections are verified very securely before any OFTP2 communication takes place. These checks are:

1. Is the TLS client certificate valid (trusted, within validity timeframe and supports all required features)?
2. Is the TLS cipher secure?
3. Is the used TLS cipher able to fulfill optional requirements regarding Perfect Forward Secrecy (PFS)?
4. In case of a used OFTP2 SmartProxy: check if the SSID contained in the client certificate is known to exactly one partner?

In the case of the Axway TSIM situation, option 2) fails. The OS4X daemon (plain or SmartProxy) offer an ordered list of usable TLS ciphers for every supported TLS version, optionally limited to only offer PFS ciphers (known to contain the text "DHE" in the cipher name). As TLS is specified, the client picks the best possible cipher from the top of the list which is supports. In most cases, it is a DHE ("Diffie-Hellman key exchange") affected cipher, which is technically good. The cryptographic library OS4X is using forces to handshake a new session key via a secure mechanism. This mechanism requires the used Diffie-Hellman key to be as big as the RSA private/public key of the OS4X TLS server certificate (which is 4096 as of time writing this document).

Axway TSIM is often running in JRE (Java Runtime Environment) version 8 which fails to support Diffie-Hellman keys of 4096 and requests to use 1024 bits instead. The TLS handshake then fails due to this inability to support a strong and secure Diffie-Hellman key size.

Solution

The best solution would be to use a modern Java Runtime Environment which supports modern TLS ciphers and most recent security requirements. Often this is not possible due to impacts unknown to data exchange partners.

Workaround

Axway TSIM and JRE8 customers can change the list of used TLS ciphers. When they prefer a non-DHE cipher first, the TLS connection can finish the handshake with a less secure cipher. For this, OS4X must not request a PFS cipher in the TLS listener (see this configuration option). The values for outgoing TLS ciphers in Axway TSIM then must be:

• TLS_RSA_WITH_AES_256_CBC_SHA
• TLS_RSA_WITH_AES_128_CBC_SHA

Beware: the session can be recorded by any party which is able to intercept network traffic and afterwards decrypt the content! So you are highly encouraged to keep security requirements high!

External resources

Java has a very good documentation about their release dates: https://www.oracle.com/de/java/technologies/java-se-support-roadmap.html

For Java 8, the mechanism of how Diffie-Hellman keys are supported, the following documentation is available (which shows the mess of the support if the required value "matched" for this configuration is used): https://docs.oracle.com/javase/8/docs/technotes/guides/security/jsse/JSSERefGuide.html#customizing_dh_keys

Meme context