Nowadays, most of the data traffic on the internet is encrypted for information security and protection. The problem: hackers are also using encryption to smuggle harmful malware past the network defence systems unnoticed. Methods such as SSL/TLS inspection or encrypted traffic analysis offer a potential remedy. These enable companies to examine encrypted data transmissions for malware.

In the age of the internet and the cloud, there is one central security measure – encryption of the data stored in the cloud and of the data while it is being transmitted. The TLS (Transport Layer Security) protocol is generally used for this purpose, also known under its previous designation, SSL (Secure Sockets Layer). In “Market Radar: Next-Generation Firewall Platforms, January 2020”, the analysts at Omdia estimate that, in today’s companies, 70 to 80 per cent of incoming network traffic is encrypted. This is an increase of around 20 per cent compared to the previous three years. According to Mozilla’s telemetry data, 70 percent of all websites currently use SSL encryption – recognisable by the padlock in the corner of the browser or the ”https” in the internet address.

But encryption also offers new possibilities for the “dark side”. Many hackers use SSL/TLS to hide their activities and movements in the network, and to attack systems or applications with malware they have smuggled in unnoticed. In this encapsulated form, it is very difficult for the companies being attacked to recognise the malware. As a matter of principle, if companies can’t see what’s entering their network, they can’t protect themselves against it. To prevent catastrophic consequences, companies ought to analyse this encrypted data traffic, so that no malware enters their network and, in addition, that no propriety data exits the network unchecked.

Analysing encrypted traffic

But this is frequently not the case. Many companies don’t analyse encrypted traffic – or don’t analyse it sufficiently. This is highlighted in a study by Flowmon and IDG Connect.  Although 99 per cent of the IT managers questioned regard encrypted network traffic as a source of security risks, two thirds of the companies fail to protect their assets from internal and external threats which misuse the SSL/TLS.

The two largest obstacles to the decryption of network traffic via SSL inspection are the fear of infringing data protection legislation (36 per cent) and concerns about reduced performance (29 per cent). SSL/TLS inspection enables companies to decrypt encrypted data traffic, classify it and check it. Encrypted traffic analysis is another way of analysing encrypted traffic. Temporarily decrypting the data is not necessary. Here is a brief description of the two approaches.

TLS/SSL inspection

Next Generation Firewalls (NGFW), such as the FortiGate High-End series from our partner Fortinet, use SSL/TLS inspection to analyse TLS 1.3-encrypted communications for malware. To do this, the firewall must analyse the data in plain text. So the encrypted data is first decrypted, analysed and then re-encrypted and sent to the target computer via a second TLS/SSL connection. Here the firewall acts as the “man in the middle” between the server of, for example, a website and the client in the company; decrypting the traffic and checking it for malware. Because decrypting and re-encrypting HTTPS data traffic takes a lot of computing power, the FortiGate NGFs have special security processors to ensure that performance is not impaired. Furthermore, it is necessary to check precisely which data is permitted to be decrypted, in order to continue adhering to the GDPR and to ensure compliance.

Here it is the certificates which are vital for security. The firewall needs a trustworthy Certificate Authority (CA) certificate so that the browser will accept the interruption of the encrypted data flow through the firewall. Otherwise, the browser will issue a warning because it no longer sees the original server certificate, but one signed by the firewall. It is also possible to shift scanning of the encrypted traffic from the firewall to the client to ensure end-to-end encryption. As soon as a potential threat is uncovered there, the endpoint notifies the firewall that it must interrupt the client’s connection to the internet.

Encrypted traffic analysis

Another approach is encrypted traffic analysis, which can analyse encrypted data transmissions for malware without having to decrypt the data first. Our partner Juniper Networks offers this function for the Juniper Advanced Threat Prevention (ATP) Cloud and the SRX series firewalls. Machine learning is used to detect anomalies in the encrypted data traffic. The benchmark is a defined baseline of the normal internet and network activity for each network host.

To detect anomalies in the traffic pattern, the solution analyses telemetry data and metadata – which are independent from the information to be transported. This includes, for example, information such as packet length or times. Further data is also correlated. Via passive monitoring, encrypted traffic analysis recognises suspicious data flows in encrypted traffic. Only these are subsequently decrypted and blocked – this means that data protection is also ensured.

Summary: Decrypting with a sense of proportion

Encrypted traffic analysis monitors the network traffic and limits decryption to critical data flows only. But GDPR compliance can also be achieved using SSL/TLS inspection, as long as it is implemented with the appropriate solution. It is important that the decryption is proportionate: companies shouldn’t decrypt all data as a matter of course. Instead, they must classify the data according to risk in advance, and define high-risk categories on which to focus their efforts. This could be, for example, newly-registered domains or recently-infected or uncategorised websites. In addition, connection to websites with expired certificates, untrusted certificates or self-signed certificates can be blocked. This protects users effectively without having to decrypt the data.