Blog

CVE-2020-15598 - ModSecurity v3 Affected By DoS (Severity HIGH)

September 14, 2020 By Christian Folini (netnea)

The OWASP ModSecurity Core Rule Set (CRS) team has identified a Denial of Service vulnerability in the underlying ModSecurity engine. This affects all releases in the ModSecurity v3 release line. The vendor Trustwave Spiderlabs did not release an update yet. However, we are providing users with a patch for ModSecurity and a workaround if they can not patch. Likewise, we are coordinating the patching with the Linux distributors.

This blog post tries to give you a comprehensive overview of the problem with all the resources you need to cope with the situation.

This is what you will find here:

Official Advisory for CVE-2020-15598

ModSecurity v3.0.x is affected by a Denial of Service vulnerability due to the global matching of regular expressions. The combination of a non-anchored regular expression and the ModSecurity “capture” action can be exploited via a specially crafted payload.

While ModSecurity v2.x used to quit the execution of a regular expression after the first match. ModSecurity v3.0.x silently changed the behavior to global matching. This results in a DoS for existing non-anchored regexes containing the “capture” action. It also fills the TX variable space beyond the documented limit of 10 instances. The defense is handicapped due to the absence of the SecRequestBodyNoFilesLimit directive. The vendor Trustwave Spiderlabs dropped this functionality for ModSecurity v3.

The vendor did not publish a new release, but there is a patch that brings back the former behavior.

CVSSv3: 7.5 HIGH - https://nvd.nist.gov/vuln-metrics/cvss/v3-calculator?vector=AV:N/AC:L/PR:N/UI:N/S:U/C:N/I:N/A:H&version=3.1

Exploitability Metrics:

Impact Metrics:

Weakness Enumeration

CWE-400: Uncontrolled Resource Consumption

Known Affected Software Configurations:

Video of the PoC (we are not sharing an exploit though)

While we are not sharing an exploit, here is a video demonstrating a proof of concept attack. This is a python script killing an NGINX server running ModSecurity v3 and CRS3.

Please note that this was a script using ten threads and NGINX runs on 8 cores. The payload is about 700K per request and the server becomes unresponsive within seconds. A positive observation is that the server recovers relatively fast once the attack stops.

Description of the Problem

Let us give you a detailed look into this problem. It may be worth to take a seat, this is going to take a while.

Consider the following ModSecurity rule:

SecRule ARGS "@rx \d" "id:1000,phase:2,deny,capture,log,msg:'Numeric payload'"

For those not familiar with ModSecurity rules, this tells the engine to inspect all GET and POST parameters and look for digits (-> \d). If there is one in any of the payloads, the request is blocked, and the match is written into a transaction variable (TX.0) thanks to the capture action.

And now we send the following payload, apparently expecting a match.

foo=123456789012345

The operator @rx will execute the regular expression \d and look for a match. What happens after match is not identical across various different ModSecurity versions, though:

ModSecurity v2.x: operator matches at “1”, transaction variable TX.0 filled with “1”, remaining payload is dropped, no additional TX variables filled, rule execution aborts and request is being blocked.

ModSecurity v3.x: operator matches at “1”, TX.0 filled with “1”, remaining payload is used for new regular expression match, operator matches at “2”, TX.1 filled with “2”, remaining payload is used for new regular expression match, operator matches at “3”, TX.2 filled with “3”, remaining payload is used for new regular expression match, … and so on until TX.14 is filled with “5” and there is no payload remaining anymore, rule execution comes to an end and the request is being blocked.

So our reference platform ModSecurity v2 stops the execution of the regular expression after the first match. ModSecurity v3, which we also try to support as good as we can, continues to execute the regular expression again and again until the whole payload is consumed. In the regex world, this is called global matching.

Essentially, ModSecurity v3 is executing the following code:

do { rc = pcre_exec(m_pc, …) … } while (rc > 0)

This does not make any sense at all in this situation, since the first match will lead to a blockade and all remaining matches are going to be ignored. Also, there is no need to fill the additional transaction variables as no other rule is going to look at them after the deny. So the while loop should simply abort after the first match.

This repetitive and futile call of the regular expression execution function leads to a DoS. In the video above, the payload was fairly big, but the deciding factor is the number of matches: the number of transaction variables that are being written.

Global regular expression matching was introduced silently with the release of ModSecurity v3. ModSecurity v2 does not do global matching. The ModSecurity v2 Reference Manual does not mention global matching anywhere and all but two instances of all the “capture” rules in the manual are affected by this vulnerability unless they are rewritten to avoid it. (For the record: This has not happened).

Documentation is lacking with ModSecurity v3. A sign of this is that there is no reference manual for v3 and there was no announcement (or release notes) that mention this fundamental change in the behavior of the regular expression matching.

When CRS developer Ervin Hegedüs contacted Trustwave with information about this vulnerability, Trustwave refused to acknowledge a security problem despite the devastating effect on a server. Instead they claimed it was the more natural form of behavior and that the rules should be adopted. But they also reverted to the old behavior - mostly that is. However, they did this for the ModSecurity v3.1 development tree. Trustwave has refused to backport the fix and they stated clearly that they will not release a new minor version (ModSecurity 3.0.5) with this security fix.

The fixed behavior in the development tree is still not 100% the same as ModSecurity v2. That version supports ten transaction variables that can be captured in the regular expressions. If you define more than ten in your regex, ModSecurity will simply ignore the capture completely (unfortunately without writing an error message). The patched ModSecurity v3 does more than ten matches if your regular expression comes with more than ten sub-groups (e.g. (\d)(\d)(\d)(\d)(\d)(\d)(\d)(\d)(\d)(\d)(\d)(\d)) and fills the variables as requested in the rule. We do not really see a security problem connected with this behavior, but we are troubled it is non-documented behavior: It is different than the description of the capture action and the TX variables in the last version of the online reference manual.

Our reference platform ModSecurity v2 also features a directive that allows you to limit the size of the incoming requests for file uploads and form submissions separately (SecRequestBodyLimit vs. SecRequestBodyNoFilesLimit). The latter directive could be useful to limit the impact of this vulnerability while still allowing larger file uploads. Unfortunately, this directive was dropped for ModSecurity v3 and the vendor Trustwave refused re-implement it for v3 (see here for a conversation on this and an alternative proposal by Trustwave that would not help much in this situation). We see this as a major shortcoming as SecRequestBodyNoFilesLimit is a useful tool in DoS situations when you want to leave large file uploads intact, but limit the size of form submissions.

We can thus conclude this is a ModSecurity v3 DoS vulnerability. But how does it impact CRS? It impacts CRS a big deal because a lot of CRS3 rules are affected. We have working exploit payloads for three of them, but we consider a total of 58 rules of the default installation to be vulnerable. This is about a third of the default rules.

While it may be theoretically possible to rewrite the rules by anchoring them, performance and especially the readability of the rules would suffer. And despite our rules being very hard to read already, we have no intention to worsen this problem by anchoring everything. And there are cases where an anchored rule design would be impossible. Instead, we think that a security fix in the form of ModSecurity 3.0.5 is due.

Unfortunately, Trustwave Spiderlabs refuses to release despite having fixed the code already. This shifts the burden from the vendor to the users and we think this is wrong. That’s why we will tryeverything to make it easier for users to cope with this challenge.

Patch for ModSecurity 3.0.4

Since Trustwave does not want to release a new version, we are providing you with a backported patch for the vanilla ModSecurity 3.0.4. Please note that this is the patch that the Debian packaging team as well as NGINX will be using for their security updates.

cve-2020-15598.patch

This patch is a backport of the git commit, that brings back the former behavior of the regular expression matching. As such, the patch is copyrighted 2020 by Trustwave Spiderlabs but we release it here thanks to the Apache license of the code.

Debian has applied this patch and they are updating the binaries. The NGINX team at F5 provides a supported build of ModSecurity for NGINX Plus. They have tested and applied the patch and made updated ModSecurity builds available for NGINX Plus subscribers. So if you are using any of these, you should be covered with a simple update. If you need to compile it yourself, here are the instructions for the patching of ModSecurity 3.0.4:

…
… download src and checking of signature
…
$> tar xvzf modsecurity-v3.0.4.tar.gz
$> cd modsecurity-v3.0.4
$> wget https://gist.githubusercontent.com/crsgists/0e1f6f7f1bd1f239ded64cecee46a11d/raw/181bc852065e9782367f1dc67c96d4d250e73a46/cve-2020-15598.patch
$> patch -p1 < cve-2020-15598.patch

After this, the sources are ready for compilation. Follow any of the online recipes to do so. Here is one at netnea.

A (limited) Workaround for CRS3

CRS depends on regular expression matching a lot and the inspection of payload parameters is necessarily a primary focus for a web application firewall. We do not use the capture action everywhere, but most of the regular expressions are not anchored. Readability is a major reason for this.

With that being said, we identified 58 rules that are likely to be affected by this DoS vulnerability in the underlying ModSecurity v3 engine. Trustwave asked us to rewrite our rules to cope with their silent change of behavior (that they since reverted). We have no inclination to do so, however.

But what do you do, when updating / patching ModSecurity is not an option for you? You could disable all the CRS rules that are affected by this. But we advise you to ignore this option, since you would disable too many important rules and end up with a stripped down CRS3 leaving you with a lot of false negatives.

As explained above, the directive SecRequestBodyNoFilesLimit could be useful as a mitigation tactic, but it is not supported in ModSecurity v3. But we can try to implement a recipe that performs something along those lines. Unfortunately, this only works as a mitigation, when you do not have large POST requests.

# Defense against CVE-2020-15598
#
# Make sure to set blocking or non-blocking in first rule
# and then limit (in bytes) in third and fourth rule.
#
# Place recipe before CRS include in your configuration.
# 
# See https://coreruleset.org/20200914/cve-2020-15598/ for more infos
#

SecAction "id:1000,phase:1,pass,nolog,setvar:tx.cve_size_counter=0,setvar:tx.cveblock=0"

SecRule ARGS|ARGS_NAMES "@unconditionalMatch" "id:1001,phase:2,pass,nolog,t:length,setvar:'tx.cve_size_counter=+%{MATCHED_VAR}'"

SecRule REQUEST_HEADERS:Content-Type "(?:application(?:/soap+|/)|text/)xml" "id:1002,phase:2,pass,nolog,chain"
   SecRule REQUEST_BODY "@unconditionalMatch" "t:length,setvar:'tx.cve_size_counter=+%{MATCHED_VAR}'"

SecRule TX:cve_size_counter "@gt 10000" "id:1010,phase:2,deny,log,msg:'ARGS size exceeded. Blocking request to mitigate CVE-2020-15598',chain"
   SecRule TX:cveblock "@eq 1" "t:none"

SecRule TX:cve_size_counter "@gt 10000" "id:1011,phase:2,pass,log,msg:'ARGS size exceeded. This could be attack via CVE-2020-15598 (or size limit too low).'"

It is probably worth to describe what we are doing here a bit. I won’t go into the details of ModSecurity rule writing, but there are a few pecularities of this recipe that are worth noting.

In rule 1000, we initialize a counter and we define if this recipe should be blocking or not (tx.cveblock). By default, it is in monitoring mode. Rule 1001 counts the length of all argument names and all argument values and adds them to the counter. We do this via the length transformation (t:length). We could also pick up the value of the Content-Length request header, but here we prefer to go with the actual parameters and their length.

The problem with this argument name and argument value counter is that it does not cover all the arguments. Some are missing in the ARGS collection. An XML request is handled by the XML parser and the result is served to ModSecurity via a tree that can be addressed via the XPath. The individual leaves of the tree are not added to the ARGS collection though. XPath is your only way to iterate them. An XML payload could thus be used to bypass our argument counter (Please note that unlike XML, JSON arguments are added to the ARGS collection). If there is no XML, one can simply deny it. But if we have to assume there might be XML, we need to count it too. We could turn to the Content-Length again, but let’s not trust that too much. Unfortunately, we can not simply iterate over the XPath tree: since XML is hierarchical, there is data that we would get multiple times in case the XML was nested. So instead, we take the size (length!) of the REQUEST_BODY. In ModSecurity2, the REQUEST_BODY is consumed by the XML body processor and it is thus no longer accessible. In ModSecurity3, however, this is another undocumented deviation from ModSecurity2: REQUEST_BODY is accessible after the body processor has finished. In rule 1002, we take advantage of this implementation oddity in ModSecurity3 (the rule won’t work in ModSecurity2, but it is simply ignored silently).

Finally, 1010 and 1011 check the size of the length counter and they block the request or at least report the rule violation - both depending on the variable initialized in the first rule of the recipe.

It is hard to tell if you should install these rules on your server right away. You should definitely know the size of your POST requests before you do. Then it’s probably best to set the rule set in monitoring mode and let it run and only if you encounter a DoS to actually set it to blocking. For one thing must be clear: A single request triggering this rule is more a sign of a large POST request in your application than of a DoS attack. It’s only if you have so many of them that they actually take down your server that you can be sure it’s a DoS attack. If that happens, install this recipe, put it into blocking mode in rule 1000 and adjust the maximum size of the POST request in rule 1010 and 1011 (in bytes).

Timeline of Our Conversation With the ModSecurity Vendor Trustwave Spiderlabs

We’d rather not go into the details of our conversation with Trustwave Spiderlabs about this weakness. We reported this to the vendor Trustwave immediately after our discovery. That was 91 days ago as of this writing. Trustwave responded that they do not see the new behavior as a vulnerability. Subsequently we declared that we would wait for the standard 90 days (that Trustwave sees as normal grace period before a disclosure) and would then disclose, asking Trustwave to fix the problem and release an update for ModSecurity v3.0 (-> ModSecurity 3.0.5) in the meantime. However, they responded they would only release this together with v3.1.0. This conversation continued, but we did not really get anywhere:

The exchange has been friendly and professional and there is little to complain about in terms of responsiveness. We just disagree whether this is a vulnerability or not. And asking us to re-design up to 60 regular expressions after the underlying engine silently changed its behavior is overly demanding - not the least because the code has been reverted to the previous behavior and this laborious exercise would be futile in the long run.

We will update this list as more infos appear online.

Christian Folini and Ervin Hegedüs for the CRS team

EDIT

Christian Folini / [@ChrFolini]