CRS and Log4j / Log4Shell / CVE-2021-44228

This is an evolving blog post with infos about the role of CRS in defending against the log4j vulnerabilities that threatens quite all logging JAVA applications. We believe the mitigations and rules suggested below will have you covered up to and including CVE-2021-45105.
In January 2022, we have consolidated our knowledge into a pull request with new rules to be merged into CRS for the next major release. The pull request can be applied to your existing installation for immediate use of the new rules.

Quick Fix

CRS rule 932130 is able to detect all known exploits targetting arbitrary GET and POST parameters. However, the rule is not inspecting HTTP headers such as the User-Agent and Referer. Starting Friday we saw attacks on these headers, so 932130 is not really adequate.

The quick fix is to add the User-Agent and the Referer to the targets of the rule. Do this by adding the following two directives after the CRS include in your configuration:

# Defense against CVE-2021-44228
SecRuleUpdateTargetById 932130 "REQUEST_HEADERS:User-Agent"
SecRuleUpdateTargetById 932130 "REQUEST_HEADERS:Referer"

There is a certain chance this update will trigger new false positives. Personally I do not expect too many, but you have been warned.

If you want to extend the coverage to all HTTP headers, then the following should be applied:

# Defense against CVE-2021-44228 
SecRuleUpdateTargetById 932130 "REQUEST_HEADERS"

The new rule to detect all log4shell payloads

There have been so many payloads published those last few days leading to more complicated rules and what not. But I have now come to the conclusion a very simple rule is the better approach. In fact that's the effectiveness of 932130: It's not overly smart, but it catches literally every payload with one exception.

So here is the approach for the new rule:

  • Detect nested use of ${ - this literally kills all the evasions we have seen
  • Detect use of ${jndi:... without the closing bracket (this is the payload that 932130 misses)

There might be legitimate uses of nested ${, but that will simply lead to false positives and they are acceptable in light of a highly critical vulnerability.
The second item can literally scan for the string jndi, as any obfuscation would depend on nested ${ make the first item hit again.

So here is your rule that does just this. It also includes the ctx lookup that can be exploited in a similar way like jndi. We also introduce a range limiter with the nesting. We have thought about whitespace in this position, but a closer look at the log4j source code revealed that whitespace will ruin the lookup and thus the exploit. Here you go (please place before your CRS include):

# Generic rule against CVE-2021-44228 (Log4j / Log4Shell)
# See https://coreruleset.org/20211213/crs-and-log4j-log4shell-cve-2021-44228/
SecRule REQUEST_LINE|ARGS|ARGS_NAMES|REQUEST_COOKIES|REQUEST_COOKIES_NAMES|REQUEST_HEADERS|XML://*|XML://@* "@rx (?:\${[^}]{0,4}\${|\${(?:jndi|ctx))" \
    "id:1005,\
    phase:2,\
    block,\
    t:none,t:urlDecodeUni,t:cmdline,\
    log,\
    msg:'Potential Remote Command Execution: Log4j CVE-2021-44228', \
    tag:'application-multi',\
    tag:'language-java',\
    tag:'platform-multi',\
    tag:'attack-rce',\
    tag:'OWASP_CRS',\
    tag:'capec/1000/152/137/6',\
    tag:'PCI/6.5.2',\
    tag:'paranoia-level/1',\
    ver:'OWASP_CRS/3.4.0-dev',\
    severity:'CRITICAL',\
    setvar:'tx.rce_score=+%{tx.critical_anomaly_score}',\
    setvar:'tx.anomaly_score_pl1=+%{tx.critical_anomaly_score}'"

Warning: Apache throws a warning when loading this rule. But I am confident it does not lead to any problems.

There is a certain chance for false positives for this rule, but so far we have not heard any reports about it. Other users confirm they did not get any so far.

Please notice that the t:cmdline transformation also puts everything in lowercase, so we do not have to do this explictly. And that we do not log the exact payload via logdata anymore. This makes it harder to identify false positives, but it also protects your logviewer from making jndi calls.

Update 2021-12-23: We've received a bypass for this rule (see our log4j hall-of-fame). The attack was detected by 932130 however. So we really suggest to address the whole problem with this rule, but also the extension of 932130 described above.

Mitigation via WAF? Should you not just patch your software?

Patching your servers should be your number 1 priority. Of course it should.

CRS can only buy you time or support you by taking out simple attacks in a layered defense. This will allow you to concentrate on the hard stuff.

We dub our project as "The 1st line of defense". It does not say the only line of defense for a good reason.

Illustration of the CVE-2021-44228 and the different mitigation techniques by GovCert.ch. Notice how a WAF is depicted as a first line of defense.

And finally, if you really think our detection is m00t, then proof it via our log4j detection bypass hallo-of-fame contest.

Former approaches to a new rule - left here for archive and further inspiration

CRS is a generic rule set that does not update when a new CVE comes out. Usually, we are covering it already, but it is true that we fear false positives on User-Agents and Referers. As a consequence a lot of very strong rules are not applied to these headers since they can take crazy forms, really. Loginjection attacks are therefore a bit of a weak spot.

Maybe this CVE is such a clusterf**k that a separate update of the rule set is due. We have not made up our mind yet, but there is now consensus, that a new rule has to be developed. New rules are always tricky because of false positives and the flexible JNDI interface that the exploits abuse make evasions really simple (and hard to spot!).

So here are two a basic rules contributed by one of our integrators and then a third one with an alternative regex from our github issue on the topic. They are candidates for a possible inclusion in the rule set:

# Generic rule against CVE-2021-44228 (Log4j)
SecRule REQUEST_LINE|ARGS|ARGS_NAMES|REQUEST_COOKIES|REQUEST_COOKIES_NAMES|REQUEST_HEADERS|XML://*|XML://@* "@rx \${[^}]*\${" \
    "id:1000,\
    phase:2,\
    block,\
    t:none,t:urlDecodeUni,t:cmdline,\
    log,\
    msg:'Potential Remote Command Execution: Log4j CVE-2021-44228', \
    logdata:'Matched Data: %{MATCHED_VAR} found within %{MATCHED_VAR_NAME}',\
    tag:'application-multi',\
    tag:'language-java',\
    tag:'platform-multi',\
    tag:'attack-rce',\
    tag:'OWASP_CRS',\
    tag:'capec/1000/152/137/6',\
    tag:'PCI/6.5.2',\
    tag:'paranoia-level/1',\
    ver:'OWASP_CRS/3.4.0-dev',\
    severity:'CRITICAL',\
    setvar:'tx.rce_score=+%{tx.critical_anomaly_score}',\
    setvar:'tx.anomaly_score_pl1=+%{tx.critical_anomaly_score}'"

# Targetted rule against CVE-2021-44228 (Log4j)
# Can be evaded
SecRule  REQUEST_LINE|ARGS|ARGS_NAMES|REQUEST_COOKIES|REQUEST_COOKIES_NAMES|REQUEST_HEADERS|XML://*|XML://@* "@rx \${jndi:(?:ldaps?|iiop|dns|rmi)://" \
    "id:1001,\
    phase:2,\
    block,\
    t:none,t:lowercase,t:urlDecodeUni,\
    log,\
    msg:'Remote Command Execution: Log4j CVE-2021-44228', \
    logdata:'Matched Data: %{MATCHED_VAR} found within %{MATCHED_VAR_NAME}',\
    tag:'application-multi',\
    tag:'language-java',\
    tag:'platform-multi',\
    tag:'attack-rce',\
    tag:'OWASP_CRS',\
    tag:'capec/1000/152/137/6',\
    tag:'PCI/6.5.2',\
    tag:'paranoia-level/1',\
    ver:'OWASP_CRS/3.3.x',\
    severity:'CRITICAL',\
    setvar:'tx.rce_score=+%{tx.critical_anomaly_score}',\
    setvar:'tx.anomaly_score_pl1=+%{tx.critical_anomaly_score}'"

# Targetted rule against CVE-2021-44228 (Log4j)
# Alternative generic regex
SecRule  REQUEST_LINE|ARGS|ARGS_NAMES|REQUEST_COOKIES|REQUEST_COOKIES_NAMES|REQUEST_HEADERS|XML://*|XML://@* "@rx \${[\w${}\-:]*j[\w${}\-:]*n[\w${}\-:]*d[\w${}\-:]*i[\w${}\-:]*:.*}" \
    "id:1002,\
    phase:2,\
    block,\
    t:none,t:lowercase,t:urlDecodeUni,\
    log,\
    msg:'Remote Command Execution: Log4j CVE-2021-44228', \
    logdata:'Matched Data: %{MATCHED_VAR} found within %{MATCHED_VAR_NAME}',\
    tag:'application-multi',\
    tag:'language-java',\
    tag:'platform-multi',\
    tag:'attack-rce',\
    tag:'OWASP_CRS',\
    tag:'capec/1000/152/137/6',\
    tag:'PCI/6.5.2',\
    tag:'paranoia-level/1',\
    ver:'OWASP_CRS/3.3.x',\
    severity:'CRITICAL',\
    setvar:'tx.rce_score=+%{tx.critical_anomaly_score}',\
    setvar:'tx.anomaly_score_pl1=+%{tx.critical_anomaly_score}'"

Both rules have to be added to the configuration before the CRS include. While the 2nd more targetted rule is unlikely to trigger false positives, they have to be expected with the first one.

Please note that rule 1000 and rule 1002 trigger several warnings on Apache when starting, but it seems to work.

If you extended the target list of 932130 as described under "Quick Fix", then the rule 1000 does not really add much value on a system that blocks at an anomaly threshold of 5 as it should. If you have a higher anomaly limit, then adding a 2nd rule that follows a very similar pattern will push the request to an anomaly score of 10.

False Positives

It's very hard to give substantial information about false positives here. However, I did run everything above against a little User-Agent value zoo of 7.5K UAs from a production server. I did not get a single false positives on the User-Agent HTTP header for rule 932130 and the three new rule proposals above.

Things might look different on other headers or cookies depending on the application. So you can not know unless you test it yourself, or you really, really know your traffic.

The Basic Auth Problem

The so called basic auth header in HTTP has a very special format with a value that starts with "Basic" and then a base64 encoded concatenation of username and password, separated by a colon. This colon is what saves the day here. Because the exploits all contain colons, splitting username and password would cut the exploit apart. So the username can not really contain the exploit. Besides RFC 7617 forbids colons in usernames for Basic Auth.

So you are pretty much save from that angle with two exceptions:

  • An application that logs passwords via log4j
  • You URL-decode the username before you log it via log4j (since an attacker could URL-decode the colons)

I think these two exceptions are quite far fetched, so I won't pursue them here.

Changelog:

2021-12-13 09:00 CET : Publication
2021-12-13 09:30 CET : Fix linebreaks in rules, add notice about Apache warning for rule 1000
2021-12-13 12:30 CET : Added rule 1002, added section on false positives
2021-12-13 21:40 CET : Added comma after msg action in rules 1000, 1001 1002 and transformations in 1000; added info about potential memory leak
2021-12-13 22:00 CET : Dropped the case insensitive prefix for the regex (since we have t:lowercase)
2021-12-13 22:10 CET : Added a variant for all HTTP headers to the Quick Fix section
2021-12-13 23:10 CET : Added section "The Basic Auth problem"
2021-12-14 22:40 CET : Removed REQUEST_BODY after reports of false positives and multimatch because it's likely no use with the transformations being used here. Remove remark about potential memory leak. Said report has been withdrawn and no further information about a memory leak.
2021-12-15 09:00 CET : Added rule 1005 as new generic rule.
2021-12-15 09:10 CET : Added section about "Mitigation via WAF? ..."
2021-12-15 11:15 CET : 3 small typos and reformatting
2021-12-15 22:30 CET : Added ctx to rule 1005, fixed backtracking / ReDoS problem of rule by introducing a range limiter after we rules out whitespace can be used to evade the pattern.
2021-12-15 22:50 CET : Add GovCert.ch illustration of mitigation options including WAF as 1st line of defense
2021-12-15 22:55 CET : Explain that t:cmdline also does t:lowercase in the same run
2021-12-16 09:15 CET : Removed logdata action from rule 1005
2021-12-16 15:50 CET : Typo with rule ID plus link to 2nd blog post
2021-12-16 15:50 CET : Added remark about other log4j CVEs up to CVE-2021-45105.
2021-12-20 10:30 CET : Updated the XML XPath selectors in all rules.
2021-12-23 16:15 CET : Notice about partial bypass and advice to extend 932130 for real.
2022-01-10 18:05 CET : Link to PR 2349 with new log4j rules


Christian Folini, CRS Co-Lead

29 thoughts on “CRS and Log4j / Log4Shell / CVE-2021-44228”

  1. Alexandre Monnot

    Hi,

    I'm trying to add one of those rule to my modsecurity but I always have a syntaxe error when I test my nginx configuration

  2. Alexandre Monnot

    Auto-reply :

    Seems like the problem is there :

    msg:'Remote Command Execution: Log4j CVE-2021-44228' \

    On my configuration (nginx with modsecurity 3.3.0) it might be :
    msg:'Remote Command Execution: Log4j CVE-2021-44228',\

    But I still got an error on the 1st rule of this post (the one who start by this comment :

    # Generic rule against CVE-2021-44228 (Log4j)

    )

  3. Christian Folini

    Thank you for pointing this out Alexandre. I only tested on Apache and said implementation is less picky when it comes to botched rules.

    I fixed the missing commas and tried on nginx now. Seems to be working fine for me.

  4. Alexandre Monnot

    Hi,

    I've got a doubt of how to implement this on our reverse proxy. Will the rule id 1001 be match on a nginx reverse proxy server ? I though rules ids should be greater than 100,000,000 in that case. But maybe I didn't understand well the rules id mechanism ?

  5. Christian Folini

    Rule IDs are irrelevant as long as they are unique.

    The name space below 100K is yours. So no fear of collisions. But check and assign the IDs wisely before you deploy - and keep track of what IDs you have used.

    In my tutorials and when doing arbitrary rules for blog posts, I usually use rule IDs around 1K or 10K.

  6. Hi. Thanks a lot for you work. Perhaps a noob question, Apache gives 6 warnings of type AH00111 "Config variable A is not defined" where A are the regex starting with the dollar sign. Why is Apache trying to interpret as a variable? Thank you.

  7. Christian Folini

    I mentioned those warnings above. Honestly, I do not really know. Maybe they can be surpressed with a smarter regex, but AFAICT, the warnings are no problem. And while I initially thought the memory leak problem could be linked to this, I have meanwhile been informed by said user that the memory leak had a different cause.

    1. Christian Folini

      REQUEST_BODY is basically covered by ARGS and ARGS_NAMES. And it is the very item where we received reports of false positives (for mime bodies with file uploads!). So while we can not really rule out a potential bypass around ARGS, we thought it acceptable to remove the REQUEST_BODY. Until somebody convinces us it's been a bad idea.

      Honestly, we're constantly re-adjusting the rules as reports come in. And we could do with more reports. FP reports very rare for example. Anything would help in this regard. Namely ratios like 1 FP in 10M requests.

  8. Charles Butterfield

    Thanks for the great work! Given the rapid rate of change would it be possible to mark each iteration of your recommendations with a version number of some sort. As well as a way of accessing the older recommendation given the version number. I would like to be able to comment my applied fixes with "... per CRS log4j quick fix version TBD"

    Perhaps the date of the posting is adequate, not sure exactly if it includes tweaks and mass revisions to this page.
    Again - thanks so much for the great work.

    1. Christian Folini

      You're totally right, Charles.

      Things have slowed down now and I do not expect too many updates anymore. Beyond the one I just did.

      But next time when we do live coverage, we'll have to do better.

      We might venture into the ModSecurity "ver:" tag to address this.

  9. Christian Haller

    Hi Christian, thank you very much!
    Deployed the rule on my ingress-nginx ingress controller on Kubernetes. Works like a charm.

  10. Using Modsecurity 2.9.5 with Apache 2.4.x with latest 3.3.x CRS. I included the 1005 rule at the end of my modsecurity.conf file, but changed the ID to be within the Service-specific Range before Core Ruls (10000 - 49999). I also included the 932130 REQUEST_HEADERS configuration in my Custom Rules which get loaded post CRS. Everything loaded, however, it does not seem to Block as I get a Response Code 200 on the client (running CURL request with the JNDI in the Header). The 1005 is triggering as I see in the Logs that the SecRule is encountered, but it still seems to be passing through. Thoughts on what I might be wrong?

    1. Christian Folini

      I see two possibilities:
      * Your anomaly threshold is too high
      * You defined 1005 / 1000x after the CRS include. That means this phase 2 rule happens after the blocking decision was taken.

      Shout out for following the numbering scheme I advocate (but don't follow here).

  11. so I am using
    # Defense against CVE-2021-44228
    SecRuleUpdateTargetById 932130 "REQUEST_HEADERS:User-Agent"
    SecRuleUpdateTargetById 932130 "REQUEST_HEADERS:Referer"

    on IIS and event viewer shows warning (pass) instead of error (deny)
    paranoia level is set to 1

    why it's not blocking?

  12. Hello, my English is not good, I hope you can understand. I would like to ask where the rules need to be placed:
    Put "SecRuleUpdateTargetById 932130 "REQUEST_HEADERS:User-Agent" and SecRuleUpdateTargetById 932130 "REQUEST_HEADERS:Referer"" in the "RESPONSE-999-EXCLUSION-RULES-AFTER-CRS.conf" file?
    Then put the new rules for detecting all log4shell loads in the "REQUEST-900-EXCLUSION-RULES-BEFORE-CRS.conf" file?
    Hope to get a reply, thank you very much!

    1. Christian Folini

      I think this is a good place to put these directives / rules. For more information, head over to my tutorials at https://netnea.com, where this precedence is explained in great detail. And an alternative configuration layout.

    1. Christian Folini

      The range limit is there to avoid DoS vulnerability via a regular expression (Lookup ReDoS on Google). We think we can do this, because after looking at the source of Log4J, we think it will not execute the jndi lookup if there is whitespace in front. So jndi is immediately or it's too late.

      If you want to be on the safe side and if you assume that DoS is the least of your concerns in light of Log4J (an assumption that I personally share), then doing the regex with * instead of the range limit is the right approach.

    1. Christian Folini

      Hey Elia, good to read you!

      These are non-official rules with temporary IDs. Once we put one or all of them, they will get a 900K rule id.

      The sub 100K space is reserved for local use. I do not think we use 1500 (or 1005 for that matter) anywhere with CRS. Our rule range is 900K + 9M for rule exclusions and in the future plugins.
      Or where did you find this rule ID being used?

  13. My nessus scan for Log4Shell triggers the below rules.
    [id "932130"] [msg "Remote Command Execution: Unix Shell Expression Found"]
    [id "932100"] [msg "Remote Command Execution: Unix Command Injection"]

    Is 932130 + 932100 enough?

Leave a Reply to Charles Butterfield Cancel Reply

Your email address will not be published.