You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@pulsar.apache.org by Lari Hotari <lh...@apache.org> on 2021/12/10 12:23:04 UTC
[Security] CVE-2021-44228 severe RCE 0-day exploit found in Log4J - affects also Pulsar - mitigation instructions
As many of you might have already heard of this, there's a severe RCE 0-day
exploit found in Log4J (2.0 <= Apache log4j <= 2.14.1).
Blog post: https://www.lunasec.io/docs/blog/log4j-zero-day/
CVE-2021-44228 in GitHub Security Advisory:
https://github.com/advisories/GHSA-jfh8-c2jp-5v3q
This also affects all Pulsar versions after 2.0.0-incubating since a
vulnerable Log4J version is used. I'm not aware of a confirmed exploit for
Pulsar. The fix to Pulsar is to upgrade to Log4J 2.15.0 . The PR is
https://github.com/apache/pulsar/pull/13226 . The fix will be release as
part of Pulsar 2.8.2 , 2.7.4 and 2.9.1 . Before the fixed version is
available, there's an immediate workaround to mitigate the security issue.
I'd like to share mitigation instructions for this severe vulnerability:
- Add -Dlog4j2.formatMsgNoLookups=true system property to the JVM arguments
of all Pulsar processes. There are multiple ways to achieve this in Pulsar.
It can be added to either OPTS, PULSAR_GC or PULSAR_MEM environment
variables.
- Upgrade to Pulsar 2.8.2 , 2.7.4 or 2.9.1 once they are available.
There's a PR to handle the adding of -Dlog4j2.formatMsgNoLookups=true
system property in the Apache Pulsar Helm chart, that is
https://github.com/apache/pulsar-helm-chart/pull/186 . Until that is
available, the recommended approach is to add
"-Dlog4j2.formatMsgNoLookups=true" to OPTS, PULSAR_GC or PULSAR_MEM
manually and ensure that the Java process picks up the system property.
It's also necessary to check that the property doesn't have typos. The
setting is case sensitive.
Please patch your productions systems asap!
BR, Lari Hotari
Re: [Security] CVE-2021-44228 severe RCE 0-day exploit found in Log4J - affects also Pulsar - mitigation instructions
Posted by Lari Hotari <lh...@apache.org>.
For mitigating current systems, there's one more Log4JShell mitigation
strategy which is explained in
https://twitter.com/brunoborges/status/1469462412679991300 . This relies on
setting an environment variable called LOG4J_FORMAT_MSG_NO_LOOKUPS to the
value of "true" .
I created a simple test application for testing mitigations,
https://github.com/lhotari/log4shell-mitigation-tester . That might be
useful in ensuring that you don't have a copy-paste error or typo in the
mitigation.
Bruno Borges has also a sample for patching a K8S deployment. Something
like that could also work for Pulsar Helm Deployments unless you cannot
upgrade your deployment using the latest Pulsar Helm Chart version 2.7.6
which contains the system property based workaround.
I also added the LOG4J_FORMAT_MSG_NO_LOOKUPS=true mitigation to the
Dockerfile overlaying solution,
https://github.com/lhotari/pulsar-docker-images-patch-CVE-2021-44228 .
I hope others have found this work useful. Happy patching! Please be quick
if you want to avoid getting hacked. Pulsar is confirmed to be exploitable
with Log4Shell. I'm not aware of actual RCE, but at least reading security
tokens is possible using Log4Shell with released Pulsar versions and that's
why you should also cycle your credentials & security tokens after your
systems are patched.
If you are running Pulsar versions older than 2.7.1, they are vulnerable to
older security issues (CVE-2021-22160, excluding 2.6.4 which contains the
fix).
BR, Lari
On Fri, Dec 10, 2021 at 2:23 PM Lari Hotari <lh...@apache.org> wrote:
>
> As many of you might have already heard of this, there's a severe RCE
> 0-day exploit found in Log4J (2.0 <= Apache log4j <= 2.14.1).
> Blog post: https://www.lunasec.io/docs/blog/log4j-zero-day/
> CVE-2021-44228 in GitHub Security Advisory:
> https://github.com/advisories/GHSA-jfh8-c2jp-5v3q
>
> This also affects all Pulsar versions after 2.0.0-incubating since a
> vulnerable Log4J version is used. I'm not aware of a confirmed exploit for
> Pulsar. The fix to Pulsar is to upgrade to Log4J 2.15.0 . The PR is
> https://github.com/apache/pulsar/pull/13226 . The fix will be release as
> part of Pulsar 2.8.2 , 2.7.4 and 2.9.1 . Before the fixed version is
> available, there's an immediate workaround to mitigate the security issue.
>
> I'd like to share mitigation instructions for this severe vulnerability:
> - Add -Dlog4j2.formatMsgNoLookups=true system property to the JVM
> arguments of all Pulsar processes. There are multiple ways to achieve this
> in Pulsar. It can be added to either OPTS, PULSAR_GC or PULSAR_MEM
> environment variables.
> - Upgrade to Pulsar 2.8.2 , 2.7.4 or 2.9.1 once they are available.
>
> There's a PR to handle the adding of -Dlog4j2.formatMsgNoLookups=true
> system property in the Apache Pulsar Helm chart, that is
> https://github.com/apache/pulsar-helm-chart/pull/186 . Until that is
> available, the recommended approach is to add
> "-Dlog4j2.formatMsgNoLookups=true" to OPTS, PULSAR_GC or PULSAR_MEM
> manually and ensure that the Java process picks up the system property.
> It's also necessary to check that the property doesn't have typos. The
> setting is case sensitive.
>
> Please patch your productions systems asap!
>
> BR, Lari Hotari
>
>
>
>
>
>
>
>
Re: [Security] CVE-2021-44228 severe RCE 0-day exploit found in Log4J - affects also Pulsar - mitigation instructions
Posted by Lari Hotari <lh...@apache.org>.
Here's an additional workaround for patching existing Docker images with an
additional overlay which upgrades Log4J to 2.15.0:
https://github.com/lhotari/pulsar-docker-images-patch-CVE-2021-44228 .
BR, Lari
On Fri, Dec 10, 2021 at 2:23 PM Lari Hotari <lh...@apache.org> wrote:
>
> As many of you might have already heard of this, there's a severe RCE
> 0-day exploit found in Log4J (2.0 <= Apache log4j <= 2.14.1).
> Blog post: https://www.lunasec.io/docs/blog/log4j-zero-day/
> CVE-2021-44228 in GitHub Security Advisory:
> https://github.com/advisories/GHSA-jfh8-c2jp-5v3q
>
> This also affects all Pulsar versions after 2.0.0-incubating since a
> vulnerable Log4J version is used. I'm not aware of a confirmed exploit for
> Pulsar. The fix to Pulsar is to upgrade to Log4J 2.15.0 . The PR is
> https://github.com/apache/pulsar/pull/13226 . The fix will be release as
> part of Pulsar 2.8.2 , 2.7.4 and 2.9.1 . Before the fixed version is
> available, there's an immediate workaround to mitigate the security issue.
>
> I'd like to share mitigation instructions for this severe vulnerability:
> - Add -Dlog4j2.formatMsgNoLookups=true system property to the JVM
> arguments of all Pulsar processes. There are multiple ways to achieve this
> in Pulsar. It can be added to either OPTS, PULSAR_GC or PULSAR_MEM
> environment variables.
> - Upgrade to Pulsar 2.8.2 , 2.7.4 or 2.9.1 once they are available.
>
> There's a PR to handle the adding of -Dlog4j2.formatMsgNoLookups=true
> system property in the Apache Pulsar Helm chart, that is
> https://github.com/apache/pulsar-helm-chart/pull/186 . Until that is
> available, the recommended approach is to add
> "-Dlog4j2.formatMsgNoLookups=true" to OPTS, PULSAR_GC or PULSAR_MEM
> manually and ensure that the Java process picks up the system property.
> It's also necessary to check that the property doesn't have typos. The
> setting is case sensitive.
>
> Please patch your productions systems asap!
>
> BR, Lari Hotari
>
>
>
>
>
>
>
>
Re: [Security] CVE-2021-44228 severe RCE 0-day exploit found in Log4J - affects also Pulsar - mitigation instructions
Posted by Lari Hotari <lh...@apache.org>.
For mitigating current systems, there's one more Log4JShell mitigation
strategy which is explained in
https://twitter.com/brunoborges/status/1469462412679991300 . This relies on
setting an environment variable called LOG4J_FORMAT_MSG_NO_LOOKUPS to the
value of "true" .
I created a simple test application for testing mitigations,
https://github.com/lhotari/log4shell-mitigation-tester . That might be
useful in ensuring that you don't have a copy-paste error or typo in the
mitigation.
Bruno Borges has also a sample for patching a K8S deployment. Something
like that could also work for Pulsar Helm Deployments unless you cannot
upgrade your deployment using the latest Pulsar Helm Chart version 2.7.6
which contains the system property based workaround.
I also added the LOG4J_FORMAT_MSG_NO_LOOKUPS=true mitigation to the
Dockerfile overlaying solution,
https://github.com/lhotari/pulsar-docker-images-patch-CVE-2021-44228 .
I hope others have found this work useful. Happy patching! Please be quick
if you want to avoid getting hacked. Pulsar is confirmed to be exploitable
with Log4Shell. I'm not aware of actual RCE, but at least reading security
tokens is possible using Log4Shell with released Pulsar versions and that's
why you should also cycle your credentials & security tokens after your
systems are patched.
If you are running Pulsar versions older than 2.7.1, they are vulnerable to
older security issues (CVE-2021-22160, excluding 2.6.4 which contains the
fix).
BR, Lari
On Fri, Dec 10, 2021 at 2:23 PM Lari Hotari <lh...@apache.org> wrote:
>
> As many of you might have already heard of this, there's a severe RCE
> 0-day exploit found in Log4J (2.0 <= Apache log4j <= 2.14.1).
> Blog post: https://www.lunasec.io/docs/blog/log4j-zero-day/
> CVE-2021-44228 in GitHub Security Advisory:
> https://github.com/advisories/GHSA-jfh8-c2jp-5v3q
>
> This also affects all Pulsar versions after 2.0.0-incubating since a
> vulnerable Log4J version is used. I'm not aware of a confirmed exploit for
> Pulsar. The fix to Pulsar is to upgrade to Log4J 2.15.0 . The PR is
> https://github.com/apache/pulsar/pull/13226 . The fix will be release as
> part of Pulsar 2.8.2 , 2.7.4 and 2.9.1 . Before the fixed version is
> available, there's an immediate workaround to mitigate the security issue.
>
> I'd like to share mitigation instructions for this severe vulnerability:
> - Add -Dlog4j2.formatMsgNoLookups=true system property to the JVM
> arguments of all Pulsar processes. There are multiple ways to achieve this
> in Pulsar. It can be added to either OPTS, PULSAR_GC or PULSAR_MEM
> environment variables.
> - Upgrade to Pulsar 2.8.2 , 2.7.4 or 2.9.1 once they are available.
>
> There's a PR to handle the adding of -Dlog4j2.formatMsgNoLookups=true
> system property in the Apache Pulsar Helm chart, that is
> https://github.com/apache/pulsar-helm-chart/pull/186 . Until that is
> available, the recommended approach is to add
> "-Dlog4j2.formatMsgNoLookups=true" to OPTS, PULSAR_GC or PULSAR_MEM
> manually and ensure that the Java process picks up the system property.
> It's also necessary to check that the property doesn't have typos. The
> setting is case sensitive.
>
> Please patch your productions systems asap!
>
> BR, Lari Hotari
>
>
>
>
>
>
>
>
Re: [Security] CVE-2021-44228 severe RCE 0-day exploit found in Log4J - affects also Pulsar - mitigation instructions
Posted by Lari Hotari <lh...@apache.org>.
Here's an additional workaround for patching existing Docker images with an
additional overlay which upgrades Log4J to 2.15.0:
https://github.com/lhotari/pulsar-docker-images-patch-CVE-2021-44228 .
BR, Lari
On Fri, Dec 10, 2021 at 2:23 PM Lari Hotari <lh...@apache.org> wrote:
>
> As many of you might have already heard of this, there's a severe RCE
> 0-day exploit found in Log4J (2.0 <= Apache log4j <= 2.14.1).
> Blog post: https://www.lunasec.io/docs/blog/log4j-zero-day/
> CVE-2021-44228 in GitHub Security Advisory:
> https://github.com/advisories/GHSA-jfh8-c2jp-5v3q
>
> This also affects all Pulsar versions after 2.0.0-incubating since a
> vulnerable Log4J version is used. I'm not aware of a confirmed exploit for
> Pulsar. The fix to Pulsar is to upgrade to Log4J 2.15.0 . The PR is
> https://github.com/apache/pulsar/pull/13226 . The fix will be release as
> part of Pulsar 2.8.2 , 2.7.4 and 2.9.1 . Before the fixed version is
> available, there's an immediate workaround to mitigate the security issue.
>
> I'd like to share mitigation instructions for this severe vulnerability:
> - Add -Dlog4j2.formatMsgNoLookups=true system property to the JVM
> arguments of all Pulsar processes. There are multiple ways to achieve this
> in Pulsar. It can be added to either OPTS, PULSAR_GC or PULSAR_MEM
> environment variables.
> - Upgrade to Pulsar 2.8.2 , 2.7.4 or 2.9.1 once they are available.
>
> There's a PR to handle the adding of -Dlog4j2.formatMsgNoLookups=true
> system property in the Apache Pulsar Helm chart, that is
> https://github.com/apache/pulsar-helm-chart/pull/186 . Until that is
> available, the recommended approach is to add
> "-Dlog4j2.formatMsgNoLookups=true" to OPTS, PULSAR_GC or PULSAR_MEM
> manually and ensure that the Java process picks up the system property.
> It's also necessary to check that the property doesn't have typos. The
> setting is case sensitive.
>
> Please patch your productions systems asap!
>
> BR, Lari Hotari
>
>
>
>
>
>
>
>