You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by humanitarian <ro...@humanitarian.com> on 2018/08/25 18:59:52 UTC
solr crypto mining hack...
Hi All,
I am struggling to fight an attack were the solr user is being used to
crate files used for mining cryptocurrencies. The files are being
created in the /var/tmp and /tmp folders.
It will use 100% of the CPU.
I am looking for help in stopping these attacks.
All files are created under the solr user.
Any help would be greatly appreciated.
Thank you!
Re: solr crypto mining hack...
Posted by Walter Underwood <wu...@wunderwood.org>.
What version of Solr are you running? On what OS? With what version of Java?
wunder
Walter Underwood
wunder@wunderwood.org
http://observer.wunderwood.org/ (my blog)
> On Aug 25, 2018, at 11:59 AM, humanitarian <ro...@humanitarian.com> wrote:
>
> Hi All,
>
> I am struggling to fight an attack were the solr user is being used to
> crate files used for mining cryptocurrencies. The files are being
> created in the /var/tmp and /tmp folders.
>
> It will use 100% of the CPU.
>
> I am looking for help in stopping these attacks.
>
> All files are created under the solr user.
>
> Any help would be greatly appreciated.
>
>
> Thank you!
>
Re: solr crypto mining hack...
Posted by Walter Underwood <wu...@wunderwood.org>.
This is exactly why I asked what Solr version they were running, to see if they
had the vulnerability. We still have no idea about Solr, OS, or JVM versions.
wunder
Walter Underwood
wunder@wunderwood.org
http://observer.wunderwood.org/ (my blog)
> On Aug 26, 2018, at 5:25 AM, Shawn Heisey <ap...@elyograg.org> wrote:
>
> On 8/25/2018 9:21 PM, Erick Erickson wrote:
>> This is probably CVE-2017-12629, see SOLR-11482, SOLR-11477 for
>> specific versions that have been patched and upgrade. You also need
>> to, as Jan suggested, figure out a way to be absolutely sure that your
>> installation is cleaned before you can be sure that you're protected.
>>
>> Also see: https://www.bleepingcomputer.com/news/security/coinminer-campaigns-target-redis-apache-solr-and-windows-servers/
>
> Erick is awesome. We can usually count on Erick to research a problem and find the likely culprit. This is a vulnerability in the way that Solr handles XML parsing. Certain operations were allowed in the name of flexibility. It was not realized at the time of implementation that it was opening a security hole.
>
> Here's the Solr announcement about the related vulnerabilities and the versions with a fix:
>
> http://mail-archives.us.apache.org/mod_mbox/www-announce/201710.mbox/%3CCAOOKt51UO_6Vy%3Dj8W%3Dx1pMbLW9VJfZyFWz7pAnXJC_OAdSZubA%40mail.gmail.com%3E
>
> In order to exploit that vulnerability, somebody must have network access to a Solr install. Such access could be obtained by first breaking into another piece of software, like a web server.
>
> The recommendation I mentioned about making sure only trusted parties can reach Solr is in the documentation, and on the wiki:
>
> https://lucene.apache.org/solr/guide/7_4/securing-solr.html
> https://wiki.apache.org/solr/SolrSecurity#Need_for_firewall
>
> That second link covers some other possible attack vectors.
>
> Thanks,
> Shawn
>
Re: solr crypto mining hack...
Posted by Shawn Heisey <ap...@elyograg.org>.
On 8/25/2018 9:21 PM, Erick Erickson wrote:
> This is probably CVE-2017-12629, see SOLR-11482, SOLR-11477 for
> specific versions that have been patched and upgrade. You also need
> to, as Jan suggested, figure out a way to be absolutely sure that your
> installation is cleaned before you can be sure that you're protected.
>
> Also see: https://www.bleepingcomputer.com/news/security/coinminer-campaigns-target-redis-apache-solr-and-windows-servers/
Erick is awesome. We can usually count on Erick to research a problem
and find the likely culprit. This is a vulnerability in the way that
Solr handles XML parsing. Certain operations were allowed in the name
of flexibility. It was not realized at the time of implementation that
it was opening a security hole.
Here's the Solr announcement about the related vulnerabilities and the
versions with a fix:
http://mail-archives.us.apache.org/mod_mbox/www-announce/201710.mbox/%3CCAOOKt51UO_6Vy%3Dj8W%3Dx1pMbLW9VJfZyFWz7pAnXJC_OAdSZubA%40mail.gmail.com%3E
In order to exploit that vulnerability, somebody must have network
access to a Solr install. Such access could be obtained by first
breaking into another piece of software, like a web server.
The recommendation I mentioned about making sure only trusted parties
can reach Solr is in the documentation, and on the wiki:
https://lucene.apache.org/solr/guide/7_4/securing-solr.html
https://wiki.apache.org/solr/SolrSecurity#Need_for_firewall
That second link covers some other possible attack vectors.
Thanks,
Shawn
Re: solr crypto mining hack...
Posted by Erick Erickson <er...@gmail.com>.
This is probably CVE-2017-12629, see SOLR-11482, SOLR-11477 for
specific versions that have been patched and upgrade. You also need
to, as Jan suggested, figure out a way to be absolutely sure that your
installation is cleaned before you can be sure that you're protected.
Also see: https://www.bleepingcomputer.com/news/security/coinminer-campaigns-target-redis-apache-solr-and-windows-servers/
Best,
Erick
On Sat, Aug 25, 2018 at 7:43 PM Tim Casey <tc...@gmail.com> wrote:
>
> I am not sure how solr is exactly set up currently, much less on any
> specific system. But, for operations which are largely reading, *maybe*
> like a query, you might be able run on a read only partition.
>
> A firewall is a lot less work and a good start, like 90% of the problem.
>
> To do this, you bring up the system with two partitions, one read-only and
> one read-write. You chroot into the readonly partition and start the query
> server. This process would only be allowed to run queries and would only
> be read only. The indexing process, if exposed to the world, would have to
> have a firewall in front of it with white listing of various parts of the
> world. (Preferably with an ssh enabled exchange, but security is hard,
> lets go shopping.)
>
> This is complicated to set up. If I recall, we had to build up the used
> parts of the OS as a sub-mount and then run there. However, once it is
> mounted read only, any subprocess in that root would not be allowed to
> write. As an simple example, this type of change requires network logging
> and then a whole lot of qualification to get to useful production.
>
> On Sat, Aug 25, 2018 at 7:10 PM Shawn Heisey <ap...@elyograg.org> wrote:
>
> > On 8/25/2018 12:59 PM, humanitarian wrote:
> > > I am struggling to fight an attack were the solr user is being used to
> > > crate files used for mining cryptocurrencies. The files are being
> > > created in the /var/tmp and /tmp folders.
> > >
> > > It will use 100% of the CPU.
> > >
> > > I am looking for help in stopping these attacks.
> > >
> > > All files are created under the solr user.
> >
> > At least some of what I'm writing is a repeat of what was said in
> > SOLR-12700 -- an issue in Jira with a description that's extremely
> > similar to the subject of this message.
> >
> > The Solr server should never be exposed to untrusted parties, especially
> > the open Internet. This is probably our number one recommendation for
> > security. If an attacker cannot reach a server, they cannot compromise it.
> >
> > There are a lot of possible vectors in Solr that could have been used to
> > compromise the system. Most of the vulnerabilities that have been found
> > are in third-party dependencies that Solr utilizes to create certain
> > functionality.
> >
> > This is not the first time I've encountered this. On at least one other
> > occasion, a user found weird software on their system running as the
> > solr user. It turned out to be a crypto-mining program.
> >
> > If you have Solr logs from when your system was compromised, we can
> > check them to see if there's anything useful. There may not be anything
> > useful. One of the better logs for tracking this sort of thing is the
> > Jetty request log, but this log is not enabled by default in the Solr
> > download. This log will be the only way you can get the IP address
> > making requests.
> >
> > Lock down your Solr server(s) so that only trusted network addresses can
> > reach them. This will need to be done outside of Solr. The operating
> > system will have a firewall available, and your network equipment might
> > also have filtering capability.
> >
> > Thanks,
> > Shawn
> >
> >
Re: solr crypto mining hack...
Posted by Tim Casey <tc...@gmail.com>.
I am not sure how solr is exactly set up currently, much less on any
specific system. But, for operations which are largely reading, *maybe*
like a query, you might be able run on a read only partition.
A firewall is a lot less work and a good start, like 90% of the problem.
To do this, you bring up the system with two partitions, one read-only and
one read-write. You chroot into the readonly partition and start the query
server. This process would only be allowed to run queries and would only
be read only. The indexing process, if exposed to the world, would have to
have a firewall in front of it with white listing of various parts of the
world. (Preferably with an ssh enabled exchange, but security is hard,
lets go shopping.)
This is complicated to set up. If I recall, we had to build up the used
parts of the OS as a sub-mount and then run there. However, once it is
mounted read only, any subprocess in that root would not be allowed to
write. As an simple example, this type of change requires network logging
and then a whole lot of qualification to get to useful production.
On Sat, Aug 25, 2018 at 7:10 PM Shawn Heisey <ap...@elyograg.org> wrote:
> On 8/25/2018 12:59 PM, humanitarian wrote:
> > I am struggling to fight an attack were the solr user is being used to
> > crate files used for mining cryptocurrencies. The files are being
> > created in the /var/tmp and /tmp folders.
> >
> > It will use 100% of the CPU.
> >
> > I am looking for help in stopping these attacks.
> >
> > All files are created under the solr user.
>
> At least some of what I'm writing is a repeat of what was said in
> SOLR-12700 -- an issue in Jira with a description that's extremely
> similar to the subject of this message.
>
> The Solr server should never be exposed to untrusted parties, especially
> the open Internet. This is probably our number one recommendation for
> security. If an attacker cannot reach a server, they cannot compromise it.
>
> There are a lot of possible vectors in Solr that could have been used to
> compromise the system. Most of the vulnerabilities that have been found
> are in third-party dependencies that Solr utilizes to create certain
> functionality.
>
> This is not the first time I've encountered this. On at least one other
> occasion, a user found weird software on their system running as the
> solr user. It turned out to be a crypto-mining program.
>
> If you have Solr logs from when your system was compromised, we can
> check them to see if there's anything useful. There may not be anything
> useful. One of the better logs for tracking this sort of thing is the
> Jetty request log, but this log is not enabled by default in the Solr
> download. This log will be the only way you can get the IP address
> making requests.
>
> Lock down your Solr server(s) so that only trusted network addresses can
> reach them. This will need to be done outside of Solr. The operating
> system will have a firewall available, and your network equipment might
> also have filtering capability.
>
> Thanks,
> Shawn
>
>
Re: solr crypto mining hack...
Posted by Shawn Heisey <ap...@elyograg.org>.
On 8/25/2018 12:59 PM, humanitarian wrote:
> I am struggling to fight an attack were the solr user is being used to
> crate files used for mining cryptocurrencies. The files are being
> created in the /var/tmp and /tmp folders.
>
> It will use 100% of the CPU.
>
> I am looking for help in stopping these attacks.
>
> All files are created under the solr user.
At least some of what I'm writing is a repeat of what was said in
SOLR-12700 -- an issue in Jira with a description that's extremely
similar to the subject of this message.
The Solr server should never be exposed to untrusted parties, especially
the open Internet. This is probably our number one recommendation for
security. If an attacker cannot reach a server, they cannot compromise it.
There are a lot of possible vectors in Solr that could have been used to
compromise the system. Most of the vulnerabilities that have been found
are in third-party dependencies that Solr utilizes to create certain
functionality.
This is not the first time I've encountered this. On at least one other
occasion, a user found weird software on their system running as the
solr user. It turned out to be a crypto-mining program.
If you have Solr logs from when your system was compromised, we can
check them to see if there's anything useful. There may not be anything
useful. One of the better logs for tracking this sort of thing is the
Jetty request log, but this log is not enabled by default in the Solr
download. This log will be the only way you can get the IP address
making requests.
Lock down your Solr server(s) so that only trusted network addresses can
reach them. This will need to be done outside of Solr. The operating
system will have a firewall available, and your network equipment might
also have filtering capability.
Thanks,
Shawn