You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Odysci <od...@gmail.com> on 2020/07/28 20:39:02 UTC
Meow attacks
Folks,
I suspect one of our Zookeeper installations on AWS was subject to a Meow
attack (
https://arstechnica.com/information-technology/2020/07/more-than-1000-databases-have-been-nuked-by-mystery-meow-attack/
)
Basically, the configuration for one of our collections disappeared from
the Zookeeper tree (when looking at the Solr interface), and it left
several files ending in "-meow"
Before I realized it, I stopped and restarted the ZK and Solr machines (as
part of ubuntu updates), and when ZK didn't find the configuration for a
collection, it deleted the collection from Solr. At least that's what I
suspect happened.
Fortunately it affected a very small index and we had backups. But it is
very worrisome.
Has anyone had any problems with this?
Is there any type of log that I can check to sort out how this happened?
The ZK log complained that the configs for the collection were not there,
but that's about it.
and, is there a better way to protect against such attacks?
Thanks
Reinaldo
Re: Meow attacks
Posted by matthew sporleder <ms...@gmail.com>.
On Tue, Jul 28, 2020 at 4:39 PM Odysci <od...@gmail.com> wrote:
>
> Folks,
>
> I suspect one of our Zookeeper installations on AWS was subject to a Meow
> attack (
> https://arstechnica.com/information-technology/2020/07/more-than-1000-databases-have-been-nuked-by-mystery-meow-attack/
> )
>
> Basically, the configuration for one of our collections disappeared from
> the Zookeeper tree (when looking at the Solr interface), and it left
> several files ending in "-meow"
> Before I realized it, I stopped and restarted the ZK and Solr machines (as
> part of ubuntu updates), and when ZK didn't find the configuration for a
> collection, it deleted the collection from Solr. At least that's what I
> suspect happened.
>
> Fortunately it affected a very small index and we had backups. But it is
> very worrisome.
> Has anyone had any problems with this?
> Is there any type of log that I can check to sort out how this happened?
> The ZK log complained that the configs for the collection were not there,
> but that's about it.
>
> and, is there a better way to protect against such attacks?
> Thanks
>
> Reinaldo
Use VPC and private networks!
ask in ##aws on freenode if you are really lost
Re: Meow attacks
Posted by Odysci <od...@gmail.com>.
Folks,
thanks for the replies. We do use VPCs in AWS and the ZK ports are only
open to the solr machines (also in the same VPC). We're using Solr 8.3 and
ZK 3.5.6
We will investigate the Kerberos authentication.
thanks
Reinaldo
On Tue, Jul 28, 2020 at 6:03 PM Jörn Franke <jo...@gmail.com> wrote:
> In Addition what has been said before (use private networks/firewall
> rules) - activate Kerberos authentication so that only Solr hosts can write
> to Zk (the Solr client needs no write access) and use encryption where
> possible.
> Upgrade Solr to the latest version, use ssl , enable Kerberos, have
> clients not having any admin access on Solr (minimum privileges only!), use
> Solr whitelists to enable only clients that should access Solr, enable Java
> security manager (* to make it work with Kerberos auth you need for it to
> wait for a newer Solr version).
>
> > Am 28.07.2020 um 22:41 schrieb Odysci <od...@gmail.com>:
> >
> > Folks,
> >
> > I suspect one of our Zookeeper installations on AWS was subject to a Meow
> > attack (
> >
> https://arstechnica.com/information-technology/2020/07/more-than-1000-databases-have-been-nuked-by-mystery-meow-attack/
> > )
> >
> > Basically, the configuration for one of our collections disappeared from
> > the Zookeeper tree (when looking at the Solr interface), and it left
> > several files ending in "-meow"
> > Before I realized it, I stopped and restarted the ZK and Solr machines
> (as
> > part of ubuntu updates), and when ZK didn't find the configuration for a
> > collection, it deleted the collection from Solr. At least that's what I
> > suspect happened.
> >
> > Fortunately it affected a very small index and we had backups. But it is
> > very worrisome.
> > Has anyone had any problems with this?
> > Is there any type of log that I can check to sort out how this happened?
> > The ZK log complained that the configs for the collection were not there,
> > but that's about it.
> >
> > and, is there a better way to protect against such attacks?
> > Thanks
> >
> > Reinaldo
>
Re: Meow attacks
Posted by Jörn Franke <jo...@gmail.com>.
In Addition what has been said before (use private networks/firewall rules) - activate Kerberos authentication so that only Solr hosts can write to Zk (the Solr client needs no write access) and use encryption where possible.
Upgrade Solr to the latest version, use ssl , enable Kerberos, have clients not having any admin access on Solr (minimum privileges only!), use Solr whitelists to enable only clients that should access Solr, enable Java security manager (* to make it work with Kerberos auth you need for it to wait for a newer Solr version).
> Am 28.07.2020 um 22:41 schrieb Odysci <od...@gmail.com>:
>
> Folks,
>
> I suspect one of our Zookeeper installations on AWS was subject to a Meow
> attack (
> https://arstechnica.com/information-technology/2020/07/more-than-1000-databases-have-been-nuked-by-mystery-meow-attack/
> )
>
> Basically, the configuration for one of our collections disappeared from
> the Zookeeper tree (when looking at the Solr interface), and it left
> several files ending in "-meow"
> Before I realized it, I stopped and restarted the ZK and Solr machines (as
> part of ubuntu updates), and when ZK didn't find the configuration for a
> collection, it deleted the collection from Solr. At least that's what I
> suspect happened.
>
> Fortunately it affected a very small index and we had backups. But it is
> very worrisome.
> Has anyone had any problems with this?
> Is there any type of log that I can check to sort out how this happened?
> The ZK log complained that the configs for the collection were not there,
> but that's about it.
>
> and, is there a better way to protect against such attacks?
> Thanks
>
> Reinaldo
Re: Meow attacks
Posted by David Hastings <ha...@gmail.com>.
so, your zookeeper/solr servers have public facing addresses/ports?
On Tue, Jul 28, 2020 at 4:41 PM Odysci <od...@gmail.com> wrote:
> Folks,
>
> I suspect one of our Zookeeper installations on AWS was subject to a Meow
> attack (
>
> https://arstechnica.com/information-technology/2020/07/more-than-1000-databases-have-been-nuked-by-mystery-meow-attack/
> )
>
> Basically, the configuration for one of our collections disappeared from
> the Zookeeper tree (when looking at the Solr interface), and it left
> several files ending in "-meow"
> Before I realized it, I stopped and restarted the ZK and Solr machines (as
> part of ubuntu updates), and when ZK didn't find the configuration for a
> collection, it deleted the collection from Solr. At least that's what I
> suspect happened.
>
> Fortunately it affected a very small index and we had backups. But it is
> very worrisome.
> Has anyone had any problems with this?
> Is there any type of log that I can check to sort out how this happened?
> The ZK log complained that the configs for the collection were not there,
> but that's about it.
>
> and, is there a better way to protect against such attacks?
> Thanks
>
> Reinaldo
>