You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by dhurandar S <dh...@gmail.com> on 2020/04/24 01:49:22 UTC
Re: using S3 as the Directory for Solr
Hi Jan,
Thank you for your reply. The reason we are looking for S3 is since the
volume is close to 10 Petabytes.
We are okay to have higher latency of say twice or thrice that of placing
data on the local disk. But we have a requirement to have long-range data
and providing Seach capability on that. Every other storage apart from S3
turned out to be very expensive at that scale.
Basically I want to replace
-Dsolr.directoryFactory=HdfsDirectoryFactory \
with S3 based implementation.
regards,
Rahul
On Thu, Apr 23, 2020 at 3:12 AM Jan Høydahl <ja...@cominvent.com> wrote:
> Hi,
>
> Is your data so partitioned that it makes sense to consider splitting up
> in multiple collections and make some arrangement that will keep only
> a few collections live at a time, loading index files from S3 on demand?
>
> I cannot see how an S3 directory would be able to effectively cache files
> in S3 and what units the index files would be stored as?
>
> Have you investigated EFS as an alternative? That would look like a
> normal filesystem to Solr but might be cheaper storage wise, but much
> slower.
>
> Jan
>
> > 23. apr. 2020 kl. 06:57 skrev dhurandar S <dh...@gmail.com>:
> >
> > Hi,
> >
> > I am looking to use S3 as the place to store indexes. Just how Solr uses
> > HdfsDirectory to store the index and all the other documents.
> >
> > We want to provide a search capability that is okay to be a little slow
> but
> > cheaper in terms of the cost. We have close to 2 petabytes of data on
> which
> > we want to provide the Search using Solr.
> >
> > Are there any open-source implementations around using S3 as the
> Directory
> > for Solr ??
> >
> > Any recommendations on this approach?
> >
> > regards,
> > Rahul
>
>
Re: using S3 as the Directory for Solr
Posted by Kevin Risden <kr...@apache.org>.
Solr's use of the HdfsDirectory may work over S3 directly if you use the
Hadoop AWS binding - s3a [1]. The idea is to replace hdfs:// with
s3a://bucket/. Since S3 is eventually consistent, the Hadoop AWS s3a
project has s3guard to help with consistent listing. If you are only doing
queries (no indexing) with Solr you may not need to worry about the
eventual consistency.
There was some previous exploration in this area with Solr 6.x/7.x, but it
should be much better with Solr 8.x due to the upgraded Hadoop 3.x
dependency. I haven't done any stress testing of this, but I made sure it
at least in theory would connect. I could index and query some small
datasets stored via s3a.
Using the HdfsDirectory with s3a will most likely be slower as already
pointed out. You might get reasonable performance depending on the nodes
used and tuning the HdfsDirectory block cache.
[1]
https://hadoop.apache.org/docs/current/hadoop-aws/tools/hadoop-aws/index.html
Kevin Risden
On Fri, Apr 24, 2020 at 1:19 PM dhurandar S <dh...@gmail.com> wrote:
> Its 10 PB of source data, But we do have indexes on most of the attributes.
> 80% or so
> We have a need to support such large data and we have use cases of finding
> a needle in the haystack kinda scenario.
> Most of our users are used to Search query language or Solr in addition to
> SQL. So we would have both the interfaces.
>
> We store the actual data in S3 in Parquet and have Presto query it using
> SQL (Presto is similar to Hive but much much faster).
>
> We also now want to store the indexes in S3 we have leeway in query
> interactivity performance, the key thing here is support finding the
> needle in the haystack pattern and supporting really long-range data in a
> cheaper fashion
>
> regards,
> Rahul
>
>
> On Thu, Apr 23, 2020 at 7:41 PM Walter Underwood <wu...@wunderwood.org>
> wrote:
>
> > It will be a lot more than 2X or 3X slower. Years ago, I accidentally put
> > Solr indexes on an NFS mounted filesystem and it was 100X slower. S3
> would
> > be a lot slower than that.
> >
> > Are you doing relevance-ranked searches on all that data? That is the
> only
> > reason to use Solr instead of some other solution.
> >
> > I’d use Apache Hive, or whatever has replaced it. That is what Facebook
> > wrote to do searches on their multi-petabyte logs.
> >
> > https://hive.apache.org
> >
> > More options.
> >
> > https://jethro.io/hadoop-hive
> > https://mapr.com/why-hadoop/sql-hadoop/sql-hadoop-details/
> >
> > wunder
> > Walter Underwood
> > wunder@wunderwood.org
> > http://observer.wunderwood.org/ (my blog)
> >
> > > On Apr 23, 2020, at 7:29 PM, Christopher Schultz <
> > chris@christopherschultz.net> wrote:
> > >
> > > -----BEGIN PGP SIGNED MESSAGE-----
> > > Hash: SHA256
> > >
> > > Rahul,
> > >
> > > On 4/23/20 21:49, dhurandar S wrote:
> > >> Thank you for your reply. The reason we are looking for S3 is since
> > >> the volume is close to 10 Petabytes. We are okay to have higher
> > >> latency of say twice or thrice that of placing data on the local
> > >> disk. But we have a requirement to have long-range data and
> > >> providing Seach capability on that. Every other storage apart from
> > >> S3 turned out to be very expensive at that scale.
> > >>
> > >> Basically I want to replace
> > >>
> > >> -Dsolr.directoryFactory=HdfsDirectoryFactory \
> > >>
> > >> with S3 based implementation.
> > >
> > > Can you clarify whether you have 10 PiB of /source data/ or 10 PiB of
> > > /index data/?
> > >
> > > You can theoretically store your source data anywhere, of course. 10
> > > PiB sounds like a truly enormous index.
> > >
> > > - -chris
> > >
> > >> On Thu, Apr 23, 2020 at 3:12 AM Jan Høydahl <ja...@cominvent.com>
> > >> wrote:
> > >>
> > >>> Hi,
> > >>>
> > >>> Is your data so partitioned that it makes sense to consider
> > >>> splitting up in multiple collections and make some arrangement
> > >>> that will keep only a few collections live at a time, loading
> > >>> index files from S3 on demand?
> > >>>
> > >>> I cannot see how an S3 directory would be able to effectively
> > >>> cache files in S3 and what units the index files would be stored
> > >>> as?
> > >>>
> > >>> Have you investigated EFS as an alternative? That would look like
> > >>> a normal filesystem to Solr but might be cheaper storage wise,
> > >>> but much slower.
> > >>>
> > >>> Jan
> > >>>
> > >>>> 23. apr. 2020 kl. 06:57 skrev dhurandar S
> > >>>> <dh...@gmail.com>:
> > >>>>
> > >>>> Hi,
> > >>>>
> > >>>> I am looking to use S3 as the place to store indexes. Just how
> > >>>> Solr uses HdfsDirectory to store the index and all the other
> > >>>> documents.
> > >>>>
> > >>>> We want to provide a search capability that is okay to be a
> > >>>> little slow
> > >>> but
> > >>>> cheaper in terms of the cost. We have close to 2 petabytes of
> > >>>> data on
> > >>> which
> > >>>> we want to provide the Search using Solr.
> > >>>>
> > >>>> Are there any open-source implementations around using S3 as
> > >>>> the
> > >>> Directory
> > >>>> for Solr ??
> > >>>>
> > >>>> Any recommendations on this approach?
> > >>>>
> > >>>> regards, Rahul
> > >>>
> > >>>
> > >>
> > > -----BEGIN PGP SIGNATURE-----
> > > Comment: Using GnuPG with Thunderbird - https://www.enigmail.net/
> > >
> > > iQIyBAEBCAAdFiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAl6iTwUACgkQHPApP6U8
> > > pFjRaw/4sGbH286gZJe+wfKsLc4JPvyJZjjwVDCdpiR2SHt50IA23wYSK97R6xRj
> > > dbWWReA7C3JNWp6x21i8Bb6sIeLDnotbc7IOSmOMuNep1BtVaYBMJ8wyW6uUtXf6
> > > hQbY0Ew93ZhDlS9CWMJqbQtWfrQEqH51Xbz+4uqqvJU8Bq9o9Vv0rnuVp/5f73lV
> > > ihek0sbA73oGle0gC5NFmrKItnn+14X8vIxUC8JRZlY4rDSiOdOcIil3DExxOQNQ
> > > UodIvwKKhzALFY77PeGSSjKiy0X3JJ1rKzLeIBrW0JCNMprYLzL2CQjZ5F09MraZ
> > > WxXdA64lEg2diEwHywNrsaaygbEZYTWd8gaeGA7kzCk78Y2KuhWuEQej6KmE3Iq2
> > > AW+K7JgFakUpzB5oorCtKNLQOqFHX85ne57gCYKr42S3Htfxmf98pBdudQy4RvuT
> > > +tJvGYx8NLqgeOoZN4u+G/8WunlzUC+u2vUxVcIoK3Ozz0usMioFDqn69vmOxxoH
> > > cN2Y4T1ZZZGtndiAGZww1JXKAbVN0U41isXg2F8tHQV9dxaeoYDQ/xYbAoWEhhlM
> > > SVtEdr76eMJ08T6h5711gtrhSK+RQFPD2Jbr8B/Xl063xPfN2TpqmcJCKXkucvpc
> > > CEDLFqeKX6qIRZDgMf8EICmbFl6aF5knbDP0MkyYk4urB+uFaw==
> > > =Y/6Y
> > > -----END PGP SIGNATURE-----
> >
> >
>
Re: using S3 as the Directory for Solr
Posted by dhurandar S <dh...@gmail.com>.
Its 10 PB of source data, But we do have indexes on most of the attributes.
80% or so
We have a need to support such large data and we have use cases of finding
a needle in the haystack kinda scenario.
Most of our users are used to Search query language or Solr in addition to
SQL. So we would have both the interfaces.
We store the actual data in S3 in Parquet and have Presto query it using
SQL (Presto is similar to Hive but much much faster).
We also now want to store the indexes in S3 we have leeway in query
interactivity performance, the key thing here is support finding the
needle in the haystack pattern and supporting really long-range data in a
cheaper fashion
regards,
Rahul
On Thu, Apr 23, 2020 at 7:41 PM Walter Underwood <wu...@wunderwood.org>
wrote:
> It will be a lot more than 2X or 3X slower. Years ago, I accidentally put
> Solr indexes on an NFS mounted filesystem and it was 100X slower. S3 would
> be a lot slower than that.
>
> Are you doing relevance-ranked searches on all that data? That is the only
> reason to use Solr instead of some other solution.
>
> I’d use Apache Hive, or whatever has replaced it. That is what Facebook
> wrote to do searches on their multi-petabyte logs.
>
> https://hive.apache.org
>
> More options.
>
> https://jethro.io/hadoop-hive
> https://mapr.com/why-hadoop/sql-hadoop/sql-hadoop-details/
>
> wunder
> Walter Underwood
> wunder@wunderwood.org
> http://observer.wunderwood.org/ (my blog)
>
> > On Apr 23, 2020, at 7:29 PM, Christopher Schultz <
> chris@christopherschultz.net> wrote:
> >
> > -----BEGIN PGP SIGNED MESSAGE-----
> > Hash: SHA256
> >
> > Rahul,
> >
> > On 4/23/20 21:49, dhurandar S wrote:
> >> Thank you for your reply. The reason we are looking for S3 is since
> >> the volume is close to 10 Petabytes. We are okay to have higher
> >> latency of say twice or thrice that of placing data on the local
> >> disk. But we have a requirement to have long-range data and
> >> providing Seach capability on that. Every other storage apart from
> >> S3 turned out to be very expensive at that scale.
> >>
> >> Basically I want to replace
> >>
> >> -Dsolr.directoryFactory=HdfsDirectoryFactory \
> >>
> >> with S3 based implementation.
> >
> > Can you clarify whether you have 10 PiB of /source data/ or 10 PiB of
> > /index data/?
> >
> > You can theoretically store your source data anywhere, of course. 10
> > PiB sounds like a truly enormous index.
> >
> > - -chris
> >
> >> On Thu, Apr 23, 2020 at 3:12 AM Jan Høydahl <ja...@cominvent.com>
> >> wrote:
> >>
> >>> Hi,
> >>>
> >>> Is your data so partitioned that it makes sense to consider
> >>> splitting up in multiple collections and make some arrangement
> >>> that will keep only a few collections live at a time, loading
> >>> index files from S3 on demand?
> >>>
> >>> I cannot see how an S3 directory would be able to effectively
> >>> cache files in S3 and what units the index files would be stored
> >>> as?
> >>>
> >>> Have you investigated EFS as an alternative? That would look like
> >>> a normal filesystem to Solr but might be cheaper storage wise,
> >>> but much slower.
> >>>
> >>> Jan
> >>>
> >>>> 23. apr. 2020 kl. 06:57 skrev dhurandar S
> >>>> <dh...@gmail.com>:
> >>>>
> >>>> Hi,
> >>>>
> >>>> I am looking to use S3 as the place to store indexes. Just how
> >>>> Solr uses HdfsDirectory to store the index and all the other
> >>>> documents.
> >>>>
> >>>> We want to provide a search capability that is okay to be a
> >>>> little slow
> >>> but
> >>>> cheaper in terms of the cost. We have close to 2 petabytes of
> >>>> data on
> >>> which
> >>>> we want to provide the Search using Solr.
> >>>>
> >>>> Are there any open-source implementations around using S3 as
> >>>> the
> >>> Directory
> >>>> for Solr ??
> >>>>
> >>>> Any recommendations on this approach?
> >>>>
> >>>> regards, Rahul
> >>>
> >>>
> >>
> > -----BEGIN PGP SIGNATURE-----
> > Comment: Using GnuPG with Thunderbird - https://www.enigmail.net/
> >
> > iQIyBAEBCAAdFiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAl6iTwUACgkQHPApP6U8
> > pFjRaw/4sGbH286gZJe+wfKsLc4JPvyJZjjwVDCdpiR2SHt50IA23wYSK97R6xRj
> > dbWWReA7C3JNWp6x21i8Bb6sIeLDnotbc7IOSmOMuNep1BtVaYBMJ8wyW6uUtXf6
> > hQbY0Ew93ZhDlS9CWMJqbQtWfrQEqH51Xbz+4uqqvJU8Bq9o9Vv0rnuVp/5f73lV
> > ihek0sbA73oGle0gC5NFmrKItnn+14X8vIxUC8JRZlY4rDSiOdOcIil3DExxOQNQ
> > UodIvwKKhzALFY77PeGSSjKiy0X3JJ1rKzLeIBrW0JCNMprYLzL2CQjZ5F09MraZ
> > WxXdA64lEg2diEwHywNrsaaygbEZYTWd8gaeGA7kzCk78Y2KuhWuEQej6KmE3Iq2
> > AW+K7JgFakUpzB5oorCtKNLQOqFHX85ne57gCYKr42S3Htfxmf98pBdudQy4RvuT
> > +tJvGYx8NLqgeOoZN4u+G/8WunlzUC+u2vUxVcIoK3Ozz0usMioFDqn69vmOxxoH
> > cN2Y4T1ZZZGtndiAGZww1JXKAbVN0U41isXg2F8tHQV9dxaeoYDQ/xYbAoWEhhlM
> > SVtEdr76eMJ08T6h5711gtrhSK+RQFPD2Jbr8B/Xl063xPfN2TpqmcJCKXkucvpc
> > CEDLFqeKX6qIRZDgMf8EICmbFl6aF5knbDP0MkyYk4urB+uFaw==
> > =Y/6Y
> > -----END PGP SIGNATURE-----
>
>
Re: using S3 as the Directory for Solr
Posted by Walter Underwood <wu...@wunderwood.org>.
It will be a lot more than 2X or 3X slower. Years ago, I accidentally put Solr indexes on an NFS mounted filesystem and it was 100X slower. S3 would be a lot slower than that.
Are you doing relevance-ranked searches on all that data? That is the only reason to use Solr instead of some other solution.
I’d use Apache Hive, or whatever has replaced it. That is what Facebook wrote to do searches on their multi-petabyte logs.
https://hive.apache.org
More options.
https://jethro.io/hadoop-hive
https://mapr.com/why-hadoop/sql-hadoop/sql-hadoop-details/
wunder
Walter Underwood
wunder@wunderwood.org
http://observer.wunderwood.org/ (my blog)
> On Apr 23, 2020, at 7:29 PM, Christopher Schultz <ch...@christopherschultz.net> wrote:
>
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA256
>
> Rahul,
>
> On 4/23/20 21:49, dhurandar S wrote:
>> Thank you for your reply. The reason we are looking for S3 is since
>> the volume is close to 10 Petabytes. We are okay to have higher
>> latency of say twice or thrice that of placing data on the local
>> disk. But we have a requirement to have long-range data and
>> providing Seach capability on that. Every other storage apart from
>> S3 turned out to be very expensive at that scale.
>>
>> Basically I want to replace
>>
>> -Dsolr.directoryFactory=HdfsDirectoryFactory \
>>
>> with S3 based implementation.
>
> Can you clarify whether you have 10 PiB of /source data/ or 10 PiB of
> /index data/?
>
> You can theoretically store your source data anywhere, of course. 10
> PiB sounds like a truly enormous index.
>
> - -chris
>
>> On Thu, Apr 23, 2020 at 3:12 AM Jan Høydahl <ja...@cominvent.com>
>> wrote:
>>
>>> Hi,
>>>
>>> Is your data so partitioned that it makes sense to consider
>>> splitting up in multiple collections and make some arrangement
>>> that will keep only a few collections live at a time, loading
>>> index files from S3 on demand?
>>>
>>> I cannot see how an S3 directory would be able to effectively
>>> cache files in S3 and what units the index files would be stored
>>> as?
>>>
>>> Have you investigated EFS as an alternative? That would look like
>>> a normal filesystem to Solr but might be cheaper storage wise,
>>> but much slower.
>>>
>>> Jan
>>>
>>>> 23. apr. 2020 kl. 06:57 skrev dhurandar S
>>>> <dh...@gmail.com>:
>>>>
>>>> Hi,
>>>>
>>>> I am looking to use S3 as the place to store indexes. Just how
>>>> Solr uses HdfsDirectory to store the index and all the other
>>>> documents.
>>>>
>>>> We want to provide a search capability that is okay to be a
>>>> little slow
>>> but
>>>> cheaper in terms of the cost. We have close to 2 petabytes of
>>>> data on
>>> which
>>>> we want to provide the Search using Solr.
>>>>
>>>> Are there any open-source implementations around using S3 as
>>>> the
>>> Directory
>>>> for Solr ??
>>>>
>>>> Any recommendations on this approach?
>>>>
>>>> regards, Rahul
>>>
>>>
>>
> -----BEGIN PGP SIGNATURE-----
> Comment: Using GnuPG with Thunderbird - https://www.enigmail.net/
>
> iQIyBAEBCAAdFiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAl6iTwUACgkQHPApP6U8
> pFjRaw/4sGbH286gZJe+wfKsLc4JPvyJZjjwVDCdpiR2SHt50IA23wYSK97R6xRj
> dbWWReA7C3JNWp6x21i8Bb6sIeLDnotbc7IOSmOMuNep1BtVaYBMJ8wyW6uUtXf6
> hQbY0Ew93ZhDlS9CWMJqbQtWfrQEqH51Xbz+4uqqvJU8Bq9o9Vv0rnuVp/5f73lV
> ihek0sbA73oGle0gC5NFmrKItnn+14X8vIxUC8JRZlY4rDSiOdOcIil3DExxOQNQ
> UodIvwKKhzALFY77PeGSSjKiy0X3JJ1rKzLeIBrW0JCNMprYLzL2CQjZ5F09MraZ
> WxXdA64lEg2diEwHywNrsaaygbEZYTWd8gaeGA7kzCk78Y2KuhWuEQej6KmE3Iq2
> AW+K7JgFakUpzB5oorCtKNLQOqFHX85ne57gCYKr42S3Htfxmf98pBdudQy4RvuT
> +tJvGYx8NLqgeOoZN4u+G/8WunlzUC+u2vUxVcIoK3Ozz0usMioFDqn69vmOxxoH
> cN2Y4T1ZZZGtndiAGZww1JXKAbVN0U41isXg2F8tHQV9dxaeoYDQ/xYbAoWEhhlM
> SVtEdr76eMJ08T6h5711gtrhSK+RQFPD2Jbr8B/Xl063xPfN2TpqmcJCKXkucvpc
> CEDLFqeKX6qIRZDgMf8EICmbFl6aF5knbDP0MkyYk4urB+uFaw==
> =Y/6Y
> -----END PGP SIGNATURE-----
Re: using S3 as the Directory for Solr
Posted by Christopher Schultz <ch...@christopherschultz.net>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256
Rahul,
On 4/23/20 21:49, dhurandar S wrote:
> Thank you for your reply. The reason we are looking for S3 is since
> the volume is close to 10 Petabytes. We are okay to have higher
> latency of say twice or thrice that of placing data on the local
> disk. But we have a requirement to have long-range data and
> providing Seach capability on that. Every other storage apart from
> S3 turned out to be very expensive at that scale.
>
> Basically I want to replace
>
> -Dsolr.directoryFactory=HdfsDirectoryFactory \
>
> with S3 based implementation.
Can you clarify whether you have 10 PiB of /source data/ or 10 PiB of
/index data/?
You can theoretically store your source data anywhere, of course. 10
PiB sounds like a truly enormous index.
- -chris
> On Thu, Apr 23, 2020 at 3:12 AM Jan Høydahl <ja...@cominvent.com>
> wrote:
>
>> Hi,
>>
>> Is your data so partitioned that it makes sense to consider
>> splitting up in multiple collections and make some arrangement
>> that will keep only a few collections live at a time, loading
>> index files from S3 on demand?
>>
>> I cannot see how an S3 directory would be able to effectively
>> cache files in S3 and what units the index files would be stored
>> as?
>>
>> Have you investigated EFS as an alternative? That would look like
>> a normal filesystem to Solr but might be cheaper storage wise,
>> but much slower.
>>
>> Jan
>>
>>> 23. apr. 2020 kl. 06:57 skrev dhurandar S
>>> <dh...@gmail.com>:
>>>
>>> Hi,
>>>
>>> I am looking to use S3 as the place to store indexes. Just how
>>> Solr uses HdfsDirectory to store the index and all the other
>>> documents.
>>>
>>> We want to provide a search capability that is okay to be a
>>> little slow
>> but
>>> cheaper in terms of the cost. We have close to 2 petabytes of
>>> data on
>> which
>>> we want to provide the Search using Solr.
>>>
>>> Are there any open-source implementations around using S3 as
>>> the
>> Directory
>>> for Solr ??
>>>
>>> Any recommendations on this approach?
>>>
>>> regards, Rahul
>>
>>
>
-----BEGIN PGP SIGNATURE-----
Comment: Using GnuPG with Thunderbird - https://www.enigmail.net/
iQIyBAEBCAAdFiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAl6iTwUACgkQHPApP6U8
pFjRaw/4sGbH286gZJe+wfKsLc4JPvyJZjjwVDCdpiR2SHt50IA23wYSK97R6xRj
dbWWReA7C3JNWp6x21i8Bb6sIeLDnotbc7IOSmOMuNep1BtVaYBMJ8wyW6uUtXf6
hQbY0Ew93ZhDlS9CWMJqbQtWfrQEqH51Xbz+4uqqvJU8Bq9o9Vv0rnuVp/5f73lV
ihek0sbA73oGle0gC5NFmrKItnn+14X8vIxUC8JRZlY4rDSiOdOcIil3DExxOQNQ
UodIvwKKhzALFY77PeGSSjKiy0X3JJ1rKzLeIBrW0JCNMprYLzL2CQjZ5F09MraZ
WxXdA64lEg2diEwHywNrsaaygbEZYTWd8gaeGA7kzCk78Y2KuhWuEQej6KmE3Iq2
AW+K7JgFakUpzB5oorCtKNLQOqFHX85ne57gCYKr42S3Htfxmf98pBdudQy4RvuT
+tJvGYx8NLqgeOoZN4u+G/8WunlzUC+u2vUxVcIoK3Ozz0usMioFDqn69vmOxxoH
cN2Y4T1ZZZGtndiAGZww1JXKAbVN0U41isXg2F8tHQV9dxaeoYDQ/xYbAoWEhhlM
SVtEdr76eMJ08T6h5711gtrhSK+RQFPD2Jbr8B/Xl063xPfN2TpqmcJCKXkucvpc
CEDLFqeKX6qIRZDgMf8EICmbFl6aF5knbDP0MkyYk4urB+uFaw==
=Y/6Y
-----END PGP SIGNATURE-----
Re: using S3 as the Directory for Solr
Posted by Ilan Ginzburg <il...@gmail.com>.
Hi Rahul,
I don't have a direct answer to your question as I don't know of any
S3 based Directory implementation. Such an implementation would likely
be more complex than an HDFS one.
Reason is S3 has eventual consistency. When an S3 file is updated you
might still read the old content for a while, and similarly a
"directory listing" in S3 might not immediately show recently added
files. This basically requires storing metadata elsewhere (see for
example https://docs.aws.amazon.com/emr/latest/ManagementGuide/emr-fs.html
and https://blog.cloudera.com/introducing-s3guard-s3-consistency-for-apache-hadoop/).
At Salesforce we are working on storing SolrCloud search indexes on
S3/GCS in a way that allows these indexes to be shared between
replicas. We use Zookeeper to deal with S3's eventual consistency.
See Activate presentation last year
https://www.youtube.com/watch?v=6fE5KvOfb6A, and code at
https://github.com/apache/lucene-solr/tree/jira/SOLR-13101 (SHARED
replica type is a good entry point for looking at the changes).
In a nutshell, we use nodes' local disks as a cache that can be lost
when a node fails, and read/write segments from/to S3 as needed.
Queries are then always served from local disk and indexing always
happens locally then pushed to S3.
Could a solution to your use case be built instead using S3 based
backup/restore? Would require the right data partitioning (to get
reasonable restore then query latency for cold data and reasonable
backup time for modified indexes) and likely a friendly indexing
pipeline that can resubmit data indexed since last backup...
Ilan
On Fri, Apr 24, 2020 at 5:06 AM dhurandar S <dh...@gmail.com> wrote:
>
> Hi Jan,
>
> Thank you for your reply. The reason we are looking for S3 is since the volume is close to 10 Petabytes.
> We are okay to have higher latency of say twice or thrice that of placing data on the local disk. But we have a requirement to have long-range data and providing Seach capability on that. Every other storage apart from S3 turned out to be very expensive at that scale.
>
> Basically I want to replace
>
> -Dsolr.directoryFactory=HdfsDirectoryFactory \
>
> with S3 based implementation.
>
>
> regards,
> Rahul
>
>
>
>
>
> On Thu, Apr 23, 2020 at 3:12 AM Jan Høydahl <ja...@cominvent.com> wrote:
>>
>> Hi,
>>
>> Is your data so partitioned that it makes sense to consider splitting up
>> in multiple collections and make some arrangement that will keep only
>> a few collections live at a time, loading index files from S3 on demand?
>>
>> I cannot see how an S3 directory would be able to effectively cache files
>> in S3 and what units the index files would be stored as?
>>
>> Have you investigated EFS as an alternative? That would look like a
>> normal filesystem to Solr but might be cheaper storage wise, but much
>> slower.
>>
>> Jan
>>
>> > 23. apr. 2020 kl. 06:57 skrev dhurandar S <dh...@gmail.com>:
>> >
>> > Hi,
>> >
>> > I am looking to use S3 as the place to store indexes. Just how Solr uses
>> > HdfsDirectory to store the index and all the other documents.
>> >
>> > We want to provide a search capability that is okay to be a little slow but
>> > cheaper in terms of the cost. We have close to 2 petabytes of data on which
>> > we want to provide the Search using Solr.
>> >
>> > Are there any open-source implementations around using S3 as the Directory
>> > for Solr ??
>> >
>> > Any recommendations on this approach?
>> >
>> > regards,
>> > Rahul
>>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org