Filesystems supported by Solr

[Ritesh Chaman <ch...@gmail.com>]

Hi team

May I know what all filesystems are supported by Solr. For eg ADLS,WASB, S3
etc. Thanks.

Ritesh

Re: Filesystems supported by Solr

[Shawn Heisey <ap...@elyograg.org>]

On 2/20/2018 3:22 PM, Ritesh Chaman wrote:
> May I know what all filesystems are supported by Solr. For eg ADLS,WASB, S3
> etc. Thanks.

Solr supports whatever your operating system supports.  It will expect
file locking to work be fully functional, so things like NFS don't
always work.  Local filesystems are very much preferred, and will
generally have the best performance.

As far as I am aware, the only filesystem that Solr has explicit support
for (outside of what the OS itself provides) is HDFS.

https://lucene.apache.org/solr/guide/7_2/running-solr-on-hdfs.html

There may be plugins available to store indexes in other stores like S3,
but if those exist, I am not immediately aware of them.  They would be
third-party plugins, not supported by the Solr project.

Thanks,
Shawn

Re: Filesystems supported by Solr

[Shawn Heisey <ap...@elyograg.org>]

On 3/2/2018 2:30 PM, Ritesh Chaman wrote:
> I am trying to deploy solr on my ADLS subscription. can you tell me if that
> is tested and is compatible.

Walter says that this is storage related to Azure.

https://azure.microsoft.com/en-us/services/data-lake-store/

If this is what you are talking about, that page says they use HDFS
APIs.  Which means that you MIGHT be able to use it with the HDFS
support built into Solr.  You'd have to ask Microsoft what they can support.

But I can almost guarantee that unless you have plenty of memory in the
system for HDFS to cache that data, that it's going to be VERY slow,
just due to the amount of latency involved in accessing data over the
Internet.  Accessing the data after a reboot or a restart of the
particular service that caches the data is probably going to also be
extremely slow, even if you DO have plenty of memory for caching.

You'll see far better performance if you just install enough disk space
in your Solr servers to hold your index data, and install enough memory
that the OS can effectively cache that index data.

Thanks,
Shawn

Re: Filesystems supported by Solr

[Walter Underwood <wu...@wunderwood.org>]

From a quick google search, ADLS seems like the Azure version of S3. Putting Solr indexes on S3 would be unbelievably slow, if it worked at all. Years ago, I accidentally put indexes on NFS and it was 100X slower.

Tell us more about what you are trying to do. It is unusual to put Solr indexes on anything but a local filesystem. HDFS is the only exception I can think of.

wunder
Walter Underwood
wunder@wunderwood.org
http://observer.wunderwood.org/  (my blog)

> On Mar 2, 2018, at 1:30 PM, Ritesh Chaman <ch...@gmail.com> wrote:
> 
> Hi team
> 
> I am trying to deploy solr on my ADLS subscription. can you tell me if that
> is tested and is compatible.
> 
> Regards
> 
> On Tue, Feb 20, 2018 at 2:22 PM, Ritesh Chaman <ch...@gmail.com>
> wrote:
> 
>> Hi team
>> 
>> May I know what all filesystems are supported by Solr. For eg ADLS,WASB,
>> S3 etc. Thanks.
>> 
>> Ritesh
>> 

Re: Filesystems supported by Solr

[Ritesh Chaman <ch...@gmail.com>]

Hi team

I am trying to deploy solr on my ADLS subscription. can you tell me if that
is tested and is compatible.

Regards

On Tue, Feb 20, 2018 at 2:22 PM, Ritesh Chaman <ch...@gmail.com>
wrote:

> Hi team
>
> May I know what all filesystems are supported by Solr. For eg ADLS,WASB,
> S3 etc. Thanks.
>
> Ritesh
>

Re: Filesystems supported by Solr

[Rick Leir <rl...@leirtech.com>]

Ritesh
The filesystems you mention are used by Spark so it can stream huge quantities of data (corrections please).

By comparison, Solr uses a more 'reasonable' sized filesystem, but needs enough memory that all the index data can be resident. The regular Linux ext3 or ext4 is fine.

If you are integrating Solr with Spark, then the filesystems you mention would be for Spark not Solr. 
Cheers -- Rick


On February 20, 2018 5:22:33 PM EST, Ritesh Chaman <ch...@gmail.com> wrote:
>Hi team
>
>May I know what all filesystems are supported by Solr. For eg
>ADLS,WASB, S3
>etc. Thanks.
>
>Ritesh

-- 
Sorry for being brief. Alternate email is rickleir at yahoo dot com