Time to Load a Solr Core with Hdfs Directory Factory

[Shashank Pedamallu <sp...@vmware.com>]

Hi,

I’m trying an experiment in which, I’m loading a core of 1.27GB with 5621600 documents on 2 Solr setups. On first setup, dataDir is pointed as a NRTCachingDirectory as a standard path in my local. On second setup, it is pointed as HdfsDirectory. As part of loading the core, I see following log:

2017-10-04 01:07:50.102 UTC INFO  (searcherExecutor-12-thread-1-processing-x:staging_1gb-core-1) [core='x:staging_1gb-core-1'] org.apache.solr.core.SolrCore@2247 [staging_1gb-core-1] Registered new searcher Searcher@10fe9415[staging_1gb-core-1] main{ExitableDirectoryReader(UninvertingDirectoryReader(Uninverting(_12bk(6.4.2):C2886542) Uninverting(_1eu5(6.4.2):C743800) Uninverting(_18kl(6.4.2):c331485) Uninverting(_1lt0(6.4.2):c284012) Uninverting(_1xx5(6.4.2):C654477) Uninverting(_1qsg(6.4.2):C658237) Uninverting(_1qf4(6.4.2):c24903) Uninverting(_1xwv(6.4.2):c16734) Uninverting(_1xwb(6.4.2):c17777) Uninverting(_1xww(6.4.2):C174) Uninverting(_1xy9(6.4.2):c878) Uninverting(_1xxf(6.4.2):c354) Uninverting(_1xxp(6.4.2):c508) Uninverting(_1xx6(6.4.2):C150) Uninverting(_1xxz(6.4.2):c545) Uninverting(_1xxg(6.4.2):C190) Uninverting(_1xyj(6.4.2):c690) Uninverting(_1xyd(6.4.2):C144)))}

This step takes about 132 milli-secods in setup 1 (i.e., with NRTCachingDirectoryFactory).
The same step takes about 21 minutes on second setup (i.e., with HdfsDirectoryFactory).

Does the load time of a Solr Core drops so badly on a HdfsFileSystem? Is this expected?

Thanks,
Shashank

Re: Time to Load a Solr Core with Hdfs Directory Factory

[Rick Leir <rl...@leirtech.com>]

Shashank,

I had a quick look at:

https://lucene.apache.org/solr/guide/6_6/running-solr-on-hdfs.html

Did you enable the Block Cache and the solr.hdfs.nrtcachingdirectory?
cheers -- Rick

On 2017-10-03 09:22 PM, Shashank Pedamallu wrote:
> Hi,
>
> I’m trying an experiment in which, I’m loading a core of 1.27GB with 5621600 documents on 2 Solr setups. On first setup, dataDir is pointed as a NRTCachingDirectory as a standard path in my local. On second setup, it is pointed as HdfsDirectory. As part of loading the core, I see following log:
>
> 2017-10-04 01:07:50.102 UTC INFO  (searcherExecutor-12-thread-1-processing-x:staging_1gb-core-1) [core='x:staging_1gb-core-1'] org.apache.solr.core.SolrCore@2247 [staging_1gb-core-1] Registered new searcher Searcher@10fe9415[staging_1gb-core-1] main{ExitableDirectoryReader(UninvertingDirectoryReader(Uninverting(_12bk(6.4.2):C2886542) Uninverting(_1eu5(6.4.2):C743800) Uninverting(_18kl(6.4.2):c331485) Uninverting(_1lt0(6.4.2):c284012) Uninverting(_1xx5(6.4.2):C654477) Uninverting(_1qsg(6.4.2):C658237) Uninverting(_1qf4(6.4.2):c24903) Uninverting(_1xwv(6.4.2):c16734) Uninverting(_1xwb(6.4.2):c17777) Uninverting(_1xww(6.4.2):C174) Uninverting(_1xy9(6.4.2):c878) Uninverting(_1xxf(6.4.2):c354) Uninverting(_1xxp(6.4.2):c508) Uninverting(_1xx6(6.4.2):C150) Uninverting(_1xxz(6.4.2):c545) Uninverting(_1xxg(6.4.2):C190) Uninverting(_1xyj(6.4.2):c690) Uninverting(_1xyd(6.4.2):C144)))}
>
> This step takes about 132 milli-secods in setup 1 (i.e., with NRTCachingDirectoryFactory).
> The same step takes about 21 minutes on second setup (i.e., with HdfsDirectoryFactory).
>
> Does the load time of a Solr Core drops so badly on a HdfsFileSystem? Is this expected?
>
> Thanks,
> Shashank


.