You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@lucene.apache.org by "Mike Drob (Jira)" <ji...@apache.org> on 2021/01/07 16:50:00 UTC
[jira] [Commented] (SOLR-14251) Shard Split on HDFS
[ https://issues.apache.org/jira/browse/SOLR-14251?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17260643#comment-17260643 ]
Mike Drob commented on SOLR-14251:
----------------------------------
This is something that should be delegated to the Directory to decide whether the disk space should be checked, no? I suppose you've got worse problems if you run out of space to split on your shared FS.
> Shard Split on HDFS
> --------------------
>
> Key: SOLR-14251
> URL: https://issues.apache.org/jira/browse/SOLR-14251
> Project: Solr
> Issue Type: Bug
> Components: hdfs
> Affects Versions: 8.4
> Reporter: Johannes Brucher
> Priority: Major
> Time Spent: 20m
> Remaining Estimate: 0h
>
> Shard Split on HDFS Index will evaluate local disc space instead of HDFS space
> When performing a shard split on an index that is stored on HDFS the SplitShardCmd however evaluates the free disc space on the local file system of the server where Solr is installed.
> SplitShardCmd assumes that its main phase (when the Lucene index is being split) always executes on the local file system of the shard leader; and indeed the ShardSplitCmd.checkDiskSpace() checks the local file system's free disk space - even though the actual data is written to the HDFS Directory so it (almost) doesn't affect the local FS (except for core.properties file).
> See also: [https://lucene.472066.n3.nabble.com/HDFS-Shard-Split-td4449920.html]
> My setup to reproduce the issue:
> * Solr deployed on Openshift with local disc of about 5GB
> * HDFS configuration based on solrconfig.xml with
> {code:java}
> <directoryFactory name="DirectoryFactory" class="solr.HdfsDirectoryFactory">
> <str name="solr.hdfs.home">hdfs://path/to/index/</str>
> ...
> {code}
> * Split command:
> {code:java}
> .../admin/collections?action=SPLITSHARD&collection=collection1&shard=shard1&async=1234{code}
> * Response:
> {code:java}
> {
> "responseHeader":{"status":0,"QTime":32},
> "Operation splitshard caused exception:":"org.apache.solr.common.SolrException:org.apache.solr.common.SolrException: not enough free disk space to perform index split on node <solr instance>:8983_solr, required: 294.64909074269235, available: 5.4632568359375",
> "exception":{
> "msg":"not enough free disk space to perform index split on node <solr instance>:8983_solr, required: 294.64909074269235, available: 5.4632568359375",
> "rspCode":500},
> "status":{"state":"failed","msg":"found [1234] in failed tasks"}
> }
> {code}
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org