[jira] [Created] (SOLR-11707) allow to configure the HDFS block size

["Hendrik Haddorp (JIRA)" <ji...@apache.org>]

Hendrik Haddorp created SOLR-11707:
--------------------------------------

             Summary: allow to configure the HDFS block size
                 Key: SOLR-11707
                 URL: https://issues.apache.org/jira/browse/SOLR-11707
             Project: Solr
          Issue Type: Improvement
      Security Level: Public (Default Security Level. Issues are Public)
          Components: hdfs
            Reporter: Hendrik Haddorp
            Priority: Minor


Currently index files are created in HDFS with the block size that is defined on the namenode. For that the HdfsFileWriter reads out the config from the server and then specifies the size (and replication factor) in the FileSystem.create call.

For the write.lock files things work slightly different. These are being created by the HdfsLockFactory without specifying a block size (or replication factor). This results in a default being picked by the HDFS client, which is 128MB.

So currently files are being created with different block sizes if the namenode is configured to something else then 128MB. It would be good if Solr would allow to configure the block size to be used. This is especially useful if the Solr admin is not the HDFS admin and if you have different applications using HDFS that have different requirements for their block size.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org