[jira] [Commented] (SOLR-11473) Make HDFSDirectoryFactory support other prefixes (besides hdfs:/)

["Kevin Risden (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SOLR-11473?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16792276#comment-16792276 ] 

Kevin Risden commented on SOLR-11473:
-------------------------------------

One comment here so I don't forget. We NEED to make sure we set "fs.SCHEME.impl.disable.cache" = true for whatever scheme is being used. Solr makes sure to close filesystems ahead of time. If we don't disable the cache for whatever scheme, you end up with closed filesystem errors.

> Make HDFSDirectoryFactory support other prefixes (besides hdfs:/)
> -----------------------------------------------------------------
>
>                 Key: SOLR-11473
>                 URL: https://issues.apache.org/jira/browse/SOLR-11473
>             Project: Solr
>          Issue Type: Improvement
>      Security Level: Public(Default Security Level. Issues are Public) 
>          Components: hdfs
>    Affects Versions: 6.6.1
>            Reporter: Radu Gheorghe
>            Priority: Minor
>         Attachments: SOLR-11473.patch
>
>
> Not sure if it's a bug or a missing feature :) I'm trying to make Solr work on Alluxio, as described by [~thelabdude] in https://www.slideshare.net/thelabdude/running-solr-in-the-cloud-at-memory-speed-with-alluxio/1
> The problem I'm facing here is with autoAddReplicas. If I have replicationFactor=1 and the node with that replica dies, the node taking over incorrectly assigns the data directory. For example:
> before
> {code}"dataDir":"alluxio://localhost:19998/solr/test/",{code}
> after
> {code}"dataDir":"alluxio://localhost:19998/solr/test/core_node1/alluxio://localhost:19998/solr/test/",{code}
> The same happens for ulogDir. Apparently, this has to do with this bit from HDFSDirectoryFactory:
> {code}  public boolean isAbsolute(String path) {
>     return path.startsWith("hdfs:/");
>   }{code}
> If I add "alluxio:/" in there, the paths are correct and the index is recovered.
> I see a few options here:
> * add "alluxio:/" to the list there
> * add a regular expression in the lines of \[a-z]*:/ I hope that's not too expensive, I'm not sure how often this method is called
> * don't do anything and expect alluxio to work with an "hdfs:/" path? I actually tried that and didn't manage to make it work
> * have a different DirectoryFactory or something else?
> What do you think?



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org