[jira] [Commented] (TINKERPOP-1331) HADOOP_GREMLIN_LIBS can only point to local file system

["Daniel Kuppitz (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/TINKERPOP-1331?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15322122#comment-15322122 ] 

Daniel Kuppitz commented on TINKERPOP-1331:
-------------------------------------------

My proposal for  {{loadsJars}}:

{code}
private final static Pattern PATH_PATTERN = Pattern.compile("([^:]|://)+");
...
private void loadJars(final JavaSparkContext sparkContext, final Configuration hadoopConfiguration) {
    if (hadoopConfiguration.getBoolean(Constants.GREMLIN_HADOOP_JARS_IN_DISTRIBUTED_CACHE, true)) {
        final String hadoopGremlinLibs = null == System.getProperty(Constants.HADOOP_GREMLIN_LIBS) ? System.getenv(Constants.HADOOP_GREMLIN_LIBS) : System.getProperty(Constants.HADOOP_GREMLIN_LIBS);
        if (null == hadoopGremlinLibs)
            this.logger.warn(Constants.HADOOP_GREMLIN_LIBS + " is not set -- proceeding regardless");
        else {
            try {
                final Matcher matcher = PATH_PATTERN.matcher(hadoopGremlinLibs);
                while (matcher.find()) {
                    final String path = matcher.group();
                    FileSystem fs;
                    try {
                        final URI uri = new URI(path);
                        fs = FileSystem.get(uri, hadoopConfiguration);
                    } catch (URISyntaxException e) {
                        fs = FileSystem.get(hadoopConfiguration);
                    }
                    final File file = AbstractHadoopGraphComputer.copyDirectoryIfNonExistent(fs, path);
                    if (file.exists())
                        Stream.of(file.listFiles()).filter(f -> f.getName().endsWith(Constants.DOT_JAR)).forEach(f -> sparkContext.addJar(f.getAbsolutePath()));
                    else
                        this.logger.warn(path + " does not reference a valid directory -- proceeding regardless");
                }
            } catch (IOException e) {
                throw new IllegalStateException(e.getMessage(), e);
            }
        }
    }
}
{code}

> HADOOP_GREMLIN_LIBS can only point to local file system
> -------------------------------------------------------
>
>                 Key: TINKERPOP-1331
>                 URL: https://issues.apache.org/jira/browse/TINKERPOP-1331
>             Project: TinkerPop
>          Issue Type: Improvement
>          Components: hadoop
>    Affects Versions: 3.2.0-incubating, 3.1.2-incubating
>            Reporter: Daniel Kuppitz
>
> These two lines in {{SparkGraphComputer}} assume that {{HADOOP_GREMLIN_LIBS}} will only contain local file system references (although it seems that the rest of the code could handle DFS references):
> {code}
> final String[] paths = hadoopGremlinLocalLibs.split(":");
> final FileSystem fs = FileSystem.get(hadoopConfiguration);
> {code}
> If, for example, {{HADOOP_GREMLIN_LIBS}} would be set to {{hdfs:///spark-gremlin-libs:/foo/bar}}, the {{split(":")}} call would obviously separate the file system scheme ({{hdfs://}}) from the path ({{/spark-gremlin-libs}}).
> Next, {{FileSystem.get(hadoopConfiguration)}} will always only return a reference to the {{FileSystem}} that is defined as the default file system.
> The same is probably true for {{GiraphGraphComputer}}.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)