[jira] [Commented] (IGNITE-3078) IgniteContext needs option to read config from HDFS

["Vladimir Ozerov (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/IGNITE-3078?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15270369#comment-15270369 ] 

Vladimir Ozerov commented on IGNITE-3078:
-----------------------------------------

Naden,

Please note that we cannot have this method on {{IgnitionEx}}. The reason is that this file is located in the "core" project, where we cannot have direct dependencies on Hadoop JARs. 
However, we already have a method {{IgnitionEx.start(InputStream)}}, which accepts a stream with Spring XML. Looks like this is exactly what we needed. Probably we can add utility method inside Spark module which will read HDFS and pass the stream to {{IgnitionEx}}.

> IgniteContext needs option to read config from HDFS
> ---------------------------------------------------
>
>                 Key: IGNITE-3078
>                 URL: https://issues.apache.org/jira/browse/IGNITE-3078
>             Project: Ignite
>          Issue Type: Bug
>          Components: Ignite RDD
>    Affects Versions: 1.5.0.final
>            Reporter: Naden Franciscus
>              Labels: bigdata
>
> If you are running Ignite in YARN mode then obviously we aren't going to have Ignite installation directories on each of the Hadoop cluster nodes. Therefore we need to store the configuration in HDFS for it to be accessible everywhere.
> What is needed is a new constructor in IgniteContext:
>     def this(
>         sc: SparkContext,
>         hdfsPath: org.apache.hadoop.fs.Path
>     ) {
>         this(sc, () ⇒ IgnitionEx.loadHDFSConfiguration(hdfsPath).get1())
>     }



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)