not working in Spark submit yarn mode.

[Edward Chen <ja...@gmail.com>]

Hi, I am trying to load Hdfs data into cache from Spark. it is working 
in local mode, but failed in spark-submit yarn mode. it tried to find 
Ignite Home path in the cluster. yes, it is true that Ignite is not 
installed in cluster. but why needed ? Ignite instance is created inside 
my java code and all of ignite jars are packaged. why need to find 
ignite info outside of my jar.

My ignite version is 2.3.

Anyone has a working example of Spark submit + Yarn mode ?  any web link 
is welcome.

Thanks. Edward

Re: not working in Spark submit yarn mode.

[Denis Magda <dm...@apache.org>]

It's possible to do that but not sure the community has any
ready-to-be-used code sample to share.

As for already available solutions, GridGain provides so-called Hadoop
Connector that solves data loading task:
https://docs.gridgain.com/docs/bdb-getting-started#section-gridgain-hadoop-connector

-
Denis


On Tue, Jun 4, 2019 at 1:49 PM Edward Chen <ja...@gmail.com> wrote:

> Hi, I am trying to load Hdfs data into cache from Spark. it is working
> in local mode, but failed in spark-submit yarn mode. it tried to find
> Ignite Home path in the cluster. yes, it is true that Ignite is not
> installed in cluster. but why needed ? Ignite instance is created inside
> my java code and all of ignite jars are packaged. why need to find
> ignite info outside of my jar.
>
> My ignite version is 2.3.
>
> Anyone has a working example of Spark submit + Yarn mode ?  any web link
> is welcome.
>
> Thanks. Edward
>
>