Use apache Ignite outside spark-shell

[luismattor <lu...@gmail.com>]

Hi,

I use spark-submit script to launch my spark application. There is a problem
with IgniteContext when starting ignite server nodes on each worker. It
seems IgniteContext is trying to launch more than one ignite node instance
per worker. Does Ignite works outiside spark-shell?



--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Use-apache-Ignite-outside-spark-shell-tp5208.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: Use apache Ignite outside spark-shell

[vkulichenko <va...@gmail.com>]

Hi Luis,

IgniteContext.ignite() method handles this situation and reuses the Ignite
client if it's already started. So if there are two partitions on the same
worker, there still will be only one Ignite started per process. Is it
working in a different way for you?

-Val



--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Use-apache-Ignite-outside-spark-shell-tp5208p5301.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: Use apache Ignite outside spark-shell

[luismattor <lu...@gmail.com>]

Hi,

The problem is in IgniteContext.scala:

// Start ignite server node on each worker in server mode.
sparkContext.parallelize(1 to workers, workers).foreachPartition(it ⇒
ignite())

As far as I understand, this line works as far as each executor holds a
single partition. That is, if executor owns more than one partition
IgniteContext will try to launch several Ignite nodes (in the same
process); if there are executors with no partitions, no Ignite instance
will be launched.

For example, Image a two worker cluster (node1, node2), each worker having
two cores. The idea is to launch an Ignite node on each executor. So we
call:
sparkContext.parallelize(1 to 2, 2).foreachPartition(it ⇒ ignite())

It might be the chance the two partitions are handled by node1. Or it might
be the case node1 handle one partition, and node2 the other. (this is what
we want). So the question is how to make sure the latter scenario happens?

Regards
Luis




On 26 May 2016 at 05:19, vkulichenko [via Apache Ignite Users] <
ml-node+s70518n5228h30@n6.nabble.com> wrote:

> Hi,
>
> Why do you think this is happening? Do you have any exception? Generally,
> IgniteContext always reuses the Ignite client if it already exists in the
> current process.
>
> Note that you can still have several clients in several processes. Usually
> you will have a client on the driver and a client per each executor (not
> worker!).
>
> -Val
>
> ------------------------------
> If you reply to this email, your message will be added to the discussion
> below:
>
> http://apache-ignite-users.70518.x6.nabble.com/Use-apache-Ignite-outside-spark-shell-tp5208p5228.html
> To unsubscribe from Use apache Ignite outside spark-shell, click here
> <http://apache-ignite-users.70518.x6.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=5208&code=bHVpc21hdHRvckBnbWFpbC5jb218NTIwOHwyNjY2MDQ0MTc=>
> .
> NAML
> <http://apache-ignite-users.70518.x6.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>




--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Use-apache-Ignite-outside-spark-shell-tp5208p5238.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: Use apache Ignite outside spark-shell

[vkulichenko <va...@gmail.com>]

Hi,

Why do you think this is happening? Do you have any exception? Generally,
IgniteContext always reuses the Ignite client if it already exists in the
current process.

Note that you can still have several clients in several processes. Usually
you will have a client on the driver and a client per each executor (not
worker!).

-Val



--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Use-apache-Ignite-outside-spark-shell-tp5208p5228.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.