You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@ignite.apache.org by mamaco <ma...@163.com> on 2018/10/12 19:44:31 UTC
'Deploy' ignite cluster into a spark cluster
When I implement a Spark streaming project, I involuntarily found Ignite
could be easily hacked into spark yarn cluster and work on the spark
datanodes as an additional ignite cluster(use spark cluster resources,
RAM,core without complex deployment), and all data process could be done
with forEachPartition on datanode, I'm not sure if it's a safe and good
practise to run a hybrid project.
This is a bit crazy, but it really makes the whole structure easy and
simple. But, except get complaint from DBA, is there any potential risk that
I'll have to worry about? do ignite have any solution for use case like
this?
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
RE: 'Deploy' ignite cluster into a spark cluster
Posted by mamaco <ma...@163.com>.
Hi Stan,
'executor die, that leads ignite cluster rebalance', I haven't expected this
before. Our solution is strange, it's a bit like borrowing spark job
scheduling API to deploy ignite and other platforms, so, how to persuade
them work together is the major consideration, below is my plan, will let
you know the actual PDT performance.
1. Spark ExecutorAllocationManager.scala
<https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/ExecutorAllocationManager.scala>
there're settings could help with the static Allocations like
spark.dynamicAllocation.enabled=false and
spark.shuffle.service.enabled=false.
2. As to the data allocation, the only anti-shuffle solution that I could
find out is to build a global partition
algorithm(for both of Spark RDD and Ignite Affinity collocation), let them
do the computing locally and forcedly.
3. In case of executor crash(rebalance), we could lower the ignite cache
size using TTL policy, and eventually ensure the realtime balance and
overall performance.
Marco
Stanislav Lukyanov wrote
> Well… I mean, if you really want to… :)
>
> The issue is that spark may start or stop its executors as it sees fit –
> or, at least,
> not taking Ignite’s lifecycle into account.
> If an executor dies, the Ignite node dies. It doesn’t always lead to data
> loss,
> but it does lead to topology changes and rebalance which are the most
> costly operations for Ignite.
> So each time spark will try to start or stop an executor to make your
> computations more
> efficient, it will actually lead to Ignite doing a lot of extra work.
>
> You can try using the embedded mode, but keep in mind possible performance
> drops
> when executors are requested or removed, possible data loss (complete or
> partial)
> and the fact that this mode is eventually going away. I’d advise to go for
> the standalone deployment.
>
> Stan
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
RE: 'Deploy' ignite cluster into a spark cluster
Posted by Stanislav Lukyanov <st...@gmail.com>.
Well… I mean, if you really want to… :)
The issue is that spark may start or stop its executors as it sees fit – or, at least,
not taking Ignite’s lifecycle into account.
If an executor dies, the Ignite node dies. It doesn’t always lead to data loss,
but it does lead to topology changes and rebalance which are the most costly operations for Ignite.
So each time spark will try to start or stop an executor to make your computations more
efficient, it will actually lead to Ignite doing a lot of extra work.
You can try using the embedded mode, but keep in mind possible performance drops
when executors are requested or removed, possible data loss (complete or partial)
and the fact that this mode is eventually going away. I’d advise to go for the standalone deployment.
Stan
From: mamaco
Sent: 15 октября 2018 г. 21:27
To: user@ignite.apache.org
Subject: RE: 'Deploy' ignite cluster into a spark cluster
Hi Stan,
Thank you for providing the information.
Yes, it's exact 'embedded deployment', but it looks like to be deprecated
because of performance issues.
Stanislav Lukyanov wrote
> Embedded Mode Deprecation
> Embedded mode implies starting Ignite server nodes within Spark executors
> which can cause unexpected rebalancing or even data loss. Therefore this
> mode is currently deprecated and will be eventually discontinued. Consider
> starting a separate Ignite cluster and using standalone mode to avoid data
> consistency and performance issues.
Sorry for my careless, I didn't notice that information before, it almost
answered my question.
But, how about if I use earlier ignite (1.9-), ensure On-Heap cache and let
Ignite handle the GC?
My case is straight forward streaming process (locality process comes with
hash partitioning, affinity collocation to ignite cache).
Is there any further potential problem? Your suggestion is appreciated.
My concerns:
This way makes deployment very easy, compare old SharedRDD solution, I don't
have to persuade DBA to initial ignite cluster one by one, and POJO classes,
that's also really a bottleneck to collect data from 'cluster' to 'driver'
and 'ignite', node to node local computing is preferable, because all I
need to do is to ship the library to spark cluster from the driver node.
If spark job stopped, Ignite cluster will be terminated and the data will
lost(I saw this actions from the spark executor logs). If that is the case,
the 'risk' is acceptable to me, I don't care about data loss issue. My
project is to receive concurrent kafka traffic data using spark streaming, I
need ignite to act a fast k/v dictionary to reduce deduplication computing
(before saving to Hbase), in the meantime ignite cache provide SQL
query/monitor service for curious users.
According to the information above, do you have any suggestion?
We have been confused about how to integrate bunch of open source projects
until I found apache spark has a big heart to supply everything.
Your help is appreciated.
Marco
Code:
#ship ignite 1.9 libraries to spark cluster
/spark-shell --master yarn --conf spark.ui.enabled=true --deploy-mode client
--num-executors 2 --jars
"/opt/ignite/assembly/1.9/Ignite-1.9-jar-with-dependencies.jar"
class LazyAgent(isClient: Boolean) extends Serializable {
lazy val conf=ic_conf()
lazy val ignite=org.apache.ignite.Ignition.start(conf)
lazy val cache=ignite.getOrCreateCache(gen_cache("test"))
def ic_conf(): org.apache.ignite.configuration.IgniteConfiguration={
val spi = new org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi()
spi.setLocalPortRange(10)
val ipfiner=new
org.apache.ignite.spi.discovery.tcp.ipfinder.jdbc.TcpDiscoveryJdbcIpFinder()
ipfiner.setDataSource(JDBCDataSource())
spi.setIpFinder(ipfiner)
val cfg = new org.apache.ignite.configuration.IgniteConfiguration()
cfg.setClientMode(isClient)
cfg.setDiscoverySpi(spi)
cfg
}
def JDBCDataSource(): javax.sql.DataSource={
val dataSource=new com.mysql.jdbc.jdbc2.optional.MysqlDataSource()
dataSource.setServerName({hostip})
......
dataSource
}
def gen_cache(cache_name:String):
org.apache.ignite.configuration.CacheConfiguration[String,String] ={
val cache_cfg = new
org.apache.ignite.configuration.CacheConfiguration[String,
String](cache_name)
......
cache_cfg
}
}
//on driver:
val instance= sc.broadcast(new LazyAgent(false))
//on datanode
rdd.foreachPartition(iter => {
#here on each data node, ignite cluster will be created and only once.
val cache = instance.value.cache
iter.foreach(el=>{
cache.put("k"+el.toString(),"value of ["+el.toString()+"]")
})
})
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
RE: 'Deploy' ignite cluster into a spark cluster
Posted by mamaco <ma...@163.com>.
Hi Stan,
Thank you for providing the information.
Yes, it's exact 'embedded deployment', but it looks like to be deprecated
because of performance issues.
Stanislav Lukyanov wrote
> Embedded Mode Deprecation
> Embedded mode implies starting Ignite server nodes within Spark executors
> which can cause unexpected rebalancing or even data loss. Therefore this
> mode is currently deprecated and will be eventually discontinued. Consider
> starting a separate Ignite cluster and using standalone mode to avoid data
> consistency and performance issues.
Sorry for my careless, I didn't notice that information before, it almost
answered my question.
But, how about if I use earlier ignite (1.9-), ensure On-Heap cache and let
Ignite handle the GC?
My case is straight forward streaming process (locality process comes with
hash partitioning, affinity collocation to ignite cache).
Is there any further potential problem? Your suggestion is appreciated.
My concerns:
This way makes deployment very easy, compare old SharedRDD solution, I don't
have to persuade DBA to initial ignite cluster one by one, and POJO classes,
that's also really a bottleneck to collect data from 'cluster' to 'driver'
and 'ignite', node to node local computing is preferable, because all I
need to do is to ship the library to spark cluster from the driver node.
If spark job stopped, Ignite cluster will be terminated and the data will
lost(I saw this actions from the spark executor logs). If that is the case,
the 'risk' is acceptable to me, I don't care about data loss issue. My
project is to receive concurrent kafka traffic data using spark streaming, I
need ignite to act a fast k/v dictionary to reduce deduplication computing
(before saving to Hbase), in the meantime ignite cache provide SQL
query/monitor service for curious users.
According to the information above, do you have any suggestion?
We have been confused about how to integrate bunch of open source projects
until I found apache spark has a big heart to supply everything.
Your help is appreciated.
Marco
Code:
#ship ignite 1.9 libraries to spark cluster
/spark-shell --master yarn --conf spark.ui.enabled=true --deploy-mode client
--num-executors 2 --jars
"/opt/ignite/assembly/1.9/Ignite-1.9-jar-with-dependencies.jar"
class LazyAgent(isClient: Boolean) extends Serializable {
lazy val conf=ic_conf()
lazy val ignite=org.apache.ignite.Ignition.start(conf)
lazy val cache=ignite.getOrCreateCache(gen_cache("test"))
def ic_conf(): org.apache.ignite.configuration.IgniteConfiguration={
val spi = new org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi()
spi.setLocalPortRange(10)
val ipfiner=new
org.apache.ignite.spi.discovery.tcp.ipfinder.jdbc.TcpDiscoveryJdbcIpFinder()
ipfiner.setDataSource(JDBCDataSource())
spi.setIpFinder(ipfiner)
val cfg = new org.apache.ignite.configuration.IgniteConfiguration()
cfg.setClientMode(isClient)
cfg.setDiscoverySpi(spi)
cfg
}
def JDBCDataSource(): javax.sql.DataSource={
val dataSource=new com.mysql.jdbc.jdbc2.optional.MysqlDataSource()
dataSource.setServerName({hostip})
......
dataSource
}
def gen_cache(cache_name:String):
org.apache.ignite.configuration.CacheConfiguration[String,String] ={
val cache_cfg = new
org.apache.ignite.configuration.CacheConfiguration[String,
String](cache_name)
......
cache_cfg
}
}
//on driver:
val instance= sc.broadcast(new LazyAgent(false))
//on datanode
rdd.foreachPartition(iter => {
#here on each data node, ignite cluster will be created and only once.
val cache = instance.value.cache
iter.foreach(el=>{
cache.put("k"+el.toString(),"value of ["+el.toString()+"]")
})
})
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
RE: 'Deploy' ignite cluster into a spark cluster
Posted by Stanislav Lukyanov <st...@gmail.com>.
Hi,
Sorry, I didn’t really get what you mean here…
Can you explain:
- What exactly you’re doing?
- How is it different from normal Ignite-Spark integration, e.g. from shared deployment (https://apacheignite-fs.readme.io/docs/installation-deployment)?
- What would be an alternative solution you’re comparing yours to?
Stan
From: mamaco
Sent: 12 октября 2018 г. 22:44
To: user@ignite.apache.org
Subject: 'Deploy' ignite cluster into a spark cluster
When I implement a Spark streaming project, I involuntarily found Ignite
could be easily hacked into spark yarn cluster and work on the spark
datanodes as an additional ignite cluster(use spark cluster resources,
RAM,core without complex deployment), and all data process could be done
with forEachPartition on datanode, I'm not sure if it's a safe and good
practise to run a hybrid project.
This is a bit crazy, but it really makes the whole structure easy and
simple. But, except get complaint from DBA, is there any potential risk that
I'll have to worry about? do ignite have any solution for use case like
this?
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/