Distribute JMS receiver jobs on YARN

[ni...@free.fr]

Hello,
I have spark application with a JMS receiver.
Basically my application does :

....
JavaDStream<String> incoming_msg = customReceiverStream.map(
				new Function<JMSEvent, String>()
				{
					public String call(JMSEvent jmsEvent)
					{				
						return jmsEvent.getText();
					}
				}
				);
incoming_msg.foreachRDD( new Function<JavaRDD<String>,  Void>() {
public Void call(JavaRDD<String> rdd) throws Exception {
	rdd.foreachPartition(new VoidFunction<Iterator<String>>() {
		public void call(Iterator<String> msg) throws Exception {
			while (msg.hasNext()) {
                            // insert msg in MongoDB
                        }
.....

It works fine in standalone, but now I want to distribute it inside a YARN cluster of 4 nodes.

Please, could you explain me how will/should be done the distribution on the cluster.
I don't understand if each node will consume JMS Queue or if master node will consume JMS Queue and message set will be distributed over the cluster nodes.

Please help me it is not clear..

Tks
Nicolas
 


---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org