HDFS example in distributed mode

[Naresh <ko...@bah.com>]

I am trying to get HDFS example to work on my local machine.  I have hadoop
installed and when I use the HDFS API I can transfer files back and forth so
I know hadoop is set up.  I tried to use the HDFS unit tests as example but
only one actually is using HDFS in distributed mode while most of the tests
are written in hadoop local mode.  I got the local mode unit tests to work
but I need the distributed mode to work.  I cannot get the distributed mode
to work using camel.  I copied the unit test code and am trying to follow
the unit test as given.  The Part I am getting stuck on is
template.sendBody("direct:start", "CIAO" + i); . The template variable has
not been created in the test.  I searched all the forums but could not find
a working example with HDFS.  Here is my code:

import java.util.List;

import org.apache.camel.CamelContext;
import org.apache.camel.Exchange;
import org.apache.camel.component.hdfs.HdfsConfiguration;
import org.apache.camel.component.hdfs.HdfsEndpoint;
import org.apache.camel.component.hdfs.HdfsProducer;
import org.apache.camel.builder.RouteBuilder;
import org.apache.camel.impl.DefaultCamelContext;
import org.apache.camel.component.mock.MockEndpoint;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.Path;


public class HDFSReadUsingCamel {
	public static void main(String args[]) throws Exception {
		
		Configuration conf = new Configuration();
		conf.addResource(new
Path("/Users/nareshkosgi/booz/hadoop/conf/core-site.xml"));
		conf.addResource(new
Path("/Users/nareshkosgi/booz/hadoop/conf/hdfs-site.xml"));
		
		CamelContext context = new DefaultCamelContext();
		
		context.addRoutes(new RouteBuilder() {
			public void configure() {
			
from("direct:start").to("hdfs://localhost:9000/tmp/test/test-camel-simple-write-file?fileSystemType=HDFS&splitStrategy=BYTES:5,IDLE:1000");
			
from("hdfs://localhost:9000/test?pattern=seg*&initialDelay=2000&fileSystemType=HDFS&chunkSize=5").to("mock:result");
			}
		});

                context.start();
		
		for(int i=0; i<10; ++i) {
			template.sendBody("direct:start", "CIAO" + i) // this is the part that
has an error and I am not able to fix
		}
		
		 MockEndpoint resultEndpoint = context.getEndpoint("mock:result",
MockEndpoint.class);
		 List<Exchange> exchanges = resultEndpoint.getExchanges();
		 for(Exchange exchange : exchanges) {
			 System.out.println(exchange.getIn().getBody(String.class));
		 }
		
		Thread.sleep(1000);
		context.stop();
		
	}

}

Sorry if this is a noob question but I searched the forms and google but
could not find any information and I am using the forum as a last resort.

Thanks
Naresh

-----
Naresh
--
View this message in context: http://camel.465427.n5.nabble.com/HDFS-example-in-distributed-mode-tp5715103.html
Sent from the Camel - Users mailing list archive at Nabble.com.

Re: HDFS example in distributed mode

[Naresh <ko...@bah.com>]

One more thing:

That example is basically the unit test called
HdfsProducerConsumerIntegrationTest.java

http://svn.apache.org/repos/asf/camel/branches/camel-2.8.x/components/camel-hdfs/src/test/java/org/apache/camel/component/hdfs/integration/HdfsProducerConsumerIntegrationTest.java

-----
Naresh
--
View this message in context: http://camel.465427.n5.nabble.com/HDFS-example-in-distributed-mode-tp5715103p5715104.html
Sent from the Camel - Users mailing list archive at Nabble.com.