You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@camel.apache.org by Geoffry Roberts <ge...@gmail.com> on 2011/08/04 20:01:04 UTC
Camel and Hadoop First Try
All,
This is my first attempt at using camel so I doing something simple. I want
to copy a file from hadoop's hdfs to the local file system. I'm getting the
following error even though I believe I'm following the examples correctly.
java.lang.IllegalArgumentException: defaultEndpoint must be specified
What am I doing wrong?
Thanks
My code:
public class Copy2Local {
// irrelevant code
String in =
"hdfs://qq000:54310/hdfs/sourcedata/consolidate/56/56crmax/epistate/part-r-00000";
String out = "file://biomobility/camel/epistate.tsv";
public void run() {
CamelContext ctx = new DefaultCamelContext();
EpiRouteBuilder erb = new EpiRouteBuilder();
ctx.start();
ProducerTemplate prod = ctx.createProducerTemplate();
prod.sendBody(i);
}
class EpiRouteBuilder extends RouteBuilder {
public void configure() {
from(in).to(out);
}
}
// more irrelevant code
}
--
Geoffry Roberts
Re: Camel and Hadoop First Try
Posted by Claus Ibsen <cl...@gmail.com>.
Hi
Sometimes it can also be a good idea to peek into the unit test of a
given component you use and want to understand better.
For example to check out some of its unit tests
https://svn.apache.org/repos/asf/camel/trunk/components/camel-hdfs/
And for testing with Camel we do have documentation at
http://camel.apache.org/testing
And you may also want to read the javadoc of the ProducerTemplate that
you use for sending message.
And read a bit about it here:
http://camel.apache.org/producertemplate.html
2011/8/5 Magnus Palmér <ma...@gmail.com>:
> You are sending the URI for as the payload message in your code.
> So Christian is right, you should change sendBody(in, payload)
> By doing this you send the payload to your HDFS file system.
> Then your Camel route is polling that and will pick up the file.
>
> If you already have files on that HDFS path you can just start Camel and the
> files will be polled from the URI you set up in the route.
>
> Also in your to() URI you seem to specify a filename, but if you check the
> file component the path is the directory and then you add the file name
> parameter: fileName=epistate.tsv
> "file://biomobility/camel?fileName=epistate.tsv"
>
> Also as Christian wrote, you need to add your RouteBuilder to the Camel
> context.
>
> Check out the getting started example perhaps?
> http://camel.apache.org/getting-started.html
>
> (Or even better, the excellent "Camel in Action"-book.)
>
> /Magnus Palmér
>
>
> 2011/8/5 Geoffry Roberts <ge...@gmail.com>
>
>> Christian,
>>
>> My mistake, the "i" should be "in".
>>
>> String in = "hdfs://qq000:5...";
>>
>> prod.sendBody(in);
>>
>> It doesn't work.
>>
>> On 4 August 2011 15:32, Christian Müller <christian.mueller@gmail.com
>> >wrote:
>>
>> > Hello Geoffry!
>> >
>> > What is "i" in your code? I assume it's your payload which you send to
>> your
>> > Hadoop file system...
>> >
>> > And in your code I miss something like:
>> > ctx.addRoutes(erb);
>> >
>> > Instead of:
>> > prod.sendBody(i);
>> >
>> > you should try:
>> > prod.sendBody(in, i);
>> >
>> > Otherwise Camel doesn't know where to send the message...
>> >
>> > Best,
>> > Christian
>> >
>>
>>
>>
>> --
>> Geoffry Roberts
>>
>
--
Claus Ibsen
-----------------
FuseSource
Email: cibsen@fusesource.com
Web: http://fusesource.com
Twitter: davsclaus, fusenews
Blog: http://davsclaus.blogspot.com/
Author of Camel in Action: http://www.manning.com/ibsen/
Re: Camel and Hadoop First Try
Posted by Magnus Palmér <ma...@gmail.com>.
You are sending the URI for as the payload message in your code.
So Christian is right, you should change sendBody(in, payload)
By doing this you send the payload to your HDFS file system.
Then your Camel route is polling that and will pick up the file.
If you already have files on that HDFS path you can just start Camel and the
files will be polled from the URI you set up in the route.
Also in your to() URI you seem to specify a filename, but if you check the
file component the path is the directory and then you add the file name
parameter: fileName=epistate.tsv
"file://biomobility/camel?fileName=epistate.tsv"
Also as Christian wrote, you need to add your RouteBuilder to the Camel
context.
Check out the getting started example perhaps?
http://camel.apache.org/getting-started.html
(Or even better, the excellent "Camel in Action"-book.)
/Magnus Palmér
2011/8/5 Geoffry Roberts <ge...@gmail.com>
> Christian,
>
> My mistake, the "i" should be "in".
>
> String in = "hdfs://qq000:5...";
>
> prod.sendBody(in);
>
> It doesn't work.
>
> On 4 August 2011 15:32, Christian Müller <christian.mueller@gmail.com
> >wrote:
>
> > Hello Geoffry!
> >
> > What is "i" in your code? I assume it's your payload which you send to
> your
> > Hadoop file system...
> >
> > And in your code I miss something like:
> > ctx.addRoutes(erb);
> >
> > Instead of:
> > prod.sendBody(i);
> >
> > you should try:
> > prod.sendBody(in, i);
> >
> > Otherwise Camel doesn't know where to send the message...
> >
> > Best,
> > Christian
> >
>
>
>
> --
> Geoffry Roberts
>
Re: Camel and Hadoop First Try
Posted by Geoffry Roberts <ge...@gmail.com>.
Christian,
My mistake, the "i" should be "in".
String in = "hdfs://qq000:5...";
prod.sendBody(in);
It doesn't work.
On 4 August 2011 15:32, Christian Müller <ch...@gmail.com>wrote:
> Hello Geoffry!
>
> What is "i" in your code? I assume it's your payload which you send to your
> Hadoop file system...
>
> And in your code I miss something like:
> ctx.addRoutes(erb);
>
> Instead of:
> prod.sendBody(i);
>
> you should try:
> prod.sendBody(in, i);
>
> Otherwise Camel doesn't know where to send the message...
>
> Best,
> Christian
>
--
Geoffry Roberts
Re: Camel and Hadoop First Try
Posted by Christian Müller <ch...@gmail.com>.
Hello Geoffry!
What is "i" in your code? I assume it's your payload which you send to your
Hadoop file system...
And in your code I miss something like:
ctx.addRoutes(erb);
Instead of:
prod.sendBody(i);
you should try:
prod.sendBody(in, i);
Otherwise Camel doesn't know where to send the message...
Best,
Christian
Re: Camel and Hadoop First Try
Posted by Sander Mak <sa...@gmail.com>.
I know how to add the route Claus, thanks :) I was just pointing out
that the code from the OP seemed to omit this step...
On Fri, Aug 5, 2011 at 10:43 AM, Claus Ibsen <cl...@gmail.com> wrote:
> On Fri, Aug 5, 2011 at 10:21 AM, Sander Mak <sa...@gmail.com> wrote:
>> May be I'm missing something, but where do you add your routebuilder
>> to the context?
>
> CamelContext ctx ..
> RoutesBuilder builder = ...
>
> ctx.addRoutes(builder);
> ctx.start();
>
> See this getting starting example
> http://camel.apache.org/walk-through-an-example.html
>
>>
>> On Thu, Aug 4, 2011 at 8:01 PM, Geoffry Roberts
>> <ge...@gmail.com> wrote:
>>> All,
>>>
>>> This is my first attempt at using camel so I doing something simple. I want
>>> to copy a file from hadoop's hdfs to the local file system. I'm getting the
>>> following error even though I believe I'm following the examples correctly.
>>>
>>> java.lang.IllegalArgumentException: defaultEndpoint must be specified
>>>
>>> What am I doing wrong?
>>>
>>> Thanks
>>>
>>> My code:
>>>
>>> public class Copy2Local {
>>>
>>> // irrelevant code
>>>
>>> String in =
>>> "hdfs://qq000:54310/hdfs/sourcedata/consolidate/56/56crmax/epistate/part-r-00000";
>>> String out = "file://biomobility/camel/epistate.tsv";
>>>
>>> public void run() {
>>> CamelContext ctx = new DefaultCamelContext();
>>> EpiRouteBuilder erb = new EpiRouteBuilder();
>>> ctx.start();
>>> ProducerTemplate prod = ctx.createProducerTemplate();
>>> prod.sendBody(i);
>>> }
>>> class EpiRouteBuilder extends RouteBuilder {
>>>
>>> public void configure() {
>>> from(in).to(out);
>>> }
>>> }
>>> // more irrelevant code
>>> }
>>>
>>> --
>>> Geoffry Roberts
>>>
>>
>
>
>
> --
> Claus Ibsen
> -----------------
> FuseSource
> Email: cibsen@fusesource.com
> Web: http://fusesource.com
> Twitter: davsclaus, fusenews
> Blog: http://davsclaus.blogspot.com/
> Author of Camel in Action: http://www.manning.com/ibsen/
>
Re: Camel and Hadoop First Try
Posted by Claus Ibsen <cl...@gmail.com>.
On Fri, Aug 5, 2011 at 10:21 AM, Sander Mak <sa...@gmail.com> wrote:
> May be I'm missing something, but where do you add your routebuilder
> to the context?
CamelContext ctx ..
RoutesBuilder builder = ...
ctx.addRoutes(builder);
ctx.start();
See this getting starting example
http://camel.apache.org/walk-through-an-example.html
>
> On Thu, Aug 4, 2011 at 8:01 PM, Geoffry Roberts
> <ge...@gmail.com> wrote:
>> All,
>>
>> This is my first attempt at using camel so I doing something simple. I want
>> to copy a file from hadoop's hdfs to the local file system. I'm getting the
>> following error even though I believe I'm following the examples correctly.
>>
>> java.lang.IllegalArgumentException: defaultEndpoint must be specified
>>
>> What am I doing wrong?
>>
>> Thanks
>>
>> My code:
>>
>> public class Copy2Local {
>>
>> // irrelevant code
>>
>> String in =
>> "hdfs://qq000:54310/hdfs/sourcedata/consolidate/56/56crmax/epistate/part-r-00000";
>> String out = "file://biomobility/camel/epistate.tsv";
>>
>> public void run() {
>> CamelContext ctx = new DefaultCamelContext();
>> EpiRouteBuilder erb = new EpiRouteBuilder();
>> ctx.start();
>> ProducerTemplate prod = ctx.createProducerTemplate();
>> prod.sendBody(i);
>> }
>> class EpiRouteBuilder extends RouteBuilder {
>>
>> public void configure() {
>> from(in).to(out);
>> }
>> }
>> // more irrelevant code
>> }
>>
>> --
>> Geoffry Roberts
>>
>
--
Claus Ibsen
-----------------
FuseSource
Email: cibsen@fusesource.com
Web: http://fusesource.com
Twitter: davsclaus, fusenews
Blog: http://davsclaus.blogspot.com/
Author of Camel in Action: http://www.manning.com/ibsen/
Re: Camel and Hadoop First Try
Posted by Sander Mak <sa...@gmail.com>.
May be I'm missing something, but where do you add your routebuilder
to the context?
On Thu, Aug 4, 2011 at 8:01 PM, Geoffry Roberts
<ge...@gmail.com> wrote:
> All,
>
> This is my first attempt at using camel so I doing something simple. I want
> to copy a file from hadoop's hdfs to the local file system. I'm getting the
> following error even though I believe I'm following the examples correctly.
>
> java.lang.IllegalArgumentException: defaultEndpoint must be specified
>
> What am I doing wrong?
>
> Thanks
>
> My code:
>
> public class Copy2Local {
>
> // irrelevant code
>
> String in =
> "hdfs://qq000:54310/hdfs/sourcedata/consolidate/56/56crmax/epistate/part-r-00000";
> String out = "file://biomobility/camel/epistate.tsv";
>
> public void run() {
> CamelContext ctx = new DefaultCamelContext();
> EpiRouteBuilder erb = new EpiRouteBuilder();
> ctx.start();
> ProducerTemplate prod = ctx.createProducerTemplate();
> prod.sendBody(i);
> }
> class EpiRouteBuilder extends RouteBuilder {
>
> public void configure() {
> from(in).to(out);
> }
> }
> // more irrelevant code
> }
>
> --
> Geoffry Roberts
>