What is the best way to use the Hadoop output data

[Huy Phan <da...@gmail.com>]

Hi everybody, I'm working on a hadoop project that processing the log files.
In the reduce part, as usual, I store the output to HDFS, but I also want
send those output data to the message queue using HTTP Post Request.
I'm wondering if there's any performance killer in this approach, I posted
the question to IRC channel and someone told me that there may be a
bottleneck.
Then I think about running a cron task to get the output data and send it to
MQ, but not sure it's the best way cause it's not synchronize with the
MapReduce process.
I wonder if there is any way to spawn a process directly from Hadoop after
all the MapReduce tasks finish ?

Re: What is the best way to use the Hadoop output data

[Chris Curtin <cu...@gmail.com>]

Hi,

have you looked at Cascading? It gives you events (
http://www.cascading.org/userguide/htmlsingle/#N20ADF) that can call back to
your code when one of your MR jobs completes. It is also embeddable inside a
daemon so you can launch a thread that runs the MR job, then writes to the
external end point. It handles all the Hadoop stuff for you so you don't
need to worry about tracking dependencies.

On Fri, Jun 26, 2009 at 4:10 AM, Zhong Wang <wa...@gmail.com> wrote:

> Hi Huy,
>
> On Thu, Jun 25, 2009 at 6:02 PM, Huy Phan<da...@gmail.com> wrote:
> > I'm wondering if there's any performance killer in this approach, I
> posted
> > the question to IRC channel and someone told me that there may be a
> > bottleneck.
>
> There may be some communication errors to block your MapReduce job
> when you post your output data. So I think it's better to do this
> after the job is done.
>
> > I wonder if there is any way to spawn a process directly from Hadoop
> after
> > all the MapReduce tasks finish ?
> >
>
> How do you submit your jobs? You can block the job submit process by
> running job using job.waitForCompletion(true) in your main driver
> class. Then the two processes are synchronous.
>
>
> --
> Zhong Wang
>

Re: What is the best way to use the Hadoop output data

[Zhong Wang <wa...@gmail.com>]

Hi Huy,

On Thu, Jun 25, 2009 at 6:02 PM, Huy Phan<da...@gmail.com> wrote:
> I'm wondering if there's any performance killer in this approach, I posted
> the question to IRC channel and someone told me that there may be a
> bottleneck.

There may be some communication errors to block your MapReduce job
when you post your output data. So I think it's better to do this
after the job is done.

> I wonder if there is any way to spawn a process directly from Hadoop after
> all the MapReduce tasks finish ?
>

How do you submit your jobs? You can block the job submit process by
running job using job.waitForCompletion(true) in your main driver
class. Then the two processes are synchronous.


-- 
Zhong Wang

Re: What is the best way to use the Hadoop output data

[Huy Phan <da...@gmail.com>]

Anybody help me on this ? :)

On Thu, Jun 25, 2009 at 5:02 PM, Huy Phan <da...@gmail.com> wrote:

> Hi everybody, I'm working on a hadoop project that processing the log
> files. In the reduce part, as usual, I store the output to HDFS, but I also
> want send those output data to the message queue using HTTP Post Request.
> I'm wondering if there's any performance killer in this approach, I posted
> the question to IRC channel and someone told me that there may be a
> bottleneck.
> Then I think about running a cron task to get the output data and send it
> to MQ, but not sure it's the best way cause it's not synchronize with the
> MapReduce process.
> I wonder if there is any way to spawn a process directly from Hadoop after
> all the MapReduce tasks finish ?
>
>