You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flume.apache.org by Nathaniel Auvil <na...@gmail.com> on 2012/11/05 16:31:26 UTC

HTTP Source

i am working with Flume 1.2.0 and creating my own HTTP source by embedding
Tomcat. I am having several issues.  Has anyone ever gotten this to work?
 It looks like my latest issue is a servlet-api conflict due to an ancient
Jetty 6 version on the Flume classpath

Re: HTTP Source

Posted by Hari Shreedharan <hs...@cloudera.com>.
No specific reason. I was familiar with Jetty and we already had a dependence on Jetty for the metrics stuff and I think some avro IPC stuff too, so I decided to just use the same. 

Thanks,
Hari

-- 
Hari Shreedharan


On Monday, November 5, 2012 at 2:09 PM, Harish Mandala wrote:

> Just curious, why doesn't the HTTP Source use Netty? 
> 
> On Mon, Nov 5, 2012 at 1:09 PM, Nathaniel Auvil <nathaniel.auvil@gmail.com (mailto:nathaniel.auvil@gmail.com)> wrote:
> > One thing i do not see with this HTTPSource is any way to customize the response.
> > 
> > 
> > On Mon, Nov 5, 2012 at 10:58 AM, Brock Noland <brock@cloudera.com (mailto:brock@cloudera.com)> wrote:
> > > Very soon. I am going to wait for the latest rev of this patch
> > > (https://reviews.apache.org/r/6377/) to be reviewed in hopes it can be
> > > committed and either way I will build the RC.
> > > 
> > > Brock
> > > 
> > > On Mon, Nov 5, 2012 at 9:47 AM, Nathaniel Auvil
> > > <nathaniel.auvil@gmail.com (mailto:nathaniel.auvil@gmail.com)> wrote:
> > > > is there a target date for 1.3?
> > > >
> > > >
> > > > On Mon, Nov 5, 2012 at 10:34 AM, Brock Noland <brock@cloudera.com (mailto:brock@cloudera.com)> wrote:
> > > >>
> > > >> There is actually an HTTP Source committed to the 1.3 branch:
> > > >>
> > > >>
> > > >> https://git-wip-us.apache.org/repos/asf?p=flume.git;a=commit;h=0bb1b21e3fa80cbb0f15c6397214d2a040fd1a5c
> > > >>
> > > >> Otherwise I would shade or jarjar your HTTP Source jars so they don't
> > > >> conflict.
> > > >>
> > > >> Brock
> > > >>
> > > >> On Mon, Nov 5, 2012 at 9:31 AM, Nathaniel Auvil
> > > >> <nathaniel.auvil@gmail.com (mailto:nathaniel.auvil@gmail.com)> wrote:
> > > >> > i am working with Flume 1.2.0 and creating my own HTTP source by
> > > >> > embedding
> > > >> > Tomcat. I am having several issues.  Has anyone ever gotten this to
> > > >> > work?
> > > >> > It looks like my latest issue is a servlet-api conflict due to an
> > > >> > ancient
> > > >> > Jetty 6 version on the Flume classpath
> > > >>
> > > >>
> > > >>
> > > >> --
> > > >> Apache MRUnit - Unit testing MapReduce -
> > > >> http://incubator.apache.org/mrunit/
> > > >
> > > >
> > > 
> > > 
> > > 
> > > --
> > > Apache MRUnit - Unit testing MapReduce - http://incubator.apache.org/mrunit/
> > 
>

Re: HTTP Source

Posted by Harish Mandala <mv...@gmail.com>.
Just curious, why doesn't the HTTP Source use Netty?

On Mon, Nov 5, 2012 at 1:09 PM, Nathaniel Auvil
<na...@gmail.com>wrote:

> One thing i do not see with this HTTPSource is any way to customize the
> response.
>
>
> On Mon, Nov 5, 2012 at 10:58 AM, Brock Noland <br...@cloudera.com> wrote:
>
>> Very soon. I am going to wait for the latest rev of this patch
>> (https://reviews.apache.org/r/6377/) to be reviewed in hopes it can be
>> committed and either way I will build the RC.
>>
>> Brock
>>
>> On Mon, Nov 5, 2012 at 9:47 AM, Nathaniel Auvil
>> <na...@gmail.com> wrote:
>> > is there a target date for 1.3?
>> >
>> >
>> > On Mon, Nov 5, 2012 at 10:34 AM, Brock Noland <br...@cloudera.com>
>> wrote:
>> >>
>> >> There is actually an HTTP Source committed to the 1.3 branch:
>> >>
>> >>
>> >>
>> https://git-wip-us.apache.org/repos/asf?p=flume.git;a=commit;h=0bb1b21e3fa80cbb0f15c6397214d2a040fd1a5c
>> >>
>> >> Otherwise I would shade or jarjar your HTTP Source jars so they don't
>> >> conflict.
>> >>
>> >> Brock
>> >>
>> >> On Mon, Nov 5, 2012 at 9:31 AM, Nathaniel Auvil
>> >> <na...@gmail.com> wrote:
>> >> > i am working with Flume 1.2.0 and creating my own HTTP source by
>> >> > embedding
>> >> > Tomcat. I am having several issues.  Has anyone ever gotten this to
>> >> > work?
>> >> > It looks like my latest issue is a servlet-api conflict due to an
>> >> > ancient
>> >> > Jetty 6 version on the Flume classpath
>> >>
>> >>
>> >>
>> >> --
>> >> Apache MRUnit - Unit testing MapReduce -
>> >> http://incubator.apache.org/mrunit/
>> >
>> >
>>
>>
>>
>> --
>> Apache MRUnit - Unit testing MapReduce -
>> http://incubator.apache.org/mrunit/
>>
>
>

Re: HTTP Source

Posted by Hari Shreedharan <hs...@cloudera.com>.
Hi Nathaniel, 

When I wrote this (and while it was being reviewed), we concluded it was not a good idea to expose the response object to the custom handler which can be plugged in because the handler can flush the buffer of the response object (which also can be flushed just because it is full) and then we could hit some exception (like channel being full) causing inconsistencies in the response. This source is not meant to be used for HTTP based RPC etc, this is to be used to send the data to the source and then wait for success or failure - on success send next event, on failure - back off for a while and retry (or similar logic).

For RPC etc, it might be best to write a source specific to your use-case. In case of this source bundled with Flume, we had to make it general enough to be useful but cautious enough to not make it confuse applications sending the data into Flume.

Hope this helps.

Thanks,
Hari

-- 
Hari Shreedharan


On Monday, November 5, 2012 at 10:09 AM, Nathaniel Auvil wrote:

> One thing i do not see with this HTTPSource is any way to customize the response.
> 
> 
> On Mon, Nov 5, 2012 at 10:58 AM, Brock Noland <brock@cloudera.com (mailto:brock@cloudera.com)> wrote:
> > Very soon. I am going to wait for the latest rev of this patch
> > (https://reviews.apache.org/r/6377/) to be reviewed in hopes it can be
> > committed and either way I will build the RC.
> > 
> > Brock
> > 
> > On Mon, Nov 5, 2012 at 9:47 AM, Nathaniel Auvil
> > <nathaniel.auvil@gmail.com (mailto:nathaniel.auvil@gmail.com)> wrote:
> > > is there a target date for 1.3?
> > >
> > >
> > > On Mon, Nov 5, 2012 at 10:34 AM, Brock Noland <brock@cloudera.com (mailto:brock@cloudera.com)> wrote:
> > >>
> > >> There is actually an HTTP Source committed to the 1.3 branch:
> > >>
> > >>
> > >> https://git-wip-us.apache.org/repos/asf?p=flume.git;a=commit;h=0bb1b21e3fa80cbb0f15c6397214d2a040fd1a5c
> > >>
> > >> Otherwise I would shade or jarjar your HTTP Source jars so they don't
> > >> conflict.
> > >>
> > >> Brock
> > >>
> > >> On Mon, Nov 5, 2012 at 9:31 AM, Nathaniel Auvil
> > >> <nathaniel.auvil@gmail.com (mailto:nathaniel.auvil@gmail.com)> wrote:
> > >> > i am working with Flume 1.2.0 and creating my own HTTP source by
> > >> > embedding
> > >> > Tomcat. I am having several issues.  Has anyone ever gotten this to
> > >> > work?
> > >> > It looks like my latest issue is a servlet-api conflict due to an
> > >> > ancient
> > >> > Jetty 6 version on the Flume classpath
> > >>
> > >>
> > >>
> > >> --
> > >> Apache MRUnit - Unit testing MapReduce -
> > >> http://incubator.apache.org/mrunit/
> > >
> > >
> > 
> > 
> > 
> > --
> > Apache MRUnit - Unit testing MapReduce - http://incubator.apache.org/mrunit/
>

Re: HTTP Source

Posted by Nathaniel Auvil <na...@gmail.com>.
One thing i do not see with this HTTPSource is any way to customize the
response.


On Mon, Nov 5, 2012 at 10:58 AM, Brock Noland <br...@cloudera.com> wrote:

> Very soon. I am going to wait for the latest rev of this patch
> (https://reviews.apache.org/r/6377/) to be reviewed in hopes it can be
> committed and either way I will build the RC.
>
> Brock
>
> On Mon, Nov 5, 2012 at 9:47 AM, Nathaniel Auvil
> <na...@gmail.com> wrote:
> > is there a target date for 1.3?
> >
> >
> > On Mon, Nov 5, 2012 at 10:34 AM, Brock Noland <br...@cloudera.com>
> wrote:
> >>
> >> There is actually an HTTP Source committed to the 1.3 branch:
> >>
> >>
> >>
> https://git-wip-us.apache.org/repos/asf?p=flume.git;a=commit;h=0bb1b21e3fa80cbb0f15c6397214d2a040fd1a5c
> >>
> >> Otherwise I would shade or jarjar your HTTP Source jars so they don't
> >> conflict.
> >>
> >> Brock
> >>
> >> On Mon, Nov 5, 2012 at 9:31 AM, Nathaniel Auvil
> >> <na...@gmail.com> wrote:
> >> > i am working with Flume 1.2.0 and creating my own HTTP source by
> >> > embedding
> >> > Tomcat. I am having several issues.  Has anyone ever gotten this to
> >> > work?
> >> > It looks like my latest issue is a servlet-api conflict due to an
> >> > ancient
> >> > Jetty 6 version on the Flume classpath
> >>
> >>
> >>
> >> --
> >> Apache MRUnit - Unit testing MapReduce -
> >> http://incubator.apache.org/mrunit/
> >
> >
>
>
>
> --
> Apache MRUnit - Unit testing MapReduce -
> http://incubator.apache.org/mrunit/
>

Re: HTTP Source

Posted by Brock Noland <br...@cloudera.com>.
Very soon. I am going to wait for the latest rev of this patch
(https://reviews.apache.org/r/6377/) to be reviewed in hopes it can be
committed and either way I will build the RC.

Brock

On Mon, Nov 5, 2012 at 9:47 AM, Nathaniel Auvil
<na...@gmail.com> wrote:
> is there a target date for 1.3?
>
>
> On Mon, Nov 5, 2012 at 10:34 AM, Brock Noland <br...@cloudera.com> wrote:
>>
>> There is actually an HTTP Source committed to the 1.3 branch:
>>
>>
>> https://git-wip-us.apache.org/repos/asf?p=flume.git;a=commit;h=0bb1b21e3fa80cbb0f15c6397214d2a040fd1a5c
>>
>> Otherwise I would shade or jarjar your HTTP Source jars so they don't
>> conflict.
>>
>> Brock
>>
>> On Mon, Nov 5, 2012 at 9:31 AM, Nathaniel Auvil
>> <na...@gmail.com> wrote:
>> > i am working with Flume 1.2.0 and creating my own HTTP source by
>> > embedding
>> > Tomcat. I am having several issues.  Has anyone ever gotten this to
>> > work?
>> > It looks like my latest issue is a servlet-api conflict due to an
>> > ancient
>> > Jetty 6 version on the Flume classpath
>>
>>
>>
>> --
>> Apache MRUnit - Unit testing MapReduce -
>> http://incubator.apache.org/mrunit/
>
>



-- 
Apache MRUnit - Unit testing MapReduce - http://incubator.apache.org/mrunit/

Re: HTTP Source

Posted by Nathaniel Auvil <na...@gmail.com>.
is there a target date for 1.3?


On Mon, Nov 5, 2012 at 10:34 AM, Brock Noland <br...@cloudera.com> wrote:

> There is actually an HTTP Source committed to the 1.3 branch:
>
>
> https://git-wip-us.apache.org/repos/asf?p=flume.git;a=commit;h=0bb1b21e3fa80cbb0f15c6397214d2a040fd1a5c
>
> Otherwise I would shade or jarjar your HTTP Source jars so they don't
> conflict.
>
> Brock
>
> On Mon, Nov 5, 2012 at 9:31 AM, Nathaniel Auvil
> <na...@gmail.com> wrote:
> > i am working with Flume 1.2.0 and creating my own HTTP source by
> embedding
> > Tomcat. I am having several issues.  Has anyone ever gotten this to work?
> > It looks like my latest issue is a servlet-api conflict due to an ancient
> > Jetty 6 version on the Flume classpath
>
>
>
> --
> Apache MRUnit - Unit testing MapReduce -
> http://incubator.apache.org/mrunit/
>

Re: HTTP Source

Posted by Brock Noland <br...@cloudera.com>.
There is actually an HTTP Source committed to the 1.3 branch:

https://git-wip-us.apache.org/repos/asf?p=flume.git;a=commit;h=0bb1b21e3fa80cbb0f15c6397214d2a040fd1a5c

Otherwise I would shade or jarjar your HTTP Source jars so they don't conflict.

Brock

On Mon, Nov 5, 2012 at 9:31 AM, Nathaniel Auvil
<na...@gmail.com> wrote:
> i am working with Flume 1.2.0 and creating my own HTTP source by embedding
> Tomcat. I am having several issues.  Has anyone ever gotten this to work?
> It looks like my latest issue is a servlet-api conflict due to an ancient
> Jetty 6 version on the Flume classpath



-- 
Apache MRUnit - Unit testing MapReduce - http://incubator.apache.org/mrunit/