You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@flume.apache.org by "Jeremy Karlson (JIRA)" <ji...@apache.org> on 2014/02/27 03:55:23 UTC
[jira] [Updated] (FLUME-2333) HTTP source handler doesn't allow for
responses
[ https://issues.apache.org/jira/browse/FLUME-2333?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jeremy Karlson updated FLUME-2333:
----------------------------------
Attachment: FLUME-2333.diff
> HTTP source handler doesn't allow for responses
> -----------------------------------------------
>
> Key: FLUME-2333
> URL: https://issues.apache.org/jira/browse/FLUME-2333
> Project: Flume
> Issue Type: Improvement
> Reporter: Jeremy Karlson
> Assignee: Jeremy Karlson
> Attachments: FLUME-2333.diff
>
>
> Existing HTTP source handlers recieve events via a HTTPServletRequest. This works, but because the handler doesn't have access to the HTTPServletResponse, there is no ability to return a response. This makes it unsuitable for some sort of protocol that relies on bidirectional communication.
> My solution: In addition to the existing HTTPSource interface, I've added a BidirectionalHTTPSource interface that is provided the servlet response as a parameter. I've made some changes in the HTTP source allow for both types to co-exist, and my changes shouldn't affect anyone who is already using the existing interface.
> Also includes minor documentation updates to reflect this.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
Re: [jira] [Updated] (FLUME-2333) HTTP source handler doesn't allow
for responses
Posted by Hari Shreedharan <hs...@cloudera.com>.
Let's keep the discussion on the jira. I posted an update on the jira.
On Fri, Feb 28, 2014 at 11:48 AM, Gabriel Commeau <gabriel.commeau@gmail.com
> wrote:
> Ok, I'll update the ticket without the 3rd point.
> I understand your concern though, and I've actually heard it before when
> this source was first designed. I'll create a different ticket for that.
> Thank you!
>
> Gabriel
>
> From: Jeremy Karlson <je...@gmail.com>
> Reply-To: <de...@flume.apache.org>
> Date: Friday, February 28, 2014 at 11:00 AM
> To: <de...@flume.apache.org>
> Subject: Re: [jira] [Updated] (FLUME-2333) HTTP source handler doesn't
> allow for responses
>
> I am not a committer, but I have no problem with additional changes to this
> report or patch item. It's true that the HTTP handler is limited in its
> openness for extension and if we can make a few changes to make it better,
> I'd be happy to look at any changes you make.
>
> The only thing that I'm uncertain on is point 3. It sounds like the plan
> is to build a simple servlet routing mechanism? My gut feeling is that
> Flume isn't really intended to be be a servlet container. If things are
> that complex, maybe a better choice is to run a standalone Tomcat or Jetty
> out front instead? Or alternatively, can the same thing be accomplished
> reasonably by running each handler on a separate port (the way it would be
> done now)?
>
> Point 3 sounds like a fair bit of work and I'd hate to see you do that and
> have it rejected. Maybe we should wait for some committers to comment
> before you start down that path?
>
> -- Jeremy
>
>
>
> On Fri, Feb 28, 2014 at 8:20 AM, Gabriel Commeau
> <ga...@gmail.com>wrote:
>
> > Hi,
> >
> > Impeccable timing! I was going to submit a change very similar.
> > I agree with you: I do think the HTTPSource ³simple" handler has its
> place,
> > because it provides a simple way to accept events via HTTP, but like
> you, I
> > would like to have more flexibility in the API - so my custom handler
> can
> > implement more complex protocols.
> > I like what you did, and I would have done something close. I would have
> > modified it more however. Would you allow me to add the following
> > requirements to this ticket:
> > - Add in the BidirectionalHTTPSource interface a method to handle the
> > exception that may be thrown while writing the events to the channel: If
> > Flume fails to write the events there, the handler may want to modify
> the
> > response it initially provided. Either way, I would like the response to
> > come from the handler, not from the FlumeHTTPServlet.
> > - Add a third type of handlers, even more generic, which basically
> extends
> > HttpServlet and implements configurable, so that we can add code to do
> > whatever we wish in the HttpSource.
> > - Allow through configuration for many handlers to be added to different
> > paths. In order not to break the backward compatibility, let¹s keep the
> > ³.handler² configuration parameter, and add to it ³.handlers², which
> > basically takes a list of handlers.
> > - Fixed the configure method to use the checkHostAndPort method instead
> of
> > copy and paste (this has been bugging me for a while!) :D
> >
> > I will implement these changes (I started yesterday), so it would be my
> > pleasure to provide a diff. Otherwise, I can create a different ticket.
> > Please let me know what you (and/or the Flume community) think is best.
> > Thank you!
> >
> > Gabriel
> >
> >
> > From: "Jeremy Karlson (JIRA)" <ji...@apache.org>
> > Reply-To: <de...@flume.apache.org>
> > Date: Wednesday, February 26, 2014 at 7:55 PM
> > To: <de...@flume.apache.org>
> > Subject: [jira] [Updated] (FLUME-2333) HTTP source handler doesn't
> allow
> > for responses
> >
> >
> > [
> >
> >
> https://issues.apache.org/jira/browse/FLUME-2333?page=com.atlassian.jira.plu
> > gin.system.issuetabpanels:all-tabpanel ]
> >
> > Jeremy Karlson updated FLUME-2333:
> > ----------------------------------
> >
> > Attachment: FLUME-2333.diff
> >
> >> > HTTP source handler doesn't allow for responses
> >> > -----------------------------------------------
> >> >
> >> > Key: FLUME-2333
> >> > URL:
> https://issues.apache.org/jira/browse/FLUME-2333
> >> > Project: Flume
> >> > Issue Type: Improvement
> >> > Reporter: Jeremy Karlson
> >> > Assignee: Jeremy Karlson
> >> > Attachments: FLUME-2333.diff
> >> >
> >> >
> >> > Existing HTTP source handlers recieve events via a
> HTTPServletRequest.
> > This
> >> > works, but because the handler doesn't have access to the
> > HTTPServletResponse,
> >> > there is no ability to return a response. This makes it unsuitable
> for
> > some
> >> > sort of protocol that relies on bidirectional communication.
> >> > My solution: In addition to the existing HTTPSource interface, I've
> > added a
> >> > BidirectionalHTTPSource interface that is provided the servlet
> response
> > as a
> >> > parameter. I've made some changes in the HTTP source allow for both
> > types to
> >> > co-exist, and my changes shouldn't affect anyone who is already
> using the
> >> > existing interface.
> >> > Also includes minor documentation updates to reflect this.
> >
> >
> >
> > --
> > This message was sent by Atlassian JIRA
> > (v6.1.5#6160)
> >
> >
> >
> >
>
>
>
>
Re: [jira] [Updated] (FLUME-2333) HTTP source handler doesn't allow
for responses
Posted by Gabriel Commeau <ga...@gmail.com>.
Ok, I’ll update the ticket without the 3rd point.
I understand your concern though, and I’ve actually heard it before when
this source was first designed. I’ll create a different ticket for that.
Thank you!
Gabriel
From: Jeremy Karlson <je...@gmail.com>
Reply-To: <de...@flume.apache.org>
Date: Friday, February 28, 2014 at 11:00 AM
To: <de...@flume.apache.org>
Subject: Re: [jira] [Updated] (FLUME-2333) HTTP source handler doesn't
allow for responses
I am not a committer, but I have no problem with additional changes to this
report or patch item. It's true that the HTTP handler is limited in its
openness for extension and if we can make a few changes to make it better,
I'd be happy to look at any changes you make.
The only thing that I'm uncertain on is point 3. It sounds like the plan
is to build a simple servlet routing mechanism? My gut feeling is that
Flume isn't really intended to be be a servlet container. If things are
that complex, maybe a better choice is to run a standalone Tomcat or Jetty
out front instead? Or alternatively, can the same thing be accomplished
reasonably by running each handler on a separate port (the way it would be
done now)?
Point 3 sounds like a fair bit of work and I'd hate to see you do that and
have it rejected. Maybe we should wait for some committers to comment
before you start down that path?
-- Jeremy
On Fri, Feb 28, 2014 at 8:20 AM, Gabriel Commeau
<ga...@gmail.com>wrote:
> Hi,
>
> Impeccable timing! I was going to submit a change very similar.
> I agree with you: I do think the HTTPSource ³simple" handler has its place,
> because it provides a simple way to accept events via HTTP, but like you, I
> would like to have more flexibility in the API so my custom handler can
> implement more complex protocols.
> I like what you did, and I would have done something close. I would have
> modified it more however. Would you allow me to add the following
> requirements to this ticket:
> - Add in the BidirectionalHTTPSource interface a method to handle the
> exception that may be thrown while writing the events to the channel: If
> Flume fails to write the events there, the handler may want to modify the
> response it initially provided. Either way, I would like the response to
> come from the handler, not from the FlumeHTTPServlet.
> - Add a third type of handlers, even more generic, which basically extends
> HttpServlet and implements configurable, so that we can add code to do
> whatever we wish in the HttpSource.
> - Allow through configuration for many handlers to be added to different
> paths. In order not to break the backward compatibility, let¹s keep the
> ³.handler² configuration parameter, and add to it ³.handlers², which
> basically takes a list of handlers.
> - Fixed the configure method to use the checkHostAndPort method instead of
> copy and paste (this has been bugging me for a while!) :D
>
> I will implement these changes (I started yesterday), so it would be my
> pleasure to provide a diff. Otherwise, I can create a different ticket.
> Please let me know what you (and/or the Flume community) think is best.
> Thank you!
>
> Gabriel
>
>
> From: "Jeremy Karlson (JIRA)" <ji...@apache.org>
> Reply-To: <de...@flume.apache.org>
> Date: Wednesday, February 26, 2014 at 7:55 PM
> To: <de...@flume.apache.org>
> Subject: [jira] [Updated] (FLUME-2333) HTTP source handler doesn't allow
> for responses
>
>
> [
>
> https://issues.apache.org/jira/browse/FLUME-2333?page=com.atlassian.jira.plu
> gin.system.issuetabpanels:all-tabpanel ]
>
> Jeremy Karlson updated FLUME-2333:
> ----------------------------------
>
> Attachment: FLUME-2333.diff
>
>> > HTTP source handler doesn't allow for responses
>> > -----------------------------------------------
>> >
>> > Key: FLUME-2333
>> > URL: https://issues.apache.org/jira/browse/FLUME-2333
>> > Project: Flume
>> > Issue Type: Improvement
>> > Reporter: Jeremy Karlson
>> > Assignee: Jeremy Karlson
>> > Attachments: FLUME-2333.diff
>> >
>> >
>> > Existing HTTP source handlers recieve events via a HTTPServletRequest.
> This
>> > works, but because the handler doesn't have access to the
> HTTPServletResponse,
>> > there is no ability to return a response. This makes it unsuitable for
> some
>> > sort of protocol that relies on bidirectional communication.
>> > My solution: In addition to the existing HTTPSource interface, I've
> added a
>> > BidirectionalHTTPSource interface that is provided the servlet response
> as a
>> > parameter. I've made some changes in the HTTP source allow for both
> types to
>> > co-exist, and my changes shouldn't affect anyone who is already using the
>> > existing interface.
>> > Also includes minor documentation updates to reflect this.
>
>
>
> --
> This message was sent by Atlassian JIRA
> (v6.1.5#6160)
>
>
>
>
Re: [jira] [Updated] (FLUME-2333) HTTP source handler doesn't allow
for responses
Posted by Jeremy Karlson <je...@gmail.com>.
I am not a committer, but I have no problem with additional changes to this
report or patch item. It's true that the HTTP handler is limited in its
openness for extension and if we can make a few changes to make it better,
I'd be happy to look at any changes you make.
The only thing that I'm uncertain on is point 3. It sounds like the plan
is to build a simple servlet routing mechanism? My gut feeling is that
Flume isn't really intended to be be a servlet container. If things are
that complex, maybe a better choice is to run a standalone Tomcat or Jetty
out front instead? Or alternatively, can the same thing be accomplished
reasonably by running each handler on a separate port (the way it would be
done now)?
Point 3 sounds like a fair bit of work and I'd hate to see you do that and
have it rejected. Maybe we should wait for some committers to comment
before you start down that path?
-- Jeremy
On Fri, Feb 28, 2014 at 8:20 AM, Gabriel Commeau
<ga...@gmail.com>wrote:
> Hi,
>
> Impeccable timing! I was going to submit a change very similar.
> I agree with you: I do think the HTTPSource ³simple" handler has its place,
> because it provides a simple way to accept events via HTTP, but like you, I
> would like to have more flexibility in the API so my custom handler can
> implement more complex protocols.
> I like what you did, and I would have done something close. I would have
> modified it more however. Would you allow me to add the following
> requirements to this ticket:
> - Add in the BidirectionalHTTPSource interface a method to handle the
> exception that may be thrown while writing the events to the channel: If
> Flume fails to write the events there, the handler may want to modify the
> response it initially provided. Either way, I would like the response to
> come from the handler, not from the FlumeHTTPServlet.
> - Add a third type of handlers, even more generic, which basically extends
> HttpServlet and implements configurable, so that we can add code to do
> whatever we wish in the HttpSource.
> - Allow through configuration for many handlers to be added to different
> paths. In order not to break the backward compatibility, let¹s keep the
> ³.handler² configuration parameter, and add to it ³.handlers², which
> basically takes a list of handlers.
> - Fixed the configure method to use the checkHostAndPort method instead of
> copy and paste (this has been bugging me for a while!) :D
>
> I will implement these changes (I started yesterday), so it would be my
> pleasure to provide a diff. Otherwise, I can create a different ticket.
> Please let me know what you (and/or the Flume community) think is best.
> Thank you!
>
> Gabriel
>
>
> From: "Jeremy Karlson (JIRA)" <ji...@apache.org>
> Reply-To: <de...@flume.apache.org>
> Date: Wednesday, February 26, 2014 at 7:55 PM
> To: <de...@flume.apache.org>
> Subject: [jira] [Updated] (FLUME-2333) HTTP source handler doesn't allow
> for responses
>
>
> [
>
> https://issues.apache.org/jira/browse/FLUME-2333?page=com.atlassian.jira.plu
> gin.system.issuetabpanels:all-tabpanel ]
>
> Jeremy Karlson updated FLUME-2333:
> ----------------------------------
>
> Attachment: FLUME-2333.diff
>
> > HTTP source handler doesn't allow for responses
> > -----------------------------------------------
> >
> > Key: FLUME-2333
> > URL: https://issues.apache.org/jira/browse/FLUME-2333
> > Project: Flume
> > Issue Type: Improvement
> > Reporter: Jeremy Karlson
> > Assignee: Jeremy Karlson
> > Attachments: FLUME-2333.diff
> >
> >
> > Existing HTTP source handlers recieve events via a HTTPServletRequest.
> This
> > works, but because the handler doesn't have access to the
> HTTPServletResponse,
> > there is no ability to return a response. This makes it unsuitable for
> some
> > sort of protocol that relies on bidirectional communication.
> > My solution: In addition to the existing HTTPSource interface, I've
> added a
> > BidirectionalHTTPSource interface that is provided the servlet response
> as a
> > parameter. I've made some changes in the HTTP source allow for both
> types to
> > co-exist, and my changes shouldn't affect anyone who is already using the
> > existing interface.
> > Also includes minor documentation updates to reflect this.
>
>
>
> --
> This message was sent by Atlassian JIRA
> (v6.1.5#6160)
>
>
>
>
Re: [jira] [Updated] (FLUME-2333) HTTP source handler doesn't allow
for responses
Posted by Gabriel Commeau <ga...@gmail.com>.
Hi,
Impeccable timing! I was going to submit a change very similar.
I agree with you: I do think the HTTPSource ³simple" handler has its place,
because it provides a simple way to accept events via HTTP, but like you, I
would like to have more flexibility in the API so my custom handler can
implement more complex protocols.
I like what you did, and I would have done something close. I would have
modified it more however. Would you allow me to add the following
requirements to this ticket:
- Add in the BidirectionalHTTPSource interface a method to handle the
exception that may be thrown while writing the events to the channel: If
Flume fails to write the events there, the handler may want to modify the
response it initially provided. Either way, I would like the response to
come from the handler, not from the FlumeHTTPServlet.
- Add a third type of handlers, even more generic, which basically extends
HttpServlet and implements configurable, so that we can add code to do
whatever we wish in the HttpSource.
- Allow through configuration for many handlers to be added to different
paths. In order not to break the backward compatibility, let¹s keep the
³.handler² configuration parameter, and add to it ³.handlers², which
basically takes a list of handlers.
- Fixed the configure method to use the checkHostAndPort method instead of
copy and paste (this has been bugging me for a while!) :D
I will implement these changes (I started yesterday), so it would be my
pleasure to provide a diff. Otherwise, I can create a different ticket.
Please let me know what you (and/or the Flume community) think is best.
Thank you!
Gabriel
From: "Jeremy Karlson (JIRA)" <ji...@apache.org>
Reply-To: <de...@flume.apache.org>
Date: Wednesday, February 26, 2014 at 7:55 PM
To: <de...@flume.apache.org>
Subject: [jira] [Updated] (FLUME-2333) HTTP source handler doesn't allow
for responses
[
https://issues.apache.org/jira/browse/FLUME-2333?page=com.atlassian.jira.plu
gin.system.issuetabpanels:all-tabpanel ]
Jeremy Karlson updated FLUME-2333:
----------------------------------
Attachment: FLUME-2333.diff
> HTTP source handler doesn't allow for responses
> -----------------------------------------------
>
> Key: FLUME-2333
> URL: https://issues.apache.org/jira/browse/FLUME-2333
> Project: Flume
> Issue Type: Improvement
> Reporter: Jeremy Karlson
> Assignee: Jeremy Karlson
> Attachments: FLUME-2333.diff
>
>
> Existing HTTP source handlers recieve events via a HTTPServletRequest. This
> works, but because the handler doesn't have access to the HTTPServletResponse,
> there is no ability to return a response. This makes it unsuitable for some
> sort of protocol that relies on bidirectional communication.
> My solution: In addition to the existing HTTPSource interface, I've added a
> BidirectionalHTTPSource interface that is provided the servlet response as a
> parameter. I've made some changes in the HTTP source allow for both types to
> co-exist, and my changes shouldn't affect anyone who is already using the
> existing interface.
> Also includes minor documentation updates to reflect this.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)