You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flume.apache.org by shushuai zhu <ss...@yahoo.com> on 2013/06/19 22:10:10 UTC
How to use HTTPS with HTTPSource?
Hi,
I am using HTTP to transfer data from remote machine to HTTPSource of Flume agent (the flume conf. file looks like below). Is HTTPS supported? Anyone has examples to use HTTPS to transfer the data?
agent1.sources = r1
agent1.sources.r1.type = org.apache.flume.source.http.HTTPSource
agent1.sources.r1.port = 41415
agent1.sources.r1.channels = ch1
agent1.sources.r1.handler = org.apache.flume.source.http.JSONHandler
Thanks.
Shushuai
Re: How to use HTTPS with HTTPSource?
Posted by Josh Myers <jo...@mydrivesolutions.com>.
Hi I have not tried this but would suggest using something like nginx to do
your ssl termination and loop back to a local port for your http source
On 19 Jun 2013 21:10, "shushuai zhu" <ss...@yahoo.com> wrote:
> Hi,
>
> I am using HTTP to transfer data from remote machine to HTTPSource of
> Flume agent (the flume conf. file looks like below). Is HTTPS supported?
> Anyone has examples to use HTTPS to transfer the data?
>
> agent1.sources = r1
> agent1.sources.r1.type = org.apache.flume.source.http.HTTPSource
> agent1.sources.r1.port = 41415
> agent1.sources.r1.channels = ch1
> agent1.sources.r1.handler = org.apache.flume.source.http.JSONHandler
> Thanks.
>
> Shushuai
>
>
--
www.mydrivesolutions.com
This email and any attachments is private and confidential. If you have
received this message in error please remove it from your systems and
notify the author.
MyDrive Solutions Limited is registered in England and Wales, No 07330334.
Registered office: Surrey Technology Centre, 40 Occam Road, Guildford GU2
7YG, UK
Re: How to use HTTPS with HTTPSource?
Posted by Ashish <pa...@gmail.com>.
Have added the patch to JIRA. Let's wait for the review.
On Thu, Jun 27, 2013 at 7:20 AM, Ashish <pa...@gmail.com> wrote:
> Sure, I am working on the test cases and Httpclient is giving me tough
> time with SSL. If it doesn't work, shall write simple SSL client to test it.
>
>
> On Wed, Jun 26, 2013 at 8:13 PM, shushuai zhu <ss...@yahoo.com> wrote:
>
>> Ashish, thx. Will try your solution. Please also kindly send a
>> notice after you submit the patch.
>>
>> Shushuai
>> *From:* Ashish <pa...@gmail.com>
>> *To:* user@flume.apache.org; shushuai zhu <ss...@yahoo.com>
>> *Sent:* Wednesday, June 26, 2013 5:28 AM
>>
>> *Subject:* Re: How to use HTTPS with HTTPSource?
>>
>> The example I shared works for Jetty 9, Flume uses 6.1.26.
>>
>> Here is a basic version. Add these lines in HTTPSource in start() method.
>> This is a basic version. You would like to read these properties from flume
>> config in configure(). This is small stuff.
>>
>> SslSocketConnector sslSocketConnector = new SslSocketConnector();
>> sslSocketConnector.setKeystore("jettykeystore");
>> sslSocketConnector.setKeyPassword("password");
>> sslSocketConnector.setPort(8443);
>> srv.setConnectors(new Connector[] { connector, sslSocketConnector });
>>
>> Here is how it would like
>>
>> @Override
>> public void start() {
>> Preconditions.checkState(srv == null,
>> "Running HTTP Server found in source: " + getName()
>> + " before I started one."
>> + "Will not attempt to start.");
>> srv = new Server();
>> SocketConnector connector = new SocketConnector();
>> connector.setPort(port);
>> connector.setHost(host);
>>
>> SslSocketConnector sslSocketConnector = new SslSocketConnector();
>> sslSocketConnector.setKeystore("jettykeystore");
>> sslSocketConnector.setKeyPassword("password");
>> sslSocketConnector.setPort(8443);
>>
>> srv.setConnectors(new Connector[] { connector, sslSocketConnector });
>> try {
>> org.mortbay.jetty.servlet.Context root =
>> new org.mortbay.jetty.servlet.Context(
>> srv, "/", org.mortbay.jetty.servlet.Context.SESSIONS);
>> root.addServlet(new ServletHolder(new FlumeHTTPServlet()), "/");
>> srv.start();
>> ........
>>
>> To submit a patch, would need to refine the code a bit and add test
>> cases. Shall take a while. HTH !
>>
>>
>> On Mon, Jun 24, 2013 at 8:24 PM, shushuai zhu <ss...@yahoo.com> wrote:
>>
>> Ashish, thanks again. Could you elaborate a little more what I should do?
>> I am relatively new to Flume (just started using it a couple of weeks ago)
>> and also new to open source code base.
>>
>> Shushuai
>> *From:* Ashish <pa...@gmail.com>
>> *To:* user@flume.apache.org; shushuai zhu <ss...@yahoo.com>
>> *Sent:* Friday, June 21, 2013 10:38 PM
>>
>> *Subject:* Re: How to use HTTPS with HTTPSource?
>>
>> HTTP source has not changed much :) One way to patch in additional SSL
>> stuff in jetty code. The second more tedious way is to use NIO framework
>> with SSL code like Netty or MINA. We are working on MINA M1 release and
>> perhaps after that shall take a tab at writing an HTTP source based on that.
>>
>> HTH !
>>
>>
>> On Sat, Jun 22, 2013 at 3:07 AM, shushuai zhu <ss...@yahoo.com> wrote:
>>
>> Ashish,
>>
>> Thanks for the pointer. So I will create my own source, say HTTPSSource,
>> which extends the HTPSource to add the https connection, then use the
>> custom HTTPSSource in flume.conf.
>>
>> Is this the right source code you referred (I am using flume 1.3.1)?
>>
>>
>> https://git-wip-us.apache.org/repos/asf?p=flume.git;a=tree;f=flume-ng-core/src/main/java/org/apache/flume/source/http;h=fa5477033ca23c075fec6af2c391b01d1d49f69a;hb=trunk
>>
>>
>> Shushuai
>>
>> *From:* Ashish <pa...@gmail.com>
>> *To:* user@flume.apache.org; shushuai zhu <ss...@yahoo.com>
>> *Sent:* Friday, June 21, 2013 5:09 AM
>> *Subject:* Re: How to use HTTPS with HTTPSource?
>>
>> AFAIK, HTTPSource doesn't support HTTPS. HTTP Source uses embedded jetty,
>> so it would extending HTTP source to add SSL configuration for this.
>>
>> This link should help
>> http://git.eclipse.org/c/jetty/org.eclipse.jetty.project.git/tree/examples/embedded/src/main/java/org/eclipse/jetty/embedded/ManyConnectors.java
>>
>> It should be just few lines of code change.
>>
>> Please note I am referring to source code in master branch.
>>
>>
>> On Thu, Jun 20, 2013 at 1:40 AM, shushuai zhu <ss...@yahoo.com> wrote:
>>
>> Hi,
>>
>> I am using HTTP to transfer data from remote machine to HTTPSource of
>> Flume agent (the flume conf. file looks like below). Is HTTPS supported?
>> Anyone has examples to use HTTPS to transfer the data?
>>
>> agent1.sources = r1
>> agent1.sources.r1.type = org.apache.flume.source.http.HTTPSource
>> agent1.sources.r1.port = 41415
>> agent1.sources.r1.channels = ch1
>> agent1.sources.r1.handler = org.apache.flume.source.http.JSONHandler
>> Thanks.
>>
>> Shushuai
>>
>>
>>
>>
>>
>> --
>> thanks
>> ashish
>>
>> Blog: http://www.ashishpaliwal.com/blog
>> My Photo Galleries: http://www.pbase.com/ashishpaliwal
>>
>>
>>
>>
>>
>> --
>> thanks
>> ashish
>>
>> Blog: http://www.ashishpaliwal.com/blog
>> My Photo Galleries: http://www.pbase.com/ashishpaliwal
>>
>>
>>
>>
>>
>> --
>> thanks
>> ashish
>>
>> Blog: http://www.ashishpaliwal.com/blog
>> My Photo Galleries: http://www.pbase.com/ashishpaliwal
>>
>>
>>
>
>
> --
> thanks
> ashish
>
> Blog: http://www.ashishpaliwal.com/blog
> My Photo Galleries: http://www.pbase.com/ashishpaliwal
>
--
thanks
ashish
Blog: http://www.ashishpaliwal.com/blog
My Photo Galleries: http://www.pbase.com/ashishpaliwal
Re: How to use HTTPS with HTTPSource?
Posted by Ashish <pa...@gmail.com>.
Sure, I am working on the test cases and Httpclient is giving me tough time
with SSL. If it doesn't work, shall write simple SSL client to test it.
On Wed, Jun 26, 2013 at 8:13 PM, shushuai zhu <ss...@yahoo.com> wrote:
> Ashish, thx. Will try your solution. Please also kindly send a
> notice after you submit the patch.
>
> Shushuai
> *From:* Ashish <pa...@gmail.com>
> *To:* user@flume.apache.org; shushuai zhu <ss...@yahoo.com>
> *Sent:* Wednesday, June 26, 2013 5:28 AM
>
> *Subject:* Re: How to use HTTPS with HTTPSource?
>
> The example I shared works for Jetty 9, Flume uses 6.1.26.
>
> Here is a basic version. Add these lines in HTTPSource in start() method.
> This is a basic version. You would like to read these properties from flume
> config in configure(). This is small stuff.
>
> SslSocketConnector sslSocketConnector = new SslSocketConnector();
> sslSocketConnector.setKeystore("jettykeystore");
> sslSocketConnector.setKeyPassword("password");
> sslSocketConnector.setPort(8443);
> srv.setConnectors(new Connector[] { connector, sslSocketConnector });
>
> Here is how it would like
>
> @Override
> public void start() {
> Preconditions.checkState(srv == null,
> "Running HTTP Server found in source: " + getName()
> + " before I started one."
> + "Will not attempt to start.");
> srv = new Server();
> SocketConnector connector = new SocketConnector();
> connector.setPort(port);
> connector.setHost(host);
>
> SslSocketConnector sslSocketConnector = new SslSocketConnector();
> sslSocketConnector.setKeystore("jettykeystore");
> sslSocketConnector.setKeyPassword("password");
> sslSocketConnector.setPort(8443);
>
> srv.setConnectors(new Connector[] { connector, sslSocketConnector });
> try {
> org.mortbay.jetty.servlet.Context root =
> new org.mortbay.jetty.servlet.Context(
> srv, "/", org.mortbay.jetty.servlet.Context.SESSIONS);
> root.addServlet(new ServletHolder(new FlumeHTTPServlet()), "/");
> srv.start();
> ........
>
> To submit a patch, would need to refine the code a bit and add test cases.
> Shall take a while. HTH !
>
>
> On Mon, Jun 24, 2013 at 8:24 PM, shushuai zhu <ss...@yahoo.com> wrote:
>
> Ashish, thanks again. Could you elaborate a little more what I should do?
> I am relatively new to Flume (just started using it a couple of weeks ago)
> and also new to open source code base.
>
> Shushuai
> *From:* Ashish <pa...@gmail.com>
> *To:* user@flume.apache.org; shushuai zhu <ss...@yahoo.com>
> *Sent:* Friday, June 21, 2013 10:38 PM
>
> *Subject:* Re: How to use HTTPS with HTTPSource?
>
> HTTP source has not changed much :) One way to patch in additional SSL
> stuff in jetty code. The second more tedious way is to use NIO framework
> with SSL code like Netty or MINA. We are working on MINA M1 release and
> perhaps after that shall take a tab at writing an HTTP source based on that.
>
> HTH !
>
>
> On Sat, Jun 22, 2013 at 3:07 AM, shushuai zhu <ss...@yahoo.com> wrote:
>
> Ashish,
>
> Thanks for the pointer. So I will create my own source, say HTTPSSource,
> which extends the HTPSource to add the https connection, then use the
> custom HTTPSSource in flume.conf.
>
> Is this the right source code you referred (I am using flume 1.3.1)?
>
>
> https://git-wip-us.apache.org/repos/asf?p=flume.git;a=tree;f=flume-ng-core/src/main/java/org/apache/flume/source/http;h=fa5477033ca23c075fec6af2c391b01d1d49f69a;hb=trunk
>
>
> Shushuai
>
> *From:* Ashish <pa...@gmail.com>
> *To:* user@flume.apache.org; shushuai zhu <ss...@yahoo.com>
> *Sent:* Friday, June 21, 2013 5:09 AM
> *Subject:* Re: How to use HTTPS with HTTPSource?
>
> AFAIK, HTTPSource doesn't support HTTPS. HTTP Source uses embedded jetty,
> so it would extending HTTP source to add SSL configuration for this.
>
> This link should help
> http://git.eclipse.org/c/jetty/org.eclipse.jetty.project.git/tree/examples/embedded/src/main/java/org/eclipse/jetty/embedded/ManyConnectors.java
>
> It should be just few lines of code change.
>
> Please note I am referring to source code in master branch.
>
>
> On Thu, Jun 20, 2013 at 1:40 AM, shushuai zhu <ss...@yahoo.com> wrote:
>
> Hi,
>
> I am using HTTP to transfer data from remote machine to HTTPSource of
> Flume agent (the flume conf. file looks like below). Is HTTPS supported?
> Anyone has examples to use HTTPS to transfer the data?
>
> agent1.sources = r1
> agent1.sources.r1.type = org.apache.flume.source.http.HTTPSource
> agent1.sources.r1.port = 41415
> agent1.sources.r1.channels = ch1
> agent1.sources.r1.handler = org.apache.flume.source.http.JSONHandler
> Thanks.
>
> Shushuai
>
>
>
>
>
> --
> thanks
> ashish
>
> Blog: http://www.ashishpaliwal.com/blog
> My Photo Galleries: http://www.pbase.com/ashishpaliwal
>
>
>
>
>
> --
> thanks
> ashish
>
> Blog: http://www.ashishpaliwal.com/blog
> My Photo Galleries: http://www.pbase.com/ashishpaliwal
>
>
>
>
>
> --
> thanks
> ashish
>
> Blog: http://www.ashishpaliwal.com/blog
> My Photo Galleries: http://www.pbase.com/ashishpaliwal
>
>
>
--
thanks
ashish
Blog: http://www.ashishpaliwal.com/blog
My Photo Galleries: http://www.pbase.com/ashishpaliwal
Re: How to use HTTPS with HTTPSource?
Posted by shushuai zhu <ss...@yahoo.com>.
Ashish, thx. Will try your solution. Please also kindly send a notice after you submit the patch.
Shushuai
________________________________
From: Ashish <pa...@gmail.com>
To: user@flume.apache.org; shushuai zhu <ss...@yahoo.com>
Sent: Wednesday, June 26, 2013 5:28 AM
Subject: Re: How to use HTTPS with HTTPSource?
The example I shared works for Jetty 9, Flume uses 6.1.26.
Here is a basic version. Add these lines in HTTPSource in start() method. This is a basic version. You would like to read these properties from flume config in configure(). This is small stuff.
SslSocketConnector sslSocketConnector = new SslSocketConnector();
sslSocketConnector.setKeystore("jettykeystore");
sslSocketConnector.setKeyPassword("password");
sslSocketConnector.setPort(8443);
srv.setConnectors(new Connector[] { connector, sslSocketConnector });
Here is how it would like
@Override
public void start() {
Preconditions.checkState(srv == null,
"Running HTTP Server found in source: " + getName()
+ " before I started one."
+ "Will not attempt to start.");
srv = new Server();
SocketConnector connector = new SocketConnector();
connector.setPort(port);
connector.setHost(host);
SslSocketConnector sslSocketConnector = new SslSocketConnector();
sslSocketConnector.setKeystore("jettykeystore");
sslSocketConnector.setKeyPassword("password");
sslSocketConnector.setPort(8443);
srv.setConnectors(new Connector[] { connector, sslSocketConnector });
try {
org.mortbay.jetty.servlet.Context root =
new org.mortbay.jetty.servlet.Context(
srv, "/", org.mortbay.jetty.servlet.Context.SESSIONS);
root.addServlet(new ServletHolder(new FlumeHTTPServlet()), "/");
srv.start();
........
To submit a patch, would need to refine the code a bit and add test cases. Shall take a while. HTH !
On Mon, Jun 24, 2013 at 8:24 PM, shushuai zhu <ss...@yahoo.com> wrote:
Ashish, thanks again. Could you elaborate a little more what I should do? I am relatively new to Flume (just started using it a couple of weeks ago) and also new to open source code base.
>
>Shushuai
>
> From: Ashish <pa...@gmail.com>
>To: user@flume.apache.org; shushuai zhu <ss...@yahoo.com>
>Sent: Friday, June 21, 2013 10:38 PM
>
>Subject: Re: How to use HTTPS with HTTPSource?
>
>
>
>HTTP source has not changed much :) One way to patch in additional SSL stuff in jetty code. The second more tedious way is to use NIO framework with SSL code like Netty or MINA. We are working on MINA M1 release and perhaps after that shall take a tab at writing an HTTP source based on that.
>
>
>HTH !
>
>
>
>On Sat, Jun 22, 2013 at 3:07 AM, shushuai zhu <ss...@yahoo.com> wrote:
>
>Ashish,
>>
>>Thanks for the pointer. So I will create my own source, say HTTPSSource, which extends the HTPSource to add the https connection, then use the custom HTTPSSource in flume.conf.
>>
>>Is this the right source code you referred (I am using flume 1.3.1)?
>>
>>https://git-wip-us.apache.org/repos/asf?p=flume.git;a=tree;f=flume-ng-core/src/main/java/org/apache/flume/source/http;h=fa5477033ca23c075fec6af2c391b01d1d49f69a;hb=trunk
>>
>>Shushuai
>>
>>
>> From: Ashish <pa...@gmail.com>
>>To: user@flume.apache.org; shushuai zhu <ss...@yahoo.com>
>>Sent: Friday, June 21, 2013 5:09 AM
>>Subject: Re: How to use HTTPS with HTTPSource?
>>
>>
>>
>>AFAIK, HTTPSource doesn't support HTTPS. HTTP Source uses embedded jetty, so it would extending HTTP source to add SSL configuration for this.
>>
>>
>>This link should help http://git.eclipse.org/c/jetty/org.eclipse.jetty.project.git/tree/examples/embedded/src/main/java/org/eclipse/jetty/embedded/ManyConnectors.java
>>
>>
>>It should be just few lines of code change.
>>
>>
>>Please note I am referring to source code in master branch.
>>
>>
>>
>>On Thu, Jun 20, 2013 at 1:40 AM, shushuai zhu <ss...@yahoo.com> wrote:
>>
>>Hi,
>>>
>>>I am using HTTP to transfer data from remote machine to HTTPSource of Flume agent (the flume conf. file looks like below). Is HTTPS supported? Anyone has examples to use HTTPS to transfer the data?
>>>
>>>agent1.sources = r1
>>>agent1.sources.r1.type = org.apache.flume.source.http.HTTPSource
>>>agent1.sources.r1.port = 41415
>>>agent1.sources.r1.channels = ch1
>>>agent1.sources.r1.handler = org.apache.flume.source.http.JSONHandler
>>>
>>>Thanks.
>>>
>>>Shushuai
>>>
>>
>>
>>
>>--
>>thanks
>>ashish
>>
>>Blog: http://www.ashishpaliwal.com/blog
>>My Photo Galleries: http://www.pbase.com/ashishpaliwal
>>
>>
>
>
>
>--
>thanks
>ashish
>
>Blog: http://www.ashishpaliwal.com/blog
>My Photo Galleries: http://www.pbase.com/ashishpaliwal
>
>
--
thanks
ashish
Blog: http://www.ashishpaliwal.com/blog
My Photo Galleries: http://www.pbase.com/ashishpaliwal
Re: How to use HTTPS with HTTPSource?
Posted by Ashish <pa...@gmail.com>.
The example I shared works for Jetty 9, Flume uses 6.1.26.
Here is a basic version. Add these lines in HTTPSource in start() method.
This is a basic version. You would like to read these properties from flume
config in configure(). This is small stuff.
SslSocketConnector sslSocketConnector = new SslSocketConnector();
sslSocketConnector.setKeystore("jettykeystore");
sslSocketConnector.setKeyPassword("password");
sslSocketConnector.setPort(8443);
srv.setConnectors(new Connector[] { connector, sslSocketConnector });
Here is how it would like
@Override
public void start() {
Preconditions.checkState(srv == null,
"Running HTTP Server found in source: " + getName()
+ " before I started one."
+ "Will not attempt to start.");
srv = new Server();
SocketConnector connector = new SocketConnector();
connector.setPort(port);
connector.setHost(host);
SslSocketConnector sslSocketConnector = new SslSocketConnector();
sslSocketConnector.setKeystore("jettykeystore");
sslSocketConnector.setKeyPassword("password");
sslSocketConnector.setPort(8443);
srv.setConnectors(new Connector[] { connector, sslSocketConnector });
try {
org.mortbay.jetty.servlet.Context root =
new org.mortbay.jetty.servlet.Context(
srv, "/", org.mortbay.jetty.servlet.Context.SESSIONS);
root.addServlet(new ServletHolder(new FlumeHTTPServlet()), "/");
srv.start();
........
To submit a patch, would need to refine the code a bit and add test cases.
Shall take a while. HTH !
On Mon, Jun 24, 2013 at 8:24 PM, shushuai zhu <ss...@yahoo.com> wrote:
> Ashish, thanks again. Could you elaborate a little more what I should do?
> I am relatively new to Flume (just started using it a couple of weeks ago)
> and also new to open source code base.
>
> Shushuai
> *From:* Ashish <pa...@gmail.com>
> *To:* user@flume.apache.org; shushuai zhu <ss...@yahoo.com>
> *Sent:* Friday, June 21, 2013 10:38 PM
>
> *Subject:* Re: How to use HTTPS with HTTPSource?
>
> HTTP source has not changed much :) One way to patch in additional SSL
> stuff in jetty code. The second more tedious way is to use NIO framework
> with SSL code like Netty or MINA. We are working on MINA M1 release and
> perhaps after that shall take a tab at writing an HTTP source based on that.
>
> HTH !
>
>
> On Sat, Jun 22, 2013 at 3:07 AM, shushuai zhu <ss...@yahoo.com> wrote:
>
> Ashish,
>
> Thanks for the pointer. So I will create my own source, say HTTPSSource,
> which extends the HTPSource to add the https connection, then use the
> custom HTTPSSource in flume.conf.
>
> Is this the right source code you referred (I am using flume 1.3.1)?
>
>
> https://git-wip-us.apache.org/repos/asf?p=flume.git;a=tree;f=flume-ng-core/src/main/java/org/apache/flume/source/http;h=fa5477033ca23c075fec6af2c391b01d1d49f69a;hb=trunk
>
>
> Shushuai
>
> *From:* Ashish <pa...@gmail.com>
> *To:* user@flume.apache.org; shushuai zhu <ss...@yahoo.com>
> *Sent:* Friday, June 21, 2013 5:09 AM
> *Subject:* Re: How to use HTTPS with HTTPSource?
>
> AFAIK, HTTPSource doesn't support HTTPS. HTTP Source uses embedded jetty,
> so it would extending HTTP source to add SSL configuration for this.
>
> This link should help
> http://git.eclipse.org/c/jetty/org.eclipse.jetty.project.git/tree/examples/embedded/src/main/java/org/eclipse/jetty/embedded/ManyConnectors.java
>
> It should be just few lines of code change.
>
> Please note I am referring to source code in master branch.
>
>
> On Thu, Jun 20, 2013 at 1:40 AM, shushuai zhu <ss...@yahoo.com> wrote:
>
> Hi,
>
> I am using HTTP to transfer data from remote machine to HTTPSource of
> Flume agent (the flume conf. file looks like below). Is HTTPS supported?
> Anyone has examples to use HTTPS to transfer the data?
>
> agent1.sources = r1
> agent1.sources.r1.type = org.apache.flume.source.http.HTTPSource
> agent1.sources.r1.port = 41415
> agent1.sources.r1.channels = ch1
> agent1.sources.r1.handler = org.apache.flume.source.http.JSONHandler
> Thanks.
>
> Shushuai
>
>
>
>
>
> --
> thanks
> ashish
>
> Blog: http://www.ashishpaliwal.com/blog
> My Photo Galleries: http://www.pbase.com/ashishpaliwal
>
>
>
>
>
> --
> thanks
> ashish
>
> Blog: http://www.ashishpaliwal.com/blog
> My Photo Galleries: http://www.pbase.com/ashishpaliwal
>
>
>
--
thanks
ashish
Blog: http://www.ashishpaliwal.com/blog
My Photo Galleries: http://www.pbase.com/ashishpaliwal
Re: How to use HTTPS with HTTPSource?
Posted by Hari Shreedharan <hs...@cloudera.com>.
I filed https://issues.apache.org/jira/browse/FLUME-2109 to track this. Feel free to submit a patch!
Cheers,
Hari
On Monday, June 24, 2013 at 8:05 AM, Ashish wrote:
> So am I :) I am just into my 2nd week of reading about Flume. If I were in your place, I would have added Jetty SSL example code to HTTP source and make it work. Other approaches are purist approaches, and are needed only when you need a very low level control of your implementation.
>
> I can try to put up an example by this weekend, if time permits.
>
> HTH !
>
>
>
> On Mon, Jun 24, 2013 at 8:24 PM, shushuai zhu <sszhu@yahoo.com (mailto:sszhu@yahoo.com)> wrote:
> > Ashish, thanks again. Could you elaborate a little more what I should do? I am relatively new to Flume (just started using it a couple of weeks ago) and also new to open source code base.
> >
> > Shushuai
> > From: Ashish <paliwalashish@gmail.com (mailto:paliwalashish@gmail.com)>
> > To: user@flume.apache.org (mailto:user@flume.apache.org); shushuai zhu <sszhu@yahoo.com (mailto:sszhu@yahoo.com)>
> > Sent: Friday, June 21, 2013 10:38 PM
> >
> > Subject: Re: How to use HTTPS with HTTPSource?
> >
> > HTTP source has not changed much :) One way to patch in additional SSL stuff in jetty code. The second more tedious way is to use NIO framework with SSL code like Netty or MINA. We are working on MINA M1 release and perhaps after that shall take a tab at writing an HTTP source based on that.
> >
> > HTH !
> >
> >
> > On Sat, Jun 22, 2013 at 3:07 AM, shushuai zhu <sszhu@yahoo.com (mailto:sszhu@yahoo.com)> wrote:
> > > Ashish,
> > >
> > > Thanks for the pointer. So I will create my own source, say HTTPSSource, which extends the HTPSource to add the https connection, then use the custom HTTPSSource in flume.conf.
> > >
> > > Is this the right source code you referred (I am using flume 1.3.1)?
> > >
> > > https://git-wip-us.apache.org/repos/asf?p=flume.git;a=tree;f=flume-ng-core/src/main/java/org/apache/flume/source/http;h=fa5477033ca23c075fec6af2c391b01d1d49f69a;hb=trunk
> > >
> > > Shushuai
> > >
> > > From: Ashish <paliwalashish@gmail.com (mailto:paliwalashish@gmail.com)>
> > > To: user@flume.apache.org (mailto:user@flume.apache.org); shushuai zhu <sszhu@yahoo.com (mailto:sszhu@yahoo.com)>
> > > Sent: Friday, June 21, 2013 5:09 AM
> > > Subject: Re: How to use HTTPS with HTTPSource?
> > >
> > > AFAIK, HTTPSource doesn't support HTTPS. HTTP Source uses embedded jetty, so it would extending HTTP source to add SSL configuration for this.
> > >
> > > This link should help http://git.eclipse.org/c/jetty/org.eclipse.jetty.project.git/tree/examples/embedded/src/main/java/org/eclipse/jetty/embedded/ManyConnectors.java
> > >
> > > It should be just few lines of code change.
> > >
> > > Please note I am referring to source code in master branch.
> > >
> > >
> > > On Thu, Jun 20, 2013 at 1:40 AM, shushuai zhu <sszhu@yahoo.com (mailto:sszhu@yahoo.com)> wrote:
> > > > Hi,
> > > >
> > > > I am using HTTP to transfer data from remote machine to HTTPSource of Flume agent (the flume conf. file looks like below). Is HTTPS supported? Anyone has examples to use HTTPS to transfer the data?
> > > >
> > > > agent1.sources = r1
> > > > agent1.sources.r1.type = org.apache.flume.source.http.HTTPSource
> > > > agent1.sources.r1.port = 41415
> > > > agent1.sources.r1.channels = ch1
> > > > agent1.sources.r1.handler = org.apache.flume.source.http.JSONHandler
> > > > Thanks.
> > > >
> > > > Shushuai
> > > >
> > >
> > >
> > > --
> > > thanks
> > > ashish
> > >
> > > Blog: http://www.ashishpaliwal.com/blog
> > > My Photo Galleries: http://www.pbase.com/ashishpaliwal
> > >
> >
> >
> >
> > --
> > thanks
> > ashish
> >
> > Blog: http://www.ashishpaliwal.com/blog
> > My Photo Galleries: http://www.pbase.com/ashishpaliwal
> >
>
>
>
> --
> thanks
> ashish
>
> Blog: http://www.ashishpaliwal.com/blog
> My Photo Galleries: http://www.pbase.com/ashishpaliwal
Re: How to use HTTPS with HTTPSource?
Posted by Ashish <pa...@gmail.com>.
So am I :) I am just into my 2nd week of reading about Flume. If I were in
your place, I would have added Jetty SSL example code to HTTP source and
make it work. Other approaches are purist approaches, and are needed only
when you need a very low level control of your implementation.
I can try to put up an example by this weekend, if time permits.
HTH !
On Mon, Jun 24, 2013 at 8:24 PM, shushuai zhu <ss...@yahoo.com> wrote:
> Ashish, thanks again. Could you elaborate a little more what I should do?
> I am relatively new to Flume (just started using it a couple of weeks ago)
> and also new to open source code base.
>
> Shushuai
> *From:* Ashish <pa...@gmail.com>
> *To:* user@flume.apache.org; shushuai zhu <ss...@yahoo.com>
> *Sent:* Friday, June 21, 2013 10:38 PM
>
> *Subject:* Re: How to use HTTPS with HTTPSource?
>
> HTTP source has not changed much :) One way to patch in additional SSL
> stuff in jetty code. The second more tedious way is to use NIO framework
> with SSL code like Netty or MINA. We are working on MINA M1 release and
> perhaps after that shall take a tab at writing an HTTP source based on that.
>
> HTH !
>
>
> On Sat, Jun 22, 2013 at 3:07 AM, shushuai zhu <ss...@yahoo.com> wrote:
>
> Ashish,
>
> Thanks for the pointer. So I will create my own source, say HTTPSSource,
> which extends the HTPSource to add the https connection, then use the
> custom HTTPSSource in flume.conf.
>
> Is this the right source code you referred (I am using flume 1.3.1)?
>
>
> https://git-wip-us.apache.org/repos/asf?p=flume.git;a=tree;f=flume-ng-core/src/main/java/org/apache/flume/source/http;h=fa5477033ca23c075fec6af2c391b01d1d49f69a;hb=trunk
>
>
> Shushuai
>
> *From:* Ashish <pa...@gmail.com>
> *To:* user@flume.apache.org; shushuai zhu <ss...@yahoo.com>
> *Sent:* Friday, June 21, 2013 5:09 AM
> *Subject:* Re: How to use HTTPS with HTTPSource?
>
> AFAIK, HTTPSource doesn't support HTTPS. HTTP Source uses embedded jetty,
> so it would extending HTTP source to add SSL configuration for this.
>
> This link should help
> http://git.eclipse.org/c/jetty/org.eclipse.jetty.project.git/tree/examples/embedded/src/main/java/org/eclipse/jetty/embedded/ManyConnectors.java
>
> It should be just few lines of code change.
>
> Please note I am referring to source code in master branch.
>
>
> On Thu, Jun 20, 2013 at 1:40 AM, shushuai zhu <ss...@yahoo.com> wrote:
>
> Hi,
>
> I am using HTTP to transfer data from remote machine to HTTPSource of
> Flume agent (the flume conf. file looks like below). Is HTTPS supported?
> Anyone has examples to use HTTPS to transfer the data?
>
> agent1.sources = r1
> agent1.sources.r1.type = org.apache.flume.source.http.HTTPSource
> agent1.sources.r1.port = 41415
> agent1.sources.r1.channels = ch1
> agent1.sources.r1.handler = org.apache.flume.source.http.JSONHandler
> Thanks.
>
> Shushuai
>
>
>
>
>
> --
> thanks
> ashish
>
> Blog: http://www.ashishpaliwal.com/blog
> My Photo Galleries: http://www.pbase.com/ashishpaliwal
>
>
>
>
>
> --
> thanks
> ashish
>
> Blog: http://www.ashishpaliwal.com/blog
> My Photo Galleries: http://www.pbase.com/ashishpaliwal
>
>
>
--
thanks
ashish
Blog: http://www.ashishpaliwal.com/blog
My Photo Galleries: http://www.pbase.com/ashishpaliwal
Re: How to use HTTPS with HTTPSource?
Posted by shushuai zhu <ss...@yahoo.com>.
Ashish, thanks again. Could you elaborate a little more what I should do? I am relatively new to Flume (just started using it a couple of weeks ago) and also new to open source code base.
Shushuai
________________________________
From: Ashish <pa...@gmail.com>
To: user@flume.apache.org; shushuai zhu <ss...@yahoo.com>
Sent: Friday, June 21, 2013 10:38 PM
Subject: Re: How to use HTTPS with HTTPSource?
HTTP source has not changed much :) One way to patch in additional SSL stuff in jetty code. The second more tedious way is to use NIO framework with SSL code like Netty or MINA. We are working on MINA M1 release and perhaps after that shall take a tab at writing an HTTP source based on that.
HTH !
On Sat, Jun 22, 2013 at 3:07 AM, shushuai zhu <ss...@yahoo.com> wrote:
Ashish,
>
>Thanks for the pointer. So I will create my own source, say HTTPSSource, which extends the HTPSource to add the https connection, then use the custom HTTPSSource in flume.conf.
>
>Is this the right source code you referred (I am using flume 1.3.1)?
>
>https://git-wip-us.apache.org/repos/asf?p=flume.git;a=tree;f=flume-ng-core/src/main/java/org/apache/flume/source/http;h=fa5477033ca23c075fec6af2c391b01d1d49f69a;hb=trunk
>
>Shushuai
>
>
> From: Ashish <pa...@gmail.com>
>To: user@flume.apache.org; shushuai zhu <ss...@yahoo.com>
>Sent: Friday, June 21, 2013 5:09 AM
>Subject: Re: How to use HTTPS with HTTPSource?
>
>
>
>AFAIK, HTTPSource doesn't support HTTPS. HTTP Source uses embedded jetty, so it would extending HTTP source to add SSL configuration for this.
>
>
>This link should help http://git.eclipse.org/c/jetty/org.eclipse.jetty.project.git/tree/examples/embedded/src/main/java/org/eclipse/jetty/embedded/ManyConnectors.java
>
>
>It should be just few lines of code change.
>
>
>Please note I am referring to source code in master branch.
>
>
>
>On Thu, Jun 20, 2013 at 1:40 AM, shushuai zhu <ss...@yahoo.com> wrote:
>
>Hi,
>>
>>I am using HTTP to transfer data from remote machine to HTTPSource of Flume agent (the flume conf. file looks like below). Is HTTPS supported? Anyone has examples to use HTTPS to transfer the data?
>>
>>agent1.sources = r1
>>agent1.sources.r1.type = org.apache.flume.source.http.HTTPSource
>>agent1.sources.r1.port = 41415
>>agent1.sources.r1.channels = ch1
>>agent1.sources.r1.handler = org.apache.flume.source.http.JSONHandler
>>
>>Thanks.
>>
>>Shushuai
>>
>
>
>
>--
>thanks
>ashish
>
>Blog: http://www.ashishpaliwal.com/blog
>My Photo Galleries: http://www.pbase.com/ashishpaliwal
>
>
--
thanks
ashish
Blog: http://www.ashishpaliwal.com/blog
My Photo Galleries: http://www.pbase.com/ashishpaliwal
Re: How to use HTTPS with HTTPSource?
Posted by Ashish <pa...@gmail.com>.
HTTP source has not changed much :) One way to patch in additional SSL
stuff in jetty code. The second more tedious way is to use NIO framework
with SSL code like Netty or MINA. We are working on MINA M1 release and
perhaps after that shall take a tab at writing an HTTP source based on that.
HTH !
On Sat, Jun 22, 2013 at 3:07 AM, shushuai zhu <ss...@yahoo.com> wrote:
> Ashish,
>
> Thanks for the pointer. So I will create my own source, say HTTPSSource,
> which extends the HTPSource to add the https connection, then use the
> custom HTTPSSource in flume.conf.
>
> Is this the right source code you referred (I am using flume 1.3.1)?
>
>
> https://git-wip-us.apache.org/repos/asf?p=flume.git;a=tree;f=flume-ng-core/src/main/java/org/apache/flume/source/http;h=fa5477033ca23c075fec6af2c391b01d1d49f69a;hb=trunk
>
>
> Shushuai
>
> *From:* Ashish <pa...@gmail.com>
> *To:* user@flume.apache.org; shushuai zhu <ss...@yahoo.com>
> *Sent:* Friday, June 21, 2013 5:09 AM
> *Subject:* Re: How to use HTTPS with HTTPSource?
>
> AFAIK, HTTPSource doesn't support HTTPS. HTTP Source uses embedded jetty,
> so it would extending HTTP source to add SSL configuration for this.
>
> This link should help
> http://git.eclipse.org/c/jetty/org.eclipse.jetty.project.git/tree/examples/embedded/src/main/java/org/eclipse/jetty/embedded/ManyConnectors.java
>
> It should be just few lines of code change.
>
> Please note I am referring to source code in master branch.
>
>
> On Thu, Jun 20, 2013 at 1:40 AM, shushuai zhu <ss...@yahoo.com> wrote:
>
> Hi,
>
> I am using HTTP to transfer data from remote machine to HTTPSource of
> Flume agent (the flume conf. file looks like below). Is HTTPS supported?
> Anyone has examples to use HTTPS to transfer the data?
>
> agent1.sources = r1
> agent1.sources.r1.type = org.apache.flume.source.http.HTTPSource
> agent1.sources.r1.port = 41415
> agent1.sources.r1.channels = ch1
> agent1.sources.r1.handler = org.apache.flume.source.http.JSONHandler
> Thanks.
>
> Shushuai
>
>
>
>
>
> --
> thanks
> ashish
>
> Blog: http://www.ashishpaliwal.com/blog
> My Photo Galleries: http://www.pbase.com/ashishpaliwal
>
>
>
--
thanks
ashish
Blog: http://www.ashishpaliwal.com/blog
My Photo Galleries: http://www.pbase.com/ashishpaliwal
Re: How to use HTTPS with HTTPSource?
Posted by shushuai zhu <ss...@yahoo.com>.
Ashish,
Thanks for the pointer. So I will create my own source, say HTTPSSource, which extends the HTPSource to add the https connection, then use the custom HTTPSSource in flume.conf.
Is this the right source code you referred (I am using flume 1.3.1)?
https://git-wip-us.apache.org/repos/asf?p=flume.git;a=tree;f=flume-ng-core/src/main/java/org/apache/flume/source/http;h=fa5477033ca23c075fec6af2c391b01d1d49f69a;hb=trunk
Shushuai
________________________________
From: Ashish <pa...@gmail.com>
To: user@flume.apache.org; shushuai zhu <ss...@yahoo.com>
Sent: Friday, June 21, 2013 5:09 AM
Subject: Re: How to use HTTPS with HTTPSource?
AFAIK, HTTPSource doesn't support HTTPS. HTTP Source uses embedded jetty, so it would extending HTTP source to add SSL configuration for this.
This link should help http://git.eclipse.org/c/jetty/org.eclipse.jetty.project.git/tree/examples/embedded/src/main/java/org/eclipse/jetty/embedded/ManyConnectors.java
It should be just few lines of code change.
Please note I am referring to source code in master branch.
On Thu, Jun 20, 2013 at 1:40 AM, shushuai zhu <ss...@yahoo.com> wrote:
Hi,
>
>I am using HTTP to transfer data from remote machine to HTTPSource of Flume agent (the flume conf. file looks like below). Is HTTPS supported? Anyone has examples to use HTTPS to transfer the data?
>
>agent1.sources = r1
>agent1.sources.r1.type = org.apache.flume.source.http.HTTPSource
>agent1.sources.r1.port = 41415
>agent1.sources.r1.channels = ch1
>agent1.sources.r1.handler = org.apache.flume.source.http.JSONHandler
>
>Thanks.
>
>Shushuai
>
--
thanks
ashish
Blog: http://www.ashishpaliwal.com/blog
My Photo Galleries: http://www.pbase.com/ashishpaliwal
Re: How to use HTTPS with HTTPSource?
Posted by Ashish <pa...@gmail.com>.
AFAIK, HTTPSource doesn't support HTTPS. HTTP Source uses embedded jetty,
so it would extending HTTP source to add SSL configuration for this.
This link should help
http://git.eclipse.org/c/jetty/org.eclipse.jetty.project.git/tree/examples/embedded/src/main/java/org/eclipse/jetty/embedded/ManyConnectors.java
It should be just few lines of code change.
Please note I am referring to source code in master branch.
On Thu, Jun 20, 2013 at 1:40 AM, shushuai zhu <ss...@yahoo.com> wrote:
> Hi,
>
> I am using HTTP to transfer data from remote machine to HTTPSource of
> Flume agent (the flume conf. file looks like below). Is HTTPS supported?
> Anyone has examples to use HTTPS to transfer the data?
>
> agent1.sources = r1
> agent1.sources.r1.type = org.apache.flume.source.http.HTTPSource
> agent1.sources.r1.port = 41415
> agent1.sources.r1.channels = ch1
> agent1.sources.r1.handler = org.apache.flume.source.http.JSONHandler
> Thanks.
>
> Shushuai
>
>
--
thanks
ashish
Blog: http://www.ashishpaliwal.com/blog
My Photo Galleries: http://www.pbase.com/ashishpaliwal