You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flume.apache.org by Carlotta Hicks <Ca...@sas.com> on 2015/01/16 19:30:29 UTC
Java API for Flume Agent
I need to create an agent using java API to load an HBase table using a CSV file. I am able to accomplish this using flume-ng and an agent with a spooldir directory source, file channel and HBase sink.
I need information on how to accomplish this via a java API.
Any help would be appreciated.
-CM
Re: Java API for Flume Agent
Posted by Hari Shreedharan <hs...@cloudera.com>.
Can you enable debug logging for Flume? I want to see what the
configuration looks like.
On Thu, Jan 22, 2015 at 8:05 PM, Carlotta Hicks <Ca...@sas.com>
wrote:
> Here is the code change in EmbeddedAgentConfiguration:
>
>
>
> /**
>
> * Avro sink which can send events to a downstream avro source. This is
> the
>
> * only supported sink for Embedded Agents.
>
> */
>
> public static final String SINK_TYPE_AVRO = SinkType.AVRO.name();
>
> public static final String SINK_TYPE_HBASE = SinkType.HBASE.name();
>
> public static final String SINK_TYPE_HDFS = SinkType.HDFS.name();
>
>
>
> private static final String[] ALLOWED_SINKS = {
>
> SINK_TYPE_AVRO,
>
> SINK_TYPE_HBASE,
>
> SINK_TYPE_HDFS
>
> };
>
>
>
> This is my code with configurations:
>
>
>
> Map<String, String> properties = new HashMap<String, String>();
>
>
>
> properties.put("channel.type", "file");
>
> properties.put("sinks", "hdfs1");
>
> properties.put("hdfs1.type", "hdfs");
>
> properties.put("hdfs1.path", "hdfs://lax07.unx.sas.com:8020/tmp");
>
> properties.put("processor.type", "default");
>
>
>
> agent = new EmbeddedAgent(“myAgent”);
>
>
>
> agent.configure(properties); ß This statement causes
> the exception below
>
>
>
> org.apache.flume.FlumeException: Expected one source and got 0
>
> at
> org.apache.flume.agent.embedded.EmbeddedAgent.doConfigure(EmbeddedAgent.java:160)
>
> at
> org.apache.flume.agent.embedded.EmbeddedAgent.configure(EmbeddedAgent.java:95)
>
>
>
>
>
>
>
> *From:* Hari Shreedharan [mailto:hshreedharan@cloudera.com]
> *Sent:* Tuesday, January 20, 2015 3:23 PM
> *To:* user@flume.apache.org
> *Cc:* user@flume.apache.org
> *Subject:* RE: Java API for Flume Agent
>
>
>
> Can you send the code you used?
>
>
> Thanks,
>
> Hari
>
>
>
> On Tue, Jan 20, 2015 at 8:48 AM, Carlotta Hicks <Ca...@sas.com>
> wrote:
>
> I added in an HDFS sink. But, I am getting this FlumeException: Expected
> one source and got 0 .
>
> The only source for the embedded agent is 'embedded', correct? Not sure
> how to proceed from here.
>
> Need help.
>
> -----Original Message-----
> From: Ashish [mailto:paliwalashish@gmail.com <pa...@gmail.com>]
> Sent: Saturday, January 17, 2015 7:58 AM
> To: user@flume.apache.org
> Subject: Re: Java API for Flume Agent
>
> Try a bit of hacking in
> org.apache.flume.agent.embedded.EmbeddedAgentConfiguration class, and add
> HBase Sink to allowed sinks in the section
>
> private static final String[] ALLOWED_SINKS = {
> SINK_TYPE_AVRO,
> // Add Hbase sink here <<<
>
> };
>
> See if it works and let us know.
>
> HTH!
>
> On Sat, Jan 17, 2015 at 10:19 AM, Carlotta Hicks <Ca...@sas.com>
> wrote:
> > Thanks Joey! I would love to use the embedded agent. It is exactly what
> I am looking for. However, I can't use the embedded agent because with the
> embedded agent, the sink must be avro. I need the sink to be hbase.
> > I am assuming that requirements has not changed. Please correct me if I
> am wrong.
> > ________________________________________
> > From: Joey Echeverria <jo...@cloudera.com>
> > Sent: Friday, January 16, 2015 3:47 PM
> > To: user@flume.apache.org
> > Subject: Re: Java API for Flume Agent
> >
> > You could use an embedded flume agent[1]. This will do the same thing
> > that the Flume flow does but run flume inside of another Java
> > application.
> >
> > Let me know if that meets your needs.
> >
> > -Joey
> >
> > [1] http://flume.apache.org/FlumeDeveloperGuide.html#embedded-agent
> >
> >
> > On Fri, Jan 16, 2015 at 10:30 AM, Carlotta Hicks <Ca...@sas.com>
> wrote:
> >> I need to create an agent using java API to load an HBase table using
> >> a CSV file. I am able to accomplish this using flume-ng and an agent
> >> with a spooldir directory source, file channel and HBase sink.
> >>
> >>
> >>
> >> I need information on how to accomplish this via a java API.
> >>
> >>
> >>
> >> Any help would be appreciated.
> >>
> >>
> >>
> >> -CM
> >>
> >>
> >
> >
> >
> > --
> > Joey Echeverria
>
>
>
> --
> thanks
> ashish
>
> Blog: http://www.ashishpaliwal.com/blog
> My Photo Galleries: http://www.pbase.com/ashishpaliwal
>
>
>
RE: Java API for Flume Agent
Posted by Carlotta Hicks <Ca...@sas.com>.
Here is the code change in EmbeddedAgentConfiguration:
/**
* Avro sink which can send events to a downstream avro source. This is the
* only supported sink for Embedded Agents.
*/
public static final String SINK_TYPE_AVRO = SinkType.AVRO.name();
public static final String SINK_TYPE_HBASE = SinkType.HBASE.name();
public static final String SINK_TYPE_HDFS = SinkType.HDFS.name();
private static final String[] ALLOWED_SINKS = {
SINK_TYPE_AVRO,
SINK_TYPE_HBASE,
SINK_TYPE_HDFS
};
This is my code with configurations:
Map<String, String> properties = new HashMap<String, String>();
properties.put("channel.type", "file");
properties.put("sinks", "hdfs1");
properties.put("hdfs1.type", "hdfs");
properties.put("hdfs1.path", "hdfs://lax07.unx.sas.com:8020/tmp");
properties.put("processor.type", "default");
agent = new EmbeddedAgent(“myAgent”);
agent.configure(properties); <-- This statement causes the exception below
org.apache.flume.FlumeException: Expected one source and got 0
at org.apache.flume.agent.embedded.EmbeddedAgent.doConfigure(EmbeddedAgent.java:160)
at org.apache.flume.agent.embedded.EmbeddedAgent.configure(EmbeddedAgent.java:95)
From: Hari Shreedharan [mailto:hshreedharan@cloudera.com]
Sent: Tuesday, January 20, 2015 3:23 PM
To: user@flume.apache.org
Cc: user@flume.apache.org
Subject: RE: Java API for Flume Agent
Can you send the code you used?
Thanks,
Hari
On Tue, Jan 20, 2015 at 8:48 AM, Carlotta Hicks <Ca...@sas.com>> wrote:
I added in an HDFS sink. But, I am getting this FlumeException: Expected one source and got 0 .
The only source for the embedded agent is 'embedded', correct? Not sure how to proceed from here.
Need help.
-----Original Message-----
From: Ashish [mailto:paliwalashish@gmail.com]
Sent: Saturday, January 17, 2015 7:58 AM
To: user@flume.apache.org<ma...@flume.apache.org>
Subject: Re: Java API for Flume Agent
Try a bit of hacking in
org.apache.flume.agent.embedded.EmbeddedAgentConfiguration class, and add HBase Sink to allowed sinks in the section
private static final String[] ALLOWED_SINKS = {
SINK_TYPE_AVRO,
// Add Hbase sink here <<<
};
See if it works and let us know.
HTH!
On Sat, Jan 17, 2015 at 10:19 AM, Carlotta Hicks <Ca...@sas.com>> wrote:
> Thanks Joey! I would love to use the embedded agent. It is exactly what I am looking for. However, I can't use the embedded agent because with the embedded agent, the sink must be avro. I need the sink to be hbase.
> I am assuming that requirements has not changed. Please correct me if I am wrong.
> ________________________________________
> From: Joey Echeverria <jo...@cloudera.com>>
> Sent: Friday, January 16, 2015 3:47 PM
> To: user@flume.apache.org<ma...@flume.apache.org>
> Subject: Re: Java API for Flume Agent
>
> You could use an embedded flume agent[1]. This will do the same thing
> that the Flume flow does but run flume inside of another Java
> application.
>
> Let me know if that meets your needs.
>
> -Joey
>
> [1] http://flume.apache.org/FlumeDeveloperGuide.html#embedded-agent
>
>
> On Fri, Jan 16, 2015 at 10:30 AM, Carlotta Hicks <Ca...@sas.com>> wrote:
>> I need to create an agent using java API to load an HBase table using
>> a CSV file. I am able to accomplish this using flume-ng and an agent
>> with a spooldir directory source, file channel and HBase sink.
>>
>>
>>
>> I need information on how to accomplish this via a java API.
>>
>>
>>
>> Any help would be appreciated.
>>
>>
>>
>> -CM
>>
>>
>
>
>
> --
> Joey Echeverria
--
thanks
ashish
Blog: http://www.ashishpaliwal.com/blog
My Photo Galleries: http://www.pbase.com/ashishpaliwal
RE: Java API for Flume Agent
Posted by Hari Shreedharan <hs...@cloudera.com>.
Can you send the code you used?
Thanks, Hari
On Tue, Jan 20, 2015 at 8:48 AM, Carlotta Hicks <Ca...@sas.com>
wrote:
> I added in an HDFS sink. But, I am getting this FlumeException: Expected one source and got 0 .
> The only source for the embedded agent is 'embedded', correct? Not sure how to proceed from here.
> Need help.
> -----Original Message-----
> From: Ashish [mailto:paliwalashish@gmail.com]
> Sent: Saturday, January 17, 2015 7:58 AM
> To: user@flume.apache.org
> Subject: Re: Java API for Flume Agent
> Try a bit of hacking in
> org.apache.flume.agent.embedded.EmbeddedAgentConfiguration class, and add HBase Sink to allowed sinks in the section
> private static final String[] ALLOWED_SINKS = {
> SINK_TYPE_AVRO,
> // Add Hbase sink here <<<
> };
> See if it works and let us know.
> HTH!
> On Sat, Jan 17, 2015 at 10:19 AM, Carlotta Hicks <Ca...@sas.com> wrote:
>> Thanks Joey! I would love to use the embedded agent. It is exactly what I am looking for. However, I can't use the embedded agent because with the embedded agent, the sink must be avro. I need the sink to be hbase.
>> I am assuming that requirements has not changed. Please correct me if I am wrong.
>> ________________________________________
>> From: Joey Echeverria <jo...@cloudera.com>
>> Sent: Friday, January 16, 2015 3:47 PM
>> To: user@flume.apache.org
>> Subject: Re: Java API for Flume Agent
>>
>> You could use an embedded flume agent[1]. This will do the same thing
>> that the Flume flow does but run flume inside of another Java
>> application.
>>
>> Let me know if that meets your needs.
>>
>> -Joey
>>
>> [1] http://flume.apache.org/FlumeDeveloperGuide.html#embedded-agent
>>
>>
>> On Fri, Jan 16, 2015 at 10:30 AM, Carlotta Hicks <Ca...@sas.com> wrote:
>>> I need to create an agent using java API to load an HBase table using
>>> a CSV file. I am able to accomplish this using flume-ng and an agent
>>> with a spooldir directory source, file channel and HBase sink.
>>>
>>>
>>>
>>> I need information on how to accomplish this via a java API.
>>>
>>>
>>>
>>> Any help would be appreciated.
>>>
>>>
>>>
>>> -CM
>>>
>>>
>>
>>
>>
>> --
>> Joey Echeverria
> --
> thanks
> ashish
> Blog: http://www.ashishpaliwal.com/blog
> My Photo Galleries: http://www.pbase.com/ashishpaliwal
Re: Java API for Flume Agent
Posted by iain wright <ia...@gmail.com>.
Also need to see the config if possible.
RE: Embedded agent not supporting hbase sink, you could avro sink from
embeedded flume to an avro source on a sperate flume "tier", which
aggregates events from your servers and then writes to hbase using hbase
sink
--
Iain Wright
This email message is confidential, intended only for the recipient(s)
named above and may contain information that is privileged, exempt from
disclosure under applicable law. If you are not the intended recipient, do
not disclose or disseminate the message to anyone except the intended
recipient. If you have received this message in error, or are not the named
recipient(s), please immediately notify the sender by return email, and
delete all copies of this message.
On Tue, Jan 20, 2015 at 8:47 AM, Carlotta Hicks <Ca...@sas.com>
wrote:
> I added in an HDFS sink. But, I am getting this FlumeException: Expected
> one source and got 0 .
>
> The only source for the embedded agent is 'embedded', correct? Not sure
> how to proceed from here.
>
> Need help.
>
> -----Original Message-----
> From: Ashish [mailto:paliwalashish@gmail.com]
> Sent: Saturday, January 17, 2015 7:58 AM
> To: user@flume.apache.org
> Subject: Re: Java API for Flume Agent
>
> Try a bit of hacking in
> org.apache.flume.agent.embedded.EmbeddedAgentConfiguration class, and add
> HBase Sink to allowed sinks in the section
>
> private static final String[] ALLOWED_SINKS = {
> SINK_TYPE_AVRO,
> // Add Hbase sink here <<<
>
> };
>
> See if it works and let us know.
>
> HTH!
>
> On Sat, Jan 17, 2015 at 10:19 AM, Carlotta Hicks <Ca...@sas.com>
> wrote:
> > Thanks Joey! I would love to use the embedded agent. It is exactly
> what I am looking for. However, I can't use the embedded agent because
> with the embedded agent, the sink must be avro. I need the sink to be
> hbase.
> > I am assuming that requirements has not changed. Please correct me if I
> am wrong.
> > ________________________________________
> > From: Joey Echeverria <jo...@cloudera.com>
> > Sent: Friday, January 16, 2015 3:47 PM
> > To: user@flume.apache.org
> > Subject: Re: Java API for Flume Agent
> >
> > You could use an embedded flume agent[1]. This will do the same thing
> > that the Flume flow does but run flume inside of another Java
> > application.
> >
> > Let me know if that meets your needs.
> >
> > -Joey
> >
> > [1] http://flume.apache.org/FlumeDeveloperGuide.html#embedded-agent
> >
> >
> > On Fri, Jan 16, 2015 at 10:30 AM, Carlotta Hicks <Ca...@sas.com>
> wrote:
> >> I need to create an agent using java API to load an HBase table using
> >> a CSV file. I am able to accomplish this using flume-ng and an agent
> >> with a spooldir directory source, file channel and HBase sink.
> >>
> >>
> >>
> >> I need information on how to accomplish this via a java API.
> >>
> >>
> >>
> >> Any help would be appreciated.
> >>
> >>
> >>
> >> -CM
> >>
> >>
> >
> >
> >
> > --
> > Joey Echeverria
>
>
>
> --
> thanks
> ashish
>
> Blog: http://www.ashishpaliwal.com/blog
> My Photo Galleries: http://www.pbase.com/ashishpaliwal
>
RE: Java API for Flume Agent
Posted by Carlotta Hicks <Ca...@sas.com>.
I added in an HDFS sink. But, I am getting this FlumeException: Expected one source and got 0 .
The only source for the embedded agent is 'embedded', correct? Not sure how to proceed from here.
Need help.
-----Original Message-----
From: Ashish [mailto:paliwalashish@gmail.com]
Sent: Saturday, January 17, 2015 7:58 AM
To: user@flume.apache.org
Subject: Re: Java API for Flume Agent
Try a bit of hacking in
org.apache.flume.agent.embedded.EmbeddedAgentConfiguration class, and add HBase Sink to allowed sinks in the section
private static final String[] ALLOWED_SINKS = {
SINK_TYPE_AVRO,
// Add Hbase sink here <<<
};
See if it works and let us know.
HTH!
On Sat, Jan 17, 2015 at 10:19 AM, Carlotta Hicks <Ca...@sas.com> wrote:
> Thanks Joey! I would love to use the embedded agent. It is exactly what I am looking for. However, I can't use the embedded agent because with the embedded agent, the sink must be avro. I need the sink to be hbase.
> I am assuming that requirements has not changed. Please correct me if I am wrong.
> ________________________________________
> From: Joey Echeverria <jo...@cloudera.com>
> Sent: Friday, January 16, 2015 3:47 PM
> To: user@flume.apache.org
> Subject: Re: Java API for Flume Agent
>
> You could use an embedded flume agent[1]. This will do the same thing
> that the Flume flow does but run flume inside of another Java
> application.
>
> Let me know if that meets your needs.
>
> -Joey
>
> [1] http://flume.apache.org/FlumeDeveloperGuide.html#embedded-agent
>
>
> On Fri, Jan 16, 2015 at 10:30 AM, Carlotta Hicks <Ca...@sas.com> wrote:
>> I need to create an agent using java API to load an HBase table using
>> a CSV file. I am able to accomplish this using flume-ng and an agent
>> with a spooldir directory source, file channel and HBase sink.
>>
>>
>>
>> I need information on how to accomplish this via a java API.
>>
>>
>>
>> Any help would be appreciated.
>>
>>
>>
>> -CM
>>
>>
>
>
>
> --
> Joey Echeverria
--
thanks
ashish
Blog: http://www.ashishpaliwal.com/blog
My Photo Galleries: http://www.pbase.com/ashishpaliwal
Re: Java API for Flume Agent
Posted by Ashish <pa...@gmail.com>.
Try a bit of hacking in
org.apache.flume.agent.embedded.EmbeddedAgentConfiguration class, and
add HBase Sink to allowed sinks in the section
private static final String[] ALLOWED_SINKS = {
SINK_TYPE_AVRO,
// Add Hbase sink here <<<
};
See if it works and let us know.
HTH!
On Sat, Jan 17, 2015 at 10:19 AM, Carlotta Hicks <Ca...@sas.com> wrote:
> Thanks Joey! I would love to use the embedded agent. It is exactly what I am looking for. However, I can't use the embedded agent because with the embedded agent, the sink must be avro. I need the sink to be hbase.
> I am assuming that requirements has not changed. Please correct me if I am wrong.
> ________________________________________
> From: Joey Echeverria <jo...@cloudera.com>
> Sent: Friday, January 16, 2015 3:47 PM
> To: user@flume.apache.org
> Subject: Re: Java API for Flume Agent
>
> You could use an embedded flume agent[1]. This will do the same thing
> that the Flume flow does but run flume inside of another Java
> application.
>
> Let me know if that meets your needs.
>
> -Joey
>
> [1] http://flume.apache.org/FlumeDeveloperGuide.html#embedded-agent
>
>
> On Fri, Jan 16, 2015 at 10:30 AM, Carlotta Hicks <Ca...@sas.com> wrote:
>> I need to create an agent using java API to load an HBase table using a CSV
>> file. I am able to accomplish this using flume-ng and an agent with a
>> spooldir directory source, file channel and HBase sink.
>>
>>
>>
>> I need information on how to accomplish this via a java API.
>>
>>
>>
>> Any help would be appreciated.
>>
>>
>>
>> -CM
>>
>>
>
>
>
> --
> Joey Echeverria
--
thanks
ashish
Blog: http://www.ashishpaliwal.com/blog
My Photo Galleries: http://www.pbase.com/ashishpaliwal
RE: Java API for Flume Agent
Posted by Carlotta Hicks <Ca...@sas.com>.
Thanks Joey! I would love to use the embedded agent. It is exactly what I am looking for. However, I can't use the embedded agent because with the embedded agent, the sink must be avro. I need the sink to be hbase.
I am assuming that requirements has not changed. Please correct me if I am wrong.
________________________________________
From: Joey Echeverria <jo...@cloudera.com>
Sent: Friday, January 16, 2015 3:47 PM
To: user@flume.apache.org
Subject: Re: Java API for Flume Agent
You could use an embedded flume agent[1]. This will do the same thing
that the Flume flow does but run flume inside of another Java
application.
Let me know if that meets your needs.
-Joey
[1] http://flume.apache.org/FlumeDeveloperGuide.html#embedded-agent
On Fri, Jan 16, 2015 at 10:30 AM, Carlotta Hicks <Ca...@sas.com> wrote:
> I need to create an agent using java API to load an HBase table using a CSV
> file. I am able to accomplish this using flume-ng and an agent with a
> spooldir directory source, file channel and HBase sink.
>
>
>
> I need information on how to accomplish this via a java API.
>
>
>
> Any help would be appreciated.
>
>
>
> -CM
>
>
--
Joey Echeverria
Re: Java API for Flume Agent
Posted by Joey Echeverria <jo...@cloudera.com>.
You could use an embedded flume agent[1]. This will do the same thing
that the Flume flow does but run flume inside of another Java
application.
Let me know if that meets your needs.
-Joey
[1] http://flume.apache.org/FlumeDeveloperGuide.html#embedded-agent
On Fri, Jan 16, 2015 at 10:30 AM, Carlotta Hicks <Ca...@sas.com> wrote:
> I need to create an agent using java API to load an HBase table using a CSV
> file. I am able to accomplish this using flume-ng and an agent with a
> spooldir directory source, file channel and HBase sink.
>
>
>
> I need information on how to accomplish this via a java API.
>
>
>
> Any help would be appreciated.
>
>
>
> -CM
>
>
--
Joey Echeverria