You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flume.apache.org by Jeff Hansen <ds...@gmail.com> on 2011/09/09 23:10:51 UTC
collectorSink is always using default from flume-conf.xml
When I configure a collectorSink to use a format different from the
default specified in the flume-conf.xml file, it continues to use the
default. I get the following error
2011-09-09 16:02:29,059 [Roll-TriggerThread-0] WARN conf.FlumeBuilder:
Deprecated syntax: Expected a format spec but instead had a (String)
raw
even though I'm using the following combination of configurations. Is
the xml file configuration supposed to supercede that of the runtime
configuration?
I'm running version "Flume 0.9.4-cdh3u1"
Here's the complete startup script I'm using:
#!/bin/sh
gnome-terminal -e "flume master"
sleep 10
flume shell -c localhost -e "exec config agent
'tail(\"/var/log/apache2/access.log\")' '[console,
collectorSink(\"hdfs://localhost/flume/avro/\",\"log\",60000,avrojson)]'"
gnome-terminal -e "flume node -n agent"
Here's the appropriate entry from my flume-conf.xml file:
<property>
<name>flume.collector.output.format</name>
<value>raw</value>
<description>The output format for the data written by a Flume
collector node. There are several formats available:
syslog - outputs events in a syslog-like format
log4j - outputs events in a pattern similar to Hadoop's log4j pattern
raw - Event body only. This is most similar to copying a file but
does not preserve any uniqifying metadata like host/timestamp/nanos.
avro - Avro Native file format. Default currently is uncompressed.
avrojson - this outputs data as json encoded by avro
avrodata - this outputs data as a avro binary encoded data
debug - used only for debugging
</description>
</property>
Re: collectorSink is always using default from flume-conf.xml
Posted by Stephen Layland <st...@gmail.com>.
FWIW, I get that Deprecated syntax error all the time when running flume
normally. It seems to occur whenever a sink is opened with a given format.
My guess is that is related to an internal API change and is probably
harmless as the String args should still work (as long as the format spec is
one that Flume knows about like "avro", "avrojson", "raw", "avrodata",
etc...
-Steve
On Tue, Sep 27, 2011 at 8:02 PM, Bao Thai Ngo <ba...@gmail.com> wrote:
> Mingjie,
>
> Thanks for the reply.
>
> I did override build() instead of create() but flume gave out the error as
> I said in previous email (WARN com.cloudera.flume.conf.**FlumeBuilder:
> Deprecated syntax: Expected a format spec but instead had a (String) raw)
>
> I then overrided create() according to the information Jonathan Hsieh
> indicated at
> https://groups.google.com/a/cloudera.org/group/flume-user/browse_thread/thread/19b3c9ebb21e4b06
>
> any suggestions?
>
> ~Thai
>
> On Wed, Sep 28, 2011 at 2:00 AM, Mingjie Lai <mj...@gmail.com> wrote:
>
>> Thai.
>>
>> I checked out the 0.9.4-branch from git://github.com/cloudera/**flume.git<http://github.com/cloudera/flume.git>,
>> but didn't see the problem.
>>
>> In the source code, you should override build() instead of create() here.
>>
>> -mingjie
>>
>>
>> On 09/27/2011 03:59 AM, Bao Thai Ngo wrote:
>>
>>> any thought?
>>>
>>> On Wed, Sep 21, 2011 at 4:19 PM, Bao Thai Ngo <baothaingo@gmail.com
>>> <ma...@gmail.com>> wrote:
>>>
>>> I'd also like to keep you informed that when compiling FLume
>>> Helloworld (modified according to some changes in version 0.9.4):
>>> ------------------------------**----
>>> public static SinkDecoBuilder builder() {
>>> return new SinkDecoBuilder() {
>>> // construct a new parameterized decorator
>>> @Override
>>> public EventSinkDecorator<EventSink> create(Context
>>> context,Object... argv) {
>>> Preconditions.checkArgument(**argv.length == 0,"usage:
>>> helloWorldDecorator");
>>>
>>> return new HelloWorldDecorator<EventSink>**(null);
>>> }
>>>
>>> };
>>> }
>>>
>>> -----------------------
>>>
>>> I get the following error:
>>>
>>> [javac]
>>> /usr/lib/flume/plugins/**helloworld/src/**
>>> HelloWorldDecorator.java:34:
>>> <anonymous helloworld.**HelloWorldDecorator$1> is not abstract and
>>> does not override abstract method
>>> build(com.cloudera.flume.conf.**Context,java.lang.String...) in
>>> com.cloudera.flume.conf.**SinkFactory.SinkDecoBuilder
>>> [javac] return new SinkDecoBuilder() {
>>>
>>> Was I missing something?
>>>
>>> ~Thai
>>>
>>>
>>> On Wed, Sep 21, 2011 at 3:41 PM, Bao Thai Ngo <baothaingo@gmail.com
>>> <ma...@gmail.com>> wrote:
>>>
>>> Hello list,
>>>
>>> After updating to Flume v0.9.4-cdh3u1, I do meet this error:
>>> WARN com.cloudera.flume.conf.**FlumeBuilder: Deprecated syntax:
>>> Expected a format spec but instead had a (String) raw
>>> As a result, collector does not save events on storage backend
>>>
>>> Before the updating, my plug-ins worked completely well. Below
>>> is my typical config:
>>> exec config node1 test
>>> 'tail("/path/to/test.txt",**startFromEnd="true")'
>>> '{deco_modiy_event_body()=>**autoDFOChain}'
>>>
>>> Any thoughts?
>>>
>>> Thanks,
>>> ~Thai
>>>
>>> On Sat, Sep 10, 2011 at 4:14 AM, Jeff Hansen <dscheffy@gmail.com
>>> <ma...@gmail.com>> wrote:
>>>
>>> Sorry, I should have also mentioned that the output itself
>>> is coming
>>> out in the raw format (consistent with the xml config rather
>>> than the
>>> runtime config)
>>>
>>> On Fri, Sep 9, 2011 at 4:10 PM, Jeff Hansen
>>> <dscheffy@gmail.com <ma...@gmail.com>> wrote:
>>> > When I configure a collectorSink to use a format
>>> different from the
>>> > default specified in the flume-conf.xml file, it
>>> continues to use the
>>> > default. I get the following error
>>> >
>>> > 2011-09-09 16:02:29,059 [Roll-TriggerThread-0] WARN
>>> conf.FlumeBuilder:
>>> > Deprecated syntax: Expected a format spec but instead had
>>> a (String)
>>> > raw
>>> >
>>> > even though I'm using the following combination of
>>> configurations. Is
>>> > the xml file configuration supposed to supercede that of
>>> the runtime
>>> > configuration?
>>> >
>>> > I'm running version "Flume 0.9.4-cdh3u1"
>>> >
>>> > Here's the complete startup script I'm using:
>>> >
>>> > #!/bin/sh
>>> > gnome-terminal -e "flume master"
>>> > sleep 10
>>> > flume shell -c localhost -e "exec config agent
>>> > 'tail(\"/var/log/apache2/**access.log\")' '[console,
>>> >
>>> collectorSink(\"hdfs://**localhost/flume/avro/\",\"log\**
>>> ",60000,avrojson)]'"
>>> > gnome-terminal -e "flume node -n agent"
>>> >
>>> > Here's the appropriate entry from my flume-conf.xml file:
>>> > <property>
>>> > <name>flume.collector.output.**format</name>
>>> > <value>raw</value>
>>> > <description>The output format for the data written by a
>>> Flume
>>> > collector node. There are several formats available:
>>> > syslog - outputs events in a syslog-like format
>>> > log4j - outputs events in a pattern similar to
>>> Hadoop's log4j pattern
>>> > raw - Event body only. This is most similar to
>>> copying a file but
>>> > does not preserve any uniqifying metadata like
>>> host/timestamp/nanos.
>>> > avro - Avro Native file format. Default currently
>>> is uncompressed.
>>> > avrojson - this outputs data as json encoded by avro
>>> > avrodata - this outputs data as a avro binary
>>> encoded data
>>> > debug - used only for debugging
>>> > </description>
>>> > </property>
>>> >
>>>
>>>
>>>
>>>
>>>
>
Re: collectorSink is always using default from flume-conf.xml
Posted by Mingjie Lai <mj...@gmail.com>.
Thai.
I got your point when looking at the source code.
index 1843248..582ae82 100644
--- a/flume-core/src/main/java/com/cloudera/flume/conf/SourceFactory.java
+++ b/flume-core/src/main/java/com/cloudera/flume/conf/SourceFactory.java
...
@@ -33,10 +35,20 @@ abstract public class SourceFactory {
return build(new Context(), argv);
}
+ @Deprecated
public abstract EventSource build(Context ctx, String... argv);
+
+ public EventSource create(Context ctx, Object... argv) {
+ return build(ctx, toStrings(argv));
+ }
};
It's about deprecating an abstract method. I'm not quite sure the best
practice to deprecate an abstract method, maybe it should get concreted
(by an empty body) in flume?
Since the build() method is still abstract, right now (for 0.9,4) you
still have to override it in your client code, and ignore the deprecate
warning. Another reason is that I didn't see the create() method really
gets called in the source code.
-mingjie
On 09/27/2011 08:02 PM, Bao Thai Ngo wrote:
> Mingjie,
>
> Thanks for the reply.
>
> I did override build() instead of create() but flume gave out the error
> as I said in previous email (WARN
> com.cloudera.flume.conf.__FlumeBuilder: Deprecated syntax: Expected a
> format spec but instead had a (String) raw)
>
> I then overrided create() according to the information Jonathan Hsieh
> indicated at
> https://groups.google.com/a/cloudera.org/group/flume-user/browse_thread/thread/19b3c9ebb21e4b06
>
> any suggestions?
>
> ~Thai
>
> On Wed, Sep 28, 2011 at 2:00 AM, Mingjie Lai <mjlai09@gmail.com
> <ma...@gmail.com>> wrote:
>
> Thai.
>
> I checked out the 0.9.4-branch from
> git://github.com/cloudera/__flume.git
> <http://github.com/cloudera/flume.git>, but didn't see the problem.
>
> In the source code, you should override build() instead of create()
> here.
>
> -mingjie
>
>
> On 09/27/2011 03:59 AM, Bao Thai Ngo wrote:
>
> any thought?
>
> On Wed, Sep 21, 2011 at 4:19 PM, Bao Thai Ngo
> <baothaingo@gmail.com <ma...@gmail.com>
> <mailto:baothaingo@gmail.com <ma...@gmail.com>>> wrote:
>
> I'd also like to keep you informed that when compiling FLume
> Helloworld (modified according to some changes in version
> 0.9.4):
> ------------------------------__----
> public static SinkDecoBuilder builder() {
> return new SinkDecoBuilder() {
> // construct a new parameterized decorator
> @Override
> public EventSinkDecorator<EventSink> create(Context
> context,Object... argv) {
> Preconditions.checkArgument(__argv.length == 0,"usage:
> helloWorldDecorator");
>
> return new HelloWorldDecorator<EventSink>__(null);
> }
>
> };
> }
>
> -----------------------
>
> I get the following error:
>
> [javac]
>
> /usr/lib/flume/plugins/__helloworld/src/__HelloWorldDecorator.java:34:
> <anonymous helloworld.__HelloWorldDecorator$1> is not abstract and
> does not override abstract method
> build(com.cloudera.flume.conf.__Context,java.lang.String...) in
> com.cloudera.flume.conf.__SinkFactory.SinkDecoBuilder
> [javac] return new SinkDecoBuilder() {
>
> Was I missing something?
>
> ~Thai
>
>
> On Wed, Sep 21, 2011 at 3:41 PM, Bao Thai Ngo
> <baothaingo@gmail.com <ma...@gmail.com>
> <mailto:baothaingo@gmail.com <ma...@gmail.com>>> wrote:
>
> Hello list,
>
> After updating to Flume v0.9.4-cdh3u1, I do meet this error:
> WARN com.cloudera.flume.conf.__FlumeBuilder: Deprecated
> syntax:
> Expected a format spec but instead had a (String) raw
> As a result, collector does not save events on storage
> backend
>
> Before the updating, my plug-ins worked completely well.
> Below
> is my typical config:
> exec config node1 test
> 'tail("/path/to/test.txt",__startFromEnd="true")'
> '{deco_modiy_event_body()=>__autoDFOChain}'
>
> Any thoughts?
>
> Thanks,
> ~Thai
>
> On Sat, Sep 10, 2011 at 4:14 AM, Jeff Hansen
> <dscheffy@gmail.com <ma...@gmail.com>
> <mailto:dscheffy@gmail.com <ma...@gmail.com>>> wrote:
>
> Sorry, I should have also mentioned that the output
> itself
> is coming
> out in the raw format (consistent with the xml
> config rather
> than the
> runtime config)
>
> On Fri, Sep 9, 2011 at 4:10 PM, Jeff Hansen
> <dscheffy@gmail.com <ma...@gmail.com>
> <mailto:dscheffy@gmail.com <ma...@gmail.com>>> wrote:
> > When I configure a collectorSink to use a format
> different from the
> > default specified in the flume-conf.xml file, it
> continues to use the
> > default. I get the following error
> >
> > 2011-09-09 16:02:29,059 [Roll-TriggerThread-0] WARN
> conf.FlumeBuilder:
> > Deprecated syntax: Expected a format spec but instead had
> a (String)
> > raw
> >
> > even though I'm using the following combination of
> configurations. Is
> > the xml file configuration supposed to supercede that of
> the runtime
> > configuration?
> >
> > I'm running version "Flume 0.9.4-cdh3u1"
> >
> > Here's the complete startup script I'm using:
> >
> > #!/bin/sh
> > gnome-terminal -e "flume master"
> > sleep 10
> > flume shell -c localhost -e "exec config agent
> > 'tail(\"/var/log/apache2/__access.log\")' '[console,
> >
>
> collectorSink(\"hdfs://__localhost/flume/avro/\",\"log\__",60000,avrojson)]'"
> > gnome-terminal -e "flume node -n agent"
> >
> > Here's the appropriate entry from my flume-conf.xml file:
> > <property>
> > <name>flume.collector.output.__format</name>
> > <value>raw</value>
> > <description>The output format for the data written by a
> Flume
> > collector node. There are several formats available:
> > syslog - outputs events in a syslog-like format
> > log4j - outputs events in a pattern similar to
> Hadoop's log4j pattern
> > raw - Event body only. This is most similar to
> copying a file but
> > does not preserve any uniqifying metadata like
> host/timestamp/nanos.
> > avro - Avro Native file format. Default currently
> is uncompressed.
> > avrojson - this outputs data as json encoded by avro
> > avrodata - this outputs data as a avro binary
> encoded data
> > debug - used only for debugging
> > </description>
> > </property>
> >
>
>
>
>
>
Re: collectorSink is always using default from flume-conf.xml
Posted by Bao Thai Ngo <ba...@gmail.com>.
Mingjie,
Thanks for the reply.
I did override build() instead of create() but flume gave out the error as I
said in previous email (WARN com.cloudera.flume.conf.**FlumeBuilder:
Deprecated syntax: Expected a format spec but instead had a (String) raw)
I then overrided create() according to the information Jonathan Hsieh
indicated at
https://groups.google.com/a/cloudera.org/group/flume-user/browse_thread/thread/19b3c9ebb21e4b06
any suggestions?
~Thai
On Wed, Sep 28, 2011 at 2:00 AM, Mingjie Lai <mj...@gmail.com> wrote:
> Thai.
>
> I checked out the 0.9.4-branch from git://github.com/cloudera/**flume.git<http://github.com/cloudera/flume.git>,
> but didn't see the problem.
>
> In the source code, you should override build() instead of create() here.
>
> -mingjie
>
>
> On 09/27/2011 03:59 AM, Bao Thai Ngo wrote:
>
>> any thought?
>>
>> On Wed, Sep 21, 2011 at 4:19 PM, Bao Thai Ngo <baothaingo@gmail.com
>> <ma...@gmail.com>> wrote:
>>
>> I'd also like to keep you informed that when compiling FLume
>> Helloworld (modified according to some changes in version 0.9.4):
>> ------------------------------**----
>> public static SinkDecoBuilder builder() {
>> return new SinkDecoBuilder() {
>> // construct a new parameterized decorator
>> @Override
>> public EventSinkDecorator<EventSink> create(Context
>> context,Object... argv) {
>> Preconditions.checkArgument(**argv.length == 0,"usage:
>> helloWorldDecorator");
>>
>> return new HelloWorldDecorator<EventSink>**(null);
>> }
>>
>> };
>> }
>>
>> -----------------------
>>
>> I get the following error:
>>
>> [javac]
>> /usr/lib/flume/plugins/**helloworld/src/**HelloWorldDecorator.java:34:
>> <anonymous helloworld.**HelloWorldDecorator$1> is not abstract and
>> does not override abstract method
>> build(com.cloudera.flume.conf.**Context,java.lang.String...) in
>> com.cloudera.flume.conf.**SinkFactory.SinkDecoBuilder
>> [javac] return new SinkDecoBuilder() {
>>
>> Was I missing something?
>>
>> ~Thai
>>
>>
>> On Wed, Sep 21, 2011 at 3:41 PM, Bao Thai Ngo <baothaingo@gmail.com
>> <ma...@gmail.com>> wrote:
>>
>> Hello list,
>>
>> After updating to Flume v0.9.4-cdh3u1, I do meet this error:
>> WARN com.cloudera.flume.conf.**FlumeBuilder: Deprecated syntax:
>> Expected a format spec but instead had a (String) raw
>> As a result, collector does not save events on storage backend
>>
>> Before the updating, my plug-ins worked completely well. Below
>> is my typical config:
>> exec config node1 test
>> 'tail("/path/to/test.txt",**startFromEnd="true")'
>> '{deco_modiy_event_body()=>**autoDFOChain}'
>>
>> Any thoughts?
>>
>> Thanks,
>> ~Thai
>>
>> On Sat, Sep 10, 2011 at 4:14 AM, Jeff Hansen <dscheffy@gmail.com
>> <ma...@gmail.com>> wrote:
>>
>> Sorry, I should have also mentioned that the output itself
>> is coming
>> out in the raw format (consistent with the xml config rather
>> than the
>> runtime config)
>>
>> On Fri, Sep 9, 2011 at 4:10 PM, Jeff Hansen
>> <dscheffy@gmail.com <ma...@gmail.com>> wrote:
>> > When I configure a collectorSink to use a format
>> different from the
>> > default specified in the flume-conf.xml file, it
>> continues to use the
>> > default. I get the following error
>> >
>> > 2011-09-09 16:02:29,059 [Roll-TriggerThread-0] WARN
>> conf.FlumeBuilder:
>> > Deprecated syntax: Expected a format spec but instead had
>> a (String)
>> > raw
>> >
>> > even though I'm using the following combination of
>> configurations. Is
>> > the xml file configuration supposed to supercede that of
>> the runtime
>> > configuration?
>> >
>> > I'm running version "Flume 0.9.4-cdh3u1"
>> >
>> > Here's the complete startup script I'm using:
>> >
>> > #!/bin/sh
>> > gnome-terminal -e "flume master"
>> > sleep 10
>> > flume shell -c localhost -e "exec config agent
>> > 'tail(\"/var/log/apache2/**access.log\")' '[console,
>> >
>> collectorSink(\"hdfs://**localhost/flume/avro/\",\"log\**
>> ",60000,avrojson)]'"
>> > gnome-terminal -e "flume node -n agent"
>> >
>> > Here's the appropriate entry from my flume-conf.xml file:
>> > <property>
>> > <name>flume.collector.output.**format</name>
>> > <value>raw</value>
>> > <description>The output format for the data written by a
>> Flume
>> > collector node. There are several formats available:
>> > syslog - outputs events in a syslog-like format
>> > log4j - outputs events in a pattern similar to
>> Hadoop's log4j pattern
>> > raw - Event body only. This is most similar to
>> copying a file but
>> > does not preserve any uniqifying metadata like
>> host/timestamp/nanos.
>> > avro - Avro Native file format. Default currently
>> is uncompressed.
>> > avrojson - this outputs data as json encoded by avro
>> > avrodata - this outputs data as a avro binary
>> encoded data
>> > debug - used only for debugging
>> > </description>
>> > </property>
>> >
>>
>>
>>
>>
>>
Re: collectorSink is always using default from flume-conf.xml
Posted by Mingjie Lai <mj...@gmail.com>.
Thai.
I checked out the 0.9.4-branch from git://github.com/cloudera/flume.git,
but didn't see the problem.
In the source code, you should override build() instead of create() here.
-mingjie
On 09/27/2011 03:59 AM, Bao Thai Ngo wrote:
> any thought?
>
> On Wed, Sep 21, 2011 at 4:19 PM, Bao Thai Ngo <baothaingo@gmail.com
> <ma...@gmail.com>> wrote:
>
> I'd also like to keep you informed that when compiling FLume
> Helloworld (modified according to some changes in version 0.9.4):
> ----------------------------------
> public static SinkDecoBuilder builder() {
> return new SinkDecoBuilder() {
> // construct a new parameterized decorator
> @Override
> public EventSinkDecorator<EventSink> create(Context
> context,Object... argv) {
> Preconditions.checkArgument(argv.length == 0,"usage:
> helloWorldDecorator");
>
> return new HelloWorldDecorator<EventSink>(null);
> }
>
> };
> }
>
> -----------------------
>
> I get the following error:
>
> [javac]
> /usr/lib/flume/plugins/helloworld/src/HelloWorldDecorator.java:34:
> <anonymous helloworld.HelloWorldDecorator$1> is not abstract and
> does not override abstract method
> build(com.cloudera.flume.conf.Context,java.lang.String...) in
> com.cloudera.flume.conf.SinkFactory.SinkDecoBuilder
> [javac] return new SinkDecoBuilder() {
>
> Was I missing something?
>
> ~Thai
>
>
> On Wed, Sep 21, 2011 at 3:41 PM, Bao Thai Ngo <baothaingo@gmail.com
> <ma...@gmail.com>> wrote:
>
> Hello list,
>
> After updating to Flume v0.9.4-cdh3u1, I do meet this error:
> WARN com.cloudera.flume.conf.FlumeBuilder: Deprecated syntax:
> Expected a format spec but instead had a (String) raw
> As a result, collector does not save events on storage backend
>
> Before the updating, my plug-ins worked completely well. Below
> is my typical config:
> exec config node1 test
> 'tail("/path/to/test.txt",startFromEnd="true")'
> '{deco_modiy_event_body()=>autoDFOChain}'
>
> Any thoughts?
>
> Thanks,
> ~Thai
>
> On Sat, Sep 10, 2011 at 4:14 AM, Jeff Hansen <dscheffy@gmail.com
> <ma...@gmail.com>> wrote:
>
> Sorry, I should have also mentioned that the output itself
> is coming
> out in the raw format (consistent with the xml config rather
> than the
> runtime config)
>
> On Fri, Sep 9, 2011 at 4:10 PM, Jeff Hansen
> <dscheffy@gmail.com <ma...@gmail.com>> wrote:
> > When I configure a collectorSink to use a format
> different from the
> > default specified in the flume-conf.xml file, it
> continues to use the
> > default. I get the following error
> >
> > 2011-09-09 16:02:29,059 [Roll-TriggerThread-0] WARN
> conf.FlumeBuilder:
> > Deprecated syntax: Expected a format spec but instead had
> a (String)
> > raw
> >
> > even though I'm using the following combination of
> configurations. Is
> > the xml file configuration supposed to supercede that of
> the runtime
> > configuration?
> >
> > I'm running version "Flume 0.9.4-cdh3u1"
> >
> > Here's the complete startup script I'm using:
> >
> > #!/bin/sh
> > gnome-terminal -e "flume master"
> > sleep 10
> > flume shell -c localhost -e "exec config agent
> > 'tail(\"/var/log/apache2/access.log\")' '[console,
> >
> collectorSink(\"hdfs://localhost/flume/avro/\",\"log\",60000,avrojson)]'"
> > gnome-terminal -e "flume node -n agent"
> >
> > Here's the appropriate entry from my flume-conf.xml file:
> > <property>
> > <name>flume.collector.output.format</name>
> > <value>raw</value>
> > <description>The output format for the data written by a
> Flume
> > collector node. There are several formats available:
> > syslog - outputs events in a syslog-like format
> > log4j - outputs events in a pattern similar to
> Hadoop's log4j pattern
> > raw - Event body only. This is most similar to
> copying a file but
> > does not preserve any uniqifying metadata like
> host/timestamp/nanos.
> > avro - Avro Native file format. Default currently
> is uncompressed.
> > avrojson - this outputs data as json encoded by avro
> > avrodata - this outputs data as a avro binary
> encoded data
> > debug - used only for debugging
> > </description>
> > </property>
> >
>
>
>
>
Re: collectorSink is always using default from flume-conf.xml
Posted by Bao Thai Ngo <ba...@gmail.com>.
any thought?
On Wed, Sep 21, 2011 at 4:19 PM, Bao Thai Ngo <ba...@gmail.com> wrote:
> I'd also like to keep you informed that when compiling FLume Helloworld
> (modified according to some changes in version 0.9.4):
> ----------------------------------
> public static SinkDecoBuilder builder() {
> return new SinkDecoBuilder() {
> // construct a new parameterized decorator
> @Override
> public EventSinkDecorator<EventSink> create(Context context,Object...
> argv) {
> Preconditions.checkArgument(argv.length == 0,"usage:
> helloWorldDecorator");
>
> return new HelloWorldDecorator<EventSink>(null);
> }
>
> };
> }
>
> -----------------------
>
> I get the following error:
>
> [javac] /usr/lib/flume/plugins/helloworld/src/HelloWorldDecorator.java:34:
> <anonymous helloworld.HelloWorldDecorator$1> is not abstract and does not
> override abstract method
> build(com.cloudera.flume.conf.Context,java.lang.String...) in
> com.cloudera.flume.conf.SinkFactory.SinkDecoBuilder
> [javac] return new SinkDecoBuilder() {
>
> Was I missing something?
>
> ~Thai
>
>
> On Wed, Sep 21, 2011 at 3:41 PM, Bao Thai Ngo <ba...@gmail.com>wrote:
>
>> Hello list,
>>
>> After updating to Flume v0.9.4-cdh3u1, I do meet this error:
>> WARN com.cloudera.flume.conf.FlumeBuilder: Deprecated syntax: Expected a
>> format spec but instead had a (String) raw
>> As a result, collector does not save events on storage backend
>>
>> Before the updating, my plug-ins worked completely well. Below is my
>> typical config:
>> exec config node1 test 'tail("/path/to/test.txt",startFromEnd="true")'
>> '{deco_modiy_event_body()=>autoDFOChain}'
>>
>> Any thoughts?
>>
>> Thanks,
>> ~Thai
>>
>> On Sat, Sep 10, 2011 at 4:14 AM, Jeff Hansen <ds...@gmail.com> wrote:
>>
>>> Sorry, I should have also mentioned that the output itself is coming
>>> out in the raw format (consistent with the xml config rather than the
>>> runtime config)
>>>
>>> On Fri, Sep 9, 2011 at 4:10 PM, Jeff Hansen <ds...@gmail.com> wrote:
>>> > When I configure a collectorSink to use a format different from the
>>> > default specified in the flume-conf.xml file, it continues to use the
>>> > default. I get the following error
>>> >
>>> > 2011-09-09 16:02:29,059 [Roll-TriggerThread-0] WARN conf.FlumeBuilder:
>>> > Deprecated syntax: Expected a format spec but instead had a (String)
>>> > raw
>>> >
>>> > even though I'm using the following combination of configurations. Is
>>> > the xml file configuration supposed to supercede that of the runtime
>>> > configuration?
>>> >
>>> > I'm running version "Flume 0.9.4-cdh3u1"
>>> >
>>> > Here's the complete startup script I'm using:
>>> >
>>> > #!/bin/sh
>>> > gnome-terminal -e "flume master"
>>> > sleep 10
>>> > flume shell -c localhost -e "exec config agent
>>> > 'tail(\"/var/log/apache2/access.log\")' '[console,
>>> >
>>> collectorSink(\"hdfs://localhost/flume/avro/\",\"log\",60000,avrojson)]'"
>>> > gnome-terminal -e "flume node -n agent"
>>> >
>>> > Here's the appropriate entry from my flume-conf.xml file:
>>> > <property>
>>> > <name>flume.collector.output.format</name>
>>> > <value>raw</value>
>>> > <description>The output format for the data written by a Flume
>>> > collector node. There are several formats available:
>>> > syslog - outputs events in a syslog-like format
>>> > log4j - outputs events in a pattern similar to Hadoop's log4j
>>> pattern
>>> > raw - Event body only. This is most similar to copying a file but
>>> > does not preserve any uniqifying metadata like
>>> host/timestamp/nanos.
>>> > avro - Avro Native file format. Default currently is
>>> uncompressed.
>>> > avrojson - this outputs data as json encoded by avro
>>> > avrodata - this outputs data as a avro binary encoded data
>>> > debug - used only for debugging
>>> > </description>
>>> > </property>
>>> >
>>>
>>
>>
>
Re: collectorSink is always using default from flume-conf.xml
Posted by Bao Thai Ngo <ba...@gmail.com>.
I'd also like to keep you informed that when compiling FLume Helloworld
(modified according to some changes in version 0.9.4):
----------------------------------
public static SinkDecoBuilder builder() {
return new SinkDecoBuilder() {
// construct a new parameterized decorator
@Override
public EventSinkDecorator<EventSink> create(Context context,Object...
argv) {
Preconditions.checkArgument(argv.length == 0,"usage:
helloWorldDecorator");
return new HelloWorldDecorator<EventSink>(null);
}
};
}
-----------------------
I get the following error:
[javac] /usr/lib/flume/plugins/helloworld/src/HelloWorldDecorator.java:34:
<anonymous helloworld.HelloWorldDecorator$1> is not abstract and does not
override abstract method
build(com.cloudera.flume.conf.Context,java.lang.String...) in
com.cloudera.flume.conf.SinkFactory.SinkDecoBuilder
[javac] return new SinkDecoBuilder() {
Was I missing something?
~Thai
On Wed, Sep 21, 2011 at 3:41 PM, Bao Thai Ngo <ba...@gmail.com> wrote:
> Hello list,
>
> After updating to Flume v0.9.4-cdh3u1, I do meet this error:
> WARN com.cloudera.flume.conf.FlumeBuilder: Deprecated syntax: Expected a
> format spec but instead had a (String) raw
> As a result, collector does not save events on storage backend
>
> Before the updating, my plug-ins worked completely well. Below is my
> typical config:
> exec config node1 test 'tail("/path/to/test.txt",startFromEnd="true")'
> '{deco_modiy_event_body()=>autoDFOChain}'
>
> Any thoughts?
>
> Thanks,
> ~Thai
>
> On Sat, Sep 10, 2011 at 4:14 AM, Jeff Hansen <ds...@gmail.com> wrote:
>
>> Sorry, I should have also mentioned that the output itself is coming
>> out in the raw format (consistent with the xml config rather than the
>> runtime config)
>>
>> On Fri, Sep 9, 2011 at 4:10 PM, Jeff Hansen <ds...@gmail.com> wrote:
>> > When I configure a collectorSink to use a format different from the
>> > default specified in the flume-conf.xml file, it continues to use the
>> > default. I get the following error
>> >
>> > 2011-09-09 16:02:29,059 [Roll-TriggerThread-0] WARN conf.FlumeBuilder:
>> > Deprecated syntax: Expected a format spec but instead had a (String)
>> > raw
>> >
>> > even though I'm using the following combination of configurations. Is
>> > the xml file configuration supposed to supercede that of the runtime
>> > configuration?
>> >
>> > I'm running version "Flume 0.9.4-cdh3u1"
>> >
>> > Here's the complete startup script I'm using:
>> >
>> > #!/bin/sh
>> > gnome-terminal -e "flume master"
>> > sleep 10
>> > flume shell -c localhost -e "exec config agent
>> > 'tail(\"/var/log/apache2/access.log\")' '[console,
>> >
>> collectorSink(\"hdfs://localhost/flume/avro/\",\"log\",60000,avrojson)]'"
>> > gnome-terminal -e "flume node -n agent"
>> >
>> > Here's the appropriate entry from my flume-conf.xml file:
>> > <property>
>> > <name>flume.collector.output.format</name>
>> > <value>raw</value>
>> > <description>The output format for the data written by a Flume
>> > collector node. There are several formats available:
>> > syslog - outputs events in a syslog-like format
>> > log4j - outputs events in a pattern similar to Hadoop's log4j
>> pattern
>> > raw - Event body only. This is most similar to copying a file but
>> > does not preserve any uniqifying metadata like
>> host/timestamp/nanos.
>> > avro - Avro Native file format. Default currently is uncompressed.
>> > avrojson - this outputs data as json encoded by avro
>> > avrodata - this outputs data as a avro binary encoded data
>> > debug - used only for debugging
>> > </description>
>> > </property>
>> >
>>
>
>
Re: collectorSink is always using default from flume-conf.xml
Posted by Bao Thai Ngo <ba...@gmail.com>.
Hello list,
After updating to Flume v0.9.4-cdh3u1, I do meet this error:
WARN com.cloudera.flume.conf.FlumeBuilder: Deprecated syntax: Expected a
format spec but instead had a (String) raw
As a result, collector does not save events on storage backend
Before the updating, my plug-ins worked completely well. Below is my typical
config:
exec config node1 test 'tail("/path/to/test.txt",startFromEnd="true")'
'{deco_modiy_event_body()=>autoDFOChain}'
Any thoughts?
Thanks,
~Thai
On Sat, Sep 10, 2011 at 4:14 AM, Jeff Hansen <ds...@gmail.com> wrote:
> Sorry, I should have also mentioned that the output itself is coming
> out in the raw format (consistent with the xml config rather than the
> runtime config)
>
> On Fri, Sep 9, 2011 at 4:10 PM, Jeff Hansen <ds...@gmail.com> wrote:
> > When I configure a collectorSink to use a format different from the
> > default specified in the flume-conf.xml file, it continues to use the
> > default. I get the following error
> >
> > 2011-09-09 16:02:29,059 [Roll-TriggerThread-0] WARN conf.FlumeBuilder:
> > Deprecated syntax: Expected a format spec but instead had a (String)
> > raw
> >
> > even though I'm using the following combination of configurations. Is
> > the xml file configuration supposed to supercede that of the runtime
> > configuration?
> >
> > I'm running version "Flume 0.9.4-cdh3u1"
> >
> > Here's the complete startup script I'm using:
> >
> > #!/bin/sh
> > gnome-terminal -e "flume master"
> > sleep 10
> > flume shell -c localhost -e "exec config agent
> > 'tail(\"/var/log/apache2/access.log\")' '[console,
> > collectorSink(\"hdfs://localhost/flume/avro/\",\"log\",60000,avrojson)]'"
> > gnome-terminal -e "flume node -n agent"
> >
> > Here's the appropriate entry from my flume-conf.xml file:
> > <property>
> > <name>flume.collector.output.format</name>
> > <value>raw</value>
> > <description>The output format for the data written by a Flume
> > collector node. There are several formats available:
> > syslog - outputs events in a syslog-like format
> > log4j - outputs events in a pattern similar to Hadoop's log4j
> pattern
> > raw - Event body only. This is most similar to copying a file but
> > does not preserve any uniqifying metadata like
> host/timestamp/nanos.
> > avro - Avro Native file format. Default currently is uncompressed.
> > avrojson - this outputs data as json encoded by avro
> > avrodata - this outputs data as a avro binary encoded data
> > debug - used only for debugging
> > </description>
> > </property>
> >
>
Re: collectorSink is always using default from flume-conf.xml
Posted by Jeff Hansen <ds...@gmail.com>.
Sorry, I should have also mentioned that the output itself is coming
out in the raw format (consistent with the xml config rather than the
runtime config)
On Fri, Sep 9, 2011 at 4:10 PM, Jeff Hansen <ds...@gmail.com> wrote:
> When I configure a collectorSink to use a format different from the
> default specified in the flume-conf.xml file, it continues to use the
> default. I get the following error
>
> 2011-09-09 16:02:29,059 [Roll-TriggerThread-0] WARN conf.FlumeBuilder:
> Deprecated syntax: Expected a format spec but instead had a (String)
> raw
>
> even though I'm using the following combination of configurations. Is
> the xml file configuration supposed to supercede that of the runtime
> configuration?
>
> I'm running version "Flume 0.9.4-cdh3u1"
>
> Here's the complete startup script I'm using:
>
> #!/bin/sh
> gnome-terminal -e "flume master"
> sleep 10
> flume shell -c localhost -e "exec config agent
> 'tail(\"/var/log/apache2/access.log\")' '[console,
> collectorSink(\"hdfs://localhost/flume/avro/\",\"log\",60000,avrojson)]'"
> gnome-terminal -e "flume node -n agent"
>
> Here's the appropriate entry from my flume-conf.xml file:
> <property>
> <name>flume.collector.output.format</name>
> <value>raw</value>
> <description>The output format for the data written by a Flume
> collector node. There are several formats available:
> syslog - outputs events in a syslog-like format
> log4j - outputs events in a pattern similar to Hadoop's log4j pattern
> raw - Event body only. This is most similar to copying a file but
> does not preserve any uniqifying metadata like host/timestamp/nanos.
> avro - Avro Native file format. Default currently is uncompressed.
> avrojson - this outputs data as json encoded by avro
> avrodata - this outputs data as a avro binary encoded data
> debug - used only for debugging
> </description>
> </property>
>