You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flume.apache.org by "Wang, Yongkun | Yongkun | BDD" <yo...@mail.rakuten.com> on 2012/07/20 12:13:10 UTC
Tried NG 1.1.0 and trunk
Hi,
Currently we are using a modified version of flume-0.9.3 in production.
I tried release 1.1.0. The basic command is still bin/flume-ng node, not bin/flume-ng agent (by doc).
No document for HDFS sink, I need to read the source code to config it. HDFS sink of 1.1.0 seems be developed with hadoop 0.20.205.0. It can rotate files by count of events, but the last hadoop file cannot be closed.
Rotating the hadoop files by time interval does not work. It seems that problem is here:
A mistake by timing 1000 in BucketWriter:
lastProcessTime = System.currentTimeMillis() * 1000;
So the lines in 'if' will never get executed:
if ((rollInterval > 0)
&& (rollInterval < (System.currentTimeMillis() - lastProcessTime) / 1000))
doRotate = true;
But the code has been changed aggressively in later versions, so it is no longer a bug in the trunk.
I tested NG trunk (1.3 snapshot). The code is aggressively changed compared to 1.1.0. It seems that it is developed with hadoop 0.23.1 or 1.0.1. It can work with hadoop 0.20.205.0 (time-interval-based rotation works perfectly, last hadoop file closed correctly).
Will keep an eye on NG and wait for later releases.
Cheers,
Yongkun Wang (Kun)
Re: Tried NG 1.1.0 and trunk
Posted by "Wang, Yongkun | Yongkun | BDD" <yo...@mail.rakuten.com>.
Hi Alex,
I created a ticket in JIRA as suggested.
https://issues.apache.org/jira/browse/FLUME-1391
Cheers,
Kun
On 12/07/23 15:33, "alo alt" <wg...@gmail.com> wrote:
>Hi Yongkun Wang,
>
>>
>> My comment inline.
>>
>> On 12/07/20 19:38, "alo alt" <wg...@gmail.com> wrote:
>>
>>> Hi,
>>>
>>> On Jul 20, 2012, at 12:13 PM, Wang, Yongkun | Yongkun | BDD wrote:
>>>
>>> inline
>>>
>>>> Hi,
>>>>
>>>> Currently we are using a modified version of flume-0.9.3 in
>>>>production.
>>>>
>>>> I tried release 1.1.0. The basic command is still bin/flume-ng node,
>>>> not bin/flume-ng agent (by doc).
>>>
>>> Did you build Flume from git? In 1.1.x the notation was changed into
>>> agent, node is deprecated.
>>
>> In the 1.1.0 package it is still bin/flume-ng node. But it has been
>> changed in the trunk.
>>
>
>Yes, thats right.
>
>
>
>>>
>>>
>>>> No document for HDFS sink, I need to read the source code to config
>>>>it.
>>>> HDFS sink of 1.1.0 seems be developed with hadoop 0.20.205.0. It can
>>>> rotate files by count of events, but the last hadoop file cannot be
>>>> closed.
>>>
>>> http://archive.cloudera.com/cdh4/cdh/4/flume-ng/
>>
>> Very nice user guide in cdh4!
>> But it is not easy to find the user guide from flume apache home page. I
>> finally find this one:
>>
>>https://people.apache.org/~mpercy/flume/flume-1.2.0-incubating-SNAPSHOT/d
>>oc
>> s/FlumeUserGuide.html
>>
>
>much better!
>
>
>>>
>>>>
>>>> Rotating the hadoop files by time interval does not work. It seems
>>>>that
>>>> problem is here:
>>>> A mistake by timing 1000 in BucketWriter:
>>>>
>>>> lastProcessTime = System.currentTimeMillis() * 1000;
>>>>
>>>> So the lines in 'if' will never get executed:
>>>>
>>>> if ((rollInterval > 0)
>>>>
>>>> && (rollInterval < (System.currentTimeMillis() -
>>>> lastProcessTime) / 1000))
>>>>
>>>> doRotate = true;
>>>>
>>>> But the code has been changed aggressively in later versions, so it is
>>>> no longer a bug in the trunk.
>>>>
>>>> I tested NG trunk (1.3 snapshot). The code is aggressively changed
>>>> compared to 1.1.0. It seems that it is developed with hadoop 0.23.1 or
>>>> 1.0.1. It can work with hadoop 0.20.205.0 (time-interval-based
>>>>rotation
>>>> works perfectly, last hadoop file closed correctly).
>>>>
>>>
>>> Flume looks into the exported classpath and determine the installed
>>> hadoop version to use it.
>>
>> Yes. I tested flume with several hadoop versions. It seems that flume
>>may
>> not work well with hadoop 0.20.2. For e.g., writer.syncFs() is not
>> supported in 0.20.2 (writer.syncFs() is implemented by calling sync(),
>> maybe using writer.sync() in flume is ok?)
>>
>
>Good point, please file a jira about. I don't think that's a blocker for
>1.2.x, but could be one for the next release. Thanks.
>
>cheers,
> Alex
Re: Tried NG 1.1.0 and trunk
Posted by alo alt <wg...@gmail.com>.
Hi Yongkun Wang,
>
> My comment inline.
>
> On 12/07/20 19:38, "alo alt" <wg...@gmail.com> wrote:
>
>> Hi,
>>
>> On Jul 20, 2012, at 12:13 PM, Wang, Yongkun | Yongkun | BDD wrote:
>>
>> inline
>>
>>> Hi,
>>>
>>> Currently we are using a modified version of flume-0.9.3 in production.
>>>
>>> I tried release 1.1.0. The basic command is still bin/flume-ng node,
>>> not bin/flume-ng agent (by doc).
>>
>> Did you build Flume from git? In 1.1.x the notation was changed into
>> agent, node is deprecated.
>
> In the 1.1.0 package it is still bin/flume-ng node. But it has been
> changed in the trunk.
>
Yes, thats right.
>>
>>
>>> No document for HDFS sink, I need to read the source code to config it.
>>> HDFS sink of 1.1.0 seems be developed with hadoop 0.20.205.0. It can
>>> rotate files by count of events, but the last hadoop file cannot be
>>> closed.
>>
>> http://archive.cloudera.com/cdh4/cdh/4/flume-ng/
>
> Very nice user guide in cdh4!
> But it is not easy to find the user guide from flume apache home page. I
> finally find this one:
> https://people.apache.org/~mpercy/flume/flume-1.2.0-incubating-SNAPSHOT/doc
> s/FlumeUserGuide.html
>
much better!
>>
>>>
>>> Rotating the hadoop files by time interval does not work. It seems that
>>> problem is here:
>>> A mistake by timing 1000 in BucketWriter:
>>>
>>> lastProcessTime = System.currentTimeMillis() * 1000;
>>>
>>> So the lines in 'if' will never get executed:
>>>
>>> if ((rollInterval > 0)
>>>
>>> && (rollInterval < (System.currentTimeMillis() -
>>> lastProcessTime) / 1000))
>>>
>>> doRotate = true;
>>>
>>> But the code has been changed aggressively in later versions, so it is
>>> no longer a bug in the trunk.
>>>
>>> I tested NG trunk (1.3 snapshot). The code is aggressively changed
>>> compared to 1.1.0. It seems that it is developed with hadoop 0.23.1 or
>>> 1.0.1. It can work with hadoop 0.20.205.0 (time-interval-based rotation
>>> works perfectly, last hadoop file closed correctly).
>>>
>>
>> Flume looks into the exported classpath and determine the installed
>> hadoop version to use it.
>
> Yes. I tested flume with several hadoop versions. It seems that flume may
> not work well with hadoop 0.20.2. For e.g., writer.syncFs() is not
> supported in 0.20.2 (writer.syncFs() is implemented by calling sync(),
> maybe using writer.sync() in flume is ok?)
>
Good point, please file a jira about. I don't think that's a blocker for 1.2.x, but could be one for the next release. Thanks.
cheers,
Alex
Re: Tried NG 1.1.0 and trunk
Posted by "Wang, Yongkun | Yongkun | BDD" <yo...@mail.rakuten.com>.
Hi Alex,
Thanks for your reply.
My comment inline.
On 12/07/20 19:38, "alo alt" <wg...@gmail.com> wrote:
>Hi,
>
>On Jul 20, 2012, at 12:13 PM, Wang, Yongkun | Yongkun | BDD wrote:
>
>inline
>
>> Hi,
>>
>> Currently we are using a modified version of flume-0.9.3 in production.
>>
>> I tried release 1.1.0. The basic command is still bin/flume-ng node,
>>not bin/flume-ng agent (by doc).
>
>Did you build Flume from git? In 1.1.x the notation was changed into
>agent, node is deprecated.
In the 1.1.0 package it is still bin/flume-ng node. But it has been
changed in the trunk.
>
>
>> No document for HDFS sink, I need to read the source code to config it.
>>HDFS sink of 1.1.0 seems be developed with hadoop 0.20.205.0. It can
>>rotate files by count of events, but the last hadoop file cannot be
>>closed.
>
>http://archive.cloudera.com/cdh4/cdh/4/flume-ng/
Very nice user guide in cdh4!
But it is not easy to find the user guide from flume apache home page. I
finally find this one:
https://people.apache.org/~mpercy/flume/flume-1.2.0-incubating-SNAPSHOT/doc
s/FlumeUserGuide.html
>
>>
>> Rotating the hadoop files by time interval does not work. It seems that
>>problem is here:
>> A mistake by timing 1000 in BucketWriter:
>>
>> lastProcessTime = System.currentTimeMillis() * 1000;
>>
>> So the lines in 'if' will never get executed:
>>
>> if ((rollInterval > 0)
>>
>> && (rollInterval < (System.currentTimeMillis() -
>>lastProcessTime) / 1000))
>>
>> doRotate = true;
>>
>> But the code has been changed aggressively in later versions, so it is
>>no longer a bug in the trunk.
>>
>> I tested NG trunk (1.3 snapshot). The code is aggressively changed
>>compared to 1.1.0. It seems that it is developed with hadoop 0.23.1 or
>>1.0.1. It can work with hadoop 0.20.205.0 (time-interval-based rotation
>>works perfectly, last hadoop file closed correctly).
>>
>
>Flume looks into the exported classpath and determine the installed
>hadoop version to use it.
Yes. I tested flume with several hadoop versions. It seems that flume may
not work well with hadoop 0.20.2. For e.g., writer.syncFs() is not
supported in 0.20.2 (writer.syncFs() is implemented by calling sync(),
maybe using writer.sync() in flume is ok?)
>
>
>> Will keep an eye on NG and wait for later releases.
>
>
>cheers,
> Alex
>
>
>--
>Alexander Alten-Lorenz
>http://mapredit.blogspot.com
>German Hadoop LinkedIn Group: http://goo.gl/N8pCF
>
>
Cheers,
Yongkun Wang (Kun)
>
Re: Tried NG 1.1.0 and trunk
Posted by alo alt <wg...@gmail.com>.
Hi,
On Jul 20, 2012, at 12:13 PM, Wang, Yongkun | Yongkun | BDD wrote:
inline
> Hi,
>
> Currently we are using a modified version of flume-0.9.3 in production.
>
> I tried release 1.1.0. The basic command is still bin/flume-ng node, not bin/flume-ng agent (by doc).
Did you build Flume from git? In 1.1.x the notation was changed into agent, node is deprecated.
> No document for HDFS sink, I need to read the source code to config it. HDFS sink of 1.1.0 seems be developed with hadoop 0.20.205.0. It can rotate files by count of events, but the last hadoop file cannot be closed.
http://archive.cloudera.com/cdh4/cdh/4/flume-ng/
>
> Rotating the hadoop files by time interval does not work. It seems that problem is here:
> A mistake by timing 1000 in BucketWriter:
>
> lastProcessTime = System.currentTimeMillis() * 1000;
>
> So the lines in 'if' will never get executed:
>
> if ((rollInterval > 0)
>
> && (rollInterval < (System.currentTimeMillis() - lastProcessTime) / 1000))
>
> doRotate = true;
>
> But the code has been changed aggressively in later versions, so it is no longer a bug in the trunk.
>
> I tested NG trunk (1.3 snapshot). The code is aggressively changed compared to 1.1.0. It seems that it is developed with hadoop 0.23.1 or 1.0.1. It can work with hadoop 0.20.205.0 (time-interval-based rotation works perfectly, last hadoop file closed correctly).
>
Flume looks into the exported classpath and determine the installed hadoop version to use it.
> Will keep an eye on NG and wait for later releases.
cheers,
Alex
--
Alexander Alten-Lorenz
http://mapredit.blogspot.com
German Hadoop LinkedIn Group: http://goo.gl/N8pCF