You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flume.apache.org by shangan chen <ch...@gmail.com> on 2013/07/05 13:28:45 UTC
hdfs sink with lzo codec failed to rename
I use hdfs sink with lzo codec, when finished injecting data to flume
agent, I found ***.lzo.tmp file in hdfs. It seems the hdfs outputstream
failed to call close, so rename is thus not called.
Anyone can explain why ? thanks a lot !
--
have a good day!
chenshang'an
Re: hdfs sink with lzo codec failed to rename
Posted by shangan chen <ch...@gmail.com>.
I think I mistook the mechanism how renaming is triggered. If I configure
the rollInterval or rollSize, when it reach the time interval or reach the
rollsize, it triggers the renaming. But if I only configure the rollSize
and the file size never reach that large, rename will never be triggered.
Before I think when I finished writing, it should trigger closing thus
call the rename. Now I add rollInterval, problem resolved.
On Mon, Jul 8, 2013 at 11:09 AM, shangan chen <ch...@gmail.com>wrote:
> hi, Lenin,
> Thanks for your quick reply. I didn't find any error log, in collector, I
> can only find the create file log as following, and no other logs...
>
> 08 Jul 2013 11:04:20,425 INFO [hdfs-sink_hdfs_0-call-runner-9]
> (org.apache.flume.sink.hdfs.BucketWriter.doOpen:215) - Creating
> /user/sankuai/flume_data/mobile_nginx/dt=20130708/hour=11/log_collector_srv03.1373252660403.lzo.tmp
>
>
> On Sat, Jul 6, 2013 at 1:54 AM, Lenin Raj <em...@gmail.com> wrote:
>
>> Hi Chen,
>>
>> Were you able to find any errors in the logs?
>>
>> --
>> Lenin
>> On Jul 5, 2013 4:59 PM, "shangan chen" <ch...@gmail.com> wrote:
>>
>>> I use hdfs sink with lzo codec, when finished injecting data to flume
>>> agent, I found ***.lzo.tmp file in hdfs. It seems the hdfs outputstream
>>> failed to call close, so rename is thus not called.
>>> Anyone can explain why ? thanks a lot !
>>>
>>> --
>>> have a good day!
>>> chenshang'an
>>>
>>>
>
>
> --
> have a good day!
> chenshang'an
>
>
--
have a good day!
chenshang'an
Re: hdfs sink with lzo codec failed to rename
Posted by shangan chen <ch...@gmail.com>.
hi, Lenin,
Thanks for your quick reply. I didn't find any error log, in collector, I
can only find the create file log as following, and no other logs...
08 Jul 2013 11:04:20,425 INFO [hdfs-sink_hdfs_0-call-runner-9]
(org.apache.flume.sink.hdfs.BucketWriter.doOpen:215) - Creating
/user/sankuai/flume_data/mobile_nginx/dt=20130708/hour=11/log_collector_srv03.1373252660403.lzo.tmp
On Sat, Jul 6, 2013 at 1:54 AM, Lenin Raj <em...@gmail.com> wrote:
> Hi Chen,
>
> Were you able to find any errors in the logs?
>
> --
> Lenin
> On Jul 5, 2013 4:59 PM, "shangan chen" <ch...@gmail.com> wrote:
>
>> I use hdfs sink with lzo codec, when finished injecting data to flume
>> agent, I found ***.lzo.tmp file in hdfs. It seems the hdfs outputstream
>> failed to call close, so rename is thus not called.
>> Anyone can explain why ? thanks a lot !
>>
>> --
>> have a good day!
>> chenshang'an
>>
>>
--
have a good day!
chenshang'an
Re: hdfs sink with lzo codec failed to rename
Posted by Lenin Raj <em...@gmail.com>.
Hi Chen,
Were you able to find any errors in the logs?
--
Lenin
On Jul 5, 2013 4:59 PM, "shangan chen" <ch...@gmail.com> wrote:
> I use hdfs sink with lzo codec, when finished injecting data to flume
> agent, I found ***.lzo.tmp file in hdfs. It seems the hdfs outputstream
> failed to call close, so rename is thus not called.
> Anyone can explain why ? thanks a lot !
>
> --
> have a good day!
> chenshang'an
>
>