You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flume.apache.org by Abraham Fine <ab...@brightroll.com> on 2014/06/10 00:56:05 UTC
File Channel Backup Checkpoints are I/O Intensive
Hello-
We are using Flume 1.4 with File Channel configured to use a very
large capacity. We keep the checkpoint and backup checkpoint on
separate disks.
Normally the file channel is mostly empty (<<1% of capacity). For the
checkpoint the disk I/O seems to be very reasonable due to the usage
of a MappedByteBuffer.
On the other hand, the backup checkpoint seems to be written to disk
in its entirety over and over again, resulting in very high disk
utilization.
I noticed that, because the checkpoint file is mostly empty, it is
very compressible. I was able to GZIP our checkpoint from 381M to
386K. I was wondering if it would be possible to always compress the
backup checkpoint before writing it to disk.
I would be happy to work on a patch to implement this functionality if
there is interest.
Thanks in Advance,
--
Abraham Fine | Software Engineer
(516) 567-2535
BrightRoll, Inc. | Smart Video Advertising | www.brightroll.com
Re: File Channel Backup Checkpoints are I/O Intensive
Posted by Hari Shreedharan <hs...@cloudera.com>.
Thanks a lot. I will take a look at this tomorrow or early next week.
On Wed, Jul 2, 2014 at 5:29 PM, Abraham Fine <ab...@brightroll.com> wrote:
> Hari-
>
> I added the new tests and created a new revision to my patch.
>
>
> https://issues.apache.org/jira/secure/attachment/12653728/compress_backup_checkpoint_new_tests.patch
>
> Thanks,
> Abe
>
> --
> Abraham Fine | Software Engineer
> (516) 567-2535
> BrightRoll, Inc. | Smart Video Advertising | www.brightroll.com
>
>
> On Wed, Jul 2, 2014 at 4:32 PM, Hari Shreedharan <
> hshreedharan@cloudera.com> wrote:
>
>> Hi Abraham,
>>
>> In general, the patch looks good. Can you add a couple of tests -
>> * Original checkpoint is uncompressed, config changes to compress
>> checkpoint - does the file channel restart from original checkpoint? are
>> new checkpoints compressed?
>> * Compressed checkpoint, config changes to not compress checkpoint - does
>> channel start up? are new checkpoints uncompressed?
>>
>>
>> Hari
>>
>>
>> On Wed, Jul 2, 2014 at 3:06 PM, Abraham Fine <ab...@brightroll.com> wrote:
>>
>>> Hi Brock and Hari-
>>>
>>> I was just wondering if either of you had a chance to take a look at the
>>> patch and if there is anything I can do to improve it.
>>>
>>> Thanks,
>>> Abe
>>>
>>> --
>>> Abraham Fine | Software Engineer
>>> (516) 567-2535
>>> BrightRoll, Inc. | Smart Video Advertising | www.brightroll.com
>>>
>>>
>>> On Wed, Jun 11, 2014 at 6:48 PM, Brock Noland <br...@cloudera.com>
>>> wrote:
>>>
>>>> This is a great suggestion Abraham!
>>>>
>>>>
>>>> On Wed, Jun 11, 2014 at 5:39 PM, Hari Shreedharan <
>>>> hshreedharan@cloudera.com> wrote:
>>>>
>>>>> Thanks. I will review it :)
>>>>>
>>>>>
>>>>> Thanks,
>>>>> Hari
>>>>>
>>>>> On Wednesday, June 11, 2014 at 5:00 PM, Abraham Fine wrote:
>>>>>
>>>>> I went ahead and created a JIRA and patch:
>>>>> https://issues.apache.org/jira/browse/FLUME-2401
>>>>>
>>>>> The option is configurable with:
>>>>> agentX.channels.ch1.compressBackupCheckpoint = true
>>>>>
>>>>> As per your recommendation, I used snappy-java. I also considered the
>>>>> snappy and lz4 implementations in Hadoop IO but noticed that the
>>>>> Hadoop IO dependency was removed in
>>>>> https://issues.apache.org/jira/browse/FLUME-1285
>>>>>
>>>>> Thanks,
>>>>> Abe
>>>>> --
>>>>> Abraham Fine | Software Engineer
>>>>> (516) 567-2535
>>>>> BrightRoll, Inc. | Smart Video Advertising | www.brightroll.com
>>>>>
>>>>>
>>>>> On Mon, Jun 9, 2014 at 4:01 PM, Hari Shreedharan
>>>>> <hs...@cloudera.com> wrote:
>>>>>
>>>>> Hi Abraham,
>>>>>
>>>>> Compressing the backup checkpoint is very possible. Since the backup is
>>>>> rarely read (only if the original one is corrupt on restarts), is it
>>>>> used.
>>>>> So I think compressing it using something like Snappy would make sense
>>>>> (GZIP
>>>>> might hit performance). Can you try using snappy-java and see if that
>>>>> gives
>>>>> good perf and reasonable compression?
>>>>>
>>>>> Patches are always welcome. I’d be glad to review and commit it. I
>>>>> would
>>>>> suggest making the compression optional via configuration so that
>>>>> anyone
>>>>> with smaller channels don’t end up using CPU for not much gain.
>>>>>
>>>>>
>>>>> Thanks,
>>>>> Hari
>>>>>
>>>>> On Monday, June 9, 2014 at 3:56 PM, Abraham Fine wrote:
>>>>>
>>>>> Hello-
>>>>>
>>>>> We are using Flume 1.4 with File Channel configured to use a very
>>>>> large capacity. We keep the checkpoint and backup checkpoint on
>>>>> separate disks.
>>>>>
>>>>> Normally the file channel is mostly empty (<<1% of capacity). For the
>>>>> checkpoint the disk I/O seems to be very reasonable due to the usage
>>>>> of a MappedByteBuffer.
>>>>>
>>>>> On the other hand, the backup checkpoint seems to be written to disk
>>>>> in its entirety over and over again, resulting in very high disk
>>>>> utilization.
>>>>>
>>>>> I noticed that, because the checkpoint file is mostly empty, it is
>>>>> very compressible. I was able to GZIP our checkpoint from 381M to
>>>>> 386K. I was wondering if it would be possible to always compress the
>>>>> backup checkpoint before writing it to disk.
>>>>>
>>>>> I would be happy to work on a patch to implement this functionality if
>>>>> there is interest.
>>>>>
>>>>> Thanks in Advance,
>>>>>
>>>>> --
>>>>> Abraham Fine | Software Engineer
>>>>> (516) 567-2535
>>>>> BrightRoll, Inc. | Smart Video Advertising | www.brightroll.com
>>>>>
>>>>>
>>>>>
>>>>
>>>
>>
>
Re: File Channel Backup Checkpoints are I/O Intensive
Posted by Abraham Fine <ab...@brightroll.com>.
Hari-
I added the new tests and created a new revision to my patch.
https://issues.apache.org/jira/secure/attachment/12653728/compress_backup_checkpoint_new_tests.patch
Thanks,
Abe
--
Abraham Fine | Software Engineer
(516) 567-2535
BrightRoll, Inc. | Smart Video Advertising | www.brightroll.com
On Wed, Jul 2, 2014 at 4:32 PM, Hari Shreedharan <hs...@cloudera.com>
wrote:
> Hi Abraham,
>
> In general, the patch looks good. Can you add a couple of tests -
> * Original checkpoint is uncompressed, config changes to compress
> checkpoint - does the file channel restart from original checkpoint? are
> new checkpoints compressed?
> * Compressed checkpoint, config changes to not compress checkpoint - does
> channel start up? are new checkpoints uncompressed?
>
>
> Hari
>
>
> On Wed, Jul 2, 2014 at 3:06 PM, Abraham Fine <ab...@brightroll.com> wrote:
>
>> Hi Brock and Hari-
>>
>> I was just wondering if either of you had a chance to take a look at the
>> patch and if there is anything I can do to improve it.
>>
>> Thanks,
>> Abe
>>
>> --
>> Abraham Fine | Software Engineer
>> (516) 567-2535
>> BrightRoll, Inc. | Smart Video Advertising | www.brightroll.com
>>
>>
>> On Wed, Jun 11, 2014 at 6:48 PM, Brock Noland <br...@cloudera.com> wrote:
>>
>>> This is a great suggestion Abraham!
>>>
>>>
>>> On Wed, Jun 11, 2014 at 5:39 PM, Hari Shreedharan <
>>> hshreedharan@cloudera.com> wrote:
>>>
>>>> Thanks. I will review it :)
>>>>
>>>>
>>>> Thanks,
>>>> Hari
>>>>
>>>> On Wednesday, June 11, 2014 at 5:00 PM, Abraham Fine wrote:
>>>>
>>>> I went ahead and created a JIRA and patch:
>>>> https://issues.apache.org/jira/browse/FLUME-2401
>>>>
>>>> The option is configurable with:
>>>> agentX.channels.ch1.compressBackupCheckpoint = true
>>>>
>>>> As per your recommendation, I used snappy-java. I also considered the
>>>> snappy and lz4 implementations in Hadoop IO but noticed that the
>>>> Hadoop IO dependency was removed in
>>>> https://issues.apache.org/jira/browse/FLUME-1285
>>>>
>>>> Thanks,
>>>> Abe
>>>> --
>>>> Abraham Fine | Software Engineer
>>>> (516) 567-2535
>>>> BrightRoll, Inc. | Smart Video Advertising | www.brightroll.com
>>>>
>>>>
>>>> On Mon, Jun 9, 2014 at 4:01 PM, Hari Shreedharan
>>>> <hs...@cloudera.com> wrote:
>>>>
>>>> Hi Abraham,
>>>>
>>>> Compressing the backup checkpoint is very possible. Since the backup is
>>>> rarely read (only if the original one is corrupt on restarts), is it
>>>> used.
>>>> So I think compressing it using something like Snappy would make sense
>>>> (GZIP
>>>> might hit performance). Can you try using snappy-java and see if that
>>>> gives
>>>> good perf and reasonable compression?
>>>>
>>>> Patches are always welcome. I’d be glad to review and commit it. I would
>>>> suggest making the compression optional via configuration so that anyone
>>>> with smaller channels don’t end up using CPU for not much gain.
>>>>
>>>>
>>>> Thanks,
>>>> Hari
>>>>
>>>> On Monday, June 9, 2014 at 3:56 PM, Abraham Fine wrote:
>>>>
>>>> Hello-
>>>>
>>>> We are using Flume 1.4 with File Channel configured to use a very
>>>> large capacity. We keep the checkpoint and backup checkpoint on
>>>> separate disks.
>>>>
>>>> Normally the file channel is mostly empty (<<1% of capacity). For the
>>>> checkpoint the disk I/O seems to be very reasonable due to the usage
>>>> of a MappedByteBuffer.
>>>>
>>>> On the other hand, the backup checkpoint seems to be written to disk
>>>> in its entirety over and over again, resulting in very high disk
>>>> utilization.
>>>>
>>>> I noticed that, because the checkpoint file is mostly empty, it is
>>>> very compressible. I was able to GZIP our checkpoint from 381M to
>>>> 386K. I was wondering if it would be possible to always compress the
>>>> backup checkpoint before writing it to disk.
>>>>
>>>> I would be happy to work on a patch to implement this functionality if
>>>> there is interest.
>>>>
>>>> Thanks in Advance,
>>>>
>>>> --
>>>> Abraham Fine | Software Engineer
>>>> (516) 567-2535
>>>> BrightRoll, Inc. | Smart Video Advertising | www.brightroll.com
>>>>
>>>>
>>>>
>>>
>>
>
Re: File Channel Backup Checkpoints are I/O Intensive
Posted by Hari Shreedharan <hs...@cloudera.com>.
Hi Abraham,
In general, the patch looks good. Can you add a couple of tests -
* Original checkpoint is uncompressed, config changes to compress
checkpoint - does the file channel restart from original checkpoint? are
new checkpoints compressed?
* Compressed checkpoint, config changes to not compress checkpoint - does
channel start up? are new checkpoints uncompressed?
Hari
On Wed, Jul 2, 2014 at 3:06 PM, Abraham Fine <ab...@brightroll.com> wrote:
> Hi Brock and Hari-
>
> I was just wondering if either of you had a chance to take a look at the
> patch and if there is anything I can do to improve it.
>
> Thanks,
> Abe
>
> --
> Abraham Fine | Software Engineer
> (516) 567-2535
> BrightRoll, Inc. | Smart Video Advertising | www.brightroll.com
>
>
> On Wed, Jun 11, 2014 at 6:48 PM, Brock Noland <br...@cloudera.com> wrote:
>
>> This is a great suggestion Abraham!
>>
>>
>> On Wed, Jun 11, 2014 at 5:39 PM, Hari Shreedharan <
>> hshreedharan@cloudera.com> wrote:
>>
>>> Thanks. I will review it :)
>>>
>>>
>>> Thanks,
>>> Hari
>>>
>>> On Wednesday, June 11, 2014 at 5:00 PM, Abraham Fine wrote:
>>>
>>> I went ahead and created a JIRA and patch:
>>> https://issues.apache.org/jira/browse/FLUME-2401
>>>
>>> The option is configurable with:
>>> agentX.channels.ch1.compressBackupCheckpoint = true
>>>
>>> As per your recommendation, I used snappy-java. I also considered the
>>> snappy and lz4 implementations in Hadoop IO but noticed that the
>>> Hadoop IO dependency was removed in
>>> https://issues.apache.org/jira/browse/FLUME-1285
>>>
>>> Thanks,
>>> Abe
>>> --
>>> Abraham Fine | Software Engineer
>>> (516) 567-2535
>>> BrightRoll, Inc. | Smart Video Advertising | www.brightroll.com
>>>
>>>
>>> On Mon, Jun 9, 2014 at 4:01 PM, Hari Shreedharan
>>> <hs...@cloudera.com> wrote:
>>>
>>> Hi Abraham,
>>>
>>> Compressing the backup checkpoint is very possible. Since the backup is
>>> rarely read (only if the original one is corrupt on restarts), is it
>>> used.
>>> So I think compressing it using something like Snappy would make sense
>>> (GZIP
>>> might hit performance). Can you try using snappy-java and see if that
>>> gives
>>> good perf and reasonable compression?
>>>
>>> Patches are always welcome. I’d be glad to review and commit it. I would
>>> suggest making the compression optional via configuration so that anyone
>>> with smaller channels don’t end up using CPU for not much gain.
>>>
>>>
>>> Thanks,
>>> Hari
>>>
>>> On Monday, June 9, 2014 at 3:56 PM, Abraham Fine wrote:
>>>
>>> Hello-
>>>
>>> We are using Flume 1.4 with File Channel configured to use a very
>>> large capacity. We keep the checkpoint and backup checkpoint on
>>> separate disks.
>>>
>>> Normally the file channel is mostly empty (<<1% of capacity). For the
>>> checkpoint the disk I/O seems to be very reasonable due to the usage
>>> of a MappedByteBuffer.
>>>
>>> On the other hand, the backup checkpoint seems to be written to disk
>>> in its entirety over and over again, resulting in very high disk
>>> utilization.
>>>
>>> I noticed that, because the checkpoint file is mostly empty, it is
>>> very compressible. I was able to GZIP our checkpoint from 381M to
>>> 386K. I was wondering if it would be possible to always compress the
>>> backup checkpoint before writing it to disk.
>>>
>>> I would be happy to work on a patch to implement this functionality if
>>> there is interest.
>>>
>>> Thanks in Advance,
>>>
>>> --
>>> Abraham Fine | Software Engineer
>>> (516) 567-2535
>>> BrightRoll, Inc. | Smart Video Advertising | www.brightroll.com
>>>
>>>
>>>
>>
>
Re: File Channel Backup Checkpoints are I/O Intensive
Posted by Abraham Fine <ab...@brightroll.com>.
Hi Brock and Hari-
I was just wondering if either of you had a chance to take a look at the
patch and if there is anything I can do to improve it.
Thanks,
Abe
--
Abraham Fine | Software Engineer
(516) 567-2535
BrightRoll, Inc. | Smart Video Advertising | www.brightroll.com
On Wed, Jun 11, 2014 at 6:48 PM, Brock Noland <br...@cloudera.com> wrote:
> This is a great suggestion Abraham!
>
>
> On Wed, Jun 11, 2014 at 5:39 PM, Hari Shreedharan <
> hshreedharan@cloudera.com> wrote:
>
>> Thanks. I will review it :)
>>
>>
>> Thanks,
>> Hari
>>
>> On Wednesday, June 11, 2014 at 5:00 PM, Abraham Fine wrote:
>>
>> I went ahead and created a JIRA and patch:
>> https://issues.apache.org/jira/browse/FLUME-2401
>>
>> The option is configurable with:
>> agentX.channels.ch1.compressBackupCheckpoint = true
>>
>> As per your recommendation, I used snappy-java. I also considered the
>> snappy and lz4 implementations in Hadoop IO but noticed that the
>> Hadoop IO dependency was removed in
>> https://issues.apache.org/jira/browse/FLUME-1285
>>
>> Thanks,
>> Abe
>> --
>> Abraham Fine | Software Engineer
>> (516) 567-2535
>> BrightRoll, Inc. | Smart Video Advertising | www.brightroll.com
>>
>>
>> On Mon, Jun 9, 2014 at 4:01 PM, Hari Shreedharan
>> <hs...@cloudera.com> wrote:
>>
>> Hi Abraham,
>>
>> Compressing the backup checkpoint is very possible. Since the backup is
>> rarely read (only if the original one is corrupt on restarts), is it used.
>> So I think compressing it using something like Snappy would make sense
>> (GZIP
>> might hit performance). Can you try using snappy-java and see if that
>> gives
>> good perf and reasonable compression?
>>
>> Patches are always welcome. I’d be glad to review and commit it. I would
>> suggest making the compression optional via configuration so that anyone
>> with smaller channels don’t end up using CPU for not much gain.
>>
>>
>> Thanks,
>> Hari
>>
>> On Monday, June 9, 2014 at 3:56 PM, Abraham Fine wrote:
>>
>> Hello-
>>
>> We are using Flume 1.4 with File Channel configured to use a very
>> large capacity. We keep the checkpoint and backup checkpoint on
>> separate disks.
>>
>> Normally the file channel is mostly empty (<<1% of capacity). For the
>> checkpoint the disk I/O seems to be very reasonable due to the usage
>> of a MappedByteBuffer.
>>
>> On the other hand, the backup checkpoint seems to be written to disk
>> in its entirety over and over again, resulting in very high disk
>> utilization.
>>
>> I noticed that, because the checkpoint file is mostly empty, it is
>> very compressible. I was able to GZIP our checkpoint from 381M to
>> 386K. I was wondering if it would be possible to always compress the
>> backup checkpoint before writing it to disk.
>>
>> I would be happy to work on a patch to implement this functionality if
>> there is interest.
>>
>> Thanks in Advance,
>>
>> --
>> Abraham Fine | Software Engineer
>> (516) 567-2535
>> BrightRoll, Inc. | Smart Video Advertising | www.brightroll.com
>>
>>
>>
>
Re: File Channel Backup Checkpoints are I/O Intensive
Posted by Brock Noland <br...@cloudera.com>.
This is a great suggestion Abraham!
On Wed, Jun 11, 2014 at 5:39 PM, Hari Shreedharan <hshreedharan@cloudera.com
> wrote:
> Thanks. I will review it :)
>
>
> Thanks,
> Hari
>
> On Wednesday, June 11, 2014 at 5:00 PM, Abraham Fine wrote:
>
> I went ahead and created a JIRA and patch:
> https://issues.apache.org/jira/browse/FLUME-2401
>
> The option is configurable with:
> agentX.channels.ch1.compressBackupCheckpoint = true
>
> As per your recommendation, I used snappy-java. I also considered the
> snappy and lz4 implementations in Hadoop IO but noticed that the
> Hadoop IO dependency was removed in
> https://issues.apache.org/jira/browse/FLUME-1285
>
> Thanks,
> Abe
> --
> Abraham Fine | Software Engineer
> (516) 567-2535
> BrightRoll, Inc. | Smart Video Advertising | www.brightroll.com
>
>
> On Mon, Jun 9, 2014 at 4:01 PM, Hari Shreedharan
> <hs...@cloudera.com> wrote:
>
> Hi Abraham,
>
> Compressing the backup checkpoint is very possible. Since the backup is
> rarely read (only if the original one is corrupt on restarts), is it used.
> So I think compressing it using something like Snappy would make sense
> (GZIP
> might hit performance). Can you try using snappy-java and see if that gives
> good perf and reasonable compression?
>
> Patches are always welcome. I’d be glad to review and commit it. I would
> suggest making the compression optional via configuration so that anyone
> with smaller channels don’t end up using CPU for not much gain.
>
>
> Thanks,
> Hari
>
> On Monday, June 9, 2014 at 3:56 PM, Abraham Fine wrote:
>
> Hello-
>
> We are using Flume 1.4 with File Channel configured to use a very
> large capacity. We keep the checkpoint and backup checkpoint on
> separate disks.
>
> Normally the file channel is mostly empty (<<1% of capacity). For the
> checkpoint the disk I/O seems to be very reasonable due to the usage
> of a MappedByteBuffer.
>
> On the other hand, the backup checkpoint seems to be written to disk
> in its entirety over and over again, resulting in very high disk
> utilization.
>
> I noticed that, because the checkpoint file is mostly empty, it is
> very compressible. I was able to GZIP our checkpoint from 381M to
> 386K. I was wondering if it would be possible to always compress the
> backup checkpoint before writing it to disk.
>
> I would be happy to work on a patch to implement this functionality if
> there is interest.
>
> Thanks in Advance,
>
> --
> Abraham Fine | Software Engineer
> (516) 567-2535
> BrightRoll, Inc. | Smart Video Advertising | www.brightroll.com
>
>
>
Re: File Channel Backup Checkpoints are I/O Intensive
Posted by Hari Shreedharan <hs...@cloudera.com>.
Thanks. I will review it :)
Thanks,
Hari
On Wednesday, June 11, 2014 at 5:00 PM, Abraham Fine wrote:
> I went ahead and created a JIRA and patch:
> https://issues.apache.org/jira/browse/FLUME-2401
>
> The option is configurable with:
> agentX.channels.ch1.compressBackupCheckpoint = true
>
> As per your recommendation, I used snappy-java. I also considered the
> snappy and lz4 implementations in Hadoop IO but noticed that the
> Hadoop IO dependency was removed in
> https://issues.apache.org/jira/browse/FLUME-1285
>
> Thanks,
> Abe
> --
> Abraham Fine | Software Engineer
> (516) 567-2535
> BrightRoll, Inc. | Smart Video Advertising | www.brightroll.com (http://www.brightroll.com)
>
>
> On Mon, Jun 9, 2014 at 4:01 PM, Hari Shreedharan
> <hshreedharan@cloudera.com (mailto:hshreedharan@cloudera.com)> wrote:
> > Hi Abraham,
> >
> > Compressing the backup checkpoint is very possible. Since the backup is
> > rarely read (only if the original one is corrupt on restarts), is it used.
> > So I think compressing it using something like Snappy would make sense (GZIP
> > might hit performance). Can you try using snappy-java and see if that gives
> > good perf and reasonable compression?
> >
> > Patches are always welcome. I’d be glad to review and commit it. I would
> > suggest making the compression optional via configuration so that anyone
> > with smaller channels don’t end up using CPU for not much gain.
> >
> >
> > Thanks,
> > Hari
> >
> > On Monday, June 9, 2014 at 3:56 PM, Abraham Fine wrote:
> >
> > Hello-
> >
> > We are using Flume 1.4 with File Channel configured to use a very
> > large capacity. We keep the checkpoint and backup checkpoint on
> > separate disks.
> >
> > Normally the file channel is mostly empty (<<1% of capacity). For the
> > checkpoint the disk I/O seems to be very reasonable due to the usage
> > of a MappedByteBuffer.
> >
> > On the other hand, the backup checkpoint seems to be written to disk
> > in its entirety over and over again, resulting in very high disk
> > utilization.
> >
> > I noticed that, because the checkpoint file is mostly empty, it is
> > very compressible. I was able to GZIP our checkpoint from 381M to
> > 386K. I was wondering if it would be possible to always compress the
> > backup checkpoint before writing it to disk.
> >
> > I would be happy to work on a patch to implement this functionality if
> > there is interest.
> >
> > Thanks in Advance,
> >
> > --
> > Abraham Fine | Software Engineer
> > (516) 567-2535
> > BrightRoll, Inc. | Smart Video Advertising | www.brightroll.com (http://www.brightroll.com)
> >
>
>
>
Re: File Channel Backup Checkpoints are I/O Intensive
Posted by Abraham Fine <ab...@brightroll.com>.
I went ahead and created a JIRA and patch:
https://issues.apache.org/jira/browse/FLUME-2401
The option is configurable with:
agentX.channels.ch1.compressBackupCheckpoint = true
As per your recommendation, I used snappy-java. I also considered the
snappy and lz4 implementations in Hadoop IO but noticed that the
Hadoop IO dependency was removed in
https://issues.apache.org/jira/browse/FLUME-1285
Thanks,
Abe
--
Abraham Fine | Software Engineer
(516) 567-2535
BrightRoll, Inc. | Smart Video Advertising | www.brightroll.com
On Mon, Jun 9, 2014 at 4:01 PM, Hari Shreedharan
<hs...@cloudera.com> wrote:
> Hi Abraham,
>
> Compressing the backup checkpoint is very possible. Since the backup is
> rarely read (only if the original one is corrupt on restarts), is it used.
> So I think compressing it using something like Snappy would make sense (GZIP
> might hit performance). Can you try using snappy-java and see if that gives
> good perf and reasonable compression?
>
> Patches are always welcome. I’d be glad to review and commit it. I would
> suggest making the compression optional via configuration so that anyone
> with smaller channels don’t end up using CPU for not much gain.
>
>
> Thanks,
> Hari
>
> On Monday, June 9, 2014 at 3:56 PM, Abraham Fine wrote:
>
> Hello-
>
> We are using Flume 1.4 with File Channel configured to use a very
> large capacity. We keep the checkpoint and backup checkpoint on
> separate disks.
>
> Normally the file channel is mostly empty (<<1% of capacity). For the
> checkpoint the disk I/O seems to be very reasonable due to the usage
> of a MappedByteBuffer.
>
> On the other hand, the backup checkpoint seems to be written to disk
> in its entirety over and over again, resulting in very high disk
> utilization.
>
> I noticed that, because the checkpoint file is mostly empty, it is
> very compressible. I was able to GZIP our checkpoint from 381M to
> 386K. I was wondering if it would be possible to always compress the
> backup checkpoint before writing it to disk.
>
> I would be happy to work on a patch to implement this functionality if
> there is interest.
>
> Thanks in Advance,
>
> --
> Abraham Fine | Software Engineer
> (516) 567-2535
> BrightRoll, Inc. | Smart Video Advertising | www.brightroll.com
>
>
Re: File Channel Backup Checkpoints are I/O Intensive
Posted by Hari Shreedharan <hs...@cloudera.com>.
Hi Abraham,
Compressing the backup checkpoint is very possible. Since the backup is rarely read (only if the original one is corrupt on restarts), is it used. So I think compressing it using something like Snappy would make sense (GZIP might hit performance). Can you try using snappy-java and see if that gives good perf and reasonable compression?
Patches are always welcome. I’d be glad to review and commit it. I would suggest making the compression optional via configuration so that anyone with smaller channels don’t end up using CPU for not much gain.
Thanks,
Hari
On Monday, June 9, 2014 at 3:56 PM, Abraham Fine wrote:
> Hello-
>
> We are using Flume 1.4 with File Channel configured to use a very
> large capacity. We keep the checkpoint and backup checkpoint on
> separate disks.
>
> Normally the file channel is mostly empty (<<1% of capacity). For the
> checkpoint the disk I/O seems to be very reasonable due to the usage
> of a MappedByteBuffer.
>
> On the other hand, the backup checkpoint seems to be written to disk
> in its entirety over and over again, resulting in very high disk
> utilization.
>
> I noticed that, because the checkpoint file is mostly empty, it is
> very compressible. I was able to GZIP our checkpoint from 381M to
> 386K. I was wondering if it would be possible to always compress the
> backup checkpoint before writing it to disk.
>
> I would be happy to work on a patch to implement this functionality if
> there is interest.
>
> Thanks in Advance,
>
> --
> Abraham Fine | Software Engineer
> (516) 567-2535
> BrightRoll, Inc. | Smart Video Advertising | www.brightroll.com (http://www.brightroll.com)
>
>