Flume stops writing data to s3

[Frank Norris <ff...@mailbox.org>]

I use flume v1.6 and write data to s3 sink. However flume stops processing (no crash still print log message) when it receives http code 204. In log it shows flume was deleting files. Any places I can check to avoid this problem or further investigation?

Thanks

Re: Flume stops writing data to s3

[Frank Norris <ff...@mailbox.org>]

Log and cry are as below:

http://paste.debian.net/901028

https://paste.debian.net/901029

Actually there is no error or exception thrown. The problem is when observing 204 then flume stops consuming(only keep printing 'checkpoint not required' msg to log), resulting in the records in source and sink are not the same. This usually happens with consuming more than 15k records. When the number of records are not big e.g. 5k everything works fine (after moving data from source to sink, the record number is the same). And at this moment no transformation, etc. involves, just plain moving stuff from source to sink.

Thanks for help.


> On 05 December 2016 at 22:36 iain wright <ia...@gmail.com> wrote:
> 
>     seems http 204 is exepected response from S3 delete, can you pastebin your log/cfg (please omit keys if present in cfg)?
> 
>     Does your sink stop consuming from the channel when that error is encountered? 
> 
> 
>     --
>     Iain Wright
> 
>     This email message is confidential, intended only for the recipient(s) named above and may contain information that is privileged, exempt from disclosure under applicable law. If you are not the intended recipient, do not disclose or disseminate the message to anyone except the intended recipient. If you have received this message in error, or are not the named recipient(s), please immediately notify the sender by return email, and delete all copies of this message.
> 
>     On Mon, Dec 5, 2016 at 12:27 PM, Frank Norris <ffrank7482@mailbox.org mailto:ffrank7482@mailbox.org > wrote:
> 
>         > > 
> >         I use flume v1.6 and write data to s3 sink. However flume stops processing (no crash still print log message) when it receives http code 204. In log it shows flume was deleting files. Any places I can check to avoid this problem or further investigation?
> > 
> >         Thanks
> > 
> >     > 
> 

Re: Flume stops writing data to s3

[iain wright <ia...@gmail.com>]

seems http 204 is exepected response from S3 delete, can you pastebin your
log/cfg (please omit keys if present in cfg)?

Does your sink stop consuming from the channel when that error is
encountered?


-- 
Iain Wright

This email message is confidential, intended only for the recipient(s)
named above and may contain information that is privileged, exempt from
disclosure under applicable law. If you are not the intended recipient, do
not disclose or disseminate the message to anyone except the intended
recipient. If you have received this message in error, or are not the named
recipient(s), please immediately notify the sender by return email, and
delete all copies of this message.

On Mon, Dec 5, 2016 at 12:27 PM, Frank Norris <ff...@mailbox.org>
wrote:

> I use flume v1.6 and write data to s3 sink. However flume stops processing
> (no crash still print log message) when it receives http code 204. In log
> it shows flume was deleting files. Any places I can check to avoid this
> problem or further investigation?
>
> Thanks
>