You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@flume.apache.org by Dmytro Popovych <dr...@gmail.com> on 2015/12/14 17:02:24 UTC

HDFS sink: inUseSuffix for s3

Hi all,

I noticed that it isn’t possible to specify empty inUseSuffix for the HDFS sink.
As a result, we can't avoid a redundant rename operation for files in s3.
But afaik, s3 native filesystem keeps a temporary file on the local fs of the machine.
So, we anyway copy on s3 the final data, right?

At the same time the rename operation for s3 is copy + delete,
which is kinda useless if we already have a final data in the "temporary" file.

The question is: how can I avoid the rename operation in a HDFS sink?

Thanks in advance!

- Dmytro Popovych