Hdfs -> hdfs ends up with .progress files

[Johan Rask <jo...@flapsdown.se>]

Hi!

I am quite new to Apache Flink but we have been evaluating it for some
weeks to read from kafka and transform and write to hdfs. With kafka to
hdfs with exactly-once configured this works as expected but when we
replace the source with kafka to hdfs files hangs in .in-progress.

We first experienced this also with kafka but after properly configuring
checkpoints we got expected results.

We have simply changed kafka source to hdfs source and kept streaming mode
although this is obviously a bounded data set so there might be some issue
with this that we do not understand.

Any help with this is highly appreciated!

Regards /Johan Rask

I have created a small gist of our program here
https://gist.github.com/jrask/ef4a8531b0563f1420ce276e7b0f59ce

And I also copy of the gist here.

 public static void main(String[] args) throws Exception {

        final StreamExecutionEnvironment env =
StreamExecutionEnvironment.getExecutionEnvironment();
        env.setStreamTimeCharacteristic(TimeCharacteristic.EventTime);
        env.enableCheckpointing(TimeUnit.MINUTES.toMillis(1),
CheckpointingMode.EXACTLY_ONCE);
        env.setStateBackend(new
FsStateBackend("hdfs://<server><dir>/checkpoints",true));
        env.getCheckpointConfig().setTolerableCheckpointFailureNumber(1);

        StreamingFileSink<JsonObject> hdfsSink = StreamingFileSink
                .<JsonObject>forRowFormat(new
Path("hdfs://<server>/<dir>"), new SimpleStringEncoder<>("UTF-8"))
                .withBucketAssigner(new
EventTimeDateTimeBuckerAssigner<>("'/year'=YYYY/'month'=MM/'day'=dd/'hour'=HH"))
                .build();

        env.readTextFile("hdfs://<server><dir><file>")
            .map(Parser::parse)
            .addSink(hdfsSink);

        env.execute("some-pipeline");
    }


hdfs -> hdfs results in the following. However if I use kafka as
source, it works properly.

rw-rw----+  3 someuser supergroup   87792789 2019-11-16 20:57
/data/some_dir/year=2019/month=05/day=21/hour=23/.part-0-62.inprogress.8f9c6104-4c6c-4eee-8650-dd5d1d12d668
-rw-rw----+  3 someuser supergroup   64696413 2019-11-16 20:58
/data/some_dir/year=2019/month=05/day=21/hour=23/.part-0-63.inprogress.42589a04-601b-496d-ae20-7db1d56089dc

... rest is removed for clarity