You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@flink.apache.org by Lasse Nedergaard <la...@gmail.com> on 2024/03/20 12:41:11 UTC

Is there any options to control the file names in file sink

Hi. 

Anyone know if it’s possible to control the file names eg change the uuid file names and extensions to something else. I know I can control the path but I would like to be able to set the file name and the extension based on the data in the stream. So I can’t use any general pre or postfix that are applied to all files. 

All ideas appreciate 

Med venlig hilsen / Best regards
Lasse Nedergaard

Re: Is there any options to control the file names in file sink

Posted by Yanfei Lei <fr...@gmail.com>.

Hi Lasse,
If the datastream job is used, you can try setting `OutputFileConfig`
for file sink, something like[1]:
```
OutputFileConfig config = OutputFileConfig
 .builder()
 .withPartPrefix("prefix")
 .withPartSuffix(".ext")
 .build();

FileSink<Tuple2<Integer, Integer>> sink = FileSink
 .forRowFormat((new Path(outputPath), new SimpleStringEncoder<>("UTF-8"))
 .withBucketAssigner(new KeyBucketAssigner())
 .withRollingPolicy(OnCheckpointRollingPolicy.build())
 .withOutputFileConfig(config)
 .build();

```

[1] https://nightlies.apache.org/flink/flink-docs-release-1.18/docs/connectors/datastream/filesystem/#bucket-assignment

Lasse Nedergaard <la...@gmail.com> 于2024年3月20日周三 20:50写道：
>
> Hi.
>
> Anyone know if it’s possible to control the file names eg change the uuid file names and extensions to something else. I know I can control the path but I would like to be able to set the file name and the extension based on the data in the stream. So I can’t use any general pre or postfix that are applied to all files.
>
> All ideas appreciate
>
> Med venlig hilsen / Best regards
> Lasse Nedergaard
>


-- 
Best,
Yanfei

Re:Is there any options to control the file names in file sink

Posted by Xuyang <xy...@163.com>.

Hi, Nedergaard.
IIUC, currently, filesystem sink is not supported to custom the name of the output files, and only custom naming of directories should be supported.


[1] https://github.com/apache/flink/blob/4142c4386a92f1ec5016583f4832f8869782765e/flink-connectors/flink-connector-files/src/main/java/org/apache/flink/connector/file/table/FileSystemTableSink.java#L232



--

    Best！
    Xuyang





At 2024-03-20 20:41:11, "Lasse Nedergaard" <la...@gmail.com> wrote:
>Hi. 
>
>Anyone know if it’s possible to control the file names eg change the uuid file names and extensions to something else. I know I can control the path but I would like to be able to set the file name and the extension based on the data in the stream. So I can’t use any general pre or postfix that are applied to all files. 
>
>All ideas appreciate 
>
>Med venlig hilsen / Best regards
>Lasse Nedergaard