You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Siew Wai Yow <wa...@hotmail.com> on 2019/03/01 08:39:56 UTC
Flink Custom SourceFunction and SinkFunction
Hi guys,
I have question regarding to the title that need your expertise,
1. I need to build a SFTP SourceFunction, may I know if hadoop SFTPFileSystem suitable?
2. I need to build a SFTP SinkFunction as well, may I know if per-defined HDFS rolling file sink accept SFTP connection since SFTP is supported by hadoop file system?
3. Any good reference on how to write custom source/sink?
4. Any similar code to share?
Thanks!
Regards,
Yow
Re: Flink Custom SourceFunction and SinkFunction
Posted by Piotr Nowojski <pi...@ververica.com>.
Hi,
I couldn’t find any references to your question neither I haven’t seen such use case, but:
Re 1.
It looks like it could work
Re 2.
It should work as well, but just try to use StreamingFileSink
Re 3.
For custom source/sink function, if you do not care data processing guarantees it’s quite easy. If you have to achieve at-least-once or exactly-once things might get more complicated.
For exactly-once sink, you should start from `TwoPhaseCommitSinkFunction`. (Example usages check test class `TwoPhaseCommitSinkFunctionTest.ContentDumpSinkFunction`, or more complicated FlinkKafkaProducer)
For at-least-once sink, you can just flush/sync the output files on snapshot/checkpoint.
For source, you would have to manually keep the input offsets on Flink’s state.
Re 4.
Regarding SFTP support: not that I’m aware of.
Regarding sources/sinks you can try to look at existing source/sinks implementations.
Piotrek
> On 1 Mar 2019, at 09:39, Siew Wai Yow <wa...@hotmail.com> wrote:
>
> Hi guys,
>
> I have question regarding to the title that need your expertise,
>
> I need to build a SFTP SourceFunction, may I know if hadoop SFTPFileSystem suitable?
> I need to build a SFTP SinkFunction as well, may I know if per-defined HDFS rolling file sink accept SFTP connection since SFTP is supported by hadoop file system?
> Any good reference on how to write custom source/sink?
> Any similar code to share?
> Thanks!
>
> Regards,
> Yow
Re: Flink Custom SourceFunction and SinkFunction
Posted by Piotr Nowojski <pi...@ververica.com>.
Hi,
I couldn’t find any references to your question neither I haven’t seen such use case, but:
Re 1.
It looks like it could work
Re 2.
It should work as well, but just try to use StreamingFileSink
Re 3.
For custom source/sink function, if you do not care data processing guarantees it’s quite easy. If you have to achieve at-least-once or exactly-once things might get more complicated.
For exactly-once sink, you should start from `TwoPhaseCommitSinkFunction`. (Example usages check test class `TwoPhaseCommitSinkFunctionTest.ContentDumpSinkFunction`, or more complicated FlinkKafkaProducer)
For at-least-once sink, you can just flush/sync the output files on snapshot/checkpoint.
For source, you would have to manually keep the input offsets on Flink’s state.
Re 4.
Regarding SFTP support: not that I’m aware of.
Regarding sources/sinks you can try to look at existing source/sinks implementations.
Piotrek
> On 1 Mar 2019, at 09:39, Siew Wai Yow <wa...@hotmail.com> wrote:
>
> Hi guys,
>
> I have question regarding to the title that need your expertise,
>
> I need to build a SFTP SourceFunction, may I know if hadoop SFTPFileSystem suitable?
> I need to build a SFTP SinkFunction as well, may I know if per-defined HDFS rolling file sink accept SFTP connection since SFTP is supported by hadoop file system?
> Any good reference on how to write custom source/sink?
> Any similar code to share?
> Thanks!
>
> Regards,
> Yow