You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by "ravi_suryavanshi.yahoo.com via user" <us...@flink.apache.org> on 2023/03/09 17:39:07 UTC
Re: Are the Table API Connectors production ready?
Hi,Can anyone help me here?
Thanks and regards,Ravi
On Monday, 27 February, 2023 at 09:33:18 am IST, ravi_suryavanshi.yahoo.com via user <us...@flink.apache.org> wrote:
Hi Team,
In Flink 1.16.0, we would like to use some of the Table API Connectors for production. Kindly let me know if the below connectors are production ready or only for testing purposes.
| Name | Version | Source | Sink |
| Filesystem | | Bounded and Unbounded Scan, Lookup | Streaming Sink, Batch Sink |
| Elasticsearch | 6.x & 7.x | Not supported | Streaming Sink, Batch Sink |
| Opensearch | 1.x & 2.x | Not supported | Streaming Sink, Batch Sink |
| Apache Kafka | 0.10+ | Unbounded Scan | Streaming Sink, Batch Sink |
| Amazon DynamoDB | | Not supported | Streaming Sink, Batch Sink |
| Amazon Kinesis Data Streams | | Unbounded Scan | Streaming Sink |
| Amazon Kinesis Data Firehose | | Not supported | Streaming Sink |
| JDBC | | Bounded Scan, Lookup | Streaming Sink, Batch Sink |
| Apache HBase | 1.4.x & 2.2.x | Bounded Scan, Lookup | Streaming Sink, Batch Sink |
| Apache Hive |
Thanks and regards
Re: Are the Table API Connectors production ready?
Posted by "ravi_suryavanshi.yahoo.com via user" <us...@flink.apache.org>.
Thank you All.
On Tuesday, 14 March, 2023 at 07:14:05 am IST, yuxia <lu...@alumni.sjtu.edu.cn> wrote:
The plan shows the filters has been pushed down. But remeber, although pused down, the filesystem table won't accept the filter. So, it'll be still like scan
all files.
Best regards,
Yuxia
发件人: "Maryam Moafimadani" <ma...@shopify.com>
收件人: "Hang Ruan" <ru...@gmail.com>
抄送: "yuxia" <lu...@alumni.sjtu.edu.cn>, "ravi suryavanshi" <ra...@yahoo.com>, "Yaroslav Tkachenko" <ya...@goldsky.com>, "Shammon FY" <zj...@gmail.com>, "User" <us...@flink.apache.org>
发送时间: 星期一, 2023年 3 月 13日 下午 10:07:57
主题: Re: Are the Table API Connectors production ready?
Hi All,It's exciting to see file filtering in the plan for development. I am curious whether the following query on a filesystem connector would actually push down the filter on metadata `file.path`?
Select score, `file.path` from MyUserTable WHERE `file.path` LIKE '%prefix_%'
== Optimized Execution Plan ==
Calc(select=[score, file.path], where=[LIKE(file.path, '%2022070611284%')])
+- TableSourceScan(table=[[default_catalog, default_database, MyUserTable, filter=[LIKE(file.path, _UTF-16LE'%2022070611284%')]]], fields=[score, file.path])
Thanks,Maryam
On Mon, Mar 13, 2023 at 8:55 AM Hang Ruan <ru...@gmail.com> wrote:
Hi, yuxia,I would like to help to complete this task.
Best,Hang
yuxia <lu...@alumni.sjtu.edu.cn> 于2023年3月13日周一 09:32写道：
Yeah, you're right. We don't provide filtering files with patterns. And actually we had already a jira[1] for it.
I was intended to do this in the past, but don't have much time. Anyone who are insterested can take it over. We're
happy to help review.
[1] https://issues.apache.org/jira/browse/FLINK-17398
Best regards,
Yuxia
发件人: "User" <us...@flink.apache.org>
收件人: "Yaroslav Tkachenko" <ya...@goldsky.com>, "Shammon FY" <zj...@gmail.com>
抄送: "User" <us...@flink.apache.org>
发送时间: 星期一, 2023年 3 月 13日 上午 12:36:46
主题: Re: Are the Table API Connectors production ready?
Thanks a lot, Yaroslav and Shammon.I want to use the Filesystem Connector. I tried it works well till it is running. If the job is restarted. It processes all the files again.
Could not find the move or delete option after collecting the files. Also, I could not find the filtering using patterns.
Pattern matching is required as different files exist in the same folder.
Regards,Ravi On Friday, 10 March, 2023 at 05:47:27 am IST, Shammon FY <zj...@gmail.com> wrote:
Hi Ravi
Agree with Yaroslav and if you find any problems in use, you can create an issue in jira https://issues.apache.org/jira/issues/?jql=project%20%3D%20FLINK . I have used kafka/jdbc/hive in production too, they work well.
Best,Shammon
On Fri, Mar 10, 2023 at 1:42 AM Yaroslav Tkachenko <ya...@goldsky.com> wrote:
Hi Ravi,
All of them should be production ready. I've personally used half of them in production.
Do you have any specific concerns?
On Thu, Mar 9, 2023 at 9:39 AM ravi_suryavanshi.yahoo.com via user <us...@flink.apache.org> wrote:
Hi,Can anyone help me here?
Thanks and regards,Ravi
On Monday, 27 February, 2023 at 09:33:18 am IST, ravi_suryavanshi.yahoo.com via user <us...@flink.apache.org> wrote:
Hi Team,
In Flink 1.16.0, we would like to use some of the Table API Connectors for production. Kindly let me know if the below connectors are production ready or only for testing purposes.
| Name | Version | Source | Sink |
| Filesystem | | Bounded and Unbounded Scan, Lookup | Streaming Sink, Batch Sink |
| Elasticsearch | 6.x & 7.x | Not supported | Streaming Sink, Batch Sink |
| Opensearch | 1.x & 2.x | Not supported | Streaming Sink, Batch Sink |
| Apache Kafka | 0.10+ | Unbounded Scan | Streaming Sink, Batch Sink |
| Amazon DynamoDB | | Not supported | Streaming Sink, Batch Sink |
| Amazon Kinesis Data Streams | | Unbounded Scan | Streaming Sink |
| Amazon Kinesis Data Firehose | | Not supported | Streaming Sink |
| JDBC | | Bounded Scan, Lookup | Streaming Sink, Batch Sink |
| Apache HBase | 1.4.x & 2.2.x | Bounded Scan, Lookup | Streaming Sink, Batch Sink |
| Apache Hive |
Thanks and regards
--
Maryam MoafimadaniSenior Data Developer @Shopify
Re: Are the Table API Connectors production ready?
Posted by yuxia <lu...@alumni.sjtu.edu.cn>.
The plan shows the filters has been pushed down. But remeber, although pused down, the filesystem table won't accept the filter. So, it'll be still like scan
all files.
Best regards,
Yuxia
发件人: "Maryam Moafimadani" <ma...@shopify.com>
收件人: "Hang Ruan" <ru...@gmail.com>
抄送: "yuxia" <lu...@alumni.sjtu.edu.cn>, "ravi suryavanshi" <ra...@yahoo.com>, "Yaroslav Tkachenko" <ya...@goldsky.com>, "Shammon FY" <zj...@gmail.com>, "User" <us...@flink.apache.org>
发送时间: 星期一, 2023年 3 月 13日 下午 10:07:57
主题: Re: Are the Table API Connectors production ready?
Hi All,
It's exciting to see file filtering in the plan for development. I am curious whether the following query on a filesystem connector would actually push down the filter on metadata `file.path`?
Select score, `file.path` from MyUserTable WHERE `file.path` LIKE '%prefix_%'
== Optimized Execution Plan ==
Calc(select=[score, file.path], where=[LIKE(file.path, '%2022070611284%')])
+- TableSourceScan(table=[[default_catalog, default_database, MyUserTable, filter=[LIKE(file.path, _UTF-16LE'%2022070611284%')]]], fields=[score, file.path])
Thanks,
Maryam
On Mon, Mar 13, 2023 at 8:55 AM Hang Ruan < [ mailto:ruanhang1993@gmail.com | ruanhang1993@gmail.com ] > wrote:
Hi, yuxia,
I would like to help to complete this task.
Best,
Hang
yuxia < [ mailto:luoyuxia@alumni.sjtu.edu.cn | luoyuxia@alumni.sjtu.edu.cn ] > 于2023年3月13日周一 09:32写道：
BQ_BEGIN
Yeah, you're right. We don't provide filtering files with patterns. And actually we had already a jira[1] for it.
I was intended to do this in the past, but don't have much time. Anyone who are insterested can take it over. We're
happy to help review.
[1] [ https://issues.apache.org/jira/browse/FLINK-17398 | https://issues.apache.org/jira/browse/FLINK-17398 ]
Best regards,
Yuxia
发件人: "User" < [ mailto:user@flink.apache.org | user@flink.apache.org ] >
收件人: "Yaroslav Tkachenko" < [ mailto:yaroslav@goldsky.com | yaroslav@goldsky.com ] >, "Shammon FY" < [ mailto:zjureel@gmail.com | zjureel@gmail.com ] >
抄送: "User" < [ mailto:user@flink.apache.org | user@flink.apache.org ] >
发送时间: 星期一, 2023年 3 月 13日 上午 12:36:46
主题: Re: Are the Table API Connectors production ready?
Thanks a lot, Yaroslav and Shammon.
I want to use the Filesystem Connector. I tried it works well till it is running. If the job is restarted. It processes all the files again.
Could not find the move or delete option after collecting the files. Also, I could not find the filtering using patterns.
Pattern matching is required as different files exist in the same folder.
Regards,
Ravi
On Friday, 10 March, 2023 at 05:47:27 am IST, Shammon FY < [ mailto:zjureel@gmail.com | zjureel@gmail.com ] > wrote:
Hi Ravi
Agree with Yaroslav and if you find any problems in use, you can create an issue in jira [ https://issues.apache.org/jira/issues/?jql=project%20%3D%20FLINK | https://issues.apache.org/jira/issues/?jql=project%20%3D%20FLINK ] . I have used kafka/jdbc/hive in production too, they work well.
Best,
Shammon
On Fri, Mar 10, 2023 at 1:42 AM Yaroslav Tkachenko < [ mailto:yaroslav@goldsky.com | yaroslav@goldsky.com ] > wrote:
BQ_BEGIN
Hi Ravi,
All of them should be production ready. I've personally used half of them in production.
Do you have any specific concerns?
On Thu, Mar 9, 2023 at 9:39 AM [ http://ravi_suryavanshi.yahoo.com/ | ravi_suryavanshi.yahoo.com ] via user < [ mailto:user@flink.apache.org | user@flink.apache.org ] > wrote:
BQ_BEGIN
Hi,
Can anyone help me here?
Thanks and regards,
Ravi
On Monday, 27 February, 2023 at 09:33:18 am IST, [ http://ravi_suryavanshi.yahoo.com/ | ravi_suryavanshi.yahoo.com ] via user < [ mailto:user@flink.apache.org | user@flink.apache.org ] > wrote:
Hi Team,
In Flink 1.16.0, we would like to use some of the Table API Connectors for production. Kindly let me know if the below connectors are production ready or only for testing purposes.
Name Version Source Sink [ https://nightlies.apache.org/flink/flink-docs-release-1.16/docs/connectors/table/filesystem/ | Filesystem ] Bounded and Unbounded Scan, Lookup Streaming Sink, Batch Sink
[ https://nightlies.apache.org/flink/flink-docs-release-1.16/docs/connectors/table/elasticsearch/ | Elasticsearch ] 6.x & 7.x Not supported Streaming Sink, Batch Sink
[ https://nightlies.apache.org/flink/flink-docs-release-1.16/docs/connectors/table/opensearch/ | Opensearch ] 1.x & 2.x Not supported Streaming Sink, Batch Sink
[ https://nightlies.apache.org/flink/flink-docs-release-1.16/docs/connectors/table/kafka/ | Apache Kafka ] 0.10+ Unbounded Scan Streaming Sink, Batch Sink
[ https://nightlies.apache.org/flink/flink-docs-release-1.16/docs/connectors/table/dynamodb/ | Amazon DynamoDB ] Not supported Streaming Sink, Batch Sink
[ https://nightlies.apache.org/flink/flink-docs-release-1.16/docs/connectors/table/kinesis/ | Amazon Kinesis Data Streams ] Unbounded Scan Streaming Sink
[ https://nightlies.apache.org/flink/flink-docs-release-1.16/docs/connectors/table/firehose/ | Amazon Kinesis Data Firehose ] Not supported Streaming Sink
[ https://nightlies.apache.org/flink/flink-docs-release-1.16/docs/connectors/table/jdbc/ | JDBC ] Bounded Scan, Lookup Streaming Sink, Batch Sink
[ https://nightlies.apache.org/flink/flink-docs-release-1.16/docs/connectors/table/hbase/ | Apache HBase ] 1.4.x & 2.2.x Bounded Scan, Lookup Streaming Sink, Batch Sink
[ https://nightlies.apache.org/flink/flink-docs-release-1.16/docs/connectors/table/hive/overview/ | Apache Hive ]
Thanks and regards
BQ_END
BQ_END
BQ_END
--
Maryam Moafimadani
Senior Data Developer @ [ http://www.shopify.com/ | Shopify ]
Re: Are the Table API Connectors production ready?
Posted by Maryam Moafimadani via user <us...@flink.apache.org>.
Hi All,
It's exciting to see file filtering in the plan for development. I am
curious whether the following query on a filesystem connector would
actually push down the filter on metadata `file.path`?
*Select score, `file.path` from MyUserTable WHERE `file.path` LIKE
'%prefix_%' *
== Optimized Execution Plan ==
Calc(select=[score, file.path], where=[LIKE(file.path, '%2022070611284%')])
+- TableSourceScan(table=[[default_catalog, default_database, MyUserTable,
filter=[LIKE(file.path, _UTF-16LE'%2022070611284%')]]], fields=[score,
file.path])
Thanks,
Maryam
On Mon, Mar 13, 2023 at 8:55 AM Hang Ruan <ru...@gmail.com> wrote:
> Hi, yuxia,
> I would like to help to complete this task.
>
> Best,
> Hang
>
> yuxia <lu...@alumni.sjtu.edu.cn> 于2023年3月13日周一 09:32写道：
>
>> Yeah, you're right. We don't provide filtering files with patterns. And
>> actually we had already a jira[1] for it.
>> I was intended to do this in the past, but don't have much time. Anyone
>> who are insterested can take it over. We're
>> happy to help review.
>>
>> [1] https://issues.apache.org/jira/browse/FLINK-17398
>>
>> Best regards,
>> Yuxia
>>
>> ------------------------------
>> *发件人: *"User" <us...@flink.apache.org>
>> *收件人: *"Yaroslav Tkachenko" <ya...@goldsky.com>, "Shammon FY" <
>> zjureel@gmail.com>
>> *抄送: *"User" <us...@flink.apache.org>
>> *发送时间: *星期一, 2023年 3 月 13日 上午 12:36:46
>> *主题: *Re: Are the Table API Connectors production ready?
>>
>> Thanks a lot, Yaroslav and Shammon.
>> I want to use the Filesystem Connector. I tried it works well till it is
>> running. If the job is restarted. It processes all the files again.
>>
>> Could not find the move or delete option after collecting the files.
>> Also, I could not find the filtering using patterns.
>>
>> Pattern matching is required as different files exist in the same folder.
>>
>> Regards,
>> Ravi
>> On Friday, 10 March, 2023 at 05:47:27 am IST, Shammon FY <
>> zjureel@gmail.com> wrote:
>>
>>
>> Hi Ravi
>>
>> Agree with Yaroslav and if you find any problems in use, you can create
>> an issue in jira
>> https://issues.apache.org/jira/issues/?jql=project%20%3D%20FLINK . I
>> have used kafka/jdbc/hive in production too, they work well.
>>
>> Best,
>> Shammon
>>
>> On Fri, Mar 10, 2023 at 1:42 AM Yaroslav Tkachenko <ya...@goldsky.com>
>> wrote:
>>
>> Hi Ravi,
>>
>> All of them should be production ready. I've personally used half of them
>> in production.
>>
>> Do you have any specific concerns?
>>
>> On Thu, Mar 9, 2023 at 9:39 AM ravi_suryavanshi.yahoo.com via user <
>> user@flink.apache.org> wrote:
>>
>> Hi,
>> Can anyone help me here?
>>
>> Thanks and regards,
>> Ravi
>>
>> On Monday, 27 February, 2023 at 09:33:18 am IST,
>> ravi_suryavanshi.yahoo.com via user <us...@flink.apache.org> wrote:
>>
>>
>> Hi Team,
>>
>>
>> In Flink 1.16.0, we would like to use some of the Table API Connectors
>> for production. Kindly let me know if the below connectors are production
>> ready or only for testing purposes.
>>
>> NameVersionSourceSink
>> Filesystem
>> <https://nightlies.apache.org/flink/flink-docs-release-1.16/docs/connectors/table/filesystem/> Bounded
>> and Unbounded Scan, Lookup Streaming Sink, Batch Sink
>> Elasticsearch
>> <https://nightlies.apache.org/flink/flink-docs-release-1.16/docs/connectors/table/elasticsearch/> 6.x
>> & 7.x Not supported Streaming Sink, Batch Sink
>> Opensearch
>> <https://nightlies.apache.org/flink/flink-docs-release-1.16/docs/connectors/table/opensearch/> 1.x
>> & 2.x Not supported Streaming Sink, Batch Sink
>> Apache Kafka
>> <https://nightlies.apache.org/flink/flink-docs-release-1.16/docs/connectors/table/kafka/>
>> 0.10+ Unbounded Scan Streaming Sink, Batch Sink
>> Amazon DynamoDB
>> <https://nightlies.apache.org/flink/flink-docs-release-1.16/docs/connectors/table/dynamodb/> Not
>> supported Streaming Sink, Batch Sink
>> Amazon Kinesis Data Streams
>> <https://nightlies.apache.org/flink/flink-docs-release-1.16/docs/connectors/table/kinesis/> Unbounded
>> Scan Streaming Sink
>> Amazon Kinesis Data Firehose
>> <https://nightlies.apache.org/flink/flink-docs-release-1.16/docs/connectors/table/firehose/> Not
>> supported Streaming Sink
>> JDBC
>> <https://nightlies.apache.org/flink/flink-docs-release-1.16/docs/connectors/table/jdbc/> Bounded
>> Scan, Lookup Streaming Sink, Batch Sink
>> Apache HBase
>> <https://nightlies.apache.org/flink/flink-docs-release-1.16/docs/connectors/table/hbase/> 1.4.x
>> & 2.2.x Bounded Scan, Lookup Streaming Sink, Batch Sink
>> Apache Hive
>> <https://nightlies.apache.org/flink/flink-docs-release-1.16/docs/connectors/table/hive/overview/>
>>
>> Thanks and regards
>>
>>
>>
--
Maryam Moafimadani
Senior Data Developer @Shopify <http://www.shopify.com/>
Re: Are the Table API Connectors production ready?
Posted by yuxia <lu...@alumni.sjtu.edu.cn>.
Thanks Hang for taking it. Assigned to you~
Best regards,
Yuxia
发件人: "Hang Ruan" <ru...@gmail.com>
收件人: "yuxia" <lu...@alumni.sjtu.edu.cn>
抄送: "ravi suryavanshi" <ra...@yahoo.com>, "Yaroslav Tkachenko" <ya...@goldsky.com>, "Shammon FY" <zj...@gmail.com>, "User" <us...@flink.apache.org>
发送时间: 星期一, 2023年 3 月 13日 下午 8:54:49
主题: Re: Are the Table API Connectors production ready?
Hi, yuxia,
I would like to help to complete this task.
Best,
Hang
yuxia < [ mailto:luoyuxia@alumni.sjtu.edu.cn | luoyuxia@alumni.sjtu.edu.cn ] > 于2023年3月13日周一 09:32写道：
Yeah, you're right. We don't provide filtering files with patterns. And actually we had already a jira[1] for it.
I was intended to do this in the past, but don't have much time. Anyone who are insterested can take it over. We're
happy to help review.
[1] [ https://issues.apache.org/jira/browse/FLINK-17398 | https://issues.apache.org/jira/browse/FLINK-17398 ]
Best regards,
Yuxia
发件人: "User" < [ mailto:user@flink.apache.org | user@flink.apache.org ] >
收件人: "Yaroslav Tkachenko" < [ mailto:yaroslav@goldsky.com | yaroslav@goldsky.com ] >, "Shammon FY" < [ mailto:zjureel@gmail.com | zjureel@gmail.com ] >
抄送: "User" < [ mailto:user@flink.apache.org | user@flink.apache.org ] >
发送时间: 星期一, 2023年 3 月 13日 上午 12:36:46
主题: Re: Are the Table API Connectors production ready?
Thanks a lot, Yaroslav and Shammon.
I want to use the Filesystem Connector. I tried it works well till it is running. If the job is restarted. It processes all the files again.
Could not find the move or delete option after collecting the files. Also, I could not find the filtering using patterns.
Pattern matching is required as different files exist in the same folder.
Regards,
Ravi
On Friday, 10 March, 2023 at 05:47:27 am IST, Shammon FY < [ mailto:zjureel@gmail.com | zjureel@gmail.com ] > wrote:
Hi Ravi
Agree with Yaroslav and if you find any problems in use, you can create an issue in jira [ https://issues.apache.org/jira/issues/?jql=project%20%3D%20FLINK | https://issues.apache.org/jira/issues/?jql=project%20%3D%20FLINK ] . I have used kafka/jdbc/hive in production too, they work well.
Best,
Shammon
On Fri, Mar 10, 2023 at 1:42 AM Yaroslav Tkachenko < [ mailto:yaroslav@goldsky.com | yaroslav@goldsky.com ] > wrote:
BQ_BEGIN
Hi Ravi,
All of them should be production ready. I've personally used half of them in production.
Do you have any specific concerns?
On Thu, Mar 9, 2023 at 9:39 AM [ http://ravi_suryavanshi.yahoo.com/ | ravi_suryavanshi.yahoo.com ] via user < [ mailto:user@flink.apache.org | user@flink.apache.org ] > wrote:
BQ_BEGIN
Hi,
Can anyone help me here?
Thanks and regards,
Ravi
On Monday, 27 February, 2023 at 09:33:18 am IST, [ http://ravi_suryavanshi.yahoo.com/ | ravi_suryavanshi.yahoo.com ] via user < [ mailto:user@flink.apache.org | user@flink.apache.org ] > wrote:
Hi Team,
In Flink 1.16.0, we would like to use some of the Table API Connectors for production. Kindly let me know if the below connectors are production ready or only for testing purposes.
Name Version Source Sink [ https://nightlies.apache.org/flink/flink-docs-release-1.16/docs/connectors/table/filesystem/ | Filesystem ] Bounded and Unbounded Scan, Lookup Streaming Sink, Batch Sink
[ https://nightlies.apache.org/flink/flink-docs-release-1.16/docs/connectors/table/elasticsearch/ | Elasticsearch ] 6.x & 7.x Not supported Streaming Sink, Batch Sink
[ https://nightlies.apache.org/flink/flink-docs-release-1.16/docs/connectors/table/opensearch/ | Opensearch ] 1.x & 2.x Not supported Streaming Sink, Batch Sink
[ https://nightlies.apache.org/flink/flink-docs-release-1.16/docs/connectors/table/kafka/ | Apache Kafka ] 0.10+ Unbounded Scan Streaming Sink, Batch Sink
[ https://nightlies.apache.org/flink/flink-docs-release-1.16/docs/connectors/table/dynamodb/ | Amazon DynamoDB ] Not supported Streaming Sink, Batch Sink
[ https://nightlies.apache.org/flink/flink-docs-release-1.16/docs/connectors/table/kinesis/ | Amazon Kinesis Data Streams ] Unbounded Scan Streaming Sink
[ https://nightlies.apache.org/flink/flink-docs-release-1.16/docs/connectors/table/firehose/ | Amazon Kinesis Data Firehose ] Not supported Streaming Sink
[ https://nightlies.apache.org/flink/flink-docs-release-1.16/docs/connectors/table/jdbc/ | JDBC ] Bounded Scan, Lookup Streaming Sink, Batch Sink
[ https://nightlies.apache.org/flink/flink-docs-release-1.16/docs/connectors/table/hbase/ | Apache HBase ] 1.4.x & 2.2.x Bounded Scan, Lookup Streaming Sink, Batch Sink
[ https://nightlies.apache.org/flink/flink-docs-release-1.16/docs/connectors/table/hive/overview/ | Apache Hive ]
Thanks and regards
BQ_END
BQ_END
Re: Are the Table API Connectors production ready?
Posted by Hang Ruan <ru...@gmail.com>.
Hi, yuxia,
I would like to help to complete this task.
Best,
Hang
yuxia <lu...@alumni.sjtu.edu.cn> 于2023年3月13日周一 09:32写道：
> Yeah, you're right. We don't provide filtering files with patterns. And
> actually we had already a jira[1] for it.
> I was intended to do this in the past, but don't have much time. Anyone
> who are insterested can take it over. We're
> happy to help review.
>
> [1] https://issues.apache.org/jira/browse/FLINK-17398
>
> Best regards,
> Yuxia
>
> ------------------------------
> *发件人: *"User" <us...@flink.apache.org>
> *收件人: *"Yaroslav Tkachenko" <ya...@goldsky.com>, "Shammon FY" <
> zjureel@gmail.com>
> *抄送: *"User" <us...@flink.apache.org>
> *发送时间: *星期一, 2023年 3 月 13日 上午 12:36:46
> *主题: *Re: Are the Table API Connectors production ready?
>
> Thanks a lot, Yaroslav and Shammon.
> I want to use the Filesystem Connector. I tried it works well till it is
> running. If the job is restarted. It processes all the files again.
>
> Could not find the move or delete option after collecting the files. Also,
> I could not find the filtering using patterns.
>
> Pattern matching is required as different files exist in the same folder.
>
> Regards,
> Ravi
> On Friday, 10 March, 2023 at 05:47:27 am IST, Shammon FY <
> zjureel@gmail.com> wrote:
>
>
> Hi Ravi
>
> Agree with Yaroslav and if you find any problems in use, you can create an
> issue in jira
> https://issues.apache.org/jira/issues/?jql=project%20%3D%20FLINK . I have
> used kafka/jdbc/hive in production too, they work well.
>
> Best,
> Shammon
>
> On Fri, Mar 10, 2023 at 1:42 AM Yaroslav Tkachenko <ya...@goldsky.com>
> wrote:
>
> Hi Ravi,
>
> All of them should be production ready. I've personally used half of them
> in production.
>
> Do you have any specific concerns?
>
> On Thu, Mar 9, 2023 at 9:39 AM ravi_suryavanshi.yahoo.com via user <
> user@flink.apache.org> wrote:
>
> Hi,
> Can anyone help me here?
>
> Thanks and regards,
> Ravi
>
> On Monday, 27 February, 2023 at 09:33:18 am IST,
> ravi_suryavanshi.yahoo.com via user <us...@flink.apache.org> wrote:
>
>
> Hi Team,
>
>
> In Flink 1.16.0, we would like to use some of the Table API Connectors for
> production. Kindly let me know if the below connectors are production ready
> or only for testing purposes.
>
> NameVersionSourceSink
> Filesystem
> <https://nightlies.apache.org/flink/flink-docs-release-1.16/docs/connectors/table/filesystem/> Bounded
> and Unbounded Scan, Lookup Streaming Sink, Batch Sink
> Elasticsearch
> <https://nightlies.apache.org/flink/flink-docs-release-1.16/docs/connectors/table/elasticsearch/> 6.x
> & 7.x Not supported Streaming Sink, Batch Sink
> Opensearch
> <https://nightlies.apache.org/flink/flink-docs-release-1.16/docs/connectors/table/opensearch/> 1.x
> & 2.x Not supported Streaming Sink, Batch Sink
> Apache Kafka
> <https://nightlies.apache.org/flink/flink-docs-release-1.16/docs/connectors/table/kafka/>
> 0.10+ Unbounded Scan Streaming Sink, Batch Sink
> Amazon DynamoDB
> <https://nightlies.apache.org/flink/flink-docs-release-1.16/docs/connectors/table/dynamodb/> Not
> supported Streaming Sink, Batch Sink
> Amazon Kinesis Data Streams
> <https://nightlies.apache.org/flink/flink-docs-release-1.16/docs/connectors/table/kinesis/> Unbounded
> Scan Streaming Sink
> Amazon Kinesis Data Firehose
> <https://nightlies.apache.org/flink/flink-docs-release-1.16/docs/connectors/table/firehose/> Not
> supported Streaming Sink
> JDBC
> <https://nightlies.apache.org/flink/flink-docs-release-1.16/docs/connectors/table/jdbc/> Bounded
> Scan, Lookup Streaming Sink, Batch Sink
> Apache HBase
> <https://nightlies.apache.org/flink/flink-docs-release-1.16/docs/connectors/table/hbase/> 1.4.x
> & 2.2.x Bounded Scan, Lookup Streaming Sink, Batch Sink
> Apache Hive
> <https://nightlies.apache.org/flink/flink-docs-release-1.16/docs/connectors/table/hive/overview/>
>
> Thanks and regards
>
>
>
Re: Are the Table API Connectors production ready?
Posted by yuxia <lu...@alumni.sjtu.edu.cn>.
Yeah, you're right. We don't provide filtering files with patterns. And actually we had already a jira[1] for it.
I was intended to do this in the past, but don't have much time. Anyone who are insterested can take it over. We're
happy to help review.
[1] https://issues.apache.org/jira/browse/FLINK-17398
Best regards,
Yuxia
发件人: "User" <us...@flink.apache.org>
收件人: "Yaroslav Tkachenko" <ya...@goldsky.com>, "Shammon FY" <zj...@gmail.com>
抄送: "User" <us...@flink.apache.org>
发送时间: 星期一, 2023年 3 月 13日 上午 12:36:46
主题: Re: Are the Table API Connectors production ready?
Thanks a lot, Yaroslav and Shammon.
I want to use the Filesystem Connector. I tried it works well till it is running. If the job is restarted. It processes all the files again.
Could not find the move or delete option after collecting the files. Also, I could not find the filtering using patterns.
Pattern matching is required as different files exist in the same folder.
Regards,
Ravi
On Friday, 10 March, 2023 at 05:47:27 am IST, Shammon FY <zj...@gmail.com> wrote:
Hi Ravi
Agree with Yaroslav and if you find any problems in use, you can create an issue in jira [ https://issues.apache.org/jira/issues/?jql=project%20%3D%20FLINK | https://issues.apache.org/jira/issues/?jql=project%20%3D%20FLINK ] . I have used kafka/jdbc/hive in production too, they work well.
Best,
Shammon
On Fri, Mar 10, 2023 at 1:42 AM Yaroslav Tkachenko < [ mailto:yaroslav@goldsky.com | yaroslav@goldsky.com ] > wrote:
Hi Ravi,
All of them should be production ready. I've personally used half of them in production.
Do you have any specific concerns?
On Thu, Mar 9, 2023 at 9:39 AM [ http://ravi_suryavanshi.yahoo.com/ | ravi_suryavanshi.yahoo.com ] via user < [ mailto:user@flink.apache.org | user@flink.apache.org ] > wrote:
BQ_BEGIN
Hi,
Can anyone help me here?
Thanks and regards,
Ravi
On Monday, 27 February, 2023 at 09:33:18 am IST, [ http://ravi_suryavanshi.yahoo.com/ | ravi_suryavanshi.yahoo.com ] via user < [ mailto:user@flink.apache.org | user@flink.apache.org ] > wrote:
Hi Team,
In Flink 1.16.0, we would like to use some of the Table API Connectors for production. Kindly let me know if the below connectors are production ready or only for testing purposes.
Name Version Source Sink [ https://nightlies.apache.org/flink/flink-docs-release-1.16/docs/connectors/table/filesystem/ | Filesystem ] Bounded and Unbounded Scan, Lookup Streaming Sink, Batch Sink
[ https://nightlies.apache.org/flink/flink-docs-release-1.16/docs/connectors/table/elasticsearch/ | Elasticsearch ] 6.x & 7.x Not supported Streaming Sink, Batch Sink
[ https://nightlies.apache.org/flink/flink-docs-release-1.16/docs/connectors/table/opensearch/ | Opensearch ] 1.x & 2.x Not supported Streaming Sink, Batch Sink
[ https://nightlies.apache.org/flink/flink-docs-release-1.16/docs/connectors/table/kafka/ | Apache Kafka ] 0.10+ Unbounded Scan Streaming Sink, Batch Sink
[ https://nightlies.apache.org/flink/flink-docs-release-1.16/docs/connectors/table/dynamodb/ | Amazon DynamoDB ] Not supported Streaming Sink, Batch Sink
[ https://nightlies.apache.org/flink/flink-docs-release-1.16/docs/connectors/table/kinesis/ | Amazon Kinesis Data Streams ] Unbounded Scan Streaming Sink
[ https://nightlies.apache.org/flink/flink-docs-release-1.16/docs/connectors/table/firehose/ | Amazon Kinesis Data Firehose ] Not supported Streaming Sink
[ https://nightlies.apache.org/flink/flink-docs-release-1.16/docs/connectors/table/jdbc/ | JDBC ] Bounded Scan, Lookup Streaming Sink, Batch Sink
[ https://nightlies.apache.org/flink/flink-docs-release-1.16/docs/connectors/table/hbase/ | Apache HBase ] 1.4.x & 2.2.x Bounded Scan, Lookup Streaming Sink, Batch Sink
[ https://nightlies.apache.org/flink/flink-docs-release-1.16/docs/connectors/table/hive/overview/ | Apache Hive ]
Thanks and regards
BQ_END
Re: Are the Table API Connectors production ready?
Posted by Shammon FY <zj...@gmail.com>.
Hi ravi
Do you read streaming data with filesystem connector? By default, the file
system is bound. If you need to read streaming data, you can look at the
configuration `source.monitor.interval` in
https://nightlies.apache.org/flink/flink-docs-master/docs/connectors/table/filesystem/#directory-watching,
and see whether your job is executed in streaming mode and trigger
checkpoint regularly.
Best,
Shammon
On Mon, Mar 13, 2023 at 12:36 AM Shammon FY <zj...@gmail.com> wrote:
>
>
Re: Are the Table API Connectors production ready?
Posted by "ravi_suryavanshi.yahoo.com via user" <us...@flink.apache.org>.
Thanks a lot, Yaroslav and Shammon.I want to use the Filesystem Connector. I tried it works well till it is running. If the job is restarted. It processes all the files again.
Could not find the move or delete option after collecting the files. Also, I could not find the filtering using patterns.
Pattern matching is required as different files exist in the same folder.
Regards,Ravi On Friday, 10 March, 2023 at 05:47:27 am IST, Shammon FY <zj...@gmail.com> wrote:
Hi Ravi
Agree with Yaroslav and if you find any problems in use, you can create an issue in jira https://issues.apache.org/jira/issues/?jql=project%20%3D%20FLINK . I have used kafka/jdbc/hive in production too, they work well.
Best,Shammon
On Fri, Mar 10, 2023 at 1:42 AM Yaroslav Tkachenko <ya...@goldsky.com> wrote:
Hi Ravi,
All of them should be production ready. I've personally used half of them in production.
Do you have any specific concerns?
On Thu, Mar 9, 2023 at 9:39 AM ravi_suryavanshi.yahoo.com via user <us...@flink.apache.org> wrote:
Hi,Can anyone help me here?
Thanks and regards,Ravi
On Monday, 27 February, 2023 at 09:33:18 am IST, ravi_suryavanshi.yahoo.com via user <us...@flink.apache.org> wrote:
Hi Team,
In Flink 1.16.0, we would like to use some of the Table API Connectors for production. Kindly let me know if the below connectors are production ready or only for testing purposes.
| Name | Version | Source | Sink |
| Filesystem | | Bounded and Unbounded Scan, Lookup | Streaming Sink, Batch Sink |
| Elasticsearch | 6.x & 7.x | Not supported | Streaming Sink, Batch Sink |
| Opensearch | 1.x & 2.x | Not supported | Streaming Sink, Batch Sink |
| Apache Kafka | 0.10+ | Unbounded Scan | Streaming Sink, Batch Sink |
| Amazon DynamoDB | | Not supported | Streaming Sink, Batch Sink |
| Amazon Kinesis Data Streams | | Unbounded Scan | Streaming Sink |
| Amazon Kinesis Data Firehose | | Not supported | Streaming Sink |
| JDBC | | Bounded Scan, Lookup | Streaming Sink, Batch Sink |
| Apache HBase | 1.4.x & 2.2.x | Bounded Scan, Lookup | Streaming Sink, Batch Sink |
| Apache Hive |
Thanks and regards
Re: Are the Table API Connectors production ready?
Posted by Shammon FY <zj...@gmail.com>.
Hi Ravi
Agree with Yaroslav and if you find any problems in use, you can create an
issue in jira
https://issues.apache.org/jira/issues/?jql=project%20%3D%20FLINK . I have
used kafka/jdbc/hive in production too, they work well.
Best,
Shammon
On Fri, Mar 10, 2023 at 1:42 AM Yaroslav Tkachenko <ya...@goldsky.com>
wrote:
> Hi Ravi,
>
> All of them should be production ready. I've personally used half of them
> in production.
>
> Do you have any specific concerns?
>
> On Thu, Mar 9, 2023 at 9:39 AM ravi_suryavanshi.yahoo.com via user <
> user@flink.apache.org> wrote:
>
>> Hi,
>> Can anyone help me here?
>>
>> Thanks and regards,
>> Ravi
>>
>> On Monday, 27 February, 2023 at 09:33:18 am IST,
>> ravi_suryavanshi.yahoo.com via user <us...@flink.apache.org> wrote:
>>
>>
>> Hi Team,
>>
>>
>> In Flink 1.16.0, we would like to use some of the Table API Connectors
>> for production. Kindly let me know if the below connectors are production
>> ready or only for testing purposes.
>>
>> NameVersionSourceSink
>> Filesystem
>> <https://nightlies.apache.org/flink/flink-docs-release-1.16/docs/connectors/table/filesystem/> Bounded
>> and Unbounded Scan, Lookup Streaming Sink, Batch Sink
>> Elasticsearch
>> <https://nightlies.apache.org/flink/flink-docs-release-1.16/docs/connectors/table/elasticsearch/> 6.x
>> & 7.x Not supported Streaming Sink, Batch Sink
>> Opensearch
>> <https://nightlies.apache.org/flink/flink-docs-release-1.16/docs/connectors/table/opensearch/> 1.x
>> & 2.x Not supported Streaming Sink, Batch Sink
>> Apache Kafka
>> <https://nightlies.apache.org/flink/flink-docs-release-1.16/docs/connectors/table/kafka/>
>> 0.10+ Unbounded Scan Streaming Sink, Batch Sink
>> Amazon DynamoDB
>> <https://nightlies.apache.org/flink/flink-docs-release-1.16/docs/connectors/table/dynamodb/> Not
>> supported Streaming Sink, Batch Sink
>> Amazon Kinesis Data Streams
>> <https://nightlies.apache.org/flink/flink-docs-release-1.16/docs/connectors/table/kinesis/> Unbounded
>> Scan Streaming Sink
>> Amazon Kinesis Data Firehose
>> <https://nightlies.apache.org/flink/flink-docs-release-1.16/docs/connectors/table/firehose/> Not
>> supported Streaming Sink
>> JDBC
>> <https://nightlies.apache.org/flink/flink-docs-release-1.16/docs/connectors/table/jdbc/> Bounded
>> Scan, Lookup Streaming Sink, Batch Sink
>> Apache HBase
>> <https://nightlies.apache.org/flink/flink-docs-release-1.16/docs/connectors/table/hbase/> 1.4.x
>> & 2.2.x Bounded Scan, Lookup Streaming Sink, Batch Sink
>> Apache Hive
>> <https://nightlies.apache.org/flink/flink-docs-release-1.16/docs/connectors/table/hive/overview/>
>>
>> Thanks and regards
>>
>
Re: Are the Table API Connectors production ready?
Posted by Yaroslav Tkachenko <ya...@goldsky.com>.
Hi Ravi,
All of them should be production ready. I've personally used half of them
in production.
Do you have any specific concerns?
On Thu, Mar 9, 2023 at 9:39 AM ravi_suryavanshi.yahoo.com via user <
user@flink.apache.org> wrote:
> Hi,
> Can anyone help me here?
>
> Thanks and regards,
> Ravi
>
> On Monday, 27 February, 2023 at 09:33:18 am IST,
> ravi_suryavanshi.yahoo.com via user <us...@flink.apache.org> wrote:
>
>
> Hi Team,
>
>
> In Flink 1.16.0, we would like to use some of the Table API Connectors for
> production. Kindly let me know if the below connectors are production ready
> or only for testing purposes.
>
> NameVersionSourceSink
> Filesystem
> <https://nightlies.apache.org/flink/flink-docs-release-1.16/docs/connectors/table/filesystem/> Bounded
> and Unbounded Scan, Lookup Streaming Sink, Batch Sink
> Elasticsearch
> <https://nightlies.apache.org/flink/flink-docs-release-1.16/docs/connectors/table/elasticsearch/> 6.x
> & 7.x Not supported Streaming Sink, Batch Sink
> Opensearch
> <https://nightlies.apache.org/flink/flink-docs-release-1.16/docs/connectors/table/opensearch/> 1.x
> & 2.x Not supported Streaming Sink, Batch Sink
> Apache Kafka
> <https://nightlies.apache.org/flink/flink-docs-release-1.16/docs/connectors/table/kafka/>
> 0.10+ Unbounded Scan Streaming Sink, Batch Sink
> Amazon DynamoDB
> <https://nightlies.apache.org/flink/flink-docs-release-1.16/docs/connectors/table/dynamodb/> Not
> supported Streaming Sink, Batch Sink
> Amazon Kinesis Data Streams
> <https://nightlies.apache.org/flink/flink-docs-release-1.16/docs/connectors/table/kinesis/> Unbounded
> Scan Streaming Sink
> Amazon Kinesis Data Firehose
> <https://nightlies.apache.org/flink/flink-docs-release-1.16/docs/connectors/table/firehose/> Not
> supported Streaming Sink
> JDBC
> <https://nightlies.apache.org/flink/flink-docs-release-1.16/docs/connectors/table/jdbc/> Bounded
> Scan, Lookup Streaming Sink, Batch Sink
> Apache HBase
> <https://nightlies.apache.org/flink/flink-docs-release-1.16/docs/connectors/table/hbase/> 1.4.x
> & 2.2.x Bounded Scan, Lookup Streaming Sink, Batch Sink
> Apache Hive
> <https://nightlies.apache.org/flink/flink-docs-release-1.16/docs/connectors/table/hive/overview/>
>
> Thanks and regards
>