You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@paimon.apache.org by "MOBIN-F (via GitHub)" <gi...@apache.org> on 2024/09/10 10:39:04 UTC
[PR] [cdc] paimon cdc support snaphost mode [paimon]
MOBIN-F opened a new pull request, #4164:
URL: https://github.com/apache/paimon/pull/4164
<!-- Please specify the module before the PR name: [core] ... or [flink] ... -->
### Purpose
Currently paimon cdc does not support scan.startup.mode=snapshot, but flink-cdc 3.x(mysql、pg、mongdb cdc) already supports shapshot mode
<!-- What is the purpose of the change -->
### Tests
<!-- List UT and IT cases to verify this change -->
### API and Format
<!-- Does this change affect API or storage format -->
### Documentation
<!-- Does this change introduce a new feature -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [cdc] paimon cdc support snapshot mode [paimon]
Posted by "LinMingQiang (via GitHub)" <gi...@apache.org>.
LinMingQiang commented on PR #4164:
URL: https://github.com/apache/paimon/pull/4164#issuecomment-2348436781
In addition, the job will not finish until checkpoint is triggered once in streaming mode.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [cdc] paimon cdc support snapshot mode [paimon]
Posted by "LinMingQiang (via GitHub)" <gi...@apache.org>.
LinMingQiang commented on PR #4164:
URL: https://github.com/apache/paimon/pull/4164#issuecomment-2348158180
Why ```MysqlSource``` can't finish when i set ```-Dexecution.runtime-mode=BATCH ```.
If in ``` scan.startup.mode=snapshot``` mode , 'execution.runtime-mode' should/can be ```Batch```?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [cdc] paimon cdc support snapshot mode [paimon]
Posted by "MOBIN-F (via GitHub)" <gi...@apache.org>.
MOBIN-F commented on PR #4164:
URL: https://github.com/apache/paimon/pull/4164#issuecomment-2348240851
> Why `MysqlSource` can't finish when i set `-Dexecution.runtime-mode=BATCH `.
>
> If in ` scan.startup.mode=snapshot` mode , 'execution.runtime-mode' should/can be `Batch`?
hi @LinMingQiang , no need to set the batch mode, the job will automatically complete and exit
```
2024-09-10 18:20:40,006 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Job MySQL-Paimon Table Sync: rt_ods.xxxx(d9998e2d7bfccd5d320fb4ac21761c3d) switched from state
RUNNING to FINISHED.
->
2024-09-10 18:20:40,007 DEBUG org.apache.flink.runtime.executiongraph.ExecutionGraph [] - ExecutionGraph d9998e2d7bfccd5d320fb4ac21761c3d reached terminal state FINISHED.
->
2024-09-10 18:20:40,343 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - Shutting YarnJobClusterEntrypoint down with application status SUCCEEDED. Diagnostics null.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [cdc] paimon cdc support snapshot mode [paimon]
Posted by "LinMingQiang (via GitHub)" <gi...@apache.org>.
LinMingQiang commented on PR #4164:
URL: https://github.com/apache/paimon/pull/4164#issuecomment-2348255742
It is normal in mode ```streaming```. My question is why can't I use batch mode?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [cdc] paimon cdc support snapshot mode [paimon]
Posted by "MOBIN-F (via GitHub)" <gi...@apache.org>.
MOBIN-F commented on PR #4164:
URL: https://github.com/apache/paimon/pull/4164#issuecomment-2348784983
@LinMingQiang See more details in discussion of [#2867](https://github.com/apache/flink-cdc/issues/2867).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [cdc] paimon cdc support snapshot mode [paimon]
Posted by "LinMingQiang (via GitHub)" <gi...@apache.org>.
LinMingQiang commented on PR #4164:
URL: https://github.com/apache/paimon/pull/4164#issuecomment-2350821576
I have reported the bug to cdc :
https://issues.apache.org/jira/browse/FLINK-36280
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [cdc] paimon cdc support snapshot mode [paimon]
Posted by "LinMingQiang (via GitHub)" <gi...@apache.org>.
LinMingQiang commented on PR #4164:
URL: https://github.com/apache/paimon/pull/4164#issuecomment-2348192874
Just 1 record in mysql. and operator ```Source: MySQL Source -> Parse -> Side Output```` will be never finish.
<img width="1529" alt="image" src="https://github.com/user-attachments/assets/04c9e2fd-35e6-4ed7-984d-a14ec3519d8b">
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [cdc] paimon cdc support snapshot mode [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi merged PR #4164:
URL: https://github.com/apache/paimon/pull/4164
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [cdc] paimon cdc support snapshot mode [paimon]
Posted by "LinMingQiang (via GitHub)" <gi...@apache.org>.
LinMingQiang commented on PR #4164:
URL: https://github.com/apache/paimon/pull/4164#issuecomment-2348191136
```flink run -Dexecution.runtime-mode=BATCH \
paimon-flink-action-1.0-SNAPSHOT.jar \
mysql_sync_table \
--warehouse xxx \
--database local_database \
--table test_paimon_cdc_opt \
--mysql_conf hostname=localhost \
--mysql_conf username=root \
--mysql_conf password=123456789 \
--mysql_conf database-name=paimon \
--mysql_conf table-name=paimon_cdc_opt \
--mysql_conf scan.startup.mode=snapshot \
--table_conf bucket=10 \
--table_conf tag.automatic-creation=watermark \
--partition_keys dt \
--primary_keys dt,id \
--type-mapping char-to-string```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org