You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "lschampion (via GitHub)" <gi...@apache.org> on 2023/11/23 06:04:57 UTC
[I] [SUPPORT] flink jdbc reading to hudi lost record for not enough checkpoints in flink stream api. [hudi]
lschampion opened a new issue, #10166:
URL: https://github.com/apache/hudi/issues/10166
**Describe the problem you faced**
using flink stream api
(1) create a flink jdbc source, which is obvious batch reading.
(2) create a flink hudi sink
(3) sink source to sink,start the flink job in streaming mode
(4) checkpoint works, and some record stored on hdfs.
(5) jdbc source run out and flink exit while records submitted to hudi without enough checkpionts completed.
(6) only a small part of records writed down, some record losted
is there any way to solve it?
**Environment Description**
* Hudi version :
0.13.1
* Spark version :
none
* Hive version :
2.1.1
* Hadoop version :
3.0.0
* Storage (HDFS/S3/GCS..) :
hdfs
* Running on Docker? (yes/no) :
no
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] flink jdbc reading to hudi lost record for not enough checkpoints in flink stream api. [hudi]
Posted by "lschampion (via GitHub)" <gi...@apache.org>.
lschampion commented on issue #10166:
URL: https://github.com/apache/hudi/issues/10166#issuecomment-1824073145
note my flink version: 1.13.6
is this require might come true when using a flink version more than 1.14?
here are flink doc for feature 'ENABLE_CHECKPOINTS_AFTER_TASKS_FINISH' :
Checkpointing with parts of the graph finished [#](https://nightlies.apache.org/flink/flink-docs-release-1.18/docs/dev/datastream/fault-tolerance/checkpointing/#checkpointing-with-parts-of-the-graph-finished)
Starting from Flink 1.14 it is possible to continue performing checkpoints even if parts of the job graph have finished processing all data, which might happen if it contains bounded sources. This feature is enabled by default since 1.15, and it could be disabled via a feature flag:
Configuration config = new Configuration();
config.set(ExecutionCheckpointingOptions.ENABLE_CHECKPOINTS_AFTER_TASKS_FINISH, false);
StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment(config);
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] flink jdbc reading to hudi lost record for not enough checkpoints in flink stream api. [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on issue #10166:
URL: https://github.com/apache/hudi/issues/10166#issuecomment-1824141827
For streaming execution mode you are right, did you try batch execution mode?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] flink jdbc reading to hudi lost record for not enough checkpoints in flink stream api. [hudi]
Posted by "ad1happy2go (via GitHub)" <gi...@apache.org>.
ad1happy2go commented on issue #10166:
URL: https://github.com/apache/hudi/issues/10166#issuecomment-1919268092
@lschampion Any updates here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] flink jdbc reading to hudi lost record for not enough checkpoints in flink stream api. [hudi]
Posted by "lschampion (via GitHub)" <gi...@apache.org>.
lschampion closed issue #10166: [SUPPORT] flink jdbc reading to hudi lost record for not enough checkpoints in flink stream api.
URL: https://github.com/apache/hudi/issues/10166
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] flink jdbc reading to hudi lost record for not enough checkpoints in flink stream api. [hudi]
Posted by "lschampion (via GitHub)" <gi...@apache.org>.
lschampion commented on issue #10166:
URL: https://github.com/apache/hudi/issues/10166#issuecomment-2890096576
oh, i forgot the reason ,but the problem has be resolved, sorrry,@ad1happy2go
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org