You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "jiangxinqi1995 (via GitHub)" <gi...@apache.org> on 2023/03/23 11:17:15 UTC
[GitHub] [hudi] jiangxinqi1995 opened a new issue, #8276: [SUPPORT] Flink Exceeded checkpoint tolerable failure threshold.
jiangxinqi1995 opened a new issue, #8276:
URL: https://github.com/apache/hudi/issues/8276
**Describe the problem you faced**
A clear and concise description of the problem.
"I use Flink cdc to read MySQL data, and then write it to S3 through hudi. I often encounter checkpoint org.apache.Flink.util.FlinkRuntimeException: Exceeded checkpoint tolerable failure threshold." "The common problem is that a checkpoint failure occurs every 20 minutes. I have no problems running on a local machine, but when I go to an EKS cluster, this problem occurs."
**To Reproduce**
Steps to reproduce the behavior:
1.
2.
3.
4.
**Expected behavior**
A clear and concise description of what you expected to happen.
```
2023-03-23 10:23:07,101 INFO org.apache.hudi.sink.StreamWriteOperatorCoordinator [] - Executor executes action [handle write metadata event for instant 20230323101927464] success!
2023-03-23 10:23:07,817 INFO org.apache.flink.fs.s3.common.writer.S3Committer [] - Committing reject/savepoint-fbea13-af1aa20f0400/_metadata with MPU ID rqLctlP9RnUBjWedNLI1bXhNB32evfVDwi7T1nNz8Gd9gDzFtYDRm615A5MCQivzMav.9yAJeD_Tp36Yp52oHitnmMZ6BqYmJV9G.JcDXEco.czZfXHRzGisnIuVzO._qSlcCQaQpSW.Qx.X0ex3LQ--
2023-03-23 10:23:08,065 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Completed checkpoint 11 for job fbea139434199b0095f544fe5c15d25f (747958 bytes, checkpointDuration=6668 ms, finalizationTime=0 ms).
2023-03-23 10:24:06,082 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Triggering checkpoint 12 (type=CheckpointType{name='Checkpoint', sharingFilesStrategy=FORWARD_BACKWARD}) @ 1679567046082 for job fbea139434199b0095f544fe5c15d25f.
2023-03-23 10:24:06,765 INFO org.apache.hudi.sink.StreamWriteOperatorCoordinator [] - Executor executes action [taking checkpoint 12] success!
2023-03-23 10:34:06,083 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint 12 of job fbea139434199b0095f544fe5c15d25f expired before completing.
2023-03-23 10:34:06,083 WARN org.apache.flink.runtime.checkpoint.CheckpointFailureManager [] - Failed to trigger or complete checkpoint 12 for job fbea139434199b0095f544fe5c15d25f. (0 consecutive failed attempts so far)
org.apache.flink.runtime.checkpoint.CheckpointException: Checkpoint expired before completing.
at org.apache.flink.runtime.checkpoint.CheckpointCoordinator$CheckpointCanceller.run(CheckpointCoordinator.java:2143) [flink-dist-1.15.3.jar:1.15.3]
at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) [?:?]
at java.util.concurrent.FutureTask.run(Unknown Source) [?:?]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) [?:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) [?:?]
at java.lang.Thread.run(Unknown Source) [?:?]
2023-03-23 10:34:06,084 INFO org.apache.flink.runtime.jobmaster.JobMaster [] - Trying to recover from a global failure.
org.apache.flink.util.FlinkRuntimeException: Exceeded checkpoint tolerable failure threshold.
at org.apache.flink.runtime.checkpoint.CheckpointFailureManager.checkFailureAgainstCounter(CheckpointFailureManager.java:206) ~[flink-dist-1.15.3.jar:1.15.3]
at org.apache.flink.runtime.checkpoint.CheckpointFailureManager.handleJobLevelCheckpointException(CheckpointFailureManager.java:169) ~[flink-dist-1.15.3.jar:1.15.3]
at org.apache.flink.runtime.checkpoint.CheckpointFailureManager.handleCheckpointException(CheckpointFailureManager.java:122) ~[flink-dist-1.15.3.jar:1.15.3]
at org.apache.flink.runtime.checkpoint.CheckpointCoordinator.abortPendingCheckpoint(CheckpointCoordinator.java:2082) ~[flink-dist-1.15.3.jar:1.15.3]
at org.apache.flink.runtime.checkpoint.CheckpointCoordinator.abortPendingCheckpoint(CheckpointCoordinator.java:2061) ~[flink-dist-1.15.3.jar:1.15.3]
at org.apache.flink.runtime.checkpoint.CheckpointCoordinator.access$600(CheckpointCoordinator.java:98) ~[flink-dist-1.15.3.jar:1.15.3]
at org.apache.flink.runtime.checkpoint.CheckpointCoordinator$CheckpointCanceller.run(CheckpointCoordinator.java:2143) ~[flink-dist-1.15.3.jar:1.15.3]
at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) ~[?:?]
at java.util.concurrent.FutureTask.run(Unknown Source) ~[?:?]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) ~[?:?]
at java.lang.Thread.run(Unknown Source) ~[?:?]
```
**Environment Description**
* Hudi version : 0.13.0
* Flink version：1.15.3
* Hadoop version : 3.2.1
* Storage (HDFS/S3/GCS..) : S3
* Running on Docker? (yes/no) :yes ,EKS
**Additional context**
```
connector = hudi,
table.type = MERGE_ON_READ,
hoodie.clean.async=true,
hoodie.compact.inline= true,
hoodie.compact.inline.max.delta.commits=2,
hoodie.clean.max.commits=2,
hoodie.cleaner.commits.retained = 3,
hoodie.cleaner.policy = KEEP_LATEST_COMMITS,
hoodie.parquet.small.file.limit=104857600,
clustering.schedule.enabled=true,
clustering.async.enabled=true,
hoodie.clustering.inline= true,
hoodie.clustering.inline.max.commits= 2,
hoodie.clustering.plan.strategy.max.bytes.per.group= 107374182400,
hoodie.clustering.plan.strategy.max.num.groups= 1,
hoodie.datasource.write.recordkey.field = id,installs,rejects,
path = s3a://xxxxxxxx/xxxxxxx
```
Add any other context about the problem here.
**Stacktrace**
```Add the stacktrace of the error.```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] jarrodcodes commented on issue #8276: [SUPPORT] Flink Exceeded checkpoint tolerable failure threshold.
Posted by "jarrodcodes (via GitHub)" <gi...@apache.org>.
jarrodcodes commented on issue #8276:
URL: https://github.com/apache/hudi/issues/8276#issuecomment-1575226550
We are experiencing this issue as well. We're also on Flink 1.15. Just curious if you've tried a newer version of Flink?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] Flink Exceeded checkpoint tolerable failure threshold. [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on issue #8276:
URL: https://github.com/apache/hudi/issues/8276#issuecomment-2586089650
@amiru3f Did you check the root cause that triggered the checkpoint failure?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on issue #8276: [SUPPORT] Flink Exceeded checkpoint tolerable failure threshold.
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on issue #8276:
URL: https://github.com/apache/hudi/issues/8276#issuecomment-1515653560
Nice findings, can you help to dig into the root cause @gfunc ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] gfunc commented on issue #8276: [SUPPORT] Flink Exceeded checkpoint tolerable failure threshold.
Posted by "gfunc (via GitHub)" <gi...@apache.org>.
gfunc commented on issue #8276:
URL: https://github.com/apache/hudi/issues/8276#issuecomment-1575957699
@danny0405 Sorry for any confusion, I meant the above-mentioned specific scenario: the first checkpoint after savepoint. Normal checkpoint is relatively slow but ok since we had a bad partition key making the number of partitions a bit too many for this small setup.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] Flink Exceeded checkpoint tolerable failure threshold. [hudi]
Posted by "JackrayWang2020 (via GitHub)" <gi...@apache.org>.
JackrayWang2020 commented on issue #8276:
URL: https://github.com/apache/hudi/issues/8276#issuecomment-2607299939
I encountered the same issue, but I didn’t see how you all resolved it. Could you give me some hints? Thank you very much!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on issue #8276: [SUPPORT] Flink Exceeded checkpoint tolerable failure threshold.
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on issue #8276:
URL: https://github.com/apache/hudi/issues/8276#issuecomment-1575962096
> the first checkpoint after savepoint
Got it, I need to figure out what's the savepoint's effect to the ensueing checkpoint.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] Flink Exceeded checkpoint tolerable failure threshold. [hudi]
Posted by "cshuo (via GitHub)" <gi...@apache.org>.
cshuo commented on issue #8276:
URL: https://github.com/apache/hudi/issues/8276#issuecomment-3044064936
Hi all, I've reproduced the problem locally, and figured out the root cause:
1) the `stuck` happens in `instantToWrite` method w/ write function, where write tasks waiting for coordinator generating a new instant to write until timeout.
2) Before https://github.com/apache/hudi/pull/13285, flink write coordinator generates instant in synchronous mode, i.e., new instant will be generated until previous instant is committed.
3) For flink ingestion, commiting happens during `notifyCheckpointComplete` in write coordinator.
4) `notifyCheckpointComplete` will not be called for Savepoint.
So when Savepoint is called asynchronously without stopping job, commit will not happen and new instant will not be generated, which leads to job stuck until checkpoint timeout exception.
The possible solution is:
1) Using https://github.com/apache/hudi/pull/13285 as @danny0405 mentioned above, which is introduced in release 1.1.
2) Disable asynchronous savepoint or using 'Stop/Resume job with savepoint'.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] jiangxinqi1995 commented on issue #8276: [SUPPORT] Flink Exceeded checkpoint tolerable failure threshold.
Posted by "jiangxinqi1995 (via GitHub)" <gi...@apache.org>.
jiangxinqi1995 commented on issue #8276:
URL: https://github.com/apache/hudi/issues/8276#issuecomment-1482297437
Every time a Flink task is restarted, every two checkpoints are performed, and the next checkpoint will fail. This phenomenon is very strange.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] jiangxinqi1995 commented on issue #8276: [SUPPORT] Flink Exceeded checkpoint tolerable failure threshold.
Posted by "jiangxinqi1995 (via GitHub)" <gi...@apache.org>.
jiangxinqi1995 commented on issue #8276:
URL: https://github.com/apache/hudi/issues/8276#issuecomment-1486350462
Flink job is deployed in AWS EKS，with a jobmanager and a taskmanager
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] jiangxinqi1995 commented on issue #8276: [SUPPORT] Flink Exceeded checkpoint tolerable failure threshold.
Posted by "jiangxinqi1995 (via GitHub)" <gi...@apache.org>.
jiangxinqi1995 commented on issue #8276:
URL: https://github.com/apache/hudi/issues/8276#issuecomment-1486324135
Once successful, a parquet file was generated, and after a period of time, a checkpoint exception occurred. Thereafter, no parquet file was generated, only log files were written, accompanied by checkpoint failures.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on issue #8276: [SUPPORT] Flink Exceeded checkpoint tolerable failure threshold.
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on issue #8276:
URL: https://github.com/apache/hudi/issues/8276#issuecomment-1482178977
Did you see any error stack trace in the logging then, what time interval the ckp triggers with?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] jiangxinqi1995 commented on issue #8276: [SUPPORT] Flink Exceeded checkpoint tolerable failure threshold.
Posted by "jiangxinqi1995 (via GitHub)" <gi...@apache.org>.
jiangxinqi1995 commented on issue #8276:
URL: https://github.com/apache/hudi/issues/8276#issuecomment-1482283168
Partitions by day, only triggers the current day's partition each time
``
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on issue #8276: [SUPPORT] Flink Exceeded checkpoint tolerable failure threshold.
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on issue #8276:
URL: https://github.com/apache/hudi/issues/8276#issuecomment-1486236095
> It may be an environmental issue because it always performs a compression operation and then never performs a compression operation again
Did the compaction succeed or failed ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on issue #8276: [SUPPORT] Flink Exceeded checkpoint tolerable failure threshold.
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on issue #8276:
URL: https://github.com/apache/hudi/issues/8276#issuecomment-1486376799
I see, I guess it is because of the resource, did you try allocating more resource to the job here, the slot/memory/parallelism.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] gfunc commented on issue #8276: [SUPPORT] Flink Exceeded checkpoint tolerable failure threshold.
Posted by "gfunc (via GitHub)" <gi...@apache.org>.
gfunc commented on issue #8276:
URL: https://github.com/apache/hudi/issues/8276#issuecomment-1514275619
I am facing the same problem, but I am still curious about the reason behind it.
I had a pipeline running for days ATM, and the checkpoint was triggered by the k8s operator every 6 hours. And every checkpoint after that would result in failure: `Checkpoint expired before completing`. Tunning up resources could resolve the issue.
Could someone elaborate on what exactly is different about the process after Checkpoint?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] jiangxinqi1995 commented on issue #8276: [SUPPORT] Flink Exceeded checkpoint tolerable failure threshold.
Posted by "jiangxinqi1995 (via GitHub)" <gi...@apache.org>.
jiangxinqi1995 commented on issue #8276:
URL: https://github.com/apache/hudi/issues/8276#issuecomment-1482293819
I don't know why. At first, it is possible to synthesize Parquet files, but after synthesizing two Parquet files, subsequent files will not be merged, and there will be errors such as org. apache. link. runtime. checkpoint. CheckpointException: Checkpoint expired before completing or Exceeded checkpoint tolerable failure threshold.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] jiangxinqi1995 commented on issue #8276: [SUPPORT] Flink Exceeded checkpoint tolerable failure threshold.
Posted by "jiangxinqi1995 (via GitHub)" <gi...@apache.org>.
jiangxinqi1995 commented on issue #8276:
URL: https://github.com/apache/hudi/issues/8276#issuecomment-1486359847
Flink images are self built，this is my Dockerfile
```
FROM flink:1.15.3-scala_2.12-java11
WORKDIR /opt/flink
RUN wget -q -O /opt/flink/lib/hudi-flink1.15-bundle-0.13.0.jar https://repo.maven.apache.org/maven2/org/apache/hudi/hudi-flink1.15-bundle/0.13.0/hudi-flink1.15-bundle-0.13.0.jar && \
wget -q -O /opt/flink/lib/hadoop-aws-3.2.1.jar https://repo1.maven.org/maven2/org/apache/hadoop/hadoop-aws/3.2.1/hadoop-aws-3.2.1.jar && \
wget -q -O /opt/flink/lib/aws-java-sdk-bundle-1.11.874.jar https://repo1.maven.org/maven2/com/amazonaws/aws-java-sdk-bundle/1.11.874/aws-java-sdk-bundle-1.11.874.jar && \
wget -q -O /opt/flink/lib/mysql-connector-java-8.0.27.jar https://repo1.maven.org/maven2/mysql/mysql-connector-java/8.0.27/mysql-connector-java-8.0.27.jar && \
wget -q -O /opt/flink/lib/flink-connector-jdbc-1.15.3.jar https://repo1.maven.org/maven2/org/apache/flink/flink-connector-jdbc/1.15.3/flink-connector-jdbc-1.15.3.jar && \
mkdir /opt/flink/plugins/s3-fs-hadoop && \
cp /opt/flink/opt/flink-s3-fs-hadoop-1.15.3.jar /opt/flink/plugins/s3-fs-hadoop/ && \
wget -q -O /opt/hadoop-3.2.1.tar.gz https://archive.apache.org/dist/hadoop/common/hadoop-3.2.1/hadoop-3.2.1.tar.gz && \
tar -zxvf /opt/hadoop-3.2.1.tar.gz -C /opt/ && \
rm /opt/hadoop-3.2.1.tar.gz
ENV HADOOP_HOME=/opt/hadoop-3.2.1
ENV HADOOP_COMMON_HOME=$HADOOP_HOME
ENV PATH=$PATH:$HADOOP_HOME/bin
ENV HADOOP_CLASSPATH=${HADOOP_HOME}/etc/hadoop:${HADOOP_HOME}/share/hadoop/common/lib/*:${HADOOP_HOME}/share/hadoop/common/*:${HADOOP_HOME}/share/hadoop/hdfs:${HADOOP_HOME}/share/hadoop/hdfs/lib/*:${HADOOP_HOME}/share/hadoop/hdfs/*:${HADOOP_HOME}/share/hadoop/mapreduce/lib/*:${HADOOP_HOME}/share/hadoop/mapreduce/*:${HADOOP_HOME}/share/hadoop/yarn:${HADOOP_HOME}/share/hadoop/yarn/lib/*:${HADOOP_HOME}/share/hadoop/yarn/*
ENV CLASSPATH=$CLASSPATH:$HADOOP_CLASSPATH
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] Flink Exceeded checkpoint tolerable failure threshold. [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on issue #8276:
URL: https://github.com/apache/hudi/issues/8276#issuecomment-3043394296
By guess is `#savepoint` actually triggers `#endInput` instead of `#snapshotState` of the function, so that the `coordinator` will not instantiate the new instant after the current instant is committed. Thus the write functions will just hang in there until timeout.
This issue should be addressed since commit: https://github.com/apache/hudi/pull/13285, because we trigger the instant time generation from the writer function instead within the coordinator checkpoint ack event handling.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] Flink Exceeded checkpoint tolerable failure threshold. [hudi]
Posted by "leilinen (via GitHub)" <gi...@apache.org>.
leilinen commented on issue #8276:
URL: https://github.com/apache/hudi/issues/8276#issuecomment-3041105196
I faced the same problem for checkpoint tolerable failure exception after savepoint triggered. he fix in PR #6cda35c appears unavailable in any branch. How can I resolve this issue?"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] Flink Exceeded checkpoint tolerable failure threshold. [hudi]
Posted by "sin668 (via GitHub)" <gi...@apache.org>.
sin668 commented on issue #8276:
URL: https://github.com/apache/hudi/issues/8276#issuecomment-2006093805
> I think it's because I set the automatic trigger for the flink savepoint, which causes checkpoint failures after each savepoint. I removed the automatic trigger for the savepoint, and now everything is normal. I think it's because of this problem, but I don't understand why this situation occurs. Has it happened before? Thank you. @danny0405 @c-f-cooper
I am facing the same problem, I don't know how to remove the automatic trigger for the savepoint, Can you help to show detail? @jiangxinqi1995
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] jiangxinqi1995 commented on issue #8276: [SUPPORT] Flink Exceeded checkpoint tolerable failure threshold.
Posted by "jiangxinqi1995 (via GitHub)" <gi...@apache.org>.
jiangxinqi1995 commented on issue #8276:
URL: https://github.com/apache/hudi/issues/8276#issuecomment-1489605250
I think it's because I set the automatic trigger for the flink savepoint, which causes checkpoint failures after each savepoint. I removed the automatic trigger for the savepoint, and now everything is normal. I think it's because of this problem, but I don't understand why this situation occurs. Has it happened before? Thank you. @danny0405 @c-f-cooper
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on issue #8276: [SUPPORT] Flink Exceeded checkpoint tolerable failure threshold.
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on issue #8276:
URL: https://github.com/apache/hudi/issues/8276#issuecomment-1486338021
What kind of env did you run the Flink job on?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on issue #8276: [SUPPORT] Flink Exceeded checkpoint tolerable failure threshold.
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on issue #8276:
URL: https://github.com/apache/hudi/issues/8276#issuecomment-1482217420
Your ckp options seem good, what is your partition path field then? How many partitions do you estimate that can be touched for one ckp write operation?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] c-f-cooper commented on issue #8276: [SUPPORT] Flink Exceeded checkpoint tolerable failure threshold.
Posted by "c-f-cooper (via GitHub)" <gi...@apache.org>.
c-f-cooper commented on issue #8276:
URL: https://github.com/apache/hudi/issues/8276#issuecomment-1486662589
It's sound like the ckp interval ,try to decrease the ckp interval!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] jiangxinqi1995 commented on issue #8276: [SUPPORT] Flink Exceeded checkpoint tolerable failure threshold.
Posted by "jiangxinqi1995 (via GitHub)" <gi...@apache.org>.
jiangxinqi1995 commented on issue #8276:
URL: https://github.com/apache/hudi/issues/8276#issuecomment-1491196499
I did a test, and there was a 10 minute interval between savepoint and checkpoint. After triggering savepoint, the checkpoint still failed because the resources were sufficient at that time
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on issue #8276: [SUPPORT] Flink Exceeded checkpoint tolerable failure threshold.
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on issue #8276:
URL: https://github.com/apache/hudi/issues/8276#issuecomment-1575952897
> my example job was always stuck at the sink part
@gfunc Is it stuck for regular writing or when the auto savepoint kicks in?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] Flink Exceeded checkpoint tolerable failure threshold. [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on issue #8276:
URL: https://github.com/apache/hudi/issues/8276#issuecomment-3043240190
This is by design for 0.x release, the error indicates the data flush in one checkpoint takes much longer time than the checkpoint timeout param, this issue has been completely resolved in 1.1 relase: https://github.com/apache/hudi/pull/13285
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] gfunc commented on issue #8276: [SUPPORT] Flink Exceeded checkpoint tolerable failure threshold.
Posted by "gfunc (via GitHub)" <gi...@apache.org>.
gfunc commented on issue #8276:
URL: https://github.com/apache/hudi/issues/8276#issuecomment-1575918513
I am with Flink 1.16 and Hudi 0.13. I tried looking into the code but was not able to identify the exact task that was responsible (my example job was always stuck at the sink part).
Since we were using S3 for checkpoint backend, we turned off auto savepoint and moved on to test other features, hoping that maybe async Hudi job execution (cleaning, etc.) would help in pinpointing the issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on issue #8276: [SUPPORT] Flink Exceeded checkpoint tolerable failure threshold.
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on issue #8276:
URL: https://github.com/apache/hudi/issues/8276#issuecomment-1489973129
Hav't learnt about the details of auto savepoint, guess it's caused by the preemption of resources, both the compaction and savepoint have rigorous requirements for resource.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] Flink Exceeded checkpoint tolerable failure threshold. [hudi]
Posted by "sin668 (via GitHub)" <gi...@apache.org>.
sin668 commented on issue #8276:
URL: https://github.com/apache/hudi/issues/8276#issuecomment-2006087060
> I did a test, and there was a 10 minute interval between savepoint and checkpoint. After triggering savepoint, the checkpoint still failed because the resources were sufficient at that time 
I am facing the same problem, But I don't know How to remove the automatic trigger for the savepoint, Can you help to show detail? @jiangxinqi1995
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] jiangxinqi1995 commented on issue #8276: [SUPPORT] Flink Exceeded checkpoint tolerable failure threshold.
Posted by "jiangxinqi1995 (via GitHub)" <gi...@apache.org>.
jiangxinqi1995 commented on issue #8276:
URL: https://github.com/apache/hudi/issues/8276#issuecomment-1482196417
This is my Flink checkpoint configuration. I don't see any other useful log information
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on issue #8276: [SUPPORT] Flink Exceeded checkpoint tolerable failure threshold.
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on issue #8276:
URL: https://github.com/apache/hudi/issues/8276#issuecomment-1482369331
Sonds like a env problem or record payload issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] jiangxinqi1995 commented on issue #8276: [SUPPORT] Flink Exceeded checkpoint tolerable failure threshold.
Posted by "jiangxinqi1995 (via GitHub)" <gi...@apache.org>.
jiangxinqi1995 commented on issue #8276:
URL: https://github.com/apache/hudi/issues/8276#issuecomment-1486118862
It may be an environmental issue because it always performs a compression operation and then never performs a compression operation again
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] Flink Exceeded checkpoint tolerable failure threshold. [hudi]
Posted by "amiru3f (via GitHub)" <gi...@apache.org>.
amiru3f commented on issue #8276:
URL: https://github.com/apache/hudi/issues/8276#issuecomment-2585356794
Facing exactly the same problem, @danny0405 haven't you managed to find out the root cause of this yet?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org