You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "manuzhang (via GitHub)" <gi...@apache.org> on 2024/05/23 11:28:07 UTC
[PR] Flink 1.19: Fix flaky `TestIcebergSourceFailover > testBoundedWithSavepoint` [iceberg]
manuzhang opened a new pull request, #10370:
URL: https://github.com/apache/iceberg/pull/10370
(no comment)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink 1.19: Fix flaky `TestIcebergSourceFailover > testBoundedWithSavepoint` [iceberg]
Posted by "stevenzwu (via GitHub)" <gi...@apache.org>.
stevenzwu commented on code in PR #10370:
URL: https://github.com/apache/iceberg/pull/10370#discussion_r1612760150
##########
flink/v1.19/flink/src/test/java/org/apache/iceberg/flink/source/TestIcebergSourceFailover.java:
##########
@@ -124,7 +124,7 @@ public void testBoundedWithSavepoint() throws Exception {
GenericAppenderHelper dataAppender =
new GenericAppenderHelper(
sourceTableResource.table(), FileFormat.PARQUET, TEMPORARY_FOLDER);
- for (int i = 0; i < 4; ++i) {
+ for (int i = 0; i < 20; ++i) {
Review Comment:
> but if one TM is slow to start, then other TMs might finish some splits, and get a new one.
I remember Flink open the operator/tasks in the backward fashion. When the source tasks are ready, all other tasks are in running state and ready to process.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink 1.19: Fix flaky `TestIcebergSourceFailover > testBoundedWithSavepoint` [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #10370:
URL: https://github.com/apache/iceberg/pull/10370#discussion_r1613105037
##########
flink/v1.19/flink/src/test/java/org/apache/iceberg/flink/source/TestIcebergSourceFailover.java:
##########
@@ -124,7 +124,7 @@ public void testBoundedWithSavepoint() throws Exception {
GenericAppenderHelper dataAppender =
new GenericAppenderHelper(
sourceTableResource.table(), FileFormat.PARQUET, TEMPORARY_FOLDER);
- for (int i = 0; i < 4; ++i) {
+ for (int i = 0; i < 20; ++i) {
Review Comment:
I was able to reproduce the issue locally with breakpoints.
I put a breakpoint to `IcebergStreamWriter.open` method: https://github.com/apache/iceberg/blob/fbcd142c5dc1ec99792ef8edc1378e3a027fecf7/flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/sink/IcebergStreamWriter.java#L53
I started the test, and only allowed to continue one of the writer threads, and kept the other blocked.
After a while I have seen that the job is cancelled:
```
switched from CANCELING to CANCELED.
```
At this point I have released the other breakpoint as well, and as a result, I have got the same exception as in https://github.com/apache/iceberg/issues/10356
Not sure about the fix yet...
I got a different exception when I put a breakpoint to [here](https://github.com/apache/iceberg/blob/b7d3a7f9d33aae2a85f9a742ed1ab87608283d28/flink/v1.19/flink/src/test/java/org/apache/iceberg/flink/source/TestIcebergSourceFailover.java#L141)
```
[flink-pekko.remote.default-remote-dispatcher-7] INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Triggering Checkpoint 1 for job 1f429c228963a080189d7cb2779db020 failed due to {}
org.apache.flink.runtime.checkpoint.CheckpointException: TaskManager received a checkpoint request for unknown task 6b830cecd3bbc9134fd9fcb2c832aa2b_cbc357ccb763df2852fee8c4fc7d55f2_1_0. Failure reason: Task local checkpoint failure.
at org.apache.flink.runtime.taskexecutor.TaskExecutor.triggerCheckpoint(TaskExecutor.java:1061)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.apache.flink.runtime.rpc.pekko.PekkoRpcActor.lambda$handleRpcInvocation$1(PekkoRpcActor.java:309)
at org.apache.flink.runtime.concurrent.ClassLoadingUtils.runWithContextClassLoader(ClassLoadingUtils.java:83)
at org.apache.flink.runtime.rpc.pekko.PekkoRpcActor.handleRpcInvocation(PekkoRpcActor.java:307)
at org.apache.flink.runtime.rpc.pekko.PekkoRpcActor.handleRpcMessage(PekkoRpcActor.java:222)
at org.apache.flink.runtime.rpc.pekko.PekkoRpcActor.handleMessage(PekkoRpcActor.java:168)
at org.apache.pekko.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:33)
at org.apache.pekko.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:29)
at scala.PartialFunction.applyOrElse(PartialFunction.scala:127)
at scala.PartialFunction.applyOrElse$(PartialFunction.scala:126)
at org.apache.pekko.japi.pf.UnitCaseStatement.applyOrElse(CaseStatements.scala:29)
at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:175)
at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:176)
at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:176)
at org.apache.pekko.actor.Actor.aroundReceive(Actor.scala:547)
at org.apache.pekko.actor.Actor.aroundReceive$(Actor.scala:545)
at org.apache.pekko.actor.AbstractActor.aroundReceive(AbstractActor.scala:229)
at org.apache.pekko.actor.ActorCell.receiveMessage(ActorCell.scala:590)
at org.apache.pekko.actor.ActorCell.invoke(ActorCell.scala:557)
at org.apache.pekko.dispatch.Mailbox.processMailbox(Mailbox.scala:280)
at org.apache.pekko.dispatch.Mailbox.run(Mailbox.scala:241)
at org.apache.pekko.dispatch.Mailbox.exec(Mailbox.scala:253)
at java.base/java.util.concurrent.ForkJoinTask.doExec$$$capture(ForkJoinTask.java:290)
at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java)
at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1020)
at java.base/java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1656)
at java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1594)
at java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:183)
[Checkpoint Timer] WARN org.apache.flink.runtime.checkpoint.CheckpointFailureManager - Failed to trigger or complete checkpoint 1 for job 1f429c228963a080189d7cb2779db020. (0 consecutive failed attempts so far)
org.apache.flink.runtime.checkpoint.CheckpointException: TaskManager received a checkpoint request for unknown task 6b830cecd3bbc9134fd9fcb2c832aa2b_cbc357ccb763df2852fee8c4fc7d55f2_1_0. Failure reason: Task local checkpoint failure.
at org.apache.flink.runtime.taskexecutor.TaskExecutor.triggerCheckpoint(TaskExecutor.java:1061)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.apache.flink.runtime.rpc.pekko.PekkoRpcActor.lambda$handleRpcInvocation$1(PekkoRpcActor.java:309)
at org.apache.flink.runtime.concurrent.ClassLoadingUtils.runWithContextClassLoader(ClassLoadingUtils.java:83)
at org.apache.flink.runtime.rpc.pekko.PekkoRpcActor.handleRpcInvocation(PekkoRpcActor.java:307)
at org.apache.flink.runtime.rpc.pekko.PekkoRpcActor.handleRpcMessage(PekkoRpcActor.java:222)
at org.apache.flink.runtime.rpc.pekko.PekkoRpcActor.handleMessage(PekkoRpcActor.java:168)
at org.apache.pekko.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:33)
at org.apache.pekko.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:29)
at scala.PartialFunction.applyOrElse(PartialFunction.scala:127)
at scala.PartialFunction.applyOrElse$(PartialFunction.scala:126)
at org.apache.pekko.japi.pf.UnitCaseStatement.applyOrElse(CaseStatements.scala:29)
at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:175)
at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:176)
at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:176)
at org.apache.pekko.actor.Actor.aroundReceive(Actor.scala:547)
at org.apache.pekko.actor.Actor.aroundReceive$(Actor.scala:545)
at org.apache.pekko.actor.AbstractActor.aroundReceive(AbstractActor.scala:229)
at org.apache.pekko.actor.ActorCell.receiveMessage(ActorCell.scala:590)
at org.apache.pekko.actor.ActorCell.invoke(ActorCell.scala:557)
at org.apache.pekko.dispatch.Mailbox.processMailbox(Mailbox.scala:280)
at org.apache.pekko.dispatch.Mailbox.run(Mailbox.scala:241)
at org.apache.pekko.dispatch.Mailbox.exec(Mailbox.scala:253)
at java.base/java.util.concurrent.ForkJoinTask.doExec$$$capture(ForkJoinTask.java:290)
at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java)
at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1020)
at java.base/java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1656)
at java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1594)
at java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:183)
[main] INFO org.apache.flink.test.util.MiniClusterWithClientResource - Finalization triggered: Cluster shutdown is going to be initiated.
[SourceCoordinator-Source: IcebergSource] INFO org.apache.flink.runtime.source.coordinator.SourceCoordinator - Marking checkpoint 1 as aborted for source Source: IcebergSource.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink 1.19: Fix flaky `TestIcebergSourceFailover > testBoundedWithSavepoint` [iceberg]
Posted by "manuzhang (via GitHub)" <gi...@apache.org>.
manuzhang commented on code in PR #10370:
URL: https://github.com/apache/iceberg/pull/10370#discussion_r1613284723
##########
flink/v1.19/flink/src/test/java/org/apache/iceberg/flink/source/TestIcebergSourceFailover.java:
##########
@@ -124,7 +124,7 @@ public void testBoundedWithSavepoint() throws Exception {
GenericAppenderHelper dataAppender =
new GenericAppenderHelper(
sourceTableResource.table(), FileFormat.PARQUET, TEMPORARY_FOLDER);
- for (int i = 0; i < 4; ++i) {
+ for (int i = 0; i < 20; ++i) {
Review Comment:
I think the fix is apparently to keep the tasks running, but not sure how to achieve it consistently.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink 1.19: Fix flaky `TestIcebergSourceFailover > testBoundedWithSavepoint` [iceberg]
Posted by "manuzhang (via GitHub)" <gi...@apache.org>.
manuzhang closed pull request #10370: Flink 1.19: Fix flaky `TestIcebergSourceFailover > testBoundedWithSavepoint`
URL: https://github.com/apache/iceberg/pull/10370
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink 1.19: Fix flaky `TestIcebergSourceFailover > testBoundedWithSavepoint` [iceberg]
Posted by "manuzhang (via GitHub)" <gi...@apache.org>.
manuzhang commented on code in PR #10370:
URL: https://github.com/apache/iceberg/pull/10370#discussion_r1612975023
##########
flink/v1.19/flink/src/test/java/org/apache/iceberg/flink/source/TestIcebergSourceFailover.java:
##########
@@ -124,7 +124,7 @@ public void testBoundedWithSavepoint() throws Exception {
GenericAppenderHelper dataAppender =
new GenericAppenderHelper(
sourceTableResource.table(), FileFormat.PARQUET, TEMPORARY_FOLDER);
- for (int i = 0; i < 4; ++i) {
+ for (int i = 0; i < 20; ++i) {
Review Comment:
From [where it failed](https://github.com/apache/flink/blob/release-1.19.0/flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/DefaultCheckpointPlanCalculator.java#L105), I think all tasks have started, but some have finished and some are still running when a savepoint is triggered. That's why I try increasing number of records to keep all tasks running.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink 1.19: Fix flaky `TestIcebergSourceFailover > testBoundedWithSavepoint` [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #10370:
URL: https://github.com/apache/iceberg/pull/10370#discussion_r1612213520
##########
flink/v1.19/flink/src/test/java/org/apache/iceberg/flink/source/TestIcebergSourceFailover.java:
##########
@@ -124,7 +124,7 @@ public void testBoundedWithSavepoint() throws Exception {
GenericAppenderHelper dataAppender =
new GenericAppenderHelper(
sourceTableResource.table(), FileFormat.PARQUET, TEMPORARY_FOLDER);
- for (int i = 0; i < 4; ++i) {
+ for (int i = 0; i < 20; ++i) {
Review Comment:
Not sure that this is the same issue that I have found before, but if one TM is slow to start, then other TMs might finish some splits, and get a new one. So when the slow starting TM is finally ready, the there are no more splits available. So it could immediately finish (no checkpoint in this case)
If I remember correctly, there is some way on the minicluster to wait until all of the TMs are started. Which could help, but there are no theoretical guarantees that the checkpoint could finish, even in this case.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink 1.19: Fix flaky `TestIcebergSourceFailover > testBoundedWithSavepoint` [iceberg]
Posted by "stevenzwu (via GitHub)" <gi...@apache.org>.
stevenzwu commented on code in PR #10370:
URL: https://github.com/apache/iceberg/pull/10370#discussion_r1613698660
##########
flink/v1.19/flink/src/test/java/org/apache/iceberg/flink/source/TestIcebergSourceFailover.java:
##########
@@ -124,7 +124,7 @@ public void testBoundedWithSavepoint() throws Exception {
GenericAppenderHelper dataAppender =
new GenericAppenderHelper(
sourceTableResource.table(), FileFormat.PARQUET, TEMPORARY_FOLDER);
- for (int i = 0; i < 4; ++i) {
+ for (int i = 0; i < 20; ++i) {
Review Comment:
> but what happens if you try to create a savepoint when some of the tasks are not started yet?
I see. so the savepoint is triggered before all tasks in running state. that would explain the problem/exception.
If we want to keep this test, we need to achieve these
- make sure the bounded test doesn't complete before a checkpoint/savepoint is triggered
- make sure all tasks are in running state before a checkpoint/savepoint is triggered
I am always fine with disabling a flaky test first until we figure out a proper solution.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink 1.19: Fix flaky `TestIcebergSourceFailover > testBoundedWithSavepoint` [iceberg]
Posted by "stevenzwu (via GitHub)" <gi...@apache.org>.
stevenzwu commented on code in PR #10370:
URL: https://github.com/apache/iceberg/pull/10370#discussion_r1611949141
##########
flink/v1.19/flink/src/test/java/org/apache/iceberg/flink/source/TestIcebergSourceFailover.java:
##########
@@ -124,7 +124,7 @@ public void testBoundedWithSavepoint() throws Exception {
GenericAppenderHelper dataAppender =
new GenericAppenderHelper(
sourceTableResource.table(), FileFormat.PARQUET, TEMPORARY_FOLDER);
- for (int i = 0; i < 4; ++i) {
+ for (int i = 0; i < 20; ++i) {
Review Comment:
from the stack trace that you included in the GitHub issue, it seems that some source tasks finished before the savepoint is triggered. old code should produce 4 files (each with 2 records). I think the real problem is probably all files was bundled in one `CombinedScanTask`. As a result, one subtask is idle and finished early. this probably won't fix the flakiness. we need to change the target split size to 1 byte to avoid bundling.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink 1.19: Fix flaky `TestIcebergSourceFailover > testBoundedWithSavepoint` [iceberg]
Posted by "stevenzwu (via GitHub)" <gi...@apache.org>.
stevenzwu commented on code in PR #10370:
URL: https://github.com/apache/iceberg/pull/10370#discussion_r1612756537
##########
flink/v1.19/flink/src/test/java/org/apache/iceberg/flink/source/TestIcebergSourceFailover.java:
##########
@@ -124,7 +124,7 @@ public void testBoundedWithSavepoint() throws Exception {
GenericAppenderHelper dataAppender =
new GenericAppenderHelper(
sourceTableResource.table(), FileFormat.PARQUET, TEMPORARY_FOLDER);
- for (int i = 0; i < 4; ++i) {
+ for (int i = 0; i < 20; ++i) {
Review Comment:
checking this test method again. we are testing savepoint trigger with bounded source. it is possible that splits are exhausted and source operator reached finished state before the savepoint was triggered.
But Flink has this feature to support checkpoint even if part of the graph finished. If that is the case, savepoint should still succeed.
https://nightlies.apache.org/flink/flink-docs-master/docs/dev/datastream/fault-tolerance/checkpointing/#checkpointing-with-parts-of-the-graph-finished
We are probably still missing sth
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink 1.19: Fix flaky `TestIcebergSourceFailover > testBoundedWithSavepoint` [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #10370:
URL: https://github.com/apache/iceberg/pull/10370#discussion_r1617641116
##########
flink/v1.19/flink/src/test/java/org/apache/iceberg/flink/source/TestIcebergSourceFailover.java:
##########
@@ -124,7 +124,7 @@ public void testBoundedWithSavepoint() throws Exception {
GenericAppenderHelper dataAppender =
new GenericAppenderHelper(
sourceTableResource.table(), FileFormat.PARQUET, TEMPORARY_FOLDER);
- for (int i = 0; i < 4; ++i) {
+ for (int i = 0; i < 20; ++i) {
Review Comment:
I think we could solve this by using `CountDownLatch` to wait for instead of a single lock. Still testing, but looks promising
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink 1.19: Fix flaky `TestIcebergSourceFailover > testBoundedWithSavepoint` [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on PR #10370:
URL: https://github.com/apache/iceberg/pull/10370#issuecomment-2136744665
Created #10393 with the `CountDownLatch` solution
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink 1.19: Fix flaky `TestIcebergSourceFailover > testBoundedWithSavepoint` [iceberg]
Posted by "manuzhang (via GitHub)" <gi...@apache.org>.
manuzhang commented on PR #10370:
URL: https://github.com/apache/iceberg/pull/10370#issuecomment-2136807372
Superseded by #10393
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink 1.19: Fix flaky `TestIcebergSourceFailover > testBoundedWithSavepoint` [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #10370:
URL: https://github.com/apache/iceberg/pull/10370#discussion_r1612804231
##########
flink/v1.19/flink/src/test/java/org/apache/iceberg/flink/source/TestIcebergSourceFailover.java:
##########
@@ -124,7 +124,7 @@ public void testBoundedWithSavepoint() throws Exception {
GenericAppenderHelper dataAppender =
new GenericAppenderHelper(
sourceTableResource.table(), FileFormat.PARQUET, TEMPORARY_FOLDER);
- for (int i = 0; i < 4; ++i) {
+ for (int i = 0; i < 20; ++i) {
Review Comment:
You can create a savepoint if some of the tasks are finished, but what happens if you try to create a savepoint when some of the tasks are not started yet?
Does the savepoint fail, or just wait until the tasks are started?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink 1.19: Fix flaky `TestIcebergSourceFailover > testBoundedWithSavepoint` [iceberg]
Posted by "stevenzwu (via GitHub)" <gi...@apache.org>.
stevenzwu commented on code in PR #10370:
URL: https://github.com/apache/iceberg/pull/10370#discussion_r1611949141
##########
flink/v1.19/flink/src/test/java/org/apache/iceberg/flink/source/TestIcebergSourceFailover.java:
##########
@@ -124,7 +124,7 @@ public void testBoundedWithSavepoint() throws Exception {
GenericAppenderHelper dataAppender =
new GenericAppenderHelper(
sourceTableResource.table(), FileFormat.PARQUET, TEMPORARY_FOLDER);
- for (int i = 0; i < 4; ++i) {
+ for (int i = 0; i < 20; ++i) {
Review Comment:
from the stack trace that you included in the GitHub issue, it seems that some source tasks finished before the savepoint is triggered. old code should produce 4 files (each with 2 records). I think the real problem is probably all files was bundled in one `CombinedScanTask`. As a result, one subtask is idle and finished early.
this will not fix the flakiness. we need to change the target split size to 1 byte to avoid bundling.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink 1.19: Fix flaky `TestIcebergSourceFailover > testBoundedWithSavepoint` [iceberg]
Posted by "manuzhang (via GitHub)" <gi...@apache.org>.
manuzhang closed pull request #10370: Flink 1.19: Fix flaky `TestIcebergSourceFailover > testBoundedWithSavepoint`
URL: https://github.com/apache/iceberg/pull/10370
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink 1.19: Fix flaky `TestIcebergSourceFailover > testBoundedWithSavepoint` [iceberg]
Posted by "manuzhang (via GitHub)" <gi...@apache.org>.
manuzhang closed pull request #10370: Flink 1.19: Fix flaky `TestIcebergSourceFailover > testBoundedWithSavepoint`
URL: https://github.com/apache/iceberg/pull/10370
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink 1.19: Fix flaky `TestIcebergSourceFailover > testBoundedWithSavepoint` [iceberg]
Posted by "stevenzwu (via GitHub)" <gi...@apache.org>.
stevenzwu commented on code in PR #10370:
URL: https://github.com/apache/iceberg/pull/10370#discussion_r1612082934
##########
flink/v1.19/flink/src/test/java/org/apache/iceberg/flink/source/TestIcebergSourceFailover.java:
##########
@@ -124,7 +124,7 @@ public void testBoundedWithSavepoint() throws Exception {
GenericAppenderHelper dataAppender =
new GenericAppenderHelper(
sourceTableResource.table(), FileFormat.PARQUET, TEMPORARY_FOLDER);
- for (int i = 0; i < 4; ++i) {
+ for (int i = 0; i < 20; ++i) {
Review Comment:
hmm. we actually already set the file open cost to avoid bundling
```
// Prevent combining splits
.set(
FlinkReadOptions.SPLIT_FILE_OPEN_COST,
Long.toString(TableProperties.SPLIT_SIZE_DEFAULT))
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org