You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by GitBox <gi...@apache.org> on 2022/04/15 16:44:42 UTC
[GitHub] [iceberg] kbendick opened a new pull request, #4418: Flink - Fix UPSERT delete file metadata [Backport to Flink 1.12]
kbendick opened a new pull request, #4418:
URL: https://github.com/apache/iceberg/pull/4418
This backports https://github.com/apache/iceberg/pull/4364 to Flink 1.12.
See also https://github.com/apache/iceberg/pull/4417 for Flink 1.13.
Co-authored-by: liliwei <hi...@gmail.com>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on pull request #4418: Flink - Fix UPSERT delete file metadata [Backport to Flink 1.12]
Posted by GitBox <gi...@apache.org>.
kbendick commented on PR #4418:
URL: https://github.com/apache/iceberg/pull/4418#issuecomment-1093124935
Closing in favor of https://github.com/apache/iceberg/pull/4519
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick closed pull request #4418: Flink - Fix UPSERT delete file metadata [Backport to Flink 1.12]
Posted by GitBox <gi...@apache.org>.
kbendick closed pull request #4418: Flink - Fix UPSERT delete file metadata [Backport to Flink 1.12]
URL: https://github.com/apache/iceberg/pull/4418
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink - Fix UPSERT delete file metadata [Backport to Flink 1.12] [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #4418:
URL: https://github.com/apache/iceberg/pull/4418#issuecomment-2290114264
This pull request has been closed due to lack of activity. This is not a judgement on the merit of the PR in any way. It is just a way of keeping the PR queue manageable. If you think that is incorrect, or the pull request requires review, you can revive the PR at any time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on pull request #4418: Flink - Fix UPSERT delete file metadata [Backport to Flink 1.12]
Posted by GitBox <gi...@apache.org>.
kbendick commented on PR #4418:
URL: https://github.com/apache/iceberg/pull/4418#issuecomment-1100293518
I updated this PR to use `DATE` instead of `TO_DATE`, as we discovered in https://github.com/apache/iceberg/pull/4532 that `TO_DATE` was causing tests to process rows out of order during `INSERT VALUES`.
However, the tests all still fail in 1.12.
I think that So I'm still not able to get 1.12 to correctly process upserts. I think we should disable `upsert` mode in Flink 1.12 in the next patch with a recommendation to upgrade to 1.13 as proposed here: https://github.com/apache/iceberg/pull/4519
```
> Task :iceberg-flink:iceberg-flink-1.12:test
org.apache.iceberg.flink.TestFlinkUpsert > testPrimaryKeyEqualToPartitionKey[catalogName=testhadoop, baseNamespace=default, format=PARQUET, isStreaming=true] FAILED
java.lang.AssertionError: expected:<[2,aaa, 3,bbb]> but was:<[1,aaa, 3,bbb]>
at org.junit.Assert.fail(Assert.java:89)
at org.junit.Assert.failNotEquals(Assert.java:835)
at org.junit.Assert.assertEquals(Assert.java:120)
at org.junit.Assert.assertEquals(Assert.java:146)
at org.apache.iceberg.flink.TestHelpers.assertRows(TestHelpers.java:140)
at org.apache.iceberg.flink.TestFlinkUpsert.testPrimaryKeyEqualToPartitionKey(TestFlinkUpsert.java:178)
org.apache.iceberg.flink.TestFlinkUpsert > testPrimaryKeyFieldsAtBeginningOfSchema[catalogName=testhadoop, baseNamespace=default, format=PARQUET, isStreaming=true] FAILED
java.lang.AssertionError: expected:<[aaa,2022-03-01,2, bbb,2022-03-01,3]> but was:<[aaa,2022-03-01,1, bbb,2022-03-01,3]>
at org.junit.Assert.fail(Assert.java:89)
at org.junit.Assert.failNotEquals(Assert.java:835)
at org.junit.Assert.assertEquals(Assert.java:120)
at org.junit.Assert.assertEquals(Assert.java:146)
at org.apache.iceberg.flink.TestHelpers.assertRows(TestHelpers.java:140)
at org.apache.iceberg.flink.TestFlinkUpsert.testPrimaryKeyFieldsAtBeginningOfSchema(TestFlinkUpsert.java:219)
org.apache.iceberg.flink.TestFlinkUpsert > testPrimaryKeyFieldsAtEndOfTableSchema[catalogName=testhadoop, baseNamespace=default, format=PARQUET, isStreaming=false] FAILED
java.lang.AssertionError: expected:<[2,aaa,2022-03-01, 3,bbb,2022-03-01]> but was:<[1,aaa,2022-03-01, 3,bbb,2022-03-01]>
at org.junit.Assert.fail(Assert.java:89)
at org.junit.Assert.failNotEquals(Assert.java:835)
at org.junit.Assert.assertEquals(Assert.java:120)
at org.junit.Assert.assertEquals(Assert.java:146)
at org.apache.iceberg.flink.TestHelpers.assertRows(TestHelpers.java:140)
at org.apache.iceberg.flink.TestFlinkUpsert.testPrimaryKeyFieldsAtEndOfTableSchema(TestFlinkUpsert.java:262)
org.apache.iceberg.flink.TestFlinkUpsert > testPrimaryKeyEqualToPartitionKey[catalogName=testhadoop, baseNamespace=default, format=PARQUET, isStreaming=false] FAILED
java.lang.AssertionError: expected:<[2,aaa, 3,bbb]> but was:<[1,aaa, 3,bbb]>
at org.junit.Assert.fail(Assert.java:89)
at org.junit.Assert.failNotEquals(Assert.java:835)
at org.junit.Assert.assertEquals(Assert.java:120)
at org.junit.Assert.assertEquals(Assert.java:146)
at org.apache.iceberg.flink.TestHelpers.assertRows(TestHelpers.java:140)
at org.apache.iceberg.flink.TestFlinkUpsert.testPrimaryKeyEqualToPartitionKey(TestFlinkUpsert.java:178)
org.apache.iceberg.flink.TestFlinkUpsert > testPrimaryKeyFieldsAtEndOfTableSchema[catalogName=testhadoop, baseNamespace=default, format=AVRO, isStreaming=true] FAILED
java.lang.AssertionError: expected:<[2,aaa,2022-03-01, 3,bbb,2022-03-01]> but was:<[1,aaa,2022-03-01, 3,bbb,2022-03-01]>
at org.junit.Assert.fail(Assert.java:89)
at org.junit.Assert.failNotEquals(Assert.java:835)
at org.junit.Assert.assertEquals(Assert.java:120)
at org.junit.Assert.assertEquals(Assert.java:146)
at org.apache.iceberg.flink.TestHelpers.assertRows(TestHelpers.java:140)
at org.apache.iceberg.flink.TestFlinkUpsert.testPrimaryKeyFieldsAtEndOfTableSchema(TestFlinkUpsert.java:262)
org.apache.iceberg.flink.TestFlinkUpsert > testPrimaryKeyEqualToPartitionKey[catalogName=testhadoop, baseNamespace=default, format=AVRO, isStreaming=true] FAILED
java.lang.AssertionError: expected:<[2,aaa, 3,bbb]> but was:<[1,aaa, 3,bbb]>
at org.junit.Assert.fail(Assert.java:89)
at org.junit.Assert.failNotEquals(Assert.java:835)
at org.junit.Assert.assertEquals(Assert.java:120)
at org.junit.Assert.assertEquals(Assert.java:146)
at org.apache.iceberg.flink.TestHelpers.assertRows(TestHelpers.java:140)
at org.apache.iceberg.flink.TestFlinkUpsert.testPrimaryKeyEqualToPartitionKey(TestFlinkUpsert.java:178)
org.apache.iceberg.flink.TestFlinkUpsert > testPrimaryKeyFieldsAtEndOfTableSchema[catalogName=testhadoop, baseNamespace=default, format=AVRO, isStreaming=false] FAILED
java.lang.AssertionError: expected:<[2,aaa,2022-03-01, 3,bbb,2022-03-01]> but was:<[1,aaa,2022-03-01, 3,bbb,2022-03-01]>
at org.junit.Assert.fail(Assert.java:89)
at org.junit.Assert.failNotEquals(Assert.java:835)
at org.junit.Assert.assertEquals(Assert.java:120)
at org.junit.Assert.assertEquals(Assert.java:146)
at org.apache.iceberg.flink.TestHelpers.assertRows(TestHelpers.java:140)
at org.apache.iceberg.flink.TestFlinkUpsert.testPrimaryKeyFieldsAtEndOfTableSchema(TestFlinkUpsert.java:262)
org.apache.iceberg.flink.TestFlinkUpsert > testPrimaryKeyEqualToPartitionKey[catalogName=testhadoop, baseNamespace=default, format=AVRO, isStreaming=false] FAILED
java.lang.AssertionError: expected:<[2,aaa, 3,bbb]> but was:<[1,aaa, 3,bbb]>
at org.junit.Assert.fail(Assert.java:89)
at org.junit.Assert.failNotEquals(Assert.java:835)
at org.junit.Assert.assertEquals(Assert.java:120)
at org.junit.Assert.assertEquals(Assert.java:146)
at org.apache.iceberg.flink.TestHelpers.assertRows(TestHelpers.java:140)
at org.apache.iceberg.flink.TestFlinkUpsert.testPrimaryKeyEqualToPartitionKey(TestFlinkUpsert.java:178)
org.apache.iceberg.flink.TestFlinkUpsert > testPrimaryKeyEqualToPartitionKey[catalogName=testhadoop, baseNamespace=default, format=ORC, isStreaming=false] FAILED
java.lang.AssertionError: expected:<[2,aaa, 3,bbb]> but was:<[1,aaa, 3,bbb]>
at org.junit.Assert.fail(Assert.java:89)
at org.junit.Assert.failNotEquals(Assert.java:835)
at org.junit.Assert.assertEquals(Assert.java:120)
at org.junit.Assert.assertEquals(Assert.java:146)
at org.apache.iceberg.flink.TestHelpers.assertRows(TestHelpers.java:140)
at org.apache.iceberg.flink.TestFlinkUpsert.testPrimaryKeyEqualToPartitionKey(TestFlinkUpsert.java:178)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink - Fix UPSERT delete file metadata [Backport to Flink 1.12] [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #4418:
URL: https://github.com/apache/iceberg/pull/4418#issuecomment-2274583879
This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@iceberg.apache.org list. Thank you for your contributions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink - Fix UPSERT delete file metadata [Backport to Flink 1.12] [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #4418: Flink - Fix UPSERT delete file metadata [Backport to Flink 1.12]
URL: https://github.com/apache/iceberg/pull/4418
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on pull request #4418: Flink - Fix UPSERT delete file metadata [Backport to Flink 1.12]
Posted by GitBox <gi...@apache.org>.
kbendick commented on PR #4418:
URL: https://github.com/apache/iceberg/pull/4418#issuecomment-1100291368
I updated this PR to use `DATE` instead of `TO_DATE`, as we discovered that `TO_DATE` was causing tests to process out of order.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on pull request #4418: Flink - Fix UPSERT delete file metadata [Backport to Flink 1.12]
Posted by GitBox <gi...@apache.org>.
rdblue commented on PR #4418:
URL: https://github.com/apache/iceberg/pull/4418#issuecomment-1115474996
@openinx have you had a chance to look into the failure? It would be great to fix this instead of disabling in 1.12.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on pull request #4418: Flink - Fix UPSERT delete file metadata [Backport to Flink 1.12]
Posted by GitBox <gi...@apache.org>.
kbendick commented on PR #4418:
URL: https://github.com/apache/iceberg/pull/4418#issuecomment-1090808573
Opened a PR to simply throw `UnsupportedOperationException` in `FlinkSink` if it's determined that the program is running in `upsert` mode: https://github.com/apache/iceberg/pull/4519
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on pull request #4418: Flink - Fix UPSERT delete file metadata [Backport to Flink 1.12]
Posted by GitBox <gi...@apache.org>.
kbendick commented on PR #4418:
URL: https://github.com/apache/iceberg/pull/4418#issuecomment-1100294375
Also, Github is saying that all checks are passing as Flink 1.12 has been removed in master. Checking out this branch and running it locally shows that the issue is still present.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on pull request #4418: Flink - Fix UPSERT delete file metadata [Backport to Flink 1.12]
Posted by GitBox <gi...@apache.org>.
kbendick commented on PR #4418:
URL: https://github.com/apache/iceberg/pull/4418#issuecomment-1100291039
I updated this PR to use `DATE` instead of the `TO_DATE` function as we discovered that `TO_DATE` was causing data to be processed out of order.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] openinx commented on pull request #4418: Flink - Fix UPSERT delete file metadata [Backport to Flink 1.12]
Posted by GitBox <gi...@apache.org>.
openinx commented on PR #4418:
URL: https://github.com/apache/iceberg/pull/4418#issuecomment-1101025838
If this failure can be reproduced locally, then we can just open the intellij idea debugger to analysis the written columnar files to see what's wrong there.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org