You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "stevenzwu (via GitHub)" <gi...@apache.org> on 2024/07/23 17:18:36 UTC
[PR] Flink: backport PR #10457 for handling rescale and statistics refactoring in smart shuffling [iceberg]
stevenzwu opened a new pull request, #10757:
URL: https://github.com/apache/iceberg/pull/10757
(no comment)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: backport PR #10331 and PR #10457 [iceberg]
Posted by "stevenzwu (via GitHub)" <gi...@apache.org>.
stevenzwu merged PR #10757:
URL: https://github.com/apache/iceberg/pull/10757
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: backport PR #10331 and PR #10457 [iceberg]
Posted by "stevenzwu (via GitHub)" <gi...@apache.org>.
stevenzwu commented on PR #10757:
URL: https://github.com/apache/iceberg/pull/10757#issuecomment-2251276689
> @stevenzwu: Could you please help me understand why are the changes in flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/sink/shuffle/SortKeySerializer.java are needed?
I assume you were asking about the diff btw 1.19 and 1.18. There were change of API in 1.19 for `TypeSerializerSnapshot` interface.
```
1.18
@Override
public TypeSerializerSchemaCompatibility<SortKey> resolveSchemaCompatibility(
TypeSerializer<SortKey> newSerializer) {
1.19
@Override
public TypeSerializerSchemaCompatibility<SortKey> resolveSchemaCompatibility(
TypeSerializerSnapshot<SortKey> oldSerializerSnapshot) {
```
But I did find a bug in the 1.18 implementation that didn't compare the SortOder for compatibility, this is fixed in the latest commit that I just pushed.
Would also need to add a unit test to test the compatibility check. It could be done in this PR or a separate one.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: backport PR #10331 and PR #10457 [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on PR #10757:
URL: https://github.com/apache/iceberg/pull/10757#issuecomment-2246943273
@stevenzwu: Could you please help me understand why are the changes in flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/sink/shuffle/SortKeySerializer.java are needed?
Thanks, Peter
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: backport PR #10457 for handling rescale and statistics refactoring in smart shuffling [iceberg]
Posted by "stevenzwu (via GitHub)" <gi...@apache.org>.
stevenzwu commented on PR #10757:
URL: https://github.com/apache/iceberg/pull/10757#issuecomment-2245809245
It is a clean back port (complete copy over from 1.19 to 1.17/1.18)
```
➜ iceberg git:(backport-10457) ✗ diff flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/sink/shuffle/ flink/v1.18/flink/src/main/java/org/apache/iceberg/flink/sink/shuffle/
➜ iceberg git:(backport-10457) ✗ diff flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/sink/shuffle/ flink/v1.17/flink/src/main/java/org/apache/iceberg/flink/sink/shuffle/
➜ iceberg git:(backport-10457) ✗ diff flink/v1.19/flink/src/test/java/org/apache/iceberg/flink/sink/shuffle/ flink/v1.18/flink/src/test/java/org/apache/iceberg/flink/sink/shuffle/
➜ iceberg git:(backport-10457) ✗ diff flink/v1.19/flink/src/test/java/org/apache/iceberg/flink/sink/shuffle/ flink/v1.17/flink/src/test/java/org/apache/iceberg/flink/sink/shuffle/
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: backport PR #10331 and PR #10457 [iceberg]
Posted by "stevenzwu (via GitHub)" <gi...@apache.org>.
stevenzwu commented on PR #10757:
URL: https://github.com/apache/iceberg/pull/10757#issuecomment-2253077198
thanks @pvary for the review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org