You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@paimon.apache.org by "Aitozi (via GitHub)" <gi...@apache.org> on 2025/05/22 15:31:01 UTC
[I] [Bug] The bucket join may not right after bucket rescaled [paimon]
Aitozi opened a new issue, #5654:
URL: https://github.com/apache/paimon/issues/5654
### Search before asking
- [x] I searched in the [issues](https://github.com/apache/paimon/issues) and found nothing similar.
### Paimon version
1.2
### Compute Engine
Spark
### Minimal reproduce step
Currently, Spark bucket join use the table properties bucket number to report the output partition. But, this information can be not right for the partitioned table.
Because, different partition can have different bucket number, this could lead to the wrong results for the bucket join.
### What doesn't meet your expectations?
.
### Anything else?
_No response_
### Are you willing to submit a PR?
- [ ] I'm willing to submit a PR!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] The bucket join may not right after bucket rescaled [paimon]
Posted by "ulysses-you (via GitHub)" <gi...@apache.org>.
ulysses-you commented on issue #5654:
URL: https://github.com/apache/paimon/issues/5654#issuecomment-2908629028
@Aitozi I did not know Paimon support `different partition can have different bucket number`. Is this a Flink feature ? Is there an example case ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] The bucket join may not right after bucket rescaled [paimon]
Posted by "Aitozi (via GitHub)" <gi...@apache.org>.
Aitozi commented on issue #5654:
URL: https://github.com/apache/paimon/issues/5654#issuecomment-2901689196
CC @ulysses-you @JingsongLi @YannByron
Do you have some good suggestion here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] The bucket join may not right after bucket rescaled [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi closed issue #5654: [Bug] The bucket join may not right after bucket rescaled
URL: https://github.com/apache/paimon/issues/5654
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] The bucket join may not right after bucket rescaled [paimon]
Posted by "Aitozi (via GitHub)" <gi...@apache.org>.
Aitozi commented on issue #5654:
URL: https://github.com/apache/paimon/issues/5654#issuecomment-2910173519
Maybe we should check the input partition's total buckets all equal to the bucket spec ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] The bucket join may not right after bucket rescaled [paimon]
Posted by "Aitozi (via GitHub)" <gi...@apache.org>.
Aitozi commented on issue #5654:
URL: https://github.com/apache/paimon/issues/5654#issuecomment-2910166033
https://paimon.apache.org/docs/master/maintenance/rescale-bucket/#rescale-overwrite
Paimon allows to rescale the bucket without rewrite the old partition's data.
I write a test to reproduce this
```scala
withTable("t1", "t2") {
spark.sql("CREATE TABLE t1 (id INT, c STRING, dt STRING) partitioned by (dt) TBLPROPERTIES ('bucket'='2', 'bucket-key' = 'id')")
spark.sql("CREATE TABLE t2 (id INT, c STRING, dt STRING) partitioned by (dt) TBLPROPERTIES ('bucket'='3', 'bucket-key' = 'id')")
spark.sql("INSERT INTO t1 VALUES (1, 'x1', '20250101'), (3, 'x2', '20250101')")
spark.sql("INSERT INTO t2 VALUES (1, 'x1', '20250101'), (4, 'x2', '20250101')")
spark.sql("SELECT bucket(2, 1), bucket(2, 3), bucket(3, 1), bucket(3, 4)").show()
checkAnswerAndShuffleSorts("SELECT * FROM t1 JOIN t2 on t1.id = t2.id and t1.dt = '20250101' and t2.dt = '20250101'", 2, 2)
spark.sql("ALTER TABLE t1 SET TBLPROPERTIES ('bucket' = '3')")
checkAnswerAndShuffleSorts("SELECT * FROM t1 JOIN t2 on t1.id = t2.id and t1.dt = t2.dt ", 2, 2)
}
-- enables "spark.sql.requireAllClusterKeysForCoPartition" -> "false",
```
```
+-----------------------------+-----------------------------+-----------------------------+-----------------------------+
|applyfunctionexpression(2, 1)|applyfunctionexpression(2, 3)|applyfunctionexpression(3, 1)|applyfunctionexpression(3, 4)|
+-----------------------------+-----------------------------+-----------------------------+-----------------------------+
| 0| 1| 1| 0|
+-----------------------------+-----------------------------+-----------------------------+-----------------------------+
```
In this case, the bucket is rescaled to 3, then the bucket join condition is matched. and the result is not right.
<img width="825" alt="Image" src="https://github.com/user-attachments/assets/09520cb0-c3e5-4bff-911b-cd33b972660b" />
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org