You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@paimon.apache.org by "zhongyujiang (via GitHub)" <gi...@apache.org> on 2025/11/24 05:18:11 UTC
[PR] [spark] fix drop partition when partition value is null or empty string [paimon]
zhongyujiang opened a new pull request, #6662:
URL: https://github.com/apache/paimon/pull/6662
<!-- Please specify the module before the PR name: [core] ... or [flink] ... -->
### Purpose
Currently, when using Spark to delete Paimon partitions, deletion fails or incorrectly removes mismatched data if the partition value is NULL or an empty string. This issue occurs across `ALTER TABLE ... DROP PARTITION`, `DELETE FROM`, and `TRUNCATE PARTITION`
**ALTER TABLE T DROP PARTITION and DELETE FROM**
Dropping a partition with partitionField = '' erroneously deletes data where partitionField = null.
The root cause is that during deletion, Spark SQL passes partition values that are replaced by `InternalRowPartitionComputer` with the default partition value. However, the actual partition filter should use the original (real) partition values for matching, because the manifest files store the real partition values—not the default ones.
This PR resolves the issue by preserving the original partition values (instead of replacing them with the default partition value) when computing partitions for data deletion: `InternalRowPartitionComputer preserveNullOrEmptyValue`
**TRUNCATE PARTITION**
Throws a NullPointerException (NPE) when the partition value is null.
Fixed by safely handling null values.
### Tests
Added in the PR.
### API and Format
No.
### Documentation
No.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [spark] fix drop partition when partition value is null or empty string [paimon]
Posted by "zhongyujiang (via GitHub)" <gi...@apache.org>.
zhongyujiang commented on PR #6662:
URL: https://github.com/apache/paimon/pull/6662#issuecomment-3568968792
cc @JingsongLi @Zouxxyy @YannByron Can you take a look when you have time? Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [spark] fix drop partition when partition value is null or empty string [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on PR #6662:
URL: https://github.com/apache/paimon/pull/6662#issuecomment-3570076302
> Oh, I wasn't aware of that. I noticed that Paimon currently places data with partition values of both null and empty string into the same partition—the default partition. Is Paimon intended to be consistent with Hive in this behavior?
My answer is, if possible, it's best for us to maintain the status quo. Can this solve your business problem?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [spark] fix drop partition when partition value is null or empty string [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on PR #6662:
URL: https://github.com/apache/paimon/pull/6662#issuecomment-3569521869
This may be a tricky issue, as Hive's previous definition was to treat empty strings and null as equivalent.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [spark] fix drop partition when partition value is null or empty string [paimon]
Posted by "zhongyujiang (via GitHub)" <gi...@apache.org>.
zhongyujiang commented on PR #6662:
URL: https://github.com/apache/paimon/pull/6662#issuecomment-3595656127
@JingsongLi Do you have time to take another look? Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [spark] fix drop partition when partition value is null or empty string [paimon]
Posted by "zhongyujiang (via GitHub)" <gi...@apache.org>.
zhongyujiang commented on PR #6662:
URL: https://github.com/apache/paimon/pull/6662#issuecomment-3569932272
> @zhongyujiang Are you referring to the issue of mixing empty strings and null?
@JingsongLi Is not that the values are "mixed", but that the data with partition column values equal to an empty string cannot be deleted when using `DELETE FROM` or `ALTER TABLE T DROP PARTITION`.
Reproduce:
```scala
test("Paimon Partition Management: partition values are empty string") {
spark.sql(s"""
|CREATE TABLE T (pt STRING, data STRING)
|PARTITIONED BY (pt)
|""".stripMargin)
sql("INSERT INTO T VALUES('', 'a'), ('2', 'b')")
sql("ALTER TABLE T DROP PARTITION (pt = '')")
spark.sql("SELECT * FROM T").show(false)
//+---+----+
//|pt |data|
//+---+----+
//|2 |b |
//| |a |
//+---+----+
}
```
> This may be a tricky issue, as Hive's previous definition was to treat empty strings and null as equivalent.
Oh, I wasn't aware of that. I noticed that Paimon currently places data with partition values of both null and empty string into the same partition—the default partition. Is Paimon intended to be consistent with Hive in this behavior?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [spark] fix drop partition when partition value is null or empty string [paimon]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #6662:
URL: https://github.com/apache/paimon/pull/6662#issuecomment-4410755211
This pull request has had no activity for 90 days. If you'd like to keep it open, please push a new commit or leave a comment. Thanks for the contribution.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [spark] fix drop partition when partition value is null or empty string [paimon]
Posted by "zhongyujiang (via GitHub)" <gi...@apache.org>.
zhongyujiang commented on PR #6662:
URL: https://github.com/apache/paimon/pull/6662#issuecomment-3569384180
@JingsongLi Thanks for looking at this.
I think this issue occurs for all catalogs. The root cause lies in Spark’s current partition-dropping logic, which incorrectly transforms partition values using InternalRowPartitionComputer—not in the catalog implementation itself. Here's how the issue unfolds:
When executing ALTER TABLE T DROP PARTITION(pt=''), PaimonPartitionManagement#toPaimonPartitions uses InternalRowPartitionComputer to convert the Spark Row into a partition spec of type Map<String, String>. During this conversion, the empty string ('') is replaced with the default partition value, resulting in (pt='__DEFAULT_PARTITION__').
This spec is then passed to FileStoreCommitImpl#dropPartitions to delete the partition.
Inside dropPartitions, the partition spec (pt='__DEFAULT_PARTITION__') is converted back into a Paimon internal row via `InternalRowPartitionComputer#convertSpecToInternalRow` to construct the partition predicate. However, at this stage, '__DEFAULT_PARTITION__' is converted to null (see code below):
https://github.com/apache/paimon/blob/4a7cdf445f0a726266ecb3dd36b36c823c79e615/paimon-common/src/main/java/org/apache/paimon/utils/InternalRowPartitionComputer.java#L135-L153
As a result, data with pt = '' is not deleted; instead, data with pt = null gets deleted (if it exists), leading to incorrect behavior.
This also happens in `DELETE FROM`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [spark] fix drop partition when partition value is null or empty string [paimon]
Posted by "zhongyujiang (via GitHub)" <gi...@apache.org>.
zhongyujiang commented on PR #6662:
URL: https://github.com/apache/paimon/pull/6662#issuecomment-3570420680
Actually, Spark's `TRUNCATE TABLE` command already works this way—it correctly passes down the partCol = '' value, so it can properly delete the partition where partCol = ''. However, when partCol = null, this command will throw a NPE.
My first commit in this PR fixes this issue.
https://github.com/apache/paimon/blob/69b815918ef353540b9999255c8138b9d49bc28c/paimon-spark/paimon-spark-common/src/main/scala/org/apache/paimon/spark/catalyst/analysis/PaimonIncompatiblePHRRules.scala#L31-L45
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [spark] fix drop partition when partition value is null or empty string [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on PR #6662:
URL: https://github.com/apache/paimon/pull/6662#issuecomment-3569518232
@zhongyujiang Are you referring to the issue of mixing empty strings and null?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [spark] fix drop partition when partition value is null or empty string [paimon]
Posted by "zhongyujiang (via GitHub)" <gi...@apache.org>.
zhongyujiang commented on PR #6662:
URL: https://github.com/apache/paimon/pull/6662#issuecomment-3570406650
> if possible, it's best for us to maintain the status quo.
I think that in Paimon's manifest, `partCol = null` and `partCol = ''` are actually treated as two distinct partitions, because their `_PARTITION` values in the manifest are different. Querying the files table or partitions table shows two separate partitions too, for both FileSystemCatalog and HiveCatalog.
In other words, in Paimon's metadata, these are currently considered two different partitions. Just that data files of both types are stored under the same path `\_\_DEFAULT_PARTITION\_\_`.
Are you saying that when using HiveCatalog, these two partitions get merged into a single partition when synced to the HMS? And when doing partition deletion, would data from both null and empty string be deleted together? I’m not clear on this point.
> Can this solve your business problem?
The issue we’re encountering now is that data with partCol = '' cannot be deleted using either DELETE FROM or ALTER TABLE ... DROP PARTITION. Instead, it erroneously deletes data where partCol = null, which is not the expected behavior. So I fixed the push down of partition values to ensure that truncatePartitions works correctly in this PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [spark] fix drop partition when partition value is null or empty string [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on PR #6662:
URL: https://github.com/apache/paimon/pull/6662#issuecomment-3600336249
I merged your first commit.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [spark] fix drop partition when partition value is null or empty string [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on PR #6662:
URL: https://github.com/apache/paimon/pull/6662#issuecomment-3600339227
Maybe can we provide a `void truncatePartitions(List<BinaryRow> partitions)` for `BatchTableCommit`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [spark] fix drop partition when partition value is null or empty string [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi closed pull request #6662: [spark] fix drop partition when partition value is null or empty string
URL: https://github.com/apache/paimon/pull/6662
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org