You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "kazantsev-maksim (via GitHub)" <gi...@apache.org> on 2026/06/18 14:49:57 UTC
[PR] Empty2null [datafusion-comet]
kazantsev-maksim opened a new pull request, #4683:
URL: https://github.com/apache/datafusion-comet/pull/4683
## Which issue does this PR close?
Part of: https://github.com/apache/datafusion-comet/pull/4670
## Rationale for this change
`Empty2Null` is an internal Spark expression that converts an empty string "" into null. The logic is trivial: if the value is null or a zero-length string, it returns null; otherwise it returns the string unchanged.
Purpose
The function is applied during partitioned file writes (parquet, orc, etc.) — specifically to the partition columns. The reason is the correctness of Hive-style partitioning.
In Hive-style directory naming, an empty string and null are indistinguishable: both would produce a path like col1=, which is ambiguous and breaks reading the data back. To avoid this, Spark runs partition columns through Empty2Null before writing, so empty strings end up in the same default partition as null: `col1=__HIVE_DEFAULT_PARTITION__`
## What changes are included in this PR?
## How are these changes tested?
Add rust unit tests
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: implement native empty2null spark inner function [datafusion-comet]
Posted by "kazantsev-maksim (via GitHub)" <gi...@apache.org>.
kazantsev-maksim commented on code in PR #4683:
URL: https://github.com/apache/datafusion-comet/pull/4683#discussion_r3508146702
##########
spark/src/test/scala/org/apache/comet/parquet/CometParquetWriterSuite.scala:
##########
@@ -37,6 +38,27 @@ class CometParquetWriterSuite extends CometTestBase {
import testImplicits._
+ test("partitioned write with empty string partition value") {
+ withTempPath { path =>
+ Seq(("", 1), ("a", 2))
+ .toDF("part", "value")
+ .write
Review Comment:
Sorry for the delayed response. This function is used for string fields during writes using partitionBy, and I encountered this while working on this: https://github.com/apache/datafusion-comet/pull/4670.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: implement native empty2null spark inner function [datafusion-comet]
Posted by "comphead (via GitHub)" <gi...@apache.org>.
comphead commented on PR #4683:
URL: https://github.com/apache/datafusion-comet/pull/4683#issuecomment-4753564173
Thanks, starting CI
@kazantsev-maksim wondering, if its possible to write a unit test for the function to make sure it works properly?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: implement native empty2null spark inner function [datafusion-comet]
Posted by "comphead (via GitHub)" <gi...@apache.org>.
comphead commented on PR #4683:
URL: https://github.com/apache/datafusion-comet/pull/4683#issuecomment-4759533810
> @comphead Spark uses this function to wrap string fields during partitioning; it cannot be called via Spark SQL, but we need this function to implement native partition writing.
I see, perhaps we can come up with unit test?
LLM suggests me something like
```
test("SPARK-XXXXX: empty partition values are written as null partitions") {
withTempDir { dir =>
val path = dir.getCanonicalPath
Seq(
(1, ""),
(2, "a"),
(3, null.asInstanceOf[String])
).toDF("id", "part")
.write
.partitionBy("part")
.parquet(path)
val fs = new Path(path).getFileSystem(spark.sessionState.newHadoopConf())
val partitions = fs.listStatus(new Path(path))
.filter(_.isDirectory)
.map(_.getPath.getName)
.sorted
assert(partitions.contains("part=a"))
// Empty string should not generate a dedicated partition directory.
assert(!partitions.contains("part="))
// Empty string and null should both map to the default partition.
assert(partitions.count(_.startsWith("part=__HIVE_DEFAULT_PARTITION__")) == 1)
checkAnswer(
spark.read.parquet(path).groupBy("part").count(),
Row(null, 2) :: Row("a", 1) :: Nil
)
}
}
```
Without unit test it would be pretty complicated to ensure no regressions happened
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: implement native empty2null spark inner function [datafusion-comet]
Posted by "kazantsev-maksim (via GitHub)" <gi...@apache.org>.
kazantsev-maksim commented on PR #4683:
URL: https://github.com/apache/datafusion-comet/pull/4683#issuecomment-4761839745
Thanks, @comphead! I've added some tests. Would you mind taking another look when you get a chance?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: implement native empty2null spark inner function [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on code in PR #4683:
URL: https://github.com/apache/datafusion-comet/pull/4683#discussion_r3475080824
##########
spark/src/test/scala/org/apache/comet/parquet/CometParquetWriterSuite.scala:
##########
@@ -37,6 +38,27 @@ class CometParquetWriterSuite extends CometTestBase {
import testImplicits._
+ test("partitioned write with empty string partition value") {
+ withTempPath { path =>
+ Seq(("", 1), ("a", 2))
+ .toDF("part", "value")
+ .write
Review Comment:
Could you add an assertion to check that the write is executed by Comet? From reading the PR description, my understanding is that we expect `empty2null` to be called during the write operation?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: implement native empty2null spark inner function [datafusion-comet]
Posted by "kazantsev-maksim (via GitHub)" <gi...@apache.org>.
kazantsev-maksim commented on PR #4683:
URL: https://github.com/apache/datafusion-comet/pull/4683#issuecomment-4753081527
Thanks @comphead, it's work fine.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: implement native empty2null spark inner function [datafusion-comet]
Posted by "kazantsev-maksim (via GitHub)" <gi...@apache.org>.
kazantsev-maksim commented on PR #4683:
URL: https://github.com/apache/datafusion-comet/pull/4683#issuecomment-4753674700
@comphead Spark uses this function to wrap string fields during partitioning; it cannot be called via Spark SQL.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: implement native empty2null spark inner function [datafusion-comet]
Posted by "comphead (via GitHub)" <gi...@apache.org>.
comphead merged PR #4683:
URL: https://github.com/apache/datafusion-comet/pull/4683
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org