You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "Smallfu666 (via GitHub)" <gi...@apache.org> on 2026/09/11 07:21:05 UTC
[PR] fix: preserve nulls for Boolean/Byte/Short/Integer columns in FuzzDataGenerator [datafusion-comet]
Smallfu666 opened a new pull request, #5855:
URL: https://github.com/apache/datafusion-comet/pull/5855
## Which issue does this PR close?
Closes #5389.
## Rationale for this change
`FuzzDataGenerator` never produces nulls for boolean, byte, short or int columns, so fuzz runs
silently skip null handling for those four types.
All four are generated from a `Long` column and then narrowed:
```scala
generateColumn(r, DataTypes.LongType, numRows, options)
.map(_.asInstanceOf[Long].toByte)
```
`asInstanceOf[Long]` on a null goes through `BoxesRunTime.unboxToLong`, which returns `0L` rather
than throwing. Every null becomes `0`, and the column comes back with no nulls at all.
## What changes are included in this PR?
Match on the element instead of casting, so nulls survive the narrowing:
```scala
generateColumn(r, DataTypes.LongType, numRows, options).map {
case x: Long => x.toByte
case null => null
}
```
Applied to the boolean, byte, short and int arms. No other type is affected.
## How are these changes tested?
`DataGeneratorSuite` gains `allowNull produces nulls in every type that honours it`. Its seed is
pinned, and it fails on all four arms before this change.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix: preserve nulls for Boolean/Byte/Short/Integer columns in FuzzDataGenerator [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove merged PR #5855:
URL: https://github.com/apache/datafusion-comet/pull/5855
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix: preserve nulls for Boolean/Byte/Short/Integer columns in FuzzDataGenerator [datafusion-comet]
Posted by "Smallfu666 (via GitHub)" <gi...@apache.org>.
Smallfu666 commented on PR #5855:
URL: https://github.com/apache/datafusion-comet/pull/5855#issuecomment-5633603071
Thanks @sunchao @rich7420.
CI is still sitting at `action_required` so nothing has actually run yet. The only green check is the label one, which feels like a low bar. Mind approving the workflow run?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org