You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "andygrove (via GitHub)" <gi...@apache.org> on 2026/10/19 21:30:25 UTC
[PR] docs: note that Text broadcast build sides decode invalid UTF-8 [datafusion-comet]
andygrove opened a new pull request, #7119:
URL: https://github.com/apache/datafusion-comet/pull/7119
## Which issue does this PR close?
Part of #7118.
## Rationale for this change
Since #6786, Comet converts a Text scan on a broadcast join's build side to Arrow by default. Text files are where invalid UTF-8 most often shows up, for example a file written in Latin-1, and the conversion replaces those bytes with `U+FFFD`. A query that ran entirely in Spark on 1.1.0 can now return different `hex`, `length` or join results. The compatibility guide says invalid UTF-8 "does not occur for text read from Parquet or produced by string expressions", which reads as if a query over Parquet and Text files is unaffected.
#7118 also asks whether to turn the conversion off by default for 1.2.0. This PR documents the current behavior either way. If the default changes, the sentence about the default changes with it.
## What changes are included in this PR?
- `compatibility/index.md`: name Text files in another encoding and strings cast from binary as sources of invalid UTF-8, say that a Text scan on a broadcast build side is converted by default, and give the config that keeps it on Spark.
- `datasources.md`: in the paragraph about the build-side conversion, link to the invalid UTF-8 section and say that a Text file in another encoding can give different results.
## How are these changes tested?
Documentation only. Prettier passes. The behavior described is the one reproduced in #7118.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] docs: note that Text broadcast build sides decode invalid UTF-8 [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on PR #7119:
URL: https://github.com/apache/datafusion-comet/pull/7119#issuecomment-6281835415
Closing in favor of #7123, which turns the broadcast build side conversion off by default and documents the invalid UTF-8 cases. The text here says the conversion is on by default, which no longer holds.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] docs: note that Text broadcast build sides decode invalid UTF-8 [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove closed pull request #7119: docs: note that Text broadcast build sides decode invalid UTF-8
URL: https://github.com/apache/datafusion-comet/pull/7119
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org