You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "llphxd (via GitHub)" <gi...@apache.org> on 2026/05/12 11:57:51 UTC
[PR] [SPARK-56819][SQL] Add option to trim CHAR trailing spaces on read [spark]
llphxd opened a new pull request, #55820:
URL: https://github.com/apache/spark/pull/55820
### What changes were proposed in this pull request?
This PR adds a new SQL configuration, `spark.sql.charTrimTrailingSpacesOnRead`, to trim trailing spaces from `CHAR(N)` columns and fields when reading table data.
The new configuration is disabled by default, so the existing Spark behavior is preserved. When it is enabled, it takes precedence over `spark.sql.readSideCharPadding`.
This is intended to provide an opt-in compatibility mode for systems such as MySQL, where `CHAR` values are commonly returned without trailing spaces unless `PAD_CHAR_TO_FULL_LENGTH` is enabled.
### Why are the changes needed?
Spark currently enforces fixed-length `CHAR(N)` semantics by padding `CHAR` values on write, and by applying read-side padding when `spark.sql.readSideCharPadding` is enabled.
I tested this behavior across several Spark versions with MySQL tables. In Spark 3.3.1 and Spark 3.4.4, MySQL `CHAR` and `VARCHAR` columns were simply treated as Spark `STRING`, so trailing-space behavior was closer to the old string-based behavior. In Spark 3.5.2 and Spark 4.0.1, Spark maps MySQL character types to more standard and stricter Spark `CHAR` types, which can expose behavior differences for `CHAR` columns compared with older Spark versions.
This makes migration or upgrade harder for workloads that rely on the previous string-like behavior or on MySQL's default `CHAR` retrieval behavior, where trailing spaces are removed on read. Users may otherwise need to wrap many `CHAR` columns with `rtrim()` manually in queries.
This PR provides an opt-in configuration to make this behavior easier to control without changing Spark's default semantics.
### Does this PR introduce _any_ user-facing change?
Yes.
This PR adds a new SQL configuration:
```text
spark.sql.charTrimTrailingSpacesOnRead
The default value is false, so existing behavior is unchanged.
When set to true, Spark trims trailing spaces from CHAR(N) columns and fields when reading table data. The option does not affect VARCHAR or STRING, and it does not change write-side CHAR/VARCHAR length checks.
Example:
SET spark.sql.charTrimTrailingSpacesOnRead=true;
CREATE TABLE t (c CHAR(4), v VARCHAR(4), s STRING) USING parquet;
INSERT INTO t VALUES ('12', '12 ', '12 ');
SELECT c, length(c), v, length(v), s, length(s) FROM t;
With the new configuration enabled, the CHAR(4) value is returned without trailing spaces, while VARCHAR and STRING remain unchanged.
### How was this patch tested?
Added test coverage in CharVarcharTestSuite for trimming trailing spaces from CHAR columns and nested CHAR fields on read, while keeping VARCHAR and STRING unchanged.
Tested with:
./dev/scalastyle
build/sbt "sql/testOnly *CharVarcharTestSuite"
### Was this patch authored or co-authored using generative AI tooling?
Assisted by ChatGPT-5.5
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-56819][SQL] Add option to trim CHAR trailing spaces on read [spark]
Posted by "llphxd (via GitHub)" <gi...@apache.org>.
llphxd commented on PR #55820:
URL: https://github.com/apache/spark/pull/55820#issuecomment-4430293750
jira is ready: [SPARK-56819](https://issues.apache.org/jira/browse/SPARK-56819#)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-56819][SQL] Add option to trim CHAR trailing spaces on read [spark]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #55820:
URL: https://github.com/apache/spark/pull/55820#issuecomment-5363642291
We're closing this PR because it hasn't been updated in a while. This isn't a judgement on the merit of the PR in any way. It's just a way of keeping the PR queue manageable.
If you'd like to revive this PR, please reopen it and ask a committer to remove the Stale tag!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-56819][SQL] Add option to trim CHAR trailing spaces on read [spark]
Posted by "llphxd (via GitHub)" <gi...@apache.org>.
llphxd commented on PR #55820:
URL: https://github.com/apache/spark/pull/55820#issuecomment-4430357180
One possible question is why this new option is needed when `spark.sql.legacy.charVarcharAsString` already exists.
I think the two options serve different purposes. `spark.sql.legacy.charVarcharAsString` disables Spark's CHAR/VARCHAR type semantics broadly by treating CHAR/VARCHAR as STRING. This restores older Spark behavior, but it also disables length checks and CHAR padding semantics, so it is a coarse-grained legacy compatibility switch.
The proposed option is narrower. It only changes the read-side representation of CHAR values by trimming trailing spaces when explicitly enabled. It does not affect VARCHAR or STRING, and it does not disable write-side CHAR/VARCHAR length checks. This allows users to keep Spark's stricter CHAR/VARCHAR type handling while opting into MySQL-compatible CHAR retrieval behavior.
This is useful for migration/upgrade scenarios where users want to preserve standard CHAR/VARCHAR validation in Spark, but need the returned CHAR values to match MySQL's default behavior or previous string-like query results more closely.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-56819][SQL] Add option to trim CHAR trailing spaces on read [spark]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #55820: [SPARK-56819][SQL] Add option to trim CHAR trailing spaces on read
URL: https://github.com/apache/spark/pull/55820
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org