You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "jxu (via GitHub)" <gi...@apache.org> on 2026/05/14 17:47:27 UTC
[I] Feature request: inferSchema do not infer digit strings that start with 0 as integer [spark]
jxu opened a new issue, #55884:
URL: https://github.com/apache/spark/issues/55884
Version: Spark 3.5
Usually, if a data source has a string like "000123", this is an ID of fixed number of digits where the leading zeros are important, so this should not be cast into an integer like 123. An example is zip codes. There should be an option to be safe and keep as a string. Otherwise I have to turn off inferSchema and manually specify the whole schema or cast afterwards.
Same request here https://stackoverflow.com/questions/50299192/pyspark-inferschema-numbers-that-start-with-0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [I] Feature request: inferSchema do not infer digit strings that start with 0 as integer [spark]
Posted by "ravikiranpagidi (via GitHub)" <gi...@apache.org>.
ravikiranpagidi commented on issue #55884:
URL: https://github.com/apache/spark/issues/55884#issuecomment-4537793242
Thanks. I will keep it small and focused.
I will add an opt-in CSV option so that values like 000123 can remain as strings during inferSchema, with the default behavior unchanged. I will include tests and docs in the PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [I] Feature request: inferSchema do not infer digit strings that start with 0 as integer [spark]
Posted by "ravikiranpagidi (via GitHub)" <gi...@apache.org>.
ravikiranpagidi commented on issue #55884:
URL: https://github.com/apache/spark/issues/55884#issuecomment-4511890926
Hi, I’m Ravikiran Pagidi. I would like to work on this issue.
My proposed approach is to add an opt-in CSV option, defaulting to false for backward compatibility, that keeps digit-only values with leading zeros as StringType during inferSchema.
Example:
000123 -> StringType
00123 -> StringType
0 -> IntegerType
123 -> IntegerType
I can add the implementation, tests in CSVInferSchemaSuite, and update the CSV data source documentation.
Does this approach sound acceptable to the maintainers?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [I] Feature request: inferSchema do not infer digit strings that start with 0 as integer [spark]
Posted by "jxu (via GitHub)" <gi...@apache.org>.
jxu commented on issue #55884:
URL: https://github.com/apache/spark/issues/55884#issuecomment-4531675031
Ok but write it without AI please
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [I] Feature request: inferSchema do not infer digit strings that start with 0 as integer [spark]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed issue #55884: Feature request: inferSchema do not infer digit strings that start with 0 as integer
URL: https://github.com/apache/spark/issues/55884
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org