You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2022/08/06 13:59:42 UTC
[GitHub] [spark] zinking opened a new pull request, #37428: [SPARK-39997][SQL] Fix ParquetSchemaConverter fails match schema by id
zinking opened a new pull request, #37428:
URL: https://github.com/apache/spark/pull/37428
currently, match parquet schema by id fails under certain case
no
new unit test added
### What changes were proposed in this pull request?
in this PR, fixed cases where ParquetSchemaConverter fails match schema by id
when converting parquet schema, SparkType is preferred instead of Converted parquet type, chances are SparkType has a name which will then fail the same type validation in later cases when running vectorized nested column read.
### Why are the changes needed?
when converting parquet schema, SparkType is preferred instead of Converted parquet type, chances are SparkType has a name which will then fail the same type validation in later cases when running vectorized nested column read.
### Does this PR introduce _any_ user-facing change?
no
### How was this patch tested?
added new ut
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] github-actions[bot] commented on pull request #37428: [SPARK-39997][SQL] Fix ParquetSchemaConverter fails match schema by id
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on PR #37428:
URL: https://github.com/apache/spark/pull/37428#issuecomment-1341790110
We're closing this PR because it hasn't been updated in a while. This isn't a judgement on the merit of the PR in any way. It's just a way of keeping the PR queue manageable.
If you'd like to revive this PR, please reopen it and ask a committer to remove the Stale tag!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #37428: [SPARK-39997][SQL] Fix ParquetSchemaConverter fails match schema by id
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on PR #37428:
URL: https://github.com/apache/spark/pull/37428#issuecomment-1207349975
cc @sunchao
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] zinking commented on pull request #37428: [SPARK-39997][SQL] Fix ParquetSchemaConverter fails match schema by id
Posted by GitBox <gi...@apache.org>.
zinking commented on PR #37428:
URL: https://github.com/apache/spark/pull/37428#issuecomment-1207374314
guess this isn't the fix intended after sunchao commented the spark type is intended for maintaining type precision, but here the spark type is carrying the name `c1` which is renamed from `c0` so carrying it forward to later phases will cause same type validation errors.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #37428: [SPARK-39997][SQL] Fix ParquetSchemaConverter fails match schema by id
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on PR #37428:
URL: https://github.com/apache/spark/pull/37428#issuecomment-1207245039
Can one of the admins verify this patch?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] github-actions[bot] closed pull request #37428: [SPARK-39997][SQL] Fix ParquetSchemaConverter fails match schema by id
Posted by GitBox <gi...@apache.org>.
github-actions[bot] closed pull request #37428: [SPARK-39997][SQL] Fix ParquetSchemaConverter fails match schema by id
URL: https://github.com/apache/spark/pull/37428
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org