You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "ahshahid (via GitHub)" <gi...@apache.org> on 2023/04/12 23:44:37 UTC
[GitHub] [spark] ahshahid opened a new pull request, #40765: [WIP][SPARK-43112]. Spark may use a column other than the actual specified partitioning column for partitioning, for Hive format tables
ahshahid opened a new pull request, #40765:
URL: https://github.com/apache/spark/pull/40765
### What changes were proposed in this pull request?
Right now only bug test. still figuring out a clean way to fix.
### Why are the changes needed?
the checked in bug test describes the issue. It appears to me that spark-hive column mapping especially for insert is severely broken. I am not sure why the issue has remained hidden so far.
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
not tested
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ahshahid closed pull request #40765: [WIP][SPARK-43112]. Spark may use a column other than the actual specified partitioning column for partitioning, for Hive format tables
Posted by "ahshahid (via GitHub)" <gi...@apache.org>.
ahshahid closed pull request #40765: [WIP][SPARK-43112]. Spark may use a column other than the actual specified partitioning column for partitioning, for Hive format tables
URL: https://github.com/apache/spark/pull/40765
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ahshahid commented on pull request #40765: [WIP][SPARK-43112]. Spark may use a column other than the actual specified partitioning column for partitioning, for Hive format tables
Posted by "ahshahid (via GitHub)" <gi...@apache.org>.
ahshahid commented on PR #40765:
URL: https://github.com/apache/spark/pull/40765#issuecomment-1515576382
I will close this PR. Though there is problematic behaviour if the column name is present in column list as well as in partition clause, and in case of hive tables, that situation should be detected . But it is not that severe a bug, as for hive table format the partition col should not be present in both the places ( i was not aware of that).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ahshahid commented on pull request #40765: [WIP][SPARK-43112]. Spark may use a column other than the actual specified partitioning column for partitioning, for Hive format tables
Posted by "ahshahid (via GitHub)" <gi...@apache.org>.
ahshahid commented on PR #40765:
URL: https://github.com/apache/spark/pull/40765#issuecomment-1515576682
I will close this PR. Though there is problematic behaviour if the column name is present in column list as well as in partition clause, and in case of hive tables, that situation should be detected . But it is not that severe a bug, as for hive table format the partition col should not be present in both the places ( i was not aware of that).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org