You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "huangxiaopingRD (via GitHub)" <gi...@apache.org> on 2023/02/27 13:00:44 UTC
[GitHub] [spark] huangxiaopingRD opened a new pull request, #40196: [SPARK-42603][SQL] Set spark.sql.legacy.createHiveTableByDefault to f…
huangxiaopingRD opened a new pull request, #40196:
URL: https://github.com/apache/spark/pull/40196
### What changes were proposed in this pull request?
Set `spark.sql.legacy.createHiveTableByDefault ` to false.
### Why are the changes needed?
In the spark [documentation](https://github.com/apache/spark/blob/master/docs/sql-ref-syntax-ddl-create-table-datasource.md?plain=1#L121), it is stated that the default creation table is parquet, but you need to set "spark.sql.legacy.createHiveTableByDefault" to false, otherwise the default is textfile.
### Does this PR introduce _any_ user-facing change?
Yes
### How was this patch tested?
tests were added
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] zzzzming95 commented on pull request #40196: [SPARK-42603][SQL] Set spark.sql.legacy.createHiveTableByDefault to false.
Posted by "zzzzming95 (via GitHub)" <gi...@apache.org>.
zzzzming95 commented on PR #40196:
URL: https://github.com/apache/spark/pull/40196#issuecomment-1449371301
In fact, the describtion of `spark.sql.legacy.createHiveTableByDefault` from `SQLConf` is wrong.
In my test, the hive table format depends on the conf `hive.default.fileformat` when set `spark.sql.legacy.createHiveTableByDefault` is true . And if `spark.sql.legacy.createHiveTableByDefault` is false, it hive table format depends on the conf `spark.sql.sources.default` .
So maybe we should fix the wrong describtion of `spark.sql.legacy.createHiveTableByDefault`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] zzzzming95 commented on pull request #40196: [SPARK-42603][SQL] Set spark.sql.legacy.createHiveTableByDefault to false.
Posted by "zzzzming95 (via GitHub)" <gi...@apache.org>.
zzzzming95 commented on PR #40196:
URL: https://github.com/apache/spark/pull/40196#issuecomment-1449375189
In my test, the hive table format depends on the conf `hive.default.fileformat` when set `spark.sql.legacy.createHiveTableByDefault` is true . And if `spark.sql.legacy.createHiveTableByDefault` is false, it hive table format depends on the conf `spark.sql.sources.default` .
I think it's reasonable .
So maybe we should only fix the describtion of the https://github.com/apache/spark/blob/master/docs/sql-ref-syntax-ddl-create-table-datasource.md?plain=1#L121 ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] huangxiaopingRD closed pull request #40196: [SPARK-42603][SQL] Set spark.sql.legacy.createHiveTableByDefault to false.
Posted by "huangxiaopingRD (via GitHub)" <gi...@apache.org>.
huangxiaopingRD closed pull request #40196: [SPARK-42603][SQL] Set spark.sql.legacy.createHiveTableByDefault to false.
URL: https://github.com/apache/spark/pull/40196
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] huangxiaopingRD commented on pull request #40196: [SPARK-42603][SQL] Set spark.sql.legacy.createHiveTableByDefault to false.
Posted by "huangxiaopingRD (via GitHub)" <gi...@apache.org>.
huangxiaopingRD commented on PR #40196:
URL: https://github.com/apache/spark/pull/40196#issuecomment-1447471031
> >
>
>
> You are suggesting a breaking change here. The existing behavior is correct and consistent with old Spark versions. I guess we need to revise the test comment instead, @huangxiaopingRD .
Sorry, I don't understand what you say we should do next. I want to make this change just because I think the default format of Spark's data source is parquet, and I think the default fileformat of "`create table`" should be consistent with the default value (parquet) of "spark.sql.sources.default".
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] huangxiaopingRD commented on pull request #40196: [SPARK-42603][SQL] Set spark.sql.legacy.createHiveTableByDefault to false.
Posted by "huangxiaopingRD (via GitHub)" <gi...@apache.org>.
huangxiaopingRD commented on PR #40196:
URL: https://github.com/apache/spark/pull/40196#issuecomment-1447470431
>
Sorry, I don't understand what you say we should do next. I want to make this change just because I think the default format of Spark's data source is parquet, and I think the default fileformat of "`create table`" should be consistent with the default value (parquet) of "spark.sql.sources.default".
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] huangxiaopingRD commented on pull request #40196: [SPARK-42603][SQL] Set spark.sql.legacy.createHiveTableByDefault to false.
Posted by "huangxiaopingRD (via GitHub)" <gi...@apache.org>.
huangxiaopingRD commented on PR #40196:
URL: https://github.com/apache/spark/pull/40196#issuecomment-1450064629
> In my test, the hive table format depends on the conf `hive.default.fileformat` when set `spark.sql.legacy.createHiveTableByDefault` is true . And if `spark.sql.legacy.createHiveTableByDefault` is false, it hive table format depends on the conf `spark.sql.sources.default` .
>
> I think it's reasonable .
>
> So maybe we should only fix the describtion of the https://github.com/apache/spark/blob/master/docs/sql-ref-syntax-ddl-create-table-datasource.md?plain=1#L121 ?
I agree with you. I will launch another PR to modify the description of the document. Thanks @zzzzming95
what do you think about this? @dongjoon-hyun
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org