You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2020/09/01 16:57:37 UTC
[GitHub] [spark] rdefreitas opened a new pull request #29613: [SPARK-32768] Add Parquet Timestamp output configuration to docs
rdefreitas opened a new pull request #29613:
URL: https://github.com/apache/spark/pull/29613
### What changes were proposed in this pull request?
Spark 2.3.0 added the spark.sql.parquet.outputTimestampType configuration option for controlling the underlying datatype used when writing Timestamp column types into parquet files.
### Why are the changes needed?
This was never exposed in the documentation. Fix should be applied to docs for both to the next 3.x release and 2.4.x release
### Does this PR introduce _any_ user-facing change?
Yes, updates the documentation on the parquet datasource page
### How was this patch tested?
Docs change only. Visual test
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29613: [SPARK-32768][docs] Add Parquet Timestamp output configuration to docs
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29613:
URL: https://github.com/apache/spark/pull/29613#issuecomment-685003676
Can one of the admins verify this patch?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #29613: [SPARK-32768][docs] Add Parquet Timestamp output configuration to docs
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #29613:
URL: https://github.com/apache/spark/pull/29613#discussion_r482191500
##########
File path: docs/sql-data-sources-parquet.md
##########
@@ -278,6 +278,19 @@ Configuration of Parquet can be done using the `setConf` method on `SparkSession
</td>
<td>1.3.0</td>
</tr>
+<tr>
+ <td><code>spark.sql.parquet.outputTimestampType</code></td>
Review comment:
Probably we could add a link to the SQL configuration page to avoid duplication.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29613: [SPARK-32768][docs] Add Parquet Timestamp output configuration to docs
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29613:
URL: https://github.com/apache/spark/pull/29613#issuecomment-685002820
Can one of the admins verify this patch?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon closed pull request #29613: [SPARK-32768][docs] Add Parquet Timestamp output configuration to docs
Posted by GitBox <gi...@apache.org>.
HyukjinKwon closed pull request #29613:
URL: https://github.com/apache/spark/pull/29613
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] rdefreitas commented on a change in pull request #29613: [SPARK-32768][docs] Add Parquet Timestamp output configuration to docs
Posted by GitBox <gi...@apache.org>.
rdefreitas commented on a change in pull request #29613:
URL: https://github.com/apache/spark/pull/29613#discussion_r482172230
##########
File path: docs/sql-data-sources-parquet.md
##########
@@ -278,6 +278,19 @@ Configuration of Parquet can be done using the `setConf` method on `SparkSession
</td>
<td>1.3.0</td>
</tr>
+<tr>
+ <td><code>spark.sql.parquet.outputTimestampType</code></td>
Review comment:
@HyukjinKwon thanks! I didn't see that.
That said, anyone following the parquet guide doesn't see all the parquet options. Perhaps we change this to link back to the general config docs and state that all parquet options can be found there?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] rdefreitas commented on pull request #29613: [SPARK-32768][docs] Add Parquet Timestamp output configuration to docs
Posted by GitBox <gi...@apache.org>.
rdefreitas commented on pull request #29613:
URL: https://github.com/apache/spark/pull/29613#issuecomment-685057544
seems the test report is failing - is this due to it being a docs only change?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #29613: [SPARK-32768][docs] Add Parquet Timestamp output configuration to docs
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #29613:
URL: https://github.com/apache/spark/pull/29613#discussion_r481521668
##########
File path: docs/sql-data-sources-parquet.md
##########
@@ -278,6 +278,19 @@ Configuration of Parquet can be done using the `setConf` method on `SparkSession
</td>
<td>1.3.0</td>
</tr>
+<tr>
+ <td><code>spark.sql.parquet.outputTimestampType</code></td>
Review comment:
I think all sql confs are now documented at https://spark.apache.org/docs/latest/configuration.html#spark-sql
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] rdefreitas commented on a change in pull request #29613: [SPARK-32768][docs] Add Parquet Timestamp output configuration to docs
Posted by GitBox <gi...@apache.org>.
rdefreitas commented on a change in pull request #29613:
URL: https://github.com/apache/spark/pull/29613#discussion_r482172230
##########
File path: docs/sql-data-sources-parquet.md
##########
@@ -278,6 +278,19 @@ Configuration of Parquet can be done using the `setConf` method on `SparkSession
</td>
<td>1.3.0</td>
</tr>
+<tr>
+ <td><code>spark.sql.parquet.outputTimestampType</code></td>
Review comment:
@HyukjinKwon thanks! I didn't see that.
That said, anyone following the parquet guide doesn't see all the parquet options. Perhaps this should be changed this to link back to the general config docs and state that all parquet options can be found there?
Not having all options on the guide page, or at least an indicator that there are other values, is a user experience issue.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29613: [SPARK-32768][docs] Add Parquet Timestamp output configuration to docs
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29613:
URL: https://github.com/apache/spark/pull/29613#issuecomment-685002820
Can one of the admins verify this patch?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org