You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2021/07/16 07:28:29 UTC

[GitHub] [spark] MaxGekk opened a new pull request #33387: [SPARK-36034][SQL][3.0] Rebase datetime in pushed down filters to parquet

MaxGekk opened a new pull request #33387:
URL: https://github.com/apache/spark/pull/33387


   ### What changes were proposed in this pull request?
   In the PR, I propose to propagate either the SQL config `spark.sql.parquet.datetimeRebaseModeInRead` or/and Parquet option `datetimeRebaseMode` to `ParquetFilters`. The `ParquetFilters` class uses the settings in conversions of dates/timestamps instances from datasource filters to values pushed via `FilterApi` to the `parquet-column` lib.
   
   Before the changes, date/timestamp values expressed as days/microseconds/milliseconds are interpreted as offsets in Proleptic Gregorian calendar, and pushed to the parquet library as is. That works fine if timestamp/dates values in parquet files were saved in the `CORRECTED` mode but in the `LEGACY` mode, filter's values could not match to actual values.
   
   After the changes, timestamp/dates values of filters pushed down to parquet libs such as `FilterApi.eq(col1, -719162)` are rebased according the rebase settings. For the example, if the rebase mode is `CORRECTED`, **-719162** is pushed down as is but if the current rebase mode is `LEGACY`, the number of days is rebased to **-719164**. For more context, the PR description https://github.com/apache/spark/pull/28067 shows the diffs between two calendars.
   
   ### Why are the changes needed?
   The changes fix the bug portrayed by the following example from SPARK-36034:
   ```scala
   In [27]: spark.conf.set("spark.sql.legacy.parquet.datetimeRebaseModeInWrite", "LEGACY")
   >>> spark.sql("SELECT DATE '0001-01-01' AS date").write.mode("overwrite").parquet("date_written_by_spark3_legacy")
   >>> spark.read.parquet("date_written_by_spark3_legacy").where("date = '0001-01-01'").show()
   +----+
   |date|
   +----+
   +----+
   ```
   The result must have the date value `0001-01-01`.
   
   ### Does this PR introduce _any_ user-facing change?
   In some sense, yes. Query results can be different in some cases. For the example above:
   ```scala
   scala> spark.conf.set("spark.sql.parquet.datetimeRebaseModeInWrite", "LEGACY")
   scala> spark.sql("SELECT DATE '0001-01-01' AS date").write.mode("overwrite").parquet("date_written_by_spark3_legacy")
   scala> spark.read.parquet("date_written_by_spark3_legacy").where("date = '0001-01-01'").show(false)
   +----------+
   |date      |
   +----------+
   |0001-01-01|
   +----------+
   ```
   
   ### How was this patch tested?
   By running the modified test suite `ParquetFilterSuite`:
   ```
   $ build/sbt "test:testOnly *ParquetV1FilterSuite"
   $ build/sbt "test:testOnly *ParquetV2FilterSuite"
   ```
   
   Authored-by: Max Gekk <max.gekkgmail.com>
   Signed-off-by: Hyukjin Kwon <gu...@apache.org>
   (cherry picked from commit b09b7f7cc024d3054debd7bdb51caec3b53764d7)
   (cherry picked from commit ba7117224c797e3ade64a281b1a165cf5040c541)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #33387: [SPARK-36034][SQL][3.0] Rebase datetime in pushed down filters to parquet

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33387:
URL: https://github.com/apache/spark/pull/33387#issuecomment-881416191


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/141142/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #33387: [SPARK-36034][SQL][3.0] Rebase datetime in pushed down filters to parquet

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33387:
URL: https://github.com/apache/spark/pull/33387#issuecomment-881314941


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/45653/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #33387: [SPARK-36034][SQL][3.0] Rebase datetime in pushed down filters to parquet

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33387:
URL: https://github.com/apache/spark/pull/33387#issuecomment-881314941


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/45653/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #33387: [SPARK-36034][SQL][3.0] Rebase datetime in pushed down filters to parquet

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33387:
URL: https://github.com/apache/spark/pull/33387#issuecomment-881243179


   **[Test build #141142 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/141142/testReport)** for PR 33387 at commit [`04732c7`](https://github.com/apache/spark/commit/04732c7ad997c559cfd5b902c985c06b75813a4a).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #33387: [SPARK-36034][SQL][3.0] Rebase datetime in pushed down filters to parquet

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33387:
URL: https://github.com/apache/spark/pull/33387#issuecomment-881276620


   Kubernetes integration test starting
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/45653/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA removed a comment on pull request #33387: [SPARK-36034][SQL][3.0] Rebase datetime in pushed down filters to parquet

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #33387:
URL: https://github.com/apache/spark/pull/33387#issuecomment-881243179


   **[Test build #141142 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/141142/testReport)** for PR 33387 at commit [`04732c7`](https://github.com/apache/spark/commit/04732c7ad997c559cfd5b902c985c06b75813a4a).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] MaxGekk commented on pull request #33387: [SPARK-36034][SQL][3.0] Rebase datetime in pushed down filters to parquet

Posted by GitBox <gi...@apache.org>.
MaxGekk commented on pull request #33387:
URL: https://github.com/apache/spark/pull/33387#issuecomment-881367810


   I cannot reproduce AQE failure locally:
   ```
   $ build/sbt "sql/test:testOnly *.AdaptiveQueryExecSuite"
   [info] AdaptiveQueryExecSuite:
   ...
   [info] - multiple joins (1 second, 187 milliseconds)
   [info] - multiple joins with aggregate (1 second, 225 milliseconds)
   [info] - multiple joins with aggregate 2 (1 second, 298 milliseconds)
   ...
   [info] Run completed in 1 minute, 12 seconds.
   [info] Total number of tests run: 33
   [info] Suites: completed 1, aborted 0
   [info] Tests: succeeded 33, failed 0, canceled 0, ignored 0, pending 0
   [info] All tests passed.
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] HyukjinKwon commented on pull request #33387: [SPARK-36034][SQL][3.0] Rebase datetime in pushed down filters to parquet

Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #33387:
URL: https://github.com/apache/spark/pull/33387#issuecomment-881411239


   Merged to bracnh-3.0!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #33387: [SPARK-36034][SQL][3.0] Rebase datetime in pushed down filters to parquet

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33387:
URL: https://github.com/apache/spark/pull/33387#issuecomment-881291113


   Kubernetes integration test status failure
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/45653/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] HyukjinKwon closed pull request #33387: [SPARK-36034][SQL][3.0] Rebase datetime in pushed down filters to parquet

Posted by GitBox <gi...@apache.org>.
HyukjinKwon closed pull request #33387:
URL: https://github.com/apache/spark/pull/33387


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] HyukjinKwon commented on pull request #33387: [SPARK-36034][SQL][3.0] Rebase datetime in pushed down filters to parquet

Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #33387:
URL: https://github.com/apache/spark/pull/33387#issuecomment-881368428


   I retriggered


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #33387: [SPARK-36034][SQL][3.0] Rebase datetime in pushed down filters to parquet

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33387:
URL: https://github.com/apache/spark/pull/33387#issuecomment-881406856


   **[Test build #141142 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/141142/testReport)** for PR 33387 at commit [`04732c7`](https://github.com/apache/spark/commit/04732c7ad997c559cfd5b902c985c06b75813a4a).
    * This patch **fails SparkR unit tests**.
    * This patch merges cleanly.
    * This patch adds no public classes.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] HyukjinKwon commented on pull request #33387: [SPARK-36034][SQL][3.0] Rebase datetime in pushed down filters to parquet

Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #33387:
URL: https://github.com/apache/spark/pull/33387#issuecomment-881352526


   I made a PR to fix the R tests (https://github.com/apache/spark/pull/33390). but AQE test failure seems new


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #33387: [SPARK-36034][SQL][3.0] Rebase datetime in pushed down filters to parquet

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33387:
URL: https://github.com/apache/spark/pull/33387#issuecomment-881416191


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/141142/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org