You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by ueshin <gi...@git.apache.org> on 2018/02/06 07:47:27 UTC

[GitHub] spark pull request #20515: [SPARK-23290][SQL][PYTHON][BACKPORT-2.3] Use date...

GitHub user ueshin opened a pull request:

    https://github.com/apache/spark/pull/20515

    [SPARK-23290][SQL][PYTHON][BACKPORT-2.3] Use datetime.date for date type when converting Spark DataFrame to Pandas DataFrame.

    ## What changes were proposed in this pull request?
    
    This is a backport of #20506.
    
    In #18664, there was a change in how `DateType` is being returned to users ([line 1968 in dataframe.py](https://github.com/apache/spark/pull/18664/files#diff-6fc344560230bf0ef711bb9b5573f1faR1968)). This can cause client code which works in Spark 2.2 to fail.
    See [SPARK-23290](https://issues.apache.org/jira/browse/SPARK-23290?focusedCommentId=16350917&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16350917) for an example.
    
    This pr modifies to use `datetime.date` for date type as Spark 2.2 does.
    
    ## How was this patch tested?
    
    Tests modified to fit the new behavior and existing tests.


You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/ueshin/apache-spark issues/SPARK-23290_2.3

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/spark/pull/20515.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #20515
    
----
commit b489f4a0d4fa25fd51d9db78bd01fc972e4e0dd4
Author: Takuya UESHIN <ue...@...>
Date:   2018-02-06T06:52:25Z

    [SPARK-23290][SQL][PYTHON] Use datetime.date for date type when converting Spark DataFrame to Pandas DataFrame.
    
    ## What changes were proposed in this pull request?
    
    In #18664, there was a change in how `DateType` is being returned to users ([line 1968 in dataframe.py](https://github.com/apache/spark/pull/18664/files#diff-6fc344560230bf0ef711bb9b5573f1faR1968)). This can cause client code which works in Spark 2.2 to fail.
    See [SPARK-23290](https://issues.apache.org/jira/browse/SPARK-23290?focusedCommentId=16350917&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16350917) for an example.
    
    This pr modifies to use `datetime.date` for date type as Spark 2.2 does.
    
    ## How was this patch tested?
    
    Tests modified to fit the new behavior and existing tests.
    
    Author: Takuya UESHIN <ue...@databricks.com>
    
    Closes #20506 from ueshin/issues/SPARK-23290.

----


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #20515: [SPARK-23290][SQL][PYTHON][BACKPORT-2.3] Use datetime.da...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/20515
  
    Merged build finished. Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #20515: [SPARK-23290][SQL][PYTHON][BACKPORT-2.3] Use datetime.da...

Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on the issue:

    https://github.com/apache/spark/pull/20515
  
    retest this please


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #20515: [SPARK-23290][SQL][PYTHON][BACKPORT-2.3] Use datetime.da...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/20515
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/621/
    Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #20515: [SPARK-23290][SQL][PYTHON][BACKPORT-2.3] Use datetime.da...

Posted by ueshin <gi...@git.apache.org>.
Github user ueshin commented on the issue:

    https://github.com/apache/spark/pull/20515
  
    cc @cloud-fan 


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark pull request #20515: [SPARK-23290][SQL][PYTHON][BACKPORT-2.3] Use date...

Posted by ueshin <gi...@git.apache.org>.
Github user ueshin closed the pull request at:

    https://github.com/apache/spark/pull/20515


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #20515: [SPARK-23290][SQL][PYTHON][BACKPORT-2.3] Use datetime.da...

Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on the issue:

    https://github.com/apache/spark/pull/20515
  
    Merged to branch-2.3.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #20515: [SPARK-23290][SQL][PYTHON][BACKPORT-2.3] Use datetime.da...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/20515
  
    Merged build finished. Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #20515: [SPARK-23290][SQL][PYTHON][BACKPORT-2.3] Use datetime.da...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/20515
  
    Merged build finished. Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #20515: [SPARK-23290][SQL][PYTHON][BACKPORT-2.3] Use datetime.da...

Posted by ueshin <gi...@git.apache.org>.
Github user ueshin commented on the issue:

    https://github.com/apache/spark/pull/20515
  
    Thanks! @HyukjinKwon @cloud-fan 


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #20515: [SPARK-23290][SQL][PYTHON][BACKPORT-2.3] Use datetime.da...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/20515
  
    **[Test build #87095 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/87095/testReport)** for PR 20515 at commit [`b489f4a`](https://github.com/apache/spark/commit/b489f4a0d4fa25fd51d9db78bd01fc972e4e0dd4).
     * This patch **fails PySpark unit tests**.
     * This patch merges cleanly.
     * This patch adds no public classes.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #20515: [SPARK-23290][SQL][PYTHON][BACKPORT-2.3] Use datetime.da...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/20515
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/618/
    Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #20515: [SPARK-23290][SQL][PYTHON][BACKPORT-2.3] Use datetime.da...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/20515
  
    Test FAILed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/87094/
    Test FAILed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #20515: [SPARK-23290][SQL][PYTHON][BACKPORT-2.3] Use datetime.da...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/20515
  
    Merged build finished. Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #20515: [SPARK-23290][SQL][PYTHON][BACKPORT-2.3] Use datetime.da...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/20515
  
    Merged build finished. Test FAILed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #20515: [SPARK-23290][SQL][PYTHON][BACKPORT-2.3] Use datetime.da...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/20515
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/615/
    Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #20515: [SPARK-23290][SQL][PYTHON][BACKPORT-2.3] Use datetime.da...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/20515
  
    **[Test build #87094 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/87094/testReport)** for PR 20515 at commit [`b489f4a`](https://github.com/apache/spark/commit/b489f4a0d4fa25fd51d9db78bd01fc972e4e0dd4).
     * This patch **fails due to an unknown error code, -9**.
     * This patch merges cleanly.
     * This patch adds no public classes.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #20515: [SPARK-23290][SQL][PYTHON][BACKPORT-2.3] Use datetime.da...

Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on the issue:

    https://github.com/apache/spark/pull/20515
  
    LGTM


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #20515: [SPARK-23290][SQL][PYTHON][BACKPORT-2.3] Use datetime.da...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/20515
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/87099/
    Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #20515: [SPARK-23290][SQL][PYTHON][BACKPORT-2.3] Use datetime.da...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/20515
  
    Merged build finished. Test FAILed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #20515: [SPARK-23290][SQL][PYTHON][BACKPORT-2.3] Use datetime.da...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/20515
  
    **[Test build #87099 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/87099/testReport)** for PR 20515 at commit [`b489f4a`](https://github.com/apache/spark/commit/b489f4a0d4fa25fd51d9db78bd01fc972e4e0dd4).


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #20515: [SPARK-23290][SQL][PYTHON][BACKPORT-2.3] Use datetime.da...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/20515
  
    Test FAILed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/87095/
    Test FAILed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #20515: [SPARK-23290][SQL][PYTHON][BACKPORT-2.3] Use datetime.da...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/20515
  
    **[Test build #87095 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/87095/testReport)** for PR 20515 at commit [`b489f4a`](https://github.com/apache/spark/commit/b489f4a0d4fa25fd51d9db78bd01fc972e4e0dd4).


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #20515: [SPARK-23290][SQL][PYTHON][BACKPORT-2.3] Use datetime.da...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/20515
  
    **[Test build #87099 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/87099/testReport)** for PR 20515 at commit [`b489f4a`](https://github.com/apache/spark/commit/b489f4a0d4fa25fd51d9db78bd01fc972e4e0dd4).
     * This patch passes all tests.
     * This patch merges cleanly.
     * This patch adds no public classes.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #20515: [SPARK-23290][SQL][PYTHON][BACKPORT-2.3] Use datetime.da...

Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on the issue:

    https://github.com/apache/spark/pull/20515
  
    retest this please


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #20515: [SPARK-23290][SQL][PYTHON][BACKPORT-2.3] Use datetime.da...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/20515
  
    **[Test build #87094 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/87094/testReport)** for PR 20515 at commit [`b489f4a`](https://github.com/apache/spark/commit/b489f4a0d4fa25fd51d9db78bd01fc972e4e0dd4).


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org