You are viewing a plain text version of this content. The canonical link for it is here.

Posted to reviews@spark.apache.org by HyukjinKwon <gi...@git.apache.org> on 2017/11/06 14:50:49 UTC

[GitHub] spark pull request #19674: [DO-NOT-MERGE] Investigate test failures related ...

GitHub user HyukjinKwon opened a pull request:

    https://github.com/apache/spark/pull/19674

    [DO-NOT-MERGE] Investigate test failures related with SPARK-22395

    ## What changes were proposed in this pull request?
    
    This PR is not for merging. There are unexpected and unrepducible test failures, specifically in Jenkins. This PR targets to test and investigate it.
    
    ## How was this patch tested?
    
    N/A


You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/HyukjinKwon/spark debug-SPARK-22395

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/spark/pull/19674.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #19674
    
----
commit 4735e5981ecf3a4bce50ce86f706e25830f4a801
Author: Takuya UESHIN <ue...@databricks.com>
Date:   2017-10-23T06:27:22Z

    Add a conf to make Pandas DataFrame respect session local timezone.

commit 1f85150dc5b26df21dca6bad2ef4eaec342c4400
Author: Takuya UESHIN <ue...@databricks.com>
Date:   2017-10-23T08:09:16Z

    Fix toPandas() behavior.

commit 5c08ecf247bfe7e14afcdef8eba1c25cb3b68634
Author: Takuya UESHIN <ue...@databricks.com>
Date:   2017-10-23T09:15:47Z

    Modify pandas UDFs to respect session timezone.

commit ee1a1c87e2a89974e4e299f4ad84e5831526d079
Author: Takuya UESHIN <ue...@databricks.com>
Date:   2017-11-01T04:38:30Z

    Workaround for old pandas.

commit b1436b8e4876838efbcb38cf0bffa7ebcc7a5544
Author: Takuya UESHIN <ue...@databricks.com>
Date:   2017-11-01T07:53:13Z

    Don't use is_datetime64tz_dtype for old pandas.

commit 6872516e8cd9d7f81929c38708571c69a0af7883
Author: Takuya UESHIN <ue...@databricks.com>
Date:   2017-11-01T14:27:34Z

    Fix one of the failed tests.

commit 1f096bf32f742945363cc7d9af978041ad77408b
Author: Takuya UESHIN <ue...@databricks.com>
Date:   2017-11-02T05:41:09Z

    Modify check_data udf for debug messages.

commit 569bb633116b35cd717e9f27b2264e1114f8e35d
Author: Takuya UESHIN <ue...@databricks.com>
Date:   2017-11-03T06:05:46Z

    Remove unused method.

commit ce07f39643ef9711419e7e11082e62c578d816f5
Author: Takuya UESHIN <ue...@databricks.com>
Date:   2017-11-03T06:15:03Z

    Modify a test.

commit ba3d6e3cf679e3db0a2e095f8cbe099ab4260532
Author: Takuya UESHIN <ue...@databricks.com>
Date:   2017-11-06T06:59:43Z

    Add debug print, which will be removed later.

commit 9101a3a12f17b5bd633756139eaa2cb3ee9bb33c
Author: Takuya UESHIN <ue...@databricks.com>
Date:   2017-11-06T07:06:07Z

    Fix style.

commit 256bf9abc4004fafb40b09cc63c68103a1664f87
Author: hyukjinkwon <gu...@gmail.com>
Date:   2017-11-06T14:47:24Z

    Debug in Jenkins

----


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #19674: [DO-NOT-MERGE] Investigate test failures related with SP...

Posted by AmplabJenkins <gi...@git.apache.org>.

Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/19674
  
    Test FAILed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/83541/
    Test FAILed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #19674: [DO-NOT-MERGE] Investigate test failures related with SP...

Posted by AmplabJenkins <gi...@git.apache.org>.

Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/19674
  
    Merged build finished. Test FAILed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #19674: [DO-NOT-MERGE] Investigate test failures related with SP...

Posted by SparkQA <gi...@git.apache.org>.

Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/19674
  
    **[Test build #83522 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83522/testReport)** for PR 19674 at commit [`d66044d`](https://github.com/apache/spark/commit/d66044d0eb19db60d752b4716046fd9810d7524b).


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #19674: [DO-NOT-MERGE] Investigate test failures related with SP...

Posted by SparkQA <gi...@git.apache.org>.

Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/19674
  
    **[Test build #83494 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83494/testReport)** for PR 19674 at commit [`256bf9a`](https://github.com/apache/spark/commit/256bf9abc4004fafb40b09cc63c68103a1664f87).


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #19674: [DO-NOT-MERGE] Investigate test failures related with SP...

Posted by SparkQA <gi...@git.apache.org>.

Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/19674
  
    **[Test build #83517 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83517/testReport)** for PR 19674 at commit [`db93034`](https://github.com/apache/spark/commit/db93034142969189412f805326b6dc40cc92a0d2).


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #19674: [DO-NOT-MERGE] Investigate test failures related with SP...

Posted by AmplabJenkins <gi...@git.apache.org>.

Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/19674
  
    Merged build finished. Test FAILed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #19674: [DO-NOT-MERGE] Investigate test failures related with SP...

Posted by HyukjinKwon <gi...@git.apache.org>.

Github user HyukjinKwon commented on the issue:

    https://github.com/apache/spark/pull/19674
  
    retest this please


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #19674: [DO-NOT-MERGE] Investigate test failures related with SP...

Posted by SparkQA <gi...@git.apache.org>.

Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/19674
  
    **[Test build #83497 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83497/testReport)** for PR 19674 at commit [`c88e5cd`](https://github.com/apache/spark/commit/c88e5cd694ddf68e65e695af54966c25c6095b71).
     * This patch **fails Python style tests**.
     * This patch merges cleanly.
     * This patch adds no public classes.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #19674: [DO-NOT-MERGE] Investigate test failures related with SP...

Posted by AmplabJenkins <gi...@git.apache.org>.

Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/19674
  
    Merged build finished. Test FAILed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #19674: [DO-NOT-MERGE] Investigate test failures related with SP...

Posted by AmplabJenkins <gi...@git.apache.org>.

Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/19674
  
    Test FAILed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/83511/
    Test FAILed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #19674: [DO-NOT-MERGE] Investigate test failures related with SP...

Posted by HyukjinKwon <gi...@git.apache.org>.

Github user HyukjinKwon commented on the issue:

    https://github.com/apache/spark/pull/19674
  
    @ueshin, now I am getting what you meant by DST. I think you roughly knew this problem but let me describe it more given my debugging.
    
    Looks the problem is in `mktime` and sounds `mktime` is platform-dependent (roughly assuming from codes and docs).
    
    I made a minimised reproducer:
    
    ```python
    import time
    import os
    from datetime import datetime
    os.environ["TZ"] = "America/Los_Angeles"
    time.tzset()
    time.mktime(datetime(2100, 4, 4, 4, 4, 4).timetuple())
    ```
    
    My local, it prints: 
    
    ```
    4110523444.0
    ```
    
    On Unbuntu 14.04:
    
    ```
    4110519844.0
    ```
    
    Jenkins, it prints:
    
    ```
    4110519844.0
    ```
    
    I am not sure if this is easily fixable within Spark as it looks dependent on Python implementation and/or the underlying C library, up to my knowledge and from my reading some docs.
    
    Could you maybe avoid this time within DST for now in your PR? I currently don't have a good idea to fix this with a simple and surgical fix


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #19674: [DO-NOT-MERGE] Investigate test failures related with SP...

Posted by AmplabJenkins <gi...@git.apache.org>.

Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/19674
  
    Merged build finished. Test FAILed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #19674: [DO-NOT-MERGE] Investigate test failures related with SP...

Posted by SparkQA <gi...@git.apache.org>.

Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/19674
  
    **[Test build #83517 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83517/testReport)** for PR 19674 at commit [`db93034`](https://github.com/apache/spark/commit/db93034142969189412f805326b6dc40cc92a0d2).
     * This patch passes all tests.
     * This patch merges cleanly.
     * This patch adds no public classes.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #19674: [DO-NOT-MERGE] Investigate test failures related with SP...

Posted by AmplabJenkins <gi...@git.apache.org>.

Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/19674
  
    Merged build finished. Test FAILed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #19674: [DO-NOT-MERGE] Investigate test failures related with SP...

Posted by AmplabJenkins <gi...@git.apache.org>.

Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/19674
  
    Merged build finished. Test FAILed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #19674: [DO-NOT-MERGE] Investigate test failures related with SP...

Posted by HyukjinKwon <gi...@git.apache.org>.

Github user HyukjinKwon commented on the issue:

    https://github.com/apache/spark/pull/19674
  
    Let me leave this closed!


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #19674: [DO-NOT-MERGE] Investigate test failures related with SP...

Posted by HyukjinKwon <gi...@git.apache.org>.

Github user HyukjinKwon commented on the issue:

    https://github.com/apache/spark/pull/19674
  
    I can directly add anything if you want to check, @ueshin. Just feel free to let me know, or I also support to open another PR to figure out if you are willing to. Meanwhile, let me print out few things I want to check.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #19674: [DO-NOT-MERGE] Investigate test failures related with SP...

Posted by SparkQA <gi...@git.apache.org>.

Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/19674
  
    **[Test build #83533 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83533/testReport)** for PR 19674 at commit [`16e0614`](https://github.com/apache/spark/commit/16e0614300b092688e1d9f09a4575918e5d7358a).
     * This patch **fails PySpark unit tests**.
     * This patch merges cleanly.
     * This patch adds no public classes.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #19674: [DO-NOT-MERGE] Investigate test failures related with SP...

Posted by SparkQA <gi...@git.apache.org>.

Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/19674
  
    **[Test build #83502 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83502/testReport)** for PR 19674 at commit [`da35a31`](https://github.com/apache/spark/commit/da35a31dafb3d961081437222bbf661a39188c72).


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #19674: [DO-NOT-MERGE] Investigate test failures related with SP...

Posted by SparkQA <gi...@git.apache.org>.

Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/19674
  
    **[Test build #83541 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83541/testReport)** for PR 19674 at commit [`3ef8ec0`](https://github.com/apache/spark/commit/3ef8ec0acb23cd6e10346dbbb6d2d6510e019ac1).


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #19674: [DO-NOT-MERGE] Investigate test failures related with SP...

Posted by AmplabJenkins <gi...@git.apache.org>.

Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/19674
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/83512/
    Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #19674: [DO-NOT-MERGE] Investigate test failures related with SP...

Posted by AmplabJenkins <gi...@git.apache.org>.

Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/19674
  
    Test FAILed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/83539/
    Test FAILed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #19674: [DO-NOT-MERGE] Investigate test failures related with SP...

Posted by AmplabJenkins <gi...@git.apache.org>.

Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/19674
  
    Merged build finished. Test FAILed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #19674: [DO-NOT-MERGE] Investigate test failures related with SP...

Posted by SparkQA <gi...@git.apache.org>.

Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/19674
  
    **[Test build #83501 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83501/testReport)** for PR 19674 at commit [`9656d8a`](https://github.com/apache/spark/commit/9656d8a624c4f115aa3759d069a118c742ead65f).
     * This patch **fails PySpark unit tests**.
     * This patch merges cleanly.
     * This patch adds no public classes.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #19674: [DO-NOT-MERGE] Investigate test failures related with SP...

Posted by HyukjinKwon <gi...@git.apache.org>.

Github user HyukjinKwon commented on the issue:

    https://github.com/apache/spark/pull/19674
  
    Thanks for quick response. Will try to produce a matrix for it while I am here.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #19674: [DO-NOT-MERGE] Investigate test failures related with SP...

Posted by AmplabJenkins <gi...@git.apache.org>.

Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/19674
  
    Merged build finished. Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #19674: [DO-NOT-MERGE] Investigate test failures related with SP...

Posted by AmplabJenkins <gi...@git.apache.org>.

Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/19674
  
    Test FAILed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/83494/
    Test FAILed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #19674: [DO-NOT-MERGE] Investigate test failures related with SP...

Posted by SparkQA <gi...@git.apache.org>.

Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/19674
  
    **[Test build #83533 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83533/testReport)** for PR 19674 at commit [`16e0614`](https://github.com/apache/spark/commit/16e0614300b092688e1d9f09a4575918e5d7358a).


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #19674: [DO-NOT-MERGE] Investigate test failures related with SP...

Posted by AmplabJenkins <gi...@git.apache.org>.

Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/19674
  
    Test FAILed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/83522/
    Test FAILed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #19674: [DO-NOT-MERGE] Investigate test failures related with SP...

Posted by AmplabJenkins <gi...@git.apache.org>.

Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/19674
  
    Test FAILed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/83499/
    Test FAILed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #19674: [DO-NOT-MERGE] Investigate test failures related with SP...

Posted by ueshin <gi...@git.apache.org>.

Github user ueshin commented on the issue:

    https://github.com/apache/spark/pull/19674
  
    @HyukjinKwon Awesome! Thanks for the investigation!
    I didn't think the "platform-dependent" is so dependent.
    I'll update my pr to avoid the time within DST. Thanks!


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #19674: [DO-NOT-MERGE] Investigate test failures related with SP...

Posted by AmplabJenkins <gi...@git.apache.org>.

Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/19674
  
    Merged build finished. Test FAILed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #19674: [DO-NOT-MERGE] Investigate test failures related with SP...

Posted by AmplabJenkins <gi...@git.apache.org>.

Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/19674
  
    Test FAILed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/83501/
    Test FAILed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org