[jira] [Commented] (LIVY-399) Enable real test for PySpark and SparkR interpreters

["ASF GitHub Bot (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/LIVY-399?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16196518#comment-16196518 ] 

ASF GitHub Bot commented on LIVY-399:
-------------------------------------

GitHub user jerryshao reopened a pull request:

    https://github.com/apache/incubator-livy/pull/49

    [LIVY-399][TEST] Enable real test for PySpark and SparkR interpreters

    Currently because we lack pyspark and sparkr dependencies in our environment, so we neglect the pyspark and sparkr integration tests, here propose a way to enable these integration tests. The solution is mainly borrowed from Zeppelin - to download Spark binary before integration tests, so that we will have a complete Spark environment to do the test. Because of this several modules like minicluster-dependencies are not required, so here removed such modules. 
    
    Besides, we proposed to remove real test implementations, since we never used real test before and it mainly did what scripts should do with Scala code, which makes it hard to maintain. Also for now mini-cluster test is quite close to real test, so it is not so useful to still keep real test.

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/jerryshao/incubator-livy LIVY-399

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/incubator-livy/pull/49.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #49
    
----
commit f91c57f348f9c7098c92cd25ff60df989eb419ad
Author: jerryshao <ss...@hortonworks.com>
Date:   2017-09-21T01:46:57Z

    Enable real test for PySpark and SparkR interpreters
    
    Change-Id: I860e30fdeded40a35479ec1823e00fb302fe8238

commit e9e70237cd0de200ff59a470a0859e6bb5f22171
Author: jerryshao <ss...@hortonworks.com>
Date:   2017-09-21T03:14:01Z

    temporary changes to travis.yml to show failure log

commit 15f84a9a915d2ca06c6cf742c40f0f2ba03873dc
Author: jerryshao <ss...@hortonworks.com>
Date:   2017-09-21T03:59:39Z

    Travis changes

commit d4cc3d517d8628ad4abba9fc053ccbfefda6bc44
Author: jerryshao <ss...@hortonworks.com>
Date:   2017-09-21T05:35:55Z

    Revert travis changes

commit e8ec20f87cbefa6898cda8cf7d56d63c33908331
Author: jerryshao <ss...@hortonworks.com>
Date:   2017-09-25T01:11:58Z

    Print out log if travis test is failed

commit d96ff77701e27ab63e46ec2e4c0be22d8220057d
Author: jerryshao <ss...@hortonworks.com>
Date:   2017-09-25T01:16:34Z

    Style fix

commit 5f223d4ee2d5cdea7f2f3200f1a587dc027f8440
Author: jerryshao <ss...@hortonworks.com>
Date:   2017-09-25T05:38:23Z

    Increase the timeout to reduce flakiness

commit f033fa9185907dbf3a3bc80dccefebc059718410
Author: jerryshao <ss...@hortonworks.com>
Date:   2017-09-25T07:33:33Z

    Add some test log

commit 3f3decbcb1664ec143e61325516c2cf87440fee4
Author: jerryshao <ss...@hortonworks.com>
Date:   2017-09-26T06:09:24Z

    Add more log to identify the issue

----


> Enable real test for PySpark and SparkR interpreters
> ----------------------------------------------------
>
>                 Key: LIVY-399
>                 URL: https://issues.apache.org/jira/browse/LIVY-399
>             Project: Livy
>          Issue Type: Sub-task
>          Components: REPL
>    Affects Versions: 0.4
>            Reporter: Saisai Shao
>            Assignee: Saisai Shao
>          Time Spent: 2h
>  Remaining Estimate: 0h
>
> Currently because we lack pyspark and sparkr dependencies in our environment, so we neglect the pyspark and sparkr integration test, also the UT for PySpark and SparkR doesn't involve Spark things, only test the plain python and R REPL, so we should figure out a way to support real test for PySpark and SparkR interpreters.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)