You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@zeppelin.apache.org by Leemoonsoo <gi...@git.apache.org> on 2015/08/23 02:16:01 UTC

[GitHub] incubator-zeppelin pull request: ZEPPELIN-256 Self diagnosis spark...

GitHub user Leemoonsoo opened a pull request:

    https://github.com/apache/incubator-zeppelin/pull/246

    ZEPPELIN-256 Self diagnosis spark configuration

    https://issues.apache.org/jira/browse/ZEPPELIN-256
    
    This patch adds capability of self diagnosis of configuration for spark interpreter.
    User can set "zeppelin.spark.diagnosis" property "true" to enable this feature, in Interpreter page.
    
    When it is turned on, SparkConfValidator checks validity of combination of environment variables and properties. It checks actual existence of specified directories (eg. SPARK_HOME, etc) and files (eg. pyspark.zip), according to given configuration.
    When it detects some problem, it prints error as a result message in a notebook screen. So user can see what was missed.
    


You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/Leemoonsoo/incubator-zeppelin spark_conf_validate

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/incubator-zeppelin/pull/246.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #246
    
----
commit c92f18534d85e318e41fe53c0cf684b3805be719
Author: Lee moon soo <mo...@apache.org>
Date:   2015-08-21T17:56:43Z

    Separate spark-dependency into submodule

commit c49c154bc1ca6ebd783022b774e4ff92c37c8d4c
Author: Lee moon soo <mo...@apache.org>
Date:   2015-08-21T17:59:47Z

    Load interpreter/spark/dep only when SPARK_HOME is undefined

commit 3cd1d4b123fd90620c30bb363d7d73986c5e4cb0
Author: Lee moon soo <mo...@apache.org>
Date:   2015-08-21T18:38:16Z

    Add hadoop jar and spark jar from HADOOP_HOME, SPARK_HOME when they are defined

commit fee333f317092e1450faf56d70ca4fe62811b546
Author: Lee moon soo <mo...@apache.org>
Date:   2015-08-21T18:52:24Z

    Add scala-library in test scope

commit ea3312a36867438afbd986a686fa59f654fc8e28
Author: Lee moon soo <mo...@apache.org>
Date:   2015-08-21T23:26:30Z

    Add dependency for compile and test

commit a026d7898f71bd83c070628d71aaa615a6f3527c
Author: Lee moon soo <mo...@apache.org>
Date:   2015-08-21T23:26:50Z

    py4j version as a property

commit d579754ad0699c75829f7955652b8daa84b1fe44
Author: Lee moon soo <mo...@apache.org>
Date:   2015-08-21T23:27:11Z

    Handle ZEPPELIN_CLASSPATH proper way

commit 454efab853df1484a527cb8a298a7fc3eb476376
Author: Lee moon soo <mo...@apache.org>
Date:   2015-08-22T01:36:44Z

    Use reflection not to use import org.apache.spark.scheduler.Stage

commit 6ed12c3f9f33b6ea62786bf2c7f6ceac7783fd2c
Author: Lee moon soo <mo...@apache.org>
Date:   2015-08-22T02:20:30Z

    update travis config

commit 4d48283d10ae801dd1fcedd4fda51ef93a11923e
Author: Lee moon soo <mo...@apache.org>
Date:   2015-08-22T03:21:51Z

    Clear interpreter directory on mvn clean

commit 6b797ea50fd1fee93bb1ede4ea37ead2d14eb9c5
Author: Lee moon soo <mo...@apache.org>
Date:   2015-08-22T04:39:01Z

    set SPARK_HOME

commit b3e8085a6622e5f7e280aa69b082bc7dd45d7f15
Author: Lee moon soo <mo...@apache.org>
Date:   2015-08-22T14:19:31Z

    Update .travis

commit 857f7fb08486b38a4ce749b0198fde4d61f66331
Author: Lee moon soo <mo...@apache.org>
Date:   2015-08-22T14:35:48Z

    test more efficiently

commit 4ebc5ee62a83b5919321c10ea98f9ff666c79d1f
Author: Lee moon soo <mo...@apache.org>
Date:   2015-08-22T15:05:11Z

    bringing travis-install.sh back

commit 918a472d635773e4308037e118f8df2faa0838d9
Author: Lee moon soo <mo...@apache.org>
Date:   2015-08-22T15:44:41Z

    rm -rf only interpreter directory instead of mvn clean

commit 0137514a9d7575c80ccda7c0505f15845823f2bb
Author: Lee moon soo <mo...@apache.org>
Date:   2015-08-22T16:38:27Z

    trying to find why travis is not closing the test

commit 669f6cf5689a1287f2124cbcb67f368fdf19b267
Author: Lee moon soo <mo...@apache.org>
Date:   2015-08-22T17:09:34Z

    enable 1.3.x test

commit 3a08b82fa0109bc211daf6b499ea048c95a92e9c
Author: Lee moon soo <mo...@apache.org>
Date:   2015-08-22T17:38:59Z

    enable 1.2.x test

commit 9fbf799d20846f0655d77f629a793ceba9b9170a
Author: Lee moon soo <mo...@apache.org>
Date:   2015-08-22T17:59:28Z

    try sys.stdout.write instead of print

commit 14b46f851afbf3d6e94096cd7884080071a9da80
Author: Lee moon soo <mo...@apache.org>
Date:   2015-08-22T18:32:45Z

    not use travis-install for mvn test

commit 3e675a5682061fac39776bd1ca026a11b2612af7
Author: Lee moon soo <mo...@apache.org>
Date:   2015-08-22T18:33:48Z

    test -> package

commit 7b3b3c6eef6b3b24ec909723c7cd64fd771b09f9
Author: Lee moon soo <mo...@apache.org>
Date:   2015-08-22T19:13:37Z

    add pyspark flag

commit e4bd76f2264fc184d86b7be311898a9c6c418dc5
Author: Lee moon soo <mo...@apache.org>
Date:   2015-08-22T20:02:42Z

    Use reflection to call addListener

commit a77d481d90166d44d5b6494ffcc8fe6768774c63
Author: Lee moon soo <mo...@apache.org>
Date:   2015-08-22T20:49:04Z

    Print all logs

commit 6a085c2138eb703e1c065eeda7f6877c77a3f18a
Author: Lee moon soo <mo...@apache.org>
Date:   2015-08-22T21:29:44Z

    call listenerBus() using reflection

commit 7ec6a03873d32627980b3c4ab4bcf1edc8eddede
Author: Lee moon soo <mo...@apache.org>
Date:   2015-08-22T22:10:15Z

    get correct method

commit 47f96aca8b87c77e15cd4b0874c08623294d6ce8
Author: Lee moon soo <mo...@apache.org>
Date:   2015-08-22T23:13:17Z

    Call sqlContext.sql using reflection

commit b200e77e537d3980ee7c36b0128ad5a35ac0d411
Author: Lee moon soo <mo...@apache.org>
Date:   2015-08-22T23:13:36Z

    Test use explicitly %spark

commit 09d5a0ed1b67d88e92726774495f9c2de51bf665
Author: Lee moon soo <mo...@apache.org>
Date:   2015-08-23T00:00:15Z

    Update readme

commit 0c9c7434342fcb6e31680f0d765f5cce2307f731
Author: Lee moon soo <mo...@apache.org>
Date:   2015-08-22T04:21:45Z

    Initial implementation of spark conf validator

----


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-zeppelin pull request: ZEPPELIN-256 Self diagnosis spark...

Posted by Leemoonsoo <gi...@git.apache.org>.
Github user Leemoonsoo commented on the pull request:

    https://github.com/apache/incubator-zeppelin/pull/246#issuecomment-133766873
  
    This branch rebased #244 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-zeppelin pull request: ZEPPELIN-256 Self diagnosis spark...

Posted by felixcheung <gi...@git.apache.org>.
Github user felixcheung commented on the pull request:

    https://github.com/apache/incubator-zeppelin/pull/246#issuecomment-194215239
  
    Let's bring to up-to-date?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-zeppelin pull request: ZEPPELIN-256 Self diagnosis spark...

Posted by khalidhuseynov <gi...@git.apache.org>.
Github user khalidhuseynov commented on the pull request:

    https://github.com/apache/incubator-zeppelin/pull/246#issuecomment-133871289
  
    That would be very helpful during Zeppelin set up! LGTM


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-zeppelin pull request: ZEPPELIN-256 Self diagnosis spark...

Posted by Leemoonsoo <gi...@git.apache.org>.
Github user Leemoonsoo commented on the pull request:

    https://github.com/apache/incubator-zeppelin/pull/246#issuecomment-133773967
  
    Ready to be reviewed and merge.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---