You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@zeppelin.apache.org by Leemoonsoo <gi...@git.apache.org> on 2015/08/23 02:16:01 UTC
[GitHub] incubator-zeppelin pull request: ZEPPELIN-256 Self diagnosis spark...
GitHub user Leemoonsoo opened a pull request:
https://github.com/apache/incubator-zeppelin/pull/246
ZEPPELIN-256 Self diagnosis spark configuration
https://issues.apache.org/jira/browse/ZEPPELIN-256
This patch adds capability of self diagnosis of configuration for spark interpreter.
User can set "zeppelin.spark.diagnosis" property "true" to enable this feature, in Interpreter page.
When it is turned on, SparkConfValidator checks validity of combination of environment variables and properties. It checks actual existence of specified directories (eg. SPARK_HOME, etc) and files (eg. pyspark.zip), according to given configuration.
When it detects some problem, it prints error as a result message in a notebook screen. So user can see what was missed.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/Leemoonsoo/incubator-zeppelin spark_conf_validate
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-zeppelin/pull/246.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #246
----
commit c92f18534d85e318e41fe53c0cf684b3805be719
Author: Lee moon soo <mo...@apache.org>
Date: 2015-08-21T17:56:43Z
Separate spark-dependency into submodule
commit c49c154bc1ca6ebd783022b774e4ff92c37c8d4c
Author: Lee moon soo <mo...@apache.org>
Date: 2015-08-21T17:59:47Z
Load interpreter/spark/dep only when SPARK_HOME is undefined
commit 3cd1d4b123fd90620c30bb363d7d73986c5e4cb0
Author: Lee moon soo <mo...@apache.org>
Date: 2015-08-21T18:38:16Z
Add hadoop jar and spark jar from HADOOP_HOME, SPARK_HOME when they are defined
commit fee333f317092e1450faf56d70ca4fe62811b546
Author: Lee moon soo <mo...@apache.org>
Date: 2015-08-21T18:52:24Z
Add scala-library in test scope
commit ea3312a36867438afbd986a686fa59f654fc8e28
Author: Lee moon soo <mo...@apache.org>
Date: 2015-08-21T23:26:30Z
Add dependency for compile and test
commit a026d7898f71bd83c070628d71aaa615a6f3527c
Author: Lee moon soo <mo...@apache.org>
Date: 2015-08-21T23:26:50Z
py4j version as a property
commit d579754ad0699c75829f7955652b8daa84b1fe44
Author: Lee moon soo <mo...@apache.org>
Date: 2015-08-21T23:27:11Z
Handle ZEPPELIN_CLASSPATH proper way
commit 454efab853df1484a527cb8a298a7fc3eb476376
Author: Lee moon soo <mo...@apache.org>
Date: 2015-08-22T01:36:44Z
Use reflection not to use import org.apache.spark.scheduler.Stage
commit 6ed12c3f9f33b6ea62786bf2c7f6ceac7783fd2c
Author: Lee moon soo <mo...@apache.org>
Date: 2015-08-22T02:20:30Z
update travis config
commit 4d48283d10ae801dd1fcedd4fda51ef93a11923e
Author: Lee moon soo <mo...@apache.org>
Date: 2015-08-22T03:21:51Z
Clear interpreter directory on mvn clean
commit 6b797ea50fd1fee93bb1ede4ea37ead2d14eb9c5
Author: Lee moon soo <mo...@apache.org>
Date: 2015-08-22T04:39:01Z
set SPARK_HOME
commit b3e8085a6622e5f7e280aa69b082bc7dd45d7f15
Author: Lee moon soo <mo...@apache.org>
Date: 2015-08-22T14:19:31Z
Update .travis
commit 857f7fb08486b38a4ce749b0198fde4d61f66331
Author: Lee moon soo <mo...@apache.org>
Date: 2015-08-22T14:35:48Z
test more efficiently
commit 4ebc5ee62a83b5919321c10ea98f9ff666c79d1f
Author: Lee moon soo <mo...@apache.org>
Date: 2015-08-22T15:05:11Z
bringing travis-install.sh back
commit 918a472d635773e4308037e118f8df2faa0838d9
Author: Lee moon soo <mo...@apache.org>
Date: 2015-08-22T15:44:41Z
rm -rf only interpreter directory instead of mvn clean
commit 0137514a9d7575c80ccda7c0505f15845823f2bb
Author: Lee moon soo <mo...@apache.org>
Date: 2015-08-22T16:38:27Z
trying to find why travis is not closing the test
commit 669f6cf5689a1287f2124cbcb67f368fdf19b267
Author: Lee moon soo <mo...@apache.org>
Date: 2015-08-22T17:09:34Z
enable 1.3.x test
commit 3a08b82fa0109bc211daf6b499ea048c95a92e9c
Author: Lee moon soo <mo...@apache.org>
Date: 2015-08-22T17:38:59Z
enable 1.2.x test
commit 9fbf799d20846f0655d77f629a793ceba9b9170a
Author: Lee moon soo <mo...@apache.org>
Date: 2015-08-22T17:59:28Z
try sys.stdout.write instead of print
commit 14b46f851afbf3d6e94096cd7884080071a9da80
Author: Lee moon soo <mo...@apache.org>
Date: 2015-08-22T18:32:45Z
not use travis-install for mvn test
commit 3e675a5682061fac39776bd1ca026a11b2612af7
Author: Lee moon soo <mo...@apache.org>
Date: 2015-08-22T18:33:48Z
test -> package
commit 7b3b3c6eef6b3b24ec909723c7cd64fd771b09f9
Author: Lee moon soo <mo...@apache.org>
Date: 2015-08-22T19:13:37Z
add pyspark flag
commit e4bd76f2264fc184d86b7be311898a9c6c418dc5
Author: Lee moon soo <mo...@apache.org>
Date: 2015-08-22T20:02:42Z
Use reflection to call addListener
commit a77d481d90166d44d5b6494ffcc8fe6768774c63
Author: Lee moon soo <mo...@apache.org>
Date: 2015-08-22T20:49:04Z
Print all logs
commit 6a085c2138eb703e1c065eeda7f6877c77a3f18a
Author: Lee moon soo <mo...@apache.org>
Date: 2015-08-22T21:29:44Z
call listenerBus() using reflection
commit 7ec6a03873d32627980b3c4ab4bcf1edc8eddede
Author: Lee moon soo <mo...@apache.org>
Date: 2015-08-22T22:10:15Z
get correct method
commit 47f96aca8b87c77e15cd4b0874c08623294d6ce8
Author: Lee moon soo <mo...@apache.org>
Date: 2015-08-22T23:13:17Z
Call sqlContext.sql using reflection
commit b200e77e537d3980ee7c36b0128ad5a35ac0d411
Author: Lee moon soo <mo...@apache.org>
Date: 2015-08-22T23:13:36Z
Test use explicitly %spark
commit 09d5a0ed1b67d88e92726774495f9c2de51bf665
Author: Lee moon soo <mo...@apache.org>
Date: 2015-08-23T00:00:15Z
Update readme
commit 0c9c7434342fcb6e31680f0d765f5cce2307f731
Author: Lee moon soo <mo...@apache.org>
Date: 2015-08-22T04:21:45Z
Initial implementation of spark conf validator
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-zeppelin pull request: ZEPPELIN-256 Self diagnosis spark...
Posted by Leemoonsoo <gi...@git.apache.org>.
Github user Leemoonsoo commented on the pull request:
https://github.com/apache/incubator-zeppelin/pull/246#issuecomment-133766873
This branch rebased #244
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-zeppelin pull request: ZEPPELIN-256 Self diagnosis spark...
Posted by felixcheung <gi...@git.apache.org>.
Github user felixcheung commented on the pull request:
https://github.com/apache/incubator-zeppelin/pull/246#issuecomment-194215239
Let's bring to up-to-date?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-zeppelin pull request: ZEPPELIN-256 Self diagnosis spark...
Posted by khalidhuseynov <gi...@git.apache.org>.
Github user khalidhuseynov commented on the pull request:
https://github.com/apache/incubator-zeppelin/pull/246#issuecomment-133871289
That would be very helpful during Zeppelin set up! LGTM
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-zeppelin pull request: ZEPPELIN-256 Self diagnosis spark...
Posted by Leemoonsoo <gi...@git.apache.org>.
Github user Leemoonsoo commented on the pull request:
https://github.com/apache/incubator-zeppelin/pull/246#issuecomment-133773967
Ready to be reviewed and merge.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---