You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@bigtop.apache.org by ktsakalozos <gi...@git.apache.org> on 2016/08/10 12:39:51 UTC
[GitHub] bigtop pull request #137: BIGTOP-2516: Add Zeppelin Charm
GitHub user ktsakalozos opened a pull request:
https://github.com/apache/bigtop/pull/137
BIGTOP-2516: Add Zeppelin Charm
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/juju-solutions/bigtop BIGTOP-2516-Zeppelin
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/bigtop/pull/137.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #137
----
commit 7650b439ec7b24d2948e668c4c08cf9aff28c9e1
Author: Kevin W Monroe <ke...@canonical.com>
Date: 2016-06-07T02:21:51Z
BIGTOP-2516: Add Zeppelin Charm
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] bigtop pull request #137: BIGTOP-2516: Add Zeppelin Charm
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/bigtop/pull/137
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] bigtop pull request #137: BIGTOP-2516: Add Zeppelin Charm
Posted by kwmonroe <gi...@git.apache.org>.
Github user kwmonroe commented on a diff in the pull request:
https://github.com/apache/bigtop/pull/137#discussion_r81859067
--- Diff: bigtop-packages/src/charm/zeppelin/layer-zeppelin/README.md ---
@@ -0,0 +1,119 @@
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+## Overview
+
+Apache Zeppelin is a web-based notebook that enables interactive data analytics.
+You can make beautiful data-driven, interactive, and collaborative documents
+with SQL, Scala and more.
+
+As a Multi-purpose Notebook, Apache Zeppelin is the place for interactive:
+
+ * Data Ingestion
+ * Data Discovery
+ * Data Analytics
+ * Data Visualization & Collaboration
+
+
+## Usage
+
+This charm is intended to be deployed alongside one of the
+[big data bundles](https://jujucharms.com/u/bigdata-charmers/#bundles).
+For example:
+
+ juju deploy hadoop-processing
+
+_**Note**: The above assumes Juju 2.0 or greater. If using an earlier version
+of Juju, the syntax is `juju quickstart cs:bundle/hadoop-processing`._
+
+Now add Zeppelin and expose the web interface:
+
+ juju deploy zeppelin
+ juju expose zeppelin
+
+Now relate Zeppelin to Java and the Hadoop cluster:
+
+ juju add-relation zeppelin openjdk
--- End diff --
Openjdk is optional now and won't be present in the Bigtop bundles... which will make this `add-relation` step fail. Yank it (and the Java comment above).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] bigtop pull request #137: BIGTOP-2516: Add Zeppelin Charm
Posted by kwmonroe <gi...@git.apache.org>.
Github user kwmonroe commented on a diff in the pull request:
https://github.com/apache/bigtop/pull/137#discussion_r81864360
--- Diff: bigtop-packages/src/charm/zeppelin/layer-zeppelin/tests/01-zeppelin-smoke.py ---
@@ -0,0 +1,66 @@
+#!/usr/bin/env python3
+
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import unittest
+import amulet
+
+
+class TestDeploy(unittest.TestCase):
+ """
+ Deployment and smoke test for the Apache Bigtop Zeppelin service.
+ """
+ @classmethod
+ def setUpClass(cls):
--- End diff --
We can simplify this class by removing all references to 'openjdk' and 'java' below. Do it!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] bigtop pull request #137: BIGTOP-2516: Add Zeppelin Charm
Posted by johnsca <gi...@git.apache.org>.
Github user johnsca commented on a diff in the pull request:
https://github.com/apache/bigtop/pull/137#discussion_r82397771
--- Diff: bigtop-packages/src/charm/zeppelin/layer-zeppelin/reactive/zeppelin.py ---
@@ -0,0 +1,158 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import hashlib
+
+from charms.reactive import when, when_not
+from charms.reactive import is_state, set_state, remove_state
+from charmhelpers.core import hookenv
+from charms.layer.bigtop_zeppelin import Zeppelin
+from charms.reactive.helpers import data_changed
+
+
+@when('zeppelin.installed')
+def update_status():
+ hadoop_joined = is_state('hadoop.joined')
+ hadoop_ready = is_state('hadoop.ready')
+ hive_joined = is_state('hive.connected')
+ hive_ready = is_state('hive.available')
+ spark_joined = is_state('spark.joined')
+ spark_ready = is_state('spark.ready')
+
+ waiting_apps = []
+ ready_apps = []
+ # Check status of the hadoop plugin
+ if hadoop_joined and not hadoop_ready:
+ waiting_apps.append('hadoop')
+ elif hadoop_ready:
+ ready_apps.append('hadoop')
+
+ # Check status of Hive
+ if hive_joined and not hive_ready:
+ waiting_apps.append('hive')
+ elif hive_ready:
+ ready_apps.append('hive')
+
+ # Check status of Spark
+ if spark_joined and not spark_ready:
+ waiting_apps.append('spark')
+ elif spark_ready:
+ ready_apps.append('spark')
+
+ # Set appropriate status based on the apps we checked above
+ if waiting_apps:
+ hookenv.status_set('waiting',
+ 'waiting for: {}'.format(' & '.join(waiting_apps)))
+ elif ready_apps:
+ hookenv.status_set('active',
+ 'ready with: {}'.format(' & '.join(ready_apps)))
+ else:
+ hookenv.status_set('active', 'ready')
+
+
+@when('bigtop.available')
+@when_not('zeppelin.installed')
+def initial_setup():
+ hookenv.status_set('maintenance', 'installing zeppelin')
+ zeppelin = Zeppelin()
+ zeppelin.install()
+ zeppelin.setup_etc_env()
+ zeppelin.open_ports()
+ set_state('zeppelin.installed')
+ update_status()
+
+
+@when('zeppelin.installed')
+@when('hadoop.ready')
+@when_not('zeppelin.hadoop.configured')
+def configure_hadoop(hadoop):
+ zeppelin = Zeppelin()
+ zeppelin.configure_hadoop()
+ zeppelin.register_hadoop_notebooks()
+ set_state('zeppelin.hadoop.configured')
+
+
+@when('zeppelin.installed')
+@when_not('hadoop.ready')
+@when('zeppelin.hadoop.configured')
+def unconfigure_hadoop():
+ zeppelin = Zeppelin()
+ zeppelin.remove_hadoop_notebooks()
+ remove_state('zeppelin.hadoop.configured')
+
+
+@when('zeppelin.installed', 'hive.ready')
+def configure_hive(hive):
+ hive_ip = hive.get_private_ip()
+ hive_port = hive.get_port()
+ hive_url = 'jdbc:hive2://%s:%s' % (hive_ip, hive_port)
+ if data_changed('hive.connect', hive_url):
+ hookenv.status_set('maintenance', 'configuring hive')
+ zeppelin = Zeppelin()
+ zeppelin.configure_hive(hive_url)
+ set_state('zeppelin.hive.configured')
+ update_status()
+
+
+@when('zeppelin.installed', 'zeppelin.hive.configured')
+@when_not('hive.ready')
+def unconfigure_hive():
+ hookenv.status_set('maintenance', 'removing hive relation')
+ zeppelin = Zeppelin()
+ zeppelin.configure_hive('jdbc:hive2://:')
+ remove_state('zeppelin.hive.configured')
+ update_status()
+
+
+@when('zeppelin.installed', 'spark.ready')
+def configure_spark(spark):
+ master_url = spark.get_master_url()
+ if data_changed('spark.master', master_url):
+ hookenv.status_set('maintenance', 'configuring spark')
+ zeppelin = Zeppelin()
+ zeppelin.configure_spark(master_url)
+ set_state('zeppelin.spark.configured')
+ update_status()
+
+
+@when('zeppelin.installed', 'zeppelin.spark.configured')
+@when_not('spark.ready')
+def unconfigure_spark():
+ hookenv.status_set('maintenance', 'removing spark relation')
+ zeppelin = Zeppelin()
+ zeppelin.configure_spark(None)
--- End diff --
If Spark isn't related to Zeppelin, there's a fairly good chance that it's not related to Hadoop, either, and thus it would fail in `yarn-client` mode.
Perhaps a better solution would be to remove the override entirely and just let Bigtop fall back to its default naturally.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] bigtop pull request #137: BIGTOP-2516: Add Zeppelin Charm
Posted by kwmonroe <gi...@git.apache.org>.
Github user kwmonroe commented on a diff in the pull request:
https://github.com/apache/bigtop/pull/137#discussion_r81858734
--- Diff: bigtop-packages/src/charm/zeppelin/layer-zeppelin/README.md ---
@@ -0,0 +1,119 @@
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+## Overview
+
+Apache Zeppelin is a web-based notebook that enables interactive data analytics.
+You can make beautiful data-driven, interactive, and collaborative documents
+with SQL, Scala and more.
+
+As a Multi-purpose Notebook, Apache Zeppelin is the place for interactive:
+
+ * Data Ingestion
+ * Data Discovery
+ * Data Analytics
+ * Data Visualization & Collaboration
+
+
+## Usage
+
+This charm is intended to be deployed alongside one of the
+[big data bundles](https://jujucharms.com/u/bigdata-charmers/#bundles).
+For example:
+
+ juju deploy hadoop-processing
+
+_**Note**: The above assumes Juju 2.0 or greater. If using an earlier version
+of Juju, the syntax is `juju quickstart cs:bundle/hadoop-processing`._
+
+Now add Zeppelin and expose the web interface:
+
+ juju deploy zeppelin
+ juju expose zeppelin
+
+Now relate Zeppelin to Java and the Hadoop cluster:
+
+ juju add-relation zeppelin openjdk
+ juju add-relation zeppelin plugin
+
+Once deployment is complete, you will have an Apache Bigtop Hadoop platform
+with Zeppelin available to create notebooks and start analyzing your data!
+You can access the web interface at http://{zeppelin_unit_ip_address}:9080.
+The ip address can be found by running`juju status zeppelin | grep public-address`.
+
+
+## Status and Smoke Test
+
+The services provide extended status reporting to indicate when they are ready:
+
+ juju status
+
+This is particularly useful when combined with `watch` to track the on-going
+progress of the deployment:
+
+ watch -n 0.5 juju status
+
+The message for each unit will provide information about that unit's state.
+Once they all indicate that they are ready, you can perform a "smoke test"
+to verify that Zeppelin is working as expected using the built-in `smoke-test`
+action:
+
+ juju run-action zeppelin/0 smoke-test
+
+_**Note**: The above assumes Juju 2.0 or greater. If using an earlier version
+of Juju, the syntax is `juju action do zeppelin/0 smoke-test`._
+
+After a minute or so, you can check the results of the smoke test:
+
+ juju show-action-status
+
+_**Note**: The above assumes Juju 2.0 or greater. If using an earlier version
+of Juju, the syntax is `juju action status`._
+
+You will see `status: completed` if the smoke test was successful, or
+`status: failed` if it was not. You can get more information on why it failed
+via:
+
+ juju show-action-output <action-id>
+
+_**Note**: The above assumes Juju 2.0 or greater. If using an earlier version
+of Juju, the syntax is `juju action fetch <action-id>`._
+
+
+## Limitations
--- End diff --
Let's remove this section until https://github.com/juju-solutions/bigtop/issues/40 is fixed. Doesn't make any sense here since you can't change spark config on the bigtop-spark charm anyway.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] bigtop pull request #137: BIGTOP-2516: Add Zeppelin Charm
Posted by kwmonroe <gi...@git.apache.org>.
Github user kwmonroe commented on a diff in the pull request:
https://github.com/apache/bigtop/pull/137#discussion_r81862021
--- Diff: bigtop-packages/src/charm/zeppelin/layer-zeppelin/lib/charms/layer/bigtop_zeppelin.py ---
@@ -0,0 +1,238 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import os
+import json
+import time
+import socket
+from urllib.parse import urljoin
+
+import requests
+from path import Path
+
+from jujubigdata import utils
+from charmhelpers.core import hookenv, host, unitdata
+from charms import layer
+from charms.layer.apache_bigtop_base import Bigtop
+
+
+class Zeppelin(object):
+ """
+ This class manages Zeppelin.
+ """
+ def __init__(self):
+ self.dist_config = utils.DistConfig(
+ data=layer.options('apache-bigtop-base'))
+
+ def _add_override(self, name, value):
+ unitdata.kv().update({
+ name: value,
+ }, prefix='zeppelin.bigtop.overrides.')
+
+ def install(self):
+ '''
+ Trigger the Bigtop puppet recipe that handles the Zepplin service.
+ '''
+ # Dirs are handled by the bigtop deb. No need to call out to
+ # dist_config to do that work.
+ self.dist_config.add_users()
+ self.trigger_bigtop()
--- End diff --
If some poor soul changes our layer.yaml ports, they're in trouble. Let's honor them by actually setting the chosen port in an override before calling `self.trigger_bigtop()` here. Something like:
```python
self._add_override('zeppelin::server::server_port', self.dist_config.port('zeppelin'))
self._add_override('zeppelin::server::web_socket_port', self.dist_config.port('zeppelin_web'))
```
Those should be the correct overrides, as defined in zeppelin's init.pp here:
https://github.com/apache/bigtop/blob/master/bigtop-deploy/puppet/modules/zeppelin/manifests/init.pp#L26
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] bigtop pull request #137: BIGTOP-2516: Add Zeppelin Charm
Posted by johnsca <gi...@git.apache.org>.
Github user johnsca commented on a diff in the pull request:
https://github.com/apache/bigtop/pull/137#discussion_r82406325
--- Diff: bigtop-packages/src/charm/zeppelin/layer-zeppelin/reactive/zeppelin.py ---
@@ -0,0 +1,158 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import hashlib
+
+from charms.reactive import when, when_not
+from charms.reactive import is_state, set_state, remove_state
+from charmhelpers.core import hookenv
+from charms.layer.bigtop_zeppelin import Zeppelin
+from charms.reactive.helpers import data_changed
+
+
+@when('zeppelin.installed')
+def update_status():
+ hadoop_joined = is_state('hadoop.joined')
+ hadoop_ready = is_state('hadoop.ready')
+ hive_joined = is_state('hive.connected')
+ hive_ready = is_state('hive.available')
+ spark_joined = is_state('spark.joined')
+ spark_ready = is_state('spark.ready')
+
+ waiting_apps = []
+ ready_apps = []
+ # Check status of the hadoop plugin
+ if hadoop_joined and not hadoop_ready:
+ waiting_apps.append('hadoop')
+ elif hadoop_ready:
+ ready_apps.append('hadoop')
+
+ # Check status of Hive
+ if hive_joined and not hive_ready:
+ waiting_apps.append('hive')
+ elif hive_ready:
+ ready_apps.append('hive')
+
+ # Check status of Spark
+ if spark_joined and not spark_ready:
+ waiting_apps.append('spark')
+ elif spark_ready:
+ ready_apps.append('spark')
+
+ # Set appropriate status based on the apps we checked above
+ if waiting_apps:
+ hookenv.status_set('waiting',
+ 'waiting for: {}'.format(' & '.join(waiting_apps)))
+ elif ready_apps:
+ hookenv.status_set('active',
+ 'ready with: {}'.format(' & '.join(ready_apps)))
+ else:
+ hookenv.status_set('active', 'ready')
+
+
+@when('bigtop.available')
+@when_not('zeppelin.installed')
+def initial_setup():
+ hookenv.status_set('maintenance', 'installing zeppelin')
+ zeppelin = Zeppelin()
+ zeppelin.install()
+ zeppelin.setup_etc_env()
+ zeppelin.open_ports()
+ set_state('zeppelin.installed')
+ update_status()
+
+
+@when('zeppelin.installed')
+@when('hadoop.ready')
+@when_not('zeppelin.hadoop.configured')
+def configure_hadoop(hadoop):
+ zeppelin = Zeppelin()
+ zeppelin.configure_hadoop()
+ zeppelin.register_hadoop_notebooks()
+ set_state('zeppelin.hadoop.configured')
+
+
+@when('zeppelin.installed')
+@when_not('hadoop.ready')
+@when('zeppelin.hadoop.configured')
+def unconfigure_hadoop():
+ zeppelin = Zeppelin()
+ zeppelin.remove_hadoop_notebooks()
+ remove_state('zeppelin.hadoop.configured')
+
+
+@when('zeppelin.installed', 'hive.ready')
+def configure_hive(hive):
+ hive_ip = hive.get_private_ip()
+ hive_port = hive.get_port()
+ hive_url = 'jdbc:hive2://%s:%s' % (hive_ip, hive_port)
+ if data_changed('hive.connect', hive_url):
+ hookenv.status_set('maintenance', 'configuring hive')
+ zeppelin = Zeppelin()
+ zeppelin.configure_hive(hive_url)
+ set_state('zeppelin.hive.configured')
+ update_status()
+
+
+@when('zeppelin.installed', 'zeppelin.hive.configured')
+@when_not('hive.ready')
+def unconfigure_hive():
+ hookenv.status_set('maintenance', 'removing hive relation')
+ zeppelin = Zeppelin()
+ zeppelin.configure_hive('jdbc:hive2://:')
+ remove_state('zeppelin.hive.configured')
+ update_status()
+
+
+@when('zeppelin.installed', 'spark.ready')
+def configure_spark(spark):
+ master_url = spark.get_master_url()
+ if data_changed('spark.master', master_url):
+ hookenv.status_set('maintenance', 'configuring spark')
+ zeppelin = Zeppelin()
+ zeppelin.configure_spark(master_url)
+ set_state('zeppelin.spark.configured')
+ update_status()
+
+
+@when('zeppelin.installed', 'zeppelin.spark.configured')
+@when_not('spark.ready')
+def unconfigure_spark():
+ hookenv.status_set('maintenance', 'removing spark relation')
+ zeppelin = Zeppelin()
+ zeppelin.configure_spark(None)
--- End diff --
Since the overrides are additive, removing one just leaves it as its current value, so we'll need to explicitly change it back to `yarn-client`.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] bigtop pull request #137: BIGTOP-2516: Add Zeppelin Charm
Posted by c0s <gi...@git.apache.org>.
Github user c0s commented on a diff in the pull request:
https://github.com/apache/bigtop/pull/137#discussion_r74286278
--- Diff: bigtop-packages/src/charm/zeppelin/layer-zeppelin/LICENSE ---
@@ -0,0 +1,177 @@
+
--- End diff --
You don't need to include LICENSE file to each charm - it is already done at the top of the project.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] bigtop pull request #137: BIGTOP-2516: Add Zeppelin Charm
Posted by kwmonroe <gi...@git.apache.org>.
Github user kwmonroe commented on a diff in the pull request:
https://github.com/apache/bigtop/pull/137#discussion_r81864099
--- Diff: bigtop-packages/src/charm/zeppelin/layer-zeppelin/reactive/zeppelin.py ---
@@ -0,0 +1,158 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import hashlib
+
+from charms.reactive import when, when_not
+from charms.reactive import is_state, set_state, remove_state
+from charmhelpers.core import hookenv
+from charms.layer.bigtop_zeppelin import Zeppelin
+from charms.reactive.helpers import data_changed
+
+
+@when('zeppelin.installed')
+def update_status():
+ hadoop_joined = is_state('hadoop.joined')
+ hadoop_ready = is_state('hadoop.ready')
+ hive_joined = is_state('hive.connected')
+ hive_ready = is_state('hive.available')
+ spark_joined = is_state('spark.joined')
+ spark_ready = is_state('spark.ready')
+
+ waiting_apps = []
+ ready_apps = []
+ # Check status of the hadoop plugin
+ if hadoop_joined and not hadoop_ready:
+ waiting_apps.append('hadoop')
+ elif hadoop_ready:
+ ready_apps.append('hadoop')
+
+ # Check status of Hive
+ if hive_joined and not hive_ready:
+ waiting_apps.append('hive')
+ elif hive_ready:
+ ready_apps.append('hive')
+
+ # Check status of Spark
+ if spark_joined and not spark_ready:
+ waiting_apps.append('spark')
+ elif spark_ready:
+ ready_apps.append('spark')
+
+ # Set appropriate status based on the apps we checked above
+ if waiting_apps:
+ hookenv.status_set('waiting',
+ 'waiting for: {}'.format(' & '.join(waiting_apps)))
+ elif ready_apps:
+ hookenv.status_set('active',
+ 'ready with: {}'.format(' & '.join(ready_apps)))
+ else:
+ hookenv.status_set('active', 'ready')
+
+
+@when('bigtop.available')
+@when_not('zeppelin.installed')
+def initial_setup():
+ hookenv.status_set('maintenance', 'installing zeppelin')
+ zeppelin = Zeppelin()
+ zeppelin.install()
+ zeppelin.setup_etc_env()
+ zeppelin.open_ports()
+ set_state('zeppelin.installed')
+ update_status()
+
+
+@when('zeppelin.installed')
+@when('hadoop.ready')
+@when_not('zeppelin.hadoop.configured')
+def configure_hadoop(hadoop):
+ zeppelin = Zeppelin()
+ zeppelin.configure_hadoop()
+ zeppelin.register_hadoop_notebooks()
+ set_state('zeppelin.hadoop.configured')
+
+
+@when('zeppelin.installed')
+@when_not('hadoop.ready')
+@when('zeppelin.hadoop.configured')
+def unconfigure_hadoop():
+ zeppelin = Zeppelin()
+ zeppelin.remove_hadoop_notebooks()
+ remove_state('zeppelin.hadoop.configured')
+
+
+@when('zeppelin.installed', 'hive.ready')
+def configure_hive(hive):
+ hive_ip = hive.get_private_ip()
+ hive_port = hive.get_port()
+ hive_url = 'jdbc:hive2://%s:%s' % (hive_ip, hive_port)
+ if data_changed('hive.connect', hive_url):
+ hookenv.status_set('maintenance', 'configuring hive')
+ zeppelin = Zeppelin()
+ zeppelin.configure_hive(hive_url)
+ set_state('zeppelin.hive.configured')
+ update_status()
+
+
+@when('zeppelin.installed', 'zeppelin.hive.configured')
+@when_not('hive.ready')
+def unconfigure_hive():
+ hookenv.status_set('maintenance', 'removing hive relation')
+ zeppelin = Zeppelin()
+ zeppelin.configure_hive('jdbc:hive2://:')
+ remove_state('zeppelin.hive.configured')
+ update_status()
+
+
+@when('zeppelin.installed', 'spark.ready')
+def configure_spark(spark):
+ master_url = spark.get_master_url()
+ if data_changed('spark.master', master_url):
+ hookenv.status_set('maintenance', 'configuring spark')
+ zeppelin = Zeppelin()
+ zeppelin.configure_spark(master_url)
+ set_state('zeppelin.spark.configured')
+ update_status()
+
+
+@when('zeppelin.installed', 'zeppelin.spark.configured')
+@when_not('spark.ready')
+def unconfigure_spark():
+ hookenv.status_set('maintenance', 'removing spark relation')
+ zeppelin = Zeppelin()
+ zeppelin.configure_spark(None)
--- End diff --
I think we should check for `hadoop.ready` here, and if that's true, use `yarn-client` as the spark master url instead of `None`. I'm pretty sure zeppelin can run it's internal spark interpreter in yarn-client mode without any external spark master.
FWIW, bigtop defines `spark_master_url` as `yarn-client` by default:
https://github.com/apache/bigtop/blob/master/bigtop-deploy/puppet/modules/zeppelin/manifests/init.pp#L25
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---