You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@bigtop.apache.org by kwmonroe <gi...@git.apache.org> on 2016/12/02 22:38:36 UTC
[GitHub] bigtop pull request #166: BIGTOP-2561: add juju bundle for hadoop-spark
GitHub user kwmonroe opened a pull request:
https://github.com/apache/bigtop/pull/166
BIGTOP-2561: add juju bundle for hadoop-spark
This combines hadoop-processing and spark-processing into a single bundle:
https://jujucharms.com/hadoop-spark/
Test results are available here across multiple clouds:
http://data.vapour.ws/cwr-tests/results/bundle_hadoop_spark/index.html
We are working through infrastructure issues atm (hence a lot of yellow triangles), but we haven't seen any failures with the charms included in this bundle for the last couple weeks. Hence I'm putting it up for review / inclusion upstream.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/juju-solutions/bigtop feature/hadoop-spark-bundle
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/bigtop/pull/166.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #166
----
commit 35b8d13c198a98296f0290e340e7bc93a7e1b62d
Author: Kevin W Monroe <ke...@canonical.com>
Date: 2016-09-22T01:46:24Z
initial hadoop-spark bundle
commit d1387eda17b3d8cb326e5d0d12500a60ac66d04a
Author: Kevin W Monroe <ke...@canonical.com>
Date: 2016-09-22T23:20:51Z
colocate bundle applications
To save resources, colocate some of the applications when it makes sense.
Put the namenode and resourcemgr on the same unit; spark and zeppelin;
ganglia and rsyslog.
Update readme formatting and callout these colocated apps. Also tweak
the restricted network section and add a section for connecting to zepp.
Note, machine spec in the bundle yaml start with machine '1' instead of
'0'. This is because someone may deploy this on juju-1.x, where machine
'0' is the bootstrap node.
commit b52a7c5a7020ffaec25ca3f240768eaf3b4c1cac
Author: Konstantinos Tsakalozos <ko...@canonical.com>
Date: 2016-10-24T09:37:24Z
Be explicit on series for CWR to deploy the right series
commit 040bab24a58d0b4eda05a8ab215342a1f21ff9a7
Author: Kevin W Monroe <ke...@canonical.com>
Date: 2016-10-29T02:19:05Z
refresh charms
commit a29bac836b7d39adabe5d3c835f3d00e28b75f8a
Author: Kevin W Monroe <ke...@canonical.com>
Date: 2016-10-29T05:03:24Z
refresh hadoop-spark bundle
- update readme to be consistent with other bigtop-based bundles
- bump all charms to latest revisions
- update amulet test to call charm smoke tests and work around machine spec
commit ef3d70542babecd69baa0618e39388854968fda8
Author: Kevin W Monroe <ke...@canonical.com>
Date: 2016-10-31T16:45:08Z
skip slave smoke tests and update smoke failure message
commit 506ba320764323f4ffb294bce4834f044626e799
Author: Kevin W Monroe <ke...@canonical.com>
Date: 2016-10-31T22:29:15Z
bump spark revno for spark-bench fixes
commit 4c0f812d736b8d4c31bfce68b450d8b77adf79a0
Author: Kevin W Monroe <ke...@canonical.com>
Date: 2016-11-02T04:25:58Z
refresh spark charm and readme for new sparkbench suite
commit ff19096dbb9bf3fb21297fcc0a258d98453ea7cc
Author: Kevin W Monroe <ke...@canonical.com>
Date: 2016-11-03T04:33:14Z
charm revision bump
- new ganglia charms report app version and fix '-broken' hook failures
- new spark charm correctly runs sparkbench regardless of spark exe mode
commit 5a961a6fb15f403f80356de62b3acdaa7aa2b10c
Author: Kevin W Monroe <ke...@canonical.com>
Date: 2016-11-05T22:39:41Z
bump spark rev for updated test case and sparkbench composite is now seconds
commit 71c5731208c637bc2cb0ed2a44179c25479dc5e2
Author: Kevin W Monroe <ke...@canonical.com>
Date: 2016-11-07T17:53:37Z
bump zk revno to pick up test rename and app version
commit 745deec545226a5c9d83d9d749101bf227b4dec6
Author: Kevin W Monroe <ke...@canonical.com>
Date: 2016-11-15T21:12:22Z
bump rsyslog for lint and zk for an amulet test enhancement (ease local testing)
commit 638221be832236641eef3ceef0a9b46e11121077
Author: Kevin W Monroe <ke...@canonical.com>
Date: 2016-11-22T21:34:35Z
add ppa, charm rev bump, new constraints
- ensure tests use juju/stable ppa
- bump zk to latest (consistent bigtop_ prefix, readme tweaks)
- bump core hadoop charms to latest (new swap handler, detect failed service
start, add spark user/group to nn/rm)
- aws defaults to 8gb root filesystem; bump this to 20gb
commit 34ffbe4bc576ba3a330e8c1da6dd08a74f3a392a
Author: Kevin W Monroe <ke...@canonical.com>
Date: 2016-11-24T19:49:56Z
rev charms to pick up new swap handler
commit 3fc9934e7723e1ab8383d1a96c093f86d546178b
Author: Kevin W Monroe <ke...@canonical.com>
Date: 2016-11-28T23:16:11Z
workaround azure root-disk: https://bugs.launchpad.net/juju/+bug/1645408
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] bigtop pull request #166: BIGTOP-2561: add juju bundle for hadoop-spark
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/bigtop/pull/166
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] bigtop issue #166: BIGTOP-2561: add juju bundle for hadoop-spark
Posted by petevg <gi...@git.apache.org>.
Github user petevg commented on the issue:
https://github.com/apache/bigtop/pull/166
LGTM/+1 (I was just able to deploy this bundle to aws without any hitches, too)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---