You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@bigtop.apache.org by "ASF GitHub Bot (JIRA)" <ji...@apache.org> on 2016/12/02 22:38:59 UTC
[jira] [Commented] (BIGTOP-2561) add juju bundle for hadoop-spark
[ https://issues.apache.org/jira/browse/BIGTOP-2561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15716753#comment-15716753 ]
ASF GitHub Bot commented on BIGTOP-2561:
----------------------------------------
GitHub user kwmonroe opened a pull request:
https://github.com/apache/bigtop/pull/166
BIGTOP-2561: add juju bundle for hadoop-spark
This combines hadoop-processing and spark-processing into a single bundle:
https://jujucharms.com/hadoop-spark/
Test results are available here across multiple clouds:
http://data.vapour.ws/cwr-tests/results/bundle_hadoop_spark/index.html
We are working through infrastructure issues atm (hence a lot of yellow triangles), but we haven't seen any failures with the charms included in this bundle for the last couple weeks. Hence I'm putting it up for review / inclusion upstream.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/juju-solutions/bigtop feature/hadoop-spark-bundle
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/bigtop/pull/166.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #166
----
commit 35b8d13c198a98296f0290e340e7bc93a7e1b62d
Author: Kevin W Monroe <ke...@canonical.com>
Date: 2016-09-22T01:46:24Z
initial hadoop-spark bundle
commit d1387eda17b3d8cb326e5d0d12500a60ac66d04a
Author: Kevin W Monroe <ke...@canonical.com>
Date: 2016-09-22T23:20:51Z
colocate bundle applications
To save resources, colocate some of the applications when it makes sense.
Put the namenode and resourcemgr on the same unit; spark and zeppelin;
ganglia and rsyslog.
Update readme formatting and callout these colocated apps. Also tweak
the restricted network section and add a section for connecting to zepp.
Note, machine spec in the bundle yaml start with machine '1' instead of
'0'. This is because someone may deploy this on juju-1.x, where machine
'0' is the bootstrap node.
commit b52a7c5a7020ffaec25ca3f240768eaf3b4c1cac
Author: Konstantinos Tsakalozos <ko...@canonical.com>
Date: 2016-10-24T09:37:24Z
Be explicit on series for CWR to deploy the right series
commit 040bab24a58d0b4eda05a8ab215342a1f21ff9a7
Author: Kevin W Monroe <ke...@canonical.com>
Date: 2016-10-29T02:19:05Z
refresh charms
commit a29bac836b7d39adabe5d3c835f3d00e28b75f8a
Author: Kevin W Monroe <ke...@canonical.com>
Date: 2016-10-29T05:03:24Z
refresh hadoop-spark bundle
- update readme to be consistent with other bigtop-based bundles
- bump all charms to latest revisions
- update amulet test to call charm smoke tests and work around machine spec
commit ef3d70542babecd69baa0618e39388854968fda8
Author: Kevin W Monroe <ke...@canonical.com>
Date: 2016-10-31T16:45:08Z
skip slave smoke tests and update smoke failure message
commit 506ba320764323f4ffb294bce4834f044626e799
Author: Kevin W Monroe <ke...@canonical.com>
Date: 2016-10-31T22:29:15Z
bump spark revno for spark-bench fixes
commit 4c0f812d736b8d4c31bfce68b450d8b77adf79a0
Author: Kevin W Monroe <ke...@canonical.com>
Date: 2016-11-02T04:25:58Z
refresh spark charm and readme for new sparkbench suite
commit ff19096dbb9bf3fb21297fcc0a258d98453ea7cc
Author: Kevin W Monroe <ke...@canonical.com>
Date: 2016-11-03T04:33:14Z
charm revision bump
- new ganglia charms report app version and fix '-broken' hook failures
- new spark charm correctly runs sparkbench regardless of spark exe mode
commit 5a961a6fb15f403f80356de62b3acdaa7aa2b10c
Author: Kevin W Monroe <ke...@canonical.com>
Date: 2016-11-05T22:39:41Z
bump spark rev for updated test case and sparkbench composite is now seconds
commit 71c5731208c637bc2cb0ed2a44179c25479dc5e2
Author: Kevin W Monroe <ke...@canonical.com>
Date: 2016-11-07T17:53:37Z
bump zk revno to pick up test rename and app version
commit 745deec545226a5c9d83d9d749101bf227b4dec6
Author: Kevin W Monroe <ke...@canonical.com>
Date: 2016-11-15T21:12:22Z
bump rsyslog for lint and zk for an amulet test enhancement (ease local testing)
commit 638221be832236641eef3ceef0a9b46e11121077
Author: Kevin W Monroe <ke...@canonical.com>
Date: 2016-11-22T21:34:35Z
add ppa, charm rev bump, new constraints
- ensure tests use juju/stable ppa
- bump zk to latest (consistent bigtop_ prefix, readme tweaks)
- bump core hadoop charms to latest (new swap handler, detect failed service
start, add spark user/group to nn/rm)
- aws defaults to 8gb root filesystem; bump this to 20gb
commit 34ffbe4bc576ba3a330e8c1da6dd08a74f3a392a
Author: Kevin W Monroe <ke...@canonical.com>
Date: 2016-11-24T19:49:56Z
rev charms to pick up new swap handler
commit 3fc9934e7723e1ab8383d1a96c093f86d546178b
Author: Kevin W Monroe <ke...@canonical.com>
Date: 2016-11-28T23:16:11Z
workaround azure root-disk: https://bugs.launchpad.net/juju/+bug/1645408
----
> add juju bundle for hadoop-spark
> --------------------------------
>
> Key: BIGTOP-2561
> URL: https://issues.apache.org/jira/browse/BIGTOP-2561
> Project: Bigtop
> Issue Type: Improvement
> Components: deployment
> Affects Versions: 1.1.0
> Reporter: Kevin W Monroe
> Assignee: Kevin W Monroe
> Priority: Trivial
>
> Building off the core hadoop and spark bundles, we've put together one that combines these two:
> http://jujucharms.com/hadoop-spark
> This deploys hadoop and spark in yarn-client mode. Tests are running now; so far, so good.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)