You are viewing a plain text version of this content. The canonical link for it is here.
Posted to general@incubator.apache.org by Steve Loughran <st...@hortonworks.com> on 2014/01/08 16:08:31 UTC
Hoya Proposal
I'm starting to put together the incubation proposal for Hoya: a tool to
dynamically deploy applications such as HBase or Accumulo on YARN
https://wiki.apache.org/incubator/HoyaProposal
It does already work to the extent that it can bring up either application,
run different clusters of different versions, and remember where containers
were allocated so that on application restart it can ask for them back.
That increases data locality and makes a big difference with HBase.
It also needs a lot more work -YARN-896 is adding YARN features that help,
but there's lots of fun to be had in Hoya including
-leading edge work in failure handling, modelling cluster unreliability
and reacting to it. Can we move beyond simple blacklisting to
"greylisting", accepting unreliable boxes if we have no altenatives
Then there's adding more providers, to support different application
installations -I'm starting to write a functional test framework which need
provider-specific workload generations
Other features: AM should have a web ui that redirects to the live
endpoints to all the app-specific UIs (e.g. HBase Master GUI), as well as
displaying cluster state itself, for people and for management tools....
To summarise: lots of fun to be had
-steve
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: Hoya Proposal
Posted by Steve Loughran <st...@hortonworks.com>.
> Andreas, to me, Twill is a library, a convenience library, that one
> can use to write Yarn apps. Hoya aims to provide a general framework
> using which one can take existing apps (HBase/Accumulo to start with),
> and make them run well in a Yarn cluster, without intruding at all
> into the App internals. The goal is to have minimal, ideally zero
> changes, in the App itself. The only glue between the App and Hoya is
> a Hoya interface that the App needs to implement for it to be
> deployable/manageable by Hoya.
>
> Although, one could argue that Hoya can be written using Twill
> libraries (which is something we should pursue as part of
> long/medium-term collaboration between the two projects),
FWIW the first iteration was 100% groovy, using my 2012 "Grumpy" code as a
starting point https://github.com/steveloughran/grumpy -
The later iterations are all Java.
The current design tries to have a clean model-view split server side, with
all the server state stored independently from the AM itself,
https://github.com/hortonworks/hoya/tree/develop/hoya-core/src/main/java/org/apache/hoya/yarn/appmaster/state
This state is something that could perhaps be worked out into something
more re-usable, as all it really does is
-take a specification of role types, the no. of instances of each, and
some options (there's an unused anti-affinity flag)
- build a model of the cluster, including live containers, queued
operations, nodes in the YARN cluster and their history
- Build up a list of actions: requests and releases aimed at keeping
the model (and hence the deployed app) consistent with the
specification.
- persist placement history to HDFS for best-effort reassignment
requests on cluster restart ( see
https://github.com/hortonworks/hoya/blob/master/src/site/markdown/rolehistory.md
)
- Respond to relayed events (container assignment, failure, rebuild
after AM restart by updating model
For any long-lived YARN service, the goal "keep the number of
instances of a role constant" is an underlying use case, from
Distributed Shell up. Hoya just effectively allows this to be driven
by an JSON document persisted to HDFS, updated via RPC, and delegates
to plugin providers the work of actually starting processes at the far
end.
I've love to see this pulled out and re-usable, indeed, I'd love to
see the YARN dshell architecture redone a bit more cleanly too:
https://issues.apache.org/jira/browse/YARN-1360
Not only would that give something for others to use, but there are
some aspect of failure handling that we can/should improve in a way
that would benefit all apps, such as better weighted moving average
failure tracking, sophisticated policies on reacting to it, and moving
beyond a simple works/doesn't work to a more nuanced greylist of
perceived server reliability:
http://steveloughran.blogspot.co.uk/2014/01/greylisting-like-blacklisting-only-more.html
If we can work with the Twill team to factor this stuff into something
broadly re-usable, that would be great for everyone.
-steve
ps. I'd a standard management service API too
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
For additional commands, e-mail: general-help@incubator.apache.org
Re: Hoya Proposal
Posted by Bernd Fondermann <be...@gmail.com>.
On Fri, Jan 17, 2014 at 1:13 PM, Steve Loughran <st...@hortonworks.com> wrote:
> On 17 January 2014 10:10, Bernd Fondermann <be...@gmail.com>wrote:
>
>> (side note: When a project graduates, a PMC is established and tasked with
>> a specific task. Two PMCs cannot have the same task. Therefore, it is
>> better in my opinion to either clearly differentiate the projects from
>> start, or join them from start.)
>>
>>
> Really? Because while I am confident that hoya and twill are different, i
> do note similarities between ant, maven and buildr, commons-logging, log4j
> and avalon-logger,..
Now if two podling entered the Incubator around the same time with the
same goal I think it would be unwise to not a. set both on parallel
tracks instead of the same or b. make them one train from start.
Bernd
---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
For additional commands, e-mail: general-help@incubator.apache.org
Re: Hoya Proposal
Posted by Steve Loughran <st...@hortonworks.com>.
On 17 January 2014 10:10, Bernd Fondermann <be...@gmail.com>wrote:
> Very informative explanation. I've got a better understanding of the
> proposal now.
>
> I suggest that a remark like this is added to the Hoya proposal, so this
> part of the discussion becomes part of the vote on the proposal.The
> proposal's "Known Risk" section lacks a "Relationships with Other Apache
> Products" paragraph, which would be a good place to add this information.
> There are more considerations missing that we have in other proposals. I'd
> suggest to add them, too, to complete the proposal.
>
I've added more on this
>
> All-in-all I'm currently -1 on the proposal as it is, but ready to change
> to +1 as this discussion reaches a consent.
>
>
Would you or anyone else like a Demo? I could do it online via google+ or
webex
> (side note: When a project graduates, a PMC is established and tasked with
> a specific task. Two PMCs cannot have the same task. Therefore, it is
> better in my opinion to either clearly differentiate the projects from
> start, or join them from start.)
>
>
Really? Because while I am confident that hoya and twill are different, i
do note similarities between ant, maven and buildr, commons-logging, log4j
and avalon-logger,..
Of course, if you look at the history there they were launched
sequentially, and apache logging is now an uber-project for everything bar
avalon, whose sole appearance is in unexplained transitive artefact
dependencies.
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: Hoya Proposal
Posted by Bernd Fondermann <be...@gmail.com>.
Very informative explanation. I've got a better understanding of the
proposal now.
I suggest that a remark like this is added to the Hoya proposal, so this
part of the discussion becomes part of the vote on the proposal.The
proposal's "Known Risk" section lacks a "Relationships with Other Apache
Products" paragraph, which would be a good place to add this information.
There are more considerations missing that we have in other proposals. I'd
suggest to add them, too, to complete the proposal.
All-in-all I'm currently -1 on the proposal as it is, but ready to change
to +1 as this discussion reaches a consent.
(side note: When a project graduates, a PMC is established and tasked with
a specific task. Two PMCs cannot have the same task. Therefore, it is
better in my opinion to either clearly differentiate the projects from
start, or join them from start.)
Thanks,
Bernd
On Fri, Jan 17, 2014 at 8:48 AM, Devaraj Das <dd...@hortonworks.com> wrote:
> Andreas, to me, Twill is a library, a convenience library, that one
> can use to write Yarn apps. Hoya aims to provide a general framework
> using which one can take existing apps (HBase/Accumulo to start with),
> and make them run well in a Yarn cluster, without intruding at all
> into the App internals. The goal is to have minimal, ideally zero
> changes, in the App itself. The only glue between the App and Hoya is
> a Hoya interface that the App needs to implement for it to be
> deployable/manageable by Hoya.
>
> Although, one could argue that Hoya can be written using Twill
> libraries (which is something we should pursue as part of
> long/medium-term collaboration between the two projects), but I'd
> argue that the goals of the two projects are different - Twill would
> continue to make Yarn app developers' lives easier, while Hoya is a
> tool that could deploy distributed-frameworks easily in a Yarn
> cluster, and be able to later do basic management (as is talked about
> in the github docs). Things like dynamic configuration patching for
> the applications like HBase to easily run in a Yarn cluster, security
> issues, failure handling and defining a model for reacting to
> failures, being able to store some state about applications to
> facilitate better application restart behavior in a Yarn cluster, etc.
> would be in the purview of Hoya. Management frameworks could use Hoya
> as a tool as well to talk to Yarn and do application
> start/stop/shrink/expand an instance of an application (e.g. HBase)
> cluster.
>
> On Thu, Jan 16, 2014 at 4:38 PM, Andreas Neumann <an...@apache.org> wrote:
>> I do see a lot of value in all the features of Hoya, and I am not trying to
>> discredit it. Yet I do think that most of these features are actually
>> already in Twill or would be great additions to Twill, and sooner or later
>> will be implemented in Twill, implying even greater overlap.
>>
>> I am not sure whether I follow the distinction between porting an existing
>> app and developing a new app, if both require similar abstractions... And
>> after all, porting an existing application is one way to start developing a
>> Yarn application.
>>
>> Anyway, I just wanted to point out the overlap and offer collaboration. If
>> the community thinks that the two projects are different beasts, that's
>> fine with me.
>>
>>
>> On Wed, Jan 15, 2014 at 10:40 AM, Josh Elser <jo...@gmail.com> wrote:
>>
>>> I wanted to weigh in on some of Steve's thoughts, I'm actually really
>>> excited about being able to leverage Hoya inside of Accumulo itself.
>>>
>>> We presently have a few system tests that rely on manual set up, which can
>>> be frustrating to deal with on a beefy boxes (running multiple Accumulo
>>> procs on a single host). Hoya drastically reduces the amount of effort it
>>> takes to run these distributed tests in a 'closer to real' environment.
>>> On Jan 15, 2014 8:36 AM, "Steve Loughran" <st...@hortonworks.com> wrote:
>>>
>>> > On 15 January 2014 02:13, Andreas Neumann <an...@apache.org> wrote:
>>> >
>>> > > I see. So is Hoya limited to HBase and Accumulo? Or is it open for any
>>> > > other type of existing application? If so, won't it have some common
>>> > > abstraction that is shared by all of them? That is where I see the
>>> > > similarity with Twill.
>>> > >
>>> > >
>>> > it started off as pure hbase, but now has the notion of a provider which
>>> > has a client-side and server side
>>> >
>>> > client side
>>> > -helps set up the template JSON file to describe the cluster (e.g. adds
>>> > default values),
>>> > -patches the configuration directory supplied at creation time with
>>> > whatever options it wants (e.g links up fs.default.name & ZK settings in
>>> > hbase-site.xml)
>>> > -does preflight validation of cluster options
>>> > -can also add its own .tar.gz to the resources of the AM (or any other
>>> > resources)
>>> >
>>> > server side
>>> > -runs in the AM and sets up everything needed to run instances of a role
>>> > (e.g. HBase master, Accumulo GC, ..)
>>> > -can run code in the AM to help set things up (e.g. Accumulo provider
>>> > service runs bin/accumulo init if needed)
>>> > -TODO: liveness monitoring
>>> >
>>> > the requirements of an app to be deployable are
>>> >
>>> >
>>> https://github.com/hortonworks/hoya/blob/master/src/site/markdown/app_needs.md
>>> >
>>> >
>>> >
>>> > > Whereas, if it is a separate effort for each existing application, say
>>> > > HBase, then what is the end goal for Hoya when it comes out of
>>> > incubation?
>>> > > To merge it back into HBase proper?
>>> > >
>>> > >
>>> > Now that it supports >1 application, it can't go into HBase. The
>>> individual
>>> > provider services can (their implementation classes are worked out via
>>> the
>>> > configuration.XML).
>>> >
>>> > But as we use the accumulo and HBase apps for testing, its really good to
>>> > have them both in the hoya project right now -a project that builds
>>> > downstream of both and needs
>>> > to be given the Hadoop filesystem paths to .tar.gz files of each app.
>>> >
>>> > There's nothing to stop 3rd party apps joining in, indeed, one thing I'd
>>> > like someone to do is actually have Hoya deploy SmartFrog .tar.gz files
>>> and
>>> > pass down configurations that deploy applications -for example
>>> jetty-based
>>> > web servers. That'd take an intern working at HP Labs over the summer,
>>> > maybe
>>> >
>>> > -steve
>>> >
>>> > --
>>> > CONFIDENTIALITY NOTICE
>>> > NOTICE: This message is intended for the use of the individual or entity
>>> to
>>> > which it is addressed and may contain information that is confidential,
>>> > privileged and exempt from disclosure under applicable law. If the reader
>>> > of this message is not the intended recipient, you are hereby notified
>>> that
>>> > any printing, copying, dissemination, distribution, disclosure or
>>> > forwarding of this communication is strictly prohibited. If you have
>>> > received this communication in error, please contact the sender
>>> immediately
>>> > and delete it from your system. Thank You.
>>> >
>>>
>
> --
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity to
> which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> For additional commands, e-mail: general-help@incubator.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
For additional commands, e-mail: general-help@incubator.apache.org
Re: Hoya Proposal
Posted by Devaraj Das <dd...@hortonworks.com>.
Andreas, to me, Twill is a library, a convenience library, that one
can use to write Yarn apps. Hoya aims to provide a general framework
using which one can take existing apps (HBase/Accumulo to start with),
and make them run well in a Yarn cluster, without intruding at all
into the App internals. The goal is to have minimal, ideally zero
changes, in the App itself. The only glue between the App and Hoya is
a Hoya interface that the App needs to implement for it to be
deployable/manageable by Hoya.
Although, one could argue that Hoya can be written using Twill
libraries (which is something we should pursue as part of
long/medium-term collaboration between the two projects), but I'd
argue that the goals of the two projects are different - Twill would
continue to make Yarn app developers' lives easier, while Hoya is a
tool that could deploy distributed-frameworks easily in a Yarn
cluster, and be able to later do basic management (as is talked about
in the github docs). Things like dynamic configuration patching for
the applications like HBase to easily run in a Yarn cluster, security
issues, failure handling and defining a model for reacting to
failures, being able to store some state about applications to
facilitate better application restart behavior in a Yarn cluster, etc.
would be in the purview of Hoya. Management frameworks could use Hoya
as a tool as well to talk to Yarn and do application
start/stop/shrink/expand an instance of an application (e.g. HBase)
cluster.
On Thu, Jan 16, 2014 at 4:38 PM, Andreas Neumann <an...@apache.org> wrote:
> I do see a lot of value in all the features of Hoya, and I am not trying to
> discredit it. Yet I do think that most of these features are actually
> already in Twill or would be great additions to Twill, and sooner or later
> will be implemented in Twill, implying even greater overlap.
>
> I am not sure whether I follow the distinction between porting an existing
> app and developing a new app, if both require similar abstractions... And
> after all, porting an existing application is one way to start developing a
> Yarn application.
>
> Anyway, I just wanted to point out the overlap and offer collaboration. If
> the community thinks that the two projects are different beasts, that's
> fine with me.
>
>
> On Wed, Jan 15, 2014 at 10:40 AM, Josh Elser <jo...@gmail.com> wrote:
>
>> I wanted to weigh in on some of Steve's thoughts, I'm actually really
>> excited about being able to leverage Hoya inside of Accumulo itself.
>>
>> We presently have a few system tests that rely on manual set up, which can
>> be frustrating to deal with on a beefy boxes (running multiple Accumulo
>> procs on a single host). Hoya drastically reduces the amount of effort it
>> takes to run these distributed tests in a 'closer to real' environment.
>> On Jan 15, 2014 8:36 AM, "Steve Loughran" <st...@hortonworks.com> wrote:
>>
>> > On 15 January 2014 02:13, Andreas Neumann <an...@apache.org> wrote:
>> >
>> > > I see. So is Hoya limited to HBase and Accumulo? Or is it open for any
>> > > other type of existing application? If so, won't it have some common
>> > > abstraction that is shared by all of them? That is where I see the
>> > > similarity with Twill.
>> > >
>> > >
>> > it started off as pure hbase, but now has the notion of a provider which
>> > has a client-side and server side
>> >
>> > client side
>> > -helps set up the template JSON file to describe the cluster (e.g. adds
>> > default values),
>> > -patches the configuration directory supplied at creation time with
>> > whatever options it wants (e.g links up fs.default.name & ZK settings in
>> > hbase-site.xml)
>> > -does preflight validation of cluster options
>> > -can also add its own .tar.gz to the resources of the AM (or any other
>> > resources)
>> >
>> > server side
>> > -runs in the AM and sets up everything needed to run instances of a role
>> > (e.g. HBase master, Accumulo GC, ..)
>> > -can run code in the AM to help set things up (e.g. Accumulo provider
>> > service runs bin/accumulo init if needed)
>> > -TODO: liveness monitoring
>> >
>> > the requirements of an app to be deployable are
>> >
>> >
>> https://github.com/hortonworks/hoya/blob/master/src/site/markdown/app_needs.md
>> >
>> >
>> >
>> > > Whereas, if it is a separate effort for each existing application, say
>> > > HBase, then what is the end goal for Hoya when it comes out of
>> > incubation?
>> > > To merge it back into HBase proper?
>> > >
>> > >
>> > Now that it supports >1 application, it can't go into HBase. The
>> individual
>> > provider services can (their implementation classes are worked out via
>> the
>> > configuration.XML).
>> >
>> > But as we use the accumulo and HBase apps for testing, its really good to
>> > have them both in the hoya project right now -a project that builds
>> > downstream of both and needs
>> > to be given the Hadoop filesystem paths to .tar.gz files of each app.
>> >
>> > There's nothing to stop 3rd party apps joining in, indeed, one thing I'd
>> > like someone to do is actually have Hoya deploy SmartFrog .tar.gz files
>> and
>> > pass down configurations that deploy applications -for example
>> jetty-based
>> > web servers. That'd take an intern working at HP Labs over the summer,
>> > maybe
>> >
>> > -steve
>> >
>> > --
>> > CONFIDENTIALITY NOTICE
>> > NOTICE: This message is intended for the use of the individual or entity
>> to
>> > which it is addressed and may contain information that is confidential,
>> > privileged and exempt from disclosure under applicable law. If the reader
>> > of this message is not the intended recipient, you are hereby notified
>> that
>> > any printing, copying, dissemination, distribution, disclosure or
>> > forwarding of this communication is strictly prohibited. If you have
>> > received this communication in error, please contact the sender
>> immediately
>> > and delete it from your system. Thank You.
>> >
>>
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
For additional commands, e-mail: general-help@incubator.apache.org
Re: Hoya Proposal
Posted by Andreas Neumann <an...@apache.org>.
I do see a lot of value in all the features of Hoya, and I am not trying to
discredit it. Yet I do think that most of these features are actually
already in Twill or would be great additions to Twill, and sooner or later
will be implemented in Twill, implying even greater overlap.
I am not sure whether I follow the distinction between porting an existing
app and developing a new app, if both require similar abstractions... And
after all, porting an existing application is one way to start developing a
Yarn application.
Anyway, I just wanted to point out the overlap and offer collaboration. If
the community thinks that the two projects are different beasts, that's
fine with me.
On Wed, Jan 15, 2014 at 10:40 AM, Josh Elser <jo...@gmail.com> wrote:
> I wanted to weigh in on some of Steve's thoughts, I'm actually really
> excited about being able to leverage Hoya inside of Accumulo itself.
>
> We presently have a few system tests that rely on manual set up, which can
> be frustrating to deal with on a beefy boxes (running multiple Accumulo
> procs on a single host). Hoya drastically reduces the amount of effort it
> takes to run these distributed tests in a 'closer to real' environment.
> On Jan 15, 2014 8:36 AM, "Steve Loughran" <st...@hortonworks.com> wrote:
>
> > On 15 January 2014 02:13, Andreas Neumann <an...@apache.org> wrote:
> >
> > > I see. So is Hoya limited to HBase and Accumulo? Or is it open for any
> > > other type of existing application? If so, won't it have some common
> > > abstraction that is shared by all of them? That is where I see the
> > > similarity with Twill.
> > >
> > >
> > it started off as pure hbase, but now has the notion of a provider which
> > has a client-side and server side
> >
> > client side
> > -helps set up the template JSON file to describe the cluster (e.g. adds
> > default values),
> > -patches the configuration directory supplied at creation time with
> > whatever options it wants (e.g links up fs.default.name & ZK settings in
> > hbase-site.xml)
> > -does preflight validation of cluster options
> > -can also add its own .tar.gz to the resources of the AM (or any other
> > resources)
> >
> > server side
> > -runs in the AM and sets up everything needed to run instances of a role
> > (e.g. HBase master, Accumulo GC, ..)
> > -can run code in the AM to help set things up (e.g. Accumulo provider
> > service runs bin/accumulo init if needed)
> > -TODO: liveness monitoring
> >
> > the requirements of an app to be deployable are
> >
> >
> https://github.com/hortonworks/hoya/blob/master/src/site/markdown/app_needs.md
> >
> >
> >
> > > Whereas, if it is a separate effort for each existing application, say
> > > HBase, then what is the end goal for Hoya when it comes out of
> > incubation?
> > > To merge it back into HBase proper?
> > >
> > >
> > Now that it supports >1 application, it can't go into HBase. The
> individual
> > provider services can (their implementation classes are worked out via
> the
> > configuration.XML).
> >
> > But as we use the accumulo and HBase apps for testing, its really good to
> > have them both in the hoya project right now -a project that builds
> > downstream of both and needs
> > to be given the Hadoop filesystem paths to .tar.gz files of each app.
> >
> > There's nothing to stop 3rd party apps joining in, indeed, one thing I'd
> > like someone to do is actually have Hoya deploy SmartFrog .tar.gz files
> and
> > pass down configurations that deploy applications -for example
> jetty-based
> > web servers. That'd take an intern working at HP Labs over the summer,
> > maybe
> >
> > -steve
> >
> > --
> > CONFIDENTIALITY NOTICE
> > NOTICE: This message is intended for the use of the individual or entity
> to
> > which it is addressed and may contain information that is confidential,
> > privileged and exempt from disclosure under applicable law. If the reader
> > of this message is not the intended recipient, you are hereby notified
> that
> > any printing, copying, dissemination, distribution, disclosure or
> > forwarding of this communication is strictly prohibited. If you have
> > received this communication in error, please contact the sender
> immediately
> > and delete it from your system. Thank You.
> >
>
Re: Hoya Proposal
Posted by Josh Elser <jo...@gmail.com>.
I wanted to weigh in on some of Steve's thoughts, I'm actually really
excited about being able to leverage Hoya inside of Accumulo itself.
We presently have a few system tests that rely on manual set up, which can
be frustrating to deal with on a beefy boxes (running multiple Accumulo
procs on a single host). Hoya drastically reduces the amount of effort it
takes to run these distributed tests in a 'closer to real' environment.
On Jan 15, 2014 8:36 AM, "Steve Loughran" <st...@hortonworks.com> wrote:
> On 15 January 2014 02:13, Andreas Neumann <an...@apache.org> wrote:
>
> > I see. So is Hoya limited to HBase and Accumulo? Or is it open for any
> > other type of existing application? If so, won't it have some common
> > abstraction that is shared by all of them? That is where I see the
> > similarity with Twill.
> >
> >
> it started off as pure hbase, but now has the notion of a provider which
> has a client-side and server side
>
> client side
> -helps set up the template JSON file to describe the cluster (e.g. adds
> default values),
> -patches the configuration directory supplied at creation time with
> whatever options it wants (e.g links up fs.default.name & ZK settings in
> hbase-site.xml)
> -does preflight validation of cluster options
> -can also add its own .tar.gz to the resources of the AM (or any other
> resources)
>
> server side
> -runs in the AM and sets up everything needed to run instances of a role
> (e.g. HBase master, Accumulo GC, ..)
> -can run code in the AM to help set things up (e.g. Accumulo provider
> service runs bin/accumulo init if needed)
> -TODO: liveness monitoring
>
> the requirements of an app to be deployable are
>
> https://github.com/hortonworks/hoya/blob/master/src/site/markdown/app_needs.md
>
>
>
> > Whereas, if it is a separate effort for each existing application, say
> > HBase, then what is the end goal for Hoya when it comes out of
> incubation?
> > To merge it back into HBase proper?
> >
> >
> Now that it supports >1 application, it can't go into HBase. The individual
> provider services can (their implementation classes are worked out via the
> configuration.XML).
>
> But as we use the accumulo and HBase apps for testing, its really good to
> have them both in the hoya project right now -a project that builds
> downstream of both and needs
> to be given the Hadoop filesystem paths to .tar.gz files of each app.
>
> There's nothing to stop 3rd party apps joining in, indeed, one thing I'd
> like someone to do is actually have Hoya deploy SmartFrog .tar.gz files and
> pass down configurations that deploy applications -for example jetty-based
> web servers. That'd take an intern working at HP Labs over the summer,
> maybe
>
> -steve
>
> --
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity to
> which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
>
Re: Hoya Proposal
Posted by Bernd Fondermann <be...@gmail.com>.
I think that Andreas has a valid point.
With the description you are giving here, there seems to be much more
overlap with the Twill podling than I initially anticipated.
In the Hoya proposal, I'd like to learn about how it compares to Twill and
why it makes sense to start another such podling while the other one is
still ongoing.
Bernd
On Wed, Jan 15, 2014 at 2:36 PM, Steve Loughran <st...@hortonworks.com>wrote:
> On 15 January 2014 02:13, Andreas Neumann <an...@apache.org> wrote:
>
> > I see. So is Hoya limited to HBase and Accumulo? Or is it open for any
> > other type of existing application? If so, won't it have some common
> > abstraction that is shared by all of them? That is where I see the
> > similarity with Twill.
> >
> >
> it started off as pure hbase, but now has the notion of a provider which
> has a client-side and server side
>
> client side
> -helps set up the template JSON file to describe the cluster (e.g. adds
> default values),
> -patches the configuration directory supplied at creation time with
> whatever options it wants (e.g links up fs.default.name & ZK settings in
> hbase-site.xml)
> -does preflight validation of cluster options
> -can also add its own .tar.gz to the resources of the AM (or any other
> resources)
>
> server side
> -runs in the AM and sets up everything needed to run instances of a role
> (e.g. HBase master, Accumulo GC, ..)
> -can run code in the AM to help set things up (e.g. Accumulo provider
> service runs bin/accumulo init if needed)
> -TODO: liveness monitoring
>
> the requirements of an app to be deployable are
>
> https://github.com/hortonworks/hoya/blob/master/src/site/markdown/app_needs.md
>
>
>
> > Whereas, if it is a separate effort for each existing application, say
> > HBase, then what is the end goal for Hoya when it comes out of
> incubation?
> > To merge it back into HBase proper?
> >
> >
> Now that it supports >1 application, it can't go into HBase. The individual
> provider services can (their implementation classes are worked out via the
> configuration.XML).
>
> But as we use the accumulo and HBase apps for testing, its really good to
> have them both in the hoya project right now -a project that builds
> downstream of both and needs
> to be given the Hadoop filesystem paths to .tar.gz files of each app.
>
> There's nothing to stop 3rd party apps joining in, indeed, one thing I'd
> like someone to do is actually have Hoya deploy SmartFrog .tar.gz files and
> pass down configurations that deploy applications -for example jetty-based
> web servers. That'd take an intern working at HP Labs over the summer,
> maybe
>
> -steve
>
> --
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity to
> which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
>
Re: Hoya Proposal
Posted by Steve Loughran <st...@hortonworks.com>.
On 15 January 2014 02:13, Andreas Neumann <an...@apache.org> wrote:
> I see. So is Hoya limited to HBase and Accumulo? Or is it open for any
> other type of existing application? If so, won't it have some common
> abstraction that is shared by all of them? That is where I see the
> similarity with Twill.
>
>
it started off as pure hbase, but now has the notion of a provider which
has a client-side and server side
client side
-helps set up the template JSON file to describe the cluster (e.g. adds
default values),
-patches the configuration directory supplied at creation time with
whatever options it wants (e.g links up fs.default.name & ZK settings in
hbase-site.xml)
-does preflight validation of cluster options
-can also add its own .tar.gz to the resources of the AM (or any other
resources)
server side
-runs in the AM and sets up everything needed to run instances of a role
(e.g. HBase master, Accumulo GC, ..)
-can run code in the AM to help set things up (e.g. Accumulo provider
service runs bin/accumulo init if needed)
-TODO: liveness monitoring
the requirements of an app to be deployable are
https://github.com/hortonworks/hoya/blob/master/src/site/markdown/app_needs.md
> Whereas, if it is a separate effort for each existing application, say
> HBase, then what is the end goal for Hoya when it comes out of incubation?
> To merge it back into HBase proper?
>
>
Now that it supports >1 application, it can't go into HBase. The individual
provider services can (their implementation classes are worked out via the
configuration.XML).
But as we use the accumulo and HBase apps for testing, its really good to
have them both in the hoya project right now -a project that builds
downstream of both and needs
to be given the Hadoop filesystem paths to .tar.gz files of each app.
There's nothing to stop 3rd party apps joining in, indeed, one thing I'd
like someone to do is actually have Hoya deploy SmartFrog .tar.gz files and
pass down configurations that deploy applications -for example jetty-based
web servers. That'd take an intern working at HP Labs over the summer, maybe
-steve
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: Hoya Proposal
Posted by Andreas Neumann <an...@apache.org>.
I see. So is Hoya limited to HBase and Accumulo? Or is it open for any
other type of existing application? If so, won't it have some common
abstraction that is shared by all of them? That is where I see the
similarity with Twill.
Whereas, if it is a separate effort for each existing application, say
HBase, then what is the end goal for Hoya when it comes out of incubation?
To merge it back into HBase proper?
On Tue, Jan 14, 2014 at 1:38 AM, Steve Loughran <st...@hortonworks.com>wrote:
> I'd put more in common with Helix than twill -twill is for building YARN
> apps, Hoya focused on deploying existing apps into YARN containers. It's a
> lot more minimal than Helix -it doesn't run its own code in any of the
> containers other than the AM
>
>
> On 14 January 2014 03:33, Andreas Neumann <an...@apache.org> wrote:
>
> > Hi Steve,
> >
> > Hoya has a lot of good ideas. How do think it relates to Twill (recently
> > incubated at http://twill.incubator.apache.org/)? I feel that there is a
> > lot of overlap in what both projects are trying to achieve. Do you think
> > they complement each other? Is there opportunity for collaboration,
> perhaps
> > cross-contributing or even converging the two projects?
> >
> > -Andreas.
> >
> >
> > On Wed, Jan 8, 2014 at 7:08 AM, Steve Loughran <stevel@hortonworks.com
> > >wrote:
> >
> > > I'm starting to put together the incubation proposal for Hoya: a tool
> to
> > > dynamically deploy applications such as HBase or Accumulo on YARN
> > >
> > > https://wiki.apache.org/incubator/HoyaProposal
> > >
> > > It does already work to the extent that it can bring up either
> > application,
> > > run different clusters of different versions, and remember where
> > containers
> > > were allocated so that on application restart it can ask for them back.
> > > That increases data locality and makes a big difference with HBase.
> > >
> > > It also needs a lot more work -YARN-896 is adding YARN features that
> > help,
> > > but there's lots of fun to be had in Hoya including
> > > -leading edge work in failure handling, modelling cluster
> unreliability
> > > and reacting to it. Can we move beyond simple blacklisting to
> > > "greylisting", accepting unreliable boxes if we have no altenatives
> > >
> > > Then there's adding more providers, to support different application
> > > installations -I'm starting to write a functional test framework which
> > need
> > > provider-specific workload generations
> > >
> > > Other features: AM should have a web ui that redirects to the live
> > > endpoints to all the app-specific UIs (e.g. HBase Master GUI), as well
> as
> > > displaying cluster state itself, for people and for management
> tools....
> > >
> > > To summarise: lots of fun to be had
> > >
> > > -steve
> > >
> > > --
> > > CONFIDENTIALITY NOTICE
> > > NOTICE: This message is intended for the use of the individual or
> entity
> > to
> > > which it is addressed and may contain information that is confidential,
> > > privileged and exempt from disclosure under applicable law. If the
> reader
> > > of this message is not the intended recipient, you are hereby notified
> > that
> > > any printing, copying, dissemination, distribution, disclosure or
> > > forwarding of this communication is strictly prohibited. If you have
> > > received this communication in error, please contact the sender
> > immediately
> > > and delete it from your system. Thank You.
> > >
> >
>
> --
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity to
> which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
>
Re: Hoya Proposal
Posted by Steve Loughran <st...@hortonworks.com>.
I'd put more in common with Helix than twill -twill is for building YARN
apps, Hoya focused on deploying existing apps into YARN containers. It's a
lot more minimal than Helix -it doesn't run its own code in any of the
containers other than the AM
On 14 January 2014 03:33, Andreas Neumann <an...@apache.org> wrote:
> Hi Steve,
>
> Hoya has a lot of good ideas. How do think it relates to Twill (recently
> incubated at http://twill.incubator.apache.org/)? I feel that there is a
> lot of overlap in what both projects are trying to achieve. Do you think
> they complement each other? Is there opportunity for collaboration, perhaps
> cross-contributing or even converging the two projects?
>
> -Andreas.
>
>
> On Wed, Jan 8, 2014 at 7:08 AM, Steve Loughran <stevel@hortonworks.com
> >wrote:
>
> > I'm starting to put together the incubation proposal for Hoya: a tool to
> > dynamically deploy applications such as HBase or Accumulo on YARN
> >
> > https://wiki.apache.org/incubator/HoyaProposal
> >
> > It does already work to the extent that it can bring up either
> application,
> > run different clusters of different versions, and remember where
> containers
> > were allocated so that on application restart it can ask for them back.
> > That increases data locality and makes a big difference with HBase.
> >
> > It also needs a lot more work -YARN-896 is adding YARN features that
> help,
> > but there's lots of fun to be had in Hoya including
> > -leading edge work in failure handling, modelling cluster unreliability
> > and reacting to it. Can we move beyond simple blacklisting to
> > "greylisting", accepting unreliable boxes if we have no altenatives
> >
> > Then there's adding more providers, to support different application
> > installations -I'm starting to write a functional test framework which
> need
> > provider-specific workload generations
> >
> > Other features: AM should have a web ui that redirects to the live
> > endpoints to all the app-specific UIs (e.g. HBase Master GUI), as well as
> > displaying cluster state itself, for people and for management tools....
> >
> > To summarise: lots of fun to be had
> >
> > -steve
> >
> > --
> > CONFIDENTIALITY NOTICE
> > NOTICE: This message is intended for the use of the individual or entity
> to
> > which it is addressed and may contain information that is confidential,
> > privileged and exempt from disclosure under applicable law. If the reader
> > of this message is not the intended recipient, you are hereby notified
> that
> > any printing, copying, dissemination, distribution, disclosure or
> > forwarding of this communication is strictly prohibited. If you have
> > received this communication in error, please contact the sender
> immediately
> > and delete it from your system. Thank You.
> >
>
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: Hoya Proposal
Posted by Andreas Neumann <an...@apache.org>.
Hi Steve,
Hoya has a lot of good ideas. How do think it relates to Twill (recently
incubated at http://twill.incubator.apache.org/)? I feel that there is a
lot of overlap in what both projects are trying to achieve. Do you think
they complement each other? Is there opportunity for collaboration, perhaps
cross-contributing or even converging the two projects?
-Andreas.
On Wed, Jan 8, 2014 at 7:08 AM, Steve Loughran <st...@hortonworks.com>wrote:
> I'm starting to put together the incubation proposal for Hoya: a tool to
> dynamically deploy applications such as HBase or Accumulo on YARN
>
> https://wiki.apache.org/incubator/HoyaProposal
>
> It does already work to the extent that it can bring up either application,
> run different clusters of different versions, and remember where containers
> were allocated so that on application restart it can ask for them back.
> That increases data locality and makes a big difference with HBase.
>
> It also needs a lot more work -YARN-896 is adding YARN features that help,
> but there's lots of fun to be had in Hoya including
> -leading edge work in failure handling, modelling cluster unreliability
> and reacting to it. Can we move beyond simple blacklisting to
> "greylisting", accepting unreliable boxes if we have no altenatives
>
> Then there's adding more providers, to support different application
> installations -I'm starting to write a functional test framework which need
> provider-specific workload generations
>
> Other features: AM should have a web ui that redirects to the live
> endpoints to all the app-specific UIs (e.g. HBase Master GUI), as well as
> displaying cluster state itself, for people and for management tools....
>
> To summarise: lots of fun to be had
>
> -steve
>
> --
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity to
> which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
>
Re: Hoya Proposal
Posted by Steve Loughran <st...@hortonworks.com>.
We're doing some redesign and need to get some more progress on that before
putting the proposal out -otherwise people wouldn't get a current view of
what's being built.
-steve
On 5 March 2014 01:07, Roman Shaposhnik <rv...@apache.org> wrote:
> Steve,
>
> I am just wondering: are you in a position to move forward with the
> proposal at this point or are you still mulling over the feedback?
>
> Thanks,
> Roman.
>
> On Wed, Jan 8, 2014 at 7:08 AM, Steve Loughran <st...@hortonworks.com>
> wrote:
> > I'm starting to put together the incubation proposal for Hoya: a tool to
> > dynamically deploy applications such as HBase or Accumulo on YARN
> >
> > https://wiki.apache.org/incubator/HoyaProposal
> >
> > It does already work to the extent that it can bring up either
> application,
> > run different clusters of different versions, and remember where
> containers
> > were allocated so that on application restart it can ask for them back.
> > That increases data locality and makes a big difference with HBase.
> >
> > It also needs a lot more work -YARN-896 is adding YARN features that
> help,
> > but there's lots of fun to be had in Hoya including
> > -leading edge work in failure handling, modelling cluster unreliability
> > and reacting to it. Can we move beyond simple blacklisting to
> > "greylisting", accepting unreliable boxes if we have no altenatives
> >
> > Then there's adding more providers, to support different application
> > installations -I'm starting to write a functional test framework which
> need
> > provider-specific workload generations
> >
> > Other features: AM should have a web ui that redirects to the live
> > endpoints to all the app-specific UIs (e.g. HBase Master GUI), as well as
> > displaying cluster state itself, for people and for management tools....
> >
> > To summarise: lots of fun to be had
> >
> > -steve
> >
> > --
> > CONFIDENTIALITY NOTICE
> > NOTICE: This message is intended for the use of the individual or entity
> to
> > which it is addressed and may contain information that is confidential,
> > privileged and exempt from disclosure under applicable law. If the reader
> > of this message is not the intended recipient, you are hereby notified
> that
> > any printing, copying, dissemination, distribution, disclosure or
> > forwarding of this communication is strictly prohibited. If you have
> > received this communication in error, please contact the sender
> immediately
> > and delete it from your system. Thank You.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> For additional commands, e-mail: general-help@incubator.apache.org
>
>
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: Hoya Proposal
Posted by Roman Shaposhnik <rv...@apache.org>.
Steve,
I am just wondering: are you in a position to move forward with the
proposal at this point or are you still mulling over the feedback?
Thanks,
Roman.
On Wed, Jan 8, 2014 at 7:08 AM, Steve Loughran <st...@hortonworks.com> wrote:
> I'm starting to put together the incubation proposal for Hoya: a tool to
> dynamically deploy applications such as HBase or Accumulo on YARN
>
> https://wiki.apache.org/incubator/HoyaProposal
>
> It does already work to the extent that it can bring up either application,
> run different clusters of different versions, and remember where containers
> were allocated so that on application restart it can ask for them back.
> That increases data locality and makes a big difference with HBase.
>
> It also needs a lot more work -YARN-896 is adding YARN features that help,
> but there's lots of fun to be had in Hoya including
> -leading edge work in failure handling, modelling cluster unreliability
> and reacting to it. Can we move beyond simple blacklisting to
> "greylisting", accepting unreliable boxes if we have no altenatives
>
> Then there's adding more providers, to support different application
> installations -I'm starting to write a functional test framework which need
> provider-specific workload generations
>
> Other features: AM should have a web ui that redirects to the live
> endpoints to all the app-specific UIs (e.g. HBase Master GUI), as well as
> displaying cluster state itself, for people and for management tools....
>
> To summarise: lots of fun to be had
>
> -steve
>
> --
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity to
> which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
For additional commands, e-mail: general-help@incubator.apache.org
Re: Hoya Proposal
Posted by Jean-Baptiste Onofré <jb...@nanthrax.net>.
Hi Steve,
Awesome, thanks.
Regards
JB
On 01/09/2014 10:20 AM, Steve Loughran wrote:
> thanks, that's a great offer -lovely to have some management oversight from
> others.
>
> I've put your name down
>
>
> On 8 January 2014 15:18, Jean-Baptiste Onofré <jb...@nanthrax.net> wrote:
>
>> Hi Steve,
>>
>> it looks interesting. It's a kind of "advanced" provisioning for Hadoop
>> cluster. I see some potential interesting "work together" with other
>> projects.
>>
>> I would be please to be mentor on Hoya.
>>
>> Regards
>> JB
>>
>>
>> On 01/08/2014 04:08 PM, Steve Loughran wrote:
>>
>>> I'm starting to put together the incubation proposal for Hoya: a tool to
>>> dynamically deploy applications such as HBase or Accumulo on YARN
>>>
>>> https://wiki.apache.org/incubator/HoyaProposal
>>>
>>> It does already work to the extent that it can bring up either
>>> application,
>>> run different clusters of different versions, and remember where
>>> containers
>>> were allocated so that on application restart it can ask for them back.
>>> That increases data locality and makes a big difference with HBase.
>>>
>>> It also needs a lot more work -YARN-896 is adding YARN features that help,
>>> but there's lots of fun to be had in Hoya including
>>> -leading edge work in failure handling, modelling cluster unreliability
>>> and reacting to it. Can we move beyond simple blacklisting to
>>> "greylisting", accepting unreliable boxes if we have no altenatives
>>>
>>> Then there's adding more providers, to support different application
>>> installations -I'm starting to write a functional test framework which
>>> need
>>> provider-specific workload generations
>>>
>>> Other features: AM should have a web ui that redirects to the live
>>> endpoints to all the app-specific UIs (e.g. HBase Master GUI), as well as
>>> displaying cluster state itself, for people and for management tools....
>>>
>>> To summarise: lots of fun to be had
>>>
>>> -steve
>>>
>>>
>> --
>> Jean-Baptiste Onofré
>> jbonofre@apache.org
>> http://blog.nanthrax.net
>> Talend - http://www.talend.com
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
>> For additional commands, e-mail: general-help@incubator.apache.org
>>
>>
>
--
Jean-Baptiste Onofré
jbonofre@apache.org
http://blog.nanthrax.net
Talend - http://www.talend.com
---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
For additional commands, e-mail: general-help@incubator.apache.org
Re: Hoya Proposal
Posted by Steve Loughran <st...@hortonworks.com>.
thanks, that's a great offer -lovely to have some management oversight from
others.
I've put your name down
On 8 January 2014 15:18, Jean-Baptiste Onofré <jb...@nanthrax.net> wrote:
> Hi Steve,
>
> it looks interesting. It's a kind of "advanced" provisioning for Hadoop
> cluster. I see some potential interesting "work together" with other
> projects.
>
> I would be please to be mentor on Hoya.
>
> Regards
> JB
>
>
> On 01/08/2014 04:08 PM, Steve Loughran wrote:
>
>> I'm starting to put together the incubation proposal for Hoya: a tool to
>> dynamically deploy applications such as HBase or Accumulo on YARN
>>
>> https://wiki.apache.org/incubator/HoyaProposal
>>
>> It does already work to the extent that it can bring up either
>> application,
>> run different clusters of different versions, and remember where
>> containers
>> were allocated so that on application restart it can ask for them back.
>> That increases data locality and makes a big difference with HBase.
>>
>> It also needs a lot more work -YARN-896 is adding YARN features that help,
>> but there's lots of fun to be had in Hoya including
>> -leading edge work in failure handling, modelling cluster unreliability
>> and reacting to it. Can we move beyond simple blacklisting to
>> "greylisting", accepting unreliable boxes if we have no altenatives
>>
>> Then there's adding more providers, to support different application
>> installations -I'm starting to write a functional test framework which
>> need
>> provider-specific workload generations
>>
>> Other features: AM should have a web ui that redirects to the live
>> endpoints to all the app-specific UIs (e.g. HBase Master GUI), as well as
>> displaying cluster state itself, for people and for management tools....
>>
>> To summarise: lots of fun to be had
>>
>> -steve
>>
>>
> --
> Jean-Baptiste Onofré
> jbonofre@apache.org
> http://blog.nanthrax.net
> Talend - http://www.talend.com
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> For additional commands, e-mail: general-help@incubator.apache.org
>
>
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: Hoya Proposal
Posted by Jean-Baptiste Onofré <jb...@nanthrax.net>.
Hi Steve,
it looks interesting. It's a kind of "advanced" provisioning for Hadoop
cluster. I see some potential interesting "work together" with other
projects.
I would be please to be mentor on Hoya.
Regards
JB
On 01/08/2014 04:08 PM, Steve Loughran wrote:
> I'm starting to put together the incubation proposal for Hoya: a tool to
> dynamically deploy applications such as HBase or Accumulo on YARN
>
> https://wiki.apache.org/incubator/HoyaProposal
>
> It does already work to the extent that it can bring up either application,
> run different clusters of different versions, and remember where containers
> were allocated so that on application restart it can ask for them back.
> That increases data locality and makes a big difference with HBase.
>
> It also needs a lot more work -YARN-896 is adding YARN features that help,
> but there's lots of fun to be had in Hoya including
> -leading edge work in failure handling, modelling cluster unreliability
> and reacting to it. Can we move beyond simple blacklisting to
> "greylisting", accepting unreliable boxes if we have no altenatives
>
> Then there's adding more providers, to support different application
> installations -I'm starting to write a functional test framework which need
> provider-specific workload generations
>
> Other features: AM should have a web ui that redirects to the live
> endpoints to all the app-specific UIs (e.g. HBase Master GUI), as well as
> displaying cluster state itself, for people and for management tools....
>
> To summarise: lots of fun to be had
>
> -steve
>
--
Jean-Baptiste Onofré
jbonofre@apache.org
http://blog.nanthrax.net
Talend - http://www.talend.com
---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
For additional commands, e-mail: general-help@incubator.apache.org
Re: Hoya Proposal
Posted by Henry Saputra <he...@gmail.com>.
Ah yes, I missed the additional hadoop in the path.
Close enough :)
On Thursday, January 9, 2014, Steve Loughran wrote:
> no its wrong, it should all be under org.apache.hoya.
>
> I had the hadoop prefix so that I could perhaps put it straight into the
> hadoop code as another tools module -no need for incubation. But as the
> actual providers and all tests are related to the deployment of hbase and
> accumulo, it really comes downstream of those.
>
> so a rename is needed.
>
> but yes, ASF headers everywhere
>
>
> On 8 January 2014 22:48, Henry Saputra <henry.saputra@gmail.com<javascript:;>>
> wrote:
>
> > I like how the initial code already put under "
> > org.apache.hadoop.hoya" with correct ASF header =)
> >
> > - Henry
> >
> > On Wed, Jan 8, 2014 at 7:08 AM, Steve Loughran <stevel@hortonworks.com<javascript:;>
> >
> > wrote:
> > > I'm starting to put together the incubation proposal for Hoya: a tool
> to
> > > dynamically deploy applications such as HBase or Accumulo on YARN
> > >
> > > https://wiki.apache.org/incubator/HoyaProposal
> > >
> > > It does already work to the extent that it can bring up either
> > application,
> > > run different clusters of different versions, and remember where
> > containers
> > > were allocated so that on application restart it can ask for them back.
> > > That increases data locality and makes a big difference with HBase.
> > >
> > > It also needs a lot more work -YARN-896 is adding YARN features that
> > help,
> > > but there's lots of fun to be had in Hoya including
> > > -leading edge work in failure handling, modelling cluster
> unreliability
> > > and reacting to it. Can we move beyond simple blacklisting to
> > > "greylisting", accepting unreliable boxes if we have no altenatives
> > >
> > > Then there's adding more providers, to support different application
> > > installations -I'm starting to write a functional test framework which
> > need
> > > provider-specific workload generations
> > >
> > > Other features: AM should have a web ui that redirects to the live
> > > endpoints to all the app-specific UIs (e.g. HBase Master GUI), as well
> as
> > > displaying cluster state itself, for people and for management
> tools....
> > >
> > > To summarise: lots of fun to be had
> > >
> > > -steve
> > >
> > > --
> > > CONFIDENTIALITY NOTICE
> > > NOTICE: This message is intended for the use of the individual or
> entity
> > to
> > > which it is addressed and may contain information that is confidential,
> > > privileged and exempt from disclosure under applicable law. If the
> reader
> > > of this message is not the intended recipient, you are hereby notified
> > that
> > > any printing, copying, dissemination, distribution, disclosure or
> > > forwarding of this communication is strictly prohibited. If you have
> > > received this communication in error, please contact the sender
> > immediately
> > > and delete it from your system. Thank You.
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org<javascript:;>
> > For additional commands, e-mail: general-help@incubator.apache.org<javascript:;>
> >
> >
>
> --
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity to
> which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
>
Re: Hoya Proposal
Posted by Jean-Baptiste Onofré <jb...@nanthrax.net>.
Steve did already indeed.
Regards
JB
On 01/09/2014 06:23 PM, larry mccay wrote:
> Hi Alejandro -
>
> I believe that Steve has already acknowledged that a package rename is
> required and that the project is likely not appropriate for Hadoop proper.
>
> thanks,
>
> --larry
>
>
> On Thu, Jan 9, 2014 at 11:46 AM, Alejandro Abdelnur <tu...@cloudera.com>wrote:
>
>> I may have not been clear enough, I was referring to using
>> 'org.apache.hadoop' as package prefix for a project other than hadoop.
>>
>> Thanks
>>
>>
>> On Thu, Jan 9, 2014 at 8:36 AM, Benson Margulies <bimargulies@gmail.com
>>> wrote:
>>
>>> If you can work out a plan to do this directly in Hadoop, there's no
>>> need for the incubator. You just build and and contribute it in
>>> cahoots with them, and earn commit over there as you go.
>>>
>>> On Thu, Jan 9, 2014 at 11:14 AM, Alejandro Abdelnur <tu...@cloudera.com>
>>> wrote:
>>>> Mmmh, if i recall correctly this has come up in the past with other
>>> projects and it was decided against it. Could you please check with the
>>> hadoop folks about it?
>>>>
>>>> Thx
>>>>
>>>>> On Jan 9, 2014, at 1:19 AM, Steve Loughran <st...@hortonworks.com>
>>> wrote:
>>>>>
>>>>> no its wrong, it should all be under org.apache.hoya.
>>>>>
>>>>> I had the hadoop prefix so that I could perhaps put it straight into
>> the
>>>>> hadoop code as another tools module -no need for incubation. But as
>> the
>>>>> actual providers and all tests are related to the deployment of hbase
>>> and
>>>>> accumulo, it really comes downstream of those.
>>>>>
>>>>> so a rename is needed.
>>>>>
>>>>> but yes, ASF headers everywhere
>>>>>
>>>>>
>>>>>> On 8 January 2014 22:48, Henry Saputra <he...@gmail.com>
>>> wrote:
>>>>>>
>>>>>> I like how the initial code already put under "
>>>>>> org.apache.hadoop.hoya" with correct ASF header =)
>>>>>>
>>>>>> - Henry
>>>>>>
>>>>>> On Wed, Jan 8, 2014 at 7:08 AM, Steve Loughran <
>> stevel@hortonworks.com
>>>>
>>>>>> wrote:
>>>>>>> I'm starting to put together the incubation proposal for Hoya: a
>> tool
>>> to
>>>>>>> dynamically deploy applications such as HBase or Accumulo on YARN
>>>>>>>
>>>>>>> https://wiki.apache.org/incubator/HoyaProposal
>>>>>>>
>>>>>>> It does already work to the extent that it can bring up either
>>>>>> application,
>>>>>>> run different clusters of different versions, and remember where
>>>>>> containers
>>>>>>> were allocated so that on application restart it can ask for them
>>> back.
>>>>>>> That increases data locality and makes a big difference with HBase.
>>>>>>>
>>>>>>> It also needs a lot more work -YARN-896 is adding YARN features that
>>>>>> help,
>>>>>>> but there's lots of fun to be had in Hoya including
>>>>>>> -leading edge work in failure handling, modelling cluster
>>> unreliability
>>>>>>> and reacting to it. Can we move beyond simple blacklisting to
>>>>>>> "greylisting", accepting unreliable boxes if we have no altenatives
>>>>>>>
>>>>>>> Then there's adding more providers, to support different application
>>>>>>> installations -I'm starting to write a functional test framework
>> which
>>>>>> need
>>>>>>> provider-specific workload generations
>>>>>>>
>>>>>>> Other features: AM should have a web ui that redirects to the live
>>>>>>> endpoints to all the app-specific UIs (e.g. HBase Master GUI), as
>>> well as
>>>>>>> displaying cluster state itself, for people and for management
>>> tools....
>>>>>>>
>>>>>>> To summarise: lots of fun to be had
>>>>>>>
>>>>>>> -steve
>>>>>>>
>>>>>>> --
>>>>>>> CONFIDENTIALITY NOTICE
>>>>>>> NOTICE: This message is intended for the use of the individual or
>>> entity
>>>>>> to
>>>>>>> which it is addressed and may contain information that is
>>> confidential,
>>>>>>> privileged and exempt from disclosure under applicable law. If the
>>> reader
>>>>>>> of this message is not the intended recipient, you are hereby
>> notified
>>>>>> that
>>>>>>> any printing, copying, dissemination, distribution, disclosure or
>>>>>>> forwarding of this communication is strictly prohibited. If you have
>>>>>>> received this communication in error, please contact the sender
>>>>>> immediately
>>>>>>> and delete it from your system. Thank You.
>>>>>>
>>>>>> ---------------------------------------------------------------------
>>>>>> To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
>>>>>> For additional commands, e-mail: general-help@incubator.apache.org
>>>>>
>>>>> --
>>>>> CONFIDENTIALITY NOTICE
>>>>> NOTICE: This message is intended for the use of the individual or
>>> entity to
>>>>> which it is addressed and may contain information that is
>> confidential,
>>>>> privileged and exempt from disclosure under applicable law. If the
>>> reader
>>>>> of this message is not the intended recipient, you are hereby notified
>>> that
>>>>> any printing, copying, dissemination, distribution, disclosure or
>>>>> forwarding of this communication is strictly prohibited. If you have
>>>>> received this communication in error, please contact the sender
>>> immediately
>>>>> and delete it from your system. Thank You.
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
>>>> For additional commands, e-mail: general-help@incubator.apache.org
>>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
>>> For additional commands, e-mail: general-help@incubator.apache.org
>>>
>>>
>>
>>
>> --
>> Alejandro
>>
>
--
Jean-Baptiste Onofré
jbonofre@apache.org
http://blog.nanthrax.net
Talend - http://www.talend.com
---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
For additional commands, e-mail: general-help@incubator.apache.org
Re: Hoya Proposal
Posted by larry mccay <lm...@apache.org>.
Hoya - Hadoop Oriented Yarn Applications.
:)
As Andrew indicates - stick with Hoya.
On Mon, Jan 13, 2014 at 7:18 PM, Andrew Purtell <ap...@apache.org> wrote:
> There isn't anything in the same domain named Hoya that I am aware of. One
> can get to Hoya from other avenues - it is a county and city in Germany; or
> it is the genus for several hundred tropical plants. According to
> http://en.wikipedia.org/wiki/Hoya, hoya flowers grow in a configuration
> referred to as an "umbel", which resembles what one might draw of the
> logical relationship of YARN application processes on the cluster.
>
>
> On Mon, Jan 13, 2014 at 11:07 AM, larry mccay <la...@gmail.com>
> wrote:
>
> > I agree with Marvin. Either works fine.
> > The original meaning of Hoya doesn't have to continue to be acknowledged.
> >
> >
> >
> > On Mon, Jan 13, 2014 at 12:02 PM, Marvin Humphrey <
> marvin@rectangular.com
> > >wrote:
> >
> > > On Mon, Jan 13, 2014 at 6:33 AM, Steve Loughran <
> stevel@hortonworks.com>
> > > wrote:
> > >
> > > > On that note, I know we've used the acronym "hoya", HBase on YARN,
> but
> > > it is
> > > > already a bit out of date.
> > > >
> > > > What do people think about a project name "Hyena"? It's close enough,
> > > lives
> > > > in the savannah...
> > >
> > > Based on Google searches for `hoya software` and `hyena software`, it
> > would
> > > seem that Hoya is more distinctive.
> > >
> > > I don't think either of them specifically conveys what the software
> > > does. Neither
> > > is hard to spell or pronounce. Both are nice and short.
> > >
> > > Marvin Humphrey
> > >
> > > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> > > For additional commands, e-mail: general-help@incubator.apache.org
> > >
> > >
> >
>
>
>
> --
> Best regards,
>
> - Andy
>
> Problems worthy of attack prove their worth by hitting back. - Piet Hein
> (via Tom White)
>
Re: Hoya Proposal
Posted by Andrew Purtell <ap...@apache.org>.
There isn't anything in the same domain named Hoya that I am aware of. One
can get to Hoya from other avenues - it is a county and city in Germany; or
it is the genus for several hundred tropical plants. According to
http://en.wikipedia.org/wiki/Hoya, hoya flowers grow in a configuration
referred to as an "umbel", which resembles what one might draw of the
logical relationship of YARN application processes on the cluster.
On Mon, Jan 13, 2014 at 11:07 AM, larry mccay <la...@gmail.com> wrote:
> I agree with Marvin. Either works fine.
> The original meaning of Hoya doesn't have to continue to be acknowledged.
>
>
>
> On Mon, Jan 13, 2014 at 12:02 PM, Marvin Humphrey <marvin@rectangular.com
> >wrote:
>
> > On Mon, Jan 13, 2014 at 6:33 AM, Steve Loughran <st...@hortonworks.com>
> > wrote:
> >
> > > On that note, I know we've used the acronym "hoya", HBase on YARN, but
> > it is
> > > already a bit out of date.
> > >
> > > What do people think about a project name "Hyena"? It's close enough,
> > lives
> > > in the savannah...
> >
> > Based on Google searches for `hoya software` and `hyena software`, it
> would
> > seem that Hoya is more distinctive.
> >
> > I don't think either of them specifically conveys what the software
> > does. Neither
> > is hard to spell or pronounce. Both are nice and short.
> >
> > Marvin Humphrey
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> > For additional commands, e-mail: general-help@incubator.apache.org
> >
> >
>
--
Best regards,
- Andy
Problems worthy of attack prove their worth by hitting back. - Piet Hein
(via Tom White)
Re: Hoya Proposal
Posted by larry mccay <la...@gmail.com>.
I agree with Marvin. Either works fine.
The original meaning of Hoya doesn't have to continue to be acknowledged.
On Mon, Jan 13, 2014 at 12:02 PM, Marvin Humphrey <ma...@rectangular.com>wrote:
> On Mon, Jan 13, 2014 at 6:33 AM, Steve Loughran <st...@hortonworks.com>
> wrote:
>
> > On that note, I know we've used the acronym "hoya", HBase on YARN, but
> it is
> > already a bit out of date.
> >
> > What do people think about a project name "Hyena"? It's close enough,
> lives
> > in the savannah...
>
> Based on Google searches for `hoya software` and `hyena software`, it would
> seem that Hoya is more distinctive.
>
> I don't think either of them specifically conveys what the software
> does. Neither
> is hard to spell or pronounce. Both are nice and short.
>
> Marvin Humphrey
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> For additional commands, e-mail: general-help@incubator.apache.org
>
>
Re: Hoya Proposal
Posted by Marvin Humphrey <ma...@rectangular.com>.
On Mon, Jan 13, 2014 at 6:33 AM, Steve Loughran <st...@hortonworks.com> wrote:
> On that note, I know we've used the acronym "hoya", HBase on YARN, but it is
> already a bit out of date.
>
> What do people think about a project name "Hyena"? It's close enough, lives
> in the savannah...
Based on Google searches for `hoya software` and `hyena software`, it would
seem that Hoya is more distinctive.
I don't think either of them specifically conveys what the software
does. Neither
is hard to spell or pronounce. Both are nice and short.
Marvin Humphrey
---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
For additional commands, e-mail: general-help@incubator.apache.org
Re: Hoya Proposal
Posted by Dave Fisher <da...@comcast.net>.
On Jan 13, 2014, at 6:33 AM, Steve Loughran wrote:
> On 9 January 2014 21:55, Enis Söztutar <en...@gmail.com> wrote:
>
>> Proposal looks good. Let me know if you need any additional help in
>> mentors.
>>
>> Enis
>>
>
> +1 -I'd really like input from the HBase and Accumulo teams as they are the
> first apps we're trying to work with.
>
> On that note, I know we've used the acronym "hoya", HBase on YARN, but it
> is already a bit out of date.
I thought someone lived in or went to Georgetown.
Regards,
Dave
>
> What do people think about a project name "Hyena"? It's close enough, lives
> in the savannah...
>
> -steve
>
> --
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity to
> which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
For additional commands, e-mail: general-help@incubator.apache.org
Re: Hoya Proposal
Posted by Andy Seaborne <an...@apache.org>.
On 13/01/14 21:57, Andy Seaborne wrote:
> On 13/01/14 20:25, Bertrand Delacretaz wrote:
>> On Mon, Jan 13, 2014 at 5:36 PM, Bertrand Delacretaz
>> <bd...@apache.org> wrote:
>>> On Mon, Jan 13, 2014 at 3:33 PM, Steve Loughran
>>> <st...@hortonworks.com> wrote:
>>>> ...What do people think about a project name "Hyena"? It's close
>>>> enough, lives
>>>> in the savannah...
>>>
>>> I think it's too close to http://jena.apache.org/
>>
>> to be clear, this is a strong -1 from me, having confusing project
>> names inside Apache is not ok. Hyena and Jena sound exactly the same
>> in several languages, unfortunately.
>
> The "j-e-n" in "Jena" is like "Jen-kins" or "Gen-er-al".
>
> It is derived from "Jennifer"
>
> http://en.wikipedia.org/wiki/Jennifer_%28given_name%29
>
> which is often, but not here, with "nn":
>
> http://en.wikipedia.org/wiki/Jenna
>
BTW I had nothing to do with the project name. Before my time. Other
names used within the project you can fault me for but not this one.
Andy
> Andy
>
>>
>> -Bertrand
>
---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
For additional commands, e-mail: general-help@incubator.apache.org
Re: Hoya Proposal
Posted by Steve Loughran <st...@hortonworks.com>.
not yet, but that should be the goal. It's already a hadoop-cliuster-ready
app, so all that is needed is patching the configs for the target cluster
(filesystem, ZK, security) and building up the commands to exec
On 2 February 2014 17:55, Edward J. Yoon <ed...@apache.org> wrote:
> Hi, Hoya looks nice.
>
> > 5. Help migrate more distributed applications into YARN clusters - such
> as Apache HAMA.
>
> BTW, this means that HAMA can be deployed on YARN cluster using HOYA
> without implement separate YARN application?
>
> On Wed, Jan 22, 2014 at 6:42 PM, Steve Loughran <st...@hortonworks.com>
> wrote:
> > thanks Jakob - we haven't had a vote yet though, still discussing the
> > proposal
> >
> >
> > On 22 January 2014 06:57, Jakob Homan <jg...@gmail.com> wrote:
> >
> >> HOYA looks like a useful bit of code,which if squinted at appropriately,
> >> might, sorta be able to be implemented with another useful bit of code
> >> already in the incubator. But could just as well be independent. Let's
> let
> >> HOYA in and see where the communities that develop around it and Twill
> want
> >> to go. Samza would certainly be interested in evaluating it as it
> matures.
> >>
> >> +1 (binding) on the proposal.
> >>
> >>
> >> On Tue, Jan 21, 2014 at 2:05 PM, Chris Riccomini
> >> <cr...@gmail.com>wrote:
> >>
> >> > Hey Guys,
> >> >
> >> > I'd like to +1 non-binding Hoya (or whatever you want to call it).
> It's
> >> > going to be really useful for the YARN ecosystem to have a project
> like
> >> > this out there, and it makes a lot of sense to put it in Apache.
> >> >
> >> > Hopefully we can move past the naming stuff, and get the project in.
> >> >
> >> > Cheers,
> >> > Chris
> >> >
> >> >
> >> > On Mon, Jan 20, 2014 at 12:32 PM, Tommaso Teofili <
> >> > tommaso.teofili@gmail.com
> >> > > wrote:
> >> >
> >> > > I just can speak for the Hama-to-Yarn use case (with my newbie hat
> on)
> >> > and
> >> > > it seems Hoya may be really helpful as the entry point to address
> it as
> >> > it
> >> > > provides a set of simple and useful tools.
> >> > > For the sake of the names I think Hoya would be good (while Hyena
> would
> >> > > not).
> >> > >
> >> > > Just my 0.02 cents,
> >> > > Tommaso
> >> > >
> >> > >
> >> > > 2014/1/17 Steve Loughran <st...@hortonworks.com>
> >> > >
> >> > > > On 16 January 2014 12:41, Tommaso Teofili <
> tommaso.teofili@gmail.com
> >> >
> >> > > > wrote:
> >> > > >
> >> > > > > Hi all,
> >> > > > >
> >> > > > > @Steve I'm curious to know if also Apache Hama may leverage
> Hoya to
> >> > be
> >> > > > > deployed on YARN.
> >> > > > >
> >> > > > >
> >> > > > If you are going to make a 100% commit to running in YARN -which,
> if
> >> > you
> >> > > > aren't ready yet, Hama should be able to do in some point in the
> >> > future-
> >> > > > then I'd say "go for it!" and then point you at Twill to get you
> >> > started.
> >> > > >
> >> > > > If all you are trying to do is deploy the BSP daemons in a YARN
> >> > cluster,
> >> > > > and they don't need any persistent localfs storage, then Hoya
> should
> >> > > work.
> >> > > > You already use ZK for the binding mechanism, don't you?
> >> > > >
> >> > > >
> >> > > > If yes, that'd hopefully bring some more people involved from
> Hama as
> >> > > well.
> >> > > > >
> >> > > >
> >> > > >
> >> > > > That'd be great!
> >> > > >
> >> > > > --
> >> > > > CONFIDENTIALITY NOTICE
> >> > > > NOTICE: This message is intended for the use of the individual or
> >> > entity
> >> > > to
> >> > > > which it is addressed and may contain information that is
> >> confidential,
> >> > > > privileged and exempt from disclosure under applicable law. If the
> >> > reader
> >> > > > of this message is not the intended recipient, you are hereby
> >> notified
> >> > > that
> >> > > > any printing, copying, dissemination, distribution, disclosure or
> >> > > > forwarding of this communication is strictly prohibited. If you
> have
> >> > > > received this communication in error, please contact the sender
> >> > > immediately
> >> > > > and delete it from your system. Thank You.
> >> > > >
> >> > >
> >> >
> >>
> >
> > --
> > CONFIDENTIALITY NOTICE
> > NOTICE: This message is intended for the use of the individual or entity
> to
> > which it is addressed and may contain information that is confidential,
> > privileged and exempt from disclosure under applicable law. If the reader
> > of this message is not the intended recipient, you are hereby notified
> that
> > any printing, copying, dissemination, distribution, disclosure or
> > forwarding of this communication is strictly prohibited. If you have
> > received this communication in error, please contact the sender
> immediately
> > and delete it from your system. Thank You.
>
>
>
> --
> Best Regards, Edward J. Yoon
> @eddieyoon
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> For additional commands, e-mail: general-help@incubator.apache.org
>
>
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: Hoya Proposal
Posted by "Edward J. Yoon" <ed...@apache.org>.
Hi, Hoya looks nice.
> 5. Help migrate more distributed applications into YARN clusters - such as Apache HAMA.
BTW, this means that HAMA can be deployed on YARN cluster using HOYA
without implement separate YARN application?
On Wed, Jan 22, 2014 at 6:42 PM, Steve Loughran <st...@hortonworks.com> wrote:
> thanks Jakob - we haven't had a vote yet though, still discussing the
> proposal
>
>
> On 22 January 2014 06:57, Jakob Homan <jg...@gmail.com> wrote:
>
>> HOYA looks like a useful bit of code,which if squinted at appropriately,
>> might, sorta be able to be implemented with another useful bit of code
>> already in the incubator. But could just as well be independent. Let's let
>> HOYA in and see where the communities that develop around it and Twill want
>> to go. Samza would certainly be interested in evaluating it as it matures.
>>
>> +1 (binding) on the proposal.
>>
>>
>> On Tue, Jan 21, 2014 at 2:05 PM, Chris Riccomini
>> <cr...@gmail.com>wrote:
>>
>> > Hey Guys,
>> >
>> > I'd like to +1 non-binding Hoya (or whatever you want to call it). It's
>> > going to be really useful for the YARN ecosystem to have a project like
>> > this out there, and it makes a lot of sense to put it in Apache.
>> >
>> > Hopefully we can move past the naming stuff, and get the project in.
>> >
>> > Cheers,
>> > Chris
>> >
>> >
>> > On Mon, Jan 20, 2014 at 12:32 PM, Tommaso Teofili <
>> > tommaso.teofili@gmail.com
>> > > wrote:
>> >
>> > > I just can speak for the Hama-to-Yarn use case (with my newbie hat on)
>> > and
>> > > it seems Hoya may be really helpful as the entry point to address it as
>> > it
>> > > provides a set of simple and useful tools.
>> > > For the sake of the names I think Hoya would be good (while Hyena would
>> > > not).
>> > >
>> > > Just my 0.02 cents,
>> > > Tommaso
>> > >
>> > >
>> > > 2014/1/17 Steve Loughran <st...@hortonworks.com>
>> > >
>> > > > On 16 January 2014 12:41, Tommaso Teofili <tommaso.teofili@gmail.com
>> >
>> > > > wrote:
>> > > >
>> > > > > Hi all,
>> > > > >
>> > > > > @Steve I'm curious to know if also Apache Hama may leverage Hoya to
>> > be
>> > > > > deployed on YARN.
>> > > > >
>> > > > >
>> > > > If you are going to make a 100% commit to running in YARN -which, if
>> > you
>> > > > aren't ready yet, Hama should be able to do in some point in the
>> > future-
>> > > > then I'd say "go for it!" and then point you at Twill to get you
>> > started.
>> > > >
>> > > > If all you are trying to do is deploy the BSP daemons in a YARN
>> > cluster,
>> > > > and they don't need any persistent localfs storage, then Hoya should
>> > > work.
>> > > > You already use ZK for the binding mechanism, don't you?
>> > > >
>> > > >
>> > > > If yes, that'd hopefully bring some more people involved from Hama as
>> > > well.
>> > > > >
>> > > >
>> > > >
>> > > > That'd be great!
>> > > >
>> > > > --
>> > > > CONFIDENTIALITY NOTICE
>> > > > NOTICE: This message is intended for the use of the individual or
>> > entity
>> > > to
>> > > > which it is addressed and may contain information that is
>> confidential,
>> > > > privileged and exempt from disclosure under applicable law. If the
>> > reader
>> > > > of this message is not the intended recipient, you are hereby
>> notified
>> > > that
>> > > > any printing, copying, dissemination, distribution, disclosure or
>> > > > forwarding of this communication is strictly prohibited. If you have
>> > > > received this communication in error, please contact the sender
>> > > immediately
>> > > > and delete it from your system. Thank You.
>> > > >
>> > >
>> >
>>
>
> --
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity to
> which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
--
Best Regards, Edward J. Yoon
@eddieyoon
---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
For additional commands, e-mail: general-help@incubator.apache.org
Re: Hoya Proposal
Posted by Steve Loughran <st...@hortonworks.com>.
thanks Jakob - we haven't had a vote yet though, still discussing the
proposal
On 22 January 2014 06:57, Jakob Homan <jg...@gmail.com> wrote:
> HOYA looks like a useful bit of code,which if squinted at appropriately,
> might, sorta be able to be implemented with another useful bit of code
> already in the incubator. But could just as well be independent. Let's let
> HOYA in and see where the communities that develop around it and Twill want
> to go. Samza would certainly be interested in evaluating it as it matures.
>
> +1 (binding) on the proposal.
>
>
> On Tue, Jan 21, 2014 at 2:05 PM, Chris Riccomini
> <cr...@gmail.com>wrote:
>
> > Hey Guys,
> >
> > I'd like to +1 non-binding Hoya (or whatever you want to call it). It's
> > going to be really useful for the YARN ecosystem to have a project like
> > this out there, and it makes a lot of sense to put it in Apache.
> >
> > Hopefully we can move past the naming stuff, and get the project in.
> >
> > Cheers,
> > Chris
> >
> >
> > On Mon, Jan 20, 2014 at 12:32 PM, Tommaso Teofili <
> > tommaso.teofili@gmail.com
> > > wrote:
> >
> > > I just can speak for the Hama-to-Yarn use case (with my newbie hat on)
> > and
> > > it seems Hoya may be really helpful as the entry point to address it as
> > it
> > > provides a set of simple and useful tools.
> > > For the sake of the names I think Hoya would be good (while Hyena would
> > > not).
> > >
> > > Just my 0.02 cents,
> > > Tommaso
> > >
> > >
> > > 2014/1/17 Steve Loughran <st...@hortonworks.com>
> > >
> > > > On 16 January 2014 12:41, Tommaso Teofili <tommaso.teofili@gmail.com
> >
> > > > wrote:
> > > >
> > > > > Hi all,
> > > > >
> > > > > @Steve I'm curious to know if also Apache Hama may leverage Hoya to
> > be
> > > > > deployed on YARN.
> > > > >
> > > > >
> > > > If you are going to make a 100% commit to running in YARN -which, if
> > you
> > > > aren't ready yet, Hama should be able to do in some point in the
> > future-
> > > > then I'd say "go for it!" and then point you at Twill to get you
> > started.
> > > >
> > > > If all you are trying to do is deploy the BSP daemons in a YARN
> > cluster,
> > > > and they don't need any persistent localfs storage, then Hoya should
> > > work.
> > > > You already use ZK for the binding mechanism, don't you?
> > > >
> > > >
> > > > If yes, that'd hopefully bring some more people involved from Hama as
> > > well.
> > > > >
> > > >
> > > >
> > > > That'd be great!
> > > >
> > > > --
> > > > CONFIDENTIALITY NOTICE
> > > > NOTICE: This message is intended for the use of the individual or
> > entity
> > > to
> > > > which it is addressed and may contain information that is
> confidential,
> > > > privileged and exempt from disclosure under applicable law. If the
> > reader
> > > > of this message is not the intended recipient, you are hereby
> notified
> > > that
> > > > any printing, copying, dissemination, distribution, disclosure or
> > > > forwarding of this communication is strictly prohibited. If you have
> > > > received this communication in error, please contact the sender
> > > immediately
> > > > and delete it from your system. Thank You.
> > > >
> > >
> >
>
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: Hoya Proposal
Posted by Jakob Homan <jg...@gmail.com>.
HOYA looks like a useful bit of code,which if squinted at appropriately,
might, sorta be able to be implemented with another useful bit of code
already in the incubator. But could just as well be independent. Let's let
HOYA in and see where the communities that develop around it and Twill want
to go. Samza would certainly be interested in evaluating it as it matures.
+1 (binding) on the proposal.
On Tue, Jan 21, 2014 at 2:05 PM, Chris Riccomini
<cr...@gmail.com>wrote:
> Hey Guys,
>
> I'd like to +1 non-binding Hoya (or whatever you want to call it). It's
> going to be really useful for the YARN ecosystem to have a project like
> this out there, and it makes a lot of sense to put it in Apache.
>
> Hopefully we can move past the naming stuff, and get the project in.
>
> Cheers,
> Chris
>
>
> On Mon, Jan 20, 2014 at 12:32 PM, Tommaso Teofili <
> tommaso.teofili@gmail.com
> > wrote:
>
> > I just can speak for the Hama-to-Yarn use case (with my newbie hat on)
> and
> > it seems Hoya may be really helpful as the entry point to address it as
> it
> > provides a set of simple and useful tools.
> > For the sake of the names I think Hoya would be good (while Hyena would
> > not).
> >
> > Just my 0.02 cents,
> > Tommaso
> >
> >
> > 2014/1/17 Steve Loughran <st...@hortonworks.com>
> >
> > > On 16 January 2014 12:41, Tommaso Teofili <to...@gmail.com>
> > > wrote:
> > >
> > > > Hi all,
> > > >
> > > > @Steve I'm curious to know if also Apache Hama may leverage Hoya to
> be
> > > > deployed on YARN.
> > > >
> > > >
> > > If you are going to make a 100% commit to running in YARN -which, if
> you
> > > aren't ready yet, Hama should be able to do in some point in the
> future-
> > > then I'd say "go for it!" and then point you at Twill to get you
> started.
> > >
> > > If all you are trying to do is deploy the BSP daemons in a YARN
> cluster,
> > > and they don't need any persistent localfs storage, then Hoya should
> > work.
> > > You already use ZK for the binding mechanism, don't you?
> > >
> > >
> > > If yes, that'd hopefully bring some more people involved from Hama as
> > well.
> > > >
> > >
> > >
> > > That'd be great!
> > >
> > > --
> > > CONFIDENTIALITY NOTICE
> > > NOTICE: This message is intended for the use of the individual or
> entity
> > to
> > > which it is addressed and may contain information that is confidential,
> > > privileged and exempt from disclosure under applicable law. If the
> reader
> > > of this message is not the intended recipient, you are hereby notified
> > that
> > > any printing, copying, dissemination, distribution, disclosure or
> > > forwarding of this communication is strictly prohibited. If you have
> > > received this communication in error, please contact the sender
> > immediately
> > > and delete it from your system. Thank You.
> > >
> >
>
Re: Hoya Proposal
Posted by Chris Riccomini <cr...@gmail.com>.
Hey Guys,
I'd like to +1 non-binding Hoya (or whatever you want to call it). It's
going to be really useful for the YARN ecosystem to have a project like
this out there, and it makes a lot of sense to put it in Apache.
Hopefully we can move past the naming stuff, and get the project in.
Cheers,
Chris
On Mon, Jan 20, 2014 at 12:32 PM, Tommaso Teofili <tommaso.teofili@gmail.com
> wrote:
> I just can speak for the Hama-to-Yarn use case (with my newbie hat on) and
> it seems Hoya may be really helpful as the entry point to address it as it
> provides a set of simple and useful tools.
> For the sake of the names I think Hoya would be good (while Hyena would
> not).
>
> Just my 0.02 cents,
> Tommaso
>
>
> 2014/1/17 Steve Loughran <st...@hortonworks.com>
>
> > On 16 January 2014 12:41, Tommaso Teofili <to...@gmail.com>
> > wrote:
> >
> > > Hi all,
> > >
> > > @Steve I'm curious to know if also Apache Hama may leverage Hoya to be
> > > deployed on YARN.
> > >
> > >
> > If you are going to make a 100% commit to running in YARN -which, if you
> > aren't ready yet, Hama should be able to do in some point in the future-
> > then I'd say "go for it!" and then point you at Twill to get you started.
> >
> > If all you are trying to do is deploy the BSP daemons in a YARN cluster,
> > and they don't need any persistent localfs storage, then Hoya should
> work.
> > You already use ZK for the binding mechanism, don't you?
> >
> >
> > If yes, that'd hopefully bring some more people involved from Hama as
> well.
> > >
> >
> >
> > That'd be great!
> >
> > --
> > CONFIDENTIALITY NOTICE
> > NOTICE: This message is intended for the use of the individual or entity
> to
> > which it is addressed and may contain information that is confidential,
> > privileged and exempt from disclosure under applicable law. If the reader
> > of this message is not the intended recipient, you are hereby notified
> that
> > any printing, copying, dissemination, distribution, disclosure or
> > forwarding of this communication is strictly prohibited. If you have
> > received this communication in error, please contact the sender
> immediately
> > and delete it from your system. Thank You.
> >
>
Re: Hoya Proposal
Posted by Tommaso Teofili <to...@gmail.com>.
I just can speak for the Hama-to-Yarn use case (with my newbie hat on) and
it seems Hoya may be really helpful as the entry point to address it as it
provides a set of simple and useful tools.
For the sake of the names I think Hoya would be good (while Hyena would
not).
Just my 0.02 cents,
Tommaso
2014/1/17 Steve Loughran <st...@hortonworks.com>
> On 16 January 2014 12:41, Tommaso Teofili <to...@gmail.com>
> wrote:
>
> > Hi all,
> >
> > @Steve I'm curious to know if also Apache Hama may leverage Hoya to be
> > deployed on YARN.
> >
> >
> If you are going to make a 100% commit to running in YARN -which, if you
> aren't ready yet, Hama should be able to do in some point in the future-
> then I'd say "go for it!" and then point you at Twill to get you started.
>
> If all you are trying to do is deploy the BSP daemons in a YARN cluster,
> and they don't need any persistent localfs storage, then Hoya should work.
> You already use ZK for the binding mechanism, don't you?
>
>
> If yes, that'd hopefully bring some more people involved from Hama as well.
> >
>
>
> That'd be great!
>
> --
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity to
> which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
>
Re: Hoya Proposal
Posted by Steve Loughran <st...@hortonworks.com>.
On 16 January 2014 12:41, Tommaso Teofili <to...@gmail.com> wrote:
> Hi all,
>
> @Steve I'm curious to know if also Apache Hama may leverage Hoya to be
> deployed on YARN.
>
>
If you are going to make a 100% commit to running in YARN -which, if you
aren't ready yet, Hama should be able to do in some point in the future-
then I'd say "go for it!" and then point you at Twill to get you started.
If all you are trying to do is deploy the BSP daemons in a YARN cluster,
and they don't need any persistent localfs storage, then Hoya should work.
You already use ZK for the binding mechanism, don't you?
If yes, that'd hopefully bring some more people involved from Hama as well.
>
That'd be great!
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: Hoya Proposal
Posted by Tommaso Teofili <to...@gmail.com>.
Hi all,
@Steve I'm curious to know if also Apache Hama may leverage Hoya to be
deployed on YARN.
If yes, that'd hopefully bring some more people involved from Hama as well.
My 0.02 cents,
Tommaso
2014/1/14 Enis Söztutar <en...@gmail.com>
> > +1 -I'd really like input from the HBase and Accumulo teams as they are
> the
> first apps we're trying to work with.
> Great, please add me as a mentor than.
>
> There is another animal named "Nyala" (
>
> http://thewebsiteofeverything.com/animals/mammals/Artiodactyla/Bovidae/Tragelaphus/Tragelaphus-angasii.html
> ).
> Not sure how similar / dissimilar to jena.
>
> For future reference to Hadoop-related projects:
> http://thewebsiteofeverything.com/habitats/Savanna.html
>
> Enis
>
>
> On Mon, Jan 13, 2014 at 2:21 PM, Ted Dunning <te...@gmail.com>
> wrote:
>
> > Too many people will read that as Jena, the city in Thuringia.
> >
> > https://en.wikipedia.org/wiki/Jena
> >
> > -1
> >
> >
> >
> >
> > On Mon, Jan 13, 2014 at 1:57 PM, Andy Seaborne <an...@apache.org> wrote:
> >
> > > On 13/01/14 20:25, Bertrand Delacretaz wrote:
> > >
> > >> On Mon, Jan 13, 2014 at 5:36 PM, Bertrand Delacretaz
> > >> <bd...@apache.org> wrote:
> > >>
> > >>> On Mon, Jan 13, 2014 at 3:33 PM, Steve Loughran <
> > stevel@hortonworks.com>
> > >>> wrote:
> > >>>
> > >>>> ...What do people think about a project name "Hyena"? It's close
> > >>>> enough, lives
> > >>>> in the savannah...
> > >>>>
> > >>>
> > >>> I think it's too close to http://jena.apache.org/
> > >>>
> > >>
> > >> to be clear, this is a strong -1 from me, having confusing project
> > >> names inside Apache is not ok. Hyena and Jena sound exactly the same
> > >> in several languages, unfortunately.
> > >>
> > >
> > > The "j-e-n" in "Jena" is like "Jen-kins" or "Gen-er-al".
> > >
> > > It is derived from "Jennifer"
> > >
> > > http://en.wikipedia.org/wiki/Jennifer_%28given_name%29
> > >
> > > which is often, but not here, with "nn":
> > >
> > > http://en.wikipedia.org/wiki/Jenna
> > >
> > > Andy
> > >
> > >
> > >> -Bertrand
> > >>
> > >
> > >
> > > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> > > For additional commands, e-mail: general-help@incubator.apache.org
> > >
> > >
> >
>
Re: Hoya Proposal
Posted by Enis Söztutar <en...@gmail.com>.
> +1 -I'd really like input from the HBase and Accumulo teams as they are
the
first apps we're trying to work with.
Great, please add me as a mentor than.
There is another animal named "Nyala" (
http://thewebsiteofeverything.com/animals/mammals/Artiodactyla/Bovidae/Tragelaphus/Tragelaphus-angasii.html).
Not sure how similar / dissimilar to jena.
For future reference to Hadoop-related projects:
http://thewebsiteofeverything.com/habitats/Savanna.html
Enis
On Mon, Jan 13, 2014 at 2:21 PM, Ted Dunning <te...@gmail.com> wrote:
> Too many people will read that as Jena, the city in Thuringia.
>
> https://en.wikipedia.org/wiki/Jena
>
> -1
>
>
>
>
> On Mon, Jan 13, 2014 at 1:57 PM, Andy Seaborne <an...@apache.org> wrote:
>
> > On 13/01/14 20:25, Bertrand Delacretaz wrote:
> >
> >> On Mon, Jan 13, 2014 at 5:36 PM, Bertrand Delacretaz
> >> <bd...@apache.org> wrote:
> >>
> >>> On Mon, Jan 13, 2014 at 3:33 PM, Steve Loughran <
> stevel@hortonworks.com>
> >>> wrote:
> >>>
> >>>> ...What do people think about a project name "Hyena"? It's close
> >>>> enough, lives
> >>>> in the savannah...
> >>>>
> >>>
> >>> I think it's too close to http://jena.apache.org/
> >>>
> >>
> >> to be clear, this is a strong -1 from me, having confusing project
> >> names inside Apache is not ok. Hyena and Jena sound exactly the same
> >> in several languages, unfortunately.
> >>
> >
> > The "j-e-n" in "Jena" is like "Jen-kins" or "Gen-er-al".
> >
> > It is derived from "Jennifer"
> >
> > http://en.wikipedia.org/wiki/Jennifer_%28given_name%29
> >
> > which is often, but not here, with "nn":
> >
> > http://en.wikipedia.org/wiki/Jenna
> >
> > Andy
> >
> >
> >> -Bertrand
> >>
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> > For additional commands, e-mail: general-help@incubator.apache.org
> >
> >
>
Re: Hoya Proposal
Posted by Andreas Neumann <an...@apache.org>.
Deveraj,
all these features that you list are not only useful for existing
applications but also for applications developed with Twill.
But I agree that the initial focus of the projects is different, and we
should keep talking in the future abut possible convergence.
-Andreas.
On Fri, Jan 17, 2014 at 5:37 AM, Bertrand Delacretaz <bdelacretaz@apache.org
> wrote:
> On Wed, Jan 15, 2014 at 1:57 AM, lars hofhansl <la...@apache.org> wrote:
> > ...I should we shouldn't rule out a name just because the English
> pronunciation is similar to another project.
> > Will people really confuse Apache Hyena and Apache Jena?...
>
> At this early stage it's very easy to switch names, so let's avoid
> potentially problematic names.
>
> -Bertrand
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> For additional commands, e-mail: general-help@incubator.apache.org
>
>
Re: Hoya Proposal
Posted by Bertrand Delacretaz <bd...@apache.org>.
On Wed, Jan 15, 2014 at 1:57 AM, lars hofhansl <la...@apache.org> wrote:
> ...I should we shouldn't rule out a name just because the English pronunciation is similar to another project.
> Will people really confuse Apache Hyena and Apache Jena?...
At this early stage it's very easy to switch names, so let's avoid
potentially problematic names.
-Bertrand
---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
For additional commands, e-mail: general-help@incubator.apache.org
Re: Hoya Proposal
Posted by lars hofhansl <la...@apache.org>.
I should we shouldn't rule out a name just because the English pronunciation is similar to another project.
Will people really confuse Apache Hyena and Apache Jena?
Would have to be a face-to-face conversation or a recording where either party do not really know what they are talking about.
That said, "Hoya" seems like a fine name to me as well.
I do wonder whether it is a large enough project to be standalone (rather then being folded into either Hadoop or HBase).
Just my $0.02.
-- Lars
________________________________
From: Andy Seaborne <an...@apache.org>
To: general@incubator.apache.org
Sent: Tuesday, January 14, 2014 4:03 AM
Subject: Re: Hoya Proposal
As a project name "Jena" has been in use for the project for about 13
years now. 1.0 was 9 Jan 2001. A mere 983K (zipped).
The discussion is about Hyena for Hoya.
Andy
On 13/01/14 23:38, sebb wrote:
> Jenny ?
>
> On 13 January 2014 22:21, Ted Dunning <te...@gmail.com> wrote:
>> Too many people will read that as Jena, the city in Thuringia.
>>
>> https://en.wikipedia.org/wiki/Jena
>>
>> -1
>>
>>
>>
>>
>> On Mon, Jan 13, 2014 at 1:57 PM, Andy Seaborne <an...@apache.org> wrote:
>>
>>> On 13/01/14 20:25, Bertrand Delacretaz wrote:
>>>
>>>> On Mon, Jan 13, 2014 at 5:36 PM, Bertrand Delacretaz
>>>> <bd...@apache.org> wrote:
>>>>
>>>>> On Mon, Jan 13, 2014 at 3:33 PM, Steve Loughran <st...@hortonworks.com>
>>>>> wrote:
>>>>>
>>>>>> ...What do people think about a project name "Hyena"? It's close
>>>>>> enough, lives
>>>>>> in the savannah...
>>>>>>
>>>>>
>>>>> I think it's too close to http://jena.apache.org/
>>>>>
>>>>
>>>> to be clear, this is a strong -1 from me, having confusing project
>>>> names inside Apache is not ok. Hyena and Jena sound exactly the same
>>>> in several languages, unfortunately.
>>>>
>>>
>>> The "j-e-n" in "Jena" is like "Jen-kins" or "Gen-er-al".
>>>
>>> It is derived from "Jennifer"
>>>
>>> http://en.wikipedia.org/wiki/Jennifer_%28given_name%29
>>>
>>> which is often, but not here, with "nn":
>>>
>>> http://en.wikipedia.org/wiki/Jenna
>>>
>>> Andy
>>>
>>>
>>>> -Bertrand
>>>>
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
>>> For additional commands, e-mail: general-help@incubator.apache.org
>>>
>>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> For additional commands, e-mail: general-help@incubator.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
For additional commands, e-mail: general-help@incubator.apache.org
Re: Hoya Proposal
Posted by Steve Loughran <st...@hortonworks.com>.
I don't this is something to worry about much -Hoya is good too, lots of
things outlive their acroynms. SOAP, for example
On 14 January 2014 12:03, Andy Seaborne <an...@apache.org> wrote:
> As a project name "Jena" has been in use for the project for about 13
> years now. 1.0 was 9 Jan 2001. A mere 983K (zipped).
>
> The discussion is about Hyena for Hoya.
>
> Andy
>
>
> On 13/01/14 23:38, sebb wrote:
>
>> Jenny ?
>>
>> On 13 January 2014 22:21, Ted Dunning <te...@gmail.com> wrote:
>>
>>> Too many people will read that as Jena, the city in Thuringia.
>>>
>>> https://en.wikipedia.org/wiki/Jena
>>>
>>> -1
>>>
>>>
>>>
>>>
>>> On Mon, Jan 13, 2014 at 1:57 PM, Andy Seaborne <an...@apache.org> wrote:
>>>
>>> On 13/01/14 20:25, Bertrand Delacretaz wrote:
>>>>
>>>> On Mon, Jan 13, 2014 at 5:36 PM, Bertrand Delacretaz
>>>>> <bd...@apache.org> wrote:
>>>>>
>>>>> On Mon, Jan 13, 2014 at 3:33 PM, Steve Loughran <
>>>>>> stevel@hortonworks.com>
>>>>>> wrote:
>>>>>>
>>>>>> ...What do people think about a project name "Hyena"? It's close
>>>>>>> enough, lives
>>>>>>> in the savannah...
>>>>>>>
>>>>>>>
>>>>>> I think it's too close to http://jena.apache.org/
>>>>>>
>>>>>>
>>>>> to be clear, this is a strong -1 from me, having confusing project
>>>>> names inside Apache is not ok. Hyena and Jena sound exactly the same
>>>>> in several languages, unfortunately.
>>>>>
>>>>>
>>>> The "j-e-n" in "Jena" is like "Jen-kins" or "Gen-er-al".
>>>>
>>>> It is derived from "Jennifer"
>>>>
>>>> http://en.wikipedia.org/wiki/Jennifer_%28given_name%29
>>>>
>>>> which is often, but not here, with "nn":
>>>>
>>>> http://en.wikipedia.org/wiki/Jenna
>>>>
>>>> Andy
>>>>
>>>>
>>>> -Bertrand
>>>>>
>>>>>
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
>>>> For additional commands, e-mail: general-help@incubator.apache.org
>>>>
>>>>
>>>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
>> For additional commands, e-mail: general-help@incubator.apache.org
>>
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> For additional commands, e-mail: general-help@incubator.apache.org
>
>
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: Hoya Proposal
Posted by Andy Seaborne <an...@apache.org>.
As a project name "Jena" has been in use for the project for about 13
years now. 1.0 was 9 Jan 2001. A mere 983K (zipped).
The discussion is about Hyena for Hoya.
Andy
On 13/01/14 23:38, sebb wrote:
> Jenny ?
>
> On 13 January 2014 22:21, Ted Dunning <te...@gmail.com> wrote:
>> Too many people will read that as Jena, the city in Thuringia.
>>
>> https://en.wikipedia.org/wiki/Jena
>>
>> -1
>>
>>
>>
>>
>> On Mon, Jan 13, 2014 at 1:57 PM, Andy Seaborne <an...@apache.org> wrote:
>>
>>> On 13/01/14 20:25, Bertrand Delacretaz wrote:
>>>
>>>> On Mon, Jan 13, 2014 at 5:36 PM, Bertrand Delacretaz
>>>> <bd...@apache.org> wrote:
>>>>
>>>>> On Mon, Jan 13, 2014 at 3:33 PM, Steve Loughran <st...@hortonworks.com>
>>>>> wrote:
>>>>>
>>>>>> ...What do people think about a project name "Hyena"? It's close
>>>>>> enough, lives
>>>>>> in the savannah...
>>>>>>
>>>>>
>>>>> I think it's too close to http://jena.apache.org/
>>>>>
>>>>
>>>> to be clear, this is a strong -1 from me, having confusing project
>>>> names inside Apache is not ok. Hyena and Jena sound exactly the same
>>>> in several languages, unfortunately.
>>>>
>>>
>>> The "j-e-n" in "Jena" is like "Jen-kins" or "Gen-er-al".
>>>
>>> It is derived from "Jennifer"
>>>
>>> http://en.wikipedia.org/wiki/Jennifer_%28given_name%29
>>>
>>> which is often, but not here, with "nn":
>>>
>>> http://en.wikipedia.org/wiki/Jenna
>>>
>>> Andy
>>>
>>>
>>>> -Bertrand
>>>>
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
>>> For additional commands, e-mail: general-help@incubator.apache.org
>>>
>>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> For additional commands, e-mail: general-help@incubator.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
For additional commands, e-mail: general-help@incubator.apache.org
Re: Hoya Proposal
Posted by sebb <se...@gmail.com>.
Jenny ?
On 13 January 2014 22:21, Ted Dunning <te...@gmail.com> wrote:
> Too many people will read that as Jena, the city in Thuringia.
>
> https://en.wikipedia.org/wiki/Jena
>
> -1
>
>
>
>
> On Mon, Jan 13, 2014 at 1:57 PM, Andy Seaborne <an...@apache.org> wrote:
>
>> On 13/01/14 20:25, Bertrand Delacretaz wrote:
>>
>>> On Mon, Jan 13, 2014 at 5:36 PM, Bertrand Delacretaz
>>> <bd...@apache.org> wrote:
>>>
>>>> On Mon, Jan 13, 2014 at 3:33 PM, Steve Loughran <st...@hortonworks.com>
>>>> wrote:
>>>>
>>>>> ...What do people think about a project name "Hyena"? It's close
>>>>> enough, lives
>>>>> in the savannah...
>>>>>
>>>>
>>>> I think it's too close to http://jena.apache.org/
>>>>
>>>
>>> to be clear, this is a strong -1 from me, having confusing project
>>> names inside Apache is not ok. Hyena and Jena sound exactly the same
>>> in several languages, unfortunately.
>>>
>>
>> The "j-e-n" in "Jena" is like "Jen-kins" or "Gen-er-al".
>>
>> It is derived from "Jennifer"
>>
>> http://en.wikipedia.org/wiki/Jennifer_%28given_name%29
>>
>> which is often, but not here, with "nn":
>>
>> http://en.wikipedia.org/wiki/Jenna
>>
>> Andy
>>
>>
>>> -Bertrand
>>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
>> For additional commands, e-mail: general-help@incubator.apache.org
>>
>>
---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
For additional commands, e-mail: general-help@incubator.apache.org
Re: Hoya Proposal
Posted by Ted Dunning <te...@gmail.com>.
Too many people will read that as Jena, the city in Thuringia.
https://en.wikipedia.org/wiki/Jena
-1
On Mon, Jan 13, 2014 at 1:57 PM, Andy Seaborne <an...@apache.org> wrote:
> On 13/01/14 20:25, Bertrand Delacretaz wrote:
>
>> On Mon, Jan 13, 2014 at 5:36 PM, Bertrand Delacretaz
>> <bd...@apache.org> wrote:
>>
>>> On Mon, Jan 13, 2014 at 3:33 PM, Steve Loughran <st...@hortonworks.com>
>>> wrote:
>>>
>>>> ...What do people think about a project name "Hyena"? It's close
>>>> enough, lives
>>>> in the savannah...
>>>>
>>>
>>> I think it's too close to http://jena.apache.org/
>>>
>>
>> to be clear, this is a strong -1 from me, having confusing project
>> names inside Apache is not ok. Hyena and Jena sound exactly the same
>> in several languages, unfortunately.
>>
>
> The "j-e-n" in "Jena" is like "Jen-kins" or "Gen-er-al".
>
> It is derived from "Jennifer"
>
> http://en.wikipedia.org/wiki/Jennifer_%28given_name%29
>
> which is often, but not here, with "nn":
>
> http://en.wikipedia.org/wiki/Jenna
>
> Andy
>
>
>> -Bertrand
>>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> For additional commands, e-mail: general-help@incubator.apache.org
>
>
Re: Hoya Proposal
Posted by Andy Seaborne <an...@apache.org>.
On 13/01/14 20:25, Bertrand Delacretaz wrote:
> On Mon, Jan 13, 2014 at 5:36 PM, Bertrand Delacretaz
> <bd...@apache.org> wrote:
>> On Mon, Jan 13, 2014 at 3:33 PM, Steve Loughran <st...@hortonworks.com> wrote:
>>> ...What do people think about a project name "Hyena"? It's close enough, lives
>>> in the savannah...
>>
>> I think it's too close to http://jena.apache.org/
>
> to be clear, this is a strong -1 from me, having confusing project
> names inside Apache is not ok. Hyena and Jena sound exactly the same
> in several languages, unfortunately.
The "j-e-n" in "Jena" is like "Jen-kins" or "Gen-er-al".
It is derived from "Jennifer"
http://en.wikipedia.org/wiki/Jennifer_%28given_name%29
which is often, but not here, with "nn":
http://en.wikipedia.org/wiki/Jenna
Andy
>
> -Bertrand
---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
For additional commands, e-mail: general-help@incubator.apache.org
Re: Hoya Proposal
Posted by Bernd Fondermann <be...@gmail.com>.
Which languages are they? I would expect the english pronounciation would
be relevant, where Hyena (see http://en.wiktionary.org/wiki/hyena) is quite
different from Jena.
Bernd
On Mon, Jan 13, 2014 at 9:25 PM, Bertrand Delacretaz <bdelacretaz@apache.org
> wrote:
> On Mon, Jan 13, 2014 at 5:36 PM, Bertrand Delacretaz
> <bd...@apache.org> wrote:
> > On Mon, Jan 13, 2014 at 3:33 PM, Steve Loughran <st...@hortonworks.com>
> wrote:
> >> ...What do people think about a project name "Hyena"? It's close
> enough, lives
> >> in the savannah...
> >
> > I think it's too close to http://jena.apache.org/
>
> to be clear, this is a strong -1 from me, having confusing project
> names inside Apache is not ok. Hyena and Jena sound exactly the same
> in several languages, unfortunately.
>
> -Bertrand
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> For additional commands, e-mail: general-help@incubator.apache.org
>
>
Re: Hoya Proposal
Posted by Bertrand Delacretaz <bd...@apache.org>.
On Mon, Jan 13, 2014 at 5:36 PM, Bertrand Delacretaz
<bd...@apache.org> wrote:
> On Mon, Jan 13, 2014 at 3:33 PM, Steve Loughran <st...@hortonworks.com> wrote:
>> ...What do people think about a project name "Hyena"? It's close enough, lives
>> in the savannah...
>
> I think it's too close to http://jena.apache.org/
to be clear, this is a strong -1 from me, having confusing project
names inside Apache is not ok. Hyena and Jena sound exactly the same
in several languages, unfortunately.
-Bertrand
---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
For additional commands, e-mail: general-help@incubator.apache.org
Re: Hoya Proposal
Posted by Bertrand Delacretaz <bd...@apache.org>.
On Mon, Jan 13, 2014 at 3:33 PM, Steve Loughran <st...@hortonworks.com> wrote:
> ...What do people think about a project name "Hyena"? It's close enough, lives
> in the savannah...
I think it's too close to http://jena.apache.org/
-Bertrand
---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
For additional commands, e-mail: general-help@incubator.apache.org
Re: Hoya Proposal
Posted by Bernd Fondermann <be...@gmail.com>.
+1
Bernd
On Mon, Jan 13, 2014 at 4:44 PM, Jean-Baptiste Onofré <jb...@nanthrax.net>wrote:
> +1 for Hyena.
>
> Regards
> JB
>
>
> On 01/13/2014 03:33 PM, Steve Loughran wrote:
>
>> On 9 January 2014 21:55, Enis Söztutar <en...@gmail.com> wrote:
>>
>> Proposal looks good. Let me know if you need any additional help in
>>> mentors.
>>>
>>> Enis
>>>
>>>
>> +1 -I'd really like input from the HBase and Accumulo teams as they are
>> the
>> first apps we're trying to work with.
>>
>> On that note, I know we've used the acronym "hoya", HBase on YARN, but it
>> is already a bit out of date.
>>
>> What do people think about a project name "Hyena"? It's close enough,
>> lives
>> in the savannah...
>>
>> -steve
>>
>>
> --
> Jean-Baptiste Onofré
> jbonofre@apache.org
> http://blog.nanthrax.net
> Talend - http://www.talend.com
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> For additional commands, e-mail: general-help@incubator.apache.org
>
>
Re: Hoya Proposal
Posted by Jean-Baptiste Onofré <jb...@nanthrax.net>.
+1 for Hyena.
Regards
JB
On 01/13/2014 03:33 PM, Steve Loughran wrote:
> On 9 January 2014 21:55, Enis Söztutar <en...@gmail.com> wrote:
>
>> Proposal looks good. Let me know if you need any additional help in
>> mentors.
>>
>> Enis
>>
>
> +1 -I'd really like input from the HBase and Accumulo teams as they are the
> first apps we're trying to work with.
>
> On that note, I know we've used the acronym "hoya", HBase on YARN, but it
> is already a bit out of date.
>
> What do people think about a project name "Hyena"? It's close enough, lives
> in the savannah...
>
> -steve
>
--
Jean-Baptiste Onofré
jbonofre@apache.org
http://blog.nanthrax.net
Talend - http://www.talend.com
---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
For additional commands, e-mail: general-help@incubator.apache.org
Re: Hoya Proposal
Posted by Steve Loughran <st...@hortonworks.com>.
On 9 January 2014 21:55, Enis Söztutar <en...@gmail.com> wrote:
> Proposal looks good. Let me know if you need any additional help in
> mentors.
>
> Enis
>
+1 -I'd really like input from the HBase and Accumulo teams as they are the
first apps we're trying to work with.
On that note, I know we've used the acronym "hoya", HBase on YARN, but it
is already a bit out of date.
What do people think about a project name "Hyena"? It's close enough, lives
in the savannah...
-steve
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: Hoya Proposal
Posted by Enis Söztutar <en...@gmail.com>.
Proposal looks good. Let me know if you need any additional help in mentors.
Enis
On Thu, Jan 9, 2014 at 9:29 AM, Alejandro Abdelnur <tu...@cloudera.com>wrote:
> argh, gmail trying to be extra smart collapsed his comment on that and i
> missed. sorry
>
> thx
>
> Alejandro
> (phone typing)
>
> > On Jan 9, 2014, at 9:23, larry mccay <lm...@apache.org> wrote:
> >
> > Hi Alejandro -
> >
> > I believe that Steve has already acknowledged that a package rename is
> > required and that the project is likely not appropriate for Hadoop
> proper.
> >
> > thanks,
> >
> > --larry
> >
> >
> > On Thu, Jan 9, 2014 at 11:46 AM, Alejandro Abdelnur <tucu@cloudera.com
> >wrote:
> >
> >> I may have not been clear enough, I was referring to using
> >> 'org.apache.hadoop' as package prefix for a project other than hadoop.
> >>
> >> Thanks
> >>
> >>
> >> On Thu, Jan 9, 2014 at 8:36 AM, Benson Margulies <bimargulies@gmail.com
> >>> wrote:
> >>
> >>> If you can work out a plan to do this directly in Hadoop, there's no
> >>> need for the incubator. You just build and and contribute it in
> >>> cahoots with them, and earn commit over there as you go.
> >>>
> >>> On Thu, Jan 9, 2014 at 11:14 AM, Alejandro Abdelnur <tucu@cloudera.com
> >
> >>> wrote:
> >>>> Mmmh, if i recall correctly this has come up in the past with other
> >>> projects and it was decided against it. Could you please check with the
> >>> hadoop folks about it?
> >>>>
> >>>> Thx
> >>>>
> >>>>> On Jan 9, 2014, at 1:19 AM, Steve Loughran <st...@hortonworks.com>
> >>> wrote:
> >>>>>
> >>>>> no its wrong, it should all be under org.apache.hoya.
> >>>>>
> >>>>> I had the hadoop prefix so that I could perhaps put it straight into
> >> the
> >>>>> hadoop code as another tools module -no need for incubation. But as
> >> the
> >>>>> actual providers and all tests are related to the deployment of hbase
> >>> and
> >>>>> accumulo, it really comes downstream of those.
> >>>>>
> >>>>> so a rename is needed.
> >>>>>
> >>>>> but yes, ASF headers everywhere
> >>>>>
> >>>>>
> >>>>>> On 8 January 2014 22:48, Henry Saputra <he...@gmail.com>
> >>> wrote:
> >>>>>>
> >>>>>> I like how the initial code already put under "
> >>>>>> org.apache.hadoop.hoya" with correct ASF header =)
> >>>>>>
> >>>>>> - Henry
> >>>>>>
> >>>>>> On Wed, Jan 8, 2014 at 7:08 AM, Steve Loughran <
> >> stevel@hortonworks.com
> >>>>
> >>>>>> wrote:
> >>>>>>> I'm starting to put together the incubation proposal for Hoya: a
> >> tool
> >>> to
> >>>>>>> dynamically deploy applications such as HBase or Accumulo on YARN
> >>>>>>>
> >>>>>>> https://wiki.apache.org/incubator/HoyaProposal
> >>>>>>>
> >>>>>>> It does already work to the extent that it can bring up either
> >>>>>> application,
> >>>>>>> run different clusters of different versions, and remember where
> >>>>>> containers
> >>>>>>> were allocated so that on application restart it can ask for them
> >>> back.
> >>>>>>> That increases data locality and makes a big difference with HBase.
> >>>>>>>
> >>>>>>> It also needs a lot more work -YARN-896 is adding YARN features
> that
> >>>>>> help,
> >>>>>>> but there's lots of fun to be had in Hoya including
> >>>>>>> -leading edge work in failure handling, modelling cluster
> >>> unreliability
> >>>>>>> and reacting to it. Can we move beyond simple blacklisting to
> >>>>>>> "greylisting", accepting unreliable boxes if we have no altenatives
> >>>>>>>
> >>>>>>> Then there's adding more providers, to support different
> application
> >>>>>>> installations -I'm starting to write a functional test framework
> >> which
> >>>>>> need
> >>>>>>> provider-specific workload generations
> >>>>>>>
> >>>>>>> Other features: AM should have a web ui that redirects to the live
> >>>>>>> endpoints to all the app-specific UIs (e.g. HBase Master GUI), as
> >>> well as
> >>>>>>> displaying cluster state itself, for people and for management
> >>> tools....
> >>>>>>>
> >>>>>>> To summarise: lots of fun to be had
> >>>>>>>
> >>>>>>> -steve
> >>>>>>>
> >>>>>>> --
> >>>>>>> CONFIDENTIALITY NOTICE
> >>>>>>> NOTICE: This message is intended for the use of the individual or
> >>> entity
> >>>>>> to
> >>>>>>> which it is addressed and may contain information that is
> >>> confidential,
> >>>>>>> privileged and exempt from disclosure under applicable law. If the
> >>> reader
> >>>>>>> of this message is not the intended recipient, you are hereby
> >> notified
> >>>>>> that
> >>>>>>> any printing, copying, dissemination, distribution, disclosure or
> >>>>>>> forwarding of this communication is strictly prohibited. If you
> have
> >>>>>>> received this communication in error, please contact the sender
> >>>>>> immediately
> >>>>>>> and delete it from your system. Thank You.
> >>>>>>
> >>>>>>
> ---------------------------------------------------------------------
> >>>>>> To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> >>>>>> For additional commands, e-mail: general-help@incubator.apache.org
> >>>>>
> >>>>> --
> >>>>> CONFIDENTIALITY NOTICE
> >>>>> NOTICE: This message is intended for the use of the individual or
> >>> entity to
> >>>>> which it is addressed and may contain information that is
> >> confidential,
> >>>>> privileged and exempt from disclosure under applicable law. If the
> >>> reader
> >>>>> of this message is not the intended recipient, you are hereby
> notified
> >>> that
> >>>>> any printing, copying, dissemination, distribution, disclosure or
> >>>>> forwarding of this communication is strictly prohibited. If you have
> >>>>> received this communication in error, please contact the sender
> >>> immediately
> >>>>> and delete it from your system. Thank You.
> >>>>
> >>>> ---------------------------------------------------------------------
> >>>> To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> >>>> For additional commands, e-mail: general-help@incubator.apache.org
> >>>
> >>> ---------------------------------------------------------------------
> >>> To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> >>> For additional commands, e-mail: general-help@incubator.apache.org
> >>
> >>
> >> --
> >> Alejandro
> >>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> For additional commands, e-mail: general-help@incubator.apache.org
>
>
Re: Hoya Proposal
Posted by Steve Loughran <st...@hortonworks.com>.
On 10 January 2014 09:54, Jean-Baptiste Onofré <jb...@nanthrax.net> wrote:
> Hi Steve,
>
> can I help during the week end ?
>
> Regards
> JB
I'm just running the post-rename tests now; functional test runner came
first...
-steve
>
>
> On 01/10/2014 10:31 AM, Steve Loughran wrote:
>
>> I'm going to do the rename over the w/end, I just need to get a functional
>> tests branch in, because merge-over-renames is one of those things you
>> don't want to have to do.
>>
>>
>> On 9 January 2014 17:29, Alejandro Abdelnur <tu...@cloudera.com> wrote:
>>
>> argh, gmail trying to be extra smart collapsed his comment on that and i
>>> missed. sorry
>>>
>>> thx
>>>
>>> Alejandro
>>> (phone typing)
>>>
>>> On Jan 9, 2014, at 9:23, larry mccay <lm...@apache.org> wrote:
>>>>
>>>> Hi Alejandro -
>>>>
>>>> I believe that Steve has already acknowledged that a package rename is
>>>> required and that the project is likely not appropriate for Hadoop
>>>>
>>> proper.
>>>
>>>>
>>>> thanks,
>>>>
>>>> --larry
>>>>
>>>>
>>>> On Thu, Jan 9, 2014 at 11:46 AM, Alejandro Abdelnur <tucu@cloudera.com
>>>> wrote:
>>>>
>>>> I may have not been clear enough, I was referring to using
>>>>> 'org.apache.hadoop' as package prefix for a project other than hadoop.
>>>>>
>>>>> Thanks
>>>>>
>>>>>
>>>>> On Thu, Jan 9, 2014 at 8:36 AM, Benson Margulies <
>>>>> bimargulies@gmail.com
>>>>>
>>>>>> wrote:
>>>>>>
>>>>>
>>>>> If you can work out a plan to do this directly in Hadoop, there's no
>>>>>> need for the incubator. You just build and and contribute it in
>>>>>> cahoots with them, and earn commit over there as you go.
>>>>>>
>>>>>> On Thu, Jan 9, 2014 at 11:14 AM, Alejandro Abdelnur <
>>>>>> tucu@cloudera.com
>>>>>>
>>>>>
>>>> wrote:
>>>>>>
>>>>>>> Mmmh, if i recall correctly this has come up in the past with other
>>>>>>>
>>>>>> projects and it was decided against it. Could you please check with
>>>>>> the
>>>>>> hadoop folks about it?
>>>>>>
>>>>>>>
>>>>>>> Thx
>>>>>>>
>>>>>>> On Jan 9, 2014, at 1:19 AM, Steve Loughran <st...@hortonworks.com>
>>>>>>>>
>>>>>>> wrote:
>>>>>>
>>>>>>>
>>>>>>>> no its wrong, it should all be under org.apache.hoya.
>>>>>>>>
>>>>>>>> I had the hadoop prefix so that I could perhaps put it straight into
>>>>>>>>
>>>>>>> the
>>>>>
>>>>>> hadoop code as another tools module -no need for incubation. But as
>>>>>>>>
>>>>>>> the
>>>>>
>>>>>> actual providers and all tests are related to the deployment of hbase
>>>>>>>>
>>>>>>> and
>>>>>>
>>>>>>> accumulo, it really comes downstream of those.
>>>>>>>>
>>>>>>>> so a rename is needed.
>>>>>>>>
>>>>>>>> but yes, ASF headers everywhere
>>>>>>>>
>>>>>>>>
>>>>>>>> On 8 January 2014 22:48, Henry Saputra <he...@gmail.com>
>>>>>>>>>
>>>>>>>> wrote:
>>>>>>
>>>>>>>
>>>>>>>>> I like how the initial code already put under "
>>>>>>>>> org.apache.hadoop.hoya" with correct ASF header =)
>>>>>>>>>
>>>>>>>>> - Henry
>>>>>>>>>
>>>>>>>>> On Wed, Jan 8, 2014 at 7:08 AM, Steve Loughran <
>>>>>>>>>
>>>>>>>> stevel@hortonworks.com
>>>>>
>>>>>>
>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> I'm starting to put together the incubation proposal for Hoya: a
>>>>>>>>>>
>>>>>>>>> tool
>>>>>
>>>>>> to
>>>>>>
>>>>>>> dynamically deploy applications such as HBase or Accumulo on YARN
>>>>>>>>>>
>>>>>>>>>> https://wiki.apache.org/incubator/HoyaProposal
>>>>>>>>>>
>>>>>>>>>> It does already work to the extent that it can bring up either
>>>>>>>>>>
>>>>>>>>> application,
>>>>>>>>>
>>>>>>>>>> run different clusters of different versions, and remember where
>>>>>>>>>>
>>>>>>>>> containers
>>>>>>>>>
>>>>>>>>>> were allocated so that on application restart it can ask for them
>>>>>>>>>>
>>>>>>>>> back.
>>>>>>
>>>>>>> That increases data locality and makes a big difference with HBase.
>>>>>>>>>>
>>>>>>>>>> It also needs a lot more work -YARN-896 is adding YARN features
>>>>>>>>>>
>>>>>>>>> that
>>>
>>>> help,
>>>>>>>>>
>>>>>>>>>> but there's lots of fun to be had in Hoya including
>>>>>>>>>> -leading edge work in failure handling, modelling cluster
>>>>>>>>>>
>>>>>>>>> unreliability
>>>>>>
>>>>>>> and reacting to it. Can we move beyond simple blacklisting to
>>>>>>>>>> "greylisting", accepting unreliable boxes if we have no
>>>>>>>>>> altenatives
>>>>>>>>>>
>>>>>>>>>> Then there's adding more providers, to support different
>>>>>>>>>>
>>>>>>>>> application
>>>
>>>> installations -I'm starting to write a functional test framework
>>>>>>>>>>
>>>>>>>>> which
>>>>>
>>>>>> need
>>>>>>>>>
>>>>>>>>>> provider-specific workload generations
>>>>>>>>>>
>>>>>>>>>> Other features: AM should have a web ui that redirects to the live
>>>>>>>>>> endpoints to all the app-specific UIs (e.g. HBase Master GUI), as
>>>>>>>>>>
>>>>>>>>> well as
>>>>>>
>>>>>>> displaying cluster state itself, for people and for management
>>>>>>>>>>
>>>>>>>>> tools....
>>>>>>
>>>>>>>
>>>>>>>>>> To summarise: lots of fun to be had
>>>>>>>>>>
>>>>>>>>>> -steve
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> CONFIDENTIALITY NOTICE
>>>>>>>>>> NOTICE: This message is intended for the use of the individual or
>>>>>>>>>>
>>>>>>>>> entity
>>>>>>
>>>>>>> to
>>>>>>>>>
>>>>>>>>>> which it is addressed and may contain information that is
>>>>>>>>>>
>>>>>>>>> confidential,
>>>>>>
>>>>>>> privileged and exempt from disclosure under applicable law. If the
>>>>>>>>>>
>>>>>>>>> reader
>>>>>>
>>>>>>> of this message is not the intended recipient, you are hereby
>>>>>>>>>>
>>>>>>>>> notified
>>>>>
>>>>>> that
>>>>>>>>>
>>>>>>>>>> any printing, copying, dissemination, distribution, disclosure or
>>>>>>>>>> forwarding of this communication is strictly prohibited. If you
>>>>>>>>>>
>>>>>>>>> have
>>>
>>>> received this communication in error, please contact the sender
>>>>>>>>>>
>>>>>>>>> immediately
>>>>>>>>>
>>>>>>>>>> and delete it from your system. Thank You.
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> ------------------------------------------------------------
>>> ---------
>>>
>>>> To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
>>>>>>>>> For additional commands, e-mail: general-help@incubator.apache.org
>>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> CONFIDENTIALITY NOTICE
>>>>>>>> NOTICE: This message is intended for the use of the individual or
>>>>>>>>
>>>>>>> entity to
>>>>>>
>>>>>>> which it is addressed and may contain information that is
>>>>>>>>
>>>>>>> confidential,
>>>>>
>>>>>> privileged and exempt from disclosure under applicable law. If the
>>>>>>>>
>>>>>>> reader
>>>>>>
>>>>>>> of this message is not the intended recipient, you are hereby
>>>>>>>>
>>>>>>> notified
>>>
>>>> that
>>>>>>
>>>>>>> any printing, copying, dissemination, distribution, disclosure or
>>>>>>>> forwarding of this communication is strictly prohibited. If you have
>>>>>>>> received this communication in error, please contact the sender
>>>>>>>>
>>>>>>> immediately
>>>>>>
>>>>>>> and delete it from your system. Thank You.
>>>>>>>>
>>>>>>>
>>>>>>> ------------------------------------------------------------
>>>>>>> ---------
>>>>>>> To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
>>>>>>> For additional commands, e-mail: general-help@incubator.apache.org
>>>>>>>
>>>>>>
>>>>>> ---------------------------------------------------------------------
>>>>>> To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
>>>>>> For additional commands, e-mail: general-help@incubator.apache.org
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Alejandro
>>>>>
>>>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
>>> For additional commands, e-mail: general-help@incubator.apache.org
>>>
>>>
>>>
>>
> --
> Jean-Baptiste Onofré
> jbonofre@apache.org
> http://blog.nanthrax.net
> Talend - http://www.talend.com
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> For additional commands, e-mail: general-help@incubator.apache.org
>
>
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: Hoya Proposal
Posted by Jean-Baptiste Onofré <jb...@nanthrax.net>.
Hi Steve,
can I help during the week end ?
Regards
JB
On 01/10/2014 10:31 AM, Steve Loughran wrote:
> I'm going to do the rename over the w/end, I just need to get a functional
> tests branch in, because merge-over-renames is one of those things you
> don't want to have to do.
>
>
> On 9 January 2014 17:29, Alejandro Abdelnur <tu...@cloudera.com> wrote:
>
>> argh, gmail trying to be extra smart collapsed his comment on that and i
>> missed. sorry
>>
>> thx
>>
>> Alejandro
>> (phone typing)
>>
>>> On Jan 9, 2014, at 9:23, larry mccay <lm...@apache.org> wrote:
>>>
>>> Hi Alejandro -
>>>
>>> I believe that Steve has already acknowledged that a package rename is
>>> required and that the project is likely not appropriate for Hadoop
>> proper.
>>>
>>> thanks,
>>>
>>> --larry
>>>
>>>
>>> On Thu, Jan 9, 2014 at 11:46 AM, Alejandro Abdelnur <tucu@cloudera.com
>>> wrote:
>>>
>>>> I may have not been clear enough, I was referring to using
>>>> 'org.apache.hadoop' as package prefix for a project other than hadoop.
>>>>
>>>> Thanks
>>>>
>>>>
>>>> On Thu, Jan 9, 2014 at 8:36 AM, Benson Margulies <bimargulies@gmail.com
>>>>> wrote:
>>>>
>>>>> If you can work out a plan to do this directly in Hadoop, there's no
>>>>> need for the incubator. You just build and and contribute it in
>>>>> cahoots with them, and earn commit over there as you go.
>>>>>
>>>>> On Thu, Jan 9, 2014 at 11:14 AM, Alejandro Abdelnur <tucu@cloudera.com
>>>
>>>>> wrote:
>>>>>> Mmmh, if i recall correctly this has come up in the past with other
>>>>> projects and it was decided against it. Could you please check with the
>>>>> hadoop folks about it?
>>>>>>
>>>>>> Thx
>>>>>>
>>>>>>> On Jan 9, 2014, at 1:19 AM, Steve Loughran <st...@hortonworks.com>
>>>>> wrote:
>>>>>>>
>>>>>>> no its wrong, it should all be under org.apache.hoya.
>>>>>>>
>>>>>>> I had the hadoop prefix so that I could perhaps put it straight into
>>>> the
>>>>>>> hadoop code as another tools module -no need for incubation. But as
>>>> the
>>>>>>> actual providers and all tests are related to the deployment of hbase
>>>>> and
>>>>>>> accumulo, it really comes downstream of those.
>>>>>>>
>>>>>>> so a rename is needed.
>>>>>>>
>>>>>>> but yes, ASF headers everywhere
>>>>>>>
>>>>>>>
>>>>>>>> On 8 January 2014 22:48, Henry Saputra <he...@gmail.com>
>>>>> wrote:
>>>>>>>>
>>>>>>>> I like how the initial code already put under "
>>>>>>>> org.apache.hadoop.hoya" with correct ASF header =)
>>>>>>>>
>>>>>>>> - Henry
>>>>>>>>
>>>>>>>> On Wed, Jan 8, 2014 at 7:08 AM, Steve Loughran <
>>>> stevel@hortonworks.com
>>>>>>
>>>>>>>> wrote:
>>>>>>>>> I'm starting to put together the incubation proposal for Hoya: a
>>>> tool
>>>>> to
>>>>>>>>> dynamically deploy applications such as HBase or Accumulo on YARN
>>>>>>>>>
>>>>>>>>> https://wiki.apache.org/incubator/HoyaProposal
>>>>>>>>>
>>>>>>>>> It does already work to the extent that it can bring up either
>>>>>>>> application,
>>>>>>>>> run different clusters of different versions, and remember where
>>>>>>>> containers
>>>>>>>>> were allocated so that on application restart it can ask for them
>>>>> back.
>>>>>>>>> That increases data locality and makes a big difference with HBase.
>>>>>>>>>
>>>>>>>>> It also needs a lot more work -YARN-896 is adding YARN features
>> that
>>>>>>>> help,
>>>>>>>>> but there's lots of fun to be had in Hoya including
>>>>>>>>> -leading edge work in failure handling, modelling cluster
>>>>> unreliability
>>>>>>>>> and reacting to it. Can we move beyond simple blacklisting to
>>>>>>>>> "greylisting", accepting unreliable boxes if we have no altenatives
>>>>>>>>>
>>>>>>>>> Then there's adding more providers, to support different
>> application
>>>>>>>>> installations -I'm starting to write a functional test framework
>>>> which
>>>>>>>> need
>>>>>>>>> provider-specific workload generations
>>>>>>>>>
>>>>>>>>> Other features: AM should have a web ui that redirects to the live
>>>>>>>>> endpoints to all the app-specific UIs (e.g. HBase Master GUI), as
>>>>> well as
>>>>>>>>> displaying cluster state itself, for people and for management
>>>>> tools....
>>>>>>>>>
>>>>>>>>> To summarise: lots of fun to be had
>>>>>>>>>
>>>>>>>>> -steve
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> CONFIDENTIALITY NOTICE
>>>>>>>>> NOTICE: This message is intended for the use of the individual or
>>>>> entity
>>>>>>>> to
>>>>>>>>> which it is addressed and may contain information that is
>>>>> confidential,
>>>>>>>>> privileged and exempt from disclosure under applicable law. If the
>>>>> reader
>>>>>>>>> of this message is not the intended recipient, you are hereby
>>>> notified
>>>>>>>> that
>>>>>>>>> any printing, copying, dissemination, distribution, disclosure or
>>>>>>>>> forwarding of this communication is strictly prohibited. If you
>> have
>>>>>>>>> received this communication in error, please contact the sender
>>>>>>>> immediately
>>>>>>>>> and delete it from your system. Thank You.
>>>>>>>>
>>>>>>>>
>> ---------------------------------------------------------------------
>>>>>>>> To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
>>>>>>>> For additional commands, e-mail: general-help@incubator.apache.org
>>>>>>>
>>>>>>> --
>>>>>>> CONFIDENTIALITY NOTICE
>>>>>>> NOTICE: This message is intended for the use of the individual or
>>>>> entity to
>>>>>>> which it is addressed and may contain information that is
>>>> confidential,
>>>>>>> privileged and exempt from disclosure under applicable law. If the
>>>>> reader
>>>>>>> of this message is not the intended recipient, you are hereby
>> notified
>>>>> that
>>>>>>> any printing, copying, dissemination, distribution, disclosure or
>>>>>>> forwarding of this communication is strictly prohibited. If you have
>>>>>>> received this communication in error, please contact the sender
>>>>> immediately
>>>>>>> and delete it from your system. Thank You.
>>>>>>
>>>>>> ---------------------------------------------------------------------
>>>>>> To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
>>>>>> For additional commands, e-mail: general-help@incubator.apache.org
>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
>>>>> For additional commands, e-mail: general-help@incubator.apache.org
>>>>
>>>>
>>>> --
>>>> Alejandro
>>>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
>> For additional commands, e-mail: general-help@incubator.apache.org
>>
>>
>
--
Jean-Baptiste Onofré
jbonofre@apache.org
http://blog.nanthrax.net
Talend - http://www.talend.com
---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
For additional commands, e-mail: general-help@incubator.apache.org
Re: Hoya Proposal
Posted by Steve Loughran <st...@hortonworks.com>.
I'm going to do the rename over the w/end, I just need to get a functional
tests branch in, because merge-over-renames is one of those things you
don't want to have to do.
On 9 January 2014 17:29, Alejandro Abdelnur <tu...@cloudera.com> wrote:
> argh, gmail trying to be extra smart collapsed his comment on that and i
> missed. sorry
>
> thx
>
> Alejandro
> (phone typing)
>
> > On Jan 9, 2014, at 9:23, larry mccay <lm...@apache.org> wrote:
> >
> > Hi Alejandro -
> >
> > I believe that Steve has already acknowledged that a package rename is
> > required and that the project is likely not appropriate for Hadoop
> proper.
> >
> > thanks,
> >
> > --larry
> >
> >
> > On Thu, Jan 9, 2014 at 11:46 AM, Alejandro Abdelnur <tucu@cloudera.com
> >wrote:
> >
> >> I may have not been clear enough, I was referring to using
> >> 'org.apache.hadoop' as package prefix for a project other than hadoop.
> >>
> >> Thanks
> >>
> >>
> >> On Thu, Jan 9, 2014 at 8:36 AM, Benson Margulies <bimargulies@gmail.com
> >>> wrote:
> >>
> >>> If you can work out a plan to do this directly in Hadoop, there's no
> >>> need for the incubator. You just build and and contribute it in
> >>> cahoots with them, and earn commit over there as you go.
> >>>
> >>> On Thu, Jan 9, 2014 at 11:14 AM, Alejandro Abdelnur <tucu@cloudera.com
> >
> >>> wrote:
> >>>> Mmmh, if i recall correctly this has come up in the past with other
> >>> projects and it was decided against it. Could you please check with the
> >>> hadoop folks about it?
> >>>>
> >>>> Thx
> >>>>
> >>>>> On Jan 9, 2014, at 1:19 AM, Steve Loughran <st...@hortonworks.com>
> >>> wrote:
> >>>>>
> >>>>> no its wrong, it should all be under org.apache.hoya.
> >>>>>
> >>>>> I had the hadoop prefix so that I could perhaps put it straight into
> >> the
> >>>>> hadoop code as another tools module -no need for incubation. But as
> >> the
> >>>>> actual providers and all tests are related to the deployment of hbase
> >>> and
> >>>>> accumulo, it really comes downstream of those.
> >>>>>
> >>>>> so a rename is needed.
> >>>>>
> >>>>> but yes, ASF headers everywhere
> >>>>>
> >>>>>
> >>>>>> On 8 January 2014 22:48, Henry Saputra <he...@gmail.com>
> >>> wrote:
> >>>>>>
> >>>>>> I like how the initial code already put under "
> >>>>>> org.apache.hadoop.hoya" with correct ASF header =)
> >>>>>>
> >>>>>> - Henry
> >>>>>>
> >>>>>> On Wed, Jan 8, 2014 at 7:08 AM, Steve Loughran <
> >> stevel@hortonworks.com
> >>>>
> >>>>>> wrote:
> >>>>>>> I'm starting to put together the incubation proposal for Hoya: a
> >> tool
> >>> to
> >>>>>>> dynamically deploy applications such as HBase or Accumulo on YARN
> >>>>>>>
> >>>>>>> https://wiki.apache.org/incubator/HoyaProposal
> >>>>>>>
> >>>>>>> It does already work to the extent that it can bring up either
> >>>>>> application,
> >>>>>>> run different clusters of different versions, and remember where
> >>>>>> containers
> >>>>>>> were allocated so that on application restart it can ask for them
> >>> back.
> >>>>>>> That increases data locality and makes a big difference with HBase.
> >>>>>>>
> >>>>>>> It also needs a lot more work -YARN-896 is adding YARN features
> that
> >>>>>> help,
> >>>>>>> but there's lots of fun to be had in Hoya including
> >>>>>>> -leading edge work in failure handling, modelling cluster
> >>> unreliability
> >>>>>>> and reacting to it. Can we move beyond simple blacklisting to
> >>>>>>> "greylisting", accepting unreliable boxes if we have no altenatives
> >>>>>>>
> >>>>>>> Then there's adding more providers, to support different
> application
> >>>>>>> installations -I'm starting to write a functional test framework
> >> which
> >>>>>> need
> >>>>>>> provider-specific workload generations
> >>>>>>>
> >>>>>>> Other features: AM should have a web ui that redirects to the live
> >>>>>>> endpoints to all the app-specific UIs (e.g. HBase Master GUI), as
> >>> well as
> >>>>>>> displaying cluster state itself, for people and for management
> >>> tools....
> >>>>>>>
> >>>>>>> To summarise: lots of fun to be had
> >>>>>>>
> >>>>>>> -steve
> >>>>>>>
> >>>>>>> --
> >>>>>>> CONFIDENTIALITY NOTICE
> >>>>>>> NOTICE: This message is intended for the use of the individual or
> >>> entity
> >>>>>> to
> >>>>>>> which it is addressed and may contain information that is
> >>> confidential,
> >>>>>>> privileged and exempt from disclosure under applicable law. If the
> >>> reader
> >>>>>>> of this message is not the intended recipient, you are hereby
> >> notified
> >>>>>> that
> >>>>>>> any printing, copying, dissemination, distribution, disclosure or
> >>>>>>> forwarding of this communication is strictly prohibited. If you
> have
> >>>>>>> received this communication in error, please contact the sender
> >>>>>> immediately
> >>>>>>> and delete it from your system. Thank You.
> >>>>>>
> >>>>>>
> ---------------------------------------------------------------------
> >>>>>> To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> >>>>>> For additional commands, e-mail: general-help@incubator.apache.org
> >>>>>
> >>>>> --
> >>>>> CONFIDENTIALITY NOTICE
> >>>>> NOTICE: This message is intended for the use of the individual or
> >>> entity to
> >>>>> which it is addressed and may contain information that is
> >> confidential,
> >>>>> privileged and exempt from disclosure under applicable law. If the
> >>> reader
> >>>>> of this message is not the intended recipient, you are hereby
> notified
> >>> that
> >>>>> any printing, copying, dissemination, distribution, disclosure or
> >>>>> forwarding of this communication is strictly prohibited. If you have
> >>>>> received this communication in error, please contact the sender
> >>> immediately
> >>>>> and delete it from your system. Thank You.
> >>>>
> >>>> ---------------------------------------------------------------------
> >>>> To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> >>>> For additional commands, e-mail: general-help@incubator.apache.org
> >>>
> >>> ---------------------------------------------------------------------
> >>> To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> >>> For additional commands, e-mail: general-help@incubator.apache.org
> >>
> >>
> >> --
> >> Alejandro
> >>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> For additional commands, e-mail: general-help@incubator.apache.org
>
>
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: Hoya Proposal
Posted by Alejandro Abdelnur <tu...@cloudera.com>.
argh, gmail trying to be extra smart collapsed his comment on that and i missed. sorry
thx
Alejandro
(phone typing)
> On Jan 9, 2014, at 9:23, larry mccay <lm...@apache.org> wrote:
>
> Hi Alejandro -
>
> I believe that Steve has already acknowledged that a package rename is
> required and that the project is likely not appropriate for Hadoop proper.
>
> thanks,
>
> --larry
>
>
> On Thu, Jan 9, 2014 at 11:46 AM, Alejandro Abdelnur <tu...@cloudera.com>wrote:
>
>> I may have not been clear enough, I was referring to using
>> 'org.apache.hadoop' as package prefix for a project other than hadoop.
>>
>> Thanks
>>
>>
>> On Thu, Jan 9, 2014 at 8:36 AM, Benson Margulies <bimargulies@gmail.com
>>> wrote:
>>
>>> If you can work out a plan to do this directly in Hadoop, there's no
>>> need for the incubator. You just build and and contribute it in
>>> cahoots with them, and earn commit over there as you go.
>>>
>>> On Thu, Jan 9, 2014 at 11:14 AM, Alejandro Abdelnur <tu...@cloudera.com>
>>> wrote:
>>>> Mmmh, if i recall correctly this has come up in the past with other
>>> projects and it was decided against it. Could you please check with the
>>> hadoop folks about it?
>>>>
>>>> Thx
>>>>
>>>>> On Jan 9, 2014, at 1:19 AM, Steve Loughran <st...@hortonworks.com>
>>> wrote:
>>>>>
>>>>> no its wrong, it should all be under org.apache.hoya.
>>>>>
>>>>> I had the hadoop prefix so that I could perhaps put it straight into
>> the
>>>>> hadoop code as another tools module -no need for incubation. But as
>> the
>>>>> actual providers and all tests are related to the deployment of hbase
>>> and
>>>>> accumulo, it really comes downstream of those.
>>>>>
>>>>> so a rename is needed.
>>>>>
>>>>> but yes, ASF headers everywhere
>>>>>
>>>>>
>>>>>> On 8 January 2014 22:48, Henry Saputra <he...@gmail.com>
>>> wrote:
>>>>>>
>>>>>> I like how the initial code already put under "
>>>>>> org.apache.hadoop.hoya" with correct ASF header =)
>>>>>>
>>>>>> - Henry
>>>>>>
>>>>>> On Wed, Jan 8, 2014 at 7:08 AM, Steve Loughran <
>> stevel@hortonworks.com
>>>>
>>>>>> wrote:
>>>>>>> I'm starting to put together the incubation proposal for Hoya: a
>> tool
>>> to
>>>>>>> dynamically deploy applications such as HBase or Accumulo on YARN
>>>>>>>
>>>>>>> https://wiki.apache.org/incubator/HoyaProposal
>>>>>>>
>>>>>>> It does already work to the extent that it can bring up either
>>>>>> application,
>>>>>>> run different clusters of different versions, and remember where
>>>>>> containers
>>>>>>> were allocated so that on application restart it can ask for them
>>> back.
>>>>>>> That increases data locality and makes a big difference with HBase.
>>>>>>>
>>>>>>> It also needs a lot more work -YARN-896 is adding YARN features that
>>>>>> help,
>>>>>>> but there's lots of fun to be had in Hoya including
>>>>>>> -leading edge work in failure handling, modelling cluster
>>> unreliability
>>>>>>> and reacting to it. Can we move beyond simple blacklisting to
>>>>>>> "greylisting", accepting unreliable boxes if we have no altenatives
>>>>>>>
>>>>>>> Then there's adding more providers, to support different application
>>>>>>> installations -I'm starting to write a functional test framework
>> which
>>>>>> need
>>>>>>> provider-specific workload generations
>>>>>>>
>>>>>>> Other features: AM should have a web ui that redirects to the live
>>>>>>> endpoints to all the app-specific UIs (e.g. HBase Master GUI), as
>>> well as
>>>>>>> displaying cluster state itself, for people and for management
>>> tools....
>>>>>>>
>>>>>>> To summarise: lots of fun to be had
>>>>>>>
>>>>>>> -steve
>>>>>>>
>>>>>>> --
>>>>>>> CONFIDENTIALITY NOTICE
>>>>>>> NOTICE: This message is intended for the use of the individual or
>>> entity
>>>>>> to
>>>>>>> which it is addressed and may contain information that is
>>> confidential,
>>>>>>> privileged and exempt from disclosure under applicable law. If the
>>> reader
>>>>>>> of this message is not the intended recipient, you are hereby
>> notified
>>>>>> that
>>>>>>> any printing, copying, dissemination, distribution, disclosure or
>>>>>>> forwarding of this communication is strictly prohibited. If you have
>>>>>>> received this communication in error, please contact the sender
>>>>>> immediately
>>>>>>> and delete it from your system. Thank You.
>>>>>>
>>>>>> ---------------------------------------------------------------------
>>>>>> To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
>>>>>> For additional commands, e-mail: general-help@incubator.apache.org
>>>>>
>>>>> --
>>>>> CONFIDENTIALITY NOTICE
>>>>> NOTICE: This message is intended for the use of the individual or
>>> entity to
>>>>> which it is addressed and may contain information that is
>> confidential,
>>>>> privileged and exempt from disclosure under applicable law. If the
>>> reader
>>>>> of this message is not the intended recipient, you are hereby notified
>>> that
>>>>> any printing, copying, dissemination, distribution, disclosure or
>>>>> forwarding of this communication is strictly prohibited. If you have
>>>>> received this communication in error, please contact the sender
>>> immediately
>>>>> and delete it from your system. Thank You.
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
>>>> For additional commands, e-mail: general-help@incubator.apache.org
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
>>> For additional commands, e-mail: general-help@incubator.apache.org
>>
>>
>> --
>> Alejandro
>>
---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
For additional commands, e-mail: general-help@incubator.apache.org
Re: Hoya Proposal
Posted by larry mccay <lm...@apache.org>.
Hi Alejandro -
I believe that Steve has already acknowledged that a package rename is
required and that the project is likely not appropriate for Hadoop proper.
thanks,
--larry
On Thu, Jan 9, 2014 at 11:46 AM, Alejandro Abdelnur <tu...@cloudera.com>wrote:
> I may have not been clear enough, I was referring to using
> 'org.apache.hadoop' as package prefix for a project other than hadoop.
>
> Thanks
>
>
> On Thu, Jan 9, 2014 at 8:36 AM, Benson Margulies <bimargulies@gmail.com
> >wrote:
>
> > If you can work out a plan to do this directly in Hadoop, there's no
> > need for the incubator. You just build and and contribute it in
> > cahoots with them, and earn commit over there as you go.
> >
> > On Thu, Jan 9, 2014 at 11:14 AM, Alejandro Abdelnur <tu...@cloudera.com>
> > wrote:
> > > Mmmh, if i recall correctly this has come up in the past with other
> > projects and it was decided against it. Could you please check with the
> > hadoop folks about it?
> > >
> > > Thx
> > >
> > >> On Jan 9, 2014, at 1:19 AM, Steve Loughran <st...@hortonworks.com>
> > wrote:
> > >>
> > >> no its wrong, it should all be under org.apache.hoya.
> > >>
> > >> I had the hadoop prefix so that I could perhaps put it straight into
> the
> > >> hadoop code as another tools module -no need for incubation. But as
> the
> > >> actual providers and all tests are related to the deployment of hbase
> > and
> > >> accumulo, it really comes downstream of those.
> > >>
> > >> so a rename is needed.
> > >>
> > >> but yes, ASF headers everywhere
> > >>
> > >>
> > >>> On 8 January 2014 22:48, Henry Saputra <he...@gmail.com>
> > wrote:
> > >>>
> > >>> I like how the initial code already put under "
> > >>> org.apache.hadoop.hoya" with correct ASF header =)
> > >>>
> > >>> - Henry
> > >>>
> > >>> On Wed, Jan 8, 2014 at 7:08 AM, Steve Loughran <
> stevel@hortonworks.com
> > >
> > >>> wrote:
> > >>>> I'm starting to put together the incubation proposal for Hoya: a
> tool
> > to
> > >>>> dynamically deploy applications such as HBase or Accumulo on YARN
> > >>>>
> > >>>> https://wiki.apache.org/incubator/HoyaProposal
> > >>>>
> > >>>> It does already work to the extent that it can bring up either
> > >>> application,
> > >>>> run different clusters of different versions, and remember where
> > >>> containers
> > >>>> were allocated so that on application restart it can ask for them
> > back.
> > >>>> That increases data locality and makes a big difference with HBase.
> > >>>>
> > >>>> It also needs a lot more work -YARN-896 is adding YARN features that
> > >>> help,
> > >>>> but there's lots of fun to be had in Hoya including
> > >>>> -leading edge work in failure handling, modelling cluster
> > unreliability
> > >>>> and reacting to it. Can we move beyond simple blacklisting to
> > >>>> "greylisting", accepting unreliable boxes if we have no altenatives
> > >>>>
> > >>>> Then there's adding more providers, to support different application
> > >>>> installations -I'm starting to write a functional test framework
> which
> > >>> need
> > >>>> provider-specific workload generations
> > >>>>
> > >>>> Other features: AM should have a web ui that redirects to the live
> > >>>> endpoints to all the app-specific UIs (e.g. HBase Master GUI), as
> > well as
> > >>>> displaying cluster state itself, for people and for management
> > tools....
> > >>>>
> > >>>> To summarise: lots of fun to be had
> > >>>>
> > >>>> -steve
> > >>>>
> > >>>> --
> > >>>> CONFIDENTIALITY NOTICE
> > >>>> NOTICE: This message is intended for the use of the individual or
> > entity
> > >>> to
> > >>>> which it is addressed and may contain information that is
> > confidential,
> > >>>> privileged and exempt from disclosure under applicable law. If the
> > reader
> > >>>> of this message is not the intended recipient, you are hereby
> notified
> > >>> that
> > >>>> any printing, copying, dissemination, distribution, disclosure or
> > >>>> forwarding of this communication is strictly prohibited. If you have
> > >>>> received this communication in error, please contact the sender
> > >>> immediately
> > >>>> and delete it from your system. Thank You.
> > >>>
> > >>> ---------------------------------------------------------------------
> > >>> To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> > >>> For additional commands, e-mail: general-help@incubator.apache.org
> > >>
> > >> --
> > >> CONFIDENTIALITY NOTICE
> > >> NOTICE: This message is intended for the use of the individual or
> > entity to
> > >> which it is addressed and may contain information that is
> confidential,
> > >> privileged and exempt from disclosure under applicable law. If the
> > reader
> > >> of this message is not the intended recipient, you are hereby notified
> > that
> > >> any printing, copying, dissemination, distribution, disclosure or
> > >> forwarding of this communication is strictly prohibited. If you have
> > >> received this communication in error, please contact the sender
> > immediately
> > >> and delete it from your system. Thank You.
> > >
> > > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> > > For additional commands, e-mail: general-help@incubator.apache.org
> > >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> > For additional commands, e-mail: general-help@incubator.apache.org
> >
> >
>
>
> --
> Alejandro
>
Re: Hoya Proposal
Posted by Alejandro Abdelnur <tu...@cloudera.com>.
I may have not been clear enough, I was referring to using
'org.apache.hadoop' as package prefix for a project other than hadoop.
Thanks
On Thu, Jan 9, 2014 at 8:36 AM, Benson Margulies <bi...@gmail.com>wrote:
> If you can work out a plan to do this directly in Hadoop, there's no
> need for the incubator. You just build and and contribute it in
> cahoots with them, and earn commit over there as you go.
>
> On Thu, Jan 9, 2014 at 11:14 AM, Alejandro Abdelnur <tu...@cloudera.com>
> wrote:
> > Mmmh, if i recall correctly this has come up in the past with other
> projects and it was decided against it. Could you please check with the
> hadoop folks about it?
> >
> > Thx
> >
> >> On Jan 9, 2014, at 1:19 AM, Steve Loughran <st...@hortonworks.com>
> wrote:
> >>
> >> no its wrong, it should all be under org.apache.hoya.
> >>
> >> I had the hadoop prefix so that I could perhaps put it straight into the
> >> hadoop code as another tools module -no need for incubation. But as the
> >> actual providers and all tests are related to the deployment of hbase
> and
> >> accumulo, it really comes downstream of those.
> >>
> >> so a rename is needed.
> >>
> >> but yes, ASF headers everywhere
> >>
> >>
> >>> On 8 January 2014 22:48, Henry Saputra <he...@gmail.com>
> wrote:
> >>>
> >>> I like how the initial code already put under "
> >>> org.apache.hadoop.hoya" with correct ASF header =)
> >>>
> >>> - Henry
> >>>
> >>> On Wed, Jan 8, 2014 at 7:08 AM, Steve Loughran <stevel@hortonworks.com
> >
> >>> wrote:
> >>>> I'm starting to put together the incubation proposal for Hoya: a tool
> to
> >>>> dynamically deploy applications such as HBase or Accumulo on YARN
> >>>>
> >>>> https://wiki.apache.org/incubator/HoyaProposal
> >>>>
> >>>> It does already work to the extent that it can bring up either
> >>> application,
> >>>> run different clusters of different versions, and remember where
> >>> containers
> >>>> were allocated so that on application restart it can ask for them
> back.
> >>>> That increases data locality and makes a big difference with HBase.
> >>>>
> >>>> It also needs a lot more work -YARN-896 is adding YARN features that
> >>> help,
> >>>> but there's lots of fun to be had in Hoya including
> >>>> -leading edge work in failure handling, modelling cluster
> unreliability
> >>>> and reacting to it. Can we move beyond simple blacklisting to
> >>>> "greylisting", accepting unreliable boxes if we have no altenatives
> >>>>
> >>>> Then there's adding more providers, to support different application
> >>>> installations -I'm starting to write a functional test framework which
> >>> need
> >>>> provider-specific workload generations
> >>>>
> >>>> Other features: AM should have a web ui that redirects to the live
> >>>> endpoints to all the app-specific UIs (e.g. HBase Master GUI), as
> well as
> >>>> displaying cluster state itself, for people and for management
> tools....
> >>>>
> >>>> To summarise: lots of fun to be had
> >>>>
> >>>> -steve
> >>>>
> >>>> --
> >>>> CONFIDENTIALITY NOTICE
> >>>> NOTICE: This message is intended for the use of the individual or
> entity
> >>> to
> >>>> which it is addressed and may contain information that is
> confidential,
> >>>> privileged and exempt from disclosure under applicable law. If the
> reader
> >>>> of this message is not the intended recipient, you are hereby notified
> >>> that
> >>>> any printing, copying, dissemination, distribution, disclosure or
> >>>> forwarding of this communication is strictly prohibited. If you have
> >>>> received this communication in error, please contact the sender
> >>> immediately
> >>>> and delete it from your system. Thank You.
> >>>
> >>> ---------------------------------------------------------------------
> >>> To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> >>> For additional commands, e-mail: general-help@incubator.apache.org
> >>
> >> --
> >> CONFIDENTIALITY NOTICE
> >> NOTICE: This message is intended for the use of the individual or
> entity to
> >> which it is addressed and may contain information that is confidential,
> >> privileged and exempt from disclosure under applicable law. If the
> reader
> >> of this message is not the intended recipient, you are hereby notified
> that
> >> any printing, copying, dissemination, distribution, disclosure or
> >> forwarding of this communication is strictly prohibited. If you have
> >> received this communication in error, please contact the sender
> immediately
> >> and delete it from your system. Thank You.
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> > For additional commands, e-mail: general-help@incubator.apache.org
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> For additional commands, e-mail: general-help@incubator.apache.org
>
>
--
Alejandro
Re: Hoya Proposal
Posted by Benson Margulies <bi...@gmail.com>.
If you can work out a plan to do this directly in Hadoop, there's no
need for the incubator. You just build and and contribute it in
cahoots with them, and earn commit over there as you go.
On Thu, Jan 9, 2014 at 11:14 AM, Alejandro Abdelnur <tu...@cloudera.com> wrote:
> Mmmh, if i recall correctly this has come up in the past with other projects and it was decided against it. Could you please check with the hadoop folks about it?
>
> Thx
>
>> On Jan 9, 2014, at 1:19 AM, Steve Loughran <st...@hortonworks.com> wrote:
>>
>> no its wrong, it should all be under org.apache.hoya.
>>
>> I had the hadoop prefix so that I could perhaps put it straight into the
>> hadoop code as another tools module -no need for incubation. But as the
>> actual providers and all tests are related to the deployment of hbase and
>> accumulo, it really comes downstream of those.
>>
>> so a rename is needed.
>>
>> but yes, ASF headers everywhere
>>
>>
>>> On 8 January 2014 22:48, Henry Saputra <he...@gmail.com> wrote:
>>>
>>> I like how the initial code already put under "
>>> org.apache.hadoop.hoya" with correct ASF header =)
>>>
>>> - Henry
>>>
>>> On Wed, Jan 8, 2014 at 7:08 AM, Steve Loughran <st...@hortonworks.com>
>>> wrote:
>>>> I'm starting to put together the incubation proposal for Hoya: a tool to
>>>> dynamically deploy applications such as HBase or Accumulo on YARN
>>>>
>>>> https://wiki.apache.org/incubator/HoyaProposal
>>>>
>>>> It does already work to the extent that it can bring up either
>>> application,
>>>> run different clusters of different versions, and remember where
>>> containers
>>>> were allocated so that on application restart it can ask for them back.
>>>> That increases data locality and makes a big difference with HBase.
>>>>
>>>> It also needs a lot more work -YARN-896 is adding YARN features that
>>> help,
>>>> but there's lots of fun to be had in Hoya including
>>>> -leading edge work in failure handling, modelling cluster unreliability
>>>> and reacting to it. Can we move beyond simple blacklisting to
>>>> "greylisting", accepting unreliable boxes if we have no altenatives
>>>>
>>>> Then there's adding more providers, to support different application
>>>> installations -I'm starting to write a functional test framework which
>>> need
>>>> provider-specific workload generations
>>>>
>>>> Other features: AM should have a web ui that redirects to the live
>>>> endpoints to all the app-specific UIs (e.g. HBase Master GUI), as well as
>>>> displaying cluster state itself, for people and for management tools....
>>>>
>>>> To summarise: lots of fun to be had
>>>>
>>>> -steve
>>>>
>>>> --
>>>> CONFIDENTIALITY NOTICE
>>>> NOTICE: This message is intended for the use of the individual or entity
>>> to
>>>> which it is addressed and may contain information that is confidential,
>>>> privileged and exempt from disclosure under applicable law. If the reader
>>>> of this message is not the intended recipient, you are hereby notified
>>> that
>>>> any printing, copying, dissemination, distribution, disclosure or
>>>> forwarding of this communication is strictly prohibited. If you have
>>>> received this communication in error, please contact the sender
>>> immediately
>>>> and delete it from your system. Thank You.
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
>>> For additional commands, e-mail: general-help@incubator.apache.org
>>
>> --
>> CONFIDENTIALITY NOTICE
>> NOTICE: This message is intended for the use of the individual or entity to
>> which it is addressed and may contain information that is confidential,
>> privileged and exempt from disclosure under applicable law. If the reader
>> of this message is not the intended recipient, you are hereby notified that
>> any printing, copying, dissemination, distribution, disclosure or
>> forwarding of this communication is strictly prohibited. If you have
>> received this communication in error, please contact the sender immediately
>> and delete it from your system. Thank You.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> For additional commands, e-mail: general-help@incubator.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
For additional commands, e-mail: general-help@incubator.apache.org
Re: Hoya Proposal
Posted by Alejandro Abdelnur <tu...@cloudera.com>.
Mmmh, if i recall correctly this has come up in the past with other projects and it was decided against it. Could you please check with the hadoop folks about it?
Thx
> On Jan 9, 2014, at 1:19 AM, Steve Loughran <st...@hortonworks.com> wrote:
>
> no its wrong, it should all be under org.apache.hoya.
>
> I had the hadoop prefix so that I could perhaps put it straight into the
> hadoop code as another tools module -no need for incubation. But as the
> actual providers and all tests are related to the deployment of hbase and
> accumulo, it really comes downstream of those.
>
> so a rename is needed.
>
> but yes, ASF headers everywhere
>
>
>> On 8 January 2014 22:48, Henry Saputra <he...@gmail.com> wrote:
>>
>> I like how the initial code already put under "
>> org.apache.hadoop.hoya" with correct ASF header =)
>>
>> - Henry
>>
>> On Wed, Jan 8, 2014 at 7:08 AM, Steve Loughran <st...@hortonworks.com>
>> wrote:
>>> I'm starting to put together the incubation proposal for Hoya: a tool to
>>> dynamically deploy applications such as HBase or Accumulo on YARN
>>>
>>> https://wiki.apache.org/incubator/HoyaProposal
>>>
>>> It does already work to the extent that it can bring up either
>> application,
>>> run different clusters of different versions, and remember where
>> containers
>>> were allocated so that on application restart it can ask for them back.
>>> That increases data locality and makes a big difference with HBase.
>>>
>>> It also needs a lot more work -YARN-896 is adding YARN features that
>> help,
>>> but there's lots of fun to be had in Hoya including
>>> -leading edge work in failure handling, modelling cluster unreliability
>>> and reacting to it. Can we move beyond simple blacklisting to
>>> "greylisting", accepting unreliable boxes if we have no altenatives
>>>
>>> Then there's adding more providers, to support different application
>>> installations -I'm starting to write a functional test framework which
>> need
>>> provider-specific workload generations
>>>
>>> Other features: AM should have a web ui that redirects to the live
>>> endpoints to all the app-specific UIs (e.g. HBase Master GUI), as well as
>>> displaying cluster state itself, for people and for management tools....
>>>
>>> To summarise: lots of fun to be had
>>>
>>> -steve
>>>
>>> --
>>> CONFIDENTIALITY NOTICE
>>> NOTICE: This message is intended for the use of the individual or entity
>> to
>>> which it is addressed and may contain information that is confidential,
>>> privileged and exempt from disclosure under applicable law. If the reader
>>> of this message is not the intended recipient, you are hereby notified
>> that
>>> any printing, copying, dissemination, distribution, disclosure or
>>> forwarding of this communication is strictly prohibited. If you have
>>> received this communication in error, please contact the sender
>> immediately
>>> and delete it from your system. Thank You.
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
>> For additional commands, e-mail: general-help@incubator.apache.org
>
> --
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity to
> which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
For additional commands, e-mail: general-help@incubator.apache.org
Re: Hoya Proposal
Posted by Steve Loughran <st...@hortonworks.com>.
no its wrong, it should all be under org.apache.hoya.
I had the hadoop prefix so that I could perhaps put it straight into the
hadoop code as another tools module -no need for incubation. But as the
actual providers and all tests are related to the deployment of hbase and
accumulo, it really comes downstream of those.
so a rename is needed.
but yes, ASF headers everywhere
On 8 January 2014 22:48, Henry Saputra <he...@gmail.com> wrote:
> I like how the initial code already put under "
> org.apache.hadoop.hoya" with correct ASF header =)
>
> - Henry
>
> On Wed, Jan 8, 2014 at 7:08 AM, Steve Loughran <st...@hortonworks.com>
> wrote:
> > I'm starting to put together the incubation proposal for Hoya: a tool to
> > dynamically deploy applications such as HBase or Accumulo on YARN
> >
> > https://wiki.apache.org/incubator/HoyaProposal
> >
> > It does already work to the extent that it can bring up either
> application,
> > run different clusters of different versions, and remember where
> containers
> > were allocated so that on application restart it can ask for them back.
> > That increases data locality and makes a big difference with HBase.
> >
> > It also needs a lot more work -YARN-896 is adding YARN features that
> help,
> > but there's lots of fun to be had in Hoya including
> > -leading edge work in failure handling, modelling cluster unreliability
> > and reacting to it. Can we move beyond simple blacklisting to
> > "greylisting", accepting unreliable boxes if we have no altenatives
> >
> > Then there's adding more providers, to support different application
> > installations -I'm starting to write a functional test framework which
> need
> > provider-specific workload generations
> >
> > Other features: AM should have a web ui that redirects to the live
> > endpoints to all the app-specific UIs (e.g. HBase Master GUI), as well as
> > displaying cluster state itself, for people and for management tools....
> >
> > To summarise: lots of fun to be had
> >
> > -steve
> >
> > --
> > CONFIDENTIALITY NOTICE
> > NOTICE: This message is intended for the use of the individual or entity
> to
> > which it is addressed and may contain information that is confidential,
> > privileged and exempt from disclosure under applicable law. If the reader
> > of this message is not the intended recipient, you are hereby notified
> that
> > any printing, copying, dissemination, distribution, disclosure or
> > forwarding of this communication is strictly prohibited. If you have
> > received this communication in error, please contact the sender
> immediately
> > and delete it from your system. Thank You.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> For additional commands, e-mail: general-help@incubator.apache.org
>
>
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: Hoya Proposal
Posted by Henry Saputra <he...@gmail.com>.
I like how the initial code already put under "
org.apache.hadoop.hoya" with correct ASF header =)
- Henry
On Wed, Jan 8, 2014 at 7:08 AM, Steve Loughran <st...@hortonworks.com> wrote:
> I'm starting to put together the incubation proposal for Hoya: a tool to
> dynamically deploy applications such as HBase or Accumulo on YARN
>
> https://wiki.apache.org/incubator/HoyaProposal
>
> It does already work to the extent that it can bring up either application,
> run different clusters of different versions, and remember where containers
> were allocated so that on application restart it can ask for them back.
> That increases data locality and makes a big difference with HBase.
>
> It also needs a lot more work -YARN-896 is adding YARN features that help,
> but there's lots of fun to be had in Hoya including
> -leading edge work in failure handling, modelling cluster unreliability
> and reacting to it. Can we move beyond simple blacklisting to
> "greylisting", accepting unreliable boxes if we have no altenatives
>
> Then there's adding more providers, to support different application
> installations -I'm starting to write a functional test framework which need
> provider-specific workload generations
>
> Other features: AM should have a web ui that redirects to the live
> endpoints to all the app-specific UIs (e.g. HBase Master GUI), as well as
> displaying cluster state itself, for people and for management tools....
>
> To summarise: lots of fun to be had
>
> -steve
>
> --
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity to
> which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
For additional commands, e-mail: general-help@incubator.apache.org