You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@accumulo.apache.org by Josh Elser <jo...@gmail.com> on 2013/10/24 01:05:08 UTC
[DISCUSS] Hadoop 2 and Accumulo 1.6.0
To ensure that we get broader community interaction than only on a Jira
issue [1], I want to get community feedback about the version of Hadoop
which the default, deployed Accumulo artifacts will be compiled against.
Currently, Accumulo builds against a Hadoop-1 series release
(1.5.1-SNAPSHOT and 1.6.0-SNAPSHOT build against 1.2.1, and 1.5.0 builds
against 1.0.4). Last week, the Apache Hadoop community voted to release
2.2.0 as GA (general availability) -- in other words, the Apache Hadoop
community is calling Hadoop-2.2.0 "stable".
As has been discussed across various issues on Jira, this means a few
different things for Accumulo. Most importantly, this serves as a
recommendation by us that users should be trying to use Hadoop-2.2.0
with Accumulo 1.6.0. This does *not* mean that we do not support Hadoop1
([2] 1.2.1 specifically). Hadoop-1 support would still be "guaranteed"
by us for 1.6.0.
- Josh
[1] https://issues.apache.org/jira/browse/ACCUMULO-1419
[2] https://issues.apache.org/jira/browse/ACCUMULO-1643
Re: [DISCUSS] Hadoop 2 and Accumulo 1.6.0
Posted by Steve Loughran <st...@hortonworks.com>.
at a lower level, Hadoop 2.2 has some other opportunities too
- Move up the dependencies (inc things like SLF4J, protobuf) to be
consistent.
-co-exist with YARN. That shouldn't take any effort, merely that
a YARN node manager can run on every server node, so that YARN-scheduled
work can be executed on it -code that may want to work with Accumulo.
-deploy under YARN. This is what we are doing with Hoya
-Move to the Hadoop 2 APIs, which for Accumulo means changes in the
filesystem
APIs (FileContext alongside the classic FileSystem). While FileContext
promises
a more consistent set of semantics, I don't see any single tangible feature
in
the API that isn't in FileSystem (I speak as someone who has just
implemented
a new FileSystem implementation for Hadoop 1 & 2)/
- Move to the new "undeprecated" configuration options. This is very much
a committing move, except in the special case that you are referencing
strings in the Hadoop source, and they change in those JARs. (remember,
strings
don't get copied at compile time, only the types where sizeof() < =8 ).
As the deprecated options get migrated, not moving merely adds more
warning messages into the logs. Which you can turn off by cranking back on
the level
of org.apache.hadoop.conf.Configuration.deprecation (Hadoop-9487 by one
Stevel)
The least traumatic options would be
-stay with the APIs and constants that work with 1.x for now
-run from the 2.0.5 APIs, switch Hadoop 2 to 2.2.0
-test and run on it: find bugreps, file them now
-help us with Hoya -we are drafting it for incubation- so that anyone with
a Hadoop 2 cluster can bring up an Accumulo cluster on top of it -without
even needing to talk to ops about it.
On 24 October 2013 05:51, William Slacum <wi...@accumulo.net>wrote:
> There wasn't any discussions in those tickets as to what Hadoop 2 provides
> Accumulo. If we're going to still support 1, then any new features only
> possible with 2 have to become optional until we ditch support for 1. Is
> there anything people have in mind, feature wise, that Hadoop 2 would help
> with?
>
>
> On Wed, Oct 23, 2013 at 7:05 PM, Josh Elser <jo...@gmail.com> wrote:
>
> > To ensure that we get broader community interaction than only on a Jira
> > issue [1], I want to get community feedback about the version of Hadoop
> > which the default, deployed Accumulo artifacts will be compiled against.
> >
> > Currently, Accumulo builds against a Hadoop-1 series release
> > (1.5.1-SNAPSHOT and 1.6.0-SNAPSHOT build against 1.2.1, and 1.5.0 builds
> > against 1.0.4). Last week, the Apache Hadoop community voted to release
> > 2.2.0 as GA (general availability) -- in other words, the Apache Hadoop
> > community is calling Hadoop-2.2.0 "stable".
> >
> > As has been discussed across various issues on Jira, this means a few
> > different things for Accumulo. Most importantly, this serves as a
> > recommendation by us that users should be trying to use Hadoop-2.2.0 with
> > Accumulo 1.6.0. This does *not* mean that we do not support Hadoop1 ([2]
> > 1.2.1 specifically). Hadoop-1 support would still be "guaranteed" by us
> for
> > 1.6.0.
> >
> > - Josh
> >
> > [1] https://issues.apache.org/**jira/browse/ACCUMULO-1419<
> https://issues.apache.org/jira/browse/ACCUMULO-1419>
> > [2] https://issues.apache.org/**jira/browse/ACCUMULO-1643<
> https://issues.apache.org/jira/browse/ACCUMULO-1643>
> >
>
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: [DISCUSS] Hadoop 2 and Accumulo 1.6.0
Posted by Steve Loughran <st...@hortonworks.com>.
On 24 October 2013 18:01, John Vines <vi...@apache.org> wrote:
> Accumulo isn't going to run in YARN,
Well it can if you want it too - what we are doing with hoya is giving the
option to do without changing the core application.
Some applications (e.g. Samza) are being written from the ground up for
YARN , so they have an Application Master that manages requesting
containers, running code in it, while the code knows it is in an isolated
container (currently: transient dir and cgroup-limited RAM, more limits
coming). But you don't need a complete rewrite to gain from running under
YAR.
Accumulo is very suited to running in YARN; if you look at the list of
things we need from an app to get it to play, it's pretty much complete:
https://github.com/hortonworks/hoya/blob/develop/src/site/markdown/app_needs.md
A key benefit is how you can react to failures. Until now: tserver down:
log it, balance workload. {master, monitor, gc} down: page someone.
In YARN, all failures become something for the automation to handle: log
it, ask for new instance, balance work back when it is up. You can
downgrade the one-per-application node to that of logged statistics rather
than events that need immediate human intervention,
Detecting a process failure is trivial: it exits and the container does
too. Harder is more byzantine failure modes -what would be ideal would just
be able to go from the ZK data to identifying which service instance needs
to be let go.
What is useful here is getting all the locations of things like web URLs
out of ZK too -as you can't predetermine the host they will be on, and if
you fix the port you may have binding problems.
> it's just a matter of replacing MR
> compatibility with YARN compatibility.
>
>
MRv1 code should work with MR-on-YARN : if not, file a bug report
What you do get long term is the option of talking to other services
running in the cluster -and for them to talk to you
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: [DISCUSS] Hadoop 2 and Accumulo 1.6.0
Posted by Josh Elser <jo...@gmail.com>.
On the contrary, it actually can. There's an example in the following
repo which I've used to actually run Accumulo on YARN. It's still a
little rough around the edges in some cases, but it does work.
https://github.com/hortonworks/hoya/
Point being though, there's nothing special that Accumulo needs to do
here. The only real concern is the compatibility with Hadoop1 and
Hadoop2 APIs (which is already done).
There are no architectural tradeoffs that I am aware of between hadoop 1
and 2.
On 10/24/13 10:01 AM, John Vines wrote:
> Accumulo isn't going to run in YARN, it's just a matter of replacing MR
> compatibility with YARN compatibility.
>
>
> On Thu, Oct 24, 2013 at 7:04 AM, David Medinets <da...@gmail.com>wrote:
>
>> Can an application work with parts in Hadoop 1 and parts in YARN? Could we
>> convert one component at a time to work inside YARN? For example, first
>> move monitor then tracer, etc. Where there any architectural tradeoffs made
>> for Hadoop 1 that no longer apply in Hadoop 2?
>>
>>
>> On Thu, Oct 24, 2013 at 12:58 AM, Josh Elser <jo...@gmail.com> wrote:
>>
>>> Highlights from hadoop.apache.org
>>>
>>> YARN - A general purpose resource management system for Hadoop to
>>> allow MapReduce and other other data processing frameworks and services
>>> High Availability for HDFS
>>> HDFS Federation
>>> HDFS Snapshots
>>> NFSv3 access to data in HDFS
>>> Support for running Hadoop on Microsoft Windows
>>> Binary Compatibility for MapReduce applications built on hadoop-1.x
>>> Substantial amount of integration testing with rest of projects in
>> the
>>> ecosystem
>>>
>>> Additionally, I believe there are substantial improvements in HDFS which
>>> should improve general performance for all filesystem ops.
>>>
>>>
>>> On 10/23/13 9:51 PM, William Slacum wrote:
>>>
>>>> There wasn't any discussions in those tickets as to what Hadoop 2
>> provides
>>>> Accumulo. If we're going to still support 1, then any new features only
>>>> possible with 2 have to become optional until we ditch support for 1. Is
>>>> there anything people have in mind, feature wise, that Hadoop 2 would
>> help
>>>> with?
>>>>
>>>>
>>>> On Wed, Oct 23, 2013 at 7:05 PM, Josh Elser <jo...@gmail.com>
>> wrote:
>>>>
>>>> To ensure that we get broader community interaction than only on a Jira
>>>>> issue [1], I want to get community feedback about the version of Hadoop
>>>>> which the default, deployed Accumulo artifacts will be compiled
>> against.
>>>>>
>>>>> Currently, Accumulo builds against a Hadoop-1 series release
>>>>> (1.5.1-SNAPSHOT and 1.6.0-SNAPSHOT build against 1.2.1, and 1.5.0
>> builds
>>>>> against 1.0.4). Last week, the Apache Hadoop community voted to release
>>>>> 2.2.0 as GA (general availability) -- in other words, the Apache Hadoop
>>>>> community is calling Hadoop-2.2.0 "stable".
>>>>>
>>>>> As has been discussed across various issues on Jira, this means a few
>>>>> different things for Accumulo. Most importantly, this serves as a
>>>>> recommendation by us that users should be trying to use Hadoop-2.2.0
>> with
>>>>> Accumulo 1.6.0. This does *not* mean that we do not support Hadoop1
>> ([2]
>>>>> 1.2.1 specifically). Hadoop-1 support would still be "guaranteed" by us
>>>>> for
>>>>> 1.6.0.
>>>>>
>>>>> - Josh
>>>>>
>>>>> [1] https://issues.apache.org/****jira/browse/ACCUMULO-1419<
>> https://issues.apache.org/**jira/browse/ACCUMULO-1419>
>>>>> <http**s://issues.apache.org/jira/**browse/ACCUMULO-1419<
>> https://issues.apache.org/jira/browse/ACCUMULO-1419>
>>>>>>
>>>>> [2] https://issues.apache.org/****jira/browse/ACCUMULO-1643<
>> https://issues.apache.org/**jira/browse/ACCUMULO-1643>
>>>>> <http**s://issues.apache.org/jira/**browse/ACCUMULO-1643<
>> https://issues.apache.org/jira/browse/ACCUMULO-1643>
>>>>>>
>>>>>
>>>>>
>>>>
>>
>
Re: [DISCUSS] Hadoop 2 and Accumulo 1.6.0
Posted by John Vines <vi...@apache.org>.
Accumulo isn't going to run in YARN, it's just a matter of replacing MR
compatibility with YARN compatibility.
On Thu, Oct 24, 2013 at 7:04 AM, David Medinets <da...@gmail.com>wrote:
> Can an application work with parts in Hadoop 1 and parts in YARN? Could we
> convert one component at a time to work inside YARN? For example, first
> move monitor then tracer, etc. Where there any architectural tradeoffs made
> for Hadoop 1 that no longer apply in Hadoop 2?
>
>
> On Thu, Oct 24, 2013 at 12:58 AM, Josh Elser <jo...@gmail.com> wrote:
>
> > Highlights from hadoop.apache.org
> >
> > YARN - A general purpose resource management system for Hadoop to
> > allow MapReduce and other other data processing frameworks and services
> > High Availability for HDFS
> > HDFS Federation
> > HDFS Snapshots
> > NFSv3 access to data in HDFS
> > Support for running Hadoop on Microsoft Windows
> > Binary Compatibility for MapReduce applications built on hadoop-1.x
> > Substantial amount of integration testing with rest of projects in
> the
> > ecosystem
> >
> > Additionally, I believe there are substantial improvements in HDFS which
> > should improve general performance for all filesystem ops.
> >
> >
> > On 10/23/13 9:51 PM, William Slacum wrote:
> >
> >> There wasn't any discussions in those tickets as to what Hadoop 2
> provides
> >> Accumulo. If we're going to still support 1, then any new features only
> >> possible with 2 have to become optional until we ditch support for 1. Is
> >> there anything people have in mind, feature wise, that Hadoop 2 would
> help
> >> with?
> >>
> >>
> >> On Wed, Oct 23, 2013 at 7:05 PM, Josh Elser <jo...@gmail.com>
> wrote:
> >>
> >> To ensure that we get broader community interaction than only on a Jira
> >>> issue [1], I want to get community feedback about the version of Hadoop
> >>> which the default, deployed Accumulo artifacts will be compiled
> against.
> >>>
> >>> Currently, Accumulo builds against a Hadoop-1 series release
> >>> (1.5.1-SNAPSHOT and 1.6.0-SNAPSHOT build against 1.2.1, and 1.5.0
> builds
> >>> against 1.0.4). Last week, the Apache Hadoop community voted to release
> >>> 2.2.0 as GA (general availability) -- in other words, the Apache Hadoop
> >>> community is calling Hadoop-2.2.0 "stable".
> >>>
> >>> As has been discussed across various issues on Jira, this means a few
> >>> different things for Accumulo. Most importantly, this serves as a
> >>> recommendation by us that users should be trying to use Hadoop-2.2.0
> with
> >>> Accumulo 1.6.0. This does *not* mean that we do not support Hadoop1
> ([2]
> >>> 1.2.1 specifically). Hadoop-1 support would still be "guaranteed" by us
> >>> for
> >>> 1.6.0.
> >>>
> >>> - Josh
> >>>
> >>> [1] https://issues.apache.org/****jira/browse/ACCUMULO-1419<
> https://issues.apache.org/**jira/browse/ACCUMULO-1419>
> >>> <http**s://issues.apache.org/jira/**browse/ACCUMULO-1419<
> https://issues.apache.org/jira/browse/ACCUMULO-1419>
> >>> >
> >>> [2] https://issues.apache.org/****jira/browse/ACCUMULO-1643<
> https://issues.apache.org/**jira/browse/ACCUMULO-1643>
> >>> <http**s://issues.apache.org/jira/**browse/ACCUMULO-1643<
> https://issues.apache.org/jira/browse/ACCUMULO-1643>
> >>> >
> >>>
> >>>
> >>
>
Re: [DISCUSS] Hadoop 2 and Accumulo 1.6.0
Posted by David Medinets <da...@gmail.com>.
Can an application work with parts in Hadoop 1 and parts in YARN? Could we
convert one component at a time to work inside YARN? For example, first
move monitor then tracer, etc. Where there any architectural tradeoffs made
for Hadoop 1 that no longer apply in Hadoop 2?
On Thu, Oct 24, 2013 at 12:58 AM, Josh Elser <jo...@gmail.com> wrote:
> Highlights from hadoop.apache.org
>
> YARN - A general purpose resource management system for Hadoop to
> allow MapReduce and other other data processing frameworks and services
> High Availability for HDFS
> HDFS Federation
> HDFS Snapshots
> NFSv3 access to data in HDFS
> Support for running Hadoop on Microsoft Windows
> Binary Compatibility for MapReduce applications built on hadoop-1.x
> Substantial amount of integration testing with rest of projects in the
> ecosystem
>
> Additionally, I believe there are substantial improvements in HDFS which
> should improve general performance for all filesystem ops.
>
>
> On 10/23/13 9:51 PM, William Slacum wrote:
>
>> There wasn't any discussions in those tickets as to what Hadoop 2 provides
>> Accumulo. If we're going to still support 1, then any new features only
>> possible with 2 have to become optional until we ditch support for 1. Is
>> there anything people have in mind, feature wise, that Hadoop 2 would help
>> with?
>>
>>
>> On Wed, Oct 23, 2013 at 7:05 PM, Josh Elser <jo...@gmail.com> wrote:
>>
>> To ensure that we get broader community interaction than only on a Jira
>>> issue [1], I want to get community feedback about the version of Hadoop
>>> which the default, deployed Accumulo artifacts will be compiled against.
>>>
>>> Currently, Accumulo builds against a Hadoop-1 series release
>>> (1.5.1-SNAPSHOT and 1.6.0-SNAPSHOT build against 1.2.1, and 1.5.0 builds
>>> against 1.0.4). Last week, the Apache Hadoop community voted to release
>>> 2.2.0 as GA (general availability) -- in other words, the Apache Hadoop
>>> community is calling Hadoop-2.2.0 "stable".
>>>
>>> As has been discussed across various issues on Jira, this means a few
>>> different things for Accumulo. Most importantly, this serves as a
>>> recommendation by us that users should be trying to use Hadoop-2.2.0 with
>>> Accumulo 1.6.0. This does *not* mean that we do not support Hadoop1 ([2]
>>> 1.2.1 specifically). Hadoop-1 support would still be "guaranteed" by us
>>> for
>>> 1.6.0.
>>>
>>> - Josh
>>>
>>> [1] https://issues.apache.org/****jira/browse/ACCUMULO-1419<https://issues.apache.org/**jira/browse/ACCUMULO-1419>
>>> <http**s://issues.apache.org/jira/**browse/ACCUMULO-1419<https://issues.apache.org/jira/browse/ACCUMULO-1419>
>>> >
>>> [2] https://issues.apache.org/****jira/browse/ACCUMULO-1643<https://issues.apache.org/**jira/browse/ACCUMULO-1643>
>>> <http**s://issues.apache.org/jira/**browse/ACCUMULO-1643<https://issues.apache.org/jira/browse/ACCUMULO-1643>
>>> >
>>>
>>>
>>
Re: [DISCUSS] Hadoop 2 and Accumulo 1.6.0
Posted by Josh Elser <jo...@gmail.com>.
Highlights from hadoop.apache.org
YARN - A general purpose resource management system for Hadoop to
allow MapReduce and other other data processing frameworks and services
High Availability for HDFS
HDFS Federation
HDFS Snapshots
NFSv3 access to data in HDFS
Support for running Hadoop on Microsoft Windows
Binary Compatibility for MapReduce applications built on hadoop-1.x
Substantial amount of integration testing with rest of projects in
the ecosystem
Additionally, I believe there are substantial improvements in HDFS which
should improve general performance for all filesystem ops.
On 10/23/13 9:51 PM, William Slacum wrote:
> There wasn't any discussions in those tickets as to what Hadoop 2 provides
> Accumulo. If we're going to still support 1, then any new features only
> possible with 2 have to become optional until we ditch support for 1. Is
> there anything people have in mind, feature wise, that Hadoop 2 would help
> with?
>
>
> On Wed, Oct 23, 2013 at 7:05 PM, Josh Elser <jo...@gmail.com> wrote:
>
>> To ensure that we get broader community interaction than only on a Jira
>> issue [1], I want to get community feedback about the version of Hadoop
>> which the default, deployed Accumulo artifacts will be compiled against.
>>
>> Currently, Accumulo builds against a Hadoop-1 series release
>> (1.5.1-SNAPSHOT and 1.6.0-SNAPSHOT build against 1.2.1, and 1.5.0 builds
>> against 1.0.4). Last week, the Apache Hadoop community voted to release
>> 2.2.0 as GA (general availability) -- in other words, the Apache Hadoop
>> community is calling Hadoop-2.2.0 "stable".
>>
>> As has been discussed across various issues on Jira, this means a few
>> different things for Accumulo. Most importantly, this serves as a
>> recommendation by us that users should be trying to use Hadoop-2.2.0 with
>> Accumulo 1.6.0. This does *not* mean that we do not support Hadoop1 ([2]
>> 1.2.1 specifically). Hadoop-1 support would still be "guaranteed" by us for
>> 1.6.0.
>>
>> - Josh
>>
>> [1] https://issues.apache.org/**jira/browse/ACCUMULO-1419<https://issues.apache.org/jira/browse/ACCUMULO-1419>
>> [2] https://issues.apache.org/**jira/browse/ACCUMULO-1643<https://issues.apache.org/jira/browse/ACCUMULO-1643>
>>
>
Re: [DISCUSS] Hadoop 2 and Accumulo 1.6.0
Posted by Josh Elser <jo...@gmail.com>.
Highlights from hadoop.apache.org
YARN - A general purpose resource management system for Hadoop to
allow MapReduce and other other data processing frameworks and services
High Availability for HDFS
HDFS Federation
HDFS Snapshots
NFSv3 access to data in HDFS
Support for running Hadoop on Microsoft Windows
Binary Compatibility for MapReduce applications built on hadoop-1.x
Substantial amount of integration testing with rest of projects in
the ecosystem
Additionally, I believe there are substantial improvements in HDFS which
should improve general performance for all filesystem ops.
On 10/23/13 9:51 PM, William Slacum wrote:
> There wasn't any discussions in those tickets as to what Hadoop 2 provides
> Accumulo. If we're going to still support 1, then any new features only
> possible with 2 have to become optional until we ditch support for 1. Is
> there anything people have in mind, feature wise, that Hadoop 2 would help
> with?
>
>
> On Wed, Oct 23, 2013 at 7:05 PM, Josh Elser <jo...@gmail.com> wrote:
>
>> To ensure that we get broader community interaction than only on a Jira
>> issue [1], I want to get community feedback about the version of Hadoop
>> which the default, deployed Accumulo artifacts will be compiled against.
>>
>> Currently, Accumulo builds against a Hadoop-1 series release
>> (1.5.1-SNAPSHOT and 1.6.0-SNAPSHOT build against 1.2.1, and 1.5.0 builds
>> against 1.0.4). Last week, the Apache Hadoop community voted to release
>> 2.2.0 as GA (general availability) -- in other words, the Apache Hadoop
>> community is calling Hadoop-2.2.0 "stable".
>>
>> As has been discussed across various issues on Jira, this means a few
>> different things for Accumulo. Most importantly, this serves as a
>> recommendation by us that users should be trying to use Hadoop-2.2.0 with
>> Accumulo 1.6.0. This does *not* mean that we do not support Hadoop1 ([2]
>> 1.2.1 specifically). Hadoop-1 support would still be "guaranteed" by us for
>> 1.6.0.
>>
>> - Josh
>>
>> [1] https://issues.apache.org/**jira/browse/ACCUMULO-1419<https://issues.apache.org/jira/browse/ACCUMULO-1419>
>> [2] https://issues.apache.org/**jira/browse/ACCUMULO-1643<https://issues.apache.org/jira/browse/ACCUMULO-1643>
>>
>
Re: [DISCUSS] Hadoop 2 and Accumulo 1.6.0
Posted by William Slacum <wi...@accumulo.net>.
There wasn't any discussions in those tickets as to what Hadoop 2 provides
Accumulo. If we're going to still support 1, then any new features only
possible with 2 have to become optional until we ditch support for 1. Is
there anything people have in mind, feature wise, that Hadoop 2 would help
with?
On Wed, Oct 23, 2013 at 7:05 PM, Josh Elser <jo...@gmail.com> wrote:
> To ensure that we get broader community interaction than only on a Jira
> issue [1], I want to get community feedback about the version of Hadoop
> which the default, deployed Accumulo artifacts will be compiled against.
>
> Currently, Accumulo builds against a Hadoop-1 series release
> (1.5.1-SNAPSHOT and 1.6.0-SNAPSHOT build against 1.2.1, and 1.5.0 builds
> against 1.0.4). Last week, the Apache Hadoop community voted to release
> 2.2.0 as GA (general availability) -- in other words, the Apache Hadoop
> community is calling Hadoop-2.2.0 "stable".
>
> As has been discussed across various issues on Jira, this means a few
> different things for Accumulo. Most importantly, this serves as a
> recommendation by us that users should be trying to use Hadoop-2.2.0 with
> Accumulo 1.6.0. This does *not* mean that we do not support Hadoop1 ([2]
> 1.2.1 specifically). Hadoop-1 support would still be "guaranteed" by us for
> 1.6.0.
>
> - Josh
>
> [1] https://issues.apache.org/**jira/browse/ACCUMULO-1419<https://issues.apache.org/jira/browse/ACCUMULO-1419>
> [2] https://issues.apache.org/**jira/browse/ACCUMULO-1643<https://issues.apache.org/jira/browse/ACCUMULO-1643>
>
Re: [DISCUSS] Hadoop 2 and Accumulo 1.6.0
Posted by Josh Elser <jo...@gmail.com>.
I just tested shell stuff
On 10/24/13 11:21 AM, Sean Busbey wrote:
>
>
>
> On Thu, Oct 24, 2013 at 1:17 PM, Josh Elser <josh.elser@gmail.com
> <ma...@gmail.com>> wrote:
>
> Hadoop-2 compiled accumulo artifacts run just fine against Hadoop-1.2.1
>
>
>
> Does this mean "shell works" or functional tests? (or a really short
> continuous run?)
>
>
> --
> Sean
Re: [DISCUSS] Hadoop 2 and Accumulo 1.6.0
Posted by Sean Busbey <bu...@cloudera.com>.
On Thu, Oct 24, 2013 at 1:17 PM, Josh Elser <jo...@gmail.com> wrote:
> Hadoop-2 compiled accumulo artifacts run just fine against Hadoop-1.2.1
>
>
>>>
Does this mean "shell works" or functional tests? (or a really short
continuous run?)
--
Sean
Re: [DISCUSS] Hadoop 2 and Accumulo 1.6.0
Posted by Josh Elser <jo...@gmail.com>.
Hadoop-2 compiled accumulo artifacts run just fine against Hadoop-1.2.1
On 10/24/13 10:14 AM, Josh Elser wrote:
> I'll try the vice versa now. It would be good to know ahead of time if
> we need to create multiple artifacts so that users can pull the correct
> GAV out of central.
>
> On 10/24/13 9:59 AM, Keith Turner wrote:
>>
>>
>>
>> On Thu, Oct 24, 2013 at 12:41 PM, Eric Newton <eric.newton@gmail.com
>> <ma...@gmail.com>> wrote:
>>
>> > Does Accumulo compiled against Hadoop-1 run against Hadoop-2 w/o
>> > recompilation
>>
>> Yes.
>>
>> > or visa versa?
>>
>> I think so, but I've not tested this.
>>
>>
>> We should test this. If accumulo jars compiled against hadoop-2 do not
>> run against hadoop-1, then it would cause users problems when they pull
>> those jars from maven central.
>>
>>
>> -Eric
>>
>>
Re: [DISCUSS] Hadoop 2 and Accumulo 1.6.0
Posted by Josh Elser <jo...@gmail.com>.
I'll try the vice versa now. It would be good to know ahead of time if
we need to create multiple artifacts so that users can pull the correct
GAV out of central.
On 10/24/13 9:59 AM, Keith Turner wrote:
>
>
>
> On Thu, Oct 24, 2013 at 12:41 PM, Eric Newton <eric.newton@gmail.com
> <ma...@gmail.com>> wrote:
>
> > Does Accumulo compiled against Hadoop-1 run against Hadoop-2 w/o
> > recompilation
>
> Yes.
>
> > or visa versa?
>
> I think so, but I've not tested this.
>
>
> We should test this. If accumulo jars compiled against hadoop-2 do not
> run against hadoop-1, then it would cause users problems when they pull
> those jars from maven central.
>
>
> -Eric
>
>
Re: [DISCUSS] Hadoop 2 and Accumulo 1.6.0
Posted by Keith Turner <ke...@deenlo.com>.
On Thu, Oct 24, 2013 at 12:41 PM, Eric Newton <er...@gmail.com> wrote:
> > Does Accumulo compiled against Hadoop-1 run against Hadoop-2 w/o
> > recompilation
>
> Yes.
>
> > or visa versa?
>
> I think so, but I've not tested this.
>
We should test this. If accumulo jars compiled against hadoop-2 do not run
against hadoop-1, then it would cause users problems when they pull those
jars from maven central.
>
> -Eric
>
Re: [DISCUSS] Hadoop 2 and Accumulo 1.6.0
Posted by Eric Newton <er...@gmail.com>.
> Does Accumulo compiled against Hadoop-1 run against Hadoop-2 w/o
> recompilation
Yes.
> or visa versa?
I think so, but I've not tested this.
-Eric
Re: [DISCUSS] Hadoop 2 and Accumulo 1.6.0
Posted by Keith Turner <ke...@deenlo.com>.
On Wed, Oct 23, 2013 at 7:05 PM, Josh Elser <jo...@gmail.com> wrote:
> To ensure that we get broader community interaction than only on a Jira
> issue [1], I want to get community feedback about the version of Hadoop
> which the default, deployed Accumulo artifacts will be compiled against.
>
Does Accumulo compiled against Hadoop-1 run against Hadoop-2 w/o
recompilation or visa versa? If this does not work, then I suppose the
jars we put into central will not work for some users.
>
> Currently, Accumulo builds against a Hadoop-1 series release
> (1.5.1-SNAPSHOT and 1.6.0-SNAPSHOT build against 1.2.1, and 1.5.0 builds
> against 1.0.4). Last week, the Apache Hadoop community voted to release
> 2.2.0 as GA (general availability) -- in other words, the Apache Hadoop
> community is calling Hadoop-2.2.0 "stable".
>
> As has been discussed across various issues on Jira, this means a few
> different things for Accumulo. Most importantly, this serves as a
> recommendation by us that users should be trying to use Hadoop-2.2.0 with
> Accumulo 1.6.0. This does *not* mean that we do not support Hadoop1 ([2]
> 1.2.1 specifically). Hadoop-1 support would still be "guaranteed" by us for
> 1.6.0.
>
> - Josh
>
> [1] https://issues.apache.org/**jira/browse/ACCUMULO-1419<https://issues.apache.org/jira/browse/ACCUMULO-1419>
> [2] https://issues.apache.org/**jira/browse/ACCUMULO-1643<https://issues.apache.org/jira/browse/ACCUMULO-1643>
>
Re: [DISCUSS] Hadoop 2 and Accumulo 1.6.0
Posted by William Slacum <wi...@accumulo.net>.
There wasn't any discussions in those tickets as to what Hadoop 2 provides
Accumulo. If we're going to still support 1, then any new features only
possible with 2 have to become optional until we ditch support for 1. Is
there anything people have in mind, feature wise, that Hadoop 2 would help
with?
On Wed, Oct 23, 2013 at 7:05 PM, Josh Elser <jo...@gmail.com> wrote:
> To ensure that we get broader community interaction than only on a Jira
> issue [1], I want to get community feedback about the version of Hadoop
> which the default, deployed Accumulo artifacts will be compiled against.
>
> Currently, Accumulo builds against a Hadoop-1 series release
> (1.5.1-SNAPSHOT and 1.6.0-SNAPSHOT build against 1.2.1, and 1.5.0 builds
> against 1.0.4). Last week, the Apache Hadoop community voted to release
> 2.2.0 as GA (general availability) -- in other words, the Apache Hadoop
> community is calling Hadoop-2.2.0 "stable".
>
> As has been discussed across various issues on Jira, this means a few
> different things for Accumulo. Most importantly, this serves as a
> recommendation by us that users should be trying to use Hadoop-2.2.0 with
> Accumulo 1.6.0. This does *not* mean that we do not support Hadoop1 ([2]
> 1.2.1 specifically). Hadoop-1 support would still be "guaranteed" by us for
> 1.6.0.
>
> - Josh
>
> [1] https://issues.apache.org/**jira/browse/ACCUMULO-1419<https://issues.apache.org/jira/browse/ACCUMULO-1419>
> [2] https://issues.apache.org/**jira/browse/ACCUMULO-1643<https://issues.apache.org/jira/browse/ACCUMULO-1643>
>