You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hawq.apache.org by Ed Espino <es...@apache.org> on 2017/03/31 23:27:07 UTC
Apache HAWQ binary release and Hadoop vendor neutrality
HAWQ,
As we get closer to producing our initial binary release, there is one
important thread (discussed in HAWQ-1421
<https://issues.apache.org/jira/browse/HAWQ-1421> and supporting PR #1208
<https://github.com/apache/incubator-hawq/pull/1208>) I would like to bring
to the forefront. All RPMs produced as part of our initial HAWQ binary
release *MUST* be Hadoop vendor agnostic. We will be able to clearly
demonstrate this by being able to successfully install and run HAWQ and
supporting components (PXF, Ranger Plugin Service, etc) using RPMs produced
from Apache Bigtop and not any specific commercial vendor. This is a core
ASF principle we continue to follow.
Thoughts? Concerns?
-=e
--
*Ed Espino*
Re: Apache HAWQ binary release and Hadoop vendor neutrality
Posted by Roman Shaposhnik <ro...@shaposhnik.org>.
On Mon, Apr 3, 2017 at 6:06 PM, Keaton Adams <ka...@pivotal.io> wrote:
>
> Probably due to a lack of understanding on my part.
>
> Is HAWQ tested with the Open Source Apache Hadoop project? Will it work with plan vanilla Hadoop?
>
> http://hadoop.apache.org/
> http://hadoop.apache.org/releases.html
That's a great question. Lets unpack what it means to be tested with
"plain vanilla Hadoop".
If you look at what kind of artifacts Apache Hadoop community is
releasing you will see
that they consist of source code and a tarball with Java binary
convenice artifacts. E.g.
here's how it looks like for Hadoop 2.7.3:
http://archive.apache.org/dist/hadoop/common/hadoop-2.7.3/
there's no RPMs of DEBs. That happens on Apache Bigtop side when Apache Bigtop
puts together a distribution around Apache Hadoop.
Putting this together means that if HAWQ wants to test its RPMs against
"plain vanilla Hadoop" the only game in town are "plain vanilla Hadoop" RPMs
produced by Bigtop.
At the same time, HAWQ may decide to utilize Hadoop Java tarball somewhere in
its pipeline -- that'd be fine too, but probably won't be super useful
to end users.
Thanks,
Roman.
Re: Apache HAWQ binary release and Hadoop vendor neutrality
Posted by Keaton Adams <ka...@pivotal.io>.
Probably due to a lack of understanding on my part.
Is HAWQ tested with the Open Source Apache Hadoop project? Will it work
with plan vanilla Hadoop?
http://hadoop.apache.org/
http://hadoop.apache.org/releases.html
Thanks again,
K
On Mon, Apr 3, 2017 at 6:50 PM Roman Shaposhnik <ro...@shaposhnik.org>
wrote:
> On Mon, Apr 3, 2017 at 5:48 PM, Keaton Adams <ka...@pivotal.io> wrote:
> > Is it Apache Hadoop compatible? Is it tested with the Apache distro?
>
> I'm not sure I understand the question.
>
> Thanks,
> Roman.
>
--
*Keaton Adams | Advisory Data Engineer | Pivotal*
Direct: 719-309-8960 | Email: kadams@pivotal.io |
<https://www.linkedin.com/in/keaton-adams-3a23785a>
Re: Apache HAWQ binary release and Hadoop vendor neutrality
Posted by Roman Shaposhnik <ro...@shaposhnik.org>.
On Mon, Apr 3, 2017 at 5:48 PM, Keaton Adams <ka...@pivotal.io> wrote:
> Is it Apache Hadoop compatible? Is it tested with the Apache distro?
I'm not sure I understand the question.
Thanks,
Roman.
Re: Apache HAWQ binary release and Hadoop vendor neutrality
Posted by Keaton Adams <ka...@pivotal.io>.
Is it Apache Hadoop compatible? Is it tested with the Apache distro?
Thanks,
Keaton
On Mon, Apr 3, 2017 at 6:44 PM Roman Shaposhnik <ro...@shaposhnik.org>
wrote:
> On Fri, Mar 31, 2017 at 7:16 PM, Ming Li <ml...@pivotal.io> wrote:
> > Hi Ed,
> >
> > What is the relationship between Apache Bigtop and any other commercial
> > vendor?
>
> Starting from Bigtop 1.2 the Bigtop release is ODPi compatible and as such
> provides a great level of compatibility with the vendors participating in
> ODPi.
>
> On top of that, anecdotal evidence suggests that different vendors that
> are not part of ODPi are using Bigtop to build distributions (this includes
> Cloudera, Amazon and Google among the most noticeable ones).
>
> > All features and codes in Bigtop are all included into all other
> > commercial vendor?
>
> In general this is never true as a blanket statement for vendors building
> products out of ASF components. Certain features may be disabled or
> not exposed depending on the kind of use case that a given vendor is
> satisfying.
>
> > If so, then we are pleased to support it because hawq
> > can be used in much wider scope.
>
> That's the main reason to test with Bigtop.
>
> Thanks,
> Roman.
>
--
*Keaton Adams | Advisory Data Engineer | Pivotal*
Direct: 719-309-8960 | Email: kadams@pivotal.io |
<https://www.linkedin.com/in/keaton-adams-3a23785a>
Re: Apache HAWQ binary release and Hadoop vendor neutrality
Posted by Roman Shaposhnik <ro...@shaposhnik.org>.
On Fri, Mar 31, 2017 at 7:16 PM, Ming Li <ml...@pivotal.io> wrote:
> Hi Ed,
>
> What is the relationship between Apache Bigtop and any other commercial
> vendor?
Starting from Bigtop 1.2 the Bigtop release is ODPi compatible and as such
provides a great level of compatibility with the vendors participating in ODPi.
On top of that, anecdotal evidence suggests that different vendors that
are not part of ODPi are using Bigtop to build distributions (this includes
Cloudera, Amazon and Google among the most noticeable ones).
> All features and codes in Bigtop are all included into all other
> commercial vendor?
In general this is never true as a blanket statement for vendors building
products out of ASF components. Certain features may be disabled or
not exposed depending on the kind of use case that a given vendor is
satisfying.
> If so, then we are pleased to support it because hawq
> can be used in much wider scope.
That's the main reason to test with Bigtop.
Thanks,
Roman.
Re: Apache HAWQ binary release and Hadoop vendor neutrality
Posted by Roman Shaposhnik <ro...@shaposhnik.org>.
On Fri, Mar 31, 2017 at 7:16 PM, Ming Li <ml...@pivotal.io> wrote:
> Hi Ed,
>
> What is the relationship between Apache Bigtop and any other commercial
> vendor?
Starting from Bigtop 1.2 the Bigtop release is ODPi compatible and as such
provides a great level of compatibility with the vendors participating in ODPi.
On top of that, anecdotal evidence suggests that different vendors that
are not part of ODPi are using Bigtop to build distributions (this includes
Cloudera, Amazon and Google among the most noticeable ones).
> All features and codes in Bigtop are all included into all other
> commercial vendor?
In general this is never true as a blanket statement for vendors building
products out of ASF components. Certain features may be disabled or
not exposed depending on the kind of use case that a given vendor is
satisfying.
> If so, then we are pleased to support it because hawq
> can be used in much wider scope.
That's the main reason to test with Bigtop.
Thanks,
Roman.
Re: Apache HAWQ binary release and Hadoop vendor neutrality
Posted by Ming Li <ml...@pivotal.io>.
Hi Ed,
What is the relationship between Apache Bigtop and any other commercial
vendor? All features and codes in Bigtop are all included into all other
commercial vendor? If so, then we are pleased to support it because hawq
can be used in much wider scope.
Thanks.
On Sat, Apr 1, 2017 at 7:31 AM, Roman Shaposhnik <ro...@shaposhnik.org>
wrote:
> On Fri, Mar 31, 2017 at 4:27 PM, Ed Espino <es...@apache.org> wrote:
> > HAWQ,
> >
> > As we get closer to producing our initial binary release, there is one
> > important thread (discussed in HAWQ-1421
> > <https://issues.apache.org/jira/browse/HAWQ-1421> and supporting PR
> #1208
> > <https://github.com/apache/incubator-hawq/pull/1208>) I would like to
> bring
> > to the forefront. All RPMs produced as part of our initial HAWQ binary
> > release *MUST* be Hadoop vendor agnostic. We will be able to clearly
> > demonstrate this by being able to successfully install and run HAWQ and
> > supporting components (PXF, Ranger Plugin Service, etc) using RPMs
> produced
> > from Apache Bigtop and not any specific commercial vendor. This is a core
> > ASF principle we continue to follow.
> >
> > Thoughts? Concerns?
>
> This is a great suggestion and perfect timing. Bigtop 1.2.0 release is
> about to
> happen next week, but all of the RPMs and DEBs are already staged. You can
> install them as any other Hadoop distro by using the following repo files:
> https://dist.apache.org/repos/dist/dev/bigtop/1.2.0-RC1/repos/
>
> In fact, Bigtop allows you to do one cool trick on a Docker (and Docker
> compose)
> capable laptop/desktop. All it takes to deploy a Bigtop cluster for
> testing is:
> you download the Bigtop release:
> https://dist.apache.org/repos/dist/dev/bigtop/1.2.0-RC1/
> bigtop-1.2.0-project.tar.gz
> you untar it and do:
> $ cd bigtop-1.2.0/provisioner/docker
> $ ./docker-hadoop.sh -C config_centos7.yaml -c 3
> or
> $ ./docker-hadoop.sh -C config_centos6.yaml -c 3
>
> That will give you a 3 nodes Hadoop cluster running in a
> 3 Docker containers in a manner of like 1 minute. All configured
> and ready to go. In order to interact with nodes in this cluster
> you can open direct shell on them via:
> $ ./docker-hadoop.sh -e <node #> bash
> and then run the usual hadoop command or install extra
> RPMs, etc.
>
> This release of Bigtop also packages Ambari 2.5 which means that
> you can even try deploying HAWQ's management pack.
>
> Thanks,
> Roman.
>
Re: Apache HAWQ binary release and Hadoop vendor neutrality
Posted by Ming Li <ml...@pivotal.io>.
Hi Ed,
What is the relationship between Apache Bigtop and any other commercial
vendor? All features and codes in Bigtop are all included into all other
commercial vendor? If so, then we are pleased to support it because hawq
can be used in much wider scope.
Thanks.
On Sat, Apr 1, 2017 at 7:31 AM, Roman Shaposhnik <ro...@shaposhnik.org>
wrote:
> On Fri, Mar 31, 2017 at 4:27 PM, Ed Espino <es...@apache.org> wrote:
> > HAWQ,
> >
> > As we get closer to producing our initial binary release, there is one
> > important thread (discussed in HAWQ-1421
> > <https://issues.apache.org/jira/browse/HAWQ-1421> and supporting PR
> #1208
> > <https://github.com/apache/incubator-hawq/pull/1208>) I would like to
> bring
> > to the forefront. All RPMs produced as part of our initial HAWQ binary
> > release *MUST* be Hadoop vendor agnostic. We will be able to clearly
> > demonstrate this by being able to successfully install and run HAWQ and
> > supporting components (PXF, Ranger Plugin Service, etc) using RPMs
> produced
> > from Apache Bigtop and not any specific commercial vendor. This is a core
> > ASF principle we continue to follow.
> >
> > Thoughts? Concerns?
>
> This is a great suggestion and perfect timing. Bigtop 1.2.0 release is
> about to
> happen next week, but all of the RPMs and DEBs are already staged. You can
> install them as any other Hadoop distro by using the following repo files:
> https://dist.apache.org/repos/dist/dev/bigtop/1.2.0-RC1/repos/
>
> In fact, Bigtop allows you to do one cool trick on a Docker (and Docker
> compose)
> capable laptop/desktop. All it takes to deploy a Bigtop cluster for
> testing is:
> you download the Bigtop release:
> https://dist.apache.org/repos/dist/dev/bigtop/1.2.0-RC1/
> bigtop-1.2.0-project.tar.gz
> you untar it and do:
> $ cd bigtop-1.2.0/provisioner/docker
> $ ./docker-hadoop.sh -C config_centos7.yaml -c 3
> or
> $ ./docker-hadoop.sh -C config_centos6.yaml -c 3
>
> That will give you a 3 nodes Hadoop cluster running in a
> 3 Docker containers in a manner of like 1 minute. All configured
> and ready to go. In order to interact with nodes in this cluster
> you can open direct shell on them via:
> $ ./docker-hadoop.sh -e <node #> bash
> and then run the usual hadoop command or install extra
> RPMs, etc.
>
> This release of Bigtop also packages Ambari 2.5 which means that
> you can even try deploying HAWQ's management pack.
>
> Thanks,
> Roman.
>
Re: Apache HAWQ binary release and Hadoop vendor neutrality
Posted by Roman Shaposhnik <ro...@shaposhnik.org>.
On Fri, Mar 31, 2017 at 4:27 PM, Ed Espino <es...@apache.org> wrote:
> HAWQ,
>
> As we get closer to producing our initial binary release, there is one
> important thread (discussed in HAWQ-1421
> <https://issues.apache.org/jira/browse/HAWQ-1421> and supporting PR #1208
> <https://github.com/apache/incubator-hawq/pull/1208>) I would like to bring
> to the forefront. All RPMs produced as part of our initial HAWQ binary
> release *MUST* be Hadoop vendor agnostic. We will be able to clearly
> demonstrate this by being able to successfully install and run HAWQ and
> supporting components (PXF, Ranger Plugin Service, etc) using RPMs produced
> from Apache Bigtop and not any specific commercial vendor. This is a core
> ASF principle we continue to follow.
>
> Thoughts? Concerns?
This is a great suggestion and perfect timing. Bigtop 1.2.0 release is about to
happen next week, but all of the RPMs and DEBs are already staged. You can
install them as any other Hadoop distro by using the following repo files:
https://dist.apache.org/repos/dist/dev/bigtop/1.2.0-RC1/repos/
In fact, Bigtop allows you to do one cool trick on a Docker (and Docker compose)
capable laptop/desktop. All it takes to deploy a Bigtop cluster for testing is:
you download the Bigtop release:
https://dist.apache.org/repos/dist/dev/bigtop/1.2.0-RC1/bigtop-1.2.0-project.tar.gz
you untar it and do:
$ cd bigtop-1.2.0/provisioner/docker
$ ./docker-hadoop.sh -C config_centos7.yaml -c 3
or
$ ./docker-hadoop.sh -C config_centos6.yaml -c 3
That will give you a 3 nodes Hadoop cluster running in a
3 Docker containers in a manner of like 1 minute. All configured
and ready to go. In order to interact with nodes in this cluster
you can open direct shell on them via:
$ ./docker-hadoop.sh -e <node #> bash
and then run the usual hadoop command or install extra
RPMs, etc.
This release of Bigtop also packages Ambari 2.5 which means that
you can even try deploying HAWQ's management pack.
Thanks,
Roman.
Re: Apache HAWQ binary release and Hadoop vendor neutrality
Posted by Roman Shaposhnik <ro...@shaposhnik.org>.
On Fri, Mar 31, 2017 at 4:27 PM, Ed Espino <es...@apache.org> wrote:
> HAWQ,
>
> As we get closer to producing our initial binary release, there is one
> important thread (discussed in HAWQ-1421
> <https://issues.apache.org/jira/browse/HAWQ-1421> and supporting PR #1208
> <https://github.com/apache/incubator-hawq/pull/1208>) I would like to bring
> to the forefront. All RPMs produced as part of our initial HAWQ binary
> release *MUST* be Hadoop vendor agnostic. We will be able to clearly
> demonstrate this by being able to successfully install and run HAWQ and
> supporting components (PXF, Ranger Plugin Service, etc) using RPMs produced
> from Apache Bigtop and not any specific commercial vendor. This is a core
> ASF principle we continue to follow.
>
> Thoughts? Concerns?
This is a great suggestion and perfect timing. Bigtop 1.2.0 release is about to
happen next week, but all of the RPMs and DEBs are already staged. You can
install them as any other Hadoop distro by using the following repo files:
https://dist.apache.org/repos/dist/dev/bigtop/1.2.0-RC1/repos/
In fact, Bigtop allows you to do one cool trick on a Docker (and Docker compose)
capable laptop/desktop. All it takes to deploy a Bigtop cluster for testing is:
you download the Bigtop release:
https://dist.apache.org/repos/dist/dev/bigtop/1.2.0-RC1/bigtop-1.2.0-project.tar.gz
you untar it and do:
$ cd bigtop-1.2.0/provisioner/docker
$ ./docker-hadoop.sh -C config_centos7.yaml -c 3
or
$ ./docker-hadoop.sh -C config_centos6.yaml -c 3
That will give you a 3 nodes Hadoop cluster running in a
3 Docker containers in a manner of like 1 minute. All configured
and ready to go. In order to interact with nodes in this cluster
you can open direct shell on them via:
$ ./docker-hadoop.sh -e <node #> bash
and then run the usual hadoop command or install extra
RPMs, etc.
This release of Bigtop also packages Ambari 2.5 which means that
you can even try deploying HAWQ's management pack.
Thanks,
Roman.