Future plans? Hadoop 3.0.

[Антон Чевычалов <ca...@arenadata.io>]

Hi Everyone!

Could someone tell me a vision about Bigtop's future. We are trying to
deliver software faster and intend to make some changes in our system to
allow our customer have new versions as fast as it possible. Does it make
sense for Bigtop Community or we should find our own way?

As first and big move we intend to switch to Hadoop 3.x and start working
with Ignite 2.x. Dead line is April 2018.

In long-term we intend to invest our time in testing and docker (or other
containers) packaging. We are interesting to add new software and care
about Centos and SLES packaging.

-- 
Anton B Chevychalov
Team Lead at Arenadata.io

Re: Future plans? Hadoop 3.0.

[Evans Ye <ev...@apache.org>]

+1 to what roman said.
For Hadoop 3, maybe we can start with a new branch for it since it's might
take quite an effort to reach a qualified state.


2017-12-19 3:29 GMT+08:00 Roman Shaposhnik <ro...@shaposhnik.org>:

> On Mon, Dec 18, 2017 at 2:26 AM, Антон Чевычалов <ca...@arenadata.io> wrote:
> > Hi Everyone!
> >
> > Could someone tell me a vision about Bigtop's future. We are trying to
> > deliver software faster and intend to make some changes in our system to
> > allow our customer have new versions as fast as it possible. Does it make
> > sense for Bigtop Community or we should find our own way?
> >
> > As first and big move we intend to switch to Hadoop 3.x and start working
> > with Ignite 2.x. Dead line is April 2018.
>
> The biggest problem with this kind of switch will be updating all the
> software
> that sits on top of Hadoop and making sure it doesn't regress in
> functionality.
>
> It isn't trivial, but if you're willing to do the work -- I'm sure
> we'll be more than
> happy to review patches.
>
> Thanks,
> Roman.
>

Re: Future plans? Hadoop 3.0.

[Evans Ye <ev...@apache.org>]

I would say you can definitely get +1s and code merges from the Bigtop
community cause eventually we’ll get there sooner or later. But as a
starter we most likely will dev it in a new branch.

As you said this is mutual beneficial. If our collaborative work finally
paid off, we can also have your case as another successful story of Bigtop,
if you agreed to share.

Can you elaborate the use case of having multiple versions? Won’t that be
hard to maintain from clients’ perspective?

Anton Chevychalov <ca...@arenadata.io>於 2017年12月19日 週二，下午3:48寫道：

> > The biggest problem with this kind of switch will be updating all the
> software
> > that sits on top of Hadoop and making sure it doesn't regress in
> functionality.
>
> That is true and it seems to me it will be a lot of pain to switch version.
> That is way it's better to start early.
>
> While our efforts seems to be too risky for Bigtop, I think it is good
> chance to use us as early adopters. But we need to be sure that we are
> moving in same direction. And any help would be nice.
>
> Our plan for next year is a switch from "Ambari way" with one big cluster
> per installation to multiple small clusters with different versions.
> For that paradigm it is nice to have not only solid rock version (like
> 1.2).
>
>
> On 18 December 2017 at 22:29, Roman Shaposhnik <ro...@shaposhnik.org>
> wrote:
>
> > On Mon, Dec 18, 2017 at 2:26 AM, Антон Чевычалов <ca...@arenadata.io>
> wrote:
> > > Hi Everyone!
> > >
> > > Could someone tell me a vision about Bigtop's future. We are trying to
> > > deliver software faster and intend to make some changes in our system
> to
> > > allow our customer have new versions as fast as it possible. Does it
> make
> > > sense for Bigtop Community or we should find our own way?
> > >
> > > As first and big move we intend to switch to Hadoop 3.x and start
> working
> > > with Ignite 2.x. Dead line is April 2018.
> >
> > The biggest problem with this kind of switch will be updating all the
> > software
> > that sits on top of Hadoop and making sure it doesn't regress in
> > functionality.
> >
> > It isn't trivial, but if you're willing to do the work -- I'm sure
> > we'll be more than
> > happy to review patches.
> >
> > Thanks,
> > Roman.
> >
>
>
>
> --
> Anton B Chevychalov
> CI/CD Engineer of ArenaData
>

Re: Future plans? Hadoop 3.0.

["김영우 (Youngwoo Kim)" <wa...@gmail.com>]

Hi Anton,

Good point!

I've been working at Telco(MNO) and semiconductor industry for last 3
years. Some prefers our way -- a kind of devops friendly but in
enterprises, IT staffs want more managed environment for their big data.
Sometimes we do not need the entire stack but management GUI and some
selected components would be sufficient. also some people want to try
bleeding edge technology and hardwares.

I believe your idea is not harmful for this project. It would be rather
complementary way for developers and users.

Thanks,

- Youngwoo


On Tue, Dec 19, 2017 at 4:48 PM, Anton Chevychalov <ca...@arenadata.io> wrote:

> > The biggest problem with this kind of switch will be updating all the
> software
> > that sits on top of Hadoop and making sure it doesn't regress in
> functionality.
>
> That is true and it seems to me it will be a lot of pain to switch version.
> That is way it's better to start early.
>
> While our efforts seems to be too risky for Bigtop, I think it is good
> chance to use us as early adopters. But we need to be sure that we are
> moving in same direction. And any help would be nice.
>
> Our plan for next year is a switch from "Ambari way" with one big cluster
> per installation to multiple small clusters with different versions.
> For that paradigm it is nice to have not only solid rock version (like
> 1.2).
>
>
> On 18 December 2017 at 22:29, Roman Shaposhnik <ro...@shaposhnik.org>
> wrote:
>
> > On Mon, Dec 18, 2017 at 2:26 AM, Антон Чевычалов <ca...@arenadata.io>
> wrote:
> > > Hi Everyone!
> > >
> > > Could someone tell me a vision about Bigtop's future. We are trying to
> > > deliver software faster and intend to make some changes in our system
> to
> > > allow our customer have new versions as fast as it possible. Does it
> make
> > > sense for Bigtop Community or we should find our own way?
> > >
> > > As first and big move we intend to switch to Hadoop 3.x and start
> working
> > > with Ignite 2.x. Dead line is April 2018.
> >
> > The biggest problem with this kind of switch will be updating all the
> > software
> > that sits on top of Hadoop and making sure it doesn't regress in
> > functionality.
> >
> > It isn't trivial, but if you're willing to do the work -- I'm sure
> > we'll be more than
> > happy to review patches.
> >
> > Thanks,
> > Roman.
> >
>
>
>
> --
> Anton B Chevychalov
> CI/CD Engineer of ArenaData
>

Re: Future plans? Hadoop 3.0.

[Anton Chevychalov <ca...@arenadata.io>]

> The biggest problem with this kind of switch will be updating all the
software
> that sits on top of Hadoop and making sure it doesn't regress in
functionality.

That is true and it seems to me it will be a lot of pain to switch version.
That is way it's better to start early.

While our efforts seems to be too risky for Bigtop, I think it is good
chance to use us as early adopters. But we need to be sure that we are
moving in same direction. And any help would be nice.

Our plan for next year is a switch from "Ambari way" with one big cluster
per installation to multiple small clusters with different versions.
For that paradigm it is nice to have not only solid rock version (like 1.2).


On 18 December 2017 at 22:29, Roman Shaposhnik <ro...@shaposhnik.org> wrote:

> On Mon, Dec 18, 2017 at 2:26 AM, Антон Чевычалов <ca...@arenadata.io> wrote:
> > Hi Everyone!
> >
> > Could someone tell me a vision about Bigtop's future. We are trying to
> > deliver software faster and intend to make some changes in our system to
> > allow our customer have new versions as fast as it possible. Does it make
> > sense for Bigtop Community or we should find our own way?
> >
> > As first and big move we intend to switch to Hadoop 3.x and start working
> > with Ignite 2.x. Dead line is April 2018.
>
> The biggest problem with this kind of switch will be updating all the
> software
> that sits on top of Hadoop and making sure it doesn't regress in
> functionality.
>
> It isn't trivial, but if you're willing to do the work -- I'm sure
> we'll be more than
> happy to review patches.
>
> Thanks,
> Roman.
>



-- 
Anton B Chevychalov
CI/CD Engineer of ArenaData

Re: Future plans? Hadoop 3.0.

[Roman Shaposhnik <ro...@shaposhnik.org>]

On Mon, Dec 18, 2017 at 2:26 AM, Антон Чевычалов <ca...@arenadata.io> wrote:
> Hi Everyone!
>
> Could someone tell me a vision about Bigtop's future. We are trying to
> deliver software faster and intend to make some changes in our system to
> allow our customer have new versions as fast as it possible. Does it make
> sense for Bigtop Community or we should find our own way?
>
> As first and big move we intend to switch to Hadoop 3.x and start working
> with Ignite 2.x. Dead line is April 2018.

The biggest problem with this kind of switch will be updating all the software
that sits on top of Hadoop and making sure it doesn't regress in functionality.

It isn't trivial, but if you're willing to do the work -- I'm sure
we'll be more than
happy to review patches.

Thanks,
Roman.

Re: Future plans? Hadoop 3.0.

["김영우 (YoungWoo Kim)" <yw...@apache.org>]

Hi Anton,

Sounds interesting to me! I believe, there is Hadoop 3 on our radar :-)
however, stabilizing infrastructure and supporting features on analytics in
bigtop are important too. at this moment, Hadoop 2.x is the kernel for our
users (at least in 1.3 release) and definitely we'll move to 3!

IMHO, 'experimental branch' or 'alpha release' on Bigtop are a good start
for big move.

Thanks,

- Youngwoo

On Mon, Dec 18, 2017 at 7:26 PM, Антон Чевычалов <ca...@arenadata.io> wrote:

> Hi Everyone!
>
> Could someone tell me a vision about Bigtop's future. We are trying to
> deliver software faster and intend to make some changes in our system to
> allow our customer have new versions as fast as it possible. Does it make
> sense for Bigtop Community or we should find our own way?
>
> As first and big move we intend to switch to Hadoop 3.x and start working
> with Ignite 2.x. Dead line is April 2018.
>
> In long-term we intend to invest our time in testing and docker (or other
> containers) packaging. We are interesting to add new software and care
> about Centos and SLES packaging.
>
> --
> Anton B Chevychalov
> Team Lead at Arenadata.io
>