You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@metron.apache.org by Michael Miklavcic <mi...@gmail.com> on 2019/09/30 21:13:49 UTC

[DISCUSS] Completing HDP 3.1 upgrade feature branch and next steps for Metron 1.x release

Hi all,

We're getting close to wrapping up the feature branch work for upgrading to
HDP 3.1. As previously discussed, this is a backwards incompatible upgrade
and is going to require a 1.x Metron release. There has already been some
discussion on this here -
https://lists.apache.org/thread.html/21e9caf80c0ec4d9db4cb44423befa87cd1e42d327860369a6a13273@%3Cdev.metron.apache.org%3E
.

I want to hash out any remaining tasks we think would be gating factors for
getting the upgrade feature branch merged into master, and subsequently a
1.x Metron release. Based on feedback in the previous dev list discussion
from above, I see a utility for helping automate the upgrade process (i.e.
config backup and restore) being the last major task. Including the upgrade
script, these are the remaining tasks I've identified:

1. Finish Metron upgrade script
2. Upgrade hadoop - PR to follow from
https://github.com/mmiklavc/metron/tree/METRON-2232 (I'm currently
running this up in Vagrant)
3. Cleanup Centos 7 dev environment documentation
4. Finish Kerberos testing with all components now upgraded (depends on
hadoop upgrade being finished) https://github.com/apache/metron/pull/1519.
This should serve as our final acceptance testing for the upgrade - we've
heavily tested the FB on each PR, so this would be a final e2e check of
Metron with Kerberos enabled.
5. Final DISCUSS to merge the FB into master (previous example of what
we did with PCAP here -
https://lists.apache.org/thread.html/94ebc9be23f6f2ec8c53f8f6b71e97d6919baf415caf534e2b25ba9b@%3Cdev.metron.apache.org%3E
)

*Note*: there are a couple tasks in the FB epic Jira around deprecating our
custom code with Kafka spouts. As it is, changing/upgrading this code is
not required for the upgrade. To reduce overall upgrade risk, we want to
consider this work as a follow on in a later point release.

Here are the implications of this upgrade work from a supported components
perspective:

- HDP 2.x is no longer supported - must upgrade
- Centos 6 is no longer supported - must upgrade to Centos 7 (Ambari
relies on binaries shipped by the HDP 3.1 distro, and they do not have any
binaries for Centos 6)
- Solr has also been upgraded from 6.6.2 to 7.4.0 -
https://issues.apache.org/jira/browse/METRON-2225 (No changes to the
Elasticsearch version have been made)

I think that covers everything. Once we get the FB merged, I would expect
we would want to move towards a discussion and vote on releasing Metron
1.0. Does this sound reasonable to everyone?

Best,
Mike

Re: [DISCUSS] Completing HDP 3.1 upgrade feature branch and next steps for Metron 1.x release

Posted by Michael Miklavcic <mi...@gmail.com>.

Hey Otto,

Thanks for taking the time and providing feedback. Some answers/questions
inline below.

On Mon, Sep 30, 2019 at 3:30 PM Otto Fowler <ot...@gmail.com> wrote:

> I think we also need:
>
>    -
>
>    Draft of upgrade guide, reviewed and tried. What is missing? What is
>    hard? Possible refactoring from there.
>

That's part of #1 I mentioned in my list. We're working with Anand and
Mohan on this as well - the expectation is that we will have guidance that
is a combination of scripted backup/upgrade along with manual steps that
should be clear enough to follow. Any platform-specific upgrade tasks (for
a Hadoop cluster, specifically) will be linked to as part of this
documentation.

>    -
>
>    Draft of the new supported versions of XXXXXX for Metron
>

Versions noted in the Jira -
https://issues.apache.org/jira/browse/METRON-2088. Are you thinking a
specific mention in our READMEs beyond what we currently have documented?
Either way, this is easy enough.

>    -
>
>    How are we going to tell current users the change in support? I am not
>    sure burying in the readme is good enough. George isn’t doing blogs
>    anymore, but something like that would be cool, we should also be ready
>    with twitter and other things we can think of. Maybe James can do
> something
>    like that?
>

This will be a major milestone for the project. I agree we should have some
fanfare not only for a 1.0 release, but around notifying users of the
upcoming changes. Wouldn't this just be part of our typical release
announcement, or are you suggesting that in advance of the next release we
make an announcement? Just to be clear, this is going to initiate a 1.0 for
the next release, but there will be a period of time where the feature
branch has been merged into master and we've not yet done a release. I
don't know how much time will pass between those 2 steps, so it's worth
noting.

   -
>
>    Should the docker images we use for building etc be centos 7 too?
>

I don't think that's strictly necessary - the artifacts all work being
built by centos 6 currently and EOL for centos 6 is nov 2020.
https://wiki.centos.org/About/Product

>    -
>
>    Have we removed the centos6 vagrant completely? updated the docs? Is
>    that in the upgrade notes?
>

We will need to add this to the upgrade notes if it is not already present.

>    -
>
>    We have to make sure the full-dev stuff is all set and solid
>

Yes - we're doing all dev testing in our Vagrant box. Anand and Mohan are
performing multi-node cluster testing (along with the upgrade testing)

>    -
>
>    What about the Amazon script and stuff, the documentation for that?
>

I'm honestly not sure how to handle the AWS stuff because it's not free to
do. Is anyone willing to donate some resources for testing this?

>
> I see if I can think of more, hopefully you all will too.
>
>
>
>
> On September 30, 2019 at 17:14:09, Michael Miklavcic (
> michael.miklavcic@gmail.com) wrote:
>
> Hi all,
>
> We're getting close to wrapping up the feature branch work for upgrading to
> HDP 3.1. As previously discussed, this is a backwards incompatible upgrade
> and is going to require a 1.x Metron release. There has already been some
> discussion on this here -
>
> https://lists.apache.org/thread.html/21e9caf80c0ec4d9db4cb44423befa87cd1e42d327860369a6a13273@%3Cdev.metron.apache.org%3E
> .
>
> I want to hash out any remaining tasks we think would be gating factors for
> getting the upgrade feature branch merged into master, and subsequently a
> 1.x Metron release. Based on feedback in the previous dev list discussion
> from above, I see a utility for helping automate the upgrade process (i.e.
> config backup and restore) being the last major task. Including the upgrade
> script, these are the remaining tasks I've identified:
>
> 1. Finish Metron upgrade script
> 2. Upgrade hadoop - PR to follow from
> https://github.com/mmiklavc/metron/tree/METRON-2232 (I'm currently
> running this up in Vagrant)
> 3. Cleanup Centos 7 dev environment documentation
> 4. Finish Kerberos testing with all components now upgraded (depends on
> hadoop upgrade being finished) https://github.com/apache/metron/pull/1519.
> This should serve as our final acceptance testing for the upgrade - we've
> heavily tested the FB on each PR, so this would be a final e2e check of
> Metron with Kerberos enabled.
> 5. Final DISCUSS to merge the FB into master (previous example of what
> we did with PCAP here -
>
> https://lists.apache.org/thread.html/94ebc9be23f6f2ec8c53f8f6b71e97d6919baf415caf534e2b25ba9b@%3Cdev.metron.apache.org%3E
> )
>
> *Note*: there are a couple tasks in the FB epic Jira around deprecating our
> custom code with Kafka spouts. As it is, changing/upgrading this code is
> not required for the upgrade. To reduce overall upgrade risk, we want to
> consider this work as a follow on in a later point release.
>
> Here are the implications of this upgrade work from a supported components
> perspective:
>
> - HDP 2.x is no longer supported - must upgrade
> - Centos 6 is no longer supported - must upgrade to Centos 7 (Ambari
> relies on binaries shipped by the HDP 3.1 distro, and they do not have any
> binaries for Centos 6)
> - Solr has also been upgraded from 6.6.2 to 7.4.0 -
> https://issues.apache.org/jira/browse/METRON-2225 (No changes to the
> Elasticsearch version have been made)
>
> I think that covers everything. Once we get the FB merged, I would expect
> we would want to move towards a discussion and vote on releasing Metron
> 1.0. Does this sound reasonable to everyone?
>
> Best,
> Mike
>

Re: [DISCUSS] Completing HDP 3.1 upgrade feature branch and next steps for Metron 1.x release

Posted by Otto Fowler <ot...@gmail.com>.

I think we also need:

Draft of upgrade guide, reviewed and tried. What is missing? What is
hard? Possible refactoring from there.
-

Draft of the new supported versions of XXXXXX for Metron
-

How are we going to tell current users the change in support? I am not
sure burying in the readme is good enough. George isn’t doing blogs
anymore, but something like that would be cool, we should also be ready
with twitter and other things we can think of. Maybe James can do something
like that?
-

Should the docker images we use for building etc be centos 7 too?
-

Have we removed the centos6 vagrant completely? updated the docs? Is
that in the upgrade notes?
-

We have to make sure the full-dev stuff is all set and solid
-

What about the Amazon script and stuff, the documentation for that?

I see if I can think of more, hopefully you all will too.

On September 30, 2019 at 17:14:09, Michael Miklavcic (
michael.miklavcic@gmail.com) wrote:

Hi all,

Here are the implications of this upgrade work from a supported components
perspective:

I think that covers everything. Once we get the FB merged, I would expect
we would want to move towards a discussion and vote on releasing Metron
1.0. Does this sound reasonable to everyone?

Best,
Mike