You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@metron.apache.org by Nick Allen <ni...@nickallen.org> on 2019/04/22 18:50:15 UTC

[DISCUSS] Upgrade to HDP 3.1.0

We currently support running Metron on an HDP 2.6.5
<https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.6.5/bk_release-notes/content/comp_versions.html>
cluster.
I'd like to get Metron updated to run in an HDP 3.1.0
<https://docs.hortonworks.com/HDPDocuments/HDP3/HDP-3.1.0/release-notes/content/comp_versions.html>
cluster.
This provides a number of significant updates to the core platform
components that we depend on like Kafka, HBase, Ambari, etc.

### Feature Branch

I'd like to create a feature branch in which to do this.  This will take a
good amount of effort and multiple PRs. To avoid any impact to master as we
progress through this, a feature branch would make sense.

If you have concerns or interest in this effort, please speak up.  Here are
some relevant discussion points based on what I know so far.

### CentOS 7

CentOS 6 RPMs are no longer distributed for HDP 3.1.0, only CentOS 7 RPMs.
Because of this we will likely need to transition Full Dev over to CentOS
7.  I don't see a downside to doing this since 6 is rather old and I assume
that most users run variants of 7 already anyways.

### HDP 2.6.5

I'd like to try and make these changes backwards compatible with HDP 2.6.5
if possible, but only as long as that does not increase our ongoing
development burden.

For example, if I can simply define a separate build profile for 3.1.0 and
things are generally backwards compatible, then I'm all for maintaining
support for 2.6.5.  On the other hand, I would not want to go as far as
maintaining separate master branches for each.  In my mind the ongoing cost
there is too high.

### HDP 2.5.6

There are some workaround in the code base that were introduced to support HDP
2.5.6
<https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.5.6/bk_release-notes/content/comp_versions.html>
when
we moved to HDP 2.6.5. There are some workarounds specifically for older
versions of Storm like 1.0.x. Rather than maintaining this going forward,
I'd prefer we remove this technical debt and not support anything older
than HDP 2.6.5.




Best,
Nick

Re: [DISCUSS] Upgrade to HDP 3.1.0

Posted by Michael Miklavcic <mi...@gmail.com>.
Hi all,

Ran through multiple rounds of testing on
https://issues.apache.org/jira/browse/METRON-2217 (PR 1483) and it has now
been merged into master. I held off a couple days before merging it because
we were dealing with some other classpath issues in the feature branch with
the revert PR that have now been addressed. Status has been updated on the
Jira epic https://issues.apache.org/jira/browse/METRON-2088. Note - the PRs
that were reverted are simply left as "done." If someone has used a
different/better protocol for handling this in the past, please feel free
to chime in :). At this point, some of the final tasks will likely roll
into the Hadoop upgrade to 3.1.1 task. I think this is where we should
focus our final rounds of testing E2E and with Kerberos (fingers crossed)
enabled.

Best,
Mike



On Tue, Sep 3, 2019 at 11:41 AM James Sirota <js...@apache.org> wrote:

> Thanks for the status update, Mike
>
> 28.08.2019, 16:48, "Michael Miklavcic" <mi...@gmail.com>:
> > Update on current HDP upgrade progress.
> >
> > A separate DISCUSS thread has been spun up around how we should handle
> the
> > upgrade, backwards compatibility issues, semantic versioning, new
> release,
> > etc. here -
> >
> https://lists.apache.org/thread.html/21e9caf80c0ec4d9db4cb44423befa87cd1e42d327860369a6a13273@%3Cdev.metron.apache.org%3E
> >
> > A few PRs have recently gone into the FB from Ryan Merriman
> >
> >    1. https://issues.apache.org/jira/browse/METRON-2169
> >    2. https://issues.apache.org/jira/browse/METRON-2225
> >    3. https://issues.apache.org/jira/browse/METRON-2224
> >
> > He's also currently making some progress towards the Hadoop upgrade, but
> > will be afk for a bit, so this work is expected to be handed off in a
> > branch - https://issues.apache.org/jira/browse/METRON-2232
> >
> > Nick Allen is currently working on reverting the 3 Hbase FB PRs. Ran into
> > some merge conflicts with the Kafka PR, and hit some classpath issues.
> >
> > I merged master into the HBase PR for changing deprecated interfaces, but
> > ran into some classpath trouble with another recent PR change. That's
> been
> > resolved. Ran through a round of testing and all looks good. Looking for
> > some help from Anand and Mohan in getting a good multi-node deployment
> test
> > done on this PR before we merge it in. I've also reached out to Dale
> > Richardson for some additional testing validation
> > https://github.com/apache/metron/pull/1483
> >
> > We'll be looking to merge the current master branch into the FB in the
> next
> > day or so and get the 3 PRs in the FB reverted. Once we have that settled
> > down, we'll have hopefully been able to get PR 1483 into master, which
> > should then be merged again into the FB. From there on, there's a bit of
> > remaining work to polish off HBase. And then it's getting through the
> final
> > Hadoop version upgrade with some final testing with Kerberos, etc.
> >
> > Best,
> > Mike
> >
> > On Fri, Aug 23, 2019 at 10:52 AM Michael Miklavcic <
> > michael.miklavcic@gmail.com> wrote:
> >
> >>  Hi devs,
> >>
> >>  We've been at this a while now, and I want to share an update with you.
> >>  Here's the current HDP 3.1 upgrade Jira again for reference -
> >>  https://issues.apache.org/jira/browse/METRON-2088. Nick, Ryan, and I
> had
> >>  a number of offline conversations in the past week discussing some of
> what
> >>  has been learned during the upgrade as well as how to best address
> some of
> >>  the feedback originating in the recent HBase PR reviews (
> >>  https://github.com/apache/metron/pull/1470#issuecomment-521033037).
> >>
> >>  *Prolog*
> >>
> >>  As this discuss thread shows, there's little debate from the community
> >>  regarding the need to upgrade to HDP 3.1 and our current version of
> Storm
> >>  is being eol'ed (
> >>
> https://lists.apache.org/thread.html/7dbac4e50159ec899d1965505a9844f503130b1526a0e577c705959c@%3Cdev.metron.apache.org%3E
> ).
> >>  Metron has been running on the same version of Hadoop ever since its
> >>  graduation to a TLP a few years ago. While there have been numerous
> >>  individual component/service upgrades: ElasticSearch, Solr, and Storm,
> to
> >>  name a few, this is our first major platform-wide upgrade.
> >>
> >>  The biggest challenge we've run into so far has been dealing with the
> >>  HBase client APIs that were deprecated in HBase 1.x and now removed in
> 2.x.
> >>  We were previously able to depend on HTableInterface and HTables fully
> >>  encapsulating the connection management logic, but this was decoupled
> in
> >>  the new API. It is now left to the user to manage connection lifecycle.
> >>  Nick Allen spent time exploring various options for how to move forward
> >>  with new abstractions that would accommodate a new connection strategy
> as
> >>  well as work within our existing Storm architecture. Fully extracting
> logic
> >>  for managing HBase connections independent of the Tables proved to have
> >>  dramatic ripple effects throughout the architecture (again, refer to
> >>  https://github.com/apache/metron/pull/1470#issuecomment-521033037 for
> >>  more details). We also encountered major changes in the core classes
> used
> >>  by our MockHTable implementation used in the integration tests. These
> two
> >>  problems resulted in interface changes that affected quite a bit of
> code.
> >>
> >>  Reflecting on what we learned from this refactoring push, we explored
> >>  other options to reduce the overall surface area impacted by the API
> >>  change. The main thrust of our work seems to hinge on how to deal with
> the
> >>  new connection management problem. We looked at options for how to
> leverage
> >>  the existing TableProvider abstraction and decided to try out a
> compromise
> >>  approach that allows us to:
> >>
> >>     1. Upgrade as much of the API as possible in the current version of
> >>     HBase against master
> >>     2. Manage connections within the TableProvider abstraction - this
> >>     would have an API feel that is similar to what had been
> encapsulated by the
> >>     HTableInterfaces we rely on currently, and remove a large chunk of
> the code
> >>     that had been necessary to finish the upgrade.
> >>
> >>  *Reducing Scope*
> >>
> >>  We have long-lived connections to HBase that don't need to be
> >>  opened/closed and pooled in the traditional request/response lifecycle
> >>  sense. We know at the time our application spins up how many processes
> and
> >>  threads there will be - this is static for us. I put together a PR (
> >>  https://issues.apache.org/jira/browse/METRON-2217) that migrates
> HTable
> >>  and HTableInterface classes to the new Table API and encapsulates
> >>  connection management within the TableProviders. We had some concerns
> about
> >>  risks associated with managing the connections this way, as opposed to
> >>  using a more robust connection management approach, so I reached out
> to the
> >>  HBase community to get some guidance. The feedback we received suggests
> >>  that managing our connections this way should be sufficient. And the
> HBase
> >>  connection objects are threadsafe, to boot.
> >>
> https://lists.apache.org/thread.html/6b83cd7548efb8c37899063affc97e4c5ce823a13359a49b477e3c07@%3Cdev.hbase.apache.org%3E
> >>
> >>  *A Revised Plan*
> >>
> >>  The alternative HBase client/connection approach is promising, and it
> >>  greatly reduces the overall architectural impact we will need to absorb
> >>  alongside a major upgrade. The following is my proposal after some
> coding
> >>  experiments and numerous conversations with Nick Allen, Ryan Merriman,
> >>  Casey Stella, Otto Fowler, and James Sirota.
> >>
> >>     1. Do as much refactoring in small chunks as possible in master.
> e.g.
> >>     the first phase of the HBase API changes. Reducing the overall
> number of
> >>     variables changing all at the same time in the same place should
> reduce the
> >>     overall risk of the upgrade. Prove stability with what we can in
> master and
> >>     the issues we run into in the FB should be easier to isolate and
> solve.
> >>     2. Target the upgrade feature branch as being a place where we
> >>     primarily have to deal with changes due to classpath problems.
> There will
> >>     be some other necessary code changes, e.g. hbase coprocessor,
> however the
> >>     changes should be well-isolated and narrower in scope.
> >>     3. Ryan and I have had numerous conversations surrounding the Maven
> >>     dependency classpath issues that frequently come up at runtime
> anytime even
> >>     the smallest change to our dependency tree occurs. I won't go into
> those
> >>     details now, but you can see the discussion and history here (
> >>     https://github.com/apache/metron/pull/1436). While there's inherent
> >>     risk in making big changes to our dep management, there is also a
> >>     substantial upside - this PR makes finding classpath problems and
> solving
> >>     them substantially easier. This PR is ready to go in master and
> should
> >>     greatly speed up our ability to rectify and cp problems we
> encounter in the
> >>     feature branch.
> >>     4. Find an analog for our port of the MockHTable (
> >>
> https://github.com/apache/metron/blob/master/metron-platform/metron-hbase/metron-hbase-common/src/test/java/org/apache/metron/hbase/mock/MockHTable.java)
> in
> >>     HBase 2.0.2. Nick has been working on a POC around this alongside
> my work
> >>     on the other API migration and he has been able to get to a point
> with the
> >>     integration tests passing. We had originally hoped this could be
> landed in
> >>     master, but the underlying low-level supporting classes have
> changed and
> >>     are not be forwards compatible the way the Table interface
> >>     and ConnectionFactory class are. We plan to land this in the
> feature branch.
> >>     5. Manage component version changes in an HDP 3.1 profile that gets
> >>     updated as PRs are submitted. This allows the modules to be
> upgraded on a
> >>     per-component basis, while still compiling and allowing tests to
> run,
> >>     without requiring a big bang all-or-nothing upgrade. We would then
> do a
> >>     final reconciliation and deprecation of the old Hadoop versions and
> profile
> >>     at the tail of the FB.
> >>     https://issues.apache.org/jira/browse/METRON-2223
> >>     6. Upgrade Kafka, Storm, Solr, and Zeppelin. PRs from Ryan are up in
> >>     the feature branch now.
> >>     7. Revert the HBase feature branch PRs that have already gone in.
> The
> >>     new approach removes the need for the HBase client changes that have
> >>     already gone in, so we should remove them before polishing off the
> HBase
> >>     upgrade.
> >>     8. Merge in master - including the Maven and HTable migration PRs
> >>     9. Finish HBase upgrade: coprocessor, integration test changes, data
> >>     management
> >>     10. Upgrade Hadoop
> >>     11. Final dependency reconciliation
> >>     https://issues.apache.org/jira/browse/METRON-2223
> >>     12. Acceptance testing
> >>     13. Beers, Profit
> >>
> >>  I think I've covered the major tasks, but if I've missed anything
> please
> >>  reach out.
> >>
> >>  Best,
> >>  Mike Miklavcic
> >>
> >>  On Tue, Apr 23, 2019 at 8:18 AM Nick Allen <ni...@nickallen.org> wrote:
> >>
> >>>  FYI - I opened a ticket to serve as an epic for this work and the
> feature
> >>>  branch.
> >>>
> >>>  https://issues.apache.org/jira/browse/METRON-2088
> >>>
> >>>  On Mon, Apr 22, 2019 at 3:32 PM Michael Miklavcic <
> >>>  michael.miklavcic@gmail.com> wrote:
> >>>
> >>>  > +1 to starting a feature branch for this.
> >>>  > +1 to removing our custom implementations if the newest revs are in
> fact
> >>>  > stable now.
> >>>  >
> >>>  > Regarding the profile option - if it's possible to keep 2.6.5 for a
> bit
> >>>  and
> >>>  > not require separate branches or code trees, this is probably OK.
> >>>  > Otherwise, I'm inclined to take the approach we've taken in the past
> >>>  with
> >>>  > every other upgrade and only support 1 version. I think we should
> >>>  prepare
> >>>  > users for the likelihood that if/when we cut over, there will be no
> more
> >>>  > updates to 2.6.x.
> >>>  >
> >>>  > I talked through this a bit with Nick and Ryan Merriman offline.
> There
> >>>  are
> >>>  > a number of major version revs of components from HDP 2.6 to 3.x
> that
> >>>  are
> >>>  > likely to have backwards compatibility problems. HBase is a big one
> that
> >>>  > comes to mind - I noticed the HTable interface was deprecated while
> >>>  working
> >>>  > through the coprocessor implementation, and Ryan found that it was
> >>>  removed
> >>>  > completely in the new version. That affects our integration tests as
> >>>  well
> >>>  > bc we have a rather large mock implementation of HBase in use that
> is
> >>>  built
> >>>  > around the removed API. We will either need to migrate to the new
> API or
> >>>  > find alternative approach to integration testing with HBase.
> >>>  >
> >>>  > I'll let Nick add more detail in the Epic/Jira and feature branch
> plan,
> >>>  but
> >>>  > here is a sampling of some of what we can expect to require some
> work to
> >>>  > upgrade:
> >>>  >
> >>>  > - Ambari - the current MPack is incompatible with Ambari 2.7.3,
> >>>  however
> >>>  > there isn't a breaking changes document, so we'll have to work
> >>>  through
> >>>  > this
> >>>  > brute force or hopefully find some help from the Ambari community.
> >>>  > - MaaS - YARN major change
> >>>  > - PCAP - HDFS, Kafka
> >>>  > - Indexing - HDFS, Solr
> >>>  > - All topologies - Kafka
> >>>  > - Stellar - HDFS, HBase
> >>>  > - Enrichment - HBase
> >>>  > - Enrichment Coprocessor (the enrichments listing) - HBase
> >>>  > - Integration tests - Kafka and HBase have changed considerably.
> >>>  > - UI, REST - Solr, HDFS, HBase
> >>>  > - Knox
> >>>  > - Kerberos (hopefully this is a kick-the-tires effort, though there
> >>>  is
> >>>  > some possible risk if Ambari and the individual components introduce
> >>>  > changes here)
> >>>  >
> >>>  > Fortunately, Zookeeper appears to have stayed the same across
> versions.
> >>>  It
> >>>  > might be worthwhile to get a chart of the versions for each platform
> >>>  added
> >>>  > to the epic Jira for reference while performing this work.
> >>>  >
> >>>  > Best,
> >>>  > Mike
> >>>  >
> >>>  >
> >>>  > On Mon, Apr 22, 2019, 12:50 PM Nick Allen <ni...@nickallen.org>
> wrote:
> >>>  >
> >>>  > > We currently support running Metron on an HDP 2.6.5
> >>>  > > <
> >>>  > >
> >>>  >
> >>>
> https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.6.5/bk_release-notes/content/comp_versions.html
> >>>  > > >
> >>>  > > cluster.
> >>>  > > I'd like to get Metron updated to run in an HDP 3.1.0
> >>>  > > <
> >>>  > >
> >>>  >
> >>>
> https://docs.hortonworks.com/HDPDocuments/HDP3/HDP-3.1.0/release-notes/content/comp_versions.html
> >>>  > > >
> >>>  > > cluster.
> >>>  > > This provides a number of significant updates to the core platform
> >>>  > > components that we depend on like Kafka, HBase, Ambari, etc.
> >>>  > >
> >>>  > > ### Feature Branch
> >>>  > >
> >>>  > > I'd like to create a feature branch in which to do this. This will
> >>>  take
> >>>  > a
> >>>  > > good amount of effort and multiple PRs. To avoid any impact to
> master
> >>>  as
> >>>  > we
> >>>  > > progress through this, a feature branch would make sense.
> >>>  > >
> >>>  > > If you have concerns or interest in this effort, please speak up.
> >>>  Here
> >>>  > are
> >>>  > > some relevant discussion points based on what I know so far.
> >>>  > >
> >>>  > > ### CentOS 7
> >>>  > >
> >>>  > > CentOS 6 RPMs are no longer distributed for HDP 3.1.0, only
> CentOS 7
> >>>  > RPMs.
> >>>  > > Because of this we will likely need to transition Full Dev over to
> >>>  CentOS
> >>>  > > 7. I don't see a downside to doing this since 6 is rather old and
> I
> >>>  > assume
> >>>  > > that most users run variants of 7 already anyways.
> >>>  > >
> >>>  > > ### HDP 2.6.5
> >>>  > >
> >>>  > > I'd like to try and make these changes backwards compatible with
> HDP
> >>>  > 2.6.5
> >>>  > > if possible, but only as long as that does not increase our
> ongoing
> >>>  > > development burden.
> >>>  > >
> >>>  > > For example, if I can simply define a separate build profile for
> 3.1.0
> >>>  > and
> >>>  > > things are generally backwards compatible, then I'm all for
> >>>  maintaining
> >>>  > > support for 2.6.5. On the other hand, I would not want to go as
> far
> >>>  as
> >>>  > > maintaining separate master branches for each. In my mind the
> ongoing
> >>>  > cost
> >>>  > > there is too high.
> >>>  > >
> >>>  > > ### HDP 2.5.6
> >>>  > >
> >>>  > > There are some workaround in the code base that were introduced to
> >>>  > support
> >>>  > > HDP
> >>>  > > 2.5.6
> >>>  > > <
> >>>  > >
> >>>  >
> >>>
> https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.5.6/bk_release-notes/content/comp_versions.html
> >>>  > > >
> >>>  > > when
> >>>  > > we moved to HDP 2.6.5. There are some workarounds specifically for
> >>>  older
> >>>  > > versions of Storm like 1.0.x. Rather than maintaining this going
> >>>  forward,
> >>>  > > I'd prefer we remove this technical debt and not support anything
> >>>  older
> >>>  > > than HDP 2.6.5.
> >>>  > >
> >>>  > >
> >>>  > >
> >>>  > >
> >>>  > > Best,
> >>>  > > Nick
> >>>  > >
> >>>  >
>
> -------------------
> Thank you,
>
> James Sirota
> PMC- Apache Metron
> jsirota AT apache DOT org
>
>

Re: [DISCUSS] Upgrade to HDP 3.1.0

Posted by James Sirota <js...@apache.org>.
Thanks for the status update, Mike 

28.08.2019, 16:48, "Michael Miklavcic" <mi...@gmail.com>:
> Update on current HDP upgrade progress.
>
> A separate DISCUSS thread has been spun up around how we should handle the
> upgrade, backwards compatibility issues, semantic versioning, new release,
> etc. here -
> https://lists.apache.org/thread.html/21e9caf80c0ec4d9db4cb44423befa87cd1e42d327860369a6a13273@%3Cdev.metron.apache.org%3E
>
> A few PRs have recently gone into the FB from Ryan Merriman
>
>    1. https://issues.apache.org/jira/browse/METRON-2169
>    2. https://issues.apache.org/jira/browse/METRON-2225
>    3. https://issues.apache.org/jira/browse/METRON-2224
>
> He's also currently making some progress towards the Hadoop upgrade, but
> will be afk for a bit, so this work is expected to be handed off in a
> branch - https://issues.apache.org/jira/browse/METRON-2232
>
> Nick Allen is currently working on reverting the 3 Hbase FB PRs. Ran into
> some merge conflicts with the Kafka PR, and hit some classpath issues.
>
> I merged master into the HBase PR for changing deprecated interfaces, but
> ran into some classpath trouble with another recent PR change. That's been
> resolved. Ran through a round of testing and all looks good. Looking for
> some help from Anand and Mohan in getting a good multi-node deployment test
> done on this PR before we merge it in. I've also reached out to Dale
> Richardson for some additional testing validation
> https://github.com/apache/metron/pull/1483
>
> We'll be looking to merge the current master branch into the FB in the next
> day or so and get the 3 PRs in the FB reverted. Once we have that settled
> down, we'll have hopefully been able to get PR 1483 into master, which
> should then be merged again into the FB. From there on, there's a bit of
> remaining work to polish off HBase. And then it's getting through the final
> Hadoop version upgrade with some final testing with Kerberos, etc.
>
> Best,
> Mike
>
> On Fri, Aug 23, 2019 at 10:52 AM Michael Miklavcic <
> michael.miklavcic@gmail.com> wrote:
>
>>  Hi devs,
>>
>>  We've been at this a while now, and I want to share an update with you.
>>  Here's the current HDP 3.1 upgrade Jira again for reference -
>>  https://issues.apache.org/jira/browse/METRON-2088. Nick, Ryan, and I had
>>  a number of offline conversations in the past week discussing some of what
>>  has been learned during the upgrade as well as how to best address some of
>>  the feedback originating in the recent HBase PR reviews (
>>  https://github.com/apache/metron/pull/1470#issuecomment-521033037).
>>
>>  *Prolog*
>>
>>  As this discuss thread shows, there's little debate from the community
>>  regarding the need to upgrade to HDP 3.1 and our current version of Storm
>>  is being eol'ed (
>>  https://lists.apache.org/thread.html/7dbac4e50159ec899d1965505a9844f503130b1526a0e577c705959c@%3Cdev.metron.apache.org%3E).
>>  Metron has been running on the same version of Hadoop ever since its
>>  graduation to a TLP a few years ago. While there have been numerous
>>  individual component/service upgrades: ElasticSearch, Solr, and Storm, to
>>  name a few, this is our first major platform-wide upgrade.
>>
>>  The biggest challenge we've run into so far has been dealing with the
>>  HBase client APIs that were deprecated in HBase 1.x and now removed in 2.x.
>>  We were previously able to depend on HTableInterface and HTables fully
>>  encapsulating the connection management logic, but this was decoupled in
>>  the new API. It is now left to the user to manage connection lifecycle.
>>  Nick Allen spent time exploring various options for how to move forward
>>  with new abstractions that would accommodate a new connection strategy as
>>  well as work within our existing Storm architecture. Fully extracting logic
>>  for managing HBase connections independent of the Tables proved to have
>>  dramatic ripple effects throughout the architecture (again, refer to
>>  https://github.com/apache/metron/pull/1470#issuecomment-521033037 for
>>  more details). We also encountered major changes in the core classes used
>>  by our MockHTable implementation used in the integration tests. These two
>>  problems resulted in interface changes that affected quite a bit of code.
>>
>>  Reflecting on what we learned from this refactoring push, we explored
>>  other options to reduce the overall surface area impacted by the API
>>  change. The main thrust of our work seems to hinge on how to deal with the
>>  new connection management problem. We looked at options for how to leverage
>>  the existing TableProvider abstraction and decided to try out a compromise
>>  approach that allows us to:
>>
>>     1. Upgrade as much of the API as possible in the current version of
>>     HBase against master
>>     2. Manage connections within the TableProvider abstraction - this
>>     would have an API feel that is similar to what had been encapsulated by the
>>     HTableInterfaces we rely on currently, and remove a large chunk of the code
>>     that had been necessary to finish the upgrade.
>>
>>  *Reducing Scope*
>>
>>  We have long-lived connections to HBase that don't need to be
>>  opened/closed and pooled in the traditional request/response lifecycle
>>  sense. We know at the time our application spins up how many processes and
>>  threads there will be - this is static for us. I put together a PR (
>>  https://issues.apache.org/jira/browse/METRON-2217) that migrates HTable
>>  and HTableInterface classes to the new Table API and encapsulates
>>  connection management within the TableProviders. We had some concerns about
>>  risks associated with managing the connections this way, as opposed to
>>  using a more robust connection management approach, so I reached out to the
>>  HBase community to get some guidance. The feedback we received suggests
>>  that managing our connections this way should be sufficient. And the HBase
>>  connection objects are threadsafe, to boot.
>>  https://lists.apache.org/thread.html/6b83cd7548efb8c37899063affc97e4c5ce823a13359a49b477e3c07@%3Cdev.hbase.apache.org%3E
>>
>>  *A Revised Plan*
>>
>>  The alternative HBase client/connection approach is promising, and it
>>  greatly reduces the overall architectural impact we will need to absorb
>>  alongside a major upgrade. The following is my proposal after some coding
>>  experiments and numerous conversations with Nick Allen, Ryan Merriman,
>>  Casey Stella, Otto Fowler, and James Sirota.
>>
>>     1. Do as much refactoring in small chunks as possible in master. e.g.
>>     the first phase of the HBase API changes. Reducing the overall number of
>>     variables changing all at the same time in the same place should reduce the
>>     overall risk of the upgrade. Prove stability with what we can in master and
>>     the issues we run into in the FB should be easier to isolate and solve.
>>     2. Target the upgrade feature branch as being a place where we
>>     primarily have to deal with changes due to classpath problems. There will
>>     be some other necessary code changes, e.g. hbase coprocessor, however the
>>     changes should be well-isolated and narrower in scope.
>>     3. Ryan and I have had numerous conversations surrounding the Maven
>>     dependency classpath issues that frequently come up at runtime anytime even
>>     the smallest change to our dependency tree occurs. I won't go into those
>>     details now, but you can see the discussion and history here (
>>     https://github.com/apache/metron/pull/1436). While there's inherent
>>     risk in making big changes to our dep management, there is also a
>>     substantial upside - this PR makes finding classpath problems and solving
>>     them substantially easier. This PR is ready to go in master and should
>>     greatly speed up our ability to rectify and cp problems we encounter in the
>>     feature branch.
>>     4. Find an analog for our port of the MockHTable (
>>     https://github.com/apache/metron/blob/master/metron-platform/metron-hbase/metron-hbase-common/src/test/java/org/apache/metron/hbase/mock/MockHTable.java) in
>>     HBase 2.0.2. Nick has been working on a POC around this alongside my work
>>     on the other API migration and he has been able to get to a point with the
>>     integration tests passing. We had originally hoped this could be landed in
>>     master, but the underlying low-level supporting classes have changed and
>>     are not be forwards compatible the way the Table interface
>>     and ConnectionFactory class are. We plan to land this in the feature branch.
>>     5. Manage component version changes in an HDP 3.1 profile that gets
>>     updated as PRs are submitted. This allows the modules to be upgraded on a
>>     per-component basis, while still compiling and allowing tests to run,
>>     without requiring a big bang all-or-nothing upgrade. We would then do a
>>     final reconciliation and deprecation of the old Hadoop versions and profile
>>     at the tail of the FB.
>>     https://issues.apache.org/jira/browse/METRON-2223
>>     6. Upgrade Kafka, Storm, Solr, and Zeppelin. PRs from Ryan are up in
>>     the feature branch now.
>>     7. Revert the HBase feature branch PRs that have already gone in. The
>>     new approach removes the need for the HBase client changes that have
>>     already gone in, so we should remove them before polishing off the HBase
>>     upgrade.
>>     8. Merge in master - including the Maven and HTable migration PRs
>>     9. Finish HBase upgrade: coprocessor, integration test changes, data
>>     management
>>     10. Upgrade Hadoop
>>     11. Final dependency reconciliation
>>     https://issues.apache.org/jira/browse/METRON-2223
>>     12. Acceptance testing
>>     13. Beers, Profit
>>
>>  I think I've covered the major tasks, but if I've missed anything please
>>  reach out.
>>
>>  Best,
>>  Mike Miklavcic
>>
>>  On Tue, Apr 23, 2019 at 8:18 AM Nick Allen <ni...@nickallen.org> wrote:
>>
>>>  FYI - I opened a ticket to serve as an epic for this work and the feature
>>>  branch.
>>>
>>>  https://issues.apache.org/jira/browse/METRON-2088
>>>
>>>  On Mon, Apr 22, 2019 at 3:32 PM Michael Miklavcic <
>>>  michael.miklavcic@gmail.com> wrote:
>>>
>>>  > +1 to starting a feature branch for this.
>>>  > +1 to removing our custom implementations if the newest revs are in fact
>>>  > stable now.
>>>  >
>>>  > Regarding the profile option - if it's possible to keep 2.6.5 for a bit
>>>  and
>>>  > not require separate branches or code trees, this is probably OK.
>>>  > Otherwise, I'm inclined to take the approach we've taken in the past
>>>  with
>>>  > every other upgrade and only support 1 version. I think we should
>>>  prepare
>>>  > users for the likelihood that if/when we cut over, there will be no more
>>>  > updates to 2.6.x.
>>>  >
>>>  > I talked through this a bit with Nick and Ryan Merriman offline. There
>>>  are
>>>  > a number of major version revs of components from HDP 2.6 to 3.x that
>>>  are
>>>  > likely to have backwards compatibility problems. HBase is a big one that
>>>  > comes to mind - I noticed the HTable interface was deprecated while
>>>  working
>>>  > through the coprocessor implementation, and Ryan found that it was
>>>  removed
>>>  > completely in the new version. That affects our integration tests as
>>>  well
>>>  > bc we have a rather large mock implementation of HBase in use that is
>>>  built
>>>  > around the removed API. We will either need to migrate to the new API or
>>>  > find alternative approach to integration testing with HBase.
>>>  >
>>>  > I'll let Nick add more detail in the Epic/Jira and feature branch plan,
>>>  but
>>>  > here is a sampling of some of what we can expect to require some work to
>>>  > upgrade:
>>>  >
>>>  > - Ambari - the current MPack is incompatible with Ambari 2.7.3,
>>>  however
>>>  > there isn't a breaking changes document, so we'll have to work
>>>  through
>>>  > this
>>>  > brute force or hopefully find some help from the Ambari community.
>>>  > - MaaS - YARN major change
>>>  > - PCAP - HDFS, Kafka
>>>  > - Indexing - HDFS, Solr
>>>  > - All topologies - Kafka
>>>  > - Stellar - HDFS, HBase
>>>  > - Enrichment - HBase
>>>  > - Enrichment Coprocessor (the enrichments listing) - HBase
>>>  > - Integration tests - Kafka and HBase have changed considerably.
>>>  > - UI, REST - Solr, HDFS, HBase
>>>  > - Knox
>>>  > - Kerberos (hopefully this is a kick-the-tires effort, though there
>>>  is
>>>  > some possible risk if Ambari and the individual components introduce
>>>  > changes here)
>>>  >
>>>  > Fortunately, Zookeeper appears to have stayed the same across versions.
>>>  It
>>>  > might be worthwhile to get a chart of the versions for each platform
>>>  added
>>>  > to the epic Jira for reference while performing this work.
>>>  >
>>>  > Best,
>>>  > Mike
>>>  >
>>>  >
>>>  > On Mon, Apr 22, 2019, 12:50 PM Nick Allen <ni...@nickallen.org> wrote:
>>>  >
>>>  > > We currently support running Metron on an HDP 2.6.5
>>>  > > <
>>>  > >
>>>  >
>>>  https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.6.5/bk_release-notes/content/comp_versions.html
>>>  > > >
>>>  > > cluster.
>>>  > > I'd like to get Metron updated to run in an HDP 3.1.0
>>>  > > <
>>>  > >
>>>  >
>>>  https://docs.hortonworks.com/HDPDocuments/HDP3/HDP-3.1.0/release-notes/content/comp_versions.html
>>>  > > >
>>>  > > cluster.
>>>  > > This provides a number of significant updates to the core platform
>>>  > > components that we depend on like Kafka, HBase, Ambari, etc.
>>>  > >
>>>  > > ### Feature Branch
>>>  > >
>>>  > > I'd like to create a feature branch in which to do this. This will
>>>  take
>>>  > a
>>>  > > good amount of effort and multiple PRs. To avoid any impact to master
>>>  as
>>>  > we
>>>  > > progress through this, a feature branch would make sense.
>>>  > >
>>>  > > If you have concerns or interest in this effort, please speak up.
>>>  Here
>>>  > are
>>>  > > some relevant discussion points based on what I know so far.
>>>  > >
>>>  > > ### CentOS 7
>>>  > >
>>>  > > CentOS 6 RPMs are no longer distributed for HDP 3.1.0, only CentOS 7
>>>  > RPMs.
>>>  > > Because of this we will likely need to transition Full Dev over to
>>>  CentOS
>>>  > > 7. I don't see a downside to doing this since 6 is rather old and I
>>>  > assume
>>>  > > that most users run variants of 7 already anyways.
>>>  > >
>>>  > > ### HDP 2.6.5
>>>  > >
>>>  > > I'd like to try and make these changes backwards compatible with HDP
>>>  > 2.6.5
>>>  > > if possible, but only as long as that does not increase our ongoing
>>>  > > development burden.
>>>  > >
>>>  > > For example, if I can simply define a separate build profile for 3.1.0
>>>  > and
>>>  > > things are generally backwards compatible, then I'm all for
>>>  maintaining
>>>  > > support for 2.6.5. On the other hand, I would not want to go as far
>>>  as
>>>  > > maintaining separate master branches for each. In my mind the ongoing
>>>  > cost
>>>  > > there is too high.
>>>  > >
>>>  > > ### HDP 2.5.6
>>>  > >
>>>  > > There are some workaround in the code base that were introduced to
>>>  > support
>>>  > > HDP
>>>  > > 2.5.6
>>>  > > <
>>>  > >
>>>  >
>>>  https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.5.6/bk_release-notes/content/comp_versions.html
>>>  > > >
>>>  > > when
>>>  > > we moved to HDP 2.6.5. There are some workarounds specifically for
>>>  older
>>>  > > versions of Storm like 1.0.x. Rather than maintaining this going
>>>  forward,
>>>  > > I'd prefer we remove this technical debt and not support anything
>>>  older
>>>  > > than HDP 2.6.5.
>>>  > >
>>>  > >
>>>  > >
>>>  > >
>>>  > > Best,
>>>  > > Nick
>>>  > >
>>>  >

------------------- 
Thank you,

James Sirota
PMC- Apache Metron
jsirota AT apache DOT org


Re: [DISCUSS] Upgrade to HDP 3.1.0

Posted by Michael Miklavcic <mi...@gmail.com>.
Update on current HDP upgrade progress.

A separate DISCUSS thread has been spun up around how we should handle the
upgrade, backwards compatibility issues, semantic versioning, new release,
etc. here -
https://lists.apache.org/thread.html/21e9caf80c0ec4d9db4cb44423befa87cd1e42d327860369a6a13273@%3Cdev.metron.apache.org%3E

A few PRs have recently gone into the FB from Ryan Merriman

   1. https://issues.apache.org/jira/browse/METRON-2169
   2. https://issues.apache.org/jira/browse/METRON-2225
   3. https://issues.apache.org/jira/browse/METRON-2224

He's also currently making some progress towards the Hadoop upgrade, but
will be afk for a bit, so this work is expected to be handed off in a
branch - https://issues.apache.org/jira/browse/METRON-2232

Nick Allen is currently working on reverting the 3 Hbase FB PRs. Ran into
some merge conflicts with the Kafka PR, and hit some classpath issues.

I merged master into the HBase PR for changing deprecated interfaces, but
ran into some classpath trouble with another recent PR change. That's been
resolved. Ran through a round of testing and all looks good. Looking for
some help from Anand and Mohan in getting a good multi-node deployment test
done on this PR before we merge it in. I've also reached out to Dale
Richardson for some additional testing validation
https://github.com/apache/metron/pull/1483

We'll be looking to merge the current master branch into the FB in the next
day or so and get the 3 PRs in the FB reverted. Once we have that settled
down, we'll have hopefully been able to get PR 1483 into master, which
should then be merged again into the FB. From there on, there's a bit of
remaining work to polish off HBase. And then it's getting through the final
Hadoop version upgrade with some final testing with Kerberos, etc.

Best,
Mike


On Fri, Aug 23, 2019 at 10:52 AM Michael Miklavcic <
michael.miklavcic@gmail.com> wrote:

> Hi devs,
>
> We've been at this a while now, and I want to share an update with you.
> Here's the current HDP 3.1 upgrade Jira again for reference -
> https://issues.apache.org/jira/browse/METRON-2088. Nick, Ryan, and I had
> a number of offline conversations in the past week discussing some of what
> has been learned during the upgrade as well as how to best address some of
> the feedback originating in the recent HBase PR reviews (
> https://github.com/apache/metron/pull/1470#issuecomment-521033037).
>
> *Prolog*
>
> As this discuss thread shows, there's little debate from the community
> regarding the need to upgrade to HDP 3.1 and our current version of Storm
> is being eol'ed (
> https://lists.apache.org/thread.html/7dbac4e50159ec899d1965505a9844f503130b1526a0e577c705959c@%3Cdev.metron.apache.org%3E).
> Metron has been running on the same version of Hadoop ever since its
> graduation to a TLP a few years ago. While there have been numerous
> individual component/service upgrades: ElasticSearch, Solr, and Storm, to
> name a few, this is our first major platform-wide upgrade.
>
> The biggest challenge we've run into so far has been dealing with the
> HBase client APIs that were deprecated in HBase 1.x and now removed in 2.x.
> We were previously able to depend on HTableInterface and HTables fully
> encapsulating the connection management logic, but this was decoupled in
> the new API. It is now left to the user to manage connection lifecycle.
> Nick Allen spent time exploring various options for how to move forward
> with new abstractions that would accommodate a new connection strategy as
> well as work within our existing Storm architecture. Fully extracting logic
> for managing HBase connections independent of the Tables proved to have
> dramatic ripple effects throughout the architecture (again, refer to
> https://github.com/apache/metron/pull/1470#issuecomment-521033037 for
> more details). We also encountered major changes in the core classes used
> by our MockHTable implementation used in the integration tests. These two
> problems resulted in interface changes that affected quite a bit of code.
>
> Reflecting on what we learned from this refactoring push, we explored
> other options to reduce the overall surface area impacted by the API
> change. The main thrust of our work seems to hinge on how to deal with the
> new connection management problem. We looked at options for how to leverage
> the existing TableProvider abstraction and decided to try out a compromise
> approach that allows us to:
>
>    1. Upgrade as much of the API as possible in the current version of
>    HBase against master
>    2. Manage connections within the TableProvider abstraction - this
>    would have an API feel that is similar to what had been encapsulated by the
>    HTableInterfaces we rely on currently, and remove a large chunk of the code
>    that had been necessary to finish the upgrade.
>
>
> *Reducing Scope*
>
> We have long-lived connections to HBase that don't need to be
> opened/closed and pooled in the traditional request/response lifecycle
> sense. We know at the time our application spins up how many processes and
> threads there will be - this is static for us. I put together a PR (
> https://issues.apache.org/jira/browse/METRON-2217) that migrates HTable
> and HTableInterface classes to the new Table API and encapsulates
> connection management within the TableProviders. We had some concerns about
> risks associated with managing the connections this way, as opposed to
> using a more robust connection management approach, so I reached out to the
> HBase community to get some guidance. The feedback we received suggests
> that managing our connections this way should be sufficient. And the HBase
> connection objects are threadsafe, to boot.
> https://lists.apache.org/thread.html/6b83cd7548efb8c37899063affc97e4c5ce823a13359a49b477e3c07@%3Cdev.hbase.apache.org%3E
>
> *A Revised Plan*
>
> The alternative HBase client/connection approach is promising, and it
> greatly reduces the overall architectural impact we will need to absorb
> alongside a major upgrade. The following is my proposal after some coding
> experiments and numerous conversations with Nick Allen, Ryan Merriman,
> Casey Stella, Otto Fowler, and James Sirota.
>
>
>    1. Do as much refactoring in small chunks as possible in master. e.g.
>    the first phase of the HBase API changes. Reducing the overall number of
>    variables changing all at the same time in the same place should reduce the
>    overall risk of the upgrade. Prove stability with what we can in master and
>    the issues we run into in the FB should be easier to isolate and solve.
>    2. Target the upgrade feature branch as being a place where we
>    primarily have to deal with changes due to classpath problems. There will
>    be some other necessary code changes, e.g. hbase coprocessor, however the
>    changes should be well-isolated and narrower in scope.
>    3. Ryan and I have had numerous conversations surrounding the Maven
>    dependency classpath issues that frequently come up at runtime anytime even
>    the smallest change to our dependency tree occurs. I won't go into those
>    details now, but you can see the discussion and history here (
>    https://github.com/apache/metron/pull/1436). While there's inherent
>    risk in making big changes to our dep management, there is also a
>    substantial upside - this PR makes finding classpath problems and solving
>    them substantially easier. This PR is ready to go in master and should
>    greatly speed up our ability to rectify and cp problems we encounter in the
>    feature branch.
>    4. Find an analog for our port of the MockHTable (
>    https://github.com/apache/metron/blob/master/metron-platform/metron-hbase/metron-hbase-common/src/test/java/org/apache/metron/hbase/mock/MockHTable.java) in
>    HBase 2.0.2. Nick has been working on a POC around this alongside my work
>    on the other API migration and he has been able to get to a point with the
>    integration tests passing. We had originally hoped this could be landed in
>    master, but the underlying low-level supporting classes have changed and
>    are not be forwards compatible the way the Table interface
>    and ConnectionFactory class are. We plan to land this in the feature branch.
>    5. Manage component version changes in an HDP 3.1 profile that gets
>    updated as PRs are submitted. This allows the modules to be upgraded on a
>    per-component basis, while still compiling and allowing tests to run,
>    without requiring a big bang all-or-nothing upgrade. We would then do a
>    final reconciliation and deprecation of the old Hadoop versions and profile
>    at the tail of the FB.
>    https://issues.apache.org/jira/browse/METRON-2223
>    6. Upgrade Kafka, Storm, Solr, and Zeppelin. PRs from Ryan are up in
>    the feature branch now.
>    7. Revert the HBase feature branch PRs that have already gone in. The
>    new approach removes the need for the HBase client changes that have
>    already gone in, so we should remove them before polishing off the HBase
>    upgrade.
>    8. Merge in master - including the Maven and HTable migration PRs
>    9. Finish HBase upgrade: coprocessor, integration test changes, data
>    management
>    10. Upgrade Hadoop
>    11. Final dependency reconciliation
>    https://issues.apache.org/jira/browse/METRON-2223
>    12. Acceptance testing
>    13. Beers, Profit
>
> I think I've covered the major tasks, but if I've missed anything please
> reach out.
>
> Best,
> Mike Miklavcic
>
>
>
> On Tue, Apr 23, 2019 at 8:18 AM Nick Allen <ni...@nickallen.org> wrote:
>
>> FYI - I opened a ticket to serve as an epic for this work and the feature
>> branch.
>>
>> https://issues.apache.org/jira/browse/METRON-2088
>>
>> On Mon, Apr 22, 2019 at 3:32 PM Michael Miklavcic <
>> michael.miklavcic@gmail.com> wrote:
>>
>> > +1 to starting a feature branch for this.
>> > +1 to removing our custom implementations if the newest revs are in fact
>> > stable now.
>> >
>> > Regarding the profile option - if it's possible to keep 2.6.5 for a bit
>> and
>> > not require separate branches or code trees, this is probably OK.
>> > Otherwise, I'm inclined to take the approach we've taken in the past
>> with
>> > every other upgrade and only support 1 version. I think we should
>> prepare
>> > users for the likelihood that if/when we cut over, there will be no more
>> > updates to 2.6.x.
>> >
>> > I talked through this a bit with Nick and Ryan Merriman offline. There
>> are
>> > a number of major version revs of components from HDP 2.6 to 3.x that
>> are
>> > likely to have backwards compatibility problems. HBase is a big one that
>> > comes to mind - I noticed the HTable interface was deprecated while
>> working
>> > through the coprocessor implementation, and Ryan found that it was
>> removed
>> > completely in the new version. That affects our integration tests as
>> well
>> > bc we have a rather large mock implementation of HBase in use that is
>> built
>> > around the removed API. We will either need to migrate to the new API or
>> > find alternative approach to integration testing with HBase.
>> >
>> > I'll let Nick add more detail in the Epic/Jira and feature branch plan,
>> but
>> > here is a sampling of some of what we can expect to require some work to
>> > upgrade:
>> >
>> >    - Ambari - the current MPack is incompatible with Ambari 2.7.3,
>> however
>> >    there isn't a breaking changes document, so we'll have to work
>> through
>> > this
>> >    brute force or hopefully find some help from the Ambari community.
>> >    - MaaS - YARN major change
>> >    - PCAP - HDFS, Kafka
>> >    - Indexing - HDFS, Solr
>> >    - All topologies - Kafka
>> >    - Stellar - HDFS, HBase
>> >    - Enrichment - HBase
>> >    - Enrichment Coprocessor (the enrichments listing) - HBase
>> >    - Integration tests - Kafka and HBase have changed considerably.
>> >    - UI, REST - Solr, HDFS, HBase
>> >    - Knox
>> >    - Kerberos (hopefully this is a kick-the-tires effort, though there
>> is
>> >    some possible risk if Ambari and the individual components introduce
>> >    changes here)
>> >
>> > Fortunately, Zookeeper appears to have stayed the same across versions.
>> It
>> > might be worthwhile to get a chart of the versions for each platform
>> added
>> > to the epic Jira for reference while performing this work.
>> >
>> > Best,
>> > Mike
>> >
>> >
>> > On Mon, Apr 22, 2019, 12:50 PM Nick Allen <ni...@nickallen.org> wrote:
>> >
>> > > We currently support running Metron on an HDP 2.6.5
>> > > <
>> > >
>> >
>> https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.6.5/bk_release-notes/content/comp_versions.html
>> > > >
>> > > cluster.
>> > > I'd like to get Metron updated to run in an HDP 3.1.0
>> > > <
>> > >
>> >
>> https://docs.hortonworks.com/HDPDocuments/HDP3/HDP-3.1.0/release-notes/content/comp_versions.html
>> > > >
>> > > cluster.
>> > > This provides a number of significant updates to the core platform
>> > > components that we depend on like Kafka, HBase, Ambari, etc.
>> > >
>> > > ### Feature Branch
>> > >
>> > > I'd like to create a feature branch in which to do this.  This will
>> take
>> > a
>> > > good amount of effort and multiple PRs. To avoid any impact to master
>> as
>> > we
>> > > progress through this, a feature branch would make sense.
>> > >
>> > > If you have concerns or interest in this effort, please speak up.
>> Here
>> > are
>> > > some relevant discussion points based on what I know so far.
>> > >
>> > > ### CentOS 7
>> > >
>> > > CentOS 6 RPMs are no longer distributed for HDP 3.1.0, only CentOS 7
>> > RPMs.
>> > > Because of this we will likely need to transition Full Dev over to
>> CentOS
>> > > 7.  I don't see a downside to doing this since 6 is rather old and I
>> > assume
>> > > that most users run variants of 7 already anyways.
>> > >
>> > > ### HDP 2.6.5
>> > >
>> > > I'd like to try and make these changes backwards compatible with HDP
>> > 2.6.5
>> > > if possible, but only as long as that does not increase our ongoing
>> > > development burden.
>> > >
>> > > For example, if I can simply define a separate build profile for 3.1.0
>> > and
>> > > things are generally backwards compatible, then I'm all for
>> maintaining
>> > > support for 2.6.5.  On the other hand, I would not want to go as far
>> as
>> > > maintaining separate master branches for each.  In my mind the ongoing
>> > cost
>> > > there is too high.
>> > >
>> > > ### HDP 2.5.6
>> > >
>> > > There are some workaround in the code base that were introduced to
>> > support
>> > > HDP
>> > > 2.5.6
>> > > <
>> > >
>> >
>> https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.5.6/bk_release-notes/content/comp_versions.html
>> > > >
>> > > when
>> > > we moved to HDP 2.6.5. There are some workarounds specifically for
>> older
>> > > versions of Storm like 1.0.x. Rather than maintaining this going
>> forward,
>> > > I'd prefer we remove this technical debt and not support anything
>> older
>> > > than HDP 2.6.5.
>> > >
>> > >
>> > >
>> > >
>> > > Best,
>> > > Nick
>> > >
>> >
>>
>

Re: [DISCUSS] Upgrade to HDP 3.1.0

Posted by Michael Miklavcic <mi...@gmail.com>.
Hi devs,

We've been at this a while now, and I want to share an update with you.
Here's the current HDP 3.1 upgrade Jira again for reference -
https://issues.apache.org/jira/browse/METRON-2088. Nick, Ryan, and I had a
number of offline conversations in the past week discussing some of what
has been learned during the upgrade as well as how to best address some of
the feedback originating in the recent HBase PR reviews (
https://github.com/apache/metron/pull/1470#issuecomment-521033037).

*Prolog*

As this discuss thread shows, there's little debate from the community
regarding the need to upgrade to HDP 3.1 and our current version of Storm
is being eol'ed (
https://lists.apache.org/thread.html/7dbac4e50159ec899d1965505a9844f503130b1526a0e577c705959c@%3Cdev.metron.apache.org%3E).
Metron has been running on the same version of Hadoop ever since its
graduation to a TLP a few years ago. While there have been numerous
individual component/service upgrades: ElasticSearch, Solr, and Storm, to
name a few, this is our first major platform-wide upgrade.

The biggest challenge we've run into so far has been dealing with the HBase
client APIs that were deprecated in HBase 1.x and now removed in 2.x. We
were previously able to depend on HTableInterface and HTables fully
encapsulating the connection management logic, but this was decoupled in
the new API. It is now left to the user to manage connection lifecycle.
Nick Allen spent time exploring various options for how to move forward
with new abstractions that would accommodate a new connection strategy as
well as work within our existing Storm architecture. Fully extracting logic
for managing HBase connections independent of the Tables proved to have
dramatic ripple effects throughout the architecture (again, refer to
https://github.com/apache/metron/pull/1470#issuecomment-521033037 for more
details). We also encountered major changes in the core classes used by our
MockHTable implementation used in the integration tests. These two problems
resulted in interface changes that affected quite a bit of code.

Reflecting on what we learned from this refactoring push, we explored other
options to reduce the overall surface area impacted by the API change. The
main thrust of our work seems to hinge on how to deal with the new
connection management problem. We looked at options for how to leverage the
existing TableProvider abstraction and decided to try out a compromise
approach that allows us to:

   1. Upgrade as much of the API as possible in the current version of
   HBase against master
   2. Manage connections within the TableProvider abstraction - this would
   have an API feel that is similar to what had been encapsulated by the
   HTableInterfaces we rely on currently, and remove a large chunk of the code
   that had been necessary to finish the upgrade.


*Reducing Scope*

We have long-lived connections to HBase that don't need to be opened/closed
and pooled in the traditional request/response lifecycle sense. We know at
the time our application spins up how many processes and threads there will
be - this is static for us. I put together a PR (
https://issues.apache.org/jira/browse/METRON-2217) that migrates HTable and
HTableInterface classes to the new Table API and encapsulates connection
management within the TableProviders. We had some concerns about risks
associated with managing the connections this way, as opposed to using a
more robust connection management approach, so I reached out to the HBase
community to get some guidance. The feedback we received suggests that
managing our connections this way should be sufficient. And the HBase
connection objects are threadsafe, to boot.
https://lists.apache.org/thread.html/6b83cd7548efb8c37899063affc97e4c5ce823a13359a49b477e3c07@%3Cdev.hbase.apache.org%3E

*A Revised Plan*

The alternative HBase client/connection approach is promising, and it
greatly reduces the overall architectural impact we will need to absorb
alongside a major upgrade. The following is my proposal after some coding
experiments and numerous conversations with Nick Allen, Ryan Merriman,
Casey Stella, Otto Fowler, and James Sirota.


   1. Do as much refactoring in small chunks as possible in master. e.g.
   the first phase of the HBase API changes. Reducing the overall number of
   variables changing all at the same time in the same place should reduce the
   overall risk of the upgrade. Prove stability with what we can in master and
   the issues we run into in the FB should be easier to isolate and solve.
   2. Target the upgrade feature branch as being a place where we primarily
   have to deal with changes due to classpath problems. There will be some
   other necessary code changes, e.g. hbase coprocessor, however the changes
   should be well-isolated and narrower in scope.
   3. Ryan and I have had numerous conversations surrounding the Maven
   dependency classpath issues that frequently come up at runtime anytime even
   the smallest change to our dependency tree occurs. I won't go into those
   details now, but you can see the discussion and history here (
   https://github.com/apache/metron/pull/1436). While there's inherent risk
   in making big changes to our dep management, there is also a substantial
   upside - this PR makes finding classpath problems and solving them
   substantially easier. This PR is ready to go in master and should greatly
   speed up our ability to rectify and cp problems we encounter in the feature
   branch.
   4. Find an analog for our port of the MockHTable (
   https://github.com/apache/metron/blob/master/metron-platform/metron-hbase/metron-hbase-common/src/test/java/org/apache/metron/hbase/mock/MockHTable.java)
in
   HBase 2.0.2. Nick has been working on a POC around this alongside my work
   on the other API migration and he has been able to get to a point with the
   integration tests passing. We had originally hoped this could be landed in
   master, but the underlying low-level supporting classes have changed and
   are not be forwards compatible the way the Table interface
   and ConnectionFactory class are. We plan to land this in the feature branch.
   5. Manage component version changes in an HDP 3.1 profile that gets
   updated as PRs are submitted. This allows the modules to be upgraded on a
   per-component basis, while still compiling and allowing tests to run,
   without requiring a big bang all-or-nothing upgrade. We would then do a
   final reconciliation and deprecation of the old Hadoop versions and profile
   at the tail of the FB. https://issues.apache.org/jira/browse/METRON-2223
   6. Upgrade Kafka, Storm, Solr, and Zeppelin. PRs from Ryan are up in the
   feature branch now.
   7. Revert the HBase feature branch PRs that have already gone in. The
   new approach removes the need for the HBase client changes that have
   already gone in, so we should remove them before polishing off the HBase
   upgrade.
   8. Merge in master - including the Maven and HTable migration PRs
   9. Finish HBase upgrade: coprocessor, integration test changes, data
   management
   10. Upgrade Hadoop
   11. Final dependency reconciliation
   https://issues.apache.org/jira/browse/METRON-2223
   12. Acceptance testing
   13. Beers, Profit

I think I've covered the major tasks, but if I've missed anything please
reach out.

Best,
Mike Miklavcic



On Tue, Apr 23, 2019 at 8:18 AM Nick Allen <ni...@nickallen.org> wrote:

> FYI - I opened a ticket to serve as an epic for this work and the feature
> branch.
>
> https://issues.apache.org/jira/browse/METRON-2088
>
> On Mon, Apr 22, 2019 at 3:32 PM Michael Miklavcic <
> michael.miklavcic@gmail.com> wrote:
>
> > +1 to starting a feature branch for this.
> > +1 to removing our custom implementations if the newest revs are in fact
> > stable now.
> >
> > Regarding the profile option - if it's possible to keep 2.6.5 for a bit
> and
> > not require separate branches or code trees, this is probably OK.
> > Otherwise, I'm inclined to take the approach we've taken in the past with
> > every other upgrade and only support 1 version. I think we should prepare
> > users for the likelihood that if/when we cut over, there will be no more
> > updates to 2.6.x.
> >
> > I talked through this a bit with Nick and Ryan Merriman offline. There
> are
> > a number of major version revs of components from HDP 2.6 to 3.x that are
> > likely to have backwards compatibility problems. HBase is a big one that
> > comes to mind - I noticed the HTable interface was deprecated while
> working
> > through the coprocessor implementation, and Ryan found that it was
> removed
> > completely in the new version. That affects our integration tests as well
> > bc we have a rather large mock implementation of HBase in use that is
> built
> > around the removed API. We will either need to migrate to the new API or
> > find alternative approach to integration testing with HBase.
> >
> > I'll let Nick add more detail in the Epic/Jira and feature branch plan,
> but
> > here is a sampling of some of what we can expect to require some work to
> > upgrade:
> >
> >    - Ambari - the current MPack is incompatible with Ambari 2.7.3,
> however
> >    there isn't a breaking changes document, so we'll have to work through
> > this
> >    brute force or hopefully find some help from the Ambari community.
> >    - MaaS - YARN major change
> >    - PCAP - HDFS, Kafka
> >    - Indexing - HDFS, Solr
> >    - All topologies - Kafka
> >    - Stellar - HDFS, HBase
> >    - Enrichment - HBase
> >    - Enrichment Coprocessor (the enrichments listing) - HBase
> >    - Integration tests - Kafka and HBase have changed considerably.
> >    - UI, REST - Solr, HDFS, HBase
> >    - Knox
> >    - Kerberos (hopefully this is a kick-the-tires effort, though there is
> >    some possible risk if Ambari and the individual components introduce
> >    changes here)
> >
> > Fortunately, Zookeeper appears to have stayed the same across versions.
> It
> > might be worthwhile to get a chart of the versions for each platform
> added
> > to the epic Jira for reference while performing this work.
> >
> > Best,
> > Mike
> >
> >
> > On Mon, Apr 22, 2019, 12:50 PM Nick Allen <ni...@nickallen.org> wrote:
> >
> > > We currently support running Metron on an HDP 2.6.5
> > > <
> > >
> >
> https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.6.5/bk_release-notes/content/comp_versions.html
> > > >
> > > cluster.
> > > I'd like to get Metron updated to run in an HDP 3.1.0
> > > <
> > >
> >
> https://docs.hortonworks.com/HDPDocuments/HDP3/HDP-3.1.0/release-notes/content/comp_versions.html
> > > >
> > > cluster.
> > > This provides a number of significant updates to the core platform
> > > components that we depend on like Kafka, HBase, Ambari, etc.
> > >
> > > ### Feature Branch
> > >
> > > I'd like to create a feature branch in which to do this.  This will
> take
> > a
> > > good amount of effort and multiple PRs. To avoid any impact to master
> as
> > we
> > > progress through this, a feature branch would make sense.
> > >
> > > If you have concerns or interest in this effort, please speak up.  Here
> > are
> > > some relevant discussion points based on what I know so far.
> > >
> > > ### CentOS 7
> > >
> > > CentOS 6 RPMs are no longer distributed for HDP 3.1.0, only CentOS 7
> > RPMs.
> > > Because of this we will likely need to transition Full Dev over to
> CentOS
> > > 7.  I don't see a downside to doing this since 6 is rather old and I
> > assume
> > > that most users run variants of 7 already anyways.
> > >
> > > ### HDP 2.6.5
> > >
> > > I'd like to try and make these changes backwards compatible with HDP
> > 2.6.5
> > > if possible, but only as long as that does not increase our ongoing
> > > development burden.
> > >
> > > For example, if I can simply define a separate build profile for 3.1.0
> > and
> > > things are generally backwards compatible, then I'm all for maintaining
> > > support for 2.6.5.  On the other hand, I would not want to go as far as
> > > maintaining separate master branches for each.  In my mind the ongoing
> > cost
> > > there is too high.
> > >
> > > ### HDP 2.5.6
> > >
> > > There are some workaround in the code base that were introduced to
> > support
> > > HDP
> > > 2.5.6
> > > <
> > >
> >
> https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.5.6/bk_release-notes/content/comp_versions.html
> > > >
> > > when
> > > we moved to HDP 2.6.5. There are some workarounds specifically for
> older
> > > versions of Storm like 1.0.x. Rather than maintaining this going
> forward,
> > > I'd prefer we remove this technical debt and not support anything older
> > > than HDP 2.6.5.
> > >
> > >
> > >
> > >
> > > Best,
> > > Nick
> > >
> >
>

Re: [DISCUSS] Upgrade to HDP 3.1.0

Posted by Nick Allen <ni...@nickallen.org>.
FYI - I opened a ticket to serve as an epic for this work and the feature
branch.

https://issues.apache.org/jira/browse/METRON-2088

On Mon, Apr 22, 2019 at 3:32 PM Michael Miklavcic <
michael.miklavcic@gmail.com> wrote:

> +1 to starting a feature branch for this.
> +1 to removing our custom implementations if the newest revs are in fact
> stable now.
>
> Regarding the profile option - if it's possible to keep 2.6.5 for a bit and
> not require separate branches or code trees, this is probably OK.
> Otherwise, I'm inclined to take the approach we've taken in the past with
> every other upgrade and only support 1 version. I think we should prepare
> users for the likelihood that if/when we cut over, there will be no more
> updates to 2.6.x.
>
> I talked through this a bit with Nick and Ryan Merriman offline. There are
> a number of major version revs of components from HDP 2.6 to 3.x that are
> likely to have backwards compatibility problems. HBase is a big one that
> comes to mind - I noticed the HTable interface was deprecated while working
> through the coprocessor implementation, and Ryan found that it was removed
> completely in the new version. That affects our integration tests as well
> bc we have a rather large mock implementation of HBase in use that is built
> around the removed API. We will either need to migrate to the new API or
> find alternative approach to integration testing with HBase.
>
> I'll let Nick add more detail in the Epic/Jira and feature branch plan, but
> here is a sampling of some of what we can expect to require some work to
> upgrade:
>
>    - Ambari - the current MPack is incompatible with Ambari 2.7.3, however
>    there isn't a breaking changes document, so we'll have to work through
> this
>    brute force or hopefully find some help from the Ambari community.
>    - MaaS - YARN major change
>    - PCAP - HDFS, Kafka
>    - Indexing - HDFS, Solr
>    - All topologies - Kafka
>    - Stellar - HDFS, HBase
>    - Enrichment - HBase
>    - Enrichment Coprocessor (the enrichments listing) - HBase
>    - Integration tests - Kafka and HBase have changed considerably.
>    - UI, REST - Solr, HDFS, HBase
>    - Knox
>    - Kerberos (hopefully this is a kick-the-tires effort, though there is
>    some possible risk if Ambari and the individual components introduce
>    changes here)
>
> Fortunately, Zookeeper appears to have stayed the same across versions. It
> might be worthwhile to get a chart of the versions for each platform added
> to the epic Jira for reference while performing this work.
>
> Best,
> Mike
>
>
> On Mon, Apr 22, 2019, 12:50 PM Nick Allen <ni...@nickallen.org> wrote:
>
> > We currently support running Metron on an HDP 2.6.5
> > <
> >
> https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.6.5/bk_release-notes/content/comp_versions.html
> > >
> > cluster.
> > I'd like to get Metron updated to run in an HDP 3.1.0
> > <
> >
> https://docs.hortonworks.com/HDPDocuments/HDP3/HDP-3.1.0/release-notes/content/comp_versions.html
> > >
> > cluster.
> > This provides a number of significant updates to the core platform
> > components that we depend on like Kafka, HBase, Ambari, etc.
> >
> > ### Feature Branch
> >
> > I'd like to create a feature branch in which to do this.  This will take
> a
> > good amount of effort and multiple PRs. To avoid any impact to master as
> we
> > progress through this, a feature branch would make sense.
> >
> > If you have concerns or interest in this effort, please speak up.  Here
> are
> > some relevant discussion points based on what I know so far.
> >
> > ### CentOS 7
> >
> > CentOS 6 RPMs are no longer distributed for HDP 3.1.0, only CentOS 7
> RPMs.
> > Because of this we will likely need to transition Full Dev over to CentOS
> > 7.  I don't see a downside to doing this since 6 is rather old and I
> assume
> > that most users run variants of 7 already anyways.
> >
> > ### HDP 2.6.5
> >
> > I'd like to try and make these changes backwards compatible with HDP
> 2.6.5
> > if possible, but only as long as that does not increase our ongoing
> > development burden.
> >
> > For example, if I can simply define a separate build profile for 3.1.0
> and
> > things are generally backwards compatible, then I'm all for maintaining
> > support for 2.6.5.  On the other hand, I would not want to go as far as
> > maintaining separate master branches for each.  In my mind the ongoing
> cost
> > there is too high.
> >
> > ### HDP 2.5.6
> >
> > There are some workaround in the code base that were introduced to
> support
> > HDP
> > 2.5.6
> > <
> >
> https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.5.6/bk_release-notes/content/comp_versions.html
> > >
> > when
> > we moved to HDP 2.6.5. There are some workarounds specifically for older
> > versions of Storm like 1.0.x. Rather than maintaining this going forward,
> > I'd prefer we remove this technical debt and not support anything older
> > than HDP 2.6.5.
> >
> >
> >
> >
> > Best,
> > Nick
> >
>

Re: [DISCUSS] Upgrade to HDP 3.1.0

Posted by Michael Miklavcic <mi...@gmail.com>.
+1 to starting a feature branch for this.
+1 to removing our custom implementations if the newest revs are in fact
stable now.

Regarding the profile option - if it's possible to keep 2.6.5 for a bit and
not require separate branches or code trees, this is probably OK.
Otherwise, I'm inclined to take the approach we've taken in the past with
every other upgrade and only support 1 version. I think we should prepare
users for the likelihood that if/when we cut over, there will be no more
updates to 2.6.x.

I talked through this a bit with Nick and Ryan Merriman offline. There are
a number of major version revs of components from HDP 2.6 to 3.x that are
likely to have backwards compatibility problems. HBase is a big one that
comes to mind - I noticed the HTable interface was deprecated while working
through the coprocessor implementation, and Ryan found that it was removed
completely in the new version. That affects our integration tests as well
bc we have a rather large mock implementation of HBase in use that is built
around the removed API. We will either need to migrate to the new API or
find alternative approach to integration testing with HBase.

I'll let Nick add more detail in the Epic/Jira and feature branch plan, but
here is a sampling of some of what we can expect to require some work to
upgrade:

   - Ambari - the current MPack is incompatible with Ambari 2.7.3, however
   there isn't a breaking changes document, so we'll have to work through this
   brute force or hopefully find some help from the Ambari community.
   - MaaS - YARN major change
   - PCAP - HDFS, Kafka
   - Indexing - HDFS, Solr
   - All topologies - Kafka
   - Stellar - HDFS, HBase
   - Enrichment - HBase
   - Enrichment Coprocessor (the enrichments listing) - HBase
   - Integration tests - Kafka and HBase have changed considerably.
   - UI, REST - Solr, HDFS, HBase
   - Knox
   - Kerberos (hopefully this is a kick-the-tires effort, though there is
   some possible risk if Ambari and the individual components introduce
   changes here)

Fortunately, Zookeeper appears to have stayed the same across versions. It
might be worthwhile to get a chart of the versions for each platform added
to the epic Jira for reference while performing this work.

Best,
Mike


On Mon, Apr 22, 2019, 12:50 PM Nick Allen <ni...@nickallen.org> wrote:

> We currently support running Metron on an HDP 2.6.5
> <
> https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.6.5/bk_release-notes/content/comp_versions.html
> >
> cluster.
> I'd like to get Metron updated to run in an HDP 3.1.0
> <
> https://docs.hortonworks.com/HDPDocuments/HDP3/HDP-3.1.0/release-notes/content/comp_versions.html
> >
> cluster.
> This provides a number of significant updates to the core platform
> components that we depend on like Kafka, HBase, Ambari, etc.
>
> ### Feature Branch
>
> I'd like to create a feature branch in which to do this.  This will take a
> good amount of effort and multiple PRs. To avoid any impact to master as we
> progress through this, a feature branch would make sense.
>
> If you have concerns or interest in this effort, please speak up.  Here are
> some relevant discussion points based on what I know so far.
>
> ### CentOS 7
>
> CentOS 6 RPMs are no longer distributed for HDP 3.1.0, only CentOS 7 RPMs.
> Because of this we will likely need to transition Full Dev over to CentOS
> 7.  I don't see a downside to doing this since 6 is rather old and I assume
> that most users run variants of 7 already anyways.
>
> ### HDP 2.6.5
>
> I'd like to try and make these changes backwards compatible with HDP 2.6.5
> if possible, but only as long as that does not increase our ongoing
> development burden.
>
> For example, if I can simply define a separate build profile for 3.1.0 and
> things are generally backwards compatible, then I'm all for maintaining
> support for 2.6.5.  On the other hand, I would not want to go as far as
> maintaining separate master branches for each.  In my mind the ongoing cost
> there is too high.
>
> ### HDP 2.5.6
>
> There are some workaround in the code base that were introduced to support
> HDP
> 2.5.6
> <
> https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.5.6/bk_release-notes/content/comp_versions.html
> >
> when
> we moved to HDP 2.6.5. There are some workarounds specifically for older
> versions of Storm like 1.0.x. Rather than maintaining this going forward,
> I'd prefer we remove this technical debt and not support anything older
> than HDP 2.6.5.
>
>
>
>
> Best,
> Nick
>