[DISCUSS] HDP 3.1 Upgrade and release strategy

[Michael Miklavcic <mi...@gmail.com>]

Hi devs and users,

Some questions were asked in the Slack channel about our ongoing HDP/Hadoop
upgrade and I'd like to get a discussion rolling. The original Hadoop
upgrade discuss thread can be found here
https://lists.apache.org/thread.html/37cc29648f0592cc39d3c78a0d07fce38521bdbbc4cf40e022a7a8ea@%3Cdev.metron.apache.org%3E

The major issue and impact with upgrading the Hadoop platform is that there
are breaking changes. Code that runs on HDP 3.1 will not run on 2.x. Here
is a sampling of core components we depend on that we know of so far that
are not backwards compatible:

   1. The Core OS - we currently base our builds and test deployment off of
   artifacts pulled from HDP. The latest rev of HDP no longer ships RPMs for
   Centos 6, which means we need to upgrade to Centos 7
   2. Ambari
   3. HBase

This differs from individual components we've upgraded in the past in that
our code could still be deployed on the old as well as new version of the
component in a backwards compatible way. Based on semantic versioning, I
don't know if we can introduce this level of change in a point release,
which is the reason for kicking off this discussion. In the past, users and
developers in the community have suggested that they are -1 on a 1.x
release that does not provide an upgrade
https://lists.apache.org/thread.html/eb1a8df2d0a6a79c5d50540d1fdbf215ec83d831ff15d3117c2592cc@%3Cdev.metron.apache.org%3E
.

Is there a way we can avoid a 1.x release? If we do need 1.x, do we still
see upgrades as a gating function? The main issue is that this has the
potential to drag out the upgrade and further couple it with other
features. And with Storm 1.x being eol'ed, I'm not sure this is something
we can wait much longer for. I'll think on this and send out my own
thoughts once folks have had a chance to review.

Best,
Mike Miklavcic
Apache Metron, PMC, committer

Re: [DISCUSS] HDP 3.1 Upgrade and release strategy

[James Sirota <js...@apache.org>]

Given what Simon pointed out, where in your view are the current feature gaps with respect to that?  We can back up the zookeeper config, any files we need on HDFS, and any custom parsers/enrichers that were produced.  The rest of it is pretty much HDP-pecific upgrade.  In my view, backing up kafka topic configs, HDFS configs, ES configs, etc., is outside of the scope of Metron and is more of a platform function.

Thanks,
James 

27.08.2019, 15:56, "Zeolla@GMail.com" <ze...@gmail.com>:
> I agree that having a scripted approach for backup and restore of Metron
> configs should be necessary for such a large change/upgrade process.
> Having been through this many times in the past I can tell you that the
> difficulty of upgrading (whether perceived or actual) holds back adoption
> of the platform.
>
> Jon Zeolla
>
> On Tue, Aug 27, 2019, 5:25 PM Simon Elliston Ball <
> simon@simonellistonball.com> wrote:
>
>>  Not sure it’s in the scope of the project to handle the HDP upgrade as
>>  well, I would scope it to metron config only, and point at the extensive
>>  upgrade capability of Ambari, rather than us trying to recreate the way the
>>  distribution works.
>>
>>  Simon
>>
>>  On Tue, 27 Aug 2019 at 22:23, Otto Fowler <ot...@gmail.com> wrote:
>>
>>  > If anyone can think of the things that need to be backed up, please
>>  > comment the jira.
>>  >
>>  >
>>  >
>>  >
>>  > On August 27, 2019 at 17:07:20, Otto Fowler (ottobackwards@gmail.com)
>>  > wrote:
>>  >
>>  > Good idea METRON–2239 [blocker].
>>  >
>>  >
>>  >
>>  > On August 27, 2019 at 16:30:13, Simon Elliston Ball (
>>  > simon@simonellistonball.com) wrote:
>>  >
>>  > You could always submit a Jira :)
>>  >
>>  > On Tue, 27 Aug 2019 at 21:27, Otto Fowler <ot...@gmail.com>
>>  wrote:
>>  >
>>  >> You are right, that is much better than backup_metron_configs.sh.
>>  >>
>>  >>
>>  >>
>>  >> On August 27, 2019 at 16:05:38, Simon Elliston Ball (
>>  >> simon@simonellistonball.com) wrote:
>>  >>
>>  >> You can do this with zk_load_configs and Ambari’s blueprint api, so we
>>  >> kinda already do.
>>  >>
>>  >> Simon
>>  >>
>>  >> On Tue, 27 Aug 2019 at 20:19, Otto Fowler <ot...@gmail.com>
>>  >> wrote:
>>  >>
>>  >>> Maybe we need some automated method to backup configurations and
>>  restore
>>  >>> them.
>>  >>>
>>  >>>
>>  >>>
>>  >>> On August 27, 2019 at 14:46:58, Michael Miklavcic (
>>  >>> michael.miklavcic@gmail.com) wrote:
>>  >>>
>>  >>> > Once you back up your metron configs, the same configs that worked on
>>  >>> the previous version will continue to work on the version running on
>>  HDP
>>  >>> 3.x. If there is any discrepancy between the two or additional
>>  settings
>>  >>> will be required, those will be documented in the release notes. From
>>  the
>>  >>> Metron perspective, this upgrade would be no different than simply
>>  >>> upgrading to the new Metron version.
>>  >>>
>>  >>> This upgrade cannot be performed the same way we've done it in the
>>  past.
>>  >>> A number of platform upgrades, including OS, are required:
>>  >>>
>>  >>> 1. Requires the OS to be updated on all nodes because there are no
>>  >>> Centos6 RPMs provided in HDP 3.1. Must bump to Centos7.
>>  >>> 2. The final new HBase code will not run on HDP 2.6
>>  >>> 3. The MPack changes for new Ambari are not backwards compatible
>>  >>> 4. YARN and HDFS/MR are also at risk to be backwards incompatible
>>  >>>
>>  >>>
>>  >>> On Tue, Aug 27, 2019 at 12:39 PM Michael Miklavcic <
>>  >>> michael.miklavcic@gmail.com> wrote:
>>  >>>
>>  >>>> Adding the dev list back into the thread (a reply-all was missed).
>>  >>>>
>>  >>>> On Tue, Aug 27, 2019 at 10:49 AM James Sirota <js...@apache.org>
>>  >>>> wrote:
>>  >>>>
>>  >>>>> I agree with Simon. HDP 2.x platform is rapidly approaching EOL and
>>  >>>>> everyone will likely need to migrate by end of year. Doing this
>>  platform
>>  >>>>> upgrade sooner will give everyone visibility into what Metron on HDP
>>  3.x
>>  >>>>> looks like so they have time to plan and upgrade at their own pace.
>>  >>>>> Feature-wise, the Metron application itself will be unchanged. It is
>>  >>>>> merely the platform underneath that is changing. HDP itself can be
>>  >>>>> upgraded per instructions here:
>>  >>>>>
>>  https://docs.hortonworks.com/HDPDocuments/HDP3/HDP-3.1.0/release-notes/content/upgrading_parent.html
>>  >>>>>
>>  >>>>> Once you back up your metron configs, the same configs that worked on
>>  >>>>> the previous version will continue to work on the version running on
>>  HDP
>>  >>>>> 3.x. If there is any discrepancy between the two or additional
>>  settings
>>  >>>>> will be required, those will be documented in the release notes.
>>  From the
>>  >>>>> Metron perspective, this upgrade would be no different than simply
>>  >>>>> upgrading to the new Metron version.
>>  >>>>>
>>  >>>>> James
>>  >>>>>
>>  >>>>>
>>  >>>>> 27.08.2019, 07:09, "Simon Elliston Ball" <
>>  simon@simonellistonball.com
>>  >>>>> >:
>>  >>>>>
>>  >>>>> Something worth noting here is that HDP 2.6.5 is quite old and
>>  >>>>> approaching EoL rapidly, so the issue of upgrade is urgent. I am
>>  aware of a
>>  >>>>> large number of users who require this upgrade ASAP, and in fact an
>>  aware
>>  >>>>> of zero users who wish to remain on HDP 2.
>>  >>>>>
>>  >>>>> Perhaps those users who want to stay on the old platform can stick
>>  >>>>> their hands up and raise concerns, but this move will likely have to
>>  happen
>>  >>>>> very soon.
>>  >>>>>
>>  >>>>> Simon
>>  >>>>>
>>  >>>>> On Tue, 27 Aug 2019 at 15:04, Otto Fowler <ot...@gmail.com>
>>  >>>>> wrote:
>>  >>>>>
>>  >>>>> Although we had the discussion, and some great ideas where passed
>>  >>>>> around, I do not believe we came to some kind of consensus on what
>>  1.0
>>  >>>>> should look like. So that discussion would have to be picked up
>>  again so
>>  >>>>> that we could know where we are at, and make it an actual thing if
>>  we were
>>  >>>>> going to make this a 1.0 release.
>>  >>>>>
>>  >>>>> I believe that the issues raised in that discussion gating 1.0 are
>>  >>>>> still largely applicable, including upgrade.
>>  >>>>>
>>  >>>>> Right now we have *ZERO* HDP 3.1 users. We will go from that to
>>  *only*
>>  >>>>> supporting 3.1 work and releases. So every user and deployment we
>>  currently
>>  >>>>> have will feel real pain, have to slay real dragons to move forward
>>  with
>>  >>>>> metron.
>>  >>>>>
>>  >>>>> With regards to support for older versions, the “backward capability”
>>  >>>>> that has been mentioned, I would not say that I have any specific
>>  plan for
>>  >>>>> that in mind. What I would say rather, is that I believe that we
>>  must be
>>  >>>>> explicit, setting expectations correctly and clearly with regards to
>>  our
>>  >>>>> intent while demonstrating that we have thought through the
>>  situation. That
>>  >>>>> discussion has not happened, at least I do not believe that the
>>  prior dev
>>  >>>>> thread really handles it in context.
>>  >>>>>
>>  >>>>> Depending on the upgrade situation for going to 3.1, it may be that a
>>  >>>>> dual stream of releases or fixes or new features to the extent that
>>  we can
>>  >>>>> do it will greatly reduce the pain for folks, or make it viable to
>>  stick
>>  >>>>> with metron until they can upgrade.
>>  >>>>>
>>  >>>>> The issue of what metron *is* features wise may be another one we
>>  >>>>> want to take up at some point. The idea being can we separate the
>>  metron
>>  >>>>> _integration parts from the metron core functionality such that we
>>  can work
>>  >>>>> on them separately and thus support multiple platforms through
>>  >>>>> integrations/applications. Of course definition of metron’s value
>>  beyond
>>  >>>>> integration, and what those features and application boundaries are
>>  would
>>  >>>>> be necessary.
>>  >>>>>
>>  >>>>>
>>  >>>>>
>>  >>>>> On August 26, 2019 at 18:52:57, Michael Miklavcic (
>>  >>>>> michael.miklavcic@gmail.com) wrote:
>>  >>>>>
>>  >>>>> Hi devs and users,
>>  >>>>>
>>  >>>>> Some questions were asked in the Slack channel about our ongoing
>>  >>>>> HDP/Hadoop upgrade and I'd like to get a discussion rolling. The
>>  original
>>  >>>>> Hadoop upgrade discuss thread can be found here
>>  >>>>>
>>  https://lists.apache.org/thread.html/37cc29648f0592cc39d3c78a0d07fce38521bdbbc4cf40e022a7a8ea@%3Cdev.metron.apache.org%3E
>>  >>>>>
>>  >>>>> The major issue and impact with upgrading the Hadoop platform is that
>>  >>>>> there are breaking changes. Code that runs on HDP 3.1 will not run
>>  on 2.x.
>>  >>>>> Here is a sampling of core components we depend on that we know of so
>>  >>>>> far that are not backwards compatible:
>>  >>>>>
>>  >>>>> 1. The Core OS - we currently base our builds and test deployment
>>  >>>>> off of artifacts pulled from HDP. The latest rev of HDP no longer
>>  ships
>>  >>>>> RPMs for Centos 6, which means we need to upgrade to Centos 7
>>  >>>>> 2. Ambari
>>  >>>>> 3. HBase
>>  >>>>>
>>  >>>>> This differs from individual components we've upgraded in the past in
>>  >>>>> that our code could still be deployed on the old as well as new
>>  version of
>>  >>>>> the component in a backwards compatible way. Based on semantic
>>  versioning,
>>  >>>>> I don't know if we can introduce this level of change in a point
>>  release,
>>  >>>>> which is the reason for kicking off this discussion. In the past,
>>  users and
>>  >>>>> developers in the community have suggested that they are -1 on a 1.x
>>  >>>>> release that does not provide an upgrade
>>  >>>>>
>>  https://lists.apache.org/thread.html/eb1a8df2d0a6a79c5d50540d1fdbf215ec83d831ff15d3117c2592cc@%3Cdev.metron.apache.org%3E
>>  >>>>> .
>>  >>>>>
>>  >>>>> Is there a way we can avoid a 1.x release? If we do need 1.x, do we
>>  >>>>> still see upgrades as a gating function? The main issue is that this
>>  has
>>  >>>>> the potential to drag out the upgrade and further couple it with
>>  other
>>  >>>>> features. And with Storm 1.x being eol'ed, I'm not sure this is
>>  something
>>  >>>>> we can wait much longer for. I'll think on this and send out my own
>>  >>>>> thoughts once folks have had a chance to review.
>>  >>>>>
>>  >>>>> Best,
>>  >>>>> Mike Miklavcic
>>  >>>>> Apache Metron, PMC, committer
>>  >>>>>
>>  >>>>>
>>  >>>>> --
>>  >>>>> --
>>  >>>>> simon elliston ball
>>  >>>>> @sireb
>>  >>>>>
>>  >>>>>
>>  >>>>>
>>  >>>>> -------------------
>>  >>>>> Thank you,
>>  >>>>>
>>  >>>>> James Sirota
>>  >>>>> PMC- Apache Metron
>>  >>>>> jsirota AT apache DOT org
>>  >>>>>
>>  >>>>> --
>>  >> --
>>  >> simon elliston ball
>>  >> @sireb
>>  >>
>>  >> --
>>  > --
>>  > simon elliston ball
>>  > @sireb
>>  >
>>  > --
>>  --
>>  simon elliston ball
>>  @sireb

------------------- 
Thank you,

James Sirota
PMC- Apache Metron
jsirota AT apache DOT org

Re: [DISCUSS] HDP 3.1 Upgrade and release strategy

["Zeolla@GMail.com" <ze...@gmail.com>]

I agree that having a scripted approach for backup and restore of Metron
configs should be necessary for such a large change/upgrade process.
Having been through this many times in the past I can tell you that the
difficulty of upgrading (whether perceived or actual) holds back adoption
of the platform.

Jon Zeolla

On Tue, Aug 27, 2019, 5:25 PM Simon Elliston Ball <
simon@simonellistonball.com> wrote:

> Not sure it’s in the scope of the project to handle the HDP upgrade as
> well, I would scope it to metron config only, and point at the extensive
> upgrade capability of Ambari, rather than us trying to recreate the way the
> distribution works.
>
> Simon
>
> On Tue, 27 Aug 2019 at 22:23, Otto Fowler <ot...@gmail.com> wrote:
>
> > If anyone can think of the things that need to be backed up, please
> > comment the jira.
> >
> >
> >
> >
> > On August 27, 2019 at 17:07:20, Otto Fowler (ottobackwards@gmail.com)
> > wrote:
> >
> > Good idea METRON–2239 [blocker].
> >
> >
> >
> > On August 27, 2019 at 16:30:13, Simon Elliston Ball (
> > simon@simonellistonball.com) wrote:
> >
> > You could always submit a Jira :)
> >
> > On Tue, 27 Aug 2019 at 21:27, Otto Fowler <ot...@gmail.com>
> wrote:
> >
> >> You are right, that is much better than backup_metron_configs.sh.
> >>
> >>
> >>
> >> On August 27, 2019 at 16:05:38, Simon Elliston Ball (
> >> simon@simonellistonball.com) wrote:
> >>
> >> You can do this with zk_load_configs and Ambari’s blueprint api, so we
> >> kinda already do.
> >>
> >> Simon
> >>
> >> On Tue, 27 Aug 2019 at 20:19, Otto Fowler <ot...@gmail.com>
> >> wrote:
> >>
> >>> Maybe we need some automated method to backup configurations and
> restore
> >>> them.
> >>>
> >>>
> >>>
> >>> On August 27, 2019 at 14:46:58, Michael Miklavcic (
> >>> michael.miklavcic@gmail.com) wrote:
> >>>
> >>> > Once you back up your metron configs, the same configs that worked on
> >>> the previous version will continue to work on the version running on
> HDP
> >>> 3.x.  If there is any discrepancy between the two or additional
> settings
> >>> will be required, those will be documented in the release notes.  From
> the
> >>> Metron perspective, this upgrade would be no different than simply
> >>> upgrading to the new Metron version.
> >>>
> >>> This upgrade cannot be performed the same way we've done it in the
> past.
> >>> A number of platform upgrades, including OS, are required:
> >>>
> >>>    1. Requires the OS to be updated on all nodes because there are no
> >>>    Centos6 RPMs provided in HDP 3.1. Must bump to Centos7.
> >>>    2. The final new HBase code will not run on HDP 2.6
> >>>    3. The MPack changes for new Ambari are not backwards compatible
> >>>    4. YARN and HDFS/MR are also at risk to be backwards incompatible
> >>>
> >>>
> >>> On Tue, Aug 27, 2019 at 12:39 PM Michael Miklavcic <
> >>> michael.miklavcic@gmail.com> wrote:
> >>>
> >>>> Adding the dev list back into the thread (a reply-all was missed).
> >>>>
> >>>> On Tue, Aug 27, 2019 at 10:49 AM James Sirota <js...@apache.org>
> >>>> wrote:
> >>>>
> >>>>> I agree with Simon.  HDP 2.x platform is rapidly approaching EOL and
> >>>>> everyone will likely need to migrate by end of year.  Doing this
> platform
> >>>>> upgrade sooner will give everyone visibility into what Metron on HDP
> 3.x
> >>>>> looks like so they have time to plan and upgrade at their own pace.
> >>>>> Feature-wise, the Metron application itself will be unchanged.  It is
> >>>>> merely the platform underneath that is changing.  HDP itself can be
> >>>>> upgraded per instructions here:
> >>>>>
> https://docs.hortonworks.com/HDPDocuments/HDP3/HDP-3.1.0/release-notes/content/upgrading_parent.html
> >>>>>
> >>>>> Once you back up your metron configs, the same configs that worked on
> >>>>> the previous version will continue to work on the version running on
> HDP
> >>>>> 3.x.  If there is any discrepancy between the two or additional
> settings
> >>>>> will be required, those will be documented in the release notes.
> From the
> >>>>> Metron perspective, this upgrade would be no different than simply
> >>>>> upgrading to the new Metron version.
> >>>>>
> >>>>> James
> >>>>>
> >>>>>
> >>>>> 27.08.2019, 07:09, "Simon Elliston Ball" <
> simon@simonellistonball.com
> >>>>> >:
> >>>>>
> >>>>> Something worth noting here is that HDP 2.6.5 is quite old and
> >>>>> approaching EoL rapidly, so the issue of upgrade is urgent. I am
> aware of a
> >>>>> large number of users who require this upgrade ASAP, and in fact an
> aware
> >>>>> of zero users who wish to remain on HDP 2.
> >>>>>
> >>>>> Perhaps those users who want to stay on the old platform can stick
> >>>>> their hands up and raise concerns, but this move will likely have to
> happen
> >>>>> very soon.
> >>>>>
> >>>>> Simon
> >>>>>
> >>>>> On Tue, 27 Aug 2019 at 15:04, Otto Fowler <ot...@gmail.com>
> >>>>> wrote:
> >>>>>
> >>>>> Although we had the discussion, and some great ideas where passed
> >>>>> around, I do not believe we came to some kind of consensus on what
> 1.0
> >>>>> should look like. So that discussion would have to be picked up
> again so
> >>>>> that we could know where we are at, and make it an actual thing if
> we were
> >>>>> going to make this a 1.0 release.
> >>>>>
> >>>>> I believe that the issues raised in that discussion gating 1.0 are
> >>>>> still largely applicable, including upgrade.
> >>>>>
> >>>>> Right now we have *ZERO* HDP 3.1 users. We will go from that to
> *only*
> >>>>> supporting 3.1 work and releases. So every user and deployment we
> currently
> >>>>> have will feel real pain, have to slay real dragons to move forward
> with
> >>>>> metron.
> >>>>>
> >>>>> With regards to support for older versions, the “backward capability”
> >>>>> that has been mentioned, I would not say that I have any specific
> plan for
> >>>>> that in mind. What I would say rather, is that I believe that we
> must be
> >>>>> explicit, setting expectations correctly and clearly with regards to
> our
> >>>>> intent while demonstrating that we have thought through the
> situation. That
> >>>>> discussion has not happened, at least I do not believe that the
> prior dev
> >>>>> thread really handles it in context.
> >>>>>
> >>>>> Depending on the upgrade situation for going to 3.1, it may be that a
> >>>>> dual stream of releases or fixes or new features to the extent that
> we can
> >>>>> do it will greatly reduce the pain for folks, or make it viable to
> stick
> >>>>> with metron until they can upgrade.
> >>>>>
> >>>>> The issue of what metron *is* features wise may be another one we
> >>>>> want to take up at some point. The idea being can we separate the
> metron
> >>>>> _integration parts from the metron core functionality such that we
> can work
> >>>>> on them separately and thus support multiple platforms through
> >>>>> integrations/applications. Of course definition of metron’s value
> beyond
> >>>>> integration, and what those features and application boundaries are
> would
> >>>>> be necessary.
> >>>>>
> >>>>>
> >>>>>
> >>>>> On August 26, 2019 at 18:52:57, Michael Miklavcic (
> >>>>> michael.miklavcic@gmail.com) wrote:
> >>>>>
> >>>>> Hi devs and users,
> >>>>>
> >>>>> Some questions were asked in the Slack channel about our ongoing
> >>>>> HDP/Hadoop upgrade and I'd like to get a discussion rolling. The
> original
> >>>>> Hadoop upgrade discuss thread can be found here
> >>>>>
> https://lists.apache.org/thread.html/37cc29648f0592cc39d3c78a0d07fce38521bdbbc4cf40e022a7a8ea@%3Cdev.metron.apache.org%3E
> >>>>>
> >>>>> The major issue and impact with upgrading the Hadoop platform is that
> >>>>> there are breaking changes. Code that runs on HDP 3.1 will not run
> on 2.x.
> >>>>> Here is a sampling of core components we depend on that we know of so
> >>>>> far that are not backwards compatible:
> >>>>>
> >>>>>    1. The Core OS - we currently base our builds and test deployment
> >>>>>    off of artifacts pulled from HDP. The latest rev of HDP no longer
> ships
> >>>>>    RPMs for Centos 6, which means we need to upgrade to Centos 7
> >>>>>    2. Ambari
> >>>>>    3. HBase
> >>>>>
> >>>>> This differs from individual components we've upgraded in the past in
> >>>>> that our code could still be deployed on the old as well as new
> version of
> >>>>> the component in a backwards compatible way. Based on semantic
> versioning,
> >>>>> I don't know if we can introduce this level of change in a point
> release,
> >>>>> which is the reason for kicking off this discussion. In the past,
> users and
> >>>>> developers in the community have suggested that they are -1 on a 1.x
> >>>>> release that does not provide an upgrade
> >>>>>
> https://lists.apache.org/thread.html/eb1a8df2d0a6a79c5d50540d1fdbf215ec83d831ff15d3117c2592cc@%3Cdev.metron.apache.org%3E
> >>>>> .
> >>>>>
> >>>>> Is there a way we can avoid a 1.x release? If we do need 1.x, do we
> >>>>> still see upgrades as a gating function? The main issue is that this
> has
> >>>>> the potential to drag out the upgrade and further couple it with
> other
> >>>>> features. And with Storm 1.x being eol'ed, I'm not sure this is
> something
> >>>>> we can wait much longer for. I'll think on this and send out my own
> >>>>> thoughts once folks have had a chance to review.
> >>>>>
> >>>>> Best,
> >>>>> Mike Miklavcic
> >>>>> Apache Metron, PMC, committer
> >>>>>
> >>>>>
> >>>>> --
> >>>>> --
> >>>>> simon elliston ball
> >>>>> @sireb
> >>>>>
> >>>>>
> >>>>>
> >>>>> -------------------
> >>>>> Thank you,
> >>>>>
> >>>>> James Sirota
> >>>>> PMC- Apache Metron
> >>>>> jsirota AT apache DOT org
> >>>>>
> >>>>> --
> >> --
> >> simon elliston ball
> >> @sireb
> >>
> >> --
> > --
> > simon elliston ball
> > @sireb
> >
> > --
> --
> simon elliston ball
> @sireb
>

Re: [DISCUSS] HDP 3.1 Upgrade and release strategy

["Zeolla@GMail.com" <ze...@gmail.com>]

I agree that having a scripted approach for backup and restore of Metron
configs should be necessary for such a large change/upgrade process.
Having been through this many times in the past I can tell you that the
difficulty of upgrading (whether perceived or actual) holds back adoption
of the platform.

Jon Zeolla

On Tue, Aug 27, 2019, 5:25 PM Simon Elliston Ball <
simon@simonellistonball.com> wrote:

> Not sure it’s in the scope of the project to handle the HDP upgrade as
> well, I would scope it to metron config only, and point at the extensive
> upgrade capability of Ambari, rather than us trying to recreate the way the
> distribution works.
>
> Simon
>
> On Tue, 27 Aug 2019 at 22:23, Otto Fowler <ot...@gmail.com> wrote:
>
> > If anyone can think of the things that need to be backed up, please
> > comment the jira.
> >
> >
> >
> >
> > On August 27, 2019 at 17:07:20, Otto Fowler (ottobackwards@gmail.com)
> > wrote:
> >
> > Good idea METRON–2239 [blocker].
> >
> >
> >
> > On August 27, 2019 at 16:30:13, Simon Elliston Ball (
> > simon@simonellistonball.com) wrote:
> >
> > You could always submit a Jira :)
> >
> > On Tue, 27 Aug 2019 at 21:27, Otto Fowler <ot...@gmail.com>
> wrote:
> >
> >> You are right, that is much better than backup_metron_configs.sh.
> >>
> >>
> >>
> >> On August 27, 2019 at 16:05:38, Simon Elliston Ball (
> >> simon@simonellistonball.com) wrote:
> >>
> >> You can do this with zk_load_configs and Ambari’s blueprint api, so we
> >> kinda already do.
> >>
> >> Simon
> >>
> >> On Tue, 27 Aug 2019 at 20:19, Otto Fowler <ot...@gmail.com>
> >> wrote:
> >>
> >>> Maybe we need some automated method to backup configurations and
> restore
> >>> them.
> >>>
> >>>
> >>>
> >>> On August 27, 2019 at 14:46:58, Michael Miklavcic (
> >>> michael.miklavcic@gmail.com) wrote:
> >>>
> >>> > Once you back up your metron configs, the same configs that worked on
> >>> the previous version will continue to work on the version running on
> HDP
> >>> 3.x.  If there is any discrepancy between the two or additional
> settings
> >>> will be required, those will be documented in the release notes.  From
> the
> >>> Metron perspective, this upgrade would be no different than simply
> >>> upgrading to the new Metron version.
> >>>
> >>> This upgrade cannot be performed the same way we've done it in the
> past.
> >>> A number of platform upgrades, including OS, are required:
> >>>
> >>>    1. Requires the OS to be updated on all nodes because there are no
> >>>    Centos6 RPMs provided in HDP 3.1. Must bump to Centos7.
> >>>    2. The final new HBase code will not run on HDP 2.6
> >>>    3. The MPack changes for new Ambari are not backwards compatible
> >>>    4. YARN and HDFS/MR are also at risk to be backwards incompatible
> >>>
> >>>
> >>> On Tue, Aug 27, 2019 at 12:39 PM Michael Miklavcic <
> >>> michael.miklavcic@gmail.com> wrote:
> >>>
> >>>> Adding the dev list back into the thread (a reply-all was missed).
> >>>>
> >>>> On Tue, Aug 27, 2019 at 10:49 AM James Sirota <js...@apache.org>
> >>>> wrote:
> >>>>
> >>>>> I agree with Simon.  HDP 2.x platform is rapidly approaching EOL and
> >>>>> everyone will likely need to migrate by end of year.  Doing this
> platform
> >>>>> upgrade sooner will give everyone visibility into what Metron on HDP
> 3.x
> >>>>> looks like so they have time to plan and upgrade at their own pace.
> >>>>> Feature-wise, the Metron application itself will be unchanged.  It is
> >>>>> merely the platform underneath that is changing.  HDP itself can be
> >>>>> upgraded per instructions here:
> >>>>>
> https://docs.hortonworks.com/HDPDocuments/HDP3/HDP-3.1.0/release-notes/content/upgrading_parent.html
> >>>>>
> >>>>> Once you back up your metron configs, the same configs that worked on
> >>>>> the previous version will continue to work on the version running on
> HDP
> >>>>> 3.x.  If there is any discrepancy between the two or additional
> settings
> >>>>> will be required, those will be documented in the release notes.
> From the
> >>>>> Metron perspective, this upgrade would be no different than simply
> >>>>> upgrading to the new Metron version.
> >>>>>
> >>>>> James
> >>>>>
> >>>>>
> >>>>> 27.08.2019, 07:09, "Simon Elliston Ball" <
> simon@simonellistonball.com
> >>>>> >:
> >>>>>
> >>>>> Something worth noting here is that HDP 2.6.5 is quite old and
> >>>>> approaching EoL rapidly, so the issue of upgrade is urgent. I am
> aware of a
> >>>>> large number of users who require this upgrade ASAP, and in fact an
> aware
> >>>>> of zero users who wish to remain on HDP 2.
> >>>>>
> >>>>> Perhaps those users who want to stay on the old platform can stick
> >>>>> their hands up and raise concerns, but this move will likely have to
> happen
> >>>>> very soon.
> >>>>>
> >>>>> Simon
> >>>>>
> >>>>> On Tue, 27 Aug 2019 at 15:04, Otto Fowler <ot...@gmail.com>
> >>>>> wrote:
> >>>>>
> >>>>> Although we had the discussion, and some great ideas where passed
> >>>>> around, I do not believe we came to some kind of consensus on what
> 1.0
> >>>>> should look like. So that discussion would have to be picked up
> again so
> >>>>> that we could know where we are at, and make it an actual thing if
> we were
> >>>>> going to make this a 1.0 release.
> >>>>>
> >>>>> I believe that the issues raised in that discussion gating 1.0 are
> >>>>> still largely applicable, including upgrade.
> >>>>>
> >>>>> Right now we have *ZERO* HDP 3.1 users. We will go from that to
> *only*
> >>>>> supporting 3.1 work and releases. So every user and deployment we
> currently
> >>>>> have will feel real pain, have to slay real dragons to move forward
> with
> >>>>> metron.
> >>>>>
> >>>>> With regards to support for older versions, the “backward capability”
> >>>>> that has been mentioned, I would not say that I have any specific
> plan for
> >>>>> that in mind. What I would say rather, is that I believe that we
> must be
> >>>>> explicit, setting expectations correctly and clearly with regards to
> our
> >>>>> intent while demonstrating that we have thought through the
> situation. That
> >>>>> discussion has not happened, at least I do not believe that the
> prior dev
> >>>>> thread really handles it in context.
> >>>>>
> >>>>> Depending on the upgrade situation for going to 3.1, it may be that a
> >>>>> dual stream of releases or fixes or new features to the extent that
> we can
> >>>>> do it will greatly reduce the pain for folks, or make it viable to
> stick
> >>>>> with metron until they can upgrade.
> >>>>>
> >>>>> The issue of what metron *is* features wise may be another one we
> >>>>> want to take up at some point. The idea being can we separate the
> metron
> >>>>> _integration parts from the metron core functionality such that we
> can work
> >>>>> on them separately and thus support multiple platforms through
> >>>>> integrations/applications. Of course definition of metron’s value
> beyond
> >>>>> integration, and what those features and application boundaries are
> would
> >>>>> be necessary.
> >>>>>
> >>>>>
> >>>>>
> >>>>> On August 26, 2019 at 18:52:57, Michael Miklavcic (
> >>>>> michael.miklavcic@gmail.com) wrote:
> >>>>>
> >>>>> Hi devs and users,
> >>>>>
> >>>>> Some questions were asked in the Slack channel about our ongoing
> >>>>> HDP/Hadoop upgrade and I'd like to get a discussion rolling. The
> original
> >>>>> Hadoop upgrade discuss thread can be found here
> >>>>>
> https://lists.apache.org/thread.html/37cc29648f0592cc39d3c78a0d07fce38521bdbbc4cf40e022a7a8ea@%3Cdev.metron.apache.org%3E
> >>>>>
> >>>>> The major issue and impact with upgrading the Hadoop platform is that
> >>>>> there are breaking changes. Code that runs on HDP 3.1 will not run
> on 2.x.
> >>>>> Here is a sampling of core components we depend on that we know of so
> >>>>> far that are not backwards compatible:
> >>>>>
> >>>>>    1. The Core OS - we currently base our builds and test deployment
> >>>>>    off of artifacts pulled from HDP. The latest rev of HDP no longer
> ships
> >>>>>    RPMs for Centos 6, which means we need to upgrade to Centos 7
> >>>>>    2. Ambari
> >>>>>    3. HBase
> >>>>>
> >>>>> This differs from individual components we've upgraded in the past in
> >>>>> that our code could still be deployed on the old as well as new
> version of
> >>>>> the component in a backwards compatible way. Based on semantic
> versioning,
> >>>>> I don't know if we can introduce this level of change in a point
> release,
> >>>>> which is the reason for kicking off this discussion. In the past,
> users and
> >>>>> developers in the community have suggested that they are -1 on a 1.x
> >>>>> release that does not provide an upgrade
> >>>>>
> https://lists.apache.org/thread.html/eb1a8df2d0a6a79c5d50540d1fdbf215ec83d831ff15d3117c2592cc@%3Cdev.metron.apache.org%3E
> >>>>> .
> >>>>>
> >>>>> Is there a way we can avoid a 1.x release? If we do need 1.x, do we
> >>>>> still see upgrades as a gating function? The main issue is that this
> has
> >>>>> the potential to drag out the upgrade and further couple it with
> other
> >>>>> features. And with Storm 1.x being eol'ed, I'm not sure this is
> something
> >>>>> we can wait much longer for. I'll think on this and send out my own
> >>>>> thoughts once folks have had a chance to review.
> >>>>>
> >>>>> Best,
> >>>>> Mike Miklavcic
> >>>>> Apache Metron, PMC, committer
> >>>>>
> >>>>>
> >>>>> --
> >>>>> --
> >>>>> simon elliston ball
> >>>>> @sireb
> >>>>>
> >>>>>
> >>>>>
> >>>>> -------------------
> >>>>> Thank you,
> >>>>>
> >>>>> James Sirota
> >>>>> PMC- Apache Metron
> >>>>> jsirota AT apache DOT org
> >>>>>
> >>>>> --
> >> --
> >> simon elliston ball
> >> @sireb
> >>
> >> --
> > --
> > simon elliston ball
> > @sireb
> >
> > --
> --
> simon elliston ball
> @sireb
>

Re: [DISCUSS] HDP 3.1 Upgrade and release strategy

[Simon Elliston Ball <si...@simonellistonball.com>]

Not sure it’s in the scope of the project to handle the HDP upgrade as
well, I would scope it to metron config only, and point at the extensive
upgrade capability of Ambari, rather than us trying to recreate the way the
distribution works.

Simon

On Tue, 27 Aug 2019 at 22:23, Otto Fowler <ot...@gmail.com> wrote:

> If anyone can think of the things that need to be backed up, please
> comment the jira.
>
>
>
>
> On August 27, 2019 at 17:07:20, Otto Fowler (ottobackwards@gmail.com)
> wrote:
>
> Good idea METRON–2239 [blocker].
>
>
>
> On August 27, 2019 at 16:30:13, Simon Elliston Ball (
> simon@simonellistonball.com) wrote:
>
> You could always submit a Jira :)
>
> On Tue, 27 Aug 2019 at 21:27, Otto Fowler <ot...@gmail.com> wrote:
>
>> You are right, that is much better than backup_metron_configs.sh.
>>
>>
>>
>> On August 27, 2019 at 16:05:38, Simon Elliston Ball (
>> simon@simonellistonball.com) wrote:
>>
>> You can do this with zk_load_configs and Ambari’s blueprint api, so we
>> kinda already do.
>>
>> Simon
>>
>> On Tue, 27 Aug 2019 at 20:19, Otto Fowler <ot...@gmail.com>
>> wrote:
>>
>>> Maybe we need some automated method to backup configurations and restore
>>> them.
>>>
>>>
>>>
>>> On August 27, 2019 at 14:46:58, Michael Miklavcic (
>>> michael.miklavcic@gmail.com) wrote:
>>>
>>> > Once you back up your metron configs, the same configs that worked on
>>> the previous version will continue to work on the version running on HDP
>>> 3.x.  If there is any discrepancy between the two or additional settings
>>> will be required, those will be documented in the release notes.  From the
>>> Metron perspective, this upgrade would be no different than simply
>>> upgrading to the new Metron version.
>>>
>>> This upgrade cannot be performed the same way we've done it in the past.
>>> A number of platform upgrades, including OS, are required:
>>>
>>>    1. Requires the OS to be updated on all nodes because there are no
>>>    Centos6 RPMs provided in HDP 3.1. Must bump to Centos7.
>>>    2. The final new HBase code will not run on HDP 2.6
>>>    3. The MPack changes for new Ambari are not backwards compatible
>>>    4. YARN and HDFS/MR are also at risk to be backwards incompatible
>>>
>>>
>>> On Tue, Aug 27, 2019 at 12:39 PM Michael Miklavcic <
>>> michael.miklavcic@gmail.com> wrote:
>>>
>>>> Adding the dev list back into the thread (a reply-all was missed).
>>>>
>>>> On Tue, Aug 27, 2019 at 10:49 AM James Sirota <js...@apache.org>
>>>> wrote:
>>>>
>>>>> I agree with Simon.  HDP 2.x platform is rapidly approaching EOL and
>>>>> everyone will likely need to migrate by end of year.  Doing this platform
>>>>> upgrade sooner will give everyone visibility into what Metron on HDP 3.x
>>>>> looks like so they have time to plan and upgrade at their own pace.
>>>>> Feature-wise, the Metron application itself will be unchanged.  It is
>>>>> merely the platform underneath that is changing.  HDP itself can be
>>>>> upgraded per instructions here:
>>>>> https://docs.hortonworks.com/HDPDocuments/HDP3/HDP-3.1.0/release-notes/content/upgrading_parent.html
>>>>>
>>>>> Once you back up your metron configs, the same configs that worked on
>>>>> the previous version will continue to work on the version running on HDP
>>>>> 3.x.  If there is any discrepancy between the two or additional settings
>>>>> will be required, those will be documented in the release notes.  From the
>>>>> Metron perspective, this upgrade would be no different than simply
>>>>> upgrading to the new Metron version.
>>>>>
>>>>> James
>>>>>
>>>>>
>>>>> 27.08.2019, 07:09, "Simon Elliston Ball" <simon@simonellistonball.com
>>>>> >:
>>>>>
>>>>> Something worth noting here is that HDP 2.6.5 is quite old and
>>>>> approaching EoL rapidly, so the issue of upgrade is urgent. I am aware of a
>>>>> large number of users who require this upgrade ASAP, and in fact an aware
>>>>> of zero users who wish to remain on HDP 2.
>>>>>
>>>>> Perhaps those users who want to stay on the old platform can stick
>>>>> their hands up and raise concerns, but this move will likely have to happen
>>>>> very soon.
>>>>>
>>>>> Simon
>>>>>
>>>>> On Tue, 27 Aug 2019 at 15:04, Otto Fowler <ot...@gmail.com>
>>>>> wrote:
>>>>>
>>>>> Although we had the discussion, and some great ideas where passed
>>>>> around, I do not believe we came to some kind of consensus on what 1.0
>>>>> should look like. So that discussion would have to be picked up again so
>>>>> that we could know where we are at, and make it an actual thing if we were
>>>>> going to make this a 1.0 release.
>>>>>
>>>>> I believe that the issues raised in that discussion gating 1.0 are
>>>>> still largely applicable, including upgrade.
>>>>>
>>>>> Right now we have *ZERO* HDP 3.1 users. We will go from that to *only*
>>>>> supporting 3.1 work and releases. So every user and deployment we currently
>>>>> have will feel real pain, have to slay real dragons to move forward with
>>>>> metron.
>>>>>
>>>>> With regards to support for older versions, the “backward capability”
>>>>> that has been mentioned, I would not say that I have any specific plan for
>>>>> that in mind. What I would say rather, is that I believe that we must be
>>>>> explicit, setting expectations correctly and clearly with regards to our
>>>>> intent while demonstrating that we have thought through the situation. That
>>>>> discussion has not happened, at least I do not believe that the prior dev
>>>>> thread really handles it in context.
>>>>>
>>>>> Depending on the upgrade situation for going to 3.1, it may be that a
>>>>> dual stream of releases or fixes or new features to the extent that we can
>>>>> do it will greatly reduce the pain for folks, or make it viable to stick
>>>>> with metron until they can upgrade.
>>>>>
>>>>> The issue of what metron *is* features wise may be another one we
>>>>> want to take up at some point. The idea being can we separate the metron
>>>>> _integration parts from the metron core functionality such that we can work
>>>>> on them separately and thus support multiple platforms through
>>>>> integrations/applications. Of course definition of metron’s value beyond
>>>>> integration, and what those features and application boundaries are would
>>>>> be necessary.
>>>>>
>>>>>
>>>>>
>>>>> On August 26, 2019 at 18:52:57, Michael Miklavcic (
>>>>> michael.miklavcic@gmail.com) wrote:
>>>>>
>>>>> Hi devs and users,
>>>>>
>>>>> Some questions were asked in the Slack channel about our ongoing
>>>>> HDP/Hadoop upgrade and I'd like to get a discussion rolling. The original
>>>>> Hadoop upgrade discuss thread can be found here
>>>>> https://lists.apache.org/thread.html/37cc29648f0592cc39d3c78a0d07fce38521bdbbc4cf40e022a7a8ea@%3Cdev.metron.apache.org%3E
>>>>>
>>>>> The major issue and impact with upgrading the Hadoop platform is that
>>>>> there are breaking changes. Code that runs on HDP 3.1 will not run on 2.x.
>>>>> Here is a sampling of core components we depend on that we know of so
>>>>> far that are not backwards compatible:
>>>>>
>>>>>    1. The Core OS - we currently base our builds and test deployment
>>>>>    off of artifacts pulled from HDP. The latest rev of HDP no longer ships
>>>>>    RPMs for Centos 6, which means we need to upgrade to Centos 7
>>>>>    2. Ambari
>>>>>    3. HBase
>>>>>
>>>>> This differs from individual components we've upgraded in the past in
>>>>> that our code could still be deployed on the old as well as new version of
>>>>> the component in a backwards compatible way. Based on semantic versioning,
>>>>> I don't know if we can introduce this level of change in a point release,
>>>>> which is the reason for kicking off this discussion. In the past, users and
>>>>> developers in the community have suggested that they are -1 on a 1.x
>>>>> release that does not provide an upgrade
>>>>> https://lists.apache.org/thread.html/eb1a8df2d0a6a79c5d50540d1fdbf215ec83d831ff15d3117c2592cc@%3Cdev.metron.apache.org%3E
>>>>> .
>>>>>
>>>>> Is there a way we can avoid a 1.x release? If we do need 1.x, do we
>>>>> still see upgrades as a gating function? The main issue is that this has
>>>>> the potential to drag out the upgrade and further couple it with other
>>>>> features. And with Storm 1.x being eol'ed, I'm not sure this is something
>>>>> we can wait much longer for. I'll think on this and send out my own
>>>>> thoughts once folks have had a chance to review.
>>>>>
>>>>> Best,
>>>>> Mike Miklavcic
>>>>> Apache Metron, PMC, committer
>>>>>
>>>>>
>>>>> --
>>>>> --
>>>>> simon elliston ball
>>>>> @sireb
>>>>>
>>>>>
>>>>>
>>>>> -------------------
>>>>> Thank you,
>>>>>
>>>>> James Sirota
>>>>> PMC- Apache Metron
>>>>> jsirota AT apache DOT org
>>>>>
>>>>> --
>> --
>> simon elliston ball
>> @sireb
>>
>> --
> --
> simon elliston ball
> @sireb
>
> --
--
simon elliston ball
@sireb

Re: [DISCUSS] HDP 3.1 Upgrade and release strategy

[Simon Elliston Ball <si...@simonellistonball.com>]

Not sure it’s in the scope of the project to handle the HDP upgrade as
well, I would scope it to metron config only, and point at the extensive
upgrade capability of Ambari, rather than us trying to recreate the way the
distribution works.

Simon

On Tue, 27 Aug 2019 at 22:23, Otto Fowler <ot...@gmail.com> wrote:

> If anyone can think of the things that need to be backed up, please
> comment the jira.
>
>
>
>
> On August 27, 2019 at 17:07:20, Otto Fowler (ottobackwards@gmail.com)
> wrote:
>
> Good idea METRON–2239 [blocker].
>
>
>
> On August 27, 2019 at 16:30:13, Simon Elliston Ball (
> simon@simonellistonball.com) wrote:
>
> You could always submit a Jira :)
>
> On Tue, 27 Aug 2019 at 21:27, Otto Fowler <ot...@gmail.com> wrote:
>
>> You are right, that is much better than backup_metron_configs.sh.
>>
>>
>>
>> On August 27, 2019 at 16:05:38, Simon Elliston Ball (
>> simon@simonellistonball.com) wrote:
>>
>> You can do this with zk_load_configs and Ambari’s blueprint api, so we
>> kinda already do.
>>
>> Simon
>>
>> On Tue, 27 Aug 2019 at 20:19, Otto Fowler <ot...@gmail.com>
>> wrote:
>>
>>> Maybe we need some automated method to backup configurations and restore
>>> them.
>>>
>>>
>>>
>>> On August 27, 2019 at 14:46:58, Michael Miklavcic (
>>> michael.miklavcic@gmail.com) wrote:
>>>
>>> > Once you back up your metron configs, the same configs that worked on
>>> the previous version will continue to work on the version running on HDP
>>> 3.x.  If there is any discrepancy between the two or additional settings
>>> will be required, those will be documented in the release notes.  From the
>>> Metron perspective, this upgrade would be no different than simply
>>> upgrading to the new Metron version.
>>>
>>> This upgrade cannot be performed the same way we've done it in the past.
>>> A number of platform upgrades, including OS, are required:
>>>
>>>    1. Requires the OS to be updated on all nodes because there are no
>>>    Centos6 RPMs provided in HDP 3.1. Must bump to Centos7.
>>>    2. The final new HBase code will not run on HDP 2.6
>>>    3. The MPack changes for new Ambari are not backwards compatible
>>>    4. YARN and HDFS/MR are also at risk to be backwards incompatible
>>>
>>>
>>> On Tue, Aug 27, 2019 at 12:39 PM Michael Miklavcic <
>>> michael.miklavcic@gmail.com> wrote:
>>>
>>>> Adding the dev list back into the thread (a reply-all was missed).
>>>>
>>>> On Tue, Aug 27, 2019 at 10:49 AM James Sirota <js...@apache.org>
>>>> wrote:
>>>>
>>>>> I agree with Simon.  HDP 2.x platform is rapidly approaching EOL and
>>>>> everyone will likely need to migrate by end of year.  Doing this platform
>>>>> upgrade sooner will give everyone visibility into what Metron on HDP 3.x
>>>>> looks like so they have time to plan and upgrade at their own pace.
>>>>> Feature-wise, the Metron application itself will be unchanged.  It is
>>>>> merely the platform underneath that is changing.  HDP itself can be
>>>>> upgraded per instructions here:
>>>>> https://docs.hortonworks.com/HDPDocuments/HDP3/HDP-3.1.0/release-notes/content/upgrading_parent.html
>>>>>
>>>>> Once you back up your metron configs, the same configs that worked on
>>>>> the previous version will continue to work on the version running on HDP
>>>>> 3.x.  If there is any discrepancy between the two or additional settings
>>>>> will be required, those will be documented in the release notes.  From the
>>>>> Metron perspective, this upgrade would be no different than simply
>>>>> upgrading to the new Metron version.
>>>>>
>>>>> James
>>>>>
>>>>>
>>>>> 27.08.2019, 07:09, "Simon Elliston Ball" <simon@simonellistonball.com
>>>>> >:
>>>>>
>>>>> Something worth noting here is that HDP 2.6.5 is quite old and
>>>>> approaching EoL rapidly, so the issue of upgrade is urgent. I am aware of a
>>>>> large number of users who require this upgrade ASAP, and in fact an aware
>>>>> of zero users who wish to remain on HDP 2.
>>>>>
>>>>> Perhaps those users who want to stay on the old platform can stick
>>>>> their hands up and raise concerns, but this move will likely have to happen
>>>>> very soon.
>>>>>
>>>>> Simon
>>>>>
>>>>> On Tue, 27 Aug 2019 at 15:04, Otto Fowler <ot...@gmail.com>
>>>>> wrote:
>>>>>
>>>>> Although we had the discussion, and some great ideas where passed
>>>>> around, I do not believe we came to some kind of consensus on what 1.0
>>>>> should look like. So that discussion would have to be picked up again so
>>>>> that we could know where we are at, and make it an actual thing if we were
>>>>> going to make this a 1.0 release.
>>>>>
>>>>> I believe that the issues raised in that discussion gating 1.0 are
>>>>> still largely applicable, including upgrade.
>>>>>
>>>>> Right now we have *ZERO* HDP 3.1 users. We will go from that to *only*
>>>>> supporting 3.1 work and releases. So every user and deployment we currently
>>>>> have will feel real pain, have to slay real dragons to move forward with
>>>>> metron.
>>>>>
>>>>> With regards to support for older versions, the “backward capability”
>>>>> that has been mentioned, I would not say that I have any specific plan for
>>>>> that in mind. What I would say rather, is that I believe that we must be
>>>>> explicit, setting expectations correctly and clearly with regards to our
>>>>> intent while demonstrating that we have thought through the situation. That
>>>>> discussion has not happened, at least I do not believe that the prior dev
>>>>> thread really handles it in context.
>>>>>
>>>>> Depending on the upgrade situation for going to 3.1, it may be that a
>>>>> dual stream of releases or fixes or new features to the extent that we can
>>>>> do it will greatly reduce the pain for folks, or make it viable to stick
>>>>> with metron until they can upgrade.
>>>>>
>>>>> The issue of what metron *is* features wise may be another one we
>>>>> want to take up at some point. The idea being can we separate the metron
>>>>> _integration parts from the metron core functionality such that we can work
>>>>> on them separately and thus support multiple platforms through
>>>>> integrations/applications. Of course definition of metron’s value beyond
>>>>> integration, and what those features and application boundaries are would
>>>>> be necessary.
>>>>>
>>>>>
>>>>>
>>>>> On August 26, 2019 at 18:52:57, Michael Miklavcic (
>>>>> michael.miklavcic@gmail.com) wrote:
>>>>>
>>>>> Hi devs and users,
>>>>>
>>>>> Some questions were asked in the Slack channel about our ongoing
>>>>> HDP/Hadoop upgrade and I'd like to get a discussion rolling. The original
>>>>> Hadoop upgrade discuss thread can be found here
>>>>> https://lists.apache.org/thread.html/37cc29648f0592cc39d3c78a0d07fce38521bdbbc4cf40e022a7a8ea@%3Cdev.metron.apache.org%3E
>>>>>
>>>>> The major issue and impact with upgrading the Hadoop platform is that
>>>>> there are breaking changes. Code that runs on HDP 3.1 will not run on 2.x.
>>>>> Here is a sampling of core components we depend on that we know of so
>>>>> far that are not backwards compatible:
>>>>>
>>>>>    1. The Core OS - we currently base our builds and test deployment
>>>>>    off of artifacts pulled from HDP. The latest rev of HDP no longer ships
>>>>>    RPMs for Centos 6, which means we need to upgrade to Centos 7
>>>>>    2. Ambari
>>>>>    3. HBase
>>>>>
>>>>> This differs from individual components we've upgraded in the past in
>>>>> that our code could still be deployed on the old as well as new version of
>>>>> the component in a backwards compatible way. Based on semantic versioning,
>>>>> I don't know if we can introduce this level of change in a point release,
>>>>> which is the reason for kicking off this discussion. In the past, users and
>>>>> developers in the community have suggested that they are -1 on a 1.x
>>>>> release that does not provide an upgrade
>>>>> https://lists.apache.org/thread.html/eb1a8df2d0a6a79c5d50540d1fdbf215ec83d831ff15d3117c2592cc@%3Cdev.metron.apache.org%3E
>>>>> .
>>>>>
>>>>> Is there a way we can avoid a 1.x release? If we do need 1.x, do we
>>>>> still see upgrades as a gating function? The main issue is that this has
>>>>> the potential to drag out the upgrade and further couple it with other
>>>>> features. And with Storm 1.x being eol'ed, I'm not sure this is something
>>>>> we can wait much longer for. I'll think on this and send out my own
>>>>> thoughts once folks have had a chance to review.
>>>>>
>>>>> Best,
>>>>> Mike Miklavcic
>>>>> Apache Metron, PMC, committer
>>>>>
>>>>>
>>>>> --
>>>>> --
>>>>> simon elliston ball
>>>>> @sireb
>>>>>
>>>>>
>>>>>
>>>>> -------------------
>>>>> Thank you,
>>>>>
>>>>> James Sirota
>>>>> PMC- Apache Metron
>>>>> jsirota AT apache DOT org
>>>>>
>>>>> --
>> --
>> simon elliston ball
>> @sireb
>>
>> --
> --
> simon elliston ball
> @sireb
>
> --
--
simon elliston ball
@sireb

Re: [DISCUSS] HDP 3.1 Upgrade and release strategy

[Otto Fowler <ot...@gmail.com>]

If anyone can think of the things that need to be backed up, please comment
the jira.




On August 27, 2019 at 17:07:20, Otto Fowler (ottobackwards@gmail.com) wrote:

Good idea METRON–2239 [blocker].



On August 27, 2019 at 16:30:13, Simon Elliston Ball (
simon@simonellistonball.com) wrote:

You could always submit a Jira :)

On Tue, 27 Aug 2019 at 21:27, Otto Fowler <ot...@gmail.com> wrote:

> You are right, that is much better than backup_metron_configs.sh.
>
>
>
> On August 27, 2019 at 16:05:38, Simon Elliston Ball (
> simon@simonellistonball.com) wrote:
>
> You can do this with zk_load_configs and Ambari’s blueprint api, so we
> kinda already do.
>
> Simon
>
> On Tue, 27 Aug 2019 at 20:19, Otto Fowler <ot...@gmail.com> wrote:
>
>> Maybe we need some automated method to backup configurations and restore
>> them.
>>
>>
>>
>> On August 27, 2019 at 14:46:58, Michael Miklavcic (
>> michael.miklavcic@gmail.com) wrote:
>>
>> > Once you back up your metron configs, the same configs that worked on
>> the previous version will continue to work on the version running on HDP
>> 3.x.  If there is any discrepancy between the two or additional settings
>> will be required, those will be documented in the release notes.  From the
>> Metron perspective, this upgrade would be no different than simply
>> upgrading to the new Metron version.
>>
>> This upgrade cannot be performed the same way we've done it in the past.
>> A number of platform upgrades, including OS, are required:
>>
>>    1. Requires the OS to be updated on all nodes because there are no
>>    Centos6 RPMs provided in HDP 3.1. Must bump to Centos7.
>>    2. The final new HBase code will not run on HDP 2.6
>>    3. The MPack changes for new Ambari are not backwards compatible
>>    4. YARN and HDFS/MR are also at risk to be backwards incompatible
>>
>>
>> On Tue, Aug 27, 2019 at 12:39 PM Michael Miklavcic <
>> michael.miklavcic@gmail.com> wrote:
>>
>>> Adding the dev list back into the thread (a reply-all was missed).
>>>
>>> On Tue, Aug 27, 2019 at 10:49 AM James Sirota <js...@apache.org>
>>> wrote:
>>>
>>>> I agree with Simon.  HDP 2.x platform is rapidly approaching EOL and
>>>> everyone will likely need to migrate by end of year.  Doing this platform
>>>> upgrade sooner will give everyone visibility into what Metron on HDP 3.x
>>>> looks like so they have time to plan and upgrade at their own pace.
>>>> Feature-wise, the Metron application itself will be unchanged.  It is
>>>> merely the platform underneath that is changing.  HDP itself can be
>>>> upgraded per instructions here:
>>>> https://docs.hortonworks.com/HDPDocuments/HDP3/HDP-3.1.0/release-notes/content/upgrading_parent.html
>>>>
>>>> Once you back up your metron configs, the same configs that worked on
>>>> the previous version will continue to work on the version running on HDP
>>>> 3.x.  If there is any discrepancy between the two or additional settings
>>>> will be required, those will be documented in the release notes.  From the
>>>> Metron perspective, this upgrade would be no different than simply
>>>> upgrading to the new Metron version.
>>>>
>>>> James
>>>>
>>>>
>>>> 27.08.2019, 07:09, "Simon Elliston Ball" <si...@simonellistonball.com>:
>>>>
>>>> Something worth noting here is that HDP 2.6.5 is quite old and
>>>> approaching EoL rapidly, so the issue of upgrade is urgent. I am aware of a
>>>> large number of users who require this upgrade ASAP, and in fact an aware
>>>> of zero users who wish to remain on HDP 2.
>>>>
>>>> Perhaps those users who want to stay on the old platform can stick
>>>> their hands up and raise concerns, but this move will likely have to happen
>>>> very soon.
>>>>
>>>> Simon
>>>>
>>>> On Tue, 27 Aug 2019 at 15:04, Otto Fowler <ot...@gmail.com>
>>>> wrote:
>>>>
>>>> Although we had the discussion, and some great ideas where passed
>>>> around, I do not believe we came to some kind of consensus on what 1.0
>>>> should look like. So that discussion would have to be picked up again so
>>>> that we could know where we are at, and make it an actual thing if we were
>>>> going to make this a 1.0 release.
>>>>
>>>> I believe that the issues raised in that discussion gating 1.0 are
>>>> still largely applicable, including upgrade.
>>>>
>>>> Right now we have *ZERO* HDP 3.1 users. We will go from that to *only*
>>>> supporting 3.1 work and releases. So every user and deployment we currently
>>>> have will feel real pain, have to slay real dragons to move forward with
>>>> metron.
>>>>
>>>> With regards to support for older versions, the “backward capability”
>>>> that has been mentioned, I would not say that I have any specific plan for
>>>> that in mind. What I would say rather, is that I believe that we must be
>>>> explicit, setting expectations correctly and clearly with regards to our
>>>> intent while demonstrating that we have thought through the situation. That
>>>> discussion has not happened, at least I do not believe that the prior dev
>>>> thread really handles it in context.
>>>>
>>>> Depending on the upgrade situation for going to 3.1, it may be that a
>>>> dual stream of releases or fixes or new features to the extent that we can
>>>> do it will greatly reduce the pain for folks, or make it viable to stick
>>>> with metron until they can upgrade.
>>>>
>>>> The issue of what metron *is* features wise may be another one we want
>>>> to take up at some point. The idea being can we separate the metron
>>>> _integration parts from the metron core functionality such that we can work
>>>> on them separately and thus support multiple platforms through
>>>> integrations/applications. Of course definition of metron’s value beyond
>>>> integration, and what those features and application boundaries are would
>>>> be necessary.
>>>>
>>>>
>>>>
>>>> On August 26, 2019 at 18:52:57, Michael Miklavcic (
>>>> michael.miklavcic@gmail.com) wrote:
>>>>
>>>> Hi devs and users,
>>>>
>>>> Some questions were asked in the Slack channel about our ongoing
>>>> HDP/Hadoop upgrade and I'd like to get a discussion rolling. The original
>>>> Hadoop upgrade discuss thread can be found here
>>>> https://lists.apache.org/thread.html/37cc29648f0592cc39d3c78a0d07fce38521bdbbc4cf40e022a7a8ea@%3Cdev.metron.apache.org%3E
>>>>
>>>> The major issue and impact with upgrading the Hadoop platform is that
>>>> there are breaking changes. Code that runs on HDP 3.1 will not run on 2.x.
>>>> Here is a sampling of core components we depend on that we know of so
>>>> far that are not backwards compatible:
>>>>
>>>>    1. The Core OS - we currently base our builds and test deployment
>>>>    off of artifacts pulled from HDP. The latest rev of HDP no longer ships
>>>>    RPMs for Centos 6, which means we need to upgrade to Centos 7
>>>>    2. Ambari
>>>>    3. HBase
>>>>
>>>> This differs from individual components we've upgraded in the past in
>>>> that our code could still be deployed on the old as well as new version of
>>>> the component in a backwards compatible way. Based on semantic versioning,
>>>> I don't know if we can introduce this level of change in a point release,
>>>> which is the reason for kicking off this discussion. In the past, users and
>>>> developers in the community have suggested that they are -1 on a 1.x
>>>> release that does not provide an upgrade
>>>> https://lists.apache.org/thread.html/eb1a8df2d0a6a79c5d50540d1fdbf215ec83d831ff15d3117c2592cc@%3Cdev.metron.apache.org%3E
>>>> .
>>>>
>>>> Is there a way we can avoid a 1.x release? If we do need 1.x, do we
>>>> still see upgrades as a gating function? The main issue is that this has
>>>> the potential to drag out the upgrade and further couple it with other
>>>> features. And with Storm 1.x being eol'ed, I'm not sure this is something
>>>> we can wait much longer for. I'll think on this and send out my own
>>>> thoughts once folks have had a chance to review.
>>>>
>>>> Best,
>>>> Mike Miklavcic
>>>> Apache Metron, PMC, committer
>>>>
>>>>
>>>> --
>>>> --
>>>> simon elliston ball
>>>> @sireb
>>>>
>>>>
>>>>
>>>> -------------------
>>>> Thank you,
>>>>
>>>> James Sirota
>>>> PMC- Apache Metron
>>>> jsirota AT apache DOT org
>>>>
>>>> --
> --
> simon elliston ball
> @sireb
>
> --
--
simon elliston ball
@sireb

Re: [DISCUSS] HDP 3.1 Upgrade and release strategy

[Otto Fowler <ot...@gmail.com>]

If anyone can think of the things that need to be backed up, please comment
the jira.




On August 27, 2019 at 17:07:20, Otto Fowler (ottobackwards@gmail.com) wrote:

Good idea METRON–2239 [blocker].



On August 27, 2019 at 16:30:13, Simon Elliston Ball (
simon@simonellistonball.com) wrote:

You could always submit a Jira :)

On Tue, 27 Aug 2019 at 21:27, Otto Fowler <ot...@gmail.com> wrote:

> You are right, that is much better than backup_metron_configs.sh.
>
>
>
> On August 27, 2019 at 16:05:38, Simon Elliston Ball (
> simon@simonellistonball.com) wrote:
>
> You can do this with zk_load_configs and Ambari’s blueprint api, so we
> kinda already do.
>
> Simon
>
> On Tue, 27 Aug 2019 at 20:19, Otto Fowler <ot...@gmail.com> wrote:
>
>> Maybe we need some automated method to backup configurations and restore
>> them.
>>
>>
>>
>> On August 27, 2019 at 14:46:58, Michael Miklavcic (
>> michael.miklavcic@gmail.com) wrote:
>>
>> > Once you back up your metron configs, the same configs that worked on
>> the previous version will continue to work on the version running on HDP
>> 3.x.  If there is any discrepancy between the two or additional settings
>> will be required, those will be documented in the release notes.  From the
>> Metron perspective, this upgrade would be no different than simply
>> upgrading to the new Metron version.
>>
>> This upgrade cannot be performed the same way we've done it in the past.
>> A number of platform upgrades, including OS, are required:
>>
>>    1. Requires the OS to be updated on all nodes because there are no
>>    Centos6 RPMs provided in HDP 3.1. Must bump to Centos7.
>>    2. The final new HBase code will not run on HDP 2.6
>>    3. The MPack changes for new Ambari are not backwards compatible
>>    4. YARN and HDFS/MR are also at risk to be backwards incompatible
>>
>>
>> On Tue, Aug 27, 2019 at 12:39 PM Michael Miklavcic <
>> michael.miklavcic@gmail.com> wrote:
>>
>>> Adding the dev list back into the thread (a reply-all was missed).
>>>
>>> On Tue, Aug 27, 2019 at 10:49 AM James Sirota <js...@apache.org>
>>> wrote:
>>>
>>>> I agree with Simon.  HDP 2.x platform is rapidly approaching EOL and
>>>> everyone will likely need to migrate by end of year.  Doing this platform
>>>> upgrade sooner will give everyone visibility into what Metron on HDP 3.x
>>>> looks like so they have time to plan and upgrade at their own pace.
>>>> Feature-wise, the Metron application itself will be unchanged.  It is
>>>> merely the platform underneath that is changing.  HDP itself can be
>>>> upgraded per instructions here:
>>>> https://docs.hortonworks.com/HDPDocuments/HDP3/HDP-3.1.0/release-notes/content/upgrading_parent.html
>>>>
>>>> Once you back up your metron configs, the same configs that worked on
>>>> the previous version will continue to work on the version running on HDP
>>>> 3.x.  If there is any discrepancy between the two or additional settings
>>>> will be required, those will be documented in the release notes.  From the
>>>> Metron perspective, this upgrade would be no different than simply
>>>> upgrading to the new Metron version.
>>>>
>>>> James
>>>>
>>>>
>>>> 27.08.2019, 07:09, "Simon Elliston Ball" <si...@simonellistonball.com>:
>>>>
>>>> Something worth noting here is that HDP 2.6.5 is quite old and
>>>> approaching EoL rapidly, so the issue of upgrade is urgent. I am aware of a
>>>> large number of users who require this upgrade ASAP, and in fact an aware
>>>> of zero users who wish to remain on HDP 2.
>>>>
>>>> Perhaps those users who want to stay on the old platform can stick
>>>> their hands up and raise concerns, but this move will likely have to happen
>>>> very soon.
>>>>
>>>> Simon
>>>>
>>>> On Tue, 27 Aug 2019 at 15:04, Otto Fowler <ot...@gmail.com>
>>>> wrote:
>>>>
>>>> Although we had the discussion, and some great ideas where passed
>>>> around, I do not believe we came to some kind of consensus on what 1.0
>>>> should look like. So that discussion would have to be picked up again so
>>>> that we could know where we are at, and make it an actual thing if we were
>>>> going to make this a 1.0 release.
>>>>
>>>> I believe that the issues raised in that discussion gating 1.0 are
>>>> still largely applicable, including upgrade.
>>>>
>>>> Right now we have *ZERO* HDP 3.1 users. We will go from that to *only*
>>>> supporting 3.1 work and releases. So every user and deployment we currently
>>>> have will feel real pain, have to slay real dragons to move forward with
>>>> metron.
>>>>
>>>> With regards to support for older versions, the “backward capability”
>>>> that has been mentioned, I would not say that I have any specific plan for
>>>> that in mind. What I would say rather, is that I believe that we must be
>>>> explicit, setting expectations correctly and clearly with regards to our
>>>> intent while demonstrating that we have thought through the situation. That
>>>> discussion has not happened, at least I do not believe that the prior dev
>>>> thread really handles it in context.
>>>>
>>>> Depending on the upgrade situation for going to 3.1, it may be that a
>>>> dual stream of releases or fixes or new features to the extent that we can
>>>> do it will greatly reduce the pain for folks, or make it viable to stick
>>>> with metron until they can upgrade.
>>>>
>>>> The issue of what metron *is* features wise may be another one we want
>>>> to take up at some point. The idea being can we separate the metron
>>>> _integration parts from the metron core functionality such that we can work
>>>> on them separately and thus support multiple platforms through
>>>> integrations/applications. Of course definition of metron’s value beyond
>>>> integration, and what those features and application boundaries are would
>>>> be necessary.
>>>>
>>>>
>>>>
>>>> On August 26, 2019 at 18:52:57, Michael Miklavcic (
>>>> michael.miklavcic@gmail.com) wrote:
>>>>
>>>> Hi devs and users,
>>>>
>>>> Some questions were asked in the Slack channel about our ongoing
>>>> HDP/Hadoop upgrade and I'd like to get a discussion rolling. The original
>>>> Hadoop upgrade discuss thread can be found here
>>>> https://lists.apache.org/thread.html/37cc29648f0592cc39d3c78a0d07fce38521bdbbc4cf40e022a7a8ea@%3Cdev.metron.apache.org%3E
>>>>
>>>> The major issue and impact with upgrading the Hadoop platform is that
>>>> there are breaking changes. Code that runs on HDP 3.1 will not run on 2.x.
>>>> Here is a sampling of core components we depend on that we know of so
>>>> far that are not backwards compatible:
>>>>
>>>>    1. The Core OS - we currently base our builds and test deployment
>>>>    off of artifacts pulled from HDP. The latest rev of HDP no longer ships
>>>>    RPMs for Centos 6, which means we need to upgrade to Centos 7
>>>>    2. Ambari
>>>>    3. HBase
>>>>
>>>> This differs from individual components we've upgraded in the past in
>>>> that our code could still be deployed on the old as well as new version of
>>>> the component in a backwards compatible way. Based on semantic versioning,
>>>> I don't know if we can introduce this level of change in a point release,
>>>> which is the reason for kicking off this discussion. In the past, users and
>>>> developers in the community have suggested that they are -1 on a 1.x
>>>> release that does not provide an upgrade
>>>> https://lists.apache.org/thread.html/eb1a8df2d0a6a79c5d50540d1fdbf215ec83d831ff15d3117c2592cc@%3Cdev.metron.apache.org%3E
>>>> .
>>>>
>>>> Is there a way we can avoid a 1.x release? If we do need 1.x, do we
>>>> still see upgrades as a gating function? The main issue is that this has
>>>> the potential to drag out the upgrade and further couple it with other
>>>> features. And with Storm 1.x being eol'ed, I'm not sure this is something
>>>> we can wait much longer for. I'll think on this and send out my own
>>>> thoughts once folks have had a chance to review.
>>>>
>>>> Best,
>>>> Mike Miklavcic
>>>> Apache Metron, PMC, committer
>>>>
>>>>
>>>> --
>>>> --
>>>> simon elliston ball
>>>> @sireb
>>>>
>>>>
>>>>
>>>> -------------------
>>>> Thank you,
>>>>
>>>> James Sirota
>>>> PMC- Apache Metron
>>>> jsirota AT apache DOT org
>>>>
>>>> --
> --
> simon elliston ball
> @sireb
>
> --
--
simon elliston ball
@sireb

Re: [DISCUSS] HDP 3.1 Upgrade and release strategy

[Otto Fowler <ot...@gmail.com>]

Good idea METRON–2239 [blocker].




On August 27, 2019 at 16:30:13, Simon Elliston Ball (
simon@simonellistonball.com) wrote:

You could always submit a Jira :)

On Tue, 27 Aug 2019 at 21:27, Otto Fowler <ot...@gmail.com> wrote:

> You are right, that is much better than backup_metron_configs.sh.
>
>
>
> On August 27, 2019 at 16:05:38, Simon Elliston Ball (
> simon@simonellistonball.com) wrote:
>
> You can do this with zk_load_configs and Ambari’s blueprint api, so we
> kinda already do.
>
> Simon
>
> On Tue, 27 Aug 2019 at 20:19, Otto Fowler <ot...@gmail.com> wrote:
>
>> Maybe we need some automated method to backup configurations and restore
>> them.
>>
>>
>>
>> On August 27, 2019 at 14:46:58, Michael Miklavcic (
>> michael.miklavcic@gmail.com) wrote:
>>
>> > Once you back up your metron configs, the same configs that worked on
>> the previous version will continue to work on the version running on HDP
>> 3.x.  If there is any discrepancy between the two or additional settings
>> will be required, those will be documented in the release notes.  From the
>> Metron perspective, this upgrade would be no different than simply
>> upgrading to the new Metron version.
>>
>> This upgrade cannot be performed the same way we've done it in the past.
>> A number of platform upgrades, including OS, are required:
>>
>>    1. Requires the OS to be updated on all nodes because there are no
>>    Centos6 RPMs provided in HDP 3.1. Must bump to Centos7.
>>    2. The final new HBase code will not run on HDP 2.6
>>    3. The MPack changes for new Ambari are not backwards compatible
>>    4. YARN and HDFS/MR are also at risk to be backwards incompatible
>>
>>
>> On Tue, Aug 27, 2019 at 12:39 PM Michael Miklavcic <
>> michael.miklavcic@gmail.com> wrote:
>>
>>> Adding the dev list back into the thread (a reply-all was missed).
>>>
>>> On Tue, Aug 27, 2019 at 10:49 AM James Sirota <js...@apache.org>
>>> wrote:
>>>
>>>> I agree with Simon.  HDP 2.x platform is rapidly approaching EOL and
>>>> everyone will likely need to migrate by end of year.  Doing this platform
>>>> upgrade sooner will give everyone visibility into what Metron on HDP 3.x
>>>> looks like so they have time to plan and upgrade at their own pace.
>>>> Feature-wise, the Metron application itself will be unchanged.  It is
>>>> merely the platform underneath that is changing.  HDP itself can be
>>>> upgraded per instructions here:
>>>> https://docs.hortonworks.com/HDPDocuments/HDP3/HDP-3.1.0/release-notes/content/upgrading_parent.html
>>>>
>>>> Once you back up your metron configs, the same configs that worked on
>>>> the previous version will continue to work on the version running on HDP
>>>> 3.x.  If there is any discrepancy between the two or additional settings
>>>> will be required, those will be documented in the release notes.  From the
>>>> Metron perspective, this upgrade would be no different than simply
>>>> upgrading to the new Metron version.
>>>>
>>>> James
>>>>
>>>>
>>>> 27.08.2019, 07:09, "Simon Elliston Ball" <si...@simonellistonball.com>:
>>>>
>>>> Something worth noting here is that HDP 2.6.5 is quite old and
>>>> approaching EoL rapidly, so the issue of upgrade is urgent. I am aware of a
>>>> large number of users who require this upgrade ASAP, and in fact an aware
>>>> of zero users who wish to remain on HDP 2.
>>>>
>>>> Perhaps those users who want to stay on the old platform can stick
>>>> their hands up and raise concerns, but this move will likely have to happen
>>>> very soon.
>>>>
>>>> Simon
>>>>
>>>> On Tue, 27 Aug 2019 at 15:04, Otto Fowler <ot...@gmail.com>
>>>> wrote:
>>>>
>>>> Although we had the discussion, and some great ideas where passed
>>>> around, I do not believe we came to some kind of consensus on what 1.0
>>>> should look like. So that discussion would have to be picked up again so
>>>> that we could know where we are at, and make it an actual thing if we were
>>>> going to make this a 1.0 release.
>>>>
>>>> I believe that the issues raised in that discussion gating 1.0 are
>>>> still largely applicable, including upgrade.
>>>>
>>>> Right now we have *ZERO* HDP 3.1 users. We will go from that to *only*
>>>> supporting 3.1 work and releases. So every user and deployment we currently
>>>> have will feel real pain, have to slay real dragons to move forward with
>>>> metron.
>>>>
>>>> With regards to support for older versions, the “backward capability”
>>>> that has been mentioned, I would not say that I have any specific plan for
>>>> that in mind. What I would say rather, is that I believe that we must be
>>>> explicit, setting expectations correctly and clearly with regards to our
>>>> intent while demonstrating that we have thought through the situation. That
>>>> discussion has not happened, at least I do not believe that the prior dev
>>>> thread really handles it in context.
>>>>
>>>> Depending on the upgrade situation for going to 3.1, it may be that a
>>>> dual stream of releases or fixes or new features to the extent that we can
>>>> do it will greatly reduce the pain for folks, or make it viable to stick
>>>> with metron until they can upgrade.
>>>>
>>>> The issue of what metron *is* features wise may be another one we want
>>>> to take up at some point. The idea being can we separate the metron
>>>> _integration parts from the metron core functionality such that we can work
>>>> on them separately and thus support multiple platforms through
>>>> integrations/applications. Of course definition of metron’s value beyond
>>>> integration, and what those features and application boundaries are would
>>>> be necessary.
>>>>
>>>>
>>>>
>>>> On August 26, 2019 at 18:52:57, Michael Miklavcic (
>>>> michael.miklavcic@gmail.com) wrote:
>>>>
>>>> Hi devs and users,
>>>>
>>>> Some questions were asked in the Slack channel about our ongoing
>>>> HDP/Hadoop upgrade and I'd like to get a discussion rolling. The original
>>>> Hadoop upgrade discuss thread can be found here
>>>> https://lists.apache.org/thread.html/37cc29648f0592cc39d3c78a0d07fce38521bdbbc4cf40e022a7a8ea@%3Cdev.metron.apache.org%3E
>>>>
>>>> The major issue and impact with upgrading the Hadoop platform is that
>>>> there are breaking changes. Code that runs on HDP 3.1 will not run on 2.x.
>>>> Here is a sampling of core components we depend on that we know of so
>>>> far that are not backwards compatible:
>>>>
>>>>    1. The Core OS - we currently base our builds and test deployment
>>>>    off of artifacts pulled from HDP. The latest rev of HDP no longer ships
>>>>    RPMs for Centos 6, which means we need to upgrade to Centos 7
>>>>    2. Ambari
>>>>    3. HBase
>>>>
>>>> This differs from individual components we've upgraded in the past in
>>>> that our code could still be deployed on the old as well as new version of
>>>> the component in a backwards compatible way. Based on semantic versioning,
>>>> I don't know if we can introduce this level of change in a point release,
>>>> which is the reason for kicking off this discussion. In the past, users and
>>>> developers in the community have suggested that they are -1 on a 1.x
>>>> release that does not provide an upgrade
>>>> https://lists.apache.org/thread.html/eb1a8df2d0a6a79c5d50540d1fdbf215ec83d831ff15d3117c2592cc@%3Cdev.metron.apache.org%3E
>>>> .
>>>>
>>>> Is there a way we can avoid a 1.x release? If we do need 1.x, do we
>>>> still see upgrades as a gating function? The main issue is that this has
>>>> the potential to drag out the upgrade and further couple it with other
>>>> features. And with Storm 1.x being eol'ed, I'm not sure this is something
>>>> we can wait much longer for. I'll think on this and send out my own
>>>> thoughts once folks have had a chance to review.
>>>>
>>>> Best,
>>>> Mike Miklavcic
>>>> Apache Metron, PMC, committer
>>>>
>>>>
>>>> --
>>>> --
>>>> simon elliston ball
>>>> @sireb
>>>>
>>>>
>>>>
>>>> -------------------
>>>> Thank you,
>>>>
>>>> James Sirota
>>>> PMC- Apache Metron
>>>> jsirota AT apache DOT org
>>>>
>>>> --
> --
> simon elliston ball
> @sireb
>
> --
--
simon elliston ball
@sireb

Re: [DISCUSS] HDP 3.1 Upgrade and release strategy

[Otto Fowler <ot...@gmail.com>]

Good idea METRON–2239 [blocker].




On August 27, 2019 at 16:30:13, Simon Elliston Ball (
simon@simonellistonball.com) wrote:

You could always submit a Jira :)

On Tue, 27 Aug 2019 at 21:27, Otto Fowler <ot...@gmail.com> wrote:

> You are right, that is much better than backup_metron_configs.sh.
>
>
>
> On August 27, 2019 at 16:05:38, Simon Elliston Ball (
> simon@simonellistonball.com) wrote:
>
> You can do this with zk_load_configs and Ambari’s blueprint api, so we
> kinda already do.
>
> Simon
>
> On Tue, 27 Aug 2019 at 20:19, Otto Fowler <ot...@gmail.com> wrote:
>
>> Maybe we need some automated method to backup configurations and restore
>> them.
>>
>>
>>
>> On August 27, 2019 at 14:46:58, Michael Miklavcic (
>> michael.miklavcic@gmail.com) wrote:
>>
>> > Once you back up your metron configs, the same configs that worked on
>> the previous version will continue to work on the version running on HDP
>> 3.x.  If there is any discrepancy between the two or additional settings
>> will be required, those will be documented in the release notes.  From the
>> Metron perspective, this upgrade would be no different than simply
>> upgrading to the new Metron version.
>>
>> This upgrade cannot be performed the same way we've done it in the past.
>> A number of platform upgrades, including OS, are required:
>>
>>    1. Requires the OS to be updated on all nodes because there are no
>>    Centos6 RPMs provided in HDP 3.1. Must bump to Centos7.
>>    2. The final new HBase code will not run on HDP 2.6
>>    3. The MPack changes for new Ambari are not backwards compatible
>>    4. YARN and HDFS/MR are also at risk to be backwards incompatible
>>
>>
>> On Tue, Aug 27, 2019 at 12:39 PM Michael Miklavcic <
>> michael.miklavcic@gmail.com> wrote:
>>
>>> Adding the dev list back into the thread (a reply-all was missed).
>>>
>>> On Tue, Aug 27, 2019 at 10:49 AM James Sirota <js...@apache.org>
>>> wrote:
>>>
>>>> I agree with Simon.  HDP 2.x platform is rapidly approaching EOL and
>>>> everyone will likely need to migrate by end of year.  Doing this platform
>>>> upgrade sooner will give everyone visibility into what Metron on HDP 3.x
>>>> looks like so they have time to plan and upgrade at their own pace.
>>>> Feature-wise, the Metron application itself will be unchanged.  It is
>>>> merely the platform underneath that is changing.  HDP itself can be
>>>> upgraded per instructions here:
>>>> https://docs.hortonworks.com/HDPDocuments/HDP3/HDP-3.1.0/release-notes/content/upgrading_parent.html
>>>>
>>>> Once you back up your metron configs, the same configs that worked on
>>>> the previous version will continue to work on the version running on HDP
>>>> 3.x.  If there is any discrepancy between the two or additional settings
>>>> will be required, those will be documented in the release notes.  From the
>>>> Metron perspective, this upgrade would be no different than simply
>>>> upgrading to the new Metron version.
>>>>
>>>> James
>>>>
>>>>
>>>> 27.08.2019, 07:09, "Simon Elliston Ball" <si...@simonellistonball.com>:
>>>>
>>>> Something worth noting here is that HDP 2.6.5 is quite old and
>>>> approaching EoL rapidly, so the issue of upgrade is urgent. I am aware of a
>>>> large number of users who require this upgrade ASAP, and in fact an aware
>>>> of zero users who wish to remain on HDP 2.
>>>>
>>>> Perhaps those users who want to stay on the old platform can stick
>>>> their hands up and raise concerns, but this move will likely have to happen
>>>> very soon.
>>>>
>>>> Simon
>>>>
>>>> On Tue, 27 Aug 2019 at 15:04, Otto Fowler <ot...@gmail.com>
>>>> wrote:
>>>>
>>>> Although we had the discussion, and some great ideas where passed
>>>> around, I do not believe we came to some kind of consensus on what 1.0
>>>> should look like. So that discussion would have to be picked up again so
>>>> that we could know where we are at, and make it an actual thing if we were
>>>> going to make this a 1.0 release.
>>>>
>>>> I believe that the issues raised in that discussion gating 1.0 are
>>>> still largely applicable, including upgrade.
>>>>
>>>> Right now we have *ZERO* HDP 3.1 users. We will go from that to *only*
>>>> supporting 3.1 work and releases. So every user and deployment we currently
>>>> have will feel real pain, have to slay real dragons to move forward with
>>>> metron.
>>>>
>>>> With regards to support for older versions, the “backward capability”
>>>> that has been mentioned, I would not say that I have any specific plan for
>>>> that in mind. What I would say rather, is that I believe that we must be
>>>> explicit, setting expectations correctly and clearly with regards to our
>>>> intent while demonstrating that we have thought through the situation. That
>>>> discussion has not happened, at least I do not believe that the prior dev
>>>> thread really handles it in context.
>>>>
>>>> Depending on the upgrade situation for going to 3.1, it may be that a
>>>> dual stream of releases or fixes or new features to the extent that we can
>>>> do it will greatly reduce the pain for folks, or make it viable to stick
>>>> with metron until they can upgrade.
>>>>
>>>> The issue of what metron *is* features wise may be another one we want
>>>> to take up at some point. The idea being can we separate the metron
>>>> _integration parts from the metron core functionality such that we can work
>>>> on them separately and thus support multiple platforms through
>>>> integrations/applications. Of course definition of metron’s value beyond
>>>> integration, and what those features and application boundaries are would
>>>> be necessary.
>>>>
>>>>
>>>>
>>>> On August 26, 2019 at 18:52:57, Michael Miklavcic (
>>>> michael.miklavcic@gmail.com) wrote:
>>>>
>>>> Hi devs and users,
>>>>
>>>> Some questions were asked in the Slack channel about our ongoing
>>>> HDP/Hadoop upgrade and I'd like to get a discussion rolling. The original
>>>> Hadoop upgrade discuss thread can be found here
>>>> https://lists.apache.org/thread.html/37cc29648f0592cc39d3c78a0d07fce38521bdbbc4cf40e022a7a8ea@%3Cdev.metron.apache.org%3E
>>>>
>>>> The major issue and impact with upgrading the Hadoop platform is that
>>>> there are breaking changes. Code that runs on HDP 3.1 will not run on 2.x.
>>>> Here is a sampling of core components we depend on that we know of so
>>>> far that are not backwards compatible:
>>>>
>>>>    1. The Core OS - we currently base our builds and test deployment
>>>>    off of artifacts pulled from HDP. The latest rev of HDP no longer ships
>>>>    RPMs for Centos 6, which means we need to upgrade to Centos 7
>>>>    2. Ambari
>>>>    3. HBase
>>>>
>>>> This differs from individual components we've upgraded in the past in
>>>> that our code could still be deployed on the old as well as new version of
>>>> the component in a backwards compatible way. Based on semantic versioning,
>>>> I don't know if we can introduce this level of change in a point release,
>>>> which is the reason for kicking off this discussion. In the past, users and
>>>> developers in the community have suggested that they are -1 on a 1.x
>>>> release that does not provide an upgrade
>>>> https://lists.apache.org/thread.html/eb1a8df2d0a6a79c5d50540d1fdbf215ec83d831ff15d3117c2592cc@%3Cdev.metron.apache.org%3E
>>>> .
>>>>
>>>> Is there a way we can avoid a 1.x release? If we do need 1.x, do we
>>>> still see upgrades as a gating function? The main issue is that this has
>>>> the potential to drag out the upgrade and further couple it with other
>>>> features. And with Storm 1.x being eol'ed, I'm not sure this is something
>>>> we can wait much longer for. I'll think on this and send out my own
>>>> thoughts once folks have had a chance to review.
>>>>
>>>> Best,
>>>> Mike Miklavcic
>>>> Apache Metron, PMC, committer
>>>>
>>>>
>>>> --
>>>> --
>>>> simon elliston ball
>>>> @sireb
>>>>
>>>>
>>>>
>>>> -------------------
>>>> Thank you,
>>>>
>>>> James Sirota
>>>> PMC- Apache Metron
>>>> jsirota AT apache DOT org
>>>>
>>>> --
> --
> simon elliston ball
> @sireb
>
> --
--
simon elliston ball
@sireb

Re: [DISCUSS] HDP 3.1 Upgrade and release strategy

[Simon Elliston Ball <si...@simonellistonball.com>]

You could always submit a Jira :)

On Tue, 27 Aug 2019 at 21:27, Otto Fowler <ot...@gmail.com> wrote:

> You are right, that is much better than backup_metron_configs.sh.
>
>
>
>
> On August 27, 2019 at 16:05:38, Simon Elliston Ball (
> simon@simonellistonball.com) wrote:
>
> You can do this with zk_load_configs and Ambari’s blueprint api, so we
> kinda already do.
>
> Simon
>
> On Tue, 27 Aug 2019 at 20:19, Otto Fowler <ot...@gmail.com> wrote:
>
>> Maybe we need some automated method to backup configurations and restore
>> them.
>>
>>
>>
>> On August 27, 2019 at 14:46:58, Michael Miklavcic (
>> michael.miklavcic@gmail.com) wrote:
>>
>> > Once you back up your metron configs, the same configs that worked on
>> the previous version will continue to work on the version running on HDP
>> 3.x.  If there is any discrepancy between the two or additional settings
>> will be required, those will be documented in the release notes.  From the
>> Metron perspective, this upgrade would be no different than simply
>> upgrading to the new Metron version.
>>
>> This upgrade cannot be performed the same way we've done it in the past.
>> A number of platform upgrades, including OS, are required:
>>
>>    1. Requires the OS to be updated on all nodes because there are no
>>    Centos6 RPMs provided in HDP 3.1. Must bump to Centos7.
>>    2. The final new HBase code will not run on HDP 2.6
>>    3. The MPack changes for new Ambari are not backwards compatible
>>    4. YARN and HDFS/MR are also at risk to be backwards incompatible
>>
>>
>> On Tue, Aug 27, 2019 at 12:39 PM Michael Miklavcic <
>> michael.miklavcic@gmail.com> wrote:
>>
>>> Adding the dev list back into the thread (a reply-all was missed).
>>>
>>> On Tue, Aug 27, 2019 at 10:49 AM James Sirota <js...@apache.org>
>>> wrote:
>>>
>>>> I agree with Simon.  HDP 2.x platform is rapidly approaching EOL and
>>>> everyone will likely need to migrate by end of year.  Doing this platform
>>>> upgrade sooner will give everyone visibility into what Metron on HDP 3.x
>>>> looks like so they have time to plan and upgrade at their own pace.
>>>> Feature-wise, the Metron application itself will be unchanged.  It is
>>>> merely the platform underneath that is changing.  HDP itself can be
>>>> upgraded per instructions here:
>>>> https://docs.hortonworks.com/HDPDocuments/HDP3/HDP-3.1.0/release-notes/content/upgrading_parent.html
>>>>
>>>> Once you back up your metron configs, the same configs that worked on
>>>> the previous version will continue to work on the version running on HDP
>>>> 3.x.  If there is any discrepancy between the two or additional settings
>>>> will be required, those will be documented in the release notes.  From the
>>>> Metron perspective, this upgrade would be no different than simply
>>>> upgrading to the new Metron version.
>>>>
>>>> James
>>>>
>>>>
>>>> 27.08.2019, 07:09, "Simon Elliston Ball" <si...@simonellistonball.com>:
>>>>
>>>> Something worth noting here is that HDP 2.6.5 is quite old and
>>>> approaching EoL rapidly, so the issue of upgrade is urgent. I am aware of a
>>>> large number of users who require this upgrade ASAP, and in fact an aware
>>>> of zero users who wish to remain on HDP 2.
>>>>
>>>> Perhaps those users who want to stay on the old platform can stick
>>>> their hands up and raise concerns, but this move will likely have to happen
>>>> very soon.
>>>>
>>>> Simon
>>>>
>>>> On Tue, 27 Aug 2019 at 15:04, Otto Fowler <ot...@gmail.com>
>>>> wrote:
>>>>
>>>> Although we had the discussion, and some great ideas where passed
>>>> around, I do not believe we came to some kind of consensus on what 1.0
>>>> should look like. So that discussion would have to be picked up again so
>>>> that we could know where we are at, and make it an actual thing if we were
>>>> going to make this a 1.0 release.
>>>>
>>>> I believe that the issues raised in that discussion gating 1.0 are
>>>> still largely applicable, including upgrade.
>>>>
>>>> Right now we have *ZERO* HDP 3.1 users. We will go from that to *only*
>>>> supporting 3.1 work and releases. So every user and deployment we currently
>>>> have will feel real pain, have to slay real dragons to move forward with
>>>> metron.
>>>>
>>>> With regards to support for older versions, the “backward capability”
>>>> that has been mentioned, I would not say that I have any specific plan for
>>>> that in mind. What I would say rather, is that I believe that we must be
>>>> explicit, setting expectations correctly and clearly with regards to our
>>>> intent while demonstrating that we have thought through the situation. That
>>>> discussion has not happened, at least I do not believe that the prior dev
>>>> thread really handles it in context.
>>>>
>>>> Depending on the upgrade situation for going to 3.1, it may be that a
>>>> dual stream of releases or fixes or new features to the extent that we can
>>>> do it will greatly reduce the pain for folks, or make it viable to stick
>>>> with metron until they can upgrade.
>>>>
>>>> The issue of what metron *is* features wise may be another one we want
>>>> to take up at some point. The idea being can we separate the metron
>>>> _integration parts from the metron core functionality such that we can work
>>>> on them separately and thus support multiple platforms through
>>>> integrations/applications. Of course definition of metron’s value beyond
>>>> integration, and what those features and application boundaries are would
>>>> be necessary.
>>>>
>>>>
>>>>
>>>> On August 26, 2019 at 18:52:57, Michael Miklavcic (
>>>> michael.miklavcic@gmail.com) wrote:
>>>>
>>>> Hi devs and users,
>>>>
>>>> Some questions were asked in the Slack channel about our ongoing
>>>> HDP/Hadoop upgrade and I'd like to get a discussion rolling. The original
>>>> Hadoop upgrade discuss thread can be found here
>>>> https://lists.apache.org/thread.html/37cc29648f0592cc39d3c78a0d07fce38521bdbbc4cf40e022a7a8ea@%3Cdev.metron.apache.org%3E
>>>>
>>>> The major issue and impact with upgrading the Hadoop platform is that
>>>> there are breaking changes. Code that runs on HDP 3.1 will not run on 2.x.
>>>> Here is a sampling of core components we depend on that we know of so
>>>> far that are not backwards compatible:
>>>>
>>>>    1. The Core OS - we currently base our builds and test deployment
>>>>    off of artifacts pulled from HDP. The latest rev of HDP no longer ships
>>>>    RPMs for Centos 6, which means we need to upgrade to Centos 7
>>>>    2. Ambari
>>>>    3. HBase
>>>>
>>>> This differs from individual components we've upgraded in the past in
>>>> that our code could still be deployed on the old as well as new version of
>>>> the component in a backwards compatible way. Based on semantic versioning,
>>>> I don't know if we can introduce this level of change in a point release,
>>>> which is the reason for kicking off this discussion. In the past, users and
>>>> developers in the community have suggested that they are -1 on a 1.x
>>>> release that does not provide an upgrade
>>>> https://lists.apache.org/thread.html/eb1a8df2d0a6a79c5d50540d1fdbf215ec83d831ff15d3117c2592cc@%3Cdev.metron.apache.org%3E
>>>> .
>>>>
>>>> Is there a way we can avoid a 1.x release? If we do need 1.x, do we
>>>> still see upgrades as a gating function? The main issue is that this has
>>>> the potential to drag out the upgrade and further couple it with other
>>>> features. And with Storm 1.x being eol'ed, I'm not sure this is something
>>>> we can wait much longer for. I'll think on this and send out my own
>>>> thoughts once folks have had a chance to review.
>>>>
>>>> Best,
>>>> Mike Miklavcic
>>>> Apache Metron, PMC, committer
>>>>
>>>>
>>>> --
>>>> --
>>>> simon elliston ball
>>>> @sireb
>>>>
>>>>
>>>>
>>>> -------------------
>>>> Thank you,
>>>>
>>>> James Sirota
>>>> PMC- Apache Metron
>>>> jsirota AT apache DOT org
>>>>
>>>> --
> --
> simon elliston ball
> @sireb
>
> --
--
simon elliston ball
@sireb

Re: [DISCUSS] HDP 3.1 Upgrade and release strategy

[Simon Elliston Ball <si...@simonellistonball.com>]

You could always submit a Jira :)

On Tue, 27 Aug 2019 at 21:27, Otto Fowler <ot...@gmail.com> wrote:

> You are right, that is much better than backup_metron_configs.sh.
>
>
>
>
> On August 27, 2019 at 16:05:38, Simon Elliston Ball (
> simon@simonellistonball.com) wrote:
>
> You can do this with zk_load_configs and Ambari’s blueprint api, so we
> kinda already do.
>
> Simon
>
> On Tue, 27 Aug 2019 at 20:19, Otto Fowler <ot...@gmail.com> wrote:
>
>> Maybe we need some automated method to backup configurations and restore
>> them.
>>
>>
>>
>> On August 27, 2019 at 14:46:58, Michael Miklavcic (
>> michael.miklavcic@gmail.com) wrote:
>>
>> > Once you back up your metron configs, the same configs that worked on
>> the previous version will continue to work on the version running on HDP
>> 3.x.  If there is any discrepancy between the two or additional settings
>> will be required, those will be documented in the release notes.  From the
>> Metron perspective, this upgrade would be no different than simply
>> upgrading to the new Metron version.
>>
>> This upgrade cannot be performed the same way we've done it in the past.
>> A number of platform upgrades, including OS, are required:
>>
>>    1. Requires the OS to be updated on all nodes because there are no
>>    Centos6 RPMs provided in HDP 3.1. Must bump to Centos7.
>>    2. The final new HBase code will not run on HDP 2.6
>>    3. The MPack changes for new Ambari are not backwards compatible
>>    4. YARN and HDFS/MR are also at risk to be backwards incompatible
>>
>>
>> On Tue, Aug 27, 2019 at 12:39 PM Michael Miklavcic <
>> michael.miklavcic@gmail.com> wrote:
>>
>>> Adding the dev list back into the thread (a reply-all was missed).
>>>
>>> On Tue, Aug 27, 2019 at 10:49 AM James Sirota <js...@apache.org>
>>> wrote:
>>>
>>>> I agree with Simon.  HDP 2.x platform is rapidly approaching EOL and
>>>> everyone will likely need to migrate by end of year.  Doing this platform
>>>> upgrade sooner will give everyone visibility into what Metron on HDP 3.x
>>>> looks like so they have time to plan and upgrade at their own pace.
>>>> Feature-wise, the Metron application itself will be unchanged.  It is
>>>> merely the platform underneath that is changing.  HDP itself can be
>>>> upgraded per instructions here:
>>>> https://docs.hortonworks.com/HDPDocuments/HDP3/HDP-3.1.0/release-notes/content/upgrading_parent.html
>>>>
>>>> Once you back up your metron configs, the same configs that worked on
>>>> the previous version will continue to work on the version running on HDP
>>>> 3.x.  If there is any discrepancy between the two or additional settings
>>>> will be required, those will be documented in the release notes.  From the
>>>> Metron perspective, this upgrade would be no different than simply
>>>> upgrading to the new Metron version.
>>>>
>>>> James
>>>>
>>>>
>>>> 27.08.2019, 07:09, "Simon Elliston Ball" <si...@simonellistonball.com>:
>>>>
>>>> Something worth noting here is that HDP 2.6.5 is quite old and
>>>> approaching EoL rapidly, so the issue of upgrade is urgent. I am aware of a
>>>> large number of users who require this upgrade ASAP, and in fact an aware
>>>> of zero users who wish to remain on HDP 2.
>>>>
>>>> Perhaps those users who want to stay on the old platform can stick
>>>> their hands up and raise concerns, but this move will likely have to happen
>>>> very soon.
>>>>
>>>> Simon
>>>>
>>>> On Tue, 27 Aug 2019 at 15:04, Otto Fowler <ot...@gmail.com>
>>>> wrote:
>>>>
>>>> Although we had the discussion, and some great ideas where passed
>>>> around, I do not believe we came to some kind of consensus on what 1.0
>>>> should look like. So that discussion would have to be picked up again so
>>>> that we could know where we are at, and make it an actual thing if we were
>>>> going to make this a 1.0 release.
>>>>
>>>> I believe that the issues raised in that discussion gating 1.0 are
>>>> still largely applicable, including upgrade.
>>>>
>>>> Right now we have *ZERO* HDP 3.1 users. We will go from that to *only*
>>>> supporting 3.1 work and releases. So every user and deployment we currently
>>>> have will feel real pain, have to slay real dragons to move forward with
>>>> metron.
>>>>
>>>> With regards to support for older versions, the “backward capability”
>>>> that has been mentioned, I would not say that I have any specific plan for
>>>> that in mind. What I would say rather, is that I believe that we must be
>>>> explicit, setting expectations correctly and clearly with regards to our
>>>> intent while demonstrating that we have thought through the situation. That
>>>> discussion has not happened, at least I do not believe that the prior dev
>>>> thread really handles it in context.
>>>>
>>>> Depending on the upgrade situation for going to 3.1, it may be that a
>>>> dual stream of releases or fixes or new features to the extent that we can
>>>> do it will greatly reduce the pain for folks, or make it viable to stick
>>>> with metron until they can upgrade.
>>>>
>>>> The issue of what metron *is* features wise may be another one we want
>>>> to take up at some point. The idea being can we separate the metron
>>>> _integration parts from the metron core functionality such that we can work
>>>> on them separately and thus support multiple platforms through
>>>> integrations/applications. Of course definition of metron’s value beyond
>>>> integration, and what those features and application boundaries are would
>>>> be necessary.
>>>>
>>>>
>>>>
>>>> On August 26, 2019 at 18:52:57, Michael Miklavcic (
>>>> michael.miklavcic@gmail.com) wrote:
>>>>
>>>> Hi devs and users,
>>>>
>>>> Some questions were asked in the Slack channel about our ongoing
>>>> HDP/Hadoop upgrade and I'd like to get a discussion rolling. The original
>>>> Hadoop upgrade discuss thread can be found here
>>>> https://lists.apache.org/thread.html/37cc29648f0592cc39d3c78a0d07fce38521bdbbc4cf40e022a7a8ea@%3Cdev.metron.apache.org%3E
>>>>
>>>> The major issue and impact with upgrading the Hadoop platform is that
>>>> there are breaking changes. Code that runs on HDP 3.1 will not run on 2.x.
>>>> Here is a sampling of core components we depend on that we know of so
>>>> far that are not backwards compatible:
>>>>
>>>>    1. The Core OS - we currently base our builds and test deployment
>>>>    off of artifacts pulled from HDP. The latest rev of HDP no longer ships
>>>>    RPMs for Centos 6, which means we need to upgrade to Centos 7
>>>>    2. Ambari
>>>>    3. HBase
>>>>
>>>> This differs from individual components we've upgraded in the past in
>>>> that our code could still be deployed on the old as well as new version of
>>>> the component in a backwards compatible way. Based on semantic versioning,
>>>> I don't know if we can introduce this level of change in a point release,
>>>> which is the reason for kicking off this discussion. In the past, users and
>>>> developers in the community have suggested that they are -1 on a 1.x
>>>> release that does not provide an upgrade
>>>> https://lists.apache.org/thread.html/eb1a8df2d0a6a79c5d50540d1fdbf215ec83d831ff15d3117c2592cc@%3Cdev.metron.apache.org%3E
>>>> .
>>>>
>>>> Is there a way we can avoid a 1.x release? If we do need 1.x, do we
>>>> still see upgrades as a gating function? The main issue is that this has
>>>> the potential to drag out the upgrade and further couple it with other
>>>> features. And with Storm 1.x being eol'ed, I'm not sure this is something
>>>> we can wait much longer for. I'll think on this and send out my own
>>>> thoughts once folks have had a chance to review.
>>>>
>>>> Best,
>>>> Mike Miklavcic
>>>> Apache Metron, PMC, committer
>>>>
>>>>
>>>> --
>>>> --
>>>> simon elliston ball
>>>> @sireb
>>>>
>>>>
>>>>
>>>> -------------------
>>>> Thank you,
>>>>
>>>> James Sirota
>>>> PMC- Apache Metron
>>>> jsirota AT apache DOT org
>>>>
>>>> --
> --
> simon elliston ball
> @sireb
>
> --
--
simon elliston ball
@sireb

Re: [DISCUSS] HDP 3.1 Upgrade and release strategy

[Otto Fowler <ot...@gmail.com>]

You are right, that is much better than backup_metron_configs.sh.




On August 27, 2019 at 16:05:38, Simon Elliston Ball (
simon@simonellistonball.com) wrote:

You can do this with zk_load_configs and Ambari’s blueprint api, so we
kinda already do.

Simon

On Tue, 27 Aug 2019 at 20:19, Otto Fowler <ot...@gmail.com> wrote:

> Maybe we need some automated method to backup configurations and restore
> them.
>
>
>
> On August 27, 2019 at 14:46:58, Michael Miklavcic (
> michael.miklavcic@gmail.com) wrote:
>
> > Once you back up your metron configs, the same configs that worked on
> the previous version will continue to work on the version running on HDP
> 3.x.  If there is any discrepancy between the two or additional settings
> will be required, those will be documented in the release notes.  From the
> Metron perspective, this upgrade would be no different than simply
> upgrading to the new Metron version.
>
> This upgrade cannot be performed the same way we've done it in the past. A
> number of platform upgrades, including OS, are required:
>
>    1. Requires the OS to be updated on all nodes because there are no
>    Centos6 RPMs provided in HDP 3.1. Must bump to Centos7.
>    2. The final new HBase code will not run on HDP 2.6
>    3. The MPack changes for new Ambari are not backwards compatible
>    4. YARN and HDFS/MR are also at risk to be backwards incompatible
>
>
> On Tue, Aug 27, 2019 at 12:39 PM Michael Miklavcic <
> michael.miklavcic@gmail.com> wrote:
>
>> Adding the dev list back into the thread (a reply-all was missed).
>>
>> On Tue, Aug 27, 2019 at 10:49 AM James Sirota <js...@apache.org> wrote:
>>
>>> I agree with Simon.  HDP 2.x platform is rapidly approaching EOL and
>>> everyone will likely need to migrate by end of year.  Doing this platform
>>> upgrade sooner will give everyone visibility into what Metron on HDP 3.x
>>> looks like so they have time to plan and upgrade at their own pace.
>>> Feature-wise, the Metron application itself will be unchanged.  It is
>>> merely the platform underneath that is changing.  HDP itself can be
>>> upgraded per instructions here:
>>> https://docs.hortonworks.com/HDPDocuments/HDP3/HDP-3.1.0/release-notes/content/upgrading_parent.html
>>>
>>> Once you back up your metron configs, the same configs that worked on
>>> the previous version will continue to work on the version running on HDP
>>> 3.x.  If there is any discrepancy between the two or additional settings
>>> will be required, those will be documented in the release notes.  From the
>>> Metron perspective, this upgrade would be no different than simply
>>> upgrading to the new Metron version.
>>>
>>> James
>>>
>>>
>>> 27.08.2019, 07:09, "Simon Elliston Ball" <si...@simonellistonball.com>:
>>>
>>> Something worth noting here is that HDP 2.6.5 is quite old and
>>> approaching EoL rapidly, so the issue of upgrade is urgent. I am aware of a
>>> large number of users who require this upgrade ASAP, and in fact an aware
>>> of zero users who wish to remain on HDP 2.
>>>
>>> Perhaps those users who want to stay on the old platform can stick their
>>> hands up and raise concerns, but this move will likely have to happen very
>>> soon.
>>>
>>> Simon
>>>
>>> On Tue, 27 Aug 2019 at 15:04, Otto Fowler <ot...@gmail.com>
>>> wrote:
>>>
>>> Although we had the discussion, and some great ideas where passed
>>> around, I do not believe we came to some kind of consensus on what 1.0
>>> should look like. So that discussion would have to be picked up again so
>>> that we could know where we are at, and make it an actual thing if we were
>>> going to make this a 1.0 release.
>>>
>>> I believe that the issues raised in that discussion gating 1.0 are still
>>> largely applicable, including upgrade.
>>>
>>> Right now we have *ZERO* HDP 3.1 users. We will go from that to *only*
>>> supporting 3.1 work and releases. So every user and deployment we currently
>>> have will feel real pain, have to slay real dragons to move forward with
>>> metron.
>>>
>>> With regards to support for older versions, the “backward capability”
>>> that has been mentioned, I would not say that I have any specific plan for
>>> that in mind. What I would say rather, is that I believe that we must be
>>> explicit, setting expectations correctly and clearly with regards to our
>>> intent while demonstrating that we have thought through the situation. That
>>> discussion has not happened, at least I do not believe that the prior dev
>>> thread really handles it in context.
>>>
>>> Depending on the upgrade situation for going to 3.1, it may be that a
>>> dual stream of releases or fixes or new features to the extent that we can
>>> do it will greatly reduce the pain for folks, or make it viable to stick
>>> with metron until they can upgrade.
>>>
>>> The issue of what metron *is* features wise may be another one we want
>>> to take up at some point. The idea being can we separate the metron
>>> _integration parts from the metron core functionality such that we can work
>>> on them separately and thus support multiple platforms through
>>> integrations/applications. Of course definition of metron’s value beyond
>>> integration, and what those features and application boundaries are would
>>> be necessary.
>>>
>>>
>>>
>>> On August 26, 2019 at 18:52:57, Michael Miklavcic (
>>> michael.miklavcic@gmail.com) wrote:
>>>
>>> Hi devs and users,
>>>
>>> Some questions were asked in the Slack channel about our ongoing
>>> HDP/Hadoop upgrade and I'd like to get a discussion rolling. The original
>>> Hadoop upgrade discuss thread can be found here
>>> https://lists.apache.org/thread.html/37cc29648f0592cc39d3c78a0d07fce38521bdbbc4cf40e022a7a8ea@%3Cdev.metron.apache.org%3E
>>>
>>> The major issue and impact with upgrading the Hadoop platform is that
>>> there are breaking changes. Code that runs on HDP 3.1 will not run on 2.x.
>>> Here is a sampling of core components we depend on that we know of so
>>> far that are not backwards compatible:
>>>
>>>    1. The Core OS - we currently base our builds and test deployment
>>>    off of artifacts pulled from HDP. The latest rev of HDP no longer ships
>>>    RPMs for Centos 6, which means we need to upgrade to Centos 7
>>>    2. Ambari
>>>    3. HBase
>>>
>>> This differs from individual components we've upgraded in the past in
>>> that our code could still be deployed on the old as well as new version of
>>> the component in a backwards compatible way. Based on semantic versioning,
>>> I don't know if we can introduce this level of change in a point release,
>>> which is the reason for kicking off this discussion. In the past, users and
>>> developers in the community have suggested that they are -1 on a 1.x
>>> release that does not provide an upgrade
>>> https://lists.apache.org/thread.html/eb1a8df2d0a6a79c5d50540d1fdbf215ec83d831ff15d3117c2592cc@%3Cdev.metron.apache.org%3E
>>> .
>>>
>>> Is there a way we can avoid a 1.x release? If we do need 1.x, do we
>>> still see upgrades as a gating function? The main issue is that this has
>>> the potential to drag out the upgrade and further couple it with other
>>> features. And with Storm 1.x being eol'ed, I'm not sure this is something
>>> we can wait much longer for. I'll think on this and send out my own
>>> thoughts once folks have had a chance to review.
>>>
>>> Best,
>>> Mike Miklavcic
>>> Apache Metron, PMC, committer
>>>
>>>
>>> --
>>> --
>>> simon elliston ball
>>> @sireb
>>>
>>>
>>>
>>> -------------------
>>> Thank you,
>>>
>>> James Sirota
>>> PMC- Apache Metron
>>> jsirota AT apache DOT org
>>>
>>> --
--
simon elliston ball
@sireb

Re: [DISCUSS] HDP 3.1 Upgrade and release strategy

[Otto Fowler <ot...@gmail.com>]

You are right, that is much better than backup_metron_configs.sh.




On August 27, 2019 at 16:05:38, Simon Elliston Ball (
simon@simonellistonball.com) wrote:

You can do this with zk_load_configs and Ambari’s blueprint api, so we
kinda already do.

Simon

On Tue, 27 Aug 2019 at 20:19, Otto Fowler <ot...@gmail.com> wrote:

> Maybe we need some automated method to backup configurations and restore
> them.
>
>
>
> On August 27, 2019 at 14:46:58, Michael Miklavcic (
> michael.miklavcic@gmail.com) wrote:
>
> > Once you back up your metron configs, the same configs that worked on
> the previous version will continue to work on the version running on HDP
> 3.x.  If there is any discrepancy between the two or additional settings
> will be required, those will be documented in the release notes.  From the
> Metron perspective, this upgrade would be no different than simply
> upgrading to the new Metron version.
>
> This upgrade cannot be performed the same way we've done it in the past. A
> number of platform upgrades, including OS, are required:
>
>    1. Requires the OS to be updated on all nodes because there are no
>    Centos6 RPMs provided in HDP 3.1. Must bump to Centos7.
>    2. The final new HBase code will not run on HDP 2.6
>    3. The MPack changes for new Ambari are not backwards compatible
>    4. YARN and HDFS/MR are also at risk to be backwards incompatible
>
>
> On Tue, Aug 27, 2019 at 12:39 PM Michael Miklavcic <
> michael.miklavcic@gmail.com> wrote:
>
>> Adding the dev list back into the thread (a reply-all was missed).
>>
>> On Tue, Aug 27, 2019 at 10:49 AM James Sirota <js...@apache.org> wrote:
>>
>>> I agree with Simon.  HDP 2.x platform is rapidly approaching EOL and
>>> everyone will likely need to migrate by end of year.  Doing this platform
>>> upgrade sooner will give everyone visibility into what Metron on HDP 3.x
>>> looks like so they have time to plan and upgrade at their own pace.
>>> Feature-wise, the Metron application itself will be unchanged.  It is
>>> merely the platform underneath that is changing.  HDP itself can be
>>> upgraded per instructions here:
>>> https://docs.hortonworks.com/HDPDocuments/HDP3/HDP-3.1.0/release-notes/content/upgrading_parent.html
>>>
>>> Once you back up your metron configs, the same configs that worked on
>>> the previous version will continue to work on the version running on HDP
>>> 3.x.  If there is any discrepancy between the two or additional settings
>>> will be required, those will be documented in the release notes.  From the
>>> Metron perspective, this upgrade would be no different than simply
>>> upgrading to the new Metron version.
>>>
>>> James
>>>
>>>
>>> 27.08.2019, 07:09, "Simon Elliston Ball" <si...@simonellistonball.com>:
>>>
>>> Something worth noting here is that HDP 2.6.5 is quite old and
>>> approaching EoL rapidly, so the issue of upgrade is urgent. I am aware of a
>>> large number of users who require this upgrade ASAP, and in fact an aware
>>> of zero users who wish to remain on HDP 2.
>>>
>>> Perhaps those users who want to stay on the old platform can stick their
>>> hands up and raise concerns, but this move will likely have to happen very
>>> soon.
>>>
>>> Simon
>>>
>>> On Tue, 27 Aug 2019 at 15:04, Otto Fowler <ot...@gmail.com>
>>> wrote:
>>>
>>> Although we had the discussion, and some great ideas where passed
>>> around, I do not believe we came to some kind of consensus on what 1.0
>>> should look like. So that discussion would have to be picked up again so
>>> that we could know where we are at, and make it an actual thing if we were
>>> going to make this a 1.0 release.
>>>
>>> I believe that the issues raised in that discussion gating 1.0 are still
>>> largely applicable, including upgrade.
>>>
>>> Right now we have *ZERO* HDP 3.1 users. We will go from that to *only*
>>> supporting 3.1 work and releases. So every user and deployment we currently
>>> have will feel real pain, have to slay real dragons to move forward with
>>> metron.
>>>
>>> With regards to support for older versions, the “backward capability”
>>> that has been mentioned, I would not say that I have any specific plan for
>>> that in mind. What I would say rather, is that I believe that we must be
>>> explicit, setting expectations correctly and clearly with regards to our
>>> intent while demonstrating that we have thought through the situation. That
>>> discussion has not happened, at least I do not believe that the prior dev
>>> thread really handles it in context.
>>>
>>> Depending on the upgrade situation for going to 3.1, it may be that a
>>> dual stream of releases or fixes or new features to the extent that we can
>>> do it will greatly reduce the pain for folks, or make it viable to stick
>>> with metron until they can upgrade.
>>>
>>> The issue of what metron *is* features wise may be another one we want
>>> to take up at some point. The idea being can we separate the metron
>>> _integration parts from the metron core functionality such that we can work
>>> on them separately and thus support multiple platforms through
>>> integrations/applications. Of course definition of metron’s value beyond
>>> integration, and what those features and application boundaries are would
>>> be necessary.
>>>
>>>
>>>
>>> On August 26, 2019 at 18:52:57, Michael Miklavcic (
>>> michael.miklavcic@gmail.com) wrote:
>>>
>>> Hi devs and users,
>>>
>>> Some questions were asked in the Slack channel about our ongoing
>>> HDP/Hadoop upgrade and I'd like to get a discussion rolling. The original
>>> Hadoop upgrade discuss thread can be found here
>>> https://lists.apache.org/thread.html/37cc29648f0592cc39d3c78a0d07fce38521bdbbc4cf40e022a7a8ea@%3Cdev.metron.apache.org%3E
>>>
>>> The major issue and impact with upgrading the Hadoop platform is that
>>> there are breaking changes. Code that runs on HDP 3.1 will not run on 2.x.
>>> Here is a sampling of core components we depend on that we know of so
>>> far that are not backwards compatible:
>>>
>>>    1. The Core OS - we currently base our builds and test deployment
>>>    off of artifacts pulled from HDP. The latest rev of HDP no longer ships
>>>    RPMs for Centos 6, which means we need to upgrade to Centos 7
>>>    2. Ambari
>>>    3. HBase
>>>
>>> This differs from individual components we've upgraded in the past in
>>> that our code could still be deployed on the old as well as new version of
>>> the component in a backwards compatible way. Based on semantic versioning,
>>> I don't know if we can introduce this level of change in a point release,
>>> which is the reason for kicking off this discussion. In the past, users and
>>> developers in the community have suggested that they are -1 on a 1.x
>>> release that does not provide an upgrade
>>> https://lists.apache.org/thread.html/eb1a8df2d0a6a79c5d50540d1fdbf215ec83d831ff15d3117c2592cc@%3Cdev.metron.apache.org%3E
>>> .
>>>
>>> Is there a way we can avoid a 1.x release? If we do need 1.x, do we
>>> still see upgrades as a gating function? The main issue is that this has
>>> the potential to drag out the upgrade and further couple it with other
>>> features. And with Storm 1.x being eol'ed, I'm not sure this is something
>>> we can wait much longer for. I'll think on this and send out my own
>>> thoughts once folks have had a chance to review.
>>>
>>> Best,
>>> Mike Miklavcic
>>> Apache Metron, PMC, committer
>>>
>>>
>>> --
>>> --
>>> simon elliston ball
>>> @sireb
>>>
>>>
>>>
>>> -------------------
>>> Thank you,
>>>
>>> James Sirota
>>> PMC- Apache Metron
>>> jsirota AT apache DOT org
>>>
>>> --
--
simon elliston ball
@sireb

Re: [DISCUSS] HDP 3.1 Upgrade and release strategy

[Otto Fowler <ot...@gmail.com>]

You are right, that is much better than backup_metron_configs.sh.




On August 27, 2019 at 16:05:38, Simon Elliston Ball (
simon@simonellistonball.com) wrote:

You can do this with zk_load_configs and Ambari’s blueprint api, so we
kinda already do.

Simon

On Tue, 27 Aug 2019 at 20:19, Otto Fowler <ot...@gmail.com> wrote:

> Maybe we need some automated method to backup configurations and restore
> them.
>
>
>
> On August 27, 2019 at 14:46:58, Michael Miklavcic (
> michael.miklavcic@gmail.com) wrote:
>
> > Once you back up your metron configs, the same configs that worked on
> the previous version will continue to work on the version running on HDP
> 3.x.  If there is any discrepancy between the two or additional settings
> will be required, those will be documented in the release notes.  From the
> Metron perspective, this upgrade would be no different than simply
> upgrading to the new Metron version.
>
> This upgrade cannot be performed the same way we've done it in the past. A
> number of platform upgrades, including OS, are required:
>
>    1. Requires the OS to be updated on all nodes because there are no
>    Centos6 RPMs provided in HDP 3.1. Must bump to Centos7.
>    2. The final new HBase code will not run on HDP 2.6
>    3. The MPack changes for new Ambari are not backwards compatible
>    4. YARN and HDFS/MR are also at risk to be backwards incompatible
>
>
> On Tue, Aug 27, 2019 at 12:39 PM Michael Miklavcic <
> michael.miklavcic@gmail.com> wrote:
>
>> Adding the dev list back into the thread (a reply-all was missed).
>>
>> On Tue, Aug 27, 2019 at 10:49 AM James Sirota <js...@apache.org> wrote:
>>
>>> I agree with Simon.  HDP 2.x platform is rapidly approaching EOL and
>>> everyone will likely need to migrate by end of year.  Doing this platform
>>> upgrade sooner will give everyone visibility into what Metron on HDP 3.x
>>> looks like so they have time to plan and upgrade at their own pace.
>>> Feature-wise, the Metron application itself will be unchanged.  It is
>>> merely the platform underneath that is changing.  HDP itself can be
>>> upgraded per instructions here:
>>> https://docs.hortonworks.com/HDPDocuments/HDP3/HDP-3.1.0/release-notes/content/upgrading_parent.html
>>>
>>> Once you back up your metron configs, the same configs that worked on
>>> the previous version will continue to work on the version running on HDP
>>> 3.x.  If there is any discrepancy between the two or additional settings
>>> will be required, those will be documented in the release notes.  From the
>>> Metron perspective, this upgrade would be no different than simply
>>> upgrading to the new Metron version.
>>>
>>> James
>>>
>>>
>>> 27.08.2019, 07:09, "Simon Elliston Ball" <si...@simonellistonball.com>:
>>>
>>> Something worth noting here is that HDP 2.6.5 is quite old and
>>> approaching EoL rapidly, so the issue of upgrade is urgent. I am aware of a
>>> large number of users who require this upgrade ASAP, and in fact an aware
>>> of zero users who wish to remain on HDP 2.
>>>
>>> Perhaps those users who want to stay on the old platform can stick their
>>> hands up and raise concerns, but this move will likely have to happen very
>>> soon.
>>>
>>> Simon
>>>
>>> On Tue, 27 Aug 2019 at 15:04, Otto Fowler <ot...@gmail.com>
>>> wrote:
>>>
>>> Although we had the discussion, and some great ideas where passed
>>> around, I do not believe we came to some kind of consensus on what 1.0
>>> should look like. So that discussion would have to be picked up again so
>>> that we could know where we are at, and make it an actual thing if we were
>>> going to make this a 1.0 release.
>>>
>>> I believe that the issues raised in that discussion gating 1.0 are still
>>> largely applicable, including upgrade.
>>>
>>> Right now we have *ZERO* HDP 3.1 users. We will go from that to *only*
>>> supporting 3.1 work and releases. So every user and deployment we currently
>>> have will feel real pain, have to slay real dragons to move forward with
>>> metron.
>>>
>>> With regards to support for older versions, the “backward capability”
>>> that has been mentioned, I would not say that I have any specific plan for
>>> that in mind. What I would say rather, is that I believe that we must be
>>> explicit, setting expectations correctly and clearly with regards to our
>>> intent while demonstrating that we have thought through the situation. That
>>> discussion has not happened, at least I do not believe that the prior dev
>>> thread really handles it in context.
>>>
>>> Depending on the upgrade situation for going to 3.1, it may be that a
>>> dual stream of releases or fixes or new features to the extent that we can
>>> do it will greatly reduce the pain for folks, or make it viable to stick
>>> with metron until they can upgrade.
>>>
>>> The issue of what metron *is* features wise may be another one we want
>>> to take up at some point. The idea being can we separate the metron
>>> _integration parts from the metron core functionality such that we can work
>>> on them separately and thus support multiple platforms through
>>> integrations/applications. Of course definition of metron’s value beyond
>>> integration, and what those features and application boundaries are would
>>> be necessary.
>>>
>>>
>>>
>>> On August 26, 2019 at 18:52:57, Michael Miklavcic (
>>> michael.miklavcic@gmail.com) wrote:
>>>
>>> Hi devs and users,
>>>
>>> Some questions were asked in the Slack channel about our ongoing
>>> HDP/Hadoop upgrade and I'd like to get a discussion rolling. The original
>>> Hadoop upgrade discuss thread can be found here
>>> https://lists.apache.org/thread.html/37cc29648f0592cc39d3c78a0d07fce38521bdbbc4cf40e022a7a8ea@%3Cdev.metron.apache.org%3E
>>>
>>> The major issue and impact with upgrading the Hadoop platform is that
>>> there are breaking changes. Code that runs on HDP 3.1 will not run on 2.x.
>>> Here is a sampling of core components we depend on that we know of so
>>> far that are not backwards compatible:
>>>
>>>    1. The Core OS - we currently base our builds and test deployment
>>>    off of artifacts pulled from HDP. The latest rev of HDP no longer ships
>>>    RPMs for Centos 6, which means we need to upgrade to Centos 7
>>>    2. Ambari
>>>    3. HBase
>>>
>>> This differs from individual components we've upgraded in the past in
>>> that our code could still be deployed on the old as well as new version of
>>> the component in a backwards compatible way. Based on semantic versioning,
>>> I don't know if we can introduce this level of change in a point release,
>>> which is the reason for kicking off this discussion. In the past, users and
>>> developers in the community have suggested that they are -1 on a 1.x
>>> release that does not provide an upgrade
>>> https://lists.apache.org/thread.html/eb1a8df2d0a6a79c5d50540d1fdbf215ec83d831ff15d3117c2592cc@%3Cdev.metron.apache.org%3E
>>> .
>>>
>>> Is there a way we can avoid a 1.x release? If we do need 1.x, do we
>>> still see upgrades as a gating function? The main issue is that this has
>>> the potential to drag out the upgrade and further couple it with other
>>> features. And with Storm 1.x being eol'ed, I'm not sure this is something
>>> we can wait much longer for. I'll think on this and send out my own
>>> thoughts once folks have had a chance to review.
>>>
>>> Best,
>>> Mike Miklavcic
>>> Apache Metron, PMC, committer
>>>
>>>
>>> --
>>> --
>>> simon elliston ball
>>> @sireb
>>>
>>>
>>>
>>> -------------------
>>> Thank you,
>>>
>>> James Sirota
>>> PMC- Apache Metron
>>> jsirota AT apache DOT org
>>>
>>> --
--
simon elliston ball
@sireb

Re: [DISCUSS] HDP 3.1 Upgrade and release strategy

[Simon Elliston Ball <si...@simonellistonball.com>]

You can do this with zk_load_configs and Ambari’s blueprint api, so we
kinda already do.

Simon

On Tue, 27 Aug 2019 at 20:19, Otto Fowler <ot...@gmail.com> wrote:

> Maybe we need some automated method to backup configurations and restore
> them.
>
>
>
>
> On August 27, 2019 at 14:46:58, Michael Miklavcic (
> michael.miklavcic@gmail.com) wrote:
>
> > Once you back up your metron configs, the same configs that worked on
> the previous version will continue to work on the version running on HDP
> 3.x.  If there is any discrepancy between the two or additional settings
> will be required, those will be documented in the release notes.  From the
> Metron perspective, this upgrade would be no different than simply
> upgrading to the new Metron version.
>
> This upgrade cannot be performed the same way we've done it in the past. A
> number of platform upgrades, including OS, are required:
>
>    1. Requires the OS to be updated on all nodes because there are no
>    Centos6 RPMs provided in HDP 3.1. Must bump to Centos7.
>    2. The final new HBase code will not run on HDP 2.6
>    3. The MPack changes for new Ambari are not backwards compatible
>    4. YARN and HDFS/MR are also at risk to be backwards incompatible
>
>
> On Tue, Aug 27, 2019 at 12:39 PM Michael Miklavcic <
> michael.miklavcic@gmail.com> wrote:
>
>> Adding the dev list back into the thread (a reply-all was missed).
>>
>> On Tue, Aug 27, 2019 at 10:49 AM James Sirota <js...@apache.org> wrote:
>>
>>> I agree with Simon.  HDP 2.x platform is rapidly approaching EOL and
>>> everyone will likely need to migrate by end of year.  Doing this platform
>>> upgrade sooner will give everyone visibility into what Metron on HDP 3.x
>>> looks like so they have time to plan and upgrade at their own pace.
>>> Feature-wise, the Metron application itself will be unchanged.  It is
>>> merely the platform underneath that is changing.  HDP itself can be
>>> upgraded per instructions here:
>>> https://docs.hortonworks.com/HDPDocuments/HDP3/HDP-3.1.0/release-notes/content/upgrading_parent.html
>>>
>>> Once you back up your metron configs, the same configs that worked on
>>> the previous version will continue to work on the version running on HDP
>>> 3.x.  If there is any discrepancy between the two or additional settings
>>> will be required, those will be documented in the release notes.  From the
>>> Metron perspective, this upgrade would be no different than simply
>>> upgrading to the new Metron version.
>>>
>>> James
>>>
>>>
>>> 27.08.2019, 07:09, "Simon Elliston Ball" <si...@simonellistonball.com>:
>>>
>>> Something worth noting here is that HDP 2.6.5 is quite old and
>>> approaching EoL rapidly, so the issue of upgrade is urgent. I am aware of a
>>> large number of users who require this upgrade ASAP, and in fact an aware
>>> of zero users who wish to remain on HDP 2.
>>>
>>> Perhaps those users who want to stay on the old platform can stick their
>>> hands up and raise concerns, but this move will likely have to happen very
>>> soon.
>>>
>>> Simon
>>>
>>> On Tue, 27 Aug 2019 at 15:04, Otto Fowler <ot...@gmail.com>
>>> wrote:
>>>
>>> Although we had the discussion, and some great ideas where passed
>>> around, I do not believe we came to some kind of consensus on what 1.0
>>> should look like. So that discussion would have to be picked up again so
>>> that we could know where we are at, and make it an actual thing if we were
>>> going to make this a 1.0 release.
>>>
>>> I believe that the issues raised in that discussion gating 1.0 are still
>>> largely applicable, including upgrade.
>>>
>>> Right now we have *ZERO* HDP 3.1 users. We will go from that to *only*
>>> supporting 3.1 work and releases. So every user and deployment we currently
>>> have will feel real pain, have to slay real dragons to move forward with
>>> metron.
>>>
>>> With regards to support for older versions, the “backward capability”
>>> that has been mentioned, I would not say that I have any specific plan for
>>> that in mind. What I would say rather, is that I believe that we must be
>>> explicit, setting expectations correctly and clearly with regards to our
>>> intent while demonstrating that we have thought through the situation. That
>>> discussion has not happened, at least I do not believe that the prior dev
>>> thread really handles it in context.
>>>
>>> Depending on the upgrade situation for going to 3.1, it may be that a
>>> dual stream of releases or fixes or new features to the extent that we can
>>> do it will greatly reduce the pain for folks, or make it viable to stick
>>> with metron until they can upgrade.
>>>
>>> The issue of what metron *is* features wise may be another one we want
>>> to take up at some point. The idea being can we separate the metron
>>> _integration parts from the metron core functionality such that we can work
>>> on them separately and thus support multiple platforms through
>>> integrations/applications. Of course definition of metron’s value beyond
>>> integration, and what those features and application boundaries are would
>>> be necessary.
>>>
>>>
>>>
>>> On August 26, 2019 at 18:52:57, Michael Miklavcic (
>>> michael.miklavcic@gmail.com) wrote:
>>>
>>> Hi devs and users,
>>>
>>> Some questions were asked in the Slack channel about our ongoing
>>> HDP/Hadoop upgrade and I'd like to get a discussion rolling. The original
>>> Hadoop upgrade discuss thread can be found here
>>> https://lists.apache.org/thread.html/37cc29648f0592cc39d3c78a0d07fce38521bdbbc4cf40e022a7a8ea@%3Cdev.metron.apache.org%3E
>>>
>>> The major issue and impact with upgrading the Hadoop platform is that
>>> there are breaking changes. Code that runs on HDP 3.1 will not run on 2.x.
>>> Here is a sampling of core components we depend on that we know of so
>>> far that are not backwards compatible:
>>>
>>>    1. The Core OS - we currently base our builds and test deployment
>>>    off of artifacts pulled from HDP. The latest rev of HDP no longer ships
>>>    RPMs for Centos 6, which means we need to upgrade to Centos 7
>>>    2. Ambari
>>>    3. HBase
>>>
>>> This differs from individual components we've upgraded in the past in
>>> that our code could still be deployed on the old as well as new version of
>>> the component in a backwards compatible way. Based on semantic versioning,
>>> I don't know if we can introduce this level of change in a point release,
>>> which is the reason for kicking off this discussion. In the past, users and
>>> developers in the community have suggested that they are -1 on a 1.x
>>> release that does not provide an upgrade
>>> https://lists.apache.org/thread.html/eb1a8df2d0a6a79c5d50540d1fdbf215ec83d831ff15d3117c2592cc@%3Cdev.metron.apache.org%3E
>>> .
>>>
>>> Is there a way we can avoid a 1.x release? If we do need 1.x, do we
>>> still see upgrades as a gating function? The main issue is that this has
>>> the potential to drag out the upgrade and further couple it with other
>>> features. And with Storm 1.x being eol'ed, I'm not sure this is something
>>> we can wait much longer for. I'll think on this and send out my own
>>> thoughts once folks have had a chance to review.
>>>
>>> Best,
>>> Mike Miklavcic
>>> Apache Metron, PMC, committer
>>>
>>>
>>> --
>>> --
>>> simon elliston ball
>>> @sireb
>>>
>>>
>>>
>>> -------------------
>>> Thank you,
>>>
>>> James Sirota
>>> PMC- Apache Metron
>>> jsirota AT apache DOT org
>>>
>>> --
--
simon elliston ball
@sireb

Re: [DISCUSS] HDP 3.1 Upgrade and release strategy

[Simon Elliston Ball <si...@simonellistonball.com>]

You can do this with zk_load_configs and Ambari’s blueprint api, so we
kinda already do.

Simon

On Tue, 27 Aug 2019 at 20:19, Otto Fowler <ot...@gmail.com> wrote:

> Maybe we need some automated method to backup configurations and restore
> them.
>
>
>
>
> On August 27, 2019 at 14:46:58, Michael Miklavcic (
> michael.miklavcic@gmail.com) wrote:
>
> > Once you back up your metron configs, the same configs that worked on
> the previous version will continue to work on the version running on HDP
> 3.x.  If there is any discrepancy between the two or additional settings
> will be required, those will be documented in the release notes.  From the
> Metron perspective, this upgrade would be no different than simply
> upgrading to the new Metron version.
>
> This upgrade cannot be performed the same way we've done it in the past. A
> number of platform upgrades, including OS, are required:
>
>    1. Requires the OS to be updated on all nodes because there are no
>    Centos6 RPMs provided in HDP 3.1. Must bump to Centos7.
>    2. The final new HBase code will not run on HDP 2.6
>    3. The MPack changes for new Ambari are not backwards compatible
>    4. YARN and HDFS/MR are also at risk to be backwards incompatible
>
>
> On Tue, Aug 27, 2019 at 12:39 PM Michael Miklavcic <
> michael.miklavcic@gmail.com> wrote:
>
>> Adding the dev list back into the thread (a reply-all was missed).
>>
>> On Tue, Aug 27, 2019 at 10:49 AM James Sirota <js...@apache.org> wrote:
>>
>>> I agree with Simon.  HDP 2.x platform is rapidly approaching EOL and
>>> everyone will likely need to migrate by end of year.  Doing this platform
>>> upgrade sooner will give everyone visibility into what Metron on HDP 3.x
>>> looks like so they have time to plan and upgrade at their own pace.
>>> Feature-wise, the Metron application itself will be unchanged.  It is
>>> merely the platform underneath that is changing.  HDP itself can be
>>> upgraded per instructions here:
>>> https://docs.hortonworks.com/HDPDocuments/HDP3/HDP-3.1.0/release-notes/content/upgrading_parent.html
>>>
>>> Once you back up your metron configs, the same configs that worked on
>>> the previous version will continue to work on the version running on HDP
>>> 3.x.  If there is any discrepancy between the two or additional settings
>>> will be required, those will be documented in the release notes.  From the
>>> Metron perspective, this upgrade would be no different than simply
>>> upgrading to the new Metron version.
>>>
>>> James
>>>
>>>
>>> 27.08.2019, 07:09, "Simon Elliston Ball" <si...@simonellistonball.com>:
>>>
>>> Something worth noting here is that HDP 2.6.5 is quite old and
>>> approaching EoL rapidly, so the issue of upgrade is urgent. I am aware of a
>>> large number of users who require this upgrade ASAP, and in fact an aware
>>> of zero users who wish to remain on HDP 2.
>>>
>>> Perhaps those users who want to stay on the old platform can stick their
>>> hands up and raise concerns, but this move will likely have to happen very
>>> soon.
>>>
>>> Simon
>>>
>>> On Tue, 27 Aug 2019 at 15:04, Otto Fowler <ot...@gmail.com>
>>> wrote:
>>>
>>> Although we had the discussion, and some great ideas where passed
>>> around, I do not believe we came to some kind of consensus on what 1.0
>>> should look like. So that discussion would have to be picked up again so
>>> that we could know where we are at, and make it an actual thing if we were
>>> going to make this a 1.0 release.
>>>
>>> I believe that the issues raised in that discussion gating 1.0 are still
>>> largely applicable, including upgrade.
>>>
>>> Right now we have *ZERO* HDP 3.1 users. We will go from that to *only*
>>> supporting 3.1 work and releases. So every user and deployment we currently
>>> have will feel real pain, have to slay real dragons to move forward with
>>> metron.
>>>
>>> With regards to support for older versions, the “backward capability”
>>> that has been mentioned, I would not say that I have any specific plan for
>>> that in mind. What I would say rather, is that I believe that we must be
>>> explicit, setting expectations correctly and clearly with regards to our
>>> intent while demonstrating that we have thought through the situation. That
>>> discussion has not happened, at least I do not believe that the prior dev
>>> thread really handles it in context.
>>>
>>> Depending on the upgrade situation for going to 3.1, it may be that a
>>> dual stream of releases or fixes or new features to the extent that we can
>>> do it will greatly reduce the pain for folks, or make it viable to stick
>>> with metron until they can upgrade.
>>>
>>> The issue of what metron *is* features wise may be another one we want
>>> to take up at some point. The idea being can we separate the metron
>>> _integration parts from the metron core functionality such that we can work
>>> on them separately and thus support multiple platforms through
>>> integrations/applications. Of course definition of metron’s value beyond
>>> integration, and what those features and application boundaries are would
>>> be necessary.
>>>
>>>
>>>
>>> On August 26, 2019 at 18:52:57, Michael Miklavcic (
>>> michael.miklavcic@gmail.com) wrote:
>>>
>>> Hi devs and users,
>>>
>>> Some questions were asked in the Slack channel about our ongoing
>>> HDP/Hadoop upgrade and I'd like to get a discussion rolling. The original
>>> Hadoop upgrade discuss thread can be found here
>>> https://lists.apache.org/thread.html/37cc29648f0592cc39d3c78a0d07fce38521bdbbc4cf40e022a7a8ea@%3Cdev.metron.apache.org%3E
>>>
>>> The major issue and impact with upgrading the Hadoop platform is that
>>> there are breaking changes. Code that runs on HDP 3.1 will not run on 2.x.
>>> Here is a sampling of core components we depend on that we know of so
>>> far that are not backwards compatible:
>>>
>>>    1. The Core OS - we currently base our builds and test deployment
>>>    off of artifacts pulled from HDP. The latest rev of HDP no longer ships
>>>    RPMs for Centos 6, which means we need to upgrade to Centos 7
>>>    2. Ambari
>>>    3. HBase
>>>
>>> This differs from individual components we've upgraded in the past in
>>> that our code could still be deployed on the old as well as new version of
>>> the component in a backwards compatible way. Based on semantic versioning,
>>> I don't know if we can introduce this level of change in a point release,
>>> which is the reason for kicking off this discussion. In the past, users and
>>> developers in the community have suggested that they are -1 on a 1.x
>>> release that does not provide an upgrade
>>> https://lists.apache.org/thread.html/eb1a8df2d0a6a79c5d50540d1fdbf215ec83d831ff15d3117c2592cc@%3Cdev.metron.apache.org%3E
>>> .
>>>
>>> Is there a way we can avoid a 1.x release? If we do need 1.x, do we
>>> still see upgrades as a gating function? The main issue is that this has
>>> the potential to drag out the upgrade and further couple it with other
>>> features. And with Storm 1.x being eol'ed, I'm not sure this is something
>>> we can wait much longer for. I'll think on this and send out my own
>>> thoughts once folks have had a chance to review.
>>>
>>> Best,
>>> Mike Miklavcic
>>> Apache Metron, PMC, committer
>>>
>>>
>>> --
>>> --
>>> simon elliston ball
>>> @sireb
>>>
>>>
>>>
>>> -------------------
>>> Thank you,
>>>
>>> James Sirota
>>> PMC- Apache Metron
>>> jsirota AT apache DOT org
>>>
>>> --
--
simon elliston ball
@sireb

Re: [DISCUSS] HDP 3.1 Upgrade and release strategy

[Otto Fowler <ot...@gmail.com>]

Maybe we need some automated method to backup configurations and restore
them.




On August 27, 2019 at 14:46:58, Michael Miklavcic (
michael.miklavcic@gmail.com) wrote:

> Once you back up your metron configs, the same configs that worked on the
previous version will continue to work on the version running on HDP 3.x.
If there is any discrepancy between the two or additional settings will be
required, those will be documented in the release notes.  From the Metron
perspective, this upgrade would be no different than simply upgrading to
the new Metron version.

This upgrade cannot be performed the same way we've done it in the past. A
number of platform upgrades, including OS, are required:

   1. Requires the OS to be updated on all nodes because there are no
   Centos6 RPMs provided in HDP 3.1. Must bump to Centos7.
   2. The final new HBase code will not run on HDP 2.6
   3. The MPack changes for new Ambari are not backwards compatible
   4. YARN and HDFS/MR are also at risk to be backwards incompatible


On Tue, Aug 27, 2019 at 12:39 PM Michael Miklavcic <
michael.miklavcic@gmail.com> wrote:

> Adding the dev list back into the thread (a reply-all was missed).
>
> On Tue, Aug 27, 2019 at 10:49 AM James Sirota <js...@apache.org> wrote:
>
>> I agree with Simon.  HDP 2.x platform is rapidly approaching EOL and
>> everyone will likely need to migrate by end of year.  Doing this platform
>> upgrade sooner will give everyone visibility into what Metron on HDP 3.x
>> looks like so they have time to plan and upgrade at their own pace.
>> Feature-wise, the Metron application itself will be unchanged.  It is
>> merely the platform underneath that is changing.  HDP itself can be
>> upgraded per instructions here:
>> https://docs.hortonworks.com/HDPDocuments/HDP3/HDP-3.1.0/release-notes/content/upgrading_parent.html
>>
>> Once you back up your metron configs, the same configs that worked on the
>> previous version will continue to work on the version running on HDP 3.x.
>> If there is any discrepancy between the two or additional settings will be
>> required, those will be documented in the release notes.  From the Metron
>> perspective, this upgrade would be no different than simply upgrading to
>> the new Metron version.
>>
>> James
>>
>>
>> 27.08.2019, 07:09, "Simon Elliston Ball" <si...@simonellistonball.com>:
>>
>> Something worth noting here is that HDP 2.6.5 is quite old and
>> approaching EoL rapidly, so the issue of upgrade is urgent. I am aware of a
>> large number of users who require this upgrade ASAP, and in fact an aware
>> of zero users who wish to remain on HDP 2.
>>
>> Perhaps those users who want to stay on the old platform can stick their
>> hands up and raise concerns, but this move will likely have to happen very
>> soon.
>>
>> Simon
>>
>> On Tue, 27 Aug 2019 at 15:04, Otto Fowler <ot...@gmail.com>
>> wrote:
>>
>> Although we had the discussion, and some great ideas where passed around,
>> I do not believe we came to some kind of consensus on what 1.0 should look
>> like. So that discussion would have to be picked up again so that we could
>> know where we are at, and make it an actual thing if we were going to make
>> this a 1.0 release.
>>
>> I believe that the issues raised in that discussion gating 1.0 are still
>> largely applicable, including upgrade.
>>
>> Right now we have *ZERO* HDP 3.1 users. We will go from that to *only*
>> supporting 3.1 work and releases. So every user and deployment we currently
>> have will feel real pain, have to slay real dragons to move forward with
>> metron.
>>
>> With regards to support for older versions, the “backward capability”
>> that has been mentioned, I would not say that I have any specific plan for
>> that in mind. What I would say rather, is that I believe that we must be
>> explicit, setting expectations correctly and clearly with regards to our
>> intent while demonstrating that we have thought through the situation. That
>> discussion has not happened, at least I do not believe that the prior dev
>> thread really handles it in context.
>>
>> Depending on the upgrade situation for going to 3.1, it may be that a
>> dual stream of releases or fixes or new features to the extent that we can
>> do it will greatly reduce the pain for folks, or make it viable to stick
>> with metron until they can upgrade.
>>
>> The issue of what metron *is* features wise may be another one we want
>> to take up at some point. The idea being can we separate the metron
>> _integration parts from the metron core functionality such that we can work
>> on them separately and thus support multiple platforms through
>> integrations/applications. Of course definition of metron’s value beyond
>> integration, and what those features and application boundaries are would
>> be necessary.
>>
>>
>>
>> On August 26, 2019 at 18:52:57, Michael Miklavcic (
>> michael.miklavcic@gmail.com) wrote:
>>
>> Hi devs and users,
>>
>> Some questions were asked in the Slack channel about our ongoing
>> HDP/Hadoop upgrade and I'd like to get a discussion rolling. The original
>> Hadoop upgrade discuss thread can be found here
>> https://lists.apache.org/thread.html/37cc29648f0592cc39d3c78a0d07fce38521bdbbc4cf40e022a7a8ea@%3Cdev.metron.apache.org%3E
>>
>> The major issue and impact with upgrading the Hadoop platform is that
>> there are breaking changes. Code that runs on HDP 3.1 will not run on 2.x.
>> Here is a sampling of core components we depend on that we know of so
>> far that are not backwards compatible:
>>
>>    1. The Core OS - we currently base our builds and test deployment off
>>    of artifacts pulled from HDP. The latest rev of HDP no longer ships RPMs
>>    for Centos 6, which means we need to upgrade to Centos 7
>>    2. Ambari
>>    3. HBase
>>
>> This differs from individual components we've upgraded in the past in
>> that our code could still be deployed on the old as well as new version of
>> the component in a backwards compatible way. Based on semantic versioning,
>> I don't know if we can introduce this level of change in a point release,
>> which is the reason for kicking off this discussion. In the past, users and
>> developers in the community have suggested that they are -1 on a 1.x
>> release that does not provide an upgrade
>> https://lists.apache.org/thread.html/eb1a8df2d0a6a79c5d50540d1fdbf215ec83d831ff15d3117c2592cc@%3Cdev.metron.apache.org%3E
>> .
>>
>> Is there a way we can avoid a 1.x release? If we do need 1.x, do we still
>> see upgrades as a gating function? The main issue is that this has the
>> potential to drag out the upgrade and further couple it with other
>> features. And with Storm 1.x being eol'ed, I'm not sure this is something
>> we can wait much longer for. I'll think on this and send out my own
>> thoughts once folks have had a chance to review.
>>
>> Best,
>> Mike Miklavcic
>> Apache Metron, PMC, committer
>>
>>
>> --
>> --
>> simon elliston ball
>> @sireb
>>
>>
>>
>> -------------------
>> Thank you,
>>
>> James Sirota
>> PMC- Apache Metron
>> jsirota AT apache DOT org
>>
>>

Re: [DISCUSS] HDP 3.1 Upgrade and release strategy

[Otto Fowler <ot...@gmail.com>]

Maybe we need some automated method to backup configurations and restore
them.




On August 27, 2019 at 14:46:58, Michael Miklavcic (
michael.miklavcic@gmail.com) wrote:

> Once you back up your metron configs, the same configs that worked on the
previous version will continue to work on the version running on HDP 3.x.
If there is any discrepancy between the two or additional settings will be
required, those will be documented in the release notes.  From the Metron
perspective, this upgrade would be no different than simply upgrading to
the new Metron version.

This upgrade cannot be performed the same way we've done it in the past. A
number of platform upgrades, including OS, are required:

   1. Requires the OS to be updated on all nodes because there are no
   Centos6 RPMs provided in HDP 3.1. Must bump to Centos7.
   2. The final new HBase code will not run on HDP 2.6
   3. The MPack changes for new Ambari are not backwards compatible
   4. YARN and HDFS/MR are also at risk to be backwards incompatible


On Tue, Aug 27, 2019 at 12:39 PM Michael Miklavcic <
michael.miklavcic@gmail.com> wrote:

> Adding the dev list back into the thread (a reply-all was missed).
>
> On Tue, Aug 27, 2019 at 10:49 AM James Sirota <js...@apache.org> wrote:
>
>> I agree with Simon.  HDP 2.x platform is rapidly approaching EOL and
>> everyone will likely need to migrate by end of year.  Doing this platform
>> upgrade sooner will give everyone visibility into what Metron on HDP 3.x
>> looks like so they have time to plan and upgrade at their own pace.
>> Feature-wise, the Metron application itself will be unchanged.  It is
>> merely the platform underneath that is changing.  HDP itself can be
>> upgraded per instructions here:
>> https://docs.hortonworks.com/HDPDocuments/HDP3/HDP-3.1.0/release-notes/content/upgrading_parent.html
>>
>> Once you back up your metron configs, the same configs that worked on the
>> previous version will continue to work on the version running on HDP 3.x.
>> If there is any discrepancy between the two or additional settings will be
>> required, those will be documented in the release notes.  From the Metron
>> perspective, this upgrade would be no different than simply upgrading to
>> the new Metron version.
>>
>> James
>>
>>
>> 27.08.2019, 07:09, "Simon Elliston Ball" <si...@simonellistonball.com>:
>>
>> Something worth noting here is that HDP 2.6.5 is quite old and
>> approaching EoL rapidly, so the issue of upgrade is urgent. I am aware of a
>> large number of users who require this upgrade ASAP, and in fact an aware
>> of zero users who wish to remain on HDP 2.
>>
>> Perhaps those users who want to stay on the old platform can stick their
>> hands up and raise concerns, but this move will likely have to happen very
>> soon.
>>
>> Simon
>>
>> On Tue, 27 Aug 2019 at 15:04, Otto Fowler <ot...@gmail.com>
>> wrote:
>>
>> Although we had the discussion, and some great ideas where passed around,
>> I do not believe we came to some kind of consensus on what 1.0 should look
>> like. So that discussion would have to be picked up again so that we could
>> know where we are at, and make it an actual thing if we were going to make
>> this a 1.0 release.
>>
>> I believe that the issues raised in that discussion gating 1.0 are still
>> largely applicable, including upgrade.
>>
>> Right now we have *ZERO* HDP 3.1 users. We will go from that to *only*
>> supporting 3.1 work and releases. So every user and deployment we currently
>> have will feel real pain, have to slay real dragons to move forward with
>> metron.
>>
>> With regards to support for older versions, the “backward capability”
>> that has been mentioned, I would not say that I have any specific plan for
>> that in mind. What I would say rather, is that I believe that we must be
>> explicit, setting expectations correctly and clearly with regards to our
>> intent while demonstrating that we have thought through the situation. That
>> discussion has not happened, at least I do not believe that the prior dev
>> thread really handles it in context.
>>
>> Depending on the upgrade situation for going to 3.1, it may be that a
>> dual stream of releases or fixes or new features to the extent that we can
>> do it will greatly reduce the pain for folks, or make it viable to stick
>> with metron until they can upgrade.
>>
>> The issue of what metron *is* features wise may be another one we want
>> to take up at some point. The idea being can we separate the metron
>> _integration parts from the metron core functionality such that we can work
>> on them separately and thus support multiple platforms through
>> integrations/applications. Of course definition of metron’s value beyond
>> integration, and what those features and application boundaries are would
>> be necessary.
>>
>>
>>
>> On August 26, 2019 at 18:52:57, Michael Miklavcic (
>> michael.miklavcic@gmail.com) wrote:
>>
>> Hi devs and users,
>>
>> Some questions were asked in the Slack channel about our ongoing
>> HDP/Hadoop upgrade and I'd like to get a discussion rolling. The original
>> Hadoop upgrade discuss thread can be found here
>> https://lists.apache.org/thread.html/37cc29648f0592cc39d3c78a0d07fce38521bdbbc4cf40e022a7a8ea@%3Cdev.metron.apache.org%3E
>>
>> The major issue and impact with upgrading the Hadoop platform is that
>> there are breaking changes. Code that runs on HDP 3.1 will not run on 2.x.
>> Here is a sampling of core components we depend on that we know of so
>> far that are not backwards compatible:
>>
>>    1. The Core OS - we currently base our builds and test deployment off
>>    of artifacts pulled from HDP. The latest rev of HDP no longer ships RPMs
>>    for Centos 6, which means we need to upgrade to Centos 7
>>    2. Ambari
>>    3. HBase
>>
>> This differs from individual components we've upgraded in the past in
>> that our code could still be deployed on the old as well as new version of
>> the component in a backwards compatible way. Based on semantic versioning,
>> I don't know if we can introduce this level of change in a point release,
>> which is the reason for kicking off this discussion. In the past, users and
>> developers in the community have suggested that they are -1 on a 1.x
>> release that does not provide an upgrade
>> https://lists.apache.org/thread.html/eb1a8df2d0a6a79c5d50540d1fdbf215ec83d831ff15d3117c2592cc@%3Cdev.metron.apache.org%3E
>> .
>>
>> Is there a way we can avoid a 1.x release? If we do need 1.x, do we still
>> see upgrades as a gating function? The main issue is that this has the
>> potential to drag out the upgrade and further couple it with other
>> features. And with Storm 1.x being eol'ed, I'm not sure this is something
>> we can wait much longer for. I'll think on this and send out my own
>> thoughts once folks have had a chance to review.
>>
>> Best,
>> Mike Miklavcic
>> Apache Metron, PMC, committer
>>
>>
>> --
>> --
>> simon elliston ball
>> @sireb
>>
>>
>>
>> -------------------
>> Thank you,
>>
>> James Sirota
>> PMC- Apache Metron
>> jsirota AT apache DOT org
>>
>>

Re: [DISCUSS] HDP 3.1 Upgrade and release strategy

[Michael Miklavcic <mi...@gmail.com>]

> Once you back up your metron configs, the same configs that worked on the
previous version will continue to work on the version running on HDP 3.x.
If there is any discrepancy between the two or additional settings will be
required, those will be documented in the release notes.  From the Metron
perspective, this upgrade would be no different than simply upgrading to
the new Metron version.

This upgrade cannot be performed the same way we've done it in the past. A
number of platform upgrades, including OS, are required:

   1. Requires the OS to be updated on all nodes because there are no
   Centos6 RPMs provided in HDP 3.1. Must bump to Centos7.
   2. The final new HBase code will not run on HDP 2.6
   3. The MPack changes for new Ambari are not backwards compatible
   4. YARN and HDFS/MR are also at risk to be backwards incompatible


On Tue, Aug 27, 2019 at 12:39 PM Michael Miklavcic <
michael.miklavcic@gmail.com> wrote:

> Adding the dev list back into the thread (a reply-all was missed).
>
> On Tue, Aug 27, 2019 at 10:49 AM James Sirota <js...@apache.org> wrote:
>
>> I agree with Simon.  HDP 2.x platform is rapidly approaching EOL and
>> everyone will likely need to migrate by end of year.  Doing this platform
>> upgrade sooner will give everyone visibility into what Metron on HDP 3.x
>> looks like so they have time to plan and upgrade at their own pace.
>> Feature-wise, the Metron application itself will be unchanged.  It is
>> merely the platform underneath that is changing.  HDP itself can be
>> upgraded per instructions here:
>> https://docs.hortonworks.com/HDPDocuments/HDP3/HDP-3.1.0/release-notes/content/upgrading_parent.html
>>
>> Once you back up your metron configs, the same configs that worked on the
>> previous version will continue to work on the version running on HDP 3.x.
>> If there is any discrepancy between the two or additional settings will be
>> required, those will be documented in the release notes.  From the Metron
>> perspective, this upgrade would be no different than simply upgrading to
>> the new Metron version.
>>
>> James
>>
>>
>> 27.08.2019, 07:09, "Simon Elliston Ball" <si...@simonellistonball.com>:
>>
>> Something worth noting here is that HDP 2.6.5 is quite old and
>> approaching EoL rapidly, so the issue of upgrade is urgent. I am aware of a
>> large number of users who require this upgrade ASAP, and in fact an aware
>> of zero users who wish to remain on HDP 2.
>>
>> Perhaps those users who want to stay on the old platform can stick their
>> hands up and raise concerns, but this move will likely have to happen very
>> soon.
>>
>> Simon
>>
>> On Tue, 27 Aug 2019 at 15:04, Otto Fowler <ot...@gmail.com>
>> wrote:
>>
>> Although we had the discussion, and some great ideas where passed around,
>> I do not believe we came to some kind of consensus on what 1.0 should look
>> like. So that discussion would have to be picked up again so that we could
>> know where we are at, and make it an actual thing if we were going to make
>> this a 1.0 release.
>>
>> I believe that the issues raised in that discussion gating 1.0 are still
>> largely applicable, including upgrade.
>>
>> Right now we have *ZERO* HDP 3.1 users. We will go from that to *only*
>> supporting 3.1 work and releases. So every user and deployment we currently
>> have will feel real pain, have to slay real dragons to move forward with
>> metron.
>>
>> With regards to support for older versions, the “backward capability”
>> that has been mentioned, I would not say that I have any specific plan for
>> that in mind. What I would say rather, is that I believe that we must be
>> explicit, setting expectations correctly and clearly with regards to our
>> intent while demonstrating that we have thought through the situation. That
>> discussion has not happened, at least I do not believe that the prior dev
>> thread really handles it in context.
>>
>> Depending on the upgrade situation for going to 3.1, it may be that a
>> dual stream of releases or fixes or new features to the extent that we can
>> do it will greatly reduce the pain for folks, or make it viable to stick
>> with metron until they can upgrade.
>>
>> The issue of what metron *is* features wise may be another one we want
>> to take up at some point. The idea being can we separate the metron
>> _integration parts from the metron core functionality such that we can work
>> on them separately and thus support multiple platforms through
>> integrations/applications. Of course definition of metron’s value beyond
>> integration, and what those features and application boundaries are would
>> be necessary.
>>
>>
>>
>>
>> On August 26, 2019 at 18:52:57, Michael Miklavcic (
>> michael.miklavcic@gmail.com) wrote:
>>
>> Hi devs and users,
>>
>> Some questions were asked in the Slack channel about our ongoing
>> HDP/Hadoop upgrade and I'd like to get a discussion rolling. The original
>> Hadoop upgrade discuss thread can be found here
>> https://lists.apache.org/thread.html/37cc29648f0592cc39d3c78a0d07fce38521bdbbc4cf40e022a7a8ea@%3Cdev.metron.apache.org%3E
>>
>> The major issue and impact with upgrading the Hadoop platform is that
>> there are breaking changes. Code that runs on HDP 3.1 will not run on 2.x.
>> Here is a sampling of core components we depend on that we know of so
>> far that are not backwards compatible:
>>
>>    1. The Core OS - we currently base our builds and test deployment off
>>    of artifacts pulled from HDP. The latest rev of HDP no longer ships RPMs
>>    for Centos 6, which means we need to upgrade to Centos 7
>>    2. Ambari
>>    3. HBase
>>
>> This differs from individual components we've upgraded in the past in
>> that our code could still be deployed on the old as well as new version of
>> the component in a backwards compatible way. Based on semantic versioning,
>> I don't know if we can introduce this level of change in a point release,
>> which is the reason for kicking off this discussion. In the past, users and
>> developers in the community have suggested that they are -1 on a 1.x
>> release that does not provide an upgrade
>> https://lists.apache.org/thread.html/eb1a8df2d0a6a79c5d50540d1fdbf215ec83d831ff15d3117c2592cc@%3Cdev.metron.apache.org%3E
>> .
>>
>> Is there a way we can avoid a 1.x release? If we do need 1.x, do we still
>> see upgrades as a gating function? The main issue is that this has the
>> potential to drag out the upgrade and further couple it with other
>> features. And with Storm 1.x being eol'ed, I'm not sure this is something
>> we can wait much longer for. I'll think on this and send out my own
>> thoughts once folks have had a chance to review.
>>
>> Best,
>> Mike Miklavcic
>> Apache Metron, PMC, committer
>>
>>
>> --
>> --
>> simon elliston ball
>> @sireb
>>
>>
>>
>> -------------------
>> Thank you,
>>
>> James Sirota
>> PMC- Apache Metron
>> jsirota AT apache DOT org
>>
>>

Re: [DISCUSS] HDP 3.1 Upgrade and release strategy

[Michael Miklavcic <mi...@gmail.com>]

> Once you back up your metron configs, the same configs that worked on the
previous version will continue to work on the version running on HDP 3.x.
If there is any discrepancy between the two or additional settings will be
required, those will be documented in the release notes.  From the Metron
perspective, this upgrade would be no different than simply upgrading to
the new Metron version.

This upgrade cannot be performed the same way we've done it in the past. A
number of platform upgrades, including OS, are required:

   1. Requires the OS to be updated on all nodes because there are no
   Centos6 RPMs provided in HDP 3.1. Must bump to Centos7.
   2. The final new HBase code will not run on HDP 2.6
   3. The MPack changes for new Ambari are not backwards compatible
   4. YARN and HDFS/MR are also at risk to be backwards incompatible


On Tue, Aug 27, 2019 at 12:39 PM Michael Miklavcic <
michael.miklavcic@gmail.com> wrote:

> Adding the dev list back into the thread (a reply-all was missed).
>
> On Tue, Aug 27, 2019 at 10:49 AM James Sirota <js...@apache.org> wrote:
>
>> I agree with Simon.  HDP 2.x platform is rapidly approaching EOL and
>> everyone will likely need to migrate by end of year.  Doing this platform
>> upgrade sooner will give everyone visibility into what Metron on HDP 3.x
>> looks like so they have time to plan and upgrade at their own pace.
>> Feature-wise, the Metron application itself will be unchanged.  It is
>> merely the platform underneath that is changing.  HDP itself can be
>> upgraded per instructions here:
>> https://docs.hortonworks.com/HDPDocuments/HDP3/HDP-3.1.0/release-notes/content/upgrading_parent.html
>>
>> Once you back up your metron configs, the same configs that worked on the
>> previous version will continue to work on the version running on HDP 3.x.
>> If there is any discrepancy between the two or additional settings will be
>> required, those will be documented in the release notes.  From the Metron
>> perspective, this upgrade would be no different than simply upgrading to
>> the new Metron version.
>>
>> James
>>
>>
>> 27.08.2019, 07:09, "Simon Elliston Ball" <si...@simonellistonball.com>:
>>
>> Something worth noting here is that HDP 2.6.5 is quite old and
>> approaching EoL rapidly, so the issue of upgrade is urgent. I am aware of a
>> large number of users who require this upgrade ASAP, and in fact an aware
>> of zero users who wish to remain on HDP 2.
>>
>> Perhaps those users who want to stay on the old platform can stick their
>> hands up and raise concerns, but this move will likely have to happen very
>> soon.
>>
>> Simon
>>
>> On Tue, 27 Aug 2019 at 15:04, Otto Fowler <ot...@gmail.com>
>> wrote:
>>
>> Although we had the discussion, and some great ideas where passed around,
>> I do not believe we came to some kind of consensus on what 1.0 should look
>> like. So that discussion would have to be picked up again so that we could
>> know where we are at, and make it an actual thing if we were going to make
>> this a 1.0 release.
>>
>> I believe that the issues raised in that discussion gating 1.0 are still
>> largely applicable, including upgrade.
>>
>> Right now we have *ZERO* HDP 3.1 users. We will go from that to *only*
>> supporting 3.1 work and releases. So every user and deployment we currently
>> have will feel real pain, have to slay real dragons to move forward with
>> metron.
>>
>> With regards to support for older versions, the “backward capability”
>> that has been mentioned, I would not say that I have any specific plan for
>> that in mind. What I would say rather, is that I believe that we must be
>> explicit, setting expectations correctly and clearly with regards to our
>> intent while demonstrating that we have thought through the situation. That
>> discussion has not happened, at least I do not believe that the prior dev
>> thread really handles it in context.
>>
>> Depending on the upgrade situation for going to 3.1, it may be that a
>> dual stream of releases or fixes or new features to the extent that we can
>> do it will greatly reduce the pain for folks, or make it viable to stick
>> with metron until they can upgrade.
>>
>> The issue of what metron *is* features wise may be another one we want
>> to take up at some point. The idea being can we separate the metron
>> _integration parts from the metron core functionality such that we can work
>> on them separately and thus support multiple platforms through
>> integrations/applications. Of course definition of metron’s value beyond
>> integration, and what those features and application boundaries are would
>> be necessary.
>>
>>
>>
>>
>> On August 26, 2019 at 18:52:57, Michael Miklavcic (
>> michael.miklavcic@gmail.com) wrote:
>>
>> Hi devs and users,
>>
>> Some questions were asked in the Slack channel about our ongoing
>> HDP/Hadoop upgrade and I'd like to get a discussion rolling. The original
>> Hadoop upgrade discuss thread can be found here
>> https://lists.apache.org/thread.html/37cc29648f0592cc39d3c78a0d07fce38521bdbbc4cf40e022a7a8ea@%3Cdev.metron.apache.org%3E
>>
>> The major issue and impact with upgrading the Hadoop platform is that
>> there are breaking changes. Code that runs on HDP 3.1 will not run on 2.x.
>> Here is a sampling of core components we depend on that we know of so
>> far that are not backwards compatible:
>>
>>    1. The Core OS - we currently base our builds and test deployment off
>>    of artifacts pulled from HDP. The latest rev of HDP no longer ships RPMs
>>    for Centos 6, which means we need to upgrade to Centos 7
>>    2. Ambari
>>    3. HBase
>>
>> This differs from individual components we've upgraded in the past in
>> that our code could still be deployed on the old as well as new version of
>> the component in a backwards compatible way. Based on semantic versioning,
>> I don't know if we can introduce this level of change in a point release,
>> which is the reason for kicking off this discussion. In the past, users and
>> developers in the community have suggested that they are -1 on a 1.x
>> release that does not provide an upgrade
>> https://lists.apache.org/thread.html/eb1a8df2d0a6a79c5d50540d1fdbf215ec83d831ff15d3117c2592cc@%3Cdev.metron.apache.org%3E
>> .
>>
>> Is there a way we can avoid a 1.x release? If we do need 1.x, do we still
>> see upgrades as a gating function? The main issue is that this has the
>> potential to drag out the upgrade and further couple it with other
>> features. And with Storm 1.x being eol'ed, I'm not sure this is something
>> we can wait much longer for. I'll think on this and send out my own
>> thoughts once folks have had a chance to review.
>>
>> Best,
>> Mike Miklavcic
>> Apache Metron, PMC, committer
>>
>>
>> --
>> --
>> simon elliston ball
>> @sireb
>>
>>
>>
>> -------------------
>> Thank you,
>>
>> James Sirota
>> PMC- Apache Metron
>> jsirota AT apache DOT org
>>
>>

Re: [DISCUSS] HDP 3.1 Upgrade and release strategy

[Michael Miklavcic <mi...@gmail.com>]

Adding the dev list back into the thread (a reply-all was missed).

On Tue, Aug 27, 2019 at 10:49 AM James Sirota <js...@apache.org> wrote:

> I agree with Simon.  HDP 2.x platform is rapidly approaching EOL and
> everyone will likely need to migrate by end of year.  Doing this platform
> upgrade sooner will give everyone visibility into what Metron on HDP 3.x
> looks like so they have time to plan and upgrade at their own pace.
> Feature-wise, the Metron application itself will be unchanged.  It is
> merely the platform underneath that is changing.  HDP itself can be
> upgraded per instructions here:
> https://docs.hortonworks.com/HDPDocuments/HDP3/HDP-3.1.0/release-notes/content/upgrading_parent.html
>
> Once you back up your metron configs, the same configs that worked on the
> previous version will continue to work on the version running on HDP 3.x.
> If there is any discrepancy between the two or additional settings will be
> required, those will be documented in the release notes.  From the Metron
> perspective, this upgrade would be no different than simply upgrading to
> the new Metron version.
>
> James
>
>
> 27.08.2019, 07:09, "Simon Elliston Ball" <si...@simonellistonball.com>:
>
> Something worth noting here is that HDP 2.6.5 is quite old and approaching
> EoL rapidly, so the issue of upgrade is urgent. I am aware of a large
> number of users who require this upgrade ASAP, and in fact an aware of zero
> users who wish to remain on HDP 2.
>
> Perhaps those users who want to stay on the old platform can stick their
> hands up and raise concerns, but this move will likely have to happen very
> soon.
>
> Simon
>
> On Tue, 27 Aug 2019 at 15:04, Otto Fowler <ot...@gmail.com> wrote:
>
> Although we had the discussion, and some great ideas where passed around,
> I do not believe we came to some kind of consensus on what 1.0 should look
> like. So that discussion would have to be picked up again so that we could
> know where we are at, and make it an actual thing if we were going to make
> this a 1.0 release.
>
> I believe that the issues raised in that discussion gating 1.0 are still
> largely applicable, including upgrade.
>
> Right now we have *ZERO* HDP 3.1 users. We will go from that to *only*
> supporting 3.1 work and releases. So every user and deployment we currently
> have will feel real pain, have to slay real dragons to move forward with
> metron.
>
> With regards to support for older versions, the “backward capability” that
> has been mentioned, I would not say that I have any specific plan for that
> in mind. What I would say rather, is that I believe that we must be
> explicit, setting expectations correctly and clearly with regards to our
> intent while demonstrating that we have thought through the situation. That
> discussion has not happened, at least I do not believe that the prior dev
> thread really handles it in context.
>
> Depending on the upgrade situation for going to 3.1, it may be that a dual
> stream of releases or fixes or new features to the extent that we can do it
> will greatly reduce the pain for folks, or make it viable to stick with
> metron until they can upgrade.
>
> The issue of what metron *is* features wise may be another one we want to
> take up at some point. The idea being can we separate the metron
> _integration parts from the metron core functionality such that we can work
> on them separately and thus support multiple platforms through
> integrations/applications. Of course definition of metron’s value beyond
> integration, and what those features and application boundaries are would
> be necessary.
>
>
>
>
> On August 26, 2019 at 18:52:57, Michael Miklavcic (
> michael.miklavcic@gmail.com) wrote:
>
> Hi devs and users,
>
> Some questions were asked in the Slack channel about our ongoing
> HDP/Hadoop upgrade and I'd like to get a discussion rolling. The original
> Hadoop upgrade discuss thread can be found here
> https://lists.apache.org/thread.html/37cc29648f0592cc39d3c78a0d07fce38521bdbbc4cf40e022a7a8ea@%3Cdev.metron.apache.org%3E
>
> The major issue and impact with upgrading the Hadoop platform is that
> there are breaking changes. Code that runs on HDP 3.1 will not run on 2.x.
> Here is a sampling of core components we depend on that we know of so
> far that are not backwards compatible:
>
>    1. The Core OS - we currently base our builds and test deployment off
>    of artifacts pulled from HDP. The latest rev of HDP no longer ships RPMs
>    for Centos 6, which means we need to upgrade to Centos 7
>    2. Ambari
>    3. HBase
>
> This differs from individual components we've upgraded in the past in that
> our code could still be deployed on the old as well as new version of the
> component in a backwards compatible way. Based on semantic versioning, I
> don't know if we can introduce this level of change in a point release,
> which is the reason for kicking off this discussion. In the past, users and
> developers in the community have suggested that they are -1 on a 1.x
> release that does not provide an upgrade
> https://lists.apache.org/thread.html/eb1a8df2d0a6a79c5d50540d1fdbf215ec83d831ff15d3117c2592cc@%3Cdev.metron.apache.org%3E
> .
>
> Is there a way we can avoid a 1.x release? If we do need 1.x, do we still
> see upgrades as a gating function? The main issue is that this has the
> potential to drag out the upgrade and further couple it with other
> features. And with Storm 1.x being eol'ed, I'm not sure this is something
> we can wait much longer for. I'll think on this and send out my own
> thoughts once folks have had a chance to review.
>
> Best,
> Mike Miklavcic
> Apache Metron, PMC, committer
>
>
> --
> --
> simon elliston ball
> @sireb
>
>
>
> -------------------
> Thank you,
>
> James Sirota
> PMC- Apache Metron
> jsirota AT apache DOT org
>
>

Re: [DISCUSS] HDP 3.1 Upgrade and release strategy

[Michael Miklavcic <mi...@gmail.com>]

Adding the dev list back into the thread (a reply-all was missed).

On Tue, Aug 27, 2019 at 10:49 AM James Sirota <js...@apache.org> wrote:

> I agree with Simon.  HDP 2.x platform is rapidly approaching EOL and
> everyone will likely need to migrate by end of year.  Doing this platform
> upgrade sooner will give everyone visibility into what Metron on HDP 3.x
> looks like so they have time to plan and upgrade at their own pace.
> Feature-wise, the Metron application itself will be unchanged.  It is
> merely the platform underneath that is changing.  HDP itself can be
> upgraded per instructions here:
> https://docs.hortonworks.com/HDPDocuments/HDP3/HDP-3.1.0/release-notes/content/upgrading_parent.html
>
> Once you back up your metron configs, the same configs that worked on the
> previous version will continue to work on the version running on HDP 3.x.
> If there is any discrepancy between the two or additional settings will be
> required, those will be documented in the release notes.  From the Metron
> perspective, this upgrade would be no different than simply upgrading to
> the new Metron version.
>
> James
>
>
> 27.08.2019, 07:09, "Simon Elliston Ball" <si...@simonellistonball.com>:
>
> Something worth noting here is that HDP 2.6.5 is quite old and approaching
> EoL rapidly, so the issue of upgrade is urgent. I am aware of a large
> number of users who require this upgrade ASAP, and in fact an aware of zero
> users who wish to remain on HDP 2.
>
> Perhaps those users who want to stay on the old platform can stick their
> hands up and raise concerns, but this move will likely have to happen very
> soon.
>
> Simon
>
> On Tue, 27 Aug 2019 at 15:04, Otto Fowler <ot...@gmail.com> wrote:
>
> Although we had the discussion, and some great ideas where passed around,
> I do not believe we came to some kind of consensus on what 1.0 should look
> like. So that discussion would have to be picked up again so that we could
> know where we are at, and make it an actual thing if we were going to make
> this a 1.0 release.
>
> I believe that the issues raised in that discussion gating 1.0 are still
> largely applicable, including upgrade.
>
> Right now we have *ZERO* HDP 3.1 users. We will go from that to *only*
> supporting 3.1 work and releases. So every user and deployment we currently
> have will feel real pain, have to slay real dragons to move forward with
> metron.
>
> With regards to support for older versions, the “backward capability” that
> has been mentioned, I would not say that I have any specific plan for that
> in mind. What I would say rather, is that I believe that we must be
> explicit, setting expectations correctly and clearly with regards to our
> intent while demonstrating that we have thought through the situation. That
> discussion has not happened, at least I do not believe that the prior dev
> thread really handles it in context.
>
> Depending on the upgrade situation for going to 3.1, it may be that a dual
> stream of releases or fixes or new features to the extent that we can do it
> will greatly reduce the pain for folks, or make it viable to stick with
> metron until they can upgrade.
>
> The issue of what metron *is* features wise may be another one we want to
> take up at some point. The idea being can we separate the metron
> _integration parts from the metron core functionality such that we can work
> on them separately and thus support multiple platforms through
> integrations/applications. Of course definition of metron’s value beyond
> integration, and what those features and application boundaries are would
> be necessary.
>
>
>
>
> On August 26, 2019 at 18:52:57, Michael Miklavcic (
> michael.miklavcic@gmail.com) wrote:
>
> Hi devs and users,
>
> Some questions were asked in the Slack channel about our ongoing
> HDP/Hadoop upgrade and I'd like to get a discussion rolling. The original
> Hadoop upgrade discuss thread can be found here
> https://lists.apache.org/thread.html/37cc29648f0592cc39d3c78a0d07fce38521bdbbc4cf40e022a7a8ea@%3Cdev.metron.apache.org%3E
>
> The major issue and impact with upgrading the Hadoop platform is that
> there are breaking changes. Code that runs on HDP 3.1 will not run on 2.x.
> Here is a sampling of core components we depend on that we know of so
> far that are not backwards compatible:
>
>    1. The Core OS - we currently base our builds and test deployment off
>    of artifacts pulled from HDP. The latest rev of HDP no longer ships RPMs
>    for Centos 6, which means we need to upgrade to Centos 7
>    2. Ambari
>    3. HBase
>
> This differs from individual components we've upgraded in the past in that
> our code could still be deployed on the old as well as new version of the
> component in a backwards compatible way. Based on semantic versioning, I
> don't know if we can introduce this level of change in a point release,
> which is the reason for kicking off this discussion. In the past, users and
> developers in the community have suggested that they are -1 on a 1.x
> release that does not provide an upgrade
> https://lists.apache.org/thread.html/eb1a8df2d0a6a79c5d50540d1fdbf215ec83d831ff15d3117c2592cc@%3Cdev.metron.apache.org%3E
> .
>
> Is there a way we can avoid a 1.x release? If we do need 1.x, do we still
> see upgrades as a gating function? The main issue is that this has the
> potential to drag out the upgrade and further couple it with other
> features. And with Storm 1.x being eol'ed, I'm not sure this is something
> we can wait much longer for. I'll think on this and send out my own
> thoughts once folks have had a chance to review.
>
> Best,
> Mike Miklavcic
> Apache Metron, PMC, committer
>
>
> --
> --
> simon elliston ball
> @sireb
>
>
>
> -------------------
> Thank you,
>
> James Sirota
> PMC- Apache Metron
> jsirota AT apache DOT org
>
>

Re: [DISCUSS] HDP 3.1 Upgrade and release strategy

[James Sirota <js...@apache.org>]

I agree with Simon. HDP 2.x platform is rapidly approaching EOL and everyone
will likely need to migrate by end of year. Doing this platform upgrade sooner
will give everyone visibility into what Metron on HDP 3.x looks like so they
have time to plan and upgrade at their own pace. Feature-wise, the Metron
application itself will be unchanged. It is merely the platform underneath
that is changing. HDP itself can be upgraded per instructions here:
<https://docs.hortonworks.com/HDPDocuments/HDP3/HDP-3.1.0/release-
notes/content/upgrading_parent.html>

Once you back up your metron configs, the same configs that worked on the
previous version will continue to work on the version running on HDP 3.x. If
there is any discrepancy between the two or additional settings will be
required, those will be documented in the release notes. From the Metron
perspective, this upgrade would be no different than simply upgrading to the
new Metron version.

James

  

  

27.08.2019, 07:09, "Simon Elliston Ball" <si...@simonellistonball.com>:

> Something worth noting here is that HDP 2.6.5 is quite old and approaching
EoL rapidly, so the issue of upgrade is urgent. I am aware of a large number
of users who require this upgrade ASAP, and in fact an aware of zero users who
wish to remain on HDP 2.

>

>  
>

>

> Perhaps those users who want to stay on the old platform can stick their
hands up and raise concerns, but this move will likely have to happen very
soon.

>

>  
>

>

> Simon

>

>  
>

>

> On Tue, 27 Aug 2019 at 15:04, Otto Fowler
<[ottobackwards@gmail.com](mailto:ottobackwards@gmail.com)> wrote:  
>

>

>> Although we had the discussion, and some great ideas where passed around, I
do not believe we came to some kind of consensus on what 1.0 should look like.
So that discussion would have to be picked up again so that we could know
where we are at, and make it an actual thing if we were going to make this a
1.0 release.

>>

>> I believe that the issues raised in that discussion gating 1.0 are still
largely applicable, including upgrade.

>>

>> Right now we have **ZERO** HDP 3.1 users. We will go from that to _only_
supporting 3.1 work and releases. So every user and deployment we currently
have will feel real pain, have to slay real dragons to move forward with
metron.

>>

>> With regards to support for older versions, the “backward capability” that
has been mentioned, I would not say that I have any specific plan for that in
mind. What I would say rather, is that I believe that we must be explicit,
setting expectations correctly and clearly with regards to our intent while
demonstrating that we have thought through the situation. That discussion has
not happened, at least I do not believe that the prior dev thread really
handles it in context.

>>

>> Depending on the upgrade situation for going to 3.1, it may be that a dual
stream of releases or fixes or new features to the extent that we can do it
will greatly reduce the pain for folks, or make it viable to stick with metron
until they can upgrade.

>>

>> The issue of what metron _is_ features wise may be another one we want to
take up at some point. The idea being can we separate the metron _integration
parts from the metron core functionality such that we can work on them
separately and thus support multiple platforms through
integrations/applications. Of course definition of metron’s value beyond
integration, and what those features and application boundaries are would be
necessary.

>>

>>  
>

>>

>>  
>

>>

>>  
>

>>

>> On August 26, 2019 at 18:52:57, Michael Miklavcic
([michael.miklavcic@gmail.com](mailto:michael.miklavcic@gmail.com)) wrote:

>>

>>> Hi devs and users,

>>>

>>>  
>

>>>

>>> Some questions were asked in the Slack channel about our ongoing
HDP/Hadoop upgrade and I'd like to get a discussion rolling. The original
Hadoop upgrade discuss thread can be found here
<https://lists.apache.org/thread.html/37cc29648f0592cc39d3c78a0d07fce38521bdbbc4cf40e022a7a8ea@%3Cdev.metron.apache.org%3E>

>>>

>>>  
>

>>>

>>> The major issue and impact with upgrading the Hadoop platform is that
there are breaking changes. Code that runs on HDP 3.1 will not run on 2.x.
Here is a sampling of core components we depend on that we know of so far that
are not backwards compatible:

>>>

>>>   1. The Core OS - we currently base our builds and test deployment off of
artifacts pulled from HDP. The latest rev of HDP no longer ships RPMs for
Centos 6, which means we need to upgrade to Centos 7

>>>   2. Ambari

>>>   3. HBase

>>>

>>

>>> This differs from individual components we've upgraded in the past in that
our code could still be deployed on the old as well as new version of the
component in a backwards compatible way. Based on semantic versioning, I don't
know if we can introduce this level of change in a point release, which is the
reason for kicking off this discussion. In the past, users and developers in
the community have suggested that they are -1 on a 1.x release that does not
provide an upgrade
<https://lists.apache.org/thread.html/eb1a8df2d0a6a79c5d50540d1fdbf215ec83d831ff15d3117c2592cc@%3Cdev.metron.apache.org%3E>.

>>>

>>>  
>

>>>

>>> Is there a way we can avoid a 1.x release? If we do need 1.x, do we still
see upgrades as a gating function? The main issue is that this has the
potential to drag out the upgrade and further couple it with other features.
And with Storm 1.x being eol'ed, I'm not sure this is something we can wait
much longer for. I'll think on this and send out my own thoughts once folks
have had a chance to review.

>>>

>>>  
>

>>>

>>> Best,

>>>

>>> Mike Miklavcic

>>>

>>> Apache Metron, PMC, committer

>>>

>>>  
>

>>>

>>>  
>

>

> \--  
>

>

> \--

>

> simon elliston ball

>

> @sireb

  

  

\-------------------

Thank you,

James Sirota

PMC- Apache Metron

jsirota AT apache DOT org

  

Re: [DISCUSS] HDP 3.1 Upgrade and release strategy

[Simon Elliston Ball <si...@simonellistonball.com>]

Something worth noting here is that HDP 2.6.5 is quite old and approaching
EoL rapidly, so the issue of upgrade is urgent. I am aware of a large
number of users who require this upgrade ASAP, and in fact an aware of zero
users who wish to remain on HDP 2.

Perhaps those users who want to stay on the old platform can stick their
hands up and raise concerns, but this move will likely have to happen very
soon.

Simon

On Tue, 27 Aug 2019 at 15:04, Otto Fowler <ot...@gmail.com> wrote:

> Although we had the discussion, and some great ideas where passed around,
> I do not believe we came to some kind of consensus on what 1.0 should look
> like. So that discussion would have to be picked up again so that we could
> know where we are at, and make it an actual thing if we were going to make
> this a 1.0 release.
>
> I believe that the issues raised in that discussion gating 1.0 are still
> largely applicable, including upgrade.
>
> Right now we have *ZERO* HDP 3.1 users. We will go from that to *only*
> supporting 3.1 work and releases. So every user and deployment we currently
> have will feel real pain, have to slay real dragons to move forward with
> metron.
>
> With regards to support for older versions, the “backward capability” that
> has been mentioned, I would not say that I have any specific plan for that
> in mind. What I would say rather, is that I believe that we must be
> explicit, setting expectations correctly and clearly with regards to our
> intent while demonstrating that we have thought through the situation. That
> discussion has not happened, at least I do not believe that the prior dev
> thread really handles it in context.
>
> Depending on the upgrade situation for going to 3.1, it may be that a dual
> stream of releases or fixes or new features to the extent that we can do it
> will greatly reduce the pain for folks, or make it viable to stick with
> metron until they can upgrade.
>
> The issue of what metron *is* features wise may be another one we want to
> take up at some point. The idea being can we separate the metron
> _integration parts from the metron core functionality such that we can work
> on them separately and thus support multiple platforms through
> integrations/applications. Of course definition of metron’s value beyond
> integration, and what those features and application boundaries are would
> be necessary.
>
>
>
>
> On August 26, 2019 at 18:52:57, Michael Miklavcic (
> michael.miklavcic@gmail.com) wrote:
>
> Hi devs and users,
>
> Some questions were asked in the Slack channel about our ongoing
> HDP/Hadoop upgrade and I'd like to get a discussion rolling. The original
> Hadoop upgrade discuss thread can be found here
> https://lists.apache.org/thread.html/37cc29648f0592cc39d3c78a0d07fce38521bdbbc4cf40e022a7a8ea@%3Cdev.metron.apache.org%3E
>
> The major issue and impact with upgrading the Hadoop platform is that
> there are breaking changes. Code that runs on HDP 3.1 will not run on 2.x.
> Here is a sampling of core components we depend on that we know of so
> far that are not backwards compatible:
>
>    1. The Core OS - we currently base our builds and test deployment off
>    of artifacts pulled from HDP. The latest rev of HDP no longer ships RPMs
>    for Centos 6, which means we need to upgrade to Centos 7
>    2. Ambari
>    3. HBase
>
> This differs from individual components we've upgraded in the past in that
> our code could still be deployed on the old as well as new version of the
> component in a backwards compatible way. Based on semantic versioning, I
> don't know if we can introduce this level of change in a point release,
> which is the reason for kicking off this discussion. In the past, users and
> developers in the community have suggested that they are -1 on a 1.x
> release that does not provide an upgrade
> https://lists.apache.org/thread.html/eb1a8df2d0a6a79c5d50540d1fdbf215ec83d831ff15d3117c2592cc@%3Cdev.metron.apache.org%3E
> .
>
> Is there a way we can avoid a 1.x release? If we do need 1.x, do we still
> see upgrades as a gating function? The main issue is that this has the
> potential to drag out the upgrade and further couple it with other
> features. And with Storm 1.x being eol'ed, I'm not sure this is something
> we can wait much longer for. I'll think on this and send out my own
> thoughts once folks have had a chance to review.
>
> Best,
> Mike Miklavcic
> Apache Metron, PMC, committer
>
>
> --
--
simon elliston ball
@sireb

Re: [DISCUSS] HDP 3.1 Upgrade and release strategy

[Otto Fowler <ot...@gmail.com>]

Although we had the discussion, and some great ideas where passed around, I
do not believe we came to some kind of consensus on what 1.0 should look
like. So that discussion would have to be picked up again so that we could
know where we are at, and make it an actual thing if we were going to make
this a 1.0 release.

I believe that the issues raised in that discussion gating 1.0 are still
largely applicable, including upgrade.

Right now we have *ZERO* HDP 3.1 users. We will go from that to *only*
supporting 3.1 work and releases. So every user and deployment we currently
have will feel real pain, have to slay real dragons to move forward with
metron.

With regards to support for older versions, the “backward capability” that
has been mentioned, I would not say that I have any specific plan for that
in mind. What I would say rather, is that I believe that we must be
explicit, setting expectations correctly and clearly with regards to our
intent while demonstrating that we have thought through the situation. That
discussion has not happened, at least I do not believe that the prior dev
thread really handles it in context.

Depending on the upgrade situation for going to 3.1, it may be that a dual
stream of releases or fixes or new features to the extent that we can do it
will greatly reduce the pain for folks, or make it viable to stick with
metron until they can upgrade.

The issue of what metron *is* features wise may be another one we want to
take up at some point. The idea being can we separate the metron
_integration parts from the metron core functionality such that we can work
on them separately and thus support multiple platforms through
integrations/applications. Of course definition of metron’s value beyond
integration, and what those features and application boundaries are would
be necessary.




On August 26, 2019 at 18:52:57, Michael Miklavcic (
michael.miklavcic@gmail.com) wrote:

Hi devs and users,

Some questions were asked in the Slack channel about our ongoing HDP/Hadoop
upgrade and I'd like to get a discussion rolling. The original Hadoop
upgrade discuss thread can be found here
https://lists.apache.org/thread.html/37cc29648f0592cc39d3c78a0d07fce38521bdbbc4cf40e022a7a8ea@%3Cdev.metron.apache.org%3E

The major issue and impact with upgrading the Hadoop platform is that there
are breaking changes. Code that runs on HDP 3.1 will not run on 2.x. Here
is a sampling of core components we depend on that we know of so far that
are not backwards compatible:

   1. The Core OS - we currently base our builds and test deployment off of
   artifacts pulled from HDP. The latest rev of HDP no longer ships RPMs for
   Centos 6, which means we need to upgrade to Centos 7
   2. Ambari
   3. HBase

This differs from individual components we've upgraded in the past in that
our code could still be deployed on the old as well as new version of the
component in a backwards compatible way. Based on semantic versioning, I
don't know if we can introduce this level of change in a point release,
which is the reason for kicking off this discussion. In the past, users and
developers in the community have suggested that they are -1 on a 1.x
release that does not provide an upgrade
https://lists.apache.org/thread.html/eb1a8df2d0a6a79c5d50540d1fdbf215ec83d831ff15d3117c2592cc@%3Cdev.metron.apache.org%3E
.

Is there a way we can avoid a 1.x release? If we do need 1.x, do we still
see upgrades as a gating function? The main issue is that this has the
potential to drag out the upgrade and further couple it with other
features. And with Storm 1.x being eol'ed, I'm not sure this is something
we can wait much longer for. I'll think on this and send out my own
thoughts once folks have had a chance to review.

Best,
Mike Miklavcic
Apache Metron, PMC, committer

Re: [DISCUSS] HDP 3.1 Upgrade and release strategy

[Otto Fowler <ot...@gmail.com>]

Although we had the discussion, and some great ideas where passed around, I
do not believe we came to some kind of consensus on what 1.0 should look
like. So that discussion would have to be picked up again so that we could
know where we are at, and make it an actual thing if we were going to make
this a 1.0 release.

I believe that the issues raised in that discussion gating 1.0 are still
largely applicable, including upgrade.

Right now we have *ZERO* HDP 3.1 users. We will go from that to *only*
supporting 3.1 work and releases. So every user and deployment we currently
have will feel real pain, have to slay real dragons to move forward with
metron.

With regards to support for older versions, the “backward capability” that
has been mentioned, I would not say that I have any specific plan for that
in mind. What I would say rather, is that I believe that we must be
explicit, setting expectations correctly and clearly with regards to our
intent while demonstrating that we have thought through the situation. That
discussion has not happened, at least I do not believe that the prior dev
thread really handles it in context.

Depending on the upgrade situation for going to 3.1, it may be that a dual
stream of releases or fixes or new features to the extent that we can do it
will greatly reduce the pain for folks, or make it viable to stick with
metron until they can upgrade.

The issue of what metron *is* features wise may be another one we want to
take up at some point. The idea being can we separate the metron
_integration parts from the metron core functionality such that we can work
on them separately and thus support multiple platforms through
integrations/applications. Of course definition of metron’s value beyond
integration, and what those features and application boundaries are would
be necessary.




On August 26, 2019 at 18:52:57, Michael Miklavcic (
michael.miklavcic@gmail.com) wrote:

Hi devs and users,

Some questions were asked in the Slack channel about our ongoing HDP/Hadoop
upgrade and I'd like to get a discussion rolling. The original Hadoop
upgrade discuss thread can be found here
https://lists.apache.org/thread.html/37cc29648f0592cc39d3c78a0d07fce38521bdbbc4cf40e022a7a8ea@%3Cdev.metron.apache.org%3E

The major issue and impact with upgrading the Hadoop platform is that there
are breaking changes. Code that runs on HDP 3.1 will not run on 2.x. Here
is a sampling of core components we depend on that we know of so far that
are not backwards compatible:

   1. The Core OS - we currently base our builds and test deployment off of
   artifacts pulled from HDP. The latest rev of HDP no longer ships RPMs for
   Centos 6, which means we need to upgrade to Centos 7
   2. Ambari
   3. HBase

This differs from individual components we've upgraded in the past in that
our code could still be deployed on the old as well as new version of the
component in a backwards compatible way. Based on semantic versioning, I
don't know if we can introduce this level of change in a point release,
which is the reason for kicking off this discussion. In the past, users and
developers in the community have suggested that they are -1 on a 1.x
release that does not provide an upgrade
https://lists.apache.org/thread.html/eb1a8df2d0a6a79c5d50540d1fdbf215ec83d831ff15d3117c2592cc@%3Cdev.metron.apache.org%3E
.

Is there a way we can avoid a 1.x release? If we do need 1.x, do we still
see upgrades as a gating function? The main issue is that this has the
potential to drag out the upgrade and further couple it with other
features. And with Storm 1.x being eol'ed, I'm not sure this is something
we can wait much longer for. I'll think on this and send out my own
thoughts once folks have had a chance to review.

Best,
Mike Miklavcic
Apache Metron, PMC, committer