You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@metron.apache.org by Nick Allen <ni...@nickallen.org> on 2017/11/06 16:16:22 UTC
[DISCUSS] Upcoming Release
I would like to start a discussion around upcoming releases. We have a
couple separate significant tracks of work that we need to reconcile in our
release schedule.
(1) We have had (and have in review) a good number of bug fixes required to
support Metaalerts on the existing Elasticsearch 2.x infrastructure.
(2) We also have ongoing work to upgrade our infrastructure to
Elasticsearch 5.x, which will not be backwards compatible.
I would like to see a release that has our best work on ES 2.x before we
migrate to 5.x. I would propose the following.
Release N+1: Introduce Metaalerts running on ES 2.x
Release N+2: Cut-over to ES 5.x
(Q) Is it worth cutting a separate release for ES 2.x? Is there a better
way to handle the cut-over to 5.x?
Re: [DISCUSS] Upcoming Release
Posted by Nick Allen <ni...@nickallen.org>.
Our last release was 0.4.1, so the next would be at least 0.4.2. We
recently have been keeping master at the next presumed release version.
On Fri, Nov 17, 2017 at 4:59 PM, Ryan Merriman <me...@gmail.com> wrote:
> Matt,
>
> I think we are currently on version 0.4.2. If that is the case would the
> next version be 0.4.3?
>
> Ryan
>
> On Fri, Nov 17, 2017 at 3:31 PM, Matt Foley <ma...@apache.org> wrote:
>
> > (With release manager hat on)
> >
> > The community has proposed a release of Metron in the near future,
> > focusing on Meta-alerts running in Elasticsearch.
> > Congrats on getting so many of the below already done. At this point,
> > only METRON-1252, and the discussion of how to handle joint release of
> the
> > Metron bro plugin, remain as gating items for the release. I project
> these
> > will be resolved next week, so let’s propose the following:
> >
> > Sometime next week, after the last bits are done, I’ll start the release
> > process and create the release branch.
> >
> > The proposed new version will be 0.4.2, unless there are backward
> > incompatible changes that support making it 0.5.0.
> > Currently there are NO included Jiras labeled ‘backward-incompatible’,
> nor
> > having Docs Text indicating so.
> > ***If anyone knows that some of the commits included since 0.4.1
> introduce
> > backward incompatibility, please say so now on this thread, and mark the
> > Jira as such.***
> >
> > The 90 or so jiras/commits already in master branch since 0.4.1 are
> listed
> > below.
> > Thanks,
> > --Matt
> >
> > METRON-1301 Alerts UI - Sorting on Triage Score Unexpectedly Filters
> > Some Records (nickwallen) closes apache/metron#832
> > METRON-1294 IP addresses are not formatted correctly in facet and
> > group results (merrimanr) closes apache/metron#827
> > METRON-1291 Kafka produce REST endpoint does not work in a Kerberized
> > cluster (merrimanr) closes apache/metron#826
> > METRON-1290 Only first 10 alerts are update when a MetaAlert status
> is
> > changed to inactive (justinleet) closes apache/metron#842
> > METRON-1311 Service Check Should Check Elasticsearch Index Templates
> > (nickwallen) closes apache/metron#839
> > METRON-1289 Alert fields are lost when a MetaAlert is created
> > (merrimanr) closes apache/metron#824
> > METRON-1309 Change metron-deployment to pull the plugin from
> > apache/metron-bro-plugin-kafka (JonZeolla) closes apache/metron#837
> > METRON-1310 Template Delete Action Deletes Search Indices
> (nickwallen)
> > closes apache/metron#838
> > METRON-1275: Fix Metron Documentation closes
> > apache/incubator-metron#833
> > METRON-1295 Unable to Configure Logging for REST API (nickwallen)
> > closes apache/metron#828
> > METRON-1307 Force install of java8 since java9 does not appear to
> work
> > with the scripts (brianhurley via ottobackwards) closes apache/metron#835
> > METRON-1296 Full Dev Fails to Deploy Index Templates (nickwallen via
> > cestella) closes apache/incubator-metron#829
> > METRON-1281 Remove hard-coded indices from the Alerts UI (merrimanr)
> > closes apache/metron#821
> > METRON-1287 Full Dev Fails When Installing EPEL Repository
> > (nickwallen) closes apache/metron#820
> > METRON-1267 Alerts UI returns a 404 when refreshing the alerts-list
> > page (iraghumitra via merrimanr) closes apache/metron#819
> > METRON-1283 Install Elasticsearch template as a part of the mpack
> > startup scripts (anandsubbu via nickwallen) closes apache/metron#817
> > METRON-1254: Conditionals as map keys do not function in Stellar
> > closes apache/incubator-metron#801
> > METRON-1261 Apply bro security patch (JonZeolla via ottobackwards)
> > closes apache/metron#805
> > METRON-1284 Remove extraneous dead query in ElasticsearchDao
> > (justinleet) closes apache/metron#818
> > METRON-1270: fix for warnings missing @return tag argument in
> > metron-analytics/metron-profiler-common and metron-profiler-client
> closes
> > apache/incubator-metron#810
> > METRON-1272 Hide child alerts from searches and grouping if they
> > belong to meta alerts (justinleet) closes apache/metron#811
> > METRON-1224 Add time range selection to search control (iraghumitra
> > via james-sirota) closes apache/metron#796
> > METRON-1280 0.4.1 -> 0.4.2 missed a couple of projects (cestella via
> > justinleet) closes apache/metron#816
> > METRON-1243: Add a REST endpoint which allows us to get a list of all
> > indice closes apache/incubator-metron#797
> > METRON-1196 Increment master version number to 0.4.2 for on-going
> > development (mattf-horton) closes apache/metron#767
> > METRON-1278 Strip "Build Status" widget from root README.md
> > in site-book build (mattf-horton) closes apache/metron#815
> > METRON-1274 Master has failure in StormControllerIntegrationTest
> > (merrimanr) closes apache/metron#813
> > METRON-1266 Profiler - SASL Authentication Failed (nickwallen) closes
> > apache/metron#809
> > METRON-1260 Include Alerts UI in Ambari Service Check (nickwallen)
> > closes apache/metron#804
> > METRON-1251: Typo and formatting fixes for metron-rest README closes
> > apache/incubator-metron#800
> > METRON-1241: Enable the REST API to use a cache for the zookeeper
> > config similar to the Bolts closes apache/incubator-metron#795
> > METRON-1267 Alerts UI returns a 404 when refreshing the alerts-list
> > page (merrimanr) closes apache/metron#808
> > METRON-1262 Unable to add comment for a alert in a meta-alert
> > (merrimanr) closes apache/metron#806
> > METRON-1263 Start Alerts UI service after Metron REST (anandsubbu via
> > nickwallen) closes apache/metron#807
> > METRON-1255 MetaAlert search is not filtering on status (merrimanr)
> > closes apache/metron#802
> > METRON-1249 Improve Metron MPack Service Checks (nickwallen) closes
> > apache/metron#799
> > METRON-1237 address javadoc warnings in metron-maas-common (dbist via
> > james-sirota) closes apache/metron#792
> > METRON-1240 address javadoc warnings in metron-platform and
> > metron-analytics (dbist via james-sirota) closes apache/metron#794
> > METRON-1226 Searching Can Errantly Query the Wrong Indices
> > (nickwallen) closes apache/metron#793
> > METRON-1081 Fix Alerts and Ops UI Notices file (james-sirota) closes
> > apache/metron#682
> > METRON-1123 Add group by option using faceted search capabilities of
> > metron-rest-api (iraghumitra via james-sirota) closes apache/metron#768
> > METRON-1223 Add support to add comments for alerts (iraghumitra via
> > james-sirota) closes apache/metron#788
> > METRON-1083 Add filters using faceted search capabilities of
> > metron-rest-api (iraghumitra via james-sirota) closes apache/metron#710
> > METRON-1232 Alert status changes are not reflected in list view
> > (iraghumitra via merrimanr) closes apache/metron#787
> > METRON-1247 REST search and findOne endpoints return unexpected or
> > incorrect results for guids (justinleet) closes apache/metron#798
> > METRON-1235: Document the properties pulled from the global
> > configuration closes apache/incubator-metron#791
> > METRON-1234: fix for WARNING 'dependencies.dependency.(
> > groupId:artifactId:type:classifier)' must be unique:
> > org.apache.hadoop:hadoop-yarn-api:jar (dbist via mmiklavc) closes
> > apache/metron#790
> > METRON-1222: fix warning for The expression ${parent.version} is
> > deprecated. Please use ${project.parent.version} instead. (dbist via
> > mmiklavc) closes apache/metron#782
> > METRON-1220 Create documentation around alert nested field
> > (justinleet) closes apache/metron#780
> > METRON-1229 Management UI type is part of the declarations of 2
> > modules (merrimanr) closes apache/metron#784
> > METRON-1228: Configuration Management PUSH immediately does DUMP
> after
> > (mmiklavc via mmiklavc) closes apache/metron#783
> > METRON-1218 Metron REST should return better error messages
> > (merrimanr) closes apache/metron#779
> > METRON-1161 Add ability to edit parser command line options in the
> > management UI (merrimanr) closes apache/metron#737
> > METRON-1209: Make stellar repl take logging properties, like other
> CLI
> > apps in metron closes apache/incubator-metron#772
> > METRON-1059 address checkstyle warning AvoidStarImport in
> > metron-stellar (dbist via ottobackwards) closes apache/metron#664
> > METRON-1204 UI does not time out after being idle, but stops
> > functioning (merrimanr) closes apache/metron#771
> > METRON-1052: Add forensic similarity hash functions to Stellar closes
> > apache/incubator-metron#781
> > METRON-632: Added validation of "shew.enrichmentType" and
> > "shew.keyColumns" closes apache/incubator-metron#732
> > METRON-1194 Add Profiler Debug Functions to Profiler README
> > (nickwallen via ottobackwards) closes apache/metron#765
> > METRON-1055 Metron 0.4.0 manual installation guide for CentOS 6
> > updates (lvets via ottobackwards) closes apache/metron#661
> > METRON-1079 STELLAR NaN should be a keyword (ottobackwards) closes
> > apache/metron#681
> > METRON-1085 Add REST endpoint to save a user profile for the Alerts
> UI
> > (merrimanr) closes apache/metron#694
> > METRON-1208 MPack for Alerts UI (merrimanr) closes apache/metron#778
> > METRON-1207 Make RPMs for Alerts UI (merrimanr) closes
> > apache/metron#777
> > METRON-1215 Fix link to RPMs chapter (DimDroll via justinleet) closes
> > apache/metron#776
> > METRON-1206 Make alerts UI conform to ops UI for install (merrimanr)
> > closes apache/metron#773
> > METRON-1195 Meta alerts improperly handle updates to non-alert fields
> > (justinleet) closes apache/metron#766
> > METRON-1189 Add alert escalation to the Alerts UI (merrimanr) closes
> > apache/metron#762
> > METRON-1156 Simulate Triage Rules in the Stellar REPL (nickwallen)
> > closes apache/metron#733
> > METRON-1198 Pycapa - No such configuration property
> > 'sasl.kerberos.principal' (nickwallen) closes apache/metron#769
> > METRON-1202 ElasticsearchDao Has extraneous sleep call (justinleet)
> > closes apache/metron#770
> > METRON-938 "service metron-rest start <password>" does not work on
> > CentOS 7. (justinleet) closes apache/metron#757
> > METRON-1182 Refactor Code in alert list to accommodate new view types
> > (iraghumitra via merrimanr) closes apache/metron#756
> > METRON-1188: Ambari global configuration management (mmiklavc) closes
> > apache/metron#760
> > METRON-1191 update public web site to point at 0.4.1 new release
> > (mattf-horton) closes apache/metron#764
> > METRON-1063 address javadoc warnings in metron-stellar (dbist via
> > ottobackwards) closes apache/metron#668
> > METRON-1190 Fix Meta Alert Type handling in calculation of scores
> > (justinleet) closes apache/metron#763
> > METRON-1187 Indexing/Profiler Kafka ACL Groups Not Setup Correctly
> > (nickwallen) closes apache/metron#759
> > METRON-1185: Stellar REPL does not work on a kerberized cluster when
> > calling functions interacting with HBase closes
> apache/incubator-metron#755
> > METRON-1186: Profiler Functions use classutils from shaded storm
> > closes apache/incubator-metron#758
> > METRON-1173: Fix pointers to old stellar docs closes
> > apache/incubator-metron#746
> > METRON-1179: Make STATS_ADD to take a list closes
> > apache/incubator-metron#750
> > METRON-1180: Make Stellar Shell accept zookeeper quorum as a CSV list
> > and not require a port closes apache/incubator-metron#751
> > METRON-1183 Improve KDC Setup Instructions (nickwallen) closes
> > apache/metron#753
> > METRON-1177 Stale running topologies seen post-kerberization and
> cause
> > exceptions (nickwallen) closes apache/metron#748
> > METRON-1158 Build backend for grouping alerts into meta alerts
> > (justinleet) closes apache/metron#734
> > METRON-1146: Add ability to parse JSON string into JSONObject for
> > stellar closes apache/incubator-metron#727
> > METRON-1176 REST: HDFS Service should support setting permissions on
> > files when writing (ottobackwards) closes apache/metron#749
> > METRON-1114 Add group by capabilities to search REST endpoint
> > (merrimanr) closes apache/metron#702
> > METRON-1167 Define Session Specific Global Configuration Values in
> the
> > REPL (nickwallen) closes apache/metron#740
> > METRON-1171: Better validation for the SUBSTRING stellar function
> > closes apache/incubator-metron#745
> >
> >
> >
> > On 11/17/17, 11:59 AM, "Nick Allen" <ni...@nickallen.org> wrote:
> >
> > I just wanted to send an update on where we are at. We've gotten a
> lot
> > done here recently as you can see below.
> >
> > ✓ DONE (1) First, METRON-1289 needs to go in. This one was a
> fairly
> > big
> > effort and I am hearing that we are pretty close.
> >
> > ✓ DONE (2) METRON-1294 fixes an issue in how field types are
> > looked-up.
> >
> > ✓ DONE (3) METRON-1290 is next. While this may have been fixed in
> > M-1289, there may be some test cases we want from this PR.
> >
> > ✓ DONE (4) METRON-1301 addresses a problem with the sorting logic.
> >
> > ✓ DONE (5) METRON-1291 fixes an issue with escalation of
> metaalerts.
> >
> > (6) That leads us to Raghu's UI work in METRON-1252. This
> > introduces the
> > UI bits that depend on all the previous backend work.
> >
> > (7) At this point, we should have our best effort at running
> > Metaalerts
> > on Elasticsearch 2.x. I propose that we cut a release here.
> >
> > (8) After we cut the release, we can introduce the work for ES 5.x
> in
> > METRON-939. I know we will need lots of help testing and reviewing
> > this
> > one.
> >
> >
> >
> > We also have an outstanding question that needs resolved BEFORE we
> > release. We need to come to a consensus on how to release having
> > moved our
> > Bro Plugin to a separate repo. I don't think we've heard from
> > everyone on
> > this. I'd urge everyone to chime in so we can choose a path forward.
> >
> > If anyone is totally confused in regards to that discussion, I can
> try
> > and
> > send an options summary again as a separate discuss thread. The
> > original
> > chain was somewhere around here [1].
> >
> > [1]
> > https://lists.apache.org/thread.html/54a4474881b97e559df24728b3a0e9
> > 23a58345a282451085eef832ef@%3Cdev.metron.apache.org%3E
> >
> >
> >
> > On Wed, Nov 15, 2017 at 10:04 AM, Nick Allen <ni...@nickallen.org>
> > wrote:
> >
> > > Hi Guys -
> > >
> > > I want to follow-up on this discussion. It sounds like most people
> > are in
> > > agreement with the general approach.
> > >
> > > A lot of people have been working hard on Metaalerts and
> > Elasticsearch. I
> > > have checked-in with those doing the heavy lifting and have
> compiled
> > a more
> > > detailed plan based on where we are at now. To the best of my
> > knowledge
> > > here is the plan of attack for finishing out this effort.
> > >
> > > (1) First, METRON-1289 needs to go in. This one was a fairly big
> > effort
> > > and I am hearing that we are pretty close.
> > >
> > > (2) METRON-1294 fixes an issue in how field types are looked-up.
> > >
> > > (3) METRON-1290 is next. While this may have been fixed in
> M-1289,
> > > there may be some test cases we want from this PR.
> > >
> > > (4) METRON-1301 addresses a problem with the sorting logic.
> > >
> > > (5) METRON-1291 fixes an issue with escalation of metaalerts.
> > >
> > > (6) That leads us to Raghu's UI work in METRON-1252. This
> > introduces
> > > the UI bits that depend on all the previous backend work.
> > >
> > > (7) At this point, we should have our best effort at running
> > Metaalerts
> > > on Elasticsearch 2.x. I propose that we cut a release here.
> > >
> > > (8) After we cut the release, we can introduce the work for ES
> 5.x
> > in
> > > METRON-939. I know we will need lots of help testing and reviewing
> > this
> > > one.
> > >
> > > Please correct me if I am wrong. I will try and send out updates
> as
> > we
> > > make progress.
> > >
> > >
> > >
> > >
> > >
> > > On Mon, Nov 6, 2017 at 1:03 PM, Zeolla@GMail.com <zeolla@gmail.com
> >
> > wrote:
> > >
> > >> I agree, I think it's very reasonable to move in line with Nick's
> > >> proposal. I would also suggest that we outline what the target
> > versions
> > >> would be to add in the METRON-777 components, since it has been
> > functional
> > >> for a very long time but not reviewed and has some really rockstar
> > >> improvements.
> > >>
> > >> Jon
> > >>
> > >> On Mon, Nov 6, 2017 at 12:56 PM Otto Fowler <
> > ottobackwards@gmail.com>
> > >> wrote:
> > >>
> > >> > I think the ES cutover should be the start of the 0.5.x series,
> > and we
> > >> > continue on with 0.4.x for the
> > >> > metadata improvements etc. We could chose to focus 0.5.x’s
> first
> > >> releases
> > >> > on not only ES but
> > >> > getting a handle on kibana and the mpack situation as well.
> > >> >
> > >> >
> > >> >
> > >> >
> > >> > On November 6, 2017 at 12:48:45, Michael Miklavcic (
> > >> > michael.miklavcic@gmail.com) wrote:
> > >> >
> > >> > I agree with your proposal, Nick. I think having a stabilizing
> > release
> > >> > prior to upgrading ES/Kibana makes sense.
> > >> >
> > >> > On Mon, Nov 6, 2017 at 9:16 AM, Nick Allen <ni...@nickallen.org>
> > wrote:
> > >> >
> > >> > > I would like to start a discussion around upcoming releases.
> We
> > have a
> > >> > > couple separate significant tracks of work that we need to
> > reconcile
> > >> in
> > >> > our
> > >> > > release schedule.
> > >> > >
> > >> > > (1) We have had (and have in review) a good number of bug
> fixes
> > >> required
> > >> > to
> > >> > > support Metaalerts on the existing Elasticsearch 2.x
> > infrastructure.
> > >> > >
> > >> > >
> > >> > > (2) We also have ongoing work to upgrade our infrastructure to
> > >> > > Elasticsearch 5.x, which will not be backwards compatible.
> > >> > >
> > >> > >
> > >> > > I would like to see a release that has our best work on ES 2.x
> > before
> > >> we
> > >> > > migrate to 5.x. I would propose the following.
> > >> > >
> > >> > > Release N+1: Introduce Metaalerts running on ES 2.x
> > >> > >
> > >> > > Release N+2: Cut-over to ES 5.x
> > >> > >
> > >> > >
> > >> > > (Q) Is it worth cutting a separate release for ES 2.x? Is
> there
> > a
> > >> better
> > >> > > way to handle the cut-over to 5.x?
> > >> > >
> > >> >
> > >> --
> > >>
> > >> Jon
> > >>
> > >
> > >
> >
> >
> >
> >
>
Re: [DISCUSS] Upcoming Release
Posted by "Zeolla@GMail.com" <ze...@gmail.com>.
Should we consider drafting some bare metal install instructions for 0.4.2
that can be a part of the release? Similar to
https://github.com/apache/metron/tree/master/metron-deployment/other-examples/manual-install
Or do we want to continue to have those instructions outside of the
releases themselves (wiki, etc.)? Just a thought, maybe something to do
next release and not this time.
I have a documentation PR that fixes some stellar documentation issues that
I would like to get in (broken links, nonexistent functions, reordering,
etc.), and I am about done with the bro 2.5.2 upgrade in full dev (PR soon,
probably Monday). If we can get that in we will have some cleaner
documentation and a nice clean bro plugin scenario without needing to do a
one off.
Jon
On Fri, Nov 17, 2017, 21:28 Ryan Merriman <me...@gmail.com> wrote:
> Makes sense now. Thanks Matt.
>
> > On Nov 17, 2017, at 4:25 PM, Matt Foley <mf...@hortonworks.com> wrote:
> >
> > Hi Ryan,
> > Yes and no. The last release (see
> https://dist.apache.org/repos/dist/release/metron/ ) was 0.4.1, announced
> on 9/19.
> > Immediately after that we bumped the <version> of builds from master
> branch, per https://issues.apache.org/jira/browse/METRON-1196 . This is
> consistent with the Release Process “Clean up” phase: “It is good practice
> to increment the build version in master immediately after a Feature
> Release, so that dev builds with new stuff from master cannot be mistaken
> for builds of the release version. So, immediately after a release,
> increment the MINOR version number (eg, with the 0.4.0 just released, set
> the new version number to 0.4.1)” (
> https://cwiki.apache.org/confluence/display/METRON/Release+Process#ReleaseProcess-Step15-Cleanup
> )
> >
> > So you’re correct that the version of development builds is currently
> set to 0.4.2. After we actually make a release of 0.4.2, we’ll change dev
> build <version> to 0.4.3 (regardless of whether we expect the following
> release to be 0.4.3 or 0.5.0).
> >
> > Hope this clarifies,
> > --Matt
> >
> > On 11/17/17, 1:59 PM, "Ryan Merriman" <me...@gmail.com> wrote:
> >
> > Matt,
> >
> > I think we are currently on version 0.4.2. If that is the case would
> the
> > next version be 0.4.3?
> >
> > Ryan
> >
> >> On Fri, Nov 17, 2017 at 3:31 PM, Matt Foley <ma...@apache.org>
> wrote:
> >>
> >> (With release manager hat on)
> >>
> >> The community has proposed a release of Metron in the near future,
> >> focusing on Meta-alerts running in Elasticsearch.
> >> Congrats on getting so many of the below already done. At this point,
> >> only METRON-1252, and the discussion of how to handle joint release of
> the
> >> Metron bro plugin, remain as gating items for the release. I project
> these
> >> will be resolved next week, so let’s propose the following:
> >>
> >> Sometime next week, after the last bits are done, I’ll start the release
> >> process and create the release branch.
> >>
> >> The proposed new version will be 0.4.2, unless there are backward
> >> incompatible changes that support making it 0.5.0.
> >> Currently there are NO included Jiras labeled ‘backward-incompatible’,
> nor
> >> having Docs Text indicating so.
> >> ***If anyone knows that some of the commits included since 0.4.1
> introduce
> >> backward incompatibility, please say so now on this thread, and mark the
> >> Jira as such.***
> >>
> >> The 90 or so jiras/commits already in master branch since 0.4.1 are
> listed
> >> below.
> >> Thanks,
> >> --Matt
> >>
> >> METRON-1301 Alerts UI - Sorting on Triage Score Unexpectedly Filters
> >> Some Records (nickwallen) closes apache/metron#832
> >> METRON-1294 IP addresses are not formatted correctly in facet and
> >> group results (merrimanr) closes apache/metron#827
> >> METRON-1291 Kafka produce REST endpoint does not work in a Kerberized
> >> cluster (merrimanr) closes apache/metron#826
> >> METRON-1290 Only first 10 alerts are update when a MetaAlert status
> is
> >> changed to inactive (justinleet) closes apache/metron#842
> >> METRON-1311 Service Check Should Check Elasticsearch Index Templates
> >> (nickwallen) closes apache/metron#839
> >> METRON-1289 Alert fields are lost when a MetaAlert is created
> >> (merrimanr) closes apache/metron#824
> >> METRON-1309 Change metron-deployment to pull the plugin from
> >> apache/metron-bro-plugin-kafka (JonZeolla) closes apache/metron#837
> >> METRON-1310 Template Delete Action Deletes Search Indices
> (nickwallen)
> >> closes apache/metron#838
> >> METRON-1275: Fix Metron Documentation closes
> >> apache/incubator-metron#833
> >> METRON-1295 Unable to Configure Logging for REST API (nickwallen)
> >> closes apache/metron#828
> >> METRON-1307 Force install of java8 since java9 does not appear to
> work
> >> with the scripts (brianhurley via ottobackwards) closes
> apache/metron#835
> >> METRON-1296 Full Dev Fails to Deploy Index Templates (nickwallen via
> >> cestella) closes apache/incubator-metron#829
> >> METRON-1281 Remove hard-coded indices from the Alerts UI (merrimanr)
> >> closes apache/metron#821
> >> METRON-1287 Full Dev Fails When Installing EPEL Repository
> >> (nickwallen) closes apache/metron#820
> >> METRON-1267 Alerts UI returns a 404 when refreshing the alerts-list
> >> page (iraghumitra via merrimanr) closes apache/metron#819
> >> METRON-1283 Install Elasticsearch template as a part of the mpack
> >> startup scripts (anandsubbu via nickwallen) closes apache/metron#817
> >> METRON-1254: Conditionals as map keys do not function in Stellar
> >> closes apache/incubator-metron#801
> >> METRON-1261 Apply bro security patch (JonZeolla via ottobackwards)
> >> closes apache/metron#805
> >> METRON-1284 Remove extraneous dead query in ElasticsearchDao
> >> (justinleet) closes apache/metron#818
> >> METRON-1270: fix for warnings missing @return tag argument in
> >> metron-analytics/metron-profiler-common and metron-profiler-client
> closes
> >> apache/incubator-metron#810
> >> METRON-1272 Hide child alerts from searches and grouping if they
> >> belong to meta alerts (justinleet) closes apache/metron#811
> >> METRON-1224 Add time range selection to search control (iraghumitra
> >> via james-sirota) closes apache/metron#796
> >> METRON-1280 0.4.1 -> 0.4.2 missed a couple of projects (cestella via
> >> justinleet) closes apache/metron#816
> >> METRON-1243: Add a REST endpoint which allows us to get a list of all
> >> indice closes apache/incubator-metron#797
> >> METRON-1196 Increment master version number to 0.4.2 for on-going
> >> development (mattf-horton) closes apache/metron#767
> >> METRON-1278 Strip "Build Status" widget from root README.md
> >> in site-book build (mattf-horton) closes apache/metron#815
> >> METRON-1274 Master has failure in StormControllerIntegrationTest
> >> (merrimanr) closes apache/metron#813
> >> METRON-1266 Profiler - SASL Authentication Failed (nickwallen) closes
> >> apache/metron#809
> >> METRON-1260 Include Alerts UI in Ambari Service Check (nickwallen)
> >> closes apache/metron#804
> >> METRON-1251: Typo and formatting fixes for metron-rest README closes
> >> apache/incubator-metron#800
> >> METRON-1241: Enable the REST API to use a cache for the zookeeper
> >> config similar to the Bolts closes apache/incubator-metron#795
> >> METRON-1267 Alerts UI returns a 404 when refreshing the alerts-list
> >> page (merrimanr) closes apache/metron#808
> >> METRON-1262 Unable to add comment for a alert in a meta-alert
> >> (merrimanr) closes apache/metron#806
> >> METRON-1263 Start Alerts UI service after Metron REST (anandsubbu via
> >> nickwallen) closes apache/metron#807
> >> METRON-1255 MetaAlert search is not filtering on status (merrimanr)
> >> closes apache/metron#802
> >> METRON-1249 Improve Metron MPack Service Checks (nickwallen) closes
> >> apache/metron#799
> >> METRON-1237 address javadoc warnings in metron-maas-common (dbist via
> >> james-sirota) closes apache/metron#792
> >> METRON-1240 address javadoc warnings in metron-platform and
> >> metron-analytics (dbist via james-sirota) closes apache/metron#794
> >> METRON-1226 Searching Can Errantly Query the Wrong Indices
> >> (nickwallen) closes apache/metron#793
> >> METRON-1081 Fix Alerts and Ops UI Notices file (james-sirota) closes
> >> apache/metron#682
> >> METRON-1123 Add group by option using faceted search capabilities of
> >> metron-rest-api (iraghumitra via james-sirota) closes apache/metron#768
> >> METRON-1223 Add support to add comments for alerts (iraghumitra via
> >> james-sirota) closes apache/metron#788
> >> METRON-1083 Add filters using faceted search capabilities of
> >> metron-rest-api (iraghumitra via james-sirota) closes apache/metron#710
> >> METRON-1232 Alert status changes are not reflected in list view
> >> (iraghumitra via merrimanr) closes apache/metron#787
> >> METRON-1247 REST search and findOne endpoints return unexpected or
> >> incorrect results for guids (justinleet) closes apache/metron#798
> >> METRON-1235: Document the properties pulled from the global
> >> configuration closes apache/incubator-metron#791
> >> METRON-1234: fix for WARNING 'dependencies.dependency.(
> >> groupId:artifactId:type:classifier)' must be unique:
> >> org.apache.hadoop:hadoop-yarn-api:jar (dbist via mmiklavc) closes
> >> apache/metron#790
> >> METRON-1222: fix warning for The expression ${parent.version} is
> >> deprecated. Please use ${project.parent.version} instead. (dbist via
> >> mmiklavc) closes apache/metron#782
> >> METRON-1220 Create documentation around alert nested field
> >> (justinleet) closes apache/metron#780
> >> METRON-1229 Management UI type is part of the declarations of 2
> >> modules (merrimanr) closes apache/metron#784
> >> METRON-1228: Configuration Management PUSH immediately does DUMP
> after
> >> (mmiklavc via mmiklavc) closes apache/metron#783
> >> METRON-1218 Metron REST should return better error messages
> >> (merrimanr) closes apache/metron#779
> >> METRON-1161 Add ability to edit parser command line options in the
> >> management UI (merrimanr) closes apache/metron#737
> >> METRON-1209: Make stellar repl take logging properties, like other
> CLI
> >> apps in metron closes apache/incubator-metron#772
> >> METRON-1059 address checkstyle warning AvoidStarImport in
> >> metron-stellar (dbist via ottobackwards) closes apache/metron#664
> >> METRON-1204 UI does not time out after being idle, but stops
> >> functioning (merrimanr) closes apache/metron#771
> >> METRON-1052: Add forensic similarity hash functions to Stellar closes
> >> apache/incubator-metron#781
> >> METRON-632: Added validation of "shew.enrichmentType" and
> >> "shew.keyColumns" closes apache/incubator-metron#732
> >> METRON-1194 Add Profiler Debug Functions to Profiler README
> >> (nickwallen via ottobackwards) closes apache/metron#765
> >> METRON-1055 Metron 0.4.0 manual installation guide for CentOS 6
> >> updates (lvets via ottobackwards) closes apache/metron#661
> >> METRON-1079 STELLAR NaN should be a keyword (ottobackwards) closes
> >> apache/metron#681
> >> METRON-1085 Add REST endpoint to save a user profile for the Alerts
> UI
> >> (merrimanr) closes apache/metron#694
> >> METRON-1208 MPack for Alerts UI (merrimanr) closes apache/metron#778
> >> METRON-1207 Make RPMs for Alerts UI (merrimanr) closes
> >> apache/metron#777
> >> METRON-1215 Fix link to RPMs chapter (DimDroll via justinleet) closes
> >> apache/metron#776
> >> METRON-1206 Make alerts UI conform to ops UI for install (merrimanr)
> >> closes apache/metron#773
> >> METRON-1195 Meta alerts improperly handle updates to non-alert fields
> >> (justinleet) closes apache/metron#766
> >> METRON-1189 Add alert escalation to the Alerts UI (merrimanr) closes
> >> apache/metron#762
> >> METRON-1156 Simulate Triage Rules in the Stellar REPL (nickwallen)
> >> closes apache/metron#733
> >> METRON-1198 Pycapa - No such configuration property
> >> 'sasl.kerberos.principal' (nickwallen) closes apache/metron#769
> >> METRON-1202 ElasticsearchDao Has extraneous sleep call (justinleet)
> >> closes apache/metron#770
> >> METRON-938 "service metron-rest start <password>" does not work on
> >> CentOS 7. (justinleet) closes apache/metron#757
> >> METRON-1182 Refactor Code in alert list to accommodate new view types
> >> (iraghumitra via merrimanr) closes apache/metron#756
> >> METRON-1188: Ambari global configuration management (mmiklavc) closes
> >> apache/metron#760
> >> METRON-1191 update public web site to point at 0.4.1 new release
> >> (mattf-horton) closes apache/metron#764
> >> METRON-1063 address javadoc warnings in metron-stellar (dbist via
> >> ottobackwards) closes apache/metron#668
> >> METRON-1190 Fix Meta Alert Type handling in calculation of scores
> >> (justinleet) closes apache/metron#763
> >> METRON-1187 Indexing/Profiler Kafka ACL Groups Not Setup Correctly
> >> (nickwallen) closes apache/metron#759
> >> METRON-1185: Stellar REPL does not work on a kerberized cluster when
> >> calling functions interacting with HBase closes
> apache/incubator-metron#755
> >> METRON-1186: Profiler Functions use classutils from shaded storm
> >> closes apache/incubator-metron#758
> >> METRON-1173: Fix pointers to old stellar docs closes
> >> apache/incubator-metron#746
> >> METRON-1179: Make STATS_ADD to take a list closes
> >> apache/incubator-metron#750
> >> METRON-1180: Make Stellar Shell accept zookeeper quorum as a CSV list
> >> and not require a port closes apache/incubator-metron#751
> >> METRON-1183 Improve KDC Setup Instructions (nickwallen) closes
> >> apache/metron#753
> >> METRON-1177 Stale running topologies seen post-kerberization and
> cause
> >> exceptions (nickwallen) closes apache/metron#748
> >> METRON-1158 Build backend for grouping alerts into meta alerts
> >> (justinleet) closes apache/metron#734
> >> METRON-1146: Add ability to parse JSON string into JSONObject for
> >> stellar closes apache/incubator-metron#727
> >> METRON-1176 REST: HDFS Service should support setting permissions on
> >> files when writing (ottobackwards) closes apache/metron#749
> >> METRON-1114 Add group by capabilities to search REST endpoint
> >> (merrimanr) closes apache/metron#702
> >> METRON-1167 Define Session Specific Global Configuration Values in
> the
> >> REPL (nickwallen) closes apache/metron#740
> >> METRON-1171: Better validation for the SUBSTRING stellar function
> >> closes apache/incubator-metron#745
> >>
> >>
> >>
> >> On 11/17/17, 11:59 AM, "Nick Allen" <ni...@nickallen.org> wrote:
> >>
> >> I just wanted to send an update on where we are at. We've gotten a
> lot
> >> done here recently as you can see below.
> >>
> >> ✓ DONE (1) First, METRON-1289 needs to go in. This one was a
> fairly
> >> big
> >> effort and I am hearing that we are pretty close.
> >>
> >> ✓ DONE (2) METRON-1294 fixes an issue in how field types are
> >> looked-up.
> >>
> >> ✓ DONE (3) METRON-1290 is next. While this may have been fixed in
> >> M-1289, there may be some test cases we want from this PR.
> >>
> >> ✓ DONE (4) METRON-1301 addresses a problem with the sorting logic.
> >>
> >> ✓ DONE (5) METRON-1291 fixes an issue with escalation of
> metaalerts.
> >>
> >> (6) That leads us to Raghu's UI work in METRON-1252. This
> >> introduces the
> >> UI bits that depend on all the previous backend work.
> >>
> >> (7) At this point, we should have our best effort at running
> >> Metaalerts
> >> on Elasticsearch 2.x. I propose that we cut a release here.
> >>
> >> (8) After we cut the release, we can introduce the work for ES 5.x
> in
> >> METRON-939. I know we will need lots of help testing and reviewing
> >> this
> >> one.
> >>
> >>
> >>
> >> We also have an outstanding question that needs resolved BEFORE we
> >> release. We need to come to a consensus on how to release having
> >> moved our
> >> Bro Plugin to a separate repo. I don't think we've heard from
> >> everyone on
> >> this. I'd urge everyone to chime in so we can choose a path forward.
> >>
> >> If anyone is totally confused in regards to that discussion, I can
> try
> >> and
> >> send an options summary again as a separate discuss thread. The
> >> original
> >> chain was somewhere around here [1].
> >>
> >> [1]
> >> https://lists.apache.org/thread.html/54a4474881b97e559df24728b3a0e9
> >> 23a58345a282451085eef832ef@%3Cdev.metron.apache.org%3E
> >>
> >>
> >>
> >> On Wed, Nov 15, 2017 at 10:04 AM, Nick Allen <ni...@nickallen.org>
> >> wrote:
> >>
> >>> Hi Guys -
> >>>
> >>> I want to follow-up on this discussion. It sounds like most people
> >> are in
> >>> agreement with the general approach.
> >>>
> >>> A lot of people have been working hard on Metaalerts and
> >> Elasticsearch. I
> >>> have checked-in with those doing the heavy lifting and have compiled
> >> a more
> >>> detailed plan based on where we are at now. To the best of my
> >> knowledge
> >>> here is the plan of attack for finishing out this effort.
> >>>
> >>> (1) First, METRON-1289 needs to go in. This one was a fairly big
> >> effort
> >>> and I am hearing that we are pretty close.
> >>>
> >>> (2) METRON-1294 fixes an issue in how field types are looked-up.
> >>>
> >>> (3) METRON-1290 is next. While this may have been fixed in M-1289,
> >>> there may be some test cases we want from this PR.
> >>>
> >>> (4) METRON-1301 addresses a problem with the sorting logic.
> >>>
> >>> (5) METRON-1291 fixes an issue with escalation of metaalerts.
> >>>
> >>> (6) That leads us to Raghu's UI work in METRON-1252. This
> >> introduces
> >>> the UI bits that depend on all the previous backend work.
> >>>
> >>> (7) At this point, we should have our best effort at running
> >> Metaalerts
> >>> on Elasticsearch 2.x. I propose that we cut a release here.
> >>>
> >>> (8) After we cut the release, we can introduce the work for ES 5.x
> >> in
> >>> METRON-939. I know we will need lots of help testing and reviewing
> >> this
> >>> one.
> >>>
> >>> Please correct me if I am wrong. I will try and send out updates as
> >> we
> >>> make progress.
> >>>
> >>>
> >>>
> >>>
> >>>
> >>> On Mon, Nov 6, 2017 at 1:03 PM, Zeolla@GMail.com <ze...@gmail.com>
> >> wrote:
> >>>
> >>>> I agree, I think it's very reasonable to move in line with Nick's
> >>>> proposal. I would also suggest that we outline what the target
> >> versions
> >>>> would be to add in the METRON-777 components, since it has been
> >> functional
> >>>> for a very long time but not reviewed and has some really rockstar
> >>>> improvements.
> >>>>
> >>>> Jon
> >>>>
> >>>> On Mon, Nov 6, 2017 at 12:56 PM Otto Fowler <
> >> ottobackwards@gmail.com>
> >>>> wrote:
> >>>>
> >>>>> I think the ES cutover should be the start of the 0.5.x series,
> >> and we
> >>>>> continue on with 0.4.x for the
> >>>>> metadata improvements etc. We could chose to focus 0.5.x’s first
> >>>> releases
> >>>>> on not only ES but
> >>>>> getting a handle on kibana and the mpack situation as well.
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>> On November 6, 2017 at 12:48:45, Michael Miklavcic (
> >>>>> michael.miklavcic@gmail.com) wrote:
> >>>>>
> >>>>> I agree with your proposal, Nick. I think having a stabilizing
> >> release
> >>>>> prior to upgrading ES/Kibana makes sense.
> >>>>>
> >>>>> On Mon, Nov 6, 2017 at 9:16 AM, Nick Allen <ni...@nickallen.org>
> >> wrote:
> >>>>>
> >>>>>> I would like to start a discussion around upcoming releases. We
> >> have a
> >>>>>> couple separate significant tracks of work that we need to
> >> reconcile
> >>>> in
> >>>>> our
> >>>>>> release schedule.
> >>>>>>
> >>>>>> (1) We have had (and have in review) a good number of bug fixes
> >>>> required
> >>>>> to
> >>>>>> support Metaalerts on the existing Elasticsearch 2.x
> >> infrastructure.
> >>>>>>
> >>>>>>
> >>>>>> (2) We also have ongoing work to upgrade our infrastructure to
> >>>>>> Elasticsearch 5.x, which will not be backwards compatible.
> >>>>>>
> >>>>>>
> >>>>>> I would like to see a release that has our best work on ES 2.x
> >> before
> >>>> we
> >>>>>> migrate to 5.x. I would propose the following.
> >>>>>>
> >>>>>> Release N+1: Introduce Metaalerts running on ES 2.x
> >>>>>>
> >>>>>> Release N+2: Cut-over to ES 5.x
> >>>>>>
> >>>>>>
> >>>>>> (Q) Is it worth cutting a separate release for ES 2.x? Is there
> >> a
> >>>> better
> >>>>>> way to handle the cut-over to 5.x?
> >>>>>>
> >>>>>
> >>>> --
> >>>>
> >>>> Jon
> >>>>
> >>>
> >>>
> >>
> >>
> >>
> >>
> >
> >
>
--
Jon
Re: [DISCUSS] Upcoming Release
Posted by Ryan Merriman <me...@gmail.com>.
Makes sense now. Thanks Matt.
> On Nov 17, 2017, at 4:25 PM, Matt Foley <mf...@hortonworks.com> wrote:
>
> Hi Ryan,
> Yes and no. The last release (see https://dist.apache.org/repos/dist/release/metron/ ) was 0.4.1, announced on 9/19.
> Immediately after that we bumped the <version> of builds from master branch, per https://issues.apache.org/jira/browse/METRON-1196 . This is consistent with the Release Process “Clean up” phase: “It is good practice to increment the build version in master immediately after a Feature Release, so that dev builds with new stuff from master cannot be mistaken for builds of the release version. So, immediately after a release, increment the MINOR version number (eg, with the 0.4.0 just released, set the new version number to 0.4.1)” (https://cwiki.apache.org/confluence/display/METRON/Release+Process#ReleaseProcess-Step15-Cleanup )
>
> So you’re correct that the version of development builds is currently set to 0.4.2. After we actually make a release of 0.4.2, we’ll change dev build <version> to 0.4.3 (regardless of whether we expect the following release to be 0.4.3 or 0.5.0).
>
> Hope this clarifies,
> --Matt
>
> On 11/17/17, 1:59 PM, "Ryan Merriman" <me...@gmail.com> wrote:
>
> Matt,
>
> I think we are currently on version 0.4.2. If that is the case would the
> next version be 0.4.3?
>
> Ryan
>
>> On Fri, Nov 17, 2017 at 3:31 PM, Matt Foley <ma...@apache.org> wrote:
>>
>> (With release manager hat on)
>>
>> The community has proposed a release of Metron in the near future,
>> focusing on Meta-alerts running in Elasticsearch.
>> Congrats on getting so many of the below already done. At this point,
>> only METRON-1252, and the discussion of how to handle joint release of the
>> Metron bro plugin, remain as gating items for the release. I project these
>> will be resolved next week, so let’s propose the following:
>>
>> Sometime next week, after the last bits are done, I’ll start the release
>> process and create the release branch.
>>
>> The proposed new version will be 0.4.2, unless there are backward
>> incompatible changes that support making it 0.5.0.
>> Currently there are NO included Jiras labeled ‘backward-incompatible’, nor
>> having Docs Text indicating so.
>> ***If anyone knows that some of the commits included since 0.4.1 introduce
>> backward incompatibility, please say so now on this thread, and mark the
>> Jira as such.***
>>
>> The 90 or so jiras/commits already in master branch since 0.4.1 are listed
>> below.
>> Thanks,
>> --Matt
>>
>> METRON-1301 Alerts UI - Sorting on Triage Score Unexpectedly Filters
>> Some Records (nickwallen) closes apache/metron#832
>> METRON-1294 IP addresses are not formatted correctly in facet and
>> group results (merrimanr) closes apache/metron#827
>> METRON-1291 Kafka produce REST endpoint does not work in a Kerberized
>> cluster (merrimanr) closes apache/metron#826
>> METRON-1290 Only first 10 alerts are update when a MetaAlert status is
>> changed to inactive (justinleet) closes apache/metron#842
>> METRON-1311 Service Check Should Check Elasticsearch Index Templates
>> (nickwallen) closes apache/metron#839
>> METRON-1289 Alert fields are lost when a MetaAlert is created
>> (merrimanr) closes apache/metron#824
>> METRON-1309 Change metron-deployment to pull the plugin from
>> apache/metron-bro-plugin-kafka (JonZeolla) closes apache/metron#837
>> METRON-1310 Template Delete Action Deletes Search Indices (nickwallen)
>> closes apache/metron#838
>> METRON-1275: Fix Metron Documentation closes
>> apache/incubator-metron#833
>> METRON-1295 Unable to Configure Logging for REST API (nickwallen)
>> closes apache/metron#828
>> METRON-1307 Force install of java8 since java9 does not appear to work
>> with the scripts (brianhurley via ottobackwards) closes apache/metron#835
>> METRON-1296 Full Dev Fails to Deploy Index Templates (nickwallen via
>> cestella) closes apache/incubator-metron#829
>> METRON-1281 Remove hard-coded indices from the Alerts UI (merrimanr)
>> closes apache/metron#821
>> METRON-1287 Full Dev Fails When Installing EPEL Repository
>> (nickwallen) closes apache/metron#820
>> METRON-1267 Alerts UI returns a 404 when refreshing the alerts-list
>> page (iraghumitra via merrimanr) closes apache/metron#819
>> METRON-1283 Install Elasticsearch template as a part of the mpack
>> startup scripts (anandsubbu via nickwallen) closes apache/metron#817
>> METRON-1254: Conditionals as map keys do not function in Stellar
>> closes apache/incubator-metron#801
>> METRON-1261 Apply bro security patch (JonZeolla via ottobackwards)
>> closes apache/metron#805
>> METRON-1284 Remove extraneous dead query in ElasticsearchDao
>> (justinleet) closes apache/metron#818
>> METRON-1270: fix for warnings missing @return tag argument in
>> metron-analytics/metron-profiler-common and metron-profiler-client closes
>> apache/incubator-metron#810
>> METRON-1272 Hide child alerts from searches and grouping if they
>> belong to meta alerts (justinleet) closes apache/metron#811
>> METRON-1224 Add time range selection to search control (iraghumitra
>> via james-sirota) closes apache/metron#796
>> METRON-1280 0.4.1 -> 0.4.2 missed a couple of projects (cestella via
>> justinleet) closes apache/metron#816
>> METRON-1243: Add a REST endpoint which allows us to get a list of all
>> indice closes apache/incubator-metron#797
>> METRON-1196 Increment master version number to 0.4.2 for on-going
>> development (mattf-horton) closes apache/metron#767
>> METRON-1278 Strip "Build Status" widget from root README.md
>> in site-book build (mattf-horton) closes apache/metron#815
>> METRON-1274 Master has failure in StormControllerIntegrationTest
>> (merrimanr) closes apache/metron#813
>> METRON-1266 Profiler - SASL Authentication Failed (nickwallen) closes
>> apache/metron#809
>> METRON-1260 Include Alerts UI in Ambari Service Check (nickwallen)
>> closes apache/metron#804
>> METRON-1251: Typo and formatting fixes for metron-rest README closes
>> apache/incubator-metron#800
>> METRON-1241: Enable the REST API to use a cache for the zookeeper
>> config similar to the Bolts closes apache/incubator-metron#795
>> METRON-1267 Alerts UI returns a 404 when refreshing the alerts-list
>> page (merrimanr) closes apache/metron#808
>> METRON-1262 Unable to add comment for a alert in a meta-alert
>> (merrimanr) closes apache/metron#806
>> METRON-1263 Start Alerts UI service after Metron REST (anandsubbu via
>> nickwallen) closes apache/metron#807
>> METRON-1255 MetaAlert search is not filtering on status (merrimanr)
>> closes apache/metron#802
>> METRON-1249 Improve Metron MPack Service Checks (nickwallen) closes
>> apache/metron#799
>> METRON-1237 address javadoc warnings in metron-maas-common (dbist via
>> james-sirota) closes apache/metron#792
>> METRON-1240 address javadoc warnings in metron-platform and
>> metron-analytics (dbist via james-sirota) closes apache/metron#794
>> METRON-1226 Searching Can Errantly Query the Wrong Indices
>> (nickwallen) closes apache/metron#793
>> METRON-1081 Fix Alerts and Ops UI Notices file (james-sirota) closes
>> apache/metron#682
>> METRON-1123 Add group by option using faceted search capabilities of
>> metron-rest-api (iraghumitra via james-sirota) closes apache/metron#768
>> METRON-1223 Add support to add comments for alerts (iraghumitra via
>> james-sirota) closes apache/metron#788
>> METRON-1083 Add filters using faceted search capabilities of
>> metron-rest-api (iraghumitra via james-sirota) closes apache/metron#710
>> METRON-1232 Alert status changes are not reflected in list view
>> (iraghumitra via merrimanr) closes apache/metron#787
>> METRON-1247 REST search and findOne endpoints return unexpected or
>> incorrect results for guids (justinleet) closes apache/metron#798
>> METRON-1235: Document the properties pulled from the global
>> configuration closes apache/incubator-metron#791
>> METRON-1234: fix for WARNING 'dependencies.dependency.(
>> groupId:artifactId:type:classifier)' must be unique:
>> org.apache.hadoop:hadoop-yarn-api:jar (dbist via mmiklavc) closes
>> apache/metron#790
>> METRON-1222: fix warning for The expression ${parent.version} is
>> deprecated. Please use ${project.parent.version} instead. (dbist via
>> mmiklavc) closes apache/metron#782
>> METRON-1220 Create documentation around alert nested field
>> (justinleet) closes apache/metron#780
>> METRON-1229 Management UI type is part of the declarations of 2
>> modules (merrimanr) closes apache/metron#784
>> METRON-1228: Configuration Management PUSH immediately does DUMP after
>> (mmiklavc via mmiklavc) closes apache/metron#783
>> METRON-1218 Metron REST should return better error messages
>> (merrimanr) closes apache/metron#779
>> METRON-1161 Add ability to edit parser command line options in the
>> management UI (merrimanr) closes apache/metron#737
>> METRON-1209: Make stellar repl take logging properties, like other CLI
>> apps in metron closes apache/incubator-metron#772
>> METRON-1059 address checkstyle warning AvoidStarImport in
>> metron-stellar (dbist via ottobackwards) closes apache/metron#664
>> METRON-1204 UI does not time out after being idle, but stops
>> functioning (merrimanr) closes apache/metron#771
>> METRON-1052: Add forensic similarity hash functions to Stellar closes
>> apache/incubator-metron#781
>> METRON-632: Added validation of "shew.enrichmentType" and
>> "shew.keyColumns" closes apache/incubator-metron#732
>> METRON-1194 Add Profiler Debug Functions to Profiler README
>> (nickwallen via ottobackwards) closes apache/metron#765
>> METRON-1055 Metron 0.4.0 manual installation guide for CentOS 6
>> updates (lvets via ottobackwards) closes apache/metron#661
>> METRON-1079 STELLAR NaN should be a keyword (ottobackwards) closes
>> apache/metron#681
>> METRON-1085 Add REST endpoint to save a user profile for the Alerts UI
>> (merrimanr) closes apache/metron#694
>> METRON-1208 MPack for Alerts UI (merrimanr) closes apache/metron#778
>> METRON-1207 Make RPMs for Alerts UI (merrimanr) closes
>> apache/metron#777
>> METRON-1215 Fix link to RPMs chapter (DimDroll via justinleet) closes
>> apache/metron#776
>> METRON-1206 Make alerts UI conform to ops UI for install (merrimanr)
>> closes apache/metron#773
>> METRON-1195 Meta alerts improperly handle updates to non-alert fields
>> (justinleet) closes apache/metron#766
>> METRON-1189 Add alert escalation to the Alerts UI (merrimanr) closes
>> apache/metron#762
>> METRON-1156 Simulate Triage Rules in the Stellar REPL (nickwallen)
>> closes apache/metron#733
>> METRON-1198 Pycapa - No such configuration property
>> 'sasl.kerberos.principal' (nickwallen) closes apache/metron#769
>> METRON-1202 ElasticsearchDao Has extraneous sleep call (justinleet)
>> closes apache/metron#770
>> METRON-938 "service metron-rest start <password>" does not work on
>> CentOS 7. (justinleet) closes apache/metron#757
>> METRON-1182 Refactor Code in alert list to accommodate new view types
>> (iraghumitra via merrimanr) closes apache/metron#756
>> METRON-1188: Ambari global configuration management (mmiklavc) closes
>> apache/metron#760
>> METRON-1191 update public web site to point at 0.4.1 new release
>> (mattf-horton) closes apache/metron#764
>> METRON-1063 address javadoc warnings in metron-stellar (dbist via
>> ottobackwards) closes apache/metron#668
>> METRON-1190 Fix Meta Alert Type handling in calculation of scores
>> (justinleet) closes apache/metron#763
>> METRON-1187 Indexing/Profiler Kafka ACL Groups Not Setup Correctly
>> (nickwallen) closes apache/metron#759
>> METRON-1185: Stellar REPL does not work on a kerberized cluster when
>> calling functions interacting with HBase closes apache/incubator-metron#755
>> METRON-1186: Profiler Functions use classutils from shaded storm
>> closes apache/incubator-metron#758
>> METRON-1173: Fix pointers to old stellar docs closes
>> apache/incubator-metron#746
>> METRON-1179: Make STATS_ADD to take a list closes
>> apache/incubator-metron#750
>> METRON-1180: Make Stellar Shell accept zookeeper quorum as a CSV list
>> and not require a port closes apache/incubator-metron#751
>> METRON-1183 Improve KDC Setup Instructions (nickwallen) closes
>> apache/metron#753
>> METRON-1177 Stale running topologies seen post-kerberization and cause
>> exceptions (nickwallen) closes apache/metron#748
>> METRON-1158 Build backend for grouping alerts into meta alerts
>> (justinleet) closes apache/metron#734
>> METRON-1146: Add ability to parse JSON string into JSONObject for
>> stellar closes apache/incubator-metron#727
>> METRON-1176 REST: HDFS Service should support setting permissions on
>> files when writing (ottobackwards) closes apache/metron#749
>> METRON-1114 Add group by capabilities to search REST endpoint
>> (merrimanr) closes apache/metron#702
>> METRON-1167 Define Session Specific Global Configuration Values in the
>> REPL (nickwallen) closes apache/metron#740
>> METRON-1171: Better validation for the SUBSTRING stellar function
>> closes apache/incubator-metron#745
>>
>>
>>
>> On 11/17/17, 11:59 AM, "Nick Allen" <ni...@nickallen.org> wrote:
>>
>> I just wanted to send an update on where we are at. We've gotten a lot
>> done here recently as you can see below.
>>
>> ✓ DONE (1) First, METRON-1289 needs to go in. This one was a fairly
>> big
>> effort and I am hearing that we are pretty close.
>>
>> ✓ DONE (2) METRON-1294 fixes an issue in how field types are
>> looked-up.
>>
>> ✓ DONE (3) METRON-1290 is next. While this may have been fixed in
>> M-1289, there may be some test cases we want from this PR.
>>
>> ✓ DONE (4) METRON-1301 addresses a problem with the sorting logic.
>>
>> ✓ DONE (5) METRON-1291 fixes an issue with escalation of metaalerts.
>>
>> (6) That leads us to Raghu's UI work in METRON-1252. This
>> introduces the
>> UI bits that depend on all the previous backend work.
>>
>> (7) At this point, we should have our best effort at running
>> Metaalerts
>> on Elasticsearch 2.x. I propose that we cut a release here.
>>
>> (8) After we cut the release, we can introduce the work for ES 5.x in
>> METRON-939. I know we will need lots of help testing and reviewing
>> this
>> one.
>>
>>
>>
>> We also have an outstanding question that needs resolved BEFORE we
>> release. We need to come to a consensus on how to release having
>> moved our
>> Bro Plugin to a separate repo. I don't think we've heard from
>> everyone on
>> this. I'd urge everyone to chime in so we can choose a path forward.
>>
>> If anyone is totally confused in regards to that discussion, I can try
>> and
>> send an options summary again as a separate discuss thread. The
>> original
>> chain was somewhere around here [1].
>>
>> [1]
>> https://lists.apache.org/thread.html/54a4474881b97e559df24728b3a0e9
>> 23a58345a282451085eef832ef@%3Cdev.metron.apache.org%3E
>>
>>
>>
>> On Wed, Nov 15, 2017 at 10:04 AM, Nick Allen <ni...@nickallen.org>
>> wrote:
>>
>>> Hi Guys -
>>>
>>> I want to follow-up on this discussion. It sounds like most people
>> are in
>>> agreement with the general approach.
>>>
>>> A lot of people have been working hard on Metaalerts and
>> Elasticsearch. I
>>> have checked-in with those doing the heavy lifting and have compiled
>> a more
>>> detailed plan based on where we are at now. To the best of my
>> knowledge
>>> here is the plan of attack for finishing out this effort.
>>>
>>> (1) First, METRON-1289 needs to go in. This one was a fairly big
>> effort
>>> and I am hearing that we are pretty close.
>>>
>>> (2) METRON-1294 fixes an issue in how field types are looked-up.
>>>
>>> (3) METRON-1290 is next. While this may have been fixed in M-1289,
>>> there may be some test cases we want from this PR.
>>>
>>> (4) METRON-1301 addresses a problem with the sorting logic.
>>>
>>> (5) METRON-1291 fixes an issue with escalation of metaalerts.
>>>
>>> (6) That leads us to Raghu's UI work in METRON-1252. This
>> introduces
>>> the UI bits that depend on all the previous backend work.
>>>
>>> (7) At this point, we should have our best effort at running
>> Metaalerts
>>> on Elasticsearch 2.x. I propose that we cut a release here.
>>>
>>> (8) After we cut the release, we can introduce the work for ES 5.x
>> in
>>> METRON-939. I know we will need lots of help testing and reviewing
>> this
>>> one.
>>>
>>> Please correct me if I am wrong. I will try and send out updates as
>> we
>>> make progress.
>>>
>>>
>>>
>>>
>>>
>>> On Mon, Nov 6, 2017 at 1:03 PM, Zeolla@GMail.com <ze...@gmail.com>
>> wrote:
>>>
>>>> I agree, I think it's very reasonable to move in line with Nick's
>>>> proposal. I would also suggest that we outline what the target
>> versions
>>>> would be to add in the METRON-777 components, since it has been
>> functional
>>>> for a very long time but not reviewed and has some really rockstar
>>>> improvements.
>>>>
>>>> Jon
>>>>
>>>> On Mon, Nov 6, 2017 at 12:56 PM Otto Fowler <
>> ottobackwards@gmail.com>
>>>> wrote:
>>>>
>>>>> I think the ES cutover should be the start of the 0.5.x series,
>> and we
>>>>> continue on with 0.4.x for the
>>>>> metadata improvements etc. We could chose to focus 0.5.x’s first
>>>> releases
>>>>> on not only ES but
>>>>> getting a handle on kibana and the mpack situation as well.
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On November 6, 2017 at 12:48:45, Michael Miklavcic (
>>>>> michael.miklavcic@gmail.com) wrote:
>>>>>
>>>>> I agree with your proposal, Nick. I think having a stabilizing
>> release
>>>>> prior to upgrading ES/Kibana makes sense.
>>>>>
>>>>> On Mon, Nov 6, 2017 at 9:16 AM, Nick Allen <ni...@nickallen.org>
>> wrote:
>>>>>
>>>>>> I would like to start a discussion around upcoming releases. We
>> have a
>>>>>> couple separate significant tracks of work that we need to
>> reconcile
>>>> in
>>>>> our
>>>>>> release schedule.
>>>>>>
>>>>>> (1) We have had (and have in review) a good number of bug fixes
>>>> required
>>>>> to
>>>>>> support Metaalerts on the existing Elasticsearch 2.x
>> infrastructure.
>>>>>>
>>>>>>
>>>>>> (2) We also have ongoing work to upgrade our infrastructure to
>>>>>> Elasticsearch 5.x, which will not be backwards compatible.
>>>>>>
>>>>>>
>>>>>> I would like to see a release that has our best work on ES 2.x
>> before
>>>> we
>>>>>> migrate to 5.x. I would propose the following.
>>>>>>
>>>>>> Release N+1: Introduce Metaalerts running on ES 2.x
>>>>>>
>>>>>> Release N+2: Cut-over to ES 5.x
>>>>>>
>>>>>>
>>>>>> (Q) Is it worth cutting a separate release for ES 2.x? Is there
>> a
>>>> better
>>>>>> way to handle the cut-over to 5.x?
>>>>>>
>>>>>
>>>> --
>>>>
>>>> Jon
>>>>
>>>
>>>
>>
>>
>>
>>
>
>
Re: [DISCUSS] Upcoming Release
Posted by Matt Foley <mf...@hortonworks.com>.
Hi Ryan,
Yes and no. The last release (see https://dist.apache.org/repos/dist/release/metron/ ) was 0.4.1, announced on 9/19.
Immediately after that we bumped the <version> of builds from master branch, per https://issues.apache.org/jira/browse/METRON-1196 . This is consistent with the Release Process “Clean up” phase: “It is good practice to increment the build version in master immediately after a Feature Release, so that dev builds with new stuff from master cannot be mistaken for builds of the release version. So, immediately after a release, increment the MINOR version number (eg, with the 0.4.0 just released, set the new version number to 0.4.1)” (https://cwiki.apache.org/confluence/display/METRON/Release+Process#ReleaseProcess-Step15-Cleanup )
So you’re correct that the version of development builds is currently set to 0.4.2. After we actually make a release of 0.4.2, we’ll change dev build <version> to 0.4.3 (regardless of whether we expect the following release to be 0.4.3 or 0.5.0).
Hope this clarifies,
--Matt
On 11/17/17, 1:59 PM, "Ryan Merriman" <me...@gmail.com> wrote:
Matt,
I think we are currently on version 0.4.2. If that is the case would the
next version be 0.4.3?
Ryan
On Fri, Nov 17, 2017 at 3:31 PM, Matt Foley <ma...@apache.org> wrote:
> (With release manager hat on)
>
> The community has proposed a release of Metron in the near future,
> focusing on Meta-alerts running in Elasticsearch.
> Congrats on getting so many of the below already done. At this point,
> only METRON-1252, and the discussion of how to handle joint release of the
> Metron bro plugin, remain as gating items for the release. I project these
> will be resolved next week, so let’s propose the following:
>
> Sometime next week, after the last bits are done, I’ll start the release
> process and create the release branch.
>
> The proposed new version will be 0.4.2, unless there are backward
> incompatible changes that support making it 0.5.0.
> Currently there are NO included Jiras labeled ‘backward-incompatible’, nor
> having Docs Text indicating so.
> ***If anyone knows that some of the commits included since 0.4.1 introduce
> backward incompatibility, please say so now on this thread, and mark the
> Jira as such.***
>
> The 90 or so jiras/commits already in master branch since 0.4.1 are listed
> below.
> Thanks,
> --Matt
>
> METRON-1301 Alerts UI - Sorting on Triage Score Unexpectedly Filters
> Some Records (nickwallen) closes apache/metron#832
> METRON-1294 IP addresses are not formatted correctly in facet and
> group results (merrimanr) closes apache/metron#827
> METRON-1291 Kafka produce REST endpoint does not work in a Kerberized
> cluster (merrimanr) closes apache/metron#826
> METRON-1290 Only first 10 alerts are update when a MetaAlert status is
> changed to inactive (justinleet) closes apache/metron#842
> METRON-1311 Service Check Should Check Elasticsearch Index Templates
> (nickwallen) closes apache/metron#839
> METRON-1289 Alert fields are lost when a MetaAlert is created
> (merrimanr) closes apache/metron#824
> METRON-1309 Change metron-deployment to pull the plugin from
> apache/metron-bro-plugin-kafka (JonZeolla) closes apache/metron#837
> METRON-1310 Template Delete Action Deletes Search Indices (nickwallen)
> closes apache/metron#838
> METRON-1275: Fix Metron Documentation closes
> apache/incubator-metron#833
> METRON-1295 Unable to Configure Logging for REST API (nickwallen)
> closes apache/metron#828
> METRON-1307 Force install of java8 since java9 does not appear to work
> with the scripts (brianhurley via ottobackwards) closes apache/metron#835
> METRON-1296 Full Dev Fails to Deploy Index Templates (nickwallen via
> cestella) closes apache/incubator-metron#829
> METRON-1281 Remove hard-coded indices from the Alerts UI (merrimanr)
> closes apache/metron#821
> METRON-1287 Full Dev Fails When Installing EPEL Repository
> (nickwallen) closes apache/metron#820
> METRON-1267 Alerts UI returns a 404 when refreshing the alerts-list
> page (iraghumitra via merrimanr) closes apache/metron#819
> METRON-1283 Install Elasticsearch template as a part of the mpack
> startup scripts (anandsubbu via nickwallen) closes apache/metron#817
> METRON-1254: Conditionals as map keys do not function in Stellar
> closes apache/incubator-metron#801
> METRON-1261 Apply bro security patch (JonZeolla via ottobackwards)
> closes apache/metron#805
> METRON-1284 Remove extraneous dead query in ElasticsearchDao
> (justinleet) closes apache/metron#818
> METRON-1270: fix for warnings missing @return tag argument in
> metron-analytics/metron-profiler-common and metron-profiler-client closes
> apache/incubator-metron#810
> METRON-1272 Hide child alerts from searches and grouping if they
> belong to meta alerts (justinleet) closes apache/metron#811
> METRON-1224 Add time range selection to search control (iraghumitra
> via james-sirota) closes apache/metron#796
> METRON-1280 0.4.1 -> 0.4.2 missed a couple of projects (cestella via
> justinleet) closes apache/metron#816
> METRON-1243: Add a REST endpoint which allows us to get a list of all
> indice closes apache/incubator-metron#797
> METRON-1196 Increment master version number to 0.4.2 for on-going
> development (mattf-horton) closes apache/metron#767
> METRON-1278 Strip "Build Status" widget from root README.md
> in site-book build (mattf-horton) closes apache/metron#815
> METRON-1274 Master has failure in StormControllerIntegrationTest
> (merrimanr) closes apache/metron#813
> METRON-1266 Profiler - SASL Authentication Failed (nickwallen) closes
> apache/metron#809
> METRON-1260 Include Alerts UI in Ambari Service Check (nickwallen)
> closes apache/metron#804
> METRON-1251: Typo and formatting fixes for metron-rest README closes
> apache/incubator-metron#800
> METRON-1241: Enable the REST API to use a cache for the zookeeper
> config similar to the Bolts closes apache/incubator-metron#795
> METRON-1267 Alerts UI returns a 404 when refreshing the alerts-list
> page (merrimanr) closes apache/metron#808
> METRON-1262 Unable to add comment for a alert in a meta-alert
> (merrimanr) closes apache/metron#806
> METRON-1263 Start Alerts UI service after Metron REST (anandsubbu via
> nickwallen) closes apache/metron#807
> METRON-1255 MetaAlert search is not filtering on status (merrimanr)
> closes apache/metron#802
> METRON-1249 Improve Metron MPack Service Checks (nickwallen) closes
> apache/metron#799
> METRON-1237 address javadoc warnings in metron-maas-common (dbist via
> james-sirota) closes apache/metron#792
> METRON-1240 address javadoc warnings in metron-platform and
> metron-analytics (dbist via james-sirota) closes apache/metron#794
> METRON-1226 Searching Can Errantly Query the Wrong Indices
> (nickwallen) closes apache/metron#793
> METRON-1081 Fix Alerts and Ops UI Notices file (james-sirota) closes
> apache/metron#682
> METRON-1123 Add group by option using faceted search capabilities of
> metron-rest-api (iraghumitra via james-sirota) closes apache/metron#768
> METRON-1223 Add support to add comments for alerts (iraghumitra via
> james-sirota) closes apache/metron#788
> METRON-1083 Add filters using faceted search capabilities of
> metron-rest-api (iraghumitra via james-sirota) closes apache/metron#710
> METRON-1232 Alert status changes are not reflected in list view
> (iraghumitra via merrimanr) closes apache/metron#787
> METRON-1247 REST search and findOne endpoints return unexpected or
> incorrect results for guids (justinleet) closes apache/metron#798
> METRON-1235: Document the properties pulled from the global
> configuration closes apache/incubator-metron#791
> METRON-1234: fix for WARNING 'dependencies.dependency.(
> groupId:artifactId:type:classifier)' must be unique:
> org.apache.hadoop:hadoop-yarn-api:jar (dbist via mmiklavc) closes
> apache/metron#790
> METRON-1222: fix warning for The expression ${parent.version} is
> deprecated. Please use ${project.parent.version} instead. (dbist via
> mmiklavc) closes apache/metron#782
> METRON-1220 Create documentation around alert nested field
> (justinleet) closes apache/metron#780
> METRON-1229 Management UI type is part of the declarations of 2
> modules (merrimanr) closes apache/metron#784
> METRON-1228: Configuration Management PUSH immediately does DUMP after
> (mmiklavc via mmiklavc) closes apache/metron#783
> METRON-1218 Metron REST should return better error messages
> (merrimanr) closes apache/metron#779
> METRON-1161 Add ability to edit parser command line options in the
> management UI (merrimanr) closes apache/metron#737
> METRON-1209: Make stellar repl take logging properties, like other CLI
> apps in metron closes apache/incubator-metron#772
> METRON-1059 address checkstyle warning AvoidStarImport in
> metron-stellar (dbist via ottobackwards) closes apache/metron#664
> METRON-1204 UI does not time out after being idle, but stops
> functioning (merrimanr) closes apache/metron#771
> METRON-1052: Add forensic similarity hash functions to Stellar closes
> apache/incubator-metron#781
> METRON-632: Added validation of "shew.enrichmentType" and
> "shew.keyColumns" closes apache/incubator-metron#732
> METRON-1194 Add Profiler Debug Functions to Profiler README
> (nickwallen via ottobackwards) closes apache/metron#765
> METRON-1055 Metron 0.4.0 manual installation guide for CentOS 6
> updates (lvets via ottobackwards) closes apache/metron#661
> METRON-1079 STELLAR NaN should be a keyword (ottobackwards) closes
> apache/metron#681
> METRON-1085 Add REST endpoint to save a user profile for the Alerts UI
> (merrimanr) closes apache/metron#694
> METRON-1208 MPack for Alerts UI (merrimanr) closes apache/metron#778
> METRON-1207 Make RPMs for Alerts UI (merrimanr) closes
> apache/metron#777
> METRON-1215 Fix link to RPMs chapter (DimDroll via justinleet) closes
> apache/metron#776
> METRON-1206 Make alerts UI conform to ops UI for install (merrimanr)
> closes apache/metron#773
> METRON-1195 Meta alerts improperly handle updates to non-alert fields
> (justinleet) closes apache/metron#766
> METRON-1189 Add alert escalation to the Alerts UI (merrimanr) closes
> apache/metron#762
> METRON-1156 Simulate Triage Rules in the Stellar REPL (nickwallen)
> closes apache/metron#733
> METRON-1198 Pycapa - No such configuration property
> 'sasl.kerberos.principal' (nickwallen) closes apache/metron#769
> METRON-1202 ElasticsearchDao Has extraneous sleep call (justinleet)
> closes apache/metron#770
> METRON-938 "service metron-rest start <password>" does not work on
> CentOS 7. (justinleet) closes apache/metron#757
> METRON-1182 Refactor Code in alert list to accommodate new view types
> (iraghumitra via merrimanr) closes apache/metron#756
> METRON-1188: Ambari global configuration management (mmiklavc) closes
> apache/metron#760
> METRON-1191 update public web site to point at 0.4.1 new release
> (mattf-horton) closes apache/metron#764
> METRON-1063 address javadoc warnings in metron-stellar (dbist via
> ottobackwards) closes apache/metron#668
> METRON-1190 Fix Meta Alert Type handling in calculation of scores
> (justinleet) closes apache/metron#763
> METRON-1187 Indexing/Profiler Kafka ACL Groups Not Setup Correctly
> (nickwallen) closes apache/metron#759
> METRON-1185: Stellar REPL does not work on a kerberized cluster when
> calling functions interacting with HBase closes apache/incubator-metron#755
> METRON-1186: Profiler Functions use classutils from shaded storm
> closes apache/incubator-metron#758
> METRON-1173: Fix pointers to old stellar docs closes
> apache/incubator-metron#746
> METRON-1179: Make STATS_ADD to take a list closes
> apache/incubator-metron#750
> METRON-1180: Make Stellar Shell accept zookeeper quorum as a CSV list
> and not require a port closes apache/incubator-metron#751
> METRON-1183 Improve KDC Setup Instructions (nickwallen) closes
> apache/metron#753
> METRON-1177 Stale running topologies seen post-kerberization and cause
> exceptions (nickwallen) closes apache/metron#748
> METRON-1158 Build backend for grouping alerts into meta alerts
> (justinleet) closes apache/metron#734
> METRON-1146: Add ability to parse JSON string into JSONObject for
> stellar closes apache/incubator-metron#727
> METRON-1176 REST: HDFS Service should support setting permissions on
> files when writing (ottobackwards) closes apache/metron#749
> METRON-1114 Add group by capabilities to search REST endpoint
> (merrimanr) closes apache/metron#702
> METRON-1167 Define Session Specific Global Configuration Values in the
> REPL (nickwallen) closes apache/metron#740
> METRON-1171: Better validation for the SUBSTRING stellar function
> closes apache/incubator-metron#745
>
>
>
> On 11/17/17, 11:59 AM, "Nick Allen" <ni...@nickallen.org> wrote:
>
> I just wanted to send an update on where we are at. We've gotten a lot
> done here recently as you can see below.
>
> ✓ DONE (1) First, METRON-1289 needs to go in. This one was a fairly
> big
> effort and I am hearing that we are pretty close.
>
> ✓ DONE (2) METRON-1294 fixes an issue in how field types are
> looked-up.
>
> ✓ DONE (3) METRON-1290 is next. While this may have been fixed in
> M-1289, there may be some test cases we want from this PR.
>
> ✓ DONE (4) METRON-1301 addresses a problem with the sorting logic.
>
> ✓ DONE (5) METRON-1291 fixes an issue with escalation of metaalerts.
>
> (6) That leads us to Raghu's UI work in METRON-1252. This
> introduces the
> UI bits that depend on all the previous backend work.
>
> (7) At this point, we should have our best effort at running
> Metaalerts
> on Elasticsearch 2.x. I propose that we cut a release here.
>
> (8) After we cut the release, we can introduce the work for ES 5.x in
> METRON-939. I know we will need lots of help testing and reviewing
> this
> one.
>
>
>
> We also have an outstanding question that needs resolved BEFORE we
> release. We need to come to a consensus on how to release having
> moved our
> Bro Plugin to a separate repo. I don't think we've heard from
> everyone on
> this. I'd urge everyone to chime in so we can choose a path forward.
>
> If anyone is totally confused in regards to that discussion, I can try
> and
> send an options summary again as a separate discuss thread. The
> original
> chain was somewhere around here [1].
>
> [1]
> https://lists.apache.org/thread.html/54a4474881b97e559df24728b3a0e9
> 23a58345a282451085eef832ef@%3Cdev.metron.apache.org%3E
>
>
>
> On Wed, Nov 15, 2017 at 10:04 AM, Nick Allen <ni...@nickallen.org>
> wrote:
>
> > Hi Guys -
> >
> > I want to follow-up on this discussion. It sounds like most people
> are in
> > agreement with the general approach.
> >
> > A lot of people have been working hard on Metaalerts and
> Elasticsearch. I
> > have checked-in with those doing the heavy lifting and have compiled
> a more
> > detailed plan based on where we are at now. To the best of my
> knowledge
> > here is the plan of attack for finishing out this effort.
> >
> > (1) First, METRON-1289 needs to go in. This one was a fairly big
> effort
> > and I am hearing that we are pretty close.
> >
> > (2) METRON-1294 fixes an issue in how field types are looked-up.
> >
> > (3) METRON-1290 is next. While this may have been fixed in M-1289,
> > there may be some test cases we want from this PR.
> >
> > (4) METRON-1301 addresses a problem with the sorting logic.
> >
> > (5) METRON-1291 fixes an issue with escalation of metaalerts.
> >
> > (6) That leads us to Raghu's UI work in METRON-1252. This
> introduces
> > the UI bits that depend on all the previous backend work.
> >
> > (7) At this point, we should have our best effort at running
> Metaalerts
> > on Elasticsearch 2.x. I propose that we cut a release here.
> >
> > (8) After we cut the release, we can introduce the work for ES 5.x
> in
> > METRON-939. I know we will need lots of help testing and reviewing
> this
> > one.
> >
> > Please correct me if I am wrong. I will try and send out updates as
> we
> > make progress.
> >
> >
> >
> >
> >
> > On Mon, Nov 6, 2017 at 1:03 PM, Zeolla@GMail.com <ze...@gmail.com>
> wrote:
> >
> >> I agree, I think it's very reasonable to move in line with Nick's
> >> proposal. I would also suggest that we outline what the target
> versions
> >> would be to add in the METRON-777 components, since it has been
> functional
> >> for a very long time but not reviewed and has some really rockstar
> >> improvements.
> >>
> >> Jon
> >>
> >> On Mon, Nov 6, 2017 at 12:56 PM Otto Fowler <
> ottobackwards@gmail.com>
> >> wrote:
> >>
> >> > I think the ES cutover should be the start of the 0.5.x series,
> and we
> >> > continue on with 0.4.x for the
> >> > metadata improvements etc. We could chose to focus 0.5.x’s first
> >> releases
> >> > on not only ES but
> >> > getting a handle on kibana and the mpack situation as well.
> >> >
> >> >
> >> >
> >> >
> >> > On November 6, 2017 at 12:48:45, Michael Miklavcic (
> >> > michael.miklavcic@gmail.com) wrote:
> >> >
> >> > I agree with your proposal, Nick. I think having a stabilizing
> release
> >> > prior to upgrading ES/Kibana makes sense.
> >> >
> >> > On Mon, Nov 6, 2017 at 9:16 AM, Nick Allen <ni...@nickallen.org>
> wrote:
> >> >
> >> > > I would like to start a discussion around upcoming releases. We
> have a
> >> > > couple separate significant tracks of work that we need to
> reconcile
> >> in
> >> > our
> >> > > release schedule.
> >> > >
> >> > > (1) We have had (and have in review) a good number of bug fixes
> >> required
> >> > to
> >> > > support Metaalerts on the existing Elasticsearch 2.x
> infrastructure.
> >> > >
> >> > >
> >> > > (2) We also have ongoing work to upgrade our infrastructure to
> >> > > Elasticsearch 5.x, which will not be backwards compatible.
> >> > >
> >> > >
> >> > > I would like to see a release that has our best work on ES 2.x
> before
> >> we
> >> > > migrate to 5.x. I would propose the following.
> >> > >
> >> > > Release N+1: Introduce Metaalerts running on ES 2.x
> >> > >
> >> > > Release N+2: Cut-over to ES 5.x
> >> > >
> >> > >
> >> > > (Q) Is it worth cutting a separate release for ES 2.x? Is there
> a
> >> better
> >> > > way to handle the cut-over to 5.x?
> >> > >
> >> >
> >> --
> >>
> >> Jon
> >>
> >
> >
>
>
>
>
Re: [DISCUSS] Upcoming Release
Posted by Ryan Merriman <me...@gmail.com>.
Matt,
I think we are currently on version 0.4.2. If that is the case would the
next version be 0.4.3?
Ryan
On Fri, Nov 17, 2017 at 3:31 PM, Matt Foley <ma...@apache.org> wrote:
> (With release manager hat on)
>
> The community has proposed a release of Metron in the near future,
> focusing on Meta-alerts running in Elasticsearch.
> Congrats on getting so many of the below already done. At this point,
> only METRON-1252, and the discussion of how to handle joint release of the
> Metron bro plugin, remain as gating items for the release. I project these
> will be resolved next week, so let’s propose the following:
>
> Sometime next week, after the last bits are done, I’ll start the release
> process and create the release branch.
>
> The proposed new version will be 0.4.2, unless there are backward
> incompatible changes that support making it 0.5.0.
> Currently there are NO included Jiras labeled ‘backward-incompatible’, nor
> having Docs Text indicating so.
> ***If anyone knows that some of the commits included since 0.4.1 introduce
> backward incompatibility, please say so now on this thread, and mark the
> Jira as such.***
>
> The 90 or so jiras/commits already in master branch since 0.4.1 are listed
> below.
> Thanks,
> --Matt
>
> METRON-1301 Alerts UI - Sorting on Triage Score Unexpectedly Filters
> Some Records (nickwallen) closes apache/metron#832
> METRON-1294 IP addresses are not formatted correctly in facet and
> group results (merrimanr) closes apache/metron#827
> METRON-1291 Kafka produce REST endpoint does not work in a Kerberized
> cluster (merrimanr) closes apache/metron#826
> METRON-1290 Only first 10 alerts are update when a MetaAlert status is
> changed to inactive (justinleet) closes apache/metron#842
> METRON-1311 Service Check Should Check Elasticsearch Index Templates
> (nickwallen) closes apache/metron#839
> METRON-1289 Alert fields are lost when a MetaAlert is created
> (merrimanr) closes apache/metron#824
> METRON-1309 Change metron-deployment to pull the plugin from
> apache/metron-bro-plugin-kafka (JonZeolla) closes apache/metron#837
> METRON-1310 Template Delete Action Deletes Search Indices (nickwallen)
> closes apache/metron#838
> METRON-1275: Fix Metron Documentation closes
> apache/incubator-metron#833
> METRON-1295 Unable to Configure Logging for REST API (nickwallen)
> closes apache/metron#828
> METRON-1307 Force install of java8 since java9 does not appear to work
> with the scripts (brianhurley via ottobackwards) closes apache/metron#835
> METRON-1296 Full Dev Fails to Deploy Index Templates (nickwallen via
> cestella) closes apache/incubator-metron#829
> METRON-1281 Remove hard-coded indices from the Alerts UI (merrimanr)
> closes apache/metron#821
> METRON-1287 Full Dev Fails When Installing EPEL Repository
> (nickwallen) closes apache/metron#820
> METRON-1267 Alerts UI returns a 404 when refreshing the alerts-list
> page (iraghumitra via merrimanr) closes apache/metron#819
> METRON-1283 Install Elasticsearch template as a part of the mpack
> startup scripts (anandsubbu via nickwallen) closes apache/metron#817
> METRON-1254: Conditionals as map keys do not function in Stellar
> closes apache/incubator-metron#801
> METRON-1261 Apply bro security patch (JonZeolla via ottobackwards)
> closes apache/metron#805
> METRON-1284 Remove extraneous dead query in ElasticsearchDao
> (justinleet) closes apache/metron#818
> METRON-1270: fix for warnings missing @return tag argument in
> metron-analytics/metron-profiler-common and metron-profiler-client closes
> apache/incubator-metron#810
> METRON-1272 Hide child alerts from searches and grouping if they
> belong to meta alerts (justinleet) closes apache/metron#811
> METRON-1224 Add time range selection to search control (iraghumitra
> via james-sirota) closes apache/metron#796
> METRON-1280 0.4.1 -> 0.4.2 missed a couple of projects (cestella via
> justinleet) closes apache/metron#816
> METRON-1243: Add a REST endpoint which allows us to get a list of all
> indice closes apache/incubator-metron#797
> METRON-1196 Increment master version number to 0.4.2 for on-going
> development (mattf-horton) closes apache/metron#767
> METRON-1278 Strip "Build Status" widget from root README.md
> in site-book build (mattf-horton) closes apache/metron#815
> METRON-1274 Master has failure in StormControllerIntegrationTest
> (merrimanr) closes apache/metron#813
> METRON-1266 Profiler - SASL Authentication Failed (nickwallen) closes
> apache/metron#809
> METRON-1260 Include Alerts UI in Ambari Service Check (nickwallen)
> closes apache/metron#804
> METRON-1251: Typo and formatting fixes for metron-rest README closes
> apache/incubator-metron#800
> METRON-1241: Enable the REST API to use a cache for the zookeeper
> config similar to the Bolts closes apache/incubator-metron#795
> METRON-1267 Alerts UI returns a 404 when refreshing the alerts-list
> page (merrimanr) closes apache/metron#808
> METRON-1262 Unable to add comment for a alert in a meta-alert
> (merrimanr) closes apache/metron#806
> METRON-1263 Start Alerts UI service after Metron REST (anandsubbu via
> nickwallen) closes apache/metron#807
> METRON-1255 MetaAlert search is not filtering on status (merrimanr)
> closes apache/metron#802
> METRON-1249 Improve Metron MPack Service Checks (nickwallen) closes
> apache/metron#799
> METRON-1237 address javadoc warnings in metron-maas-common (dbist via
> james-sirota) closes apache/metron#792
> METRON-1240 address javadoc warnings in metron-platform and
> metron-analytics (dbist via james-sirota) closes apache/metron#794
> METRON-1226 Searching Can Errantly Query the Wrong Indices
> (nickwallen) closes apache/metron#793
> METRON-1081 Fix Alerts and Ops UI Notices file (james-sirota) closes
> apache/metron#682
> METRON-1123 Add group by option using faceted search capabilities of
> metron-rest-api (iraghumitra via james-sirota) closes apache/metron#768
> METRON-1223 Add support to add comments for alerts (iraghumitra via
> james-sirota) closes apache/metron#788
> METRON-1083 Add filters using faceted search capabilities of
> metron-rest-api (iraghumitra via james-sirota) closes apache/metron#710
> METRON-1232 Alert status changes are not reflected in list view
> (iraghumitra via merrimanr) closes apache/metron#787
> METRON-1247 REST search and findOne endpoints return unexpected or
> incorrect results for guids (justinleet) closes apache/metron#798
> METRON-1235: Document the properties pulled from the global
> configuration closes apache/incubator-metron#791
> METRON-1234: fix for WARNING 'dependencies.dependency.(
> groupId:artifactId:type:classifier)' must be unique:
> org.apache.hadoop:hadoop-yarn-api:jar (dbist via mmiklavc) closes
> apache/metron#790
> METRON-1222: fix warning for The expression ${parent.version} is
> deprecated. Please use ${project.parent.version} instead. (dbist via
> mmiklavc) closes apache/metron#782
> METRON-1220 Create documentation around alert nested field
> (justinleet) closes apache/metron#780
> METRON-1229 Management UI type is part of the declarations of 2
> modules (merrimanr) closes apache/metron#784
> METRON-1228: Configuration Management PUSH immediately does DUMP after
> (mmiklavc via mmiklavc) closes apache/metron#783
> METRON-1218 Metron REST should return better error messages
> (merrimanr) closes apache/metron#779
> METRON-1161 Add ability to edit parser command line options in the
> management UI (merrimanr) closes apache/metron#737
> METRON-1209: Make stellar repl take logging properties, like other CLI
> apps in metron closes apache/incubator-metron#772
> METRON-1059 address checkstyle warning AvoidStarImport in
> metron-stellar (dbist via ottobackwards) closes apache/metron#664
> METRON-1204 UI does not time out after being idle, but stops
> functioning (merrimanr) closes apache/metron#771
> METRON-1052: Add forensic similarity hash functions to Stellar closes
> apache/incubator-metron#781
> METRON-632: Added validation of "shew.enrichmentType" and
> "shew.keyColumns" closes apache/incubator-metron#732
> METRON-1194 Add Profiler Debug Functions to Profiler README
> (nickwallen via ottobackwards) closes apache/metron#765
> METRON-1055 Metron 0.4.0 manual installation guide for CentOS 6
> updates (lvets via ottobackwards) closes apache/metron#661
> METRON-1079 STELLAR NaN should be a keyword (ottobackwards) closes
> apache/metron#681
> METRON-1085 Add REST endpoint to save a user profile for the Alerts UI
> (merrimanr) closes apache/metron#694
> METRON-1208 MPack for Alerts UI (merrimanr) closes apache/metron#778
> METRON-1207 Make RPMs for Alerts UI (merrimanr) closes
> apache/metron#777
> METRON-1215 Fix link to RPMs chapter (DimDroll via justinleet) closes
> apache/metron#776
> METRON-1206 Make alerts UI conform to ops UI for install (merrimanr)
> closes apache/metron#773
> METRON-1195 Meta alerts improperly handle updates to non-alert fields
> (justinleet) closes apache/metron#766
> METRON-1189 Add alert escalation to the Alerts UI (merrimanr) closes
> apache/metron#762
> METRON-1156 Simulate Triage Rules in the Stellar REPL (nickwallen)
> closes apache/metron#733
> METRON-1198 Pycapa - No such configuration property
> 'sasl.kerberos.principal' (nickwallen) closes apache/metron#769
> METRON-1202 ElasticsearchDao Has extraneous sleep call (justinleet)
> closes apache/metron#770
> METRON-938 "service metron-rest start <password>" does not work on
> CentOS 7. (justinleet) closes apache/metron#757
> METRON-1182 Refactor Code in alert list to accommodate new view types
> (iraghumitra via merrimanr) closes apache/metron#756
> METRON-1188: Ambari global configuration management (mmiklavc) closes
> apache/metron#760
> METRON-1191 update public web site to point at 0.4.1 new release
> (mattf-horton) closes apache/metron#764
> METRON-1063 address javadoc warnings in metron-stellar (dbist via
> ottobackwards) closes apache/metron#668
> METRON-1190 Fix Meta Alert Type handling in calculation of scores
> (justinleet) closes apache/metron#763
> METRON-1187 Indexing/Profiler Kafka ACL Groups Not Setup Correctly
> (nickwallen) closes apache/metron#759
> METRON-1185: Stellar REPL does not work on a kerberized cluster when
> calling functions interacting with HBase closes apache/incubator-metron#755
> METRON-1186: Profiler Functions use classutils from shaded storm
> closes apache/incubator-metron#758
> METRON-1173: Fix pointers to old stellar docs closes
> apache/incubator-metron#746
> METRON-1179: Make STATS_ADD to take a list closes
> apache/incubator-metron#750
> METRON-1180: Make Stellar Shell accept zookeeper quorum as a CSV list
> and not require a port closes apache/incubator-metron#751
> METRON-1183 Improve KDC Setup Instructions (nickwallen) closes
> apache/metron#753
> METRON-1177 Stale running topologies seen post-kerberization and cause
> exceptions (nickwallen) closes apache/metron#748
> METRON-1158 Build backend for grouping alerts into meta alerts
> (justinleet) closes apache/metron#734
> METRON-1146: Add ability to parse JSON string into JSONObject for
> stellar closes apache/incubator-metron#727
> METRON-1176 REST: HDFS Service should support setting permissions on
> files when writing (ottobackwards) closes apache/metron#749
> METRON-1114 Add group by capabilities to search REST endpoint
> (merrimanr) closes apache/metron#702
> METRON-1167 Define Session Specific Global Configuration Values in the
> REPL (nickwallen) closes apache/metron#740
> METRON-1171: Better validation for the SUBSTRING stellar function
> closes apache/incubator-metron#745
>
>
>
> On 11/17/17, 11:59 AM, "Nick Allen" <ni...@nickallen.org> wrote:
>
> I just wanted to send an update on where we are at. We've gotten a lot
> done here recently as you can see below.
>
> ✓ DONE (1) First, METRON-1289 needs to go in. This one was a fairly
> big
> effort and I am hearing that we are pretty close.
>
> ✓ DONE (2) METRON-1294 fixes an issue in how field types are
> looked-up.
>
> ✓ DONE (3) METRON-1290 is next. While this may have been fixed in
> M-1289, there may be some test cases we want from this PR.
>
> ✓ DONE (4) METRON-1301 addresses a problem with the sorting logic.
>
> ✓ DONE (5) METRON-1291 fixes an issue with escalation of metaalerts.
>
> (6) That leads us to Raghu's UI work in METRON-1252. This
> introduces the
> UI bits that depend on all the previous backend work.
>
> (7) At this point, we should have our best effort at running
> Metaalerts
> on Elasticsearch 2.x. I propose that we cut a release here.
>
> (8) After we cut the release, we can introduce the work for ES 5.x in
> METRON-939. I know we will need lots of help testing and reviewing
> this
> one.
>
>
>
> We also have an outstanding question that needs resolved BEFORE we
> release. We need to come to a consensus on how to release having
> moved our
> Bro Plugin to a separate repo. I don't think we've heard from
> everyone on
> this. I'd urge everyone to chime in so we can choose a path forward.
>
> If anyone is totally confused in regards to that discussion, I can try
> and
> send an options summary again as a separate discuss thread. The
> original
> chain was somewhere around here [1].
>
> [1]
> https://lists.apache.org/thread.html/54a4474881b97e559df24728b3a0e9
> 23a58345a282451085eef832ef@%3Cdev.metron.apache.org%3E
>
>
>
> On Wed, Nov 15, 2017 at 10:04 AM, Nick Allen <ni...@nickallen.org>
> wrote:
>
> > Hi Guys -
> >
> > I want to follow-up on this discussion. It sounds like most people
> are in
> > agreement with the general approach.
> >
> > A lot of people have been working hard on Metaalerts and
> Elasticsearch. I
> > have checked-in with those doing the heavy lifting and have compiled
> a more
> > detailed plan based on where we are at now. To the best of my
> knowledge
> > here is the plan of attack for finishing out this effort.
> >
> > (1) First, METRON-1289 needs to go in. This one was a fairly big
> effort
> > and I am hearing that we are pretty close.
> >
> > (2) METRON-1294 fixes an issue in how field types are looked-up.
> >
> > (3) METRON-1290 is next. While this may have been fixed in M-1289,
> > there may be some test cases we want from this PR.
> >
> > (4) METRON-1301 addresses a problem with the sorting logic.
> >
> > (5) METRON-1291 fixes an issue with escalation of metaalerts.
> >
> > (6) That leads us to Raghu's UI work in METRON-1252. This
> introduces
> > the UI bits that depend on all the previous backend work.
> >
> > (7) At this point, we should have our best effort at running
> Metaalerts
> > on Elasticsearch 2.x. I propose that we cut a release here.
> >
> > (8) After we cut the release, we can introduce the work for ES 5.x
> in
> > METRON-939. I know we will need lots of help testing and reviewing
> this
> > one.
> >
> > Please correct me if I am wrong. I will try and send out updates as
> we
> > make progress.
> >
> >
> >
> >
> >
> > On Mon, Nov 6, 2017 at 1:03 PM, Zeolla@GMail.com <ze...@gmail.com>
> wrote:
> >
> >> I agree, I think it's very reasonable to move in line with Nick's
> >> proposal. I would also suggest that we outline what the target
> versions
> >> would be to add in the METRON-777 components, since it has been
> functional
> >> for a very long time but not reviewed and has some really rockstar
> >> improvements.
> >>
> >> Jon
> >>
> >> On Mon, Nov 6, 2017 at 12:56 PM Otto Fowler <
> ottobackwards@gmail.com>
> >> wrote:
> >>
> >> > I think the ES cutover should be the start of the 0.5.x series,
> and we
> >> > continue on with 0.4.x for the
> >> > metadata improvements etc. We could chose to focus 0.5.x’s first
> >> releases
> >> > on not only ES but
> >> > getting a handle on kibana and the mpack situation as well.
> >> >
> >> >
> >> >
> >> >
> >> > On November 6, 2017 at 12:48:45, Michael Miklavcic (
> >> > michael.miklavcic@gmail.com) wrote:
> >> >
> >> > I agree with your proposal, Nick. I think having a stabilizing
> release
> >> > prior to upgrading ES/Kibana makes sense.
> >> >
> >> > On Mon, Nov 6, 2017 at 9:16 AM, Nick Allen <ni...@nickallen.org>
> wrote:
> >> >
> >> > > I would like to start a discussion around upcoming releases. We
> have a
> >> > > couple separate significant tracks of work that we need to
> reconcile
> >> in
> >> > our
> >> > > release schedule.
> >> > >
> >> > > (1) We have had (and have in review) a good number of bug fixes
> >> required
> >> > to
> >> > > support Metaalerts on the existing Elasticsearch 2.x
> infrastructure.
> >> > >
> >> > >
> >> > > (2) We also have ongoing work to upgrade our infrastructure to
> >> > > Elasticsearch 5.x, which will not be backwards compatible.
> >> > >
> >> > >
> >> > > I would like to see a release that has our best work on ES 2.x
> before
> >> we
> >> > > migrate to 5.x. I would propose the following.
> >> > >
> >> > > Release N+1: Introduce Metaalerts running on ES 2.x
> >> > >
> >> > > Release N+2: Cut-over to ES 5.x
> >> > >
> >> > >
> >> > > (Q) Is it worth cutting a separate release for ES 2.x? Is there
> a
> >> better
> >> > > way to handle the cut-over to 5.x?
> >> > >
> >> >
> >> --
> >>
> >> Jon
> >>
> >
> >
>
>
>
>
Re: [DISCUSS] Upcoming Release
Posted by "Zeolla@GMail.com" <ze...@gmail.com>.
So I was looking at some of the docs and saw that Upgrading.md has a
dangling section that maybe should be removed. Link
<https://github.com/apache/metron/blob/master/Upgrading.md#description>,
link <https://github.com/apache/metron/pull/780>
Jon
On Tue, Dec 12, 2017 at 2:21 PM Matt Foley <ma...@apache.org> wrote:
> Thanks to Jon, Otto, and Nick for looking over RC1. Tonight I’ll fix the
> RAT glitch, build RC2, and put it to formal vote.
> Regards,
> --Matt
>
> On 12/12/17, 5:14 AM, "Nick Allen" <ni...@nickallen.org> wrote:
>
> RC1 is looking good to me. I validated the MD5s, built Metron, built
> the
> Bro plugin and reviewed the other artifacts like release notes.
>
> Running the RAT check on a 'clean' Metron does not produce any errors
> for
> me. It is only after building Metron, which pulls in additional Node
> dependencies, does the RAT check fail.
>
>
> On Sun, Dec 10, 2017 at 4:41 PM Matt Foley <ma...@apache.org> wrote:
>
> > Yes, but let’s see if anyone else find other issues.
> >
> >
> >
> > From: Otto Fowler <ot...@gmail.com>
> > Date: Saturday, December 9, 2017 at 6:16 AM
> > To: Matt Foley <mf...@hortonworks.com>, "dev@metron.apache.org" <
> > dev@metron.apache.org>
> > Subject: Re: [DISCUSS] Upcoming Release
> >
> >
> >
> > So RC2 then?
> >
> >
> >
> > On December 8, 2017 at 20:43:21, Matt Foley (mfoley@hortonworks.com)
> > wrote:
> >
> > Hah, here it is: https://github.com/apache/metron/pull/743
> > “This problem seems to only reproduce when one unrolls a tarball
> rather
> > than cloning from github.”
> >
> > Heh, the exclusion at
> > https://github.com/apache/metron/blob/master/pom.xml#L351 is still
> there,
> > but the hashcode in the bundle.css file name has changed from
> > a0b6b99c10d9a13dc67e to f56deed131e58bd7ee04. Sigh. Did the version
> of Font
> > Awesome fonts change?
> >
> >
> > On 12/8/17, 5:26 PM, "Matt Foley" <ma...@apache.org> wrote:
> >
> > I remember having trouble with this bundle.css file on the last
> release,
> > but I can’t remember what we did about it. Anybody?
> >
> > On 12/8/17, 1:41 PM, "Otto Fowler" <ot...@gmail.com> wrote:
> >
> > Steps
> >
> > - Downloaded tar.gz’s, asc files and KEYS
> > - Verified signing of both tar.gz’s
> > - searched for rouge 0.4.1 entries
> > - verified the main pom.xml
> > - built :
> >
> > mvn clean && time mvn -q -T 2C -DskipTests install && time mvn -q -T
> > 2C surefire:test@unit-tests && time mvn -q
> > surefire:test@integration-tests && time mvn -q test --projects
> > metron-interface/metron-config && time build_utils/verify_licenses.sh
> >
> > Found rat error:
> >
> >
> > *****************************************************
> > Summary
> > -------
> > Generated at: 2017-12-08T16:33:27-05:00
> >
> > Notes: 3
> > Binaries: 193
> > Archives: 0
> > Standards: 75
> >
> > Apache Licensed: 74
> > Generated Documents: 0
> >
> > JavaDocs are generated, thus a license header is optional.
> > Generated files do not require license headers.
> >
> > 1 Unknown Licenses
> >
> > *****************************************************
> >
> > Files with unapproved licenses:
> >
> >
> >
> /Users/batman/tmp/release_ver/apache-metron-0.4.2-rc1/metron-interface/metron-alerts/dist/styles.f56deed131e58bd7ee04.bundle.css
> >
> > *****************************************************
> >
> >
> >
> >
> >
> > *****************************************************
> > Summary
> > -------
> > Generated at: 2017-12-08T16:33:27-05:00
> >
> > Notes: 3
> > Binaries: 193
> > Archives: 0
> > Standards: 75
> >
> > Apache Licensed: 74
> > Generated Documents: 0
> >
> > JavaDocs are generated, thus a license header is optional.
> > Generated files do not require license headers.
> >
> > 1 Unknown Licenses
> >
> > *****************************************************
> >
> > Files with unapproved licenses:
> >
> >
> >
> >
> /Users/ottofowler/tmp/release_ver/apache-metron-0.4.2-rc1/metron-interface/metron-alerts/dist/styles.f56deed131e58bd7ee04.bundle.css
> >
> > *****************************************************
> >
> >
> >
> > On December 8, 2017 at 04:34:24, Matt Foley (mattf@apache.org)
> wrote:
> >
> > Colleagues,
> > I’ve posted Metron-0.4.2-RC1 and Metron-bro-plugin-kafka-0.1 to
> > https://dist.apache.org/repos/dist/dev/metron/0.4.2-RC1/
> >
> > Given the complexity of this RC, I’d appreciate if a couple people
> would be
> > willing to kick the tires before we put it up for a vote.
> >
> > I will myself be going thru the Verify Build process this weekend,
> as I
> > won’t be able to do it Friday.
> >
> > Thanks,
> > --Matt
> >
> >
> > On 12/4/17, 2:05 PM, "Zeolla@GMail.com" <ze...@gmail.com> wrote:
> >
> > Can we resolve the conversation regarding the second repo? I was
> waiting
> > to get more input/preferences from people There's also a
> documentation
> > update that fixes a few broken Stellar docs that already has aa +1,
> I just
> > need to merge it.
> >
> > Jon
> >
> > On Mon, Dec 4, 2017, 17:01 Casey Stella <ce...@gmail.com> wrote:
> >
> > > I would be in favor of a release at this point.
> > >
> > > On Mon, Dec 4, 2017 at 4:57 PM, Matt Foley <ma...@apache.org>
> wrote:
> > >
> > > > Hey all,
> > > > I see METRON-1252 was resolved over the weekend. Shall I go
> ahead and
> > > > start the process with 0.4.2 release?
> > > > Does anyone have any commits they feel strongly should go in
> before
> > 0.4.2
> > > > is done, or are we ready to call it good?
> > > >
> > > > I believe there is consensus the 0.4.2 release should include a
> release
> > > of
> > > > the current state of the metron-bro-plugin-kafka. I will
> continue the
> > > > discussion in that thread as to the process for accomplishing
> that, but
> > > > plan on it happening.
> > > >
> > > > Regards,
> > > > --Matt
> > > >
> > > > On 11/26/17, 6:26 PM, "Matt Foley" <ma...@apache.org> wrote:
> > > >
> > > > Hope everyone (at least in the U.S.) had a great Thanksgiving
> > > holiday.
> > > > Regarding status of the release effort, still pending
> METRON-1252, so
> > > > not making the release branch yet.
> > > >
> > > > Regards,
> > > > --Matt
> > > >
> > > > On 11/17/17, 1:32 PM, "Matt Foley" <ma...@apache.org> wrote:
> > > >
> > > > (With release manager hat on)
> > > >
> > > > The community has proposed a release of Metron in the near
> > > future,
> > > > focusing on Meta-alerts running in Elasticsearch.
> > > > Congrats on getting so many of the below already done. At this
> > > > point, only METRON-1252, and the discussion of how to handle
> joint
> > > release
> > > > of the Metron bro plugin, remain as gating items for the
> release. I
> > > > project these will be resolved next week, so let’s propose the
> > following:
> > > >
> > > > Sometime next week, after the last bits are done, I’ll start the
> > > > release process and create the release branch.
> > > >
> > > > The proposed new version will be 0.4.2, unless there are backward
> > > > incompatible changes that support making it 0.5.0.
> > > > Currently there are NO included Jiras labeled
> > > > ‘backward-incompatible’, nor having Docs Text indicating so.
> > > > ***If anyone knows that some of the commits included since 0.4.1
> > > > introduce backward incompatibility, please say so now on this
> thread,
> > and
> > > > mark the Jira as such.***
> > > >
> > > > The 90 or so jiras/commits already in master branch since 0.4.1
> > > > are listed below.
> > > > Thanks,
> > > > --Matt
> > > >
> > > > METRON-1301 Alerts UI - Sorting on Triage Score Unexpectedly
> > > > Filters Some Records (nickwallen) closes apache/metron#832
> > > > METRON-1294 IP addresses are not formatted correctly in facet
> > > > and group results (merrimanr) closes apache/metron#827
> > > > METRON-1291 Kafka produce REST endpoint does not work in a
> > > > Kerberized cluster (merrimanr) closes apache/metron#826
> > > > METRON-1290 Only first 10 alerts are update when a MetaAlert
> > > > status is changed to inactive (justinleet) closes
> apache/metron#842
> > > > METRON-1311 Service Check Should Check Elasticsearch Index
> > > > Templates (nickwallen) closes apache/metron#839
> > > > METRON-1289 Alert fields are lost when a MetaAlert is created
> > > > (merrimanr) closes apache/metron#824
> > > > METRON-1309 Change metron-deployment to pull the plugin from
> > > > apache/metron-bro-plugin-kafka (JonZeolla) closes
> apache/metron#837
> > > > METRON-1310 Template Delete Action Deletes Search Indices
> > > > (nickwallen) closes apache/metron#838
> > > > METRON-1275: Fix Metron Documentation closes
> > > > apache/incubator-metron#833
> > > > METRON-1295 Unable to Configure Logging for REST API
> > > > (nickwallen) closes apache/metron#828
> > > > METRON-1307 Force install of java8 since java9 does not
> > > appear
> > > > to work with the scripts (brianhurley via ottobackwards) closes
> > > > apache/metron#835
> > > > METRON-1296 Full Dev Fails to Deploy Index Templates
> > > > (nickwallen via cestella) closes apache/incubator-metron#829
> > > > METRON-1281 Remove hard-coded indices from the Alerts UI
> > > > (merrimanr) closes apache/metron#821
> > > > METRON-1287 Full Dev Fails When Installing EPEL Repository
> > > > (nickwallen) closes apache/metron#820
> > > > METRON-1267 Alerts UI returns a 404 when refreshing the
> > > > alerts-list page (iraghumitra via merrimanr) closes
> apache/metron#819
> > > > METRON-1283 Install Elasticsearch template as a part of the
> > > > mpack startup scripts (anandsubbu via nickwallen) closes
> > > apache/metron#817
> > > > METRON-1254: Conditionals as map keys do not function in
> > > > Stellar closes apache/incubator-metron#801
> > > > METRON-1261 Apply bro security patch (JonZeolla via
> > > > ottobackwards) closes apache/metron#805
> > > > METRON-1284 Remove extraneous dead query in ElasticsearchDao
> > > > (justinleet) closes apache/metron#818
> > > > METRON-1270: fix for warnings missing @return tag argument in
> > > > metron-analytics/metron-profiler-common and
> metron-profiler-client
> > closes
> > > > apache/incubator-metron#810
> > > > METRON-1272 Hide child alerts from searches and grouping if
> > > > they belong to meta alerts (justinleet) closes apache/metron#811
> > > > METRON-1224 Add time range selection to search control
> > > > (iraghumitra via james-sirota) closes apache/metron#796
> > > > METRON-1280 0.4.1 -> 0.4.2 missed a couple of projects
> > > > (cestella via justinleet) closes apache/metron#816
> > > > METRON-1243: Add a REST endpoint which allows us to get a
> > > list
> > > > of all indice closes apache/incubator-metron#797
> > > > METRON-1196 Increment master version number to 0.4.2 for
> > > > on-going development (mattf-horton) closes apache/metron#767
> > > > METRON-1278 Strip "Build Status" widget from root
> > > > README.md in site-book build (mattf-horton) closes
> apache/metron#815
> > > > METRON-1274 Master has failure in
> > > > StormControllerIntegrationTest (merrimanr) closes
> apache/metron#813
> > > > METRON-1266 Profiler - SASL Authentication Failed
> > > (nickwallen)
> > > > closes apache/metron#809
> > > > METRON-1260 Include Alerts UI in Ambari Service Check
> > > > (nickwallen) closes apache/metron#804
> > > > METRON-1251: Typo and formatting fixes for metron-rest README
> > > > closes apache/incubator-metron#800
> > > > METRON-1241: Enable the REST API to use a cache for the
> > > > zookeeper config similar to the Bolts closes
> > apache/incubator-metron#795
> > > > METRON-1267 Alerts UI returns a 404 when refreshing the
> > > > alerts-list page (merrimanr) closes apache/metron#808
> > > > METRON-1262 Unable to add comment for a alert in a meta-alert
> > > > (merrimanr) closes apache/metron#806
> > > > METRON-1263 Start Alerts UI service after Metron REST
> > > > (anandsubbu via nickwallen) closes apache/metron#807
> > > > METRON-1255 MetaAlert search is not filtering on status
> > > > (merrimanr) closes apache/metron#802
> > > > METRON-1249 Improve Metron MPack Service Checks (nickwallen)
> > > > closes apache/metron#799
> > > > METRON-1237 address javadoc warnings in metron-maas-common
> > > > (dbist via james-sirota) closes apache/metron#792
> > > > METRON-1240 address javadoc warnings in metron-platform and
> > > > metron-analytics (dbist via james-sirota) closes
> apache/metron#794
> > > > METRON-1226 Searching Can Errantly Query the Wrong Indices
> > > > (nickwallen) closes apache/metron#793
> > > > METRON-1081 Fix Alerts and Ops UI Notices file (james-sirota)
> > > > closes apache/metron#682
> > > > METRON-1123 Add group by option using faceted search
> > > > capabilities of metron-rest-api (iraghumitra via james-sirota)
> closes
> > > > apache/metron#768
> > > > METRON-1223 Add support to add comments for alerts
> > > > (iraghumitra via james-sirota) closes apache/metron#788
> > > > METRON-1083 Add filters using faceted search capabilities of
> > > > metron-rest-api (iraghumitra via james-sirota) closes
> apache/metron#710
> > > > METRON-1232 Alert status changes are not reflected in list
> > > > view (iraghumitra via merrimanr) closes apache/metron#787
> > > > METRON-1247 REST search and findOne endpoints return
> > > > unexpected or incorrect results for guids (justinleet) closes
> > > > apache/metron#798
> > > > METRON-1235: Document the properties pulled from the global
> > > > configuration closes apache/incubator-metron#791
> > > > METRON-1234: fix for WARNING 'dependencies.dependency.(
> > > > groupId:artifactId:type:classifier)' must be unique:
> > > > org.apache.hadoop:hadoop-yarn-api:jar (dbist via mmiklavc) closes
> > > > apache/metron#790
> > > > METRON-1222: fix warning for The expression ${parent.version}
> > > > is deprecated. Please use ${project.parent.version} instead.
> (dbist via
> > > > mmiklavc) closes apache/metron#782
> > > > METRON-1220 Create documentation around alert nested field
> > > > (justinleet) closes apache/metron#780
> > > > METRON-1229 Management UI type is part of the declarations of
> > > > 2 modules (merrimanr) closes apache/metron#784
> > > > METRON-1228: Configuration Management PUSH immediately does
> > > > DUMP after (mmiklavc via mmiklavc) closes apache/metron#783
> > > > METRON-1218 Metron REST should return better error messages
> > > > (merrimanr) closes apache/metron#779
> > > > METRON-1161 Add ability to edit parser command line options
> > > in
> > > > the management UI (merrimanr) closes apache/metron#737
> > > > METRON-1209: Make stellar repl take logging properties, like
> > > > other CLI apps in metron closes apache/incubator-metron#772
> > > > METRON-1059 address checkstyle warning AvoidStarImport in
> > > > metron-stellar (dbist via ottobackwards) closes apache/metron#664
> > > > METRON-1204 UI does not time out after being idle, but stops
> > > > functioning (merrimanr) closes apache/metron#771
> > > > METRON-1052: Add forensic similarity hash functions to
> > > Stellar
> > > > closes apache/incubator-metron#781
> > > > METRON-632: Added validation of "shew.enrichmentType" and
> > > > "shew.keyColumns" closes apache/incubator-metron#732
> > > > METRON-1194 Add Profiler Debug Functions to Profiler README
> > > > (nickwallen via ottobackwards) closes apache/metron#765
> > > > METRON-1055 Metron 0.4.0 manual installation guide for CentOS
> > > > 6 updates (lvets via ottobackwards) closes apache/metron#661
> > > > METRON-1079 STELLAR NaN should be a keyword (ottobackwards)
> > > > closes apache/metron#681
> > > > METRON-1085 Add REST endpoint to save a user profile for the
> > > > Alerts UI (merrimanr) closes apache/metron#694
> > > > METRON-1208 MPack for Alerts UI (merrimanr) closes
> > > > apache/metron#778
> > > > METRON-1207 Make RPMs for Alerts UI (merrimanr) closes
> > > > apache/metron#777
> > > > METRON-1215 Fix link to RPMs chapter (DimDroll via
> > > justinleet)
> > > > closes apache/metron#776
> > > > METRON-1206 Make alerts UI conform to ops UI for install
> > > > (merrimanr) closes apache/metron#773
> > > > METRON-1195 Meta alerts improperly handle updates to
> > > non-alert
> > > > fields (justinleet) closes apache/metron#766
> > > > METRON-1189 Add alert escalation to the Alerts UI (merrimanr)
> > > > closes apache/metron#762
> > > > METRON-1156 Simulate Triage Rules in the Stellar REPL
> > > > (nickwallen) closes apache/metron#733
> > > > METRON-1198 Pycapa - No such configuration property
> > > > 'sasl.kerberos.principal' (nickwallen) closes apache/metron#769
> > > > METRON-1202 ElasticsearchDao Has extraneous sleep call
> > > > (justinleet) closes apache/metron#770
> > > > METRON-938 "service metron-rest start <password>" does not
> > > > work on CentOS 7. (justinleet) closes apache/metron#757
> > > > METRON-1182 Refactor Code in alert list to accommodate new
> > > > view types (iraghumitra via merrimanr) closes apache/metron#756
> > > > METRON-1188: Ambari global configuration management
> > > (mmiklavc)
> > > > closes apache/metron#760
> > > > METRON-1191 update public web site to point at 0.4.1 new
> > > > release (mattf-horton) closes apache/metron#764
> > > > METRON-1063 address javadoc warnings in metron-stellar (dbist
> > > > via ottobackwards) closes apache/metron#668
> > > > METRON-1190 Fix Meta Alert Type handling in calculation of
> > > > scores (justinleet) closes apache/metron#763
> > > > METRON-1187 Indexing/Profiler Kafka ACL Groups Not Setup
> > > > Correctly (nickwallen) closes apache/metron#759
> > > > METRON-1185: Stellar REPL does not work on a kerberized
> > > > cluster when calling functions interacting with HBase closes
> > > > apache/incubator-metron#755
> > > > METRON-1186: Profiler Functions use classutils from shaded
> > > > storm closes apache/incubator-metron#758
> > > > METRON-1173: Fix pointers to old stellar docs closes
> > > > apache/incubator-metron#746
> > > > METRON-1179: Make STATS_ADD to take a list closes
> > > > apache/incubator-metron#750
> > > > METRON-1180: Make Stellar Shell accept zookeeper quorum as a
> > > > CSV list and not require a port closes
> apache/incubator-metron#751
> > > > METRON-1183 Improve KDC Setup Instructions (nickwallen)
> > > closes
> > > > apache/metron#753
> > > > METRON-1177 Stale running topologies seen post-kerberization
> > > > and cause exceptions (nickwallen) closes apache/metron#748
> > > > METRON-1158 Build backend for grouping alerts into meta
> > > alerts
> > > > (justinleet) closes apache/metron#734
> > > > METRON-1146: Add ability to parse JSON string into JSONObject
> > > > for stellar closes apache/incubator-metron#727
> > > > METRON-1176 REST: HDFS Service should support setting
> > > > permissions on files when writing (ottobackwards) closes
> > > apache/metron#749
> > > > METRON-1114 Add group by capabilities to search REST endpoint
> > > > (merrimanr) closes apache/metron#702
> > > > METRON-1167 Define Session Specific Global Configuration
> > > > Values in the REPL (nickwallen) closes apache/metron#740
> > > > METRON-1171: Better validation for the SUBSTRING stellar
> > > > function closes apache/incubator-metron#745
> > > >
> > > >
> > > >
> > > > On 11/17/17, 11:59 AM, "Nick Allen" <ni...@nickallen.org> wrote:
> > > >
> > > > I just wanted to send an update on where we are at. We've
> > > > gotten a lot
> > > > done here recently as you can see below.
> > > >
> > > > ✓ DONE (1) First, METRON-1289 needs to go in. This one was
> > > > a fairly big
> > > > effort and I am hearing that we are pretty close.
> > > >
> > > > ✓ DONE (2) METRON-1294 fixes an issue in how field types
> > > are
> > > > looked-up.
> > > >
> > > > ✓ DONE (3) METRON-1290 is next. While this may have been
> > > > fixed in
> > > > M-1289, there may be some test cases we want from this PR.
> > > >
> > > > ✓ DONE (4) METRON-1301 addresses a problem with the sorting
> > > > logic.
> > > >
> > > > ✓ DONE (5) METRON-1291 fixes an issue with escalation of
> > > > metaalerts.
> > > >
> > > > (6) That leads us to Raghu's UI work in METRON-1252. This
> > > > introduces the
> > > > UI bits that depend on all the previous backend work.
> > > >
> > > > (7) At this point, we should have our best effort at
> > > running
> > > > Metaalerts
> > > > on Elasticsearch 2.x. I propose that we cut a release here.
> > > >
> > > > (8) After we cut the release, we can introduce the work for
> > > > ES 5.x in
> > > > METRON-939. I know we will need lots of help testing and
> > > > reviewing this
> > > > one.
> > > >
> > > >
> > > >
> > > > We also have an outstanding question that needs resolved
> > > > BEFORE we
> > > > release. We need to come to a consensus on how to release
> > > > having moved our
> > > > Bro Plugin to a separate repo. I don't think we've heard
> > > from
> > > > everyone on
> > > > this. I'd urge everyone to chime in so we can choose a path
> > > > forward.
> > > >
> > > > If anyone is totally confused in regards to that discussion,
> > > I
> > > > can try and
> > > > send an options summary again as a separate discuss thread.
> > > > The original
> > > > chain was somewhere around here [1].
> > > >
> > > > [1]
> > > > https://lists.apache.org/thread.html/
> > > > 54a4474881b97e559df24728b3a0e923a58345a282451085eef832ef@%
> > > > 3Cdev.metron.apache.org%3E
> > > >
> > > >
> > > >
> > > > On Wed, Nov 15, 2017 at 10:04 AM, Nick Allen <
> > > > nick@nickallen.org> wrote:
> > > >
> > > > > Hi Guys -
> > > > >
> > > > > I want to follow-up on this discussion. It sounds like
> > > most
> > > > people are in
> > > > > agreement with the general approach.
> > > > >
> > > > > A lot of people have been working hard on Metaalerts and
> > > > Elasticsearch. I
> > > > > have checked-in with those doing the heavy lifting and have
> > > > compiled a more
> > > > > detailed plan based on where we are at now. To the best of
> > > > my knowledge
> > > > > here is the plan of attack for finishing out this effort.
> > > > >
> > > > > (1) First, METRON-1289 needs to go in. This one was a
> > > > fairly big effort
> > > > > and I am hearing that we are pretty close.
> > > > >
> > > > > (2) METRON-1294 fixes an issue in how field types are
> > > > looked-up.
> > > > >
> > > > > (3) METRON-1290 is next. While this may have been fixed
> > > > in M-1289,
> > > > > there may be some test cases we want from this PR.
> > > > >
> > > > > (4) METRON-1301 addresses a problem with the sorting
> > > logic.
> > > > >
> > > > > (5) METRON-1291 fixes an issue with escalation of
> > > > metaalerts.
> > > > >
> > > > > (6) That leads us to Raghu's UI work in METRON-1252.
> > > This
> > > > introduces
> > > > > the UI bits that depend on all the previous backend work.
> > > > >
> > > > > (7) At this point, we should have our best effort at
> > > > running Metaalerts
> > > > > on Elasticsearch 2.x. I propose that we cut a release here.
> > > > >
> > > > > (8) After we cut the release, we can introduce the work
> > > > for ES 5.x in
> > > > > METRON-939. I know we will need lots of help testing and
> > > > reviewing this
> > > > > one.
> > > > >
> > > > > Please correct me if I am wrong. I will try and send out
> > > > updates as we
> > > > > make progress.
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > On Mon, Nov 6, 2017 at 1:03 PM, Zeolla@GMail.com <
> > > > zeolla@gmail.com> wrote:
> > > > >
> > > > >> I agree, I think it's very reasonable to move in line with
> > > > Nick's
> > > > >> proposal. I would also suggest that we outline what the
> > > > target versions
> > > > >> would be to add in the METRON-777 components, since it has
> > > > been functional
> > > > >> for a very long time but not reviewed and has some really
> > > > rockstar
> > > > >> improvements.
> > > > >>
> > > > >> Jon
> > > > >>
> > > > >> On Mon, Nov 6, 2017 at 12:56 PM Otto Fowler <
> > > > ottobackwards@gmail.com>
> > > > >> wrote:
> > > > >>
> > > > >> > I think the ES cutover should be the start of the 0.5.x
> > > > series, and we
> > > > >> > continue on with 0.4.x for the
> > > > >> > metadata improvements etc. We could chose to focus
> > > > 0.5.x’s first
> > > > >> releases
> > > > >> > on not only ES but
> > > > >> > getting a handle on kibana and the mpack situation as
> > > > well.
> > > > >> >
> > > > >> >
> > > > >> >
> > > > >> >
> > > > >> > On November 6, 2017 at 12:48:45, Michael Miklavcic (
> > > > >> > michael.miklavcic@gmail.com) wrote:
> > > > >> >
> > > > >> > I agree with your proposal, Nick. I think having a
> > > > stabilizing release
> > > > >> > prior to upgrading ES/Kibana makes sense.
> > > > >> >
> > > > >> > On Mon, Nov 6, 2017 at 9:16 AM, Nick Allen <
> > > > nick@nickallen.org> wrote:
> > > > >> >
> > > > >> > > I would like to start a discussion around upcoming
> > > > releases. We have a
> > > > >> > > couple separate significant tracks of work that we
> > > need
> > > > to reconcile
> > > > >> in
> > > > >> > our
> > > > >> > > release schedule.
> > > > >> > >
> > > > >> > > (1) We have had (and have in review) a good number of
> > > > bug fixes
> > > > >> required
> > > > >> > to
> > > > >> > > support Metaalerts on the existing Elasticsearch 2.x
> > > > infrastructure.
> > > > >> > >
> > > > >> > >
> > > > >> > > (2) We also have ongoing work to upgrade our
> > > > infrastructure to
> > > > >> > > Elasticsearch 5.x, which will not be backwards
> > > > compatible.
> > > > >> > >
> > > > >> > >
> > > > >> > > I would like to see a release that has our best work
> > > on
> > > > ES 2.x before
> > > > >> we
> > > > >> > > migrate to 5.x. I would propose the following.
> > > > >> > >
> > > > >> > > Release N+1: Introduce Metaalerts running on ES 2.x
> > > > >> > >
> > > > >> > > Release N+2: Cut-over to ES 5.x
> > > > >> > >
> > > > >> > >
> > > > >> > > (Q) Is it worth cutting a separate release for ES 2.x?
> > > > Is there a
> > > > >> better
> > > > >> > > way to handle the cut-over to 5.x?
> > > > >> > >
> > > > >> >
> > > > >> --
> > > > >>
> > > > >> Jon
> > > > >>
> > > > >
> > > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > >
> > --
> >
> > Jon
> >
> >
> >
> >
> >
> >
>
>
>
> --
Jon
Re: [DISCUSS] Upcoming Release
Posted by Otto Fowler <ot...@gmail.com>.
Once we have the area, I can do the same for the RC check script
On December 18, 2017 at 11:11:18, Nick Allen (nick@nickallen.org) wrote:
Sure, I can clean up the script a bit and submit a PR for it.
Jon and Otto asked that I open a PR on the script that I use for merging
PRs too.
On Fri, Dec 15, 2017 at 2:30 PM, Matt Foley <mf...@hortonworks.com> wrote:
> Perhaps under “build_utils” we should add a subdirectory for
> “release_utils”.
>
> From: Casey Stella <ce...@gmail.com>
> Date: Friday, December 15, 2017 at 10:50 AM
> To: "dev@metron.apache.org" <de...@metron.apache.org>
> Cc: Matt Foley <mf...@hortonworks.com>
> Subject: Re: [DISCUSS] Upcoming Release
>
> That script seems great, nick! Perhaps we should adjust the wiki around
> releases to point to it? Thoughts?
>
> On Fri, Dec 15, 2017 at 1:47 PM, Nick Allen <nick@nickallen.org<mailto:nic
> k@nickallen.org>> wrote:
> Thanks, Matt.
>
> Maybe you already have something that does this. I wrote a quick script
> that validates each JIRA since the last release tag to make sure they are
> marked "Done" and with the correct fix version. I would expect that for
> the next release, each JIRA should have status="Done",
fix-version="0.4.2".
>
> Unless I am mistaken, we have quite a few that need cleaned up. In the
> following output, any line that has a URL indicates that a fix is needed.
> Or it least, **I think** a fix is needed.
>
> To the community.... If you see your name with a URL next to it, it would
> be great if you could follow that link and fix the JIRA. Otherwise, I
will
> volunteer to help clean some of these up should some not get addressed.
>
>
> *$ ./validate-jira-for-release*
> *Cloning into 'metron-0.4.2'...*
> *remote: Counting objects: 35046, done.*
> *remote: Compressing objects: 100% (13698/13698), done.*
> *remote: Total 35046 (delta 15708), reused 31645 (delta 12822)*
> *Receiving objects: 100% (35046/35046), 53.05 MiB | 6.48 MiB/s, done.*
> *Resolving deltas: 100% (15708/15708), done.*
> *Fetching origin*
> * JIRA STATUS FIX VERSION
> ASSIGNEE FIX*
> * METRON-1345 Done Michael
> Miklavcic https://issues.apache.org/jira/browse/METRON-1345
> <https://issues.apache.org/jira/browse/METRON-1345>*
> * METRON-1349 Done Next + 1 Nick
> Allen https://issues.apache.org/jira/browse/METRON-1349
> <https://issues.apache.org/jira/browse/METRON-1349>*
> * METRON-1343 Done
> Mohan https://issues.apache.org/jira/browse/METRON-1343
> <https://issues.apache.org/jira/browse/METRON-1343>*
> * METRON-1306 To Do
> Unassigned https://issues.apache.org/jira/browse/METRON-1306
> <https://issues.apache.org/jira/browse/METRON-1306>*
> * METRON-1341 Done Simon Elliston
> Ball https://issues.apache.org/jira/browse/METRON-1341
> <https://issues.apache.org/jira/browse/METRON-1341>*
> * METRON-1313 Done Jon
> Zeolla https://issues.apache.org/jira/browse/METRON-1313
> <https://issues.apache.org/jira/browse/METRON-1313>*
> * METRON-1346 Done Otto
> Fowler https://issues.apache.org/jira/browse/METRON-1346
> <https://issues.apache.org/jira/browse/METRON-1346>*
> * METRON-1336 Done 0.4.2 Nick
> Allen*
> * METRON-1335 Done Anand
> Subramanian https://issues.apache.org/jira/browse/METRON-1335
> <https://issues.apache.org/jira/browse/METRON-1335>*
> * METRON-1308 Done Jon
> Zeolla https://issues.apache.org/jira/browse/METRON-1308
> <https://issues.apache.org/jira/browse/METRON-1308>*
> * METRON-1338 Done 0.4.2 Nick
> Allen*
> * METRON-1286 To Do 0.4.2
> Unassigned https://issues.apache.org/jira/browse/METRON-1286
> <https://issues.apache.org/jira/browse/METRON-1286>*
> * METRON-1334 Done 0.4.2 Nick
> Allen*
> * METRON-1277 Done Otto
> Fowler https://issues.apache.org/jira/browse/METRON-1277
> <https://issues.apache.org/jira/browse/METRON-1277>*
> * METRON-1239 To Do
> Unassigned https://issues.apache.org/jira/browse/METRON-1239
> <https://issues.apache.org/jira/browse/METRON-1239>*
> * METRON-1328 Done Anand
> Subramanian https://issues.apache.org/jira/browse/METRON-1328
> <https://issues.apache.org/jira/browse/METRON-1328>*
> * METRON-1333 Done Otto
> Fowler https://issues.apache.org/jira/browse/METRON-1333
> <https://issues.apache.org/jira/browse/METRON-1333>*
> * METRON-1252 Done
> RaghuMitra https://issues.apache.org/jira/browse/METRON-1252
> <https://issues.apache.org/jira/browse/METRON-1252>*
> * METRON-1316 To Do Next + 1
> Unassigned https://issues.apache.org/jira/browse/METRON-1316
> <https://issues.apache.org/jira/browse/METRON-1316>*
> * METRON-1088 Done Jon
> Zeolla https://issues.apache.org/jira/browse/METRON-1088
> <https://issues.apache.org/jira/browse/METRON-1088>*
> * METRON-1319 To Do Ryan
> Merriman https://issues.apache.org/jira/browse/METRON-1319
> <https://issues.apache.org/jira/browse/METRON-1319>*
> * METRON-1321 To Do
> Unassigned https://issues.apache.org/jira/browse/METRON-1321
> <https://issues.apache.org/jira/browse/METRON-1321>*
> * METRON-1301 Done 0.4.2 Nick
> Allen*
> * METRON-1294 To Do Ryan
> Merriman https://issues.apache.org/jira/browse/METRON-1294
> <https://issues.apache.org/jira/browse/METRON-1294>*
> * METRON-1291 To Do Ryan
> Merriman https://issues.apache.org/jira/browse/METRON-1291
> <https://issues.apache.org/jira/browse/METRON-1291>*
> * METRON-1290 Done Justin
> Leet https://issues.apache.org/jira/browse/METRON-1290
> <https://issues.apache.org/jira/browse/METRON-1290>*
> * METRON-1311 Done 0.4.2 Nick
> Allen*
> * METRON-1289 To Do Ryan
> Merriman https://issues.apache.org/jira/browse/METRON-1289
> <https://issues.apache.org/jira/browse/METRON-1289>*
> * METRON-1309 Done Jon
> Zeolla https://issues.apache.org/jira/browse/METRON-1309
> <https://issues.apache.org/jira/browse/METRON-1309>*
> * METRON-1310 Done 0.4.2 Nick
> Allen*
> * METRON-1275 Done Jon
> Zeolla https://issues.apache.org/jira/browse/METRON-1275
> <https://issues.apache.org/jira/browse/METRON-1275>*
> * METRON-1295 Done 0.4.2 Nick
> Allen*
> * METRON-1307 Done Otto
> Fowler https://issues.apache.org/jira/browse/METRON-1307
> <https://issues.apache.org/jira/browse/METRON-1307>*
> * METRON-1296 Done 0.4.2 Nick
> Allen*
> * METRON-1281 To Do Ryan
> Merriman https://issues.apache.org/jira/browse/METRON-1281
> <https://issues.apache.org/jira/browse/METRON-1281>*
> * METRON-1287 Done 0.4.2 Nick
> Allen*
> * METRON-1267 To Do
> Unassigned https://issues.apache.org/jira/browse/METRON-1267
> <https://issues.apache.org/jira/browse/METRON-1267>*
> * METRON-1283 Done Anand
> Subramanian https://issues.apache.org/jira/browse/METRON-1283
> <https://issues.apache.org/jira/browse/METRON-1283>*
> * METRON-1254 To Do
> Unassigned https://issues.apache.org/jira/browse/METRON-1254
> <https://issues.apache.org/jira/browse/METRON-1254>*
> * METRON-1261 Done Jon
> Zeolla https://issues.apache.org/jira/browse/METRON-1261
> <https://issues.apache.org/jira/browse/METRON-1261>*
> * METRON-1284 Done Justin
> Leet https://issues.apache.org/jira/browse/METRON-1284
> <https://issues.apache.org/jira/browse/METRON-1284>*
> * METRON-1270 Done Artem
> Ervits https://issues.apache.org/jira/browse/METRON-1270
> <https://issues.apache.org/jira/browse/METRON-1270>*
> * METRON-1272 In Progress Justin
> Leet https://issues.apache.org/jira/browse/METRON-1272
> <https://issues.apache.org/jira/browse/METRON-1272>*
> * METRON-1224 Done
> RaghuMitra https://issues.apache.org/jira/browse/METRON-1224
> <https://issues.apache.org/jira/browse/METRON-1224>*
> * METRON-1280 To Do
> Unassigned https://issues.apache.org/jira/browse/METRON-1280
> <https://issues.apache.org/jira/browse/METRON-1280>*
> * METRON-1243 To Do
> Unassigned https://issues.apache.org/jira/browse/METRON-1243
> <https://issues.apache.org/jira/browse/METRON-1243>*
> * METRON-1196 Done Matt
> Foley https://issues.apache.org/jira/browse/METRON-1196
> <https://issues.apache.org/jira/browse/METRON-1196>*
> * METRON-1278 Done 0.4.2 Matt
> Foley*
> * METRON-1274 To Do
> Unassigned https://issues.apache.org/jira/browse/METRON-1274
> <https://issues.apache.org/jira/browse/METRON-1274>*
> * METRON-1266 Done 0.4.2 Nick
> Allen*
> * METRON-1260 Done 0.4.2 Nick
> Allen*
> * METRON-1251 Done Jon
> Zeolla https://issues.apache.org/jira/browse/METRON-1251
> <https://issues.apache.org/jira/browse/METRON-1251>*
> * METRON-1241 To Do
> Unassigned https://issues.apache.org/jira/browse/METRON-1241
> <https://issues.apache.org/jira/browse/METRON-1241>*
> * METRON-1267 To Do
> Unassigned https://issues.apache.org/jira/browse/METRON-1267
> <https://issues.apache.org/jira/browse/METRON-1267>*
> * METRON-1262 To Do Ryan
> Merriman https://issues.apache.org/jira/browse/METRON-1262
> <https://issues.apache.org/jira/browse/METRON-1262>*
> * METRON-1263 Done Anand
> Subramanian https://issues.apache.org/jira/browse/METRON-1263
> <https://issues.apache.org/jira/browse/METRON-1263>*
> * METRON-1255 To Do
> Unassigned https://issues.apache.org/jira/browse/METRON-1255
> <https://issues.apache.org/jira/browse/METRON-1255>*
> * METRON-1249 Done 0.4.1 Nick
> Allen https://issues.apache.org/jira/browse/METRON-1249
> <https://issues.apache.org/jira/browse/METRON-1249>*
> * METRON-1237 To Do Artem
> Ervits https://issues.apache.org/jira/browse/METRON-1237
> <https://issues.apache.org/jira/browse/METRON-1237>*
> * METRON-1240 Done Artem
> Ervits https://issues.apache.org/jira/browse/METRON-1240
> <https://issues.apache.org/jira/browse/METRON-1240>*
> * METRON-1226 Done 0.4.2 Nick
> Allen*
> * METRON-1081 Done James
> Sirota https://issues.apache.org/jira/browse/METRON-1081
> <https://issues.apache.org/jira/browse/METRON-1081>*
> * METRON-1123 Done
> RaghuMitra https://issues.apache.org/jira/browse/METRON-1123
> <https://issues.apache.org/jira/browse/METRON-1123>*
> * METRON-1223 Done
> RaghuMitra https://issues.apache.org/jira/browse/METRON-1223
> <https://issues.apache.org/jira/browse/METRON-1223>*
> * METRON-1083 Done
> RaghuMitra https://issues.apache.org/jira/browse/METRON-1083
> <https://issues.apache.org/jira/browse/METRON-1083>*
> * METRON-1232 Done
> RaghuMitra https://issues.apache.org/jira/browse/METRON-1232
> <https://issues.apache.org/jira/browse/METRON-1232>*
> * METRON-1247 Done Justin
> Leet https://issues.apache.org/jira/browse/METRON-1247
> <https://issues.apache.org/jira/browse/METRON-1247>*
> * METRON-1235 To Do
> Unassigned https://issues.apache.org/jira/browse/METRON-1235
> <https://issues.apache.org/jira/browse/METRON-1235>*
> * METRON-1234 Done Artem
> Ervits https://issues.apache.org/jira/browse/METRON-1234
> <https://issues.apache.org/jira/browse/METRON-1234>*
> * METRON-1222 Done Artem
> Ervits https://issues.apache.org/jira/browse/METRON-1222
> <https://issues.apache.org/jira/browse/METRON-1222>*
> * METRON-1220 In Progress Justin
> Leet https://issues.apache.org/jira/browse/METRON-1220
> <https://issues.apache.org/jira/browse/METRON-1220>*
> * METRON-1229 To Do
> Unassigned https://issues.apache.org/jira/browse/METRON-1229
> <https://issues.apache.org/jira/browse/METRON-1229>*
> * METRON-1228 Done
> Unassigned https://issues.apache.org/jira/browse/METRON-1228
> <https://issues.apache.org/jira/browse/METRON-1228>*
> * METRON-1218 To Do Ryan
> Merriman https://issues.apache.org/jira/browse/METRON-1218
> <https://issues.apache.org/jira/browse/METRON-1218>*
> * METRON-1161 In Progress Ryan
> Merriman https://issues.apache.org/jira/browse/METRON-1161
> <https://issues.apache.org/jira/browse/METRON-1161>*
> * METRON-1209 To Do
> Unassigned https://issues.apache.org/jira/browse/METRON-1209
> <https://issues.apache.org/jira/browse/METRON-1209>*
> * METRON-1059 Done Next + 1
> Unassigned https://issues.apache.org/jira/browse/METRON-1059
> <https://issues.apache.org/jira/browse/METRON-1059>*
> * METRON-1204 To Do Ryan
> Merriman https://issues.apache.org/jira/browse/METRON-1204
> <https://issues.apache.org/jira/browse/METRON-1204>*
> * METRON-1052 Done Casey
> Stella https://issues.apache.org/jira/browse/METRON-1052
> <https://issues.apache.org/jira/browse/METRON-1052>*
> * METRON-632 In Progress Tomas
> Zezula https://issues.apache.org/jira/browse/METRON-632
> <https://issues.apache.org/jira/browse/METRON-632>*
> * METRON-1194 Done 0.4.2 Nick
> Allen*
> * METRON-1055 To Do Laurens
> Vets https://issues.apache.org/jira/browse/METRON-1055
> <https://issues.apache.org/jira/browse/METRON-1055>*
> * METRON-1079 Done Otto
> Fowler https://issues.apache.org/jira/browse/METRON-1079
> <https://issues.apache.org/jira/browse/METRON-1079>*
> * METRON-1085 To Do Ryan
> Merriman https://issues.apache.org/jira/browse/METRON-1085
> <https://issues.apache.org/jira/browse/METRON-1085>*
> * METRON-1208 To Do Ryan
> Merriman https://issues.apache.org/jira/browse/METRON-1208
> <https://issues.apache.org/jira/browse/METRON-1208>*
> * METRON-1207 To Do Ryan
> Merriman https://issues.apache.org/jira/browse/METRON-1207
> <https://issues.apache.org/jira/browse/METRON-1207>*
> * METRON-1215 To Do
> Unassigned https://issues.apache.org/jira/browse/METRON-1215
> <https://issues.apache.org/jira/browse/METRON-1215>*
> * METRON-1206 To Do Ryan
> Merriman https://issues.apache.org/jira/browse/METRON-1206
> <https://issues.apache.org/jira/browse/METRON-1206>*
> * METRON-1195 To Do Justin
> Leet https://issues.apache.org/jira/browse/METRON-1195
> <https://issues.apache.org/jira/browse/METRON-1195>*
> * METRON-1189 To Do Ryan
> Merriman https://issues.apache.org/jira/browse/METRON-1189
> <https://issues.apache.org/jira/browse/METRON-1189>*
> * METRON-1156 Done 0.4.2 Nick
> Allen*
> * METRON-1198 Done 0.4.2 Nick
> Allen*
> * METRON-1202 Done Next + 1 Justin
> Leet https://issues.apache.org/jira/browse/METRON-1202
> <https://issues.apache.org/jira/browse/METRON-1202>*
> * METRON-938 Done Next + 1 Justin
> Leet https://issues.apache.org/jira/browse/METRON-938
> <https://issues.apache.org/jira/browse/METRON-938>*
> * METRON-1182 Done
> RaghuMitra https://issues.apache.org/jira/browse/METRON-1182
> <https://issues.apache.org/jira/browse/METRON-1182>*
> * METRON-1188 Done Michael
> Miklavcic https://issues.apache.org/jira/browse/METRON-1188
> <https://issues.apache.org/jira/browse/METRON-1188>*
> * METRON-1191 Done Matt
> Foley https://issues.apache.org/jira/browse/METRON-1191
> <https://issues.apache.org/jira/browse/METRON-1191>*
> * METRON-1063 Done Next + 1 Artem
> Ervits https://issues.apache.org/jira/browse/METRON-1063
> <https://issues.apache.org/jira/browse/METRON-1063>*
> * METRON-1190 In Progress Justin
> Leet https://issues.apache.org/jira/browse/METRON-1190
> <https://issues.apache.org/jira/browse/METRON-1190>*
> * METRON-1187 Done 0.4.2 Nick
> Allen*
> * METRON-1185 To Do
> Unassigned https://issues.apache.org/jira/browse/METRON-1185
> <https://issues.apache.org/jira/browse/METRON-1185>*
> * METRON-1186 To Do Casey
> Stella https://issues.apache.org/jira/browse/METRON-1186
> <https://issues.apache.org/jira/browse/METRON-1186>*
> * METRON-1173 Done Next + 1 Jon
> Zeolla https://issues.apache.org/jira/browse/METRON-1173
> <https://issues.apache.org/jira/browse/METRON-1173>*
> * METRON-1179 To Do
> Unassigned https://issues.apache.org/jira/browse/METRON-1179
> <https://issues.apache.org/jira/browse/METRON-1179>*
> * METRON-1180 To Do
> Unassigned https://issues.apache.org/jira/browse/METRON-1180
> <https://issues.apache.org/jira/browse/METRON-1180>*
> * METRON-1183 Done 0.4.2 Nick
> Allen*
> * METRON-1177 Done 0.4.2 Nick
> Allen*
> * METRON-1158 To Do Justin
> Leet https://issues.apache.org/jira/browse/METRON-1158
> <https://issues.apache.org/jira/browse/METRON-1158>*
> * METRON-1146 Done Anand
> Subramanian https://issues.apache.org/jira/browse/METRON-1146
> <https://issues.apache.org/jira/browse/METRON-1146>*
> * METRON-1176 Done Otto
> Fowler https://issues.apache.org/jira/browse/METRON-1176
> <https://issues.apache.org/jira/browse/METRON-1176>*
> * METRON-1114 Done Ryan
> Merriman https://issues.apache.org/jira/browse/METRON-1114
> <https://issues.apache.org/jira/browse/METRON-1114>*
> * METRON-1167 Done Next + 1 Nick
> Allen https://issues.apache.org/jira/browse/METRON-1167
> <https://issues.apache.org/jira/browse/METRON-1167>*
> * METRON-1171 To Do Casey
> Stella https://issues.apache.org/jira/browse/METRON-1171
> <https://issues.apache.org/jira/browse/METRON-1171>*
>
>
>
> The script is here [1] if anyone wants to run it themselves and grep for
> their own name.
>
> [1]
> https://github.com/nickwallen/metron-commit-stuff/blob/
> master/validate-jira-for-release
>
> On Fri, Dec 15, 2017 at 1:18 PM, Matt Foley <mfoley@hortonworks.com<
> mailto:mfoley@hortonworks.com>> wrote:
>
> > Hi Nick,
> > Good timing, you’ve saved me some work! :-)
> >
> > Origin of the list: The approach was defined before I started managing
> > releases, but I think it’s a reasonable approach. It’s specified in our
> > Release Process document, https://cwiki.apache.org/
> > confluence/display/METRON/Release+Process , Step 5, the last bullet:
> > “The artifacts for a release or RC [include]… A CHANGES file
> > denoting the changes [since the last release].
> > We usually construct this by taking the output of git log | grep
> > METRON | sed 's/\[//g' | sed 's/\]//g' | grep -v "http" and removing
the
> > JIRAs from the previous releases (it’s in time sorted order so this is
> > easy).”
> >
> > However, the release manager also has a responsibility to “Monitor and
> > Verify Jiras” (Step 2 and various other places in the doc). Altho the
> doc
> > isn’t explicit about it, I take this responsibility a little farther
and
> do
> > the following as part of Jira verification:
> >
> > 1. Extract the jira id’s from the CHANGES file with a simple awk
script,
> > put them in a Jira query, and confirm they are all actually marked
> > “Fixed/Resolved” with Fix Version = <the current release> in Jira. If
> > there are exceptions, I dun the contributors to fix it. You’ve seen
> these
> > emails from me in the past couple releases. I don’t just mark them
fixed
> > myself, because some contributions (perfectly legitimately) only
> partially
> > address a problem, and just marking a ticket “Fixed” may not be the
right
> > thing to do.
> >
> > 2. Query jira for any tickets NOT included in the prior list, that ARE
> > marked fixed in the current release (or later, or “Next”, or similar
> > “future” version constructs). These don’t usually happen (considering
> that
> > Metron contributors mostly live inside github PR processes rather than
> > Jiras) but when they do they also need to be resolved. Cases I’ve seen
> (in
> > Metron and other projects) include:
> > a) If they were indeed fixed in the current release, but not reflected
in
> > the commit message, I’ll annotate this fact in the Jira ticket, and
> > hand-edit the CHANGES file for this release. (Doesn’t seem worth trying
> to
> > edit old commit messages, since that mucks up the SHA1’s.)
> > b) If they were fixed in a previous release but mis-marked as to Fix
> > Version in the ticket, fix the ticket.
> > c) If they aren’t really fixed, fix the ticket.
> >
> > 3. Once the set of Fixed tickets is complete and correct, query them
for
> > any that have “labels in (backward-incompatible,
backwards-compatibility,
> > backwards-incompatible)” -- yes, there shouldn’t be 3 labels, but it’s
> hard
> > to fix because Jira never forgets, so every time someone gets it wrong,
> it
> > adds to the set; fortunately, query completion helps here – OR "Docs
> Text"
> > is not EMPTY. Each of these needs to be looked at carefully, verified,
> and
> > if Doc Text is missing, the contributor and I cooperate to write a
couple
> > sentences. The Doc Text then needs to go in the RELEASE_NOTES for the
> > release, in the section “## Non-backward-compatible Changes in this
> > Release, and Upgrade Suggestions”.
> >
> > That’s about it. I guess I should add the above as an appendix doc to
> the
> > Release Process documentation. I’ll do that.
> > Cheers,
> > --Matt
> >
> >
> > On 12/15/17, 9:15 AM, "Nick Allen" <nick@nickallen.org<mailto:nic
> k@nickallen.org>> wrote:
> >
> > Hi Matt -
> >
> > I just updated like 15+ JIRAs of my own JIRAs that I completed and
> > merged,
> > but failed to mark as resolved. All of these will be included in
> > 0.4.2. I
> > updated each to be "fixed" in version 0.4.2 and marked as resolved.
> > Hopefully the next RC will report those as fixed.
> >
> > (Q) Where does the list of JIRAs that get attached to the release
> > originate
> > from? Does it get pulled out of JIRA or do they come from the commit
> > log?
> >
> > My apologies for not staying on top of my JIRAs.
> >
> >
> >
> >
> > On Tue, Dec 12, 2017 at 2:21 PM, Matt Foley <mattf@apache.org
> <ma...@apache.org>> wrote:
> >
> > > Thanks to Jon, Otto, and Nick for looking over RC1. Tonight I’ll
> > fix the
> > > RAT glitch, build RC2, and put it to formal vote.
> > > Regards,
> > > --Matt
> > >
> > > On 12/12/17, 5:14 AM, "Nick Allen" <nick@nickallen.org<mailto:nic
> k@nickallen.org>> wrote:
> > >
> > > RC1 is looking good to me. I validated the MD5s, built Metron,
> > built
> > > the
> > > Bro plugin and reviewed the other artifacts like release notes.
> > >
> > > Running the RAT check on a 'clean' Metron does not produce any
> > errors
> > > for
> > > me. It is only after building Metron, which pulls in
> additional
> > Node
> > > dependencies, does the RAT check fail.
> > >
> > >
> > > On Sun, Dec 10, 2017 at 4:41 PM Matt Foley <mattf@apache.org
> <ma...@apache.org>>
> > wrote:
> > >
> > > > Yes, but let’s see if anyone else find other issues.
> > > >
> > > >
> > > >
> > > > From: Otto Fowler <ottobackwards@gmail.com<mailto:
> ottobackwards@gmail.com>>
> > > > Date: Saturday, December 9, 2017 at 6:16 AM
> > > > To: Matt Foley <mfoley@hortonworks.com<mailto:
> mfoley@hortonworks.com>>, "
> > dev@metron.apache.org<ma...@metron.apache.org>" <
> > > > dev@metron.apache.org<ma...@metron.apache.org>>
> > > > Subject: Re: [DISCUSS] Upcoming Release
> > > >
> > > >
> > > >
> > > > So RC2 then?
> > > >
> > > >
> > > >
> > > > On December 8, 2017 at 20:43:21, Matt Foley (
> > mfoley@hortonworks.com<ma...@hortonworks.com>)
> > > > wrote:
> > > >
> > > > Hah, here it is: https://github.com/apache/metron/pull/743
> > > > “This problem seems to only reproduce when one unrolls a
> > tarball
> > > rather
> > > > than cloning from github.”
> > > >
> > > > Heh, the exclusion at
> > > > https://github.com/apache/metron/blob/master/pom.xml#L351 is
> > still
> > > there,
> > > > but the hashcode in the bundle.css file name has changed from
> > > > a0b6b99c10d9a13dc67e to f56deed131e58bd7ee04. Sigh. Did the
> > version
> > > of Font
> > > > Awesome fonts change?
> > > >
> > > >
> > > > On 12/8/17, 5:26 PM, "Matt Foley" <mattf@apache.org<mailto:
> mattf@apache.org>> wrote:
> > > >
> > > > I remember having trouble with this bundle.css file on the
> last
> > > release,
> > > > but I can’t remember what we did about it. Anybody?
> > > >
> > > > On 12/8/17, 1:41 PM, "Otto Fowler" <ottobackwards@gmail.com<
> mailto:ottobackwards@gmail.com>>
> > wrote:
> > > >
> > > > Steps
> > > >
> > > > - Downloaded tar.gz’s, asc files and KEYS
> > > > - Verified signing of both tar.gz’s
> > > > - searched for rouge 0.4.1 entries
> > > > - verified the main pom.xml
> > > > - built :
> > > >
> > > > mvn clean && time mvn -q -T 2C -DskipTests install && time
> mvn
> > -q -T
> > > > 2C surefire:test@unit-tests && time mvn -q
> > > > surefire:test@integration-tests && time mvn -q test
> --projects
> > > > metron-interface/metron-config && time
> > build_utils/verify_licenses.sh
> > > >
> > > > Found rat error:
> > > >
> > > >
> > > > *****************************************************
> > > > Summary
> > > > -------
> > > > Generated at: 2017-12-08T16:33:27-05:00
> > > >
> > > > Notes: 3
> > > > Binaries: 193
> > > > Archives: 0
> > > > Standards: 75
> > > >
> > > > Apache Licensed: 74
> > > > Generated Documents: 0
> > > >
> > > > JavaDocs are generated, thus a license header is optional.
> > > > Generated files do not require license headers.
> > > >
> > > > 1 Unknown Licenses
> > > >
> > > > *****************************************************
> > > >
> > > > Files with unapproved licenses:
> > > >
> > > >
> > > > /Users/batman/tmp/release_ver/apache-metron-0.4.2-rc1/
> > > metron-interface/metron-alerts/dist/styles.
> > f56deed131e58bd7ee04.bundle.css
> > > >
> > > > *****************************************************
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > *****************************************************
> > > > Summary
> > > > -------
> > > > Generated at: 2017-12-08T16:33:27-05:00
> > > >
> > > > Notes: 3
> > > > Binaries: 193
> > > > Archives: 0
> > > > Standards: 75
> > > >
> > > > Apache Licensed: 74
> > > > Generated Documents: 0
> > > >
> > > > JavaDocs are generated, thus a license header is optional.
> > > > Generated files do not require license headers.
> > > >
> > > > 1 Unknown Licenses
> > > >
> > > > *****************************************************
> > > >
> > > > Files with unapproved licenses:
> > > >
> > > >
> > > >
> > > > /Users/ottofowler/tmp/release_ver/apache-metron-0.4.2-rc1/
> > > metron-interface/metron-alerts/dist/styles.
> > f56deed131e58bd7ee04.bundle.css
> > > >
> > > > *****************************************************
> > > >
> > > >
> > > >
> > > > On December 8, 2017 at 04:34:24, Matt Foley (
> mattf@apache.org<ma...@apache.org>)
> > > wrote:
> > > >
> > > > Colleagues,
> > > > I’ve posted Metron-0.4.2-RC1 and Metron-bro-plugin-kafka-0.1
> to
> > > > https://dist.apache.org/repos/dist/dev/metron/0.4.2-RC1/
> > > >
> > > > Given the complexity of this RC, I’d appreciate if a couple
> > people
> > > would be
> > > > willing to kick the tires before we put it up for a vote.
> > > >
> > > > I will myself be going thru the Verify Build process this
> > weekend,
> > > as I
> > > > won’t be able to do it Friday.
> > > >
> > > > Thanks,
> > > > --Matt
> > > >
> > > >
> > > > On 12/4/17, 2:05 PM, "Zeolla@GMail.com" <zeolla@gmail.com
> <ma...@gmail.com>>
> > wrote:
> > > >
> > > > Can we resolve the conversation regarding the second repo? I
> > was
> > > waiting
> > > > to get more input/preferences from people There's also a
> > > documentation
> > > > update that fixes a few broken Stellar docs that already has
> > aa +1,
> > > I just
> > > > need to merge it.
> > > >
> > > > Jon
> > > >
> > > > On Mon, Dec 4, 2017, 17:01 Casey Stella <cestella@gmail.com
> <ma...@gmail.com>>
> > wrote:
> > > >
> > > > > I would be in favor of a release at this point.
> > > > >
> > > > > On Mon, Dec 4, 2017 at 4:57 PM, Matt Foley <
> mattf@apache.org<ma...@apache.org>
> > >
> > > wrote:
> > > > >
> > > > > > Hey all,
> > > > > > I see METRON-1252 was resolved over the weekend. Shall I
> go
> > > ahead and
> > > > > > start the process with 0.4.2 release?
> > > > > > Does anyone have any commits they feel strongly should go
> > in
> > > before
> > > > 0.4.2
> > > > > > is done, or are we ready to call it good?
> > > > > >
> > > > > > I believe there is consensus the 0.4.2 release should
> > include a
> > > release
> > > > > of
> > > > > > the current state of the metron-bro-plugin-kafka. I will
> > > continue the
> > > > > > discussion in that thread as to the process for
> > accomplishing
> > > that, but
> > > > > > plan on it happening.
> > > > > >
> > > > > > Regards,
> > > > > > --Matt
> > > > > >
> > > > > > On 11/26/17, 6:26 PM, "Matt Foley" <mattf@apache.org
> <ma...@apache.org>>
> > wrote:
> > > > > >
> > > > > > Hope everyone (at least in the U.S.) had a great
> > Thanksgiving
> > > > > holiday.
> > > > > > Regarding status of the release effort, still pending
> > > METRON-1252, so
> > > > > > not making the release branch yet.
> > > > > >
> > > > > > Regards,
> > > > > > --Matt
> > > > > >
> > > > > > On 11/17/17, 1:32 PM, "Matt Foley" <mattf@apache.org
> <ma...@apache.org>>
> > wrote:
> > > > > >
> > > > > > (With release manager hat on)
> > > > > >
> > > > > > The community has proposed a release of Metron in the
> near
> > > > > future,
> > > > > > focusing on Meta-alerts running in Elasticsearch.
> > > > > > Congrats on getting so many of the below already done. At
> > this
> > > > > > point, only METRON-1252, and the discussion of how to
> > handle
> > > joint
> > > > > release
> > > > > > of the Metron bro plugin, remain as gating items for the
> > > release. I
> > > > > > project these will be resolved next week, so let’s
> propose
> > the
> > > > following:
> > > > > >
> > > > > > Sometime next week, after the last bits are done, I’ll
> > start the
> > > > > > release process and create the release branch.
> > > > > >
> > > > > > The proposed new version will be 0.4.2, unless there are
> > backward
> > > > > > incompatible changes that support making it 0.5.0.
> > > > > > Currently there are NO included Jiras labeled
> > > > > > ‘backward-incompatible’, nor having Docs Text indicating
> > so.
> > > > > > ***If anyone knows that some of the commits included
> since
> > 0.4.1
> > > > > > introduce backward incompatibility, please say so now on
> > this
> > > thread,
> > > > and
> > > > > > mark the Jira as such.***
> > > > > >
> > > > > > The 90 or so jiras/commits already in master branch since
> > 0.4.1
> > > > > > are listed below.
> > > > > > Thanks,
> > > > > > --Matt
> > > > > >
> > > > > > METRON-1301 Alerts UI - Sorting on Triage Score
> > Unexpectedly
> > > > > > Filters Some Records (nickwallen) closes
> apache/metron#832
> > > > > > METRON-1294 IP addresses are not formatted correctly in
> > facet
> > > > > > and group results (merrimanr) closes apache/metron#827
> > > > > > METRON-1291 Kafka produce REST endpoint does not work in
> a
> > > > > > Kerberized cluster (merrimanr) closes apache/metron#826
> > > > > > METRON-1290 Only first 10 alerts are update when a
> > MetaAlert
> > > > > > status is changed to inactive (justinleet) closes
> > > apache/metron#842
> > > > > > METRON-1311 Service Check Should Check Elasticsearch
> Index
> > > > > > Templates (nickwallen) closes apache/metron#839
> > > > > > METRON-1289 Alert fields are lost when a MetaAlert is
> > created
> > > > > > (merrimanr) closes apache/metron#824
> > > > > > METRON-1309 Change metron-deployment to pull the plugin
> > from
> > > > > > apache/metron-bro-plugin-kafka (JonZeolla) closes
> > > apache/metron#837
> > > > > > METRON-1310 Template Delete Action Deletes Search Indices
> > > > > > (nickwallen) closes apache/metron#838
> > > > > > METRON-1275: Fix Metron Documentation closes
> > > > > > apache/incubator-metron#833
> > > > > > METRON-1295 Unable to Configure Logging for REST API
> > > > > > (nickwallen) closes apache/metron#828
> > > > > > METRON-1307 Force install of java8 since java9 does not
> > > > > appear
> > > > > > to work with the scripts (brianhurley via ottobackwards)
> > closes
> > > > > > apache/metron#835
> > > > > > METRON-1296 Full Dev Fails to Deploy Index Templates
> > > > > > (nickwallen via cestella) closes
> > apache/incubator-metron#829
> > > > > > METRON-1281 Remove hard-coded indices from the Alerts UI
> > > > > > (merrimanr) closes apache/metron#821
> > > > > > METRON-1287 Full Dev Fails When Installing EPEL
> Repository
> > > > > > (nickwallen) closes apache/metron#820
> > > > > > METRON-1267 Alerts UI returns a 404 when refreshing the
> > > > > > alerts-list page (iraghumitra via merrimanr) closes
> > > apache/metron#819
> > > > > > METRON-1283 Install Elasticsearch template as a part of
> the
> > > > > > mpack startup scripts (anandsubbu via nickwallen) closes
> > > > > apache/metron#817
> > > > > > METRON-1254: Conditionals as map keys do not function in
> > > > > > Stellar closes apache/incubator-metron#801
> > > > > > METRON-1261 Apply bro security patch (JonZeolla via
> > > > > > ottobackwards) closes apache/metron#805
> > > > > > METRON-1284 Remove extraneous dead query in
> > ElasticsearchDao
> > > > > > (justinleet) closes apache/metron#818
> > > > > > METRON-1270: fix for warnings missing @return tag
> argument
> > in
> > > > > > metron-analytics/metron-profiler-common and
> > > metron-profiler-client
> > > > closes
> > > > > > apache/incubator-metron#810
> > > > > > METRON-1272 Hide child alerts from searches and grouping
> if
> > > > > > they belong to meta alerts (justinleet) closes
> > apache/metron#811
> > > > > > METRON-1224 Add time range selection to search control
> > > > > > (iraghumitra via james-sirota) closes apache/metron#796
> > > > > > METRON-1280 0.4.1 -> 0.4.2 missed a couple of projects
> > > > > > (cestella via justinleet) closes apache/metron#816
> > > > > > METRON-1243: Add a REST endpoint which allows us to get a
> > > > > list
> > > > > > of all indice closes apache/incubator-metron#797
> > > > > > METRON-1196 Increment master version number to 0.4.2 for
> > > > > > on-going development (mattf-horton) closes
> > apache/metron#767
> > > > > > METRON-1278 Strip "Build Status" widget from
> root
> > > > > > README.md in site-book build (mattf-horton) closes
> > > apache/metron#815
> > > > > > METRON-1274 Master has failure in
> > > > > > StormControllerIntegrationTest (merrimanr) closes
> > > apache/metron#813
> > > > > > METRON-1266 Profiler - SASL Authentication Failed
> > > > > (nickwallen)
> > > > > > closes apache/metron#809
> > > > > > METRON-1260 Include Alerts UI in Ambari Service Check
> > > > > > (nickwallen) closes apache/metron#804
> > > > > > METRON-1251: Typo and formatting fixes for metron-rest
> > README
> > > > > > closes apache/incubator-metron#800
> > > > > > METRON-1241: Enable the REST API to use a cache for the
> > > > > > zookeeper config similar to the Bolts closes
> > > > apache/incubator-metron#795
> > > > > > METRON-1267 Alerts UI returns a 404 when refreshing the
> > > > > > alerts-list page (merrimanr) closes apache/metron#808
> > > > > > METRON-1262 Unable to add comment for a alert in a
> > meta-alert
> > > > > > (merrimanr) closes apache/metron#806
> > > > > > METRON-1263 Start Alerts UI service after Metron REST
> > > > > > (anandsubbu via nickwallen) closes apache/metron#807
> > > > > > METRON-1255 MetaAlert search is not filtering on status
> > > > > > (merrimanr) closes apache/metron#802
> > > > > > METRON-1249 Improve Metron MPack Service Checks
> > (nickwallen)
> > > > > > closes apache/metron#799
> > > > > > METRON-1237 address javadoc warnings in
> metron-maas-common
> > > > > > (dbist via james-sirota) closes apache/metron#792
> > > > > > METRON-1240 address javadoc warnings in metron-platform
> and
> > > > > > metron-analytics (dbist via james-sirota) closes
> > > apache/metron#794
> > > > > > METRON-1226 Searching Can Errantly Query the Wrong
> Indices
> > > > > > (nickwallen) closes apache/metron#793
> > > > > > METRON-1081 Fix Alerts and Ops UI Notices file
> > (james-sirota)
> > > > > > closes apache/metron#682
> > > > > > METRON-1123 Add group by option using faceted search
> > > > > > capabilities of metron-rest-api (iraghumitra via
> > james-sirota)
> > > closes
> > > > > > apache/metron#768
> > > > > > METRON-1223 Add support to add comments for alerts
> > > > > > (iraghumitra via james-sirota) closes apache/metron#788
> > > > > > METRON-1083 Add filters using faceted search capabilities
> > of
> > > > > > metron-rest-api (iraghumitra via james-sirota) closes
> > > apache/metron#710
> > > > > > METRON-1232 Alert status changes are not reflected in
> list
> > > > > > view (iraghumitra via merrimanr) closes apache/metron#787
> > > > > > METRON-1247 REST search and findOne endpoints return
> > > > > > unexpected or incorrect results for guids (justinleet)
> > closes
> > > > > > apache/metron#798
> > > > > > METRON-1235: Document the properties pulled from the
> global
> > > > > > configuration closes apache/incubator-metron#791
> > > > > > METRON-1234: fix for WARNING 'dependencies.dependency.(
> > > > > > groupId:artifactId:type:classifier)' must be unique:
> > > > > > org.apache.hadoop:hadoop-yarn-api:jar (dbist via
> mmiklavc)
> > > closes
> > > > > > apache/metron#790
> > > > > > METRON-1222: fix warning for The expression
> > ${parent.version}
> > > > > > is deprecated. Please use ${project.parent.version}
> > instead.
> > > (dbist via
> > > > > > mmiklavc) closes apache/metron#782
> > > > > > METRON-1220 Create documentation around alert nested
> field
> > > > > > (justinleet) closes apache/metron#780
> > > > > > METRON-1229 Management UI type is part of the
> declarations
> > of
> > > > > > 2 modules (merrimanr) closes apache/metron#784
> > > > > > METRON-1228: Configuration Management PUSH immediately
> does
> > > > > > DUMP after (mmiklavc via mmiklavc) closes
> apache/metron#783
> > > > > > METRON-1218 Metron REST should return better error
> messages
> > > > > > (merrimanr) closes apache/metron#779
> > > > > > METRON-1161 Add ability to edit parser command line
> options
> > > > > in
> > > > > > the management UI (merrimanr) closes apache/metron#737
> > > > > > METRON-1209: Make stellar repl take logging properties,
> > like
> > > > > > other CLI apps in metron closes
> apache/incubator-metron#772
> > > > > > METRON-1059 address checkstyle warning AvoidStarImport in
> > > > > > metron-stellar (dbist via ottobackwards) closes
> > apache/metron#664
> > > > > > METRON-1204 UI does not time out after being idle, but
> > stops
> > > > > > functioning (merrimanr) closes apache/metron#771
> > > > > > METRON-1052: Add forensic similarity hash functions to
> > > > > Stellar
> > > > > > closes apache/incubator-metron#781
> > > > > > METRON-632: Added validation of "shew.enrichmentType" and
> > > > > > "shew.keyColumns" closes apache/incubator-metron#732
> > > > > > METRON-1194 Add Profiler Debug Functions to Profiler
> README
> > > > > > (nickwallen via ottobackwards) closes apache/metron#765
> > > > > > METRON-1055 Metron 0.4.0 manual installation guide for
> > CentOS
> > > > > > 6 updates (lvets via ottobackwards) closes
> > apache/metron#661
> > > > > > METRON-1079 STELLAR NaN should be a keyword
> (ottobackwards)
> > > > > > closes apache/metron#681
> > > > > > METRON-1085 Add REST endpoint to save a user profile for
> > the
> > > > > > Alerts UI (merrimanr) closes apache/metron#694
> > > > > > METRON-1208 MPack for Alerts UI (merrimanr) closes
> > > > > > apache/metron#778
> > > > > > METRON-1207 Make RPMs for Alerts UI (merrimanr) closes
> > > > > > apache/metron#777
> > > > > > METRON-1215 Fix link to RPMs chapter (DimDroll via
> > > > > justinleet)
> > > > > > closes apache/metron#776
> > > > > > METRON-1206 Make alerts UI conform to ops UI for install
> > > > > > (merrimanr) closes apache/metron#773
> > > > > > METRON-1195 Meta alerts improperly handle updates to
> > > > > non-alert
> > > > > > fields (justinleet) closes apache/metron#766
> > > > > > METRON-1189 Add alert escalation to the Alerts UI
> > (merrimanr)
> > > > > > closes apache/metron#762
> > > > > > METRON-1156 Simulate Triage Rules in the Stellar REPL
> > > > > > (nickwallen) closes apache/metron#733
> > > > > > METRON-1198 Pycapa - No such configuration property
> > > > > > 'sasl.kerberos.principal' (nickwallen) closes
> > apache/metron#769
> > > > > > METRON-1202 ElasticsearchDao Has extraneous sleep call
> > > > > > (justinleet) closes apache/metron#770
> > > > > > METRON-938 "service metron-rest start <password>" does
> not
> > > > > > work on CentOS 7. (justinleet) closes apache/metron#757
> > > > > > METRON-1182 Refactor Code in alert list to accommodate
> new
> > > > > > view types (iraghumitra via merrimanr) closes
> > apache/metron#756
> > > > > > METRON-1188: Ambari global configuration management
> > > > > (mmiklavc)
> > > > > > closes apache/metron#760
> > > > > > METRON-1191 update public web site to point at 0.4.1 new
> > > > > > release (mattf-horton) closes apache/metron#764
> > > > > > METRON-1063 address javadoc warnings in metron-stellar
> > (dbist
> > > > > > via ottobackwards) closes apache/metron#668
> > > > > > METRON-1190 Fix Meta Alert Type handling in calculation
> of
> > > > > > scores (justinleet) closes apache/metron#763
> > > > > > METRON-1187 Indexing/Profiler Kafka ACL Groups Not Setup
> > > > > > Correctly (nickwallen) closes apache/metron#759
> > > > > > METRON-1185: Stellar REPL does not work on a kerberized
> > > > > > cluster when calling functions interacting with HBase
> > closes
> > > > > > apache/incubator-metron#755
> > > > > > METRON-1186: Profiler Functions use classutils from
> shaded
> > > > > > storm closes apache/incubator-metron#758
> > > > > > METRON-1173: Fix pointers to old stellar docs closes
> > > > > > apache/incubator-metron#746
> > > > > > METRON-1179: Make STATS_ADD to take a list closes
> > > > > > apache/incubator-metron#750
> > > > > > METRON-1180: Make Stellar Shell accept zookeeper quorum
> as
> > a
> > > > > > CSV list and not require a port closes
> > > apache/incubator-metron#751
> > > > > > METRON-1183 Improve KDC Setup Instructions (nickwallen)
> > > > > closes
> > > > > > apache/metron#753
> > > > > > METRON-1177 Stale running topologies seen
> > post-kerberization
> > > > > > and cause exceptions (nickwallen) closes
> apache/metron#748
> > > > > > METRON-1158 Build backend for grouping alerts into meta
> > > > > alerts
> > > > > > (justinleet) closes apache/metron#734
> > > > > > METRON-1146: Add ability to parse JSON string into
> > JSONObject
> > > > > > for stellar closes apache/incubator-metron#727
> > > > > > METRON-1176 REST: HDFS Service should support setting
> > > > > > permissions on files when writing (ottobackwards) closes
> > > > > apache/metron#749
> > > > > > METRON-1114 Add group by capabilities to search REST
> > endpoint
> > > > > > (merrimanr) closes apache/metron#702
> > > > > > METRON-1167 Define Session Specific Global Configuration
> > > > > > Values in the REPL (nickwallen) closes apache/metron#740
> > > > > > METRON-1171: Better validation for the SUBSTRING stellar
> > > > > > function closes apache/incubator-metron#745
> > > > > >
> > > > > >
> > > > > >
> > > > > > On 11/17/17, 11:59 AM, "Nick Allen" <nick@nickallen.org
> <ma...@nickallen.org>>
> > wrote:
> > > > > >
> > > > > > I just wanted to send an update on where we are at. We've
> > > > > > gotten a lot
> > > > > > done here recently as you can see below.
> > > > > >
> > > > > > ✓ DONE (1) First, METRON-1289 needs to go in. This one
> was
> > > > > > a fairly big
> > > > > > effort and I am hearing that we are pretty close.
> > > > > >
> > > > > > ✓ DONE (2) METRON-1294 fixes an issue in how field types
> > > > > are
> > > > > > looked-up.
> > > > > >
> > > > > > ✓ DONE (3) METRON-1290 is next. While this may have been
> > > > > > fixed in
> > > > > > M-1289, there may be some test cases we want from this
> PR.
> > > > > >
> > > > > > ✓ DONE (4) METRON-1301 addresses a problem with the
> sorting
> > > > > > logic.
> > > > > >
> > > > > > ✓ DONE (5) METRON-1291 fixes an issue with escalation of
> > > > > > metaalerts.
> > > > > >
> > > > > > (6) That leads us to Raghu's UI work in METRON-1252. This
> > > > > > introduces the
> > > > > > UI bits that depend on all the previous backend work.
> > > > > >
> > > > > > (7) At this point, we should have our best effort at
> > > > > running
> > > > > > Metaalerts
> > > > > > on Elasticsearch 2.x. I propose that we cut a release
> here.
> > > > > >
> > > > > > (8) After we cut the release, we can introduce the work
> for
> > > > > > ES 5.x in
> > > > > > METRON-939. I know we will need lots of help testing and
> > > > > > reviewing this
> > > > > > one.
> > > > > >
> > > > > >
> > > > > >
> > > > > > We also have an outstanding question that needs resolved
> > > > > > BEFORE we
> > > > > > release. We need to come to a consensus on how to release
> > > > > > having moved our
> > > > > > Bro Plugin to a separate repo. I don't think we've heard
> > > > > from
> > > > > > everyone on
> > > > > > this. I'd urge everyone to chime in so we can choose a
> path
> > > > > > forward.
> > > > > >
> > > > > > If anyone is totally confused in regards to that
> > discussion,
> > > > > I
> > > > > > can try and
> > > > > > send an options summary again as a separate discuss
> thread.
> > > > > > The original
> > > > > > chain was somewhere around here [1].
> > > > > >
> > > > > > [1]
> > > > > > https://lists.apache.org/thread.html/
> > > > > > 54a4474881b97e559df24728b3a0e9
> 23a58345a282451085eef832ef@%
> > > > > > 3Cdev.metron.apache.org<http://3Cdev.metron.apache.org
> >%3E
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Wed, Nov 15, 2017 at 10:04 AM, Nick Allen <
> > > > > > nick@nickallen.org<ma...@nickallen.org>> wrote:
> > > > > >
> > > > > > > Hi Guys -
> > > > > > >
> > > > > > > I want to follow-up on this discussion. It sounds like
> > > > > most
> > > > > > people are in
> > > > > > > agreement with the general approach.
> > > > > > >
> > > > > > > A lot of people have been working hard on Metaalerts
> and
> > > > > > Elasticsearch. I
> > > > > > > have checked-in with those doing the heavy lifting and
> > have
> > > > > > compiled a more
> > > > > > > detailed plan based on where we are at now. To the best
> > of
> > > > > > my knowledge
> > > > > > > here is the plan of attack for finishing out this
> effort.
> > > > > > >
> > > > > > > (1) First, METRON-1289 needs to go in. This one was a
> > > > > > fairly big effort
> > > > > > > and I am hearing that we are pretty close.
> > > > > > >
> > > > > > > (2) METRON-1294 fixes an issue in how field types are
> > > > > > looked-up.
> > > > > > >
> > > > > > > (3) METRON-1290 is next. While this may have been fixed
> > > > > > in M-1289,
> > > > > > > there may be some test cases we want from this PR.
> > > > > > >
> > > > > > > (4) METRON-1301 addresses a problem with the sorting
> > > > > logic.
> > > > > > >
> > > > > > > (5) METRON-1291 fixes an issue with escalation of
> > > > > > metaalerts.
> > > > > > >
> > > > > > > (6) That leads us to Raghu's UI work in METRON-1252.
> > > > > This
> > > > > > introduces
> > > > > > > the UI bits that depend on all the previous backend
> work.
> > > > > > >
> > > > > > > (7) At this point, we should have our best effort at
> > > > > > running Metaalerts
> > > > > > > on Elasticsearch 2.x. I propose that we cut a release
> > here.
> > > > > > >
> > > > > > > (8) After we cut the release, we can introduce the work
> > > > > > for ES 5.x in
> > > > > > > METRON-939. I know we will need lots of help testing
> and
> > > > > > reviewing this
> > > > > > > one.
> > > > > > >
> > > > > > > Please correct me if I am wrong. I will try and send
> out
> > > > > > updates as we
> > > > > > > make progress.
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > On Mon, Nov 6, 2017 at 1:03 PM, Zeolla@GMail.com <
> > > > > > zeolla@gmail.com<ma...@gmail.com>> wrote:
> > > > > > >
> > > > > > >> I agree, I think it's very reasonable to move in line
> > with
> > > > > > Nick's
> > > > > > >> proposal. I would also suggest that we outline what
> the
> > > > > > target versions
> > > > > > >> would be to add in the METRON-777 components, since it
> > has
> > > > > > been functional
> > > > > > >> for a very long time but not reviewed and has some
> > really
> > > > > > rockstar
> > > > > > >> improvements.
> > > > > > >>
> > > > > > >> Jon
> > > > > > >>
> > > > > > >> On Mon, Nov 6, 2017 at 12:56 PM Otto Fowler <
> > > > > > ottobackwards@gmail.com<ma...@gmail.com>>
> > > > > > >> wrote:
> > > > > > >>
> > > > > > >> > I think the ES cutover should be the start of the
> > 0.5.x
> > > > > > series, and we
> > > > > > >> > continue on with 0.4.x for the
> > > > > > >> > metadata improvements etc. We could chose to focus
> > > > > > 0.5.x’s first
> > > > > > >> releases
> > > > > > >> > on not only ES but
> > > > > > >> > getting a handle on kibana and the mpack situation
> as
> > > > > > well.
> > > > > > >> >
> > > > > > >> >
> > > > > > >> >
> > > > > > >> >
> > > > > > >> > On November 6, 2017 at 12:48:45, Michael Miklavcic (
> > > > > > >> > michael.miklavcic@gmail.com<mailto:
> michael.miklavcic@gmail.com>) wrote:
> > > > > > >> >
> > > > > > >> > I agree with your proposal, Nick. I think having a
> > > > > > stabilizing release
> > > > > > >> > prior to upgrading ES/Kibana makes sense.
> > > > > > >> >
> > > > > > >> > On Mon, Nov 6, 2017 at 9:16 AM, Nick Allen <
> > > > > > nick@nickallen.org<ma...@nickallen.org>> wrote:
> > > > > > >> >
> > > > > > >> > > I would like to start a discussion around upcoming
> > > > > > releases. We have a
> > > > > > >> > > couple separate significant tracks of work that we
> > > > > need
> > > > > > to reconcile
> > > > > > >> in
> > > > > > >> > our
> > > > > > >> > > release schedule.
> > > > > > >> > >
> > > > > > >> > > (1) We have had (and have in review) a good number
> > of
> > > > > > bug fixes
> > > > > > >> required
> > > > > > >> > to
> > > > > > >> > > support Metaalerts on the existing Elasticsearch
> 2.x
> > > > > > infrastructure.
> > > > > > >> > >
> > > > > > >> > >
> > > > > > >> > > (2) We also have ongoing work to upgrade our
> > > > > > infrastructure to
> > > > > > >> > > Elasticsearch 5.x, which will not be backwards
> > > > > > compatible.
> > > > > > >> > >
> > > > > > >> > >
> > > > > > >> > > I would like to see a release that has our best
> work
> > > > > on
> > > > > > ES 2.x before
> > > > > > >> we
> > > > > > >> > > migrate to 5.x. I would propose the following.
> > > > > > >> > >
> > > > > > >> > > Release N+1: Introduce Metaalerts running on ES
> 2.x
> > > > > > >> > >
> > > > > > >> > > Release N+2: Cut-over to ES 5.x
> > > > > > >> > >
> > > > > > >> > >
> > > > > > >> > > (Q) Is it worth cutting a separate release for ES
> > 2.x?
> > > > > > Is there a
> > > > > > >> better
> > > > > > >> > > way to handle the cut-over to 5.x?
> > > > > > >> > >
> > > > > > >> >
> > > > > > >> --
> > > > > > >>
> > > > > > >> Jon
> > > > > > >>
> > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > >
> > > > --
> > > >
> > > > Jon
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > >
> > >
> > >
> > >
> >
> >
> >
>
>
Re: [DISCUSS] Upcoming Release
Posted by Nick Allen <ni...@nickallen.org>.
Sure, I can clean up the script a bit and submit a PR for it.
Jon and Otto asked that I open a PR on the script that I use for merging
PRs too.
On Fri, Dec 15, 2017 at 2:30 PM, Matt Foley <mf...@hortonworks.com> wrote:
> Perhaps under “build_utils” we should add a subdirectory for
> “release_utils”.
>
> From: Casey Stella <ce...@gmail.com>
> Date: Friday, December 15, 2017 at 10:50 AM
> To: "dev@metron.apache.org" <de...@metron.apache.org>
> Cc: Matt Foley <mf...@hortonworks.com>
> Subject: Re: [DISCUSS] Upcoming Release
>
> That script seems great, nick! Perhaps we should adjust the wiki around
> releases to point to it? Thoughts?
>
> On Fri, Dec 15, 2017 at 1:47 PM, Nick Allen <nick@nickallen.org<mailto:nic
> k@nickallen.org>> wrote:
> Thanks, Matt.
>
> Maybe you already have something that does this. I wrote a quick script
> that validates each JIRA since the last release tag to make sure they are
> marked "Done" and with the correct fix version. I would expect that for
> the next release, each JIRA should have status="Done", fix-version="0.4.2".
>
> Unless I am mistaken, we have quite a few that need cleaned up. In the
> following output, any line that has a URL indicates that a fix is needed.
> Or it least, **I think** a fix is needed.
>
> To the community.... If you see your name with a URL next to it, it would
> be great if you could follow that link and fix the JIRA. Otherwise, I will
> volunteer to help clean some of these up should some not get addressed.
>
>
> *$ ./validate-jira-for-release*
> *Cloning into 'metron-0.4.2'...*
> *remote: Counting objects: 35046, done.*
> *remote: Compressing objects: 100% (13698/13698), done.*
> *remote: Total 35046 (delta 15708), reused 31645 (delta 12822)*
> *Receiving objects: 100% (35046/35046), 53.05 MiB | 6.48 MiB/s, done.*
> *Resolving deltas: 100% (15708/15708), done.*
> *Fetching origin*
> * JIRA STATUS FIX VERSION
> ASSIGNEE FIX*
> * METRON-1345 Done Michael
> Miklavcic https://issues.apache.org/jira/browse/METRON-1345
> <https://issues.apache.org/jira/browse/METRON-1345>*
> * METRON-1349 Done Next + 1 Nick
> Allen https://issues.apache.org/jira/browse/METRON-1349
> <https://issues.apache.org/jira/browse/METRON-1349>*
> * METRON-1343 Done
> Mohan https://issues.apache.org/jira/browse/METRON-1343
> <https://issues.apache.org/jira/browse/METRON-1343>*
> * METRON-1306 To Do
> Unassigned https://issues.apache.org/jira/browse/METRON-1306
> <https://issues.apache.org/jira/browse/METRON-1306>*
> * METRON-1341 Done Simon Elliston
> Ball https://issues.apache.org/jira/browse/METRON-1341
> <https://issues.apache.org/jira/browse/METRON-1341>*
> * METRON-1313 Done Jon
> Zeolla https://issues.apache.org/jira/browse/METRON-1313
> <https://issues.apache.org/jira/browse/METRON-1313>*
> * METRON-1346 Done Otto
> Fowler https://issues.apache.org/jira/browse/METRON-1346
> <https://issues.apache.org/jira/browse/METRON-1346>*
> * METRON-1336 Done 0.4.2 Nick
> Allen*
> * METRON-1335 Done Anand
> Subramanian https://issues.apache.org/jira/browse/METRON-1335
> <https://issues.apache.org/jira/browse/METRON-1335>*
> * METRON-1308 Done Jon
> Zeolla https://issues.apache.org/jira/browse/METRON-1308
> <https://issues.apache.org/jira/browse/METRON-1308>*
> * METRON-1338 Done 0.4.2 Nick
> Allen*
> * METRON-1286 To Do 0.4.2
> Unassigned https://issues.apache.org/jira/browse/METRON-1286
> <https://issues.apache.org/jira/browse/METRON-1286>*
> * METRON-1334 Done 0.4.2 Nick
> Allen*
> * METRON-1277 Done Otto
> Fowler https://issues.apache.org/jira/browse/METRON-1277
> <https://issues.apache.org/jira/browse/METRON-1277>*
> * METRON-1239 To Do
> Unassigned https://issues.apache.org/jira/browse/METRON-1239
> <https://issues.apache.org/jira/browse/METRON-1239>*
> * METRON-1328 Done Anand
> Subramanian https://issues.apache.org/jira/browse/METRON-1328
> <https://issues.apache.org/jira/browse/METRON-1328>*
> * METRON-1333 Done Otto
> Fowler https://issues.apache.org/jira/browse/METRON-1333
> <https://issues.apache.org/jira/browse/METRON-1333>*
> * METRON-1252 Done
> RaghuMitra https://issues.apache.org/jira/browse/METRON-1252
> <https://issues.apache.org/jira/browse/METRON-1252>*
> * METRON-1316 To Do Next + 1
> Unassigned https://issues.apache.org/jira/browse/METRON-1316
> <https://issues.apache.org/jira/browse/METRON-1316>*
> * METRON-1088 Done Jon
> Zeolla https://issues.apache.org/jira/browse/METRON-1088
> <https://issues.apache.org/jira/browse/METRON-1088>*
> * METRON-1319 To Do Ryan
> Merriman https://issues.apache.org/jira/browse/METRON-1319
> <https://issues.apache.org/jira/browse/METRON-1319>*
> * METRON-1321 To Do
> Unassigned https://issues.apache.org/jira/browse/METRON-1321
> <https://issues.apache.org/jira/browse/METRON-1321>*
> * METRON-1301 Done 0.4.2 Nick
> Allen*
> * METRON-1294 To Do Ryan
> Merriman https://issues.apache.org/jira/browse/METRON-1294
> <https://issues.apache.org/jira/browse/METRON-1294>*
> * METRON-1291 To Do Ryan
> Merriman https://issues.apache.org/jira/browse/METRON-1291
> <https://issues.apache.org/jira/browse/METRON-1291>*
> * METRON-1290 Done Justin
> Leet https://issues.apache.org/jira/browse/METRON-1290
> <https://issues.apache.org/jira/browse/METRON-1290>*
> * METRON-1311 Done 0.4.2 Nick
> Allen*
> * METRON-1289 To Do Ryan
> Merriman https://issues.apache.org/jira/browse/METRON-1289
> <https://issues.apache.org/jira/browse/METRON-1289>*
> * METRON-1309 Done Jon
> Zeolla https://issues.apache.org/jira/browse/METRON-1309
> <https://issues.apache.org/jira/browse/METRON-1309>*
> * METRON-1310 Done 0.4.2 Nick
> Allen*
> * METRON-1275 Done Jon
> Zeolla https://issues.apache.org/jira/browse/METRON-1275
> <https://issues.apache.org/jira/browse/METRON-1275>*
> * METRON-1295 Done 0.4.2 Nick
> Allen*
> * METRON-1307 Done Otto
> Fowler https://issues.apache.org/jira/browse/METRON-1307
> <https://issues.apache.org/jira/browse/METRON-1307>*
> * METRON-1296 Done 0.4.2 Nick
> Allen*
> * METRON-1281 To Do Ryan
> Merriman https://issues.apache.org/jira/browse/METRON-1281
> <https://issues.apache.org/jira/browse/METRON-1281>*
> * METRON-1287 Done 0.4.2 Nick
> Allen*
> * METRON-1267 To Do
> Unassigned https://issues.apache.org/jira/browse/METRON-1267
> <https://issues.apache.org/jira/browse/METRON-1267>*
> * METRON-1283 Done Anand
> Subramanian https://issues.apache.org/jira/browse/METRON-1283
> <https://issues.apache.org/jira/browse/METRON-1283>*
> * METRON-1254 To Do
> Unassigned https://issues.apache.org/jira/browse/METRON-1254
> <https://issues.apache.org/jira/browse/METRON-1254>*
> * METRON-1261 Done Jon
> Zeolla https://issues.apache.org/jira/browse/METRON-1261
> <https://issues.apache.org/jira/browse/METRON-1261>*
> * METRON-1284 Done Justin
> Leet https://issues.apache.org/jira/browse/METRON-1284
> <https://issues.apache.org/jira/browse/METRON-1284>*
> * METRON-1270 Done Artem
> Ervits https://issues.apache.org/jira/browse/METRON-1270
> <https://issues.apache.org/jira/browse/METRON-1270>*
> * METRON-1272 In Progress Justin
> Leet https://issues.apache.org/jira/browse/METRON-1272
> <https://issues.apache.org/jira/browse/METRON-1272>*
> * METRON-1224 Done
> RaghuMitra https://issues.apache.org/jira/browse/METRON-1224
> <https://issues.apache.org/jira/browse/METRON-1224>*
> * METRON-1280 To Do
> Unassigned https://issues.apache.org/jira/browse/METRON-1280
> <https://issues.apache.org/jira/browse/METRON-1280>*
> * METRON-1243 To Do
> Unassigned https://issues.apache.org/jira/browse/METRON-1243
> <https://issues.apache.org/jira/browse/METRON-1243>*
> * METRON-1196 Done Matt
> Foley https://issues.apache.org/jira/browse/METRON-1196
> <https://issues.apache.org/jira/browse/METRON-1196>*
> * METRON-1278 Done 0.4.2 Matt
> Foley*
> * METRON-1274 To Do
> Unassigned https://issues.apache.org/jira/browse/METRON-1274
> <https://issues.apache.org/jira/browse/METRON-1274>*
> * METRON-1266 Done 0.4.2 Nick
> Allen*
> * METRON-1260 Done 0.4.2 Nick
> Allen*
> * METRON-1251 Done Jon
> Zeolla https://issues.apache.org/jira/browse/METRON-1251
> <https://issues.apache.org/jira/browse/METRON-1251>*
> * METRON-1241 To Do
> Unassigned https://issues.apache.org/jira/browse/METRON-1241
> <https://issues.apache.org/jira/browse/METRON-1241>*
> * METRON-1267 To Do
> Unassigned https://issues.apache.org/jira/browse/METRON-1267
> <https://issues.apache.org/jira/browse/METRON-1267>*
> * METRON-1262 To Do Ryan
> Merriman https://issues.apache.org/jira/browse/METRON-1262
> <https://issues.apache.org/jira/browse/METRON-1262>*
> * METRON-1263 Done Anand
> Subramanian https://issues.apache.org/jira/browse/METRON-1263
> <https://issues.apache.org/jira/browse/METRON-1263>*
> * METRON-1255 To Do
> Unassigned https://issues.apache.org/jira/browse/METRON-1255
> <https://issues.apache.org/jira/browse/METRON-1255>*
> * METRON-1249 Done 0.4.1 Nick
> Allen https://issues.apache.org/jira/browse/METRON-1249
> <https://issues.apache.org/jira/browse/METRON-1249>*
> * METRON-1237 To Do Artem
> Ervits https://issues.apache.org/jira/browse/METRON-1237
> <https://issues.apache.org/jira/browse/METRON-1237>*
> * METRON-1240 Done Artem
> Ervits https://issues.apache.org/jira/browse/METRON-1240
> <https://issues.apache.org/jira/browse/METRON-1240>*
> * METRON-1226 Done 0.4.2 Nick
> Allen*
> * METRON-1081 Done James
> Sirota https://issues.apache.org/jira/browse/METRON-1081
> <https://issues.apache.org/jira/browse/METRON-1081>*
> * METRON-1123 Done
> RaghuMitra https://issues.apache.org/jira/browse/METRON-1123
> <https://issues.apache.org/jira/browse/METRON-1123>*
> * METRON-1223 Done
> RaghuMitra https://issues.apache.org/jira/browse/METRON-1223
> <https://issues.apache.org/jira/browse/METRON-1223>*
> * METRON-1083 Done
> RaghuMitra https://issues.apache.org/jira/browse/METRON-1083
> <https://issues.apache.org/jira/browse/METRON-1083>*
> * METRON-1232 Done
> RaghuMitra https://issues.apache.org/jira/browse/METRON-1232
> <https://issues.apache.org/jira/browse/METRON-1232>*
> * METRON-1247 Done Justin
> Leet https://issues.apache.org/jira/browse/METRON-1247
> <https://issues.apache.org/jira/browse/METRON-1247>*
> * METRON-1235 To Do
> Unassigned https://issues.apache.org/jira/browse/METRON-1235
> <https://issues.apache.org/jira/browse/METRON-1235>*
> * METRON-1234 Done Artem
> Ervits https://issues.apache.org/jira/browse/METRON-1234
> <https://issues.apache.org/jira/browse/METRON-1234>*
> * METRON-1222 Done Artem
> Ervits https://issues.apache.org/jira/browse/METRON-1222
> <https://issues.apache.org/jira/browse/METRON-1222>*
> * METRON-1220 In Progress Justin
> Leet https://issues.apache.org/jira/browse/METRON-1220
> <https://issues.apache.org/jira/browse/METRON-1220>*
> * METRON-1229 To Do
> Unassigned https://issues.apache.org/jira/browse/METRON-1229
> <https://issues.apache.org/jira/browse/METRON-1229>*
> * METRON-1228 Done
> Unassigned https://issues.apache.org/jira/browse/METRON-1228
> <https://issues.apache.org/jira/browse/METRON-1228>*
> * METRON-1218 To Do Ryan
> Merriman https://issues.apache.org/jira/browse/METRON-1218
> <https://issues.apache.org/jira/browse/METRON-1218>*
> * METRON-1161 In Progress Ryan
> Merriman https://issues.apache.org/jira/browse/METRON-1161
> <https://issues.apache.org/jira/browse/METRON-1161>*
> * METRON-1209 To Do
> Unassigned https://issues.apache.org/jira/browse/METRON-1209
> <https://issues.apache.org/jira/browse/METRON-1209>*
> * METRON-1059 Done Next + 1
> Unassigned https://issues.apache.org/jira/browse/METRON-1059
> <https://issues.apache.org/jira/browse/METRON-1059>*
> * METRON-1204 To Do Ryan
> Merriman https://issues.apache.org/jira/browse/METRON-1204
> <https://issues.apache.org/jira/browse/METRON-1204>*
> * METRON-1052 Done Casey
> Stella https://issues.apache.org/jira/browse/METRON-1052
> <https://issues.apache.org/jira/browse/METRON-1052>*
> * METRON-632 In Progress Tomas
> Zezula https://issues.apache.org/jira/browse/METRON-632
> <https://issues.apache.org/jira/browse/METRON-632>*
> * METRON-1194 Done 0.4.2 Nick
> Allen*
> * METRON-1055 To Do Laurens
> Vets https://issues.apache.org/jira/browse/METRON-1055
> <https://issues.apache.org/jira/browse/METRON-1055>*
> * METRON-1079 Done Otto
> Fowler https://issues.apache.org/jira/browse/METRON-1079
> <https://issues.apache.org/jira/browse/METRON-1079>*
> * METRON-1085 To Do Ryan
> Merriman https://issues.apache.org/jira/browse/METRON-1085
> <https://issues.apache.org/jira/browse/METRON-1085>*
> * METRON-1208 To Do Ryan
> Merriman https://issues.apache.org/jira/browse/METRON-1208
> <https://issues.apache.org/jira/browse/METRON-1208>*
> * METRON-1207 To Do Ryan
> Merriman https://issues.apache.org/jira/browse/METRON-1207
> <https://issues.apache.org/jira/browse/METRON-1207>*
> * METRON-1215 To Do
> Unassigned https://issues.apache.org/jira/browse/METRON-1215
> <https://issues.apache.org/jira/browse/METRON-1215>*
> * METRON-1206 To Do Ryan
> Merriman https://issues.apache.org/jira/browse/METRON-1206
> <https://issues.apache.org/jira/browse/METRON-1206>*
> * METRON-1195 To Do Justin
> Leet https://issues.apache.org/jira/browse/METRON-1195
> <https://issues.apache.org/jira/browse/METRON-1195>*
> * METRON-1189 To Do Ryan
> Merriman https://issues.apache.org/jira/browse/METRON-1189
> <https://issues.apache.org/jira/browse/METRON-1189>*
> * METRON-1156 Done 0.4.2 Nick
> Allen*
> * METRON-1198 Done 0.4.2 Nick
> Allen*
> * METRON-1202 Done Next + 1 Justin
> Leet https://issues.apache.org/jira/browse/METRON-1202
> <https://issues.apache.org/jira/browse/METRON-1202>*
> * METRON-938 Done Next + 1 Justin
> Leet https://issues.apache.org/jira/browse/METRON-938
> <https://issues.apache.org/jira/browse/METRON-938>*
> * METRON-1182 Done
> RaghuMitra https://issues.apache.org/jira/browse/METRON-1182
> <https://issues.apache.org/jira/browse/METRON-1182>*
> * METRON-1188 Done Michael
> Miklavcic https://issues.apache.org/jira/browse/METRON-1188
> <https://issues.apache.org/jira/browse/METRON-1188>*
> * METRON-1191 Done Matt
> Foley https://issues.apache.org/jira/browse/METRON-1191
> <https://issues.apache.org/jira/browse/METRON-1191>*
> * METRON-1063 Done Next + 1 Artem
> Ervits https://issues.apache.org/jira/browse/METRON-1063
> <https://issues.apache.org/jira/browse/METRON-1063>*
> * METRON-1190 In Progress Justin
> Leet https://issues.apache.org/jira/browse/METRON-1190
> <https://issues.apache.org/jira/browse/METRON-1190>*
> * METRON-1187 Done 0.4.2 Nick
> Allen*
> * METRON-1185 To Do
> Unassigned https://issues.apache.org/jira/browse/METRON-1185
> <https://issues.apache.org/jira/browse/METRON-1185>*
> * METRON-1186 To Do Casey
> Stella https://issues.apache.org/jira/browse/METRON-1186
> <https://issues.apache.org/jira/browse/METRON-1186>*
> * METRON-1173 Done Next + 1 Jon
> Zeolla https://issues.apache.org/jira/browse/METRON-1173
> <https://issues.apache.org/jira/browse/METRON-1173>*
> * METRON-1179 To Do
> Unassigned https://issues.apache.org/jira/browse/METRON-1179
> <https://issues.apache.org/jira/browse/METRON-1179>*
> * METRON-1180 To Do
> Unassigned https://issues.apache.org/jira/browse/METRON-1180
> <https://issues.apache.org/jira/browse/METRON-1180>*
> * METRON-1183 Done 0.4.2 Nick
> Allen*
> * METRON-1177 Done 0.4.2 Nick
> Allen*
> * METRON-1158 To Do Justin
> Leet https://issues.apache.org/jira/browse/METRON-1158
> <https://issues.apache.org/jira/browse/METRON-1158>*
> * METRON-1146 Done Anand
> Subramanian https://issues.apache.org/jira/browse/METRON-1146
> <https://issues.apache.org/jira/browse/METRON-1146>*
> * METRON-1176 Done Otto
> Fowler https://issues.apache.org/jira/browse/METRON-1176
> <https://issues.apache.org/jira/browse/METRON-1176>*
> * METRON-1114 Done Ryan
> Merriman https://issues.apache.org/jira/browse/METRON-1114
> <https://issues.apache.org/jira/browse/METRON-1114>*
> * METRON-1167 Done Next + 1 Nick
> Allen https://issues.apache.org/jira/browse/METRON-1167
> <https://issues.apache.org/jira/browse/METRON-1167>*
> * METRON-1171 To Do Casey
> Stella https://issues.apache.org/jira/browse/METRON-1171
> <https://issues.apache.org/jira/browse/METRON-1171>*
>
>
>
> The script is here [1] if anyone wants to run it themselves and grep for
> their own name.
>
> [1]
> https://github.com/nickwallen/metron-commit-stuff/blob/
> master/validate-jira-for-release
>
> On Fri, Dec 15, 2017 at 1:18 PM, Matt Foley <mfoley@hortonworks.com<
> mailto:mfoley@hortonworks.com>> wrote:
>
> > Hi Nick,
> > Good timing, you’ve saved me some work! :-)
> >
> > Origin of the list: The approach was defined before I started managing
> > releases, but I think it’s a reasonable approach. It’s specified in our
> > Release Process document, https://cwiki.apache.org/
> > confluence/display/METRON/Release+Process , Step 5, the last bullet:
> > “The artifacts for a release or RC [include]… A CHANGES file
> > denoting the changes [since the last release].
> > We usually construct this by taking the output of git log | grep
> > METRON | sed 's/\[//g' | sed 's/\]//g' | grep -v "http" and removing the
> > JIRAs from the previous releases (it’s in time sorted order so this is
> > easy).”
> >
> > However, the release manager also has a responsibility to “Monitor and
> > Verify Jiras” (Step 2 and various other places in the doc). Altho the
> doc
> > isn’t explicit about it, I take this responsibility a little farther and
> do
> > the following as part of Jira verification:
> >
> > 1. Extract the jira id’s from the CHANGES file with a simple awk script,
> > put them in a Jira query, and confirm they are all actually marked
> > “Fixed/Resolved” with Fix Version = <the current release> in Jira. If
> > there are exceptions, I dun the contributors to fix it. You’ve seen
> these
> > emails from me in the past couple releases. I don’t just mark them fixed
> > myself, because some contributions (perfectly legitimately) only
> partially
> > address a problem, and just marking a ticket “Fixed” may not be the right
> > thing to do.
> >
> > 2. Query jira for any tickets NOT included in the prior list, that ARE
> > marked fixed in the current release (or later, or “Next”, or similar
> > “future” version constructs). These don’t usually happen (considering
> that
> > Metron contributors mostly live inside github PR processes rather than
> > Jiras) but when they do they also need to be resolved. Cases I’ve seen
> (in
> > Metron and other projects) include:
> > a) If they were indeed fixed in the current release, but not reflected in
> > the commit message, I’ll annotate this fact in the Jira ticket, and
> > hand-edit the CHANGES file for this release. (Doesn’t seem worth trying
> to
> > edit old commit messages, since that mucks up the SHA1’s.)
> > b) If they were fixed in a previous release but mis-marked as to Fix
> > Version in the ticket, fix the ticket.
> > c) If they aren’t really fixed, fix the ticket.
> >
> > 3. Once the set of Fixed tickets is complete and correct, query them for
> > any that have “labels in (backward-incompatible, backwards-compatibility,
> > backwards-incompatible)” -- yes, there shouldn’t be 3 labels, but it’s
> hard
> > to fix because Jira never forgets, so every time someone gets it wrong,
> it
> > adds to the set; fortunately, query completion helps here – OR "Docs
> Text"
> > is not EMPTY. Each of these needs to be looked at carefully, verified,
> and
> > if Doc Text is missing, the contributor and I cooperate to write a couple
> > sentences. The Doc Text then needs to go in the RELEASE_NOTES for the
> > release, in the section “## Non-backward-compatible Changes in this
> > Release, and Upgrade Suggestions”.
> >
> > That’s about it. I guess I should add the above as an appendix doc to
> the
> > Release Process documentation. I’ll do that.
> > Cheers,
> > --Matt
> >
> >
> > On 12/15/17, 9:15 AM, "Nick Allen" <nick@nickallen.org<mailto:nic
> k@nickallen.org>> wrote:
> >
> > Hi Matt -
> >
> > I just updated like 15+ JIRAs of my own JIRAs that I completed and
> > merged,
> > but failed to mark as resolved. All of these will be included in
> > 0.4.2. I
> > updated each to be "fixed" in version 0.4.2 and marked as resolved.
> > Hopefully the next RC will report those as fixed.
> >
> > (Q) Where does the list of JIRAs that get attached to the release
> > originate
> > from? Does it get pulled out of JIRA or do they come from the commit
> > log?
> >
> > My apologies for not staying on top of my JIRAs.
> >
> >
> >
> >
> > On Tue, Dec 12, 2017 at 2:21 PM, Matt Foley <mattf@apache.org
> <ma...@apache.org>> wrote:
> >
> > > Thanks to Jon, Otto, and Nick for looking over RC1. Tonight I’ll
> > fix the
> > > RAT glitch, build RC2, and put it to formal vote.
> > > Regards,
> > > --Matt
> > >
> > > On 12/12/17, 5:14 AM, "Nick Allen" <nick@nickallen.org<mailto:nic
> k@nickallen.org>> wrote:
> > >
> > > RC1 is looking good to me. I validated the MD5s, built Metron,
> > built
> > > the
> > > Bro plugin and reviewed the other artifacts like release notes.
> > >
> > > Running the RAT check on a 'clean' Metron does not produce any
> > errors
> > > for
> > > me. It is only after building Metron, which pulls in
> additional
> > Node
> > > dependencies, does the RAT check fail.
> > >
> > >
> > > On Sun, Dec 10, 2017 at 4:41 PM Matt Foley <mattf@apache.org
> <ma...@apache.org>>
> > wrote:
> > >
> > > > Yes, but let’s see if anyone else find other issues.
> > > >
> > > >
> > > >
> > > > From: Otto Fowler <ottobackwards@gmail.com<mailto:
> ottobackwards@gmail.com>>
> > > > Date: Saturday, December 9, 2017 at 6:16 AM
> > > > To: Matt Foley <mfoley@hortonworks.com<mailto:
> mfoley@hortonworks.com>>, "
> > dev@metron.apache.org<ma...@metron.apache.org>" <
> > > > dev@metron.apache.org<ma...@metron.apache.org>>
> > > > Subject: Re: [DISCUSS] Upcoming Release
> > > >
> > > >
> > > >
> > > > So RC2 then?
> > > >
> > > >
> > > >
> > > > On December 8, 2017 at 20:43:21, Matt Foley (
> > mfoley@hortonworks.com<ma...@hortonworks.com>)
> > > > wrote:
> > > >
> > > > Hah, here it is: https://github.com/apache/metron/pull/743
> > > > “This problem seems to only reproduce when one unrolls a
> > tarball
> > > rather
> > > > than cloning from github.”
> > > >
> > > > Heh, the exclusion at
> > > > https://github.com/apache/metron/blob/master/pom.xml#L351 is
> > still
> > > there,
> > > > but the hashcode in the bundle.css file name has changed from
> > > > a0b6b99c10d9a13dc67e to f56deed131e58bd7ee04. Sigh. Did the
> > version
> > > of Font
> > > > Awesome fonts change?
> > > >
> > > >
> > > > On 12/8/17, 5:26 PM, "Matt Foley" <mattf@apache.org<mailto:
> mattf@apache.org>> wrote:
> > > >
> > > > I remember having trouble with this bundle.css file on the
> last
> > > release,
> > > > but I can’t remember what we did about it. Anybody?
> > > >
> > > > On 12/8/17, 1:41 PM, "Otto Fowler" <ottobackwards@gmail.com<
> mailto:ottobackwards@gmail.com>>
> > wrote:
> > > >
> > > > Steps
> > > >
> > > > - Downloaded tar.gz’s, asc files and KEYS
> > > > - Verified signing of both tar.gz’s
> > > > - searched for rouge 0.4.1 entries
> > > > - verified the main pom.xml
> > > > - built :
> > > >
> > > > mvn clean && time mvn -q -T 2C -DskipTests install && time
> mvn
> > -q -T
> > > > 2C surefire:test@unit-tests && time mvn -q
> > > > surefire:test@integration-tests && time mvn -q test
> --projects
> > > > metron-interface/metron-config && time
> > build_utils/verify_licenses.sh
> > > >
> > > > Found rat error:
> > > >
> > > >
> > > > *****************************************************
> > > > Summary
> > > > -------
> > > > Generated at: 2017-12-08T16:33:27-05:00
> > > >
> > > > Notes: 3
> > > > Binaries: 193
> > > > Archives: 0
> > > > Standards: 75
> > > >
> > > > Apache Licensed: 74
> > > > Generated Documents: 0
> > > >
> > > > JavaDocs are generated, thus a license header is optional.
> > > > Generated files do not require license headers.
> > > >
> > > > 1 Unknown Licenses
> > > >
> > > > *****************************************************
> > > >
> > > > Files with unapproved licenses:
> > > >
> > > >
> > > > /Users/batman/tmp/release_ver/apache-metron-0.4.2-rc1/
> > > metron-interface/metron-alerts/dist/styles.
> > f56deed131e58bd7ee04.bundle.css
> > > >
> > > > *****************************************************
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > *****************************************************
> > > > Summary
> > > > -------
> > > > Generated at: 2017-12-08T16:33:27-05:00
> > > >
> > > > Notes: 3
> > > > Binaries: 193
> > > > Archives: 0
> > > > Standards: 75
> > > >
> > > > Apache Licensed: 74
> > > > Generated Documents: 0
> > > >
> > > > JavaDocs are generated, thus a license header is optional.
> > > > Generated files do not require license headers.
> > > >
> > > > 1 Unknown Licenses
> > > >
> > > > *****************************************************
> > > >
> > > > Files with unapproved licenses:
> > > >
> > > >
> > > >
> > > > /Users/ottofowler/tmp/release_ver/apache-metron-0.4.2-rc1/
> > > metron-interface/metron-alerts/dist/styles.
> > f56deed131e58bd7ee04.bundle.css
> > > >
> > > > *****************************************************
> > > >
> > > >
> > > >
> > > > On December 8, 2017 at 04:34:24, Matt Foley (
> mattf@apache.org<ma...@apache.org>)
> > > wrote:
> > > >
> > > > Colleagues,
> > > > I’ve posted Metron-0.4.2-RC1 and Metron-bro-plugin-kafka-0.1
> to
> > > > https://dist.apache.org/repos/dist/dev/metron/0.4.2-RC1/
> > > >
> > > > Given the complexity of this RC, I’d appreciate if a couple
> > people
> > > would be
> > > > willing to kick the tires before we put it up for a vote.
> > > >
> > > > I will myself be going thru the Verify Build process this
> > weekend,
> > > as I
> > > > won’t be able to do it Friday.
> > > >
> > > > Thanks,
> > > > --Matt
> > > >
> > > >
> > > > On 12/4/17, 2:05 PM, "Zeolla@GMail.com" <zeolla@gmail.com
> <ma...@gmail.com>>
> > wrote:
> > > >
> > > > Can we resolve the conversation regarding the second repo? I
> > was
> > > waiting
> > > > to get more input/preferences from people There's also a
> > > documentation
> > > > update that fixes a few broken Stellar docs that already has
> > aa +1,
> > > I just
> > > > need to merge it.
> > > >
> > > > Jon
> > > >
> > > > On Mon, Dec 4, 2017, 17:01 Casey Stella <cestella@gmail.com
> <ma...@gmail.com>>
> > wrote:
> > > >
> > > > > I would be in favor of a release at this point.
> > > > >
> > > > > On Mon, Dec 4, 2017 at 4:57 PM, Matt Foley <
> mattf@apache.org<ma...@apache.org>
> > >
> > > wrote:
> > > > >
> > > > > > Hey all,
> > > > > > I see METRON-1252 was resolved over the weekend. Shall I
> go
> > > ahead and
> > > > > > start the process with 0.4.2 release?
> > > > > > Does anyone have any commits they feel strongly should go
> > in
> > > before
> > > > 0.4.2
> > > > > > is done, or are we ready to call it good?
> > > > > >
> > > > > > I believe there is consensus the 0.4.2 release should
> > include a
> > > release
> > > > > of
> > > > > > the current state of the metron-bro-plugin-kafka. I will
> > > continue the
> > > > > > discussion in that thread as to the process for
> > accomplishing
> > > that, but
> > > > > > plan on it happening.
> > > > > >
> > > > > > Regards,
> > > > > > --Matt
> > > > > >
> > > > > > On 11/26/17, 6:26 PM, "Matt Foley" <mattf@apache.org
> <ma...@apache.org>>
> > wrote:
> > > > > >
> > > > > > Hope everyone (at least in the U.S.) had a great
> > Thanksgiving
> > > > > holiday.
> > > > > > Regarding status of the release effort, still pending
> > > METRON-1252, so
> > > > > > not making the release branch yet.
> > > > > >
> > > > > > Regards,
> > > > > > --Matt
> > > > > >
> > > > > > On 11/17/17, 1:32 PM, "Matt Foley" <mattf@apache.org
> <ma...@apache.org>>
> > wrote:
> > > > > >
> > > > > > (With release manager hat on)
> > > > > >
> > > > > > The community has proposed a release of Metron in the
> near
> > > > > future,
> > > > > > focusing on Meta-alerts running in Elasticsearch.
> > > > > > Congrats on getting so many of the below already done. At
> > this
> > > > > > point, only METRON-1252, and the discussion of how to
> > handle
> > > joint
> > > > > release
> > > > > > of the Metron bro plugin, remain as gating items for the
> > > release. I
> > > > > > project these will be resolved next week, so let’s
> propose
> > the
> > > > following:
> > > > > >
> > > > > > Sometime next week, after the last bits are done, I’ll
> > start the
> > > > > > release process and create the release branch.
> > > > > >
> > > > > > The proposed new version will be 0.4.2, unless there are
> > backward
> > > > > > incompatible changes that support making it 0.5.0.
> > > > > > Currently there are NO included Jiras labeled
> > > > > > ‘backward-incompatible’, nor having Docs Text indicating
> > so.
> > > > > > ***If anyone knows that some of the commits included
> since
> > 0.4.1
> > > > > > introduce backward incompatibility, please say so now on
> > this
> > > thread,
> > > > and
> > > > > > mark the Jira as such.***
> > > > > >
> > > > > > The 90 or so jiras/commits already in master branch since
> > 0.4.1
> > > > > > are listed below.
> > > > > > Thanks,
> > > > > > --Matt
> > > > > >
> > > > > > METRON-1301 Alerts UI - Sorting on Triage Score
> > Unexpectedly
> > > > > > Filters Some Records (nickwallen) closes
> apache/metron#832
> > > > > > METRON-1294 IP addresses are not formatted correctly in
> > facet
> > > > > > and group results (merrimanr) closes apache/metron#827
> > > > > > METRON-1291 Kafka produce REST endpoint does not work in
> a
> > > > > > Kerberized cluster (merrimanr) closes apache/metron#826
> > > > > > METRON-1290 Only first 10 alerts are update when a
> > MetaAlert
> > > > > > status is changed to inactive (justinleet) closes
> > > apache/metron#842
> > > > > > METRON-1311 Service Check Should Check Elasticsearch
> Index
> > > > > > Templates (nickwallen) closes apache/metron#839
> > > > > > METRON-1289 Alert fields are lost when a MetaAlert is
> > created
> > > > > > (merrimanr) closes apache/metron#824
> > > > > > METRON-1309 Change metron-deployment to pull the plugin
> > from
> > > > > > apache/metron-bro-plugin-kafka (JonZeolla) closes
> > > apache/metron#837
> > > > > > METRON-1310 Template Delete Action Deletes Search Indices
> > > > > > (nickwallen) closes apache/metron#838
> > > > > > METRON-1275: Fix Metron Documentation closes
> > > > > > apache/incubator-metron#833
> > > > > > METRON-1295 Unable to Configure Logging for REST API
> > > > > > (nickwallen) closes apache/metron#828
> > > > > > METRON-1307 Force install of java8 since java9 does not
> > > > > appear
> > > > > > to work with the scripts (brianhurley via ottobackwards)
> > closes
> > > > > > apache/metron#835
> > > > > > METRON-1296 Full Dev Fails to Deploy Index Templates
> > > > > > (nickwallen via cestella) closes
> > apache/incubator-metron#829
> > > > > > METRON-1281 Remove hard-coded indices from the Alerts UI
> > > > > > (merrimanr) closes apache/metron#821
> > > > > > METRON-1287 Full Dev Fails When Installing EPEL
> Repository
> > > > > > (nickwallen) closes apache/metron#820
> > > > > > METRON-1267 Alerts UI returns a 404 when refreshing the
> > > > > > alerts-list page (iraghumitra via merrimanr) closes
> > > apache/metron#819
> > > > > > METRON-1283 Install Elasticsearch template as a part of
> the
> > > > > > mpack startup scripts (anandsubbu via nickwallen) closes
> > > > > apache/metron#817
> > > > > > METRON-1254: Conditionals as map keys do not function in
> > > > > > Stellar closes apache/incubator-metron#801
> > > > > > METRON-1261 Apply bro security patch (JonZeolla via
> > > > > > ottobackwards) closes apache/metron#805
> > > > > > METRON-1284 Remove extraneous dead query in
> > ElasticsearchDao
> > > > > > (justinleet) closes apache/metron#818
> > > > > > METRON-1270: fix for warnings missing @return tag
> argument
> > in
> > > > > > metron-analytics/metron-profiler-common and
> > > metron-profiler-client
> > > > closes
> > > > > > apache/incubator-metron#810
> > > > > > METRON-1272 Hide child alerts from searches and grouping
> if
> > > > > > they belong to meta alerts (justinleet) closes
> > apache/metron#811
> > > > > > METRON-1224 Add time range selection to search control
> > > > > > (iraghumitra via james-sirota) closes apache/metron#796
> > > > > > METRON-1280 0.4.1 -> 0.4.2 missed a couple of projects
> > > > > > (cestella via justinleet) closes apache/metron#816
> > > > > > METRON-1243: Add a REST endpoint which allows us to get a
> > > > > list
> > > > > > of all indice closes apache/incubator-metron#797
> > > > > > METRON-1196 Increment master version number to 0.4.2 for
> > > > > > on-going development (mattf-horton) closes
> > apache/metron#767
> > > > > > METRON-1278 Strip "Build Status" widget from
> root
> > > > > > README.md in site-book build (mattf-horton) closes
> > > apache/metron#815
> > > > > > METRON-1274 Master has failure in
> > > > > > StormControllerIntegrationTest (merrimanr) closes
> > > apache/metron#813
> > > > > > METRON-1266 Profiler - SASL Authentication Failed
> > > > > (nickwallen)
> > > > > > closes apache/metron#809
> > > > > > METRON-1260 Include Alerts UI in Ambari Service Check
> > > > > > (nickwallen) closes apache/metron#804
> > > > > > METRON-1251: Typo and formatting fixes for metron-rest
> > README
> > > > > > closes apache/incubator-metron#800
> > > > > > METRON-1241: Enable the REST API to use a cache for the
> > > > > > zookeeper config similar to the Bolts closes
> > > > apache/incubator-metron#795
> > > > > > METRON-1267 Alerts UI returns a 404 when refreshing the
> > > > > > alerts-list page (merrimanr) closes apache/metron#808
> > > > > > METRON-1262 Unable to add comment for a alert in a
> > meta-alert
> > > > > > (merrimanr) closes apache/metron#806
> > > > > > METRON-1263 Start Alerts UI service after Metron REST
> > > > > > (anandsubbu via nickwallen) closes apache/metron#807
> > > > > > METRON-1255 MetaAlert search is not filtering on status
> > > > > > (merrimanr) closes apache/metron#802
> > > > > > METRON-1249 Improve Metron MPack Service Checks
> > (nickwallen)
> > > > > > closes apache/metron#799
> > > > > > METRON-1237 address javadoc warnings in
> metron-maas-common
> > > > > > (dbist via james-sirota) closes apache/metron#792
> > > > > > METRON-1240 address javadoc warnings in metron-platform
> and
> > > > > > metron-analytics (dbist via james-sirota) closes
> > > apache/metron#794
> > > > > > METRON-1226 Searching Can Errantly Query the Wrong
> Indices
> > > > > > (nickwallen) closes apache/metron#793
> > > > > > METRON-1081 Fix Alerts and Ops UI Notices file
> > (james-sirota)
> > > > > > closes apache/metron#682
> > > > > > METRON-1123 Add group by option using faceted search
> > > > > > capabilities of metron-rest-api (iraghumitra via
> > james-sirota)
> > > closes
> > > > > > apache/metron#768
> > > > > > METRON-1223 Add support to add comments for alerts
> > > > > > (iraghumitra via james-sirota) closes apache/metron#788
> > > > > > METRON-1083 Add filters using faceted search capabilities
> > of
> > > > > > metron-rest-api (iraghumitra via james-sirota) closes
> > > apache/metron#710
> > > > > > METRON-1232 Alert status changes are not reflected in
> list
> > > > > > view (iraghumitra via merrimanr) closes apache/metron#787
> > > > > > METRON-1247 REST search and findOne endpoints return
> > > > > > unexpected or incorrect results for guids (justinleet)
> > closes
> > > > > > apache/metron#798
> > > > > > METRON-1235: Document the properties pulled from the
> global
> > > > > > configuration closes apache/incubator-metron#791
> > > > > > METRON-1234: fix for WARNING 'dependencies.dependency.(
> > > > > > groupId:artifactId:type:classifier)' must be unique:
> > > > > > org.apache.hadoop:hadoop-yarn-api:jar (dbist via
> mmiklavc)
> > > closes
> > > > > > apache/metron#790
> > > > > > METRON-1222: fix warning for The expression
> > ${parent.version}
> > > > > > is deprecated. Please use ${project.parent.version}
> > instead.
> > > (dbist via
> > > > > > mmiklavc) closes apache/metron#782
> > > > > > METRON-1220 Create documentation around alert nested
> field
> > > > > > (justinleet) closes apache/metron#780
> > > > > > METRON-1229 Management UI type is part of the
> declarations
> > of
> > > > > > 2 modules (merrimanr) closes apache/metron#784
> > > > > > METRON-1228: Configuration Management PUSH immediately
> does
> > > > > > DUMP after (mmiklavc via mmiklavc) closes
> apache/metron#783
> > > > > > METRON-1218 Metron REST should return better error
> messages
> > > > > > (merrimanr) closes apache/metron#779
> > > > > > METRON-1161 Add ability to edit parser command line
> options
> > > > > in
> > > > > > the management UI (merrimanr) closes apache/metron#737
> > > > > > METRON-1209: Make stellar repl take logging properties,
> > like
> > > > > > other CLI apps in metron closes
> apache/incubator-metron#772
> > > > > > METRON-1059 address checkstyle warning AvoidStarImport in
> > > > > > metron-stellar (dbist via ottobackwards) closes
> > apache/metron#664
> > > > > > METRON-1204 UI does not time out after being idle, but
> > stops
> > > > > > functioning (merrimanr) closes apache/metron#771
> > > > > > METRON-1052: Add forensic similarity hash functions to
> > > > > Stellar
> > > > > > closes apache/incubator-metron#781
> > > > > > METRON-632: Added validation of "shew.enrichmentType" and
> > > > > > "shew.keyColumns" closes apache/incubator-metron#732
> > > > > > METRON-1194 Add Profiler Debug Functions to Profiler
> README
> > > > > > (nickwallen via ottobackwards) closes apache/metron#765
> > > > > > METRON-1055 Metron 0.4.0 manual installation guide for
> > CentOS
> > > > > > 6 updates (lvets via ottobackwards) closes
> > apache/metron#661
> > > > > > METRON-1079 STELLAR NaN should be a keyword
> (ottobackwards)
> > > > > > closes apache/metron#681
> > > > > > METRON-1085 Add REST endpoint to save a user profile for
> > the
> > > > > > Alerts UI (merrimanr) closes apache/metron#694
> > > > > > METRON-1208 MPack for Alerts UI (merrimanr) closes
> > > > > > apache/metron#778
> > > > > > METRON-1207 Make RPMs for Alerts UI (merrimanr) closes
> > > > > > apache/metron#777
> > > > > > METRON-1215 Fix link to RPMs chapter (DimDroll via
> > > > > justinleet)
> > > > > > closes apache/metron#776
> > > > > > METRON-1206 Make alerts UI conform to ops UI for install
> > > > > > (merrimanr) closes apache/metron#773
> > > > > > METRON-1195 Meta alerts improperly handle updates to
> > > > > non-alert
> > > > > > fields (justinleet) closes apache/metron#766
> > > > > > METRON-1189 Add alert escalation to the Alerts UI
> > (merrimanr)
> > > > > > closes apache/metron#762
> > > > > > METRON-1156 Simulate Triage Rules in the Stellar REPL
> > > > > > (nickwallen) closes apache/metron#733
> > > > > > METRON-1198 Pycapa - No such configuration property
> > > > > > 'sasl.kerberos.principal' (nickwallen) closes
> > apache/metron#769
> > > > > > METRON-1202 ElasticsearchDao Has extraneous sleep call
> > > > > > (justinleet) closes apache/metron#770
> > > > > > METRON-938 "service metron-rest start <password>" does
> not
> > > > > > work on CentOS 7. (justinleet) closes apache/metron#757
> > > > > > METRON-1182 Refactor Code in alert list to accommodate
> new
> > > > > > view types (iraghumitra via merrimanr) closes
> > apache/metron#756
> > > > > > METRON-1188: Ambari global configuration management
> > > > > (mmiklavc)
> > > > > > closes apache/metron#760
> > > > > > METRON-1191 update public web site to point at 0.4.1 new
> > > > > > release (mattf-horton) closes apache/metron#764
> > > > > > METRON-1063 address javadoc warnings in metron-stellar
> > (dbist
> > > > > > via ottobackwards) closes apache/metron#668
> > > > > > METRON-1190 Fix Meta Alert Type handling in calculation
> of
> > > > > > scores (justinleet) closes apache/metron#763
> > > > > > METRON-1187 Indexing/Profiler Kafka ACL Groups Not Setup
> > > > > > Correctly (nickwallen) closes apache/metron#759
> > > > > > METRON-1185: Stellar REPL does not work on a kerberized
> > > > > > cluster when calling functions interacting with HBase
> > closes
> > > > > > apache/incubator-metron#755
> > > > > > METRON-1186: Profiler Functions use classutils from
> shaded
> > > > > > storm closes apache/incubator-metron#758
> > > > > > METRON-1173: Fix pointers to old stellar docs closes
> > > > > > apache/incubator-metron#746
> > > > > > METRON-1179: Make STATS_ADD to take a list closes
> > > > > > apache/incubator-metron#750
> > > > > > METRON-1180: Make Stellar Shell accept zookeeper quorum
> as
> > a
> > > > > > CSV list and not require a port closes
> > > apache/incubator-metron#751
> > > > > > METRON-1183 Improve KDC Setup Instructions (nickwallen)
> > > > > closes
> > > > > > apache/metron#753
> > > > > > METRON-1177 Stale running topologies seen
> > post-kerberization
> > > > > > and cause exceptions (nickwallen) closes
> apache/metron#748
> > > > > > METRON-1158 Build backend for grouping alerts into meta
> > > > > alerts
> > > > > > (justinleet) closes apache/metron#734
> > > > > > METRON-1146: Add ability to parse JSON string into
> > JSONObject
> > > > > > for stellar closes apache/incubator-metron#727
> > > > > > METRON-1176 REST: HDFS Service should support setting
> > > > > > permissions on files when writing (ottobackwards) closes
> > > > > apache/metron#749
> > > > > > METRON-1114 Add group by capabilities to search REST
> > endpoint
> > > > > > (merrimanr) closes apache/metron#702
> > > > > > METRON-1167 Define Session Specific Global Configuration
> > > > > > Values in the REPL (nickwallen) closes apache/metron#740
> > > > > > METRON-1171: Better validation for the SUBSTRING stellar
> > > > > > function closes apache/incubator-metron#745
> > > > > >
> > > > > >
> > > > > >
> > > > > > On 11/17/17, 11:59 AM, "Nick Allen" <nick@nickallen.org
> <ma...@nickallen.org>>
> > wrote:
> > > > > >
> > > > > > I just wanted to send an update on where we are at. We've
> > > > > > gotten a lot
> > > > > > done here recently as you can see below.
> > > > > >
> > > > > > ✓ DONE (1) First, METRON-1289 needs to go in. This one
> was
> > > > > > a fairly big
> > > > > > effort and I am hearing that we are pretty close.
> > > > > >
> > > > > > ✓ DONE (2) METRON-1294 fixes an issue in how field types
> > > > > are
> > > > > > looked-up.
> > > > > >
> > > > > > ✓ DONE (3) METRON-1290 is next. While this may have been
> > > > > > fixed in
> > > > > > M-1289, there may be some test cases we want from this
> PR.
> > > > > >
> > > > > > ✓ DONE (4) METRON-1301 addresses a problem with the
> sorting
> > > > > > logic.
> > > > > >
> > > > > > ✓ DONE (5) METRON-1291 fixes an issue with escalation of
> > > > > > metaalerts.
> > > > > >
> > > > > > (6) That leads us to Raghu's UI work in METRON-1252. This
> > > > > > introduces the
> > > > > > UI bits that depend on all the previous backend work.
> > > > > >
> > > > > > (7) At this point, we should have our best effort at
> > > > > running
> > > > > > Metaalerts
> > > > > > on Elasticsearch 2.x. I propose that we cut a release
> here.
> > > > > >
> > > > > > (8) After we cut the release, we can introduce the work
> for
> > > > > > ES 5.x in
> > > > > > METRON-939. I know we will need lots of help testing and
> > > > > > reviewing this
> > > > > > one.
> > > > > >
> > > > > >
> > > > > >
> > > > > > We also have an outstanding question that needs resolved
> > > > > > BEFORE we
> > > > > > release. We need to come to a consensus on how to release
> > > > > > having moved our
> > > > > > Bro Plugin to a separate repo. I don't think we've heard
> > > > > from
> > > > > > everyone on
> > > > > > this. I'd urge everyone to chime in so we can choose a
> path
> > > > > > forward.
> > > > > >
> > > > > > If anyone is totally confused in regards to that
> > discussion,
> > > > > I
> > > > > > can try and
> > > > > > send an options summary again as a separate discuss
> thread.
> > > > > > The original
> > > > > > chain was somewhere around here [1].
> > > > > >
> > > > > > [1]
> > > > > > https://lists.apache.org/thread.html/
> > > > > > 54a4474881b97e559df24728b3a0e9
> 23a58345a282451085eef832ef@%
> > > > > > 3Cdev.metron.apache.org<http://3Cdev.metron.apache.org
> >%3E
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Wed, Nov 15, 2017 at 10:04 AM, Nick Allen <
> > > > > > nick@nickallen.org<ma...@nickallen.org>> wrote:
> > > > > >
> > > > > > > Hi Guys -
> > > > > > >
> > > > > > > I want to follow-up on this discussion. It sounds like
> > > > > most
> > > > > > people are in
> > > > > > > agreement with the general approach.
> > > > > > >
> > > > > > > A lot of people have been working hard on Metaalerts
> and
> > > > > > Elasticsearch. I
> > > > > > > have checked-in with those doing the heavy lifting and
> > have
> > > > > > compiled a more
> > > > > > > detailed plan based on where we are at now. To the best
> > of
> > > > > > my knowledge
> > > > > > > here is the plan of attack for finishing out this
> effort.
> > > > > > >
> > > > > > > (1) First, METRON-1289 needs to go in. This one was a
> > > > > > fairly big effort
> > > > > > > and I am hearing that we are pretty close.
> > > > > > >
> > > > > > > (2) METRON-1294 fixes an issue in how field types are
> > > > > > looked-up.
> > > > > > >
> > > > > > > (3) METRON-1290 is next. While this may have been fixed
> > > > > > in M-1289,
> > > > > > > there may be some test cases we want from this PR.
> > > > > > >
> > > > > > > (4) METRON-1301 addresses a problem with the sorting
> > > > > logic.
> > > > > > >
> > > > > > > (5) METRON-1291 fixes an issue with escalation of
> > > > > > metaalerts.
> > > > > > >
> > > > > > > (6) That leads us to Raghu's UI work in METRON-1252.
> > > > > This
> > > > > > introduces
> > > > > > > the UI bits that depend on all the previous backend
> work.
> > > > > > >
> > > > > > > (7) At this point, we should have our best effort at
> > > > > > running Metaalerts
> > > > > > > on Elasticsearch 2.x. I propose that we cut a release
> > here.
> > > > > > >
> > > > > > > (8) After we cut the release, we can introduce the work
> > > > > > for ES 5.x in
> > > > > > > METRON-939. I know we will need lots of help testing
> and
> > > > > > reviewing this
> > > > > > > one.
> > > > > > >
> > > > > > > Please correct me if I am wrong. I will try and send
> out
> > > > > > updates as we
> > > > > > > make progress.
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > On Mon, Nov 6, 2017 at 1:03 PM, Zeolla@GMail.com <
> > > > > > zeolla@gmail.com<ma...@gmail.com>> wrote:
> > > > > > >
> > > > > > >> I agree, I think it's very reasonable to move in line
> > with
> > > > > > Nick's
> > > > > > >> proposal. I would also suggest that we outline what
> the
> > > > > > target versions
> > > > > > >> would be to add in the METRON-777 components, since it
> > has
> > > > > > been functional
> > > > > > >> for a very long time but not reviewed and has some
> > really
> > > > > > rockstar
> > > > > > >> improvements.
> > > > > > >>
> > > > > > >> Jon
> > > > > > >>
> > > > > > >> On Mon, Nov 6, 2017 at 12:56 PM Otto Fowler <
> > > > > > ottobackwards@gmail.com<ma...@gmail.com>>
> > > > > > >> wrote:
> > > > > > >>
> > > > > > >> > I think the ES cutover should be the start of the
> > 0.5.x
> > > > > > series, and we
> > > > > > >> > continue on with 0.4.x for the
> > > > > > >> > metadata improvements etc. We could chose to focus
> > > > > > 0.5.x’s first
> > > > > > >> releases
> > > > > > >> > on not only ES but
> > > > > > >> > getting a handle on kibana and the mpack situation
> as
> > > > > > well.
> > > > > > >> >
> > > > > > >> >
> > > > > > >> >
> > > > > > >> >
> > > > > > >> > On November 6, 2017 at 12:48:45, Michael Miklavcic (
> > > > > > >> > michael.miklavcic@gmail.com<mailto:
> michael.miklavcic@gmail.com>) wrote:
> > > > > > >> >
> > > > > > >> > I agree with your proposal, Nick. I think having a
> > > > > > stabilizing release
> > > > > > >> > prior to upgrading ES/Kibana makes sense.
> > > > > > >> >
> > > > > > >> > On Mon, Nov 6, 2017 at 9:16 AM, Nick Allen <
> > > > > > nick@nickallen.org<ma...@nickallen.org>> wrote:
> > > > > > >> >
> > > > > > >> > > I would like to start a discussion around upcoming
> > > > > > releases. We have a
> > > > > > >> > > couple separate significant tracks of work that we
> > > > > need
> > > > > > to reconcile
> > > > > > >> in
> > > > > > >> > our
> > > > > > >> > > release schedule.
> > > > > > >> > >
> > > > > > >> > > (1) We have had (and have in review) a good number
> > of
> > > > > > bug fixes
> > > > > > >> required
> > > > > > >> > to
> > > > > > >> > > support Metaalerts on the existing Elasticsearch
> 2.x
> > > > > > infrastructure.
> > > > > > >> > >
> > > > > > >> > >
> > > > > > >> > > (2) We also have ongoing work to upgrade our
> > > > > > infrastructure to
> > > > > > >> > > Elasticsearch 5.x, which will not be backwards
> > > > > > compatible.
> > > > > > >> > >
> > > > > > >> > >
> > > > > > >> > > I would like to see a release that has our best
> work
> > > > > on
> > > > > > ES 2.x before
> > > > > > >> we
> > > > > > >> > > migrate to 5.x. I would propose the following.
> > > > > > >> > >
> > > > > > >> > > Release N+1: Introduce Metaalerts running on ES
> 2.x
> > > > > > >> > >
> > > > > > >> > > Release N+2: Cut-over to ES 5.x
> > > > > > >> > >
> > > > > > >> > >
> > > > > > >> > > (Q) Is it worth cutting a separate release for ES
> > 2.x?
> > > > > > Is there a
> > > > > > >> better
> > > > > > >> > > way to handle the cut-over to 5.x?
> > > > > > >> > >
> > > > > > >> >
> > > > > > >> --
> > > > > > >>
> > > > > > >> Jon
> > > > > > >>
> > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > >
> > > > --
> > > >
> > > > Jon
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > >
> > >
> > >
> > >
> >
> >
> >
>
>
Re: [DISCUSS] Upcoming Release
Posted by Casey Stella <ce...@gmail.com>.
Yeah, that might work too. Let's also not forget otto's scripting around
release activity: https://github.com/ottobackwards/Metron-and-Nifi-
Scripts/blob/master/metron/metron-rc-check
We seem to be accumulating a lot of automation to do this, which is great.
On Fri, Dec 15, 2017 at 2:30 PM, Matt Foley <mf...@hortonworks.com> wrote:
> Perhaps under “build_utils” we should add a subdirectory for
> “release_utils”.
>
> From: Casey Stella <ce...@gmail.com>
> Date: Friday, December 15, 2017 at 10:50 AM
> To: "dev@metron.apache.org" <de...@metron.apache.org>
> Cc: Matt Foley <mf...@hortonworks.com>
> Subject: Re: [DISCUSS] Upcoming Release
>
> That script seems great, nick! Perhaps we should adjust the wiki around
> releases to point to it? Thoughts?
>
> On Fri, Dec 15, 2017 at 1:47 PM, Nick Allen <nick@nickallen.org<mailto:nic
> k@nickallen.org>> wrote:
> Thanks, Matt.
>
> Maybe you already have something that does this. I wrote a quick script
> that validates each JIRA since the last release tag to make sure they are
> marked "Done" and with the correct fix version. I would expect that for
> the next release, each JIRA should have status="Done", fix-version="0.4.2".
>
> Unless I am mistaken, we have quite a few that need cleaned up. In the
> following output, any line that has a URL indicates that a fix is needed.
> Or it least, **I think** a fix is needed.
>
> To the community.... If you see your name with a URL next to it, it would
> be great if you could follow that link and fix the JIRA. Otherwise, I will
> volunteer to help clean some of these up should some not get addressed.
>
>
> *$ ./validate-jira-for-release*
> *Cloning into 'metron-0.4.2'...*
> *remote: Counting objects: 35046, done.*
> *remote: Compressing objects: 100% (13698/13698), done.*
> *remote: Total 35046 (delta 15708), reused 31645 (delta 12822)*
> *Receiving objects: 100% (35046/35046), 53.05 MiB | 6.48 MiB/s, done.*
> *Resolving deltas: 100% (15708/15708), done.*
> *Fetching origin*
> * JIRA STATUS FIX VERSION
> ASSIGNEE FIX*
> * METRON-1345 Done Michael
> Miklavcic https://issues.apache.org/jira/browse/METRON-1345
> <https://issues.apache.org/jira/browse/METRON-1345>*
> * METRON-1349 Done Next + 1 Nick
> Allen https://issues.apache.org/jira/browse/METRON-1349
> <https://issues.apache.org/jira/browse/METRON-1349>*
> * METRON-1343 Done
> Mohan https://issues.apache.org/jira/browse/METRON-1343
> <https://issues.apache.org/jira/browse/METRON-1343>*
> * METRON-1306 To Do
> Unassigned https://issues.apache.org/jira/browse/METRON-1306
> <https://issues.apache.org/jira/browse/METRON-1306>*
> * METRON-1341 Done Simon Elliston
> Ball https://issues.apache.org/jira/browse/METRON-1341
> <https://issues.apache.org/jira/browse/METRON-1341>*
> * METRON-1313 Done Jon
> Zeolla https://issues.apache.org/jira/browse/METRON-1313
> <https://issues.apache.org/jira/browse/METRON-1313>*
> * METRON-1346 Done Otto
> Fowler https://issues.apache.org/jira/browse/METRON-1346
> <https://issues.apache.org/jira/browse/METRON-1346>*
> * METRON-1336 Done 0.4.2 Nick
> Allen*
> * METRON-1335 Done Anand
> Subramanian https://issues.apache.org/jira/browse/METRON-1335
> <https://issues.apache.org/jira/browse/METRON-1335>*
> * METRON-1308 Done Jon
> Zeolla https://issues.apache.org/jira/browse/METRON-1308
> <https://issues.apache.org/jira/browse/METRON-1308>*
> * METRON-1338 Done 0.4.2 Nick
> Allen*
> * METRON-1286 To Do 0.4.2
> Unassigned https://issues.apache.org/jira/browse/METRON-1286
> <https://issues.apache.org/jira/browse/METRON-1286>*
> * METRON-1334 Done 0.4.2 Nick
> Allen*
> * METRON-1277 Done Otto
> Fowler https://issues.apache.org/jira/browse/METRON-1277
> <https://issues.apache.org/jira/browse/METRON-1277>*
> * METRON-1239 To Do
> Unassigned https://issues.apache.org/jira/browse/METRON-1239
> <https://issues.apache.org/jira/browse/METRON-1239>*
> * METRON-1328 Done Anand
> Subramanian https://issues.apache.org/jira/browse/METRON-1328
> <https://issues.apache.org/jira/browse/METRON-1328>*
> * METRON-1333 Done Otto
> Fowler https://issues.apache.org/jira/browse/METRON-1333
> <https://issues.apache.org/jira/browse/METRON-1333>*
> * METRON-1252 Done
> RaghuMitra https://issues.apache.org/jira/browse/METRON-1252
> <https://issues.apache.org/jira/browse/METRON-1252>*
> * METRON-1316 To Do Next + 1
> Unassigned https://issues.apache.org/jira/browse/METRON-1316
> <https://issues.apache.org/jira/browse/METRON-1316>*
> * METRON-1088 Done Jon
> Zeolla https://issues.apache.org/jira/browse/METRON-1088
> <https://issues.apache.org/jira/browse/METRON-1088>*
> * METRON-1319 To Do Ryan
> Merriman https://issues.apache.org/jira/browse/METRON-1319
> <https://issues.apache.org/jira/browse/METRON-1319>*
> * METRON-1321 To Do
> Unassigned https://issues.apache.org/jira/browse/METRON-1321
> <https://issues.apache.org/jira/browse/METRON-1321>*
> * METRON-1301 Done 0.4.2 Nick
> Allen*
> * METRON-1294 To Do Ryan
> Merriman https://issues.apache.org/jira/browse/METRON-1294
> <https://issues.apache.org/jira/browse/METRON-1294>*
> * METRON-1291 To Do Ryan
> Merriman https://issues.apache.org/jira/browse/METRON-1291
> <https://issues.apache.org/jira/browse/METRON-1291>*
> * METRON-1290 Done Justin
> Leet https://issues.apache.org/jira/browse/METRON-1290
> <https://issues.apache.org/jira/browse/METRON-1290>*
> * METRON-1311 Done 0.4.2 Nick
> Allen*
> * METRON-1289 To Do Ryan
> Merriman https://issues.apache.org/jira/browse/METRON-1289
> <https://issues.apache.org/jira/browse/METRON-1289>*
> * METRON-1309 Done Jon
> Zeolla https://issues.apache.org/jira/browse/METRON-1309
> <https://issues.apache.org/jira/browse/METRON-1309>*
> * METRON-1310 Done 0.4.2 Nick
> Allen*
> * METRON-1275 Done Jon
> Zeolla https://issues.apache.org/jira/browse/METRON-1275
> <https://issues.apache.org/jira/browse/METRON-1275>*
> * METRON-1295 Done 0.4.2 Nick
> Allen*
> * METRON-1307 Done Otto
> Fowler https://issues.apache.org/jira/browse/METRON-1307
> <https://issues.apache.org/jira/browse/METRON-1307>*
> * METRON-1296 Done 0.4.2 Nick
> Allen*
> * METRON-1281 To Do Ryan
> Merriman https://issues.apache.org/jira/browse/METRON-1281
> <https://issues.apache.org/jira/browse/METRON-1281>*
> * METRON-1287 Done 0.4.2 Nick
> Allen*
> * METRON-1267 To Do
> Unassigned https://issues.apache.org/jira/browse/METRON-1267
> <https://issues.apache.org/jira/browse/METRON-1267>*
> * METRON-1283 Done Anand
> Subramanian https://issues.apache.org/jira/browse/METRON-1283
> <https://issues.apache.org/jira/browse/METRON-1283>*
> * METRON-1254 To Do
> Unassigned https://issues.apache.org/jira/browse/METRON-1254
> <https://issues.apache.org/jira/browse/METRON-1254>*
> * METRON-1261 Done Jon
> Zeolla https://issues.apache.org/jira/browse/METRON-1261
> <https://issues.apache.org/jira/browse/METRON-1261>*
> * METRON-1284 Done Justin
> Leet https://issues.apache.org/jira/browse/METRON-1284
> <https://issues.apache.org/jira/browse/METRON-1284>*
> * METRON-1270 Done Artem
> Ervits https://issues.apache.org/jira/browse/METRON-1270
> <https://issues.apache.org/jira/browse/METRON-1270>*
> * METRON-1272 In Progress Justin
> Leet https://issues.apache.org/jira/browse/METRON-1272
> <https://issues.apache.org/jira/browse/METRON-1272>*
> * METRON-1224 Done
> RaghuMitra https://issues.apache.org/jira/browse/METRON-1224
> <https://issues.apache.org/jira/browse/METRON-1224>*
> * METRON-1280 To Do
> Unassigned https://issues.apache.org/jira/browse/METRON-1280
> <https://issues.apache.org/jira/browse/METRON-1280>*
> * METRON-1243 To Do
> Unassigned https://issues.apache.org/jira/browse/METRON-1243
> <https://issues.apache.org/jira/browse/METRON-1243>*
> * METRON-1196 Done Matt
> Foley https://issues.apache.org/jira/browse/METRON-1196
> <https://issues.apache.org/jira/browse/METRON-1196>*
> * METRON-1278 Done 0.4.2 Matt
> Foley*
> * METRON-1274 To Do
> Unassigned https://issues.apache.org/jira/browse/METRON-1274
> <https://issues.apache.org/jira/browse/METRON-1274>*
> * METRON-1266 Done 0.4.2 Nick
> Allen*
> * METRON-1260 Done 0.4.2 Nick
> Allen*
> * METRON-1251 Done Jon
> Zeolla https://issues.apache.org/jira/browse/METRON-1251
> <https://issues.apache.org/jira/browse/METRON-1251>*
> * METRON-1241 To Do
> Unassigned https://issues.apache.org/jira/browse/METRON-1241
> <https://issues.apache.org/jira/browse/METRON-1241>*
> * METRON-1267 To Do
> Unassigned https://issues.apache.org/jira/browse/METRON-1267
> <https://issues.apache.org/jira/browse/METRON-1267>*
> * METRON-1262 To Do Ryan
> Merriman https://issues.apache.org/jira/browse/METRON-1262
> <https://issues.apache.org/jira/browse/METRON-1262>*
> * METRON-1263 Done Anand
> Subramanian https://issues.apache.org/jira/browse/METRON-1263
> <https://issues.apache.org/jira/browse/METRON-1263>*
> * METRON-1255 To Do
> Unassigned https://issues.apache.org/jira/browse/METRON-1255
> <https://issues.apache.org/jira/browse/METRON-1255>*
> * METRON-1249 Done 0.4.1 Nick
> Allen https://issues.apache.org/jira/browse/METRON-1249
> <https://issues.apache.org/jira/browse/METRON-1249>*
> * METRON-1237 To Do Artem
> Ervits https://issues.apache.org/jira/browse/METRON-1237
> <https://issues.apache.org/jira/browse/METRON-1237>*
> * METRON-1240 Done Artem
> Ervits https://issues.apache.org/jira/browse/METRON-1240
> <https://issues.apache.org/jira/browse/METRON-1240>*
> * METRON-1226 Done 0.4.2 Nick
> Allen*
> * METRON-1081 Done James
> Sirota https://issues.apache.org/jira/browse/METRON-1081
> <https://issues.apache.org/jira/browse/METRON-1081>*
> * METRON-1123 Done
> RaghuMitra https://issues.apache.org/jira/browse/METRON-1123
> <https://issues.apache.org/jira/browse/METRON-1123>*
> * METRON-1223 Done
> RaghuMitra https://issues.apache.org/jira/browse/METRON-1223
> <https://issues.apache.org/jira/browse/METRON-1223>*
> * METRON-1083 Done
> RaghuMitra https://issues.apache.org/jira/browse/METRON-1083
> <https://issues.apache.org/jira/browse/METRON-1083>*
> * METRON-1232 Done
> RaghuMitra https://issues.apache.org/jira/browse/METRON-1232
> <https://issues.apache.org/jira/browse/METRON-1232>*
> * METRON-1247 Done Justin
> Leet https://issues.apache.org/jira/browse/METRON-1247
> <https://issues.apache.org/jira/browse/METRON-1247>*
> * METRON-1235 To Do
> Unassigned https://issues.apache.org/jira/browse/METRON-1235
> <https://issues.apache.org/jira/browse/METRON-1235>*
> * METRON-1234 Done Artem
> Ervits https://issues.apache.org/jira/browse/METRON-1234
> <https://issues.apache.org/jira/browse/METRON-1234>*
> * METRON-1222 Done Artem
> Ervits https://issues.apache.org/jira/browse/METRON-1222
> <https://issues.apache.org/jira/browse/METRON-1222>*
> * METRON-1220 In Progress Justin
> Leet https://issues.apache.org/jira/browse/METRON-1220
> <https://issues.apache.org/jira/browse/METRON-1220>*
> * METRON-1229 To Do
> Unassigned https://issues.apache.org/jira/browse/METRON-1229
> <https://issues.apache.org/jira/browse/METRON-1229>*
> * METRON-1228 Done
> Unassigned https://issues.apache.org/jira/browse/METRON-1228
> <https://issues.apache.org/jira/browse/METRON-1228>*
> * METRON-1218 To Do Ryan
> Merriman https://issues.apache.org/jira/browse/METRON-1218
> <https://issues.apache.org/jira/browse/METRON-1218>*
> * METRON-1161 In Progress Ryan
> Merriman https://issues.apache.org/jira/browse/METRON-1161
> <https://issues.apache.org/jira/browse/METRON-1161>*
> * METRON-1209 To Do
> Unassigned https://issues.apache.org/jira/browse/METRON-1209
> <https://issues.apache.org/jira/browse/METRON-1209>*
> * METRON-1059 Done Next + 1
> Unassigned https://issues.apache.org/jira/browse/METRON-1059
> <https://issues.apache.org/jira/browse/METRON-1059>*
> * METRON-1204 To Do Ryan
> Merriman https://issues.apache.org/jira/browse/METRON-1204
> <https://issues.apache.org/jira/browse/METRON-1204>*
> * METRON-1052 Done Casey
> Stella https://issues.apache.org/jira/browse/METRON-1052
> <https://issues.apache.org/jira/browse/METRON-1052>*
> * METRON-632 In Progress Tomas
> Zezula https://issues.apache.org/jira/browse/METRON-632
> <https://issues.apache.org/jira/browse/METRON-632>*
> * METRON-1194 Done 0.4.2 Nick
> Allen*
> * METRON-1055 To Do Laurens
> Vets https://issues.apache.org/jira/browse/METRON-1055
> <https://issues.apache.org/jira/browse/METRON-1055>*
> * METRON-1079 Done Otto
> Fowler https://issues.apache.org/jira/browse/METRON-1079
> <https://issues.apache.org/jira/browse/METRON-1079>*
> * METRON-1085 To Do Ryan
> Merriman https://issues.apache.org/jira/browse/METRON-1085
> <https://issues.apache.org/jira/browse/METRON-1085>*
> * METRON-1208 To Do Ryan
> Merriman https://issues.apache.org/jira/browse/METRON-1208
> <https://issues.apache.org/jira/browse/METRON-1208>*
> * METRON-1207 To Do Ryan
> Merriman https://issues.apache.org/jira/browse/METRON-1207
> <https://issues.apache.org/jira/browse/METRON-1207>*
> * METRON-1215 To Do
> Unassigned https://issues.apache.org/jira/browse/METRON-1215
> <https://issues.apache.org/jira/browse/METRON-1215>*
> * METRON-1206 To Do Ryan
> Merriman https://issues.apache.org/jira/browse/METRON-1206
> <https://issues.apache.org/jira/browse/METRON-1206>*
> * METRON-1195 To Do Justin
> Leet https://issues.apache.org/jira/browse/METRON-1195
> <https://issues.apache.org/jira/browse/METRON-1195>*
> * METRON-1189 To Do Ryan
> Merriman https://issues.apache.org/jira/browse/METRON-1189
> <https://issues.apache.org/jira/browse/METRON-1189>*
> * METRON-1156 Done 0.4.2 Nick
> Allen*
> * METRON-1198 Done 0.4.2 Nick
> Allen*
> * METRON-1202 Done Next + 1 Justin
> Leet https://issues.apache.org/jira/browse/METRON-1202
> <https://issues.apache.org/jira/browse/METRON-1202>*
> * METRON-938 Done Next + 1 Justin
> Leet https://issues.apache.org/jira/browse/METRON-938
> <https://issues.apache.org/jira/browse/METRON-938>*
> * METRON-1182 Done
> RaghuMitra https://issues.apache.org/jira/browse/METRON-1182
> <https://issues.apache.org/jira/browse/METRON-1182>*
> * METRON-1188 Done Michael
> Miklavcic https://issues.apache.org/jira/browse/METRON-1188
> <https://issues.apache.org/jira/browse/METRON-1188>*
> * METRON-1191 Done Matt
> Foley https://issues.apache.org/jira/browse/METRON-1191
> <https://issues.apache.org/jira/browse/METRON-1191>*
> * METRON-1063 Done Next + 1 Artem
> Ervits https://issues.apache.org/jira/browse/METRON-1063
> <https://issues.apache.org/jira/browse/METRON-1063>*
> * METRON-1190 In Progress Justin
> Leet https://issues.apache.org/jira/browse/METRON-1190
> <https://issues.apache.org/jira/browse/METRON-1190>*
> * METRON-1187 Done 0.4.2 Nick
> Allen*
> * METRON-1185 To Do
> Unassigned https://issues.apache.org/jira/browse/METRON-1185
> <https://issues.apache.org/jira/browse/METRON-1185>*
> * METRON-1186 To Do Casey
> Stella https://issues.apache.org/jira/browse/METRON-1186
> <https://issues.apache.org/jira/browse/METRON-1186>*
> * METRON-1173 Done Next + 1 Jon
> Zeolla https://issues.apache.org/jira/browse/METRON-1173
> <https://issues.apache.org/jira/browse/METRON-1173>*
> * METRON-1179 To Do
> Unassigned https://issues.apache.org/jira/browse/METRON-1179
> <https://issues.apache.org/jira/browse/METRON-1179>*
> * METRON-1180 To Do
> Unassigned https://issues.apache.org/jira/browse/METRON-1180
> <https://issues.apache.org/jira/browse/METRON-1180>*
> * METRON-1183 Done 0.4.2 Nick
> Allen*
> * METRON-1177 Done 0.4.2 Nick
> Allen*
> * METRON-1158 To Do Justin
> Leet https://issues.apache.org/jira/browse/METRON-1158
> <https://issues.apache.org/jira/browse/METRON-1158>*
> * METRON-1146 Done Anand
> Subramanian https://issues.apache.org/jira/browse/METRON-1146
> <https://issues.apache.org/jira/browse/METRON-1146>*
> * METRON-1176 Done Otto
> Fowler https://issues.apache.org/jira/browse/METRON-1176
> <https://issues.apache.org/jira/browse/METRON-1176>*
> * METRON-1114 Done Ryan
> Merriman https://issues.apache.org/jira/browse/METRON-1114
> <https://issues.apache.org/jira/browse/METRON-1114>*
> * METRON-1167 Done Next + 1 Nick
> Allen https://issues.apache.org/jira/browse/METRON-1167
> <https://issues.apache.org/jira/browse/METRON-1167>*
> * METRON-1171 To Do Casey
> Stella https://issues.apache.org/jira/browse/METRON-1171
> <https://issues.apache.org/jira/browse/METRON-1171>*
>
>
>
> The script is here [1] if anyone wants to run it themselves and grep for
> their own name.
>
> [1]
> https://github.com/nickwallen/metron-commit-stuff/blob/
> master/validate-jira-for-release
>
> On Fri, Dec 15, 2017 at 1:18 PM, Matt Foley <mfoley@hortonworks.com<
> mailto:mfoley@hortonworks.com>> wrote:
>
> > Hi Nick,
> > Good timing, you’ve saved me some work! :-)
> >
> > Origin of the list: The approach was defined before I started managing
> > releases, but I think it’s a reasonable approach. It’s specified in our
> > Release Process document, https://cwiki.apache.org/
> > confluence/display/METRON/Release+Process , Step 5, the last bullet:
> > “The artifacts for a release or RC [include]… A CHANGES file
> > denoting the changes [since the last release].
> > We usually construct this by taking the output of git log | grep
> > METRON | sed 's/\[//g' | sed 's/\]//g' | grep -v "http" and removing the
> > JIRAs from the previous releases (it’s in time sorted order so this is
> > easy).”
> >
> > However, the release manager also has a responsibility to “Monitor and
> > Verify Jiras” (Step 2 and various other places in the doc). Altho the
> doc
> > isn’t explicit about it, I take this responsibility a little farther and
> do
> > the following as part of Jira verification:
> >
> > 1. Extract the jira id’s from the CHANGES file with a simple awk script,
> > put them in a Jira query, and confirm they are all actually marked
> > “Fixed/Resolved” with Fix Version = <the current release> in Jira. If
> > there are exceptions, I dun the contributors to fix it. You’ve seen
> these
> > emails from me in the past couple releases. I don’t just mark them fixed
> > myself, because some contributions (perfectly legitimately) only
> partially
> > address a problem, and just marking a ticket “Fixed” may not be the right
> > thing to do.
> >
> > 2. Query jira for any tickets NOT included in the prior list, that ARE
> > marked fixed in the current release (or later, or “Next”, or similar
> > “future” version constructs). These don’t usually happen (considering
> that
> > Metron contributors mostly live inside github PR processes rather than
> > Jiras) but when they do they also need to be resolved. Cases I’ve seen
> (in
> > Metron and other projects) include:
> > a) If they were indeed fixed in the current release, but not reflected in
> > the commit message, I’ll annotate this fact in the Jira ticket, and
> > hand-edit the CHANGES file for this release. (Doesn’t seem worth trying
> to
> > edit old commit messages, since that mucks up the SHA1’s.)
> > b) If they were fixed in a previous release but mis-marked as to Fix
> > Version in the ticket, fix the ticket.
> > c) If they aren’t really fixed, fix the ticket.
> >
> > 3. Once the set of Fixed tickets is complete and correct, query them for
> > any that have “labels in (backward-incompatible, backwards-compatibility,
> > backwards-incompatible)” -- yes, there shouldn’t be 3 labels, but it’s
> hard
> > to fix because Jira never forgets, so every time someone gets it wrong,
> it
> > adds to the set; fortunately, query completion helps here – OR "Docs
> Text"
> > is not EMPTY. Each of these needs to be looked at carefully, verified,
> and
> > if Doc Text is missing, the contributor and I cooperate to write a couple
> > sentences. The Doc Text then needs to go in the RELEASE_NOTES for the
> > release, in the section “## Non-backward-compatible Changes in this
> > Release, and Upgrade Suggestions”.
> >
> > That’s about it. I guess I should add the above as an appendix doc to
> the
> > Release Process documentation. I’ll do that.
> > Cheers,
> > --Matt
> >
> >
> > On 12/15/17, 9:15 AM, "Nick Allen" <nick@nickallen.org<mailto:nic
> k@nickallen.org>> wrote:
> >
> > Hi Matt -
> >
> > I just updated like 15+ JIRAs of my own JIRAs that I completed and
> > merged,
> > but failed to mark as resolved. All of these will be included in
> > 0.4.2. I
> > updated each to be "fixed" in version 0.4.2 and marked as resolved.
> > Hopefully the next RC will report those as fixed.
> >
> > (Q) Where does the list of JIRAs that get attached to the release
> > originate
> > from? Does it get pulled out of JIRA or do they come from the commit
> > log?
> >
> > My apologies for not staying on top of my JIRAs.
> >
> >
> >
> >
> > On Tue, Dec 12, 2017 at 2:21 PM, Matt Foley <mattf@apache.org
> <ma...@apache.org>> wrote:
> >
> > > Thanks to Jon, Otto, and Nick for looking over RC1. Tonight I’ll
> > fix the
> > > RAT glitch, build RC2, and put it to formal vote.
> > > Regards,
> > > --Matt
> > >
> > > On 12/12/17, 5:14 AM, "Nick Allen" <nick@nickallen.org<mailto:nic
> k@nickallen.org>> wrote:
> > >
> > > RC1 is looking good to me. I validated the MD5s, built Metron,
> > built
> > > the
> > > Bro plugin and reviewed the other artifacts like release notes.
> > >
> > > Running the RAT check on a 'clean' Metron does not produce any
> > errors
> > > for
> > > me. It is only after building Metron, which pulls in
> additional
> > Node
> > > dependencies, does the RAT check fail.
> > >
> > >
> > > On Sun, Dec 10, 2017 at 4:41 PM Matt Foley <mattf@apache.org
> <ma...@apache.org>>
> > wrote:
> > >
> > > > Yes, but let’s see if anyone else find other issues.
> > > >
> > > >
> > > >
> > > > From: Otto Fowler <ottobackwards@gmail.com<mailto:
> ottobackwards@gmail.com>>
> > > > Date: Saturday, December 9, 2017 at 6:16 AM
> > > > To: Matt Foley <mfoley@hortonworks.com<mailto:
> mfoley@hortonworks.com>>, "
> > dev@metron.apache.org<ma...@metron.apache.org>" <
> > > > dev@metron.apache.org<ma...@metron.apache.org>>
> > > > Subject: Re: [DISCUSS] Upcoming Release
> > > >
> > > >
> > > >
> > > > So RC2 then?
> > > >
> > > >
> > > >
> > > > On December 8, 2017 at 20:43:21, Matt Foley (
> > mfoley@hortonworks.com<ma...@hortonworks.com>)
> > > > wrote:
> > > >
> > > > Hah, here it is: https://github.com/apache/metron/pull/743
> > > > “This problem seems to only reproduce when one unrolls a
> > tarball
> > > rather
> > > > than cloning from github.”
> > > >
> > > > Heh, the exclusion at
> > > > https://github.com/apache/metron/blob/master/pom.xml#L351 is
> > still
> > > there,
> > > > but the hashcode in the bundle.css file name has changed from
> > > > a0b6b99c10d9a13dc67e to f56deed131e58bd7ee04. Sigh. Did the
> > version
> > > of Font
> > > > Awesome fonts change?
> > > >
> > > >
> > > > On 12/8/17, 5:26 PM, "Matt Foley" <mattf@apache.org<mailto:
> mattf@apache.org>> wrote:
> > > >
> > > > I remember having trouble with this bundle.css file on the
> last
> > > release,
> > > > but I can’t remember what we did about it. Anybody?
> > > >
> > > > On 12/8/17, 1:41 PM, "Otto Fowler" <ottobackwards@gmail.com<
> mailto:ottobackwards@gmail.com>>
> > wrote:
> > > >
> > > > Steps
> > > >
> > > > - Downloaded tar.gz’s, asc files and KEYS
> > > > - Verified signing of both tar.gz’s
> > > > - searched for rouge 0.4.1 entries
> > > > - verified the main pom.xml
> > > > - built :
> > > >
> > > > mvn clean && time mvn -q -T 2C -DskipTests install && time
> mvn
> > -q -T
> > > > 2C surefire:test@unit-tests && time mvn -q
> > > > surefire:test@integration-tests && time mvn -q test
> --projects
> > > > metron-interface/metron-config && time
> > build_utils/verify_licenses.sh
> > > >
> > > > Found rat error:
> > > >
> > > >
> > > > *****************************************************
> > > > Summary
> > > > -------
> > > > Generated at: 2017-12-08T16:33:27-05:00
> > > >
> > > > Notes: 3
> > > > Binaries: 193
> > > > Archives: 0
> > > > Standards: 75
> > > >
> > > > Apache Licensed: 74
> > > > Generated Documents: 0
> > > >
> > > > JavaDocs are generated, thus a license header is optional.
> > > > Generated files do not require license headers.
> > > >
> > > > 1 Unknown Licenses
> > > >
> > > > *****************************************************
> > > >
> > > > Files with unapproved licenses:
> > > >
> > > >
> > > > /Users/batman/tmp/release_ver/apache-metron-0.4.2-rc1/
> > > metron-interface/metron-alerts/dist/styles.
> > f56deed131e58bd7ee04.bundle.css
> > > >
> > > > *****************************************************
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > *****************************************************
> > > > Summary
> > > > -------
> > > > Generated at: 2017-12-08T16:33:27-05:00
> > > >
> > > > Notes: 3
> > > > Binaries: 193
> > > > Archives: 0
> > > > Standards: 75
> > > >
> > > > Apache Licensed: 74
> > > > Generated Documents: 0
> > > >
> > > > JavaDocs are generated, thus a license header is optional.
> > > > Generated files do not require license headers.
> > > >
> > > > 1 Unknown Licenses
> > > >
> > > > *****************************************************
> > > >
> > > > Files with unapproved licenses:
> > > >
> > > >
> > > >
> > > > /Users/ottofowler/tmp/release_ver/apache-metron-0.4.2-rc1/
> > > metron-interface/metron-alerts/dist/styles.
> > f56deed131e58bd7ee04.bundle.css
> > > >
> > > > *****************************************************
> > > >
> > > >
> > > >
> > > > On December 8, 2017 at 04:34:24, Matt Foley (
> mattf@apache.org<ma...@apache.org>)
> > > wrote:
> > > >
> > > > Colleagues,
> > > > I’ve posted Metron-0.4.2-RC1 and Metron-bro-plugin-kafka-0.1
> to
> > > > https://dist.apache.org/repos/dist/dev/metron/0.4.2-RC1/
> > > >
> > > > Given the complexity of this RC, I’d appreciate if a couple
> > people
> > > would be
> > > > willing to kick the tires before we put it up for a vote.
> > > >
> > > > I will myself be going thru the Verify Build process this
> > weekend,
> > > as I
> > > > won’t be able to do it Friday.
> > > >
> > > > Thanks,
> > > > --Matt
> > > >
> > > >
> > > > On 12/4/17, 2:05 PM, "Zeolla@GMail.com" <zeolla@gmail.com
> <ma...@gmail.com>>
> > wrote:
> > > >
> > > > Can we resolve the conversation regarding the second repo? I
> > was
> > > waiting
> > > > to get more input/preferences from people There's also a
> > > documentation
> > > > update that fixes a few broken Stellar docs that already has
> > aa +1,
> > > I just
> > > > need to merge it.
> > > >
> > > > Jon
> > > >
> > > > On Mon, Dec 4, 2017, 17:01 Casey Stella <cestella@gmail.com
> <ma...@gmail.com>>
> > wrote:
> > > >
> > > > > I would be in favor of a release at this point.
> > > > >
> > > > > On Mon, Dec 4, 2017 at 4:57 PM, Matt Foley <
> mattf@apache.org<ma...@apache.org>
> > >
> > > wrote:
> > > > >
> > > > > > Hey all,
> > > > > > I see METRON-1252 was resolved over the weekend. Shall I
> go
> > > ahead and
> > > > > > start the process with 0.4.2 release?
> > > > > > Does anyone have any commits they feel strongly should go
> > in
> > > before
> > > > 0.4.2
> > > > > > is done, or are we ready to call it good?
> > > > > >
> > > > > > I believe there is consensus the 0.4.2 release should
> > include a
> > > release
> > > > > of
> > > > > > the current state of the metron-bro-plugin-kafka. I will
> > > continue the
> > > > > > discussion in that thread as to the process for
> > accomplishing
> > > that, but
> > > > > > plan on it happening.
> > > > > >
> > > > > > Regards,
> > > > > > --Matt
> > > > > >
> > > > > > On 11/26/17, 6:26 PM, "Matt Foley" <mattf@apache.org
> <ma...@apache.org>>
> > wrote:
> > > > > >
> > > > > > Hope everyone (at least in the U.S.) had a great
> > Thanksgiving
> > > > > holiday.
> > > > > > Regarding status of the release effort, still pending
> > > METRON-1252, so
> > > > > > not making the release branch yet.
> > > > > >
> > > > > > Regards,
> > > > > > --Matt
> > > > > >
> > > > > > On 11/17/17, 1:32 PM, "Matt Foley" <mattf@apache.org
> <ma...@apache.org>>
> > wrote:
> > > > > >
> > > > > > (With release manager hat on)
> > > > > >
> > > > > > The community has proposed a release of Metron in the
> near
> > > > > future,
> > > > > > focusing on Meta-alerts running in Elasticsearch.
> > > > > > Congrats on getting so many of the below already done. At
> > this
> > > > > > point, only METRON-1252, and the discussion of how to
> > handle
> > > joint
> > > > > release
> > > > > > of the Metron bro plugin, remain as gating items for the
> > > release. I
> > > > > > project these will be resolved next week, so let’s
> propose
> > the
> > > > following:
> > > > > >
> > > > > > Sometime next week, after the last bits are done, I’ll
> > start the
> > > > > > release process and create the release branch.
> > > > > >
> > > > > > The proposed new version will be 0.4.2, unless there are
> > backward
> > > > > > incompatible changes that support making it 0.5.0.
> > > > > > Currently there are NO included Jiras labeled
> > > > > > ‘backward-incompatible’, nor having Docs Text indicating
> > so.
> > > > > > ***If anyone knows that some of the commits included
> since
> > 0.4.1
> > > > > > introduce backward incompatibility, please say so now on
> > this
> > > thread,
> > > > and
> > > > > > mark the Jira as such.***
> > > > > >
> > > > > > The 90 or so jiras/commits already in master branch since
> > 0.4.1
> > > > > > are listed below.
> > > > > > Thanks,
> > > > > > --Matt
> > > > > >
> > > > > > METRON-1301 Alerts UI - Sorting on Triage Score
> > Unexpectedly
> > > > > > Filters Some Records (nickwallen) closes
> apache/metron#832
> > > > > > METRON-1294 IP addresses are not formatted correctly in
> > facet
> > > > > > and group results (merrimanr) closes apache/metron#827
> > > > > > METRON-1291 Kafka produce REST endpoint does not work in
> a
> > > > > > Kerberized cluster (merrimanr) closes apache/metron#826
> > > > > > METRON-1290 Only first 10 alerts are update when a
> > MetaAlert
> > > > > > status is changed to inactive (justinleet) closes
> > > apache/metron#842
> > > > > > METRON-1311 Service Check Should Check Elasticsearch
> Index
> > > > > > Templates (nickwallen) closes apache/metron#839
> > > > > > METRON-1289 Alert fields are lost when a MetaAlert is
> > created
> > > > > > (merrimanr) closes apache/metron#824
> > > > > > METRON-1309 Change metron-deployment to pull the plugin
> > from
> > > > > > apache/metron-bro-plugin-kafka (JonZeolla) closes
> > > apache/metron#837
> > > > > > METRON-1310 Template Delete Action Deletes Search Indices
> > > > > > (nickwallen) closes apache/metron#838
> > > > > > METRON-1275: Fix Metron Documentation closes
> > > > > > apache/incubator-metron#833
> > > > > > METRON-1295 Unable to Configure Logging for REST API
> > > > > > (nickwallen) closes apache/metron#828
> > > > > > METRON-1307 Force install of java8 since java9 does not
> > > > > appear
> > > > > > to work with the scripts (brianhurley via ottobackwards)
> > closes
> > > > > > apache/metron#835
> > > > > > METRON-1296 Full Dev Fails to Deploy Index Templates
> > > > > > (nickwallen via cestella) closes
> > apache/incubator-metron#829
> > > > > > METRON-1281 Remove hard-coded indices from the Alerts UI
> > > > > > (merrimanr) closes apache/metron#821
> > > > > > METRON-1287 Full Dev Fails When Installing EPEL
> Repository
> > > > > > (nickwallen) closes apache/metron#820
> > > > > > METRON-1267 Alerts UI returns a 404 when refreshing the
> > > > > > alerts-list page (iraghumitra via merrimanr) closes
> > > apache/metron#819
> > > > > > METRON-1283 Install Elasticsearch template as a part of
> the
> > > > > > mpack startup scripts (anandsubbu via nickwallen) closes
> > > > > apache/metron#817
> > > > > > METRON-1254: Conditionals as map keys do not function in
> > > > > > Stellar closes apache/incubator-metron#801
> > > > > > METRON-1261 Apply bro security patch (JonZeolla via
> > > > > > ottobackwards) closes apache/metron#805
> > > > > > METRON-1284 Remove extraneous dead query in
> > ElasticsearchDao
> > > > > > (justinleet) closes apache/metron#818
> > > > > > METRON-1270: fix for warnings missing @return tag
> argument
> > in
> > > > > > metron-analytics/metron-profiler-common and
> > > metron-profiler-client
> > > > closes
> > > > > > apache/incubator-metron#810
> > > > > > METRON-1272 Hide child alerts from searches and grouping
> if
> > > > > > they belong to meta alerts (justinleet) closes
> > apache/metron#811
> > > > > > METRON-1224 Add time range selection to search control
> > > > > > (iraghumitra via james-sirota) closes apache/metron#796
> > > > > > METRON-1280 0.4.1 -> 0.4.2 missed a couple of projects
> > > > > > (cestella via justinleet) closes apache/metron#816
> > > > > > METRON-1243: Add a REST endpoint which allows us to get a
> > > > > list
> > > > > > of all indice closes apache/incubator-metron#797
> > > > > > METRON-1196 Increment master version number to 0.4.2 for
> > > > > > on-going development (mattf-horton) closes
> > apache/metron#767
> > > > > > METRON-1278 Strip "Build Status" widget from
> root
> > > > > > README.md in site-book build (mattf-horton) closes
> > > apache/metron#815
> > > > > > METRON-1274 Master has failure in
> > > > > > StormControllerIntegrationTest (merrimanr) closes
> > > apache/metron#813
> > > > > > METRON-1266 Profiler - SASL Authentication Failed
> > > > > (nickwallen)
> > > > > > closes apache/metron#809
> > > > > > METRON-1260 Include Alerts UI in Ambari Service Check
> > > > > > (nickwallen) closes apache/metron#804
> > > > > > METRON-1251: Typo and formatting fixes for metron-rest
> > README
> > > > > > closes apache/incubator-metron#800
> > > > > > METRON-1241: Enable the REST API to use a cache for the
> > > > > > zookeeper config similar to the Bolts closes
> > > > apache/incubator-metron#795
> > > > > > METRON-1267 Alerts UI returns a 404 when refreshing the
> > > > > > alerts-list page (merrimanr) closes apache/metron#808
> > > > > > METRON-1262 Unable to add comment for a alert in a
> > meta-alert
> > > > > > (merrimanr) closes apache/metron#806
> > > > > > METRON-1263 Start Alerts UI service after Metron REST
> > > > > > (anandsubbu via nickwallen) closes apache/metron#807
> > > > > > METRON-1255 MetaAlert search is not filtering on status
> > > > > > (merrimanr) closes apache/metron#802
> > > > > > METRON-1249 Improve Metron MPack Service Checks
> > (nickwallen)
> > > > > > closes apache/metron#799
> > > > > > METRON-1237 address javadoc warnings in
> metron-maas-common
> > > > > > (dbist via james-sirota) closes apache/metron#792
> > > > > > METRON-1240 address javadoc warnings in metron-platform
> and
> > > > > > metron-analytics (dbist via james-sirota) closes
> > > apache/metron#794
> > > > > > METRON-1226 Searching Can Errantly Query the Wrong
> Indices
> > > > > > (nickwallen) closes apache/metron#793
> > > > > > METRON-1081 Fix Alerts and Ops UI Notices file
> > (james-sirota)
> > > > > > closes apache/metron#682
> > > > > > METRON-1123 Add group by option using faceted search
> > > > > > capabilities of metron-rest-api (iraghumitra via
> > james-sirota)
> > > closes
> > > > > > apache/metron#768
> > > > > > METRON-1223 Add support to add comments for alerts
> > > > > > (iraghumitra via james-sirota) closes apache/metron#788
> > > > > > METRON-1083 Add filters using faceted search capabilities
> > of
> > > > > > metron-rest-api (iraghumitra via james-sirota) closes
> > > apache/metron#710
> > > > > > METRON-1232 Alert status changes are not reflected in
> list
> > > > > > view (iraghumitra via merrimanr) closes apache/metron#787
> > > > > > METRON-1247 REST search and findOne endpoints return
> > > > > > unexpected or incorrect results for guids (justinleet)
> > closes
> > > > > > apache/metron#798
> > > > > > METRON-1235: Document the properties pulled from the
> global
> > > > > > configuration closes apache/incubator-metron#791
> > > > > > METRON-1234: fix for WARNING 'dependencies.dependency.(
> > > > > > groupId:artifactId:type:classifier)' must be unique:
> > > > > > org.apache.hadoop:hadoop-yarn-api:jar (dbist via
> mmiklavc)
> > > closes
> > > > > > apache/metron#790
> > > > > > METRON-1222: fix warning for The expression
> > ${parent.version}
> > > > > > is deprecated. Please use ${project.parent.version}
> > instead.
> > > (dbist via
> > > > > > mmiklavc) closes apache/metron#782
> > > > > > METRON-1220 Create documentation around alert nested
> field
> > > > > > (justinleet) closes apache/metron#780
> > > > > > METRON-1229 Management UI type is part of the
> declarations
> > of
> > > > > > 2 modules (merrimanr) closes apache/metron#784
> > > > > > METRON-1228: Configuration Management PUSH immediately
> does
> > > > > > DUMP after (mmiklavc via mmiklavc) closes
> apache/metron#783
> > > > > > METRON-1218 Metron REST should return better error
> messages
> > > > > > (merrimanr) closes apache/metron#779
> > > > > > METRON-1161 Add ability to edit parser command line
> options
> > > > > in
> > > > > > the management UI (merrimanr) closes apache/metron#737
> > > > > > METRON-1209: Make stellar repl take logging properties,
> > like
> > > > > > other CLI apps in metron closes
> apache/incubator-metron#772
> > > > > > METRON-1059 address checkstyle warning AvoidStarImport in
> > > > > > metron-stellar (dbist via ottobackwards) closes
> > apache/metron#664
> > > > > > METRON-1204 UI does not time out after being idle, but
> > stops
> > > > > > functioning (merrimanr) closes apache/metron#771
> > > > > > METRON-1052: Add forensic similarity hash functions to
> > > > > Stellar
> > > > > > closes apache/incubator-metron#781
> > > > > > METRON-632: Added validation of "shew.enrichmentType" and
> > > > > > "shew.keyColumns" closes apache/incubator-metron#732
> > > > > > METRON-1194 Add Profiler Debug Functions to Profiler
> README
> > > > > > (nickwallen via ottobackwards) closes apache/metron#765
> > > > > > METRON-1055 Metron 0.4.0 manual installation guide for
> > CentOS
> > > > > > 6 updates (lvets via ottobackwards) closes
> > apache/metron#661
> > > > > > METRON-1079 STELLAR NaN should be a keyword
> (ottobackwards)
> > > > > > closes apache/metron#681
> > > > > > METRON-1085 Add REST endpoint to save a user profile for
> > the
> > > > > > Alerts UI (merrimanr) closes apache/metron#694
> > > > > > METRON-1208 MPack for Alerts UI (merrimanr) closes
> > > > > > apache/metron#778
> > > > > > METRON-1207 Make RPMs for Alerts UI (merrimanr) closes
> > > > > > apache/metron#777
> > > > > > METRON-1215 Fix link to RPMs chapter (DimDroll via
> > > > > justinleet)
> > > > > > closes apache/metron#776
> > > > > > METRON-1206 Make alerts UI conform to ops UI for install
> > > > > > (merrimanr) closes apache/metron#773
> > > > > > METRON-1195 Meta alerts improperly handle updates to
> > > > > non-alert
> > > > > > fields (justinleet) closes apache/metron#766
> > > > > > METRON-1189 Add alert escalation to the Alerts UI
> > (merrimanr)
> > > > > > closes apache/metron#762
> > > > > > METRON-1156 Simulate Triage Rules in the Stellar REPL
> > > > > > (nickwallen) closes apache/metron#733
> > > > > > METRON-1198 Pycapa - No such configuration property
> > > > > > 'sasl.kerberos.principal' (nickwallen) closes
> > apache/metron#769
> > > > > > METRON-1202 ElasticsearchDao Has extraneous sleep call
> > > > > > (justinleet) closes apache/metron#770
> > > > > > METRON-938 "service metron-rest start <password>" does
> not
> > > > > > work on CentOS 7. (justinleet) closes apache/metron#757
> > > > > > METRON-1182 Refactor Code in alert list to accommodate
> new
> > > > > > view types (iraghumitra via merrimanr) closes
> > apache/metron#756
> > > > > > METRON-1188: Ambari global configuration management
> > > > > (mmiklavc)
> > > > > > closes apache/metron#760
> > > > > > METRON-1191 update public web site to point at 0.4.1 new
> > > > > > release (mattf-horton) closes apache/metron#764
> > > > > > METRON-1063 address javadoc warnings in metron-stellar
> > (dbist
> > > > > > via ottobackwards) closes apache/metron#668
> > > > > > METRON-1190 Fix Meta Alert Type handling in calculation
> of
> > > > > > scores (justinleet) closes apache/metron#763
> > > > > > METRON-1187 Indexing/Profiler Kafka ACL Groups Not Setup
> > > > > > Correctly (nickwallen) closes apache/metron#759
> > > > > > METRON-1185: Stellar REPL does not work on a kerberized
> > > > > > cluster when calling functions interacting with HBase
> > closes
> > > > > > apache/incubator-metron#755
> > > > > > METRON-1186: Profiler Functions use classutils from
> shaded
> > > > > > storm closes apache/incubator-metron#758
> > > > > > METRON-1173: Fix pointers to old stellar docs closes
> > > > > > apache/incubator-metron#746
> > > > > > METRON-1179: Make STATS_ADD to take a list closes
> > > > > > apache/incubator-metron#750
> > > > > > METRON-1180: Make Stellar Shell accept zookeeper quorum
> as
> > a
> > > > > > CSV list and not require a port closes
> > > apache/incubator-metron#751
> > > > > > METRON-1183 Improve KDC Setup Instructions (nickwallen)
> > > > > closes
> > > > > > apache/metron#753
> > > > > > METRON-1177 Stale running topologies seen
> > post-kerberization
> > > > > > and cause exceptions (nickwallen) closes
> apache/metron#748
> > > > > > METRON-1158 Build backend for grouping alerts into meta
> > > > > alerts
> > > > > > (justinleet) closes apache/metron#734
> > > > > > METRON-1146: Add ability to parse JSON string into
> > JSONObject
> > > > > > for stellar closes apache/incubator-metron#727
> > > > > > METRON-1176 REST: HDFS Service should support setting
> > > > > > permissions on files when writing (ottobackwards) closes
> > > > > apache/metron#749
> > > > > > METRON-1114 Add group by capabilities to search REST
> > endpoint
> > > > > > (merrimanr) closes apache/metron#702
> > > > > > METRON-1167 Define Session Specific Global Configuration
> > > > > > Values in the REPL (nickwallen) closes apache/metron#740
> > > > > > METRON-1171: Better validation for the SUBSTRING stellar
> > > > > > function closes apache/incubator-metron#745
> > > > > >
> > > > > >
> > > > > >
> > > > > > On 11/17/17, 11:59 AM, "Nick Allen" <nick@nickallen.org
> <ma...@nickallen.org>>
> > wrote:
> > > > > >
> > > > > > I just wanted to send an update on where we are at. We've
> > > > > > gotten a lot
> > > > > > done here recently as you can see below.
> > > > > >
> > > > > > ✓ DONE (1) First, METRON-1289 needs to go in. This one
> was
> > > > > > a fairly big
> > > > > > effort and I am hearing that we are pretty close.
> > > > > >
> > > > > > ✓ DONE (2) METRON-1294 fixes an issue in how field types
> > > > > are
> > > > > > looked-up.
> > > > > >
> > > > > > ✓ DONE (3) METRON-1290 is next. While this may have been
> > > > > > fixed in
> > > > > > M-1289, there may be some test cases we want from this
> PR.
> > > > > >
> > > > > > ✓ DONE (4) METRON-1301 addresses a problem with the
> sorting
> > > > > > logic.
> > > > > >
> > > > > > ✓ DONE (5) METRON-1291 fixes an issue with escalation of
> > > > > > metaalerts.
> > > > > >
> > > > > > (6) That leads us to Raghu's UI work in METRON-1252. This
> > > > > > introduces the
> > > > > > UI bits that depend on all the previous backend work.
> > > > > >
> > > > > > (7) At this point, we should have our best effort at
> > > > > running
> > > > > > Metaalerts
> > > > > > on Elasticsearch 2.x. I propose that we cut a release
> here.
> > > > > >
> > > > > > (8) After we cut the release, we can introduce the work
> for
> > > > > > ES 5.x in
> > > > > > METRON-939. I know we will need lots of help testing and
> > > > > > reviewing this
> > > > > > one.
> > > > > >
> > > > > >
> > > > > >
> > > > > > We also have an outstanding question that needs resolved
> > > > > > BEFORE we
> > > > > > release. We need to come to a consensus on how to release
> > > > > > having moved our
> > > > > > Bro Plugin to a separate repo. I don't think we've heard
> > > > > from
> > > > > > everyone on
> > > > > > this. I'd urge everyone to chime in so we can choose a
> path
> > > > > > forward.
> > > > > >
> > > > > > If anyone is totally confused in regards to that
> > discussion,
> > > > > I
> > > > > > can try and
> > > > > > send an options summary again as a separate discuss
> thread.
> > > > > > The original
> > > > > > chain was somewhere around here [1].
> > > > > >
> > > > > > [1]
> > > > > > https://lists.apache.org/thread.html/
> > > > > > 54a4474881b97e559df24728b3a0e9
> 23a58345a282451085eef832ef@%
> > > > > > 3Cdev.metron.apache.org<http://3Cdev.metron.apache.org
> >%3E
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Wed, Nov 15, 2017 at 10:04 AM, Nick Allen <
> > > > > > nick@nickallen.org<ma...@nickallen.org>> wrote:
> > > > > >
> > > > > > > Hi Guys -
> > > > > > >
> > > > > > > I want to follow-up on this discussion. It sounds like
> > > > > most
> > > > > > people are in
> > > > > > > agreement with the general approach.
> > > > > > >
> > > > > > > A lot of people have been working hard on Metaalerts
> and
> > > > > > Elasticsearch. I
> > > > > > > have checked-in with those doing the heavy lifting and
> > have
> > > > > > compiled a more
> > > > > > > detailed plan based on where we are at now. To the best
> > of
> > > > > > my knowledge
> > > > > > > here is the plan of attack for finishing out this
> effort.
> > > > > > >
> > > > > > > (1) First, METRON-1289 needs to go in. This one was a
> > > > > > fairly big effort
> > > > > > > and I am hearing that we are pretty close.
> > > > > > >
> > > > > > > (2) METRON-1294 fixes an issue in how field types are
> > > > > > looked-up.
> > > > > > >
> > > > > > > (3) METRON-1290 is next. While this may have been fixed
> > > > > > in M-1289,
> > > > > > > there may be some test cases we want from this PR.
> > > > > > >
> > > > > > > (4) METRON-1301 addresses a problem with the sorting
> > > > > logic.
> > > > > > >
> > > > > > > (5) METRON-1291 fixes an issue with escalation of
> > > > > > metaalerts.
> > > > > > >
> > > > > > > (6) That leads us to Raghu's UI work in METRON-1252.
> > > > > This
> > > > > > introduces
> > > > > > > the UI bits that depend on all the previous backend
> work.
> > > > > > >
> > > > > > > (7) At this point, we should have our best effort at
> > > > > > running Metaalerts
> > > > > > > on Elasticsearch 2.x. I propose that we cut a release
> > here.
> > > > > > >
> > > > > > > (8) After we cut the release, we can introduce the work
> > > > > > for ES 5.x in
> > > > > > > METRON-939. I know we will need lots of help testing
> and
> > > > > > reviewing this
> > > > > > > one.
> > > > > > >
> > > > > > > Please correct me if I am wrong. I will try and send
> out
> > > > > > updates as we
> > > > > > > make progress.
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > On Mon, Nov 6, 2017 at 1:03 PM, Zeolla@GMail.com <
> > > > > > zeolla@gmail.com<ma...@gmail.com>> wrote:
> > > > > > >
> > > > > > >> I agree, I think it's very reasonable to move in line
> > with
> > > > > > Nick's
> > > > > > >> proposal. I would also suggest that we outline what
> the
> > > > > > target versions
> > > > > > >> would be to add in the METRON-777 components, since it
> > has
> > > > > > been functional
> > > > > > >> for a very long time but not reviewed and has some
> > really
> > > > > > rockstar
> > > > > > >> improvements.
> > > > > > >>
> > > > > > >> Jon
> > > > > > >>
> > > > > > >> On Mon, Nov 6, 2017 at 12:56 PM Otto Fowler <
> > > > > > ottobackwards@gmail.com<ma...@gmail.com>>
> > > > > > >> wrote:
> > > > > > >>
> > > > > > >> > I think the ES cutover should be the start of the
> > 0.5.x
> > > > > > series, and we
> > > > > > >> > continue on with 0.4.x for the
> > > > > > >> > metadata improvements etc. We could chose to focus
> > > > > > 0.5.x’s first
> > > > > > >> releases
> > > > > > >> > on not only ES but
> > > > > > >> > getting a handle on kibana and the mpack situation
> as
> > > > > > well.
> > > > > > >> >
> > > > > > >> >
> > > > > > >> >
> > > > > > >> >
> > > > > > >> > On November 6, 2017 at 12:48:45, Michael Miklavcic (
> > > > > > >> > michael.miklavcic@gmail.com<mailto:
> michael.miklavcic@gmail.com>) wrote:
> > > > > > >> >
> > > > > > >> > I agree with your proposal, Nick. I think having a
> > > > > > stabilizing release
> > > > > > >> > prior to upgrading ES/Kibana makes sense.
> > > > > > >> >
> > > > > > >> > On Mon, Nov 6, 2017 at 9:16 AM, Nick Allen <
> > > > > > nick@nickallen.org<ma...@nickallen.org>> wrote:
> > > > > > >> >
> > > > > > >> > > I would like to start a discussion around upcoming
> > > > > > releases. We have a
> > > > > > >> > > couple separate significant tracks of work that we
> > > > > need
> > > > > > to reconcile
> > > > > > >> in
> > > > > > >> > our
> > > > > > >> > > release schedule.
> > > > > > >> > >
> > > > > > >> > > (1) We have had (and have in review) a good number
> > of
> > > > > > bug fixes
> > > > > > >> required
> > > > > > >> > to
> > > > > > >> > > support Metaalerts on the existing Elasticsearch
> 2.x
> > > > > > infrastructure.
> > > > > > >> > >
> > > > > > >> > >
> > > > > > >> > > (2) We also have ongoing work to upgrade our
> > > > > > infrastructure to
> > > > > > >> > > Elasticsearch 5.x, which will not be backwards
> > > > > > compatible.
> > > > > > >> > >
> > > > > > >> > >
> > > > > > >> > > I would like to see a release that has our best
> work
> > > > > on
> > > > > > ES 2.x before
> > > > > > >> we
> > > > > > >> > > migrate to 5.x. I would propose the following.
> > > > > > >> > >
> > > > > > >> > > Release N+1: Introduce Metaalerts running on ES
> 2.x
> > > > > > >> > >
> > > > > > >> > > Release N+2: Cut-over to ES 5.x
> > > > > > >> > >
> > > > > > >> > >
> > > > > > >> > > (Q) Is it worth cutting a separate release for ES
> > 2.x?
> > > > > > Is there a
> > > > > > >> better
> > > > > > >> > > way to handle the cut-over to 5.x?
> > > > > > >> > >
> > > > > > >> >
> > > > > > >> --
> > > > > > >>
> > > > > > >> Jon
> > > > > > >>
> > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > >
> > > > --
> > > >
> > > > Jon
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > >
> > >
> > >
> > >
> >
> >
> >
>
>
Re: [DISCUSS] Upcoming Release
Posted by Matt Foley <mf...@hortonworks.com>.
Perhaps under “build_utils” we should add a subdirectory for “release_utils”.
From: Casey Stella <ce...@gmail.com>
Date: Friday, December 15, 2017 at 10:50 AM
To: "dev@metron.apache.org" <de...@metron.apache.org>
Cc: Matt Foley <mf...@hortonworks.com>
Subject: Re: [DISCUSS] Upcoming Release
That script seems great, nick! Perhaps we should adjust the wiki around releases to point to it? Thoughts?
On Fri, Dec 15, 2017 at 1:47 PM, Nick Allen <ni...@nickallen.org>> wrote:
Thanks, Matt.
Maybe you already have something that does this. I wrote a quick script
that validates each JIRA since the last release tag to make sure they are
marked "Done" and with the correct fix version. I would expect that for
the next release, each JIRA should have status="Done", fix-version="0.4.2".
Unless I am mistaken, we have quite a few that need cleaned up. In the
following output, any line that has a URL indicates that a fix is needed.
Or it least, **I think** a fix is needed.
To the community.... If you see your name with a URL next to it, it would
be great if you could follow that link and fix the JIRA. Otherwise, I will
volunteer to help clean some of these up should some not get addressed.
*$ ./validate-jira-for-release*
*Cloning into 'metron-0.4.2'...*
*remote: Counting objects: 35046, done.*
*remote: Compressing objects: 100% (13698/13698), done.*
*remote: Total 35046 (delta 15708), reused 31645 (delta 12822)*
*Receiving objects: 100% (35046/35046), 53.05 MiB | 6.48 MiB/s, done.*
*Resolving deltas: 100% (15708/15708), done.*
*Fetching origin*
* JIRA STATUS FIX VERSION
ASSIGNEE FIX*
* METRON-1345 Done Michael
Miklavcic https://issues.apache.org/jira/browse/METRON-1345
<https://issues.apache.org/jira/browse/METRON-1345>*
* METRON-1349 Done Next + 1 Nick
Allen https://issues.apache.org/jira/browse/METRON-1349
<https://issues.apache.org/jira/browse/METRON-1349>*
* METRON-1343 Done
Mohan https://issues.apache.org/jira/browse/METRON-1343
<https://issues.apache.org/jira/browse/METRON-1343>*
* METRON-1306 To Do
Unassigned https://issues.apache.org/jira/browse/METRON-1306
<https://issues.apache.org/jira/browse/METRON-1306>*
* METRON-1341 Done Simon Elliston
Ball https://issues.apache.org/jira/browse/METRON-1341
<https://issues.apache.org/jira/browse/METRON-1341>*
* METRON-1313 Done Jon
Zeolla https://issues.apache.org/jira/browse/METRON-1313
<https://issues.apache.org/jira/browse/METRON-1313>*
* METRON-1346 Done Otto
Fowler https://issues.apache.org/jira/browse/METRON-1346
<https://issues.apache.org/jira/browse/METRON-1346>*
* METRON-1336 Done 0.4.2 Nick
Allen*
* METRON-1335 Done Anand
Subramanian https://issues.apache.org/jira/browse/METRON-1335
<https://issues.apache.org/jira/browse/METRON-1335>*
* METRON-1308 Done Jon
Zeolla https://issues.apache.org/jira/browse/METRON-1308
<https://issues.apache.org/jira/browse/METRON-1308>*
* METRON-1338 Done 0.4.2 Nick
Allen*
* METRON-1286 To Do 0.4.2
Unassigned https://issues.apache.org/jira/browse/METRON-1286
<https://issues.apache.org/jira/browse/METRON-1286>*
* METRON-1334 Done 0.4.2 Nick
Allen*
* METRON-1277 Done Otto
Fowler https://issues.apache.org/jira/browse/METRON-1277
<https://issues.apache.org/jira/browse/METRON-1277>*
* METRON-1239 To Do
Unassigned https://issues.apache.org/jira/browse/METRON-1239
<https://issues.apache.org/jira/browse/METRON-1239>*
* METRON-1328 Done Anand
Subramanian https://issues.apache.org/jira/browse/METRON-1328
<https://issues.apache.org/jira/browse/METRON-1328>*
* METRON-1333 Done Otto
Fowler https://issues.apache.org/jira/browse/METRON-1333
<https://issues.apache.org/jira/browse/METRON-1333>*
* METRON-1252 Done
RaghuMitra https://issues.apache.org/jira/browse/METRON-1252
<https://issues.apache.org/jira/browse/METRON-1252>*
* METRON-1316 To Do Next + 1
Unassigned https://issues.apache.org/jira/browse/METRON-1316
<https://issues.apache.org/jira/browse/METRON-1316>*
* METRON-1088 Done Jon
Zeolla https://issues.apache.org/jira/browse/METRON-1088
<https://issues.apache.org/jira/browse/METRON-1088>*
* METRON-1319 To Do Ryan
Merriman https://issues.apache.org/jira/browse/METRON-1319
<https://issues.apache.org/jira/browse/METRON-1319>*
* METRON-1321 To Do
Unassigned https://issues.apache.org/jira/browse/METRON-1321
<https://issues.apache.org/jira/browse/METRON-1321>*
* METRON-1301 Done 0.4.2 Nick
Allen*
* METRON-1294 To Do Ryan
Merriman https://issues.apache.org/jira/browse/METRON-1294
<https://issues.apache.org/jira/browse/METRON-1294>*
* METRON-1291 To Do Ryan
Merriman https://issues.apache.org/jira/browse/METRON-1291
<https://issues.apache.org/jira/browse/METRON-1291>*
* METRON-1290 Done Justin
Leet https://issues.apache.org/jira/browse/METRON-1290
<https://issues.apache.org/jira/browse/METRON-1290>*
* METRON-1311 Done 0.4.2 Nick
Allen*
* METRON-1289 To Do Ryan
Merriman https://issues.apache.org/jira/browse/METRON-1289
<https://issues.apache.org/jira/browse/METRON-1289>*
* METRON-1309 Done Jon
Zeolla https://issues.apache.org/jira/browse/METRON-1309
<https://issues.apache.org/jira/browse/METRON-1309>*
* METRON-1310 Done 0.4.2 Nick
Allen*
* METRON-1275 Done Jon
Zeolla https://issues.apache.org/jira/browse/METRON-1275
<https://issues.apache.org/jira/browse/METRON-1275>*
* METRON-1295 Done 0.4.2 Nick
Allen*
* METRON-1307 Done Otto
Fowler https://issues.apache.org/jira/browse/METRON-1307
<https://issues.apache.org/jira/browse/METRON-1307>*
* METRON-1296 Done 0.4.2 Nick
Allen*
* METRON-1281 To Do Ryan
Merriman https://issues.apache.org/jira/browse/METRON-1281
<https://issues.apache.org/jira/browse/METRON-1281>*
* METRON-1287 Done 0.4.2 Nick
Allen*
* METRON-1267 To Do
Unassigned https://issues.apache.org/jira/browse/METRON-1267
<https://issues.apache.org/jira/browse/METRON-1267>*
* METRON-1283 Done Anand
Subramanian https://issues.apache.org/jira/browse/METRON-1283
<https://issues.apache.org/jira/browse/METRON-1283>*
* METRON-1254 To Do
Unassigned https://issues.apache.org/jira/browse/METRON-1254
<https://issues.apache.org/jira/browse/METRON-1254>*
* METRON-1261 Done Jon
Zeolla https://issues.apache.org/jira/browse/METRON-1261
<https://issues.apache.org/jira/browse/METRON-1261>*
* METRON-1284 Done Justin
Leet https://issues.apache.org/jira/browse/METRON-1284
<https://issues.apache.org/jira/browse/METRON-1284>*
* METRON-1270 Done Artem
Ervits https://issues.apache.org/jira/browse/METRON-1270
<https://issues.apache.org/jira/browse/METRON-1270>*
* METRON-1272 In Progress Justin
Leet https://issues.apache.org/jira/browse/METRON-1272
<https://issues.apache.org/jira/browse/METRON-1272>*
* METRON-1224 Done
RaghuMitra https://issues.apache.org/jira/browse/METRON-1224
<https://issues.apache.org/jira/browse/METRON-1224>*
* METRON-1280 To Do
Unassigned https://issues.apache.org/jira/browse/METRON-1280
<https://issues.apache.org/jira/browse/METRON-1280>*
* METRON-1243 To Do
Unassigned https://issues.apache.org/jira/browse/METRON-1243
<https://issues.apache.org/jira/browse/METRON-1243>*
* METRON-1196 Done Matt
Foley https://issues.apache.org/jira/browse/METRON-1196
<https://issues.apache.org/jira/browse/METRON-1196>*
* METRON-1278 Done 0.4.2 Matt
Foley*
* METRON-1274 To Do
Unassigned https://issues.apache.org/jira/browse/METRON-1274
<https://issues.apache.org/jira/browse/METRON-1274>*
* METRON-1266 Done 0.4.2 Nick
Allen*
* METRON-1260 Done 0.4.2 Nick
Allen*
* METRON-1251 Done Jon
Zeolla https://issues.apache.org/jira/browse/METRON-1251
<https://issues.apache.org/jira/browse/METRON-1251>*
* METRON-1241 To Do
Unassigned https://issues.apache.org/jira/browse/METRON-1241
<https://issues.apache.org/jira/browse/METRON-1241>*
* METRON-1267 To Do
Unassigned https://issues.apache.org/jira/browse/METRON-1267
<https://issues.apache.org/jira/browse/METRON-1267>*
* METRON-1262 To Do Ryan
Merriman https://issues.apache.org/jira/browse/METRON-1262
<https://issues.apache.org/jira/browse/METRON-1262>*
* METRON-1263 Done Anand
Subramanian https://issues.apache.org/jira/browse/METRON-1263
<https://issues.apache.org/jira/browse/METRON-1263>*
* METRON-1255 To Do
Unassigned https://issues.apache.org/jira/browse/METRON-1255
<https://issues.apache.org/jira/browse/METRON-1255>*
* METRON-1249 Done 0.4.1 Nick
Allen https://issues.apache.org/jira/browse/METRON-1249
<https://issues.apache.org/jira/browse/METRON-1249>*
* METRON-1237 To Do Artem
Ervits https://issues.apache.org/jira/browse/METRON-1237
<https://issues.apache.org/jira/browse/METRON-1237>*
* METRON-1240 Done Artem
Ervits https://issues.apache.org/jira/browse/METRON-1240
<https://issues.apache.org/jira/browse/METRON-1240>*
* METRON-1226 Done 0.4.2 Nick
Allen*
* METRON-1081 Done James
Sirota https://issues.apache.org/jira/browse/METRON-1081
<https://issues.apache.org/jira/browse/METRON-1081>*
* METRON-1123 Done
RaghuMitra https://issues.apache.org/jira/browse/METRON-1123
<https://issues.apache.org/jira/browse/METRON-1123>*
* METRON-1223 Done
RaghuMitra https://issues.apache.org/jira/browse/METRON-1223
<https://issues.apache.org/jira/browse/METRON-1223>*
* METRON-1083 Done
RaghuMitra https://issues.apache.org/jira/browse/METRON-1083
<https://issues.apache.org/jira/browse/METRON-1083>*
* METRON-1232 Done
RaghuMitra https://issues.apache.org/jira/browse/METRON-1232
<https://issues.apache.org/jira/browse/METRON-1232>*
* METRON-1247 Done Justin
Leet https://issues.apache.org/jira/browse/METRON-1247
<https://issues.apache.org/jira/browse/METRON-1247>*
* METRON-1235 To Do
Unassigned https://issues.apache.org/jira/browse/METRON-1235
<https://issues.apache.org/jira/browse/METRON-1235>*
* METRON-1234 Done Artem
Ervits https://issues.apache.org/jira/browse/METRON-1234
<https://issues.apache.org/jira/browse/METRON-1234>*
* METRON-1222 Done Artem
Ervits https://issues.apache.org/jira/browse/METRON-1222
<https://issues.apache.org/jira/browse/METRON-1222>*
* METRON-1220 In Progress Justin
Leet https://issues.apache.org/jira/browse/METRON-1220
<https://issues.apache.org/jira/browse/METRON-1220>*
* METRON-1229 To Do
Unassigned https://issues.apache.org/jira/browse/METRON-1229
<https://issues.apache.org/jira/browse/METRON-1229>*
* METRON-1228 Done
Unassigned https://issues.apache.org/jira/browse/METRON-1228
<https://issues.apache.org/jira/browse/METRON-1228>*
* METRON-1218 To Do Ryan
Merriman https://issues.apache.org/jira/browse/METRON-1218
<https://issues.apache.org/jira/browse/METRON-1218>*
* METRON-1161 In Progress Ryan
Merriman https://issues.apache.org/jira/browse/METRON-1161
<https://issues.apache.org/jira/browse/METRON-1161>*
* METRON-1209 To Do
Unassigned https://issues.apache.org/jira/browse/METRON-1209
<https://issues.apache.org/jira/browse/METRON-1209>*
* METRON-1059 Done Next + 1
Unassigned https://issues.apache.org/jira/browse/METRON-1059
<https://issues.apache.org/jira/browse/METRON-1059>*
* METRON-1204 To Do Ryan
Merriman https://issues.apache.org/jira/browse/METRON-1204
<https://issues.apache.org/jira/browse/METRON-1204>*
* METRON-1052 Done Casey
Stella https://issues.apache.org/jira/browse/METRON-1052
<https://issues.apache.org/jira/browse/METRON-1052>*
* METRON-632 In Progress Tomas
Zezula https://issues.apache.org/jira/browse/METRON-632
<https://issues.apache.org/jira/browse/METRON-632>*
* METRON-1194 Done 0.4.2 Nick
Allen*
* METRON-1055 To Do Laurens
Vets https://issues.apache.org/jira/browse/METRON-1055
<https://issues.apache.org/jira/browse/METRON-1055>*
* METRON-1079 Done Otto
Fowler https://issues.apache.org/jira/browse/METRON-1079
<https://issues.apache.org/jira/browse/METRON-1079>*
* METRON-1085 To Do Ryan
Merriman https://issues.apache.org/jira/browse/METRON-1085
<https://issues.apache.org/jira/browse/METRON-1085>*
* METRON-1208 To Do Ryan
Merriman https://issues.apache.org/jira/browse/METRON-1208
<https://issues.apache.org/jira/browse/METRON-1208>*
* METRON-1207 To Do Ryan
Merriman https://issues.apache.org/jira/browse/METRON-1207
<https://issues.apache.org/jira/browse/METRON-1207>*
* METRON-1215 To Do
Unassigned https://issues.apache.org/jira/browse/METRON-1215
<https://issues.apache.org/jira/browse/METRON-1215>*
* METRON-1206 To Do Ryan
Merriman https://issues.apache.org/jira/browse/METRON-1206
<https://issues.apache.org/jira/browse/METRON-1206>*
* METRON-1195 To Do Justin
Leet https://issues.apache.org/jira/browse/METRON-1195
<https://issues.apache.org/jira/browse/METRON-1195>*
* METRON-1189 To Do Ryan
Merriman https://issues.apache.org/jira/browse/METRON-1189
<https://issues.apache.org/jira/browse/METRON-1189>*
* METRON-1156 Done 0.4.2 Nick
Allen*
* METRON-1198 Done 0.4.2 Nick
Allen*
* METRON-1202 Done Next + 1 Justin
Leet https://issues.apache.org/jira/browse/METRON-1202
<https://issues.apache.org/jira/browse/METRON-1202>*
* METRON-938 Done Next + 1 Justin
Leet https://issues.apache.org/jira/browse/METRON-938
<https://issues.apache.org/jira/browse/METRON-938>*
* METRON-1182 Done
RaghuMitra https://issues.apache.org/jira/browse/METRON-1182
<https://issues.apache.org/jira/browse/METRON-1182>*
* METRON-1188 Done Michael
Miklavcic https://issues.apache.org/jira/browse/METRON-1188
<https://issues.apache.org/jira/browse/METRON-1188>*
* METRON-1191 Done Matt
Foley https://issues.apache.org/jira/browse/METRON-1191
<https://issues.apache.org/jira/browse/METRON-1191>*
* METRON-1063 Done Next + 1 Artem
Ervits https://issues.apache.org/jira/browse/METRON-1063
<https://issues.apache.org/jira/browse/METRON-1063>*
* METRON-1190 In Progress Justin
Leet https://issues.apache.org/jira/browse/METRON-1190
<https://issues.apache.org/jira/browse/METRON-1190>*
* METRON-1187 Done 0.4.2 Nick
Allen*
* METRON-1185 To Do
Unassigned https://issues.apache.org/jira/browse/METRON-1185
<https://issues.apache.org/jira/browse/METRON-1185>*
* METRON-1186 To Do Casey
Stella https://issues.apache.org/jira/browse/METRON-1186
<https://issues.apache.org/jira/browse/METRON-1186>*
* METRON-1173 Done Next + 1 Jon
Zeolla https://issues.apache.org/jira/browse/METRON-1173
<https://issues.apache.org/jira/browse/METRON-1173>*
* METRON-1179 To Do
Unassigned https://issues.apache.org/jira/browse/METRON-1179
<https://issues.apache.org/jira/browse/METRON-1179>*
* METRON-1180 To Do
Unassigned https://issues.apache.org/jira/browse/METRON-1180
<https://issues.apache.org/jira/browse/METRON-1180>*
* METRON-1183 Done 0.4.2 Nick
Allen*
* METRON-1177 Done 0.4.2 Nick
Allen*
* METRON-1158 To Do Justin
Leet https://issues.apache.org/jira/browse/METRON-1158
<https://issues.apache.org/jira/browse/METRON-1158>*
* METRON-1146 Done Anand
Subramanian https://issues.apache.org/jira/browse/METRON-1146
<https://issues.apache.org/jira/browse/METRON-1146>*
* METRON-1176 Done Otto
Fowler https://issues.apache.org/jira/browse/METRON-1176
<https://issues.apache.org/jira/browse/METRON-1176>*
* METRON-1114 Done Ryan
Merriman https://issues.apache.org/jira/browse/METRON-1114
<https://issues.apache.org/jira/browse/METRON-1114>*
* METRON-1167 Done Next + 1 Nick
Allen https://issues.apache.org/jira/browse/METRON-1167
<https://issues.apache.org/jira/browse/METRON-1167>*
* METRON-1171 To Do Casey
Stella https://issues.apache.org/jira/browse/METRON-1171
<https://issues.apache.org/jira/browse/METRON-1171>*
The script is here [1] if anyone wants to run it themselves and grep for
their own name.
[1]
https://github.com/nickwallen/metron-commit-stuff/blob/master/validate-jira-for-release
On Fri, Dec 15, 2017 at 1:18 PM, Matt Foley <mf...@hortonworks.com>> wrote:
> Hi Nick,
> Good timing, you’ve saved me some work! :-)
>
> Origin of the list: The approach was defined before I started managing
> releases, but I think it’s a reasonable approach. It’s specified in our
> Release Process document, https://cwiki.apache.org/
> confluence/display/METRON/Release+Process , Step 5, the last bullet:
> “The artifacts for a release or RC [include]… A CHANGES file
> denoting the changes [since the last release].
> We usually construct this by taking the output of git log | grep
> METRON | sed 's/\[//g' | sed 's/\]//g' | grep -v "http" and removing the
> JIRAs from the previous releases (it’s in time sorted order so this is
> easy).”
>
> However, the release manager also has a responsibility to “Monitor and
> Verify Jiras” (Step 2 and various other places in the doc). Altho the doc
> isn’t explicit about it, I take this responsibility a little farther and do
> the following as part of Jira verification:
>
> 1. Extract the jira id’s from the CHANGES file with a simple awk script,
> put them in a Jira query, and confirm they are all actually marked
> “Fixed/Resolved” with Fix Version = <the current release> in Jira. If
> there are exceptions, I dun the contributors to fix it. You’ve seen these
> emails from me in the past couple releases. I don’t just mark them fixed
> myself, because some contributions (perfectly legitimately) only partially
> address a problem, and just marking a ticket “Fixed” may not be the right
> thing to do.
>
> 2. Query jira for any tickets NOT included in the prior list, that ARE
> marked fixed in the current release (or later, or “Next”, or similar
> “future” version constructs). These don’t usually happen (considering that
> Metron contributors mostly live inside github PR processes rather than
> Jiras) but when they do they also need to be resolved. Cases I’ve seen (in
> Metron and other projects) include:
> a) If they were indeed fixed in the current release, but not reflected in
> the commit message, I’ll annotate this fact in the Jira ticket, and
> hand-edit the CHANGES file for this release. (Doesn’t seem worth trying to
> edit old commit messages, since that mucks up the SHA1’s.)
> b) If they were fixed in a previous release but mis-marked as to Fix
> Version in the ticket, fix the ticket.
> c) If they aren’t really fixed, fix the ticket.
>
> 3. Once the set of Fixed tickets is complete and correct, query them for
> any that have “labels in (backward-incompatible, backwards-compatibility,
> backwards-incompatible)” -- yes, there shouldn’t be 3 labels, but it’s hard
> to fix because Jira never forgets, so every time someone gets it wrong, it
> adds to the set; fortunately, query completion helps here – OR "Docs Text"
> is not EMPTY. Each of these needs to be looked at carefully, verified, and
> if Doc Text is missing, the contributor and I cooperate to write a couple
> sentences. The Doc Text then needs to go in the RELEASE_NOTES for the
> release, in the section “## Non-backward-compatible Changes in this
> Release, and Upgrade Suggestions”.
>
> That’s about it. I guess I should add the above as an appendix doc to the
> Release Process documentation. I’ll do that.
> Cheers,
> --Matt
>
>
> On 12/15/17, 9:15 AM, "Nick Allen" <ni...@nickallen.org>> wrote:
>
> Hi Matt -
>
> I just updated like 15+ JIRAs of my own JIRAs that I completed and
> merged,
> but failed to mark as resolved. All of these will be included in
> 0.4.2. I
> updated each to be "fixed" in version 0.4.2 and marked as resolved.
> Hopefully the next RC will report those as fixed.
>
> (Q) Where does the list of JIRAs that get attached to the release
> originate
> from? Does it get pulled out of JIRA or do they come from the commit
> log?
>
> My apologies for not staying on top of my JIRAs.
>
>
>
>
> On Tue, Dec 12, 2017 at 2:21 PM, Matt Foley <ma...@apache.org>> wrote:
>
> > Thanks to Jon, Otto, and Nick for looking over RC1. Tonight I’ll
> fix the
> > RAT glitch, build RC2, and put it to formal vote.
> > Regards,
> > --Matt
> >
> > On 12/12/17, 5:14 AM, "Nick Allen" <ni...@nickallen.org>> wrote:
> >
> > RC1 is looking good to me. I validated the MD5s, built Metron,
> built
> > the
> > Bro plugin and reviewed the other artifacts like release notes.
> >
> > Running the RAT check on a 'clean' Metron does not produce any
> errors
> > for
> > me. It is only after building Metron, which pulls in additional
> Node
> > dependencies, does the RAT check fail.
> >
> >
> > On Sun, Dec 10, 2017 at 4:41 PM Matt Foley <ma...@apache.org>>
> wrote:
> >
> > > Yes, but let’s see if anyone else find other issues.
> > >
> > >
> > >
> > > From: Otto Fowler <ot...@gmail.com>>
> > > Date: Saturday, December 9, 2017 at 6:16 AM
> > > To: Matt Foley <mf...@hortonworks.com>>, "
> dev@metron.apache.org<ma...@metron.apache.org>" <
> > > dev@metron.apache.org<ma...@metron.apache.org>>
> > > Subject: Re: [DISCUSS] Upcoming Release
> > >
> > >
> > >
> > > So RC2 then?
> > >
> > >
> > >
> > > On December 8, 2017 at 20:43:21, Matt Foley (
> mfoley@hortonworks.com<ma...@hortonworks.com>)
> > > wrote:
> > >
> > > Hah, here it is: https://github.com/apache/metron/pull/743
> > > “This problem seems to only reproduce when one unrolls a
> tarball
> > rather
> > > than cloning from github.”
> > >
> > > Heh, the exclusion at
> > > https://github.com/apache/metron/blob/master/pom.xml#L351 is
> still
> > there,
> > > but the hashcode in the bundle.css file name has changed from
> > > a0b6b99c10d9a13dc67e to f56deed131e58bd7ee04. Sigh. Did the
> version
> > of Font
> > > Awesome fonts change?
> > >
> > >
> > > On 12/8/17, 5:26 PM, "Matt Foley" <ma...@apache.org>> wrote:
> > >
> > > I remember having trouble with this bundle.css file on the last
> > release,
> > > but I can’t remember what we did about it. Anybody?
> > >
> > > On 12/8/17, 1:41 PM, "Otto Fowler" <ot...@gmail.com>>
> wrote:
> > >
> > > Steps
> > >
> > > - Downloaded tar.gz’s, asc files and KEYS
> > > - Verified signing of both tar.gz’s
> > > - searched for rouge 0.4.1 entries
> > > - verified the main pom.xml
> > > - built :
> > >
> > > mvn clean && time mvn -q -T 2C -DskipTests install && time mvn
> -q -T
> > > 2C surefire:test@unit-tests && time mvn -q
> > > surefire:test@integration-tests && time mvn -q test --projects
> > > metron-interface/metron-config && time
> build_utils/verify_licenses.sh
> > >
> > > Found rat error:
> > >
> > >
> > > *****************************************************
> > > Summary
> > > -------
> > > Generated at: 2017-12-08T16:33:27-05:00
> > >
> > > Notes: 3
> > > Binaries: 193
> > > Archives: 0
> > > Standards: 75
> > >
> > > Apache Licensed: 74
> > > Generated Documents: 0
> > >
> > > JavaDocs are generated, thus a license header is optional.
> > > Generated files do not require license headers.
> > >
> > > 1 Unknown Licenses
> > >
> > > *****************************************************
> > >
> > > Files with unapproved licenses:
> > >
> > >
> > > /Users/batman/tmp/release_ver/apache-metron-0.4.2-rc1/
> > metron-interface/metron-alerts/dist/styles.
> f56deed131e58bd7ee04.bundle.css
> > >
> > > *****************************************************
> > >
> > >
> > >
> > >
> > >
> > > *****************************************************
> > > Summary
> > > -------
> > > Generated at: 2017-12-08T16:33:27-05:00
> > >
> > > Notes: 3
> > > Binaries: 193
> > > Archives: 0
> > > Standards: 75
> > >
> > > Apache Licensed: 74
> > > Generated Documents: 0
> > >
> > > JavaDocs are generated, thus a license header is optional.
> > > Generated files do not require license headers.
> > >
> > > 1 Unknown Licenses
> > >
> > > *****************************************************
> > >
> > > Files with unapproved licenses:
> > >
> > >
> > >
> > > /Users/ottofowler/tmp/release_ver/apache-metron-0.4.2-rc1/
> > metron-interface/metron-alerts/dist/styles.
> f56deed131e58bd7ee04.bundle.css
> > >
> > > *****************************************************
> > >
> > >
> > >
> > > On December 8, 2017 at 04:34:24, Matt Foley (mattf@apache.org<ma...@apache.org>)
> > wrote:
> > >
> > > Colleagues,
> > > I’ve posted Metron-0.4.2-RC1 and Metron-bro-plugin-kafka-0.1 to
> > > https://dist.apache.org/repos/dist/dev/metron/0.4.2-RC1/
> > >
> > > Given the complexity of this RC, I’d appreciate if a couple
> people
> > would be
> > > willing to kick the tires before we put it up for a vote.
> > >
> > > I will myself be going thru the Verify Build process this
> weekend,
> > as I
> > > won’t be able to do it Friday.
> > >
> > > Thanks,
> > > --Matt
> > >
> > >
> > > On 12/4/17, 2:05 PM, "Zeolla@GMail.com" <ze...@gmail.com>>
> wrote:
> > >
> > > Can we resolve the conversation regarding the second repo? I
> was
> > waiting
> > > to get more input/preferences from people There's also a
> > documentation
> > > update that fixes a few broken Stellar docs that already has
> aa +1,
> > I just
> > > need to merge it.
> > >
> > > Jon
> > >
> > > On Mon, Dec 4, 2017, 17:01 Casey Stella <ce...@gmail.com>>
> wrote:
> > >
> > > > I would be in favor of a release at this point.
> > > >
> > > > On Mon, Dec 4, 2017 at 4:57 PM, Matt Foley <ma...@apache.org>
> >
> > wrote:
> > > >
> > > > > Hey all,
> > > > > I see METRON-1252 was resolved over the weekend. Shall I go
> > ahead and
> > > > > start the process with 0.4.2 release?
> > > > > Does anyone have any commits they feel strongly should go
> in
> > before
> > > 0.4.2
> > > > > is done, or are we ready to call it good?
> > > > >
> > > > > I believe there is consensus the 0.4.2 release should
> include a
> > release
> > > > of
> > > > > the current state of the metron-bro-plugin-kafka. I will
> > continue the
> > > > > discussion in that thread as to the process for
> accomplishing
> > that, but
> > > > > plan on it happening.
> > > > >
> > > > > Regards,
> > > > > --Matt
> > > > >
> > > > > On 11/26/17, 6:26 PM, "Matt Foley" <ma...@apache.org>>
> wrote:
> > > > >
> > > > > Hope everyone (at least in the U.S.) had a great
> Thanksgiving
> > > > holiday.
> > > > > Regarding status of the release effort, still pending
> > METRON-1252, so
> > > > > not making the release branch yet.
> > > > >
> > > > > Regards,
> > > > > --Matt
> > > > >
> > > > > On 11/17/17, 1:32 PM, "Matt Foley" <ma...@apache.org>>
> wrote:
> > > > >
> > > > > (With release manager hat on)
> > > > >
> > > > > The community has proposed a release of Metron in the near
> > > > future,
> > > > > focusing on Meta-alerts running in Elasticsearch.
> > > > > Congrats on getting so many of the below already done. At
> this
> > > > > point, only METRON-1252, and the discussion of how to
> handle
> > joint
> > > > release
> > > > > of the Metron bro plugin, remain as gating items for the
> > release. I
> > > > > project these will be resolved next week, so let’s propose
> the
> > > following:
> > > > >
> > > > > Sometime next week, after the last bits are done, I’ll
> start the
> > > > > release process and create the release branch.
> > > > >
> > > > > The proposed new version will be 0.4.2, unless there are
> backward
> > > > > incompatible changes that support making it 0.5.0.
> > > > > Currently there are NO included Jiras labeled
> > > > > ‘backward-incompatible’, nor having Docs Text indicating
> so.
> > > > > ***If anyone knows that some of the commits included since
> 0.4.1
> > > > > introduce backward incompatibility, please say so now on
> this
> > thread,
> > > and
> > > > > mark the Jira as such.***
> > > > >
> > > > > The 90 or so jiras/commits already in master branch since
> 0.4.1
> > > > > are listed below.
> > > > > Thanks,
> > > > > --Matt
> > > > >
> > > > > METRON-1301 Alerts UI - Sorting on Triage Score
> Unexpectedly
> > > > > Filters Some Records (nickwallen) closes apache/metron#832
> > > > > METRON-1294 IP addresses are not formatted correctly in
> facet
> > > > > and group results (merrimanr) closes apache/metron#827
> > > > > METRON-1291 Kafka produce REST endpoint does not work in a
> > > > > Kerberized cluster (merrimanr) closes apache/metron#826
> > > > > METRON-1290 Only first 10 alerts are update when a
> MetaAlert
> > > > > status is changed to inactive (justinleet) closes
> > apache/metron#842
> > > > > METRON-1311 Service Check Should Check Elasticsearch Index
> > > > > Templates (nickwallen) closes apache/metron#839
> > > > > METRON-1289 Alert fields are lost when a MetaAlert is
> created
> > > > > (merrimanr) closes apache/metron#824
> > > > > METRON-1309 Change metron-deployment to pull the plugin
> from
> > > > > apache/metron-bro-plugin-kafka (JonZeolla) closes
> > apache/metron#837
> > > > > METRON-1310 Template Delete Action Deletes Search Indices
> > > > > (nickwallen) closes apache/metron#838
> > > > > METRON-1275: Fix Metron Documentation closes
> > > > > apache/incubator-metron#833
> > > > > METRON-1295 Unable to Configure Logging for REST API
> > > > > (nickwallen) closes apache/metron#828
> > > > > METRON-1307 Force install of java8 since java9 does not
> > > > appear
> > > > > to work with the scripts (brianhurley via ottobackwards)
> closes
> > > > > apache/metron#835
> > > > > METRON-1296 Full Dev Fails to Deploy Index Templates
> > > > > (nickwallen via cestella) closes
> apache/incubator-metron#829
> > > > > METRON-1281 Remove hard-coded indices from the Alerts UI
> > > > > (merrimanr) closes apache/metron#821
> > > > > METRON-1287 Full Dev Fails When Installing EPEL Repository
> > > > > (nickwallen) closes apache/metron#820
> > > > > METRON-1267 Alerts UI returns a 404 when refreshing the
> > > > > alerts-list page (iraghumitra via merrimanr) closes
> > apache/metron#819
> > > > > METRON-1283 Install Elasticsearch template as a part of the
> > > > > mpack startup scripts (anandsubbu via nickwallen) closes
> > > > apache/metron#817
> > > > > METRON-1254: Conditionals as map keys do not function in
> > > > > Stellar closes apache/incubator-metron#801
> > > > > METRON-1261 Apply bro security patch (JonZeolla via
> > > > > ottobackwards) closes apache/metron#805
> > > > > METRON-1284 Remove extraneous dead query in
> ElasticsearchDao
> > > > > (justinleet) closes apache/metron#818
> > > > > METRON-1270: fix for warnings missing @return tag argument
> in
> > > > > metron-analytics/metron-profiler-common and
> > metron-profiler-client
> > > closes
> > > > > apache/incubator-metron#810
> > > > > METRON-1272 Hide child alerts from searches and grouping if
> > > > > they belong to meta alerts (justinleet) closes
> apache/metron#811
> > > > > METRON-1224 Add time range selection to search control
> > > > > (iraghumitra via james-sirota) closes apache/metron#796
> > > > > METRON-1280 0.4.1 -> 0.4.2 missed a couple of projects
> > > > > (cestella via justinleet) closes apache/metron#816
> > > > > METRON-1243: Add a REST endpoint which allows us to get a
> > > > list
> > > > > of all indice closes apache/incubator-metron#797
> > > > > METRON-1196 Increment master version number to 0.4.2 for
> > > > > on-going development (mattf-horton) closes
> apache/metron#767
> > > > > METRON-1278 Strip "Build Status" widget from root
> > > > > README.md in site-book build (mattf-horton) closes
> > apache/metron#815
> > > > > METRON-1274 Master has failure in
> > > > > StormControllerIntegrationTest (merrimanr) closes
> > apache/metron#813
> > > > > METRON-1266 Profiler - SASL Authentication Failed
> > > > (nickwallen)
> > > > > closes apache/metron#809
> > > > > METRON-1260 Include Alerts UI in Ambari Service Check
> > > > > (nickwallen) closes apache/metron#804
> > > > > METRON-1251: Typo and formatting fixes for metron-rest
> README
> > > > > closes apache/incubator-metron#800
> > > > > METRON-1241: Enable the REST API to use a cache for the
> > > > > zookeeper config similar to the Bolts closes
> > > apache/incubator-metron#795
> > > > > METRON-1267 Alerts UI returns a 404 when refreshing the
> > > > > alerts-list page (merrimanr) closes apache/metron#808
> > > > > METRON-1262 Unable to add comment for a alert in a
> meta-alert
> > > > > (merrimanr) closes apache/metron#806
> > > > > METRON-1263 Start Alerts UI service after Metron REST
> > > > > (anandsubbu via nickwallen) closes apache/metron#807
> > > > > METRON-1255 MetaAlert search is not filtering on status
> > > > > (merrimanr) closes apache/metron#802
> > > > > METRON-1249 Improve Metron MPack Service Checks
> (nickwallen)
> > > > > closes apache/metron#799
> > > > > METRON-1237 address javadoc warnings in metron-maas-common
> > > > > (dbist via james-sirota) closes apache/metron#792
> > > > > METRON-1240 address javadoc warnings in metron-platform and
> > > > > metron-analytics (dbist via james-sirota) closes
> > apache/metron#794
> > > > > METRON-1226 Searching Can Errantly Query the Wrong Indices
> > > > > (nickwallen) closes apache/metron#793
> > > > > METRON-1081 Fix Alerts and Ops UI Notices file
> (james-sirota)
> > > > > closes apache/metron#682
> > > > > METRON-1123 Add group by option using faceted search
> > > > > capabilities of metron-rest-api (iraghumitra via
> james-sirota)
> > closes
> > > > > apache/metron#768
> > > > > METRON-1223 Add support to add comments for alerts
> > > > > (iraghumitra via james-sirota) closes apache/metron#788
> > > > > METRON-1083 Add filters using faceted search capabilities
> of
> > > > > metron-rest-api (iraghumitra via james-sirota) closes
> > apache/metron#710
> > > > > METRON-1232 Alert status changes are not reflected in list
> > > > > view (iraghumitra via merrimanr) closes apache/metron#787
> > > > > METRON-1247 REST search and findOne endpoints return
> > > > > unexpected or incorrect results for guids (justinleet)
> closes
> > > > > apache/metron#798
> > > > > METRON-1235: Document the properties pulled from the global
> > > > > configuration closes apache/incubator-metron#791
> > > > > METRON-1234: fix for WARNING 'dependencies.dependency.(
> > > > > groupId:artifactId:type:classifier)' must be unique:
> > > > > org.apache.hadoop:hadoop-yarn-api:jar (dbist via mmiklavc)
> > closes
> > > > > apache/metron#790
> > > > > METRON-1222: fix warning for The expression
> ${parent.version}
> > > > > is deprecated. Please use ${project.parent.version}
> instead.
> > (dbist via
> > > > > mmiklavc) closes apache/metron#782
> > > > > METRON-1220 Create documentation around alert nested field
> > > > > (justinleet) closes apache/metron#780
> > > > > METRON-1229 Management UI type is part of the declarations
> of
> > > > > 2 modules (merrimanr) closes apache/metron#784
> > > > > METRON-1228: Configuration Management PUSH immediately does
> > > > > DUMP after (mmiklavc via mmiklavc) closes apache/metron#783
> > > > > METRON-1218 Metron REST should return better error messages
> > > > > (merrimanr) closes apache/metron#779
> > > > > METRON-1161 Add ability to edit parser command line options
> > > > in
> > > > > the management UI (merrimanr) closes apache/metron#737
> > > > > METRON-1209: Make stellar repl take logging properties,
> like
> > > > > other CLI apps in metron closes apache/incubator-metron#772
> > > > > METRON-1059 address checkstyle warning AvoidStarImport in
> > > > > metron-stellar (dbist via ottobackwards) closes
> apache/metron#664
> > > > > METRON-1204 UI does not time out after being idle, but
> stops
> > > > > functioning (merrimanr) closes apache/metron#771
> > > > > METRON-1052: Add forensic similarity hash functions to
> > > > Stellar
> > > > > closes apache/incubator-metron#781
> > > > > METRON-632: Added validation of "shew.enrichmentType" and
> > > > > "shew.keyColumns" closes apache/incubator-metron#732
> > > > > METRON-1194 Add Profiler Debug Functions to Profiler README
> > > > > (nickwallen via ottobackwards) closes apache/metron#765
> > > > > METRON-1055 Metron 0.4.0 manual installation guide for
> CentOS
> > > > > 6 updates (lvets via ottobackwards) closes
> apache/metron#661
> > > > > METRON-1079 STELLAR NaN should be a keyword (ottobackwards)
> > > > > closes apache/metron#681
> > > > > METRON-1085 Add REST endpoint to save a user profile for
> the
> > > > > Alerts UI (merrimanr) closes apache/metron#694
> > > > > METRON-1208 MPack for Alerts UI (merrimanr) closes
> > > > > apache/metron#778
> > > > > METRON-1207 Make RPMs for Alerts UI (merrimanr) closes
> > > > > apache/metron#777
> > > > > METRON-1215 Fix link to RPMs chapter (DimDroll via
> > > > justinleet)
> > > > > closes apache/metron#776
> > > > > METRON-1206 Make alerts UI conform to ops UI for install
> > > > > (merrimanr) closes apache/metron#773
> > > > > METRON-1195 Meta alerts improperly handle updates to
> > > > non-alert
> > > > > fields (justinleet) closes apache/metron#766
> > > > > METRON-1189 Add alert escalation to the Alerts UI
> (merrimanr)
> > > > > closes apache/metron#762
> > > > > METRON-1156 Simulate Triage Rules in the Stellar REPL
> > > > > (nickwallen) closes apache/metron#733
> > > > > METRON-1198 Pycapa - No such configuration property
> > > > > 'sasl.kerberos.principal' (nickwallen) closes
> apache/metron#769
> > > > > METRON-1202 ElasticsearchDao Has extraneous sleep call
> > > > > (justinleet) closes apache/metron#770
> > > > > METRON-938 "service metron-rest start <password>" does not
> > > > > work on CentOS 7. (justinleet) closes apache/metron#757
> > > > > METRON-1182 Refactor Code in alert list to accommodate new
> > > > > view types (iraghumitra via merrimanr) closes
> apache/metron#756
> > > > > METRON-1188: Ambari global configuration management
> > > > (mmiklavc)
> > > > > closes apache/metron#760
> > > > > METRON-1191 update public web site to point at 0.4.1 new
> > > > > release (mattf-horton) closes apache/metron#764
> > > > > METRON-1063 address javadoc warnings in metron-stellar
> (dbist
> > > > > via ottobackwards) closes apache/metron#668
> > > > > METRON-1190 Fix Meta Alert Type handling in calculation of
> > > > > scores (justinleet) closes apache/metron#763
> > > > > METRON-1187 Indexing/Profiler Kafka ACL Groups Not Setup
> > > > > Correctly (nickwallen) closes apache/metron#759
> > > > > METRON-1185: Stellar REPL does not work on a kerberized
> > > > > cluster when calling functions interacting with HBase
> closes
> > > > > apache/incubator-metron#755
> > > > > METRON-1186: Profiler Functions use classutils from shaded
> > > > > storm closes apache/incubator-metron#758
> > > > > METRON-1173: Fix pointers to old stellar docs closes
> > > > > apache/incubator-metron#746
> > > > > METRON-1179: Make STATS_ADD to take a list closes
> > > > > apache/incubator-metron#750
> > > > > METRON-1180: Make Stellar Shell accept zookeeper quorum as
> a
> > > > > CSV list and not require a port closes
> > apache/incubator-metron#751
> > > > > METRON-1183 Improve KDC Setup Instructions (nickwallen)
> > > > closes
> > > > > apache/metron#753
> > > > > METRON-1177 Stale running topologies seen
> post-kerberization
> > > > > and cause exceptions (nickwallen) closes apache/metron#748
> > > > > METRON-1158 Build backend for grouping alerts into meta
> > > > alerts
> > > > > (justinleet) closes apache/metron#734
> > > > > METRON-1146: Add ability to parse JSON string into
> JSONObject
> > > > > for stellar closes apache/incubator-metron#727
> > > > > METRON-1176 REST: HDFS Service should support setting
> > > > > permissions on files when writing (ottobackwards) closes
> > > > apache/metron#749
> > > > > METRON-1114 Add group by capabilities to search REST
> endpoint
> > > > > (merrimanr) closes apache/metron#702
> > > > > METRON-1167 Define Session Specific Global Configuration
> > > > > Values in the REPL (nickwallen) closes apache/metron#740
> > > > > METRON-1171: Better validation for the SUBSTRING stellar
> > > > > function closes apache/incubator-metron#745
> > > > >
> > > > >
> > > > >
> > > > > On 11/17/17, 11:59 AM, "Nick Allen" <ni...@nickallen.org>>
> wrote:
> > > > >
> > > > > I just wanted to send an update on where we are at. We've
> > > > > gotten a lot
> > > > > done here recently as you can see below.
> > > > >
> > > > > ✓ DONE (1) First, METRON-1289 needs to go in. This one was
> > > > > a fairly big
> > > > > effort and I am hearing that we are pretty close.
> > > > >
> > > > > ✓ DONE (2) METRON-1294 fixes an issue in how field types
> > > > are
> > > > > looked-up.
> > > > >
> > > > > ✓ DONE (3) METRON-1290 is next. While this may have been
> > > > > fixed in
> > > > > M-1289, there may be some test cases we want from this PR.
> > > > >
> > > > > ✓ DONE (4) METRON-1301 addresses a problem with the sorting
> > > > > logic.
> > > > >
> > > > > ✓ DONE (5) METRON-1291 fixes an issue with escalation of
> > > > > metaalerts.
> > > > >
> > > > > (6) That leads us to Raghu's UI work in METRON-1252. This
> > > > > introduces the
> > > > > UI bits that depend on all the previous backend work.
> > > > >
> > > > > (7) At this point, we should have our best effort at
> > > > running
> > > > > Metaalerts
> > > > > on Elasticsearch 2.x. I propose that we cut a release here.
> > > > >
> > > > > (8) After we cut the release, we can introduce the work for
> > > > > ES 5.x in
> > > > > METRON-939. I know we will need lots of help testing and
> > > > > reviewing this
> > > > > one.
> > > > >
> > > > >
> > > > >
> > > > > We also have an outstanding question that needs resolved
> > > > > BEFORE we
> > > > > release. We need to come to a consensus on how to release
> > > > > having moved our
> > > > > Bro Plugin to a separate repo. I don't think we've heard
> > > > from
> > > > > everyone on
> > > > > this. I'd urge everyone to chime in so we can choose a path
> > > > > forward.
> > > > >
> > > > > If anyone is totally confused in regards to that
> discussion,
> > > > I
> > > > > can try and
> > > > > send an options summary again as a separate discuss thread.
> > > > > The original
> > > > > chain was somewhere around here [1].
> > > > >
> > > > > [1]
> > > > > https://lists.apache.org/thread.html/
> > > > > 54a4474881b97e559df24728b3a0e923a58345a282451085eef832ef@%
> > > > > 3Cdev.metron.apache.org<http://3Cdev.metron.apache.org>%3E
> > > > >
> > > > >
> > > > >
> > > > > On Wed, Nov 15, 2017 at 10:04 AM, Nick Allen <
> > > > > nick@nickallen.org<ma...@nickallen.org>> wrote:
> > > > >
> > > > > > Hi Guys -
> > > > > >
> > > > > > I want to follow-up on this discussion. It sounds like
> > > > most
> > > > > people are in
> > > > > > agreement with the general approach.
> > > > > >
> > > > > > A lot of people have been working hard on Metaalerts and
> > > > > Elasticsearch. I
> > > > > > have checked-in with those doing the heavy lifting and
> have
> > > > > compiled a more
> > > > > > detailed plan based on where we are at now. To the best
> of
> > > > > my knowledge
> > > > > > here is the plan of attack for finishing out this effort.
> > > > > >
> > > > > > (1) First, METRON-1289 needs to go in. This one was a
> > > > > fairly big effort
> > > > > > and I am hearing that we are pretty close.
> > > > > >
> > > > > > (2) METRON-1294 fixes an issue in how field types are
> > > > > looked-up.
> > > > > >
> > > > > > (3) METRON-1290 is next. While this may have been fixed
> > > > > in M-1289,
> > > > > > there may be some test cases we want from this PR.
> > > > > >
> > > > > > (4) METRON-1301 addresses a problem with the sorting
> > > > logic.
> > > > > >
> > > > > > (5) METRON-1291 fixes an issue with escalation of
> > > > > metaalerts.
> > > > > >
> > > > > > (6) That leads us to Raghu's UI work in METRON-1252.
> > > > This
> > > > > introduces
> > > > > > the UI bits that depend on all the previous backend work.
> > > > > >
> > > > > > (7) At this point, we should have our best effort at
> > > > > running Metaalerts
> > > > > > on Elasticsearch 2.x. I propose that we cut a release
> here.
> > > > > >
> > > > > > (8) After we cut the release, we can introduce the work
> > > > > for ES 5.x in
> > > > > > METRON-939. I know we will need lots of help testing and
> > > > > reviewing this
> > > > > > one.
> > > > > >
> > > > > > Please correct me if I am wrong. I will try and send out
> > > > > updates as we
> > > > > > make progress.
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Mon, Nov 6, 2017 at 1:03 PM, Zeolla@GMail.com <
> > > > > zeolla@gmail.com<ma...@gmail.com>> wrote:
> > > > > >
> > > > > >> I agree, I think it's very reasonable to move in line
> with
> > > > > Nick's
> > > > > >> proposal. I would also suggest that we outline what the
> > > > > target versions
> > > > > >> would be to add in the METRON-777 components, since it
> has
> > > > > been functional
> > > > > >> for a very long time but not reviewed and has some
> really
> > > > > rockstar
> > > > > >> improvements.
> > > > > >>
> > > > > >> Jon
> > > > > >>
> > > > > >> On Mon, Nov 6, 2017 at 12:56 PM Otto Fowler <
> > > > > ottobackwards@gmail.com<ma...@gmail.com>>
> > > > > >> wrote:
> > > > > >>
> > > > > >> > I think the ES cutover should be the start of the
> 0.5.x
> > > > > series, and we
> > > > > >> > continue on with 0.4.x for the
> > > > > >> > metadata improvements etc. We could chose to focus
> > > > > 0.5.x’s first
> > > > > >> releases
> > > > > >> > on not only ES but
> > > > > >> > getting a handle on kibana and the mpack situation as
> > > > > well.
> > > > > >> >
> > > > > >> >
> > > > > >> >
> > > > > >> >
> > > > > >> > On November 6, 2017 at 12:48:45, Michael Miklavcic (
> > > > > >> > michael.miklavcic@gmail.com<ma...@gmail.com>) wrote:
> > > > > >> >
> > > > > >> > I agree with your proposal, Nick. I think having a
> > > > > stabilizing release
> > > > > >> > prior to upgrading ES/Kibana makes sense.
> > > > > >> >
> > > > > >> > On Mon, Nov 6, 2017 at 9:16 AM, Nick Allen <
> > > > > nick@nickallen.org<ma...@nickallen.org>> wrote:
> > > > > >> >
> > > > > >> > > I would like to start a discussion around upcoming
> > > > > releases. We have a
> > > > > >> > > couple separate significant tracks of work that we
> > > > need
> > > > > to reconcile
> > > > > >> in
> > > > > >> > our
> > > > > >> > > release schedule.
> > > > > >> > >
> > > > > >> > > (1) We have had (and have in review) a good number
> of
> > > > > bug fixes
> > > > > >> required
> > > > > >> > to
> > > > > >> > > support Metaalerts on the existing Elasticsearch 2.x
> > > > > infrastructure.
> > > > > >> > >
> > > > > >> > >
> > > > > >> > > (2) We also have ongoing work to upgrade our
> > > > > infrastructure to
> > > > > >> > > Elasticsearch 5.x, which will not be backwards
> > > > > compatible.
> > > > > >> > >
> > > > > >> > >
> > > > > >> > > I would like to see a release that has our best work
> > > > on
> > > > > ES 2.x before
> > > > > >> we
> > > > > >> > > migrate to 5.x. I would propose the following.
> > > > > >> > >
> > > > > >> > > Release N+1: Introduce Metaalerts running on ES 2.x
> > > > > >> > >
> > > > > >> > > Release N+2: Cut-over to ES 5.x
> > > > > >> > >
> > > > > >> > >
> > > > > >> > > (Q) Is it worth cutting a separate release for ES
> 2.x?
> > > > > Is there a
> > > > > >> better
> > > > > >> > > way to handle the cut-over to 5.x?
> > > > > >> > >
> > > > > >> >
> > > > > >> --
> > > > > >>
> > > > > >> Jon
> > > > > >>
> > > > > >
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > >
> > > --
> > >
> > > Jon
> > >
> > >
> > >
> > >
> > >
> > >
> >
> >
> >
> >
>
>
>
Re: [DISCUSS] Upcoming Release
Posted by Casey Stella <ce...@gmail.com>.
That script seems great, nick! Perhaps we should adjust the wiki around
releases to point to it? Thoughts?
On Fri, Dec 15, 2017 at 1:47 PM, Nick Allen <ni...@nickallen.org> wrote:
> Thanks, Matt.
>
> Maybe you already have something that does this. I wrote a quick script
> that validates each JIRA since the last release tag to make sure they are
> marked "Done" and with the correct fix version. I would expect that for
> the next release, each JIRA should have status="Done", fix-version="0.4.2".
>
> Unless I am mistaken, we have quite a few that need cleaned up. In the
> following output, any line that has a URL indicates that a fix is needed.
> Or it least, **I think** a fix is needed.
>
> To the community.... If you see your name with a URL next to it, it would
> be great if you could follow that link and fix the JIRA. Otherwise, I will
> volunteer to help clean some of these up should some not get addressed.
>
>
> *$ ./validate-jira-for-release*
> *Cloning into 'metron-0.4.2'...*
> *remote: Counting objects: 35046, done.*
> *remote: Compressing objects: 100% (13698/13698), done.*
> *remote: Total 35046 (delta 15708), reused 31645 (delta 12822)*
> *Receiving objects: 100% (35046/35046), 53.05 MiB | 6.48 MiB/s, done.*
> *Resolving deltas: 100% (15708/15708), done.*
> *Fetching origin*
> * JIRA STATUS FIX VERSION
> ASSIGNEE FIX*
> * METRON-1345 Done Michael
> Miklavcic https://issues.apache.org/jira/browse/METRON-1345
> <https://issues.apache.org/jira/browse/METRON-1345>*
> * METRON-1349 Done Next + 1 Nick
> Allen https://issues.apache.org/jira/browse/METRON-1349
> <https://issues.apache.org/jira/browse/METRON-1349>*
> * METRON-1343 Done
> Mohan https://issues.apache.org/jira/browse/METRON-1343
> <https://issues.apache.org/jira/browse/METRON-1343>*
> * METRON-1306 To Do
> Unassigned https://issues.apache.org/jira/browse/METRON-1306
> <https://issues.apache.org/jira/browse/METRON-1306>*
> * METRON-1341 Done Simon Elliston
> Ball https://issues.apache.org/jira/browse/METRON-1341
> <https://issues.apache.org/jira/browse/METRON-1341>*
> * METRON-1313 Done Jon
> Zeolla https://issues.apache.org/jira/browse/METRON-1313
> <https://issues.apache.org/jira/browse/METRON-1313>*
> * METRON-1346 Done Otto
> Fowler https://issues.apache.org/jira/browse/METRON-1346
> <https://issues.apache.org/jira/browse/METRON-1346>*
> * METRON-1336 Done 0.4.2 Nick
> Allen*
> * METRON-1335 Done Anand
> Subramanian https://issues.apache.org/jira/browse/METRON-1335
> <https://issues.apache.org/jira/browse/METRON-1335>*
> * METRON-1308 Done Jon
> Zeolla https://issues.apache.org/jira/browse/METRON-1308
> <https://issues.apache.org/jira/browse/METRON-1308>*
> * METRON-1338 Done 0.4.2 Nick
> Allen*
> * METRON-1286 To Do 0.4.2
> Unassigned https://issues.apache.org/jira/browse/METRON-1286
> <https://issues.apache.org/jira/browse/METRON-1286>*
> * METRON-1334 Done 0.4.2 Nick
> Allen*
> * METRON-1277 Done Otto
> Fowler https://issues.apache.org/jira/browse/METRON-1277
> <https://issues.apache.org/jira/browse/METRON-1277>*
> * METRON-1239 To Do
> Unassigned https://issues.apache.org/jira/browse/METRON-1239
> <https://issues.apache.org/jira/browse/METRON-1239>*
> * METRON-1328 Done Anand
> Subramanian https://issues.apache.org/jira/browse/METRON-1328
> <https://issues.apache.org/jira/browse/METRON-1328>*
> * METRON-1333 Done Otto
> Fowler https://issues.apache.org/jira/browse/METRON-1333
> <https://issues.apache.org/jira/browse/METRON-1333>*
> * METRON-1252 Done
> RaghuMitra https://issues.apache.org/jira/browse/METRON-1252
> <https://issues.apache.org/jira/browse/METRON-1252>*
> * METRON-1316 To Do Next + 1
> Unassigned https://issues.apache.org/jira/browse/METRON-1316
> <https://issues.apache.org/jira/browse/METRON-1316>*
> * METRON-1088 Done Jon
> Zeolla https://issues.apache.org/jira/browse/METRON-1088
> <https://issues.apache.org/jira/browse/METRON-1088>*
> * METRON-1319 To Do Ryan
> Merriman https://issues.apache.org/jira/browse/METRON-1319
> <https://issues.apache.org/jira/browse/METRON-1319>*
> * METRON-1321 To Do
> Unassigned https://issues.apache.org/jira/browse/METRON-1321
> <https://issues.apache.org/jira/browse/METRON-1321>*
> * METRON-1301 Done 0.4.2 Nick
> Allen*
> * METRON-1294 To Do Ryan
> Merriman https://issues.apache.org/jira/browse/METRON-1294
> <https://issues.apache.org/jira/browse/METRON-1294>*
> * METRON-1291 To Do Ryan
> Merriman https://issues.apache.org/jira/browse/METRON-1291
> <https://issues.apache.org/jira/browse/METRON-1291>*
> * METRON-1290 Done Justin
> Leet https://issues.apache.org/jira/browse/METRON-1290
> <https://issues.apache.org/jira/browse/METRON-1290>*
> * METRON-1311 Done 0.4.2 Nick
> Allen*
> * METRON-1289 To Do Ryan
> Merriman https://issues.apache.org/jira/browse/METRON-1289
> <https://issues.apache.org/jira/browse/METRON-1289>*
> * METRON-1309 Done Jon
> Zeolla https://issues.apache.org/jira/browse/METRON-1309
> <https://issues.apache.org/jira/browse/METRON-1309>*
> * METRON-1310 Done 0.4.2 Nick
> Allen*
> * METRON-1275 Done Jon
> Zeolla https://issues.apache.org/jira/browse/METRON-1275
> <https://issues.apache.org/jira/browse/METRON-1275>*
> * METRON-1295 Done 0.4.2 Nick
> Allen*
> * METRON-1307 Done Otto
> Fowler https://issues.apache.org/jira/browse/METRON-1307
> <https://issues.apache.org/jira/browse/METRON-1307>*
> * METRON-1296 Done 0.4.2 Nick
> Allen*
> * METRON-1281 To Do Ryan
> Merriman https://issues.apache.org/jira/browse/METRON-1281
> <https://issues.apache.org/jira/browse/METRON-1281>*
> * METRON-1287 Done 0.4.2 Nick
> Allen*
> * METRON-1267 To Do
> Unassigned https://issues.apache.org/jira/browse/METRON-1267
> <https://issues.apache.org/jira/browse/METRON-1267>*
> * METRON-1283 Done Anand
> Subramanian https://issues.apache.org/jira/browse/METRON-1283
> <https://issues.apache.org/jira/browse/METRON-1283>*
> * METRON-1254 To Do
> Unassigned https://issues.apache.org/jira/browse/METRON-1254
> <https://issues.apache.org/jira/browse/METRON-1254>*
> * METRON-1261 Done Jon
> Zeolla https://issues.apache.org/jira/browse/METRON-1261
> <https://issues.apache.org/jira/browse/METRON-1261>*
> * METRON-1284 Done Justin
> Leet https://issues.apache.org/jira/browse/METRON-1284
> <https://issues.apache.org/jira/browse/METRON-1284>*
> * METRON-1270 Done Artem
> Ervits https://issues.apache.org/jira/browse/METRON-1270
> <https://issues.apache.org/jira/browse/METRON-1270>*
> * METRON-1272 In Progress Justin
> Leet https://issues.apache.org/jira/browse/METRON-1272
> <https://issues.apache.org/jira/browse/METRON-1272>*
> * METRON-1224 Done
> RaghuMitra https://issues.apache.org/jira/browse/METRON-1224
> <https://issues.apache.org/jira/browse/METRON-1224>*
> * METRON-1280 To Do
> Unassigned https://issues.apache.org/jira/browse/METRON-1280
> <https://issues.apache.org/jira/browse/METRON-1280>*
> * METRON-1243 To Do
> Unassigned https://issues.apache.org/jira/browse/METRON-1243
> <https://issues.apache.org/jira/browse/METRON-1243>*
> * METRON-1196 Done Matt
> Foley https://issues.apache.org/jira/browse/METRON-1196
> <https://issues.apache.org/jira/browse/METRON-1196>*
> * METRON-1278 Done 0.4.2 Matt
> Foley*
> * METRON-1274 To Do
> Unassigned https://issues.apache.org/jira/browse/METRON-1274
> <https://issues.apache.org/jira/browse/METRON-1274>*
> * METRON-1266 Done 0.4.2 Nick
> Allen*
> * METRON-1260 Done 0.4.2 Nick
> Allen*
> * METRON-1251 Done Jon
> Zeolla https://issues.apache.org/jira/browse/METRON-1251
> <https://issues.apache.org/jira/browse/METRON-1251>*
> * METRON-1241 To Do
> Unassigned https://issues.apache.org/jira/browse/METRON-1241
> <https://issues.apache.org/jira/browse/METRON-1241>*
> * METRON-1267 To Do
> Unassigned https://issues.apache.org/jira/browse/METRON-1267
> <https://issues.apache.org/jira/browse/METRON-1267>*
> * METRON-1262 To Do Ryan
> Merriman https://issues.apache.org/jira/browse/METRON-1262
> <https://issues.apache.org/jira/browse/METRON-1262>*
> * METRON-1263 Done Anand
> Subramanian https://issues.apache.org/jira/browse/METRON-1263
> <https://issues.apache.org/jira/browse/METRON-1263>*
> * METRON-1255 To Do
> Unassigned https://issues.apache.org/jira/browse/METRON-1255
> <https://issues.apache.org/jira/browse/METRON-1255>*
> * METRON-1249 Done 0.4.1 Nick
> Allen https://issues.apache.org/jira/browse/METRON-1249
> <https://issues.apache.org/jira/browse/METRON-1249>*
> * METRON-1237 To Do Artem
> Ervits https://issues.apache.org/jira/browse/METRON-1237
> <https://issues.apache.org/jira/browse/METRON-1237>*
> * METRON-1240 Done Artem
> Ervits https://issues.apache.org/jira/browse/METRON-1240
> <https://issues.apache.org/jira/browse/METRON-1240>*
> * METRON-1226 Done 0.4.2 Nick
> Allen*
> * METRON-1081 Done James
> Sirota https://issues.apache.org/jira/browse/METRON-1081
> <https://issues.apache.org/jira/browse/METRON-1081>*
> * METRON-1123 Done
> RaghuMitra https://issues.apache.org/jira/browse/METRON-1123
> <https://issues.apache.org/jira/browse/METRON-1123>*
> * METRON-1223 Done
> RaghuMitra https://issues.apache.org/jira/browse/METRON-1223
> <https://issues.apache.org/jira/browse/METRON-1223>*
> * METRON-1083 Done
> RaghuMitra https://issues.apache.org/jira/browse/METRON-1083
> <https://issues.apache.org/jira/browse/METRON-1083>*
> * METRON-1232 Done
> RaghuMitra https://issues.apache.org/jira/browse/METRON-1232
> <https://issues.apache.org/jira/browse/METRON-1232>*
> * METRON-1247 Done Justin
> Leet https://issues.apache.org/jira/browse/METRON-1247
> <https://issues.apache.org/jira/browse/METRON-1247>*
> * METRON-1235 To Do
> Unassigned https://issues.apache.org/jira/browse/METRON-1235
> <https://issues.apache.org/jira/browse/METRON-1235>*
> * METRON-1234 Done Artem
> Ervits https://issues.apache.org/jira/browse/METRON-1234
> <https://issues.apache.org/jira/browse/METRON-1234>*
> * METRON-1222 Done Artem
> Ervits https://issues.apache.org/jira/browse/METRON-1222
> <https://issues.apache.org/jira/browse/METRON-1222>*
> * METRON-1220 In Progress Justin
> Leet https://issues.apache.org/jira/browse/METRON-1220
> <https://issues.apache.org/jira/browse/METRON-1220>*
> * METRON-1229 To Do
> Unassigned https://issues.apache.org/jira/browse/METRON-1229
> <https://issues.apache.org/jira/browse/METRON-1229>*
> * METRON-1228 Done
> Unassigned https://issues.apache.org/jira/browse/METRON-1228
> <https://issues.apache.org/jira/browse/METRON-1228>*
> * METRON-1218 To Do Ryan
> Merriman https://issues.apache.org/jira/browse/METRON-1218
> <https://issues.apache.org/jira/browse/METRON-1218>*
> * METRON-1161 In Progress Ryan
> Merriman https://issues.apache.org/jira/browse/METRON-1161
> <https://issues.apache.org/jira/browse/METRON-1161>*
> * METRON-1209 To Do
> Unassigned https://issues.apache.org/jira/browse/METRON-1209
> <https://issues.apache.org/jira/browse/METRON-1209>*
> * METRON-1059 Done Next + 1
> Unassigned https://issues.apache.org/jira/browse/METRON-1059
> <https://issues.apache.org/jira/browse/METRON-1059>*
> * METRON-1204 To Do Ryan
> Merriman https://issues.apache.org/jira/browse/METRON-1204
> <https://issues.apache.org/jira/browse/METRON-1204>*
> * METRON-1052 Done Casey
> Stella https://issues.apache.org/jira/browse/METRON-1052
> <https://issues.apache.org/jira/browse/METRON-1052>*
> * METRON-632 In Progress Tomas
> Zezula https://issues.apache.org/jira/browse/METRON-632
> <https://issues.apache.org/jira/browse/METRON-632>*
> * METRON-1194 Done 0.4.2 Nick
> Allen*
> * METRON-1055 To Do Laurens
> Vets https://issues.apache.org/jira/browse/METRON-1055
> <https://issues.apache.org/jira/browse/METRON-1055>*
> * METRON-1079 Done Otto
> Fowler https://issues.apache.org/jira/browse/METRON-1079
> <https://issues.apache.org/jira/browse/METRON-1079>*
> * METRON-1085 To Do Ryan
> Merriman https://issues.apache.org/jira/browse/METRON-1085
> <https://issues.apache.org/jira/browse/METRON-1085>*
> * METRON-1208 To Do Ryan
> Merriman https://issues.apache.org/jira/browse/METRON-1208
> <https://issues.apache.org/jira/browse/METRON-1208>*
> * METRON-1207 To Do Ryan
> Merriman https://issues.apache.org/jira/browse/METRON-1207
> <https://issues.apache.org/jira/browse/METRON-1207>*
> * METRON-1215 To Do
> Unassigned https://issues.apache.org/jira/browse/METRON-1215
> <https://issues.apache.org/jira/browse/METRON-1215>*
> * METRON-1206 To Do Ryan
> Merriman https://issues.apache.org/jira/browse/METRON-1206
> <https://issues.apache.org/jira/browse/METRON-1206>*
> * METRON-1195 To Do Justin
> Leet https://issues.apache.org/jira/browse/METRON-1195
> <https://issues.apache.org/jira/browse/METRON-1195>*
> * METRON-1189 To Do Ryan
> Merriman https://issues.apache.org/jira/browse/METRON-1189
> <https://issues.apache.org/jira/browse/METRON-1189>*
> * METRON-1156 Done 0.4.2 Nick
> Allen*
> * METRON-1198 Done 0.4.2 Nick
> Allen*
> * METRON-1202 Done Next + 1 Justin
> Leet https://issues.apache.org/jira/browse/METRON-1202
> <https://issues.apache.org/jira/browse/METRON-1202>*
> * METRON-938 Done Next + 1 Justin
> Leet https://issues.apache.org/jira/browse/METRON-938
> <https://issues.apache.org/jira/browse/METRON-938>*
> * METRON-1182 Done
> RaghuMitra https://issues.apache.org/jira/browse/METRON-1182
> <https://issues.apache.org/jira/browse/METRON-1182>*
> * METRON-1188 Done Michael
> Miklavcic https://issues.apache.org/jira/browse/METRON-1188
> <https://issues.apache.org/jira/browse/METRON-1188>*
> * METRON-1191 Done Matt
> Foley https://issues.apache.org/jira/browse/METRON-1191
> <https://issues.apache.org/jira/browse/METRON-1191>*
> * METRON-1063 Done Next + 1 Artem
> Ervits https://issues.apache.org/jira/browse/METRON-1063
> <https://issues.apache.org/jira/browse/METRON-1063>*
> * METRON-1190 In Progress Justin
> Leet https://issues.apache.org/jira/browse/METRON-1190
> <https://issues.apache.org/jira/browse/METRON-1190>*
> * METRON-1187 Done 0.4.2 Nick
> Allen*
> * METRON-1185 To Do
> Unassigned https://issues.apache.org/jira/browse/METRON-1185
> <https://issues.apache.org/jira/browse/METRON-1185>*
> * METRON-1186 To Do Casey
> Stella https://issues.apache.org/jira/browse/METRON-1186
> <https://issues.apache.org/jira/browse/METRON-1186>*
> * METRON-1173 Done Next + 1 Jon
> Zeolla https://issues.apache.org/jira/browse/METRON-1173
> <https://issues.apache.org/jira/browse/METRON-1173>*
> * METRON-1179 To Do
> Unassigned https://issues.apache.org/jira/browse/METRON-1179
> <https://issues.apache.org/jira/browse/METRON-1179>*
> * METRON-1180 To Do
> Unassigned https://issues.apache.org/jira/browse/METRON-1180
> <https://issues.apache.org/jira/browse/METRON-1180>*
> * METRON-1183 Done 0.4.2 Nick
> Allen*
> * METRON-1177 Done 0.4.2 Nick
> Allen*
> * METRON-1158 To Do Justin
> Leet https://issues.apache.org/jira/browse/METRON-1158
> <https://issues.apache.org/jira/browse/METRON-1158>*
> * METRON-1146 Done Anand
> Subramanian https://issues.apache.org/jira/browse/METRON-1146
> <https://issues.apache.org/jira/browse/METRON-1146>*
> * METRON-1176 Done Otto
> Fowler https://issues.apache.org/jira/browse/METRON-1176
> <https://issues.apache.org/jira/browse/METRON-1176>*
> * METRON-1114 Done Ryan
> Merriman https://issues.apache.org/jira/browse/METRON-1114
> <https://issues.apache.org/jira/browse/METRON-1114>*
> * METRON-1167 Done Next + 1 Nick
> Allen https://issues.apache.org/jira/browse/METRON-1167
> <https://issues.apache.org/jira/browse/METRON-1167>*
> * METRON-1171 To Do Casey
> Stella https://issues.apache.org/jira/browse/METRON-1171
> <https://issues.apache.org/jira/browse/METRON-1171>*
>
>
>
> The script is here [1] if anyone wants to run it themselves and grep for
> their own name.
>
> [1]
> https://github.com/nickwallen/metron-commit-stuff/blob/
> master/validate-jira-for-release
>
> On Fri, Dec 15, 2017 at 1:18 PM, Matt Foley <mf...@hortonworks.com>
> wrote:
>
> > Hi Nick,
> > Good timing, you’ve saved me some work! :-)
> >
> > Origin of the list: The approach was defined before I started managing
> > releases, but I think it’s a reasonable approach. It’s specified in our
> > Release Process document, https://cwiki.apache.org/
> > confluence/display/METRON/Release+Process , Step 5, the last bullet:
> > “The artifacts for a release or RC [include]… A CHANGES file
> > denoting the changes [since the last release].
> > We usually construct this by taking the output of git log | grep
> > METRON | sed 's/\[//g' | sed 's/\]//g' | grep -v "http" and removing the
> > JIRAs from the previous releases (it’s in time sorted order so this is
> > easy).”
> >
> > However, the release manager also has a responsibility to “Monitor and
> > Verify Jiras” (Step 2 and various other places in the doc). Altho the
> doc
> > isn’t explicit about it, I take this responsibility a little farther and
> do
> > the following as part of Jira verification:
> >
> > 1. Extract the jira id’s from the CHANGES file with a simple awk script,
> > put them in a Jira query, and confirm they are all actually marked
> > “Fixed/Resolved” with Fix Version = <the current release> in Jira. If
> > there are exceptions, I dun the contributors to fix it. You’ve seen
> these
> > emails from me in the past couple releases. I don’t just mark them fixed
> > myself, because some contributions (perfectly legitimately) only
> partially
> > address a problem, and just marking a ticket “Fixed” may not be the right
> > thing to do.
> >
> > 2. Query jira for any tickets NOT included in the prior list, that ARE
> > marked fixed in the current release (or later, or “Next”, or similar
> > “future” version constructs). These don’t usually happen (considering
> that
> > Metron contributors mostly live inside github PR processes rather than
> > Jiras) but when they do they also need to be resolved. Cases I’ve seen
> (in
> > Metron and other projects) include:
> > a) If they were indeed fixed in the current release, but not reflected in
> > the commit message, I’ll annotate this fact in the Jira ticket, and
> > hand-edit the CHANGES file for this release. (Doesn’t seem worth trying
> to
> > edit old commit messages, since that mucks up the SHA1’s.)
> > b) If they were fixed in a previous release but mis-marked as to Fix
> > Version in the ticket, fix the ticket.
> > c) If they aren’t really fixed, fix the ticket.
> >
> > 3. Once the set of Fixed tickets is complete and correct, query them for
> > any that have “labels in (backward-incompatible, backwards-compatibility,
> > backwards-incompatible)” -- yes, there shouldn’t be 3 labels, but it’s
> hard
> > to fix because Jira never forgets, so every time someone gets it wrong,
> it
> > adds to the set; fortunately, query completion helps here – OR "Docs
> Text"
> > is not EMPTY. Each of these needs to be looked at carefully, verified,
> and
> > if Doc Text is missing, the contributor and I cooperate to write a couple
> > sentences. The Doc Text then needs to go in the RELEASE_NOTES for the
> > release, in the section “## Non-backward-compatible Changes in this
> > Release, and Upgrade Suggestions”.
> >
> > That’s about it. I guess I should add the above as an appendix doc to
> the
> > Release Process documentation. I’ll do that.
> > Cheers,
> > --Matt
> >
> >
> > On 12/15/17, 9:15 AM, "Nick Allen" <ni...@nickallen.org> wrote:
> >
> > Hi Matt -
> >
> > I just updated like 15+ JIRAs of my own JIRAs that I completed and
> > merged,
> > but failed to mark as resolved. All of these will be included in
> > 0.4.2. I
> > updated each to be "fixed" in version 0.4.2 and marked as resolved.
> > Hopefully the next RC will report those as fixed.
> >
> > (Q) Where does the list of JIRAs that get attached to the release
> > originate
> > from? Does it get pulled out of JIRA or do they come from the commit
> > log?
> >
> > My apologies for not staying on top of my JIRAs.
> >
> >
> >
> >
> > On Tue, Dec 12, 2017 at 2:21 PM, Matt Foley <ma...@apache.org>
> wrote:
> >
> > > Thanks to Jon, Otto, and Nick for looking over RC1. Tonight I’ll
> > fix the
> > > RAT glitch, build RC2, and put it to formal vote.
> > > Regards,
> > > --Matt
> > >
> > > On 12/12/17, 5:14 AM, "Nick Allen" <ni...@nickallen.org> wrote:
> > >
> > > RC1 is looking good to me. I validated the MD5s, built Metron,
> > built
> > > the
> > > Bro plugin and reviewed the other artifacts like release notes.
> > >
> > > Running the RAT check on a 'clean' Metron does not produce any
> > errors
> > > for
> > > me. It is only after building Metron, which pulls in
> additional
> > Node
> > > dependencies, does the RAT check fail.
> > >
> > >
> > > On Sun, Dec 10, 2017 at 4:41 PM Matt Foley <ma...@apache.org>
> > wrote:
> > >
> > > > Yes, but let’s see if anyone else find other issues.
> > > >
> > > >
> > > >
> > > > From: Otto Fowler <ot...@gmail.com>
> > > > Date: Saturday, December 9, 2017 at 6:16 AM
> > > > To: Matt Foley <mf...@hortonworks.com>, "
> > dev@metron.apache.org" <
> > > > dev@metron.apache.org>
> > > > Subject: Re: [DISCUSS] Upcoming Release
> > > >
> > > >
> > > >
> > > > So RC2 then?
> > > >
> > > >
> > > >
> > > > On December 8, 2017 at 20:43:21, Matt Foley (
> > mfoley@hortonworks.com)
> > > > wrote:
> > > >
> > > > Hah, here it is: https://github.com/apache/metron/pull/743
> > > > “This problem seems to only reproduce when one unrolls a
> > tarball
> > > rather
> > > > than cloning from github.”
> > > >
> > > > Heh, the exclusion at
> > > > https://github.com/apache/metron/blob/master/pom.xml#L351 is
> > still
> > > there,
> > > > but the hashcode in the bundle.css file name has changed from
> > > > a0b6b99c10d9a13dc67e to f56deed131e58bd7ee04. Sigh. Did the
> > version
> > > of Font
> > > > Awesome fonts change?
> > > >
> > > >
> > > > On 12/8/17, 5:26 PM, "Matt Foley" <ma...@apache.org> wrote:
> > > >
> > > > I remember having trouble with this bundle.css file on the
> last
> > > release,
> > > > but I can’t remember what we did about it. Anybody?
> > > >
> > > > On 12/8/17, 1:41 PM, "Otto Fowler" <ot...@gmail.com>
> > wrote:
> > > >
> > > > Steps
> > > >
> > > > - Downloaded tar.gz’s, asc files and KEYS
> > > > - Verified signing of both tar.gz’s
> > > > - searched for rouge 0.4.1 entries
> > > > - verified the main pom.xml
> > > > - built :
> > > >
> > > > mvn clean && time mvn -q -T 2C -DskipTests install && time
> mvn
> > -q -T
> > > > 2C surefire:test@unit-tests && time mvn -q
> > > > surefire:test@integration-tests && time mvn -q test
> --projects
> > > > metron-interface/metron-config && time
> > build_utils/verify_licenses.sh
> > > >
> > > > Found rat error:
> > > >
> > > >
> > > > *****************************************************
> > > > Summary
> > > > -------
> > > > Generated at: 2017-12-08T16:33:27-05:00
> > > >
> > > > Notes: 3
> > > > Binaries: 193
> > > > Archives: 0
> > > > Standards: 75
> > > >
> > > > Apache Licensed: 74
> > > > Generated Documents: 0
> > > >
> > > > JavaDocs are generated, thus a license header is optional.
> > > > Generated files do not require license headers.
> > > >
> > > > 1 Unknown Licenses
> > > >
> > > > *****************************************************
> > > >
> > > > Files with unapproved licenses:
> > > >
> > > >
> > > > /Users/batman/tmp/release_ver/apache-metron-0.4.2-rc1/
> > > metron-interface/metron-alerts/dist/styles.
> > f56deed131e58bd7ee04.bundle.css
> > > >
> > > > *****************************************************
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > *****************************************************
> > > > Summary
> > > > -------
> > > > Generated at: 2017-12-08T16:33:27-05:00
> > > >
> > > > Notes: 3
> > > > Binaries: 193
> > > > Archives: 0
> > > > Standards: 75
> > > >
> > > > Apache Licensed: 74
> > > > Generated Documents: 0
> > > >
> > > > JavaDocs are generated, thus a license header is optional.
> > > > Generated files do not require license headers.
> > > >
> > > > 1 Unknown Licenses
> > > >
> > > > *****************************************************
> > > >
> > > > Files with unapproved licenses:
> > > >
> > > >
> > > >
> > > > /Users/ottofowler/tmp/release_ver/apache-metron-0.4.2-rc1/
> > > metron-interface/metron-alerts/dist/styles.
> > f56deed131e58bd7ee04.bundle.css
> > > >
> > > > *****************************************************
> > > >
> > > >
> > > >
> > > > On December 8, 2017 at 04:34:24, Matt Foley (
> mattf@apache.org)
> > > wrote:
> > > >
> > > > Colleagues,
> > > > I’ve posted Metron-0.4.2-RC1 and Metron-bro-plugin-kafka-0.1
> to
> > > > https://dist.apache.org/repos/dist/dev/metron/0.4.2-RC1/
> > > >
> > > > Given the complexity of this RC, I’d appreciate if a couple
> > people
> > > would be
> > > > willing to kick the tires before we put it up for a vote.
> > > >
> > > > I will myself be going thru the Verify Build process this
> > weekend,
> > > as I
> > > > won’t be able to do it Friday.
> > > >
> > > > Thanks,
> > > > --Matt
> > > >
> > > >
> > > > On 12/4/17, 2:05 PM, "Zeolla@GMail.com" <ze...@gmail.com>
> > wrote:
> > > >
> > > > Can we resolve the conversation regarding the second repo? I
> > was
> > > waiting
> > > > to get more input/preferences from people There's also a
> > > documentation
> > > > update that fixes a few broken Stellar docs that already has
> > aa +1,
> > > I just
> > > > need to merge it.
> > > >
> > > > Jon
> > > >
> > > > On Mon, Dec 4, 2017, 17:01 Casey Stella <ce...@gmail.com>
> > wrote:
> > > >
> > > > > I would be in favor of a release at this point.
> > > > >
> > > > > On Mon, Dec 4, 2017 at 4:57 PM, Matt Foley <
> mattf@apache.org
> > >
> > > wrote:
> > > > >
> > > > > > Hey all,
> > > > > > I see METRON-1252 was resolved over the weekend. Shall I
> go
> > > ahead and
> > > > > > start the process with 0.4.2 release?
> > > > > > Does anyone have any commits they feel strongly should go
> > in
> > > before
> > > > 0.4.2
> > > > > > is done, or are we ready to call it good?
> > > > > >
> > > > > > I believe there is consensus the 0.4.2 release should
> > include a
> > > release
> > > > > of
> > > > > > the current state of the metron-bro-plugin-kafka. I will
> > > continue the
> > > > > > discussion in that thread as to the process for
> > accomplishing
> > > that, but
> > > > > > plan on it happening.
> > > > > >
> > > > > > Regards,
> > > > > > --Matt
> > > > > >
> > > > > > On 11/26/17, 6:26 PM, "Matt Foley" <ma...@apache.org>
> > wrote:
> > > > > >
> > > > > > Hope everyone (at least in the U.S.) had a great
> > Thanksgiving
> > > > > holiday.
> > > > > > Regarding status of the release effort, still pending
> > > METRON-1252, so
> > > > > > not making the release branch yet.
> > > > > >
> > > > > > Regards,
> > > > > > --Matt
> > > > > >
> > > > > > On 11/17/17, 1:32 PM, "Matt Foley" <ma...@apache.org>
> > wrote:
> > > > > >
> > > > > > (With release manager hat on)
> > > > > >
> > > > > > The community has proposed a release of Metron in the
> near
> > > > > future,
> > > > > > focusing on Meta-alerts running in Elasticsearch.
> > > > > > Congrats on getting so many of the below already done. At
> > this
> > > > > > point, only METRON-1252, and the discussion of how to
> > handle
> > > joint
> > > > > release
> > > > > > of the Metron bro plugin, remain as gating items for the
> > > release. I
> > > > > > project these will be resolved next week, so let’s
> propose
> > the
> > > > following:
> > > > > >
> > > > > > Sometime next week, after the last bits are done, I’ll
> > start the
> > > > > > release process and create the release branch.
> > > > > >
> > > > > > The proposed new version will be 0.4.2, unless there are
> > backward
> > > > > > incompatible changes that support making it 0.5.0.
> > > > > > Currently there are NO included Jiras labeled
> > > > > > ‘backward-incompatible’, nor having Docs Text indicating
> > so.
> > > > > > ***If anyone knows that some of the commits included
> since
> > 0.4.1
> > > > > > introduce backward incompatibility, please say so now on
> > this
> > > thread,
> > > > and
> > > > > > mark the Jira as such.***
> > > > > >
> > > > > > The 90 or so jiras/commits already in master branch since
> > 0.4.1
> > > > > > are listed below.
> > > > > > Thanks,
> > > > > > --Matt
> > > > > >
> > > > > > METRON-1301 Alerts UI - Sorting on Triage Score
> > Unexpectedly
> > > > > > Filters Some Records (nickwallen) closes
> apache/metron#832
> > > > > > METRON-1294 IP addresses are not formatted correctly in
> > facet
> > > > > > and group results (merrimanr) closes apache/metron#827
> > > > > > METRON-1291 Kafka produce REST endpoint does not work in
> a
> > > > > > Kerberized cluster (merrimanr) closes apache/metron#826
> > > > > > METRON-1290 Only first 10 alerts are update when a
> > MetaAlert
> > > > > > status is changed to inactive (justinleet) closes
> > > apache/metron#842
> > > > > > METRON-1311 Service Check Should Check Elasticsearch
> Index
> > > > > > Templates (nickwallen) closes apache/metron#839
> > > > > > METRON-1289 Alert fields are lost when a MetaAlert is
> > created
> > > > > > (merrimanr) closes apache/metron#824
> > > > > > METRON-1309 Change metron-deployment to pull the plugin
> > from
> > > > > > apache/metron-bro-plugin-kafka (JonZeolla) closes
> > > apache/metron#837
> > > > > > METRON-1310 Template Delete Action Deletes Search Indices
> > > > > > (nickwallen) closes apache/metron#838
> > > > > > METRON-1275: Fix Metron Documentation closes
> > > > > > apache/incubator-metron#833
> > > > > > METRON-1295 Unable to Configure Logging for REST API
> > > > > > (nickwallen) closes apache/metron#828
> > > > > > METRON-1307 Force install of java8 since java9 does not
> > > > > appear
> > > > > > to work with the scripts (brianhurley via ottobackwards)
> > closes
> > > > > > apache/metron#835
> > > > > > METRON-1296 Full Dev Fails to Deploy Index Templates
> > > > > > (nickwallen via cestella) closes
> > apache/incubator-metron#829
> > > > > > METRON-1281 Remove hard-coded indices from the Alerts UI
> > > > > > (merrimanr) closes apache/metron#821
> > > > > > METRON-1287 Full Dev Fails When Installing EPEL
> Repository
> > > > > > (nickwallen) closes apache/metron#820
> > > > > > METRON-1267 Alerts UI returns a 404 when refreshing the
> > > > > > alerts-list page (iraghumitra via merrimanr) closes
> > > apache/metron#819
> > > > > > METRON-1283 Install Elasticsearch template as a part of
> the
> > > > > > mpack startup scripts (anandsubbu via nickwallen) closes
> > > > > apache/metron#817
> > > > > > METRON-1254: Conditionals as map keys do not function in
> > > > > > Stellar closes apache/incubator-metron#801
> > > > > > METRON-1261 Apply bro security patch (JonZeolla via
> > > > > > ottobackwards) closes apache/metron#805
> > > > > > METRON-1284 Remove extraneous dead query in
> > ElasticsearchDao
> > > > > > (justinleet) closes apache/metron#818
> > > > > > METRON-1270: fix for warnings missing @return tag
> argument
> > in
> > > > > > metron-analytics/metron-profiler-common and
> > > metron-profiler-client
> > > > closes
> > > > > > apache/incubator-metron#810
> > > > > > METRON-1272 Hide child alerts from searches and grouping
> if
> > > > > > they belong to meta alerts (justinleet) closes
> > apache/metron#811
> > > > > > METRON-1224 Add time range selection to search control
> > > > > > (iraghumitra via james-sirota) closes apache/metron#796
> > > > > > METRON-1280 0.4.1 -> 0.4.2 missed a couple of projects
> > > > > > (cestella via justinleet) closes apache/metron#816
> > > > > > METRON-1243: Add a REST endpoint which allows us to get a
> > > > > list
> > > > > > of all indice closes apache/incubator-metron#797
> > > > > > METRON-1196 Increment master version number to 0.4.2 for
> > > > > > on-going development (mattf-horton) closes
> > apache/metron#767
> > > > > > METRON-1278 Strip "Build Status" widget from
> root
> > > > > > README.md in site-book build (mattf-horton) closes
> > > apache/metron#815
> > > > > > METRON-1274 Master has failure in
> > > > > > StormControllerIntegrationTest (merrimanr) closes
> > > apache/metron#813
> > > > > > METRON-1266 Profiler - SASL Authentication Failed
> > > > > (nickwallen)
> > > > > > closes apache/metron#809
> > > > > > METRON-1260 Include Alerts UI in Ambari Service Check
> > > > > > (nickwallen) closes apache/metron#804
> > > > > > METRON-1251: Typo and formatting fixes for metron-rest
> > README
> > > > > > closes apache/incubator-metron#800
> > > > > > METRON-1241: Enable the REST API to use a cache for the
> > > > > > zookeeper config similar to the Bolts closes
> > > > apache/incubator-metron#795
> > > > > > METRON-1267 Alerts UI returns a 404 when refreshing the
> > > > > > alerts-list page (merrimanr) closes apache/metron#808
> > > > > > METRON-1262 Unable to add comment for a alert in a
> > meta-alert
> > > > > > (merrimanr) closes apache/metron#806
> > > > > > METRON-1263 Start Alerts UI service after Metron REST
> > > > > > (anandsubbu via nickwallen) closes apache/metron#807
> > > > > > METRON-1255 MetaAlert search is not filtering on status
> > > > > > (merrimanr) closes apache/metron#802
> > > > > > METRON-1249 Improve Metron MPack Service Checks
> > (nickwallen)
> > > > > > closes apache/metron#799
> > > > > > METRON-1237 address javadoc warnings in
> metron-maas-common
> > > > > > (dbist via james-sirota) closes apache/metron#792
> > > > > > METRON-1240 address javadoc warnings in metron-platform
> and
> > > > > > metron-analytics (dbist via james-sirota) closes
> > > apache/metron#794
> > > > > > METRON-1226 Searching Can Errantly Query the Wrong
> Indices
> > > > > > (nickwallen) closes apache/metron#793
> > > > > > METRON-1081 Fix Alerts and Ops UI Notices file
> > (james-sirota)
> > > > > > closes apache/metron#682
> > > > > > METRON-1123 Add group by option using faceted search
> > > > > > capabilities of metron-rest-api (iraghumitra via
> > james-sirota)
> > > closes
> > > > > > apache/metron#768
> > > > > > METRON-1223 Add support to add comments for alerts
> > > > > > (iraghumitra via james-sirota) closes apache/metron#788
> > > > > > METRON-1083 Add filters using faceted search capabilities
> > of
> > > > > > metron-rest-api (iraghumitra via james-sirota) closes
> > > apache/metron#710
> > > > > > METRON-1232 Alert status changes are not reflected in
> list
> > > > > > view (iraghumitra via merrimanr) closes apache/metron#787
> > > > > > METRON-1247 REST search and findOne endpoints return
> > > > > > unexpected or incorrect results for guids (justinleet)
> > closes
> > > > > > apache/metron#798
> > > > > > METRON-1235: Document the properties pulled from the
> global
> > > > > > configuration closes apache/incubator-metron#791
> > > > > > METRON-1234: fix for WARNING 'dependencies.dependency.(
> > > > > > groupId:artifactId:type:classifier)' must be unique:
> > > > > > org.apache.hadoop:hadoop-yarn-api:jar (dbist via
> mmiklavc)
> > > closes
> > > > > > apache/metron#790
> > > > > > METRON-1222: fix warning for The expression
> > ${parent.version}
> > > > > > is deprecated. Please use ${project.parent.version}
> > instead.
> > > (dbist via
> > > > > > mmiklavc) closes apache/metron#782
> > > > > > METRON-1220 Create documentation around alert nested
> field
> > > > > > (justinleet) closes apache/metron#780
> > > > > > METRON-1229 Management UI type is part of the
> declarations
> > of
> > > > > > 2 modules (merrimanr) closes apache/metron#784
> > > > > > METRON-1228: Configuration Management PUSH immediately
> does
> > > > > > DUMP after (mmiklavc via mmiklavc) closes
> apache/metron#783
> > > > > > METRON-1218 Metron REST should return better error
> messages
> > > > > > (merrimanr) closes apache/metron#779
> > > > > > METRON-1161 Add ability to edit parser command line
> options
> > > > > in
> > > > > > the management UI (merrimanr) closes apache/metron#737
> > > > > > METRON-1209: Make stellar repl take logging properties,
> > like
> > > > > > other CLI apps in metron closes
> apache/incubator-metron#772
> > > > > > METRON-1059 address checkstyle warning AvoidStarImport in
> > > > > > metron-stellar (dbist via ottobackwards) closes
> > apache/metron#664
> > > > > > METRON-1204 UI does not time out after being idle, but
> > stops
> > > > > > functioning (merrimanr) closes apache/metron#771
> > > > > > METRON-1052: Add forensic similarity hash functions to
> > > > > Stellar
> > > > > > closes apache/incubator-metron#781
> > > > > > METRON-632: Added validation of "shew.enrichmentType" and
> > > > > > "shew.keyColumns" closes apache/incubator-metron#732
> > > > > > METRON-1194 Add Profiler Debug Functions to Profiler
> README
> > > > > > (nickwallen via ottobackwards) closes apache/metron#765
> > > > > > METRON-1055 Metron 0.4.0 manual installation guide for
> > CentOS
> > > > > > 6 updates (lvets via ottobackwards) closes
> > apache/metron#661
> > > > > > METRON-1079 STELLAR NaN should be a keyword
> (ottobackwards)
> > > > > > closes apache/metron#681
> > > > > > METRON-1085 Add REST endpoint to save a user profile for
> > the
> > > > > > Alerts UI (merrimanr) closes apache/metron#694
> > > > > > METRON-1208 MPack for Alerts UI (merrimanr) closes
> > > > > > apache/metron#778
> > > > > > METRON-1207 Make RPMs for Alerts UI (merrimanr) closes
> > > > > > apache/metron#777
> > > > > > METRON-1215 Fix link to RPMs chapter (DimDroll via
> > > > > justinleet)
> > > > > > closes apache/metron#776
> > > > > > METRON-1206 Make alerts UI conform to ops UI for install
> > > > > > (merrimanr) closes apache/metron#773
> > > > > > METRON-1195 Meta alerts improperly handle updates to
> > > > > non-alert
> > > > > > fields (justinleet) closes apache/metron#766
> > > > > > METRON-1189 Add alert escalation to the Alerts UI
> > (merrimanr)
> > > > > > closes apache/metron#762
> > > > > > METRON-1156 Simulate Triage Rules in the Stellar REPL
> > > > > > (nickwallen) closes apache/metron#733
> > > > > > METRON-1198 Pycapa - No such configuration property
> > > > > > 'sasl.kerberos.principal' (nickwallen) closes
> > apache/metron#769
> > > > > > METRON-1202 ElasticsearchDao Has extraneous sleep call
> > > > > > (justinleet) closes apache/metron#770
> > > > > > METRON-938 "service metron-rest start <password>" does
> not
> > > > > > work on CentOS 7. (justinleet) closes apache/metron#757
> > > > > > METRON-1182 Refactor Code in alert list to accommodate
> new
> > > > > > view types (iraghumitra via merrimanr) closes
> > apache/metron#756
> > > > > > METRON-1188: Ambari global configuration management
> > > > > (mmiklavc)
> > > > > > closes apache/metron#760
> > > > > > METRON-1191 update public web site to point at 0.4.1 new
> > > > > > release (mattf-horton) closes apache/metron#764
> > > > > > METRON-1063 address javadoc warnings in metron-stellar
> > (dbist
> > > > > > via ottobackwards) closes apache/metron#668
> > > > > > METRON-1190 Fix Meta Alert Type handling in calculation
> of
> > > > > > scores (justinleet) closes apache/metron#763
> > > > > > METRON-1187 Indexing/Profiler Kafka ACL Groups Not Setup
> > > > > > Correctly (nickwallen) closes apache/metron#759
> > > > > > METRON-1185: Stellar REPL does not work on a kerberized
> > > > > > cluster when calling functions interacting with HBase
> > closes
> > > > > > apache/incubator-metron#755
> > > > > > METRON-1186: Profiler Functions use classutils from
> shaded
> > > > > > storm closes apache/incubator-metron#758
> > > > > > METRON-1173: Fix pointers to old stellar docs closes
> > > > > > apache/incubator-metron#746
> > > > > > METRON-1179: Make STATS_ADD to take a list closes
> > > > > > apache/incubator-metron#750
> > > > > > METRON-1180: Make Stellar Shell accept zookeeper quorum
> as
> > a
> > > > > > CSV list and not require a port closes
> > > apache/incubator-metron#751
> > > > > > METRON-1183 Improve KDC Setup Instructions (nickwallen)
> > > > > closes
> > > > > > apache/metron#753
> > > > > > METRON-1177 Stale running topologies seen
> > post-kerberization
> > > > > > and cause exceptions (nickwallen) closes
> apache/metron#748
> > > > > > METRON-1158 Build backend for grouping alerts into meta
> > > > > alerts
> > > > > > (justinleet) closes apache/metron#734
> > > > > > METRON-1146: Add ability to parse JSON string into
> > JSONObject
> > > > > > for stellar closes apache/incubator-metron#727
> > > > > > METRON-1176 REST: HDFS Service should support setting
> > > > > > permissions on files when writing (ottobackwards) closes
> > > > > apache/metron#749
> > > > > > METRON-1114 Add group by capabilities to search REST
> > endpoint
> > > > > > (merrimanr) closes apache/metron#702
> > > > > > METRON-1167 Define Session Specific Global Configuration
> > > > > > Values in the REPL (nickwallen) closes apache/metron#740
> > > > > > METRON-1171: Better validation for the SUBSTRING stellar
> > > > > > function closes apache/incubator-metron#745
> > > > > >
> > > > > >
> > > > > >
> > > > > > On 11/17/17, 11:59 AM, "Nick Allen" <ni...@nickallen.org>
> > wrote:
> > > > > >
> > > > > > I just wanted to send an update on where we are at. We've
> > > > > > gotten a lot
> > > > > > done here recently as you can see below.
> > > > > >
> > > > > > ✓ DONE (1) First, METRON-1289 needs to go in. This one
> was
> > > > > > a fairly big
> > > > > > effort and I am hearing that we are pretty close.
> > > > > >
> > > > > > ✓ DONE (2) METRON-1294 fixes an issue in how field types
> > > > > are
> > > > > > looked-up.
> > > > > >
> > > > > > ✓ DONE (3) METRON-1290 is next. While this may have been
> > > > > > fixed in
> > > > > > M-1289, there may be some test cases we want from this
> PR.
> > > > > >
> > > > > > ✓ DONE (4) METRON-1301 addresses a problem with the
> sorting
> > > > > > logic.
> > > > > >
> > > > > > ✓ DONE (5) METRON-1291 fixes an issue with escalation of
> > > > > > metaalerts.
> > > > > >
> > > > > > (6) That leads us to Raghu's UI work in METRON-1252. This
> > > > > > introduces the
> > > > > > UI bits that depend on all the previous backend work.
> > > > > >
> > > > > > (7) At this point, we should have our best effort at
> > > > > running
> > > > > > Metaalerts
> > > > > > on Elasticsearch 2.x. I propose that we cut a release
> here.
> > > > > >
> > > > > > (8) After we cut the release, we can introduce the work
> for
> > > > > > ES 5.x in
> > > > > > METRON-939. I know we will need lots of help testing and
> > > > > > reviewing this
> > > > > > one.
> > > > > >
> > > > > >
> > > > > >
> > > > > > We also have an outstanding question that needs resolved
> > > > > > BEFORE we
> > > > > > release. We need to come to a consensus on how to release
> > > > > > having moved our
> > > > > > Bro Plugin to a separate repo. I don't think we've heard
> > > > > from
> > > > > > everyone on
> > > > > > this. I'd urge everyone to chime in so we can choose a
> path
> > > > > > forward.
> > > > > >
> > > > > > If anyone is totally confused in regards to that
> > discussion,
> > > > > I
> > > > > > can try and
> > > > > > send an options summary again as a separate discuss
> thread.
> > > > > > The original
> > > > > > chain was somewhere around here [1].
> > > > > >
> > > > > > [1]
> > > > > > https://lists.apache.org/thread.html/
> > > > > > 54a4474881b97e559df24728b3a0e9
> 23a58345a282451085eef832ef@%
> > > > > > 3Cdev.metron.apache.org%3E
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Wed, Nov 15, 2017 at 10:04 AM, Nick Allen <
> > > > > > nick@nickallen.org> wrote:
> > > > > >
> > > > > > > Hi Guys -
> > > > > > >
> > > > > > > I want to follow-up on this discussion. It sounds like
> > > > > most
> > > > > > people are in
> > > > > > > agreement with the general approach.
> > > > > > >
> > > > > > > A lot of people have been working hard on Metaalerts
> and
> > > > > > Elasticsearch. I
> > > > > > > have checked-in with those doing the heavy lifting and
> > have
> > > > > > compiled a more
> > > > > > > detailed plan based on where we are at now. To the best
> > of
> > > > > > my knowledge
> > > > > > > here is the plan of attack for finishing out this
> effort.
> > > > > > >
> > > > > > > (1) First, METRON-1289 needs to go in. This one was a
> > > > > > fairly big effort
> > > > > > > and I am hearing that we are pretty close.
> > > > > > >
> > > > > > > (2) METRON-1294 fixes an issue in how field types are
> > > > > > looked-up.
> > > > > > >
> > > > > > > (3) METRON-1290 is next. While this may have been fixed
> > > > > > in M-1289,
> > > > > > > there may be some test cases we want from this PR.
> > > > > > >
> > > > > > > (4) METRON-1301 addresses a problem with the sorting
> > > > > logic.
> > > > > > >
> > > > > > > (5) METRON-1291 fixes an issue with escalation of
> > > > > > metaalerts.
> > > > > > >
> > > > > > > (6) That leads us to Raghu's UI work in METRON-1252.
> > > > > This
> > > > > > introduces
> > > > > > > the UI bits that depend on all the previous backend
> work.
> > > > > > >
> > > > > > > (7) At this point, we should have our best effort at
> > > > > > running Metaalerts
> > > > > > > on Elasticsearch 2.x. I propose that we cut a release
> > here.
> > > > > > >
> > > > > > > (8) After we cut the release, we can introduce the work
> > > > > > for ES 5.x in
> > > > > > > METRON-939. I know we will need lots of help testing
> and
> > > > > > reviewing this
> > > > > > > one.
> > > > > > >
> > > > > > > Please correct me if I am wrong. I will try and send
> out
> > > > > > updates as we
> > > > > > > make progress.
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > On Mon, Nov 6, 2017 at 1:03 PM, Zeolla@GMail.com <
> > > > > > zeolla@gmail.com> wrote:
> > > > > > >
> > > > > > >> I agree, I think it's very reasonable to move in line
> > with
> > > > > > Nick's
> > > > > > >> proposal. I would also suggest that we outline what
> the
> > > > > > target versions
> > > > > > >> would be to add in the METRON-777 components, since it
> > has
> > > > > > been functional
> > > > > > >> for a very long time but not reviewed and has some
> > really
> > > > > > rockstar
> > > > > > >> improvements.
> > > > > > >>
> > > > > > >> Jon
> > > > > > >>
> > > > > > >> On Mon, Nov 6, 2017 at 12:56 PM Otto Fowler <
> > > > > > ottobackwards@gmail.com>
> > > > > > >> wrote:
> > > > > > >>
> > > > > > >> > I think the ES cutover should be the start of the
> > 0.5.x
> > > > > > series, and we
> > > > > > >> > continue on with 0.4.x for the
> > > > > > >> > metadata improvements etc. We could chose to focus
> > > > > > 0.5.x’s first
> > > > > > >> releases
> > > > > > >> > on not only ES but
> > > > > > >> > getting a handle on kibana and the mpack situation
> as
> > > > > > well.
> > > > > > >> >
> > > > > > >> >
> > > > > > >> >
> > > > > > >> >
> > > > > > >> > On November 6, 2017 at 12:48:45, Michael Miklavcic (
> > > > > > >> > michael.miklavcic@gmail.com) wrote:
> > > > > > >> >
> > > > > > >> > I agree with your proposal, Nick. I think having a
> > > > > > stabilizing release
> > > > > > >> > prior to upgrading ES/Kibana makes sense.
> > > > > > >> >
> > > > > > >> > On Mon, Nov 6, 2017 at 9:16 AM, Nick Allen <
> > > > > > nick@nickallen.org> wrote:
> > > > > > >> >
> > > > > > >> > > I would like to start a discussion around upcoming
> > > > > > releases. We have a
> > > > > > >> > > couple separate significant tracks of work that we
> > > > > need
> > > > > > to reconcile
> > > > > > >> in
> > > > > > >> > our
> > > > > > >> > > release schedule.
> > > > > > >> > >
> > > > > > >> > > (1) We have had (and have in review) a good number
> > of
> > > > > > bug fixes
> > > > > > >> required
> > > > > > >> > to
> > > > > > >> > > support Metaalerts on the existing Elasticsearch
> 2.x
> > > > > > infrastructure.
> > > > > > >> > >
> > > > > > >> > >
> > > > > > >> > > (2) We also have ongoing work to upgrade our
> > > > > > infrastructure to
> > > > > > >> > > Elasticsearch 5.x, which will not be backwards
> > > > > > compatible.
> > > > > > >> > >
> > > > > > >> > >
> > > > > > >> > > I would like to see a release that has our best
> work
> > > > > on
> > > > > > ES 2.x before
> > > > > > >> we
> > > > > > >> > > migrate to 5.x. I would propose the following.
> > > > > > >> > >
> > > > > > >> > > Release N+1: Introduce Metaalerts running on ES
> 2.x
> > > > > > >> > >
> > > > > > >> > > Release N+2: Cut-over to ES 5.x
> > > > > > >> > >
> > > > > > >> > >
> > > > > > >> > > (Q) Is it worth cutting a separate release for ES
> > 2.x?
> > > > > > Is there a
> > > > > > >> better
> > > > > > >> > > way to handle the cut-over to 5.x?
> > > > > > >> > >
> > > > > > >> >
> > > > > > >> --
> > > > > > >>
> > > > > > >> Jon
> > > > > > >>
> > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > >
> > > > --
> > > >
> > > > Jon
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > >
> > >
> > >
> > >
> >
> >
> >
>
Re: [DISCUSS] Upcoming Release
Posted by Nick Allen <ni...@nickallen.org>.
Thanks, Matt.
Maybe you already have something that does this. I wrote a quick script
that validates each JIRA since the last release tag to make sure they are
marked "Done" and with the correct fix version. I would expect that for
the next release, each JIRA should have status="Done", fix-version="0.4.2".
Unless I am mistaken, we have quite a few that need cleaned up. In the
following output, any line that has a URL indicates that a fix is needed.
Or it least, **I think** a fix is needed.
To the community.... If you see your name with a URL next to it, it would
be great if you could follow that link and fix the JIRA. Otherwise, I will
volunteer to help clean some of these up should some not get addressed.
*$ ./validate-jira-for-release*
*Cloning into 'metron-0.4.2'...*
*remote: Counting objects: 35046, done.*
*remote: Compressing objects: 100% (13698/13698), done.*
*remote: Total 35046 (delta 15708), reused 31645 (delta 12822)*
*Receiving objects: 100% (35046/35046), 53.05 MiB | 6.48 MiB/s, done.*
*Resolving deltas: 100% (15708/15708), done.*
*Fetching origin*
* JIRA STATUS FIX VERSION
ASSIGNEE FIX*
* METRON-1345 Done Michael
Miklavcic https://issues.apache.org/jira/browse/METRON-1345
<https://issues.apache.org/jira/browse/METRON-1345>*
* METRON-1349 Done Next + 1 Nick
Allen https://issues.apache.org/jira/browse/METRON-1349
<https://issues.apache.org/jira/browse/METRON-1349>*
* METRON-1343 Done
Mohan https://issues.apache.org/jira/browse/METRON-1343
<https://issues.apache.org/jira/browse/METRON-1343>*
* METRON-1306 To Do
Unassigned https://issues.apache.org/jira/browse/METRON-1306
<https://issues.apache.org/jira/browse/METRON-1306>*
* METRON-1341 Done Simon Elliston
Ball https://issues.apache.org/jira/browse/METRON-1341
<https://issues.apache.org/jira/browse/METRON-1341>*
* METRON-1313 Done Jon
Zeolla https://issues.apache.org/jira/browse/METRON-1313
<https://issues.apache.org/jira/browse/METRON-1313>*
* METRON-1346 Done Otto
Fowler https://issues.apache.org/jira/browse/METRON-1346
<https://issues.apache.org/jira/browse/METRON-1346>*
* METRON-1336 Done 0.4.2 Nick
Allen*
* METRON-1335 Done Anand
Subramanian https://issues.apache.org/jira/browse/METRON-1335
<https://issues.apache.org/jira/browse/METRON-1335>*
* METRON-1308 Done Jon
Zeolla https://issues.apache.org/jira/browse/METRON-1308
<https://issues.apache.org/jira/browse/METRON-1308>*
* METRON-1338 Done 0.4.2 Nick
Allen*
* METRON-1286 To Do 0.4.2
Unassigned https://issues.apache.org/jira/browse/METRON-1286
<https://issues.apache.org/jira/browse/METRON-1286>*
* METRON-1334 Done 0.4.2 Nick
Allen*
* METRON-1277 Done Otto
Fowler https://issues.apache.org/jira/browse/METRON-1277
<https://issues.apache.org/jira/browse/METRON-1277>*
* METRON-1239 To Do
Unassigned https://issues.apache.org/jira/browse/METRON-1239
<https://issues.apache.org/jira/browse/METRON-1239>*
* METRON-1328 Done Anand
Subramanian https://issues.apache.org/jira/browse/METRON-1328
<https://issues.apache.org/jira/browse/METRON-1328>*
* METRON-1333 Done Otto
Fowler https://issues.apache.org/jira/browse/METRON-1333
<https://issues.apache.org/jira/browse/METRON-1333>*
* METRON-1252 Done
RaghuMitra https://issues.apache.org/jira/browse/METRON-1252
<https://issues.apache.org/jira/browse/METRON-1252>*
* METRON-1316 To Do Next + 1
Unassigned https://issues.apache.org/jira/browse/METRON-1316
<https://issues.apache.org/jira/browse/METRON-1316>*
* METRON-1088 Done Jon
Zeolla https://issues.apache.org/jira/browse/METRON-1088
<https://issues.apache.org/jira/browse/METRON-1088>*
* METRON-1319 To Do Ryan
Merriman https://issues.apache.org/jira/browse/METRON-1319
<https://issues.apache.org/jira/browse/METRON-1319>*
* METRON-1321 To Do
Unassigned https://issues.apache.org/jira/browse/METRON-1321
<https://issues.apache.org/jira/browse/METRON-1321>*
* METRON-1301 Done 0.4.2 Nick
Allen*
* METRON-1294 To Do Ryan
Merriman https://issues.apache.org/jira/browse/METRON-1294
<https://issues.apache.org/jira/browse/METRON-1294>*
* METRON-1291 To Do Ryan
Merriman https://issues.apache.org/jira/browse/METRON-1291
<https://issues.apache.org/jira/browse/METRON-1291>*
* METRON-1290 Done Justin
Leet https://issues.apache.org/jira/browse/METRON-1290
<https://issues.apache.org/jira/browse/METRON-1290>*
* METRON-1311 Done 0.4.2 Nick
Allen*
* METRON-1289 To Do Ryan
Merriman https://issues.apache.org/jira/browse/METRON-1289
<https://issues.apache.org/jira/browse/METRON-1289>*
* METRON-1309 Done Jon
Zeolla https://issues.apache.org/jira/browse/METRON-1309
<https://issues.apache.org/jira/browse/METRON-1309>*
* METRON-1310 Done 0.4.2 Nick
Allen*
* METRON-1275 Done Jon
Zeolla https://issues.apache.org/jira/browse/METRON-1275
<https://issues.apache.org/jira/browse/METRON-1275>*
* METRON-1295 Done 0.4.2 Nick
Allen*
* METRON-1307 Done Otto
Fowler https://issues.apache.org/jira/browse/METRON-1307
<https://issues.apache.org/jira/browse/METRON-1307>*
* METRON-1296 Done 0.4.2 Nick
Allen*
* METRON-1281 To Do Ryan
Merriman https://issues.apache.org/jira/browse/METRON-1281
<https://issues.apache.org/jira/browse/METRON-1281>*
* METRON-1287 Done 0.4.2 Nick
Allen*
* METRON-1267 To Do
Unassigned https://issues.apache.org/jira/browse/METRON-1267
<https://issues.apache.org/jira/browse/METRON-1267>*
* METRON-1283 Done Anand
Subramanian https://issues.apache.org/jira/browse/METRON-1283
<https://issues.apache.org/jira/browse/METRON-1283>*
* METRON-1254 To Do
Unassigned https://issues.apache.org/jira/browse/METRON-1254
<https://issues.apache.org/jira/browse/METRON-1254>*
* METRON-1261 Done Jon
Zeolla https://issues.apache.org/jira/browse/METRON-1261
<https://issues.apache.org/jira/browse/METRON-1261>*
* METRON-1284 Done Justin
Leet https://issues.apache.org/jira/browse/METRON-1284
<https://issues.apache.org/jira/browse/METRON-1284>*
* METRON-1270 Done Artem
Ervits https://issues.apache.org/jira/browse/METRON-1270
<https://issues.apache.org/jira/browse/METRON-1270>*
* METRON-1272 In Progress Justin
Leet https://issues.apache.org/jira/browse/METRON-1272
<https://issues.apache.org/jira/browse/METRON-1272>*
* METRON-1224 Done
RaghuMitra https://issues.apache.org/jira/browse/METRON-1224
<https://issues.apache.org/jira/browse/METRON-1224>*
* METRON-1280 To Do
Unassigned https://issues.apache.org/jira/browse/METRON-1280
<https://issues.apache.org/jira/browse/METRON-1280>*
* METRON-1243 To Do
Unassigned https://issues.apache.org/jira/browse/METRON-1243
<https://issues.apache.org/jira/browse/METRON-1243>*
* METRON-1196 Done Matt
Foley https://issues.apache.org/jira/browse/METRON-1196
<https://issues.apache.org/jira/browse/METRON-1196>*
* METRON-1278 Done 0.4.2 Matt
Foley*
* METRON-1274 To Do
Unassigned https://issues.apache.org/jira/browse/METRON-1274
<https://issues.apache.org/jira/browse/METRON-1274>*
* METRON-1266 Done 0.4.2 Nick
Allen*
* METRON-1260 Done 0.4.2 Nick
Allen*
* METRON-1251 Done Jon
Zeolla https://issues.apache.org/jira/browse/METRON-1251
<https://issues.apache.org/jira/browse/METRON-1251>*
* METRON-1241 To Do
Unassigned https://issues.apache.org/jira/browse/METRON-1241
<https://issues.apache.org/jira/browse/METRON-1241>*
* METRON-1267 To Do
Unassigned https://issues.apache.org/jira/browse/METRON-1267
<https://issues.apache.org/jira/browse/METRON-1267>*
* METRON-1262 To Do Ryan
Merriman https://issues.apache.org/jira/browse/METRON-1262
<https://issues.apache.org/jira/browse/METRON-1262>*
* METRON-1263 Done Anand
Subramanian https://issues.apache.org/jira/browse/METRON-1263
<https://issues.apache.org/jira/browse/METRON-1263>*
* METRON-1255 To Do
Unassigned https://issues.apache.org/jira/browse/METRON-1255
<https://issues.apache.org/jira/browse/METRON-1255>*
* METRON-1249 Done 0.4.1 Nick
Allen https://issues.apache.org/jira/browse/METRON-1249
<https://issues.apache.org/jira/browse/METRON-1249>*
* METRON-1237 To Do Artem
Ervits https://issues.apache.org/jira/browse/METRON-1237
<https://issues.apache.org/jira/browse/METRON-1237>*
* METRON-1240 Done Artem
Ervits https://issues.apache.org/jira/browse/METRON-1240
<https://issues.apache.org/jira/browse/METRON-1240>*
* METRON-1226 Done 0.4.2 Nick
Allen*
* METRON-1081 Done James
Sirota https://issues.apache.org/jira/browse/METRON-1081
<https://issues.apache.org/jira/browse/METRON-1081>*
* METRON-1123 Done
RaghuMitra https://issues.apache.org/jira/browse/METRON-1123
<https://issues.apache.org/jira/browse/METRON-1123>*
* METRON-1223 Done
RaghuMitra https://issues.apache.org/jira/browse/METRON-1223
<https://issues.apache.org/jira/browse/METRON-1223>*
* METRON-1083 Done
RaghuMitra https://issues.apache.org/jira/browse/METRON-1083
<https://issues.apache.org/jira/browse/METRON-1083>*
* METRON-1232 Done
RaghuMitra https://issues.apache.org/jira/browse/METRON-1232
<https://issues.apache.org/jira/browse/METRON-1232>*
* METRON-1247 Done Justin
Leet https://issues.apache.org/jira/browse/METRON-1247
<https://issues.apache.org/jira/browse/METRON-1247>*
* METRON-1235 To Do
Unassigned https://issues.apache.org/jira/browse/METRON-1235
<https://issues.apache.org/jira/browse/METRON-1235>*
* METRON-1234 Done Artem
Ervits https://issues.apache.org/jira/browse/METRON-1234
<https://issues.apache.org/jira/browse/METRON-1234>*
* METRON-1222 Done Artem
Ervits https://issues.apache.org/jira/browse/METRON-1222
<https://issues.apache.org/jira/browse/METRON-1222>*
* METRON-1220 In Progress Justin
Leet https://issues.apache.org/jira/browse/METRON-1220
<https://issues.apache.org/jira/browse/METRON-1220>*
* METRON-1229 To Do
Unassigned https://issues.apache.org/jira/browse/METRON-1229
<https://issues.apache.org/jira/browse/METRON-1229>*
* METRON-1228 Done
Unassigned https://issues.apache.org/jira/browse/METRON-1228
<https://issues.apache.org/jira/browse/METRON-1228>*
* METRON-1218 To Do Ryan
Merriman https://issues.apache.org/jira/browse/METRON-1218
<https://issues.apache.org/jira/browse/METRON-1218>*
* METRON-1161 In Progress Ryan
Merriman https://issues.apache.org/jira/browse/METRON-1161
<https://issues.apache.org/jira/browse/METRON-1161>*
* METRON-1209 To Do
Unassigned https://issues.apache.org/jira/browse/METRON-1209
<https://issues.apache.org/jira/browse/METRON-1209>*
* METRON-1059 Done Next + 1
Unassigned https://issues.apache.org/jira/browse/METRON-1059
<https://issues.apache.org/jira/browse/METRON-1059>*
* METRON-1204 To Do Ryan
Merriman https://issues.apache.org/jira/browse/METRON-1204
<https://issues.apache.org/jira/browse/METRON-1204>*
* METRON-1052 Done Casey
Stella https://issues.apache.org/jira/browse/METRON-1052
<https://issues.apache.org/jira/browse/METRON-1052>*
* METRON-632 In Progress Tomas
Zezula https://issues.apache.org/jira/browse/METRON-632
<https://issues.apache.org/jira/browse/METRON-632>*
* METRON-1194 Done 0.4.2 Nick
Allen*
* METRON-1055 To Do Laurens
Vets https://issues.apache.org/jira/browse/METRON-1055
<https://issues.apache.org/jira/browse/METRON-1055>*
* METRON-1079 Done Otto
Fowler https://issues.apache.org/jira/browse/METRON-1079
<https://issues.apache.org/jira/browse/METRON-1079>*
* METRON-1085 To Do Ryan
Merriman https://issues.apache.org/jira/browse/METRON-1085
<https://issues.apache.org/jira/browse/METRON-1085>*
* METRON-1208 To Do Ryan
Merriman https://issues.apache.org/jira/browse/METRON-1208
<https://issues.apache.org/jira/browse/METRON-1208>*
* METRON-1207 To Do Ryan
Merriman https://issues.apache.org/jira/browse/METRON-1207
<https://issues.apache.org/jira/browse/METRON-1207>*
* METRON-1215 To Do
Unassigned https://issues.apache.org/jira/browse/METRON-1215
<https://issues.apache.org/jira/browse/METRON-1215>*
* METRON-1206 To Do Ryan
Merriman https://issues.apache.org/jira/browse/METRON-1206
<https://issues.apache.org/jira/browse/METRON-1206>*
* METRON-1195 To Do Justin
Leet https://issues.apache.org/jira/browse/METRON-1195
<https://issues.apache.org/jira/browse/METRON-1195>*
* METRON-1189 To Do Ryan
Merriman https://issues.apache.org/jira/browse/METRON-1189
<https://issues.apache.org/jira/browse/METRON-1189>*
* METRON-1156 Done 0.4.2 Nick
Allen*
* METRON-1198 Done 0.4.2 Nick
Allen*
* METRON-1202 Done Next + 1 Justin
Leet https://issues.apache.org/jira/browse/METRON-1202
<https://issues.apache.org/jira/browse/METRON-1202>*
* METRON-938 Done Next + 1 Justin
Leet https://issues.apache.org/jira/browse/METRON-938
<https://issues.apache.org/jira/browse/METRON-938>*
* METRON-1182 Done
RaghuMitra https://issues.apache.org/jira/browse/METRON-1182
<https://issues.apache.org/jira/browse/METRON-1182>*
* METRON-1188 Done Michael
Miklavcic https://issues.apache.org/jira/browse/METRON-1188
<https://issues.apache.org/jira/browse/METRON-1188>*
* METRON-1191 Done Matt
Foley https://issues.apache.org/jira/browse/METRON-1191
<https://issues.apache.org/jira/browse/METRON-1191>*
* METRON-1063 Done Next + 1 Artem
Ervits https://issues.apache.org/jira/browse/METRON-1063
<https://issues.apache.org/jira/browse/METRON-1063>*
* METRON-1190 In Progress Justin
Leet https://issues.apache.org/jira/browse/METRON-1190
<https://issues.apache.org/jira/browse/METRON-1190>*
* METRON-1187 Done 0.4.2 Nick
Allen*
* METRON-1185 To Do
Unassigned https://issues.apache.org/jira/browse/METRON-1185
<https://issues.apache.org/jira/browse/METRON-1185>*
* METRON-1186 To Do Casey
Stella https://issues.apache.org/jira/browse/METRON-1186
<https://issues.apache.org/jira/browse/METRON-1186>*
* METRON-1173 Done Next + 1 Jon
Zeolla https://issues.apache.org/jira/browse/METRON-1173
<https://issues.apache.org/jira/browse/METRON-1173>*
* METRON-1179 To Do
Unassigned https://issues.apache.org/jira/browse/METRON-1179
<https://issues.apache.org/jira/browse/METRON-1179>*
* METRON-1180 To Do
Unassigned https://issues.apache.org/jira/browse/METRON-1180
<https://issues.apache.org/jira/browse/METRON-1180>*
* METRON-1183 Done 0.4.2 Nick
Allen*
* METRON-1177 Done 0.4.2 Nick
Allen*
* METRON-1158 To Do Justin
Leet https://issues.apache.org/jira/browse/METRON-1158
<https://issues.apache.org/jira/browse/METRON-1158>*
* METRON-1146 Done Anand
Subramanian https://issues.apache.org/jira/browse/METRON-1146
<https://issues.apache.org/jira/browse/METRON-1146>*
* METRON-1176 Done Otto
Fowler https://issues.apache.org/jira/browse/METRON-1176
<https://issues.apache.org/jira/browse/METRON-1176>*
* METRON-1114 Done Ryan
Merriman https://issues.apache.org/jira/browse/METRON-1114
<https://issues.apache.org/jira/browse/METRON-1114>*
* METRON-1167 Done Next + 1 Nick
Allen https://issues.apache.org/jira/browse/METRON-1167
<https://issues.apache.org/jira/browse/METRON-1167>*
* METRON-1171 To Do Casey
Stella https://issues.apache.org/jira/browse/METRON-1171
<https://issues.apache.org/jira/browse/METRON-1171>*
The script is here [1] if anyone wants to run it themselves and grep for
their own name.
[1]
https://github.com/nickwallen/metron-commit-stuff/blob/master/validate-jira-for-release
On Fri, Dec 15, 2017 at 1:18 PM, Matt Foley <mf...@hortonworks.com> wrote:
> Hi Nick,
> Good timing, you’ve saved me some work! :-)
>
> Origin of the list: The approach was defined before I started managing
> releases, but I think it’s a reasonable approach. It’s specified in our
> Release Process document, https://cwiki.apache.org/
> confluence/display/METRON/Release+Process , Step 5, the last bullet:
> “The artifacts for a release or RC [include]… A CHANGES file
> denoting the changes [since the last release].
> We usually construct this by taking the output of git log | grep
> METRON | sed 's/\[//g' | sed 's/\]//g' | grep -v "http" and removing the
> JIRAs from the previous releases (it’s in time sorted order so this is
> easy).”
>
> However, the release manager also has a responsibility to “Monitor and
> Verify Jiras” (Step 2 and various other places in the doc). Altho the doc
> isn’t explicit about it, I take this responsibility a little farther and do
> the following as part of Jira verification:
>
> 1. Extract the jira id’s from the CHANGES file with a simple awk script,
> put them in a Jira query, and confirm they are all actually marked
> “Fixed/Resolved” with Fix Version = <the current release> in Jira. If
> there are exceptions, I dun the contributors to fix it. You’ve seen these
> emails from me in the past couple releases. I don’t just mark them fixed
> myself, because some contributions (perfectly legitimately) only partially
> address a problem, and just marking a ticket “Fixed” may not be the right
> thing to do.
>
> 2. Query jira for any tickets NOT included in the prior list, that ARE
> marked fixed in the current release (or later, or “Next”, or similar
> “future” version constructs). These don’t usually happen (considering that
> Metron contributors mostly live inside github PR processes rather than
> Jiras) but when they do they also need to be resolved. Cases I’ve seen (in
> Metron and other projects) include:
> a) If they were indeed fixed in the current release, but not reflected in
> the commit message, I’ll annotate this fact in the Jira ticket, and
> hand-edit the CHANGES file for this release. (Doesn’t seem worth trying to
> edit old commit messages, since that mucks up the SHA1’s.)
> b) If they were fixed in a previous release but mis-marked as to Fix
> Version in the ticket, fix the ticket.
> c) If they aren’t really fixed, fix the ticket.
>
> 3. Once the set of Fixed tickets is complete and correct, query them for
> any that have “labels in (backward-incompatible, backwards-compatibility,
> backwards-incompatible)” -- yes, there shouldn’t be 3 labels, but it’s hard
> to fix because Jira never forgets, so every time someone gets it wrong, it
> adds to the set; fortunately, query completion helps here – OR "Docs Text"
> is not EMPTY. Each of these needs to be looked at carefully, verified, and
> if Doc Text is missing, the contributor and I cooperate to write a couple
> sentences. The Doc Text then needs to go in the RELEASE_NOTES for the
> release, in the section “## Non-backward-compatible Changes in this
> Release, and Upgrade Suggestions”.
>
> That’s about it. I guess I should add the above as an appendix doc to the
> Release Process documentation. I’ll do that.
> Cheers,
> --Matt
>
>
> On 12/15/17, 9:15 AM, "Nick Allen" <ni...@nickallen.org> wrote:
>
> Hi Matt -
>
> I just updated like 15+ JIRAs of my own JIRAs that I completed and
> merged,
> but failed to mark as resolved. All of these will be included in
> 0.4.2. I
> updated each to be "fixed" in version 0.4.2 and marked as resolved.
> Hopefully the next RC will report those as fixed.
>
> (Q) Where does the list of JIRAs that get attached to the release
> originate
> from? Does it get pulled out of JIRA or do they come from the commit
> log?
>
> My apologies for not staying on top of my JIRAs.
>
>
>
>
> On Tue, Dec 12, 2017 at 2:21 PM, Matt Foley <ma...@apache.org> wrote:
>
> > Thanks to Jon, Otto, and Nick for looking over RC1. Tonight I’ll
> fix the
> > RAT glitch, build RC2, and put it to formal vote.
> > Regards,
> > --Matt
> >
> > On 12/12/17, 5:14 AM, "Nick Allen" <ni...@nickallen.org> wrote:
> >
> > RC1 is looking good to me. I validated the MD5s, built Metron,
> built
> > the
> > Bro plugin and reviewed the other artifacts like release notes.
> >
> > Running the RAT check on a 'clean' Metron does not produce any
> errors
> > for
> > me. It is only after building Metron, which pulls in additional
> Node
> > dependencies, does the RAT check fail.
> >
> >
> > On Sun, Dec 10, 2017 at 4:41 PM Matt Foley <ma...@apache.org>
> wrote:
> >
> > > Yes, but let’s see if anyone else find other issues.
> > >
> > >
> > >
> > > From: Otto Fowler <ot...@gmail.com>
> > > Date: Saturday, December 9, 2017 at 6:16 AM
> > > To: Matt Foley <mf...@hortonworks.com>, "
> dev@metron.apache.org" <
> > > dev@metron.apache.org>
> > > Subject: Re: [DISCUSS] Upcoming Release
> > >
> > >
> > >
> > > So RC2 then?
> > >
> > >
> > >
> > > On December 8, 2017 at 20:43:21, Matt Foley (
> mfoley@hortonworks.com)
> > > wrote:
> > >
> > > Hah, here it is: https://github.com/apache/metron/pull/743
> > > “This problem seems to only reproduce when one unrolls a
> tarball
> > rather
> > > than cloning from github.”
> > >
> > > Heh, the exclusion at
> > > https://github.com/apache/metron/blob/master/pom.xml#L351 is
> still
> > there,
> > > but the hashcode in the bundle.css file name has changed from
> > > a0b6b99c10d9a13dc67e to f56deed131e58bd7ee04. Sigh. Did the
> version
> > of Font
> > > Awesome fonts change?
> > >
> > >
> > > On 12/8/17, 5:26 PM, "Matt Foley" <ma...@apache.org> wrote:
> > >
> > > I remember having trouble with this bundle.css file on the last
> > release,
> > > but I can’t remember what we did about it. Anybody?
> > >
> > > On 12/8/17, 1:41 PM, "Otto Fowler" <ot...@gmail.com>
> wrote:
> > >
> > > Steps
> > >
> > > - Downloaded tar.gz’s, asc files and KEYS
> > > - Verified signing of both tar.gz’s
> > > - searched for rouge 0.4.1 entries
> > > - verified the main pom.xml
> > > - built :
> > >
> > > mvn clean && time mvn -q -T 2C -DskipTests install && time mvn
> -q -T
> > > 2C surefire:test@unit-tests && time mvn -q
> > > surefire:test@integration-tests && time mvn -q test --projects
> > > metron-interface/metron-config && time
> build_utils/verify_licenses.sh
> > >
> > > Found rat error:
> > >
> > >
> > > *****************************************************
> > > Summary
> > > -------
> > > Generated at: 2017-12-08T16:33:27-05:00
> > >
> > > Notes: 3
> > > Binaries: 193
> > > Archives: 0
> > > Standards: 75
> > >
> > > Apache Licensed: 74
> > > Generated Documents: 0
> > >
> > > JavaDocs are generated, thus a license header is optional.
> > > Generated files do not require license headers.
> > >
> > > 1 Unknown Licenses
> > >
> > > *****************************************************
> > >
> > > Files with unapproved licenses:
> > >
> > >
> > > /Users/batman/tmp/release_ver/apache-metron-0.4.2-rc1/
> > metron-interface/metron-alerts/dist/styles.
> f56deed131e58bd7ee04.bundle.css
> > >
> > > *****************************************************
> > >
> > >
> > >
> > >
> > >
> > > *****************************************************
> > > Summary
> > > -------
> > > Generated at: 2017-12-08T16:33:27-05:00
> > >
> > > Notes: 3
> > > Binaries: 193
> > > Archives: 0
> > > Standards: 75
> > >
> > > Apache Licensed: 74
> > > Generated Documents: 0
> > >
> > > JavaDocs are generated, thus a license header is optional.
> > > Generated files do not require license headers.
> > >
> > > 1 Unknown Licenses
> > >
> > > *****************************************************
> > >
> > > Files with unapproved licenses:
> > >
> > >
> > >
> > > /Users/ottofowler/tmp/release_ver/apache-metron-0.4.2-rc1/
> > metron-interface/metron-alerts/dist/styles.
> f56deed131e58bd7ee04.bundle.css
> > >
> > > *****************************************************
> > >
> > >
> > >
> > > On December 8, 2017 at 04:34:24, Matt Foley (mattf@apache.org)
> > wrote:
> > >
> > > Colleagues,
> > > I’ve posted Metron-0.4.2-RC1 and Metron-bro-plugin-kafka-0.1 to
> > > https://dist.apache.org/repos/dist/dev/metron/0.4.2-RC1/
> > >
> > > Given the complexity of this RC, I’d appreciate if a couple
> people
> > would be
> > > willing to kick the tires before we put it up for a vote.
> > >
> > > I will myself be going thru the Verify Build process this
> weekend,
> > as I
> > > won’t be able to do it Friday.
> > >
> > > Thanks,
> > > --Matt
> > >
> > >
> > > On 12/4/17, 2:05 PM, "Zeolla@GMail.com" <ze...@gmail.com>
> wrote:
> > >
> > > Can we resolve the conversation regarding the second repo? I
> was
> > waiting
> > > to get more input/preferences from people There's also a
> > documentation
> > > update that fixes a few broken Stellar docs that already has
> aa +1,
> > I just
> > > need to merge it.
> > >
> > > Jon
> > >
> > > On Mon, Dec 4, 2017, 17:01 Casey Stella <ce...@gmail.com>
> wrote:
> > >
> > > > I would be in favor of a release at this point.
> > > >
> > > > On Mon, Dec 4, 2017 at 4:57 PM, Matt Foley <mattf@apache.org
> >
> > wrote:
> > > >
> > > > > Hey all,
> > > > > I see METRON-1252 was resolved over the weekend. Shall I go
> > ahead and
> > > > > start the process with 0.4.2 release?
> > > > > Does anyone have any commits they feel strongly should go
> in
> > before
> > > 0.4.2
> > > > > is done, or are we ready to call it good?
> > > > >
> > > > > I believe there is consensus the 0.4.2 release should
> include a
> > release
> > > > of
> > > > > the current state of the metron-bro-plugin-kafka. I will
> > continue the
> > > > > discussion in that thread as to the process for
> accomplishing
> > that, but
> > > > > plan on it happening.
> > > > >
> > > > > Regards,
> > > > > --Matt
> > > > >
> > > > > On 11/26/17, 6:26 PM, "Matt Foley" <ma...@apache.org>
> wrote:
> > > > >
> > > > > Hope everyone (at least in the U.S.) had a great
> Thanksgiving
> > > > holiday.
> > > > > Regarding status of the release effort, still pending
> > METRON-1252, so
> > > > > not making the release branch yet.
> > > > >
> > > > > Regards,
> > > > > --Matt
> > > > >
> > > > > On 11/17/17, 1:32 PM, "Matt Foley" <ma...@apache.org>
> wrote:
> > > > >
> > > > > (With release manager hat on)
> > > > >
> > > > > The community has proposed a release of Metron in the near
> > > > future,
> > > > > focusing on Meta-alerts running in Elasticsearch.
> > > > > Congrats on getting so many of the below already done. At
> this
> > > > > point, only METRON-1252, and the discussion of how to
> handle
> > joint
> > > > release
> > > > > of the Metron bro plugin, remain as gating items for the
> > release. I
> > > > > project these will be resolved next week, so let’s propose
> the
> > > following:
> > > > >
> > > > > Sometime next week, after the last bits are done, I’ll
> start the
> > > > > release process and create the release branch.
> > > > >
> > > > > The proposed new version will be 0.4.2, unless there are
> backward
> > > > > incompatible changes that support making it 0.5.0.
> > > > > Currently there are NO included Jiras labeled
> > > > > ‘backward-incompatible’, nor having Docs Text indicating
> so.
> > > > > ***If anyone knows that some of the commits included since
> 0.4.1
> > > > > introduce backward incompatibility, please say so now on
> this
> > thread,
> > > and
> > > > > mark the Jira as such.***
> > > > >
> > > > > The 90 or so jiras/commits already in master branch since
> 0.4.1
> > > > > are listed below.
> > > > > Thanks,
> > > > > --Matt
> > > > >
> > > > > METRON-1301 Alerts UI - Sorting on Triage Score
> Unexpectedly
> > > > > Filters Some Records (nickwallen) closes apache/metron#832
> > > > > METRON-1294 IP addresses are not formatted correctly in
> facet
> > > > > and group results (merrimanr) closes apache/metron#827
> > > > > METRON-1291 Kafka produce REST endpoint does not work in a
> > > > > Kerberized cluster (merrimanr) closes apache/metron#826
> > > > > METRON-1290 Only first 10 alerts are update when a
> MetaAlert
> > > > > status is changed to inactive (justinleet) closes
> > apache/metron#842
> > > > > METRON-1311 Service Check Should Check Elasticsearch Index
> > > > > Templates (nickwallen) closes apache/metron#839
> > > > > METRON-1289 Alert fields are lost when a MetaAlert is
> created
> > > > > (merrimanr) closes apache/metron#824
> > > > > METRON-1309 Change metron-deployment to pull the plugin
> from
> > > > > apache/metron-bro-plugin-kafka (JonZeolla) closes
> > apache/metron#837
> > > > > METRON-1310 Template Delete Action Deletes Search Indices
> > > > > (nickwallen) closes apache/metron#838
> > > > > METRON-1275: Fix Metron Documentation closes
> > > > > apache/incubator-metron#833
> > > > > METRON-1295 Unable to Configure Logging for REST API
> > > > > (nickwallen) closes apache/metron#828
> > > > > METRON-1307 Force install of java8 since java9 does not
> > > > appear
> > > > > to work with the scripts (brianhurley via ottobackwards)
> closes
> > > > > apache/metron#835
> > > > > METRON-1296 Full Dev Fails to Deploy Index Templates
> > > > > (nickwallen via cestella) closes
> apache/incubator-metron#829
> > > > > METRON-1281 Remove hard-coded indices from the Alerts UI
> > > > > (merrimanr) closes apache/metron#821
> > > > > METRON-1287 Full Dev Fails When Installing EPEL Repository
> > > > > (nickwallen) closes apache/metron#820
> > > > > METRON-1267 Alerts UI returns a 404 when refreshing the
> > > > > alerts-list page (iraghumitra via merrimanr) closes
> > apache/metron#819
> > > > > METRON-1283 Install Elasticsearch template as a part of the
> > > > > mpack startup scripts (anandsubbu via nickwallen) closes
> > > > apache/metron#817
> > > > > METRON-1254: Conditionals as map keys do not function in
> > > > > Stellar closes apache/incubator-metron#801
> > > > > METRON-1261 Apply bro security patch (JonZeolla via
> > > > > ottobackwards) closes apache/metron#805
> > > > > METRON-1284 Remove extraneous dead query in
> ElasticsearchDao
> > > > > (justinleet) closes apache/metron#818
> > > > > METRON-1270: fix for warnings missing @return tag argument
> in
> > > > > metron-analytics/metron-profiler-common and
> > metron-profiler-client
> > > closes
> > > > > apache/incubator-metron#810
> > > > > METRON-1272 Hide child alerts from searches and grouping if
> > > > > they belong to meta alerts (justinleet) closes
> apache/metron#811
> > > > > METRON-1224 Add time range selection to search control
> > > > > (iraghumitra via james-sirota) closes apache/metron#796
> > > > > METRON-1280 0.4.1 -> 0.4.2 missed a couple of projects
> > > > > (cestella via justinleet) closes apache/metron#816
> > > > > METRON-1243: Add a REST endpoint which allows us to get a
> > > > list
> > > > > of all indice closes apache/incubator-metron#797
> > > > > METRON-1196 Increment master version number to 0.4.2 for
> > > > > on-going development (mattf-horton) closes
> apache/metron#767
> > > > > METRON-1278 Strip "Build Status" widget from root
> > > > > README.md in site-book build (mattf-horton) closes
> > apache/metron#815
> > > > > METRON-1274 Master has failure in
> > > > > StormControllerIntegrationTest (merrimanr) closes
> > apache/metron#813
> > > > > METRON-1266 Profiler - SASL Authentication Failed
> > > > (nickwallen)
> > > > > closes apache/metron#809
> > > > > METRON-1260 Include Alerts UI in Ambari Service Check
> > > > > (nickwallen) closes apache/metron#804
> > > > > METRON-1251: Typo and formatting fixes for metron-rest
> README
> > > > > closes apache/incubator-metron#800
> > > > > METRON-1241: Enable the REST API to use a cache for the
> > > > > zookeeper config similar to the Bolts closes
> > > apache/incubator-metron#795
> > > > > METRON-1267 Alerts UI returns a 404 when refreshing the
> > > > > alerts-list page (merrimanr) closes apache/metron#808
> > > > > METRON-1262 Unable to add comment for a alert in a
> meta-alert
> > > > > (merrimanr) closes apache/metron#806
> > > > > METRON-1263 Start Alerts UI service after Metron REST
> > > > > (anandsubbu via nickwallen) closes apache/metron#807
> > > > > METRON-1255 MetaAlert search is not filtering on status
> > > > > (merrimanr) closes apache/metron#802
> > > > > METRON-1249 Improve Metron MPack Service Checks
> (nickwallen)
> > > > > closes apache/metron#799
> > > > > METRON-1237 address javadoc warnings in metron-maas-common
> > > > > (dbist via james-sirota) closes apache/metron#792
> > > > > METRON-1240 address javadoc warnings in metron-platform and
> > > > > metron-analytics (dbist via james-sirota) closes
> > apache/metron#794
> > > > > METRON-1226 Searching Can Errantly Query the Wrong Indices
> > > > > (nickwallen) closes apache/metron#793
> > > > > METRON-1081 Fix Alerts and Ops UI Notices file
> (james-sirota)
> > > > > closes apache/metron#682
> > > > > METRON-1123 Add group by option using faceted search
> > > > > capabilities of metron-rest-api (iraghumitra via
> james-sirota)
> > closes
> > > > > apache/metron#768
> > > > > METRON-1223 Add support to add comments for alerts
> > > > > (iraghumitra via james-sirota) closes apache/metron#788
> > > > > METRON-1083 Add filters using faceted search capabilities
> of
> > > > > metron-rest-api (iraghumitra via james-sirota) closes
> > apache/metron#710
> > > > > METRON-1232 Alert status changes are not reflected in list
> > > > > view (iraghumitra via merrimanr) closes apache/metron#787
> > > > > METRON-1247 REST search and findOne endpoints return
> > > > > unexpected or incorrect results for guids (justinleet)
> closes
> > > > > apache/metron#798
> > > > > METRON-1235: Document the properties pulled from the global
> > > > > configuration closes apache/incubator-metron#791
> > > > > METRON-1234: fix for WARNING 'dependencies.dependency.(
> > > > > groupId:artifactId:type:classifier)' must be unique:
> > > > > org.apache.hadoop:hadoop-yarn-api:jar (dbist via mmiklavc)
> > closes
> > > > > apache/metron#790
> > > > > METRON-1222: fix warning for The expression
> ${parent.version}
> > > > > is deprecated. Please use ${project.parent.version}
> instead.
> > (dbist via
> > > > > mmiklavc) closes apache/metron#782
> > > > > METRON-1220 Create documentation around alert nested field
> > > > > (justinleet) closes apache/metron#780
> > > > > METRON-1229 Management UI type is part of the declarations
> of
> > > > > 2 modules (merrimanr) closes apache/metron#784
> > > > > METRON-1228: Configuration Management PUSH immediately does
> > > > > DUMP after (mmiklavc via mmiklavc) closes apache/metron#783
> > > > > METRON-1218 Metron REST should return better error messages
> > > > > (merrimanr) closes apache/metron#779
> > > > > METRON-1161 Add ability to edit parser command line options
> > > > in
> > > > > the management UI (merrimanr) closes apache/metron#737
> > > > > METRON-1209: Make stellar repl take logging properties,
> like
> > > > > other CLI apps in metron closes apache/incubator-metron#772
> > > > > METRON-1059 address checkstyle warning AvoidStarImport in
> > > > > metron-stellar (dbist via ottobackwards) closes
> apache/metron#664
> > > > > METRON-1204 UI does not time out after being idle, but
> stops
> > > > > functioning (merrimanr) closes apache/metron#771
> > > > > METRON-1052: Add forensic similarity hash functions to
> > > > Stellar
> > > > > closes apache/incubator-metron#781
> > > > > METRON-632: Added validation of "shew.enrichmentType" and
> > > > > "shew.keyColumns" closes apache/incubator-metron#732
> > > > > METRON-1194 Add Profiler Debug Functions to Profiler README
> > > > > (nickwallen via ottobackwards) closes apache/metron#765
> > > > > METRON-1055 Metron 0.4.0 manual installation guide for
> CentOS
> > > > > 6 updates (lvets via ottobackwards) closes
> apache/metron#661
> > > > > METRON-1079 STELLAR NaN should be a keyword (ottobackwards)
> > > > > closes apache/metron#681
> > > > > METRON-1085 Add REST endpoint to save a user profile for
> the
> > > > > Alerts UI (merrimanr) closes apache/metron#694
> > > > > METRON-1208 MPack for Alerts UI (merrimanr) closes
> > > > > apache/metron#778
> > > > > METRON-1207 Make RPMs for Alerts UI (merrimanr) closes
> > > > > apache/metron#777
> > > > > METRON-1215 Fix link to RPMs chapter (DimDroll via
> > > > justinleet)
> > > > > closes apache/metron#776
> > > > > METRON-1206 Make alerts UI conform to ops UI for install
> > > > > (merrimanr) closes apache/metron#773
> > > > > METRON-1195 Meta alerts improperly handle updates to
> > > > non-alert
> > > > > fields (justinleet) closes apache/metron#766
> > > > > METRON-1189 Add alert escalation to the Alerts UI
> (merrimanr)
> > > > > closes apache/metron#762
> > > > > METRON-1156 Simulate Triage Rules in the Stellar REPL
> > > > > (nickwallen) closes apache/metron#733
> > > > > METRON-1198 Pycapa - No such configuration property
> > > > > 'sasl.kerberos.principal' (nickwallen) closes
> apache/metron#769
> > > > > METRON-1202 ElasticsearchDao Has extraneous sleep call
> > > > > (justinleet) closes apache/metron#770
> > > > > METRON-938 "service metron-rest start <password>" does not
> > > > > work on CentOS 7. (justinleet) closes apache/metron#757
> > > > > METRON-1182 Refactor Code in alert list to accommodate new
> > > > > view types (iraghumitra via merrimanr) closes
> apache/metron#756
> > > > > METRON-1188: Ambari global configuration management
> > > > (mmiklavc)
> > > > > closes apache/metron#760
> > > > > METRON-1191 update public web site to point at 0.4.1 new
> > > > > release (mattf-horton) closes apache/metron#764
> > > > > METRON-1063 address javadoc warnings in metron-stellar
> (dbist
> > > > > via ottobackwards) closes apache/metron#668
> > > > > METRON-1190 Fix Meta Alert Type handling in calculation of
> > > > > scores (justinleet) closes apache/metron#763
> > > > > METRON-1187 Indexing/Profiler Kafka ACL Groups Not Setup
> > > > > Correctly (nickwallen) closes apache/metron#759
> > > > > METRON-1185: Stellar REPL does not work on a kerberized
> > > > > cluster when calling functions interacting with HBase
> closes
> > > > > apache/incubator-metron#755
> > > > > METRON-1186: Profiler Functions use classutils from shaded
> > > > > storm closes apache/incubator-metron#758
> > > > > METRON-1173: Fix pointers to old stellar docs closes
> > > > > apache/incubator-metron#746
> > > > > METRON-1179: Make STATS_ADD to take a list closes
> > > > > apache/incubator-metron#750
> > > > > METRON-1180: Make Stellar Shell accept zookeeper quorum as
> a
> > > > > CSV list and not require a port closes
> > apache/incubator-metron#751
> > > > > METRON-1183 Improve KDC Setup Instructions (nickwallen)
> > > > closes
> > > > > apache/metron#753
> > > > > METRON-1177 Stale running topologies seen
> post-kerberization
> > > > > and cause exceptions (nickwallen) closes apache/metron#748
> > > > > METRON-1158 Build backend for grouping alerts into meta
> > > > alerts
> > > > > (justinleet) closes apache/metron#734
> > > > > METRON-1146: Add ability to parse JSON string into
> JSONObject
> > > > > for stellar closes apache/incubator-metron#727
> > > > > METRON-1176 REST: HDFS Service should support setting
> > > > > permissions on files when writing (ottobackwards) closes
> > > > apache/metron#749
> > > > > METRON-1114 Add group by capabilities to search REST
> endpoint
> > > > > (merrimanr) closes apache/metron#702
> > > > > METRON-1167 Define Session Specific Global Configuration
> > > > > Values in the REPL (nickwallen) closes apache/metron#740
> > > > > METRON-1171: Better validation for the SUBSTRING stellar
> > > > > function closes apache/incubator-metron#745
> > > > >
> > > > >
> > > > >
> > > > > On 11/17/17, 11:59 AM, "Nick Allen" <ni...@nickallen.org>
> wrote:
> > > > >
> > > > > I just wanted to send an update on where we are at. We've
> > > > > gotten a lot
> > > > > done here recently as you can see below.
> > > > >
> > > > > ✓ DONE (1) First, METRON-1289 needs to go in. This one was
> > > > > a fairly big
> > > > > effort and I am hearing that we are pretty close.
> > > > >
> > > > > ✓ DONE (2) METRON-1294 fixes an issue in how field types
> > > > are
> > > > > looked-up.
> > > > >
> > > > > ✓ DONE (3) METRON-1290 is next. While this may have been
> > > > > fixed in
> > > > > M-1289, there may be some test cases we want from this PR.
> > > > >
> > > > > ✓ DONE (4) METRON-1301 addresses a problem with the sorting
> > > > > logic.
> > > > >
> > > > > ✓ DONE (5) METRON-1291 fixes an issue with escalation of
> > > > > metaalerts.
> > > > >
> > > > > (6) That leads us to Raghu's UI work in METRON-1252. This
> > > > > introduces the
> > > > > UI bits that depend on all the previous backend work.
> > > > >
> > > > > (7) At this point, we should have our best effort at
> > > > running
> > > > > Metaalerts
> > > > > on Elasticsearch 2.x. I propose that we cut a release here.
> > > > >
> > > > > (8) After we cut the release, we can introduce the work for
> > > > > ES 5.x in
> > > > > METRON-939. I know we will need lots of help testing and
> > > > > reviewing this
> > > > > one.
> > > > >
> > > > >
> > > > >
> > > > > We also have an outstanding question that needs resolved
> > > > > BEFORE we
> > > > > release. We need to come to a consensus on how to release
> > > > > having moved our
> > > > > Bro Plugin to a separate repo. I don't think we've heard
> > > > from
> > > > > everyone on
> > > > > this. I'd urge everyone to chime in so we can choose a path
> > > > > forward.
> > > > >
> > > > > If anyone is totally confused in regards to that
> discussion,
> > > > I
> > > > > can try and
> > > > > send an options summary again as a separate discuss thread.
> > > > > The original
> > > > > chain was somewhere around here [1].
> > > > >
> > > > > [1]
> > > > > https://lists.apache.org/thread.html/
> > > > > 54a4474881b97e559df24728b3a0e923a58345a282451085eef832ef@%
> > > > > 3Cdev.metron.apache.org%3E
> > > > >
> > > > >
> > > > >
> > > > > On Wed, Nov 15, 2017 at 10:04 AM, Nick Allen <
> > > > > nick@nickallen.org> wrote:
> > > > >
> > > > > > Hi Guys -
> > > > > >
> > > > > > I want to follow-up on this discussion. It sounds like
> > > > most
> > > > > people are in
> > > > > > agreement with the general approach.
> > > > > >
> > > > > > A lot of people have been working hard on Metaalerts and
> > > > > Elasticsearch. I
> > > > > > have checked-in with those doing the heavy lifting and
> have
> > > > > compiled a more
> > > > > > detailed plan based on where we are at now. To the best
> of
> > > > > my knowledge
> > > > > > here is the plan of attack for finishing out this effort.
> > > > > >
> > > > > > (1) First, METRON-1289 needs to go in. This one was a
> > > > > fairly big effort
> > > > > > and I am hearing that we are pretty close.
> > > > > >
> > > > > > (2) METRON-1294 fixes an issue in how field types are
> > > > > looked-up.
> > > > > >
> > > > > > (3) METRON-1290 is next. While this may have been fixed
> > > > > in M-1289,
> > > > > > there may be some test cases we want from this PR.
> > > > > >
> > > > > > (4) METRON-1301 addresses a problem with the sorting
> > > > logic.
> > > > > >
> > > > > > (5) METRON-1291 fixes an issue with escalation of
> > > > > metaalerts.
> > > > > >
> > > > > > (6) That leads us to Raghu's UI work in METRON-1252.
> > > > This
> > > > > introduces
> > > > > > the UI bits that depend on all the previous backend work.
> > > > > >
> > > > > > (7) At this point, we should have our best effort at
> > > > > running Metaalerts
> > > > > > on Elasticsearch 2.x. I propose that we cut a release
> here.
> > > > > >
> > > > > > (8) After we cut the release, we can introduce the work
> > > > > for ES 5.x in
> > > > > > METRON-939. I know we will need lots of help testing and
> > > > > reviewing this
> > > > > > one.
> > > > > >
> > > > > > Please correct me if I am wrong. I will try and send out
> > > > > updates as we
> > > > > > make progress.
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Mon, Nov 6, 2017 at 1:03 PM, Zeolla@GMail.com <
> > > > > zeolla@gmail.com> wrote:
> > > > > >
> > > > > >> I agree, I think it's very reasonable to move in line
> with
> > > > > Nick's
> > > > > >> proposal. I would also suggest that we outline what the
> > > > > target versions
> > > > > >> would be to add in the METRON-777 components, since it
> has
> > > > > been functional
> > > > > >> for a very long time but not reviewed and has some
> really
> > > > > rockstar
> > > > > >> improvements.
> > > > > >>
> > > > > >> Jon
> > > > > >>
> > > > > >> On Mon, Nov 6, 2017 at 12:56 PM Otto Fowler <
> > > > > ottobackwards@gmail.com>
> > > > > >> wrote:
> > > > > >>
> > > > > >> > I think the ES cutover should be the start of the
> 0.5.x
> > > > > series, and we
> > > > > >> > continue on with 0.4.x for the
> > > > > >> > metadata improvements etc. We could chose to focus
> > > > > 0.5.x’s first
> > > > > >> releases
> > > > > >> > on not only ES but
> > > > > >> > getting a handle on kibana and the mpack situation as
> > > > > well.
> > > > > >> >
> > > > > >> >
> > > > > >> >
> > > > > >> >
> > > > > >> > On November 6, 2017 at 12:48:45, Michael Miklavcic (
> > > > > >> > michael.miklavcic@gmail.com) wrote:
> > > > > >> >
> > > > > >> > I agree with your proposal, Nick. I think having a
> > > > > stabilizing release
> > > > > >> > prior to upgrading ES/Kibana makes sense.
> > > > > >> >
> > > > > >> > On Mon, Nov 6, 2017 at 9:16 AM, Nick Allen <
> > > > > nick@nickallen.org> wrote:
> > > > > >> >
> > > > > >> > > I would like to start a discussion around upcoming
> > > > > releases. We have a
> > > > > >> > > couple separate significant tracks of work that we
> > > > need
> > > > > to reconcile
> > > > > >> in
> > > > > >> > our
> > > > > >> > > release schedule.
> > > > > >> > >
> > > > > >> > > (1) We have had (and have in review) a good number
> of
> > > > > bug fixes
> > > > > >> required
> > > > > >> > to
> > > > > >> > > support Metaalerts on the existing Elasticsearch 2.x
> > > > > infrastructure.
> > > > > >> > >
> > > > > >> > >
> > > > > >> > > (2) We also have ongoing work to upgrade our
> > > > > infrastructure to
> > > > > >> > > Elasticsearch 5.x, which will not be backwards
> > > > > compatible.
> > > > > >> > >
> > > > > >> > >
> > > > > >> > > I would like to see a release that has our best work
> > > > on
> > > > > ES 2.x before
> > > > > >> we
> > > > > >> > > migrate to 5.x. I would propose the following.
> > > > > >> > >
> > > > > >> > > Release N+1: Introduce Metaalerts running on ES 2.x
> > > > > >> > >
> > > > > >> > > Release N+2: Cut-over to ES 5.x
> > > > > >> > >
> > > > > >> > >
> > > > > >> > > (Q) Is it worth cutting a separate release for ES
> 2.x?
> > > > > Is there a
> > > > > >> better
> > > > > >> > > way to handle the cut-over to 5.x?
> > > > > >> > >
> > > > > >> >
> > > > > >> --
> > > > > >>
> > > > > >> Jon
> > > > > >>
> > > > > >
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > >
> > > --
> > >
> > > Jon
> > >
> > >
> > >
> > >
> > >
> > >
> >
> >
> >
> >
>
>
>
Re: [DISCUSS] Upcoming Release
Posted by Matt Foley <mf...@hortonworks.com>.
Hi Nick,
Good timing, you’ve saved me some work! :-)
Origin of the list: The approach was defined before I started managing releases, but I think it’s a reasonable approach. It’s specified in our Release Process document, https://cwiki.apache.org/confluence/display/METRON/Release+Process , Step 5, the last bullet:
“The artifacts for a release or RC [include]… A CHANGES file denoting the changes [since the last release].
We usually construct this by taking the output of git log | grep METRON | sed 's/\[//g' | sed 's/\]//g' | grep -v "http" and removing the JIRAs from the previous releases (it’s in time sorted order so this is easy).”
However, the release manager also has a responsibility to “Monitor and Verify Jiras” (Step 2 and various other places in the doc). Altho the doc isn’t explicit about it, I take this responsibility a little farther and do the following as part of Jira verification:
1. Extract the jira id’s from the CHANGES file with a simple awk script, put them in a Jira query, and confirm they are all actually marked “Fixed/Resolved” with Fix Version = <the current release> in Jira. If there are exceptions, I dun the contributors to fix it. You’ve seen these emails from me in the past couple releases. I don’t just mark them fixed myself, because some contributions (perfectly legitimately) only partially address a problem, and just marking a ticket “Fixed” may not be the right thing to do.
2. Query jira for any tickets NOT included in the prior list, that ARE marked fixed in the current release (or later, or “Next”, or similar “future” version constructs). These don’t usually happen (considering that Metron contributors mostly live inside github PR processes rather than Jiras) but when they do they also need to be resolved. Cases I’ve seen (in Metron and other projects) include:
a) If they were indeed fixed in the current release, but not reflected in the commit message, I’ll annotate this fact in the Jira ticket, and hand-edit the CHANGES file for this release. (Doesn’t seem worth trying to edit old commit messages, since that mucks up the SHA1’s.)
b) If they were fixed in a previous release but mis-marked as to Fix Version in the ticket, fix the ticket.
c) If they aren’t really fixed, fix the ticket.
3. Once the set of Fixed tickets is complete and correct, query them for any that have “labels in (backward-incompatible, backwards-compatibility, backwards-incompatible)” -- yes, there shouldn’t be 3 labels, but it’s hard to fix because Jira never forgets, so every time someone gets it wrong, it adds to the set; fortunately, query completion helps here – OR "Docs Text" is not EMPTY. Each of these needs to be looked at carefully, verified, and if Doc Text is missing, the contributor and I cooperate to write a couple sentences. The Doc Text then needs to go in the RELEASE_NOTES for the release, in the section “## Non-backward-compatible Changes in this Release, and Upgrade Suggestions”.
That’s about it. I guess I should add the above as an appendix doc to the Release Process documentation. I’ll do that.
Cheers,
--Matt
On 12/15/17, 9:15 AM, "Nick Allen" <ni...@nickallen.org> wrote:
Hi Matt -
I just updated like 15+ JIRAs of my own JIRAs that I completed and merged,
but failed to mark as resolved. All of these will be included in 0.4.2. I
updated each to be "fixed" in version 0.4.2 and marked as resolved.
Hopefully the next RC will report those as fixed.
(Q) Where does the list of JIRAs that get attached to the release originate
from? Does it get pulled out of JIRA or do they come from the commit log?
My apologies for not staying on top of my JIRAs.
On Tue, Dec 12, 2017 at 2:21 PM, Matt Foley <ma...@apache.org> wrote:
> Thanks to Jon, Otto, and Nick for looking over RC1. Tonight I’ll fix the
> RAT glitch, build RC2, and put it to formal vote.
> Regards,
> --Matt
>
> On 12/12/17, 5:14 AM, "Nick Allen" <ni...@nickallen.org> wrote:
>
> RC1 is looking good to me. I validated the MD5s, built Metron, built
> the
> Bro plugin and reviewed the other artifacts like release notes.
>
> Running the RAT check on a 'clean' Metron does not produce any errors
> for
> me. It is only after building Metron, which pulls in additional Node
> dependencies, does the RAT check fail.
>
>
> On Sun, Dec 10, 2017 at 4:41 PM Matt Foley <ma...@apache.org> wrote:
>
> > Yes, but let’s see if anyone else find other issues.
> >
> >
> >
> > From: Otto Fowler <ot...@gmail.com>
> > Date: Saturday, December 9, 2017 at 6:16 AM
> > To: Matt Foley <mf...@hortonworks.com>, "dev@metron.apache.org" <
> > dev@metron.apache.org>
> > Subject: Re: [DISCUSS] Upcoming Release
> >
> >
> >
> > So RC2 then?
> >
> >
> >
> > On December 8, 2017 at 20:43:21, Matt Foley (mfoley@hortonworks.com)
> > wrote:
> >
> > Hah, here it is: https://github.com/apache/metron/pull/743
> > “This problem seems to only reproduce when one unrolls a tarball
> rather
> > than cloning from github.”
> >
> > Heh, the exclusion at
> > https://github.com/apache/metron/blob/master/pom.xml#L351 is still
> there,
> > but the hashcode in the bundle.css file name has changed from
> > a0b6b99c10d9a13dc67e to f56deed131e58bd7ee04. Sigh. Did the version
> of Font
> > Awesome fonts change?
> >
> >
> > On 12/8/17, 5:26 PM, "Matt Foley" <ma...@apache.org> wrote:
> >
> > I remember having trouble with this bundle.css file on the last
> release,
> > but I can’t remember what we did about it. Anybody?
> >
> > On 12/8/17, 1:41 PM, "Otto Fowler" <ot...@gmail.com> wrote:
> >
> > Steps
> >
> > - Downloaded tar.gz’s, asc files and KEYS
> > - Verified signing of both tar.gz’s
> > - searched for rouge 0.4.1 entries
> > - verified the main pom.xml
> > - built :
> >
> > mvn clean && time mvn -q -T 2C -DskipTests install && time mvn -q -T
> > 2C surefire:test@unit-tests && time mvn -q
> > surefire:test@integration-tests && time mvn -q test --projects
> > metron-interface/metron-config && time build_utils/verify_licenses.sh
> >
> > Found rat error:
> >
> >
> > *****************************************************
> > Summary
> > -------
> > Generated at: 2017-12-08T16:33:27-05:00
> >
> > Notes: 3
> > Binaries: 193
> > Archives: 0
> > Standards: 75
> >
> > Apache Licensed: 74
> > Generated Documents: 0
> >
> > JavaDocs are generated, thus a license header is optional.
> > Generated files do not require license headers.
> >
> > 1 Unknown Licenses
> >
> > *****************************************************
> >
> > Files with unapproved licenses:
> >
> >
> > /Users/batman/tmp/release_ver/apache-metron-0.4.2-rc1/
> metron-interface/metron-alerts/dist/styles.f56deed131e58bd7ee04.bundle.css
> >
> > *****************************************************
> >
> >
> >
> >
> >
> > *****************************************************
> > Summary
> > -------
> > Generated at: 2017-12-08T16:33:27-05:00
> >
> > Notes: 3
> > Binaries: 193
> > Archives: 0
> > Standards: 75
> >
> > Apache Licensed: 74
> > Generated Documents: 0
> >
> > JavaDocs are generated, thus a license header is optional.
> > Generated files do not require license headers.
> >
> > 1 Unknown Licenses
> >
> > *****************************************************
> >
> > Files with unapproved licenses:
> >
> >
> >
> > /Users/ottofowler/tmp/release_ver/apache-metron-0.4.2-rc1/
> metron-interface/metron-alerts/dist/styles.f56deed131e58bd7ee04.bundle.css
> >
> > *****************************************************
> >
> >
> >
> > On December 8, 2017 at 04:34:24, Matt Foley (mattf@apache.org)
> wrote:
> >
> > Colleagues,
> > I’ve posted Metron-0.4.2-RC1 and Metron-bro-plugin-kafka-0.1 to
> > https://dist.apache.org/repos/dist/dev/metron/0.4.2-RC1/
> >
> > Given the complexity of this RC, I’d appreciate if a couple people
> would be
> > willing to kick the tires before we put it up for a vote.
> >
> > I will myself be going thru the Verify Build process this weekend,
> as I
> > won’t be able to do it Friday.
> >
> > Thanks,
> > --Matt
> >
> >
> > On 12/4/17, 2:05 PM, "Zeolla@GMail.com" <ze...@gmail.com> wrote:
> >
> > Can we resolve the conversation regarding the second repo? I was
> waiting
> > to get more input/preferences from people There's also a
> documentation
> > update that fixes a few broken Stellar docs that already has aa +1,
> I just
> > need to merge it.
> >
> > Jon
> >
> > On Mon, Dec 4, 2017, 17:01 Casey Stella <ce...@gmail.com> wrote:
> >
> > > I would be in favor of a release at this point.
> > >
> > > On Mon, Dec 4, 2017 at 4:57 PM, Matt Foley <ma...@apache.org>
> wrote:
> > >
> > > > Hey all,
> > > > I see METRON-1252 was resolved over the weekend. Shall I go
> ahead and
> > > > start the process with 0.4.2 release?
> > > > Does anyone have any commits they feel strongly should go in
> before
> > 0.4.2
> > > > is done, or are we ready to call it good?
> > > >
> > > > I believe there is consensus the 0.4.2 release should include a
> release
> > > of
> > > > the current state of the metron-bro-plugin-kafka. I will
> continue the
> > > > discussion in that thread as to the process for accomplishing
> that, but
> > > > plan on it happening.
> > > >
> > > > Regards,
> > > > --Matt
> > > >
> > > > On 11/26/17, 6:26 PM, "Matt Foley" <ma...@apache.org> wrote:
> > > >
> > > > Hope everyone (at least in the U.S.) had a great Thanksgiving
> > > holiday.
> > > > Regarding status of the release effort, still pending
> METRON-1252, so
> > > > not making the release branch yet.
> > > >
> > > > Regards,
> > > > --Matt
> > > >
> > > > On 11/17/17, 1:32 PM, "Matt Foley" <ma...@apache.org> wrote:
> > > >
> > > > (With release manager hat on)
> > > >
> > > > The community has proposed a release of Metron in the near
> > > future,
> > > > focusing on Meta-alerts running in Elasticsearch.
> > > > Congrats on getting so many of the below already done. At this
> > > > point, only METRON-1252, and the discussion of how to handle
> joint
> > > release
> > > > of the Metron bro plugin, remain as gating items for the
> release. I
> > > > project these will be resolved next week, so let’s propose the
> > following:
> > > >
> > > > Sometime next week, after the last bits are done, I’ll start the
> > > > release process and create the release branch.
> > > >
> > > > The proposed new version will be 0.4.2, unless there are backward
> > > > incompatible changes that support making it 0.5.0.
> > > > Currently there are NO included Jiras labeled
> > > > ‘backward-incompatible’, nor having Docs Text indicating so.
> > > > ***If anyone knows that some of the commits included since 0.4.1
> > > > introduce backward incompatibility, please say so now on this
> thread,
> > and
> > > > mark the Jira as such.***
> > > >
> > > > The 90 or so jiras/commits already in master branch since 0.4.1
> > > > are listed below.
> > > > Thanks,
> > > > --Matt
> > > >
> > > > METRON-1301 Alerts UI - Sorting on Triage Score Unexpectedly
> > > > Filters Some Records (nickwallen) closes apache/metron#832
> > > > METRON-1294 IP addresses are not formatted correctly in facet
> > > > and group results (merrimanr) closes apache/metron#827
> > > > METRON-1291 Kafka produce REST endpoint does not work in a
> > > > Kerberized cluster (merrimanr) closes apache/metron#826
> > > > METRON-1290 Only first 10 alerts are update when a MetaAlert
> > > > status is changed to inactive (justinleet) closes
> apache/metron#842
> > > > METRON-1311 Service Check Should Check Elasticsearch Index
> > > > Templates (nickwallen) closes apache/metron#839
> > > > METRON-1289 Alert fields are lost when a MetaAlert is created
> > > > (merrimanr) closes apache/metron#824
> > > > METRON-1309 Change metron-deployment to pull the plugin from
> > > > apache/metron-bro-plugin-kafka (JonZeolla) closes
> apache/metron#837
> > > > METRON-1310 Template Delete Action Deletes Search Indices
> > > > (nickwallen) closes apache/metron#838
> > > > METRON-1275: Fix Metron Documentation closes
> > > > apache/incubator-metron#833
> > > > METRON-1295 Unable to Configure Logging for REST API
> > > > (nickwallen) closes apache/metron#828
> > > > METRON-1307 Force install of java8 since java9 does not
> > > appear
> > > > to work with the scripts (brianhurley via ottobackwards) closes
> > > > apache/metron#835
> > > > METRON-1296 Full Dev Fails to Deploy Index Templates
> > > > (nickwallen via cestella) closes apache/incubator-metron#829
> > > > METRON-1281 Remove hard-coded indices from the Alerts UI
> > > > (merrimanr) closes apache/metron#821
> > > > METRON-1287 Full Dev Fails When Installing EPEL Repository
> > > > (nickwallen) closes apache/metron#820
> > > > METRON-1267 Alerts UI returns a 404 when refreshing the
> > > > alerts-list page (iraghumitra via merrimanr) closes
> apache/metron#819
> > > > METRON-1283 Install Elasticsearch template as a part of the
> > > > mpack startup scripts (anandsubbu via nickwallen) closes
> > > apache/metron#817
> > > > METRON-1254: Conditionals as map keys do not function in
> > > > Stellar closes apache/incubator-metron#801
> > > > METRON-1261 Apply bro security patch (JonZeolla via
> > > > ottobackwards) closes apache/metron#805
> > > > METRON-1284 Remove extraneous dead query in ElasticsearchDao
> > > > (justinleet) closes apache/metron#818
> > > > METRON-1270: fix for warnings missing @return tag argument in
> > > > metron-analytics/metron-profiler-common and
> metron-profiler-client
> > closes
> > > > apache/incubator-metron#810
> > > > METRON-1272 Hide child alerts from searches and grouping if
> > > > they belong to meta alerts (justinleet) closes apache/metron#811
> > > > METRON-1224 Add time range selection to search control
> > > > (iraghumitra via james-sirota) closes apache/metron#796
> > > > METRON-1280 0.4.1 -> 0.4.2 missed a couple of projects
> > > > (cestella via justinleet) closes apache/metron#816
> > > > METRON-1243: Add a REST endpoint which allows us to get a
> > > list
> > > > of all indice closes apache/incubator-metron#797
> > > > METRON-1196 Increment master version number to 0.4.2 for
> > > > on-going development (mattf-horton) closes apache/metron#767
> > > > METRON-1278 Strip "Build Status" widget from root
> > > > README.md in site-book build (mattf-horton) closes
> apache/metron#815
> > > > METRON-1274 Master has failure in
> > > > StormControllerIntegrationTest (merrimanr) closes
> apache/metron#813
> > > > METRON-1266 Profiler - SASL Authentication Failed
> > > (nickwallen)
> > > > closes apache/metron#809
> > > > METRON-1260 Include Alerts UI in Ambari Service Check
> > > > (nickwallen) closes apache/metron#804
> > > > METRON-1251: Typo and formatting fixes for metron-rest README
> > > > closes apache/incubator-metron#800
> > > > METRON-1241: Enable the REST API to use a cache for the
> > > > zookeeper config similar to the Bolts closes
> > apache/incubator-metron#795
> > > > METRON-1267 Alerts UI returns a 404 when refreshing the
> > > > alerts-list page (merrimanr) closes apache/metron#808
> > > > METRON-1262 Unable to add comment for a alert in a meta-alert
> > > > (merrimanr) closes apache/metron#806
> > > > METRON-1263 Start Alerts UI service after Metron REST
> > > > (anandsubbu via nickwallen) closes apache/metron#807
> > > > METRON-1255 MetaAlert search is not filtering on status
> > > > (merrimanr) closes apache/metron#802
> > > > METRON-1249 Improve Metron MPack Service Checks (nickwallen)
> > > > closes apache/metron#799
> > > > METRON-1237 address javadoc warnings in metron-maas-common
> > > > (dbist via james-sirota) closes apache/metron#792
> > > > METRON-1240 address javadoc warnings in metron-platform and
> > > > metron-analytics (dbist via james-sirota) closes
> apache/metron#794
> > > > METRON-1226 Searching Can Errantly Query the Wrong Indices
> > > > (nickwallen) closes apache/metron#793
> > > > METRON-1081 Fix Alerts and Ops UI Notices file (james-sirota)
> > > > closes apache/metron#682
> > > > METRON-1123 Add group by option using faceted search
> > > > capabilities of metron-rest-api (iraghumitra via james-sirota)
> closes
> > > > apache/metron#768
> > > > METRON-1223 Add support to add comments for alerts
> > > > (iraghumitra via james-sirota) closes apache/metron#788
> > > > METRON-1083 Add filters using faceted search capabilities of
> > > > metron-rest-api (iraghumitra via james-sirota) closes
> apache/metron#710
> > > > METRON-1232 Alert status changes are not reflected in list
> > > > view (iraghumitra via merrimanr) closes apache/metron#787
> > > > METRON-1247 REST search and findOne endpoints return
> > > > unexpected or incorrect results for guids (justinleet) closes
> > > > apache/metron#798
> > > > METRON-1235: Document the properties pulled from the global
> > > > configuration closes apache/incubator-metron#791
> > > > METRON-1234: fix for WARNING 'dependencies.dependency.(
> > > > groupId:artifactId:type:classifier)' must be unique:
> > > > org.apache.hadoop:hadoop-yarn-api:jar (dbist via mmiklavc)
> closes
> > > > apache/metron#790
> > > > METRON-1222: fix warning for The expression ${parent.version}
> > > > is deprecated. Please use ${project.parent.version} instead.
> (dbist via
> > > > mmiklavc) closes apache/metron#782
> > > > METRON-1220 Create documentation around alert nested field
> > > > (justinleet) closes apache/metron#780
> > > > METRON-1229 Management UI type is part of the declarations of
> > > > 2 modules (merrimanr) closes apache/metron#784
> > > > METRON-1228: Configuration Management PUSH immediately does
> > > > DUMP after (mmiklavc via mmiklavc) closes apache/metron#783
> > > > METRON-1218 Metron REST should return better error messages
> > > > (merrimanr) closes apache/metron#779
> > > > METRON-1161 Add ability to edit parser command line options
> > > in
> > > > the management UI (merrimanr) closes apache/metron#737
> > > > METRON-1209: Make stellar repl take logging properties, like
> > > > other CLI apps in metron closes apache/incubator-metron#772
> > > > METRON-1059 address checkstyle warning AvoidStarImport in
> > > > metron-stellar (dbist via ottobackwards) closes apache/metron#664
> > > > METRON-1204 UI does not time out after being idle, but stops
> > > > functioning (merrimanr) closes apache/metron#771
> > > > METRON-1052: Add forensic similarity hash functions to
> > > Stellar
> > > > closes apache/incubator-metron#781
> > > > METRON-632: Added validation of "shew.enrichmentType" and
> > > > "shew.keyColumns" closes apache/incubator-metron#732
> > > > METRON-1194 Add Profiler Debug Functions to Profiler README
> > > > (nickwallen via ottobackwards) closes apache/metron#765
> > > > METRON-1055 Metron 0.4.0 manual installation guide for CentOS
> > > > 6 updates (lvets via ottobackwards) closes apache/metron#661
> > > > METRON-1079 STELLAR NaN should be a keyword (ottobackwards)
> > > > closes apache/metron#681
> > > > METRON-1085 Add REST endpoint to save a user profile for the
> > > > Alerts UI (merrimanr) closes apache/metron#694
> > > > METRON-1208 MPack for Alerts UI (merrimanr) closes
> > > > apache/metron#778
> > > > METRON-1207 Make RPMs for Alerts UI (merrimanr) closes
> > > > apache/metron#777
> > > > METRON-1215 Fix link to RPMs chapter (DimDroll via
> > > justinleet)
> > > > closes apache/metron#776
> > > > METRON-1206 Make alerts UI conform to ops UI for install
> > > > (merrimanr) closes apache/metron#773
> > > > METRON-1195 Meta alerts improperly handle updates to
> > > non-alert
> > > > fields (justinleet) closes apache/metron#766
> > > > METRON-1189 Add alert escalation to the Alerts UI (merrimanr)
> > > > closes apache/metron#762
> > > > METRON-1156 Simulate Triage Rules in the Stellar REPL
> > > > (nickwallen) closes apache/metron#733
> > > > METRON-1198 Pycapa - No such configuration property
> > > > 'sasl.kerberos.principal' (nickwallen) closes apache/metron#769
> > > > METRON-1202 ElasticsearchDao Has extraneous sleep call
> > > > (justinleet) closes apache/metron#770
> > > > METRON-938 "service metron-rest start <password>" does not
> > > > work on CentOS 7. (justinleet) closes apache/metron#757
> > > > METRON-1182 Refactor Code in alert list to accommodate new
> > > > view types (iraghumitra via merrimanr) closes apache/metron#756
> > > > METRON-1188: Ambari global configuration management
> > > (mmiklavc)
> > > > closes apache/metron#760
> > > > METRON-1191 update public web site to point at 0.4.1 new
> > > > release (mattf-horton) closes apache/metron#764
> > > > METRON-1063 address javadoc warnings in metron-stellar (dbist
> > > > via ottobackwards) closes apache/metron#668
> > > > METRON-1190 Fix Meta Alert Type handling in calculation of
> > > > scores (justinleet) closes apache/metron#763
> > > > METRON-1187 Indexing/Profiler Kafka ACL Groups Not Setup
> > > > Correctly (nickwallen) closes apache/metron#759
> > > > METRON-1185: Stellar REPL does not work on a kerberized
> > > > cluster when calling functions interacting with HBase closes
> > > > apache/incubator-metron#755
> > > > METRON-1186: Profiler Functions use classutils from shaded
> > > > storm closes apache/incubator-metron#758
> > > > METRON-1173: Fix pointers to old stellar docs closes
> > > > apache/incubator-metron#746
> > > > METRON-1179: Make STATS_ADD to take a list closes
> > > > apache/incubator-metron#750
> > > > METRON-1180: Make Stellar Shell accept zookeeper quorum as a
> > > > CSV list and not require a port closes
> apache/incubator-metron#751
> > > > METRON-1183 Improve KDC Setup Instructions (nickwallen)
> > > closes
> > > > apache/metron#753
> > > > METRON-1177 Stale running topologies seen post-kerberization
> > > > and cause exceptions (nickwallen) closes apache/metron#748
> > > > METRON-1158 Build backend for grouping alerts into meta
> > > alerts
> > > > (justinleet) closes apache/metron#734
> > > > METRON-1146: Add ability to parse JSON string into JSONObject
> > > > for stellar closes apache/incubator-metron#727
> > > > METRON-1176 REST: HDFS Service should support setting
> > > > permissions on files when writing (ottobackwards) closes
> > > apache/metron#749
> > > > METRON-1114 Add group by capabilities to search REST endpoint
> > > > (merrimanr) closes apache/metron#702
> > > > METRON-1167 Define Session Specific Global Configuration
> > > > Values in the REPL (nickwallen) closes apache/metron#740
> > > > METRON-1171: Better validation for the SUBSTRING stellar
> > > > function closes apache/incubator-metron#745
> > > >
> > > >
> > > >
> > > > On 11/17/17, 11:59 AM, "Nick Allen" <ni...@nickallen.org> wrote:
> > > >
> > > > I just wanted to send an update on where we are at. We've
> > > > gotten a lot
> > > > done here recently as you can see below.
> > > >
> > > > ✓ DONE (1) First, METRON-1289 needs to go in. This one was
> > > > a fairly big
> > > > effort and I am hearing that we are pretty close.
> > > >
> > > > ✓ DONE (2) METRON-1294 fixes an issue in how field types
> > > are
> > > > looked-up.
> > > >
> > > > ✓ DONE (3) METRON-1290 is next. While this may have been
> > > > fixed in
> > > > M-1289, there may be some test cases we want from this PR.
> > > >
> > > > ✓ DONE (4) METRON-1301 addresses a problem with the sorting
> > > > logic.
> > > >
> > > > ✓ DONE (5) METRON-1291 fixes an issue with escalation of
> > > > metaalerts.
> > > >
> > > > (6) That leads us to Raghu's UI work in METRON-1252. This
> > > > introduces the
> > > > UI bits that depend on all the previous backend work.
> > > >
> > > > (7) At this point, we should have our best effort at
> > > running
> > > > Metaalerts
> > > > on Elasticsearch 2.x. I propose that we cut a release here.
> > > >
> > > > (8) After we cut the release, we can introduce the work for
> > > > ES 5.x in
> > > > METRON-939. I know we will need lots of help testing and
> > > > reviewing this
> > > > one.
> > > >
> > > >
> > > >
> > > > We also have an outstanding question that needs resolved
> > > > BEFORE we
> > > > release. We need to come to a consensus on how to release
> > > > having moved our
> > > > Bro Plugin to a separate repo. I don't think we've heard
> > > from
> > > > everyone on
> > > > this. I'd urge everyone to chime in so we can choose a path
> > > > forward.
> > > >
> > > > If anyone is totally confused in regards to that discussion,
> > > I
> > > > can try and
> > > > send an options summary again as a separate discuss thread.
> > > > The original
> > > > chain was somewhere around here [1].
> > > >
> > > > [1]
> > > > https://lists.apache.org/thread.html/
> > > > 54a4474881b97e559df24728b3a0e923a58345a282451085eef832ef@%
> > > > 3Cdev.metron.apache.org%3E
> > > >
> > > >
> > > >
> > > > On Wed, Nov 15, 2017 at 10:04 AM, Nick Allen <
> > > > nick@nickallen.org> wrote:
> > > >
> > > > > Hi Guys -
> > > > >
> > > > > I want to follow-up on this discussion. It sounds like
> > > most
> > > > people are in
> > > > > agreement with the general approach.
> > > > >
> > > > > A lot of people have been working hard on Metaalerts and
> > > > Elasticsearch. I
> > > > > have checked-in with those doing the heavy lifting and have
> > > > compiled a more
> > > > > detailed plan based on where we are at now. To the best of
> > > > my knowledge
> > > > > here is the plan of attack for finishing out this effort.
> > > > >
> > > > > (1) First, METRON-1289 needs to go in. This one was a
> > > > fairly big effort
> > > > > and I am hearing that we are pretty close.
> > > > >
> > > > > (2) METRON-1294 fixes an issue in how field types are
> > > > looked-up.
> > > > >
> > > > > (3) METRON-1290 is next. While this may have been fixed
> > > > in M-1289,
> > > > > there may be some test cases we want from this PR.
> > > > >
> > > > > (4) METRON-1301 addresses a problem with the sorting
> > > logic.
> > > > >
> > > > > (5) METRON-1291 fixes an issue with escalation of
> > > > metaalerts.
> > > > >
> > > > > (6) That leads us to Raghu's UI work in METRON-1252.
> > > This
> > > > introduces
> > > > > the UI bits that depend on all the previous backend work.
> > > > >
> > > > > (7) At this point, we should have our best effort at
> > > > running Metaalerts
> > > > > on Elasticsearch 2.x. I propose that we cut a release here.
> > > > >
> > > > > (8) After we cut the release, we can introduce the work
> > > > for ES 5.x in
> > > > > METRON-939. I know we will need lots of help testing and
> > > > reviewing this
> > > > > one.
> > > > >
> > > > > Please correct me if I am wrong. I will try and send out
> > > > updates as we
> > > > > make progress.
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > On Mon, Nov 6, 2017 at 1:03 PM, Zeolla@GMail.com <
> > > > zeolla@gmail.com> wrote:
> > > > >
> > > > >> I agree, I think it's very reasonable to move in line with
> > > > Nick's
> > > > >> proposal. I would also suggest that we outline what the
> > > > target versions
> > > > >> would be to add in the METRON-777 components, since it has
> > > > been functional
> > > > >> for a very long time but not reviewed and has some really
> > > > rockstar
> > > > >> improvements.
> > > > >>
> > > > >> Jon
> > > > >>
> > > > >> On Mon, Nov 6, 2017 at 12:56 PM Otto Fowler <
> > > > ottobackwards@gmail.com>
> > > > >> wrote:
> > > > >>
> > > > >> > I think the ES cutover should be the start of the 0.5.x
> > > > series, and we
> > > > >> > continue on with 0.4.x for the
> > > > >> > metadata improvements etc. We could chose to focus
> > > > 0.5.x’s first
> > > > >> releases
> > > > >> > on not only ES but
> > > > >> > getting a handle on kibana and the mpack situation as
> > > > well.
> > > > >> >
> > > > >> >
> > > > >> >
> > > > >> >
> > > > >> > On November 6, 2017 at 12:48:45, Michael Miklavcic (
> > > > >> > michael.miklavcic@gmail.com) wrote:
> > > > >> >
> > > > >> > I agree with your proposal, Nick. I think having a
> > > > stabilizing release
> > > > >> > prior to upgrading ES/Kibana makes sense.
> > > > >> >
> > > > >> > On Mon, Nov 6, 2017 at 9:16 AM, Nick Allen <
> > > > nick@nickallen.org> wrote:
> > > > >> >
> > > > >> > > I would like to start a discussion around upcoming
> > > > releases. We have a
> > > > >> > > couple separate significant tracks of work that we
> > > need
> > > > to reconcile
> > > > >> in
> > > > >> > our
> > > > >> > > release schedule.
> > > > >> > >
> > > > >> > > (1) We have had (and have in review) a good number of
> > > > bug fixes
> > > > >> required
> > > > >> > to
> > > > >> > > support Metaalerts on the existing Elasticsearch 2.x
> > > > infrastructure.
> > > > >> > >
> > > > >> > >
> > > > >> > > (2) We also have ongoing work to upgrade our
> > > > infrastructure to
> > > > >> > > Elasticsearch 5.x, which will not be backwards
> > > > compatible.
> > > > >> > >
> > > > >> > >
> > > > >> > > I would like to see a release that has our best work
> > > on
> > > > ES 2.x before
> > > > >> we
> > > > >> > > migrate to 5.x. I would propose the following.
> > > > >> > >
> > > > >> > > Release N+1: Introduce Metaalerts running on ES 2.x
> > > > >> > >
> > > > >> > > Release N+2: Cut-over to ES 5.x
> > > > >> > >
> > > > >> > >
> > > > >> > > (Q) Is it worth cutting a separate release for ES 2.x?
> > > > Is there a
> > > > >> better
> > > > >> > > way to handle the cut-over to 5.x?
> > > > >> > >
> > > > >> >
> > > > >> --
> > > > >>
> > > > >> Jon
> > > > >>
> > > > >
> > > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > >
> > --
> >
> > Jon
> >
> >
> >
> >
> >
> >
>
>
>
>
Re: [DISCUSS] Upcoming Release
Posted by Nick Allen <ni...@nickallen.org>.
Also, METRON-1349 [1] was merged into master, but I am not sure if that
will be included in the RC. So I resolved it and marked it as "Next + 1".
[1] https://issues.apache.org/jira/browse/METRON-1349
On Fri, Dec 15, 2017 at 12:14 PM, Nick Allen <ni...@nickallen.org> wrote:
> Hi Matt -
>
> I just updated like 15+ JIRAs of my own JIRAs that I completed and merged,
> but failed to mark as resolved. All of these will be included in 0.4.2. I
> updated each to be "fixed" in version 0.4.2 and marked as resolved.
> Hopefully the next RC will report those as fixed.
>
> (Q) Where does the list of JIRAs that get attached to the release
> originate from? Does it get pulled out of JIRA or do they come from the
> commit log?
>
> My apologies for not staying on top of my JIRAs.
>
>
>
>
> On Tue, Dec 12, 2017 at 2:21 PM, Matt Foley <ma...@apache.org> wrote:
>
>> Thanks to Jon, Otto, and Nick for looking over RC1. Tonight I’ll fix the
>> RAT glitch, build RC2, and put it to formal vote.
>> Regards,
>> --Matt
>>
>> On 12/12/17, 5:14 AM, "Nick Allen" <ni...@nickallen.org> wrote:
>>
>> RC1 is looking good to me. I validated the MD5s, built Metron, built
>> the
>> Bro plugin and reviewed the other artifacts like release notes.
>>
>> Running the RAT check on a 'clean' Metron does not produce any errors
>> for
>> me. It is only after building Metron, which pulls in additional Node
>> dependencies, does the RAT check fail.
>>
>>
>> On Sun, Dec 10, 2017 at 4:41 PM Matt Foley <ma...@apache.org> wrote:
>>
>> > Yes, but let’s see if anyone else find other issues.
>> >
>> >
>> >
>> > From: Otto Fowler <ot...@gmail.com>
>> > Date: Saturday, December 9, 2017 at 6:16 AM
>> > To: Matt Foley <mf...@hortonworks.com>, "dev@metron.apache.org" <
>> > dev@metron.apache.org>
>> > Subject: Re: [DISCUSS] Upcoming Release
>> >
>> >
>> >
>> > So RC2 then?
>> >
>> >
>> >
>> > On December 8, 2017 at 20:43:21, Matt Foley (mfoley@hortonworks.com
>> )
>> > wrote:
>> >
>> > Hah, here it is: https://github.com/apache/metron/pull/743
>> > “This problem seems to only reproduce when one unrolls a tarball
>> rather
>> > than cloning from github.”
>> >
>> > Heh, the exclusion at
>> > https://github.com/apache/metron/blob/master/pom.xml#L351 is still
>> there,
>> > but the hashcode in the bundle.css file name has changed from
>> > a0b6b99c10d9a13dc67e to f56deed131e58bd7ee04. Sigh. Did the version
>> of Font
>> > Awesome fonts change?
>> >
>> >
>> > On 12/8/17, 5:26 PM, "Matt Foley" <ma...@apache.org> wrote:
>> >
>> > I remember having trouble with this bundle.css file on the last
>> release,
>> > but I can’t remember what we did about it. Anybody?
>> >
>> > On 12/8/17, 1:41 PM, "Otto Fowler" <ot...@gmail.com> wrote:
>> >
>> > Steps
>> >
>> > - Downloaded tar.gz’s, asc files and KEYS
>> > - Verified signing of both tar.gz’s
>> > - searched for rouge 0.4.1 entries
>> > - verified the main pom.xml
>> > - built :
>> >
>> > mvn clean && time mvn -q -T 2C -DskipTests install && time mvn -q -T
>> > 2C surefire:test@unit-tests && time mvn -q
>> > surefire:test@integration-tests && time mvn -q test --projects
>> > metron-interface/metron-config && time
>> build_utils/verify_licenses.sh
>> >
>> > Found rat error:
>> >
>> >
>> > *****************************************************
>> > Summary
>> > -------
>> > Generated at: 2017-12-08T16:33:27-05:00
>> >
>> > Notes: 3
>> > Binaries: 193
>> > Archives: 0
>> > Standards: 75
>> >
>> > Apache Licensed: 74
>> > Generated Documents: 0
>> >
>> > JavaDocs are generated, thus a license header is optional.
>> > Generated files do not require license headers.
>> >
>> > 1 Unknown Licenses
>> >
>> > *****************************************************
>> >
>> > Files with unapproved licenses:
>> >
>> >
>> > /Users/batman/tmp/release_ver/apache-metron-0.4.2-rc1/metron
>> -interface/metron-alerts/dist/styles.f56deed131e58bd7ee04.bundle.css
>> >
>> > *****************************************************
>> >
>> >
>> >
>> >
>> >
>> > *****************************************************
>> > Summary
>> > -------
>> > Generated at: 2017-12-08T16:33:27-05:00
>> >
>> > Notes: 3
>> > Binaries: 193
>> > Archives: 0
>> > Standards: 75
>> >
>> > Apache Licensed: 74
>> > Generated Documents: 0
>> >
>> > JavaDocs are generated, thus a license header is optional.
>> > Generated files do not require license headers.
>> >
>> > 1 Unknown Licenses
>> >
>> > *****************************************************
>> >
>> > Files with unapproved licenses:
>> >
>> >
>> >
>> > /Users/ottofowler/tmp/release_ver/apache-metron-0.4.2-rc1/me
>> tron-interface/metron-alerts/dist/styles.f56deed131e58bd7ee04.bundle.css
>> >
>> > *****************************************************
>> >
>> >
>> >
>> > On December 8, 2017 at 04:34:24, Matt Foley (mattf@apache.org)
>> wrote:
>> >
>> > Colleagues,
>> > I’ve posted Metron-0.4.2-RC1 and Metron-bro-plugin-kafka-0.1 to
>> > https://dist.apache.org/repos/dist/dev/metron/0.4.2-RC1/
>> >
>> > Given the complexity of this RC, I’d appreciate if a couple people
>> would be
>> > willing to kick the tires before we put it up for a vote.
>> >
>> > I will myself be going thru the Verify Build process this weekend,
>> as I
>> > won’t be able to do it Friday.
>> >
>> > Thanks,
>> > --Matt
>> >
>> >
>> > On 12/4/17, 2:05 PM, "Zeolla@GMail.com" <ze...@gmail.com> wrote:
>> >
>> > Can we resolve the conversation regarding the second repo? I was
>> waiting
>> > to get more input/preferences from people There's also a
>> documentation
>> > update that fixes a few broken Stellar docs that already has aa +1,
>> I just
>> > need to merge it.
>> >
>> > Jon
>> >
>> > On Mon, Dec 4, 2017, 17:01 Casey Stella <ce...@gmail.com> wrote:
>> >
>> > > I would be in favor of a release at this point.
>> > >
>> > > On Mon, Dec 4, 2017 at 4:57 PM, Matt Foley <ma...@apache.org>
>> wrote:
>> > >
>> > > > Hey all,
>> > > > I see METRON-1252 was resolved over the weekend. Shall I go
>> ahead and
>> > > > start the process with 0.4.2 release?
>> > > > Does anyone have any commits they feel strongly should go in
>> before
>> > 0.4.2
>> > > > is done, or are we ready to call it good?
>> > > >
>> > > > I believe there is consensus the 0.4.2 release should include a
>> release
>> > > of
>> > > > the current state of the metron-bro-plugin-kafka. I will
>> continue the
>> > > > discussion in that thread as to the process for accomplishing
>> that, but
>> > > > plan on it happening.
>> > > >
>> > > > Regards,
>> > > > --Matt
>> > > >
>> > > > On 11/26/17, 6:26 PM, "Matt Foley" <ma...@apache.org> wrote:
>> > > >
>> > > > Hope everyone (at least in the U.S.) had a great Thanksgiving
>> > > holiday.
>> > > > Regarding status of the release effort, still pending
>> METRON-1252, so
>> > > > not making the release branch yet.
>> > > >
>> > > > Regards,
>> > > > --Matt
>> > > >
>> > > > On 11/17/17, 1:32 PM, "Matt Foley" <ma...@apache.org> wrote:
>> > > >
>> > > > (With release manager hat on)
>> > > >
>> > > > The community has proposed a release of Metron in the near
>> > > future,
>> > > > focusing on Meta-alerts running in Elasticsearch.
>> > > > Congrats on getting so many of the below already done. At this
>> > > > point, only METRON-1252, and the discussion of how to handle
>> joint
>> > > release
>> > > > of the Metron bro plugin, remain as gating items for the
>> release. I
>> > > > project these will be resolved next week, so let’s propose the
>> > following:
>> > > >
>> > > > Sometime next week, after the last bits are done, I’ll start the
>> > > > release process and create the release branch.
>> > > >
>> > > > The proposed new version will be 0.4.2, unless there are
>> backward
>> > > > incompatible changes that support making it 0.5.0.
>> > > > Currently there are NO included Jiras labeled
>> > > > ‘backward-incompatible’, nor having Docs Text indicating so.
>> > > > ***If anyone knows that some of the commits included since 0.4.1
>> > > > introduce backward incompatibility, please say so now on this
>> thread,
>> > and
>> > > > mark the Jira as such.***
>> > > >
>> > > > The 90 or so jiras/commits already in master branch since 0.4.1
>> > > > are listed below.
>> > > > Thanks,
>> > > > --Matt
>> > > >
>> > > > METRON-1301 Alerts UI - Sorting on Triage Score Unexpectedly
>> > > > Filters Some Records (nickwallen) closes apache/metron#832
>> > > > METRON-1294 IP addresses are not formatted correctly in facet
>> > > > and group results (merrimanr) closes apache/metron#827
>> > > > METRON-1291 Kafka produce REST endpoint does not work in a
>> > > > Kerberized cluster (merrimanr) closes apache/metron#826
>> > > > METRON-1290 Only first 10 alerts are update when a MetaAlert
>> > > > status is changed to inactive (justinleet) closes
>> apache/metron#842
>> > > > METRON-1311 Service Check Should Check Elasticsearch Index
>> > > > Templates (nickwallen) closes apache/metron#839
>> > > > METRON-1289 Alert fields are lost when a MetaAlert is created
>> > > > (merrimanr) closes apache/metron#824
>> > > > METRON-1309 Change metron-deployment to pull the plugin from
>> > > > apache/metron-bro-plugin-kafka (JonZeolla) closes
>> apache/metron#837
>> > > > METRON-1310 Template Delete Action Deletes Search Indices
>> > > > (nickwallen) closes apache/metron#838
>> > > > METRON-1275: Fix Metron Documentation closes
>> > > > apache/incubator-metron#833
>> > > > METRON-1295 Unable to Configure Logging for REST API
>> > > > (nickwallen) closes apache/metron#828
>> > > > METRON-1307 Force install of java8 since java9 does not
>> > > appear
>> > > > to work with the scripts (brianhurley via ottobackwards) closes
>> > > > apache/metron#835
>> > > > METRON-1296 Full Dev Fails to Deploy Index Templates
>> > > > (nickwallen via cestella) closes apache/incubator-metron#829
>> > > > METRON-1281 Remove hard-coded indices from the Alerts UI
>> > > > (merrimanr) closes apache/metron#821
>> > > > METRON-1287 Full Dev Fails When Installing EPEL Repository
>> > > > (nickwallen) closes apache/metron#820
>> > > > METRON-1267 Alerts UI returns a 404 when refreshing the
>> > > > alerts-list page (iraghumitra via merrimanr) closes
>> apache/metron#819
>> > > > METRON-1283 Install Elasticsearch template as a part of the
>> > > > mpack startup scripts (anandsubbu via nickwallen) closes
>> > > apache/metron#817
>> > > > METRON-1254: Conditionals as map keys do not function in
>> > > > Stellar closes apache/incubator-metron#801
>> > > > METRON-1261 Apply bro security patch (JonZeolla via
>> > > > ottobackwards) closes apache/metron#805
>> > > > METRON-1284 Remove extraneous dead query in ElasticsearchDao
>> > > > (justinleet) closes apache/metron#818
>> > > > METRON-1270: fix for warnings missing @return tag argument in
>> > > > metron-analytics/metron-profiler-common and
>> metron-profiler-client
>> > closes
>> > > > apache/incubator-metron#810
>> > > > METRON-1272 Hide child alerts from searches and grouping if
>> > > > they belong to meta alerts (justinleet) closes apache/metron#811
>> > > > METRON-1224 Add time range selection to search control
>> > > > (iraghumitra via james-sirota) closes apache/metron#796
>> > > > METRON-1280 0.4.1 -> 0.4.2 missed a couple of projects
>> > > > (cestella via justinleet) closes apache/metron#816
>> > > > METRON-1243: Add a REST endpoint which allows us to get a
>> > > list
>> > > > of all indice closes apache/incubator-metron#797
>> > > > METRON-1196 Increment master version number to 0.4.2 for
>> > > > on-going development (mattf-horton) closes apache/metron#767
>> > > > METRON-1278 Strip "Build Status" widget from root
>> > > > README.md in site-book build (mattf-horton) closes
>> apache/metron#815
>> > > > METRON-1274 Master has failure in
>> > > > StormControllerIntegrationTest (merrimanr) closes
>> apache/metron#813
>> > > > METRON-1266 Profiler - SASL Authentication Failed
>> > > (nickwallen)
>> > > > closes apache/metron#809
>> > > > METRON-1260 Include Alerts UI in Ambari Service Check
>> > > > (nickwallen) closes apache/metron#804
>> > > > METRON-1251: Typo and formatting fixes for metron-rest README
>> > > > closes apache/incubator-metron#800
>> > > > METRON-1241: Enable the REST API to use a cache for the
>> > > > zookeeper config similar to the Bolts closes
>> > apache/incubator-metron#795
>> > > > METRON-1267 Alerts UI returns a 404 when refreshing the
>> > > > alerts-list page (merrimanr) closes apache/metron#808
>> > > > METRON-1262 Unable to add comment for a alert in a meta-alert
>> > > > (merrimanr) closes apache/metron#806
>> > > > METRON-1263 Start Alerts UI service after Metron REST
>> > > > (anandsubbu via nickwallen) closes apache/metron#807
>> > > > METRON-1255 MetaAlert search is not filtering on status
>> > > > (merrimanr) closes apache/metron#802
>> > > > METRON-1249 Improve Metron MPack Service Checks (nickwallen)
>> > > > closes apache/metron#799
>> > > > METRON-1237 address javadoc warnings in metron-maas-common
>> > > > (dbist via james-sirota) closes apache/metron#792
>> > > > METRON-1240 address javadoc warnings in metron-platform and
>> > > > metron-analytics (dbist via james-sirota) closes
>> apache/metron#794
>> > > > METRON-1226 Searching Can Errantly Query the Wrong Indices
>> > > > (nickwallen) closes apache/metron#793
>> > > > METRON-1081 Fix Alerts and Ops UI Notices file (james-sirota)
>> > > > closes apache/metron#682
>> > > > METRON-1123 Add group by option using faceted search
>> > > > capabilities of metron-rest-api (iraghumitra via james-sirota)
>> closes
>> > > > apache/metron#768
>> > > > METRON-1223 Add support to add comments for alerts
>> > > > (iraghumitra via james-sirota) closes apache/metron#788
>> > > > METRON-1083 Add filters using faceted search capabilities of
>> > > > metron-rest-api (iraghumitra via james-sirota) closes
>> apache/metron#710
>> > > > METRON-1232 Alert status changes are not reflected in list
>> > > > view (iraghumitra via merrimanr) closes apache/metron#787
>> > > > METRON-1247 REST search and findOne endpoints return
>> > > > unexpected or incorrect results for guids (justinleet) closes
>> > > > apache/metron#798
>> > > > METRON-1235: Document the properties pulled from the global
>> > > > configuration closes apache/incubator-metron#791
>> > > > METRON-1234: fix for WARNING 'dependencies.dependency.(
>> > > > groupId:artifactId:type:classifier)' must be unique:
>> > > > org.apache.hadoop:hadoop-yarn-api:jar (dbist via mmiklavc)
>> closes
>> > > > apache/metron#790
>> > > > METRON-1222: fix warning for The expression ${parent.version}
>> > > > is deprecated. Please use ${project.parent.version} instead.
>> (dbist via
>> > > > mmiklavc) closes apache/metron#782
>> > > > METRON-1220 Create documentation around alert nested field
>> > > > (justinleet) closes apache/metron#780
>> > > > METRON-1229 Management UI type is part of the declarations of
>> > > > 2 modules (merrimanr) closes apache/metron#784
>> > > > METRON-1228: Configuration Management PUSH immediately does
>> > > > DUMP after (mmiklavc via mmiklavc) closes apache/metron#783
>> > > > METRON-1218 Metron REST should return better error messages
>> > > > (merrimanr) closes apache/metron#779
>> > > > METRON-1161 Add ability to edit parser command line options
>> > > in
>> > > > the management UI (merrimanr) closes apache/metron#737
>> > > > METRON-1209: Make stellar repl take logging properties, like
>> > > > other CLI apps in metron closes apache/incubator-metron#772
>> > > > METRON-1059 address checkstyle warning AvoidStarImport in
>> > > > metron-stellar (dbist via ottobackwards) closes
>> apache/metron#664
>> > > > METRON-1204 UI does not time out after being idle, but stops
>> > > > functioning (merrimanr) closes apache/metron#771
>> > > > METRON-1052: Add forensic similarity hash functions to
>> > > Stellar
>> > > > closes apache/incubator-metron#781
>> > > > METRON-632: Added validation of "shew.enrichmentType" and
>> > > > "shew.keyColumns" closes apache/incubator-metron#732
>> > > > METRON-1194 Add Profiler Debug Functions to Profiler README
>> > > > (nickwallen via ottobackwards) closes apache/metron#765
>> > > > METRON-1055 Metron 0.4.0 manual installation guide for CentOS
>> > > > 6 updates (lvets via ottobackwards) closes apache/metron#661
>> > > > METRON-1079 STELLAR NaN should be a keyword (ottobackwards)
>> > > > closes apache/metron#681
>> > > > METRON-1085 Add REST endpoint to save a user profile for the
>> > > > Alerts UI (merrimanr) closes apache/metron#694
>> > > > METRON-1208 MPack for Alerts UI (merrimanr) closes
>> > > > apache/metron#778
>> > > > METRON-1207 Make RPMs for Alerts UI (merrimanr) closes
>> > > > apache/metron#777
>> > > > METRON-1215 Fix link to RPMs chapter (DimDroll via
>> > > justinleet)
>> > > > closes apache/metron#776
>> > > > METRON-1206 Make alerts UI conform to ops UI for install
>> > > > (merrimanr) closes apache/metron#773
>> > > > METRON-1195 Meta alerts improperly handle updates to
>> > > non-alert
>> > > > fields (justinleet) closes apache/metron#766
>> > > > METRON-1189 Add alert escalation to the Alerts UI (merrimanr)
>> > > > closes apache/metron#762
>> > > > METRON-1156 Simulate Triage Rules in the Stellar REPL
>> > > > (nickwallen) closes apache/metron#733
>> > > > METRON-1198 Pycapa - No such configuration property
>> > > > 'sasl.kerberos.principal' (nickwallen) closes apache/metron#769
>> > > > METRON-1202 ElasticsearchDao Has extraneous sleep call
>> > > > (justinleet) closes apache/metron#770
>> > > > METRON-938 "service metron-rest start <password>" does not
>> > > > work on CentOS 7. (justinleet) closes apache/metron#757
>> > > > METRON-1182 Refactor Code in alert list to accommodate new
>> > > > view types (iraghumitra via merrimanr) closes apache/metron#756
>> > > > METRON-1188: Ambari global configuration management
>> > > (mmiklavc)
>> > > > closes apache/metron#760
>> > > > METRON-1191 update public web site to point at 0.4.1 new
>> > > > release (mattf-horton) closes apache/metron#764
>> > > > METRON-1063 address javadoc warnings in metron-stellar (dbist
>> > > > via ottobackwards) closes apache/metron#668
>> > > > METRON-1190 Fix Meta Alert Type handling in calculation of
>> > > > scores (justinleet) closes apache/metron#763
>> > > > METRON-1187 Indexing/Profiler Kafka ACL Groups Not Setup
>> > > > Correctly (nickwallen) closes apache/metron#759
>> > > > METRON-1185: Stellar REPL does not work on a kerberized
>> > > > cluster when calling functions interacting with HBase closes
>> > > > apache/incubator-metron#755
>> > > > METRON-1186: Profiler Functions use classutils from shaded
>> > > > storm closes apache/incubator-metron#758
>> > > > METRON-1173: Fix pointers to old stellar docs closes
>> > > > apache/incubator-metron#746
>> > > > METRON-1179: Make STATS_ADD to take a list closes
>> > > > apache/incubator-metron#750
>> > > > METRON-1180: Make Stellar Shell accept zookeeper quorum as a
>> > > > CSV list and not require a port closes
>> apache/incubator-metron#751
>> > > > METRON-1183 Improve KDC Setup Instructions (nickwallen)
>> > > closes
>> > > > apache/metron#753
>> > > > METRON-1177 Stale running topologies seen post-kerberization
>> > > > and cause exceptions (nickwallen) closes apache/metron#748
>> > > > METRON-1158 Build backend for grouping alerts into meta
>> > > alerts
>> > > > (justinleet) closes apache/metron#734
>> > > > METRON-1146: Add ability to parse JSON string into JSONObject
>> > > > for stellar closes apache/incubator-metron#727
>> > > > METRON-1176 REST: HDFS Service should support setting
>> > > > permissions on files when writing (ottobackwards) closes
>> > > apache/metron#749
>> > > > METRON-1114 Add group by capabilities to search REST endpoint
>> > > > (merrimanr) closes apache/metron#702
>> > > > METRON-1167 Define Session Specific Global Configuration
>> > > > Values in the REPL (nickwallen) closes apache/metron#740
>> > > > METRON-1171: Better validation for the SUBSTRING stellar
>> > > > function closes apache/incubator-metron#745
>> > > >
>> > > >
>> > > >
>> > > > On 11/17/17, 11:59 AM, "Nick Allen" <ni...@nickallen.org> wrote:
>> > > >
>> > > > I just wanted to send an update on where we are at. We've
>> > > > gotten a lot
>> > > > done here recently as you can see below.
>> > > >
>> > > > ✓ DONE (1) First, METRON-1289 needs to go in. This one was
>> > > > a fairly big
>> > > > effort and I am hearing that we are pretty close.
>> > > >
>> > > > ✓ DONE (2) METRON-1294 fixes an issue in how field types
>> > > are
>> > > > looked-up.
>> > > >
>> > > > ✓ DONE (3) METRON-1290 is next. While this may have been
>> > > > fixed in
>> > > > M-1289, there may be some test cases we want from this PR.
>> > > >
>> > > > ✓ DONE (4) METRON-1301 addresses a problem with the sorting
>> > > > logic.
>> > > >
>> > > > ✓ DONE (5) METRON-1291 fixes an issue with escalation of
>> > > > metaalerts.
>> > > >
>> > > > (6) That leads us to Raghu's UI work in METRON-1252. This
>> > > > introduces the
>> > > > UI bits that depend on all the previous backend work.
>> > > >
>> > > > (7) At this point, we should have our best effort at
>> > > running
>> > > > Metaalerts
>> > > > on Elasticsearch 2.x. I propose that we cut a release here.
>> > > >
>> > > > (8) After we cut the release, we can introduce the work for
>> > > > ES 5.x in
>> > > > METRON-939. I know we will need lots of help testing and
>> > > > reviewing this
>> > > > one.
>> > > >
>> > > >
>> > > >
>> > > > We also have an outstanding question that needs resolved
>> > > > BEFORE we
>> > > > release. We need to come to a consensus on how to release
>> > > > having moved our
>> > > > Bro Plugin to a separate repo. I don't think we've heard
>> > > from
>> > > > everyone on
>> > > > this. I'd urge everyone to chime in so we can choose a path
>> > > > forward.
>> > > >
>> > > > If anyone is totally confused in regards to that discussion,
>> > > I
>> > > > can try and
>> > > > send an options summary again as a separate discuss thread.
>> > > > The original
>> > > > chain was somewhere around here [1].
>> > > >
>> > > > [1]
>> > > > https://lists.apache.org/thread.html/
>> > > > 54a4474881b97e559df24728b3a0e923a58345a282451085eef832ef@%
>> > > > 3Cdev.metron.apache.org%3E
>> > > >
>> > > >
>> > > >
>> > > > On Wed, Nov 15, 2017 at 10:04 AM, Nick Allen <
>> > > > nick@nickallen.org> wrote:
>> > > >
>> > > > > Hi Guys -
>> > > > >
>> > > > > I want to follow-up on this discussion. It sounds like
>> > > most
>> > > > people are in
>> > > > > agreement with the general approach.
>> > > > >
>> > > > > A lot of people have been working hard on Metaalerts and
>> > > > Elasticsearch. I
>> > > > > have checked-in with those doing the heavy lifting and have
>> > > > compiled a more
>> > > > > detailed plan based on where we are at now. To the best of
>> > > > my knowledge
>> > > > > here is the plan of attack for finishing out this effort.
>> > > > >
>> > > > > (1) First, METRON-1289 needs to go in. This one was a
>> > > > fairly big effort
>> > > > > and I am hearing that we are pretty close.
>> > > > >
>> > > > > (2) METRON-1294 fixes an issue in how field types are
>> > > > looked-up.
>> > > > >
>> > > > > (3) METRON-1290 is next. While this may have been fixed
>> > > > in M-1289,
>> > > > > there may be some test cases we want from this PR.
>> > > > >
>> > > > > (4) METRON-1301 addresses a problem with the sorting
>> > > logic.
>> > > > >
>> > > > > (5) METRON-1291 fixes an issue with escalation of
>> > > > metaalerts.
>> > > > >
>> > > > > (6) That leads us to Raghu's UI work in METRON-1252.
>> > > This
>> > > > introduces
>> > > > > the UI bits that depend on all the previous backend work.
>> > > > >
>> > > > > (7) At this point, we should have our best effort at
>> > > > running Metaalerts
>> > > > > on Elasticsearch 2.x. I propose that we cut a release here.
>> > > > >
>> > > > > (8) After we cut the release, we can introduce the work
>> > > > for ES 5.x in
>> > > > > METRON-939. I know we will need lots of help testing and
>> > > > reviewing this
>> > > > > one.
>> > > > >
>> > > > > Please correct me if I am wrong. I will try and send out
>> > > > updates as we
>> > > > > make progress.
>> > > > >
>> > > > >
>> > > > >
>> > > > >
>> > > > >
>> > > > > On Mon, Nov 6, 2017 at 1:03 PM, Zeolla@GMail.com <
>> > > > zeolla@gmail.com> wrote:
>> > > > >
>> > > > >> I agree, I think it's very reasonable to move in line with
>> > > > Nick's
>> > > > >> proposal. I would also suggest that we outline what the
>> > > > target versions
>> > > > >> would be to add in the METRON-777 components, since it has
>> > > > been functional
>> > > > >> for a very long time but not reviewed and has some really
>> > > > rockstar
>> > > > >> improvements.
>> > > > >>
>> > > > >> Jon
>> > > > >>
>> > > > >> On Mon, Nov 6, 2017 at 12:56 PM Otto Fowler <
>> > > > ottobackwards@gmail.com>
>> > > > >> wrote:
>> > > > >>
>> > > > >> > I think the ES cutover should be the start of the 0.5.x
>> > > > series, and we
>> > > > >> > continue on with 0.4.x for the
>> > > > >> > metadata improvements etc. We could chose to focus
>> > > > 0.5.x’s first
>> > > > >> releases
>> > > > >> > on not only ES but
>> > > > >> > getting a handle on kibana and the mpack situation as
>> > > > well.
>> > > > >> >
>> > > > >> >
>> > > > >> >
>> > > > >> >
>> > > > >> > On November 6, 2017 at 12:48:45, Michael Miklavcic (
>> > > > >> > michael.miklavcic@gmail.com) wrote:
>> > > > >> >
>> > > > >> > I agree with your proposal, Nick. I think having a
>> > > > stabilizing release
>> > > > >> > prior to upgrading ES/Kibana makes sense.
>> > > > >> >
>> > > > >> > On Mon, Nov 6, 2017 at 9:16 AM, Nick Allen <
>> > > > nick@nickallen.org> wrote:
>> > > > >> >
>> > > > >> > > I would like to start a discussion around upcoming
>> > > > releases. We have a
>> > > > >> > > couple separate significant tracks of work that we
>> > > need
>> > > > to reconcile
>> > > > >> in
>> > > > >> > our
>> > > > >> > > release schedule.
>> > > > >> > >
>> > > > >> > > (1) We have had (and have in review) a good number of
>> > > > bug fixes
>> > > > >> required
>> > > > >> > to
>> > > > >> > > support Metaalerts on the existing Elasticsearch 2.x
>> > > > infrastructure.
>> > > > >> > >
>> > > > >> > >
>> > > > >> > > (2) We also have ongoing work to upgrade our
>> > > > infrastructure to
>> > > > >> > > Elasticsearch 5.x, which will not be backwards
>> > > > compatible.
>> > > > >> > >
>> > > > >> > >
>> > > > >> > > I would like to see a release that has our best work
>> > > on
>> > > > ES 2.x before
>> > > > >> we
>> > > > >> > > migrate to 5.x. I would propose the following.
>> > > > >> > >
>> > > > >> > > Release N+1: Introduce Metaalerts running on ES 2.x
>> > > > >> > >
>> > > > >> > > Release N+2: Cut-over to ES 5.x
>> > > > >> > >
>> > > > >> > >
>> > > > >> > > (Q) Is it worth cutting a separate release for ES 2.x?
>> > > > Is there a
>> > > > >> better
>> > > > >> > > way to handle the cut-over to 5.x?
>> > > > >> > >
>> > > > >> >
>> > > > >> --
>> > > > >>
>> > > > >> Jon
>> > > > >>
>> > > > >
>> > > > >
>> > > >
>> > > >
>> > > >
>> > > >
>> > > >
>> > > >
>> > > >
>> > > >
>> > >
>> > --
>> >
>> > Jon
>> >
>> >
>> >
>> >
>> >
>> >
>>
>>
>>
>>
>
Re: [DISCUSS] Upcoming Release
Posted by Nick Allen <ni...@nickallen.org>.
Hi Matt -
I just updated like 15+ JIRAs of my own JIRAs that I completed and merged,
but failed to mark as resolved. All of these will be included in 0.4.2. I
updated each to be "fixed" in version 0.4.2 and marked as resolved.
Hopefully the next RC will report those as fixed.
(Q) Where does the list of JIRAs that get attached to the release originate
from? Does it get pulled out of JIRA or do they come from the commit log?
My apologies for not staying on top of my JIRAs.
On Tue, Dec 12, 2017 at 2:21 PM, Matt Foley <ma...@apache.org> wrote:
> Thanks to Jon, Otto, and Nick for looking over RC1. Tonight I’ll fix the
> RAT glitch, build RC2, and put it to formal vote.
> Regards,
> --Matt
>
> On 12/12/17, 5:14 AM, "Nick Allen" <ni...@nickallen.org> wrote:
>
> RC1 is looking good to me. I validated the MD5s, built Metron, built
> the
> Bro plugin and reviewed the other artifacts like release notes.
>
> Running the RAT check on a 'clean' Metron does not produce any errors
> for
> me. It is only after building Metron, which pulls in additional Node
> dependencies, does the RAT check fail.
>
>
> On Sun, Dec 10, 2017 at 4:41 PM Matt Foley <ma...@apache.org> wrote:
>
> > Yes, but let’s see if anyone else find other issues.
> >
> >
> >
> > From: Otto Fowler <ot...@gmail.com>
> > Date: Saturday, December 9, 2017 at 6:16 AM
> > To: Matt Foley <mf...@hortonworks.com>, "dev@metron.apache.org" <
> > dev@metron.apache.org>
> > Subject: Re: [DISCUSS] Upcoming Release
> >
> >
> >
> > So RC2 then?
> >
> >
> >
> > On December 8, 2017 at 20:43:21, Matt Foley (mfoley@hortonworks.com)
> > wrote:
> >
> > Hah, here it is: https://github.com/apache/metron/pull/743
> > “This problem seems to only reproduce when one unrolls a tarball
> rather
> > than cloning from github.”
> >
> > Heh, the exclusion at
> > https://github.com/apache/metron/blob/master/pom.xml#L351 is still
> there,
> > but the hashcode in the bundle.css file name has changed from
> > a0b6b99c10d9a13dc67e to f56deed131e58bd7ee04. Sigh. Did the version
> of Font
> > Awesome fonts change?
> >
> >
> > On 12/8/17, 5:26 PM, "Matt Foley" <ma...@apache.org> wrote:
> >
> > I remember having trouble with this bundle.css file on the last
> release,
> > but I can’t remember what we did about it. Anybody?
> >
> > On 12/8/17, 1:41 PM, "Otto Fowler" <ot...@gmail.com> wrote:
> >
> > Steps
> >
> > - Downloaded tar.gz’s, asc files and KEYS
> > - Verified signing of both tar.gz’s
> > - searched for rouge 0.4.1 entries
> > - verified the main pom.xml
> > - built :
> >
> > mvn clean && time mvn -q -T 2C -DskipTests install && time mvn -q -T
> > 2C surefire:test@unit-tests && time mvn -q
> > surefire:test@integration-tests && time mvn -q test --projects
> > metron-interface/metron-config && time build_utils/verify_licenses.sh
> >
> > Found rat error:
> >
> >
> > *****************************************************
> > Summary
> > -------
> > Generated at: 2017-12-08T16:33:27-05:00
> >
> > Notes: 3
> > Binaries: 193
> > Archives: 0
> > Standards: 75
> >
> > Apache Licensed: 74
> > Generated Documents: 0
> >
> > JavaDocs are generated, thus a license header is optional.
> > Generated files do not require license headers.
> >
> > 1 Unknown Licenses
> >
> > *****************************************************
> >
> > Files with unapproved licenses:
> >
> >
> > /Users/batman/tmp/release_ver/apache-metron-0.4.2-rc1/
> metron-interface/metron-alerts/dist/styles.f56deed131e58bd7ee04.bundle.css
> >
> > *****************************************************
> >
> >
> >
> >
> >
> > *****************************************************
> > Summary
> > -------
> > Generated at: 2017-12-08T16:33:27-05:00
> >
> > Notes: 3
> > Binaries: 193
> > Archives: 0
> > Standards: 75
> >
> > Apache Licensed: 74
> > Generated Documents: 0
> >
> > JavaDocs are generated, thus a license header is optional.
> > Generated files do not require license headers.
> >
> > 1 Unknown Licenses
> >
> > *****************************************************
> >
> > Files with unapproved licenses:
> >
> >
> >
> > /Users/ottofowler/tmp/release_ver/apache-metron-0.4.2-rc1/
> metron-interface/metron-alerts/dist/styles.f56deed131e58bd7ee04.bundle.css
> >
> > *****************************************************
> >
> >
> >
> > On December 8, 2017 at 04:34:24, Matt Foley (mattf@apache.org)
> wrote:
> >
> > Colleagues,
> > I’ve posted Metron-0.4.2-RC1 and Metron-bro-plugin-kafka-0.1 to
> > https://dist.apache.org/repos/dist/dev/metron/0.4.2-RC1/
> >
> > Given the complexity of this RC, I’d appreciate if a couple people
> would be
> > willing to kick the tires before we put it up for a vote.
> >
> > I will myself be going thru the Verify Build process this weekend,
> as I
> > won’t be able to do it Friday.
> >
> > Thanks,
> > --Matt
> >
> >
> > On 12/4/17, 2:05 PM, "Zeolla@GMail.com" <ze...@gmail.com> wrote:
> >
> > Can we resolve the conversation regarding the second repo? I was
> waiting
> > to get more input/preferences from people There's also a
> documentation
> > update that fixes a few broken Stellar docs that already has aa +1,
> I just
> > need to merge it.
> >
> > Jon
> >
> > On Mon, Dec 4, 2017, 17:01 Casey Stella <ce...@gmail.com> wrote:
> >
> > > I would be in favor of a release at this point.
> > >
> > > On Mon, Dec 4, 2017 at 4:57 PM, Matt Foley <ma...@apache.org>
> wrote:
> > >
> > > > Hey all,
> > > > I see METRON-1252 was resolved over the weekend. Shall I go
> ahead and
> > > > start the process with 0.4.2 release?
> > > > Does anyone have any commits they feel strongly should go in
> before
> > 0.4.2
> > > > is done, or are we ready to call it good?
> > > >
> > > > I believe there is consensus the 0.4.2 release should include a
> release
> > > of
> > > > the current state of the metron-bro-plugin-kafka. I will
> continue the
> > > > discussion in that thread as to the process for accomplishing
> that, but
> > > > plan on it happening.
> > > >
> > > > Regards,
> > > > --Matt
> > > >
> > > > On 11/26/17, 6:26 PM, "Matt Foley" <ma...@apache.org> wrote:
> > > >
> > > > Hope everyone (at least in the U.S.) had a great Thanksgiving
> > > holiday.
> > > > Regarding status of the release effort, still pending
> METRON-1252, so
> > > > not making the release branch yet.
> > > >
> > > > Regards,
> > > > --Matt
> > > >
> > > > On 11/17/17, 1:32 PM, "Matt Foley" <ma...@apache.org> wrote:
> > > >
> > > > (With release manager hat on)
> > > >
> > > > The community has proposed a release of Metron in the near
> > > future,
> > > > focusing on Meta-alerts running in Elasticsearch.
> > > > Congrats on getting so many of the below already done. At this
> > > > point, only METRON-1252, and the discussion of how to handle
> joint
> > > release
> > > > of the Metron bro plugin, remain as gating items for the
> release. I
> > > > project these will be resolved next week, so let’s propose the
> > following:
> > > >
> > > > Sometime next week, after the last bits are done, I’ll start the
> > > > release process and create the release branch.
> > > >
> > > > The proposed new version will be 0.4.2, unless there are backward
> > > > incompatible changes that support making it 0.5.0.
> > > > Currently there are NO included Jiras labeled
> > > > ‘backward-incompatible’, nor having Docs Text indicating so.
> > > > ***If anyone knows that some of the commits included since 0.4.1
> > > > introduce backward incompatibility, please say so now on this
> thread,
> > and
> > > > mark the Jira as such.***
> > > >
> > > > The 90 or so jiras/commits already in master branch since 0.4.1
> > > > are listed below.
> > > > Thanks,
> > > > --Matt
> > > >
> > > > METRON-1301 Alerts UI - Sorting on Triage Score Unexpectedly
> > > > Filters Some Records (nickwallen) closes apache/metron#832
> > > > METRON-1294 IP addresses are not formatted correctly in facet
> > > > and group results (merrimanr) closes apache/metron#827
> > > > METRON-1291 Kafka produce REST endpoint does not work in a
> > > > Kerberized cluster (merrimanr) closes apache/metron#826
> > > > METRON-1290 Only first 10 alerts are update when a MetaAlert
> > > > status is changed to inactive (justinleet) closes
> apache/metron#842
> > > > METRON-1311 Service Check Should Check Elasticsearch Index
> > > > Templates (nickwallen) closes apache/metron#839
> > > > METRON-1289 Alert fields are lost when a MetaAlert is created
> > > > (merrimanr) closes apache/metron#824
> > > > METRON-1309 Change metron-deployment to pull the plugin from
> > > > apache/metron-bro-plugin-kafka (JonZeolla) closes
> apache/metron#837
> > > > METRON-1310 Template Delete Action Deletes Search Indices
> > > > (nickwallen) closes apache/metron#838
> > > > METRON-1275: Fix Metron Documentation closes
> > > > apache/incubator-metron#833
> > > > METRON-1295 Unable to Configure Logging for REST API
> > > > (nickwallen) closes apache/metron#828
> > > > METRON-1307 Force install of java8 since java9 does not
> > > appear
> > > > to work with the scripts (brianhurley via ottobackwards) closes
> > > > apache/metron#835
> > > > METRON-1296 Full Dev Fails to Deploy Index Templates
> > > > (nickwallen via cestella) closes apache/incubator-metron#829
> > > > METRON-1281 Remove hard-coded indices from the Alerts UI
> > > > (merrimanr) closes apache/metron#821
> > > > METRON-1287 Full Dev Fails When Installing EPEL Repository
> > > > (nickwallen) closes apache/metron#820
> > > > METRON-1267 Alerts UI returns a 404 when refreshing the
> > > > alerts-list page (iraghumitra via merrimanr) closes
> apache/metron#819
> > > > METRON-1283 Install Elasticsearch template as a part of the
> > > > mpack startup scripts (anandsubbu via nickwallen) closes
> > > apache/metron#817
> > > > METRON-1254: Conditionals as map keys do not function in
> > > > Stellar closes apache/incubator-metron#801
> > > > METRON-1261 Apply bro security patch (JonZeolla via
> > > > ottobackwards) closes apache/metron#805
> > > > METRON-1284 Remove extraneous dead query in ElasticsearchDao
> > > > (justinleet) closes apache/metron#818
> > > > METRON-1270: fix for warnings missing @return tag argument in
> > > > metron-analytics/metron-profiler-common and
> metron-profiler-client
> > closes
> > > > apache/incubator-metron#810
> > > > METRON-1272 Hide child alerts from searches and grouping if
> > > > they belong to meta alerts (justinleet) closes apache/metron#811
> > > > METRON-1224 Add time range selection to search control
> > > > (iraghumitra via james-sirota) closes apache/metron#796
> > > > METRON-1280 0.4.1 -> 0.4.2 missed a couple of projects
> > > > (cestella via justinleet) closes apache/metron#816
> > > > METRON-1243: Add a REST endpoint which allows us to get a
> > > list
> > > > of all indice closes apache/incubator-metron#797
> > > > METRON-1196 Increment master version number to 0.4.2 for
> > > > on-going development (mattf-horton) closes apache/metron#767
> > > > METRON-1278 Strip "Build Status" widget from root
> > > > README.md in site-book build (mattf-horton) closes
> apache/metron#815
> > > > METRON-1274 Master has failure in
> > > > StormControllerIntegrationTest (merrimanr) closes
> apache/metron#813
> > > > METRON-1266 Profiler - SASL Authentication Failed
> > > (nickwallen)
> > > > closes apache/metron#809
> > > > METRON-1260 Include Alerts UI in Ambari Service Check
> > > > (nickwallen) closes apache/metron#804
> > > > METRON-1251: Typo and formatting fixes for metron-rest README
> > > > closes apache/incubator-metron#800
> > > > METRON-1241: Enable the REST API to use a cache for the
> > > > zookeeper config similar to the Bolts closes
> > apache/incubator-metron#795
> > > > METRON-1267 Alerts UI returns a 404 when refreshing the
> > > > alerts-list page (merrimanr) closes apache/metron#808
> > > > METRON-1262 Unable to add comment for a alert in a meta-alert
> > > > (merrimanr) closes apache/metron#806
> > > > METRON-1263 Start Alerts UI service after Metron REST
> > > > (anandsubbu via nickwallen) closes apache/metron#807
> > > > METRON-1255 MetaAlert search is not filtering on status
> > > > (merrimanr) closes apache/metron#802
> > > > METRON-1249 Improve Metron MPack Service Checks (nickwallen)
> > > > closes apache/metron#799
> > > > METRON-1237 address javadoc warnings in metron-maas-common
> > > > (dbist via james-sirota) closes apache/metron#792
> > > > METRON-1240 address javadoc warnings in metron-platform and
> > > > metron-analytics (dbist via james-sirota) closes
> apache/metron#794
> > > > METRON-1226 Searching Can Errantly Query the Wrong Indices
> > > > (nickwallen) closes apache/metron#793
> > > > METRON-1081 Fix Alerts and Ops UI Notices file (james-sirota)
> > > > closes apache/metron#682
> > > > METRON-1123 Add group by option using faceted search
> > > > capabilities of metron-rest-api (iraghumitra via james-sirota)
> closes
> > > > apache/metron#768
> > > > METRON-1223 Add support to add comments for alerts
> > > > (iraghumitra via james-sirota) closes apache/metron#788
> > > > METRON-1083 Add filters using faceted search capabilities of
> > > > metron-rest-api (iraghumitra via james-sirota) closes
> apache/metron#710
> > > > METRON-1232 Alert status changes are not reflected in list
> > > > view (iraghumitra via merrimanr) closes apache/metron#787
> > > > METRON-1247 REST search and findOne endpoints return
> > > > unexpected or incorrect results for guids (justinleet) closes
> > > > apache/metron#798
> > > > METRON-1235: Document the properties pulled from the global
> > > > configuration closes apache/incubator-metron#791
> > > > METRON-1234: fix for WARNING 'dependencies.dependency.(
> > > > groupId:artifactId:type:classifier)' must be unique:
> > > > org.apache.hadoop:hadoop-yarn-api:jar (dbist via mmiklavc)
> closes
> > > > apache/metron#790
> > > > METRON-1222: fix warning for The expression ${parent.version}
> > > > is deprecated. Please use ${project.parent.version} instead.
> (dbist via
> > > > mmiklavc) closes apache/metron#782
> > > > METRON-1220 Create documentation around alert nested field
> > > > (justinleet) closes apache/metron#780
> > > > METRON-1229 Management UI type is part of the declarations of
> > > > 2 modules (merrimanr) closes apache/metron#784
> > > > METRON-1228: Configuration Management PUSH immediately does
> > > > DUMP after (mmiklavc via mmiklavc) closes apache/metron#783
> > > > METRON-1218 Metron REST should return better error messages
> > > > (merrimanr) closes apache/metron#779
> > > > METRON-1161 Add ability to edit parser command line options
> > > in
> > > > the management UI (merrimanr) closes apache/metron#737
> > > > METRON-1209: Make stellar repl take logging properties, like
> > > > other CLI apps in metron closes apache/incubator-metron#772
> > > > METRON-1059 address checkstyle warning AvoidStarImport in
> > > > metron-stellar (dbist via ottobackwards) closes apache/metron#664
> > > > METRON-1204 UI does not time out after being idle, but stops
> > > > functioning (merrimanr) closes apache/metron#771
> > > > METRON-1052: Add forensic similarity hash functions to
> > > Stellar
> > > > closes apache/incubator-metron#781
> > > > METRON-632: Added validation of "shew.enrichmentType" and
> > > > "shew.keyColumns" closes apache/incubator-metron#732
> > > > METRON-1194 Add Profiler Debug Functions to Profiler README
> > > > (nickwallen via ottobackwards) closes apache/metron#765
> > > > METRON-1055 Metron 0.4.0 manual installation guide for CentOS
> > > > 6 updates (lvets via ottobackwards) closes apache/metron#661
> > > > METRON-1079 STELLAR NaN should be a keyword (ottobackwards)
> > > > closes apache/metron#681
> > > > METRON-1085 Add REST endpoint to save a user profile for the
> > > > Alerts UI (merrimanr) closes apache/metron#694
> > > > METRON-1208 MPack for Alerts UI (merrimanr) closes
> > > > apache/metron#778
> > > > METRON-1207 Make RPMs for Alerts UI (merrimanr) closes
> > > > apache/metron#777
> > > > METRON-1215 Fix link to RPMs chapter (DimDroll via
> > > justinleet)
> > > > closes apache/metron#776
> > > > METRON-1206 Make alerts UI conform to ops UI for install
> > > > (merrimanr) closes apache/metron#773
> > > > METRON-1195 Meta alerts improperly handle updates to
> > > non-alert
> > > > fields (justinleet) closes apache/metron#766
> > > > METRON-1189 Add alert escalation to the Alerts UI (merrimanr)
> > > > closes apache/metron#762
> > > > METRON-1156 Simulate Triage Rules in the Stellar REPL
> > > > (nickwallen) closes apache/metron#733
> > > > METRON-1198 Pycapa - No such configuration property
> > > > 'sasl.kerberos.principal' (nickwallen) closes apache/metron#769
> > > > METRON-1202 ElasticsearchDao Has extraneous sleep call
> > > > (justinleet) closes apache/metron#770
> > > > METRON-938 "service metron-rest start <password>" does not
> > > > work on CentOS 7. (justinleet) closes apache/metron#757
> > > > METRON-1182 Refactor Code in alert list to accommodate new
> > > > view types (iraghumitra via merrimanr) closes apache/metron#756
> > > > METRON-1188: Ambari global configuration management
> > > (mmiklavc)
> > > > closes apache/metron#760
> > > > METRON-1191 update public web site to point at 0.4.1 new
> > > > release (mattf-horton) closes apache/metron#764
> > > > METRON-1063 address javadoc warnings in metron-stellar (dbist
> > > > via ottobackwards) closes apache/metron#668
> > > > METRON-1190 Fix Meta Alert Type handling in calculation of
> > > > scores (justinleet) closes apache/metron#763
> > > > METRON-1187 Indexing/Profiler Kafka ACL Groups Not Setup
> > > > Correctly (nickwallen) closes apache/metron#759
> > > > METRON-1185: Stellar REPL does not work on a kerberized
> > > > cluster when calling functions interacting with HBase closes
> > > > apache/incubator-metron#755
> > > > METRON-1186: Profiler Functions use classutils from shaded
> > > > storm closes apache/incubator-metron#758
> > > > METRON-1173: Fix pointers to old stellar docs closes
> > > > apache/incubator-metron#746
> > > > METRON-1179: Make STATS_ADD to take a list closes
> > > > apache/incubator-metron#750
> > > > METRON-1180: Make Stellar Shell accept zookeeper quorum as a
> > > > CSV list and not require a port closes
> apache/incubator-metron#751
> > > > METRON-1183 Improve KDC Setup Instructions (nickwallen)
> > > closes
> > > > apache/metron#753
> > > > METRON-1177 Stale running topologies seen post-kerberization
> > > > and cause exceptions (nickwallen) closes apache/metron#748
> > > > METRON-1158 Build backend for grouping alerts into meta
> > > alerts
> > > > (justinleet) closes apache/metron#734
> > > > METRON-1146: Add ability to parse JSON string into JSONObject
> > > > for stellar closes apache/incubator-metron#727
> > > > METRON-1176 REST: HDFS Service should support setting
> > > > permissions on files when writing (ottobackwards) closes
> > > apache/metron#749
> > > > METRON-1114 Add group by capabilities to search REST endpoint
> > > > (merrimanr) closes apache/metron#702
> > > > METRON-1167 Define Session Specific Global Configuration
> > > > Values in the REPL (nickwallen) closes apache/metron#740
> > > > METRON-1171: Better validation for the SUBSTRING stellar
> > > > function closes apache/incubator-metron#745
> > > >
> > > >
> > > >
> > > > On 11/17/17, 11:59 AM, "Nick Allen" <ni...@nickallen.org> wrote:
> > > >
> > > > I just wanted to send an update on where we are at. We've
> > > > gotten a lot
> > > > done here recently as you can see below.
> > > >
> > > > ✓ DONE (1) First, METRON-1289 needs to go in. This one was
> > > > a fairly big
> > > > effort and I am hearing that we are pretty close.
> > > >
> > > > ✓ DONE (2) METRON-1294 fixes an issue in how field types
> > > are
> > > > looked-up.
> > > >
> > > > ✓ DONE (3) METRON-1290 is next. While this may have been
> > > > fixed in
> > > > M-1289, there may be some test cases we want from this PR.
> > > >
> > > > ✓ DONE (4) METRON-1301 addresses a problem with the sorting
> > > > logic.
> > > >
> > > > ✓ DONE (5) METRON-1291 fixes an issue with escalation of
> > > > metaalerts.
> > > >
> > > > (6) That leads us to Raghu's UI work in METRON-1252. This
> > > > introduces the
> > > > UI bits that depend on all the previous backend work.
> > > >
> > > > (7) At this point, we should have our best effort at
> > > running
> > > > Metaalerts
> > > > on Elasticsearch 2.x. I propose that we cut a release here.
> > > >
> > > > (8) After we cut the release, we can introduce the work for
> > > > ES 5.x in
> > > > METRON-939. I know we will need lots of help testing and
> > > > reviewing this
> > > > one.
> > > >
> > > >
> > > >
> > > > We also have an outstanding question that needs resolved
> > > > BEFORE we
> > > > release. We need to come to a consensus on how to release
> > > > having moved our
> > > > Bro Plugin to a separate repo. I don't think we've heard
> > > from
> > > > everyone on
> > > > this. I'd urge everyone to chime in so we can choose a path
> > > > forward.
> > > >
> > > > If anyone is totally confused in regards to that discussion,
> > > I
> > > > can try and
> > > > send an options summary again as a separate discuss thread.
> > > > The original
> > > > chain was somewhere around here [1].
> > > >
> > > > [1]
> > > > https://lists.apache.org/thread.html/
> > > > 54a4474881b97e559df24728b3a0e923a58345a282451085eef832ef@%
> > > > 3Cdev.metron.apache.org%3E
> > > >
> > > >
> > > >
> > > > On Wed, Nov 15, 2017 at 10:04 AM, Nick Allen <
> > > > nick@nickallen.org> wrote:
> > > >
> > > > > Hi Guys -
> > > > >
> > > > > I want to follow-up on this discussion. It sounds like
> > > most
> > > > people are in
> > > > > agreement with the general approach.
> > > > >
> > > > > A lot of people have been working hard on Metaalerts and
> > > > Elasticsearch. I
> > > > > have checked-in with those doing the heavy lifting and have
> > > > compiled a more
> > > > > detailed plan based on where we are at now. To the best of
> > > > my knowledge
> > > > > here is the plan of attack for finishing out this effort.
> > > > >
> > > > > (1) First, METRON-1289 needs to go in. This one was a
> > > > fairly big effort
> > > > > and I am hearing that we are pretty close.
> > > > >
> > > > > (2) METRON-1294 fixes an issue in how field types are
> > > > looked-up.
> > > > >
> > > > > (3) METRON-1290 is next. While this may have been fixed
> > > > in M-1289,
> > > > > there may be some test cases we want from this PR.
> > > > >
> > > > > (4) METRON-1301 addresses a problem with the sorting
> > > logic.
> > > > >
> > > > > (5) METRON-1291 fixes an issue with escalation of
> > > > metaalerts.
> > > > >
> > > > > (6) That leads us to Raghu's UI work in METRON-1252.
> > > This
> > > > introduces
> > > > > the UI bits that depend on all the previous backend work.
> > > > >
> > > > > (7) At this point, we should have our best effort at
> > > > running Metaalerts
> > > > > on Elasticsearch 2.x. I propose that we cut a release here.
> > > > >
> > > > > (8) After we cut the release, we can introduce the work
> > > > for ES 5.x in
> > > > > METRON-939. I know we will need lots of help testing and
> > > > reviewing this
> > > > > one.
> > > > >
> > > > > Please correct me if I am wrong. I will try and send out
> > > > updates as we
> > > > > make progress.
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > On Mon, Nov 6, 2017 at 1:03 PM, Zeolla@GMail.com <
> > > > zeolla@gmail.com> wrote:
> > > > >
> > > > >> I agree, I think it's very reasonable to move in line with
> > > > Nick's
> > > > >> proposal. I would also suggest that we outline what the
> > > > target versions
> > > > >> would be to add in the METRON-777 components, since it has
> > > > been functional
> > > > >> for a very long time but not reviewed and has some really
> > > > rockstar
> > > > >> improvements.
> > > > >>
> > > > >> Jon
> > > > >>
> > > > >> On Mon, Nov 6, 2017 at 12:56 PM Otto Fowler <
> > > > ottobackwards@gmail.com>
> > > > >> wrote:
> > > > >>
> > > > >> > I think the ES cutover should be the start of the 0.5.x
> > > > series, and we
> > > > >> > continue on with 0.4.x for the
> > > > >> > metadata improvements etc. We could chose to focus
> > > > 0.5.x’s first
> > > > >> releases
> > > > >> > on not only ES but
> > > > >> > getting a handle on kibana and the mpack situation as
> > > > well.
> > > > >> >
> > > > >> >
> > > > >> >
> > > > >> >
> > > > >> > On November 6, 2017 at 12:48:45, Michael Miklavcic (
> > > > >> > michael.miklavcic@gmail.com) wrote:
> > > > >> >
> > > > >> > I agree with your proposal, Nick. I think having a
> > > > stabilizing release
> > > > >> > prior to upgrading ES/Kibana makes sense.
> > > > >> >
> > > > >> > On Mon, Nov 6, 2017 at 9:16 AM, Nick Allen <
> > > > nick@nickallen.org> wrote:
> > > > >> >
> > > > >> > > I would like to start a discussion around upcoming
> > > > releases. We have a
> > > > >> > > couple separate significant tracks of work that we
> > > need
> > > > to reconcile
> > > > >> in
> > > > >> > our
> > > > >> > > release schedule.
> > > > >> > >
> > > > >> > > (1) We have had (and have in review) a good number of
> > > > bug fixes
> > > > >> required
> > > > >> > to
> > > > >> > > support Metaalerts on the existing Elasticsearch 2.x
> > > > infrastructure.
> > > > >> > >
> > > > >> > >
> > > > >> > > (2) We also have ongoing work to upgrade our
> > > > infrastructure to
> > > > >> > > Elasticsearch 5.x, which will not be backwards
> > > > compatible.
> > > > >> > >
> > > > >> > >
> > > > >> > > I would like to see a release that has our best work
> > > on
> > > > ES 2.x before
> > > > >> we
> > > > >> > > migrate to 5.x. I would propose the following.
> > > > >> > >
> > > > >> > > Release N+1: Introduce Metaalerts running on ES 2.x
> > > > >> > >
> > > > >> > > Release N+2: Cut-over to ES 5.x
> > > > >> > >
> > > > >> > >
> > > > >> > > (Q) Is it worth cutting a separate release for ES 2.x?
> > > > Is there a
> > > > >> better
> > > > >> > > way to handle the cut-over to 5.x?
> > > > >> > >
> > > > >> >
> > > > >> --
> > > > >>
> > > > >> Jon
> > > > >>
> > > > >
> > > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > >
> > --
> >
> > Jon
> >
> >
> >
> >
> >
> >
>
>
>
>
Re: [DISCUSS] Upcoming Release
Posted by Matt Foley <ma...@apache.org>.
Thanks to Jon, Otto, and Nick for looking over RC1. Tonight I’ll fix the RAT glitch, build RC2, and put it to formal vote.
Regards,
--Matt
On 12/12/17, 5:14 AM, "Nick Allen" <ni...@nickallen.org> wrote:
RC1 is looking good to me. I validated the MD5s, built Metron, built the
Bro plugin and reviewed the other artifacts like release notes.
Running the RAT check on a 'clean' Metron does not produce any errors for
me. It is only after building Metron, which pulls in additional Node
dependencies, does the RAT check fail.
On Sun, Dec 10, 2017 at 4:41 PM Matt Foley <ma...@apache.org> wrote:
> Yes, but let’s see if anyone else find other issues.
>
>
>
> From: Otto Fowler <ot...@gmail.com>
> Date: Saturday, December 9, 2017 at 6:16 AM
> To: Matt Foley <mf...@hortonworks.com>, "dev@metron.apache.org" <
> dev@metron.apache.org>
> Subject: Re: [DISCUSS] Upcoming Release
>
>
>
> So RC2 then?
>
>
>
> On December 8, 2017 at 20:43:21, Matt Foley (mfoley@hortonworks.com)
> wrote:
>
> Hah, here it is: https://github.com/apache/metron/pull/743
> “This problem seems to only reproduce when one unrolls a tarball rather
> than cloning from github.”
>
> Heh, the exclusion at
> https://github.com/apache/metron/blob/master/pom.xml#L351 is still there,
> but the hashcode in the bundle.css file name has changed from
> a0b6b99c10d9a13dc67e to f56deed131e58bd7ee04. Sigh. Did the version of Font
> Awesome fonts change?
>
>
> On 12/8/17, 5:26 PM, "Matt Foley" <ma...@apache.org> wrote:
>
> I remember having trouble with this bundle.css file on the last release,
> but I can’t remember what we did about it. Anybody?
>
> On 12/8/17, 1:41 PM, "Otto Fowler" <ot...@gmail.com> wrote:
>
> Steps
>
> - Downloaded tar.gz’s, asc files and KEYS
> - Verified signing of both tar.gz’s
> - searched for rouge 0.4.1 entries
> - verified the main pom.xml
> - built :
>
> mvn clean && time mvn -q -T 2C -DskipTests install && time mvn -q -T
> 2C surefire:test@unit-tests && time mvn -q
> surefire:test@integration-tests && time mvn -q test --projects
> metron-interface/metron-config && time build_utils/verify_licenses.sh
>
> Found rat error:
>
>
> *****************************************************
> Summary
> -------
> Generated at: 2017-12-08T16:33:27-05:00
>
> Notes: 3
> Binaries: 193
> Archives: 0
> Standards: 75
>
> Apache Licensed: 74
> Generated Documents: 0
>
> JavaDocs are generated, thus a license header is optional.
> Generated files do not require license headers.
>
> 1 Unknown Licenses
>
> *****************************************************
>
> Files with unapproved licenses:
>
>
> /Users/batman/tmp/release_ver/apache-metron-0.4.2-rc1/metron-interface/metron-alerts/dist/styles.f56deed131e58bd7ee04.bundle.css
>
> *****************************************************
>
>
>
>
>
> *****************************************************
> Summary
> -------
> Generated at: 2017-12-08T16:33:27-05:00
>
> Notes: 3
> Binaries: 193
> Archives: 0
> Standards: 75
>
> Apache Licensed: 74
> Generated Documents: 0
>
> JavaDocs are generated, thus a license header is optional.
> Generated files do not require license headers.
>
> 1 Unknown Licenses
>
> *****************************************************
>
> Files with unapproved licenses:
>
>
>
> /Users/ottofowler/tmp/release_ver/apache-metron-0.4.2-rc1/metron-interface/metron-alerts/dist/styles.f56deed131e58bd7ee04.bundle.css
>
> *****************************************************
>
>
>
> On December 8, 2017 at 04:34:24, Matt Foley (mattf@apache.org) wrote:
>
> Colleagues,
> I’ve posted Metron-0.4.2-RC1 and Metron-bro-plugin-kafka-0.1 to
> https://dist.apache.org/repos/dist/dev/metron/0.4.2-RC1/
>
> Given the complexity of this RC, I’d appreciate if a couple people would be
> willing to kick the tires before we put it up for a vote.
>
> I will myself be going thru the Verify Build process this weekend, as I
> won’t be able to do it Friday.
>
> Thanks,
> --Matt
>
>
> On 12/4/17, 2:05 PM, "Zeolla@GMail.com" <ze...@gmail.com> wrote:
>
> Can we resolve the conversation regarding the second repo? I was waiting
> to get more input/preferences from people There's also a documentation
> update that fixes a few broken Stellar docs that already has aa +1, I just
> need to merge it.
>
> Jon
>
> On Mon, Dec 4, 2017, 17:01 Casey Stella <ce...@gmail.com> wrote:
>
> > I would be in favor of a release at this point.
> >
> > On Mon, Dec 4, 2017 at 4:57 PM, Matt Foley <ma...@apache.org> wrote:
> >
> > > Hey all,
> > > I see METRON-1252 was resolved over the weekend. Shall I go ahead and
> > > start the process with 0.4.2 release?
> > > Does anyone have any commits they feel strongly should go in before
> 0.4.2
> > > is done, or are we ready to call it good?
> > >
> > > I believe there is consensus the 0.4.2 release should include a release
> > of
> > > the current state of the metron-bro-plugin-kafka. I will continue the
> > > discussion in that thread as to the process for accomplishing that, but
> > > plan on it happening.
> > >
> > > Regards,
> > > --Matt
> > >
> > > On 11/26/17, 6:26 PM, "Matt Foley" <ma...@apache.org> wrote:
> > >
> > > Hope everyone (at least in the U.S.) had a great Thanksgiving
> > holiday.
> > > Regarding status of the release effort, still pending METRON-1252, so
> > > not making the release branch yet.
> > >
> > > Regards,
> > > --Matt
> > >
> > > On 11/17/17, 1:32 PM, "Matt Foley" <ma...@apache.org> wrote:
> > >
> > > (With release manager hat on)
> > >
> > > The community has proposed a release of Metron in the near
> > future,
> > > focusing on Meta-alerts running in Elasticsearch.
> > > Congrats on getting so many of the below already done. At this
> > > point, only METRON-1252, and the discussion of how to handle joint
> > release
> > > of the Metron bro plugin, remain as gating items for the release. I
> > > project these will be resolved next week, so let’s propose the
> following:
> > >
> > > Sometime next week, after the last bits are done, I’ll start the
> > > release process and create the release branch.
> > >
> > > The proposed new version will be 0.4.2, unless there are backward
> > > incompatible changes that support making it 0.5.0.
> > > Currently there are NO included Jiras labeled
> > > ‘backward-incompatible’, nor having Docs Text indicating so.
> > > ***If anyone knows that some of the commits included since 0.4.1
> > > introduce backward incompatibility, please say so now on this thread,
> and
> > > mark the Jira as such.***
> > >
> > > The 90 or so jiras/commits already in master branch since 0.4.1
> > > are listed below.
> > > Thanks,
> > > --Matt
> > >
> > > METRON-1301 Alerts UI - Sorting on Triage Score Unexpectedly
> > > Filters Some Records (nickwallen) closes apache/metron#832
> > > METRON-1294 IP addresses are not formatted correctly in facet
> > > and group results (merrimanr) closes apache/metron#827
> > > METRON-1291 Kafka produce REST endpoint does not work in a
> > > Kerberized cluster (merrimanr) closes apache/metron#826
> > > METRON-1290 Only first 10 alerts are update when a MetaAlert
> > > status is changed to inactive (justinleet) closes apache/metron#842
> > > METRON-1311 Service Check Should Check Elasticsearch Index
> > > Templates (nickwallen) closes apache/metron#839
> > > METRON-1289 Alert fields are lost when a MetaAlert is created
> > > (merrimanr) closes apache/metron#824
> > > METRON-1309 Change metron-deployment to pull the plugin from
> > > apache/metron-bro-plugin-kafka (JonZeolla) closes apache/metron#837
> > > METRON-1310 Template Delete Action Deletes Search Indices
> > > (nickwallen) closes apache/metron#838
> > > METRON-1275: Fix Metron Documentation closes
> > > apache/incubator-metron#833
> > > METRON-1295 Unable to Configure Logging for REST API
> > > (nickwallen) closes apache/metron#828
> > > METRON-1307 Force install of java8 since java9 does not
> > appear
> > > to work with the scripts (brianhurley via ottobackwards) closes
> > > apache/metron#835
> > > METRON-1296 Full Dev Fails to Deploy Index Templates
> > > (nickwallen via cestella) closes apache/incubator-metron#829
> > > METRON-1281 Remove hard-coded indices from the Alerts UI
> > > (merrimanr) closes apache/metron#821
> > > METRON-1287 Full Dev Fails When Installing EPEL Repository
> > > (nickwallen) closes apache/metron#820
> > > METRON-1267 Alerts UI returns a 404 when refreshing the
> > > alerts-list page (iraghumitra via merrimanr) closes apache/metron#819
> > > METRON-1283 Install Elasticsearch template as a part of the
> > > mpack startup scripts (anandsubbu via nickwallen) closes
> > apache/metron#817
> > > METRON-1254: Conditionals as map keys do not function in
> > > Stellar closes apache/incubator-metron#801
> > > METRON-1261 Apply bro security patch (JonZeolla via
> > > ottobackwards) closes apache/metron#805
> > > METRON-1284 Remove extraneous dead query in ElasticsearchDao
> > > (justinleet) closes apache/metron#818
> > > METRON-1270: fix for warnings missing @return tag argument in
> > > metron-analytics/metron-profiler-common and metron-profiler-client
> closes
> > > apache/incubator-metron#810
> > > METRON-1272 Hide child alerts from searches and grouping if
> > > they belong to meta alerts (justinleet) closes apache/metron#811
> > > METRON-1224 Add time range selection to search control
> > > (iraghumitra via james-sirota) closes apache/metron#796
> > > METRON-1280 0.4.1 -> 0.4.2 missed a couple of projects
> > > (cestella via justinleet) closes apache/metron#816
> > > METRON-1243: Add a REST endpoint which allows us to get a
> > list
> > > of all indice closes apache/incubator-metron#797
> > > METRON-1196 Increment master version number to 0.4.2 for
> > > on-going development (mattf-horton) closes apache/metron#767
> > > METRON-1278 Strip "Build Status" widget from root
> > > README.md in site-book build (mattf-horton) closes apache/metron#815
> > > METRON-1274 Master has failure in
> > > StormControllerIntegrationTest (merrimanr) closes apache/metron#813
> > > METRON-1266 Profiler - SASL Authentication Failed
> > (nickwallen)
> > > closes apache/metron#809
> > > METRON-1260 Include Alerts UI in Ambari Service Check
> > > (nickwallen) closes apache/metron#804
> > > METRON-1251: Typo and formatting fixes for metron-rest README
> > > closes apache/incubator-metron#800
> > > METRON-1241: Enable the REST API to use a cache for the
> > > zookeeper config similar to the Bolts closes
> apache/incubator-metron#795
> > > METRON-1267 Alerts UI returns a 404 when refreshing the
> > > alerts-list page (merrimanr) closes apache/metron#808
> > > METRON-1262 Unable to add comment for a alert in a meta-alert
> > > (merrimanr) closes apache/metron#806
> > > METRON-1263 Start Alerts UI service after Metron REST
> > > (anandsubbu via nickwallen) closes apache/metron#807
> > > METRON-1255 MetaAlert search is not filtering on status
> > > (merrimanr) closes apache/metron#802
> > > METRON-1249 Improve Metron MPack Service Checks (nickwallen)
> > > closes apache/metron#799
> > > METRON-1237 address javadoc warnings in metron-maas-common
> > > (dbist via james-sirota) closes apache/metron#792
> > > METRON-1240 address javadoc warnings in metron-platform and
> > > metron-analytics (dbist via james-sirota) closes apache/metron#794
> > > METRON-1226 Searching Can Errantly Query the Wrong Indices
> > > (nickwallen) closes apache/metron#793
> > > METRON-1081 Fix Alerts and Ops UI Notices file (james-sirota)
> > > closes apache/metron#682
> > > METRON-1123 Add group by option using faceted search
> > > capabilities of metron-rest-api (iraghumitra via james-sirota) closes
> > > apache/metron#768
> > > METRON-1223 Add support to add comments for alerts
> > > (iraghumitra via james-sirota) closes apache/metron#788
> > > METRON-1083 Add filters using faceted search capabilities of
> > > metron-rest-api (iraghumitra via james-sirota) closes apache/metron#710
> > > METRON-1232 Alert status changes are not reflected in list
> > > view (iraghumitra via merrimanr) closes apache/metron#787
> > > METRON-1247 REST search and findOne endpoints return
> > > unexpected or incorrect results for guids (justinleet) closes
> > > apache/metron#798
> > > METRON-1235: Document the properties pulled from the global
> > > configuration closes apache/incubator-metron#791
> > > METRON-1234: fix for WARNING 'dependencies.dependency.(
> > > groupId:artifactId:type:classifier)' must be unique:
> > > org.apache.hadoop:hadoop-yarn-api:jar (dbist via mmiklavc) closes
> > > apache/metron#790
> > > METRON-1222: fix warning for The expression ${parent.version}
> > > is deprecated. Please use ${project.parent.version} instead. (dbist via
> > > mmiklavc) closes apache/metron#782
> > > METRON-1220 Create documentation around alert nested field
> > > (justinleet) closes apache/metron#780
> > > METRON-1229 Management UI type is part of the declarations of
> > > 2 modules (merrimanr) closes apache/metron#784
> > > METRON-1228: Configuration Management PUSH immediately does
> > > DUMP after (mmiklavc via mmiklavc) closes apache/metron#783
> > > METRON-1218 Metron REST should return better error messages
> > > (merrimanr) closes apache/metron#779
> > > METRON-1161 Add ability to edit parser command line options
> > in
> > > the management UI (merrimanr) closes apache/metron#737
> > > METRON-1209: Make stellar repl take logging properties, like
> > > other CLI apps in metron closes apache/incubator-metron#772
> > > METRON-1059 address checkstyle warning AvoidStarImport in
> > > metron-stellar (dbist via ottobackwards) closes apache/metron#664
> > > METRON-1204 UI does not time out after being idle, but stops
> > > functioning (merrimanr) closes apache/metron#771
> > > METRON-1052: Add forensic similarity hash functions to
> > Stellar
> > > closes apache/incubator-metron#781
> > > METRON-632: Added validation of "shew.enrichmentType" and
> > > "shew.keyColumns" closes apache/incubator-metron#732
> > > METRON-1194 Add Profiler Debug Functions to Profiler README
> > > (nickwallen via ottobackwards) closes apache/metron#765
> > > METRON-1055 Metron 0.4.0 manual installation guide for CentOS
> > > 6 updates (lvets via ottobackwards) closes apache/metron#661
> > > METRON-1079 STELLAR NaN should be a keyword (ottobackwards)
> > > closes apache/metron#681
> > > METRON-1085 Add REST endpoint to save a user profile for the
> > > Alerts UI (merrimanr) closes apache/metron#694
> > > METRON-1208 MPack for Alerts UI (merrimanr) closes
> > > apache/metron#778
> > > METRON-1207 Make RPMs for Alerts UI (merrimanr) closes
> > > apache/metron#777
> > > METRON-1215 Fix link to RPMs chapter (DimDroll via
> > justinleet)
> > > closes apache/metron#776
> > > METRON-1206 Make alerts UI conform to ops UI for install
> > > (merrimanr) closes apache/metron#773
> > > METRON-1195 Meta alerts improperly handle updates to
> > non-alert
> > > fields (justinleet) closes apache/metron#766
> > > METRON-1189 Add alert escalation to the Alerts UI (merrimanr)
> > > closes apache/metron#762
> > > METRON-1156 Simulate Triage Rules in the Stellar REPL
> > > (nickwallen) closes apache/metron#733
> > > METRON-1198 Pycapa - No such configuration property
> > > 'sasl.kerberos.principal' (nickwallen) closes apache/metron#769
> > > METRON-1202 ElasticsearchDao Has extraneous sleep call
> > > (justinleet) closes apache/metron#770
> > > METRON-938 "service metron-rest start <password>" does not
> > > work on CentOS 7. (justinleet) closes apache/metron#757
> > > METRON-1182 Refactor Code in alert list to accommodate new
> > > view types (iraghumitra via merrimanr) closes apache/metron#756
> > > METRON-1188: Ambari global configuration management
> > (mmiklavc)
> > > closes apache/metron#760
> > > METRON-1191 update public web site to point at 0.4.1 new
> > > release (mattf-horton) closes apache/metron#764
> > > METRON-1063 address javadoc warnings in metron-stellar (dbist
> > > via ottobackwards) closes apache/metron#668
> > > METRON-1190 Fix Meta Alert Type handling in calculation of
> > > scores (justinleet) closes apache/metron#763
> > > METRON-1187 Indexing/Profiler Kafka ACL Groups Not Setup
> > > Correctly (nickwallen) closes apache/metron#759
> > > METRON-1185: Stellar REPL does not work on a kerberized
> > > cluster when calling functions interacting with HBase closes
> > > apache/incubator-metron#755
> > > METRON-1186: Profiler Functions use classutils from shaded
> > > storm closes apache/incubator-metron#758
> > > METRON-1173: Fix pointers to old stellar docs closes
> > > apache/incubator-metron#746
> > > METRON-1179: Make STATS_ADD to take a list closes
> > > apache/incubator-metron#750
> > > METRON-1180: Make Stellar Shell accept zookeeper quorum as a
> > > CSV list and not require a port closes apache/incubator-metron#751
> > > METRON-1183 Improve KDC Setup Instructions (nickwallen)
> > closes
> > > apache/metron#753
> > > METRON-1177 Stale running topologies seen post-kerberization
> > > and cause exceptions (nickwallen) closes apache/metron#748
> > > METRON-1158 Build backend for grouping alerts into meta
> > alerts
> > > (justinleet) closes apache/metron#734
> > > METRON-1146: Add ability to parse JSON string into JSONObject
> > > for stellar closes apache/incubator-metron#727
> > > METRON-1176 REST: HDFS Service should support setting
> > > permissions on files when writing (ottobackwards) closes
> > apache/metron#749
> > > METRON-1114 Add group by capabilities to search REST endpoint
> > > (merrimanr) closes apache/metron#702
> > > METRON-1167 Define Session Specific Global Configuration
> > > Values in the REPL (nickwallen) closes apache/metron#740
> > > METRON-1171: Better validation for the SUBSTRING stellar
> > > function closes apache/incubator-metron#745
> > >
> > >
> > >
> > > On 11/17/17, 11:59 AM, "Nick Allen" <ni...@nickallen.org> wrote:
> > >
> > > I just wanted to send an update on where we are at. We've
> > > gotten a lot
> > > done here recently as you can see below.
> > >
> > > ✓ DONE (1) First, METRON-1289 needs to go in. This one was
> > > a fairly big
> > > effort and I am hearing that we are pretty close.
> > >
> > > ✓ DONE (2) METRON-1294 fixes an issue in how field types
> > are
> > > looked-up.
> > >
> > > ✓ DONE (3) METRON-1290 is next. While this may have been
> > > fixed in
> > > M-1289, there may be some test cases we want from this PR.
> > >
> > > ✓ DONE (4) METRON-1301 addresses a problem with the sorting
> > > logic.
> > >
> > > ✓ DONE (5) METRON-1291 fixes an issue with escalation of
> > > metaalerts.
> > >
> > > (6) That leads us to Raghu's UI work in METRON-1252. This
> > > introduces the
> > > UI bits that depend on all the previous backend work.
> > >
> > > (7) At this point, we should have our best effort at
> > running
> > > Metaalerts
> > > on Elasticsearch 2.x. I propose that we cut a release here.
> > >
> > > (8) After we cut the release, we can introduce the work for
> > > ES 5.x in
> > > METRON-939. I know we will need lots of help testing and
> > > reviewing this
> > > one.
> > >
> > >
> > >
> > > We also have an outstanding question that needs resolved
> > > BEFORE we
> > > release. We need to come to a consensus on how to release
> > > having moved our
> > > Bro Plugin to a separate repo. I don't think we've heard
> > from
> > > everyone on
> > > this. I'd urge everyone to chime in so we can choose a path
> > > forward.
> > >
> > > If anyone is totally confused in regards to that discussion,
> > I
> > > can try and
> > > send an options summary again as a separate discuss thread.
> > > The original
> > > chain was somewhere around here [1].
> > >
> > > [1]
> > > https://lists.apache.org/thread.html/
> > > 54a4474881b97e559df24728b3a0e923a58345a282451085eef832ef@%
> > > 3Cdev.metron.apache.org%3E
> > >
> > >
> > >
> > > On Wed, Nov 15, 2017 at 10:04 AM, Nick Allen <
> > > nick@nickallen.org> wrote:
> > >
> > > > Hi Guys -
> > > >
> > > > I want to follow-up on this discussion. It sounds like
> > most
> > > people are in
> > > > agreement with the general approach.
> > > >
> > > > A lot of people have been working hard on Metaalerts and
> > > Elasticsearch. I
> > > > have checked-in with those doing the heavy lifting and have
> > > compiled a more
> > > > detailed plan based on where we are at now. To the best of
> > > my knowledge
> > > > here is the plan of attack for finishing out this effort.
> > > >
> > > > (1) First, METRON-1289 needs to go in. This one was a
> > > fairly big effort
> > > > and I am hearing that we are pretty close.
> > > >
> > > > (2) METRON-1294 fixes an issue in how field types are
> > > looked-up.
> > > >
> > > > (3) METRON-1290 is next. While this may have been fixed
> > > in M-1289,
> > > > there may be some test cases we want from this PR.
> > > >
> > > > (4) METRON-1301 addresses a problem with the sorting
> > logic.
> > > >
> > > > (5) METRON-1291 fixes an issue with escalation of
> > > metaalerts.
> > > >
> > > > (6) That leads us to Raghu's UI work in METRON-1252.
> > This
> > > introduces
> > > > the UI bits that depend on all the previous backend work.
> > > >
> > > > (7) At this point, we should have our best effort at
> > > running Metaalerts
> > > > on Elasticsearch 2.x. I propose that we cut a release here.
> > > >
> > > > (8) After we cut the release, we can introduce the work
> > > for ES 5.x in
> > > > METRON-939. I know we will need lots of help testing and
> > > reviewing this
> > > > one.
> > > >
> > > > Please correct me if I am wrong. I will try and send out
> > > updates as we
> > > > make progress.
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > On Mon, Nov 6, 2017 at 1:03 PM, Zeolla@GMail.com <
> > > zeolla@gmail.com> wrote:
> > > >
> > > >> I agree, I think it's very reasonable to move in line with
> > > Nick's
> > > >> proposal. I would also suggest that we outline what the
> > > target versions
> > > >> would be to add in the METRON-777 components, since it has
> > > been functional
> > > >> for a very long time but not reviewed and has some really
> > > rockstar
> > > >> improvements.
> > > >>
> > > >> Jon
> > > >>
> > > >> On Mon, Nov 6, 2017 at 12:56 PM Otto Fowler <
> > > ottobackwards@gmail.com>
> > > >> wrote:
> > > >>
> > > >> > I think the ES cutover should be the start of the 0.5.x
> > > series, and we
> > > >> > continue on with 0.4.x for the
> > > >> > metadata improvements etc. We could chose to focus
> > > 0.5.x’s first
> > > >> releases
> > > >> > on not only ES but
> > > >> > getting a handle on kibana and the mpack situation as
> > > well.
> > > >> >
> > > >> >
> > > >> >
> > > >> >
> > > >> > On November 6, 2017 at 12:48:45, Michael Miklavcic (
> > > >> > michael.miklavcic@gmail.com) wrote:
> > > >> >
> > > >> > I agree with your proposal, Nick. I think having a
> > > stabilizing release
> > > >> > prior to upgrading ES/Kibana makes sense.
> > > >> >
> > > >> > On Mon, Nov 6, 2017 at 9:16 AM, Nick Allen <
> > > nick@nickallen.org> wrote:
> > > >> >
> > > >> > > I would like to start a discussion around upcoming
> > > releases. We have a
> > > >> > > couple separate significant tracks of work that we
> > need
> > > to reconcile
> > > >> in
> > > >> > our
> > > >> > > release schedule.
> > > >> > >
> > > >> > > (1) We have had (and have in review) a good number of
> > > bug fixes
> > > >> required
> > > >> > to
> > > >> > > support Metaalerts on the existing Elasticsearch 2.x
> > > infrastructure.
> > > >> > >
> > > >> > >
> > > >> > > (2) We also have ongoing work to upgrade our
> > > infrastructure to
> > > >> > > Elasticsearch 5.x, which will not be backwards
> > > compatible.
> > > >> > >
> > > >> > >
> > > >> > > I would like to see a release that has our best work
> > on
> > > ES 2.x before
> > > >> we
> > > >> > > migrate to 5.x. I would propose the following.
> > > >> > >
> > > >> > > Release N+1: Introduce Metaalerts running on ES 2.x
> > > >> > >
> > > >> > > Release N+2: Cut-over to ES 5.x
> > > >> > >
> > > >> > >
> > > >> > > (Q) Is it worth cutting a separate release for ES 2.x?
> > > Is there a
> > > >> better
> > > >> > > way to handle the cut-over to 5.x?
> > > >> > >
> > > >> >
> > > >> --
> > > >>
> > > >> Jon
> > > >>
> > > >
> > > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> >
> --
>
> Jon
>
>
>
>
>
>
Re: [DISCUSS] Upcoming Release
Posted by Nick Allen <ni...@nickallen.org>.
RC1 is looking good to me. I validated the MD5s, built Metron, built the
Bro plugin and reviewed the other artifacts like release notes.
Running the RAT check on a 'clean' Metron does not produce any errors for
me. It is only after building Metron, which pulls in additional Node
dependencies, does the RAT check fail.
On Sun, Dec 10, 2017 at 4:41 PM Matt Foley <ma...@apache.org> wrote:
> Yes, but let’s see if anyone else find other issues.
>
>
>
> From: Otto Fowler <ot...@gmail.com>
> Date: Saturday, December 9, 2017 at 6:16 AM
> To: Matt Foley <mf...@hortonworks.com>, "dev@metron.apache.org" <
> dev@metron.apache.org>
> Subject: Re: [DISCUSS] Upcoming Release
>
>
>
> So RC2 then?
>
>
>
> On December 8, 2017 at 20:43:21, Matt Foley (mfoley@hortonworks.com)
> wrote:
>
> Hah, here it is: https://github.com/apache/metron/pull/743
> “This problem seems to only reproduce when one unrolls a tarball rather
> than cloning from github.”
>
> Heh, the exclusion at
> https://github.com/apache/metron/blob/master/pom.xml#L351 is still there,
> but the hashcode in the bundle.css file name has changed from
> a0b6b99c10d9a13dc67e to f56deed131e58bd7ee04. Sigh. Did the version of Font
> Awesome fonts change?
>
>
> On 12/8/17, 5:26 PM, "Matt Foley" <ma...@apache.org> wrote:
>
> I remember having trouble with this bundle.css file on the last release,
> but I can’t remember what we did about it. Anybody?
>
> On 12/8/17, 1:41 PM, "Otto Fowler" <ot...@gmail.com> wrote:
>
> Steps
>
> - Downloaded tar.gz’s, asc files and KEYS
> - Verified signing of both tar.gz’s
> - searched for rouge 0.4.1 entries
> - verified the main pom.xml
> - built :
>
> mvn clean && time mvn -q -T 2C -DskipTests install && time mvn -q -T
> 2C surefire:test@unit-tests && time mvn -q
> surefire:test@integration-tests && time mvn -q test --projects
> metron-interface/metron-config && time build_utils/verify_licenses.sh
>
> Found rat error:
>
>
> *****************************************************
> Summary
> -------
> Generated at: 2017-12-08T16:33:27-05:00
>
> Notes: 3
> Binaries: 193
> Archives: 0
> Standards: 75
>
> Apache Licensed: 74
> Generated Documents: 0
>
> JavaDocs are generated, thus a license header is optional.
> Generated files do not require license headers.
>
> 1 Unknown Licenses
>
> *****************************************************
>
> Files with unapproved licenses:
>
>
> /Users/batman/tmp/release_ver/apache-metron-0.4.2-rc1/metron-interface/metron-alerts/dist/styles.f56deed131e58bd7ee04.bundle.css
>
> *****************************************************
>
>
>
>
>
> *****************************************************
> Summary
> -------
> Generated at: 2017-12-08T16:33:27-05:00
>
> Notes: 3
> Binaries: 193
> Archives: 0
> Standards: 75
>
> Apache Licensed: 74
> Generated Documents: 0
>
> JavaDocs are generated, thus a license header is optional.
> Generated files do not require license headers.
>
> 1 Unknown Licenses
>
> *****************************************************
>
> Files with unapproved licenses:
>
>
>
> /Users/ottofowler/tmp/release_ver/apache-metron-0.4.2-rc1/metron-interface/metron-alerts/dist/styles.f56deed131e58bd7ee04.bundle.css
>
> *****************************************************
>
>
>
> On December 8, 2017 at 04:34:24, Matt Foley (mattf@apache.org) wrote:
>
> Colleagues,
> I’ve posted Metron-0.4.2-RC1 and Metron-bro-plugin-kafka-0.1 to
> https://dist.apache.org/repos/dist/dev/metron/0.4.2-RC1/
>
> Given the complexity of this RC, I’d appreciate if a couple people would be
> willing to kick the tires before we put it up for a vote.
>
> I will myself be going thru the Verify Build process this weekend, as I
> won’t be able to do it Friday.
>
> Thanks,
> --Matt
>
>
> On 12/4/17, 2:05 PM, "Zeolla@GMail.com" <ze...@gmail.com> wrote:
>
> Can we resolve the conversation regarding the second repo? I was waiting
> to get more input/preferences from people There's also a documentation
> update that fixes a few broken Stellar docs that already has aa +1, I just
> need to merge it.
>
> Jon
>
> On Mon, Dec 4, 2017, 17:01 Casey Stella <ce...@gmail.com> wrote:
>
> > I would be in favor of a release at this point.
> >
> > On Mon, Dec 4, 2017 at 4:57 PM, Matt Foley <ma...@apache.org> wrote:
> >
> > > Hey all,
> > > I see METRON-1252 was resolved over the weekend. Shall I go ahead and
> > > start the process with 0.4.2 release?
> > > Does anyone have any commits they feel strongly should go in before
> 0.4.2
> > > is done, or are we ready to call it good?
> > >
> > > I believe there is consensus the 0.4.2 release should include a release
> > of
> > > the current state of the metron-bro-plugin-kafka. I will continue the
> > > discussion in that thread as to the process for accomplishing that, but
> > > plan on it happening.
> > >
> > > Regards,
> > > --Matt
> > >
> > > On 11/26/17, 6:26 PM, "Matt Foley" <ma...@apache.org> wrote:
> > >
> > > Hope everyone (at least in the U.S.) had a great Thanksgiving
> > holiday.
> > > Regarding status of the release effort, still pending METRON-1252, so
> > > not making the release branch yet.
> > >
> > > Regards,
> > > --Matt
> > >
> > > On 11/17/17, 1:32 PM, "Matt Foley" <ma...@apache.org> wrote:
> > >
> > > (With release manager hat on)
> > >
> > > The community has proposed a release of Metron in the near
> > future,
> > > focusing on Meta-alerts running in Elasticsearch.
> > > Congrats on getting so many of the below already done. At this
> > > point, only METRON-1252, and the discussion of how to handle joint
> > release
> > > of the Metron bro plugin, remain as gating items for the release. I
> > > project these will be resolved next week, so let’s propose the
> following:
> > >
> > > Sometime next week, after the last bits are done, I’ll start the
> > > release process and create the release branch.
> > >
> > > The proposed new version will be 0.4.2, unless there are backward
> > > incompatible changes that support making it 0.5.0.
> > > Currently there are NO included Jiras labeled
> > > ‘backward-incompatible’, nor having Docs Text indicating so.
> > > ***If anyone knows that some of the commits included since 0.4.1
> > > introduce backward incompatibility, please say so now on this thread,
> and
> > > mark the Jira as such.***
> > >
> > > The 90 or so jiras/commits already in master branch since 0.4.1
> > > are listed below.
> > > Thanks,
> > > --Matt
> > >
> > > METRON-1301 Alerts UI - Sorting on Triage Score Unexpectedly
> > > Filters Some Records (nickwallen) closes apache/metron#832
> > > METRON-1294 IP addresses are not formatted correctly in facet
> > > and group results (merrimanr) closes apache/metron#827
> > > METRON-1291 Kafka produce REST endpoint does not work in a
> > > Kerberized cluster (merrimanr) closes apache/metron#826
> > > METRON-1290 Only first 10 alerts are update when a MetaAlert
> > > status is changed to inactive (justinleet) closes apache/metron#842
> > > METRON-1311 Service Check Should Check Elasticsearch Index
> > > Templates (nickwallen) closes apache/metron#839
> > > METRON-1289 Alert fields are lost when a MetaAlert is created
> > > (merrimanr) closes apache/metron#824
> > > METRON-1309 Change metron-deployment to pull the plugin from
> > > apache/metron-bro-plugin-kafka (JonZeolla) closes apache/metron#837
> > > METRON-1310 Template Delete Action Deletes Search Indices
> > > (nickwallen) closes apache/metron#838
> > > METRON-1275: Fix Metron Documentation closes
> > > apache/incubator-metron#833
> > > METRON-1295 Unable to Configure Logging for REST API
> > > (nickwallen) closes apache/metron#828
> > > METRON-1307 Force install of java8 since java9 does not
> > appear
> > > to work with the scripts (brianhurley via ottobackwards) closes
> > > apache/metron#835
> > > METRON-1296 Full Dev Fails to Deploy Index Templates
> > > (nickwallen via cestella) closes apache/incubator-metron#829
> > > METRON-1281 Remove hard-coded indices from the Alerts UI
> > > (merrimanr) closes apache/metron#821
> > > METRON-1287 Full Dev Fails When Installing EPEL Repository
> > > (nickwallen) closes apache/metron#820
> > > METRON-1267 Alerts UI returns a 404 when refreshing the
> > > alerts-list page (iraghumitra via merrimanr) closes apache/metron#819
> > > METRON-1283 Install Elasticsearch template as a part of the
> > > mpack startup scripts (anandsubbu via nickwallen) closes
> > apache/metron#817
> > > METRON-1254: Conditionals as map keys do not function in
> > > Stellar closes apache/incubator-metron#801
> > > METRON-1261 Apply bro security patch (JonZeolla via
> > > ottobackwards) closes apache/metron#805
> > > METRON-1284 Remove extraneous dead query in ElasticsearchDao
> > > (justinleet) closes apache/metron#818
> > > METRON-1270: fix for warnings missing @return tag argument in
> > > metron-analytics/metron-profiler-common and metron-profiler-client
> closes
> > > apache/incubator-metron#810
> > > METRON-1272 Hide child alerts from searches and grouping if
> > > they belong to meta alerts (justinleet) closes apache/metron#811
> > > METRON-1224 Add time range selection to search control
> > > (iraghumitra via james-sirota) closes apache/metron#796
> > > METRON-1280 0.4.1 -> 0.4.2 missed a couple of projects
> > > (cestella via justinleet) closes apache/metron#816
> > > METRON-1243: Add a REST endpoint which allows us to get a
> > list
> > > of all indice closes apache/incubator-metron#797
> > > METRON-1196 Increment master version number to 0.4.2 for
> > > on-going development (mattf-horton) closes apache/metron#767
> > > METRON-1278 Strip "Build Status" widget from root
> > > README.md in site-book build (mattf-horton) closes apache/metron#815
> > > METRON-1274 Master has failure in
> > > StormControllerIntegrationTest (merrimanr) closes apache/metron#813
> > > METRON-1266 Profiler - SASL Authentication Failed
> > (nickwallen)
> > > closes apache/metron#809
> > > METRON-1260 Include Alerts UI in Ambari Service Check
> > > (nickwallen) closes apache/metron#804
> > > METRON-1251: Typo and formatting fixes for metron-rest README
> > > closes apache/incubator-metron#800
> > > METRON-1241: Enable the REST API to use a cache for the
> > > zookeeper config similar to the Bolts closes
> apache/incubator-metron#795
> > > METRON-1267 Alerts UI returns a 404 when refreshing the
> > > alerts-list page (merrimanr) closes apache/metron#808
> > > METRON-1262 Unable to add comment for a alert in a meta-alert
> > > (merrimanr) closes apache/metron#806
> > > METRON-1263 Start Alerts UI service after Metron REST
> > > (anandsubbu via nickwallen) closes apache/metron#807
> > > METRON-1255 MetaAlert search is not filtering on status
> > > (merrimanr) closes apache/metron#802
> > > METRON-1249 Improve Metron MPack Service Checks (nickwallen)
> > > closes apache/metron#799
> > > METRON-1237 address javadoc warnings in metron-maas-common
> > > (dbist via james-sirota) closes apache/metron#792
> > > METRON-1240 address javadoc warnings in metron-platform and
> > > metron-analytics (dbist via james-sirota) closes apache/metron#794
> > > METRON-1226 Searching Can Errantly Query the Wrong Indices
> > > (nickwallen) closes apache/metron#793
> > > METRON-1081 Fix Alerts and Ops UI Notices file (james-sirota)
> > > closes apache/metron#682
> > > METRON-1123 Add group by option using faceted search
> > > capabilities of metron-rest-api (iraghumitra via james-sirota) closes
> > > apache/metron#768
> > > METRON-1223 Add support to add comments for alerts
> > > (iraghumitra via james-sirota) closes apache/metron#788
> > > METRON-1083 Add filters using faceted search capabilities of
> > > metron-rest-api (iraghumitra via james-sirota) closes apache/metron#710
> > > METRON-1232 Alert status changes are not reflected in list
> > > view (iraghumitra via merrimanr) closes apache/metron#787
> > > METRON-1247 REST search and findOne endpoints return
> > > unexpected or incorrect results for guids (justinleet) closes
> > > apache/metron#798
> > > METRON-1235: Document the properties pulled from the global
> > > configuration closes apache/incubator-metron#791
> > > METRON-1234: fix for WARNING 'dependencies.dependency.(
> > > groupId:artifactId:type:classifier)' must be unique:
> > > org.apache.hadoop:hadoop-yarn-api:jar (dbist via mmiklavc) closes
> > > apache/metron#790
> > > METRON-1222: fix warning for The expression ${parent.version}
> > > is deprecated. Please use ${project.parent.version} instead. (dbist via
> > > mmiklavc) closes apache/metron#782
> > > METRON-1220 Create documentation around alert nested field
> > > (justinleet) closes apache/metron#780
> > > METRON-1229 Management UI type is part of the declarations of
> > > 2 modules (merrimanr) closes apache/metron#784
> > > METRON-1228: Configuration Management PUSH immediately does
> > > DUMP after (mmiklavc via mmiklavc) closes apache/metron#783
> > > METRON-1218 Metron REST should return better error messages
> > > (merrimanr) closes apache/metron#779
> > > METRON-1161 Add ability to edit parser command line options
> > in
> > > the management UI (merrimanr) closes apache/metron#737
> > > METRON-1209: Make stellar repl take logging properties, like
> > > other CLI apps in metron closes apache/incubator-metron#772
> > > METRON-1059 address checkstyle warning AvoidStarImport in
> > > metron-stellar (dbist via ottobackwards) closes apache/metron#664
> > > METRON-1204 UI does not time out after being idle, but stops
> > > functioning (merrimanr) closes apache/metron#771
> > > METRON-1052: Add forensic similarity hash functions to
> > Stellar
> > > closes apache/incubator-metron#781
> > > METRON-632: Added validation of "shew.enrichmentType" and
> > > "shew.keyColumns" closes apache/incubator-metron#732
> > > METRON-1194 Add Profiler Debug Functions to Profiler README
> > > (nickwallen via ottobackwards) closes apache/metron#765
> > > METRON-1055 Metron 0.4.0 manual installation guide for CentOS
> > > 6 updates (lvets via ottobackwards) closes apache/metron#661
> > > METRON-1079 STELLAR NaN should be a keyword (ottobackwards)
> > > closes apache/metron#681
> > > METRON-1085 Add REST endpoint to save a user profile for the
> > > Alerts UI (merrimanr) closes apache/metron#694
> > > METRON-1208 MPack for Alerts UI (merrimanr) closes
> > > apache/metron#778
> > > METRON-1207 Make RPMs for Alerts UI (merrimanr) closes
> > > apache/metron#777
> > > METRON-1215 Fix link to RPMs chapter (DimDroll via
> > justinleet)
> > > closes apache/metron#776
> > > METRON-1206 Make alerts UI conform to ops UI for install
> > > (merrimanr) closes apache/metron#773
> > > METRON-1195 Meta alerts improperly handle updates to
> > non-alert
> > > fields (justinleet) closes apache/metron#766
> > > METRON-1189 Add alert escalation to the Alerts UI (merrimanr)
> > > closes apache/metron#762
> > > METRON-1156 Simulate Triage Rules in the Stellar REPL
> > > (nickwallen) closes apache/metron#733
> > > METRON-1198 Pycapa - No such configuration property
> > > 'sasl.kerberos.principal' (nickwallen) closes apache/metron#769
> > > METRON-1202 ElasticsearchDao Has extraneous sleep call
> > > (justinleet) closes apache/metron#770
> > > METRON-938 "service metron-rest start <password>" does not
> > > work on CentOS 7. (justinleet) closes apache/metron#757
> > > METRON-1182 Refactor Code in alert list to accommodate new
> > > view types (iraghumitra via merrimanr) closes apache/metron#756
> > > METRON-1188: Ambari global configuration management
> > (mmiklavc)
> > > closes apache/metron#760
> > > METRON-1191 update public web site to point at 0.4.1 new
> > > release (mattf-horton) closes apache/metron#764
> > > METRON-1063 address javadoc warnings in metron-stellar (dbist
> > > via ottobackwards) closes apache/metron#668
> > > METRON-1190 Fix Meta Alert Type handling in calculation of
> > > scores (justinleet) closes apache/metron#763
> > > METRON-1187 Indexing/Profiler Kafka ACL Groups Not Setup
> > > Correctly (nickwallen) closes apache/metron#759
> > > METRON-1185: Stellar REPL does not work on a kerberized
> > > cluster when calling functions interacting with HBase closes
> > > apache/incubator-metron#755
> > > METRON-1186: Profiler Functions use classutils from shaded
> > > storm closes apache/incubator-metron#758
> > > METRON-1173: Fix pointers to old stellar docs closes
> > > apache/incubator-metron#746
> > > METRON-1179: Make STATS_ADD to take a list closes
> > > apache/incubator-metron#750
> > > METRON-1180: Make Stellar Shell accept zookeeper quorum as a
> > > CSV list and not require a port closes apache/incubator-metron#751
> > > METRON-1183 Improve KDC Setup Instructions (nickwallen)
> > closes
> > > apache/metron#753
> > > METRON-1177 Stale running topologies seen post-kerberization
> > > and cause exceptions (nickwallen) closes apache/metron#748
> > > METRON-1158 Build backend for grouping alerts into meta
> > alerts
> > > (justinleet) closes apache/metron#734
> > > METRON-1146: Add ability to parse JSON string into JSONObject
> > > for stellar closes apache/incubator-metron#727
> > > METRON-1176 REST: HDFS Service should support setting
> > > permissions on files when writing (ottobackwards) closes
> > apache/metron#749
> > > METRON-1114 Add group by capabilities to search REST endpoint
> > > (merrimanr) closes apache/metron#702
> > > METRON-1167 Define Session Specific Global Configuration
> > > Values in the REPL (nickwallen) closes apache/metron#740
> > > METRON-1171: Better validation for the SUBSTRING stellar
> > > function closes apache/incubator-metron#745
> > >
> > >
> > >
> > > On 11/17/17, 11:59 AM, "Nick Allen" <ni...@nickallen.org> wrote:
> > >
> > > I just wanted to send an update on where we are at. We've
> > > gotten a lot
> > > done here recently as you can see below.
> > >
> > > ✓ DONE (1) First, METRON-1289 needs to go in. This one was
> > > a fairly big
> > > effort and I am hearing that we are pretty close.
> > >
> > > ✓ DONE (2) METRON-1294 fixes an issue in how field types
> > are
> > > looked-up.
> > >
> > > ✓ DONE (3) METRON-1290 is next. While this may have been
> > > fixed in
> > > M-1289, there may be some test cases we want from this PR.
> > >
> > > ✓ DONE (4) METRON-1301 addresses a problem with the sorting
> > > logic.
> > >
> > > ✓ DONE (5) METRON-1291 fixes an issue with escalation of
> > > metaalerts.
> > >
> > > (6) That leads us to Raghu's UI work in METRON-1252. This
> > > introduces the
> > > UI bits that depend on all the previous backend work.
> > >
> > > (7) At this point, we should have our best effort at
> > running
> > > Metaalerts
> > > on Elasticsearch 2.x. I propose that we cut a release here.
> > >
> > > (8) After we cut the release, we can introduce the work for
> > > ES 5.x in
> > > METRON-939. I know we will need lots of help testing and
> > > reviewing this
> > > one.
> > >
> > >
> > >
> > > We also have an outstanding question that needs resolved
> > > BEFORE we
> > > release. We need to come to a consensus on how to release
> > > having moved our
> > > Bro Plugin to a separate repo. I don't think we've heard
> > from
> > > everyone on
> > > this. I'd urge everyone to chime in so we can choose a path
> > > forward.
> > >
> > > If anyone is totally confused in regards to that discussion,
> > I
> > > can try and
> > > send an options summary again as a separate discuss thread.
> > > The original
> > > chain was somewhere around here [1].
> > >
> > > [1]
> > > https://lists.apache.org/thread.html/
> > > 54a4474881b97e559df24728b3a0e923a58345a282451085eef832ef@%
> > > 3Cdev.metron.apache.org%3E
> > >
> > >
> > >
> > > On Wed, Nov 15, 2017 at 10:04 AM, Nick Allen <
> > > nick@nickallen.org> wrote:
> > >
> > > > Hi Guys -
> > > >
> > > > I want to follow-up on this discussion. It sounds like
> > most
> > > people are in
> > > > agreement with the general approach.
> > > >
> > > > A lot of people have been working hard on Metaalerts and
> > > Elasticsearch. I
> > > > have checked-in with those doing the heavy lifting and have
> > > compiled a more
> > > > detailed plan based on where we are at now. To the best of
> > > my knowledge
> > > > here is the plan of attack for finishing out this effort.
> > > >
> > > > (1) First, METRON-1289 needs to go in. This one was a
> > > fairly big effort
> > > > and I am hearing that we are pretty close.
> > > >
> > > > (2) METRON-1294 fixes an issue in how field types are
> > > looked-up.
> > > >
> > > > (3) METRON-1290 is next. While this may have been fixed
> > > in M-1289,
> > > > there may be some test cases we want from this PR.
> > > >
> > > > (4) METRON-1301 addresses a problem with the sorting
> > logic.
> > > >
> > > > (5) METRON-1291 fixes an issue with escalation of
> > > metaalerts.
> > > >
> > > > (6) That leads us to Raghu's UI work in METRON-1252.
> > This
> > > introduces
> > > > the UI bits that depend on all the previous backend work.
> > > >
> > > > (7) At this point, we should have our best effort at
> > > running Metaalerts
> > > > on Elasticsearch 2.x. I propose that we cut a release here.
> > > >
> > > > (8) After we cut the release, we can introduce the work
> > > for ES 5.x in
> > > > METRON-939. I know we will need lots of help testing and
> > > reviewing this
> > > > one.
> > > >
> > > > Please correct me if I am wrong. I will try and send out
> > > updates as we
> > > > make progress.
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > On Mon, Nov 6, 2017 at 1:03 PM, Zeolla@GMail.com <
> > > zeolla@gmail.com> wrote:
> > > >
> > > >> I agree, I think it's very reasonable to move in line with
> > > Nick's
> > > >> proposal. I would also suggest that we outline what the
> > > target versions
> > > >> would be to add in the METRON-777 components, since it has
> > > been functional
> > > >> for a very long time but not reviewed and has some really
> > > rockstar
> > > >> improvements.
> > > >>
> > > >> Jon
> > > >>
> > > >> On Mon, Nov 6, 2017 at 12:56 PM Otto Fowler <
> > > ottobackwards@gmail.com>
> > > >> wrote:
> > > >>
> > > >> > I think the ES cutover should be the start of the 0.5.x
> > > series, and we
> > > >> > continue on with 0.4.x for the
> > > >> > metadata improvements etc. We could chose to focus
> > > 0.5.x’s first
> > > >> releases
> > > >> > on not only ES but
> > > >> > getting a handle on kibana and the mpack situation as
> > > well.
> > > >> >
> > > >> >
> > > >> >
> > > >> >
> > > >> > On November 6, 2017 at 12:48:45, Michael Miklavcic (
> > > >> > michael.miklavcic@gmail.com) wrote:
> > > >> >
> > > >> > I agree with your proposal, Nick. I think having a
> > > stabilizing release
> > > >> > prior to upgrading ES/Kibana makes sense.
> > > >> >
> > > >> > On Mon, Nov 6, 2017 at 9:16 AM, Nick Allen <
> > > nick@nickallen.org> wrote:
> > > >> >
> > > >> > > I would like to start a discussion around upcoming
> > > releases. We have a
> > > >> > > couple separate significant tracks of work that we
> > need
> > > to reconcile
> > > >> in
> > > >> > our
> > > >> > > release schedule.
> > > >> > >
> > > >> > > (1) We have had (and have in review) a good number of
> > > bug fixes
> > > >> required
> > > >> > to
> > > >> > > support Metaalerts on the existing Elasticsearch 2.x
> > > infrastructure.
> > > >> > >
> > > >> > >
> > > >> > > (2) We also have ongoing work to upgrade our
> > > infrastructure to
> > > >> > > Elasticsearch 5.x, which will not be backwards
> > > compatible.
> > > >> > >
> > > >> > >
> > > >> > > I would like to see a release that has our best work
> > on
> > > ES 2.x before
> > > >> we
> > > >> > > migrate to 5.x. I would propose the following.
> > > >> > >
> > > >> > > Release N+1: Introduce Metaalerts running on ES 2.x
> > > >> > >
> > > >> > > Release N+2: Cut-over to ES 5.x
> > > >> > >
> > > >> > >
> > > >> > > (Q) Is it worth cutting a separate release for ES 2.x?
> > > Is there a
> > > >> better
> > > >> > > way to handle the cut-over to 5.x?
> > > >> > >
> > > >> >
> > > >> --
> > > >>
> > > >> Jon
> > > >>
> > > >
> > > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> >
> --
>
> Jon
>
>
>
>
>
>
Re: [DISCUSS] Upcoming Release
Posted by Matt Foley <ma...@apache.org>.
Yes, but let’s see if anyone else find other issues.
From: Otto Fowler <ot...@gmail.com>
Date: Saturday, December 9, 2017 at 6:16 AM
To: Matt Foley <mf...@hortonworks.com>, "dev@metron.apache.org" <de...@metron.apache.org>
Subject: Re: [DISCUSS] Upcoming Release
So RC2 then?
On December 8, 2017 at 20:43:21, Matt Foley (mfoley@hortonworks.com) wrote:
Hah, here it is: https://github.com/apache/metron/pull/743
“This problem seems to only reproduce when one unrolls a tarball rather than cloning from github.”
Heh, the exclusion at https://github.com/apache/metron/blob/master/pom.xml#L351 is still there, but the hashcode in the bundle.css file name has changed from a0b6b99c10d9a13dc67e to f56deed131e58bd7ee04. Sigh. Did the version of Font Awesome fonts change?
On 12/8/17, 5:26 PM, "Matt Foley" <ma...@apache.org> wrote:
I remember having trouble with this bundle.css file on the last release, but I can’t remember what we did about it. Anybody?
On 12/8/17, 1:41 PM, "Otto Fowler" <ot...@gmail.com> wrote:
Steps
- Downloaded tar.gz’s, asc files and KEYS
- Verified signing of both tar.gz’s
- searched for rouge 0.4.1 entries
- verified the main pom.xml
- built :
mvn clean && time mvn -q -T 2C -DskipTests install && time mvn -q -T
2C surefire:test@unit-tests && time mvn -q
surefire:test@integration-tests && time mvn -q test --projects
metron-interface/metron-config && time build_utils/verify_licenses.sh
Found rat error:
*****************************************************
Summary
-------
Generated at: 2017-12-08T16:33:27-05:00
Notes: 3
Binaries: 193
Archives: 0
Standards: 75
Apache Licensed: 74
Generated Documents: 0
JavaDocs are generated, thus a license header is optional.
Generated files do not require license headers.
1 Unknown Licenses
*****************************************************
Files with unapproved licenses:
/Users/batman/tmp/release_ver/apache-metron-0.4.2-rc1/metron-interface/metron-alerts/dist/styles.f56deed131e58bd7ee04.bundle.css
*****************************************************
*****************************************************
Summary
-------
Generated at: 2017-12-08T16:33:27-05:00
Notes: 3
Binaries: 193
Archives: 0
Standards: 75
Apache Licensed: 74
Generated Documents: 0
JavaDocs are generated, thus a license header is optional.
Generated files do not require license headers.
1 Unknown Licenses
*****************************************************
Files with unapproved licenses:
/Users/ottofowler/tmp/release_ver/apache-metron-0.4.2-rc1/metron-interface/metron-alerts/dist/styles.f56deed131e58bd7ee04.bundle.css
*****************************************************
On December 8, 2017 at 04:34:24, Matt Foley (mattf@apache.org) wrote:
Colleagues,
I’ve posted Metron-0.4.2-RC1 and Metron-bro-plugin-kafka-0.1 to
https://dist.apache.org/repos/dist/dev/metron/0.4.2-RC1/
Given the complexity of this RC, I’d appreciate if a couple people would be
willing to kick the tires before we put it up for a vote.
I will myself be going thru the Verify Build process this weekend, as I
won’t be able to do it Friday.
Thanks,
--Matt
On 12/4/17, 2:05 PM, "Zeolla@GMail.com" <ze...@gmail.com> wrote:
Can we resolve the conversation regarding the second repo? I was waiting
to get more input/preferences from people There's also a documentation
update that fixes a few broken Stellar docs that already has aa +1, I just
need to merge it.
Jon
On Mon, Dec 4, 2017, 17:01 Casey Stella <ce...@gmail.com> wrote:
> I would be in favor of a release at this point.
>
> On Mon, Dec 4, 2017 at 4:57 PM, Matt Foley <ma...@apache.org> wrote:
>
> > Hey all,
> > I see METRON-1252 was resolved over the weekend. Shall I go ahead and
> > start the process with 0.4.2 release?
> > Does anyone have any commits they feel strongly should go in before
0.4.2
> > is done, or are we ready to call it good?
> >
> > I believe there is consensus the 0.4.2 release should include a release
> of
> > the current state of the metron-bro-plugin-kafka. I will continue the
> > discussion in that thread as to the process for accomplishing that, but
> > plan on it happening.
> >
> > Regards,
> > --Matt
> >
> > On 11/26/17, 6:26 PM, "Matt Foley" <ma...@apache.org> wrote:
> >
> > Hope everyone (at least in the U.S.) had a great Thanksgiving
> holiday.
> > Regarding status of the release effort, still pending METRON-1252, so
> > not making the release branch yet.
> >
> > Regards,
> > --Matt
> >
> > On 11/17/17, 1:32 PM, "Matt Foley" <ma...@apache.org> wrote:
> >
> > (With release manager hat on)
> >
> > The community has proposed a release of Metron in the near
> future,
> > focusing on Meta-alerts running in Elasticsearch.
> > Congrats on getting so many of the below already done. At this
> > point, only METRON-1252, and the discussion of how to handle joint
> release
> > of the Metron bro plugin, remain as gating items for the release. I
> > project these will be resolved next week, so let’s propose the
following:
> >
> > Sometime next week, after the last bits are done, I’ll start the
> > release process and create the release branch.
> >
> > The proposed new version will be 0.4.2, unless there are backward
> > incompatible changes that support making it 0.5.0.
> > Currently there are NO included Jiras labeled
> > ‘backward-incompatible’, nor having Docs Text indicating so.
> > ***If anyone knows that some of the commits included since 0.4.1
> > introduce backward incompatibility, please say so now on this thread,
and
> > mark the Jira as such.***
> >
> > The 90 or so jiras/commits already in master branch since 0.4.1
> > are listed below.
> > Thanks,
> > --Matt
> >
> > METRON-1301 Alerts UI - Sorting on Triage Score Unexpectedly
> > Filters Some Records (nickwallen) closes apache/metron#832
> > METRON-1294 IP addresses are not formatted correctly in facet
> > and group results (merrimanr) closes apache/metron#827
> > METRON-1291 Kafka produce REST endpoint does not work in a
> > Kerberized cluster (merrimanr) closes apache/metron#826
> > METRON-1290 Only first 10 alerts are update when a MetaAlert
> > status is changed to inactive (justinleet) closes apache/metron#842
> > METRON-1311 Service Check Should Check Elasticsearch Index
> > Templates (nickwallen) closes apache/metron#839
> > METRON-1289 Alert fields are lost when a MetaAlert is created
> > (merrimanr) closes apache/metron#824
> > METRON-1309 Change metron-deployment to pull the plugin from
> > apache/metron-bro-plugin-kafka (JonZeolla) closes apache/metron#837
> > METRON-1310 Template Delete Action Deletes Search Indices
> > (nickwallen) closes apache/metron#838
> > METRON-1275: Fix Metron Documentation closes
> > apache/incubator-metron#833
> > METRON-1295 Unable to Configure Logging for REST API
> > (nickwallen) closes apache/metron#828
> > METRON-1307 Force install of java8 since java9 does not
> appear
> > to work with the scripts (brianhurley via ottobackwards) closes
> > apache/metron#835
> > METRON-1296 Full Dev Fails to Deploy Index Templates
> > (nickwallen via cestella) closes apache/incubator-metron#829
> > METRON-1281 Remove hard-coded indices from the Alerts UI
> > (merrimanr) closes apache/metron#821
> > METRON-1287 Full Dev Fails When Installing EPEL Repository
> > (nickwallen) closes apache/metron#820
> > METRON-1267 Alerts UI returns a 404 when refreshing the
> > alerts-list page (iraghumitra via merrimanr) closes apache/metron#819
> > METRON-1283 Install Elasticsearch template as a part of the
> > mpack startup scripts (anandsubbu via nickwallen) closes
> apache/metron#817
> > METRON-1254: Conditionals as map keys do not function in
> > Stellar closes apache/incubator-metron#801
> > METRON-1261 Apply bro security patch (JonZeolla via
> > ottobackwards) closes apache/metron#805
> > METRON-1284 Remove extraneous dead query in ElasticsearchDao
> > (justinleet) closes apache/metron#818
> > METRON-1270: fix for warnings missing @return tag argument in
> > metron-analytics/metron-profiler-common and metron-profiler-client
closes
> > apache/incubator-metron#810
> > METRON-1272 Hide child alerts from searches and grouping if
> > they belong to meta alerts (justinleet) closes apache/metron#811
> > METRON-1224 Add time range selection to search control
> > (iraghumitra via james-sirota) closes apache/metron#796
> > METRON-1280 0.4.1 -> 0.4.2 missed a couple of projects
> > (cestella via justinleet) closes apache/metron#816
> > METRON-1243: Add a REST endpoint which allows us to get a
> list
> > of all indice closes apache/incubator-metron#797
> > METRON-1196 Increment master version number to 0.4.2 for
> > on-going development (mattf-horton) closes apache/metron#767
> > METRON-1278 Strip "Build Status" widget from root
> > README.md in site-book build (mattf-horton) closes apache/metron#815
> > METRON-1274 Master has failure in
> > StormControllerIntegrationTest (merrimanr) closes apache/metron#813
> > METRON-1266 Profiler - SASL Authentication Failed
> (nickwallen)
> > closes apache/metron#809
> > METRON-1260 Include Alerts UI in Ambari Service Check
> > (nickwallen) closes apache/metron#804
> > METRON-1251: Typo and formatting fixes for metron-rest README
> > closes apache/incubator-metron#800
> > METRON-1241: Enable the REST API to use a cache for the
> > zookeeper config similar to the Bolts closes
apache/incubator-metron#795
> > METRON-1267 Alerts UI returns a 404 when refreshing the
> > alerts-list page (merrimanr) closes apache/metron#808
> > METRON-1262 Unable to add comment for a alert in a meta-alert
> > (merrimanr) closes apache/metron#806
> > METRON-1263 Start Alerts UI service after Metron REST
> > (anandsubbu via nickwallen) closes apache/metron#807
> > METRON-1255 MetaAlert search is not filtering on status
> > (merrimanr) closes apache/metron#802
> > METRON-1249 Improve Metron MPack Service Checks (nickwallen)
> > closes apache/metron#799
> > METRON-1237 address javadoc warnings in metron-maas-common
> > (dbist via james-sirota) closes apache/metron#792
> > METRON-1240 address javadoc warnings in metron-platform and
> > metron-analytics (dbist via james-sirota) closes apache/metron#794
> > METRON-1226 Searching Can Errantly Query the Wrong Indices
> > (nickwallen) closes apache/metron#793
> > METRON-1081 Fix Alerts and Ops UI Notices file (james-sirota)
> > closes apache/metron#682
> > METRON-1123 Add group by option using faceted search
> > capabilities of metron-rest-api (iraghumitra via james-sirota) closes
> > apache/metron#768
> > METRON-1223 Add support to add comments for alerts
> > (iraghumitra via james-sirota) closes apache/metron#788
> > METRON-1083 Add filters using faceted search capabilities of
> > metron-rest-api (iraghumitra via james-sirota) closes apache/metron#710
> > METRON-1232 Alert status changes are not reflected in list
> > view (iraghumitra via merrimanr) closes apache/metron#787
> > METRON-1247 REST search and findOne endpoints return
> > unexpected or incorrect results for guids (justinleet) closes
> > apache/metron#798
> > METRON-1235: Document the properties pulled from the global
> > configuration closes apache/incubator-metron#791
> > METRON-1234: fix for WARNING 'dependencies.dependency.(
> > groupId:artifactId:type:classifier)' must be unique:
> > org.apache.hadoop:hadoop-yarn-api:jar (dbist via mmiklavc) closes
> > apache/metron#790
> > METRON-1222: fix warning for The expression ${parent.version}
> > is deprecated. Please use ${project.parent.version} instead. (dbist via
> > mmiklavc) closes apache/metron#782
> > METRON-1220 Create documentation around alert nested field
> > (justinleet) closes apache/metron#780
> > METRON-1229 Management UI type is part of the declarations of
> > 2 modules (merrimanr) closes apache/metron#784
> > METRON-1228: Configuration Management PUSH immediately does
> > DUMP after (mmiklavc via mmiklavc) closes apache/metron#783
> > METRON-1218 Metron REST should return better error messages
> > (merrimanr) closes apache/metron#779
> > METRON-1161 Add ability to edit parser command line options
> in
> > the management UI (merrimanr) closes apache/metron#737
> > METRON-1209: Make stellar repl take logging properties, like
> > other CLI apps in metron closes apache/incubator-metron#772
> > METRON-1059 address checkstyle warning AvoidStarImport in
> > metron-stellar (dbist via ottobackwards) closes apache/metron#664
> > METRON-1204 UI does not time out after being idle, but stops
> > functioning (merrimanr) closes apache/metron#771
> > METRON-1052: Add forensic similarity hash functions to
> Stellar
> > closes apache/incubator-metron#781
> > METRON-632: Added validation of "shew.enrichmentType" and
> > "shew.keyColumns" closes apache/incubator-metron#732
> > METRON-1194 Add Profiler Debug Functions to Profiler README
> > (nickwallen via ottobackwards) closes apache/metron#765
> > METRON-1055 Metron 0.4.0 manual installation guide for CentOS
> > 6 updates (lvets via ottobackwards) closes apache/metron#661
> > METRON-1079 STELLAR NaN should be a keyword (ottobackwards)
> > closes apache/metron#681
> > METRON-1085 Add REST endpoint to save a user profile for the
> > Alerts UI (merrimanr) closes apache/metron#694
> > METRON-1208 MPack for Alerts UI (merrimanr) closes
> > apache/metron#778
> > METRON-1207 Make RPMs for Alerts UI (merrimanr) closes
> > apache/metron#777
> > METRON-1215 Fix link to RPMs chapter (DimDroll via
> justinleet)
> > closes apache/metron#776
> > METRON-1206 Make alerts UI conform to ops UI for install
> > (merrimanr) closes apache/metron#773
> > METRON-1195 Meta alerts improperly handle updates to
> non-alert
> > fields (justinleet) closes apache/metron#766
> > METRON-1189 Add alert escalation to the Alerts UI (merrimanr)
> > closes apache/metron#762
> > METRON-1156 Simulate Triage Rules in the Stellar REPL
> > (nickwallen) closes apache/metron#733
> > METRON-1198 Pycapa - No such configuration property
> > 'sasl.kerberos.principal' (nickwallen) closes apache/metron#769
> > METRON-1202 ElasticsearchDao Has extraneous sleep call
> > (justinleet) closes apache/metron#770
> > METRON-938 "service metron-rest start <password>" does not
> > work on CentOS 7. (justinleet) closes apache/metron#757
> > METRON-1182 Refactor Code in alert list to accommodate new
> > view types (iraghumitra via merrimanr) closes apache/metron#756
> > METRON-1188: Ambari global configuration management
> (mmiklavc)
> > closes apache/metron#760
> > METRON-1191 update public web site to point at 0.4.1 new
> > release (mattf-horton) closes apache/metron#764
> > METRON-1063 address javadoc warnings in metron-stellar (dbist
> > via ottobackwards) closes apache/metron#668
> > METRON-1190 Fix Meta Alert Type handling in calculation of
> > scores (justinleet) closes apache/metron#763
> > METRON-1187 Indexing/Profiler Kafka ACL Groups Not Setup
> > Correctly (nickwallen) closes apache/metron#759
> > METRON-1185: Stellar REPL does not work on a kerberized
> > cluster when calling functions interacting with HBase closes
> > apache/incubator-metron#755
> > METRON-1186: Profiler Functions use classutils from shaded
> > storm closes apache/incubator-metron#758
> > METRON-1173: Fix pointers to old stellar docs closes
> > apache/incubator-metron#746
> > METRON-1179: Make STATS_ADD to take a list closes
> > apache/incubator-metron#750
> > METRON-1180: Make Stellar Shell accept zookeeper quorum as a
> > CSV list and not require a port closes apache/incubator-metron#751
> > METRON-1183 Improve KDC Setup Instructions (nickwallen)
> closes
> > apache/metron#753
> > METRON-1177 Stale running topologies seen post-kerberization
> > and cause exceptions (nickwallen) closes apache/metron#748
> > METRON-1158 Build backend for grouping alerts into meta
> alerts
> > (justinleet) closes apache/metron#734
> > METRON-1146: Add ability to parse JSON string into JSONObject
> > for stellar closes apache/incubator-metron#727
> > METRON-1176 REST: HDFS Service should support setting
> > permissions on files when writing (ottobackwards) closes
> apache/metron#749
> > METRON-1114 Add group by capabilities to search REST endpoint
> > (merrimanr) closes apache/metron#702
> > METRON-1167 Define Session Specific Global Configuration
> > Values in the REPL (nickwallen) closes apache/metron#740
> > METRON-1171: Better validation for the SUBSTRING stellar
> > function closes apache/incubator-metron#745
> >
> >
> >
> > On 11/17/17, 11:59 AM, "Nick Allen" <ni...@nickallen.org> wrote:
> >
> > I just wanted to send an update on where we are at. We've
> > gotten a lot
> > done here recently as you can see below.
> >
> > ✓ DONE (1) First, METRON-1289 needs to go in. This one was
> > a fairly big
> > effort and I am hearing that we are pretty close.
> >
> > ✓ DONE (2) METRON-1294 fixes an issue in how field types
> are
> > looked-up.
> >
> > ✓ DONE (3) METRON-1290 is next. While this may have been
> > fixed in
> > M-1289, there may be some test cases we want from this PR.
> >
> > ✓ DONE (4) METRON-1301 addresses a problem with the sorting
> > logic.
> >
> > ✓ DONE (5) METRON-1291 fixes an issue with escalation of
> > metaalerts.
> >
> > (6) That leads us to Raghu's UI work in METRON-1252. This
> > introduces the
> > UI bits that depend on all the previous backend work.
> >
> > (7) At this point, we should have our best effort at
> running
> > Metaalerts
> > on Elasticsearch 2.x. I propose that we cut a release here.
> >
> > (8) After we cut the release, we can introduce the work for
> > ES 5.x in
> > METRON-939. I know we will need lots of help testing and
> > reviewing this
> > one.
> >
> >
> >
> > We also have an outstanding question that needs resolved
> > BEFORE we
> > release. We need to come to a consensus on how to release
> > having moved our
> > Bro Plugin to a separate repo. I don't think we've heard
> from
> > everyone on
> > this. I'd urge everyone to chime in so we can choose a path
> > forward.
> >
> > If anyone is totally confused in regards to that discussion,
> I
> > can try and
> > send an options summary again as a separate discuss thread.
> > The original
> > chain was somewhere around here [1].
> >
> > [1]
> > https://lists.apache.org/thread.html/
> > 54a4474881b97e559df24728b3a0e923a58345a282451085eef832ef@%
> > 3Cdev.metron.apache.org%3E
> >
> >
> >
> > On Wed, Nov 15, 2017 at 10:04 AM, Nick Allen <
> > nick@nickallen.org> wrote:
> >
> > > Hi Guys -
> > >
> > > I want to follow-up on this discussion. It sounds like
> most
> > people are in
> > > agreement with the general approach.
> > >
> > > A lot of people have been working hard on Metaalerts and
> > Elasticsearch. I
> > > have checked-in with those doing the heavy lifting and have
> > compiled a more
> > > detailed plan based on where we are at now. To the best of
> > my knowledge
> > > here is the plan of attack for finishing out this effort.
> > >
> > > (1) First, METRON-1289 needs to go in. This one was a
> > fairly big effort
> > > and I am hearing that we are pretty close.
> > >
> > > (2) METRON-1294 fixes an issue in how field types are
> > looked-up.
> > >
> > > (3) METRON-1290 is next. While this may have been fixed
> > in M-1289,
> > > there may be some test cases we want from this PR.
> > >
> > > (4) METRON-1301 addresses a problem with the sorting
> logic.
> > >
> > > (5) METRON-1291 fixes an issue with escalation of
> > metaalerts.
> > >
> > > (6) That leads us to Raghu's UI work in METRON-1252.
> This
> > introduces
> > > the UI bits that depend on all the previous backend work.
> > >
> > > (7) At this point, we should have our best effort at
> > running Metaalerts
> > > on Elasticsearch 2.x. I propose that we cut a release here.
> > >
> > > (8) After we cut the release, we can introduce the work
> > for ES 5.x in
> > > METRON-939. I know we will need lots of help testing and
> > reviewing this
> > > one.
> > >
> > > Please correct me if I am wrong. I will try and send out
> > updates as we
> > > make progress.
> > >
> > >
> > >
> > >
> > >
> > > On Mon, Nov 6, 2017 at 1:03 PM, Zeolla@GMail.com <
> > zeolla@gmail.com> wrote:
> > >
> > >> I agree, I think it's very reasonable to move in line with
> > Nick's
> > >> proposal. I would also suggest that we outline what the
> > target versions
> > >> would be to add in the METRON-777 components, since it has
> > been functional
> > >> for a very long time but not reviewed and has some really
> > rockstar
> > >> improvements.
> > >>
> > >> Jon
> > >>
> > >> On Mon, Nov 6, 2017 at 12:56 PM Otto Fowler <
> > ottobackwards@gmail.com>
> > >> wrote:
> > >>
> > >> > I think the ES cutover should be the start of the 0.5.x
> > series, and we
> > >> > continue on with 0.4.x for the
> > >> > metadata improvements etc. We could chose to focus
> > 0.5.x’s first
> > >> releases
> > >> > on not only ES but
> > >> > getting a handle on kibana and the mpack situation as
> > well.
> > >> >
> > >> >
> > >> >
> > >> >
> > >> > On November 6, 2017 at 12:48:45, Michael Miklavcic (
> > >> > michael.miklavcic@gmail.com) wrote:
> > >> >
> > >> > I agree with your proposal, Nick. I think having a
> > stabilizing release
> > >> > prior to upgrading ES/Kibana makes sense.
> > >> >
> > >> > On Mon, Nov 6, 2017 at 9:16 AM, Nick Allen <
> > nick@nickallen.org> wrote:
> > >> >
> > >> > > I would like to start a discussion around upcoming
> > releases. We have a
> > >> > > couple separate significant tracks of work that we
> need
> > to reconcile
> > >> in
> > >> > our
> > >> > > release schedule.
> > >> > >
> > >> > > (1) We have had (and have in review) a good number of
> > bug fixes
> > >> required
> > >> > to
> > >> > > support Metaalerts on the existing Elasticsearch 2.x
> > infrastructure.
> > >> > >
> > >> > >
> > >> > > (2) We also have ongoing work to upgrade our
> > infrastructure to
> > >> > > Elasticsearch 5.x, which will not be backwards
> > compatible.
> > >> > >
> > >> > >
> > >> > > I would like to see a release that has our best work
> on
> > ES 2.x before
> > >> we
> > >> > > migrate to 5.x. I would propose the following.
> > >> > >
> > >> > > Release N+1: Introduce Metaalerts running on ES 2.x
> > >> > >
> > >> > > Release N+2: Cut-over to ES 5.x
> > >> > >
> > >> > >
> > >> > > (Q) Is it worth cutting a separate release for ES 2.x?
> > Is there a
> > >> better
> > >> > > way to handle the cut-over to 5.x?
> > >> > >
> > >> >
> > >> --
> > >>
> > >> Jon
> > >>
> > >
> > >
> >
> >
> >
> >
> >
> >
> >
> >
>
--
Jon
Re: [DISCUSS] Upcoming Release
Posted by Otto Fowler <ot...@gmail.com>.
So RC2 then?
On December 8, 2017 at 20:43:21, Matt Foley (mfoley@hortonworks.com) wrote:
Hah, here it is: https://github.com/apache/metron/pull/743
“This problem seems to only reproduce when one unrolls a tarball rather
than cloning from github.”
Heh, the exclusion at
https://github.com/apache/metron/blob/master/pom.xml#L351 is still there,
but the hashcode in the bundle.css file name has changed from
a0b6b99c10d9a13dc67e to f56deed131e58bd7ee04. Sigh. Did the version of Font
Awesome fonts change?
On 12/8/17, 5:26 PM, "Matt Foley" <ma...@apache.org> wrote:
I remember having trouble with this bundle.css file on the last release,
but I can’t remember what we did about it. Anybody?
On 12/8/17, 1:41 PM, "Otto Fowler" <ot...@gmail.com> wrote:
Steps
- Downloaded tar.gz’s, asc files and KEYS
- Verified signing of both tar.gz’s
- searched for rouge 0.4.1 entries
- verified the main pom.xml
- built :
mvn clean && time mvn -q -T 2C -DskipTests install && time mvn -q -T
2C surefire:test@unit-tests && time mvn -q
surefire:test@integration-tests && time mvn -q test --projects
metron-interface/metron-config && time build_utils/verify_licenses.sh
Found rat error:
*****************************************************
Summary
-------
Generated at: 2017-12-08T16:33:27-05:00
Notes: 3
Binaries: 193
Archives: 0
Standards: 75
Apache Licensed: 74
Generated Documents: 0
JavaDocs are generated, thus a license header is optional.
Generated files do not require license headers.
1 Unknown Licenses
*****************************************************
Files with unapproved licenses:
/Users/batman/tmp/release_ver/apache-metron-0.4.2-rc1/metron-interface/metron-alerts/dist/styles.f56deed131e58bd7ee04.bundle.css
*****************************************************
*****************************************************
Summary
-------
Generated at: 2017-12-08T16:33:27-05:00
Notes: 3
Binaries: 193
Archives: 0
Standards: 75
Apache Licensed: 74
Generated Documents: 0
JavaDocs are generated, thus a license header is optional.
Generated files do not require license headers.
1 Unknown Licenses
*****************************************************
Files with unapproved licenses:
/Users/ottofowler/tmp/release_ver/apache-metron-0.4.2-rc1/metron-interface/metron-alerts/dist/styles.f56deed131e58bd7ee04.bundle.css
*****************************************************
On December 8, 2017 at 04:34:24, Matt Foley (mattf@apache.org) wrote:
Colleagues,
I’ve posted Metron-0.4.2-RC1 and Metron-bro-plugin-kafka-0.1 to
https://dist.apache.org/repos/dist/dev/metron/0.4.2-RC1/
Given the complexity of this RC, I’d appreciate if a couple people would be
willing to kick the tires before we put it up for a vote.
I will myself be going thru the Verify Build process this weekend, as I
won’t be able to do it Friday.
Thanks,
--Matt
On 12/4/17, 2:05 PM, "Zeolla@GMail.com" <ze...@gmail.com> wrote:
Can we resolve the conversation regarding the second repo? I was waiting
to get more input/preferences from people There's also a documentation
update that fixes a few broken Stellar docs that already has aa +1, I just
need to merge it.
Jon
On Mon, Dec 4, 2017, 17:01 Casey Stella <ce...@gmail.com> wrote:
> I would be in favor of a release at this point.
>
> On Mon, Dec 4, 2017 at 4:57 PM, Matt Foley <ma...@apache.org> wrote:
>
> > Hey all,
> > I see METRON-1252 was resolved over the weekend. Shall I go ahead and
> > start the process with 0.4.2 release?
> > Does anyone have any commits they feel strongly should go in before
0.4.2
> > is done, or are we ready to call it good?
> >
> > I believe there is consensus the 0.4.2 release should include a release
> of
> > the current state of the metron-bro-plugin-kafka. I will continue the
> > discussion in that thread as to the process for accomplishing that, but
> > plan on it happening.
> >
> > Regards,
> > --Matt
> >
> > On 11/26/17, 6:26 PM, "Matt Foley" <ma...@apache.org> wrote:
> >
> > Hope everyone (at least in the U.S.) had a great Thanksgiving
> holiday.
> > Regarding status of the release effort, still pending METRON-1252, so
> > not making the release branch yet.
> >
> > Regards,
> > --Matt
> >
> > On 11/17/17, 1:32 PM, "Matt Foley" <ma...@apache.org> wrote:
> >
> > (With release manager hat on)
> >
> > The community has proposed a release of Metron in the near
> future,
> > focusing on Meta-alerts running in Elasticsearch.
> > Congrats on getting so many of the below already done. At this
> > point, only METRON-1252, and the discussion of how to handle joint
> release
> > of the Metron bro plugin, remain as gating items for the release. I
> > project these will be resolved next week, so let’s propose the
following:
> >
> > Sometime next week, after the last bits are done, I’ll start the
> > release process and create the release branch.
> >
> > The proposed new version will be 0.4.2, unless there are backward
> > incompatible changes that support making it 0.5.0.
> > Currently there are NO included Jiras labeled
> > ‘backward-incompatible’, nor having Docs Text indicating so.
> > ***If anyone knows that some of the commits included since 0.4.1
> > introduce backward incompatibility, please say so now on this thread,
and
> > mark the Jira as such.***
> >
> > The 90 or so jiras/commits already in master branch since 0.4.1
> > are listed below.
> > Thanks,
> > --Matt
> >
> > METRON-1301 Alerts UI - Sorting on Triage Score Unexpectedly
> > Filters Some Records (nickwallen) closes apache/metron#832
> > METRON-1294 IP addresses are not formatted correctly in facet
> > and group results (merrimanr) closes apache/metron#827
> > METRON-1291 Kafka produce REST endpoint does not work in a
> > Kerberized cluster (merrimanr) closes apache/metron#826
> > METRON-1290 Only first 10 alerts are update when a MetaAlert
> > status is changed to inactive (justinleet) closes apache/metron#842
> > METRON-1311 Service Check Should Check Elasticsearch Index
> > Templates (nickwallen) closes apache/metron#839
> > METRON-1289 Alert fields are lost when a MetaAlert is created
> > (merrimanr) closes apache/metron#824
> > METRON-1309 Change metron-deployment to pull the plugin from
> > apache/metron-bro-plugin-kafka (JonZeolla) closes apache/metron#837
> > METRON-1310 Template Delete Action Deletes Search Indices
> > (nickwallen) closes apache/metron#838
> > METRON-1275: Fix Metron Documentation closes
> > apache/incubator-metron#833
> > METRON-1295 Unable to Configure Logging for REST API
> > (nickwallen) closes apache/metron#828
> > METRON-1307 Force install of java8 since java9 does not
> appear
> > to work with the scripts (brianhurley via ottobackwards) closes
> > apache/metron#835
> > METRON-1296 Full Dev Fails to Deploy Index Templates
> > (nickwallen via cestella) closes apache/incubator-metron#829
> > METRON-1281 Remove hard-coded indices from the Alerts UI
> > (merrimanr) closes apache/metron#821
> > METRON-1287 Full Dev Fails When Installing EPEL Repository
> > (nickwallen) closes apache/metron#820
> > METRON-1267 Alerts UI returns a 404 when refreshing the
> > alerts-list page (iraghumitra via merrimanr) closes apache/metron#819
> > METRON-1283 Install Elasticsearch template as a part of the
> > mpack startup scripts (anandsubbu via nickwallen) closes
> apache/metron#817
> > METRON-1254: Conditionals as map keys do not function in
> > Stellar closes apache/incubator-metron#801
> > METRON-1261 Apply bro security patch (JonZeolla via
> > ottobackwards) closes apache/metron#805
> > METRON-1284 Remove extraneous dead query in ElasticsearchDao
> > (justinleet) closes apache/metron#818
> > METRON-1270: fix for warnings missing @return tag argument in
> > metron-analytics/metron-profiler-common and metron-profiler-client
closes
> > apache/incubator-metron#810
> > METRON-1272 Hide child alerts from searches and grouping if
> > they belong to meta alerts (justinleet) closes apache/metron#811
> > METRON-1224 Add time range selection to search control
> > (iraghumitra via james-sirota) closes apache/metron#796
> > METRON-1280 0.4.1 -> 0.4.2 missed a couple of projects
> > (cestella via justinleet) closes apache/metron#816
> > METRON-1243: Add a REST endpoint which allows us to get a
> list
> > of all indice closes apache/incubator-metron#797
> > METRON-1196 Increment master version number to 0.4.2 for
> > on-going development (mattf-horton) closes apache/metron#767
> > METRON-1278 Strip "Build Status" widget from root
> > README.md in site-book build (mattf-horton) closes apache/metron#815
> > METRON-1274 Master has failure in
> > StormControllerIntegrationTest (merrimanr) closes apache/metron#813
> > METRON-1266 Profiler - SASL Authentication Failed
> (nickwallen)
> > closes apache/metron#809
> > METRON-1260 Include Alerts UI in Ambari Service Check
> > (nickwallen) closes apache/metron#804
> > METRON-1251: Typo and formatting fixes for metron-rest README
> > closes apache/incubator-metron#800
> > METRON-1241: Enable the REST API to use a cache for the
> > zookeeper config similar to the Bolts closes
apache/incubator-metron#795
> > METRON-1267 Alerts UI returns a 404 when refreshing the
> > alerts-list page (merrimanr) closes apache/metron#808
> > METRON-1262 Unable to add comment for a alert in a meta-alert
> > (merrimanr) closes apache/metron#806
> > METRON-1263 Start Alerts UI service after Metron REST
> > (anandsubbu via nickwallen) closes apache/metron#807
> > METRON-1255 MetaAlert search is not filtering on status
> > (merrimanr) closes apache/metron#802
> > METRON-1249 Improve Metron MPack Service Checks (nickwallen)
> > closes apache/metron#799
> > METRON-1237 address javadoc warnings in metron-maas-common
> > (dbist via james-sirota) closes apache/metron#792
> > METRON-1240 address javadoc warnings in metron-platform and
> > metron-analytics (dbist via james-sirota) closes apache/metron#794
> > METRON-1226 Searching Can Errantly Query the Wrong Indices
> > (nickwallen) closes apache/metron#793
> > METRON-1081 Fix Alerts and Ops UI Notices file (james-sirota)
> > closes apache/metron#682
> > METRON-1123 Add group by option using faceted search
> > capabilities of metron-rest-api (iraghumitra via james-sirota) closes
> > apache/metron#768
> > METRON-1223 Add support to add comments for alerts
> > (iraghumitra via james-sirota) closes apache/metron#788
> > METRON-1083 Add filters using faceted search capabilities of
> > metron-rest-api (iraghumitra via james-sirota) closes apache/metron#710
> > METRON-1232 Alert status changes are not reflected in list
> > view (iraghumitra via merrimanr) closes apache/metron#787
> > METRON-1247 REST search and findOne endpoints return
> > unexpected or incorrect results for guids (justinleet) closes
> > apache/metron#798
> > METRON-1235: Document the properties pulled from the global
> > configuration closes apache/incubator-metron#791
> > METRON-1234: fix for WARNING 'dependencies.dependency.(
> > groupId:artifactId:type:classifier)' must be unique:
> > org.apache.hadoop:hadoop-yarn-api:jar (dbist via mmiklavc) closes
> > apache/metron#790
> > METRON-1222: fix warning for The expression ${parent.version}
> > is deprecated. Please use ${project.parent.version} instead. (dbist via
> > mmiklavc) closes apache/metron#782
> > METRON-1220 Create documentation around alert nested field
> > (justinleet) closes apache/metron#780
> > METRON-1229 Management UI type is part of the declarations of
> > 2 modules (merrimanr) closes apache/metron#784
> > METRON-1228: Configuration Management PUSH immediately does
> > DUMP after (mmiklavc via mmiklavc) closes apache/metron#783
> > METRON-1218 Metron REST should return better error messages
> > (merrimanr) closes apache/metron#779
> > METRON-1161 Add ability to edit parser command line options
> in
> > the management UI (merrimanr) closes apache/metron#737
> > METRON-1209: Make stellar repl take logging properties, like
> > other CLI apps in metron closes apache/incubator-metron#772
> > METRON-1059 address checkstyle warning AvoidStarImport in
> > metron-stellar (dbist via ottobackwards) closes apache/metron#664
> > METRON-1204 UI does not time out after being idle, but stops
> > functioning (merrimanr) closes apache/metron#771
> > METRON-1052: Add forensic similarity hash functions to
> Stellar
> > closes apache/incubator-metron#781
> > METRON-632: Added validation of "shew.enrichmentType" and
> > "shew.keyColumns" closes apache/incubator-metron#732
> > METRON-1194 Add Profiler Debug Functions to Profiler README
> > (nickwallen via ottobackwards) closes apache/metron#765
> > METRON-1055 Metron 0.4.0 manual installation guide for CentOS
> > 6 updates (lvets via ottobackwards) closes apache/metron#661
> > METRON-1079 STELLAR NaN should be a keyword (ottobackwards)
> > closes apache/metron#681
> > METRON-1085 Add REST endpoint to save a user profile for the
> > Alerts UI (merrimanr) closes apache/metron#694
> > METRON-1208 MPack for Alerts UI (merrimanr) closes
> > apache/metron#778
> > METRON-1207 Make RPMs for Alerts UI (merrimanr) closes
> > apache/metron#777
> > METRON-1215 Fix link to RPMs chapter (DimDroll via
> justinleet)
> > closes apache/metron#776
> > METRON-1206 Make alerts UI conform to ops UI for install
> > (merrimanr) closes apache/metron#773
> > METRON-1195 Meta alerts improperly handle updates to
> non-alert
> > fields (justinleet) closes apache/metron#766
> > METRON-1189 Add alert escalation to the Alerts UI (merrimanr)
> > closes apache/metron#762
> > METRON-1156 Simulate Triage Rules in the Stellar REPL
> > (nickwallen) closes apache/metron#733
> > METRON-1198 Pycapa - No such configuration property
> > 'sasl.kerberos.principal' (nickwallen) closes apache/metron#769
> > METRON-1202 ElasticsearchDao Has extraneous sleep call
> > (justinleet) closes apache/metron#770
> > METRON-938 "service metron-rest start <password>" does not
> > work on CentOS 7. (justinleet) closes apache/metron#757
> > METRON-1182 Refactor Code in alert list to accommodate new
> > view types (iraghumitra via merrimanr) closes apache/metron#756
> > METRON-1188: Ambari global configuration management
> (mmiklavc)
> > closes apache/metron#760
> > METRON-1191 update public web site to point at 0.4.1 new
> > release (mattf-horton) closes apache/metron#764
> > METRON-1063 address javadoc warnings in metron-stellar (dbist
> > via ottobackwards) closes apache/metron#668
> > METRON-1190 Fix Meta Alert Type handling in calculation of
> > scores (justinleet) closes apache/metron#763
> > METRON-1187 Indexing/Profiler Kafka ACL Groups Not Setup
> > Correctly (nickwallen) closes apache/metron#759
> > METRON-1185: Stellar REPL does not work on a kerberized
> > cluster when calling functions interacting with HBase closes
> > apache/incubator-metron#755
> > METRON-1186: Profiler Functions use classutils from shaded
> > storm closes apache/incubator-metron#758
> > METRON-1173: Fix pointers to old stellar docs closes
> > apache/incubator-metron#746
> > METRON-1179: Make STATS_ADD to take a list closes
> > apache/incubator-metron#750
> > METRON-1180: Make Stellar Shell accept zookeeper quorum as a
> > CSV list and not require a port closes apache/incubator-metron#751
> > METRON-1183 Improve KDC Setup Instructions (nickwallen)
> closes
> > apache/metron#753
> > METRON-1177 Stale running topologies seen post-kerberization
> > and cause exceptions (nickwallen) closes apache/metron#748
> > METRON-1158 Build backend for grouping alerts into meta
> alerts
> > (justinleet) closes apache/metron#734
> > METRON-1146: Add ability to parse JSON string into JSONObject
> > for stellar closes apache/incubator-metron#727
> > METRON-1176 REST: HDFS Service should support setting
> > permissions on files when writing (ottobackwards) closes
> apache/metron#749
> > METRON-1114 Add group by capabilities to search REST endpoint
> > (merrimanr) closes apache/metron#702
> > METRON-1167 Define Session Specific Global Configuration
> > Values in the REPL (nickwallen) closes apache/metron#740
> > METRON-1171: Better validation for the SUBSTRING stellar
> > function closes apache/incubator-metron#745
> >
> >
> >
> > On 11/17/17, 11:59 AM, "Nick Allen" <ni...@nickallen.org> wrote:
> >
> > I just wanted to send an update on where we are at. We've
> > gotten a lot
> > done here recently as you can see below.
> >
> > ✓ DONE (1) First, METRON-1289 needs to go in. This one was
> > a fairly big
> > effort and I am hearing that we are pretty close.
> >
> > ✓ DONE (2) METRON-1294 fixes an issue in how field types
> are
> > looked-up.
> >
> > ✓ DONE (3) METRON-1290 is next. While this may have been
> > fixed in
> > M-1289, there may be some test cases we want from this PR.
> >
> > ✓ DONE (4) METRON-1301 addresses a problem with the sorting
> > logic.
> >
> > ✓ DONE (5) METRON-1291 fixes an issue with escalation of
> > metaalerts.
> >
> > (6) That leads us to Raghu's UI work in METRON-1252. This
> > introduces the
> > UI bits that depend on all the previous backend work.
> >
> > (7) At this point, we should have our best effort at
> running
> > Metaalerts
> > on Elasticsearch 2.x. I propose that we cut a release here.
> >
> > (8) After we cut the release, we can introduce the work for
> > ES 5.x in
> > METRON-939. I know we will need lots of help testing and
> > reviewing this
> > one.
> >
> >
> >
> > We also have an outstanding question that needs resolved
> > BEFORE we
> > release. We need to come to a consensus on how to release
> > having moved our
> > Bro Plugin to a separate repo. I don't think we've heard
> from
> > everyone on
> > this. I'd urge everyone to chime in so we can choose a path
> > forward.
> >
> > If anyone is totally confused in regards to that discussion,
> I
> > can try and
> > send an options summary again as a separate discuss thread.
> > The original
> > chain was somewhere around here [1].
> >
> > [1]
> > https://lists.apache.org/thread.html/
> > 54a4474881b97e559df24728b3a0e923a58345a282451085eef832ef@%
> > 3Cdev.metron.apache.org%3E
> >
> >
> >
> > On Wed, Nov 15, 2017 at 10:04 AM, Nick Allen <
> > nick@nickallen.org> wrote:
> >
> > > Hi Guys -
> > >
> > > I want to follow-up on this discussion. It sounds like
> most
> > people are in
> > > agreement with the general approach.
> > >
> > > A lot of people have been working hard on Metaalerts and
> > Elasticsearch. I
> > > have checked-in with those doing the heavy lifting and have
> > compiled a more
> > > detailed plan based on where we are at now. To the best of
> > my knowledge
> > > here is the plan of attack for finishing out this effort.
> > >
> > > (1) First, METRON-1289 needs to go in. This one was a
> > fairly big effort
> > > and I am hearing that we are pretty close.
> > >
> > > (2) METRON-1294 fixes an issue in how field types are
> > looked-up.
> > >
> > > (3) METRON-1290 is next. While this may have been fixed
> > in M-1289,
> > > there may be some test cases we want from this PR.
> > >
> > > (4) METRON-1301 addresses a problem with the sorting
> logic.
> > >
> > > (5) METRON-1291 fixes an issue with escalation of
> > metaalerts.
> > >
> > > (6) That leads us to Raghu's UI work in METRON-1252.
> This
> > introduces
> > > the UI bits that depend on all the previous backend work.
> > >
> > > (7) At this point, we should have our best effort at
> > running Metaalerts
> > > on Elasticsearch 2.x. I propose that we cut a release here.
> > >
> > > (8) After we cut the release, we can introduce the work
> > for ES 5.x in
> > > METRON-939. I know we will need lots of help testing and
> > reviewing this
> > > one.
> > >
> > > Please correct me if I am wrong. I will try and send out
> > updates as we
> > > make progress.
> > >
> > >
> > >
> > >
> > >
> > > On Mon, Nov 6, 2017 at 1:03 PM, Zeolla@GMail.com <
> > zeolla@gmail.com> wrote:
> > >
> > >> I agree, I think it's very reasonable to move in line with
> > Nick's
> > >> proposal. I would also suggest that we outline what the
> > target versions
> > >> would be to add in the METRON-777 components, since it has
> > been functional
> > >> for a very long time but not reviewed and has some really
> > rockstar
> > >> improvements.
> > >>
> > >> Jon
> > >>
> > >> On Mon, Nov 6, 2017 at 12:56 PM Otto Fowler <
> > ottobackwards@gmail.com>
> > >> wrote:
> > >>
> > >> > I think the ES cutover should be the start of the 0.5.x
> > series, and we
> > >> > continue on with 0.4.x for the
> > >> > metadata improvements etc. We could chose to focus
> > 0.5.x’s first
> > >> releases
> > >> > on not only ES but
> > >> > getting a handle on kibana and the mpack situation as
> > well.
> > >> >
> > >> >
> > >> >
> > >> >
> > >> > On November 6, 2017 at 12:48:45, Michael Miklavcic (
> > >> > michael.miklavcic@gmail.com) wrote:
> > >> >
> > >> > I agree with your proposal, Nick. I think having a
> > stabilizing release
> > >> > prior to upgrading ES/Kibana makes sense.
> > >> >
> > >> > On Mon, Nov 6, 2017 at 9:16 AM, Nick Allen <
> > nick@nickallen.org> wrote:
> > >> >
> > >> > > I would like to start a discussion around upcoming
> > releases. We have a
> > >> > > couple separate significant tracks of work that we
> need
> > to reconcile
> > >> in
> > >> > our
> > >> > > release schedule.
> > >> > >
> > >> > > (1) We have had (and have in review) a good number of
> > bug fixes
> > >> required
> > >> > to
> > >> > > support Metaalerts on the existing Elasticsearch 2.x
> > infrastructure.
> > >> > >
> > >> > >
> > >> > > (2) We also have ongoing work to upgrade our
> > infrastructure to
> > >> > > Elasticsearch 5.x, which will not be backwards
> > compatible.
> > >> > >
> > >> > >
> > >> > > I would like to see a release that has our best work
> on
> > ES 2.x before
> > >> we
> > >> > > migrate to 5.x. I would propose the following.
> > >> > >
> > >> > > Release N+1: Introduce Metaalerts running on ES 2.x
> > >> > >
> > >> > > Release N+2: Cut-over to ES 5.x
> > >> > >
> > >> > >
> > >> > > (Q) Is it worth cutting a separate release for ES 2.x?
> > Is there a
> > >> better
> > >> > > way to handle the cut-over to 5.x?
> > >> > >
> > >> >
> > >> --
> > >>
> > >> Jon
> > >>
> > >
> > >
> >
> >
> >
> >
> >
> >
> >
> >
>
--
Jon
Re: [DISCUSS] Upcoming Release
Posted by Matt Foley <mf...@hortonworks.com>.
Hah, here it is: https://github.com/apache/metron/pull/743
“This problem seems to only reproduce when one unrolls a tarball rather than cloning from github.”
Heh, the exclusion at https://github.com/apache/metron/blob/master/pom.xml#L351 is still there, but the hashcode in the bundle.css file name has changed from a0b6b99c10d9a13dc67e to f56deed131e58bd7ee04. Sigh. Did the version of Font Awesome fonts change?
On 12/8/17, 5:26 PM, "Matt Foley" <ma...@apache.org> wrote:
I remember having trouble with this bundle.css file on the last release, but I can’t remember what we did about it. Anybody?
On 12/8/17, 1:41 PM, "Otto Fowler" <ot...@gmail.com> wrote:
Steps
- Downloaded tar.gz’s, asc files and KEYS
- Verified signing of both tar.gz’s
- searched for rouge 0.4.1 entries
- verified the main pom.xml
- built :
mvn clean && time mvn -q -T 2C -DskipTests install && time mvn -q -T
2C surefire:test@unit-tests && time mvn -q
surefire:test@integration-tests && time mvn -q test --projects
metron-interface/metron-config && time build_utils/verify_licenses.sh
Found rat error:
*****************************************************
Summary
-------
Generated at: 2017-12-08T16:33:27-05:00
Notes: 3
Binaries: 193
Archives: 0
Standards: 75
Apache Licensed: 74
Generated Documents: 0
JavaDocs are generated, thus a license header is optional.
Generated files do not require license headers.
1 Unknown Licenses
*****************************************************
Files with unapproved licenses:
/Users/batman/tmp/release_ver/apache-metron-0.4.2-rc1/metron-interface/metron-alerts/dist/styles.f56deed131e58bd7ee04.bundle.css
*****************************************************
*****************************************************
Summary
-------
Generated at: 2017-12-08T16:33:27-05:00
Notes: 3
Binaries: 193
Archives: 0
Standards: 75
Apache Licensed: 74
Generated Documents: 0
JavaDocs are generated, thus a license header is optional.
Generated files do not require license headers.
1 Unknown Licenses
*****************************************************
Files with unapproved licenses:
/Users/ottofowler/tmp/release_ver/apache-metron-0.4.2-rc1/metron-interface/metron-alerts/dist/styles.f56deed131e58bd7ee04.bundle.css
*****************************************************
On December 8, 2017 at 04:34:24, Matt Foley (mattf@apache.org) wrote:
Colleagues,
I’ve posted Metron-0.4.2-RC1 and Metron-bro-plugin-kafka-0.1 to
https://dist.apache.org/repos/dist/dev/metron/0.4.2-RC1/
Given the complexity of this RC, I’d appreciate if a couple people would be
willing to kick the tires before we put it up for a vote.
I will myself be going thru the Verify Build process this weekend, as I
won’t be able to do it Friday.
Thanks,
--Matt
On 12/4/17, 2:05 PM, "Zeolla@GMail.com" <ze...@gmail.com> wrote:
Can we resolve the conversation regarding the second repo? I was waiting
to get more input/preferences from people There's also a documentation
update that fixes a few broken Stellar docs that already has aa +1, I just
need to merge it.
Jon
On Mon, Dec 4, 2017, 17:01 Casey Stella <ce...@gmail.com> wrote:
> I would be in favor of a release at this point.
>
> On Mon, Dec 4, 2017 at 4:57 PM, Matt Foley <ma...@apache.org> wrote:
>
> > Hey all,
> > I see METRON-1252 was resolved over the weekend. Shall I go ahead and
> > start the process with 0.4.2 release?
> > Does anyone have any commits they feel strongly should go in before
0.4.2
> > is done, or are we ready to call it good?
> >
> > I believe there is consensus the 0.4.2 release should include a release
> of
> > the current state of the metron-bro-plugin-kafka. I will continue the
> > discussion in that thread as to the process for accomplishing that, but
> > plan on it happening.
> >
> > Regards,
> > --Matt
> >
> > On 11/26/17, 6:26 PM, "Matt Foley" <ma...@apache.org> wrote:
> >
> > Hope everyone (at least in the U.S.) had a great Thanksgiving
> holiday.
> > Regarding status of the release effort, still pending METRON-1252, so
> > not making the release branch yet.
> >
> > Regards,
> > --Matt
> >
> > On 11/17/17, 1:32 PM, "Matt Foley" <ma...@apache.org> wrote:
> >
> > (With release manager hat on)
> >
> > The community has proposed a release of Metron in the near
> future,
> > focusing on Meta-alerts running in Elasticsearch.
> > Congrats on getting so many of the below already done. At this
> > point, only METRON-1252, and the discussion of how to handle joint
> release
> > of the Metron bro plugin, remain as gating items for the release. I
> > project these will be resolved next week, so let’s propose the
following:
> >
> > Sometime next week, after the last bits are done, I’ll start the
> > release process and create the release branch.
> >
> > The proposed new version will be 0.4.2, unless there are backward
> > incompatible changes that support making it 0.5.0.
> > Currently there are NO included Jiras labeled
> > ‘backward-incompatible’, nor having Docs Text indicating so.
> > ***If anyone knows that some of the commits included since 0.4.1
> > introduce backward incompatibility, please say so now on this thread,
and
> > mark the Jira as such.***
> >
> > The 90 or so jiras/commits already in master branch since 0.4.1
> > are listed below.
> > Thanks,
> > --Matt
> >
> > METRON-1301 Alerts UI - Sorting on Triage Score Unexpectedly
> > Filters Some Records (nickwallen) closes apache/metron#832
> > METRON-1294 IP addresses are not formatted correctly in facet
> > and group results (merrimanr) closes apache/metron#827
> > METRON-1291 Kafka produce REST endpoint does not work in a
> > Kerberized cluster (merrimanr) closes apache/metron#826
> > METRON-1290 Only first 10 alerts are update when a MetaAlert
> > status is changed to inactive (justinleet) closes apache/metron#842
> > METRON-1311 Service Check Should Check Elasticsearch Index
> > Templates (nickwallen) closes apache/metron#839
> > METRON-1289 Alert fields are lost when a MetaAlert is created
> > (merrimanr) closes apache/metron#824
> > METRON-1309 Change metron-deployment to pull the plugin from
> > apache/metron-bro-plugin-kafka (JonZeolla) closes apache/metron#837
> > METRON-1310 Template Delete Action Deletes Search Indices
> > (nickwallen) closes apache/metron#838
> > METRON-1275: Fix Metron Documentation closes
> > apache/incubator-metron#833
> > METRON-1295 Unable to Configure Logging for REST API
> > (nickwallen) closes apache/metron#828
> > METRON-1307 Force install of java8 since java9 does not
> appear
> > to work with the scripts (brianhurley via ottobackwards) closes
> > apache/metron#835
> > METRON-1296 Full Dev Fails to Deploy Index Templates
> > (nickwallen via cestella) closes apache/incubator-metron#829
> > METRON-1281 Remove hard-coded indices from the Alerts UI
> > (merrimanr) closes apache/metron#821
> > METRON-1287 Full Dev Fails When Installing EPEL Repository
> > (nickwallen) closes apache/metron#820
> > METRON-1267 Alerts UI returns a 404 when refreshing the
> > alerts-list page (iraghumitra via merrimanr) closes apache/metron#819
> > METRON-1283 Install Elasticsearch template as a part of the
> > mpack startup scripts (anandsubbu via nickwallen) closes
> apache/metron#817
> > METRON-1254: Conditionals as map keys do not function in
> > Stellar closes apache/incubator-metron#801
> > METRON-1261 Apply bro security patch (JonZeolla via
> > ottobackwards) closes apache/metron#805
> > METRON-1284 Remove extraneous dead query in ElasticsearchDao
> > (justinleet) closes apache/metron#818
> > METRON-1270: fix for warnings missing @return tag argument in
> > metron-analytics/metron-profiler-common and metron-profiler-client
closes
> > apache/incubator-metron#810
> > METRON-1272 Hide child alerts from searches and grouping if
> > they belong to meta alerts (justinleet) closes apache/metron#811
> > METRON-1224 Add time range selection to search control
> > (iraghumitra via james-sirota) closes apache/metron#796
> > METRON-1280 0.4.1 -> 0.4.2 missed a couple of projects
> > (cestella via justinleet) closes apache/metron#816
> > METRON-1243: Add a REST endpoint which allows us to get a
> list
> > of all indice closes apache/incubator-metron#797
> > METRON-1196 Increment master version number to 0.4.2 for
> > on-going development (mattf-horton) closes apache/metron#767
> > METRON-1278 Strip "Build Status" widget from root
> > README.md in site-book build (mattf-horton) closes apache/metron#815
> > METRON-1274 Master has failure in
> > StormControllerIntegrationTest (merrimanr) closes apache/metron#813
> > METRON-1266 Profiler - SASL Authentication Failed
> (nickwallen)
> > closes apache/metron#809
> > METRON-1260 Include Alerts UI in Ambari Service Check
> > (nickwallen) closes apache/metron#804
> > METRON-1251: Typo and formatting fixes for metron-rest README
> > closes apache/incubator-metron#800
> > METRON-1241: Enable the REST API to use a cache for the
> > zookeeper config similar to the Bolts closes
apache/incubator-metron#795
> > METRON-1267 Alerts UI returns a 404 when refreshing the
> > alerts-list page (merrimanr) closes apache/metron#808
> > METRON-1262 Unable to add comment for a alert in a meta-alert
> > (merrimanr) closes apache/metron#806
> > METRON-1263 Start Alerts UI service after Metron REST
> > (anandsubbu via nickwallen) closes apache/metron#807
> > METRON-1255 MetaAlert search is not filtering on status
> > (merrimanr) closes apache/metron#802
> > METRON-1249 Improve Metron MPack Service Checks (nickwallen)
> > closes apache/metron#799
> > METRON-1237 address javadoc warnings in metron-maas-common
> > (dbist via james-sirota) closes apache/metron#792
> > METRON-1240 address javadoc warnings in metron-platform and
> > metron-analytics (dbist via james-sirota) closes apache/metron#794
> > METRON-1226 Searching Can Errantly Query the Wrong Indices
> > (nickwallen) closes apache/metron#793
> > METRON-1081 Fix Alerts and Ops UI Notices file (james-sirota)
> > closes apache/metron#682
> > METRON-1123 Add group by option using faceted search
> > capabilities of metron-rest-api (iraghumitra via james-sirota) closes
> > apache/metron#768
> > METRON-1223 Add support to add comments for alerts
> > (iraghumitra via james-sirota) closes apache/metron#788
> > METRON-1083 Add filters using faceted search capabilities of
> > metron-rest-api (iraghumitra via james-sirota) closes apache/metron#710
> > METRON-1232 Alert status changes are not reflected in list
> > view (iraghumitra via merrimanr) closes apache/metron#787
> > METRON-1247 REST search and findOne endpoints return
> > unexpected or incorrect results for guids (justinleet) closes
> > apache/metron#798
> > METRON-1235: Document the properties pulled from the global
> > configuration closes apache/incubator-metron#791
> > METRON-1234: fix for WARNING 'dependencies.dependency.(
> > groupId:artifactId:type:classifier)' must be unique:
> > org.apache.hadoop:hadoop-yarn-api:jar (dbist via mmiklavc) closes
> > apache/metron#790
> > METRON-1222: fix warning for The expression ${parent.version}
> > is deprecated. Please use ${project.parent.version} instead. (dbist via
> > mmiklavc) closes apache/metron#782
> > METRON-1220 Create documentation around alert nested field
> > (justinleet) closes apache/metron#780
> > METRON-1229 Management UI type is part of the declarations of
> > 2 modules (merrimanr) closes apache/metron#784
> > METRON-1228: Configuration Management PUSH immediately does
> > DUMP after (mmiklavc via mmiklavc) closes apache/metron#783
> > METRON-1218 Metron REST should return better error messages
> > (merrimanr) closes apache/metron#779
> > METRON-1161 Add ability to edit parser command line options
> in
> > the management UI (merrimanr) closes apache/metron#737
> > METRON-1209: Make stellar repl take logging properties, like
> > other CLI apps in metron closes apache/incubator-metron#772
> > METRON-1059 address checkstyle warning AvoidStarImport in
> > metron-stellar (dbist via ottobackwards) closes apache/metron#664
> > METRON-1204 UI does not time out after being idle, but stops
> > functioning (merrimanr) closes apache/metron#771
> > METRON-1052: Add forensic similarity hash functions to
> Stellar
> > closes apache/incubator-metron#781
> > METRON-632: Added validation of "shew.enrichmentType" and
> > "shew.keyColumns" closes apache/incubator-metron#732
> > METRON-1194 Add Profiler Debug Functions to Profiler README
> > (nickwallen via ottobackwards) closes apache/metron#765
> > METRON-1055 Metron 0.4.0 manual installation guide for CentOS
> > 6 updates (lvets via ottobackwards) closes apache/metron#661
> > METRON-1079 STELLAR NaN should be a keyword (ottobackwards)
> > closes apache/metron#681
> > METRON-1085 Add REST endpoint to save a user profile for the
> > Alerts UI (merrimanr) closes apache/metron#694
> > METRON-1208 MPack for Alerts UI (merrimanr) closes
> > apache/metron#778
> > METRON-1207 Make RPMs for Alerts UI (merrimanr) closes
> > apache/metron#777
> > METRON-1215 Fix link to RPMs chapter (DimDroll via
> justinleet)
> > closes apache/metron#776
> > METRON-1206 Make alerts UI conform to ops UI for install
> > (merrimanr) closes apache/metron#773
> > METRON-1195 Meta alerts improperly handle updates to
> non-alert
> > fields (justinleet) closes apache/metron#766
> > METRON-1189 Add alert escalation to the Alerts UI (merrimanr)
> > closes apache/metron#762
> > METRON-1156 Simulate Triage Rules in the Stellar REPL
> > (nickwallen) closes apache/metron#733
> > METRON-1198 Pycapa - No such configuration property
> > 'sasl.kerberos.principal' (nickwallen) closes apache/metron#769
> > METRON-1202 ElasticsearchDao Has extraneous sleep call
> > (justinleet) closes apache/metron#770
> > METRON-938 "service metron-rest start <password>" does not
> > work on CentOS 7. (justinleet) closes apache/metron#757
> > METRON-1182 Refactor Code in alert list to accommodate new
> > view types (iraghumitra via merrimanr) closes apache/metron#756
> > METRON-1188: Ambari global configuration management
> (mmiklavc)
> > closes apache/metron#760
> > METRON-1191 update public web site to point at 0.4.1 new
> > release (mattf-horton) closes apache/metron#764
> > METRON-1063 address javadoc warnings in metron-stellar (dbist
> > via ottobackwards) closes apache/metron#668
> > METRON-1190 Fix Meta Alert Type handling in calculation of
> > scores (justinleet) closes apache/metron#763
> > METRON-1187 Indexing/Profiler Kafka ACL Groups Not Setup
> > Correctly (nickwallen) closes apache/metron#759
> > METRON-1185: Stellar REPL does not work on a kerberized
> > cluster when calling functions interacting with HBase closes
> > apache/incubator-metron#755
> > METRON-1186: Profiler Functions use classutils from shaded
> > storm closes apache/incubator-metron#758
> > METRON-1173: Fix pointers to old stellar docs closes
> > apache/incubator-metron#746
> > METRON-1179: Make STATS_ADD to take a list closes
> > apache/incubator-metron#750
> > METRON-1180: Make Stellar Shell accept zookeeper quorum as a
> > CSV list and not require a port closes apache/incubator-metron#751
> > METRON-1183 Improve KDC Setup Instructions (nickwallen)
> closes
> > apache/metron#753
> > METRON-1177 Stale running topologies seen post-kerberization
> > and cause exceptions (nickwallen) closes apache/metron#748
> > METRON-1158 Build backend for grouping alerts into meta
> alerts
> > (justinleet) closes apache/metron#734
> > METRON-1146: Add ability to parse JSON string into JSONObject
> > for stellar closes apache/incubator-metron#727
> > METRON-1176 REST: HDFS Service should support setting
> > permissions on files when writing (ottobackwards) closes
> apache/metron#749
> > METRON-1114 Add group by capabilities to search REST endpoint
> > (merrimanr) closes apache/metron#702
> > METRON-1167 Define Session Specific Global Configuration
> > Values in the REPL (nickwallen) closes apache/metron#740
> > METRON-1171: Better validation for the SUBSTRING stellar
> > function closes apache/incubator-metron#745
> >
> >
> >
> > On 11/17/17, 11:59 AM, "Nick Allen" <ni...@nickallen.org> wrote:
> >
> > I just wanted to send an update on where we are at. We've
> > gotten a lot
> > done here recently as you can see below.
> >
> > ✓ DONE (1) First, METRON-1289 needs to go in. This one was
> > a fairly big
> > effort and I am hearing that we are pretty close.
> >
> > ✓ DONE (2) METRON-1294 fixes an issue in how field types
> are
> > looked-up.
> >
> > ✓ DONE (3) METRON-1290 is next. While this may have been
> > fixed in
> > M-1289, there may be some test cases we want from this PR.
> >
> > ✓ DONE (4) METRON-1301 addresses a problem with the sorting
> > logic.
> >
> > ✓ DONE (5) METRON-1291 fixes an issue with escalation of
> > metaalerts.
> >
> > (6) That leads us to Raghu's UI work in METRON-1252. This
> > introduces the
> > UI bits that depend on all the previous backend work.
> >
> > (7) At this point, we should have our best effort at
> running
> > Metaalerts
> > on Elasticsearch 2.x. I propose that we cut a release here.
> >
> > (8) After we cut the release, we can introduce the work for
> > ES 5.x in
> > METRON-939. I know we will need lots of help testing and
> > reviewing this
> > one.
> >
> >
> >
> > We also have an outstanding question that needs resolved
> > BEFORE we
> > release. We need to come to a consensus on how to release
> > having moved our
> > Bro Plugin to a separate repo. I don't think we've heard
> from
> > everyone on
> > this. I'd urge everyone to chime in so we can choose a path
> > forward.
> >
> > If anyone is totally confused in regards to that discussion,
> I
> > can try and
> > send an options summary again as a separate discuss thread.
> > The original
> > chain was somewhere around here [1].
> >
> > [1]
> > https://lists.apache.org/thread.html/
> > 54a4474881b97e559df24728b3a0e923a58345a282451085eef832ef@%
> > 3Cdev.metron.apache.org%3E
> >
> >
> >
> > On Wed, Nov 15, 2017 at 10:04 AM, Nick Allen <
> > nick@nickallen.org> wrote:
> >
> > > Hi Guys -
> > >
> > > I want to follow-up on this discussion. It sounds like
> most
> > people are in
> > > agreement with the general approach.
> > >
> > > A lot of people have been working hard on Metaalerts and
> > Elasticsearch. I
> > > have checked-in with those doing the heavy lifting and have
> > compiled a more
> > > detailed plan based on where we are at now. To the best of
> > my knowledge
> > > here is the plan of attack for finishing out this effort.
> > >
> > > (1) First, METRON-1289 needs to go in. This one was a
> > fairly big effort
> > > and I am hearing that we are pretty close.
> > >
> > > (2) METRON-1294 fixes an issue in how field types are
> > looked-up.
> > >
> > > (3) METRON-1290 is next. While this may have been fixed
> > in M-1289,
> > > there may be some test cases we want from this PR.
> > >
> > > (4) METRON-1301 addresses a problem with the sorting
> logic.
> > >
> > > (5) METRON-1291 fixes an issue with escalation of
> > metaalerts.
> > >
> > > (6) That leads us to Raghu's UI work in METRON-1252.
> This
> > introduces
> > > the UI bits that depend on all the previous backend work.
> > >
> > > (7) At this point, we should have our best effort at
> > running Metaalerts
> > > on Elasticsearch 2.x. I propose that we cut a release here.
> > >
> > > (8) After we cut the release, we can introduce the work
> > for ES 5.x in
> > > METRON-939. I know we will need lots of help testing and
> > reviewing this
> > > one.
> > >
> > > Please correct me if I am wrong. I will try and send out
> > updates as we
> > > make progress.
> > >
> > >
> > >
> > >
> > >
> > > On Mon, Nov 6, 2017 at 1:03 PM, Zeolla@GMail.com <
> > zeolla@gmail.com> wrote:
> > >
> > >> I agree, I think it's very reasonable to move in line with
> > Nick's
> > >> proposal. I would also suggest that we outline what the
> > target versions
> > >> would be to add in the METRON-777 components, since it has
> > been functional
> > >> for a very long time but not reviewed and has some really
> > rockstar
> > >> improvements.
> > >>
> > >> Jon
> > >>
> > >> On Mon, Nov 6, 2017 at 12:56 PM Otto Fowler <
> > ottobackwards@gmail.com>
> > >> wrote:
> > >>
> > >> > I think the ES cutover should be the start of the 0.5.x
> > series, and we
> > >> > continue on with 0.4.x for the
> > >> > metadata improvements etc. We could chose to focus
> > 0.5.x’s first
> > >> releases
> > >> > on not only ES but
> > >> > getting a handle on kibana and the mpack situation as
> > well.
> > >> >
> > >> >
> > >> >
> > >> >
> > >> > On November 6, 2017 at 12:48:45, Michael Miklavcic (
> > >> > michael.miklavcic@gmail.com) wrote:
> > >> >
> > >> > I agree with your proposal, Nick. I think having a
> > stabilizing release
> > >> > prior to upgrading ES/Kibana makes sense.
> > >> >
> > >> > On Mon, Nov 6, 2017 at 9:16 AM, Nick Allen <
> > nick@nickallen.org> wrote:
> > >> >
> > >> > > I would like to start a discussion around upcoming
> > releases. We have a
> > >> > > couple separate significant tracks of work that we
> need
> > to reconcile
> > >> in
> > >> > our
> > >> > > release schedule.
> > >> > >
> > >> > > (1) We have had (and have in review) a good number of
> > bug fixes
> > >> required
> > >> > to
> > >> > > support Metaalerts on the existing Elasticsearch 2.x
> > infrastructure.
> > >> > >
> > >> > >
> > >> > > (2) We also have ongoing work to upgrade our
> > infrastructure to
> > >> > > Elasticsearch 5.x, which will not be backwards
> > compatible.
> > >> > >
> > >> > >
> > >> > > I would like to see a release that has our best work
> on
> > ES 2.x before
> > >> we
> > >> > > migrate to 5.x. I would propose the following.
> > >> > >
> > >> > > Release N+1: Introduce Metaalerts running on ES 2.x
> > >> > >
> > >> > > Release N+2: Cut-over to ES 5.x
> > >> > >
> > >> > >
> > >> > > (Q) Is it worth cutting a separate release for ES 2.x?
> > Is there a
> > >> better
> > >> > > way to handle the cut-over to 5.x?
> > >> > >
> > >> >
> > >> --
> > >>
> > >> Jon
> > >>
> > >
> > >
> >
> >
> >
> >
> >
> >
> >
> >
>
--
Jon
Re: [DISCUSS] Upcoming Release
Posted by Matt Foley <ma...@apache.org>.
I remember having trouble with this bundle.css file on the last release, but I can’t remember what we did about it. Anybody?
On 12/8/17, 1:41 PM, "Otto Fowler" <ot...@gmail.com> wrote:
Steps
- Downloaded tar.gz’s, asc files and KEYS
- Verified signing of both tar.gz’s
- searched for rouge 0.4.1 entries
- verified the main pom.xml
- built :
mvn clean && time mvn -q -T 2C -DskipTests install && time mvn -q -T
2C surefire:test@unit-tests && time mvn -q
surefire:test@integration-tests && time mvn -q test --projects
metron-interface/metron-config && time build_utils/verify_licenses.sh
Found rat error:
*****************************************************
Summary
-------
Generated at: 2017-12-08T16:33:27-05:00
Notes: 3
Binaries: 193
Archives: 0
Standards: 75
Apache Licensed: 74
Generated Documents: 0
JavaDocs are generated, thus a license header is optional.
Generated files do not require license headers.
1 Unknown Licenses
*****************************************************
Files with unapproved licenses:
/Users/batman/tmp/release_ver/apache-metron-0.4.2-rc1/metron-interface/metron-alerts/dist/styles.f56deed131e58bd7ee04.bundle.css
*****************************************************
*****************************************************
Summary
-------
Generated at: 2017-12-08T16:33:27-05:00
Notes: 3
Binaries: 193
Archives: 0
Standards: 75
Apache Licensed: 74
Generated Documents: 0
JavaDocs are generated, thus a license header is optional.
Generated files do not require license headers.
1 Unknown Licenses
*****************************************************
Files with unapproved licenses:
/Users/ottofowler/tmp/release_ver/apache-metron-0.4.2-rc1/metron-interface/metron-alerts/dist/styles.f56deed131e58bd7ee04.bundle.css
*****************************************************
On December 8, 2017 at 04:34:24, Matt Foley (mattf@apache.org) wrote:
Colleagues,
I’ve posted Metron-0.4.2-RC1 and Metron-bro-plugin-kafka-0.1 to
https://dist.apache.org/repos/dist/dev/metron/0.4.2-RC1/
Given the complexity of this RC, I’d appreciate if a couple people would be
willing to kick the tires before we put it up for a vote.
I will myself be going thru the Verify Build process this weekend, as I
won’t be able to do it Friday.
Thanks,
--Matt
On 12/4/17, 2:05 PM, "Zeolla@GMail.com" <ze...@gmail.com> wrote:
Can we resolve the conversation regarding the second repo? I was waiting
to get more input/preferences from people There's also a documentation
update that fixes a few broken Stellar docs that already has aa +1, I just
need to merge it.
Jon
On Mon, Dec 4, 2017, 17:01 Casey Stella <ce...@gmail.com> wrote:
> I would be in favor of a release at this point.
>
> On Mon, Dec 4, 2017 at 4:57 PM, Matt Foley <ma...@apache.org> wrote:
>
> > Hey all,
> > I see METRON-1252 was resolved over the weekend. Shall I go ahead and
> > start the process with 0.4.2 release?
> > Does anyone have any commits they feel strongly should go in before
0.4.2
> > is done, or are we ready to call it good?
> >
> > I believe there is consensus the 0.4.2 release should include a release
> of
> > the current state of the metron-bro-plugin-kafka. I will continue the
> > discussion in that thread as to the process for accomplishing that, but
> > plan on it happening.
> >
> > Regards,
> > --Matt
> >
> > On 11/26/17, 6:26 PM, "Matt Foley" <ma...@apache.org> wrote:
> >
> > Hope everyone (at least in the U.S.) had a great Thanksgiving
> holiday.
> > Regarding status of the release effort, still pending METRON-1252, so
> > not making the release branch yet.
> >
> > Regards,
> > --Matt
> >
> > On 11/17/17, 1:32 PM, "Matt Foley" <ma...@apache.org> wrote:
> >
> > (With release manager hat on)
> >
> > The community has proposed a release of Metron in the near
> future,
> > focusing on Meta-alerts running in Elasticsearch.
> > Congrats on getting so many of the below already done. At this
> > point, only METRON-1252, and the discussion of how to handle joint
> release
> > of the Metron bro plugin, remain as gating items for the release. I
> > project these will be resolved next week, so let’s propose the
following:
> >
> > Sometime next week, after the last bits are done, I’ll start the
> > release process and create the release branch.
> >
> > The proposed new version will be 0.4.2, unless there are backward
> > incompatible changes that support making it 0.5.0.
> > Currently there are NO included Jiras labeled
> > ‘backward-incompatible’, nor having Docs Text indicating so.
> > ***If anyone knows that some of the commits included since 0.4.1
> > introduce backward incompatibility, please say so now on this thread,
and
> > mark the Jira as such.***
> >
> > The 90 or so jiras/commits already in master branch since 0.4.1
> > are listed below.
> > Thanks,
> > --Matt
> >
> > METRON-1301 Alerts UI - Sorting on Triage Score Unexpectedly
> > Filters Some Records (nickwallen) closes apache/metron#832
> > METRON-1294 IP addresses are not formatted correctly in facet
> > and group results (merrimanr) closes apache/metron#827
> > METRON-1291 Kafka produce REST endpoint does not work in a
> > Kerberized cluster (merrimanr) closes apache/metron#826
> > METRON-1290 Only first 10 alerts are update when a MetaAlert
> > status is changed to inactive (justinleet) closes apache/metron#842
> > METRON-1311 Service Check Should Check Elasticsearch Index
> > Templates (nickwallen) closes apache/metron#839
> > METRON-1289 Alert fields are lost when a MetaAlert is created
> > (merrimanr) closes apache/metron#824
> > METRON-1309 Change metron-deployment to pull the plugin from
> > apache/metron-bro-plugin-kafka (JonZeolla) closes apache/metron#837
> > METRON-1310 Template Delete Action Deletes Search Indices
> > (nickwallen) closes apache/metron#838
> > METRON-1275: Fix Metron Documentation closes
> > apache/incubator-metron#833
> > METRON-1295 Unable to Configure Logging for REST API
> > (nickwallen) closes apache/metron#828
> > METRON-1307 Force install of java8 since java9 does not
> appear
> > to work with the scripts (brianhurley via ottobackwards) closes
> > apache/metron#835
> > METRON-1296 Full Dev Fails to Deploy Index Templates
> > (nickwallen via cestella) closes apache/incubator-metron#829
> > METRON-1281 Remove hard-coded indices from the Alerts UI
> > (merrimanr) closes apache/metron#821
> > METRON-1287 Full Dev Fails When Installing EPEL Repository
> > (nickwallen) closes apache/metron#820
> > METRON-1267 Alerts UI returns a 404 when refreshing the
> > alerts-list page (iraghumitra via merrimanr) closes apache/metron#819
> > METRON-1283 Install Elasticsearch template as a part of the
> > mpack startup scripts (anandsubbu via nickwallen) closes
> apache/metron#817
> > METRON-1254: Conditionals as map keys do not function in
> > Stellar closes apache/incubator-metron#801
> > METRON-1261 Apply bro security patch (JonZeolla via
> > ottobackwards) closes apache/metron#805
> > METRON-1284 Remove extraneous dead query in ElasticsearchDao
> > (justinleet) closes apache/metron#818
> > METRON-1270: fix for warnings missing @return tag argument in
> > metron-analytics/metron-profiler-common and metron-profiler-client
closes
> > apache/incubator-metron#810
> > METRON-1272 Hide child alerts from searches and grouping if
> > they belong to meta alerts (justinleet) closes apache/metron#811
> > METRON-1224 Add time range selection to search control
> > (iraghumitra via james-sirota) closes apache/metron#796
> > METRON-1280 0.4.1 -> 0.4.2 missed a couple of projects
> > (cestella via justinleet) closes apache/metron#816
> > METRON-1243: Add a REST endpoint which allows us to get a
> list
> > of all indice closes apache/incubator-metron#797
> > METRON-1196 Increment master version number to 0.4.2 for
> > on-going development (mattf-horton) closes apache/metron#767
> > METRON-1278 Strip "Build Status" widget from root
> > README.md in site-book build (mattf-horton) closes apache/metron#815
> > METRON-1274 Master has failure in
> > StormControllerIntegrationTest (merrimanr) closes apache/metron#813
> > METRON-1266 Profiler - SASL Authentication Failed
> (nickwallen)
> > closes apache/metron#809
> > METRON-1260 Include Alerts UI in Ambari Service Check
> > (nickwallen) closes apache/metron#804
> > METRON-1251: Typo and formatting fixes for metron-rest README
> > closes apache/incubator-metron#800
> > METRON-1241: Enable the REST API to use a cache for the
> > zookeeper config similar to the Bolts closes
apache/incubator-metron#795
> > METRON-1267 Alerts UI returns a 404 when refreshing the
> > alerts-list page (merrimanr) closes apache/metron#808
> > METRON-1262 Unable to add comment for a alert in a meta-alert
> > (merrimanr) closes apache/metron#806
> > METRON-1263 Start Alerts UI service after Metron REST
> > (anandsubbu via nickwallen) closes apache/metron#807
> > METRON-1255 MetaAlert search is not filtering on status
> > (merrimanr) closes apache/metron#802
> > METRON-1249 Improve Metron MPack Service Checks (nickwallen)
> > closes apache/metron#799
> > METRON-1237 address javadoc warnings in metron-maas-common
> > (dbist via james-sirota) closes apache/metron#792
> > METRON-1240 address javadoc warnings in metron-platform and
> > metron-analytics (dbist via james-sirota) closes apache/metron#794
> > METRON-1226 Searching Can Errantly Query the Wrong Indices
> > (nickwallen) closes apache/metron#793
> > METRON-1081 Fix Alerts and Ops UI Notices file (james-sirota)
> > closes apache/metron#682
> > METRON-1123 Add group by option using faceted search
> > capabilities of metron-rest-api (iraghumitra via james-sirota) closes
> > apache/metron#768
> > METRON-1223 Add support to add comments for alerts
> > (iraghumitra via james-sirota) closes apache/metron#788
> > METRON-1083 Add filters using faceted search capabilities of
> > metron-rest-api (iraghumitra via james-sirota) closes apache/metron#710
> > METRON-1232 Alert status changes are not reflected in list
> > view (iraghumitra via merrimanr) closes apache/metron#787
> > METRON-1247 REST search and findOne endpoints return
> > unexpected or incorrect results for guids (justinleet) closes
> > apache/metron#798
> > METRON-1235: Document the properties pulled from the global
> > configuration closes apache/incubator-metron#791
> > METRON-1234: fix for WARNING 'dependencies.dependency.(
> > groupId:artifactId:type:classifier)' must be unique:
> > org.apache.hadoop:hadoop-yarn-api:jar (dbist via mmiklavc) closes
> > apache/metron#790
> > METRON-1222: fix warning for The expression ${parent.version}
> > is deprecated. Please use ${project.parent.version} instead. (dbist via
> > mmiklavc) closes apache/metron#782
> > METRON-1220 Create documentation around alert nested field
> > (justinleet) closes apache/metron#780
> > METRON-1229 Management UI type is part of the declarations of
> > 2 modules (merrimanr) closes apache/metron#784
> > METRON-1228: Configuration Management PUSH immediately does
> > DUMP after (mmiklavc via mmiklavc) closes apache/metron#783
> > METRON-1218 Metron REST should return better error messages
> > (merrimanr) closes apache/metron#779
> > METRON-1161 Add ability to edit parser command line options
> in
> > the management UI (merrimanr) closes apache/metron#737
> > METRON-1209: Make stellar repl take logging properties, like
> > other CLI apps in metron closes apache/incubator-metron#772
> > METRON-1059 address checkstyle warning AvoidStarImport in
> > metron-stellar (dbist via ottobackwards) closes apache/metron#664
> > METRON-1204 UI does not time out after being idle, but stops
> > functioning (merrimanr) closes apache/metron#771
> > METRON-1052: Add forensic similarity hash functions to
> Stellar
> > closes apache/incubator-metron#781
> > METRON-632: Added validation of "shew.enrichmentType" and
> > "shew.keyColumns" closes apache/incubator-metron#732
> > METRON-1194 Add Profiler Debug Functions to Profiler README
> > (nickwallen via ottobackwards) closes apache/metron#765
> > METRON-1055 Metron 0.4.0 manual installation guide for CentOS
> > 6 updates (lvets via ottobackwards) closes apache/metron#661
> > METRON-1079 STELLAR NaN should be a keyword (ottobackwards)
> > closes apache/metron#681
> > METRON-1085 Add REST endpoint to save a user profile for the
> > Alerts UI (merrimanr) closes apache/metron#694
> > METRON-1208 MPack for Alerts UI (merrimanr) closes
> > apache/metron#778
> > METRON-1207 Make RPMs for Alerts UI (merrimanr) closes
> > apache/metron#777
> > METRON-1215 Fix link to RPMs chapter (DimDroll via
> justinleet)
> > closes apache/metron#776
> > METRON-1206 Make alerts UI conform to ops UI for install
> > (merrimanr) closes apache/metron#773
> > METRON-1195 Meta alerts improperly handle updates to
> non-alert
> > fields (justinleet) closes apache/metron#766
> > METRON-1189 Add alert escalation to the Alerts UI (merrimanr)
> > closes apache/metron#762
> > METRON-1156 Simulate Triage Rules in the Stellar REPL
> > (nickwallen) closes apache/metron#733
> > METRON-1198 Pycapa - No such configuration property
> > 'sasl.kerberos.principal' (nickwallen) closes apache/metron#769
> > METRON-1202 ElasticsearchDao Has extraneous sleep call
> > (justinleet) closes apache/metron#770
> > METRON-938 "service metron-rest start <password>" does not
> > work on CentOS 7. (justinleet) closes apache/metron#757
> > METRON-1182 Refactor Code in alert list to accommodate new
> > view types (iraghumitra via merrimanr) closes apache/metron#756
> > METRON-1188: Ambari global configuration management
> (mmiklavc)
> > closes apache/metron#760
> > METRON-1191 update public web site to point at 0.4.1 new
> > release (mattf-horton) closes apache/metron#764
> > METRON-1063 address javadoc warnings in metron-stellar (dbist
> > via ottobackwards) closes apache/metron#668
> > METRON-1190 Fix Meta Alert Type handling in calculation of
> > scores (justinleet) closes apache/metron#763
> > METRON-1187 Indexing/Profiler Kafka ACL Groups Not Setup
> > Correctly (nickwallen) closes apache/metron#759
> > METRON-1185: Stellar REPL does not work on a kerberized
> > cluster when calling functions interacting with HBase closes
> > apache/incubator-metron#755
> > METRON-1186: Profiler Functions use classutils from shaded
> > storm closes apache/incubator-metron#758
> > METRON-1173: Fix pointers to old stellar docs closes
> > apache/incubator-metron#746
> > METRON-1179: Make STATS_ADD to take a list closes
> > apache/incubator-metron#750
> > METRON-1180: Make Stellar Shell accept zookeeper quorum as a
> > CSV list and not require a port closes apache/incubator-metron#751
> > METRON-1183 Improve KDC Setup Instructions (nickwallen)
> closes
> > apache/metron#753
> > METRON-1177 Stale running topologies seen post-kerberization
> > and cause exceptions (nickwallen) closes apache/metron#748
> > METRON-1158 Build backend for grouping alerts into meta
> alerts
> > (justinleet) closes apache/metron#734
> > METRON-1146: Add ability to parse JSON string into JSONObject
> > for stellar closes apache/incubator-metron#727
> > METRON-1176 REST: HDFS Service should support setting
> > permissions on files when writing (ottobackwards) closes
> apache/metron#749
> > METRON-1114 Add group by capabilities to search REST endpoint
> > (merrimanr) closes apache/metron#702
> > METRON-1167 Define Session Specific Global Configuration
> > Values in the REPL (nickwallen) closes apache/metron#740
> > METRON-1171: Better validation for the SUBSTRING stellar
> > function closes apache/incubator-metron#745
> >
> >
> >
> > On 11/17/17, 11:59 AM, "Nick Allen" <ni...@nickallen.org> wrote:
> >
> > I just wanted to send an update on where we are at. We've
> > gotten a lot
> > done here recently as you can see below.
> >
> > ✓ DONE (1) First, METRON-1289 needs to go in. This one was
> > a fairly big
> > effort and I am hearing that we are pretty close.
> >
> > ✓ DONE (2) METRON-1294 fixes an issue in how field types
> are
> > looked-up.
> >
> > ✓ DONE (3) METRON-1290 is next. While this may have been
> > fixed in
> > M-1289, there may be some test cases we want from this PR.
> >
> > ✓ DONE (4) METRON-1301 addresses a problem with the sorting
> > logic.
> >
> > ✓ DONE (5) METRON-1291 fixes an issue with escalation of
> > metaalerts.
> >
> > (6) That leads us to Raghu's UI work in METRON-1252. This
> > introduces the
> > UI bits that depend on all the previous backend work.
> >
> > (7) At this point, we should have our best effort at
> running
> > Metaalerts
> > on Elasticsearch 2.x. I propose that we cut a release here.
> >
> > (8) After we cut the release, we can introduce the work for
> > ES 5.x in
> > METRON-939. I know we will need lots of help testing and
> > reviewing this
> > one.
> >
> >
> >
> > We also have an outstanding question that needs resolved
> > BEFORE we
> > release. We need to come to a consensus on how to release
> > having moved our
> > Bro Plugin to a separate repo. I don't think we've heard
> from
> > everyone on
> > this. I'd urge everyone to chime in so we can choose a path
> > forward.
> >
> > If anyone is totally confused in regards to that discussion,
> I
> > can try and
> > send an options summary again as a separate discuss thread.
> > The original
> > chain was somewhere around here [1].
> >
> > [1]
> > https://lists.apache.org/thread.html/
> > 54a4474881b97e559df24728b3a0e923a58345a282451085eef832ef@%
> > 3Cdev.metron.apache.org%3E
> >
> >
> >
> > On Wed, Nov 15, 2017 at 10:04 AM, Nick Allen <
> > nick@nickallen.org> wrote:
> >
> > > Hi Guys -
> > >
> > > I want to follow-up on this discussion. It sounds like
> most
> > people are in
> > > agreement with the general approach.
> > >
> > > A lot of people have been working hard on Metaalerts and
> > Elasticsearch. I
> > > have checked-in with those doing the heavy lifting and have
> > compiled a more
> > > detailed plan based on where we are at now. To the best of
> > my knowledge
> > > here is the plan of attack for finishing out this effort.
> > >
> > > (1) First, METRON-1289 needs to go in. This one was a
> > fairly big effort
> > > and I am hearing that we are pretty close.
> > >
> > > (2) METRON-1294 fixes an issue in how field types are
> > looked-up.
> > >
> > > (3) METRON-1290 is next. While this may have been fixed
> > in M-1289,
> > > there may be some test cases we want from this PR.
> > >
> > > (4) METRON-1301 addresses a problem with the sorting
> logic.
> > >
> > > (5) METRON-1291 fixes an issue with escalation of
> > metaalerts.
> > >
> > > (6) That leads us to Raghu's UI work in METRON-1252.
> This
> > introduces
> > > the UI bits that depend on all the previous backend work.
> > >
> > > (7) At this point, we should have our best effort at
> > running Metaalerts
> > > on Elasticsearch 2.x. I propose that we cut a release here.
> > >
> > > (8) After we cut the release, we can introduce the work
> > for ES 5.x in
> > > METRON-939. I know we will need lots of help testing and
> > reviewing this
> > > one.
> > >
> > > Please correct me if I am wrong. I will try and send out
> > updates as we
> > > make progress.
> > >
> > >
> > >
> > >
> > >
> > > On Mon, Nov 6, 2017 at 1:03 PM, Zeolla@GMail.com <
> > zeolla@gmail.com> wrote:
> > >
> > >> I agree, I think it's very reasonable to move in line with
> > Nick's
> > >> proposal. I would also suggest that we outline what the
> > target versions
> > >> would be to add in the METRON-777 components, since it has
> > been functional
> > >> for a very long time but not reviewed and has some really
> > rockstar
> > >> improvements.
> > >>
> > >> Jon
> > >>
> > >> On Mon, Nov 6, 2017 at 12:56 PM Otto Fowler <
> > ottobackwards@gmail.com>
> > >> wrote:
> > >>
> > >> > I think the ES cutover should be the start of the 0.5.x
> > series, and we
> > >> > continue on with 0.4.x for the
> > >> > metadata improvements etc. We could chose to focus
> > 0.5.x’s first
> > >> releases
> > >> > on not only ES but
> > >> > getting a handle on kibana and the mpack situation as
> > well.
> > >> >
> > >> >
> > >> >
> > >> >
> > >> > On November 6, 2017 at 12:48:45, Michael Miklavcic (
> > >> > michael.miklavcic@gmail.com) wrote:
> > >> >
> > >> > I agree with your proposal, Nick. I think having a
> > stabilizing release
> > >> > prior to upgrading ES/Kibana makes sense.
> > >> >
> > >> > On Mon, Nov 6, 2017 at 9:16 AM, Nick Allen <
> > nick@nickallen.org> wrote:
> > >> >
> > >> > > I would like to start a discussion around upcoming
> > releases. We have a
> > >> > > couple separate significant tracks of work that we
> need
> > to reconcile
> > >> in
> > >> > our
> > >> > > release schedule.
> > >> > >
> > >> > > (1) We have had (and have in review) a good number of
> > bug fixes
> > >> required
> > >> > to
> > >> > > support Metaalerts on the existing Elasticsearch 2.x
> > infrastructure.
> > >> > >
> > >> > >
> > >> > > (2) We also have ongoing work to upgrade our
> > infrastructure to
> > >> > > Elasticsearch 5.x, which will not be backwards
> > compatible.
> > >> > >
> > >> > >
> > >> > > I would like to see a release that has our best work
> on
> > ES 2.x before
> > >> we
> > >> > > migrate to 5.x. I would propose the following.
> > >> > >
> > >> > > Release N+1: Introduce Metaalerts running on ES 2.x
> > >> > >
> > >> > > Release N+2: Cut-over to ES 5.x
> > >> > >
> > >> > >
> > >> > > (Q) Is it worth cutting a separate release for ES 2.x?
> > Is there a
> > >> better
> > >> > > way to handle the cut-over to 5.x?
> > >> > >
> > >> >
> > >> --
> > >>
> > >> Jon
> > >>
> > >
> > >
> >
> >
> >
> >
> >
> >
> >
> >
>
--
Jon
Re: [DISCUSS] Upcoming Release
Posted by Otto Fowler <ot...@gmail.com>.
Steps
- Downloaded tar.gz’s, asc files and KEYS
- Verified signing of both tar.gz’s
- searched for rouge 0.4.1 entries
- verified the main pom.xml
- built :
mvn clean && time mvn -q -T 2C -DskipTests install && time mvn -q -T
2C surefire:test@unit-tests && time mvn -q
surefire:test@integration-tests && time mvn -q test --projects
metron-interface/metron-config && time build_utils/verify_licenses.sh
Found rat error:
*****************************************************
Summary
-------
Generated at: 2017-12-08T16:33:27-05:00
Notes: 3
Binaries: 193
Archives: 0
Standards: 75
Apache Licensed: 74
Generated Documents: 0
JavaDocs are generated, thus a license header is optional.
Generated files do not require license headers.
1 Unknown Licenses
*****************************************************
Files with unapproved licenses:
/Users/batman/tmp/release_ver/apache-metron-0.4.2-rc1/metron-interface/metron-alerts/dist/styles.f56deed131e58bd7ee04.bundle.css
*****************************************************
*****************************************************
Summary
-------
Generated at: 2017-12-08T16:33:27-05:00
Notes: 3
Binaries: 193
Archives: 0
Standards: 75
Apache Licensed: 74
Generated Documents: 0
JavaDocs are generated, thus a license header is optional.
Generated files do not require license headers.
1 Unknown Licenses
*****************************************************
Files with unapproved licenses:
/Users/ottofowler/tmp/release_ver/apache-metron-0.4.2-rc1/metron-interface/metron-alerts/dist/styles.f56deed131e58bd7ee04.bundle.css
*****************************************************
On December 8, 2017 at 04:34:24, Matt Foley (mattf@apache.org) wrote:
Colleagues,
I’ve posted Metron-0.4.2-RC1 and Metron-bro-plugin-kafka-0.1 to
https://dist.apache.org/repos/dist/dev/metron/0.4.2-RC1/
Given the complexity of this RC, I’d appreciate if a couple people would be
willing to kick the tires before we put it up for a vote.
I will myself be going thru the Verify Build process this weekend, as I
won’t be able to do it Friday.
Thanks,
--Matt
On 12/4/17, 2:05 PM, "Zeolla@GMail.com" <ze...@gmail.com> wrote:
Can we resolve the conversation regarding the second repo? I was waiting
to get more input/preferences from people There's also a documentation
update that fixes a few broken Stellar docs that already has aa +1, I just
need to merge it.
Jon
On Mon, Dec 4, 2017, 17:01 Casey Stella <ce...@gmail.com> wrote:
> I would be in favor of a release at this point.
>
> On Mon, Dec 4, 2017 at 4:57 PM, Matt Foley <ma...@apache.org> wrote:
>
> > Hey all,
> > I see METRON-1252 was resolved over the weekend. Shall I go ahead and
> > start the process with 0.4.2 release?
> > Does anyone have any commits they feel strongly should go in before
0.4.2
> > is done, or are we ready to call it good?
> >
> > I believe there is consensus the 0.4.2 release should include a release
> of
> > the current state of the metron-bro-plugin-kafka. I will continue the
> > discussion in that thread as to the process for accomplishing that, but
> > plan on it happening.
> >
> > Regards,
> > --Matt
> >
> > On 11/26/17, 6:26 PM, "Matt Foley" <ma...@apache.org> wrote:
> >
> > Hope everyone (at least in the U.S.) had a great Thanksgiving
> holiday.
> > Regarding status of the release effort, still pending METRON-1252, so
> > not making the release branch yet.
> >
> > Regards,
> > --Matt
> >
> > On 11/17/17, 1:32 PM, "Matt Foley" <ma...@apache.org> wrote:
> >
> > (With release manager hat on)
> >
> > The community has proposed a release of Metron in the near
> future,
> > focusing on Meta-alerts running in Elasticsearch.
> > Congrats on getting so many of the below already done. At this
> > point, only METRON-1252, and the discussion of how to handle joint
> release
> > of the Metron bro plugin, remain as gating items for the release. I
> > project these will be resolved next week, so let’s propose the
following:
> >
> > Sometime next week, after the last bits are done, I’ll start the
> > release process and create the release branch.
> >
> > The proposed new version will be 0.4.2, unless there are backward
> > incompatible changes that support making it 0.5.0.
> > Currently there are NO included Jiras labeled
> > ‘backward-incompatible’, nor having Docs Text indicating so.
> > ***If anyone knows that some of the commits included since 0.4.1
> > introduce backward incompatibility, please say so now on this thread,
and
> > mark the Jira as such.***
> >
> > The 90 or so jiras/commits already in master branch since 0.4.1
> > are listed below.
> > Thanks,
> > --Matt
> >
> > METRON-1301 Alerts UI - Sorting on Triage Score Unexpectedly
> > Filters Some Records (nickwallen) closes apache/metron#832
> > METRON-1294 IP addresses are not formatted correctly in facet
> > and group results (merrimanr) closes apache/metron#827
> > METRON-1291 Kafka produce REST endpoint does not work in a
> > Kerberized cluster (merrimanr) closes apache/metron#826
> > METRON-1290 Only first 10 alerts are update when a MetaAlert
> > status is changed to inactive (justinleet) closes apache/metron#842
> > METRON-1311 Service Check Should Check Elasticsearch Index
> > Templates (nickwallen) closes apache/metron#839
> > METRON-1289 Alert fields are lost when a MetaAlert is created
> > (merrimanr) closes apache/metron#824
> > METRON-1309 Change metron-deployment to pull the plugin from
> > apache/metron-bro-plugin-kafka (JonZeolla) closes apache/metron#837
> > METRON-1310 Template Delete Action Deletes Search Indices
> > (nickwallen) closes apache/metron#838
> > METRON-1275: Fix Metron Documentation closes
> > apache/incubator-metron#833
> > METRON-1295 Unable to Configure Logging for REST API
> > (nickwallen) closes apache/metron#828
> > METRON-1307 Force install of java8 since java9 does not
> appear
> > to work with the scripts (brianhurley via ottobackwards) closes
> > apache/metron#835
> > METRON-1296 Full Dev Fails to Deploy Index Templates
> > (nickwallen via cestella) closes apache/incubator-metron#829
> > METRON-1281 Remove hard-coded indices from the Alerts UI
> > (merrimanr) closes apache/metron#821
> > METRON-1287 Full Dev Fails When Installing EPEL Repository
> > (nickwallen) closes apache/metron#820
> > METRON-1267 Alerts UI returns a 404 when refreshing the
> > alerts-list page (iraghumitra via merrimanr) closes apache/metron#819
> > METRON-1283 Install Elasticsearch template as a part of the
> > mpack startup scripts (anandsubbu via nickwallen) closes
> apache/metron#817
> > METRON-1254: Conditionals as map keys do not function in
> > Stellar closes apache/incubator-metron#801
> > METRON-1261 Apply bro security patch (JonZeolla via
> > ottobackwards) closes apache/metron#805
> > METRON-1284 Remove extraneous dead query in ElasticsearchDao
> > (justinleet) closes apache/metron#818
> > METRON-1270: fix for warnings missing @return tag argument in
> > metron-analytics/metron-profiler-common and metron-profiler-client
closes
> > apache/incubator-metron#810
> > METRON-1272 Hide child alerts from searches and grouping if
> > they belong to meta alerts (justinleet) closes apache/metron#811
> > METRON-1224 Add time range selection to search control
> > (iraghumitra via james-sirota) closes apache/metron#796
> > METRON-1280 0.4.1 -> 0.4.2 missed a couple of projects
> > (cestella via justinleet) closes apache/metron#816
> > METRON-1243: Add a REST endpoint which allows us to get a
> list
> > of all indice closes apache/incubator-metron#797
> > METRON-1196 Increment master version number to 0.4.2 for
> > on-going development (mattf-horton) closes apache/metron#767
> > METRON-1278 Strip "Build Status" widget from root
> > README.md in site-book build (mattf-horton) closes apache/metron#815
> > METRON-1274 Master has failure in
> > StormControllerIntegrationTest (merrimanr) closes apache/metron#813
> > METRON-1266 Profiler - SASL Authentication Failed
> (nickwallen)
> > closes apache/metron#809
> > METRON-1260 Include Alerts UI in Ambari Service Check
> > (nickwallen) closes apache/metron#804
> > METRON-1251: Typo and formatting fixes for metron-rest README
> > closes apache/incubator-metron#800
> > METRON-1241: Enable the REST API to use a cache for the
> > zookeeper config similar to the Bolts closes
apache/incubator-metron#795
> > METRON-1267 Alerts UI returns a 404 when refreshing the
> > alerts-list page (merrimanr) closes apache/metron#808
> > METRON-1262 Unable to add comment for a alert in a meta-alert
> > (merrimanr) closes apache/metron#806
> > METRON-1263 Start Alerts UI service after Metron REST
> > (anandsubbu via nickwallen) closes apache/metron#807
> > METRON-1255 MetaAlert search is not filtering on status
> > (merrimanr) closes apache/metron#802
> > METRON-1249 Improve Metron MPack Service Checks (nickwallen)
> > closes apache/metron#799
> > METRON-1237 address javadoc warnings in metron-maas-common
> > (dbist via james-sirota) closes apache/metron#792
> > METRON-1240 address javadoc warnings in metron-platform and
> > metron-analytics (dbist via james-sirota) closes apache/metron#794
> > METRON-1226 Searching Can Errantly Query the Wrong Indices
> > (nickwallen) closes apache/metron#793
> > METRON-1081 Fix Alerts and Ops UI Notices file (james-sirota)
> > closes apache/metron#682
> > METRON-1123 Add group by option using faceted search
> > capabilities of metron-rest-api (iraghumitra via james-sirota) closes
> > apache/metron#768
> > METRON-1223 Add support to add comments for alerts
> > (iraghumitra via james-sirota) closes apache/metron#788
> > METRON-1083 Add filters using faceted search capabilities of
> > metron-rest-api (iraghumitra via james-sirota) closes apache/metron#710
> > METRON-1232 Alert status changes are not reflected in list
> > view (iraghumitra via merrimanr) closes apache/metron#787
> > METRON-1247 REST search and findOne endpoints return
> > unexpected or incorrect results for guids (justinleet) closes
> > apache/metron#798
> > METRON-1235: Document the properties pulled from the global
> > configuration closes apache/incubator-metron#791
> > METRON-1234: fix for WARNING 'dependencies.dependency.(
> > groupId:artifactId:type:classifier)' must be unique:
> > org.apache.hadoop:hadoop-yarn-api:jar (dbist via mmiklavc) closes
> > apache/metron#790
> > METRON-1222: fix warning for The expression ${parent.version}
> > is deprecated. Please use ${project.parent.version} instead. (dbist via
> > mmiklavc) closes apache/metron#782
> > METRON-1220 Create documentation around alert nested field
> > (justinleet) closes apache/metron#780
> > METRON-1229 Management UI type is part of the declarations of
> > 2 modules (merrimanr) closes apache/metron#784
> > METRON-1228: Configuration Management PUSH immediately does
> > DUMP after (mmiklavc via mmiklavc) closes apache/metron#783
> > METRON-1218 Metron REST should return better error messages
> > (merrimanr) closes apache/metron#779
> > METRON-1161 Add ability to edit parser command line options
> in
> > the management UI (merrimanr) closes apache/metron#737
> > METRON-1209: Make stellar repl take logging properties, like
> > other CLI apps in metron closes apache/incubator-metron#772
> > METRON-1059 address checkstyle warning AvoidStarImport in
> > metron-stellar (dbist via ottobackwards) closes apache/metron#664
> > METRON-1204 UI does not time out after being idle, but stops
> > functioning (merrimanr) closes apache/metron#771
> > METRON-1052: Add forensic similarity hash functions to
> Stellar
> > closes apache/incubator-metron#781
> > METRON-632: Added validation of "shew.enrichmentType" and
> > "shew.keyColumns" closes apache/incubator-metron#732
> > METRON-1194 Add Profiler Debug Functions to Profiler README
> > (nickwallen via ottobackwards) closes apache/metron#765
> > METRON-1055 Metron 0.4.0 manual installation guide for CentOS
> > 6 updates (lvets via ottobackwards) closes apache/metron#661
> > METRON-1079 STELLAR NaN should be a keyword (ottobackwards)
> > closes apache/metron#681
> > METRON-1085 Add REST endpoint to save a user profile for the
> > Alerts UI (merrimanr) closes apache/metron#694
> > METRON-1208 MPack for Alerts UI (merrimanr) closes
> > apache/metron#778
> > METRON-1207 Make RPMs for Alerts UI (merrimanr) closes
> > apache/metron#777
> > METRON-1215 Fix link to RPMs chapter (DimDroll via
> justinleet)
> > closes apache/metron#776
> > METRON-1206 Make alerts UI conform to ops UI for install
> > (merrimanr) closes apache/metron#773
> > METRON-1195 Meta alerts improperly handle updates to
> non-alert
> > fields (justinleet) closes apache/metron#766
> > METRON-1189 Add alert escalation to the Alerts UI (merrimanr)
> > closes apache/metron#762
> > METRON-1156 Simulate Triage Rules in the Stellar REPL
> > (nickwallen) closes apache/metron#733
> > METRON-1198 Pycapa - No such configuration property
> > 'sasl.kerberos.principal' (nickwallen) closes apache/metron#769
> > METRON-1202 ElasticsearchDao Has extraneous sleep call
> > (justinleet) closes apache/metron#770
> > METRON-938 "service metron-rest start <password>" does not
> > work on CentOS 7. (justinleet) closes apache/metron#757
> > METRON-1182 Refactor Code in alert list to accommodate new
> > view types (iraghumitra via merrimanr) closes apache/metron#756
> > METRON-1188: Ambari global configuration management
> (mmiklavc)
> > closes apache/metron#760
> > METRON-1191 update public web site to point at 0.4.1 new
> > release (mattf-horton) closes apache/metron#764
> > METRON-1063 address javadoc warnings in metron-stellar (dbist
> > via ottobackwards) closes apache/metron#668
> > METRON-1190 Fix Meta Alert Type handling in calculation of
> > scores (justinleet) closes apache/metron#763
> > METRON-1187 Indexing/Profiler Kafka ACL Groups Not Setup
> > Correctly (nickwallen) closes apache/metron#759
> > METRON-1185: Stellar REPL does not work on a kerberized
> > cluster when calling functions interacting with HBase closes
> > apache/incubator-metron#755
> > METRON-1186: Profiler Functions use classutils from shaded
> > storm closes apache/incubator-metron#758
> > METRON-1173: Fix pointers to old stellar docs closes
> > apache/incubator-metron#746
> > METRON-1179: Make STATS_ADD to take a list closes
> > apache/incubator-metron#750
> > METRON-1180: Make Stellar Shell accept zookeeper quorum as a
> > CSV list and not require a port closes apache/incubator-metron#751
> > METRON-1183 Improve KDC Setup Instructions (nickwallen)
> closes
> > apache/metron#753
> > METRON-1177 Stale running topologies seen post-kerberization
> > and cause exceptions (nickwallen) closes apache/metron#748
> > METRON-1158 Build backend for grouping alerts into meta
> alerts
> > (justinleet) closes apache/metron#734
> > METRON-1146: Add ability to parse JSON string into JSONObject
> > for stellar closes apache/incubator-metron#727
> > METRON-1176 REST: HDFS Service should support setting
> > permissions on files when writing (ottobackwards) closes
> apache/metron#749
> > METRON-1114 Add group by capabilities to search REST endpoint
> > (merrimanr) closes apache/metron#702
> > METRON-1167 Define Session Specific Global Configuration
> > Values in the REPL (nickwallen) closes apache/metron#740
> > METRON-1171: Better validation for the SUBSTRING stellar
> > function closes apache/incubator-metron#745
> >
> >
> >
> > On 11/17/17, 11:59 AM, "Nick Allen" <ni...@nickallen.org> wrote:
> >
> > I just wanted to send an update on where we are at. We've
> > gotten a lot
> > done here recently as you can see below.
> >
> > ✓ DONE (1) First, METRON-1289 needs to go in. This one was
> > a fairly big
> > effort and I am hearing that we are pretty close.
> >
> > ✓ DONE (2) METRON-1294 fixes an issue in how field types
> are
> > looked-up.
> >
> > ✓ DONE (3) METRON-1290 is next. While this may have been
> > fixed in
> > M-1289, there may be some test cases we want from this PR.
> >
> > ✓ DONE (4) METRON-1301 addresses a problem with the sorting
> > logic.
> >
> > ✓ DONE (5) METRON-1291 fixes an issue with escalation of
> > metaalerts.
> >
> > (6) That leads us to Raghu's UI work in METRON-1252. This
> > introduces the
> > UI bits that depend on all the previous backend work.
> >
> > (7) At this point, we should have our best effort at
> running
> > Metaalerts
> > on Elasticsearch 2.x. I propose that we cut a release here.
> >
> > (8) After we cut the release, we can introduce the work for
> > ES 5.x in
> > METRON-939. I know we will need lots of help testing and
> > reviewing this
> > one.
> >
> >
> >
> > We also have an outstanding question that needs resolved
> > BEFORE we
> > release. We need to come to a consensus on how to release
> > having moved our
> > Bro Plugin to a separate repo. I don't think we've heard
> from
> > everyone on
> > this. I'd urge everyone to chime in so we can choose a path
> > forward.
> >
> > If anyone is totally confused in regards to that discussion,
> I
> > can try and
> > send an options summary again as a separate discuss thread.
> > The original
> > chain was somewhere around here [1].
> >
> > [1]
> > https://lists.apache.org/thread.html/
> > 54a4474881b97e559df24728b3a0e923a58345a282451085eef832ef@%
> > 3Cdev.metron.apache.org%3E
> >
> >
> >
> > On Wed, Nov 15, 2017 at 10:04 AM, Nick Allen <
> > nick@nickallen.org> wrote:
> >
> > > Hi Guys -
> > >
> > > I want to follow-up on this discussion. It sounds like
> most
> > people are in
> > > agreement with the general approach.
> > >
> > > A lot of people have been working hard on Metaalerts and
> > Elasticsearch. I
> > > have checked-in with those doing the heavy lifting and have
> > compiled a more
> > > detailed plan based on where we are at now. To the best of
> > my knowledge
> > > here is the plan of attack for finishing out this effort.
> > >
> > > (1) First, METRON-1289 needs to go in. This one was a
> > fairly big effort
> > > and I am hearing that we are pretty close.
> > >
> > > (2) METRON-1294 fixes an issue in how field types are
> > looked-up.
> > >
> > > (3) METRON-1290 is next. While this may have been fixed
> > in M-1289,
> > > there may be some test cases we want from this PR.
> > >
> > > (4) METRON-1301 addresses a problem with the sorting
> logic.
> > >
> > > (5) METRON-1291 fixes an issue with escalation of
> > metaalerts.
> > >
> > > (6) That leads us to Raghu's UI work in METRON-1252.
> This
> > introduces
> > > the UI bits that depend on all the previous backend work.
> > >
> > > (7) At this point, we should have our best effort at
> > running Metaalerts
> > > on Elasticsearch 2.x. I propose that we cut a release here.
> > >
> > > (8) After we cut the release, we can introduce the work
> > for ES 5.x in
> > > METRON-939. I know we will need lots of help testing and
> > reviewing this
> > > one.
> > >
> > > Please correct me if I am wrong. I will try and send out
> > updates as we
> > > make progress.
> > >
> > >
> > >
> > >
> > >
> > > On Mon, Nov 6, 2017 at 1:03 PM, Zeolla@GMail.com <
> > zeolla@gmail.com> wrote:
> > >
> > >> I agree, I think it's very reasonable to move in line with
> > Nick's
> > >> proposal. I would also suggest that we outline what the
> > target versions
> > >> would be to add in the METRON-777 components, since it has
> > been functional
> > >> for a very long time but not reviewed and has some really
> > rockstar
> > >> improvements.
> > >>
> > >> Jon
> > >>
> > >> On Mon, Nov 6, 2017 at 12:56 PM Otto Fowler <
> > ottobackwards@gmail.com>
> > >> wrote:
> > >>
> > >> > I think the ES cutover should be the start of the 0.5.x
> > series, and we
> > >> > continue on with 0.4.x for the
> > >> > metadata improvements etc. We could chose to focus
> > 0.5.x’s first
> > >> releases
> > >> > on not only ES but
> > >> > getting a handle on kibana and the mpack situation as
> > well.
> > >> >
> > >> >
> > >> >
> > >> >
> > >> > On November 6, 2017 at 12:48:45, Michael Miklavcic (
> > >> > michael.miklavcic@gmail.com) wrote:
> > >> >
> > >> > I agree with your proposal, Nick. I think having a
> > stabilizing release
> > >> > prior to upgrading ES/Kibana makes sense.
> > >> >
> > >> > On Mon, Nov 6, 2017 at 9:16 AM, Nick Allen <
> > nick@nickallen.org> wrote:
> > >> >
> > >> > > I would like to start a discussion around upcoming
> > releases. We have a
> > >> > > couple separate significant tracks of work that we
> need
> > to reconcile
> > >> in
> > >> > our
> > >> > > release schedule.
> > >> > >
> > >> > > (1) We have had (and have in review) a good number of
> > bug fixes
> > >> required
> > >> > to
> > >> > > support Metaalerts on the existing Elasticsearch 2.x
> > infrastructure.
> > >> > >
> > >> > >
> > >> > > (2) We also have ongoing work to upgrade our
> > infrastructure to
> > >> > > Elasticsearch 5.x, which will not be backwards
> > compatible.
> > >> > >
> > >> > >
> > >> > > I would like to see a release that has our best work
> on
> > ES 2.x before
> > >> we
> > >> > > migrate to 5.x. I would propose the following.
> > >> > >
> > >> > > Release N+1: Introduce Metaalerts running on ES 2.x
> > >> > >
> > >> > > Release N+2: Cut-over to ES 5.x
> > >> > >
> > >> > >
> > >> > > (Q) Is it worth cutting a separate release for ES 2.x?
> > Is there a
> > >> better
> > >> > > way to handle the cut-over to 5.x?
> > >> > >
> > >> >
> > >> --
> > >>
> > >> Jon
> > >>
> > >
> > >
> >
> >
> >
> >
> >
> >
> >
> >
>
--
Jon
Re: [DISCUSS] Upcoming Release
Posted by "Zeolla@GMail.com" <ze...@gmail.com>.
I poked around the RC briefly and spun up full dev with and without
sensors, no issues so far.
Jon
On Fri, Dec 8, 2017 at 4:34 AM Matt Foley <ma...@apache.org> wrote:
> Colleagues,
> I’ve posted Metron-0.4.2-RC1 and Metron-bro-plugin-kafka-0.1 to
> https://dist.apache.org/repos/dist/dev/metron/0.4.2-RC1/
>
> Given the complexity of this RC, I’d appreciate if a couple people would
> be willing to kick the tires before we put it up for a vote.
>
> I will myself be going thru the Verify Build process this weekend, as I
> won’t be able to do it Friday.
>
> Thanks,
> --Matt
>
>
> On 12/4/17, 2:05 PM, "Zeolla@GMail.com" <ze...@gmail.com> wrote:
>
> Can we resolve the conversation regarding the second repo? I was
> waiting
> to get more input/preferences from people There's also a documentation
> update that fixes a few broken Stellar docs that already has aa +1, I
> just
> need to merge it.
>
> Jon
>
> On Mon, Dec 4, 2017, 17:01 Casey Stella <ce...@gmail.com> wrote:
>
> > I would be in favor of a release at this point.
> >
> > On Mon, Dec 4, 2017 at 4:57 PM, Matt Foley <ma...@apache.org> wrote:
> >
> > > Hey all,
> > > I see METRON-1252 was resolved over the weekend. Shall I go ahead
> and
> > > start the process with 0.4.2 release?
> > > Does anyone have any commits they feel strongly should go in
> before 0.4.2
> > > is done, or are we ready to call it good?
> > >
> > > I believe there is consensus the 0.4.2 release should include a
> release
> > of
> > > the current state of the metron-bro-plugin-kafka. I will continue
> the
> > > discussion in that thread as to the process for accomplishing
> that, but
> > > plan on it happening.
> > >
> > > Regards,
> > > --Matt
> > >
> > > On 11/26/17, 6:26 PM, "Matt Foley" <ma...@apache.org> wrote:
> > >
> > > Hope everyone (at least in the U.S.) had a great Thanksgiving
> > holiday.
> > > Regarding status of the release effort, still pending
> METRON-1252, so
> > > not making the release branch yet.
> > >
> > > Regards,
> > > --Matt
> > >
> > > On 11/17/17, 1:32 PM, "Matt Foley" <ma...@apache.org> wrote:
> > >
> > > (With release manager hat on)
> > >
> > > The community has proposed a release of Metron in the near
> > future,
> > > focusing on Meta-alerts running in Elasticsearch.
> > > Congrats on getting so many of the below already done. At
> this
> > > point, only METRON-1252, and the discussion of how to handle joint
> > release
> > > of the Metron bro plugin, remain as gating items for the release.
> I
> > > project these will be resolved next week, so let’s propose the
> following:
> > >
> > > Sometime next week, after the last bits are done, I’ll
> start the
> > > release process and create the release branch.
> > >
> > > The proposed new version will be 0.4.2, unless there are
> backward
> > > incompatible changes that support making it 0.5.0.
> > > Currently there are NO included Jiras labeled
> > > ‘backward-incompatible’, nor having Docs Text indicating so.
> > > ***If anyone knows that some of the commits included since
> 0.4.1
> > > introduce backward incompatibility, please say so now on this
> thread, and
> > > mark the Jira as such.***
> > >
> > > The 90 or so jiras/commits already in master branch since
> 0.4.1
> > > are listed below.
> > > Thanks,
> > > --Matt
> > >
> > > METRON-1301 Alerts UI - Sorting on Triage Score
> Unexpectedly
> > > Filters Some Records (nickwallen) closes apache/metron#832
> > > METRON-1294 IP addresses are not formatted correctly
> in facet
> > > and group results (merrimanr) closes apache/metron#827
> > > METRON-1291 Kafka produce REST endpoint does not work
> in a
> > > Kerberized cluster (merrimanr) closes apache/metron#826
> > > METRON-1290 Only first 10 alerts are update when a
> MetaAlert
> > > status is changed to inactive (justinleet) closes apache/metron#842
> > > METRON-1311 Service Check Should Check Elasticsearch
> Index
> > > Templates (nickwallen) closes apache/metron#839
> > > METRON-1289 Alert fields are lost when a MetaAlert is
> created
> > > (merrimanr) closes apache/metron#824
> > > METRON-1309 Change metron-deployment to pull the
> plugin from
> > > apache/metron-bro-plugin-kafka (JonZeolla) closes apache/metron#837
> > > METRON-1310 Template Delete Action Deletes Search
> Indices
> > > (nickwallen) closes apache/metron#838
> > > METRON-1275: Fix Metron Documentation closes
> > > apache/incubator-metron#833
> > > METRON-1295 Unable to Configure Logging for REST API
> > > (nickwallen) closes apache/metron#828
> > > METRON-1307 Force install of java8 since java9 does not
> > appear
> > > to work with the scripts (brianhurley via ottobackwards) closes
> > > apache/metron#835
> > > METRON-1296 Full Dev Fails to Deploy Index Templates
> > > (nickwallen via cestella) closes apache/incubator-metron#829
> > > METRON-1281 Remove hard-coded indices from the Alerts
> UI
> > > (merrimanr) closes apache/metron#821
> > > METRON-1287 Full Dev Fails When Installing EPEL
> Repository
> > > (nickwallen) closes apache/metron#820
> > > METRON-1267 Alerts UI returns a 404 when refreshing the
> > > alerts-list page (iraghumitra via merrimanr) closes
> apache/metron#819
> > > METRON-1283 Install Elasticsearch template as a part
> of the
> > > mpack startup scripts (anandsubbu via nickwallen) closes
> > apache/metron#817
> > > METRON-1254: Conditionals as map keys do not function
> in
> > > Stellar closes apache/incubator-metron#801
> > > METRON-1261 Apply bro security patch (JonZeolla via
> > > ottobackwards) closes apache/metron#805
> > > METRON-1284 Remove extraneous dead query in
> ElasticsearchDao
> > > (justinleet) closes apache/metron#818
> > > METRON-1270: fix for warnings missing @return tag
> argument in
> > > metron-analytics/metron-profiler-common and metron-profiler-client
> closes
> > > apache/incubator-metron#810
> > > METRON-1272 Hide child alerts from searches and
> grouping if
> > > they belong to meta alerts (justinleet) closes apache/metron#811
> > > METRON-1224 Add time range selection to search control
> > > (iraghumitra via james-sirota) closes apache/metron#796
> > > METRON-1280 0.4.1 -> 0.4.2 missed a couple of projects
> > > (cestella via justinleet) closes apache/metron#816
> > > METRON-1243: Add a REST endpoint which allows us to
> get a
> > list
> > > of all indice closes apache/incubator-metron#797
> > > METRON-1196 Increment master version number to 0.4.2
> for
> > > on-going development (mattf-horton) closes apache/metron#767
> > > METRON-1278 Strip "Build Status" widget from
> root
> > > README.md in site-book build (mattf-horton) closes
> apache/metron#815
> > > METRON-1274 Master has failure in
> > > StormControllerIntegrationTest (merrimanr) closes apache/metron#813
> > > METRON-1266 Profiler - SASL Authentication Failed
> > (nickwallen)
> > > closes apache/metron#809
> > > METRON-1260 Include Alerts UI in Ambari Service Check
> > > (nickwallen) closes apache/metron#804
> > > METRON-1251: Typo and formatting fixes for metron-rest
> README
> > > closes apache/incubator-metron#800
> > > METRON-1241: Enable the REST API to use a cache for the
> > > zookeeper config similar to the Bolts closes
> apache/incubator-metron#795
> > > METRON-1267 Alerts UI returns a 404 when refreshing the
> > > alerts-list page (merrimanr) closes apache/metron#808
> > > METRON-1262 Unable to add comment for a alert in a
> meta-alert
> > > (merrimanr) closes apache/metron#806
> > > METRON-1263 Start Alerts UI service after Metron REST
> > > (anandsubbu via nickwallen) closes apache/metron#807
> > > METRON-1255 MetaAlert search is not filtering on status
> > > (merrimanr) closes apache/metron#802
> > > METRON-1249 Improve Metron MPack Service Checks
> (nickwallen)
> > > closes apache/metron#799
> > > METRON-1237 address javadoc warnings in
> metron-maas-common
> > > (dbist via james-sirota) closes apache/metron#792
> > > METRON-1240 address javadoc warnings in
> metron-platform and
> > > metron-analytics (dbist via james-sirota) closes apache/metron#794
> > > METRON-1226 Searching Can Errantly Query the Wrong
> Indices
> > > (nickwallen) closes apache/metron#793
> > > METRON-1081 Fix Alerts and Ops UI Notices file
> (james-sirota)
> > > closes apache/metron#682
> > > METRON-1123 Add group by option using faceted search
> > > capabilities of metron-rest-api (iraghumitra via james-sirota)
> closes
> > > apache/metron#768
> > > METRON-1223 Add support to add comments for alerts
> > > (iraghumitra via james-sirota) closes apache/metron#788
> > > METRON-1083 Add filters using faceted search
> capabilities of
> > > metron-rest-api (iraghumitra via james-sirota) closes
> apache/metron#710
> > > METRON-1232 Alert status changes are not reflected in
> list
> > > view (iraghumitra via merrimanr) closes apache/metron#787
> > > METRON-1247 REST search and findOne endpoints return
> > > unexpected or incorrect results for guids (justinleet) closes
> > > apache/metron#798
> > > METRON-1235: Document the properties pulled from the
> global
> > > configuration closes apache/incubator-metron#791
> > > METRON-1234: fix for WARNING 'dependencies.dependency.(
> > > groupId:artifactId:type:classifier)' must be unique:
> > > org.apache.hadoop:hadoop-yarn-api:jar (dbist via mmiklavc) closes
> > > apache/metron#790
> > > METRON-1222: fix warning for The expression
> ${parent.version}
> > > is deprecated. Please use ${project.parent.version} instead.
> (dbist via
> > > mmiklavc) closes apache/metron#782
> > > METRON-1220 Create documentation around alert nested
> field
> > > (justinleet) closes apache/metron#780
> > > METRON-1229 Management UI type is part of the
> declarations of
> > > 2 modules (merrimanr) closes apache/metron#784
> > > METRON-1228: Configuration Management PUSH immediately
> does
> > > DUMP after (mmiklavc via mmiklavc) closes apache/metron#783
> > > METRON-1218 Metron REST should return better error
> messages
> > > (merrimanr) closes apache/metron#779
> > > METRON-1161 Add ability to edit parser command line
> options
> > in
> > > the management UI (merrimanr) closes apache/metron#737
> > > METRON-1209: Make stellar repl take logging
> properties, like
> > > other CLI apps in metron closes apache/incubator-metron#772
> > > METRON-1059 address checkstyle warning AvoidStarImport
> in
> > > metron-stellar (dbist via ottobackwards) closes apache/metron#664
> > > METRON-1204 UI does not time out after being idle, but
> stops
> > > functioning (merrimanr) closes apache/metron#771
> > > METRON-1052: Add forensic similarity hash functions to
> > Stellar
> > > closes apache/incubator-metron#781
> > > METRON-632: Added validation of "shew.enrichmentType"
> and
> > > "shew.keyColumns" closes apache/incubator-metron#732
> > > METRON-1194 Add Profiler Debug Functions to Profiler
> README
> > > (nickwallen via ottobackwards) closes apache/metron#765
> > > METRON-1055 Metron 0.4.0 manual installation guide for
> CentOS
> > > 6 updates (lvets via ottobackwards) closes apache/metron#661
> > > METRON-1079 STELLAR NaN should be a keyword
> (ottobackwards)
> > > closes apache/metron#681
> > > METRON-1085 Add REST endpoint to save a user profile
> for the
> > > Alerts UI (merrimanr) closes apache/metron#694
> > > METRON-1208 MPack for Alerts UI (merrimanr) closes
> > > apache/metron#778
> > > METRON-1207 Make RPMs for Alerts UI (merrimanr) closes
> > > apache/metron#777
> > > METRON-1215 Fix link to RPMs chapter (DimDroll via
> > justinleet)
> > > closes apache/metron#776
> > > METRON-1206 Make alerts UI conform to ops UI for
> install
> > > (merrimanr) closes apache/metron#773
> > > METRON-1195 Meta alerts improperly handle updates to
> > non-alert
> > > fields (justinleet) closes apache/metron#766
> > > METRON-1189 Add alert escalation to the Alerts UI
> (merrimanr)
> > > closes apache/metron#762
> > > METRON-1156 Simulate Triage Rules in the Stellar REPL
> > > (nickwallen) closes apache/metron#733
> > > METRON-1198 Pycapa - No such configuration property
> > > 'sasl.kerberos.principal' (nickwallen) closes apache/metron#769
> > > METRON-1202 ElasticsearchDao Has extraneous sleep call
> > > (justinleet) closes apache/metron#770
> > > METRON-938 "service metron-rest start <password>" does
> not
> > > work on CentOS 7. (justinleet) closes apache/metron#757
> > > METRON-1182 Refactor Code in alert list to accommodate
> new
> > > view types (iraghumitra via merrimanr) closes apache/metron#756
> > > METRON-1188: Ambari global configuration management
> > (mmiklavc)
> > > closes apache/metron#760
> > > METRON-1191 update public web site to point at 0.4.1
> new
> > > release (mattf-horton) closes apache/metron#764
> > > METRON-1063 address javadoc warnings in metron-stellar
> (dbist
> > > via ottobackwards) closes apache/metron#668
> > > METRON-1190 Fix Meta Alert Type handling in
> calculation of
> > > scores (justinleet) closes apache/metron#763
> > > METRON-1187 Indexing/Profiler Kafka ACL Groups Not
> Setup
> > > Correctly (nickwallen) closes apache/metron#759
> > > METRON-1185: Stellar REPL does not work on a kerberized
> > > cluster when calling functions interacting with HBase closes
> > > apache/incubator-metron#755
> > > METRON-1186: Profiler Functions use classutils from
> shaded
> > > storm closes apache/incubator-metron#758
> > > METRON-1173: Fix pointers to old stellar docs closes
> > > apache/incubator-metron#746
> > > METRON-1179: Make STATS_ADD to take a list closes
> > > apache/incubator-metron#750
> > > METRON-1180: Make Stellar Shell accept zookeeper
> quorum as a
> > > CSV list and not require a port closes apache/incubator-metron#751
> > > METRON-1183 Improve KDC Setup Instructions (nickwallen)
> > closes
> > > apache/metron#753
> > > METRON-1177 Stale running topologies seen
> post-kerberization
> > > and cause exceptions (nickwallen) closes apache/metron#748
> > > METRON-1158 Build backend for grouping alerts into meta
> > alerts
> > > (justinleet) closes apache/metron#734
> > > METRON-1146: Add ability to parse JSON string into
> JSONObject
> > > for stellar closes apache/incubator-metron#727
> > > METRON-1176 REST: HDFS Service should support setting
> > > permissions on files when writing (ottobackwards) closes
> > apache/metron#749
> > > METRON-1114 Add group by capabilities to search REST
> endpoint
> > > (merrimanr) closes apache/metron#702
> > > METRON-1167 Define Session Specific Global
> Configuration
> > > Values in the REPL (nickwallen) closes apache/metron#740
> > > METRON-1171: Better validation for the SUBSTRING
> stellar
> > > function closes apache/incubator-metron#745
> > >
> > >
> > >
> > > On 11/17/17, 11:59 AM, "Nick Allen" <ni...@nickallen.org>
> wrote:
> > >
> > > I just wanted to send an update on where we are at.
> We've
> > > gotten a lot
> > > done here recently as you can see below.
> > >
> > > ✓ DONE (1) First, METRON-1289 needs to go in. This
> one was
> > > a fairly big
> > > effort and I am hearing that we are pretty close.
> > >
> > > ✓ DONE (2) METRON-1294 fixes an issue in how field
> types
> > are
> > > looked-up.
> > >
> > > ✓ DONE (3) METRON-1290 is next. While this may have
> been
> > > fixed in
> > > M-1289, there may be some test cases we want from this
> PR.
> > >
> > > ✓ DONE (4) METRON-1301 addresses a problem with the
> sorting
> > > logic.
> > >
> > > ✓ DONE (5) METRON-1291 fixes an issue with
> escalation of
> > > metaalerts.
> > >
> > > (6) That leads us to Raghu's UI work in
> METRON-1252. This
> > > introduces the
> > > UI bits that depend on all the previous backend work.
> > >
> > > (7) At this point, we should have our best effort at
> > running
> > > Metaalerts
> > > on Elasticsearch 2.x. I propose that we cut a release
> here.
> > >
> > > (8) After we cut the release, we can introduce the
> work for
> > > ES 5.x in
> > > METRON-939. I know we will need lots of help testing
> and
> > > reviewing this
> > > one.
> > >
> > >
> > >
> > > We also have an outstanding question that needs
> resolved
> > > BEFORE we
> > > release. We need to come to a consensus on how to
> release
> > > having moved our
> > > Bro Plugin to a separate repo. I don't think we've
> heard
> > from
> > > everyone on
> > > this. I'd urge everyone to chime in so we can choose
> a path
> > > forward.
> > >
> > > If anyone is totally confused in regards to that
> discussion,
> > I
> > > can try and
> > > send an options summary again as a separate discuss
> thread.
> > > The original
> > > chain was somewhere around here [1].
> > >
> > > [1]
> > > https://lists.apache.org/thread.html/
> > > 54a4474881b97e559df24728b3a0e923a58345a282451085eef832ef@%
> > > 3Cdev.metron.apache.org%3E
> > >
> > >
> > >
> > > On Wed, Nov 15, 2017 at 10:04 AM, Nick Allen <
> > > nick@nickallen.org> wrote:
> > >
> > > > Hi Guys -
> > > >
> > > > I want to follow-up on this discussion. It sounds
> like
> > most
> > > people are in
> > > > agreement with the general approach.
> > > >
> > > > A lot of people have been working hard on Metaalerts
> and
> > > Elasticsearch. I
> > > > have checked-in with those doing the heavy lifting
> and have
> > > compiled a more
> > > > detailed plan based on where we are at now. To the
> best of
> > > my knowledge
> > > > here is the plan of attack for finishing out this
> effort.
> > > >
> > > > (1) First, METRON-1289 needs to go in. This one
> was a
> > > fairly big effort
> > > > and I am hearing that we are pretty close.
> > > >
> > > > (2) METRON-1294 fixes an issue in how field types
> are
> > > looked-up.
> > > >
> > > > (3) METRON-1290 is next. While this may have been
> fixed
> > > in M-1289,
> > > > there may be some test cases we want from this PR.
> > > >
> > > > (4) METRON-1301 addresses a problem with the
> sorting
> > logic.
> > > >
> > > > (5) METRON-1291 fixes an issue with escalation of
> > > metaalerts.
> > > >
> > > > (6) That leads us to Raghu's UI work in
> METRON-1252.
> > This
> > > introduces
> > > > the UI bits that depend on all the previous backend
> work.
> > > >
> > > > (7) At this point, we should have our best effort
> at
> > > running Metaalerts
> > > > on Elasticsearch 2.x. I propose that we cut a
> release here.
> > > >
> > > > (8) After we cut the release, we can introduce the
> work
> > > for ES 5.x in
> > > > METRON-939. I know we will need lots of help
> testing and
> > > reviewing this
> > > > one.
> > > >
> > > > Please correct me if I am wrong. I will try and
> send out
> > > updates as we
> > > > make progress.
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > On Mon, Nov 6, 2017 at 1:03 PM, Zeolla@GMail.com <
> > > zeolla@gmail.com> wrote:
> > > >
> > > >> I agree, I think it's very reasonable to move in
> line with
> > > Nick's
> > > >> proposal. I would also suggest that we outline
> what the
> > > target versions
> > > >> would be to add in the METRON-777 components, since
> it has
> > > been functional
> > > >> for a very long time but not reviewed and has some
> really
> > > rockstar
> > > >> improvements.
> > > >>
> > > >> Jon
> > > >>
> > > >> On Mon, Nov 6, 2017 at 12:56 PM Otto Fowler <
> > > ottobackwards@gmail.com>
> > > >> wrote:
> > > >>
> > > >> > I think the ES cutover should be the start of the
> 0.5.x
> > > series, and we
> > > >> > continue on with 0.4.x for the
> > > >> > metadata improvements etc. We could chose to
> focus
> > > 0.5.x’s first
> > > >> releases
> > > >> > on not only ES but
> > > >> > getting a handle on kibana and the mpack
> situation as
> > > well.
> > > >> >
> > > >> >
> > > >> >
> > > >> >
> > > >> > On November 6, 2017 at 12:48:45, Michael
> Miklavcic (
> > > >> > michael.miklavcic@gmail.com) wrote:
> > > >> >
> > > >> > I agree with your proposal, Nick. I think having a
> > > stabilizing release
> > > >> > prior to upgrading ES/Kibana makes sense.
> > > >> >
> > > >> > On Mon, Nov 6, 2017 at 9:16 AM, Nick Allen <
> > > nick@nickallen.org> wrote:
> > > >> >
> > > >> > > I would like to start a discussion around
> upcoming
> > > releases. We have a
> > > >> > > couple separate significant tracks of work that
> we
> > need
> > > to reconcile
> > > >> in
> > > >> > our
> > > >> > > release schedule.
> > > >> > >
> > > >> > > (1) We have had (and have in review) a good
> number of
> > > bug fixes
> > > >> required
> > > >> > to
> > > >> > > support Metaalerts on the existing
> Elasticsearch 2.x
> > > infrastructure.
> > > >> > >
> > > >> > >
> > > >> > > (2) We also have ongoing work to upgrade our
> > > infrastructure to
> > > >> > > Elasticsearch 5.x, which will not be backwards
> > > compatible.
> > > >> > >
> > > >> > >
> > > >> > > I would like to see a release that has our best
> work
> > on
> > > ES 2.x before
> > > >> we
> > > >> > > migrate to 5.x. I would propose the following.
> > > >> > >
> > > >> > > Release N+1: Introduce Metaalerts running on ES
> 2.x
> > > >> > >
> > > >> > > Release N+2: Cut-over to ES 5.x
> > > >> > >
> > > >> > >
> > > >> > > (Q) Is it worth cutting a separate release for
> ES 2.x?
> > > Is there a
> > > >> better
> > > >> > > way to handle the cut-over to 5.x?
> > > >> > >
> > > >> >
> > > >> --
> > > >>
> > > >> Jon
> > > >>
> > > >
> > > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> >
> --
>
> Jon
>
>
>
> --
Jon
Re: [DISCUSS] Upcoming Release
Posted by Matt Foley <ma...@apache.org>.
Colleagues,
I’ve posted Metron-0.4.2-RC1 and Metron-bro-plugin-kafka-0.1 to https://dist.apache.org/repos/dist/dev/metron/0.4.2-RC1/
Given the complexity of this RC, I’d appreciate if a couple people would be willing to kick the tires before we put it up for a vote.
I will myself be going thru the Verify Build process this weekend, as I won’t be able to do it Friday.
Thanks,
--Matt
On 12/4/17, 2:05 PM, "Zeolla@GMail.com" <ze...@gmail.com> wrote:
Can we resolve the conversation regarding the second repo? I was waiting
to get more input/preferences from people There's also a documentation
update that fixes a few broken Stellar docs that already has aa +1, I just
need to merge it.
Jon
On Mon, Dec 4, 2017, 17:01 Casey Stella <ce...@gmail.com> wrote:
> I would be in favor of a release at this point.
>
> On Mon, Dec 4, 2017 at 4:57 PM, Matt Foley <ma...@apache.org> wrote:
>
> > Hey all,
> > I see METRON-1252 was resolved over the weekend. Shall I go ahead and
> > start the process with 0.4.2 release?
> > Does anyone have any commits they feel strongly should go in before 0.4.2
> > is done, or are we ready to call it good?
> >
> > I believe there is consensus the 0.4.2 release should include a release
> of
> > the current state of the metron-bro-plugin-kafka. I will continue the
> > discussion in that thread as to the process for accomplishing that, but
> > plan on it happening.
> >
> > Regards,
> > --Matt
> >
> > On 11/26/17, 6:26 PM, "Matt Foley" <ma...@apache.org> wrote:
> >
> > Hope everyone (at least in the U.S.) had a great Thanksgiving
> holiday.
> > Regarding status of the release effort, still pending METRON-1252, so
> > not making the release branch yet.
> >
> > Regards,
> > --Matt
> >
> > On 11/17/17, 1:32 PM, "Matt Foley" <ma...@apache.org> wrote:
> >
> > (With release manager hat on)
> >
> > The community has proposed a release of Metron in the near
> future,
> > focusing on Meta-alerts running in Elasticsearch.
> > Congrats on getting so many of the below already done. At this
> > point, only METRON-1252, and the discussion of how to handle joint
> release
> > of the Metron bro plugin, remain as gating items for the release. I
> > project these will be resolved next week, so let’s propose the following:
> >
> > Sometime next week, after the last bits are done, I’ll start the
> > release process and create the release branch.
> >
> > The proposed new version will be 0.4.2, unless there are backward
> > incompatible changes that support making it 0.5.0.
> > Currently there are NO included Jiras labeled
> > ‘backward-incompatible’, nor having Docs Text indicating so.
> > ***If anyone knows that some of the commits included since 0.4.1
> > introduce backward incompatibility, please say so now on this thread, and
> > mark the Jira as such.***
> >
> > The 90 or so jiras/commits already in master branch since 0.4.1
> > are listed below.
> > Thanks,
> > --Matt
> >
> > METRON-1301 Alerts UI - Sorting on Triage Score Unexpectedly
> > Filters Some Records (nickwallen) closes apache/metron#832
> > METRON-1294 IP addresses are not formatted correctly in facet
> > and group results (merrimanr) closes apache/metron#827
> > METRON-1291 Kafka produce REST endpoint does not work in a
> > Kerberized cluster (merrimanr) closes apache/metron#826
> > METRON-1290 Only first 10 alerts are update when a MetaAlert
> > status is changed to inactive (justinleet) closes apache/metron#842
> > METRON-1311 Service Check Should Check Elasticsearch Index
> > Templates (nickwallen) closes apache/metron#839
> > METRON-1289 Alert fields are lost when a MetaAlert is created
> > (merrimanr) closes apache/metron#824
> > METRON-1309 Change metron-deployment to pull the plugin from
> > apache/metron-bro-plugin-kafka (JonZeolla) closes apache/metron#837
> > METRON-1310 Template Delete Action Deletes Search Indices
> > (nickwallen) closes apache/metron#838
> > METRON-1275: Fix Metron Documentation closes
> > apache/incubator-metron#833
> > METRON-1295 Unable to Configure Logging for REST API
> > (nickwallen) closes apache/metron#828
> > METRON-1307 Force install of java8 since java9 does not
> appear
> > to work with the scripts (brianhurley via ottobackwards) closes
> > apache/metron#835
> > METRON-1296 Full Dev Fails to Deploy Index Templates
> > (nickwallen via cestella) closes apache/incubator-metron#829
> > METRON-1281 Remove hard-coded indices from the Alerts UI
> > (merrimanr) closes apache/metron#821
> > METRON-1287 Full Dev Fails When Installing EPEL Repository
> > (nickwallen) closes apache/metron#820
> > METRON-1267 Alerts UI returns a 404 when refreshing the
> > alerts-list page (iraghumitra via merrimanr) closes apache/metron#819
> > METRON-1283 Install Elasticsearch template as a part of the
> > mpack startup scripts (anandsubbu via nickwallen) closes
> apache/metron#817
> > METRON-1254: Conditionals as map keys do not function in
> > Stellar closes apache/incubator-metron#801
> > METRON-1261 Apply bro security patch (JonZeolla via
> > ottobackwards) closes apache/metron#805
> > METRON-1284 Remove extraneous dead query in ElasticsearchDao
> > (justinleet) closes apache/metron#818
> > METRON-1270: fix for warnings missing @return tag argument in
> > metron-analytics/metron-profiler-common and metron-profiler-client closes
> > apache/incubator-metron#810
> > METRON-1272 Hide child alerts from searches and grouping if
> > they belong to meta alerts (justinleet) closes apache/metron#811
> > METRON-1224 Add time range selection to search control
> > (iraghumitra via james-sirota) closes apache/metron#796
> > METRON-1280 0.4.1 -> 0.4.2 missed a couple of projects
> > (cestella via justinleet) closes apache/metron#816
> > METRON-1243: Add a REST endpoint which allows us to get a
> list
> > of all indice closes apache/incubator-metron#797
> > METRON-1196 Increment master version number to 0.4.2 for
> > on-going development (mattf-horton) closes apache/metron#767
> > METRON-1278 Strip "Build Status" widget from root
> > README.md in site-book build (mattf-horton) closes apache/metron#815
> > METRON-1274 Master has failure in
> > StormControllerIntegrationTest (merrimanr) closes apache/metron#813
> > METRON-1266 Profiler - SASL Authentication Failed
> (nickwallen)
> > closes apache/metron#809
> > METRON-1260 Include Alerts UI in Ambari Service Check
> > (nickwallen) closes apache/metron#804
> > METRON-1251: Typo and formatting fixes for metron-rest README
> > closes apache/incubator-metron#800
> > METRON-1241: Enable the REST API to use a cache for the
> > zookeeper config similar to the Bolts closes apache/incubator-metron#795
> > METRON-1267 Alerts UI returns a 404 when refreshing the
> > alerts-list page (merrimanr) closes apache/metron#808
> > METRON-1262 Unable to add comment for a alert in a meta-alert
> > (merrimanr) closes apache/metron#806
> > METRON-1263 Start Alerts UI service after Metron REST
> > (anandsubbu via nickwallen) closes apache/metron#807
> > METRON-1255 MetaAlert search is not filtering on status
> > (merrimanr) closes apache/metron#802
> > METRON-1249 Improve Metron MPack Service Checks (nickwallen)
> > closes apache/metron#799
> > METRON-1237 address javadoc warnings in metron-maas-common
> > (dbist via james-sirota) closes apache/metron#792
> > METRON-1240 address javadoc warnings in metron-platform and
> > metron-analytics (dbist via james-sirota) closes apache/metron#794
> > METRON-1226 Searching Can Errantly Query the Wrong Indices
> > (nickwallen) closes apache/metron#793
> > METRON-1081 Fix Alerts and Ops UI Notices file (james-sirota)
> > closes apache/metron#682
> > METRON-1123 Add group by option using faceted search
> > capabilities of metron-rest-api (iraghumitra via james-sirota) closes
> > apache/metron#768
> > METRON-1223 Add support to add comments for alerts
> > (iraghumitra via james-sirota) closes apache/metron#788
> > METRON-1083 Add filters using faceted search capabilities of
> > metron-rest-api (iraghumitra via james-sirota) closes apache/metron#710
> > METRON-1232 Alert status changes are not reflected in list
> > view (iraghumitra via merrimanr) closes apache/metron#787
> > METRON-1247 REST search and findOne endpoints return
> > unexpected or incorrect results for guids (justinleet) closes
> > apache/metron#798
> > METRON-1235: Document the properties pulled from the global
> > configuration closes apache/incubator-metron#791
> > METRON-1234: fix for WARNING 'dependencies.dependency.(
> > groupId:artifactId:type:classifier)' must be unique:
> > org.apache.hadoop:hadoop-yarn-api:jar (dbist via mmiklavc) closes
> > apache/metron#790
> > METRON-1222: fix warning for The expression ${parent.version}
> > is deprecated. Please use ${project.parent.version} instead. (dbist via
> > mmiklavc) closes apache/metron#782
> > METRON-1220 Create documentation around alert nested field
> > (justinleet) closes apache/metron#780
> > METRON-1229 Management UI type is part of the declarations of
> > 2 modules (merrimanr) closes apache/metron#784
> > METRON-1228: Configuration Management PUSH immediately does
> > DUMP after (mmiklavc via mmiklavc) closes apache/metron#783
> > METRON-1218 Metron REST should return better error messages
> > (merrimanr) closes apache/metron#779
> > METRON-1161 Add ability to edit parser command line options
> in
> > the management UI (merrimanr) closes apache/metron#737
> > METRON-1209: Make stellar repl take logging properties, like
> > other CLI apps in metron closes apache/incubator-metron#772
> > METRON-1059 address checkstyle warning AvoidStarImport in
> > metron-stellar (dbist via ottobackwards) closes apache/metron#664
> > METRON-1204 UI does not time out after being idle, but stops
> > functioning (merrimanr) closes apache/metron#771
> > METRON-1052: Add forensic similarity hash functions to
> Stellar
> > closes apache/incubator-metron#781
> > METRON-632: Added validation of "shew.enrichmentType" and
> > "shew.keyColumns" closes apache/incubator-metron#732
> > METRON-1194 Add Profiler Debug Functions to Profiler README
> > (nickwallen via ottobackwards) closes apache/metron#765
> > METRON-1055 Metron 0.4.0 manual installation guide for CentOS
> > 6 updates (lvets via ottobackwards) closes apache/metron#661
> > METRON-1079 STELLAR NaN should be a keyword (ottobackwards)
> > closes apache/metron#681
> > METRON-1085 Add REST endpoint to save a user profile for the
> > Alerts UI (merrimanr) closes apache/metron#694
> > METRON-1208 MPack for Alerts UI (merrimanr) closes
> > apache/metron#778
> > METRON-1207 Make RPMs for Alerts UI (merrimanr) closes
> > apache/metron#777
> > METRON-1215 Fix link to RPMs chapter (DimDroll via
> justinleet)
> > closes apache/metron#776
> > METRON-1206 Make alerts UI conform to ops UI for install
> > (merrimanr) closes apache/metron#773
> > METRON-1195 Meta alerts improperly handle updates to
> non-alert
> > fields (justinleet) closes apache/metron#766
> > METRON-1189 Add alert escalation to the Alerts UI (merrimanr)
> > closes apache/metron#762
> > METRON-1156 Simulate Triage Rules in the Stellar REPL
> > (nickwallen) closes apache/metron#733
> > METRON-1198 Pycapa - No such configuration property
> > 'sasl.kerberos.principal' (nickwallen) closes apache/metron#769
> > METRON-1202 ElasticsearchDao Has extraneous sleep call
> > (justinleet) closes apache/metron#770
> > METRON-938 "service metron-rest start <password>" does not
> > work on CentOS 7. (justinleet) closes apache/metron#757
> > METRON-1182 Refactor Code in alert list to accommodate new
> > view types (iraghumitra via merrimanr) closes apache/metron#756
> > METRON-1188: Ambari global configuration management
> (mmiklavc)
> > closes apache/metron#760
> > METRON-1191 update public web site to point at 0.4.1 new
> > release (mattf-horton) closes apache/metron#764
> > METRON-1063 address javadoc warnings in metron-stellar (dbist
> > via ottobackwards) closes apache/metron#668
> > METRON-1190 Fix Meta Alert Type handling in calculation of
> > scores (justinleet) closes apache/metron#763
> > METRON-1187 Indexing/Profiler Kafka ACL Groups Not Setup
> > Correctly (nickwallen) closes apache/metron#759
> > METRON-1185: Stellar REPL does not work on a kerberized
> > cluster when calling functions interacting with HBase closes
> > apache/incubator-metron#755
> > METRON-1186: Profiler Functions use classutils from shaded
> > storm closes apache/incubator-metron#758
> > METRON-1173: Fix pointers to old stellar docs closes
> > apache/incubator-metron#746
> > METRON-1179: Make STATS_ADD to take a list closes
> > apache/incubator-metron#750
> > METRON-1180: Make Stellar Shell accept zookeeper quorum as a
> > CSV list and not require a port closes apache/incubator-metron#751
> > METRON-1183 Improve KDC Setup Instructions (nickwallen)
> closes
> > apache/metron#753
> > METRON-1177 Stale running topologies seen post-kerberization
> > and cause exceptions (nickwallen) closes apache/metron#748
> > METRON-1158 Build backend for grouping alerts into meta
> alerts
> > (justinleet) closes apache/metron#734
> > METRON-1146: Add ability to parse JSON string into JSONObject
> > for stellar closes apache/incubator-metron#727
> > METRON-1176 REST: HDFS Service should support setting
> > permissions on files when writing (ottobackwards) closes
> apache/metron#749
> > METRON-1114 Add group by capabilities to search REST endpoint
> > (merrimanr) closes apache/metron#702
> > METRON-1167 Define Session Specific Global Configuration
> > Values in the REPL (nickwallen) closes apache/metron#740
> > METRON-1171: Better validation for the SUBSTRING stellar
> > function closes apache/incubator-metron#745
> >
> >
> >
> > On 11/17/17, 11:59 AM, "Nick Allen" <ni...@nickallen.org> wrote:
> >
> > I just wanted to send an update on where we are at. We've
> > gotten a lot
> > done here recently as you can see below.
> >
> > ✓ DONE (1) First, METRON-1289 needs to go in. This one was
> > a fairly big
> > effort and I am hearing that we are pretty close.
> >
> > ✓ DONE (2) METRON-1294 fixes an issue in how field types
> are
> > looked-up.
> >
> > ✓ DONE (3) METRON-1290 is next. While this may have been
> > fixed in
> > M-1289, there may be some test cases we want from this PR.
> >
> > ✓ DONE (4) METRON-1301 addresses a problem with the sorting
> > logic.
> >
> > ✓ DONE (5) METRON-1291 fixes an issue with escalation of
> > metaalerts.
> >
> > (6) That leads us to Raghu's UI work in METRON-1252. This
> > introduces the
> > UI bits that depend on all the previous backend work.
> >
> > (7) At this point, we should have our best effort at
> running
> > Metaalerts
> > on Elasticsearch 2.x. I propose that we cut a release here.
> >
> > (8) After we cut the release, we can introduce the work for
> > ES 5.x in
> > METRON-939. I know we will need lots of help testing and
> > reviewing this
> > one.
> >
> >
> >
> > We also have an outstanding question that needs resolved
> > BEFORE we
> > release. We need to come to a consensus on how to release
> > having moved our
> > Bro Plugin to a separate repo. I don't think we've heard
> from
> > everyone on
> > this. I'd urge everyone to chime in so we can choose a path
> > forward.
> >
> > If anyone is totally confused in regards to that discussion,
> I
> > can try and
> > send an options summary again as a separate discuss thread.
> > The original
> > chain was somewhere around here [1].
> >
> > [1]
> > https://lists.apache.org/thread.html/
> > 54a4474881b97e559df24728b3a0e923a58345a282451085eef832ef@%
> > 3Cdev.metron.apache.org%3E
> >
> >
> >
> > On Wed, Nov 15, 2017 at 10:04 AM, Nick Allen <
> > nick@nickallen.org> wrote:
> >
> > > Hi Guys -
> > >
> > > I want to follow-up on this discussion. It sounds like
> most
> > people are in
> > > agreement with the general approach.
> > >
> > > A lot of people have been working hard on Metaalerts and
> > Elasticsearch. I
> > > have checked-in with those doing the heavy lifting and have
> > compiled a more
> > > detailed plan based on where we are at now. To the best of
> > my knowledge
> > > here is the plan of attack for finishing out this effort.
> > >
> > > (1) First, METRON-1289 needs to go in. This one was a
> > fairly big effort
> > > and I am hearing that we are pretty close.
> > >
> > > (2) METRON-1294 fixes an issue in how field types are
> > looked-up.
> > >
> > > (3) METRON-1290 is next. While this may have been fixed
> > in M-1289,
> > > there may be some test cases we want from this PR.
> > >
> > > (4) METRON-1301 addresses a problem with the sorting
> logic.
> > >
> > > (5) METRON-1291 fixes an issue with escalation of
> > metaalerts.
> > >
> > > (6) That leads us to Raghu's UI work in METRON-1252.
> This
> > introduces
> > > the UI bits that depend on all the previous backend work.
> > >
> > > (7) At this point, we should have our best effort at
> > running Metaalerts
> > > on Elasticsearch 2.x. I propose that we cut a release here.
> > >
> > > (8) After we cut the release, we can introduce the work
> > for ES 5.x in
> > > METRON-939. I know we will need lots of help testing and
> > reviewing this
> > > one.
> > >
> > > Please correct me if I am wrong. I will try and send out
> > updates as we
> > > make progress.
> > >
> > >
> > >
> > >
> > >
> > > On Mon, Nov 6, 2017 at 1:03 PM, Zeolla@GMail.com <
> > zeolla@gmail.com> wrote:
> > >
> > >> I agree, I think it's very reasonable to move in line with
> > Nick's
> > >> proposal. I would also suggest that we outline what the
> > target versions
> > >> would be to add in the METRON-777 components, since it has
> > been functional
> > >> for a very long time but not reviewed and has some really
> > rockstar
> > >> improvements.
> > >>
> > >> Jon
> > >>
> > >> On Mon, Nov 6, 2017 at 12:56 PM Otto Fowler <
> > ottobackwards@gmail.com>
> > >> wrote:
> > >>
> > >> > I think the ES cutover should be the start of the 0.5.x
> > series, and we
> > >> > continue on with 0.4.x for the
> > >> > metadata improvements etc. We could chose to focus
> > 0.5.x’s first
> > >> releases
> > >> > on not only ES but
> > >> > getting a handle on kibana and the mpack situation as
> > well.
> > >> >
> > >> >
> > >> >
> > >> >
> > >> > On November 6, 2017 at 12:48:45, Michael Miklavcic (
> > >> > michael.miklavcic@gmail.com) wrote:
> > >> >
> > >> > I agree with your proposal, Nick. I think having a
> > stabilizing release
> > >> > prior to upgrading ES/Kibana makes sense.
> > >> >
> > >> > On Mon, Nov 6, 2017 at 9:16 AM, Nick Allen <
> > nick@nickallen.org> wrote:
> > >> >
> > >> > > I would like to start a discussion around upcoming
> > releases. We have a
> > >> > > couple separate significant tracks of work that we
> need
> > to reconcile
> > >> in
> > >> > our
> > >> > > release schedule.
> > >> > >
> > >> > > (1) We have had (and have in review) a good number of
> > bug fixes
> > >> required
> > >> > to
> > >> > > support Metaalerts on the existing Elasticsearch 2.x
> > infrastructure.
> > >> > >
> > >> > >
> > >> > > (2) We also have ongoing work to upgrade our
> > infrastructure to
> > >> > > Elasticsearch 5.x, which will not be backwards
> > compatible.
> > >> > >
> > >> > >
> > >> > > I would like to see a release that has our best work
> on
> > ES 2.x before
> > >> we
> > >> > > migrate to 5.x. I would propose the following.
> > >> > >
> > >> > > Release N+1: Introduce Metaalerts running on ES 2.x
> > >> > >
> > >> > > Release N+2: Cut-over to ES 5.x
> > >> > >
> > >> > >
> > >> > > (Q) Is it worth cutting a separate release for ES 2.x?
> > Is there a
> > >> better
> > >> > > way to handle the cut-over to 5.x?
> > >> > >
> > >> >
> > >> --
> > >>
> > >> Jon
> > >>
> > >
> > >
> >
> >
> >
> >
> >
> >
> >
> >
>
--
Jon
Re: [DISCUSS] Upcoming Release
Posted by "Zeolla@GMail.com" <ze...@gmail.com>.
Can we resolve the conversation regarding the second repo? I was waiting
to get more input/preferences from people There's also a documentation
update that fixes a few broken Stellar docs that already has aa +1, I just
need to merge it.
Jon
On Mon, Dec 4, 2017, 17:01 Casey Stella <ce...@gmail.com> wrote:
> I would be in favor of a release at this point.
>
> On Mon, Dec 4, 2017 at 4:57 PM, Matt Foley <ma...@apache.org> wrote:
>
> > Hey all,
> > I see METRON-1252 was resolved over the weekend. Shall I go ahead and
> > start the process with 0.4.2 release?
> > Does anyone have any commits they feel strongly should go in before 0.4.2
> > is done, or are we ready to call it good?
> >
> > I believe there is consensus the 0.4.2 release should include a release
> of
> > the current state of the metron-bro-plugin-kafka. I will continue the
> > discussion in that thread as to the process for accomplishing that, but
> > plan on it happening.
> >
> > Regards,
> > --Matt
> >
> > On 11/26/17, 6:26 PM, "Matt Foley" <ma...@apache.org> wrote:
> >
> > Hope everyone (at least in the U.S.) had a great Thanksgiving
> holiday.
> > Regarding status of the release effort, still pending METRON-1252, so
> > not making the release branch yet.
> >
> > Regards,
> > --Matt
> >
> > On 11/17/17, 1:32 PM, "Matt Foley" <ma...@apache.org> wrote:
> >
> > (With release manager hat on)
> >
> > The community has proposed a release of Metron in the near
> future,
> > focusing on Meta-alerts running in Elasticsearch.
> > Congrats on getting so many of the below already done. At this
> > point, only METRON-1252, and the discussion of how to handle joint
> release
> > of the Metron bro plugin, remain as gating items for the release. I
> > project these will be resolved next week, so let’s propose the following:
> >
> > Sometime next week, after the last bits are done, I’ll start the
> > release process and create the release branch.
> >
> > The proposed new version will be 0.4.2, unless there are backward
> > incompatible changes that support making it 0.5.0.
> > Currently there are NO included Jiras labeled
> > ‘backward-incompatible’, nor having Docs Text indicating so.
> > ***If anyone knows that some of the commits included since 0.4.1
> > introduce backward incompatibility, please say so now on this thread, and
> > mark the Jira as such.***
> >
> > The 90 or so jiras/commits already in master branch since 0.4.1
> > are listed below.
> > Thanks,
> > --Matt
> >
> > METRON-1301 Alerts UI - Sorting on Triage Score Unexpectedly
> > Filters Some Records (nickwallen) closes apache/metron#832
> > METRON-1294 IP addresses are not formatted correctly in facet
> > and group results (merrimanr) closes apache/metron#827
> > METRON-1291 Kafka produce REST endpoint does not work in a
> > Kerberized cluster (merrimanr) closes apache/metron#826
> > METRON-1290 Only first 10 alerts are update when a MetaAlert
> > status is changed to inactive (justinleet) closes apache/metron#842
> > METRON-1311 Service Check Should Check Elasticsearch Index
> > Templates (nickwallen) closes apache/metron#839
> > METRON-1289 Alert fields are lost when a MetaAlert is created
> > (merrimanr) closes apache/metron#824
> > METRON-1309 Change metron-deployment to pull the plugin from
> > apache/metron-bro-plugin-kafka (JonZeolla) closes apache/metron#837
> > METRON-1310 Template Delete Action Deletes Search Indices
> > (nickwallen) closes apache/metron#838
> > METRON-1275: Fix Metron Documentation closes
> > apache/incubator-metron#833
> > METRON-1295 Unable to Configure Logging for REST API
> > (nickwallen) closes apache/metron#828
> > METRON-1307 Force install of java8 since java9 does not
> appear
> > to work with the scripts (brianhurley via ottobackwards) closes
> > apache/metron#835
> > METRON-1296 Full Dev Fails to Deploy Index Templates
> > (nickwallen via cestella) closes apache/incubator-metron#829
> > METRON-1281 Remove hard-coded indices from the Alerts UI
> > (merrimanr) closes apache/metron#821
> > METRON-1287 Full Dev Fails When Installing EPEL Repository
> > (nickwallen) closes apache/metron#820
> > METRON-1267 Alerts UI returns a 404 when refreshing the
> > alerts-list page (iraghumitra via merrimanr) closes apache/metron#819
> > METRON-1283 Install Elasticsearch template as a part of the
> > mpack startup scripts (anandsubbu via nickwallen) closes
> apache/metron#817
> > METRON-1254: Conditionals as map keys do not function in
> > Stellar closes apache/incubator-metron#801
> > METRON-1261 Apply bro security patch (JonZeolla via
> > ottobackwards) closes apache/metron#805
> > METRON-1284 Remove extraneous dead query in ElasticsearchDao
> > (justinleet) closes apache/metron#818
> > METRON-1270: fix for warnings missing @return tag argument in
> > metron-analytics/metron-profiler-common and metron-profiler-client closes
> > apache/incubator-metron#810
> > METRON-1272 Hide child alerts from searches and grouping if
> > they belong to meta alerts (justinleet) closes apache/metron#811
> > METRON-1224 Add time range selection to search control
> > (iraghumitra via james-sirota) closes apache/metron#796
> > METRON-1280 0.4.1 -> 0.4.2 missed a couple of projects
> > (cestella via justinleet) closes apache/metron#816
> > METRON-1243: Add a REST endpoint which allows us to get a
> list
> > of all indice closes apache/incubator-metron#797
> > METRON-1196 Increment master version number to 0.4.2 for
> > on-going development (mattf-horton) closes apache/metron#767
> > METRON-1278 Strip "Build Status" widget from root
> > README.md in site-book build (mattf-horton) closes apache/metron#815
> > METRON-1274 Master has failure in
> > StormControllerIntegrationTest (merrimanr) closes apache/metron#813
> > METRON-1266 Profiler - SASL Authentication Failed
> (nickwallen)
> > closes apache/metron#809
> > METRON-1260 Include Alerts UI in Ambari Service Check
> > (nickwallen) closes apache/metron#804
> > METRON-1251: Typo and formatting fixes for metron-rest README
> > closes apache/incubator-metron#800
> > METRON-1241: Enable the REST API to use a cache for the
> > zookeeper config similar to the Bolts closes apache/incubator-metron#795
> > METRON-1267 Alerts UI returns a 404 when refreshing the
> > alerts-list page (merrimanr) closes apache/metron#808
> > METRON-1262 Unable to add comment for a alert in a meta-alert
> > (merrimanr) closes apache/metron#806
> > METRON-1263 Start Alerts UI service after Metron REST
> > (anandsubbu via nickwallen) closes apache/metron#807
> > METRON-1255 MetaAlert search is not filtering on status
> > (merrimanr) closes apache/metron#802
> > METRON-1249 Improve Metron MPack Service Checks (nickwallen)
> > closes apache/metron#799
> > METRON-1237 address javadoc warnings in metron-maas-common
> > (dbist via james-sirota) closes apache/metron#792
> > METRON-1240 address javadoc warnings in metron-platform and
> > metron-analytics (dbist via james-sirota) closes apache/metron#794
> > METRON-1226 Searching Can Errantly Query the Wrong Indices
> > (nickwallen) closes apache/metron#793
> > METRON-1081 Fix Alerts and Ops UI Notices file (james-sirota)
> > closes apache/metron#682
> > METRON-1123 Add group by option using faceted search
> > capabilities of metron-rest-api (iraghumitra via james-sirota) closes
> > apache/metron#768
> > METRON-1223 Add support to add comments for alerts
> > (iraghumitra via james-sirota) closes apache/metron#788
> > METRON-1083 Add filters using faceted search capabilities of
> > metron-rest-api (iraghumitra via james-sirota) closes apache/metron#710
> > METRON-1232 Alert status changes are not reflected in list
> > view (iraghumitra via merrimanr) closes apache/metron#787
> > METRON-1247 REST search and findOne endpoints return
> > unexpected or incorrect results for guids (justinleet) closes
> > apache/metron#798
> > METRON-1235: Document the properties pulled from the global
> > configuration closes apache/incubator-metron#791
> > METRON-1234: fix for WARNING 'dependencies.dependency.(
> > groupId:artifactId:type:classifier)' must be unique:
> > org.apache.hadoop:hadoop-yarn-api:jar (dbist via mmiklavc) closes
> > apache/metron#790
> > METRON-1222: fix warning for The expression ${parent.version}
> > is deprecated. Please use ${project.parent.version} instead. (dbist via
> > mmiklavc) closes apache/metron#782
> > METRON-1220 Create documentation around alert nested field
> > (justinleet) closes apache/metron#780
> > METRON-1229 Management UI type is part of the declarations of
> > 2 modules (merrimanr) closes apache/metron#784
> > METRON-1228: Configuration Management PUSH immediately does
> > DUMP after (mmiklavc via mmiklavc) closes apache/metron#783
> > METRON-1218 Metron REST should return better error messages
> > (merrimanr) closes apache/metron#779
> > METRON-1161 Add ability to edit parser command line options
> in
> > the management UI (merrimanr) closes apache/metron#737
> > METRON-1209: Make stellar repl take logging properties, like
> > other CLI apps in metron closes apache/incubator-metron#772
> > METRON-1059 address checkstyle warning AvoidStarImport in
> > metron-stellar (dbist via ottobackwards) closes apache/metron#664
> > METRON-1204 UI does not time out after being idle, but stops
> > functioning (merrimanr) closes apache/metron#771
> > METRON-1052: Add forensic similarity hash functions to
> Stellar
> > closes apache/incubator-metron#781
> > METRON-632: Added validation of "shew.enrichmentType" and
> > "shew.keyColumns" closes apache/incubator-metron#732
> > METRON-1194 Add Profiler Debug Functions to Profiler README
> > (nickwallen via ottobackwards) closes apache/metron#765
> > METRON-1055 Metron 0.4.0 manual installation guide for CentOS
> > 6 updates (lvets via ottobackwards) closes apache/metron#661
> > METRON-1079 STELLAR NaN should be a keyword (ottobackwards)
> > closes apache/metron#681
> > METRON-1085 Add REST endpoint to save a user profile for the
> > Alerts UI (merrimanr) closes apache/metron#694
> > METRON-1208 MPack for Alerts UI (merrimanr) closes
> > apache/metron#778
> > METRON-1207 Make RPMs for Alerts UI (merrimanr) closes
> > apache/metron#777
> > METRON-1215 Fix link to RPMs chapter (DimDroll via
> justinleet)
> > closes apache/metron#776
> > METRON-1206 Make alerts UI conform to ops UI for install
> > (merrimanr) closes apache/metron#773
> > METRON-1195 Meta alerts improperly handle updates to
> non-alert
> > fields (justinleet) closes apache/metron#766
> > METRON-1189 Add alert escalation to the Alerts UI (merrimanr)
> > closes apache/metron#762
> > METRON-1156 Simulate Triage Rules in the Stellar REPL
> > (nickwallen) closes apache/metron#733
> > METRON-1198 Pycapa - No such configuration property
> > 'sasl.kerberos.principal' (nickwallen) closes apache/metron#769
> > METRON-1202 ElasticsearchDao Has extraneous sleep call
> > (justinleet) closes apache/metron#770
> > METRON-938 "service metron-rest start <password>" does not
> > work on CentOS 7. (justinleet) closes apache/metron#757
> > METRON-1182 Refactor Code in alert list to accommodate new
> > view types (iraghumitra via merrimanr) closes apache/metron#756
> > METRON-1188: Ambari global configuration management
> (mmiklavc)
> > closes apache/metron#760
> > METRON-1191 update public web site to point at 0.4.1 new
> > release (mattf-horton) closes apache/metron#764
> > METRON-1063 address javadoc warnings in metron-stellar (dbist
> > via ottobackwards) closes apache/metron#668
> > METRON-1190 Fix Meta Alert Type handling in calculation of
> > scores (justinleet) closes apache/metron#763
> > METRON-1187 Indexing/Profiler Kafka ACL Groups Not Setup
> > Correctly (nickwallen) closes apache/metron#759
> > METRON-1185: Stellar REPL does not work on a kerberized
> > cluster when calling functions interacting with HBase closes
> > apache/incubator-metron#755
> > METRON-1186: Profiler Functions use classutils from shaded
> > storm closes apache/incubator-metron#758
> > METRON-1173: Fix pointers to old stellar docs closes
> > apache/incubator-metron#746
> > METRON-1179: Make STATS_ADD to take a list closes
> > apache/incubator-metron#750
> > METRON-1180: Make Stellar Shell accept zookeeper quorum as a
> > CSV list and not require a port closes apache/incubator-metron#751
> > METRON-1183 Improve KDC Setup Instructions (nickwallen)
> closes
> > apache/metron#753
> > METRON-1177 Stale running topologies seen post-kerberization
> > and cause exceptions (nickwallen) closes apache/metron#748
> > METRON-1158 Build backend for grouping alerts into meta
> alerts
> > (justinleet) closes apache/metron#734
> > METRON-1146: Add ability to parse JSON string into JSONObject
> > for stellar closes apache/incubator-metron#727
> > METRON-1176 REST: HDFS Service should support setting
> > permissions on files when writing (ottobackwards) closes
> apache/metron#749
> > METRON-1114 Add group by capabilities to search REST endpoint
> > (merrimanr) closes apache/metron#702
> > METRON-1167 Define Session Specific Global Configuration
> > Values in the REPL (nickwallen) closes apache/metron#740
> > METRON-1171: Better validation for the SUBSTRING stellar
> > function closes apache/incubator-metron#745
> >
> >
> >
> > On 11/17/17, 11:59 AM, "Nick Allen" <ni...@nickallen.org> wrote:
> >
> > I just wanted to send an update on where we are at. We've
> > gotten a lot
> > done here recently as you can see below.
> >
> > ✓ DONE (1) First, METRON-1289 needs to go in. This one was
> > a fairly big
> > effort and I am hearing that we are pretty close.
> >
> > ✓ DONE (2) METRON-1294 fixes an issue in how field types
> are
> > looked-up.
> >
> > ✓ DONE (3) METRON-1290 is next. While this may have been
> > fixed in
> > M-1289, there may be some test cases we want from this PR.
> >
> > ✓ DONE (4) METRON-1301 addresses a problem with the sorting
> > logic.
> >
> > ✓ DONE (5) METRON-1291 fixes an issue with escalation of
> > metaalerts.
> >
> > (6) That leads us to Raghu's UI work in METRON-1252. This
> > introduces the
> > UI bits that depend on all the previous backend work.
> >
> > (7) At this point, we should have our best effort at
> running
> > Metaalerts
> > on Elasticsearch 2.x. I propose that we cut a release here.
> >
> > (8) After we cut the release, we can introduce the work for
> > ES 5.x in
> > METRON-939. I know we will need lots of help testing and
> > reviewing this
> > one.
> >
> >
> >
> > We also have an outstanding question that needs resolved
> > BEFORE we
> > release. We need to come to a consensus on how to release
> > having moved our
> > Bro Plugin to a separate repo. I don't think we've heard
> from
> > everyone on
> > this. I'd urge everyone to chime in so we can choose a path
> > forward.
> >
> > If anyone is totally confused in regards to that discussion,
> I
> > can try and
> > send an options summary again as a separate discuss thread.
> > The original
> > chain was somewhere around here [1].
> >
> > [1]
> > https://lists.apache.org/thread.html/
> > 54a4474881b97e559df24728b3a0e923a58345a282451085eef832ef@%
> > 3Cdev.metron.apache.org%3E
> >
> >
> >
> > On Wed, Nov 15, 2017 at 10:04 AM, Nick Allen <
> > nick@nickallen.org> wrote:
> >
> > > Hi Guys -
> > >
> > > I want to follow-up on this discussion. It sounds like
> most
> > people are in
> > > agreement with the general approach.
> > >
> > > A lot of people have been working hard on Metaalerts and
> > Elasticsearch. I
> > > have checked-in with those doing the heavy lifting and have
> > compiled a more
> > > detailed plan based on where we are at now. To the best of
> > my knowledge
> > > here is the plan of attack for finishing out this effort.
> > >
> > > (1) First, METRON-1289 needs to go in. This one was a
> > fairly big effort
> > > and I am hearing that we are pretty close.
> > >
> > > (2) METRON-1294 fixes an issue in how field types are
> > looked-up.
> > >
> > > (3) METRON-1290 is next. While this may have been fixed
> > in M-1289,
> > > there may be some test cases we want from this PR.
> > >
> > > (4) METRON-1301 addresses a problem with the sorting
> logic.
> > >
> > > (5) METRON-1291 fixes an issue with escalation of
> > metaalerts.
> > >
> > > (6) That leads us to Raghu's UI work in METRON-1252.
> This
> > introduces
> > > the UI bits that depend on all the previous backend work.
> > >
> > > (7) At this point, we should have our best effort at
> > running Metaalerts
> > > on Elasticsearch 2.x. I propose that we cut a release here.
> > >
> > > (8) After we cut the release, we can introduce the work
> > for ES 5.x in
> > > METRON-939. I know we will need lots of help testing and
> > reviewing this
> > > one.
> > >
> > > Please correct me if I am wrong. I will try and send out
> > updates as we
> > > make progress.
> > >
> > >
> > >
> > >
> > >
> > > On Mon, Nov 6, 2017 at 1:03 PM, Zeolla@GMail.com <
> > zeolla@gmail.com> wrote:
> > >
> > >> I agree, I think it's very reasonable to move in line with
> > Nick's
> > >> proposal. I would also suggest that we outline what the
> > target versions
> > >> would be to add in the METRON-777 components, since it has
> > been functional
> > >> for a very long time but not reviewed and has some really
> > rockstar
> > >> improvements.
> > >>
> > >> Jon
> > >>
> > >> On Mon, Nov 6, 2017 at 12:56 PM Otto Fowler <
> > ottobackwards@gmail.com>
> > >> wrote:
> > >>
> > >> > I think the ES cutover should be the start of the 0.5.x
> > series, and we
> > >> > continue on with 0.4.x for the
> > >> > metadata improvements etc. We could chose to focus
> > 0.5.x’s first
> > >> releases
> > >> > on not only ES but
> > >> > getting a handle on kibana and the mpack situation as
> > well.
> > >> >
> > >> >
> > >> >
> > >> >
> > >> > On November 6, 2017 at 12:48:45, Michael Miklavcic (
> > >> > michael.miklavcic@gmail.com) wrote:
> > >> >
> > >> > I agree with your proposal, Nick. I think having a
> > stabilizing release
> > >> > prior to upgrading ES/Kibana makes sense.
> > >> >
> > >> > On Mon, Nov 6, 2017 at 9:16 AM, Nick Allen <
> > nick@nickallen.org> wrote:
> > >> >
> > >> > > I would like to start a discussion around upcoming
> > releases. We have a
> > >> > > couple separate significant tracks of work that we
> need
> > to reconcile
> > >> in
> > >> > our
> > >> > > release schedule.
> > >> > >
> > >> > > (1) We have had (and have in review) a good number of
> > bug fixes
> > >> required
> > >> > to
> > >> > > support Metaalerts on the existing Elasticsearch 2.x
> > infrastructure.
> > >> > >
> > >> > >
> > >> > > (2) We also have ongoing work to upgrade our
> > infrastructure to
> > >> > > Elasticsearch 5.x, which will not be backwards
> > compatible.
> > >> > >
> > >> > >
> > >> > > I would like to see a release that has our best work
> on
> > ES 2.x before
> > >> we
> > >> > > migrate to 5.x. I would propose the following.
> > >> > >
> > >> > > Release N+1: Introduce Metaalerts running on ES 2.x
> > >> > >
> > >> > > Release N+2: Cut-over to ES 5.x
> > >> > >
> > >> > >
> > >> > > (Q) Is it worth cutting a separate release for ES 2.x?
> > Is there a
> > >> better
> > >> > > way to handle the cut-over to 5.x?
> > >> > >
> > >> >
> > >> --
> > >>
> > >> Jon
> > >>
> > >
> > >
> >
> >
> >
> >
> >
> >
> >
> >
>
--
Jon
Re: [DISCUSS] Upcoming Release
Posted by Casey Stella <ce...@gmail.com>.
I would be in favor of a release at this point.
On Mon, Dec 4, 2017 at 4:57 PM, Matt Foley <ma...@apache.org> wrote:
> Hey all,
> I see METRON-1252 was resolved over the weekend. Shall I go ahead and
> start the process with 0.4.2 release?
> Does anyone have any commits they feel strongly should go in before 0.4.2
> is done, or are we ready to call it good?
>
> I believe there is consensus the 0.4.2 release should include a release of
> the current state of the metron-bro-plugin-kafka. I will continue the
> discussion in that thread as to the process for accomplishing that, but
> plan on it happening.
>
> Regards,
> --Matt
>
> On 11/26/17, 6:26 PM, "Matt Foley" <ma...@apache.org> wrote:
>
> Hope everyone (at least in the U.S.) had a great Thanksgiving holiday.
> Regarding status of the release effort, still pending METRON-1252, so
> not making the release branch yet.
>
> Regards,
> --Matt
>
> On 11/17/17, 1:32 PM, "Matt Foley" <ma...@apache.org> wrote:
>
> (With release manager hat on)
>
> The community has proposed a release of Metron in the near future,
> focusing on Meta-alerts running in Elasticsearch.
> Congrats on getting so many of the below already done. At this
> point, only METRON-1252, and the discussion of how to handle joint release
> of the Metron bro plugin, remain as gating items for the release. I
> project these will be resolved next week, so let’s propose the following:
>
> Sometime next week, after the last bits are done, I’ll start the
> release process and create the release branch.
>
> The proposed new version will be 0.4.2, unless there are backward
> incompatible changes that support making it 0.5.0.
> Currently there are NO included Jiras labeled
> ‘backward-incompatible’, nor having Docs Text indicating so.
> ***If anyone knows that some of the commits included since 0.4.1
> introduce backward incompatibility, please say so now on this thread, and
> mark the Jira as such.***
>
> The 90 or so jiras/commits already in master branch since 0.4.1
> are listed below.
> Thanks,
> --Matt
>
> METRON-1301 Alerts UI - Sorting on Triage Score Unexpectedly
> Filters Some Records (nickwallen) closes apache/metron#832
> METRON-1294 IP addresses are not formatted correctly in facet
> and group results (merrimanr) closes apache/metron#827
> METRON-1291 Kafka produce REST endpoint does not work in a
> Kerberized cluster (merrimanr) closes apache/metron#826
> METRON-1290 Only first 10 alerts are update when a MetaAlert
> status is changed to inactive (justinleet) closes apache/metron#842
> METRON-1311 Service Check Should Check Elasticsearch Index
> Templates (nickwallen) closes apache/metron#839
> METRON-1289 Alert fields are lost when a MetaAlert is created
> (merrimanr) closes apache/metron#824
> METRON-1309 Change metron-deployment to pull the plugin from
> apache/metron-bro-plugin-kafka (JonZeolla) closes apache/metron#837
> METRON-1310 Template Delete Action Deletes Search Indices
> (nickwallen) closes apache/metron#838
> METRON-1275: Fix Metron Documentation closes
> apache/incubator-metron#833
> METRON-1295 Unable to Configure Logging for REST API
> (nickwallen) closes apache/metron#828
> METRON-1307 Force install of java8 since java9 does not appear
> to work with the scripts (brianhurley via ottobackwards) closes
> apache/metron#835
> METRON-1296 Full Dev Fails to Deploy Index Templates
> (nickwallen via cestella) closes apache/incubator-metron#829
> METRON-1281 Remove hard-coded indices from the Alerts UI
> (merrimanr) closes apache/metron#821
> METRON-1287 Full Dev Fails When Installing EPEL Repository
> (nickwallen) closes apache/metron#820
> METRON-1267 Alerts UI returns a 404 when refreshing the
> alerts-list page (iraghumitra via merrimanr) closes apache/metron#819
> METRON-1283 Install Elasticsearch template as a part of the
> mpack startup scripts (anandsubbu via nickwallen) closes apache/metron#817
> METRON-1254: Conditionals as map keys do not function in
> Stellar closes apache/incubator-metron#801
> METRON-1261 Apply bro security patch (JonZeolla via
> ottobackwards) closes apache/metron#805
> METRON-1284 Remove extraneous dead query in ElasticsearchDao
> (justinleet) closes apache/metron#818
> METRON-1270: fix for warnings missing @return tag argument in
> metron-analytics/metron-profiler-common and metron-profiler-client closes
> apache/incubator-metron#810
> METRON-1272 Hide child alerts from searches and grouping if
> they belong to meta alerts (justinleet) closes apache/metron#811
> METRON-1224 Add time range selection to search control
> (iraghumitra via james-sirota) closes apache/metron#796
> METRON-1280 0.4.1 -> 0.4.2 missed a couple of projects
> (cestella via justinleet) closes apache/metron#816
> METRON-1243: Add a REST endpoint which allows us to get a list
> of all indice closes apache/incubator-metron#797
> METRON-1196 Increment master version number to 0.4.2 for
> on-going development (mattf-horton) closes apache/metron#767
> METRON-1278 Strip "Build Status" widget from root
> README.md in site-book build (mattf-horton) closes apache/metron#815
> METRON-1274 Master has failure in
> StormControllerIntegrationTest (merrimanr) closes apache/metron#813
> METRON-1266 Profiler - SASL Authentication Failed (nickwallen)
> closes apache/metron#809
> METRON-1260 Include Alerts UI in Ambari Service Check
> (nickwallen) closes apache/metron#804
> METRON-1251: Typo and formatting fixes for metron-rest README
> closes apache/incubator-metron#800
> METRON-1241: Enable the REST API to use a cache for the
> zookeeper config similar to the Bolts closes apache/incubator-metron#795
> METRON-1267 Alerts UI returns a 404 when refreshing the
> alerts-list page (merrimanr) closes apache/metron#808
> METRON-1262 Unable to add comment for a alert in a meta-alert
> (merrimanr) closes apache/metron#806
> METRON-1263 Start Alerts UI service after Metron REST
> (anandsubbu via nickwallen) closes apache/metron#807
> METRON-1255 MetaAlert search is not filtering on status
> (merrimanr) closes apache/metron#802
> METRON-1249 Improve Metron MPack Service Checks (nickwallen)
> closes apache/metron#799
> METRON-1237 address javadoc warnings in metron-maas-common
> (dbist via james-sirota) closes apache/metron#792
> METRON-1240 address javadoc warnings in metron-platform and
> metron-analytics (dbist via james-sirota) closes apache/metron#794
> METRON-1226 Searching Can Errantly Query the Wrong Indices
> (nickwallen) closes apache/metron#793
> METRON-1081 Fix Alerts and Ops UI Notices file (james-sirota)
> closes apache/metron#682
> METRON-1123 Add group by option using faceted search
> capabilities of metron-rest-api (iraghumitra via james-sirota) closes
> apache/metron#768
> METRON-1223 Add support to add comments for alerts
> (iraghumitra via james-sirota) closes apache/metron#788
> METRON-1083 Add filters using faceted search capabilities of
> metron-rest-api (iraghumitra via james-sirota) closes apache/metron#710
> METRON-1232 Alert status changes are not reflected in list
> view (iraghumitra via merrimanr) closes apache/metron#787
> METRON-1247 REST search and findOne endpoints return
> unexpected or incorrect results for guids (justinleet) closes
> apache/metron#798
> METRON-1235: Document the properties pulled from the global
> configuration closes apache/incubator-metron#791
> METRON-1234: fix for WARNING 'dependencies.dependency.(
> groupId:artifactId:type:classifier)' must be unique:
> org.apache.hadoop:hadoop-yarn-api:jar (dbist via mmiklavc) closes
> apache/metron#790
> METRON-1222: fix warning for The expression ${parent.version}
> is deprecated. Please use ${project.parent.version} instead. (dbist via
> mmiklavc) closes apache/metron#782
> METRON-1220 Create documentation around alert nested field
> (justinleet) closes apache/metron#780
> METRON-1229 Management UI type is part of the declarations of
> 2 modules (merrimanr) closes apache/metron#784
> METRON-1228: Configuration Management PUSH immediately does
> DUMP after (mmiklavc via mmiklavc) closes apache/metron#783
> METRON-1218 Metron REST should return better error messages
> (merrimanr) closes apache/metron#779
> METRON-1161 Add ability to edit parser command line options in
> the management UI (merrimanr) closes apache/metron#737
> METRON-1209: Make stellar repl take logging properties, like
> other CLI apps in metron closes apache/incubator-metron#772
> METRON-1059 address checkstyle warning AvoidStarImport in
> metron-stellar (dbist via ottobackwards) closes apache/metron#664
> METRON-1204 UI does not time out after being idle, but stops
> functioning (merrimanr) closes apache/metron#771
> METRON-1052: Add forensic similarity hash functions to Stellar
> closes apache/incubator-metron#781
> METRON-632: Added validation of "shew.enrichmentType" and
> "shew.keyColumns" closes apache/incubator-metron#732
> METRON-1194 Add Profiler Debug Functions to Profiler README
> (nickwallen via ottobackwards) closes apache/metron#765
> METRON-1055 Metron 0.4.0 manual installation guide for CentOS
> 6 updates (lvets via ottobackwards) closes apache/metron#661
> METRON-1079 STELLAR NaN should be a keyword (ottobackwards)
> closes apache/metron#681
> METRON-1085 Add REST endpoint to save a user profile for the
> Alerts UI (merrimanr) closes apache/metron#694
> METRON-1208 MPack for Alerts UI (merrimanr) closes
> apache/metron#778
> METRON-1207 Make RPMs for Alerts UI (merrimanr) closes
> apache/metron#777
> METRON-1215 Fix link to RPMs chapter (DimDroll via justinleet)
> closes apache/metron#776
> METRON-1206 Make alerts UI conform to ops UI for install
> (merrimanr) closes apache/metron#773
> METRON-1195 Meta alerts improperly handle updates to non-alert
> fields (justinleet) closes apache/metron#766
> METRON-1189 Add alert escalation to the Alerts UI (merrimanr)
> closes apache/metron#762
> METRON-1156 Simulate Triage Rules in the Stellar REPL
> (nickwallen) closes apache/metron#733
> METRON-1198 Pycapa - No such configuration property
> 'sasl.kerberos.principal' (nickwallen) closes apache/metron#769
> METRON-1202 ElasticsearchDao Has extraneous sleep call
> (justinleet) closes apache/metron#770
> METRON-938 "service metron-rest start <password>" does not
> work on CentOS 7. (justinleet) closes apache/metron#757
> METRON-1182 Refactor Code in alert list to accommodate new
> view types (iraghumitra via merrimanr) closes apache/metron#756
> METRON-1188: Ambari global configuration management (mmiklavc)
> closes apache/metron#760
> METRON-1191 update public web site to point at 0.4.1 new
> release (mattf-horton) closes apache/metron#764
> METRON-1063 address javadoc warnings in metron-stellar (dbist
> via ottobackwards) closes apache/metron#668
> METRON-1190 Fix Meta Alert Type handling in calculation of
> scores (justinleet) closes apache/metron#763
> METRON-1187 Indexing/Profiler Kafka ACL Groups Not Setup
> Correctly (nickwallen) closes apache/metron#759
> METRON-1185: Stellar REPL does not work on a kerberized
> cluster when calling functions interacting with HBase closes
> apache/incubator-metron#755
> METRON-1186: Profiler Functions use classutils from shaded
> storm closes apache/incubator-metron#758
> METRON-1173: Fix pointers to old stellar docs closes
> apache/incubator-metron#746
> METRON-1179: Make STATS_ADD to take a list closes
> apache/incubator-metron#750
> METRON-1180: Make Stellar Shell accept zookeeper quorum as a
> CSV list and not require a port closes apache/incubator-metron#751
> METRON-1183 Improve KDC Setup Instructions (nickwallen) closes
> apache/metron#753
> METRON-1177 Stale running topologies seen post-kerberization
> and cause exceptions (nickwallen) closes apache/metron#748
> METRON-1158 Build backend for grouping alerts into meta alerts
> (justinleet) closes apache/metron#734
> METRON-1146: Add ability to parse JSON string into JSONObject
> for stellar closes apache/incubator-metron#727
> METRON-1176 REST: HDFS Service should support setting
> permissions on files when writing (ottobackwards) closes apache/metron#749
> METRON-1114 Add group by capabilities to search REST endpoint
> (merrimanr) closes apache/metron#702
> METRON-1167 Define Session Specific Global Configuration
> Values in the REPL (nickwallen) closes apache/metron#740
> METRON-1171: Better validation for the SUBSTRING stellar
> function closes apache/incubator-metron#745
>
>
>
> On 11/17/17, 11:59 AM, "Nick Allen" <ni...@nickallen.org> wrote:
>
> I just wanted to send an update on where we are at. We've
> gotten a lot
> done here recently as you can see below.
>
> ✓ DONE (1) First, METRON-1289 needs to go in. This one was
> a fairly big
> effort and I am hearing that we are pretty close.
>
> ✓ DONE (2) METRON-1294 fixes an issue in how field types are
> looked-up.
>
> ✓ DONE (3) METRON-1290 is next. While this may have been
> fixed in
> M-1289, there may be some test cases we want from this PR.
>
> ✓ DONE (4) METRON-1301 addresses a problem with the sorting
> logic.
>
> ✓ DONE (5) METRON-1291 fixes an issue with escalation of
> metaalerts.
>
> (6) That leads us to Raghu's UI work in METRON-1252. This
> introduces the
> UI bits that depend on all the previous backend work.
>
> (7) At this point, we should have our best effort at running
> Metaalerts
> on Elasticsearch 2.x. I propose that we cut a release here.
>
> (8) After we cut the release, we can introduce the work for
> ES 5.x in
> METRON-939. I know we will need lots of help testing and
> reviewing this
> one.
>
>
>
> We also have an outstanding question that needs resolved
> BEFORE we
> release. We need to come to a consensus on how to release
> having moved our
> Bro Plugin to a separate repo. I don't think we've heard from
> everyone on
> this. I'd urge everyone to chime in so we can choose a path
> forward.
>
> If anyone is totally confused in regards to that discussion, I
> can try and
> send an options summary again as a separate discuss thread.
> The original
> chain was somewhere around here [1].
>
> [1]
> https://lists.apache.org/thread.html/
> 54a4474881b97e559df24728b3a0e923a58345a282451085eef832ef@%
> 3Cdev.metron.apache.org%3E
>
>
>
> On Wed, Nov 15, 2017 at 10:04 AM, Nick Allen <
> nick@nickallen.org> wrote:
>
> > Hi Guys -
> >
> > I want to follow-up on this discussion. It sounds like most
> people are in
> > agreement with the general approach.
> >
> > A lot of people have been working hard on Metaalerts and
> Elasticsearch. I
> > have checked-in with those doing the heavy lifting and have
> compiled a more
> > detailed plan based on where we are at now. To the best of
> my knowledge
> > here is the plan of attack for finishing out this effort.
> >
> > (1) First, METRON-1289 needs to go in. This one was a
> fairly big effort
> > and I am hearing that we are pretty close.
> >
> > (2) METRON-1294 fixes an issue in how field types are
> looked-up.
> >
> > (3) METRON-1290 is next. While this may have been fixed
> in M-1289,
> > there may be some test cases we want from this PR.
> >
> > (4) METRON-1301 addresses a problem with the sorting logic.
> >
> > (5) METRON-1291 fixes an issue with escalation of
> metaalerts.
> >
> > (6) That leads us to Raghu's UI work in METRON-1252. This
> introduces
> > the UI bits that depend on all the previous backend work.
> >
> > (7) At this point, we should have our best effort at
> running Metaalerts
> > on Elasticsearch 2.x. I propose that we cut a release here.
> >
> > (8) After we cut the release, we can introduce the work
> for ES 5.x in
> > METRON-939. I know we will need lots of help testing and
> reviewing this
> > one.
> >
> > Please correct me if I am wrong. I will try and send out
> updates as we
> > make progress.
> >
> >
> >
> >
> >
> > On Mon, Nov 6, 2017 at 1:03 PM, Zeolla@GMail.com <
> zeolla@gmail.com> wrote:
> >
> >> I agree, I think it's very reasonable to move in line with
> Nick's
> >> proposal. I would also suggest that we outline what the
> target versions
> >> would be to add in the METRON-777 components, since it has
> been functional
> >> for a very long time but not reviewed and has some really
> rockstar
> >> improvements.
> >>
> >> Jon
> >>
> >> On Mon, Nov 6, 2017 at 12:56 PM Otto Fowler <
> ottobackwards@gmail.com>
> >> wrote:
> >>
> >> > I think the ES cutover should be the start of the 0.5.x
> series, and we
> >> > continue on with 0.4.x for the
> >> > metadata improvements etc. We could chose to focus
> 0.5.x’s first
> >> releases
> >> > on not only ES but
> >> > getting a handle on kibana and the mpack situation as
> well.
> >> >
> >> >
> >> >
> >> >
> >> > On November 6, 2017 at 12:48:45, Michael Miklavcic (
> >> > michael.miklavcic@gmail.com) wrote:
> >> >
> >> > I agree with your proposal, Nick. I think having a
> stabilizing release
> >> > prior to upgrading ES/Kibana makes sense.
> >> >
> >> > On Mon, Nov 6, 2017 at 9:16 AM, Nick Allen <
> nick@nickallen.org> wrote:
> >> >
> >> > > I would like to start a discussion around upcoming
> releases. We have a
> >> > > couple separate significant tracks of work that we need
> to reconcile
> >> in
> >> > our
> >> > > release schedule.
> >> > >
> >> > > (1) We have had (and have in review) a good number of
> bug fixes
> >> required
> >> > to
> >> > > support Metaalerts on the existing Elasticsearch 2.x
> infrastructure.
> >> > >
> >> > >
> >> > > (2) We also have ongoing work to upgrade our
> infrastructure to
> >> > > Elasticsearch 5.x, which will not be backwards
> compatible.
> >> > >
> >> > >
> >> > > I would like to see a release that has our best work on
> ES 2.x before
> >> we
> >> > > migrate to 5.x. I would propose the following.
> >> > >
> >> > > Release N+1: Introduce Metaalerts running on ES 2.x
> >> > >
> >> > > Release N+2: Cut-over to ES 5.x
> >> > >
> >> > >
> >> > > (Q) Is it worth cutting a separate release for ES 2.x?
> Is there a
> >> better
> >> > > way to handle the cut-over to 5.x?
> >> > >
> >> >
> >> --
> >>
> >> Jon
> >>
> >
> >
>
>
>
>
>
>
>
>
Re: [DISCUSS] Upcoming Release
Posted by Matt Foley <ma...@apache.org>.
Hey all,
I see METRON-1252 was resolved over the weekend. Shall I go ahead and start the process with 0.4.2 release?
Does anyone have any commits they feel strongly should go in before 0.4.2 is done, or are we ready to call it good?
I believe there is consensus the 0.4.2 release should include a release of the current state of the metron-bro-plugin-kafka. I will continue the discussion in that thread as to the process for accomplishing that, but plan on it happening.
Regards,
--Matt
On 11/26/17, 6:26 PM, "Matt Foley" <ma...@apache.org> wrote:
Hope everyone (at least in the U.S.) had a great Thanksgiving holiday.
Regarding status of the release effort, still pending METRON-1252, so not making the release branch yet.
Regards,
--Matt
On 11/17/17, 1:32 PM, "Matt Foley" <ma...@apache.org> wrote:
(With release manager hat on)
The community has proposed a release of Metron in the near future, focusing on Meta-alerts running in Elasticsearch.
Congrats on getting so many of the below already done. At this point, only METRON-1252, and the discussion of how to handle joint release of the Metron bro plugin, remain as gating items for the release. I project these will be resolved next week, so let’s propose the following:
Sometime next week, after the last bits are done, I’ll start the release process and create the release branch.
The proposed new version will be 0.4.2, unless there are backward incompatible changes that support making it 0.5.0.
Currently there are NO included Jiras labeled ‘backward-incompatible’, nor having Docs Text indicating so.
***If anyone knows that some of the commits included since 0.4.1 introduce backward incompatibility, please say so now on this thread, and mark the Jira as such.***
The 90 or so jiras/commits already in master branch since 0.4.1 are listed below.
Thanks,
--Matt
METRON-1301 Alerts UI - Sorting on Triage Score Unexpectedly Filters Some Records (nickwallen) closes apache/metron#832
METRON-1294 IP addresses are not formatted correctly in facet and group results (merrimanr) closes apache/metron#827
METRON-1291 Kafka produce REST endpoint does not work in a Kerberized cluster (merrimanr) closes apache/metron#826
METRON-1290 Only first 10 alerts are update when a MetaAlert status is changed to inactive (justinleet) closes apache/metron#842
METRON-1311 Service Check Should Check Elasticsearch Index Templates (nickwallen) closes apache/metron#839
METRON-1289 Alert fields are lost when a MetaAlert is created (merrimanr) closes apache/metron#824
METRON-1309 Change metron-deployment to pull the plugin from apache/metron-bro-plugin-kafka (JonZeolla) closes apache/metron#837
METRON-1310 Template Delete Action Deletes Search Indices (nickwallen) closes apache/metron#838
METRON-1275: Fix Metron Documentation closes apache/incubator-metron#833
METRON-1295 Unable to Configure Logging for REST API (nickwallen) closes apache/metron#828
METRON-1307 Force install of java8 since java9 does not appear to work with the scripts (brianhurley via ottobackwards) closes apache/metron#835
METRON-1296 Full Dev Fails to Deploy Index Templates (nickwallen via cestella) closes apache/incubator-metron#829
METRON-1281 Remove hard-coded indices from the Alerts UI (merrimanr) closes apache/metron#821
METRON-1287 Full Dev Fails When Installing EPEL Repository (nickwallen) closes apache/metron#820
METRON-1267 Alerts UI returns a 404 when refreshing the alerts-list page (iraghumitra via merrimanr) closes apache/metron#819
METRON-1283 Install Elasticsearch template as a part of the mpack startup scripts (anandsubbu via nickwallen) closes apache/metron#817
METRON-1254: Conditionals as map keys do not function in Stellar closes apache/incubator-metron#801
METRON-1261 Apply bro security patch (JonZeolla via ottobackwards) closes apache/metron#805
METRON-1284 Remove extraneous dead query in ElasticsearchDao (justinleet) closes apache/metron#818
METRON-1270: fix for warnings missing @return tag argument in metron-analytics/metron-profiler-common and metron-profiler-client closes apache/incubator-metron#810
METRON-1272 Hide child alerts from searches and grouping if they belong to meta alerts (justinleet) closes apache/metron#811
METRON-1224 Add time range selection to search control (iraghumitra via james-sirota) closes apache/metron#796
METRON-1280 0.4.1 -> 0.4.2 missed a couple of projects (cestella via justinleet) closes apache/metron#816
METRON-1243: Add a REST endpoint which allows us to get a list of all indice closes apache/incubator-metron#797
METRON-1196 Increment master version number to 0.4.2 for on-going development (mattf-horton) closes apache/metron#767
METRON-1278 Strip "Build Status" widget from root README.md in site-book build (mattf-horton) closes apache/metron#815
METRON-1274 Master has failure in StormControllerIntegrationTest (merrimanr) closes apache/metron#813
METRON-1266 Profiler - SASL Authentication Failed (nickwallen) closes apache/metron#809
METRON-1260 Include Alerts UI in Ambari Service Check (nickwallen) closes apache/metron#804
METRON-1251: Typo and formatting fixes for metron-rest README closes apache/incubator-metron#800
METRON-1241: Enable the REST API to use a cache for the zookeeper config similar to the Bolts closes apache/incubator-metron#795
METRON-1267 Alerts UI returns a 404 when refreshing the alerts-list page (merrimanr) closes apache/metron#808
METRON-1262 Unable to add comment for a alert in a meta-alert (merrimanr) closes apache/metron#806
METRON-1263 Start Alerts UI service after Metron REST (anandsubbu via nickwallen) closes apache/metron#807
METRON-1255 MetaAlert search is not filtering on status (merrimanr) closes apache/metron#802
METRON-1249 Improve Metron MPack Service Checks (nickwallen) closes apache/metron#799
METRON-1237 address javadoc warnings in metron-maas-common (dbist via james-sirota) closes apache/metron#792
METRON-1240 address javadoc warnings in metron-platform and metron-analytics (dbist via james-sirota) closes apache/metron#794
METRON-1226 Searching Can Errantly Query the Wrong Indices (nickwallen) closes apache/metron#793
METRON-1081 Fix Alerts and Ops UI Notices file (james-sirota) closes apache/metron#682
METRON-1123 Add group by option using faceted search capabilities of metron-rest-api (iraghumitra via james-sirota) closes apache/metron#768
METRON-1223 Add support to add comments for alerts (iraghumitra via james-sirota) closes apache/metron#788
METRON-1083 Add filters using faceted search capabilities of metron-rest-api (iraghumitra via james-sirota) closes apache/metron#710
METRON-1232 Alert status changes are not reflected in list view (iraghumitra via merrimanr) closes apache/metron#787
METRON-1247 REST search and findOne endpoints return unexpected or incorrect results for guids (justinleet) closes apache/metron#798
METRON-1235: Document the properties pulled from the global configuration closes apache/incubator-metron#791
METRON-1234: fix for WARNING 'dependencies.dependency.(groupId:artifactId:type:classifier)' must be unique: org.apache.hadoop:hadoop-yarn-api:jar (dbist via mmiklavc) closes apache/metron#790
METRON-1222: fix warning for The expression ${parent.version} is deprecated. Please use ${project.parent.version} instead. (dbist via mmiklavc) closes apache/metron#782
METRON-1220 Create documentation around alert nested field (justinleet) closes apache/metron#780
METRON-1229 Management UI type is part of the declarations of 2 modules (merrimanr) closes apache/metron#784
METRON-1228: Configuration Management PUSH immediately does DUMP after (mmiklavc via mmiklavc) closes apache/metron#783
METRON-1218 Metron REST should return better error messages (merrimanr) closes apache/metron#779
METRON-1161 Add ability to edit parser command line options in the management UI (merrimanr) closes apache/metron#737
METRON-1209: Make stellar repl take logging properties, like other CLI apps in metron closes apache/incubator-metron#772
METRON-1059 address checkstyle warning AvoidStarImport in metron-stellar (dbist via ottobackwards) closes apache/metron#664
METRON-1204 UI does not time out after being idle, but stops functioning (merrimanr) closes apache/metron#771
METRON-1052: Add forensic similarity hash functions to Stellar closes apache/incubator-metron#781
METRON-632: Added validation of "shew.enrichmentType" and "shew.keyColumns" closes apache/incubator-metron#732
METRON-1194 Add Profiler Debug Functions to Profiler README (nickwallen via ottobackwards) closes apache/metron#765
METRON-1055 Metron 0.4.0 manual installation guide for CentOS 6 updates (lvets via ottobackwards) closes apache/metron#661
METRON-1079 STELLAR NaN should be a keyword (ottobackwards) closes apache/metron#681
METRON-1085 Add REST endpoint to save a user profile for the Alerts UI (merrimanr) closes apache/metron#694
METRON-1208 MPack for Alerts UI (merrimanr) closes apache/metron#778
METRON-1207 Make RPMs for Alerts UI (merrimanr) closes apache/metron#777
METRON-1215 Fix link to RPMs chapter (DimDroll via justinleet) closes apache/metron#776
METRON-1206 Make alerts UI conform to ops UI for install (merrimanr) closes apache/metron#773
METRON-1195 Meta alerts improperly handle updates to non-alert fields (justinleet) closes apache/metron#766
METRON-1189 Add alert escalation to the Alerts UI (merrimanr) closes apache/metron#762
METRON-1156 Simulate Triage Rules in the Stellar REPL (nickwallen) closes apache/metron#733
METRON-1198 Pycapa - No such configuration property 'sasl.kerberos.principal' (nickwallen) closes apache/metron#769
METRON-1202 ElasticsearchDao Has extraneous sleep call (justinleet) closes apache/metron#770
METRON-938 "service metron-rest start <password>" does not work on CentOS 7. (justinleet) closes apache/metron#757
METRON-1182 Refactor Code in alert list to accommodate new view types (iraghumitra via merrimanr) closes apache/metron#756
METRON-1188: Ambari global configuration management (mmiklavc) closes apache/metron#760
METRON-1191 update public web site to point at 0.4.1 new release (mattf-horton) closes apache/metron#764
METRON-1063 address javadoc warnings in metron-stellar (dbist via ottobackwards) closes apache/metron#668
METRON-1190 Fix Meta Alert Type handling in calculation of scores (justinleet) closes apache/metron#763
METRON-1187 Indexing/Profiler Kafka ACL Groups Not Setup Correctly (nickwallen) closes apache/metron#759
METRON-1185: Stellar REPL does not work on a kerberized cluster when calling functions interacting with HBase closes apache/incubator-metron#755
METRON-1186: Profiler Functions use classutils from shaded storm closes apache/incubator-metron#758
METRON-1173: Fix pointers to old stellar docs closes apache/incubator-metron#746
METRON-1179: Make STATS_ADD to take a list closes apache/incubator-metron#750
METRON-1180: Make Stellar Shell accept zookeeper quorum as a CSV list and not require a port closes apache/incubator-metron#751
METRON-1183 Improve KDC Setup Instructions (nickwallen) closes apache/metron#753
METRON-1177 Stale running topologies seen post-kerberization and cause exceptions (nickwallen) closes apache/metron#748
METRON-1158 Build backend for grouping alerts into meta alerts (justinleet) closes apache/metron#734
METRON-1146: Add ability to parse JSON string into JSONObject for stellar closes apache/incubator-metron#727
METRON-1176 REST: HDFS Service should support setting permissions on files when writing (ottobackwards) closes apache/metron#749
METRON-1114 Add group by capabilities to search REST endpoint (merrimanr) closes apache/metron#702
METRON-1167 Define Session Specific Global Configuration Values in the REPL (nickwallen) closes apache/metron#740
METRON-1171: Better validation for the SUBSTRING stellar function closes apache/incubator-metron#745
On 11/17/17, 11:59 AM, "Nick Allen" <ni...@nickallen.org> wrote:
I just wanted to send an update on where we are at. We've gotten a lot
done here recently as you can see below.
✓ DONE (1) First, METRON-1289 needs to go in. This one was a fairly big
effort and I am hearing that we are pretty close.
✓ DONE (2) METRON-1294 fixes an issue in how field types are looked-up.
✓ DONE (3) METRON-1290 is next. While this may have been fixed in
M-1289, there may be some test cases we want from this PR.
✓ DONE (4) METRON-1301 addresses a problem with the sorting logic.
✓ DONE (5) METRON-1291 fixes an issue with escalation of metaalerts.
(6) That leads us to Raghu's UI work in METRON-1252. This introduces the
UI bits that depend on all the previous backend work.
(7) At this point, we should have our best effort at running Metaalerts
on Elasticsearch 2.x. I propose that we cut a release here.
(8) After we cut the release, we can introduce the work for ES 5.x in
METRON-939. I know we will need lots of help testing and reviewing this
one.
We also have an outstanding question that needs resolved BEFORE we
release. We need to come to a consensus on how to release having moved our
Bro Plugin to a separate repo. I don't think we've heard from everyone on
this. I'd urge everyone to chime in so we can choose a path forward.
If anyone is totally confused in regards to that discussion, I can try and
send an options summary again as a separate discuss thread. The original
chain was somewhere around here [1].
[1]
https://lists.apache.org/thread.html/54a4474881b97e559df24728b3a0e923a58345a282451085eef832ef@%3Cdev.metron.apache.org%3E
On Wed, Nov 15, 2017 at 10:04 AM, Nick Allen <ni...@nickallen.org> wrote:
> Hi Guys -
>
> I want to follow-up on this discussion. It sounds like most people are in
> agreement with the general approach.
>
> A lot of people have been working hard on Metaalerts and Elasticsearch. I
> have checked-in with those doing the heavy lifting and have compiled a more
> detailed plan based on where we are at now. To the best of my knowledge
> here is the plan of attack for finishing out this effort.
>
> (1) First, METRON-1289 needs to go in. This one was a fairly big effort
> and I am hearing that we are pretty close.
>
> (2) METRON-1294 fixes an issue in how field types are looked-up.
>
> (3) METRON-1290 is next. While this may have been fixed in M-1289,
> there may be some test cases we want from this PR.
>
> (4) METRON-1301 addresses a problem with the sorting logic.
>
> (5) METRON-1291 fixes an issue with escalation of metaalerts.
>
> (6) That leads us to Raghu's UI work in METRON-1252. This introduces
> the UI bits that depend on all the previous backend work.
>
> (7) At this point, we should have our best effort at running Metaalerts
> on Elasticsearch 2.x. I propose that we cut a release here.
>
> (8) After we cut the release, we can introduce the work for ES 5.x in
> METRON-939. I know we will need lots of help testing and reviewing this
> one.
>
> Please correct me if I am wrong. I will try and send out updates as we
> make progress.
>
>
>
>
>
> On Mon, Nov 6, 2017 at 1:03 PM, Zeolla@GMail.com <ze...@gmail.com> wrote:
>
>> I agree, I think it's very reasonable to move in line with Nick's
>> proposal. I would also suggest that we outline what the target versions
>> would be to add in the METRON-777 components, since it has been functional
>> for a very long time but not reviewed and has some really rockstar
>> improvements.
>>
>> Jon
>>
>> On Mon, Nov 6, 2017 at 12:56 PM Otto Fowler <ot...@gmail.com>
>> wrote:
>>
>> > I think the ES cutover should be the start of the 0.5.x series, and we
>> > continue on with 0.4.x for the
>> > metadata improvements etc. We could chose to focus 0.5.x’s first
>> releases
>> > on not only ES but
>> > getting a handle on kibana and the mpack situation as well.
>> >
>> >
>> >
>> >
>> > On November 6, 2017 at 12:48:45, Michael Miklavcic (
>> > michael.miklavcic@gmail.com) wrote:
>> >
>> > I agree with your proposal, Nick. I think having a stabilizing release
>> > prior to upgrading ES/Kibana makes sense.
>> >
>> > On Mon, Nov 6, 2017 at 9:16 AM, Nick Allen <ni...@nickallen.org> wrote:
>> >
>> > > I would like to start a discussion around upcoming releases. We have a
>> > > couple separate significant tracks of work that we need to reconcile
>> in
>> > our
>> > > release schedule.
>> > >
>> > > (1) We have had (and have in review) a good number of bug fixes
>> required
>> > to
>> > > support Metaalerts on the existing Elasticsearch 2.x infrastructure.
>> > >
>> > >
>> > > (2) We also have ongoing work to upgrade our infrastructure to
>> > > Elasticsearch 5.x, which will not be backwards compatible.
>> > >
>> > >
>> > > I would like to see a release that has our best work on ES 2.x before
>> we
>> > > migrate to 5.x. I would propose the following.
>> > >
>> > > Release N+1: Introduce Metaalerts running on ES 2.x
>> > >
>> > > Release N+2: Cut-over to ES 5.x
>> > >
>> > >
>> > > (Q) Is it worth cutting a separate release for ES 2.x? Is there a
>> better
>> > > way to handle the cut-over to 5.x?
>> > >
>> >
>> --
>>
>> Jon
>>
>
>
Re: [DISCUSS] Upcoming Release
Posted by Otto Fowler <ot...@gmail.com>.
Considering the problems we are having with people building the node stuff
on centos, would it make sense to wait to take
a potential PR that allows full metron builds in our ansible docker image?
On November 26, 2017 at 21:26:27, Matt Foley (mattf@apache.org) wrote:
Hope everyone (at least in the U.S.) had a great Thanksgiving holiday.
Regarding status of the release effort, still pending METRON-1252, so not
making the release branch yet.
Regards,
--Matt
On 11/17/17, 1:32 PM, "Matt Foley" <ma...@apache.org> wrote:
(With release manager hat on)
The community has proposed a release of Metron in the near future, focusing
on Meta-alerts running in Elasticsearch.
Congrats on getting so many of the below already done. At this point, only
METRON-1252, and the discussion of how to handle joint release of the
Metron bro plugin, remain as gating items for the release. I project these
will be resolved next week, so let’s propose the following:
Sometime next week, after the last bits are done, I’ll start the release
process and create the release branch.
The proposed new version will be 0.4.2, unless there are backward
incompatible changes that support making it 0.5.0.
Currently there are NO included Jiras labeled ‘backward-incompatible’, nor
having Docs Text indicating so.
***If anyone knows that some of the commits included since 0.4.1 introduce
backward incompatibility, please say so now on this thread, and mark the
Jira as such.***
The 90 or so jiras/commits already in master branch since 0.4.1 are listed
below.
Thanks,
--Matt
METRON-1301 Alerts UI - Sorting on Triage Score Unexpectedly Filters Some
Records (nickwallen) closes apache/metron#832
METRON-1294 IP addresses are not formatted correctly in facet and group
results (merrimanr) closes apache/metron#827
METRON-1291 Kafka produce REST endpoint does not work in a Kerberized
cluster (merrimanr) closes apache/metron#826
METRON-1290 Only first 10 alerts are update when a MetaAlert status is
changed to inactive (justinleet) closes apache/metron#842
METRON-1311 Service Check Should Check Elasticsearch Index Templates
(nickwallen) closes apache/metron#839
METRON-1289 Alert fields are lost when a MetaAlert is created (merrimanr)
closes apache/metron#824
METRON-1309 Change metron-deployment to pull the plugin from
apache/metron-bro-plugin-kafka (JonZeolla) closes apache/metron#837
METRON-1310 Template Delete Action Deletes Search Indices (nickwallen)
closes apache/metron#838
METRON-1275: Fix Metron Documentation closes apache/incubator-metron#833
METRON-1295 Unable to Configure Logging for REST API (nickwallen) closes
apache/metron#828
METRON-1307 Force install of java8 since java9 does not appear to work with
the scripts (brianhurley via ottobackwards) closes apache/metron#835
METRON-1296 Full Dev Fails to Deploy Index Templates (nickwallen via
cestella) closes apache/incubator-metron#829
METRON-1281 Remove hard-coded indices from the Alerts UI (merrimanr) closes
apache/metron#821
METRON-1287 Full Dev Fails When Installing EPEL Repository (nickwallen)
closes apache/metron#820
METRON-1267 Alerts UI returns a 404 when refreshing the alerts-list page
(iraghumitra via merrimanr) closes apache/metron#819
METRON-1283 Install Elasticsearch template as a part of the mpack startup
scripts (anandsubbu via nickwallen) closes apache/metron#817
METRON-1254: Conditionals as map keys do not function in Stellar closes
apache/incubator-metron#801
METRON-1261 Apply bro security patch (JonZeolla via ottobackwards) closes
apache/metron#805
METRON-1284 Remove extraneous dead query in ElasticsearchDao (justinleet)
closes apache/metron#818
METRON-1270: fix for warnings missing @return tag argument in
metron-analytics/metron-profiler-common and metron-profiler-client closes
apache/incubator-metron#810
METRON-1272 Hide child alerts from searches and grouping if they belong to
meta alerts (justinleet) closes apache/metron#811
METRON-1224 Add time range selection to search control (iraghumitra via
james-sirota) closes apache/metron#796
METRON-1280 0.4.1 -> 0.4.2 missed a couple of projects (cestella via
justinleet) closes apache/metron#816
METRON-1243: Add a REST endpoint which allows us to get a list of all
indice closes apache/incubator-metron#797
METRON-1196 Increment master version number to 0.4.2 for on-going
development (mattf-horton) closes apache/metron#767
METRON-1278 Strip "Build Status" widget from root README.md in
site-book build (mattf-horton) closes apache/metron#815
METRON-1274 Master has failure in StormControllerIntegrationTest
(merrimanr) closes apache/metron#813
METRON-1266 Profiler - SASL Authentication Failed (nickwallen) closes
apache/metron#809
METRON-1260 Include Alerts UI in Ambari Service Check (nickwallen) closes
apache/metron#804
METRON-1251: Typo and formatting fixes for metron-rest README closes
apache/incubator-metron#800
METRON-1241: Enable the REST API to use a cache for the zookeeper config
similar to the Bolts closes apache/incubator-metron#795
METRON-1267 Alerts UI returns a 404 when refreshing the alerts-list page
(merrimanr) closes apache/metron#808
METRON-1262 Unable to add comment for a alert in a meta-alert (merrimanr)
closes apache/metron#806
METRON-1263 Start Alerts UI service after Metron REST (anandsubbu via
nickwallen) closes apache/metron#807
METRON-1255 MetaAlert search is not filtering on status (merrimanr) closes
apache/metron#802
METRON-1249 Improve Metron MPack Service Checks (nickwallen) closes
apache/metron#799
METRON-1237 address javadoc warnings in metron-maas-common (dbist via
james-sirota) closes apache/metron#792
METRON-1240 address javadoc warnings in metron-platform and
metron-analytics (dbist via james-sirota) closes apache/metron#794
METRON-1226 Searching Can Errantly Query the Wrong Indices (nickwallen)
closes apache/metron#793
METRON-1081 Fix Alerts and Ops UI Notices file (james-sirota) closes
apache/metron#682
METRON-1123 Add group by option using faceted search capabilities of
metron-rest-api (iraghumitra via james-sirota) closes apache/metron#768
METRON-1223 Add support to add comments for alerts (iraghumitra via
james-sirota) closes apache/metron#788
METRON-1083 Add filters using faceted search capabilities of
metron-rest-api (iraghumitra via james-sirota) closes apache/metron#710
METRON-1232 Alert status changes are not reflected in list view
(iraghumitra via merrimanr) closes apache/metron#787
METRON-1247 REST search and findOne endpoints return unexpected or
incorrect results for guids (justinleet) closes apache/metron#798
METRON-1235: Document the properties pulled from the global configuration
closes apache/incubator-metron#791
METRON-1234: fix for WARNING
'dependencies.dependency.(groupId:artifactId:type:classifier)' must be
unique: org.apache.hadoop:hadoop-yarn-api:jar (dbist via mmiklavc) closes
apache/metron#790
METRON-1222: fix warning for The expression ${parent.version} is
deprecated. Please use ${project.parent.version} instead. (dbist via
mmiklavc) closes apache/metron#782
METRON-1220 Create documentation around alert nested field (justinleet)
closes apache/metron#780
METRON-1229 Management UI type is part of the declarations of 2 modules
(merrimanr) closes apache/metron#784
METRON-1228: Configuration Management PUSH immediately does DUMP after
(mmiklavc via mmiklavc) closes apache/metron#783
METRON-1218 Metron REST should return better error messages (merrimanr)
closes apache/metron#779
METRON-1161 Add ability to edit parser command line options in the
management UI (merrimanr) closes apache/metron#737
METRON-1209: Make stellar repl take logging properties, like other CLI apps
in metron closes apache/incubator-metron#772
METRON-1059 address checkstyle warning AvoidStarImport in metron-stellar
(dbist via ottobackwards) closes apache/metron#664
METRON-1204 UI does not time out after being idle, but stops functioning
(merrimanr) closes apache/metron#771
METRON-1052: Add forensic similarity hash functions to Stellar closes
apache/incubator-metron#781
METRON-632: Added validation of "shew.enrichmentType" and "shew.keyColumns"
closes apache/incubator-metron#732
METRON-1194 Add Profiler Debug Functions to Profiler README (nickwallen via
ottobackwards) closes apache/metron#765
METRON-1055 Metron 0.4.0 manual installation guide for CentOS 6 updates
(lvets via ottobackwards) closes apache/metron#661
METRON-1079 STELLAR NaN should be a keyword (ottobackwards) closes
apache/metron#681
METRON-1085 Add REST endpoint to save a user profile for the Alerts UI
(merrimanr) closes apache/metron#694
METRON-1208 MPack for Alerts UI (merrimanr) closes apache/metron#778
METRON-1207 Make RPMs for Alerts UI (merrimanr) closes apache/metron#777
METRON-1215 Fix link to RPMs chapter (DimDroll via justinleet) closes
apache/metron#776
METRON-1206 Make alerts UI conform to ops UI for install (merrimanr) closes
apache/metron#773
METRON-1195 Meta alerts improperly handle updates to non-alert fields
(justinleet) closes apache/metron#766
METRON-1189 Add alert escalation to the Alerts UI (merrimanr) closes
apache/metron#762
METRON-1156 Simulate Triage Rules in the Stellar REPL (nickwallen) closes
apache/metron#733
METRON-1198 Pycapa - No such configuration property
'sasl.kerberos.principal' (nickwallen) closes apache/metron#769
METRON-1202 ElasticsearchDao Has extraneous sleep call (justinleet) closes
apache/metron#770
METRON-938 "service metron-rest start <password>" does not work on CentOS
7. (justinleet) closes apache/metron#757
METRON-1182 Refactor Code in alert list to accommodate new view types
(iraghumitra via merrimanr) closes apache/metron#756
METRON-1188: Ambari global configuration management (mmiklavc) closes
apache/metron#760
METRON-1191 update public web site to point at 0.4.1 new release
(mattf-horton) closes apache/metron#764
METRON-1063 address javadoc warnings in metron-stellar (dbist via
ottobackwards) closes apache/metron#668
METRON-1190 Fix Meta Alert Type handling in calculation of scores
(justinleet) closes apache/metron#763
METRON-1187 Indexing/Profiler Kafka ACL Groups Not Setup Correctly
(nickwallen) closes apache/metron#759
METRON-1185: Stellar REPL does not work on a kerberized cluster when
calling functions interacting with HBase closes apache/incubator-metron#755
METRON-1186: Profiler Functions use classutils from shaded storm closes
apache/incubator-metron#758
METRON-1173: Fix pointers to old stellar docs closes
apache/incubator-metron#746
METRON-1179: Make STATS_ADD to take a list closes
apache/incubator-metron#750
METRON-1180: Make Stellar Shell accept zookeeper quorum as a CSV list and
not require a port closes apache/incubator-metron#751
METRON-1183 Improve KDC Setup Instructions (nickwallen) closes
apache/metron#753
METRON-1177 Stale running topologies seen post-kerberization and cause
exceptions (nickwallen) closes apache/metron#748
METRON-1158 Build backend for grouping alerts into meta alerts (justinleet)
closes apache/metron#734
METRON-1146: Add ability to parse JSON string into JSONObject for stellar
closes apache/incubator-metron#727
METRON-1176 REST: HDFS Service should support setting permissions on files
when writing (ottobackwards) closes apache/metron#749
METRON-1114 Add group by capabilities to search REST endpoint (merrimanr)
closes apache/metron#702
METRON-1167 Define Session Specific Global Configuration Values in the REPL
(nickwallen) closes apache/metron#740
METRON-1171: Better validation for the SUBSTRING stellar function closes
apache/incubator-metron#745
On 11/17/17, 11:59 AM, "Nick Allen" <ni...@nickallen.org> wrote:
I just wanted to send an update on where we are at. We've gotten a lot
done here recently as you can see below.
✓ DONE (1) First, METRON-1289 needs to go in. This one was a fairly big
effort and I am hearing that we are pretty close.
✓ DONE (2) METRON-1294 fixes an issue in how field types are looked-up.
✓ DONE (3) METRON-1290 is next. While this may have been fixed in
M-1289, there may be some test cases we want from this PR.
✓ DONE (4) METRON-1301 addresses a problem with the sorting logic.
✓ DONE (5) METRON-1291 fixes an issue with escalation of metaalerts.
(6) That leads us to Raghu's UI work in METRON-1252. This introduces the
UI bits that depend on all the previous backend work.
(7) At this point, we should have our best effort at running Metaalerts
on Elasticsearch 2.x. I propose that we cut a release here.
(8) After we cut the release, we can introduce the work for ES 5.x in
METRON-939. I know we will need lots of help testing and reviewing this
one.
We also have an outstanding question that needs resolved BEFORE we
release. We need to come to a consensus on how to release having moved our
Bro Plugin to a separate repo. I don't think we've heard from everyone on
this. I'd urge everyone to chime in so we can choose a path forward.
If anyone is totally confused in regards to that discussion, I can try and
send an options summary again as a separate discuss thread. The original
chain was somewhere around here [1].
[1]
https://lists.apache.org/thread.html/54a4474881b97e559df24728b3a0e923a58345a282451085eef832ef@%3Cdev.metron.apache.org%3E
On Wed, Nov 15, 2017 at 10:04 AM, Nick Allen <ni...@nickallen.org> wrote:
> Hi Guys -
>
> I want to follow-up on this discussion. It sounds like most people are in
> agreement with the general approach.
>
> A lot of people have been working hard on Metaalerts and Elasticsearch. I
> have checked-in with those doing the heavy lifting and have compiled a
more
> detailed plan based on where we are at now. To the best of my knowledge
> here is the plan of attack for finishing out this effort.
>
> (1) First, METRON-1289 needs to go in. This one was a fairly big effort
> and I am hearing that we are pretty close.
>
> (2) METRON-1294 fixes an issue in how field types are looked-up.
>
> (3) METRON-1290 is next. While this may have been fixed in M-1289,
> there may be some test cases we want from this PR.
>
> (4) METRON-1301 addresses a problem with the sorting logic.
>
> (5) METRON-1291 fixes an issue with escalation of metaalerts.
>
> (6) That leads us to Raghu's UI work in METRON-1252. This introduces
> the UI bits that depend on all the previous backend work.
>
> (7) At this point, we should have our best effort at running Metaalerts
> on Elasticsearch 2.x. I propose that we cut a release here.
>
> (8) After we cut the release, we can introduce the work for ES 5.x in
> METRON-939. I know we will need lots of help testing and reviewing this
> one.
>
> Please correct me if I am wrong. I will try and send out updates as we
> make progress.
>
>
>
>
>
> On Mon, Nov 6, 2017 at 1:03 PM, Zeolla@GMail.com <ze...@gmail.com>
wrote:
>
>> I agree, I think it's very reasonable to move in line with Nick's
>> proposal. I would also suggest that we outline what the target versions
>> would be to add in the METRON-777 components, since it has been
functional
>> for a very long time but not reviewed and has some really rockstar
>> improvements.
>>
>> Jon
>>
>> On Mon, Nov 6, 2017 at 12:56 PM Otto Fowler <ot...@gmail.com>
>> wrote:
>>
>> > I think the ES cutover should be the start of the 0.5.x series, and we
>> > continue on with 0.4.x for the
>> > metadata improvements etc. We could chose to focus 0.5.x’s first
>> releases
>> > on not only ES but
>> > getting a handle on kibana and the mpack situation as well.
>> >
>> >
>> >
>> >
>> > On November 6, 2017 at 12:48:45, Michael Miklavcic (
>> > michael.miklavcic@gmail.com) wrote:
>> >
>> > I agree with your proposal, Nick. I think having a stabilizing release
>> > prior to upgrading ES/Kibana makes sense.
>> >
>> > On Mon, Nov 6, 2017 at 9:16 AM, Nick Allen <ni...@nickallen.org> wrote:
>> >
>> > > I would like to start a discussion around upcoming releases. We have
a
>> > > couple separate significant tracks of work that we need to reconcile
>> in
>> > our
>> > > release schedule.
>> > >
>> > > (1) We have had (and have in review) a good number of bug fixes
>> required
>> > to
>> > > support Metaalerts on the existing Elasticsearch 2.x infrastructure.
>> > >
>> > >
>> > > (2) We also have ongoing work to upgrade our infrastructure to
>> > > Elasticsearch 5.x, which will not be backwards compatible.
>> > >
>> > >
>> > > I would like to see a release that has our best work on ES 2.x
before
>> we
>> > > migrate to 5.x. I would propose the following.
>> > >
>> > > Release N+1: Introduce Metaalerts running on ES 2.x
>> > >
>> > > Release N+2: Cut-over to ES 5.x
>> > >
>> > >
>> > > (Q) Is it worth cutting a separate release for ES 2.x? Is there a
>> better
>> > > way to handle the cut-over to 5.x?
>> > >
>> >
>> --
>>
>> Jon
>>
>
>
Re: [DISCUSS] Upcoming Release
Posted by Matt Foley <ma...@apache.org>.
Hope everyone (at least in the U.S.) had a great Thanksgiving holiday.
Regarding status of the release effort, still pending METRON-1252, so not making the release branch yet.
Regards,
--Matt
On 11/17/17, 1:32 PM, "Matt Foley" <ma...@apache.org> wrote:
(With release manager hat on)
The community has proposed a release of Metron in the near future, focusing on Meta-alerts running in Elasticsearch.
Congrats on getting so many of the below already done. At this point, only METRON-1252, and the discussion of how to handle joint release of the Metron bro plugin, remain as gating items for the release. I project these will be resolved next week, so let’s propose the following:
Sometime next week, after the last bits are done, I’ll start the release process and create the release branch.
The proposed new version will be 0.4.2, unless there are backward incompatible changes that support making it 0.5.0.
Currently there are NO included Jiras labeled ‘backward-incompatible’, nor having Docs Text indicating so.
***If anyone knows that some of the commits included since 0.4.1 introduce backward incompatibility, please say so now on this thread, and mark the Jira as such.***
The 90 or so jiras/commits already in master branch since 0.4.1 are listed below.
Thanks,
--Matt
METRON-1301 Alerts UI - Sorting on Triage Score Unexpectedly Filters Some Records (nickwallen) closes apache/metron#832
METRON-1294 IP addresses are not formatted correctly in facet and group results (merrimanr) closes apache/metron#827
METRON-1291 Kafka produce REST endpoint does not work in a Kerberized cluster (merrimanr) closes apache/metron#826
METRON-1290 Only first 10 alerts are update when a MetaAlert status is changed to inactive (justinleet) closes apache/metron#842
METRON-1311 Service Check Should Check Elasticsearch Index Templates (nickwallen) closes apache/metron#839
METRON-1289 Alert fields are lost when a MetaAlert is created (merrimanr) closes apache/metron#824
METRON-1309 Change metron-deployment to pull the plugin from apache/metron-bro-plugin-kafka (JonZeolla) closes apache/metron#837
METRON-1310 Template Delete Action Deletes Search Indices (nickwallen) closes apache/metron#838
METRON-1275: Fix Metron Documentation closes apache/incubator-metron#833
METRON-1295 Unable to Configure Logging for REST API (nickwallen) closes apache/metron#828
METRON-1307 Force install of java8 since java9 does not appear to work with the scripts (brianhurley via ottobackwards) closes apache/metron#835
METRON-1296 Full Dev Fails to Deploy Index Templates (nickwallen via cestella) closes apache/incubator-metron#829
METRON-1281 Remove hard-coded indices from the Alerts UI (merrimanr) closes apache/metron#821
METRON-1287 Full Dev Fails When Installing EPEL Repository (nickwallen) closes apache/metron#820
METRON-1267 Alerts UI returns a 404 when refreshing the alerts-list page (iraghumitra via merrimanr) closes apache/metron#819
METRON-1283 Install Elasticsearch template as a part of the mpack startup scripts (anandsubbu via nickwallen) closes apache/metron#817
METRON-1254: Conditionals as map keys do not function in Stellar closes apache/incubator-metron#801
METRON-1261 Apply bro security patch (JonZeolla via ottobackwards) closes apache/metron#805
METRON-1284 Remove extraneous dead query in ElasticsearchDao (justinleet) closes apache/metron#818
METRON-1270: fix for warnings missing @return tag argument in metron-analytics/metron-profiler-common and metron-profiler-client closes apache/incubator-metron#810
METRON-1272 Hide child alerts from searches and grouping if they belong to meta alerts (justinleet) closes apache/metron#811
METRON-1224 Add time range selection to search control (iraghumitra via james-sirota) closes apache/metron#796
METRON-1280 0.4.1 -> 0.4.2 missed a couple of projects (cestella via justinleet) closes apache/metron#816
METRON-1243: Add a REST endpoint which allows us to get a list of all indice closes apache/incubator-metron#797
METRON-1196 Increment master version number to 0.4.2 for on-going development (mattf-horton) closes apache/metron#767
METRON-1278 Strip "Build Status" widget from root README.md in site-book build (mattf-horton) closes apache/metron#815
METRON-1274 Master has failure in StormControllerIntegrationTest (merrimanr) closes apache/metron#813
METRON-1266 Profiler - SASL Authentication Failed (nickwallen) closes apache/metron#809
METRON-1260 Include Alerts UI in Ambari Service Check (nickwallen) closes apache/metron#804
METRON-1251: Typo and formatting fixes for metron-rest README closes apache/incubator-metron#800
METRON-1241: Enable the REST API to use a cache for the zookeeper config similar to the Bolts closes apache/incubator-metron#795
METRON-1267 Alerts UI returns a 404 when refreshing the alerts-list page (merrimanr) closes apache/metron#808
METRON-1262 Unable to add comment for a alert in a meta-alert (merrimanr) closes apache/metron#806
METRON-1263 Start Alerts UI service after Metron REST (anandsubbu via nickwallen) closes apache/metron#807
METRON-1255 MetaAlert search is not filtering on status (merrimanr) closes apache/metron#802
METRON-1249 Improve Metron MPack Service Checks (nickwallen) closes apache/metron#799
METRON-1237 address javadoc warnings in metron-maas-common (dbist via james-sirota) closes apache/metron#792
METRON-1240 address javadoc warnings in metron-platform and metron-analytics (dbist via james-sirota) closes apache/metron#794
METRON-1226 Searching Can Errantly Query the Wrong Indices (nickwallen) closes apache/metron#793
METRON-1081 Fix Alerts and Ops UI Notices file (james-sirota) closes apache/metron#682
METRON-1123 Add group by option using faceted search capabilities of metron-rest-api (iraghumitra via james-sirota) closes apache/metron#768
METRON-1223 Add support to add comments for alerts (iraghumitra via james-sirota) closes apache/metron#788
METRON-1083 Add filters using faceted search capabilities of metron-rest-api (iraghumitra via james-sirota) closes apache/metron#710
METRON-1232 Alert status changes are not reflected in list view (iraghumitra via merrimanr) closes apache/metron#787
METRON-1247 REST search and findOne endpoints return unexpected or incorrect results for guids (justinleet) closes apache/metron#798
METRON-1235: Document the properties pulled from the global configuration closes apache/incubator-metron#791
METRON-1234: fix for WARNING 'dependencies.dependency.(groupId:artifactId:type:classifier)' must be unique: org.apache.hadoop:hadoop-yarn-api:jar (dbist via mmiklavc) closes apache/metron#790
METRON-1222: fix warning for The expression ${parent.version} is deprecated. Please use ${project.parent.version} instead. (dbist via mmiklavc) closes apache/metron#782
METRON-1220 Create documentation around alert nested field (justinleet) closes apache/metron#780
METRON-1229 Management UI type is part of the declarations of 2 modules (merrimanr) closes apache/metron#784
METRON-1228: Configuration Management PUSH immediately does DUMP after (mmiklavc via mmiklavc) closes apache/metron#783
METRON-1218 Metron REST should return better error messages (merrimanr) closes apache/metron#779
METRON-1161 Add ability to edit parser command line options in the management UI (merrimanr) closes apache/metron#737
METRON-1209: Make stellar repl take logging properties, like other CLI apps in metron closes apache/incubator-metron#772
METRON-1059 address checkstyle warning AvoidStarImport in metron-stellar (dbist via ottobackwards) closes apache/metron#664
METRON-1204 UI does not time out after being idle, but stops functioning (merrimanr) closes apache/metron#771
METRON-1052: Add forensic similarity hash functions to Stellar closes apache/incubator-metron#781
METRON-632: Added validation of "shew.enrichmentType" and "shew.keyColumns" closes apache/incubator-metron#732
METRON-1194 Add Profiler Debug Functions to Profiler README (nickwallen via ottobackwards) closes apache/metron#765
METRON-1055 Metron 0.4.0 manual installation guide for CentOS 6 updates (lvets via ottobackwards) closes apache/metron#661
METRON-1079 STELLAR NaN should be a keyword (ottobackwards) closes apache/metron#681
METRON-1085 Add REST endpoint to save a user profile for the Alerts UI (merrimanr) closes apache/metron#694
METRON-1208 MPack for Alerts UI (merrimanr) closes apache/metron#778
METRON-1207 Make RPMs for Alerts UI (merrimanr) closes apache/metron#777
METRON-1215 Fix link to RPMs chapter (DimDroll via justinleet) closes apache/metron#776
METRON-1206 Make alerts UI conform to ops UI for install (merrimanr) closes apache/metron#773
METRON-1195 Meta alerts improperly handle updates to non-alert fields (justinleet) closes apache/metron#766
METRON-1189 Add alert escalation to the Alerts UI (merrimanr) closes apache/metron#762
METRON-1156 Simulate Triage Rules in the Stellar REPL (nickwallen) closes apache/metron#733
METRON-1198 Pycapa - No such configuration property 'sasl.kerberos.principal' (nickwallen) closes apache/metron#769
METRON-1202 ElasticsearchDao Has extraneous sleep call (justinleet) closes apache/metron#770
METRON-938 "service metron-rest start <password>" does not work on CentOS 7. (justinleet) closes apache/metron#757
METRON-1182 Refactor Code in alert list to accommodate new view types (iraghumitra via merrimanr) closes apache/metron#756
METRON-1188: Ambari global configuration management (mmiklavc) closes apache/metron#760
METRON-1191 update public web site to point at 0.4.1 new release (mattf-horton) closes apache/metron#764
METRON-1063 address javadoc warnings in metron-stellar (dbist via ottobackwards) closes apache/metron#668
METRON-1190 Fix Meta Alert Type handling in calculation of scores (justinleet) closes apache/metron#763
METRON-1187 Indexing/Profiler Kafka ACL Groups Not Setup Correctly (nickwallen) closes apache/metron#759
METRON-1185: Stellar REPL does not work on a kerberized cluster when calling functions interacting with HBase closes apache/incubator-metron#755
METRON-1186: Profiler Functions use classutils from shaded storm closes apache/incubator-metron#758
METRON-1173: Fix pointers to old stellar docs closes apache/incubator-metron#746
METRON-1179: Make STATS_ADD to take a list closes apache/incubator-metron#750
METRON-1180: Make Stellar Shell accept zookeeper quorum as a CSV list and not require a port closes apache/incubator-metron#751
METRON-1183 Improve KDC Setup Instructions (nickwallen) closes apache/metron#753
METRON-1177 Stale running topologies seen post-kerberization and cause exceptions (nickwallen) closes apache/metron#748
METRON-1158 Build backend for grouping alerts into meta alerts (justinleet) closes apache/metron#734
METRON-1146: Add ability to parse JSON string into JSONObject for stellar closes apache/incubator-metron#727
METRON-1176 REST: HDFS Service should support setting permissions on files when writing (ottobackwards) closes apache/metron#749
METRON-1114 Add group by capabilities to search REST endpoint (merrimanr) closes apache/metron#702
METRON-1167 Define Session Specific Global Configuration Values in the REPL (nickwallen) closes apache/metron#740
METRON-1171: Better validation for the SUBSTRING stellar function closes apache/incubator-metron#745
On 11/17/17, 11:59 AM, "Nick Allen" <ni...@nickallen.org> wrote:
I just wanted to send an update on where we are at. We've gotten a lot
done here recently as you can see below.
✓ DONE (1) First, METRON-1289 needs to go in. This one was a fairly big
effort and I am hearing that we are pretty close.
✓ DONE (2) METRON-1294 fixes an issue in how field types are looked-up.
✓ DONE (3) METRON-1290 is next. While this may have been fixed in
M-1289, there may be some test cases we want from this PR.
✓ DONE (4) METRON-1301 addresses a problem with the sorting logic.
✓ DONE (5) METRON-1291 fixes an issue with escalation of metaalerts.
(6) That leads us to Raghu's UI work in METRON-1252. This introduces the
UI bits that depend on all the previous backend work.
(7) At this point, we should have our best effort at running Metaalerts
on Elasticsearch 2.x. I propose that we cut a release here.
(8) After we cut the release, we can introduce the work for ES 5.x in
METRON-939. I know we will need lots of help testing and reviewing this
one.
We also have an outstanding question that needs resolved BEFORE we
release. We need to come to a consensus on how to release having moved our
Bro Plugin to a separate repo. I don't think we've heard from everyone on
this. I'd urge everyone to chime in so we can choose a path forward.
If anyone is totally confused in regards to that discussion, I can try and
send an options summary again as a separate discuss thread. The original
chain was somewhere around here [1].
[1]
https://lists.apache.org/thread.html/54a4474881b97e559df24728b3a0e923a58345a282451085eef832ef@%3Cdev.metron.apache.org%3E
On Wed, Nov 15, 2017 at 10:04 AM, Nick Allen <ni...@nickallen.org> wrote:
> Hi Guys -
>
> I want to follow-up on this discussion. It sounds like most people are in
> agreement with the general approach.
>
> A lot of people have been working hard on Metaalerts and Elasticsearch. I
> have checked-in with those doing the heavy lifting and have compiled a more
> detailed plan based on where we are at now. To the best of my knowledge
> here is the plan of attack for finishing out this effort.
>
> (1) First, METRON-1289 needs to go in. This one was a fairly big effort
> and I am hearing that we are pretty close.
>
> (2) METRON-1294 fixes an issue in how field types are looked-up.
>
> (3) METRON-1290 is next. While this may have been fixed in M-1289,
> there may be some test cases we want from this PR.
>
> (4) METRON-1301 addresses a problem with the sorting logic.
>
> (5) METRON-1291 fixes an issue with escalation of metaalerts.
>
> (6) That leads us to Raghu's UI work in METRON-1252. This introduces
> the UI bits that depend on all the previous backend work.
>
> (7) At this point, we should have our best effort at running Metaalerts
> on Elasticsearch 2.x. I propose that we cut a release here.
>
> (8) After we cut the release, we can introduce the work for ES 5.x in
> METRON-939. I know we will need lots of help testing and reviewing this
> one.
>
> Please correct me if I am wrong. I will try and send out updates as we
> make progress.
>
>
>
>
>
> On Mon, Nov 6, 2017 at 1:03 PM, Zeolla@GMail.com <ze...@gmail.com> wrote:
>
>> I agree, I think it's very reasonable to move in line with Nick's
>> proposal. I would also suggest that we outline what the target versions
>> would be to add in the METRON-777 components, since it has been functional
>> for a very long time but not reviewed and has some really rockstar
>> improvements.
>>
>> Jon
>>
>> On Mon, Nov 6, 2017 at 12:56 PM Otto Fowler <ot...@gmail.com>
>> wrote:
>>
>> > I think the ES cutover should be the start of the 0.5.x series, and we
>> > continue on with 0.4.x for the
>> > metadata improvements etc. We could chose to focus 0.5.x’s first
>> releases
>> > on not only ES but
>> > getting a handle on kibana and the mpack situation as well.
>> >
>> >
>> >
>> >
>> > On November 6, 2017 at 12:48:45, Michael Miklavcic (
>> > michael.miklavcic@gmail.com) wrote:
>> >
>> > I agree with your proposal, Nick. I think having a stabilizing release
>> > prior to upgrading ES/Kibana makes sense.
>> >
>> > On Mon, Nov 6, 2017 at 9:16 AM, Nick Allen <ni...@nickallen.org> wrote:
>> >
>> > > I would like to start a discussion around upcoming releases. We have a
>> > > couple separate significant tracks of work that we need to reconcile
>> in
>> > our
>> > > release schedule.
>> > >
>> > > (1) We have had (and have in review) a good number of bug fixes
>> required
>> > to
>> > > support Metaalerts on the existing Elasticsearch 2.x infrastructure.
>> > >
>> > >
>> > > (2) We also have ongoing work to upgrade our infrastructure to
>> > > Elasticsearch 5.x, which will not be backwards compatible.
>> > >
>> > >
>> > > I would like to see a release that has our best work on ES 2.x before
>> we
>> > > migrate to 5.x. I would propose the following.
>> > >
>> > > Release N+1: Introduce Metaalerts running on ES 2.x
>> > >
>> > > Release N+2: Cut-over to ES 5.x
>> > >
>> > >
>> > > (Q) Is it worth cutting a separate release for ES 2.x? Is there a
>> better
>> > > way to handle the cut-over to 5.x?
>> > >
>> >
>> --
>>
>> Jon
>>
>
>
Re: [DISCUSS] Upcoming Release
Posted by Matt Foley <ma...@apache.org>.
(With release manager hat on)
The community has proposed a release of Metron in the near future, focusing on Meta-alerts running in Elasticsearch.
Congrats on getting so many of the below already done. At this point, only METRON-1252, and the discussion of how to handle joint release of the Metron bro plugin, remain as gating items for the release. I project these will be resolved next week, so let’s propose the following:
Sometime next week, after the last bits are done, I’ll start the release process and create the release branch.
The proposed new version will be 0.4.2, unless there are backward incompatible changes that support making it 0.5.0.
Currently there are NO included Jiras labeled ‘backward-incompatible’, nor having Docs Text indicating so.
***If anyone knows that some of the commits included since 0.4.1 introduce backward incompatibility, please say so now on this thread, and mark the Jira as such.***
The 90 or so jiras/commits already in master branch since 0.4.1 are listed below.
Thanks,
--Matt
METRON-1301 Alerts UI - Sorting on Triage Score Unexpectedly Filters Some Records (nickwallen) closes apache/metron#832
METRON-1294 IP addresses are not formatted correctly in facet and group results (merrimanr) closes apache/metron#827
METRON-1291 Kafka produce REST endpoint does not work in a Kerberized cluster (merrimanr) closes apache/metron#826
METRON-1290 Only first 10 alerts are update when a MetaAlert status is changed to inactive (justinleet) closes apache/metron#842
METRON-1311 Service Check Should Check Elasticsearch Index Templates (nickwallen) closes apache/metron#839
METRON-1289 Alert fields are lost when a MetaAlert is created (merrimanr) closes apache/metron#824
METRON-1309 Change metron-deployment to pull the plugin from apache/metron-bro-plugin-kafka (JonZeolla) closes apache/metron#837
METRON-1310 Template Delete Action Deletes Search Indices (nickwallen) closes apache/metron#838
METRON-1275: Fix Metron Documentation closes apache/incubator-metron#833
METRON-1295 Unable to Configure Logging for REST API (nickwallen) closes apache/metron#828
METRON-1307 Force install of java8 since java9 does not appear to work with the scripts (brianhurley via ottobackwards) closes apache/metron#835
METRON-1296 Full Dev Fails to Deploy Index Templates (nickwallen via cestella) closes apache/incubator-metron#829
METRON-1281 Remove hard-coded indices from the Alerts UI (merrimanr) closes apache/metron#821
METRON-1287 Full Dev Fails When Installing EPEL Repository (nickwallen) closes apache/metron#820
METRON-1267 Alerts UI returns a 404 when refreshing the alerts-list page (iraghumitra via merrimanr) closes apache/metron#819
METRON-1283 Install Elasticsearch template as a part of the mpack startup scripts (anandsubbu via nickwallen) closes apache/metron#817
METRON-1254: Conditionals as map keys do not function in Stellar closes apache/incubator-metron#801
METRON-1261 Apply bro security patch (JonZeolla via ottobackwards) closes apache/metron#805
METRON-1284 Remove extraneous dead query in ElasticsearchDao (justinleet) closes apache/metron#818
METRON-1270: fix for warnings missing @return tag argument in metron-analytics/metron-profiler-common and metron-profiler-client closes apache/incubator-metron#810
METRON-1272 Hide child alerts from searches and grouping if they belong to meta alerts (justinleet) closes apache/metron#811
METRON-1224 Add time range selection to search control (iraghumitra via james-sirota) closes apache/metron#796
METRON-1280 0.4.1 -> 0.4.2 missed a couple of projects (cestella via justinleet) closes apache/metron#816
METRON-1243: Add a REST endpoint which allows us to get a list of all indice closes apache/incubator-metron#797
METRON-1196 Increment master version number to 0.4.2 for on-going development (mattf-horton) closes apache/metron#767
METRON-1278 Strip "Build Status" widget from root README.md in site-book build (mattf-horton) closes apache/metron#815
METRON-1274 Master has failure in StormControllerIntegrationTest (merrimanr) closes apache/metron#813
METRON-1266 Profiler - SASL Authentication Failed (nickwallen) closes apache/metron#809
METRON-1260 Include Alerts UI in Ambari Service Check (nickwallen) closes apache/metron#804
METRON-1251: Typo and formatting fixes for metron-rest README closes apache/incubator-metron#800
METRON-1241: Enable the REST API to use a cache for the zookeeper config similar to the Bolts closes apache/incubator-metron#795
METRON-1267 Alerts UI returns a 404 when refreshing the alerts-list page (merrimanr) closes apache/metron#808
METRON-1262 Unable to add comment for a alert in a meta-alert (merrimanr) closes apache/metron#806
METRON-1263 Start Alerts UI service after Metron REST (anandsubbu via nickwallen) closes apache/metron#807
METRON-1255 MetaAlert search is not filtering on status (merrimanr) closes apache/metron#802
METRON-1249 Improve Metron MPack Service Checks (nickwallen) closes apache/metron#799
METRON-1237 address javadoc warnings in metron-maas-common (dbist via james-sirota) closes apache/metron#792
METRON-1240 address javadoc warnings in metron-platform and metron-analytics (dbist via james-sirota) closes apache/metron#794
METRON-1226 Searching Can Errantly Query the Wrong Indices (nickwallen) closes apache/metron#793
METRON-1081 Fix Alerts and Ops UI Notices file (james-sirota) closes apache/metron#682
METRON-1123 Add group by option using faceted search capabilities of metron-rest-api (iraghumitra via james-sirota) closes apache/metron#768
METRON-1223 Add support to add comments for alerts (iraghumitra via james-sirota) closes apache/metron#788
METRON-1083 Add filters using faceted search capabilities of metron-rest-api (iraghumitra via james-sirota) closes apache/metron#710
METRON-1232 Alert status changes are not reflected in list view (iraghumitra via merrimanr) closes apache/metron#787
METRON-1247 REST search and findOne endpoints return unexpected or incorrect results for guids (justinleet) closes apache/metron#798
METRON-1235: Document the properties pulled from the global configuration closes apache/incubator-metron#791
METRON-1234: fix for WARNING 'dependencies.dependency.(groupId:artifactId:type:classifier)' must be unique: org.apache.hadoop:hadoop-yarn-api:jar (dbist via mmiklavc) closes apache/metron#790
METRON-1222: fix warning for The expression ${parent.version} is deprecated. Please use ${project.parent.version} instead. (dbist via mmiklavc) closes apache/metron#782
METRON-1220 Create documentation around alert nested field (justinleet) closes apache/metron#780
METRON-1229 Management UI type is part of the declarations of 2 modules (merrimanr) closes apache/metron#784
METRON-1228: Configuration Management PUSH immediately does DUMP after (mmiklavc via mmiklavc) closes apache/metron#783
METRON-1218 Metron REST should return better error messages (merrimanr) closes apache/metron#779
METRON-1161 Add ability to edit parser command line options in the management UI (merrimanr) closes apache/metron#737
METRON-1209: Make stellar repl take logging properties, like other CLI apps in metron closes apache/incubator-metron#772
METRON-1059 address checkstyle warning AvoidStarImport in metron-stellar (dbist via ottobackwards) closes apache/metron#664
METRON-1204 UI does not time out after being idle, but stops functioning (merrimanr) closes apache/metron#771
METRON-1052: Add forensic similarity hash functions to Stellar closes apache/incubator-metron#781
METRON-632: Added validation of "shew.enrichmentType" and "shew.keyColumns" closes apache/incubator-metron#732
METRON-1194 Add Profiler Debug Functions to Profiler README (nickwallen via ottobackwards) closes apache/metron#765
METRON-1055 Metron 0.4.0 manual installation guide for CentOS 6 updates (lvets via ottobackwards) closes apache/metron#661
METRON-1079 STELLAR NaN should be a keyword (ottobackwards) closes apache/metron#681
METRON-1085 Add REST endpoint to save a user profile for the Alerts UI (merrimanr) closes apache/metron#694
METRON-1208 MPack for Alerts UI (merrimanr) closes apache/metron#778
METRON-1207 Make RPMs for Alerts UI (merrimanr) closes apache/metron#777
METRON-1215 Fix link to RPMs chapter (DimDroll via justinleet) closes apache/metron#776
METRON-1206 Make alerts UI conform to ops UI for install (merrimanr) closes apache/metron#773
METRON-1195 Meta alerts improperly handle updates to non-alert fields (justinleet) closes apache/metron#766
METRON-1189 Add alert escalation to the Alerts UI (merrimanr) closes apache/metron#762
METRON-1156 Simulate Triage Rules in the Stellar REPL (nickwallen) closes apache/metron#733
METRON-1198 Pycapa - No such configuration property 'sasl.kerberos.principal' (nickwallen) closes apache/metron#769
METRON-1202 ElasticsearchDao Has extraneous sleep call (justinleet) closes apache/metron#770
METRON-938 "service metron-rest start <password>" does not work on CentOS 7. (justinleet) closes apache/metron#757
METRON-1182 Refactor Code in alert list to accommodate new view types (iraghumitra via merrimanr) closes apache/metron#756
METRON-1188: Ambari global configuration management (mmiklavc) closes apache/metron#760
METRON-1191 update public web site to point at 0.4.1 new release (mattf-horton) closes apache/metron#764
METRON-1063 address javadoc warnings in metron-stellar (dbist via ottobackwards) closes apache/metron#668
METRON-1190 Fix Meta Alert Type handling in calculation of scores (justinleet) closes apache/metron#763
METRON-1187 Indexing/Profiler Kafka ACL Groups Not Setup Correctly (nickwallen) closes apache/metron#759
METRON-1185: Stellar REPL does not work on a kerberized cluster when calling functions interacting with HBase closes apache/incubator-metron#755
METRON-1186: Profiler Functions use classutils from shaded storm closes apache/incubator-metron#758
METRON-1173: Fix pointers to old stellar docs closes apache/incubator-metron#746
METRON-1179: Make STATS_ADD to take a list closes apache/incubator-metron#750
METRON-1180: Make Stellar Shell accept zookeeper quorum as a CSV list and not require a port closes apache/incubator-metron#751
METRON-1183 Improve KDC Setup Instructions (nickwallen) closes apache/metron#753
METRON-1177 Stale running topologies seen post-kerberization and cause exceptions (nickwallen) closes apache/metron#748
METRON-1158 Build backend for grouping alerts into meta alerts (justinleet) closes apache/metron#734
METRON-1146: Add ability to parse JSON string into JSONObject for stellar closes apache/incubator-metron#727
METRON-1176 REST: HDFS Service should support setting permissions on files when writing (ottobackwards) closes apache/metron#749
METRON-1114 Add group by capabilities to search REST endpoint (merrimanr) closes apache/metron#702
METRON-1167 Define Session Specific Global Configuration Values in the REPL (nickwallen) closes apache/metron#740
METRON-1171: Better validation for the SUBSTRING stellar function closes apache/incubator-metron#745
On 11/17/17, 11:59 AM, "Nick Allen" <ni...@nickallen.org> wrote:
I just wanted to send an update on where we are at. We've gotten a lot
done here recently as you can see below.
✓ DONE (1) First, METRON-1289 needs to go in. This one was a fairly big
effort and I am hearing that we are pretty close.
✓ DONE (2) METRON-1294 fixes an issue in how field types are looked-up.
✓ DONE (3) METRON-1290 is next. While this may have been fixed in
M-1289, there may be some test cases we want from this PR.
✓ DONE (4) METRON-1301 addresses a problem with the sorting logic.
✓ DONE (5) METRON-1291 fixes an issue with escalation of metaalerts.
(6) That leads us to Raghu's UI work in METRON-1252. This introduces the
UI bits that depend on all the previous backend work.
(7) At this point, we should have our best effort at running Metaalerts
on Elasticsearch 2.x. I propose that we cut a release here.
(8) After we cut the release, we can introduce the work for ES 5.x in
METRON-939. I know we will need lots of help testing and reviewing this
one.
We also have an outstanding question that needs resolved BEFORE we
release. We need to come to a consensus on how to release having moved our
Bro Plugin to a separate repo. I don't think we've heard from everyone on
this. I'd urge everyone to chime in so we can choose a path forward.
If anyone is totally confused in regards to that discussion, I can try and
send an options summary again as a separate discuss thread. The original
chain was somewhere around here [1].
[1]
https://lists.apache.org/thread.html/54a4474881b97e559df24728b3a0e923a58345a282451085eef832ef@%3Cdev.metron.apache.org%3E
On Wed, Nov 15, 2017 at 10:04 AM, Nick Allen <ni...@nickallen.org> wrote:
> Hi Guys -
>
> I want to follow-up on this discussion. It sounds like most people are in
> agreement with the general approach.
>
> A lot of people have been working hard on Metaalerts and Elasticsearch. I
> have checked-in with those doing the heavy lifting and have compiled a more
> detailed plan based on where we are at now. To the best of my knowledge
> here is the plan of attack for finishing out this effort.
>
> (1) First, METRON-1289 needs to go in. This one was a fairly big effort
> and I am hearing that we are pretty close.
>
> (2) METRON-1294 fixes an issue in how field types are looked-up.
>
> (3) METRON-1290 is next. While this may have been fixed in M-1289,
> there may be some test cases we want from this PR.
>
> (4) METRON-1301 addresses a problem with the sorting logic.
>
> (5) METRON-1291 fixes an issue with escalation of metaalerts.
>
> (6) That leads us to Raghu's UI work in METRON-1252. This introduces
> the UI bits that depend on all the previous backend work.
>
> (7) At this point, we should have our best effort at running Metaalerts
> on Elasticsearch 2.x. I propose that we cut a release here.
>
> (8) After we cut the release, we can introduce the work for ES 5.x in
> METRON-939. I know we will need lots of help testing and reviewing this
> one.
>
> Please correct me if I am wrong. I will try and send out updates as we
> make progress.
>
>
>
>
>
> On Mon, Nov 6, 2017 at 1:03 PM, Zeolla@GMail.com <ze...@gmail.com> wrote:
>
>> I agree, I think it's very reasonable to move in line with Nick's
>> proposal. I would also suggest that we outline what the target versions
>> would be to add in the METRON-777 components, since it has been functional
>> for a very long time but not reviewed and has some really rockstar
>> improvements.
>>
>> Jon
>>
>> On Mon, Nov 6, 2017 at 12:56 PM Otto Fowler <ot...@gmail.com>
>> wrote:
>>
>> > I think the ES cutover should be the start of the 0.5.x series, and we
>> > continue on with 0.4.x for the
>> > metadata improvements etc. We could chose to focus 0.5.x’s first
>> releases
>> > on not only ES but
>> > getting a handle on kibana and the mpack situation as well.
>> >
>> >
>> >
>> >
>> > On November 6, 2017 at 12:48:45, Michael Miklavcic (
>> > michael.miklavcic@gmail.com) wrote:
>> >
>> > I agree with your proposal, Nick. I think having a stabilizing release
>> > prior to upgrading ES/Kibana makes sense.
>> >
>> > On Mon, Nov 6, 2017 at 9:16 AM, Nick Allen <ni...@nickallen.org> wrote:
>> >
>> > > I would like to start a discussion around upcoming releases. We have a
>> > > couple separate significant tracks of work that we need to reconcile
>> in
>> > our
>> > > release schedule.
>> > >
>> > > (1) We have had (and have in review) a good number of bug fixes
>> required
>> > to
>> > > support Metaalerts on the existing Elasticsearch 2.x infrastructure.
>> > >
>> > >
>> > > (2) We also have ongoing work to upgrade our infrastructure to
>> > > Elasticsearch 5.x, which will not be backwards compatible.
>> > >
>> > >
>> > > I would like to see a release that has our best work on ES 2.x before
>> we
>> > > migrate to 5.x. I would propose the following.
>> > >
>> > > Release N+1: Introduce Metaalerts running on ES 2.x
>> > >
>> > > Release N+2: Cut-over to ES 5.x
>> > >
>> > >
>> > > (Q) Is it worth cutting a separate release for ES 2.x? Is there a
>> better
>> > > way to handle the cut-over to 5.x?
>> > >
>> >
>> --
>>
>> Jon
>>
>
>
Re: [DISCUSS] Upcoming Release
Posted by Nick Allen <ni...@nickallen.org>.
I just wanted to send an update on where we are at. We've gotten a lot
done here recently as you can see below.
✓ DONE (1) First, METRON-1289 needs to go in. This one was a fairly big
effort and I am hearing that we are pretty close.
✓ DONE (2) METRON-1294 fixes an issue in how field types are looked-up.
✓ DONE (3) METRON-1290 is next. While this may have been fixed in
M-1289, there may be some test cases we want from this PR.
✓ DONE (4) METRON-1301 addresses a problem with the sorting logic.
✓ DONE (5) METRON-1291 fixes an issue with escalation of metaalerts.
(6) That leads us to Raghu's UI work in METRON-1252. This introduces the
UI bits that depend on all the previous backend work.
(7) At this point, we should have our best effort at running Metaalerts
on Elasticsearch 2.x. I propose that we cut a release here.
(8) After we cut the release, we can introduce the work for ES 5.x in
METRON-939. I know we will need lots of help testing and reviewing this
one.
We also have an outstanding question that needs resolved BEFORE we
release. We need to come to a consensus on how to release having moved our
Bro Plugin to a separate repo. I don't think we've heard from everyone on
this. I'd urge everyone to chime in so we can choose a path forward.
If anyone is totally confused in regards to that discussion, I can try and
send an options summary again as a separate discuss thread. The original
chain was somewhere around here [1].
[1]
https://lists.apache.org/thread.html/54a4474881b97e559df24728b3a0e923a58345a282451085eef832ef@%3Cdev.metron.apache.org%3E
On Wed, Nov 15, 2017 at 10:04 AM, Nick Allen <ni...@nickallen.org> wrote:
> Hi Guys -
>
> I want to follow-up on this discussion. It sounds like most people are in
> agreement with the general approach.
>
> A lot of people have been working hard on Metaalerts and Elasticsearch. I
> have checked-in with those doing the heavy lifting and have compiled a more
> detailed plan based on where we are at now. To the best of my knowledge
> here is the plan of attack for finishing out this effort.
>
> (1) First, METRON-1289 needs to go in. This one was a fairly big effort
> and I am hearing that we are pretty close.
>
> (2) METRON-1294 fixes an issue in how field types are looked-up.
>
> (3) METRON-1290 is next. While this may have been fixed in M-1289,
> there may be some test cases we want from this PR.
>
> (4) METRON-1301 addresses a problem with the sorting logic.
>
> (5) METRON-1291 fixes an issue with escalation of metaalerts.
>
> (6) That leads us to Raghu's UI work in METRON-1252. This introduces
> the UI bits that depend on all the previous backend work.
>
> (7) At this point, we should have our best effort at running Metaalerts
> on Elasticsearch 2.x. I propose that we cut a release here.
>
> (8) After we cut the release, we can introduce the work for ES 5.x in
> METRON-939. I know we will need lots of help testing and reviewing this
> one.
>
> Please correct me if I am wrong. I will try and send out updates as we
> make progress.
>
>
>
>
>
> On Mon, Nov 6, 2017 at 1:03 PM, Zeolla@GMail.com <ze...@gmail.com> wrote:
>
>> I agree, I think it's very reasonable to move in line with Nick's
>> proposal. I would also suggest that we outline what the target versions
>> would be to add in the METRON-777 components, since it has been functional
>> for a very long time but not reviewed and has some really rockstar
>> improvements.
>>
>> Jon
>>
>> On Mon, Nov 6, 2017 at 12:56 PM Otto Fowler <ot...@gmail.com>
>> wrote:
>>
>> > I think the ES cutover should be the start of the 0.5.x series, and we
>> > continue on with 0.4.x for the
>> > metadata improvements etc. We could chose to focus 0.5.x’s first
>> releases
>> > on not only ES but
>> > getting a handle on kibana and the mpack situation as well.
>> >
>> >
>> >
>> >
>> > On November 6, 2017 at 12:48:45, Michael Miklavcic (
>> > michael.miklavcic@gmail.com) wrote:
>> >
>> > I agree with your proposal, Nick. I think having a stabilizing release
>> > prior to upgrading ES/Kibana makes sense.
>> >
>> > On Mon, Nov 6, 2017 at 9:16 AM, Nick Allen <ni...@nickallen.org> wrote:
>> >
>> > > I would like to start a discussion around upcoming releases. We have a
>> > > couple separate significant tracks of work that we need to reconcile
>> in
>> > our
>> > > release schedule.
>> > >
>> > > (1) We have had (and have in review) a good number of bug fixes
>> required
>> > to
>> > > support Metaalerts on the existing Elasticsearch 2.x infrastructure.
>> > >
>> > >
>> > > (2) We also have ongoing work to upgrade our infrastructure to
>> > > Elasticsearch 5.x, which will not be backwards compatible.
>> > >
>> > >
>> > > I would like to see a release that has our best work on ES 2.x before
>> we
>> > > migrate to 5.x. I would propose the following.
>> > >
>> > > Release N+1: Introduce Metaalerts running on ES 2.x
>> > >
>> > > Release N+2: Cut-over to ES 5.x
>> > >
>> > >
>> > > (Q) Is it worth cutting a separate release for ES 2.x? Is there a
>> better
>> > > way to handle the cut-over to 5.x?
>> > >
>> >
>> --
>>
>> Jon
>>
>
>
Re: [DISCUSS] Upcoming Release
Posted by "Zeolla@GMail.com" <ze...@gmail.com>.
For #3 we would need people to recursively clone our repo to pull in
submodules properly. I work with other projects that do that and it is
perpetually an issue for new users. I know it doesn't seem like a big
deal, but it is definitely a deterrent to some who forget this step or use
old instructions that don't include it.
I'm interested to see how other apache projects handle releases for
multiple repos. I honestly think we can have a much lower bar for a
release on a secondary repo, but I don't know if that's acceptable.
Regarding 4, we could also fix forward. I already have a solution that
could be very slightly repurposed in my full-dev testing instructions here
<https://github.com/apache/metron-bro-plugin-kafka/pull/3>.
Yes, I still prefer #5 but I'll stop talking so much and let also chime
in. =)
Jon
On Thu, Nov 16, 2017 at 10:40 AM Nick Allen <ni...@nickallen.org> wrote:
> > (1) [Jon] Make a release in apache/metron-bro-plugin-kafka, then update
> Ansible to pin to that release.
>
> The problem with this approach, as I see it, is that it commits us to
> having separate releases for the Bro plugin. Which I am not sure if or how
> long it will take to come to a consensus on that. We also would need to
> invent that release process rather quickly.
>
> > (4) Revert PR #837 because as you pointed out this approach does not work
> well with releases. That would give us enough time to come up with a
> sensible approach and do that.
>
> I am kind of leaning towards (4) right now. The approach taken in this PR,
> doesn't work. Let's just revert it and give ourselves some time to come up
> with an approach that does work and has consensus.
>
>
>
>
>
>
>
> On Thu, Nov 16, 2017 at 10:37 AM, Nick Allen <ni...@nickallen.org> wrote:
>
> > Just to layout some other alternatives...
> >
> > (1) [Jon] Make a release in apache/metron-bro-plugin-kafka, then update
> > Ansible to pin to that release.
> >
> > (2) [Jon] Update Ansible to pin to a specific commit.
> >
> > (3) Wouldn't the submodule approach solve this? I imagine that if we use
> > a submodule, then all of the Bro plugin code will be contained directly
> in
> > each Metron release. We would just need to change that small bit of
> > Ansible code so that it uses the code directly (like before) instead of
> > going to Git and checking it out.
> >
> > (4) Revert PR #837 because as you pointed out this approach does not work
> > well with releases. That would give us enough time to come up with a
> > sensible approach and do that.
> >
> >
> >
> >
> >
> >
> > On Thu, Nov 16, 2017 at 10:21 AM, Zeolla@GMail.com <ze...@gmail.com>
> > wrote:
> >
> >> The problem is that we're currently pinning to master and if something
> in
> >> the plugin breaks backward compatibility, our prior releases will be
> >> perpetually broken, which is why I suggest it blocks the upcoming
> release.
> >>
> >> The alternative would be to update ansible to pin to a specific commit
> or
> >> to make a release in apache/metron-bro-plugin-kafka sooner rather than
> >> later and pin to its branch/tag. That feels like a waste of time
> though,
> >> as the 2.5.2 upgrade is somewhat trivial.
> >>
> >> Jon
> >>
> >> On Thu, Nov 16, 2017 at 10:14 AM Nick Allen <ni...@nickallen.org> wrote:
> >>
> >> > While I think that the Bro work is super valuable, Jon, I am not sure
> >> that
> >> > we need to block the next release for it. In my mind, the "theme" of
> >> the
> >> > next release is Metaalerts running on ES 2.x. I'd prefer to just
> stick
> >> > with that.
> >> >
> >> > I was hoping we could get the remaining "Metaalerts + ES 2.x" PRs
> merged
> >> > and start cutting release candidates ASAP. That could be possible by
> >> end
> >> > of week based on the "big one" (METRON-1289) getting merged in last
> >> night.
> >> >
> >> > Of course, if you happen to get all the Bro work done in time, then
> >> > definitely let's include it in the next release. But I am wary of
> >> blocking
> >> > the release for that work. No need for you to rush through it.
> >> >
> >> > Just one man's opinion. Would like to hear feedback from more of the
> >> > community.
> >> >
> >> >
> >> >
> >> > On Thu, Nov 16, 2017 at 8:01 AM, Zeolla@GMail.com <ze...@gmail.com>
> >> > wrote:
> >> >
> >> > > The way master's full-dev is set up right now is non optimal for the
> >> bro
> >> > > plugin configuration, and I would like to complete the roadmap I
> >> outlined
> >> > > in my discuss thread before a release. I have a PR open against
> >> > > Apache/metron-bro-plugin-kafka right now to turn it into a plugin,
> >> and I
> >> > > expect it will take me until end of next week at the latest to have
> >> the
> >> > > rest of the work done to move us to 2.5.2, and to pin to a specific
> >> > package
> >> > > version. At that point I'm comfortable with a release.
> >> > >
> >> > > Jon
> >> > >
> >> > > On Wed, Nov 15, 2017, 18:57 Matt Foley <ma...@apache.org> wrote:
> >> > >
> >> > > > I’ve been listening. Looks like there are still a number of major
> >> > issues
> >> > > > to be committed first, right?
> >> > > > The discussion on this thread constitutes sufficient engagement, I
> >> > think,
> >> > > > especially given the Subject line :-)
> >> > > > Would the folks working on the 6 issues listed by Nick care to
> >> suggest
> >> > a
> >> > > > cut-off date by which they’ll probably have those fixes in?
> >> > > > I’ll be happy to run the release process, if the community so
> >> wishes,
> >> > as
> >> > > > soon as those issues are committed.
> >> > > >
> >> > > > I guess I should, pro forma, send the list of commits already in
> >> since
> >> > > the
> >> > > > last release. I’ll do that today.
> >> > > > Also, if anyone wishes to raise a hand and propose additional
> >> commits
> >> > are
> >> > > > needed, please do so on this thread.
> >> > > > Thanks,
> >> > > > --Matt
> >> > > >
> >> > > >
> >> > > > On 11/15/17, 2:09 PM, "Casey Stella" <ce...@gmail.com> wrote:
> >> > > >
> >> > > > I'd say that if a release is this imminent that we had better
> >> > notify
> >> > > > the
> >> > > > release manager who will make a release announcement, Nick.
> >> Matt,
> >> > > are
> >> > > > you
> >> > > > tuning in to this?
> >> > > >
> >> > > > On Wed, Nov 15, 2017 at 10:04 AM, Nick Allen <
> >> nick@nickallen.org>
> >> > > > wrote:
> >> > > >
> >> > > > > Hi Guys -
> >> > > > >
> >> > > > > I want to follow-up on this discussion. It sounds like most
> >> > people
> >> > > > are in
> >> > > > > agreement with the general approach.
> >> > > > >
> >> > > > > A lot of people have been working hard on Metaalerts and
> >> > > > Elasticsearch. I
> >> > > > > have checked-in with those doing the heavy lifting and have
> >> > > compiled
> >> > > > a more
> >> > > > > detailed plan based on where we are at now. To the best of
> my
> >> > > > knowledge
> >> > > > > here is the plan of attack for finishing out this effort.
> >> > > > >
> >> > > > > (1) First, METRON-1289 needs to go in. This one was a
> >> fairly
> >> > big
> >> > > > effort
> >> > > > > and I am hearing that we are pretty close.
> >> > > > >
> >> > > > > (2) METRON-1294 fixes an issue in how field types are
> >> > looked-up.
> >> > > > >
> >> > > > > (3) METRON-1290 is next. While this may have been fixed
> in
> >> > > > M-1289, there
> >> > > > > may be some test cases we want from this PR.
> >> > > > >
> >> > > > > (4) METRON-1301 addresses a problem with the sorting
> logic.
> >> > > > >
> >> > > > > (5) METRON-1291 fixes an issue with escalation of
> >> metaalerts.
> >> > > > >
> >> > > > > (6) That leads us to Raghu's UI work in METRON-1252. This
> >> > > > introduces the
> >> > > > > UI bits that depend on all the previous backend work.
> >> > > > >
> >> > > > > (7) At this point, we should have our best effort at
> running
> >> > > > Metaalerts
> >> > > > > on Elasticsearch 2.x. I propose that we cut a release here.
> >> > > > >
> >> > > > > (8) After we cut the release, we can introduce the work
> for
> >> ES
> >> > > 5.x
> >> > > > in
> >> > > > > METRON-939. I know we will need lots of help testing and
> >> > reviewing
> >> > > > this
> >> > > > > one.
> >> > > > >
> >> > > > > Please correct me if I am wrong. I will try and send out
> >> updates
> >> > > as
> >> > > > we
> >> > > > > make progress.
> >> > > > >
> >> > > > >
> >> > > > >
> >> > > > >
> >> > > > >
> >> > > > > On Mon, Nov 6, 2017 at 1:03 PM, Zeolla@GMail.com <
> >> > zeolla@gmail.com
> >> > > >
> >> > > > wrote:
> >> > > > >
> >> > > > > > I agree, I think it's very reasonable to move in line with
> >> > Nick's
> >> > > > > > proposal. I would also suggest that we outline what the
> >> target
> >> > > > versions
> >> > > > > > would be to add in the METRON-777 components, since it has
> >> been
> >> > > > > functional
> >> > > > > > for a very long time but not reviewed and has some really
> >> > > rockstar
> >> > > > > > improvements.
> >> > > > > >
> >> > > > > > Jon
> >> > > > > >
> >> > > > > > On Mon, Nov 6, 2017 at 12:56 PM Otto Fowler <
> >> > > > ottobackwards@gmail.com>
> >> > > > > > wrote:
> >> > > > > >
> >> > > > > > > I think the ES cutover should be the start of the 0.5.x
> >> > series,
> >> > > > and we
> >> > > > > > > continue on with 0.4.x for the
> >> > > > > > > metadata improvements etc. We could chose to focus
> >> 0.5.x’s
> >> > > first
> >> > > > > > releases
> >> > > > > > > on not only ES but
> >> > > > > > > getting a handle on kibana and the mpack situation as
> >> well.
> >> > > > > > >
> >> > > > > > >
> >> > > > > > >
> >> > > > > > >
> >> > > > > > > On November 6, 2017 at 12:48:45, Michael Miklavcic (
> >> > > > > > > michael.miklavcic@gmail.com) wrote:
> >> > > > > > >
> >> > > > > > > I agree with your proposal, Nick. I think having a
> >> > stabilizing
> >> > > > release
> >> > > > > > > prior to upgrading ES/Kibana makes sense.
> >> > > > > > >
> >> > > > > > > On Mon, Nov 6, 2017 at 9:16 AM, Nick Allen <
> >> > nick@nickallen.org
> >> > > >
> >> > > > wrote:
> >> > > > > > >
> >> > > > > > > > I would like to start a discussion around upcoming
> >> > releases.
> >> > > > We have
> >> > > > > a
> >> > > > > > > > couple separate significant tracks of work that we
> need
> >> to
> >> > > > reconcile
> >> > > > > in
> >> > > > > > > our
> >> > > > > > > > release schedule.
> >> > > > > > > >
> >> > > > > > > > (1) We have had (and have in review) a good number of
> >> bug
> >> > > fixes
> >> > > > > > required
> >> > > > > > > to
> >> > > > > > > > support Metaalerts on the existing Elasticsearch 2.x
> >> > > > infrastructure.
> >> > > > > > > >
> >> > > > > > > >
> >> > > > > > > > (2) We also have ongoing work to upgrade our
> >> infrastructure
> >> > > to
> >> > > > > > > > Elasticsearch 5.x, which will not be backwards
> >> compatible.
> >> > > > > > > >
> >> > > > > > > >
> >> > > > > > > > I would like to see a release that has our best work
> on
> >> ES
> >> > > 2.x
> >> > > > before
> >> > > > > > we
> >> > > > > > > > migrate to 5.x. I would propose the following.
> >> > > > > > > >
> >> > > > > > > > Release N+1: Introduce Metaalerts running on ES 2.x
> >> > > > > > > >
> >> > > > > > > > Release N+2: Cut-over to ES 5.x
> >> > > > > > > >
> >> > > > > > > >
> >> > > > > > > > (Q) Is it worth cutting a separate release for ES 2.x?
> >> Is
> >> > > > there a
> >> > > > > > better
> >> > > > > > > > way to handle the cut-over to 5.x?
> >> > > > > > > >
> >> > > > > > >
> >> > > > > > --
> >> > > > > >
> >> > > > > > Jon
> >> > > > > >
> >> > > > >
> >> > > >
> >> > > >
> >> > > >
> >> > > > --
> >> > >
> >> > > Jon
> >> > >
> >> >
> >> --
> >>
> >> Jon
> >>
> >
> >
>
--
Jon
Re: [DISCUSS] Upcoming Release
Posted by Nick Allen <ni...@nickallen.org>.
> (1) [Jon] Make a release in apache/metron-bro-plugin-kafka, then update
Ansible to pin to that release.
The problem with this approach, as I see it, is that it commits us to
having separate releases for the Bro plugin. Which I am not sure if or how
long it will take to come to a consensus on that. We also would need to
invent that release process rather quickly.
> (4) Revert PR #837 because as you pointed out this approach does not work
well with releases. That would give us enough time to come up with a
sensible approach and do that.
I am kind of leaning towards (4) right now. The approach taken in this PR,
doesn't work. Let's just revert it and give ourselves some time to come up
with an approach that does work and has consensus.
On Thu, Nov 16, 2017 at 10:37 AM, Nick Allen <ni...@nickallen.org> wrote:
> Just to layout some other alternatives...
>
> (1) [Jon] Make a release in apache/metron-bro-plugin-kafka, then update
> Ansible to pin to that release.
>
> (2) [Jon] Update Ansible to pin to a specific commit.
>
> (3) Wouldn't the submodule approach solve this? I imagine that if we use
> a submodule, then all of the Bro plugin code will be contained directly in
> each Metron release. We would just need to change that small bit of
> Ansible code so that it uses the code directly (like before) instead of
> going to Git and checking it out.
>
> (4) Revert PR #837 because as you pointed out this approach does not work
> well with releases. That would give us enough time to come up with a
> sensible approach and do that.
>
>
>
>
>
>
> On Thu, Nov 16, 2017 at 10:21 AM, Zeolla@GMail.com <ze...@gmail.com>
> wrote:
>
>> The problem is that we're currently pinning to master and if something in
>> the plugin breaks backward compatibility, our prior releases will be
>> perpetually broken, which is why I suggest it blocks the upcoming release.
>>
>> The alternative would be to update ansible to pin to a specific commit or
>> to make a release in apache/metron-bro-plugin-kafka sooner rather than
>> later and pin to its branch/tag. That feels like a waste of time though,
>> as the 2.5.2 upgrade is somewhat trivial.
>>
>> Jon
>>
>> On Thu, Nov 16, 2017 at 10:14 AM Nick Allen <ni...@nickallen.org> wrote:
>>
>> > While I think that the Bro work is super valuable, Jon, I am not sure
>> that
>> > we need to block the next release for it. In my mind, the "theme" of
>> the
>> > next release is Metaalerts running on ES 2.x. I'd prefer to just stick
>> > with that.
>> >
>> > I was hoping we could get the remaining "Metaalerts + ES 2.x" PRs merged
>> > and start cutting release candidates ASAP. That could be possible by
>> end
>> > of week based on the "big one" (METRON-1289) getting merged in last
>> night.
>> >
>> > Of course, if you happen to get all the Bro work done in time, then
>> > definitely let's include it in the next release. But I am wary of
>> blocking
>> > the release for that work. No need for you to rush through it.
>> >
>> > Just one man's opinion. Would like to hear feedback from more of the
>> > community.
>> >
>> >
>> >
>> > On Thu, Nov 16, 2017 at 8:01 AM, Zeolla@GMail.com <ze...@gmail.com>
>> > wrote:
>> >
>> > > The way master's full-dev is set up right now is non optimal for the
>> bro
>> > > plugin configuration, and I would like to complete the roadmap I
>> outlined
>> > > in my discuss thread before a release. I have a PR open against
>> > > Apache/metron-bro-plugin-kafka right now to turn it into a plugin,
>> and I
>> > > expect it will take me until end of next week at the latest to have
>> the
>> > > rest of the work done to move us to 2.5.2, and to pin to a specific
>> > package
>> > > version. At that point I'm comfortable with a release.
>> > >
>> > > Jon
>> > >
>> > > On Wed, Nov 15, 2017, 18:57 Matt Foley <ma...@apache.org> wrote:
>> > >
>> > > > I’ve been listening. Looks like there are still a number of major
>> > issues
>> > > > to be committed first, right?
>> > > > The discussion on this thread constitutes sufficient engagement, I
>> > think,
>> > > > especially given the Subject line :-)
>> > > > Would the folks working on the 6 issues listed by Nick care to
>> suggest
>> > a
>> > > > cut-off date by which they’ll probably have those fixes in?
>> > > > I’ll be happy to run the release process, if the community so
>> wishes,
>> > as
>> > > > soon as those issues are committed.
>> > > >
>> > > > I guess I should, pro forma, send the list of commits already in
>> since
>> > > the
>> > > > last release. I’ll do that today.
>> > > > Also, if anyone wishes to raise a hand and propose additional
>> commits
>> > are
>> > > > needed, please do so on this thread.
>> > > > Thanks,
>> > > > --Matt
>> > > >
>> > > >
>> > > > On 11/15/17, 2:09 PM, "Casey Stella" <ce...@gmail.com> wrote:
>> > > >
>> > > > I'd say that if a release is this imminent that we had better
>> > notify
>> > > > the
>> > > > release manager who will make a release announcement, Nick.
>> Matt,
>> > > are
>> > > > you
>> > > > tuning in to this?
>> > > >
>> > > > On Wed, Nov 15, 2017 at 10:04 AM, Nick Allen <
>> nick@nickallen.org>
>> > > > wrote:
>> > > >
>> > > > > Hi Guys -
>> > > > >
>> > > > > I want to follow-up on this discussion. It sounds like most
>> > people
>> > > > are in
>> > > > > agreement with the general approach.
>> > > > >
>> > > > > A lot of people have been working hard on Metaalerts and
>> > > > Elasticsearch. I
>> > > > > have checked-in with those doing the heavy lifting and have
>> > > compiled
>> > > > a more
>> > > > > detailed plan based on where we are at now. To the best of my
>> > > > knowledge
>> > > > > here is the plan of attack for finishing out this effort.
>> > > > >
>> > > > > (1) First, METRON-1289 needs to go in. This one was a
>> fairly
>> > big
>> > > > effort
>> > > > > and I am hearing that we are pretty close.
>> > > > >
>> > > > > (2) METRON-1294 fixes an issue in how field types are
>> > looked-up.
>> > > > >
>> > > > > (3) METRON-1290 is next. While this may have been fixed in
>> > > > M-1289, there
>> > > > > may be some test cases we want from this PR.
>> > > > >
>> > > > > (4) METRON-1301 addresses a problem with the sorting logic.
>> > > > >
>> > > > > (5) METRON-1291 fixes an issue with escalation of
>> metaalerts.
>> > > > >
>> > > > > (6) That leads us to Raghu's UI work in METRON-1252. This
>> > > > introduces the
>> > > > > UI bits that depend on all the previous backend work.
>> > > > >
>> > > > > (7) At this point, we should have our best effort at running
>> > > > Metaalerts
>> > > > > on Elasticsearch 2.x. I propose that we cut a release here.
>> > > > >
>> > > > > (8) After we cut the release, we can introduce the work for
>> ES
>> > > 5.x
>> > > > in
>> > > > > METRON-939. I know we will need lots of help testing and
>> > reviewing
>> > > > this
>> > > > > one.
>> > > > >
>> > > > > Please correct me if I am wrong. I will try and send out
>> updates
>> > > as
>> > > > we
>> > > > > make progress.
>> > > > >
>> > > > >
>> > > > >
>> > > > >
>> > > > >
>> > > > > On Mon, Nov 6, 2017 at 1:03 PM, Zeolla@GMail.com <
>> > zeolla@gmail.com
>> > > >
>> > > > wrote:
>> > > > >
>> > > > > > I agree, I think it's very reasonable to move in line with
>> > Nick's
>> > > > > > proposal. I would also suggest that we outline what the
>> target
>> > > > versions
>> > > > > > would be to add in the METRON-777 components, since it has
>> been
>> > > > > functional
>> > > > > > for a very long time but not reviewed and has some really
>> > > rockstar
>> > > > > > improvements.
>> > > > > >
>> > > > > > Jon
>> > > > > >
>> > > > > > On Mon, Nov 6, 2017 at 12:56 PM Otto Fowler <
>> > > > ottobackwards@gmail.com>
>> > > > > > wrote:
>> > > > > >
>> > > > > > > I think the ES cutover should be the start of the 0.5.x
>> > series,
>> > > > and we
>> > > > > > > continue on with 0.4.x for the
>> > > > > > > metadata improvements etc. We could chose to focus
>> 0.5.x’s
>> > > first
>> > > > > > releases
>> > > > > > > on not only ES but
>> > > > > > > getting a handle on kibana and the mpack situation as
>> well.
>> > > > > > >
>> > > > > > >
>> > > > > > >
>> > > > > > >
>> > > > > > > On November 6, 2017 at 12:48:45, Michael Miklavcic (
>> > > > > > > michael.miklavcic@gmail.com) wrote:
>> > > > > > >
>> > > > > > > I agree with your proposal, Nick. I think having a
>> > stabilizing
>> > > > release
>> > > > > > > prior to upgrading ES/Kibana makes sense.
>> > > > > > >
>> > > > > > > On Mon, Nov 6, 2017 at 9:16 AM, Nick Allen <
>> > nick@nickallen.org
>> > > >
>> > > > wrote:
>> > > > > > >
>> > > > > > > > I would like to start a discussion around upcoming
>> > releases.
>> > > > We have
>> > > > > a
>> > > > > > > > couple separate significant tracks of work that we need
>> to
>> > > > reconcile
>> > > > > in
>> > > > > > > our
>> > > > > > > > release schedule.
>> > > > > > > >
>> > > > > > > > (1) We have had (and have in review) a good number of
>> bug
>> > > fixes
>> > > > > > required
>> > > > > > > to
>> > > > > > > > support Metaalerts on the existing Elasticsearch 2.x
>> > > > infrastructure.
>> > > > > > > >
>> > > > > > > >
>> > > > > > > > (2) We also have ongoing work to upgrade our
>> infrastructure
>> > > to
>> > > > > > > > Elasticsearch 5.x, which will not be backwards
>> compatible.
>> > > > > > > >
>> > > > > > > >
>> > > > > > > > I would like to see a release that has our best work on
>> ES
>> > > 2.x
>> > > > before
>> > > > > > we
>> > > > > > > > migrate to 5.x. I would propose the following.
>> > > > > > > >
>> > > > > > > > Release N+1: Introduce Metaalerts running on ES 2.x
>> > > > > > > >
>> > > > > > > > Release N+2: Cut-over to ES 5.x
>> > > > > > > >
>> > > > > > > >
>> > > > > > > > (Q) Is it worth cutting a separate release for ES 2.x?
>> Is
>> > > > there a
>> > > > > > better
>> > > > > > > > way to handle the cut-over to 5.x?
>> > > > > > > >
>> > > > > > >
>> > > > > > --
>> > > > > >
>> > > > > > Jon
>> > > > > >
>> > > > >
>> > > >
>> > > >
>> > > >
>> > > > --
>> > >
>> > > Jon
>> > >
>> >
>> --
>>
>> Jon
>>
>
>
Re: [DISCUSS] Upcoming Release
Posted by Nick Allen <ni...@nickallen.org>.
Sorry, missed one. I think this is the one Jon prefers.
(5) The other alternative is just to complete Jon's roadmap BEFORE the next
release.
On Thu, Nov 16, 2017 at 10:37 AM, Nick Allen <ni...@nickallen.org> wrote:
> Just to layout some other alternatives...
>
> (1) [Jon] Make a release in apache/metron-bro-plugin-kafka, then update
> Ansible to pin to that release.
>
> (2) [Jon] Update Ansible to pin to a specific commit.
>
> (3) Wouldn't the submodule approach solve this? I imagine that if we use
> a submodule, then all of the Bro plugin code will be contained directly in
> each Metron release. We would just need to change that small bit of
> Ansible code so that it uses the code directly (like before) instead of
> going to Git and checking it out.
>
> (4) Revert PR #837 because as you pointed out this approach does not work
> well with releases. That would give us enough time to come up with a
> sensible approach and do that.
>
>
>
>
>
>
> On Thu, Nov 16, 2017 at 10:21 AM, Zeolla@GMail.com <ze...@gmail.com>
> wrote:
>
>> The problem is that we're currently pinning to master and if something in
>> the plugin breaks backward compatibility, our prior releases will be
>> perpetually broken, which is why I suggest it blocks the upcoming release.
>>
>> The alternative would be to update ansible to pin to a specific commit or
>> to make a release in apache/metron-bro-plugin-kafka sooner rather than
>> later and pin to its branch/tag. That feels like a waste of time though,
>> as the 2.5.2 upgrade is somewhat trivial.
>>
>> Jon
>>
>> On Thu, Nov 16, 2017 at 10:14 AM Nick Allen <ni...@nickallen.org> wrote:
>>
>> > While I think that the Bro work is super valuable, Jon, I am not sure
>> that
>> > we need to block the next release for it. In my mind, the "theme" of
>> the
>> > next release is Metaalerts running on ES 2.x. I'd prefer to just stick
>> > with that.
>> >
>> > I was hoping we could get the remaining "Metaalerts + ES 2.x" PRs merged
>> > and start cutting release candidates ASAP. That could be possible by
>> end
>> > of week based on the "big one" (METRON-1289) getting merged in last
>> night.
>> >
>> > Of course, if you happen to get all the Bro work done in time, then
>> > definitely let's include it in the next release. But I am wary of
>> blocking
>> > the release for that work. No need for you to rush through it.
>> >
>> > Just one man's opinion. Would like to hear feedback from more of the
>> > community.
>> >
>> >
>> >
>> > On Thu, Nov 16, 2017 at 8:01 AM, Zeolla@GMail.com <ze...@gmail.com>
>> > wrote:
>> >
>> > > The way master's full-dev is set up right now is non optimal for the
>> bro
>> > > plugin configuration, and I would like to complete the roadmap I
>> outlined
>> > > in my discuss thread before a release. I have a PR open against
>> > > Apache/metron-bro-plugin-kafka right now to turn it into a plugin,
>> and I
>> > > expect it will take me until end of next week at the latest to have
>> the
>> > > rest of the work done to move us to 2.5.2, and to pin to a specific
>> > package
>> > > version. At that point I'm comfortable with a release.
>> > >
>> > > Jon
>> > >
>> > > On Wed, Nov 15, 2017, 18:57 Matt Foley <ma...@apache.org> wrote:
>> > >
>> > > > I’ve been listening. Looks like there are still a number of major
>> > issues
>> > > > to be committed first, right?
>> > > > The discussion on this thread constitutes sufficient engagement, I
>> > think,
>> > > > especially given the Subject line :-)
>> > > > Would the folks working on the 6 issues listed by Nick care to
>> suggest
>> > a
>> > > > cut-off date by which they’ll probably have those fixes in?
>> > > > I’ll be happy to run the release process, if the community so
>> wishes,
>> > as
>> > > > soon as those issues are committed.
>> > > >
>> > > > I guess I should, pro forma, send the list of commits already in
>> since
>> > > the
>> > > > last release. I’ll do that today.
>> > > > Also, if anyone wishes to raise a hand and propose additional
>> commits
>> > are
>> > > > needed, please do so on this thread.
>> > > > Thanks,
>> > > > --Matt
>> > > >
>> > > >
>> > > > On 11/15/17, 2:09 PM, "Casey Stella" <ce...@gmail.com> wrote:
>> > > >
>> > > > I'd say that if a release is this imminent that we had better
>> > notify
>> > > > the
>> > > > release manager who will make a release announcement, Nick.
>> Matt,
>> > > are
>> > > > you
>> > > > tuning in to this?
>> > > >
>> > > > On Wed, Nov 15, 2017 at 10:04 AM, Nick Allen <
>> nick@nickallen.org>
>> > > > wrote:
>> > > >
>> > > > > Hi Guys -
>> > > > >
>> > > > > I want to follow-up on this discussion. It sounds like most
>> > people
>> > > > are in
>> > > > > agreement with the general approach.
>> > > > >
>> > > > > A lot of people have been working hard on Metaalerts and
>> > > > Elasticsearch. I
>> > > > > have checked-in with those doing the heavy lifting and have
>> > > compiled
>> > > > a more
>> > > > > detailed plan based on where we are at now. To the best of my
>> > > > knowledge
>> > > > > here is the plan of attack for finishing out this effort.
>> > > > >
>> > > > > (1) First, METRON-1289 needs to go in. This one was a
>> fairly
>> > big
>> > > > effort
>> > > > > and I am hearing that we are pretty close.
>> > > > >
>> > > > > (2) METRON-1294 fixes an issue in how field types are
>> > looked-up.
>> > > > >
>> > > > > (3) METRON-1290 is next. While this may have been fixed in
>> > > > M-1289, there
>> > > > > may be some test cases we want from this PR.
>> > > > >
>> > > > > (4) METRON-1301 addresses a problem with the sorting logic.
>> > > > >
>> > > > > (5) METRON-1291 fixes an issue with escalation of
>> metaalerts.
>> > > > >
>> > > > > (6) That leads us to Raghu's UI work in METRON-1252. This
>> > > > introduces the
>> > > > > UI bits that depend on all the previous backend work.
>> > > > >
>> > > > > (7) At this point, we should have our best effort at running
>> > > > Metaalerts
>> > > > > on Elasticsearch 2.x. I propose that we cut a release here.
>> > > > >
>> > > > > (8) After we cut the release, we can introduce the work for
>> ES
>> > > 5.x
>> > > > in
>> > > > > METRON-939. I know we will need lots of help testing and
>> > reviewing
>> > > > this
>> > > > > one.
>> > > > >
>> > > > > Please correct me if I am wrong. I will try and send out
>> updates
>> > > as
>> > > > we
>> > > > > make progress.
>> > > > >
>> > > > >
>> > > > >
>> > > > >
>> > > > >
>> > > > > On Mon, Nov 6, 2017 at 1:03 PM, Zeolla@GMail.com <
>> > zeolla@gmail.com
>> > > >
>> > > > wrote:
>> > > > >
>> > > > > > I agree, I think it's very reasonable to move in line with
>> > Nick's
>> > > > > > proposal. I would also suggest that we outline what the
>> target
>> > > > versions
>> > > > > > would be to add in the METRON-777 components, since it has
>> been
>> > > > > functional
>> > > > > > for a very long time but not reviewed and has some really
>> > > rockstar
>> > > > > > improvements.
>> > > > > >
>> > > > > > Jon
>> > > > > >
>> > > > > > On Mon, Nov 6, 2017 at 12:56 PM Otto Fowler <
>> > > > ottobackwards@gmail.com>
>> > > > > > wrote:
>> > > > > >
>> > > > > > > I think the ES cutover should be the start of the 0.5.x
>> > series,
>> > > > and we
>> > > > > > > continue on with 0.4.x for the
>> > > > > > > metadata improvements etc. We could chose to focus
>> 0.5.x’s
>> > > first
>> > > > > > releases
>> > > > > > > on not only ES but
>> > > > > > > getting a handle on kibana and the mpack situation as
>> well.
>> > > > > > >
>> > > > > > >
>> > > > > > >
>> > > > > > >
>> > > > > > > On November 6, 2017 at 12:48:45, Michael Miklavcic (
>> > > > > > > michael.miklavcic@gmail.com) wrote:
>> > > > > > >
>> > > > > > > I agree with your proposal, Nick. I think having a
>> > stabilizing
>> > > > release
>> > > > > > > prior to upgrading ES/Kibana makes sense.
>> > > > > > >
>> > > > > > > On Mon, Nov 6, 2017 at 9:16 AM, Nick Allen <
>> > nick@nickallen.org
>> > > >
>> > > > wrote:
>> > > > > > >
>> > > > > > > > I would like to start a discussion around upcoming
>> > releases.
>> > > > We have
>> > > > > a
>> > > > > > > > couple separate significant tracks of work that we need
>> to
>> > > > reconcile
>> > > > > in
>> > > > > > > our
>> > > > > > > > release schedule.
>> > > > > > > >
>> > > > > > > > (1) We have had (and have in review) a good number of
>> bug
>> > > fixes
>> > > > > > required
>> > > > > > > to
>> > > > > > > > support Metaalerts on the existing Elasticsearch 2.x
>> > > > infrastructure.
>> > > > > > > >
>> > > > > > > >
>> > > > > > > > (2) We also have ongoing work to upgrade our
>> infrastructure
>> > > to
>> > > > > > > > Elasticsearch 5.x, which will not be backwards
>> compatible.
>> > > > > > > >
>> > > > > > > >
>> > > > > > > > I would like to see a release that has our best work on
>> ES
>> > > 2.x
>> > > > before
>> > > > > > we
>> > > > > > > > migrate to 5.x. I would propose the following.
>> > > > > > > >
>> > > > > > > > Release N+1: Introduce Metaalerts running on ES 2.x
>> > > > > > > >
>> > > > > > > > Release N+2: Cut-over to ES 5.x
>> > > > > > > >
>> > > > > > > >
>> > > > > > > > (Q) Is it worth cutting a separate release for ES 2.x?
>> Is
>> > > > there a
>> > > > > > better
>> > > > > > > > way to handle the cut-over to 5.x?
>> > > > > > > >
>> > > > > > >
>> > > > > > --
>> > > > > >
>> > > > > > Jon
>> > > > > >
>> > > > >
>> > > >
>> > > >
>> > > >
>> > > > --
>> > >
>> > > Jon
>> > >
>> >
>> --
>>
>> Jon
>>
>
>
Re: [DISCUSS] Upcoming Release
Posted by Nick Allen <ni...@nickallen.org>.
Just to layout some other alternatives...
(1) [Jon] Make a release in apache/metron-bro-plugin-kafka, then update
Ansible to pin to that release.
(2) [Jon] Update Ansible to pin to a specific commit.
(3) Wouldn't the submodule approach solve this? I imagine that if we use a
submodule, then all of the Bro plugin code will be contained directly in
each Metron release. We would just need to change that small bit of
Ansible code so that it uses the code directly (like before) instead of
going to Git and checking it out.
(4) Revert PR #837 because as you pointed out this approach does not work
well with releases. That would give us enough time to come up with a
sensible approach and do that.
On Thu, Nov 16, 2017 at 10:21 AM, Zeolla@GMail.com <ze...@gmail.com> wrote:
> The problem is that we're currently pinning to master and if something in
> the plugin breaks backward compatibility, our prior releases will be
> perpetually broken, which is why I suggest it blocks the upcoming release.
>
> The alternative would be to update ansible to pin to a specific commit or
> to make a release in apache/metron-bro-plugin-kafka sooner rather than
> later and pin to its branch/tag. That feels like a waste of time though,
> as the 2.5.2 upgrade is somewhat trivial.
>
> Jon
>
> On Thu, Nov 16, 2017 at 10:14 AM Nick Allen <ni...@nickallen.org> wrote:
>
> > While I think that the Bro work is super valuable, Jon, I am not sure
> that
> > we need to block the next release for it. In my mind, the "theme" of the
> > next release is Metaalerts running on ES 2.x. I'd prefer to just stick
> > with that.
> >
> > I was hoping we could get the remaining "Metaalerts + ES 2.x" PRs merged
> > and start cutting release candidates ASAP. That could be possible by end
> > of week based on the "big one" (METRON-1289) getting merged in last
> night.
> >
> > Of course, if you happen to get all the Bro work done in time, then
> > definitely let's include it in the next release. But I am wary of
> blocking
> > the release for that work. No need for you to rush through it.
> >
> > Just one man's opinion. Would like to hear feedback from more of the
> > community.
> >
> >
> >
> > On Thu, Nov 16, 2017 at 8:01 AM, Zeolla@GMail.com <ze...@gmail.com>
> > wrote:
> >
> > > The way master's full-dev is set up right now is non optimal for the
> bro
> > > plugin configuration, and I would like to complete the roadmap I
> outlined
> > > in my discuss thread before a release. I have a PR open against
> > > Apache/metron-bro-plugin-kafka right now to turn it into a plugin, and
> I
> > > expect it will take me until end of next week at the latest to have the
> > > rest of the work done to move us to 2.5.2, and to pin to a specific
> > package
> > > version. At that point I'm comfortable with a release.
> > >
> > > Jon
> > >
> > > On Wed, Nov 15, 2017, 18:57 Matt Foley <ma...@apache.org> wrote:
> > >
> > > > I’ve been listening. Looks like there are still a number of major
> > issues
> > > > to be committed first, right?
> > > > The discussion on this thread constitutes sufficient engagement, I
> > think,
> > > > especially given the Subject line :-)
> > > > Would the folks working on the 6 issues listed by Nick care to
> suggest
> > a
> > > > cut-off date by which they’ll probably have those fixes in?
> > > > I’ll be happy to run the release process, if the community so wishes,
> > as
> > > > soon as those issues are committed.
> > > >
> > > > I guess I should, pro forma, send the list of commits already in
> since
> > > the
> > > > last release. I’ll do that today.
> > > > Also, if anyone wishes to raise a hand and propose additional commits
> > are
> > > > needed, please do so on this thread.
> > > > Thanks,
> > > > --Matt
> > > >
> > > >
> > > > On 11/15/17, 2:09 PM, "Casey Stella" <ce...@gmail.com> wrote:
> > > >
> > > > I'd say that if a release is this imminent that we had better
> > notify
> > > > the
> > > > release manager who will make a release announcement, Nick.
> Matt,
> > > are
> > > > you
> > > > tuning in to this?
> > > >
> > > > On Wed, Nov 15, 2017 at 10:04 AM, Nick Allen <nick@nickallen.org
> >
> > > > wrote:
> > > >
> > > > > Hi Guys -
> > > > >
> > > > > I want to follow-up on this discussion. It sounds like most
> > people
> > > > are in
> > > > > agreement with the general approach.
> > > > >
> > > > > A lot of people have been working hard on Metaalerts and
> > > > Elasticsearch. I
> > > > > have checked-in with those doing the heavy lifting and have
> > > compiled
> > > > a more
> > > > > detailed plan based on where we are at now. To the best of my
> > > > knowledge
> > > > > here is the plan of attack for finishing out this effort.
> > > > >
> > > > > (1) First, METRON-1289 needs to go in. This one was a fairly
> > big
> > > > effort
> > > > > and I am hearing that we are pretty close.
> > > > >
> > > > > (2) METRON-1294 fixes an issue in how field types are
> > looked-up.
> > > > >
> > > > > (3) METRON-1290 is next. While this may have been fixed in
> > > > M-1289, there
> > > > > may be some test cases we want from this PR.
> > > > >
> > > > > (4) METRON-1301 addresses a problem with the sorting logic.
> > > > >
> > > > > (5) METRON-1291 fixes an issue with escalation of metaalerts.
> > > > >
> > > > > (6) That leads us to Raghu's UI work in METRON-1252. This
> > > > introduces the
> > > > > UI bits that depend on all the previous backend work.
> > > > >
> > > > > (7) At this point, we should have our best effort at running
> > > > Metaalerts
> > > > > on Elasticsearch 2.x. I propose that we cut a release here.
> > > > >
> > > > > (8) After we cut the release, we can introduce the work for
> ES
> > > 5.x
> > > > in
> > > > > METRON-939. I know we will need lots of help testing and
> > reviewing
> > > > this
> > > > > one.
> > > > >
> > > > > Please correct me if I am wrong. I will try and send out
> updates
> > > as
> > > > we
> > > > > make progress.
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > On Mon, Nov 6, 2017 at 1:03 PM, Zeolla@GMail.com <
> > zeolla@gmail.com
> > > >
> > > > wrote:
> > > > >
> > > > > > I agree, I think it's very reasonable to move in line with
> > Nick's
> > > > > > proposal. I would also suggest that we outline what the
> target
> > > > versions
> > > > > > would be to add in the METRON-777 components, since it has
> been
> > > > > functional
> > > > > > for a very long time but not reviewed and has some really
> > > rockstar
> > > > > > improvements.
> > > > > >
> > > > > > Jon
> > > > > >
> > > > > > On Mon, Nov 6, 2017 at 12:56 PM Otto Fowler <
> > > > ottobackwards@gmail.com>
> > > > > > wrote:
> > > > > >
> > > > > > > I think the ES cutover should be the start of the 0.5.x
> > series,
> > > > and we
> > > > > > > continue on with 0.4.x for the
> > > > > > > metadata improvements etc. We could chose to focus 0.5.x’s
> > > first
> > > > > > releases
> > > > > > > on not only ES but
> > > > > > > getting a handle on kibana and the mpack situation as well.
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > On November 6, 2017 at 12:48:45, Michael Miklavcic (
> > > > > > > michael.miklavcic@gmail.com) wrote:
> > > > > > >
> > > > > > > I agree with your proposal, Nick. I think having a
> > stabilizing
> > > > release
> > > > > > > prior to upgrading ES/Kibana makes sense.
> > > > > > >
> > > > > > > On Mon, Nov 6, 2017 at 9:16 AM, Nick Allen <
> > nick@nickallen.org
> > > >
> > > > wrote:
> > > > > > >
> > > > > > > > I would like to start a discussion around upcoming
> > releases.
> > > > We have
> > > > > a
> > > > > > > > couple separate significant tracks of work that we need
> to
> > > > reconcile
> > > > > in
> > > > > > > our
> > > > > > > > release schedule.
> > > > > > > >
> > > > > > > > (1) We have had (and have in review) a good number of bug
> > > fixes
> > > > > > required
> > > > > > > to
> > > > > > > > support Metaalerts on the existing Elasticsearch 2.x
> > > > infrastructure.
> > > > > > > >
> > > > > > > >
> > > > > > > > (2) We also have ongoing work to upgrade our
> infrastructure
> > > to
> > > > > > > > Elasticsearch 5.x, which will not be backwards
> compatible.
> > > > > > > >
> > > > > > > >
> > > > > > > > I would like to see a release that has our best work on
> ES
> > > 2.x
> > > > before
> > > > > > we
> > > > > > > > migrate to 5.x. I would propose the following.
> > > > > > > >
> > > > > > > > Release N+1: Introduce Metaalerts running on ES 2.x
> > > > > > > >
> > > > > > > > Release N+2: Cut-over to ES 5.x
> > > > > > > >
> > > > > > > >
> > > > > > > > (Q) Is it worth cutting a separate release for ES 2.x? Is
> > > > there a
> > > > > > better
> > > > > > > > way to handle the cut-over to 5.x?
> > > > > > > >
> > > > > > >
> > > > > > --
> > > > > >
> > > > > > Jon
> > > > > >
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > >
> > > Jon
> > >
> >
> --
>
> Jon
>
Re: [DISCUSS] Upcoming Release
Posted by "Zeolla@GMail.com" <ze...@gmail.com>.
The problem is that we're currently pinning to master and if something in
the plugin breaks backward compatibility, our prior releases will be
perpetually broken, which is why I suggest it blocks the upcoming release.
The alternative would be to update ansible to pin to a specific commit or
to make a release in apache/metron-bro-plugin-kafka sooner rather than
later and pin to its branch/tag. That feels like a waste of time though,
as the 2.5.2 upgrade is somewhat trivial.
Jon
On Thu, Nov 16, 2017 at 10:14 AM Nick Allen <ni...@nickallen.org> wrote:
> While I think that the Bro work is super valuable, Jon, I am not sure that
> we need to block the next release for it. In my mind, the "theme" of the
> next release is Metaalerts running on ES 2.x. I'd prefer to just stick
> with that.
>
> I was hoping we could get the remaining "Metaalerts + ES 2.x" PRs merged
> and start cutting release candidates ASAP. That could be possible by end
> of week based on the "big one" (METRON-1289) getting merged in last night.
>
> Of course, if you happen to get all the Bro work done in time, then
> definitely let's include it in the next release. But I am wary of blocking
> the release for that work. No need for you to rush through it.
>
> Just one man's opinion. Would like to hear feedback from more of the
> community.
>
>
>
> On Thu, Nov 16, 2017 at 8:01 AM, Zeolla@GMail.com <ze...@gmail.com>
> wrote:
>
> > The way master's full-dev is set up right now is non optimal for the bro
> > plugin configuration, and I would like to complete the roadmap I outlined
> > in my discuss thread before a release. I have a PR open against
> > Apache/metron-bro-plugin-kafka right now to turn it into a plugin, and I
> > expect it will take me until end of next week at the latest to have the
> > rest of the work done to move us to 2.5.2, and to pin to a specific
> package
> > version. At that point I'm comfortable with a release.
> >
> > Jon
> >
> > On Wed, Nov 15, 2017, 18:57 Matt Foley <ma...@apache.org> wrote:
> >
> > > I’ve been listening. Looks like there are still a number of major
> issues
> > > to be committed first, right?
> > > The discussion on this thread constitutes sufficient engagement, I
> think,
> > > especially given the Subject line :-)
> > > Would the folks working on the 6 issues listed by Nick care to suggest
> a
> > > cut-off date by which they’ll probably have those fixes in?
> > > I’ll be happy to run the release process, if the community so wishes,
> as
> > > soon as those issues are committed.
> > >
> > > I guess I should, pro forma, send the list of commits already in since
> > the
> > > last release. I’ll do that today.
> > > Also, if anyone wishes to raise a hand and propose additional commits
> are
> > > needed, please do so on this thread.
> > > Thanks,
> > > --Matt
> > >
> > >
> > > On 11/15/17, 2:09 PM, "Casey Stella" <ce...@gmail.com> wrote:
> > >
> > > I'd say that if a release is this imminent that we had better
> notify
> > > the
> > > release manager who will make a release announcement, Nick. Matt,
> > are
> > > you
> > > tuning in to this?
> > >
> > > On Wed, Nov 15, 2017 at 10:04 AM, Nick Allen <ni...@nickallen.org>
> > > wrote:
> > >
> > > > Hi Guys -
> > > >
> > > > I want to follow-up on this discussion. It sounds like most
> people
> > > are in
> > > > agreement with the general approach.
> > > >
> > > > A lot of people have been working hard on Metaalerts and
> > > Elasticsearch. I
> > > > have checked-in with those doing the heavy lifting and have
> > compiled
> > > a more
> > > > detailed plan based on where we are at now. To the best of my
> > > knowledge
> > > > here is the plan of attack for finishing out this effort.
> > > >
> > > > (1) First, METRON-1289 needs to go in. This one was a fairly
> big
> > > effort
> > > > and I am hearing that we are pretty close.
> > > >
> > > > (2) METRON-1294 fixes an issue in how field types are
> looked-up.
> > > >
> > > > (3) METRON-1290 is next. While this may have been fixed in
> > > M-1289, there
> > > > may be some test cases we want from this PR.
> > > >
> > > > (4) METRON-1301 addresses a problem with the sorting logic.
> > > >
> > > > (5) METRON-1291 fixes an issue with escalation of metaalerts.
> > > >
> > > > (6) That leads us to Raghu's UI work in METRON-1252. This
> > > introduces the
> > > > UI bits that depend on all the previous backend work.
> > > >
> > > > (7) At this point, we should have our best effort at running
> > > Metaalerts
> > > > on Elasticsearch 2.x. I propose that we cut a release here.
> > > >
> > > > (8) After we cut the release, we can introduce the work for ES
> > 5.x
> > > in
> > > > METRON-939. I know we will need lots of help testing and
> reviewing
> > > this
> > > > one.
> > > >
> > > > Please correct me if I am wrong. I will try and send out updates
> > as
> > > we
> > > > make progress.
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > On Mon, Nov 6, 2017 at 1:03 PM, Zeolla@GMail.com <
> zeolla@gmail.com
> > >
> > > wrote:
> > > >
> > > > > I agree, I think it's very reasonable to move in line with
> Nick's
> > > > > proposal. I would also suggest that we outline what the target
> > > versions
> > > > > would be to add in the METRON-777 components, since it has been
> > > > functional
> > > > > for a very long time but not reviewed and has some really
> > rockstar
> > > > > improvements.
> > > > >
> > > > > Jon
> > > > >
> > > > > On Mon, Nov 6, 2017 at 12:56 PM Otto Fowler <
> > > ottobackwards@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > I think the ES cutover should be the start of the 0.5.x
> series,
> > > and we
> > > > > > continue on with 0.4.x for the
> > > > > > metadata improvements etc. We could chose to focus 0.5.x’s
> > first
> > > > > releases
> > > > > > on not only ES but
> > > > > > getting a handle on kibana and the mpack situation as well.
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > On November 6, 2017 at 12:48:45, Michael Miklavcic (
> > > > > > michael.miklavcic@gmail.com) wrote:
> > > > > >
> > > > > > I agree with your proposal, Nick. I think having a
> stabilizing
> > > release
> > > > > > prior to upgrading ES/Kibana makes sense.
> > > > > >
> > > > > > On Mon, Nov 6, 2017 at 9:16 AM, Nick Allen <
> nick@nickallen.org
> > >
> > > wrote:
> > > > > >
> > > > > > > I would like to start a discussion around upcoming
> releases.
> > > We have
> > > > a
> > > > > > > couple separate significant tracks of work that we need to
> > > reconcile
> > > > in
> > > > > > our
> > > > > > > release schedule.
> > > > > > >
> > > > > > > (1) We have had (and have in review) a good number of bug
> > fixes
> > > > > required
> > > > > > to
> > > > > > > support Metaalerts on the existing Elasticsearch 2.x
> > > infrastructure.
> > > > > > >
> > > > > > >
> > > > > > > (2) We also have ongoing work to upgrade our infrastructure
> > to
> > > > > > > Elasticsearch 5.x, which will not be backwards compatible.
> > > > > > >
> > > > > > >
> > > > > > > I would like to see a release that has our best work on ES
> > 2.x
> > > before
> > > > > we
> > > > > > > migrate to 5.x. I would propose the following.
> > > > > > >
> > > > > > > Release N+1: Introduce Metaalerts running on ES 2.x
> > > > > > >
> > > > > > > Release N+2: Cut-over to ES 5.x
> > > > > > >
> > > > > > >
> > > > > > > (Q) Is it worth cutting a separate release for ES 2.x? Is
> > > there a
> > > > > better
> > > > > > > way to handle the cut-over to 5.x?
> > > > > > >
> > > > > >
> > > > > --
> > > > >
> > > > > Jon
> > > > >
> > > >
> > >
> > >
> > >
> > > --
> >
> > Jon
> >
>
--
Jon
Re: [DISCUSS] Upcoming Release
Posted by Nick Allen <ni...@nickallen.org>.
While I think that the Bro work is super valuable, Jon, I am not sure that
we need to block the next release for it. In my mind, the "theme" of the
next release is Metaalerts running on ES 2.x. I'd prefer to just stick
with that.
I was hoping we could get the remaining "Metaalerts + ES 2.x" PRs merged
and start cutting release candidates ASAP. That could be possible by end
of week based on the "big one" (METRON-1289) getting merged in last night.
Of course, if you happen to get all the Bro work done in time, then
definitely let's include it in the next release. But I am wary of blocking
the release for that work. No need for you to rush through it.
Just one man's opinion. Would like to hear feedback from more of the
community.
On Thu, Nov 16, 2017 at 8:01 AM, Zeolla@GMail.com <ze...@gmail.com> wrote:
> The way master's full-dev is set up right now is non optimal for the bro
> plugin configuration, and I would like to complete the roadmap I outlined
> in my discuss thread before a release. I have a PR open against
> Apache/metron-bro-plugin-kafka right now to turn it into a plugin, and I
> expect it will take me until end of next week at the latest to have the
> rest of the work done to move us to 2.5.2, and to pin to a specific package
> version. At that point I'm comfortable with a release.
>
> Jon
>
> On Wed, Nov 15, 2017, 18:57 Matt Foley <ma...@apache.org> wrote:
>
> > I’ve been listening. Looks like there are still a number of major issues
> > to be committed first, right?
> > The discussion on this thread constitutes sufficient engagement, I think,
> > especially given the Subject line :-)
> > Would the folks working on the 6 issues listed by Nick care to suggest a
> > cut-off date by which they’ll probably have those fixes in?
> > I’ll be happy to run the release process, if the community so wishes, as
> > soon as those issues are committed.
> >
> > I guess I should, pro forma, send the list of commits already in since
> the
> > last release. I’ll do that today.
> > Also, if anyone wishes to raise a hand and propose additional commits are
> > needed, please do so on this thread.
> > Thanks,
> > --Matt
> >
> >
> > On 11/15/17, 2:09 PM, "Casey Stella" <ce...@gmail.com> wrote:
> >
> > I'd say that if a release is this imminent that we had better notify
> > the
> > release manager who will make a release announcement, Nick. Matt,
> are
> > you
> > tuning in to this?
> >
> > On Wed, Nov 15, 2017 at 10:04 AM, Nick Allen <ni...@nickallen.org>
> > wrote:
> >
> > > Hi Guys -
> > >
> > > I want to follow-up on this discussion. It sounds like most people
> > are in
> > > agreement with the general approach.
> > >
> > > A lot of people have been working hard on Metaalerts and
> > Elasticsearch. I
> > > have checked-in with those doing the heavy lifting and have
> compiled
> > a more
> > > detailed plan based on where we are at now. To the best of my
> > knowledge
> > > here is the plan of attack for finishing out this effort.
> > >
> > > (1) First, METRON-1289 needs to go in. This one was a fairly big
> > effort
> > > and I am hearing that we are pretty close.
> > >
> > > (2) METRON-1294 fixes an issue in how field types are looked-up.
> > >
> > > (3) METRON-1290 is next. While this may have been fixed in
> > M-1289, there
> > > may be some test cases we want from this PR.
> > >
> > > (4) METRON-1301 addresses a problem with the sorting logic.
> > >
> > > (5) METRON-1291 fixes an issue with escalation of metaalerts.
> > >
> > > (6) That leads us to Raghu's UI work in METRON-1252. This
> > introduces the
> > > UI bits that depend on all the previous backend work.
> > >
> > > (7) At this point, we should have our best effort at running
> > Metaalerts
> > > on Elasticsearch 2.x. I propose that we cut a release here.
> > >
> > > (8) After we cut the release, we can introduce the work for ES
> 5.x
> > in
> > > METRON-939. I know we will need lots of help testing and reviewing
> > this
> > > one.
> > >
> > > Please correct me if I am wrong. I will try and send out updates
> as
> > we
> > > make progress.
> > >
> > >
> > >
> > >
> > >
> > > On Mon, Nov 6, 2017 at 1:03 PM, Zeolla@GMail.com <zeolla@gmail.com
> >
> > wrote:
> > >
> > > > I agree, I think it's very reasonable to move in line with Nick's
> > > > proposal. I would also suggest that we outline what the target
> > versions
> > > > would be to add in the METRON-777 components, since it has been
> > > functional
> > > > for a very long time but not reviewed and has some really
> rockstar
> > > > improvements.
> > > >
> > > > Jon
> > > >
> > > > On Mon, Nov 6, 2017 at 12:56 PM Otto Fowler <
> > ottobackwards@gmail.com>
> > > > wrote:
> > > >
> > > > > I think the ES cutover should be the start of the 0.5.x series,
> > and we
> > > > > continue on with 0.4.x for the
> > > > > metadata improvements etc. We could chose to focus 0.5.x’s
> first
> > > > releases
> > > > > on not only ES but
> > > > > getting a handle on kibana and the mpack situation as well.
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > On November 6, 2017 at 12:48:45, Michael Miklavcic (
> > > > > michael.miklavcic@gmail.com) wrote:
> > > > >
> > > > > I agree with your proposal, Nick. I think having a stabilizing
> > release
> > > > > prior to upgrading ES/Kibana makes sense.
> > > > >
> > > > > On Mon, Nov 6, 2017 at 9:16 AM, Nick Allen <nick@nickallen.org
> >
> > wrote:
> > > > >
> > > > > > I would like to start a discussion around upcoming releases.
> > We have
> > > a
> > > > > > couple separate significant tracks of work that we need to
> > reconcile
> > > in
> > > > > our
> > > > > > release schedule.
> > > > > >
> > > > > > (1) We have had (and have in review) a good number of bug
> fixes
> > > > required
> > > > > to
> > > > > > support Metaalerts on the existing Elasticsearch 2.x
> > infrastructure.
> > > > > >
> > > > > >
> > > > > > (2) We also have ongoing work to upgrade our infrastructure
> to
> > > > > > Elasticsearch 5.x, which will not be backwards compatible.
> > > > > >
> > > > > >
> > > > > > I would like to see a release that has our best work on ES
> 2.x
> > before
> > > > we
> > > > > > migrate to 5.x. I would propose the following.
> > > > > >
> > > > > > Release N+1: Introduce Metaalerts running on ES 2.x
> > > > > >
> > > > > > Release N+2: Cut-over to ES 5.x
> > > > > >
> > > > > >
> > > > > > (Q) Is it worth cutting a separate release for ES 2.x? Is
> > there a
> > > > better
> > > > > > way to handle the cut-over to 5.x?
> > > > > >
> > > > >
> > > > --
> > > >
> > > > Jon
> > > >
> > >
> >
> >
> >
> > --
>
> Jon
>
Re: [DISCUSS] Upcoming Release
Posted by "Zeolla@GMail.com" <ze...@gmail.com>.
My PR is to turn it into a package containing a plugin*
On Thu, Nov 16, 2017, 08:01 Zeolla@GMail.com <ze...@gmail.com> wrote:
> The way master's full-dev is set up right now is non optimal for the bro
> plugin configuration, and I would like to complete the roadmap I outlined
> in my discuss thread before a release. I have a PR open against
> Apache/metron-bro-plugin-kafka right now to turn it into a plugin, and I
> expect it will take me until end of next week at the latest to have the
> rest of the work done to move us to 2.5.2, and to pin to a specific package
> version. At that point I'm comfortable with a release.
>
> Jon
>
> On Wed, Nov 15, 2017, 18:57 Matt Foley <ma...@apache.org> wrote:
>
>> I’ve been listening. Looks like there are still a number of major issues
>> to be committed first, right?
>> The discussion on this thread constitutes sufficient engagement, I think,
>> especially given the Subject line :-)
>> Would the folks working on the 6 issues listed by Nick care to suggest a
>> cut-off date by which they’ll probably have those fixes in?
>> I’ll be happy to run the release process, if the community so wishes, as
>> soon as those issues are committed.
>>
>> I guess I should, pro forma, send the list of commits already in since
>> the last release. I’ll do that today.
>> Also, if anyone wishes to raise a hand and propose additional commits are
>> needed, please do so on this thread.
>> Thanks,
>> --Matt
>>
>>
>> On 11/15/17, 2:09 PM, "Casey Stella" <ce...@gmail.com> wrote:
>>
>> I'd say that if a release is this imminent that we had better notify
>> the
>> release manager who will make a release announcement, Nick. Matt,
>> are you
>> tuning in to this?
>>
>> On Wed, Nov 15, 2017 at 10:04 AM, Nick Allen <ni...@nickallen.org>
>> wrote:
>>
>> > Hi Guys -
>> >
>> > I want to follow-up on this discussion. It sounds like most people
>> are in
>> > agreement with the general approach.
>> >
>> > A lot of people have been working hard on Metaalerts and
>> Elasticsearch. I
>> > have checked-in with those doing the heavy lifting and have
>> compiled a more
>> > detailed plan based on where we are at now. To the best of my
>> knowledge
>> > here is the plan of attack for finishing out this effort.
>> >
>> > (1) First, METRON-1289 needs to go in. This one was a fairly big
>> effort
>> > and I am hearing that we are pretty close.
>> >
>> > (2) METRON-1294 fixes an issue in how field types are looked-up.
>> >
>> > (3) METRON-1290 is next. While this may have been fixed in
>> M-1289, there
>> > may be some test cases we want from this PR.
>> >
>> > (4) METRON-1301 addresses a problem with the sorting logic.
>> >
>> > (5) METRON-1291 fixes an issue with escalation of metaalerts.
>> >
>> > (6) That leads us to Raghu's UI work in METRON-1252. This
>> introduces the
>> > UI bits that depend on all the previous backend work.
>> >
>> > (7) At this point, we should have our best effort at running
>> Metaalerts
>> > on Elasticsearch 2.x. I propose that we cut a release here.
>> >
>> > (8) After we cut the release, we can introduce the work for ES
>> 5.x in
>> > METRON-939. I know we will need lots of help testing and reviewing
>> this
>> > one.
>> >
>> > Please correct me if I am wrong. I will try and send out updates
>> as we
>> > make progress.
>> >
>> >
>> >
>> >
>> >
>> > On Mon, Nov 6, 2017 at 1:03 PM, Zeolla@GMail.com <ze...@gmail.com>
>> wrote:
>> >
>> > > I agree, I think it's very reasonable to move in line with Nick's
>> > > proposal. I would also suggest that we outline what the target
>> versions
>> > > would be to add in the METRON-777 components, since it has been
>> > functional
>> > > for a very long time but not reviewed and has some really rockstar
>> > > improvements.
>> > >
>> > > Jon
>> > >
>> > > On Mon, Nov 6, 2017 at 12:56 PM Otto Fowler <
>> ottobackwards@gmail.com>
>> > > wrote:
>> > >
>> > > > I think the ES cutover should be the start of the 0.5.x series,
>> and we
>> > > > continue on with 0.4.x for the
>> > > > metadata improvements etc. We could chose to focus 0.5.x’s
>> first
>> > > releases
>> > > > on not only ES but
>> > > > getting a handle on kibana and the mpack situation as well.
>> > > >
>> > > >
>> > > >
>> > > >
>> > > > On November 6, 2017 at 12:48:45, Michael Miklavcic (
>> > > > michael.miklavcic@gmail.com) wrote:
>> > > >
>> > > > I agree with your proposal, Nick. I think having a stabilizing
>> release
>> > > > prior to upgrading ES/Kibana makes sense.
>> > > >
>> > > > On Mon, Nov 6, 2017 at 9:16 AM, Nick Allen <ni...@nickallen.org>
>> wrote:
>> > > >
>> > > > > I would like to start a discussion around upcoming releases.
>> We have
>> > a
>> > > > > couple separate significant tracks of work that we need to
>> reconcile
>> > in
>> > > > our
>> > > > > release schedule.
>> > > > >
>> > > > > (1) We have had (and have in review) a good number of bug
>> fixes
>> > > required
>> > > > to
>> > > > > support Metaalerts on the existing Elasticsearch 2.x
>> infrastructure.
>> > > > >
>> > > > >
>> > > > > (2) We also have ongoing work to upgrade our infrastructure to
>> > > > > Elasticsearch 5.x, which will not be backwards compatible.
>> > > > >
>> > > > >
>> > > > > I would like to see a release that has our best work on ES
>> 2.x before
>> > > we
>> > > > > migrate to 5.x. I would propose the following.
>> > > > >
>> > > > > Release N+1: Introduce Metaalerts running on ES 2.x
>> > > > >
>> > > > > Release N+2: Cut-over to ES 5.x
>> > > > >
>> > > > >
>> > > > > (Q) Is it worth cutting a separate release for ES 2.x? Is
>> there a
>> > > better
>> > > > > way to handle the cut-over to 5.x?
>> > > > >
>> > > >
>> > > --
>> > >
>> > > Jon
>> > >
>> >
>>
>>
>>
>> --
>
> Jon
>
--
Jon
Re: [DISCUSS] Upcoming Release
Posted by "Zeolla@GMail.com" <ze...@gmail.com>.
The way master's full-dev is set up right now is non optimal for the bro
plugin configuration, and I would like to complete the roadmap I outlined
in my discuss thread before a release. I have a PR open against
Apache/metron-bro-plugin-kafka right now to turn it into a plugin, and I
expect it will take me until end of next week at the latest to have the
rest of the work done to move us to 2.5.2, and to pin to a specific package
version. At that point I'm comfortable with a release.
Jon
On Wed, Nov 15, 2017, 18:57 Matt Foley <ma...@apache.org> wrote:
> I’ve been listening. Looks like there are still a number of major issues
> to be committed first, right?
> The discussion on this thread constitutes sufficient engagement, I think,
> especially given the Subject line :-)
> Would the folks working on the 6 issues listed by Nick care to suggest a
> cut-off date by which they’ll probably have those fixes in?
> I’ll be happy to run the release process, if the community so wishes, as
> soon as those issues are committed.
>
> I guess I should, pro forma, send the list of commits already in since the
> last release. I’ll do that today.
> Also, if anyone wishes to raise a hand and propose additional commits are
> needed, please do so on this thread.
> Thanks,
> --Matt
>
>
> On 11/15/17, 2:09 PM, "Casey Stella" <ce...@gmail.com> wrote:
>
> I'd say that if a release is this imminent that we had better notify
> the
> release manager who will make a release announcement, Nick. Matt, are
> you
> tuning in to this?
>
> On Wed, Nov 15, 2017 at 10:04 AM, Nick Allen <ni...@nickallen.org>
> wrote:
>
> > Hi Guys -
> >
> > I want to follow-up on this discussion. It sounds like most people
> are in
> > agreement with the general approach.
> >
> > A lot of people have been working hard on Metaalerts and
> Elasticsearch. I
> > have checked-in with those doing the heavy lifting and have compiled
> a more
> > detailed plan based on where we are at now. To the best of my
> knowledge
> > here is the plan of attack for finishing out this effort.
> >
> > (1) First, METRON-1289 needs to go in. This one was a fairly big
> effort
> > and I am hearing that we are pretty close.
> >
> > (2) METRON-1294 fixes an issue in how field types are looked-up.
> >
> > (3) METRON-1290 is next. While this may have been fixed in
> M-1289, there
> > may be some test cases we want from this PR.
> >
> > (4) METRON-1301 addresses a problem with the sorting logic.
> >
> > (5) METRON-1291 fixes an issue with escalation of metaalerts.
> >
> > (6) That leads us to Raghu's UI work in METRON-1252. This
> introduces the
> > UI bits that depend on all the previous backend work.
> >
> > (7) At this point, we should have our best effort at running
> Metaalerts
> > on Elasticsearch 2.x. I propose that we cut a release here.
> >
> > (8) After we cut the release, we can introduce the work for ES 5.x
> in
> > METRON-939. I know we will need lots of help testing and reviewing
> this
> > one.
> >
> > Please correct me if I am wrong. I will try and send out updates as
> we
> > make progress.
> >
> >
> >
> >
> >
> > On Mon, Nov 6, 2017 at 1:03 PM, Zeolla@GMail.com <ze...@gmail.com>
> wrote:
> >
> > > I agree, I think it's very reasonable to move in line with Nick's
> > > proposal. I would also suggest that we outline what the target
> versions
> > > would be to add in the METRON-777 components, since it has been
> > functional
> > > for a very long time but not reviewed and has some really rockstar
> > > improvements.
> > >
> > > Jon
> > >
> > > On Mon, Nov 6, 2017 at 12:56 PM Otto Fowler <
> ottobackwards@gmail.com>
> > > wrote:
> > >
> > > > I think the ES cutover should be the start of the 0.5.x series,
> and we
> > > > continue on with 0.4.x for the
> > > > metadata improvements etc. We could chose to focus 0.5.x’s first
> > > releases
> > > > on not only ES but
> > > > getting a handle on kibana and the mpack situation as well.
> > > >
> > > >
> > > >
> > > >
> > > > On November 6, 2017 at 12:48:45, Michael Miklavcic (
> > > > michael.miklavcic@gmail.com) wrote:
> > > >
> > > > I agree with your proposal, Nick. I think having a stabilizing
> release
> > > > prior to upgrading ES/Kibana makes sense.
> > > >
> > > > On Mon, Nov 6, 2017 at 9:16 AM, Nick Allen <ni...@nickallen.org>
> wrote:
> > > >
> > > > > I would like to start a discussion around upcoming releases.
> We have
> > a
> > > > > couple separate significant tracks of work that we need to
> reconcile
> > in
> > > > our
> > > > > release schedule.
> > > > >
> > > > > (1) We have had (and have in review) a good number of bug fixes
> > > required
> > > > to
> > > > > support Metaalerts on the existing Elasticsearch 2.x
> infrastructure.
> > > > >
> > > > >
> > > > > (2) We also have ongoing work to upgrade our infrastructure to
> > > > > Elasticsearch 5.x, which will not be backwards compatible.
> > > > >
> > > > >
> > > > > I would like to see a release that has our best work on ES 2.x
> before
> > > we
> > > > > migrate to 5.x. I would propose the following.
> > > > >
> > > > > Release N+1: Introduce Metaalerts running on ES 2.x
> > > > >
> > > > > Release N+2: Cut-over to ES 5.x
> > > > >
> > > > >
> > > > > (Q) Is it worth cutting a separate release for ES 2.x? Is
> there a
> > > better
> > > > > way to handle the cut-over to 5.x?
> > > > >
> > > >
> > > --
> > >
> > > Jon
> > >
> >
>
>
>
> --
Jon
Re: [DISCUSS] Upcoming Release
Posted by Matt Foley <ma...@apache.org>.
I’ve been listening. Looks like there are still a number of major issues to be committed first, right?
The discussion on this thread constitutes sufficient engagement, I think, especially given the Subject line :-)
Would the folks working on the 6 issues listed by Nick care to suggest a cut-off date by which they’ll probably have those fixes in?
I’ll be happy to run the release process, if the community so wishes, as soon as those issues are committed.
I guess I should, pro forma, send the list of commits already in since the last release. I’ll do that today.
Also, if anyone wishes to raise a hand and propose additional commits are needed, please do so on this thread.
Thanks,
--Matt
On 11/15/17, 2:09 PM, "Casey Stella" <ce...@gmail.com> wrote:
I'd say that if a release is this imminent that we had better notify the
release manager who will make a release announcement, Nick. Matt, are you
tuning in to this?
On Wed, Nov 15, 2017 at 10:04 AM, Nick Allen <ni...@nickallen.org> wrote:
> Hi Guys -
>
> I want to follow-up on this discussion. It sounds like most people are in
> agreement with the general approach.
>
> A lot of people have been working hard on Metaalerts and Elasticsearch. I
> have checked-in with those doing the heavy lifting and have compiled a more
> detailed plan based on where we are at now. To the best of my knowledge
> here is the plan of attack for finishing out this effort.
>
> (1) First, METRON-1289 needs to go in. This one was a fairly big effort
> and I am hearing that we are pretty close.
>
> (2) METRON-1294 fixes an issue in how field types are looked-up.
>
> (3) METRON-1290 is next. While this may have been fixed in M-1289, there
> may be some test cases we want from this PR.
>
> (4) METRON-1301 addresses a problem with the sorting logic.
>
> (5) METRON-1291 fixes an issue with escalation of metaalerts.
>
> (6) That leads us to Raghu's UI work in METRON-1252. This introduces the
> UI bits that depend on all the previous backend work.
>
> (7) At this point, we should have our best effort at running Metaalerts
> on Elasticsearch 2.x. I propose that we cut a release here.
>
> (8) After we cut the release, we can introduce the work for ES 5.x in
> METRON-939. I know we will need lots of help testing and reviewing this
> one.
>
> Please correct me if I am wrong. I will try and send out updates as we
> make progress.
>
>
>
>
>
> On Mon, Nov 6, 2017 at 1:03 PM, Zeolla@GMail.com <ze...@gmail.com> wrote:
>
> > I agree, I think it's very reasonable to move in line with Nick's
> > proposal. I would also suggest that we outline what the target versions
> > would be to add in the METRON-777 components, since it has been
> functional
> > for a very long time but not reviewed and has some really rockstar
> > improvements.
> >
> > Jon
> >
> > On Mon, Nov 6, 2017 at 12:56 PM Otto Fowler <ot...@gmail.com>
> > wrote:
> >
> > > I think the ES cutover should be the start of the 0.5.x series, and we
> > > continue on with 0.4.x for the
> > > metadata improvements etc. We could chose to focus 0.5.x’s first
> > releases
> > > on not only ES but
> > > getting a handle on kibana and the mpack situation as well.
> > >
> > >
> > >
> > >
> > > On November 6, 2017 at 12:48:45, Michael Miklavcic (
> > > michael.miklavcic@gmail.com) wrote:
> > >
> > > I agree with your proposal, Nick. I think having a stabilizing release
> > > prior to upgrading ES/Kibana makes sense.
> > >
> > > On Mon, Nov 6, 2017 at 9:16 AM, Nick Allen <ni...@nickallen.org> wrote:
> > >
> > > > I would like to start a discussion around upcoming releases. We have
> a
> > > > couple separate significant tracks of work that we need to reconcile
> in
> > > our
> > > > release schedule.
> > > >
> > > > (1) We have had (and have in review) a good number of bug fixes
> > required
> > > to
> > > > support Metaalerts on the existing Elasticsearch 2.x infrastructure.
> > > >
> > > >
> > > > (2) We also have ongoing work to upgrade our infrastructure to
> > > > Elasticsearch 5.x, which will not be backwards compatible.
> > > >
> > > >
> > > > I would like to see a release that has our best work on ES 2.x before
> > we
> > > > migrate to 5.x. I would propose the following.
> > > >
> > > > Release N+1: Introduce Metaalerts running on ES 2.x
> > > >
> > > > Release N+2: Cut-over to ES 5.x
> > > >
> > > >
> > > > (Q) Is it worth cutting a separate release for ES 2.x? Is there a
> > better
> > > > way to handle the cut-over to 5.x?
> > > >
> > >
> > --
> >
> > Jon
> >
>
Re: [DISCUSS] Upcoming Release
Posted by Casey Stella <ce...@gmail.com>.
I'd say that if a release is this imminent that we had better notify the
release manager who will make a release announcement, Nick. Matt, are you
tuning in to this?
On Wed, Nov 15, 2017 at 10:04 AM, Nick Allen <ni...@nickallen.org> wrote:
> Hi Guys -
>
> I want to follow-up on this discussion. It sounds like most people are in
> agreement with the general approach.
>
> A lot of people have been working hard on Metaalerts and Elasticsearch. I
> have checked-in with those doing the heavy lifting and have compiled a more
> detailed plan based on where we are at now. To the best of my knowledge
> here is the plan of attack for finishing out this effort.
>
> (1) First, METRON-1289 needs to go in. This one was a fairly big effort
> and I am hearing that we are pretty close.
>
> (2) METRON-1294 fixes an issue in how field types are looked-up.
>
> (3) METRON-1290 is next. While this may have been fixed in M-1289, there
> may be some test cases we want from this PR.
>
> (4) METRON-1301 addresses a problem with the sorting logic.
>
> (5) METRON-1291 fixes an issue with escalation of metaalerts.
>
> (6) That leads us to Raghu's UI work in METRON-1252. This introduces the
> UI bits that depend on all the previous backend work.
>
> (7) At this point, we should have our best effort at running Metaalerts
> on Elasticsearch 2.x. I propose that we cut a release here.
>
> (8) After we cut the release, we can introduce the work for ES 5.x in
> METRON-939. I know we will need lots of help testing and reviewing this
> one.
>
> Please correct me if I am wrong. I will try and send out updates as we
> make progress.
>
>
>
>
>
> On Mon, Nov 6, 2017 at 1:03 PM, Zeolla@GMail.com <ze...@gmail.com> wrote:
>
> > I agree, I think it's very reasonable to move in line with Nick's
> > proposal. I would also suggest that we outline what the target versions
> > would be to add in the METRON-777 components, since it has been
> functional
> > for a very long time but not reviewed and has some really rockstar
> > improvements.
> >
> > Jon
> >
> > On Mon, Nov 6, 2017 at 12:56 PM Otto Fowler <ot...@gmail.com>
> > wrote:
> >
> > > I think the ES cutover should be the start of the 0.5.x series, and we
> > > continue on with 0.4.x for the
> > > metadata improvements etc. We could chose to focus 0.5.x’s first
> > releases
> > > on not only ES but
> > > getting a handle on kibana and the mpack situation as well.
> > >
> > >
> > >
> > >
> > > On November 6, 2017 at 12:48:45, Michael Miklavcic (
> > > michael.miklavcic@gmail.com) wrote:
> > >
> > > I agree with your proposal, Nick. I think having a stabilizing release
> > > prior to upgrading ES/Kibana makes sense.
> > >
> > > On Mon, Nov 6, 2017 at 9:16 AM, Nick Allen <ni...@nickallen.org> wrote:
> > >
> > > > I would like to start a discussion around upcoming releases. We have
> a
> > > > couple separate significant tracks of work that we need to reconcile
> in
> > > our
> > > > release schedule.
> > > >
> > > > (1) We have had (and have in review) a good number of bug fixes
> > required
> > > to
> > > > support Metaalerts on the existing Elasticsearch 2.x infrastructure.
> > > >
> > > >
> > > > (2) We also have ongoing work to upgrade our infrastructure to
> > > > Elasticsearch 5.x, which will not be backwards compatible.
> > > >
> > > >
> > > > I would like to see a release that has our best work on ES 2.x before
> > we
> > > > migrate to 5.x. I would propose the following.
> > > >
> > > > Release N+1: Introduce Metaalerts running on ES 2.x
> > > >
> > > > Release N+2: Cut-over to ES 5.x
> > > >
> > > >
> > > > (Q) Is it worth cutting a separate release for ES 2.x? Is there a
> > better
> > > > way to handle the cut-over to 5.x?
> > > >
> > >
> > --
> >
> > Jon
> >
>
Re: [DISCUSS] Upcoming Release
Posted by Nick Allen <ni...@nickallen.org>.
Hi Guys -
I want to follow-up on this discussion. It sounds like most people are in
agreement with the general approach.
A lot of people have been working hard on Metaalerts and Elasticsearch. I
have checked-in with those doing the heavy lifting and have compiled a more
detailed plan based on where we are at now. To the best of my knowledge
here is the plan of attack for finishing out this effort.
(1) First, METRON-1289 needs to go in. This one was a fairly big effort
and I am hearing that we are pretty close.
(2) METRON-1294 fixes an issue in how field types are looked-up.
(3) METRON-1290 is next. While this may have been fixed in M-1289, there
may be some test cases we want from this PR.
(4) METRON-1301 addresses a problem with the sorting logic.
(5) METRON-1291 fixes an issue with escalation of metaalerts.
(6) That leads us to Raghu's UI work in METRON-1252. This introduces the
UI bits that depend on all the previous backend work.
(7) At this point, we should have our best effort at running Metaalerts
on Elasticsearch 2.x. I propose that we cut a release here.
(8) After we cut the release, we can introduce the work for ES 5.x in
METRON-939. I know we will need lots of help testing and reviewing this
one.
Please correct me if I am wrong. I will try and send out updates as we
make progress.
On Mon, Nov 6, 2017 at 1:03 PM, Zeolla@GMail.com <ze...@gmail.com> wrote:
> I agree, I think it's very reasonable to move in line with Nick's
> proposal. I would also suggest that we outline what the target versions
> would be to add in the METRON-777 components, since it has been functional
> for a very long time but not reviewed and has some really rockstar
> improvements.
>
> Jon
>
> On Mon, Nov 6, 2017 at 12:56 PM Otto Fowler <ot...@gmail.com>
> wrote:
>
> > I think the ES cutover should be the start of the 0.5.x series, and we
> > continue on with 0.4.x for the
> > metadata improvements etc. We could chose to focus 0.5.x’s first
> releases
> > on not only ES but
> > getting a handle on kibana and the mpack situation as well.
> >
> >
> >
> >
> > On November 6, 2017 at 12:48:45, Michael Miklavcic (
> > michael.miklavcic@gmail.com) wrote:
> >
> > I agree with your proposal, Nick. I think having a stabilizing release
> > prior to upgrading ES/Kibana makes sense.
> >
> > On Mon, Nov 6, 2017 at 9:16 AM, Nick Allen <ni...@nickallen.org> wrote:
> >
> > > I would like to start a discussion around upcoming releases. We have a
> > > couple separate significant tracks of work that we need to reconcile in
> > our
> > > release schedule.
> > >
> > > (1) We have had (and have in review) a good number of bug fixes
> required
> > to
> > > support Metaalerts on the existing Elasticsearch 2.x infrastructure.
> > >
> > >
> > > (2) We also have ongoing work to upgrade our infrastructure to
> > > Elasticsearch 5.x, which will not be backwards compatible.
> > >
> > >
> > > I would like to see a release that has our best work on ES 2.x before
> we
> > > migrate to 5.x. I would propose the following.
> > >
> > > Release N+1: Introduce Metaalerts running on ES 2.x
> > >
> > > Release N+2: Cut-over to ES 5.x
> > >
> > >
> > > (Q) Is it worth cutting a separate release for ES 2.x? Is there a
> better
> > > way to handle the cut-over to 5.x?
> > >
> >
> --
>
> Jon
>
Re: [DISCUSS] Upcoming Release
Posted by "Zeolla@GMail.com" <ze...@gmail.com>.
I agree, I think it's very reasonable to move in line with Nick's
proposal. I would also suggest that we outline what the target versions
would be to add in the METRON-777 components, since it has been functional
for a very long time but not reviewed and has some really rockstar
improvements.
Jon
On Mon, Nov 6, 2017 at 12:56 PM Otto Fowler <ot...@gmail.com> wrote:
> I think the ES cutover should be the start of the 0.5.x series, and we
> continue on with 0.4.x for the
> metadata improvements etc. We could chose to focus 0.5.x’s first releases
> on not only ES but
> getting a handle on kibana and the mpack situation as well.
>
>
>
>
> On November 6, 2017 at 12:48:45, Michael Miklavcic (
> michael.miklavcic@gmail.com) wrote:
>
> I agree with your proposal, Nick. I think having a stabilizing release
> prior to upgrading ES/Kibana makes sense.
>
> On Mon, Nov 6, 2017 at 9:16 AM, Nick Allen <ni...@nickallen.org> wrote:
>
> > I would like to start a discussion around upcoming releases. We have a
> > couple separate significant tracks of work that we need to reconcile in
> our
> > release schedule.
> >
> > (1) We have had (and have in review) a good number of bug fixes required
> to
> > support Metaalerts on the existing Elasticsearch 2.x infrastructure.
> >
> >
> > (2) We also have ongoing work to upgrade our infrastructure to
> > Elasticsearch 5.x, which will not be backwards compatible.
> >
> >
> > I would like to see a release that has our best work on ES 2.x before we
> > migrate to 5.x. I would propose the following.
> >
> > Release N+1: Introduce Metaalerts running on ES 2.x
> >
> > Release N+2: Cut-over to ES 5.x
> >
> >
> > (Q) Is it worth cutting a separate release for ES 2.x? Is there a better
> > way to handle the cut-over to 5.x?
> >
>
--
Jon
Re: [DISCUSS] Upcoming Release
Posted by Otto Fowler <ot...@gmail.com>.
When I speak of the mpack situation, I’m speaking of the discussion around
removing them.
On November 6, 2017 at 15:14:59, Michael Miklavcic (
michael.miklavcic@gmail.com) wrote:
Kibana and MPack are currently being handled as part of the ES upgrade. I'm
recreating the Kibana dashboards right now. Long story short, there was an
issue upgrading from our pickle file, so I'm recreating it from scratch and
will attempt an alternative approach to storing the dashboard templates.
On Mon, Nov 6, 2017 at 10:56 AM, Otto Fowler <ot...@gmail.com>
wrote:
> I think the ES cutover should be the start of the 0.5.x series, and we
> continue on with 0.4.x for the
> metadata improvements etc. We could chose to focus 0.5.x’s first releases
> on not only ES but
> getting a handle on kibana and the mpack situation as well.
>
>
>
>
> On November 6, 2017 at 12:48:45, Michael Miklavcic (
> michael.miklavcic@gmail.com) wrote:
>
> I agree with your proposal, Nick. I think having a stabilizing release
> prior to upgrading ES/Kibana makes sense.
>
> On Mon, Nov 6, 2017 at 9:16 AM, Nick Allen <ni...@nickallen.org> wrote:
>
> > I would like to start a discussion around upcoming releases. We have a
> > couple separate significant tracks of work that we need to reconcile in
> our
> > release schedule.
> >
> > (1) We have had (and have in review) a good number of bug fixes required
> to
> > support Metaalerts on the existing Elasticsearch 2.x infrastructure.
> >
> >
> > (2) We also have ongoing work to upgrade our infrastructure to
> > Elasticsearch 5.x, which will not be backwards compatible.
> >
> >
> > I would like to see a release that has our best work on ES 2.x before we
> > migrate to 5.x. I would propose the following.
> >
> > Release N+1: Introduce Metaalerts running on ES 2.x
> >
> > Release N+2: Cut-over to ES 5.x
> >
> >
> > (Q) Is it worth cutting a separate release for ES 2.x? Is there a better
> > way to handle the cut-over to 5.x?
> >
>
>
Re: [DISCUSS] Upcoming Release
Posted by Michael Miklavcic <mi...@gmail.com>.
Kibana and MPack are currently being handled as part of the ES upgrade. I'm
recreating the Kibana dashboards right now. Long story short, there was an
issue upgrading from our pickle file, so I'm recreating it from scratch and
will attempt an alternative approach to storing the dashboard templates.
On Mon, Nov 6, 2017 at 10:56 AM, Otto Fowler <ot...@gmail.com>
wrote:
> I think the ES cutover should be the start of the 0.5.x series, and we
> continue on with 0.4.x for the
> metadata improvements etc. We could chose to focus 0.5.x’s first releases
> on not only ES but
> getting a handle on kibana and the mpack situation as well.
>
>
>
>
> On November 6, 2017 at 12:48:45, Michael Miklavcic (
> michael.miklavcic@gmail.com) wrote:
>
> I agree with your proposal, Nick. I think having a stabilizing release
> prior to upgrading ES/Kibana makes sense.
>
> On Mon, Nov 6, 2017 at 9:16 AM, Nick Allen <ni...@nickallen.org> wrote:
>
> > I would like to start a discussion around upcoming releases. We have a
> > couple separate significant tracks of work that we need to reconcile in
> our
> > release schedule.
> >
> > (1) We have had (and have in review) a good number of bug fixes required
> to
> > support Metaalerts on the existing Elasticsearch 2.x infrastructure.
> >
> >
> > (2) We also have ongoing work to upgrade our infrastructure to
> > Elasticsearch 5.x, which will not be backwards compatible.
> >
> >
> > I would like to see a release that has our best work on ES 2.x before we
> > migrate to 5.x. I would propose the following.
> >
> > Release N+1: Introduce Metaalerts running on ES 2.x
> >
> > Release N+2: Cut-over to ES 5.x
> >
> >
> > (Q) Is it worth cutting a separate release for ES 2.x? Is there a better
> > way to handle the cut-over to 5.x?
> >
>
>
Re: [DISCUSS] Upcoming Release
Posted by Otto Fowler <ot...@gmail.com>.
I think the ES cutover should be the start of the 0.5.x series, and we
continue on with 0.4.x for the
metadata improvements etc. We could chose to focus 0.5.x’s first releases
on not only ES but
getting a handle on kibana and the mpack situation as well.
On November 6, 2017 at 12:48:45, Michael Miklavcic (
michael.miklavcic@gmail.com) wrote:
I agree with your proposal, Nick. I think having a stabilizing release
prior to upgrading ES/Kibana makes sense.
On Mon, Nov 6, 2017 at 9:16 AM, Nick Allen <ni...@nickallen.org> wrote:
> I would like to start a discussion around upcoming releases. We have a
> couple separate significant tracks of work that we need to reconcile in
our
> release schedule.
>
> (1) We have had (and have in review) a good number of bug fixes required
to
> support Metaalerts on the existing Elasticsearch 2.x infrastructure.
>
>
> (2) We also have ongoing work to upgrade our infrastructure to
> Elasticsearch 5.x, which will not be backwards compatible.
>
>
> I would like to see a release that has our best work on ES 2.x before we
> migrate to 5.x. I would propose the following.
>
> Release N+1: Introduce Metaalerts running on ES 2.x
>
> Release N+2: Cut-over to ES 5.x
>
>
> (Q) Is it worth cutting a separate release for ES 2.x? Is there a better
> way to handle the cut-over to 5.x?
>
Re: [DISCUSS] Upcoming Release
Posted by Michael Miklavcic <mi...@gmail.com>.
I agree with your proposal, Nick. I think having a stabilizing release
prior to upgrading ES/Kibana makes sense.
On Mon, Nov 6, 2017 at 9:16 AM, Nick Allen <ni...@nickallen.org> wrote:
> I would like to start a discussion around upcoming releases. We have a
> couple separate significant tracks of work that we need to reconcile in our
> release schedule.
>
> (1) We have had (and have in review) a good number of bug fixes required to
> support Metaalerts on the existing Elasticsearch 2.x infrastructure.
>
>
> (2) We also have ongoing work to upgrade our infrastructure to
> Elasticsearch 5.x, which will not be backwards compatible.
>
>
> I would like to see a release that has our best work on ES 2.x before we
> migrate to 5.x. I would propose the following.
>
> Release N+1: Introduce Metaalerts running on ES 2.x
>
> Release N+2: Cut-over to ES 5.x
>
>
> (Q) Is it worth cutting a separate release for ES 2.x? Is there a better
> way to handle the cut-over to 5.x?
>