You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@metron.apache.org by Casey Stella <ce...@gmail.com> on 2017/01/26 23:06:39 UTC

[DISCUSS] Next Release (0.3.1) Content

Hello Everyone!

It's been almost 2 months since the last major release and I think it might
be time to do a minor release of 0.3.1. The purpose of this email is
multifold:

   - to take stock of what we have already committed
   - figure out what (if anything) is missing to make a release that we're
   proud of
   - find volunteers for the missing items

For those who get email alerts on the JIRA changes, it should be no
surprise that I have gone through and put the committed items in the 0.3.1
release bucket.  So, let's take stock of what we have.

*What's made it so far into the next release*



   - METRON-283 Migrate Geo Enrichment outside of MySQL (justinleet) closes
   apache/incubator-metron#421
   - METRON-668: Remove the "tickUpdate" profile config and make the "init"
   phase not reset variables closes apache/incubator-metron#420
   - METRON-672: SolrIndexingIntegrationTest fails intermittently closes
   apache/incubator-metron#424
   - METRON-664: Make the index configuration per-writer with
   enabled/disabled closes apache/incubator-metron#419
   - METRON-600: Fix Metron Website (iraghumitra via cestella) closes
   apache/incubator-metron#399
   - METRON-666 Fix javadoc doclint errors closes
   apache/incubator-metron#418
   - METRON-659 Emulate Sensors in Development Environments (nickwallen)
   closes apache/incubator-metron#417
   - METRON-652: Extract indexing config from enrichment config closes
   apache/incubator-metron#415
   - METRON-654 Create RPM Installer for Profiler (nickwallen) closes
   apache/incubator-metron#413
   - METRON-656: Make Stellar 'in' closer to functioning like python closes
   apache/incubator-metron#416
   - METRON-532 Define Profile Period When Calling PROFILE_GET closes
   apache/incubator-metron#414
   - METRON-624: Updated Comparison/Equality Evaluations in Stellar closes
   apache/incubator-metron#404
   - METRON-644 RPM builds only work with Docker for Mac (kylerichardson)
   closes apache/incubator-metron#409
   - METRON-640: Add a Stellar function to compute shannon entropy for
   strings closes apache/incubator-metron#403
   - METRON-637: Add a STATS_BIN function to Stellar. closes
   apache/incubator-metron#401
   - METRON-622 Create a Metron Docker Compose application  (merrimanr)
   closes apache/incubator-metron#393
   - METRON-643: Stellar function documentation needs to be updated closes
   apache/incubator-metron#407
   - METRON-645 Unable to Start Fastcapa Test Environment (nickwallen)
   closes apache/incubator-metron#410
   - METRON-647 Parser unit test failures due to assumed year values
   (justinleet) closes apache/incubator-metron#412
   - METRON-639: The Network Stellar functions need to have better unit
   testing closes apache/incubator-metron#402
   - METRON-631 Broken link on fastcapa README (JonZeolla via nickwallen)
   closes apache/incubator-metron#398
   - METRON-625: Parser Filters cannot be specified from the sensor config
   closes apache/incubator-metron#396
   - METRON-587 Integration tests should use common processor
   implementations where possible (ottobackwards) closes
   apache/incubator-metron#374
   - METRON-616: Added support for float and long literals in Stellar
   closes apache/incubator-metron#392
   - METRON-580: Remove hard-coded Metron version from Ambari MPack code
   (mmiklavc) closes apache/incubator-metron#364
   - METRON-618 Eliminate Javac Warnings in metron-analytics (justinleet)
   closes apache/incubator-metron#391
   - METRON-364: Preserve Type for Arithmetic Expressions in Stellar closes
   apache/incubator-metron#390
   - METRON-585 Pcap Replay Fails to Stop (nickwallen) closes
   apache/incubator-metron#369
   - METRON-586 STELLAR should have FILL_LEFT and FILL_RIGHT functions
   (ottobackwards) closes apache/incubator-metron#370
   - METRON-612 Clean up Error Prone generated warnings (justinleet) closes
   apache/incubator-metron#389
   - METRON-610: OnlineStatisticsProvider serialization is broken at random
   in the REPL closes apache/incubator-metron#388
   - METRON-596 Eliminate Maven warnings from build (justinleet) closes
   apache/incubator-metron#378
   - METRON-604 Mpack installs do not work on clean machines due to missing
   Elastic Curator repo (justinleet) closes apache/incubator-metron#385
   - METRON-607: Enrichment doc improvement and test cleanup (mmiklavc)
   closes apache/incubator-metron#386
   - METRON-606 Profiler Overwriting Previously Written Values (nickwallen)
   closes apache/incubator-metron#387
   - METRON-591: Make the website in compliance with ASF standards closes
   apache/incubator-metron#373
   - METRON-597 Sporadic Failures of Profiler Integration Tests
   (nickwallen) closes apache/incubator-metron#383
   - METRON-595 Elasticsearch Writer only uses One IP Address
   (JonathanRider via dlyle65535) closes apache/incubator-metron#379
   - METRON-593 Enable an automated static analysis tool in the build
   (justinleet) closes apache/incubator-metron#376
   - METRON-592 Change popup to Warning message in Assign Master page for
   Client check (anandsubbu via dlyle65535) closes apache/incubator-metron#375
   - METRON-598 Add Kyle Richardson to committers (kylerichardson) closes
   apache/incubator-metron#382
   - METRON-565: apps/metron/enrichment/indexed directory path does not get
   created for metron cluster deployed via Ambari (mmiklavc) closes
   apache/incubator-metron#365
   - METRON-588 Remove ASF License Headers from directly copied Chef Bento
   Files closes apache/incubator-metron#372
   - METRON-576 Stellar function resolution takes too long on running
   cluster (nickwallen) closes apache/incubator-metron#366
   - METRON-570 Add Profiler Link to README (nickwallen) closes
   apache/incubator-metron#371
   - METRON-578 Missing error handling bolts for enrichment and threat
   intel closes apache/incubator-metron#363
   - METRON-522 Need to mandate Client installation on Metron Host closes
   apache/incubator-metron#367
   - METRON-584 STELLAR should have a TO_LONG Function (ottobackwards)
   closes apache/incubator-metron#368
   - METRON-575 State from different profiles can be co-mingled incorrectly
   (nickwallen) closes apache/incubator-metron#362
   - METRON-538 Ensure proper shutdown, exceptions of integration test
   components (ottobackwards) closes apache/incubator-metron#344
   - METRON-567 Usernames as numerics strings attempted to be parsed and
   compared as numbers (ottobackwards) closes apache/incubator-metron#360
   - METRON-562: Add rudimentary statistical outlier detection closes
   apache/incubator-metron#352
   - METRON-557 Create Stellar Functions for Kafka (nickwallen) closes
   apache/incubator-metron#354
   - METRON-556 Profiler - Refactor 'Group By' Calculation (nickwallen)
   closes apache/incubator-metron#348
   - METRON-547 MaasIntegrationTests should integrate with the Metron
   Integration classes (ottobackwards) closes apache/incubator-metron#350
   - METRON-561 ShellEditor tests hang if vim is set as EDITOR in your
   profile (ottobackwards) closes apache/incubator-metron#351

*What else will make it?*

Ok, so now that we know what's in there already, I need to ask your help in
determining what you want in the next release that is not currently being
worked on.  In order to do that, please help me by responding to this email
and

   - proposing and justifying a JIRA from the list of unresolved JIRAs
   <https://issues.apache.org/jira/browse/METRON-469?jql=project%20%3D%20METRON%20AND%20resolution%20%3D%20Unresolved%20AND%20fixVersion%20is%20EMPTY%20ORDER%20BY%20priority%20DESC>
that
   are not currently scheduled and let me know which ones should be included.
   Please include whether the JIRA is "Nice to Have" or "Critical to Have" and
   if you volunteer to do it.
   - proposing and justifying that one of the in-progress tasks
   <https://issues.apache.org/jira/browse/METRON-650?jql=project%20%3D%20METRON%20AND%20resolution%20%3D%20Unresolved%20AND%20fixVersion%20%3D%200.3.1%20ORDER%20BY%20priority%20DESC>
    should *not* be included for release

I'd like to aim to get release candidates out as soon as possible, so let's
look forward a vigorous discussion and a great release.

Thanks for the contributions and for your continued engagement in the
Metron community.
If you see something that you don't like in this process, please let me
know.  Feedback is very welcome.

Best,

Casey

Re: [DISCUSS] Next Release (0.3.1) Content

Posted by Casey Stella <ce...@gmail.com>.
Thanks Mike,

I saw a +1 come in for 660 this morning and I'll give the weekend for
people to comment on 692 and add anything that is a breaking change in the
comments of that PR.  I'd like the first release candidate to go out monday
or tuesday for a vote.

On Fri, Feb 3, 2017 at 9:23 AM, Michael Miklavcic <
michael.miklavcic@gmail.com> wrote:

> Hi Casey,
>
> My understanding from legal and our mentors is that as it currently stands,
> we would not have to wait.
>
> http://mail-archives.apache.org/mod_mbox/incubator-metron-
> dev/201701.mbox/%3CCAF1jEfCGVGd9tVDJfRwoxFjEmwHD59DuXTdMu-_OJQwcLvryog%
> 40mail.gmail.com%3E
>
> and
>
> http://mail-archives.apache.org/mod_mbox/www-legal-discuss/201702.mbox/%
> 3CCAOqetn-kB_ekCxB4z%2Bb0J%3DG0qJJKFepkfmMGa8OvBhxmPcvjvw
> %40mail.gmail.com%3E
>
>
> On Thu, Feb 2, 2017 at 3:03 PM, Casey Stella <ce...@gmail.com> wrote:
>
> > Just a quick release update.  As of now, we are waiting on
> >
> >    - METRON-660 to get reviewed and make it in
> >    - METRON-692 to get our upgrade.md completed for this release
> >
> > Mike Miklavcic, you sent an email to the legal-discuss about our kraken
> > dependency and it looked like we didn't have to change, but could you
> > comment on this thread in the dev list so I know if we need to wait for
> > METRON-650.
> >
> > Casey
> >
> > On Thu, Feb 2, 2017 at 4:58 PM, Casey Stella <ce...@gmail.com> wrote:
> >
> > > Ok, I've created the upgrading document for 0.3.0 to 0.3.1 and included
> > > the things that I know about and the things Jon mentioned here.
> Please,
> > if
> > > you have knowledge of other breaking/non-compatible changes between the
> > > 0.3.0 release and master, comment on this PR (
> https://github.com/apache/
> > > incubator-metron/pull/437) and I will incorporate them.
> > >
> > > On Fri, Jan 27, 2017 at 10:04 AM, Zeolla@GMail.com <ze...@gmail.com>
> > > wrote:
> > >
> > >> To start I was mostly concerned with having a per-version list of
> > >> non-backwards-compatible changes, so upgrades that may skip a version
> of
> > >> two can look at what all may be impacted.  We should also probably
> > >> document
> > >> any sort of upgrade flaws as well, such as METRON-447
> > >> <https://issues.apache.org/jira/browse/METRON-447>, METRON-448
> > >> <https://issues.apache.org/jira/browse/METRON-448>, etc.  I do think
> > that
> > >> we should have a more rigorous document, but I wouldn't push that for
> > the
> > >> 0.3.1 release.  I see that (along with the Management UI, API, etc.)
> as
> > >> key
> > >> (required) components of a 1.0 release.  I'd just like to see the
> > >> foundation begin to be laid and iterated on.
> > >>
> > >> That said, this probably constitutes a mention in the development
> > >> guidelines
> > >> <https://cwiki.apache.org/confluence/pages/viewpage.action?
> > >> pageId=61332235>
> > >> once
> > >> it's in master.
> > >>
> > >> Jon
> > >>
> > >> On Fri, Jan 27, 2017 at 9:05 AM Casey Stella <ce...@gmail.com>
> > wrote:
> > >>
> > >> > I should add, you may be thinking something more rigorous and
> > >> > step-by-step.  If so, you think you might be interested in
> > volunteering
> > >> to
> > >> > do a first draft as a PR that we can adjust?
> > >> >
> > >> > On Fri, Jan 27, 2017 at 9:01 AM, Casey Stella <ce...@gmail.com>
> > >> wrote:
> > >> >
> > >> > > So, I agree with the Upgrading.md and I was going to submit a PR
> at
> > >> least
> > >> > > to describe the the changes to indexing configurations that I made
> > >> during
> > >> > > the 3.0.1 release.
> > >> > >
> > >> > >
> > >> > > On Thu, Jan 26, 2017 at 10:50 PM, Zeolla@GMail.com <
> > zeolla@gmail.com>
> > >> > > wrote:
> > >> > >
> > >> > >> I haven't had a chance to look through the unresolved JIRAs but I
> > did
> > >> > want
> > >> > >> to mention a few quick things.
> > >> > >>
> > >> > >> First, when we released 0.3.0 and dropped the BETA flag, one of
> the
> > >> > things
> > >> > >> that was discussed was putting together a method of documenting
> > >> upgrades
> > >> > >> from one version to the next.  As one of the first steps toward
> > >> making
> > >> > >> that
> > >> > >> a reasonable process, I think we should assemble more detailed
> > >> release
> > >> > >> notes, especially outlining non-backwards compatible changes.  In
> > the
> > >> > >> "[DISCUSS] Next Release Name" email thread Kyle Richardson
> > suggested
> > >> we
> > >> > >> use
> > >> > >> "UPGRADING.md" to do this, and I still agree with that thought,
> but
> > >> I'm
> > >> > >> open to alternatives.
> > >> > >>
> > >> > >> Separately, a *nice to have* would be *METRON-660*, which was
> > >> discussed
> > >> > in
> > >> > >> the "[PROPOSAL] up-to-date versioned documentation" thread, to
> give
> > >> us
> > >> > >> some
> > >> > >> cleaner documentation using the existing READMEs.  I'd be happy
> to
> > >> help
> > >> > >> with this one, I'm just not sure what the next steps are, aside
> > from
> > >> the
> > >> > >> start that Matt has here
> > >> > >> <https://github.com/mattf-horton/incubator-metron/tree/
> METRON-660
> > >.
> > >> > >>
> > >> > >> My last *nice to have* is *METRON-635*, which I have a *PR open
> for
> > >> here
> > >> > >> <https://github.com/apache/incubator-metron/pull/411>*.  If I
> > could
> > >> get
> > >> > >> someone else to reproduce the error that I'm seeing I would be
> > happy
> > >> to
> > >> > >> pursue additional testing, troubleshooting, etc.  I've seen
> others
> > >> > report
> > >> > >> the same issue
> > >> > >> <https://community.hortonworks.com/search.html?f=&type=
> > >> > >> question&redirect=search%2Fsearch&sort=relevance&q=scp_if_ssh>
> > >> > >> on the HCC boards, so I'm fairly confident that it is not an
> issue
> > >> with
> > >> > my
> > >> > >> local environment.
> > >> > >>
> > >> > >> Jon
> > >> > >>
> > >> > >> On Thu, Jan 26, 2017 at 6:18 PM Casey Stella <cestella@gmail.com
> >
> > >> > wrote:
> > >> > >>
> > >> > >> I took the liberty of adding the pull requests since the new year
> > >> into
> > >> > the
> > >> > >> in-progress list in the previous email.  If you own one of these
> > and
> > >> do
> > >> > >> not
> > >> > >> believe that you can complete the review in the next week or so,
> > let
> > >> me
> > >> > >> know and we can remove.  The only one of these that I see as
> > >> mandatory
> > >> > is
> > >> > >> METRON-650 because the reliance on the opensoc github repo might
> > >> cause
> > >> > >> issues with the release being acceptable (see discussion by Mike
> > >> > Miklavcic
> > >> > >> with the mentors a couple weeks ago).
> > >> > >>
> > >> > >> On Thu, Jan 26, 2017 at 6:06 PM, Casey Stella <
> cestella@gmail.com>
> > >> > wrote:
> > >> > >>
> > >> > >> > Hello Everyone!
> > >> > >> >
> > >> > >> > It's been almost 2 months since the last major release and I
> > think
> > >> it
> > >> > >> > might be time to do a minor release of 0.3.1. The purpose of
> this
> > >> > email
> > >> > >> is
> > >> > >> > multifold:
> > >> > >> >
> > >> > >> >    - to take stock of what we have already committed
> > >> > >> >    - figure out what (if anything) is missing to make a release
> > >> that
> > >> > >> >    we're proud of
> > >> > >> >    - find volunteers for the missing items
> > >> > >> >
> > >> > >> > For those who get email alerts on the JIRA changes, it should
> be
> > no
> > >> > >> > surprise that I have gone through and put the committed items
> in
> > >> the
> > >> > >> 0.3.1
> > >> > >> > release bucket.  So, let's take stock of what we have.
> > >> > >> >
> > >> > >> > *What's made it so far into the next release*
> > >> > >> >
> > >> > >> >
> > >> > >> >
> > >> > >> >    - METRON-283 Migrate Geo Enrichment outside of MySQL
> > >> (justinleet)
> > >> > >> >    closes apache/incubator-metron#421
> > >> > >> >    - METRON-668: Remove the "tickUpdate" profile config and
> make
> > >> the
> > >> > >> >    "init" phase not reset variables closes
> > >> apache/incubator-metron#420
> > >> > >> >    - METRON-672: SolrIndexingIntegrationTest fails
> intermittently
> > >> > closes
> > >> > >> >    apache/incubator-metron#424
> > >> > >> >    - METRON-664: Make the index configuration per-writer with
> > >> > >> >    enabled/disabled closes apache/incubator-metron#419
> > >> > >> >    - METRON-600: Fix Metron Website (iraghumitra via cestella)
> > >> closes
> > >> > >> >    apache/incubator-metron#399
> > >> > >> >    - METRON-666 Fix javadoc doclint errors closes
> > >> > >> >    apache/incubator-metron#418
> > >> > >> >    - METRON-659 Emulate Sensors in Development Environments
> > >> > (nickwallen)
> > >> > >> >    closes apache/incubator-metron#417
> > >> > >> >    - METRON-652: Extract indexing config from enrichment config
> > >> closes
> > >> > >> >    apache/incubator-metron#415
> > >> > >> >    - METRON-654 Create RPM Installer for Profiler (nickwallen)
> > >> closes
> > >> > >> >    apache/incubator-metron#413
> > >> > >> >    - METRON-656: Make Stellar 'in' closer to functioning like
> > >> python
> > >> > >> >    closes apache/incubator-metron#416
> > >> > >> >    - METRON-532 Define Profile Period When Calling PROFILE_GET
> > >> closes
> > >> > >> >    apache/incubator-metron#414
> > >> > >> >    - METRON-624: Updated Comparison/Equality Evaluations in
> > Stellar
> > >> > >> >    closes apache/incubator-metron#404
> > >> > >> >    - METRON-644 RPM builds only work with Docker for Mac
> > >> > >> (kylerichardson)
> > >> > >> >    closes apache/incubator-metron#409
> > >> > >> >    - METRON-640: Add a Stellar function to compute shannon
> > entropy
> > >> for
> > >> > >> >    strings closes apache/incubator-metron#403
> > >> > >> >    - METRON-637: Add a STATS_BIN function to Stellar. closes
> > >> > >> >    apache/incubator-metron#401
> > >> > >> >    - METRON-622 Create a Metron Docker Compose application
> > >> > (merrimanr)
> > >> > >> >    closes apache/incubator-metron#393
> > >> > >> >    - METRON-643: Stellar function documentation needs to be
> > updated
> > >> > >> >    closes apache/incubator-metron#407
> > >> > >> >    - METRON-645 Unable to Start Fastcapa Test Environment
> > >> (nickwallen)
> > >> > >> >    closes apache/incubator-metron#410
> > >> > >> >    - METRON-647 Parser unit test failures due to assumed year
> > >> values
> > >> > >> >    (justinleet) closes apache/incubator-metron#412
> > >> > >> >    - METRON-639: The Network Stellar functions need to have
> > better
> > >> > unit
> > >> > >> >    testing closes apache/incubator-metron#402
> > >> > >> >    - METRON-631 Broken link on fastcapa README (JonZeolla via
> > >> > >> nickwallen)
> > >> > >> >    closes apache/incubator-metron#398
> > >> > >> >    - METRON-625: Parser Filters cannot be specified from the
> > sensor
> > >> > >> >    config closes apache/incubator-metron#396
> > >> > >> >    - METRON-587 Integration tests should use common processor
> > >> > >> >    implementations where possible (ottobackwards) closes
> > >> > >> >    apache/incubator-metron#374
> > >> > >> >    - METRON-616: Added support for float and long literals in
> > >> Stellar
> > >> > >> >    closes apache/incubator-metron#392
> > >> > >> >    - METRON-580: Remove hard-coded Metron version from Ambari
> > MPack
> > >> > code
> > >> > >> >    (mmiklavc) closes apache/incubator-metron#364
> > >> > >> >    - METRON-618 Eliminate Javac Warnings in metron-analytics
> > >> > >> (justinleet)
> > >> > >> >    closes apache/incubator-metron#391
> > >> > >> >    - METRON-364: Preserve Type for Arithmetic Expressions in
> > >> Stellar
> > >> > >> >    closes apache/incubator-metron#390
> > >> > >> >    - METRON-585 Pcap Replay Fails to Stop (nickwallen) closes
> > >> > >> >    apache/incubator-metron#369
> > >> > >> >    - METRON-586 STELLAR should have FILL_LEFT and FILL_RIGHT
> > >> functions
> > >> > >> >    (ottobackwards) closes apache/incubator-metron#370
> > >> > >> >    - METRON-612 Clean up Error Prone generated warnings
> > >> (justinleet)
> > >> > >> >    closes apache/incubator-metron#389
> > >> > >> >    - METRON-610: OnlineStatisticsProvider serialization is
> broken
> > >> at
> > >> > >> >    random in the REPL closes apache/incubator-metron#388
> > >> > >> >    - METRON-596 Eliminate Maven warnings from build
> (justinleet)
> > >> > closes
> > >> > >> >    apache/incubator-metron#378
> > >> > >> >    - METRON-604 Mpack installs do not work on clean machines
> due
> > to
> > >> > >> >    missing Elastic Curator repo (justinleet) closes
> > >> > >> apache/incubator-metron#385
> > >> > >> >    - METRON-607: Enrichment doc improvement and test cleanup
> > >> > (mmiklavc)
> > >> > >> >    closes apache/incubator-metron#386
> > >> > >> >    - METRON-606 Profiler Overwriting Previously Written Values
> > >> > >> >    (nickwallen) closes apache/incubator-metron#387
> > >> > >> >    - METRON-591: Make the website in compliance with ASF
> > standards
> > >> > >> closes
> > >> > >> >    apache/incubator-metron#373
> > >> > >> >    - METRON-597 Sporadic Failures of Profiler Integration Tests
> > >> > >> >    (nickwallen) closes apache/incubator-metron#383
> > >> > >> >    - METRON-595 Elasticsearch Writer only uses One IP Address
> > >> > >> >    (JonathanRider via dlyle65535) closes
> > >> apache/incubator-metron#379
> > >> > >> >    - METRON-593 Enable an automated static analysis tool in the
> > >> build
> > >> > >> >    (justinleet) closes apache/incubator-metron#376
> > >> > >> >    - METRON-592 Change popup to Warning message in Assign
> Master
> > >> page
> > >> > >> for
> > >> > >> >    Client check (anandsubbu via dlyle65535) closes
> > >> > >> apache/incubator-metron#375
> > >> > >> >    - METRON-598 Add Kyle Richardson to committers
> > (kylerichardson)
> > >> > >> closes
> > >> > >> >    apache/incubator-metron#382
> > >> > >> >    - METRON-565: apps/metron/enrichment/indexed directory path
> > does
> > >> > not
> > >> > >> >    get created for metron cluster deployed via Ambari
> (mmiklavc)
> > >> > closes
> > >> > >> >    apache/incubator-metron#365
> > >> > >> >    - METRON-588 Remove ASF License Headers from directly copied
> > >> Chef
> > >> > >> >    Bento Files closes apache/incubator-metron#372
> > >> > >> >    - METRON-576 Stellar function resolution takes too long on
> > >> running
> > >> > >> >    cluster (nickwallen) closes apache/incubator-metron#366
> > >> > >> >    - METRON-570 Add Profiler Link to README (nickwallen) closes
> > >> > >> >    apache/incubator-metron#371
> > >> > >> >    - METRON-578 Missing error handling bolts for enrichment and
> > >> threat
> > >> > >> >    intel closes apache/incubator-metron#363
> > >> > >> >    - METRON-522 Need to mandate Client installation on Metron
> > Host
> > >> > >> closes
> > >> > >> >    apache/incubator-metron#367
> > >> > >> >    - METRON-584 STELLAR should have a TO_LONG Function
> > >> (ottobackwards)
> > >> > >> >    closes apache/incubator-metron#368
> > >> > >> >    - METRON-575 State from different profiles can be co-mingled
> > >> > >> >    incorrectly (nickwallen) closes apache/incubator-metron#362
> > >> > >> >    - METRON-538 Ensure proper shutdown, exceptions of
> integration
> > >> test
> > >> > >> >    components (ottobackwards) closes
> apache/incubator-metron#344
> > >> > >> >    - METRON-567 Usernames as numerics strings attempted to be
> > >> parsed
> > >> > and
> > >> > >> >    compared as numbers (ottobackwards) closes
> > >> > >> apache/incubator-metron#360
> > >> > >> >    - METRON-562: Add rudimentary statistical outlier detection
> > >> closes
> > >> > >> >    apache/incubator-metron#352
> > >> > >> >    - METRON-557 Create Stellar Functions for Kafka (nickwallen)
> > >> closes
> > >> > >> >    apache/incubator-metron#354
> > >> > >> >    - METRON-556 Profiler - Refactor 'Group By' Calculation
> > >> > (nickwallen)
> > >> > >> >    closes apache/incubator-metron#348
> > >> > >> >    - METRON-547 MaasIntegrationTests should integrate with the
> > >> Metron
> > >> > >> >    Integration classes (ottobackwards) closes
> > >> > >> apache/incubator-metron#350
> > >> > >> >    - METRON-561 ShellEditor tests hang if vim is set as EDITOR
> in
> > >> your
> > >> > >> >    profile (ottobackwards) closes apache/incubator-metron#351
> > >> > >> >
> > >> > >> > *What else will make it?*
> > >> > >> >
> > >> > >> > Ok, so now that we know what's in there already, I need to ask
> > your
> > >> > help
> > >> > >> > in determining what you want in the next release that is not
> > >> currently
> > >> > >> > being worked on.  In order to do that, please help me by
> > >> responding to
> > >> > >> this
> > >> > >> > email and
> > >> > >> >
> > >> > >> >    - proposing and justifying a JIRA from the list of
> unresolved
> > >> JIRAs
> > >> > >> >    <
> > >> > >> https://issues.apache.org/jira/browse/METRON-469?jql=project
> > >> > >> %20%3D%20METRON%20AND%20resolution%20%3D%20Unresolved%20AND%
> > >> > >> 20fixVersion%20is%20EMPTY%20ORDER%20BY%20priority%20DESC>
> > >> > >> that
> > >> > >> >    are not currently scheduled and let me know which ones
> should
> > be
> > >> > >> included.
> > >> > >> >    Please include whether the JIRA is "Nice to Have" or
> "Critical
> > >> to
> > >> > >> Have" and
> > >> > >> >    if you volunteer to do it.
> > >> > >> >    - proposing and justifying that one of the in-progress tasks
> > >> > >> >    <
> > >> > >> https://issues.apache.org/jira/browse/METRON-650?jql=project
> > >> > >> %20%3D%20METRON%20AND%20resolution%20%3D%20Unresolved%20AND%
> > >> > >> 20fixVersion%20%3D%200.3.1%20ORDER%20BY%20priority%20DESC
> > >> > >> >
> > >> > >> >     should *not* be included for release
> > >> > >> >
> > >> > >> > I'd like to aim to get release candidates out as soon as
> > possible,
> > >> so
> > >> > >> > let's look forward a vigorous discussion and a great release.
> > >> > >> >
> > >> > >> > Thanks for the contributions and for your continued engagement
> in
> > >> the
> > >> > >> > Metron community.  If you see something that you don't like in
> > this
> > >> > >> > process, please let me know.  Feedback is very welcome.
> > >> > >> >
> > >> > >> > Best,
> > >> > >> >
> > >> > >> > Casey
> > >> > >> >
> > >> > >>
> > >> > >> --
> > >> > >>
> > >> > >> Jon
> > >> > >>
> > >> > >> Sent from my mobile device
> > >> > >>
> > >> > >
> > >> > >
> > >> >
> > >> --
> > >>
> > >> Jon
> > >>
> > >> Sent from my mobile device
> > >>
> > >
> > >
> >
>

Re: [DISCUSS] Next Release (0.3.1) Content

Posted by Michael Miklavcic <mi...@gmail.com>.
Hi Casey,

My understanding from legal and our mentors is that as it currently stands,
we would not have to wait.

http://mail-archives.apache.org/mod_mbox/incubator-metron-dev/201701.mbox/%3CCAF1jEfCGVGd9tVDJfRwoxFjEmwHD59DuXTdMu-_OJQwcLvryog%40mail.gmail.com%3E

and

http://mail-archives.apache.org/mod_mbox/www-legal-discuss/201702.mbox/%3CCAOqetn-kB_ekCxB4z%2Bb0J%3DG0qJJKFepkfmMGa8OvBhxmPcvjvw%40mail.gmail.com%3E


On Thu, Feb 2, 2017 at 3:03 PM, Casey Stella <ce...@gmail.com> wrote:

> Just a quick release update.  As of now, we are waiting on
>
>    - METRON-660 to get reviewed and make it in
>    - METRON-692 to get our upgrade.md completed for this release
>
> Mike Miklavcic, you sent an email to the legal-discuss about our kraken
> dependency and it looked like we didn't have to change, but could you
> comment on this thread in the dev list so I know if we need to wait for
> METRON-650.
>
> Casey
>
> On Thu, Feb 2, 2017 at 4:58 PM, Casey Stella <ce...@gmail.com> wrote:
>
> > Ok, I've created the upgrading document for 0.3.0 to 0.3.1 and included
> > the things that I know about and the things Jon mentioned here.  Please,
> if
> > you have knowledge of other breaking/non-compatible changes between the
> > 0.3.0 release and master, comment on this PR (https://github.com/apache/
> > incubator-metron/pull/437) and I will incorporate them.
> >
> > On Fri, Jan 27, 2017 at 10:04 AM, Zeolla@GMail.com <ze...@gmail.com>
> > wrote:
> >
> >> To start I was mostly concerned with having a per-version list of
> >> non-backwards-compatible changes, so upgrades that may skip a version of
> >> two can look at what all may be impacted.  We should also probably
> >> document
> >> any sort of upgrade flaws as well, such as METRON-447
> >> <https://issues.apache.org/jira/browse/METRON-447>, METRON-448
> >> <https://issues.apache.org/jira/browse/METRON-448>, etc.  I do think
> that
> >> we should have a more rigorous document, but I wouldn't push that for
> the
> >> 0.3.1 release.  I see that (along with the Management UI, API, etc.) as
> >> key
> >> (required) components of a 1.0 release.  I'd just like to see the
> >> foundation begin to be laid and iterated on.
> >>
> >> That said, this probably constitutes a mention in the development
> >> guidelines
> >> <https://cwiki.apache.org/confluence/pages/viewpage.action?
> >> pageId=61332235>
> >> once
> >> it's in master.
> >>
> >> Jon
> >>
> >> On Fri, Jan 27, 2017 at 9:05 AM Casey Stella <ce...@gmail.com>
> wrote:
> >>
> >> > I should add, you may be thinking something more rigorous and
> >> > step-by-step.  If so, you think you might be interested in
> volunteering
> >> to
> >> > do a first draft as a PR that we can adjust?
> >> >
> >> > On Fri, Jan 27, 2017 at 9:01 AM, Casey Stella <ce...@gmail.com>
> >> wrote:
> >> >
> >> > > So, I agree with the Upgrading.md and I was going to submit a PR at
> >> least
> >> > > to describe the the changes to indexing configurations that I made
> >> during
> >> > > the 3.0.1 release.
> >> > >
> >> > >
> >> > > On Thu, Jan 26, 2017 at 10:50 PM, Zeolla@GMail.com <
> zeolla@gmail.com>
> >> > > wrote:
> >> > >
> >> > >> I haven't had a chance to look through the unresolved JIRAs but I
> did
> >> > want
> >> > >> to mention a few quick things.
> >> > >>
> >> > >> First, when we released 0.3.0 and dropped the BETA flag, one of the
> >> > things
> >> > >> that was discussed was putting together a method of documenting
> >> upgrades
> >> > >> from one version to the next.  As one of the first steps toward
> >> making
> >> > >> that
> >> > >> a reasonable process, I think we should assemble more detailed
> >> release
> >> > >> notes, especially outlining non-backwards compatible changes.  In
> the
> >> > >> "[DISCUSS] Next Release Name" email thread Kyle Richardson
> suggested
> >> we
> >> > >> use
> >> > >> "UPGRADING.md" to do this, and I still agree with that thought, but
> >> I'm
> >> > >> open to alternatives.
> >> > >>
> >> > >> Separately, a *nice to have* would be *METRON-660*, which was
> >> discussed
> >> > in
> >> > >> the "[PROPOSAL] up-to-date versioned documentation" thread, to give
> >> us
> >> > >> some
> >> > >> cleaner documentation using the existing READMEs.  I'd be happy to
> >> help
> >> > >> with this one, I'm just not sure what the next steps are, aside
> from
> >> the
> >> > >> start that Matt has here
> >> > >> <https://github.com/mattf-horton/incubator-metron/tree/METRON-660
> >.
> >> > >>
> >> > >> My last *nice to have* is *METRON-635*, which I have a *PR open for
> >> here
> >> > >> <https://github.com/apache/incubator-metron/pull/411>*.  If I
> could
> >> get
> >> > >> someone else to reproduce the error that I'm seeing I would be
> happy
> >> to
> >> > >> pursue additional testing, troubleshooting, etc.  I've seen others
> >> > report
> >> > >> the same issue
> >> > >> <https://community.hortonworks.com/search.html?f=&type=
> >> > >> question&redirect=search%2Fsearch&sort=relevance&q=scp_if_ssh>
> >> > >> on the HCC boards, so I'm fairly confident that it is not an issue
> >> with
> >> > my
> >> > >> local environment.
> >> > >>
> >> > >> Jon
> >> > >>
> >> > >> On Thu, Jan 26, 2017 at 6:18 PM Casey Stella <ce...@gmail.com>
> >> > wrote:
> >> > >>
> >> > >> I took the liberty of adding the pull requests since the new year
> >> into
> >> > the
> >> > >> in-progress list in the previous email.  If you own one of these
> and
> >> do
> >> > >> not
> >> > >> believe that you can complete the review in the next week or so,
> let
> >> me
> >> > >> know and we can remove.  The only one of these that I see as
> >> mandatory
> >> > is
> >> > >> METRON-650 because the reliance on the opensoc github repo might
> >> cause
> >> > >> issues with the release being acceptable (see discussion by Mike
> >> > Miklavcic
> >> > >> with the mentors a couple weeks ago).
> >> > >>
> >> > >> On Thu, Jan 26, 2017 at 6:06 PM, Casey Stella <ce...@gmail.com>
> >> > wrote:
> >> > >>
> >> > >> > Hello Everyone!
> >> > >> >
> >> > >> > It's been almost 2 months since the last major release and I
> think
> >> it
> >> > >> > might be time to do a minor release of 0.3.1. The purpose of this
> >> > email
> >> > >> is
> >> > >> > multifold:
> >> > >> >
> >> > >> >    - to take stock of what we have already committed
> >> > >> >    - figure out what (if anything) is missing to make a release
> >> that
> >> > >> >    we're proud of
> >> > >> >    - find volunteers for the missing items
> >> > >> >
> >> > >> > For those who get email alerts on the JIRA changes, it should be
> no
> >> > >> > surprise that I have gone through and put the committed items in
> >> the
> >> > >> 0.3.1
> >> > >> > release bucket.  So, let's take stock of what we have.
> >> > >> >
> >> > >> > *What's made it so far into the next release*
> >> > >> >
> >> > >> >
> >> > >> >
> >> > >> >    - METRON-283 Migrate Geo Enrichment outside of MySQL
> >> (justinleet)
> >> > >> >    closes apache/incubator-metron#421
> >> > >> >    - METRON-668: Remove the "tickUpdate" profile config and make
> >> the
> >> > >> >    "init" phase not reset variables closes
> >> apache/incubator-metron#420
> >> > >> >    - METRON-672: SolrIndexingIntegrationTest fails intermittently
> >> > closes
> >> > >> >    apache/incubator-metron#424
> >> > >> >    - METRON-664: Make the index configuration per-writer with
> >> > >> >    enabled/disabled closes apache/incubator-metron#419
> >> > >> >    - METRON-600: Fix Metron Website (iraghumitra via cestella)
> >> closes
> >> > >> >    apache/incubator-metron#399
> >> > >> >    - METRON-666 Fix javadoc doclint errors closes
> >> > >> >    apache/incubator-metron#418
> >> > >> >    - METRON-659 Emulate Sensors in Development Environments
> >> > (nickwallen)
> >> > >> >    closes apache/incubator-metron#417
> >> > >> >    - METRON-652: Extract indexing config from enrichment config
> >> closes
> >> > >> >    apache/incubator-metron#415
> >> > >> >    - METRON-654 Create RPM Installer for Profiler (nickwallen)
> >> closes
> >> > >> >    apache/incubator-metron#413
> >> > >> >    - METRON-656: Make Stellar 'in' closer to functioning like
> >> python
> >> > >> >    closes apache/incubator-metron#416
> >> > >> >    - METRON-532 Define Profile Period When Calling PROFILE_GET
> >> closes
> >> > >> >    apache/incubator-metron#414
> >> > >> >    - METRON-624: Updated Comparison/Equality Evaluations in
> Stellar
> >> > >> >    closes apache/incubator-metron#404
> >> > >> >    - METRON-644 RPM builds only work with Docker for Mac
> >> > >> (kylerichardson)
> >> > >> >    closes apache/incubator-metron#409
> >> > >> >    - METRON-640: Add a Stellar function to compute shannon
> entropy
> >> for
> >> > >> >    strings closes apache/incubator-metron#403
> >> > >> >    - METRON-637: Add a STATS_BIN function to Stellar. closes
> >> > >> >    apache/incubator-metron#401
> >> > >> >    - METRON-622 Create a Metron Docker Compose application
> >> > (merrimanr)
> >> > >> >    closes apache/incubator-metron#393
> >> > >> >    - METRON-643: Stellar function documentation needs to be
> updated
> >> > >> >    closes apache/incubator-metron#407
> >> > >> >    - METRON-645 Unable to Start Fastcapa Test Environment
> >> (nickwallen)
> >> > >> >    closes apache/incubator-metron#410
> >> > >> >    - METRON-647 Parser unit test failures due to assumed year
> >> values
> >> > >> >    (justinleet) closes apache/incubator-metron#412
> >> > >> >    - METRON-639: The Network Stellar functions need to have
> better
> >> > unit
> >> > >> >    testing closes apache/incubator-metron#402
> >> > >> >    - METRON-631 Broken link on fastcapa README (JonZeolla via
> >> > >> nickwallen)
> >> > >> >    closes apache/incubator-metron#398
> >> > >> >    - METRON-625: Parser Filters cannot be specified from the
> sensor
> >> > >> >    config closes apache/incubator-metron#396
> >> > >> >    - METRON-587 Integration tests should use common processor
> >> > >> >    implementations where possible (ottobackwards) closes
> >> > >> >    apache/incubator-metron#374
> >> > >> >    - METRON-616: Added support for float and long literals in
> >> Stellar
> >> > >> >    closes apache/incubator-metron#392
> >> > >> >    - METRON-580: Remove hard-coded Metron version from Ambari
> MPack
> >> > code
> >> > >> >    (mmiklavc) closes apache/incubator-metron#364
> >> > >> >    - METRON-618 Eliminate Javac Warnings in metron-analytics
> >> > >> (justinleet)
> >> > >> >    closes apache/incubator-metron#391
> >> > >> >    - METRON-364: Preserve Type for Arithmetic Expressions in
> >> Stellar
> >> > >> >    closes apache/incubator-metron#390
> >> > >> >    - METRON-585 Pcap Replay Fails to Stop (nickwallen) closes
> >> > >> >    apache/incubator-metron#369
> >> > >> >    - METRON-586 STELLAR should have FILL_LEFT and FILL_RIGHT
> >> functions
> >> > >> >    (ottobackwards) closes apache/incubator-metron#370
> >> > >> >    - METRON-612 Clean up Error Prone generated warnings
> >> (justinleet)
> >> > >> >    closes apache/incubator-metron#389
> >> > >> >    - METRON-610: OnlineStatisticsProvider serialization is broken
> >> at
> >> > >> >    random in the REPL closes apache/incubator-metron#388
> >> > >> >    - METRON-596 Eliminate Maven warnings from build (justinleet)
> >> > closes
> >> > >> >    apache/incubator-metron#378
> >> > >> >    - METRON-604 Mpack installs do not work on clean machines due
> to
> >> > >> >    missing Elastic Curator repo (justinleet) closes
> >> > >> apache/incubator-metron#385
> >> > >> >    - METRON-607: Enrichment doc improvement and test cleanup
> >> > (mmiklavc)
> >> > >> >    closes apache/incubator-metron#386
> >> > >> >    - METRON-606 Profiler Overwriting Previously Written Values
> >> > >> >    (nickwallen) closes apache/incubator-metron#387
> >> > >> >    - METRON-591: Make the website in compliance with ASF
> standards
> >> > >> closes
> >> > >> >    apache/incubator-metron#373
> >> > >> >    - METRON-597 Sporadic Failures of Profiler Integration Tests
> >> > >> >    (nickwallen) closes apache/incubator-metron#383
> >> > >> >    - METRON-595 Elasticsearch Writer only uses One IP Address
> >> > >> >    (JonathanRider via dlyle65535) closes
> >> apache/incubator-metron#379
> >> > >> >    - METRON-593 Enable an automated static analysis tool in the
> >> build
> >> > >> >    (justinleet) closes apache/incubator-metron#376
> >> > >> >    - METRON-592 Change popup to Warning message in Assign Master
> >> page
> >> > >> for
> >> > >> >    Client check (anandsubbu via dlyle65535) closes
> >> > >> apache/incubator-metron#375
> >> > >> >    - METRON-598 Add Kyle Richardson to committers
> (kylerichardson)
> >> > >> closes
> >> > >> >    apache/incubator-metron#382
> >> > >> >    - METRON-565: apps/metron/enrichment/indexed directory path
> does
> >> > not
> >> > >> >    get created for metron cluster deployed via Ambari (mmiklavc)
> >> > closes
> >> > >> >    apache/incubator-metron#365
> >> > >> >    - METRON-588 Remove ASF License Headers from directly copied
> >> Chef
> >> > >> >    Bento Files closes apache/incubator-metron#372
> >> > >> >    - METRON-576 Stellar function resolution takes too long on
> >> running
> >> > >> >    cluster (nickwallen) closes apache/incubator-metron#366
> >> > >> >    - METRON-570 Add Profiler Link to README (nickwallen) closes
> >> > >> >    apache/incubator-metron#371
> >> > >> >    - METRON-578 Missing error handling bolts for enrichment and
> >> threat
> >> > >> >    intel closes apache/incubator-metron#363
> >> > >> >    - METRON-522 Need to mandate Client installation on Metron
> Host
> >> > >> closes
> >> > >> >    apache/incubator-metron#367
> >> > >> >    - METRON-584 STELLAR should have a TO_LONG Function
> >> (ottobackwards)
> >> > >> >    closes apache/incubator-metron#368
> >> > >> >    - METRON-575 State from different profiles can be co-mingled
> >> > >> >    incorrectly (nickwallen) closes apache/incubator-metron#362
> >> > >> >    - METRON-538 Ensure proper shutdown, exceptions of integration
> >> test
> >> > >> >    components (ottobackwards) closes apache/incubator-metron#344
> >> > >> >    - METRON-567 Usernames as numerics strings attempted to be
> >> parsed
> >> > and
> >> > >> >    compared as numbers (ottobackwards) closes
> >> > >> apache/incubator-metron#360
> >> > >> >    - METRON-562: Add rudimentary statistical outlier detection
> >> closes
> >> > >> >    apache/incubator-metron#352
> >> > >> >    - METRON-557 Create Stellar Functions for Kafka (nickwallen)
> >> closes
> >> > >> >    apache/incubator-metron#354
> >> > >> >    - METRON-556 Profiler - Refactor 'Group By' Calculation
> >> > (nickwallen)
> >> > >> >    closes apache/incubator-metron#348
> >> > >> >    - METRON-547 MaasIntegrationTests should integrate with the
> >> Metron
> >> > >> >    Integration classes (ottobackwards) closes
> >> > >> apache/incubator-metron#350
> >> > >> >    - METRON-561 ShellEditor tests hang if vim is set as EDITOR in
> >> your
> >> > >> >    profile (ottobackwards) closes apache/incubator-metron#351
> >> > >> >
> >> > >> > *What else will make it?*
> >> > >> >
> >> > >> > Ok, so now that we know what's in there already, I need to ask
> your
> >> > help
> >> > >> > in determining what you want in the next release that is not
> >> currently
> >> > >> > being worked on.  In order to do that, please help me by
> >> responding to
> >> > >> this
> >> > >> > email and
> >> > >> >
> >> > >> >    - proposing and justifying a JIRA from the list of unresolved
> >> JIRAs
> >> > >> >    <
> >> > >> https://issues.apache.org/jira/browse/METRON-469?jql=project
> >> > >> %20%3D%20METRON%20AND%20resolution%20%3D%20Unresolved%20AND%
> >> > >> 20fixVersion%20is%20EMPTY%20ORDER%20BY%20priority%20DESC>
> >> > >> that
> >> > >> >    are not currently scheduled and let me know which ones should
> be
> >> > >> included.
> >> > >> >    Please include whether the JIRA is "Nice to Have" or "Critical
> >> to
> >> > >> Have" and
> >> > >> >    if you volunteer to do it.
> >> > >> >    - proposing and justifying that one of the in-progress tasks
> >> > >> >    <
> >> > >> https://issues.apache.org/jira/browse/METRON-650?jql=project
> >> > >> %20%3D%20METRON%20AND%20resolution%20%3D%20Unresolved%20AND%
> >> > >> 20fixVersion%20%3D%200.3.1%20ORDER%20BY%20priority%20DESC
> >> > >> >
> >> > >> >     should *not* be included for release
> >> > >> >
> >> > >> > I'd like to aim to get release candidates out as soon as
> possible,
> >> so
> >> > >> > let's look forward a vigorous discussion and a great release.
> >> > >> >
> >> > >> > Thanks for the contributions and for your continued engagement in
> >> the
> >> > >> > Metron community.  If you see something that you don't like in
> this
> >> > >> > process, please let me know.  Feedback is very welcome.
> >> > >> >
> >> > >> > Best,
> >> > >> >
> >> > >> > Casey
> >> > >> >
> >> > >>
> >> > >> --
> >> > >>
> >> > >> Jon
> >> > >>
> >> > >> Sent from my mobile device
> >> > >>
> >> > >
> >> > >
> >> >
> >> --
> >>
> >> Jon
> >>
> >> Sent from my mobile device
> >>
> >
> >
>

Re: [DISCUSS] Next Release (0.3.1) Content

Posted by Casey Stella <ce...@gmail.com>.
Just a quick release update.  As of now, we are waiting on

   - METRON-660 to get reviewed and make it in
   - METRON-692 to get our upgrade.md completed for this release

Mike Miklavcic, you sent an email to the legal-discuss about our kraken
dependency and it looked like we didn't have to change, but could you
comment on this thread in the dev list so I know if we need to wait for
METRON-650.

Casey

On Thu, Feb 2, 2017 at 4:58 PM, Casey Stella <ce...@gmail.com> wrote:

> Ok, I've created the upgrading document for 0.3.0 to 0.3.1 and included
> the things that I know about and the things Jon mentioned here.  Please, if
> you have knowledge of other breaking/non-compatible changes between the
> 0.3.0 release and master, comment on this PR (https://github.com/apache/
> incubator-metron/pull/437) and I will incorporate them.
>
> On Fri, Jan 27, 2017 at 10:04 AM, Zeolla@GMail.com <ze...@gmail.com>
> wrote:
>
>> To start I was mostly concerned with having a per-version list of
>> non-backwards-compatible changes, so upgrades that may skip a version of
>> two can look at what all may be impacted.  We should also probably
>> document
>> any sort of upgrade flaws as well, such as METRON-447
>> <https://issues.apache.org/jira/browse/METRON-447>, METRON-448
>> <https://issues.apache.org/jira/browse/METRON-448>, etc.  I do think that
>> we should have a more rigorous document, but I wouldn't push that for the
>> 0.3.1 release.  I see that (along with the Management UI, API, etc.) as
>> key
>> (required) components of a 1.0 release.  I'd just like to see the
>> foundation begin to be laid and iterated on.
>>
>> That said, this probably constitutes a mention in the development
>> guidelines
>> <https://cwiki.apache.org/confluence/pages/viewpage.action?
>> pageId=61332235>
>> once
>> it's in master.
>>
>> Jon
>>
>> On Fri, Jan 27, 2017 at 9:05 AM Casey Stella <ce...@gmail.com> wrote:
>>
>> > I should add, you may be thinking something more rigorous and
>> > step-by-step.  If so, you think you might be interested in volunteering
>> to
>> > do a first draft as a PR that we can adjust?
>> >
>> > On Fri, Jan 27, 2017 at 9:01 AM, Casey Stella <ce...@gmail.com>
>> wrote:
>> >
>> > > So, I agree with the Upgrading.md and I was going to submit a PR at
>> least
>> > > to describe the the changes to indexing configurations that I made
>> during
>> > > the 3.0.1 release.
>> > >
>> > >
>> > > On Thu, Jan 26, 2017 at 10:50 PM, Zeolla@GMail.com <ze...@gmail.com>
>> > > wrote:
>> > >
>> > >> I haven't had a chance to look through the unresolved JIRAs but I did
>> > want
>> > >> to mention a few quick things.
>> > >>
>> > >> First, when we released 0.3.0 and dropped the BETA flag, one of the
>> > things
>> > >> that was discussed was putting together a method of documenting
>> upgrades
>> > >> from one version to the next.  As one of the first steps toward
>> making
>> > >> that
>> > >> a reasonable process, I think we should assemble more detailed
>> release
>> > >> notes, especially outlining non-backwards compatible changes.  In the
>> > >> "[DISCUSS] Next Release Name" email thread Kyle Richardson suggested
>> we
>> > >> use
>> > >> "UPGRADING.md" to do this, and I still agree with that thought, but
>> I'm
>> > >> open to alternatives.
>> > >>
>> > >> Separately, a *nice to have* would be *METRON-660*, which was
>> discussed
>> > in
>> > >> the "[PROPOSAL] up-to-date versioned documentation" thread, to give
>> us
>> > >> some
>> > >> cleaner documentation using the existing READMEs.  I'd be happy to
>> help
>> > >> with this one, I'm just not sure what the next steps are, aside from
>> the
>> > >> start that Matt has here
>> > >> <https://github.com/mattf-horton/incubator-metron/tree/METRON-660>.
>> > >>
>> > >> My last *nice to have* is *METRON-635*, which I have a *PR open for
>> here
>> > >> <https://github.com/apache/incubator-metron/pull/411>*.  If I could
>> get
>> > >> someone else to reproduce the error that I'm seeing I would be happy
>> to
>> > >> pursue additional testing, troubleshooting, etc.  I've seen others
>> > report
>> > >> the same issue
>> > >> <https://community.hortonworks.com/search.html?f=&type=
>> > >> question&redirect=search%2Fsearch&sort=relevance&q=scp_if_ssh>
>> > >> on the HCC boards, so I'm fairly confident that it is not an issue
>> with
>> > my
>> > >> local environment.
>> > >>
>> > >> Jon
>> > >>
>> > >> On Thu, Jan 26, 2017 at 6:18 PM Casey Stella <ce...@gmail.com>
>> > wrote:
>> > >>
>> > >> I took the liberty of adding the pull requests since the new year
>> into
>> > the
>> > >> in-progress list in the previous email.  If you own one of these and
>> do
>> > >> not
>> > >> believe that you can complete the review in the next week or so, let
>> me
>> > >> know and we can remove.  The only one of these that I see as
>> mandatory
>> > is
>> > >> METRON-650 because the reliance on the opensoc github repo might
>> cause
>> > >> issues with the release being acceptable (see discussion by Mike
>> > Miklavcic
>> > >> with the mentors a couple weeks ago).
>> > >>
>> > >> On Thu, Jan 26, 2017 at 6:06 PM, Casey Stella <ce...@gmail.com>
>> > wrote:
>> > >>
>> > >> > Hello Everyone!
>> > >> >
>> > >> > It's been almost 2 months since the last major release and I think
>> it
>> > >> > might be time to do a minor release of 0.3.1. The purpose of this
>> > email
>> > >> is
>> > >> > multifold:
>> > >> >
>> > >> >    - to take stock of what we have already committed
>> > >> >    - figure out what (if anything) is missing to make a release
>> that
>> > >> >    we're proud of
>> > >> >    - find volunteers for the missing items
>> > >> >
>> > >> > For those who get email alerts on the JIRA changes, it should be no
>> > >> > surprise that I have gone through and put the committed items in
>> the
>> > >> 0.3.1
>> > >> > release bucket.  So, let's take stock of what we have.
>> > >> >
>> > >> > *What's made it so far into the next release*
>> > >> >
>> > >> >
>> > >> >
>> > >> >    - METRON-283 Migrate Geo Enrichment outside of MySQL
>> (justinleet)
>> > >> >    closes apache/incubator-metron#421
>> > >> >    - METRON-668: Remove the "tickUpdate" profile config and make
>> the
>> > >> >    "init" phase not reset variables closes
>> apache/incubator-metron#420
>> > >> >    - METRON-672: SolrIndexingIntegrationTest fails intermittently
>> > closes
>> > >> >    apache/incubator-metron#424
>> > >> >    - METRON-664: Make the index configuration per-writer with
>> > >> >    enabled/disabled closes apache/incubator-metron#419
>> > >> >    - METRON-600: Fix Metron Website (iraghumitra via cestella)
>> closes
>> > >> >    apache/incubator-metron#399
>> > >> >    - METRON-666 Fix javadoc doclint errors closes
>> > >> >    apache/incubator-metron#418
>> > >> >    - METRON-659 Emulate Sensors in Development Environments
>> > (nickwallen)
>> > >> >    closes apache/incubator-metron#417
>> > >> >    - METRON-652: Extract indexing config from enrichment config
>> closes
>> > >> >    apache/incubator-metron#415
>> > >> >    - METRON-654 Create RPM Installer for Profiler (nickwallen)
>> closes
>> > >> >    apache/incubator-metron#413
>> > >> >    - METRON-656: Make Stellar 'in' closer to functioning like
>> python
>> > >> >    closes apache/incubator-metron#416
>> > >> >    - METRON-532 Define Profile Period When Calling PROFILE_GET
>> closes
>> > >> >    apache/incubator-metron#414
>> > >> >    - METRON-624: Updated Comparison/Equality Evaluations in Stellar
>> > >> >    closes apache/incubator-metron#404
>> > >> >    - METRON-644 RPM builds only work with Docker for Mac
>> > >> (kylerichardson)
>> > >> >    closes apache/incubator-metron#409
>> > >> >    - METRON-640: Add a Stellar function to compute shannon entropy
>> for
>> > >> >    strings closes apache/incubator-metron#403
>> > >> >    - METRON-637: Add a STATS_BIN function to Stellar. closes
>> > >> >    apache/incubator-metron#401
>> > >> >    - METRON-622 Create a Metron Docker Compose application
>> > (merrimanr)
>> > >> >    closes apache/incubator-metron#393
>> > >> >    - METRON-643: Stellar function documentation needs to be updated
>> > >> >    closes apache/incubator-metron#407
>> > >> >    - METRON-645 Unable to Start Fastcapa Test Environment
>> (nickwallen)
>> > >> >    closes apache/incubator-metron#410
>> > >> >    - METRON-647 Parser unit test failures due to assumed year
>> values
>> > >> >    (justinleet) closes apache/incubator-metron#412
>> > >> >    - METRON-639: The Network Stellar functions need to have better
>> > unit
>> > >> >    testing closes apache/incubator-metron#402
>> > >> >    - METRON-631 Broken link on fastcapa README (JonZeolla via
>> > >> nickwallen)
>> > >> >    closes apache/incubator-metron#398
>> > >> >    - METRON-625: Parser Filters cannot be specified from the sensor
>> > >> >    config closes apache/incubator-metron#396
>> > >> >    - METRON-587 Integration tests should use common processor
>> > >> >    implementations where possible (ottobackwards) closes
>> > >> >    apache/incubator-metron#374
>> > >> >    - METRON-616: Added support for float and long literals in
>> Stellar
>> > >> >    closes apache/incubator-metron#392
>> > >> >    - METRON-580: Remove hard-coded Metron version from Ambari MPack
>> > code
>> > >> >    (mmiklavc) closes apache/incubator-metron#364
>> > >> >    - METRON-618 Eliminate Javac Warnings in metron-analytics
>> > >> (justinleet)
>> > >> >    closes apache/incubator-metron#391
>> > >> >    - METRON-364: Preserve Type for Arithmetic Expressions in
>> Stellar
>> > >> >    closes apache/incubator-metron#390
>> > >> >    - METRON-585 Pcap Replay Fails to Stop (nickwallen) closes
>> > >> >    apache/incubator-metron#369
>> > >> >    - METRON-586 STELLAR should have FILL_LEFT and FILL_RIGHT
>> functions
>> > >> >    (ottobackwards) closes apache/incubator-metron#370
>> > >> >    - METRON-612 Clean up Error Prone generated warnings
>> (justinleet)
>> > >> >    closes apache/incubator-metron#389
>> > >> >    - METRON-610: OnlineStatisticsProvider serialization is broken
>> at
>> > >> >    random in the REPL closes apache/incubator-metron#388
>> > >> >    - METRON-596 Eliminate Maven warnings from build (justinleet)
>> > closes
>> > >> >    apache/incubator-metron#378
>> > >> >    - METRON-604 Mpack installs do not work on clean machines due to
>> > >> >    missing Elastic Curator repo (justinleet) closes
>> > >> apache/incubator-metron#385
>> > >> >    - METRON-607: Enrichment doc improvement and test cleanup
>> > (mmiklavc)
>> > >> >    closes apache/incubator-metron#386
>> > >> >    - METRON-606 Profiler Overwriting Previously Written Values
>> > >> >    (nickwallen) closes apache/incubator-metron#387
>> > >> >    - METRON-591: Make the website in compliance with ASF standards
>> > >> closes
>> > >> >    apache/incubator-metron#373
>> > >> >    - METRON-597 Sporadic Failures of Profiler Integration Tests
>> > >> >    (nickwallen) closes apache/incubator-metron#383
>> > >> >    - METRON-595 Elasticsearch Writer only uses One IP Address
>> > >> >    (JonathanRider via dlyle65535) closes
>> apache/incubator-metron#379
>> > >> >    - METRON-593 Enable an automated static analysis tool in the
>> build
>> > >> >    (justinleet) closes apache/incubator-metron#376
>> > >> >    - METRON-592 Change popup to Warning message in Assign Master
>> page
>> > >> for
>> > >> >    Client check (anandsubbu via dlyle65535) closes
>> > >> apache/incubator-metron#375
>> > >> >    - METRON-598 Add Kyle Richardson to committers (kylerichardson)
>> > >> closes
>> > >> >    apache/incubator-metron#382
>> > >> >    - METRON-565: apps/metron/enrichment/indexed directory path does
>> > not
>> > >> >    get created for metron cluster deployed via Ambari (mmiklavc)
>> > closes
>> > >> >    apache/incubator-metron#365
>> > >> >    - METRON-588 Remove ASF License Headers from directly copied
>> Chef
>> > >> >    Bento Files closes apache/incubator-metron#372
>> > >> >    - METRON-576 Stellar function resolution takes too long on
>> running
>> > >> >    cluster (nickwallen) closes apache/incubator-metron#366
>> > >> >    - METRON-570 Add Profiler Link to README (nickwallen) closes
>> > >> >    apache/incubator-metron#371
>> > >> >    - METRON-578 Missing error handling bolts for enrichment and
>> threat
>> > >> >    intel closes apache/incubator-metron#363
>> > >> >    - METRON-522 Need to mandate Client installation on Metron Host
>> > >> closes
>> > >> >    apache/incubator-metron#367
>> > >> >    - METRON-584 STELLAR should have a TO_LONG Function
>> (ottobackwards)
>> > >> >    closes apache/incubator-metron#368
>> > >> >    - METRON-575 State from different profiles can be co-mingled
>> > >> >    incorrectly (nickwallen) closes apache/incubator-metron#362
>> > >> >    - METRON-538 Ensure proper shutdown, exceptions of integration
>> test
>> > >> >    components (ottobackwards) closes apache/incubator-metron#344
>> > >> >    - METRON-567 Usernames as numerics strings attempted to be
>> parsed
>> > and
>> > >> >    compared as numbers (ottobackwards) closes
>> > >> apache/incubator-metron#360
>> > >> >    - METRON-562: Add rudimentary statistical outlier detection
>> closes
>> > >> >    apache/incubator-metron#352
>> > >> >    - METRON-557 Create Stellar Functions for Kafka (nickwallen)
>> closes
>> > >> >    apache/incubator-metron#354
>> > >> >    - METRON-556 Profiler - Refactor 'Group By' Calculation
>> > (nickwallen)
>> > >> >    closes apache/incubator-metron#348
>> > >> >    - METRON-547 MaasIntegrationTests should integrate with the
>> Metron
>> > >> >    Integration classes (ottobackwards) closes
>> > >> apache/incubator-metron#350
>> > >> >    - METRON-561 ShellEditor tests hang if vim is set as EDITOR in
>> your
>> > >> >    profile (ottobackwards) closes apache/incubator-metron#351
>> > >> >
>> > >> > *What else will make it?*
>> > >> >
>> > >> > Ok, so now that we know what's in there already, I need to ask your
>> > help
>> > >> > in determining what you want in the next release that is not
>> currently
>> > >> > being worked on.  In order to do that, please help me by
>> responding to
>> > >> this
>> > >> > email and
>> > >> >
>> > >> >    - proposing and justifying a JIRA from the list of unresolved
>> JIRAs
>> > >> >    <
>> > >> https://issues.apache.org/jira/browse/METRON-469?jql=project
>> > >> %20%3D%20METRON%20AND%20resolution%20%3D%20Unresolved%20AND%
>> > >> 20fixVersion%20is%20EMPTY%20ORDER%20BY%20priority%20DESC>
>> > >> that
>> > >> >    are not currently scheduled and let me know which ones should be
>> > >> included.
>> > >> >    Please include whether the JIRA is "Nice to Have" or "Critical
>> to
>> > >> Have" and
>> > >> >    if you volunteer to do it.
>> > >> >    - proposing and justifying that one of the in-progress tasks
>> > >> >    <
>> > >> https://issues.apache.org/jira/browse/METRON-650?jql=project
>> > >> %20%3D%20METRON%20AND%20resolution%20%3D%20Unresolved%20AND%
>> > >> 20fixVersion%20%3D%200.3.1%20ORDER%20BY%20priority%20DESC
>> > >> >
>> > >> >     should *not* be included for release
>> > >> >
>> > >> > I'd like to aim to get release candidates out as soon as possible,
>> so
>> > >> > let's look forward a vigorous discussion and a great release.
>> > >> >
>> > >> > Thanks for the contributions and for your continued engagement in
>> the
>> > >> > Metron community.  If you see something that you don't like in this
>> > >> > process, please let me know.  Feedback is very welcome.
>> > >> >
>> > >> > Best,
>> > >> >
>> > >> > Casey
>> > >> >
>> > >>
>> > >> --
>> > >>
>> > >> Jon
>> > >>
>> > >> Sent from my mobile device
>> > >>
>> > >
>> > >
>> >
>> --
>>
>> Jon
>>
>> Sent from my mobile device
>>
>
>

Re: [DISCUSS] Next Release (0.3.1) Content

Posted by Casey Stella <ce...@gmail.com>.
Ok, I've created the upgrading document for 0.3.0 to 0.3.1 and included the
things that I know about and the things Jon mentioned here.  Please, if you
have knowledge of other breaking/non-compatible changes between the 0.3.0
release and master, comment on this PR (
https://github.com/apache/incubator-metron/pull/437) and I will incorporate
them.

On Fri, Jan 27, 2017 at 10:04 AM, Zeolla@GMail.com <ze...@gmail.com> wrote:

> To start I was mostly concerned with having a per-version list of
> non-backwards-compatible changes, so upgrades that may skip a version of
> two can look at what all may be impacted.  We should also probably document
> any sort of upgrade flaws as well, such as METRON-447
> <https://issues.apache.org/jira/browse/METRON-447>, METRON-448
> <https://issues.apache.org/jira/browse/METRON-448>, etc.  I do think that
> we should have a more rigorous document, but I wouldn't push that for the
> 0.3.1 release.  I see that (along with the Management UI, API, etc.) as key
> (required) components of a 1.0 release.  I'd just like to see the
> foundation begin to be laid and iterated on.
>
> That said, this probably constitutes a mention in the development
> guidelines
> <https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235
> >
> once
> it's in master.
>
> Jon
>
> On Fri, Jan 27, 2017 at 9:05 AM Casey Stella <ce...@gmail.com> wrote:
>
> > I should add, you may be thinking something more rigorous and
> > step-by-step.  If so, you think you might be interested in volunteering
> to
> > do a first draft as a PR that we can adjust?
> >
> > On Fri, Jan 27, 2017 at 9:01 AM, Casey Stella <ce...@gmail.com>
> wrote:
> >
> > > So, I agree with the Upgrading.md and I was going to submit a PR at
> least
> > > to describe the the changes to indexing configurations that I made
> during
> > > the 3.0.1 release.
> > >
> > >
> > > On Thu, Jan 26, 2017 at 10:50 PM, Zeolla@GMail.com <ze...@gmail.com>
> > > wrote:
> > >
> > >> I haven't had a chance to look through the unresolved JIRAs but I did
> > want
> > >> to mention a few quick things.
> > >>
> > >> First, when we released 0.3.0 and dropped the BETA flag, one of the
> > things
> > >> that was discussed was putting together a method of documenting
> upgrades
> > >> from one version to the next.  As one of the first steps toward making
> > >> that
> > >> a reasonable process, I think we should assemble more detailed release
> > >> notes, especially outlining non-backwards compatible changes.  In the
> > >> "[DISCUSS] Next Release Name" email thread Kyle Richardson suggested
> we
> > >> use
> > >> "UPGRADING.md" to do this, and I still agree with that thought, but
> I'm
> > >> open to alternatives.
> > >>
> > >> Separately, a *nice to have* would be *METRON-660*, which was
> discussed
> > in
> > >> the "[PROPOSAL] up-to-date versioned documentation" thread, to give us
> > >> some
> > >> cleaner documentation using the existing READMEs.  I'd be happy to
> help
> > >> with this one, I'm just not sure what the next steps are, aside from
> the
> > >> start that Matt has here
> > >> <https://github.com/mattf-horton/incubator-metron/tree/METRON-660>.
> > >>
> > >> My last *nice to have* is *METRON-635*, which I have a *PR open for
> here
> > >> <https://github.com/apache/incubator-metron/pull/411>*.  If I could
> get
> > >> someone else to reproduce the error that I'm seeing I would be happy
> to
> > >> pursue additional testing, troubleshooting, etc.  I've seen others
> > report
> > >> the same issue
> > >> <https://community.hortonworks.com/search.html?f=&type=
> > >> question&redirect=search%2Fsearch&sort=relevance&q=scp_if_ssh>
> > >> on the HCC boards, so I'm fairly confident that it is not an issue
> with
> > my
> > >> local environment.
> > >>
> > >> Jon
> > >>
> > >> On Thu, Jan 26, 2017 at 6:18 PM Casey Stella <ce...@gmail.com>
> > wrote:
> > >>
> > >> I took the liberty of adding the pull requests since the new year into
> > the
> > >> in-progress list in the previous email.  If you own one of these and
> do
> > >> not
> > >> believe that you can complete the review in the next week or so, let
> me
> > >> know and we can remove.  The only one of these that I see as mandatory
> > is
> > >> METRON-650 because the reliance on the opensoc github repo might cause
> > >> issues with the release being acceptable (see discussion by Mike
> > Miklavcic
> > >> with the mentors a couple weeks ago).
> > >>
> > >> On Thu, Jan 26, 2017 at 6:06 PM, Casey Stella <ce...@gmail.com>
> > wrote:
> > >>
> > >> > Hello Everyone!
> > >> >
> > >> > It's been almost 2 months since the last major release and I think
> it
> > >> > might be time to do a minor release of 0.3.1. The purpose of this
> > email
> > >> is
> > >> > multifold:
> > >> >
> > >> >    - to take stock of what we have already committed
> > >> >    - figure out what (if anything) is missing to make a release that
> > >> >    we're proud of
> > >> >    - find volunteers for the missing items
> > >> >
> > >> > For those who get email alerts on the JIRA changes, it should be no
> > >> > surprise that I have gone through and put the committed items in the
> > >> 0.3.1
> > >> > release bucket.  So, let's take stock of what we have.
> > >> >
> > >> > *What's made it so far into the next release*
> > >> >
> > >> >
> > >> >
> > >> >    - METRON-283 Migrate Geo Enrichment outside of MySQL (justinleet)
> > >> >    closes apache/incubator-metron#421
> > >> >    - METRON-668: Remove the "tickUpdate" profile config and make the
> > >> >    "init" phase not reset variables closes
> apache/incubator-metron#420
> > >> >    - METRON-672: SolrIndexingIntegrationTest fails intermittently
> > closes
> > >> >    apache/incubator-metron#424
> > >> >    - METRON-664: Make the index configuration per-writer with
> > >> >    enabled/disabled closes apache/incubator-metron#419
> > >> >    - METRON-600: Fix Metron Website (iraghumitra via cestella)
> closes
> > >> >    apache/incubator-metron#399
> > >> >    - METRON-666 Fix javadoc doclint errors closes
> > >> >    apache/incubator-metron#418
> > >> >    - METRON-659 Emulate Sensors in Development Environments
> > (nickwallen)
> > >> >    closes apache/incubator-metron#417
> > >> >    - METRON-652: Extract indexing config from enrichment config
> closes
> > >> >    apache/incubator-metron#415
> > >> >    - METRON-654 Create RPM Installer for Profiler (nickwallen)
> closes
> > >> >    apache/incubator-metron#413
> > >> >    - METRON-656: Make Stellar 'in' closer to functioning like python
> > >> >    closes apache/incubator-metron#416
> > >> >    - METRON-532 Define Profile Period When Calling PROFILE_GET
> closes
> > >> >    apache/incubator-metron#414
> > >> >    - METRON-624: Updated Comparison/Equality Evaluations in Stellar
> > >> >    closes apache/incubator-metron#404
> > >> >    - METRON-644 RPM builds only work with Docker for Mac
> > >> (kylerichardson)
> > >> >    closes apache/incubator-metron#409
> > >> >    - METRON-640: Add a Stellar function to compute shannon entropy
> for
> > >> >    strings closes apache/incubator-metron#403
> > >> >    - METRON-637: Add a STATS_BIN function to Stellar. closes
> > >> >    apache/incubator-metron#401
> > >> >    - METRON-622 Create a Metron Docker Compose application
> > (merrimanr)
> > >> >    closes apache/incubator-metron#393
> > >> >    - METRON-643: Stellar function documentation needs to be updated
> > >> >    closes apache/incubator-metron#407
> > >> >    - METRON-645 Unable to Start Fastcapa Test Environment
> (nickwallen)
> > >> >    closes apache/incubator-metron#410
> > >> >    - METRON-647 Parser unit test failures due to assumed year values
> > >> >    (justinleet) closes apache/incubator-metron#412
> > >> >    - METRON-639: The Network Stellar functions need to have better
> > unit
> > >> >    testing closes apache/incubator-metron#402
> > >> >    - METRON-631 Broken link on fastcapa README (JonZeolla via
> > >> nickwallen)
> > >> >    closes apache/incubator-metron#398
> > >> >    - METRON-625: Parser Filters cannot be specified from the sensor
> > >> >    config closes apache/incubator-metron#396
> > >> >    - METRON-587 Integration tests should use common processor
> > >> >    implementations where possible (ottobackwards) closes
> > >> >    apache/incubator-metron#374
> > >> >    - METRON-616: Added support for float and long literals in
> Stellar
> > >> >    closes apache/incubator-metron#392
> > >> >    - METRON-580: Remove hard-coded Metron version from Ambari MPack
> > code
> > >> >    (mmiklavc) closes apache/incubator-metron#364
> > >> >    - METRON-618 Eliminate Javac Warnings in metron-analytics
> > >> (justinleet)
> > >> >    closes apache/incubator-metron#391
> > >> >    - METRON-364: Preserve Type for Arithmetic Expressions in Stellar
> > >> >    closes apache/incubator-metron#390
> > >> >    - METRON-585 Pcap Replay Fails to Stop (nickwallen) closes
> > >> >    apache/incubator-metron#369
> > >> >    - METRON-586 STELLAR should have FILL_LEFT and FILL_RIGHT
> functions
> > >> >    (ottobackwards) closes apache/incubator-metron#370
> > >> >    - METRON-612 Clean up Error Prone generated warnings (justinleet)
> > >> >    closes apache/incubator-metron#389
> > >> >    - METRON-610: OnlineStatisticsProvider serialization is broken at
> > >> >    random in the REPL closes apache/incubator-metron#388
> > >> >    - METRON-596 Eliminate Maven warnings from build (justinleet)
> > closes
> > >> >    apache/incubator-metron#378
> > >> >    - METRON-604 Mpack installs do not work on clean machines due to
> > >> >    missing Elastic Curator repo (justinleet) closes
> > >> apache/incubator-metron#385
> > >> >    - METRON-607: Enrichment doc improvement and test cleanup
> > (mmiklavc)
> > >> >    closes apache/incubator-metron#386
> > >> >    - METRON-606 Profiler Overwriting Previously Written Values
> > >> >    (nickwallen) closes apache/incubator-metron#387
> > >> >    - METRON-591: Make the website in compliance with ASF standards
> > >> closes
> > >> >    apache/incubator-metron#373
> > >> >    - METRON-597 Sporadic Failures of Profiler Integration Tests
> > >> >    (nickwallen) closes apache/incubator-metron#383
> > >> >    - METRON-595 Elasticsearch Writer only uses One IP Address
> > >> >    (JonathanRider via dlyle65535) closes apache/incubator-metron#379
> > >> >    - METRON-593 Enable an automated static analysis tool in the
> build
> > >> >    (justinleet) closes apache/incubator-metron#376
> > >> >    - METRON-592 Change popup to Warning message in Assign Master
> page
> > >> for
> > >> >    Client check (anandsubbu via dlyle65535) closes
> > >> apache/incubator-metron#375
> > >> >    - METRON-598 Add Kyle Richardson to committers (kylerichardson)
> > >> closes
> > >> >    apache/incubator-metron#382
> > >> >    - METRON-565: apps/metron/enrichment/indexed directory path does
> > not
> > >> >    get created for metron cluster deployed via Ambari (mmiklavc)
> > closes
> > >> >    apache/incubator-metron#365
> > >> >    - METRON-588 Remove ASF License Headers from directly copied Chef
> > >> >    Bento Files closes apache/incubator-metron#372
> > >> >    - METRON-576 Stellar function resolution takes too long on
> running
> > >> >    cluster (nickwallen) closes apache/incubator-metron#366
> > >> >    - METRON-570 Add Profiler Link to README (nickwallen) closes
> > >> >    apache/incubator-metron#371
> > >> >    - METRON-578 Missing error handling bolts for enrichment and
> threat
> > >> >    intel closes apache/incubator-metron#363
> > >> >    - METRON-522 Need to mandate Client installation on Metron Host
> > >> closes
> > >> >    apache/incubator-metron#367
> > >> >    - METRON-584 STELLAR should have a TO_LONG Function
> (ottobackwards)
> > >> >    closes apache/incubator-metron#368
> > >> >    - METRON-575 State from different profiles can be co-mingled
> > >> >    incorrectly (nickwallen) closes apache/incubator-metron#362
> > >> >    - METRON-538 Ensure proper shutdown, exceptions of integration
> test
> > >> >    components (ottobackwards) closes apache/incubator-metron#344
> > >> >    - METRON-567 Usernames as numerics strings attempted to be parsed
> > and
> > >> >    compared as numbers (ottobackwards) closes
> > >> apache/incubator-metron#360
> > >> >    - METRON-562: Add rudimentary statistical outlier detection
> closes
> > >> >    apache/incubator-metron#352
> > >> >    - METRON-557 Create Stellar Functions for Kafka (nickwallen)
> closes
> > >> >    apache/incubator-metron#354
> > >> >    - METRON-556 Profiler - Refactor 'Group By' Calculation
> > (nickwallen)
> > >> >    closes apache/incubator-metron#348
> > >> >    - METRON-547 MaasIntegrationTests should integrate with the
> Metron
> > >> >    Integration classes (ottobackwards) closes
> > >> apache/incubator-metron#350
> > >> >    - METRON-561 ShellEditor tests hang if vim is set as EDITOR in
> your
> > >> >    profile (ottobackwards) closes apache/incubator-metron#351
> > >> >
> > >> > *What else will make it?*
> > >> >
> > >> > Ok, so now that we know what's in there already, I need to ask your
> > help
> > >> > in determining what you want in the next release that is not
> currently
> > >> > being worked on.  In order to do that, please help me by responding
> to
> > >> this
> > >> > email and
> > >> >
> > >> >    - proposing and justifying a JIRA from the list of unresolved
> JIRAs
> > >> >    <
> > >> https://issues.apache.org/jira/browse/METRON-469?jql=project
> > >> %20%3D%20METRON%20AND%20resolution%20%3D%20Unresolved%20AND%
> > >> 20fixVersion%20is%20EMPTY%20ORDER%20BY%20priority%20DESC>
> > >> that
> > >> >    are not currently scheduled and let me know which ones should be
> > >> included.
> > >> >    Please include whether the JIRA is "Nice to Have" or "Critical to
> > >> Have" and
> > >> >    if you volunteer to do it.
> > >> >    - proposing and justifying that one of the in-progress tasks
> > >> >    <
> > >> https://issues.apache.org/jira/browse/METRON-650?jql=project
> > >> %20%3D%20METRON%20AND%20resolution%20%3D%20Unresolved%20AND%
> > >> 20fixVersion%20%3D%200.3.1%20ORDER%20BY%20priority%20DESC
> > >> >
> > >> >     should *not* be included for release
> > >> >
> > >> > I'd like to aim to get release candidates out as soon as possible,
> so
> > >> > let's look forward a vigorous discussion and a great release.
> > >> >
> > >> > Thanks for the contributions and for your continued engagement in
> the
> > >> > Metron community.  If you see something that you don't like in this
> > >> > process, please let me know.  Feedback is very welcome.
> > >> >
> > >> > Best,
> > >> >
> > >> > Casey
> > >> >
> > >>
> > >> --
> > >>
> > >> Jon
> > >>
> > >> Sent from my mobile device
> > >>
> > >
> > >
> >
> --
>
> Jon
>
> Sent from my mobile device
>

Re: [DISCUSS] Next Release (0.3.1) Content

Posted by "Zeolla@GMail.com" <ze...@gmail.com>.
To start I was mostly concerned with having a per-version list of
non-backwards-compatible changes, so upgrades that may skip a version of
two can look at what all may be impacted.  We should also probably document
any sort of upgrade flaws as well, such as METRON-447
<https://issues.apache.org/jira/browse/METRON-447>, METRON-448
<https://issues.apache.org/jira/browse/METRON-448>, etc.  I do think that
we should have a more rigorous document, but I wouldn't push that for the
0.3.1 release.  I see that (along with the Management UI, API, etc.) as key
(required) components of a 1.0 release.  I'd just like to see the
foundation begin to be laid and iterated on.

That said, this probably constitutes a mention in the development guidelines
<https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235>
once
it's in master.

Jon

On Fri, Jan 27, 2017 at 9:05 AM Casey Stella <ce...@gmail.com> wrote:

> I should add, you may be thinking something more rigorous and
> step-by-step.  If so, you think you might be interested in volunteering to
> do a first draft as a PR that we can adjust?
>
> On Fri, Jan 27, 2017 at 9:01 AM, Casey Stella <ce...@gmail.com> wrote:
>
> > So, I agree with the Upgrading.md and I was going to submit a PR at least
> > to describe the the changes to indexing configurations that I made during
> > the 3.0.1 release.
> >
> >
> > On Thu, Jan 26, 2017 at 10:50 PM, Zeolla@GMail.com <ze...@gmail.com>
> > wrote:
> >
> >> I haven't had a chance to look through the unresolved JIRAs but I did
> want
> >> to mention a few quick things.
> >>
> >> First, when we released 0.3.0 and dropped the BETA flag, one of the
> things
> >> that was discussed was putting together a method of documenting upgrades
> >> from one version to the next.  As one of the first steps toward making
> >> that
> >> a reasonable process, I think we should assemble more detailed release
> >> notes, especially outlining non-backwards compatible changes.  In the
> >> "[DISCUSS] Next Release Name" email thread Kyle Richardson suggested we
> >> use
> >> "UPGRADING.md" to do this, and I still agree with that thought, but I'm
> >> open to alternatives.
> >>
> >> Separately, a *nice to have* would be *METRON-660*, which was discussed
> in
> >> the "[PROPOSAL] up-to-date versioned documentation" thread, to give us
> >> some
> >> cleaner documentation using the existing READMEs.  I'd be happy to help
> >> with this one, I'm just not sure what the next steps are, aside from the
> >> start that Matt has here
> >> <https://github.com/mattf-horton/incubator-metron/tree/METRON-660>.
> >>
> >> My last *nice to have* is *METRON-635*, which I have a *PR open for here
> >> <https://github.com/apache/incubator-metron/pull/411>*.  If I could get
> >> someone else to reproduce the error that I'm seeing I would be happy to
> >> pursue additional testing, troubleshooting, etc.  I've seen others
> report
> >> the same issue
> >> <https://community.hortonworks.com/search.html?f=&type=
> >> question&redirect=search%2Fsearch&sort=relevance&q=scp_if_ssh>
> >> on the HCC boards, so I'm fairly confident that it is not an issue with
> my
> >> local environment.
> >>
> >> Jon
> >>
> >> On Thu, Jan 26, 2017 at 6:18 PM Casey Stella <ce...@gmail.com>
> wrote:
> >>
> >> I took the liberty of adding the pull requests since the new year into
> the
> >> in-progress list in the previous email.  If you own one of these and do
> >> not
> >> believe that you can complete the review in the next week or so, let me
> >> know and we can remove.  The only one of these that I see as mandatory
> is
> >> METRON-650 because the reliance on the opensoc github repo might cause
> >> issues with the release being acceptable (see discussion by Mike
> Miklavcic
> >> with the mentors a couple weeks ago).
> >>
> >> On Thu, Jan 26, 2017 at 6:06 PM, Casey Stella <ce...@gmail.com>
> wrote:
> >>
> >> > Hello Everyone!
> >> >
> >> > It's been almost 2 months since the last major release and I think it
> >> > might be time to do a minor release of 0.3.1. The purpose of this
> email
> >> is
> >> > multifold:
> >> >
> >> >    - to take stock of what we have already committed
> >> >    - figure out what (if anything) is missing to make a release that
> >> >    we're proud of
> >> >    - find volunteers for the missing items
> >> >
> >> > For those who get email alerts on the JIRA changes, it should be no
> >> > surprise that I have gone through and put the committed items in the
> >> 0.3.1
> >> > release bucket.  So, let's take stock of what we have.
> >> >
> >> > *What's made it so far into the next release*
> >> >
> >> >
> >> >
> >> >    - METRON-283 Migrate Geo Enrichment outside of MySQL (justinleet)
> >> >    closes apache/incubator-metron#421
> >> >    - METRON-668: Remove the "tickUpdate" profile config and make the
> >> >    "init" phase not reset variables closes apache/incubator-metron#420
> >> >    - METRON-672: SolrIndexingIntegrationTest fails intermittently
> closes
> >> >    apache/incubator-metron#424
> >> >    - METRON-664: Make the index configuration per-writer with
> >> >    enabled/disabled closes apache/incubator-metron#419
> >> >    - METRON-600: Fix Metron Website (iraghumitra via cestella) closes
> >> >    apache/incubator-metron#399
> >> >    - METRON-666 Fix javadoc doclint errors closes
> >> >    apache/incubator-metron#418
> >> >    - METRON-659 Emulate Sensors in Development Environments
> (nickwallen)
> >> >    closes apache/incubator-metron#417
> >> >    - METRON-652: Extract indexing config from enrichment config closes
> >> >    apache/incubator-metron#415
> >> >    - METRON-654 Create RPM Installer for Profiler (nickwallen) closes
> >> >    apache/incubator-metron#413
> >> >    - METRON-656: Make Stellar 'in' closer to functioning like python
> >> >    closes apache/incubator-metron#416
> >> >    - METRON-532 Define Profile Period When Calling PROFILE_GET closes
> >> >    apache/incubator-metron#414
> >> >    - METRON-624: Updated Comparison/Equality Evaluations in Stellar
> >> >    closes apache/incubator-metron#404
> >> >    - METRON-644 RPM builds only work with Docker for Mac
> >> (kylerichardson)
> >> >    closes apache/incubator-metron#409
> >> >    - METRON-640: Add a Stellar function to compute shannon entropy for
> >> >    strings closes apache/incubator-metron#403
> >> >    - METRON-637: Add a STATS_BIN function to Stellar. closes
> >> >    apache/incubator-metron#401
> >> >    - METRON-622 Create a Metron Docker Compose application
> (merrimanr)
> >> >    closes apache/incubator-metron#393
> >> >    - METRON-643: Stellar function documentation needs to be updated
> >> >    closes apache/incubator-metron#407
> >> >    - METRON-645 Unable to Start Fastcapa Test Environment (nickwallen)
> >> >    closes apache/incubator-metron#410
> >> >    - METRON-647 Parser unit test failures due to assumed year values
> >> >    (justinleet) closes apache/incubator-metron#412
> >> >    - METRON-639: The Network Stellar functions need to have better
> unit
> >> >    testing closes apache/incubator-metron#402
> >> >    - METRON-631 Broken link on fastcapa README (JonZeolla via
> >> nickwallen)
> >> >    closes apache/incubator-metron#398
> >> >    - METRON-625: Parser Filters cannot be specified from the sensor
> >> >    config closes apache/incubator-metron#396
> >> >    - METRON-587 Integration tests should use common processor
> >> >    implementations where possible (ottobackwards) closes
> >> >    apache/incubator-metron#374
> >> >    - METRON-616: Added support for float and long literals in Stellar
> >> >    closes apache/incubator-metron#392
> >> >    - METRON-580: Remove hard-coded Metron version from Ambari MPack
> code
> >> >    (mmiklavc) closes apache/incubator-metron#364
> >> >    - METRON-618 Eliminate Javac Warnings in metron-analytics
> >> (justinleet)
> >> >    closes apache/incubator-metron#391
> >> >    - METRON-364: Preserve Type for Arithmetic Expressions in Stellar
> >> >    closes apache/incubator-metron#390
> >> >    - METRON-585 Pcap Replay Fails to Stop (nickwallen) closes
> >> >    apache/incubator-metron#369
> >> >    - METRON-586 STELLAR should have FILL_LEFT and FILL_RIGHT functions
> >> >    (ottobackwards) closes apache/incubator-metron#370
> >> >    - METRON-612 Clean up Error Prone generated warnings (justinleet)
> >> >    closes apache/incubator-metron#389
> >> >    - METRON-610: OnlineStatisticsProvider serialization is broken at
> >> >    random in the REPL closes apache/incubator-metron#388
> >> >    - METRON-596 Eliminate Maven warnings from build (justinleet)
> closes
> >> >    apache/incubator-metron#378
> >> >    - METRON-604 Mpack installs do not work on clean machines due to
> >> >    missing Elastic Curator repo (justinleet) closes
> >> apache/incubator-metron#385
> >> >    - METRON-607: Enrichment doc improvement and test cleanup
> (mmiklavc)
> >> >    closes apache/incubator-metron#386
> >> >    - METRON-606 Profiler Overwriting Previously Written Values
> >> >    (nickwallen) closes apache/incubator-metron#387
> >> >    - METRON-591: Make the website in compliance with ASF standards
> >> closes
> >> >    apache/incubator-metron#373
> >> >    - METRON-597 Sporadic Failures of Profiler Integration Tests
> >> >    (nickwallen) closes apache/incubator-metron#383
> >> >    - METRON-595 Elasticsearch Writer only uses One IP Address
> >> >    (JonathanRider via dlyle65535) closes apache/incubator-metron#379
> >> >    - METRON-593 Enable an automated static analysis tool in the build
> >> >    (justinleet) closes apache/incubator-metron#376
> >> >    - METRON-592 Change popup to Warning message in Assign Master page
> >> for
> >> >    Client check (anandsubbu via dlyle65535) closes
> >> apache/incubator-metron#375
> >> >    - METRON-598 Add Kyle Richardson to committers (kylerichardson)
> >> closes
> >> >    apache/incubator-metron#382
> >> >    - METRON-565: apps/metron/enrichment/indexed directory path does
> not
> >> >    get created for metron cluster deployed via Ambari (mmiklavc)
> closes
> >> >    apache/incubator-metron#365
> >> >    - METRON-588 Remove ASF License Headers from directly copied Chef
> >> >    Bento Files closes apache/incubator-metron#372
> >> >    - METRON-576 Stellar function resolution takes too long on running
> >> >    cluster (nickwallen) closes apache/incubator-metron#366
> >> >    - METRON-570 Add Profiler Link to README (nickwallen) closes
> >> >    apache/incubator-metron#371
> >> >    - METRON-578 Missing error handling bolts for enrichment and threat
> >> >    intel closes apache/incubator-metron#363
> >> >    - METRON-522 Need to mandate Client installation on Metron Host
> >> closes
> >> >    apache/incubator-metron#367
> >> >    - METRON-584 STELLAR should have a TO_LONG Function (ottobackwards)
> >> >    closes apache/incubator-metron#368
> >> >    - METRON-575 State from different profiles can be co-mingled
> >> >    incorrectly (nickwallen) closes apache/incubator-metron#362
> >> >    - METRON-538 Ensure proper shutdown, exceptions of integration test
> >> >    components (ottobackwards) closes apache/incubator-metron#344
> >> >    - METRON-567 Usernames as numerics strings attempted to be parsed
> and
> >> >    compared as numbers (ottobackwards) closes
> >> apache/incubator-metron#360
> >> >    - METRON-562: Add rudimentary statistical outlier detection closes
> >> >    apache/incubator-metron#352
> >> >    - METRON-557 Create Stellar Functions for Kafka (nickwallen) closes
> >> >    apache/incubator-metron#354
> >> >    - METRON-556 Profiler - Refactor 'Group By' Calculation
> (nickwallen)
> >> >    closes apache/incubator-metron#348
> >> >    - METRON-547 MaasIntegrationTests should integrate with the Metron
> >> >    Integration classes (ottobackwards) closes
> >> apache/incubator-metron#350
> >> >    - METRON-561 ShellEditor tests hang if vim is set as EDITOR in your
> >> >    profile (ottobackwards) closes apache/incubator-metron#351
> >> >
> >> > *What else will make it?*
> >> >
> >> > Ok, so now that we know what's in there already, I need to ask your
> help
> >> > in determining what you want in the next release that is not currently
> >> > being worked on.  In order to do that, please help me by responding to
> >> this
> >> > email and
> >> >
> >> >    - proposing and justifying a JIRA from the list of unresolved JIRAs
> >> >    <
> >> https://issues.apache.org/jira/browse/METRON-469?jql=project
> >> %20%3D%20METRON%20AND%20resolution%20%3D%20Unresolved%20AND%
> >> 20fixVersion%20is%20EMPTY%20ORDER%20BY%20priority%20DESC>
> >> that
> >> >    are not currently scheduled and let me know which ones should be
> >> included.
> >> >    Please include whether the JIRA is "Nice to Have" or "Critical to
> >> Have" and
> >> >    if you volunteer to do it.
> >> >    - proposing and justifying that one of the in-progress tasks
> >> >    <
> >> https://issues.apache.org/jira/browse/METRON-650?jql=project
> >> %20%3D%20METRON%20AND%20resolution%20%3D%20Unresolved%20AND%
> >> 20fixVersion%20%3D%200.3.1%20ORDER%20BY%20priority%20DESC
> >> >
> >> >     should *not* be included for release
> >> >
> >> > I'd like to aim to get release candidates out as soon as possible, so
> >> > let's look forward a vigorous discussion and a great release.
> >> >
> >> > Thanks for the contributions and for your continued engagement in the
> >> > Metron community.  If you see something that you don't like in this
> >> > process, please let me know.  Feedback is very welcome.
> >> >
> >> > Best,
> >> >
> >> > Casey
> >> >
> >>
> >> --
> >>
> >> Jon
> >>
> >> Sent from my mobile device
> >>
> >
> >
>
-- 

Jon

Sent from my mobile device

Re: [DISCUSS] Next Release (0.3.1) Content

Posted by Casey Stella <ce...@gmail.com>.
I should add, you may be thinking something more rigorous and
step-by-step.  If so, you think you might be interested in volunteering to
do a first draft as a PR that we can adjust?

On Fri, Jan 27, 2017 at 9:01 AM, Casey Stella <ce...@gmail.com> wrote:

> So, I agree with the Upgrading.md and I was going to submit a PR at least
> to describe the the changes to indexing configurations that I made during
> the 3.0.1 release.
>
>
> On Thu, Jan 26, 2017 at 10:50 PM, Zeolla@GMail.com <ze...@gmail.com>
> wrote:
>
>> I haven't had a chance to look through the unresolved JIRAs but I did want
>> to mention a few quick things.
>>
>> First, when we released 0.3.0 and dropped the BETA flag, one of the things
>> that was discussed was putting together a method of documenting upgrades
>> from one version to the next.  As one of the first steps toward making
>> that
>> a reasonable process, I think we should assemble more detailed release
>> notes, especially outlining non-backwards compatible changes.  In the
>> "[DISCUSS] Next Release Name" email thread Kyle Richardson suggested we
>> use
>> "UPGRADING.md" to do this, and I still agree with that thought, but I'm
>> open to alternatives.
>>
>> Separately, a *nice to have* would be *METRON-660*, which was discussed in
>> the "[PROPOSAL] up-to-date versioned documentation" thread, to give us
>> some
>> cleaner documentation using the existing READMEs.  I'd be happy to help
>> with this one, I'm just not sure what the next steps are, aside from the
>> start that Matt has here
>> <https://github.com/mattf-horton/incubator-metron/tree/METRON-660>.
>>
>> My last *nice to have* is *METRON-635*, which I have a *PR open for here
>> <https://github.com/apache/incubator-metron/pull/411>*.  If I could get
>> someone else to reproduce the error that I'm seeing I would be happy to
>> pursue additional testing, troubleshooting, etc.  I've seen others report
>> the same issue
>> <https://community.hortonworks.com/search.html?f=&type=
>> question&redirect=search%2Fsearch&sort=relevance&q=scp_if_ssh>
>> on the HCC boards, so I'm fairly confident that it is not an issue with my
>> local environment.
>>
>> Jon
>>
>> On Thu, Jan 26, 2017 at 6:18 PM Casey Stella <ce...@gmail.com> wrote:
>>
>> I took the liberty of adding the pull requests since the new year into the
>> in-progress list in the previous email.  If you own one of these and do
>> not
>> believe that you can complete the review in the next week or so, let me
>> know and we can remove.  The only one of these that I see as mandatory is
>> METRON-650 because the reliance on the opensoc github repo might cause
>> issues with the release being acceptable (see discussion by Mike Miklavcic
>> with the mentors a couple weeks ago).
>>
>> On Thu, Jan 26, 2017 at 6:06 PM, Casey Stella <ce...@gmail.com> wrote:
>>
>> > Hello Everyone!
>> >
>> > It's been almost 2 months since the last major release and I think it
>> > might be time to do a minor release of 0.3.1. The purpose of this email
>> is
>> > multifold:
>> >
>> >    - to take stock of what we have already committed
>> >    - figure out what (if anything) is missing to make a release that
>> >    we're proud of
>> >    - find volunteers for the missing items
>> >
>> > For those who get email alerts on the JIRA changes, it should be no
>> > surprise that I have gone through and put the committed items in the
>> 0.3.1
>> > release bucket.  So, let's take stock of what we have.
>> >
>> > *What's made it so far into the next release*
>> >
>> >
>> >
>> >    - METRON-283 Migrate Geo Enrichment outside of MySQL (justinleet)
>> >    closes apache/incubator-metron#421
>> >    - METRON-668: Remove the "tickUpdate" profile config and make the
>> >    "init" phase not reset variables closes apache/incubator-metron#420
>> >    - METRON-672: SolrIndexingIntegrationTest fails intermittently closes
>> >    apache/incubator-metron#424
>> >    - METRON-664: Make the index configuration per-writer with
>> >    enabled/disabled closes apache/incubator-metron#419
>> >    - METRON-600: Fix Metron Website (iraghumitra via cestella) closes
>> >    apache/incubator-metron#399
>> >    - METRON-666 Fix javadoc doclint errors closes
>> >    apache/incubator-metron#418
>> >    - METRON-659 Emulate Sensors in Development Environments (nickwallen)
>> >    closes apache/incubator-metron#417
>> >    - METRON-652: Extract indexing config from enrichment config closes
>> >    apache/incubator-metron#415
>> >    - METRON-654 Create RPM Installer for Profiler (nickwallen) closes
>> >    apache/incubator-metron#413
>> >    - METRON-656: Make Stellar 'in' closer to functioning like python
>> >    closes apache/incubator-metron#416
>> >    - METRON-532 Define Profile Period When Calling PROFILE_GET closes
>> >    apache/incubator-metron#414
>> >    - METRON-624: Updated Comparison/Equality Evaluations in Stellar
>> >    closes apache/incubator-metron#404
>> >    - METRON-644 RPM builds only work with Docker for Mac
>> (kylerichardson)
>> >    closes apache/incubator-metron#409
>> >    - METRON-640: Add a Stellar function to compute shannon entropy for
>> >    strings closes apache/incubator-metron#403
>> >    - METRON-637: Add a STATS_BIN function to Stellar. closes
>> >    apache/incubator-metron#401
>> >    - METRON-622 Create a Metron Docker Compose application  (merrimanr)
>> >    closes apache/incubator-metron#393
>> >    - METRON-643: Stellar function documentation needs to be updated
>> >    closes apache/incubator-metron#407
>> >    - METRON-645 Unable to Start Fastcapa Test Environment (nickwallen)
>> >    closes apache/incubator-metron#410
>> >    - METRON-647 Parser unit test failures due to assumed year values
>> >    (justinleet) closes apache/incubator-metron#412
>> >    - METRON-639: The Network Stellar functions need to have better unit
>> >    testing closes apache/incubator-metron#402
>> >    - METRON-631 Broken link on fastcapa README (JonZeolla via
>> nickwallen)
>> >    closes apache/incubator-metron#398
>> >    - METRON-625: Parser Filters cannot be specified from the sensor
>> >    config closes apache/incubator-metron#396
>> >    - METRON-587 Integration tests should use common processor
>> >    implementations where possible (ottobackwards) closes
>> >    apache/incubator-metron#374
>> >    - METRON-616: Added support for float and long literals in Stellar
>> >    closes apache/incubator-metron#392
>> >    - METRON-580: Remove hard-coded Metron version from Ambari MPack code
>> >    (mmiklavc) closes apache/incubator-metron#364
>> >    - METRON-618 Eliminate Javac Warnings in metron-analytics
>> (justinleet)
>> >    closes apache/incubator-metron#391
>> >    - METRON-364: Preserve Type for Arithmetic Expressions in Stellar
>> >    closes apache/incubator-metron#390
>> >    - METRON-585 Pcap Replay Fails to Stop (nickwallen) closes
>> >    apache/incubator-metron#369
>> >    - METRON-586 STELLAR should have FILL_LEFT and FILL_RIGHT functions
>> >    (ottobackwards) closes apache/incubator-metron#370
>> >    - METRON-612 Clean up Error Prone generated warnings (justinleet)
>> >    closes apache/incubator-metron#389
>> >    - METRON-610: OnlineStatisticsProvider serialization is broken at
>> >    random in the REPL closes apache/incubator-metron#388
>> >    - METRON-596 Eliminate Maven warnings from build (justinleet) closes
>> >    apache/incubator-metron#378
>> >    - METRON-604 Mpack installs do not work on clean machines due to
>> >    missing Elastic Curator repo (justinleet) closes
>> apache/incubator-metron#385
>> >    - METRON-607: Enrichment doc improvement and test cleanup (mmiklavc)
>> >    closes apache/incubator-metron#386
>> >    - METRON-606 Profiler Overwriting Previously Written Values
>> >    (nickwallen) closes apache/incubator-metron#387
>> >    - METRON-591: Make the website in compliance with ASF standards
>> closes
>> >    apache/incubator-metron#373
>> >    - METRON-597 Sporadic Failures of Profiler Integration Tests
>> >    (nickwallen) closes apache/incubator-metron#383
>> >    - METRON-595 Elasticsearch Writer only uses One IP Address
>> >    (JonathanRider via dlyle65535) closes apache/incubator-metron#379
>> >    - METRON-593 Enable an automated static analysis tool in the build
>> >    (justinleet) closes apache/incubator-metron#376
>> >    - METRON-592 Change popup to Warning message in Assign Master page
>> for
>> >    Client check (anandsubbu via dlyle65535) closes
>> apache/incubator-metron#375
>> >    - METRON-598 Add Kyle Richardson to committers (kylerichardson)
>> closes
>> >    apache/incubator-metron#382
>> >    - METRON-565: apps/metron/enrichment/indexed directory path does not
>> >    get created for metron cluster deployed via Ambari (mmiklavc) closes
>> >    apache/incubator-metron#365
>> >    - METRON-588 Remove ASF License Headers from directly copied Chef
>> >    Bento Files closes apache/incubator-metron#372
>> >    - METRON-576 Stellar function resolution takes too long on running
>> >    cluster (nickwallen) closes apache/incubator-metron#366
>> >    - METRON-570 Add Profiler Link to README (nickwallen) closes
>> >    apache/incubator-metron#371
>> >    - METRON-578 Missing error handling bolts for enrichment and threat
>> >    intel closes apache/incubator-metron#363
>> >    - METRON-522 Need to mandate Client installation on Metron Host
>> closes
>> >    apache/incubator-metron#367
>> >    - METRON-584 STELLAR should have a TO_LONG Function (ottobackwards)
>> >    closes apache/incubator-metron#368
>> >    - METRON-575 State from different profiles can be co-mingled
>> >    incorrectly (nickwallen) closes apache/incubator-metron#362
>> >    - METRON-538 Ensure proper shutdown, exceptions of integration test
>> >    components (ottobackwards) closes apache/incubator-metron#344
>> >    - METRON-567 Usernames as numerics strings attempted to be parsed and
>> >    compared as numbers (ottobackwards) closes
>> apache/incubator-metron#360
>> >    - METRON-562: Add rudimentary statistical outlier detection closes
>> >    apache/incubator-metron#352
>> >    - METRON-557 Create Stellar Functions for Kafka (nickwallen) closes
>> >    apache/incubator-metron#354
>> >    - METRON-556 Profiler - Refactor 'Group By' Calculation (nickwallen)
>> >    closes apache/incubator-metron#348
>> >    - METRON-547 MaasIntegrationTests should integrate with the Metron
>> >    Integration classes (ottobackwards) closes
>> apache/incubator-metron#350
>> >    - METRON-561 ShellEditor tests hang if vim is set as EDITOR in your
>> >    profile (ottobackwards) closes apache/incubator-metron#351
>> >
>> > *What else will make it?*
>> >
>> > Ok, so now that we know what's in there already, I need to ask your help
>> > in determining what you want in the next release that is not currently
>> > being worked on.  In order to do that, please help me by responding to
>> this
>> > email and
>> >
>> >    - proposing and justifying a JIRA from the list of unresolved JIRAs
>> >    <
>> https://issues.apache.org/jira/browse/METRON-469?jql=project
>> %20%3D%20METRON%20AND%20resolution%20%3D%20Unresolved%20AND%
>> 20fixVersion%20is%20EMPTY%20ORDER%20BY%20priority%20DESC>
>> that
>> >    are not currently scheduled and let me know which ones should be
>> included.
>> >    Please include whether the JIRA is "Nice to Have" or "Critical to
>> Have" and
>> >    if you volunteer to do it.
>> >    - proposing and justifying that one of the in-progress tasks
>> >    <
>> https://issues.apache.org/jira/browse/METRON-650?jql=project
>> %20%3D%20METRON%20AND%20resolution%20%3D%20Unresolved%20AND%
>> 20fixVersion%20%3D%200.3.1%20ORDER%20BY%20priority%20DESC
>> >
>> >     should *not* be included for release
>> >
>> > I'd like to aim to get release candidates out as soon as possible, so
>> > let's look forward a vigorous discussion and a great release.
>> >
>> > Thanks for the contributions and for your continued engagement in the
>> > Metron community.  If you see something that you don't like in this
>> > process, please let me know.  Feedback is very welcome.
>> >
>> > Best,
>> >
>> > Casey
>> >
>>
>> --
>>
>> Jon
>>
>> Sent from my mobile device
>>
>
>

Re: [DISCUSS] Next Release (0.3.1) Content

Posted by Casey Stella <ce...@gmail.com>.
So, I agree with the Upgrading.md and I was going to submit a PR at least
to describe the the changes to indexing configurations that I made during
the 3.0.1 release.


On Thu, Jan 26, 2017 at 10:50 PM, Zeolla@GMail.com <ze...@gmail.com> wrote:

> I haven't had a chance to look through the unresolved JIRAs but I did want
> to mention a few quick things.
>
> First, when we released 0.3.0 and dropped the BETA flag, one of the things
> that was discussed was putting together a method of documenting upgrades
> from one version to the next.  As one of the first steps toward making that
> a reasonable process, I think we should assemble more detailed release
> notes, especially outlining non-backwards compatible changes.  In the
> "[DISCUSS] Next Release Name" email thread Kyle Richardson suggested we use
> "UPGRADING.md" to do this, and I still agree with that thought, but I'm
> open to alternatives.
>
> Separately, a *nice to have* would be *METRON-660*, which was discussed in
> the "[PROPOSAL] up-to-date versioned documentation" thread, to give us some
> cleaner documentation using the existing READMEs.  I'd be happy to help
> with this one, I'm just not sure what the next steps are, aside from the
> start that Matt has here
> <https://github.com/mattf-horton/incubator-metron/tree/METRON-660>.
>
> My last *nice to have* is *METRON-635*, which I have a *PR open for here
> <https://github.com/apache/incubator-metron/pull/411>*.  If I could get
> someone else to reproduce the error that I'm seeing I would be happy to
> pursue additional testing, troubleshooting, etc.  I've seen others report
> the same issue
> <https://community.hortonworks.com/search.html?f=&type=question&redirect=
> search%2Fsearch&sort=relevance&q=scp_if_ssh>
> on the HCC boards, so I'm fairly confident that it is not an issue with my
> local environment.
>
> Jon
>
> On Thu, Jan 26, 2017 at 6:18 PM Casey Stella <ce...@gmail.com> wrote:
>
> I took the liberty of adding the pull requests since the new year into the
> in-progress list in the previous email.  If you own one of these and do not
> believe that you can complete the review in the next week or so, let me
> know and we can remove.  The only one of these that I see as mandatory is
> METRON-650 because the reliance on the opensoc github repo might cause
> issues with the release being acceptable (see discussion by Mike Miklavcic
> with the mentors a couple weeks ago).
>
> On Thu, Jan 26, 2017 at 6:06 PM, Casey Stella <ce...@gmail.com> wrote:
>
> > Hello Everyone!
> >
> > It's been almost 2 months since the last major release and I think it
> > might be time to do a minor release of 0.3.1. The purpose of this email
> is
> > multifold:
> >
> >    - to take stock of what we have already committed
> >    - figure out what (if anything) is missing to make a release that
> >    we're proud of
> >    - find volunteers for the missing items
> >
> > For those who get email alerts on the JIRA changes, it should be no
> > surprise that I have gone through and put the committed items in the
> 0.3.1
> > release bucket.  So, let's take stock of what we have.
> >
> > *What's made it so far into the next release*
> >
> >
> >
> >    - METRON-283 Migrate Geo Enrichment outside of MySQL (justinleet)
> >    closes apache/incubator-metron#421
> >    - METRON-668: Remove the "tickUpdate" profile config and make the
> >    "init" phase not reset variables closes apache/incubator-metron#420
> >    - METRON-672: SolrIndexingIntegrationTest fails intermittently closes
> >    apache/incubator-metron#424
> >    - METRON-664: Make the index configuration per-writer with
> >    enabled/disabled closes apache/incubator-metron#419
> >    - METRON-600: Fix Metron Website (iraghumitra via cestella) closes
> >    apache/incubator-metron#399
> >    - METRON-666 Fix javadoc doclint errors closes
> >    apache/incubator-metron#418
> >    - METRON-659 Emulate Sensors in Development Environments (nickwallen)
> >    closes apache/incubator-metron#417
> >    - METRON-652: Extract indexing config from enrichment config closes
> >    apache/incubator-metron#415
> >    - METRON-654 Create RPM Installer for Profiler (nickwallen) closes
> >    apache/incubator-metron#413
> >    - METRON-656: Make Stellar 'in' closer to functioning like python
> >    closes apache/incubator-metron#416
> >    - METRON-532 Define Profile Period When Calling PROFILE_GET closes
> >    apache/incubator-metron#414
> >    - METRON-624: Updated Comparison/Equality Evaluations in Stellar
> >    closes apache/incubator-metron#404
> >    - METRON-644 RPM builds only work with Docker for Mac (kylerichardson)
> >    closes apache/incubator-metron#409
> >    - METRON-640: Add a Stellar function to compute shannon entropy for
> >    strings closes apache/incubator-metron#403
> >    - METRON-637: Add a STATS_BIN function to Stellar. closes
> >    apache/incubator-metron#401
> >    - METRON-622 Create a Metron Docker Compose application  (merrimanr)
> >    closes apache/incubator-metron#393
> >    - METRON-643: Stellar function documentation needs to be updated
> >    closes apache/incubator-metron#407
> >    - METRON-645 Unable to Start Fastcapa Test Environment (nickwallen)
> >    closes apache/incubator-metron#410
> >    - METRON-647 Parser unit test failures due to assumed year values
> >    (justinleet) closes apache/incubator-metron#412
> >    - METRON-639: The Network Stellar functions need to have better unit
> >    testing closes apache/incubator-metron#402
> >    - METRON-631 Broken link on fastcapa README (JonZeolla via nickwallen)
> >    closes apache/incubator-metron#398
> >    - METRON-625: Parser Filters cannot be specified from the sensor
> >    config closes apache/incubator-metron#396
> >    - METRON-587 Integration tests should use common processor
> >    implementations where possible (ottobackwards) closes
> >    apache/incubator-metron#374
> >    - METRON-616: Added support for float and long literals in Stellar
> >    closes apache/incubator-metron#392
> >    - METRON-580: Remove hard-coded Metron version from Ambari MPack code
> >    (mmiklavc) closes apache/incubator-metron#364
> >    - METRON-618 Eliminate Javac Warnings in metron-analytics (justinleet)
> >    closes apache/incubator-metron#391
> >    - METRON-364: Preserve Type for Arithmetic Expressions in Stellar
> >    closes apache/incubator-metron#390
> >    - METRON-585 Pcap Replay Fails to Stop (nickwallen) closes
> >    apache/incubator-metron#369
> >    - METRON-586 STELLAR should have FILL_LEFT and FILL_RIGHT functions
> >    (ottobackwards) closes apache/incubator-metron#370
> >    - METRON-612 Clean up Error Prone generated warnings (justinleet)
> >    closes apache/incubator-metron#389
> >    - METRON-610: OnlineStatisticsProvider serialization is broken at
> >    random in the REPL closes apache/incubator-metron#388
> >    - METRON-596 Eliminate Maven warnings from build (justinleet) closes
> >    apache/incubator-metron#378
> >    - METRON-604 Mpack installs do not work on clean machines due to
> >    missing Elastic Curator repo (justinleet) closes
> apache/incubator-metron#385
> >    - METRON-607: Enrichment doc improvement and test cleanup (mmiklavc)
> >    closes apache/incubator-metron#386
> >    - METRON-606 Profiler Overwriting Previously Written Values
> >    (nickwallen) closes apache/incubator-metron#387
> >    - METRON-591: Make the website in compliance with ASF standards closes
> >    apache/incubator-metron#373
> >    - METRON-597 Sporadic Failures of Profiler Integration Tests
> >    (nickwallen) closes apache/incubator-metron#383
> >    - METRON-595 Elasticsearch Writer only uses One IP Address
> >    (JonathanRider via dlyle65535) closes apache/incubator-metron#379
> >    - METRON-593 Enable an automated static analysis tool in the build
> >    (justinleet) closes apache/incubator-metron#376
> >    - METRON-592 Change popup to Warning message in Assign Master page for
> >    Client check (anandsubbu via dlyle65535) closes
> apache/incubator-metron#375
> >    - METRON-598 Add Kyle Richardson to committers (kylerichardson) closes
> >    apache/incubator-metron#382
> >    - METRON-565: apps/metron/enrichment/indexed directory path does not
> >    get created for metron cluster deployed via Ambari (mmiklavc) closes
> >    apache/incubator-metron#365
> >    - METRON-588 Remove ASF License Headers from directly copied Chef
> >    Bento Files closes apache/incubator-metron#372
> >    - METRON-576 Stellar function resolution takes too long on running
> >    cluster (nickwallen) closes apache/incubator-metron#366
> >    - METRON-570 Add Profiler Link to README (nickwallen) closes
> >    apache/incubator-metron#371
> >    - METRON-578 Missing error handling bolts for enrichment and threat
> >    intel closes apache/incubator-metron#363
> >    - METRON-522 Need to mandate Client installation on Metron Host closes
> >    apache/incubator-metron#367
> >    - METRON-584 STELLAR should have a TO_LONG Function (ottobackwards)
> >    closes apache/incubator-metron#368
> >    - METRON-575 State from different profiles can be co-mingled
> >    incorrectly (nickwallen) closes apache/incubator-metron#362
> >    - METRON-538 Ensure proper shutdown, exceptions of integration test
> >    components (ottobackwards) closes apache/incubator-metron#344
> >    - METRON-567 Usernames as numerics strings attempted to be parsed and
> >    compared as numbers (ottobackwards) closes apache/incubator-metron#360
> >    - METRON-562: Add rudimentary statistical outlier detection closes
> >    apache/incubator-metron#352
> >    - METRON-557 Create Stellar Functions for Kafka (nickwallen) closes
> >    apache/incubator-metron#354
> >    - METRON-556 Profiler - Refactor 'Group By' Calculation (nickwallen)
> >    closes apache/incubator-metron#348
> >    - METRON-547 MaasIntegrationTests should integrate with the Metron
> >    Integration classes (ottobackwards) closes apache/incubator-metron#350
> >    - METRON-561 ShellEditor tests hang if vim is set as EDITOR in your
> >    profile (ottobackwards) closes apache/incubator-metron#351
> >
> > *What else will make it?*
> >
> > Ok, so now that we know what's in there already, I need to ask your help
> > in determining what you want in the next release that is not currently
> > being worked on.  In order to do that, please help me by responding to
> this
> > email and
> >
> >    - proposing and justifying a JIRA from the list of unresolved JIRAs
> >    <
> https://issues.apache.org/jira/browse/METRON-469?jql=
> project%20%3D%20METRON%20AND%20resolution%20%3D%20Unresolved%20AND%
> 20fixVersion%20is%20EMPTY%20ORDER%20BY%20priority%20DESC>
> that
> >    are not currently scheduled and let me know which ones should be
> included.
> >    Please include whether the JIRA is "Nice to Have" or "Critical to
> Have" and
> >    if you volunteer to do it.
> >    - proposing and justifying that one of the in-progress tasks
> >    <
> https://issues.apache.org/jira/browse/METRON-650?jql=
> project%20%3D%20METRON%20AND%20resolution%20%3D%20Unresolved%20AND%
> 20fixVersion%20%3D%200.3.1%20ORDER%20BY%20priority%20DESC
> >
> >     should *not* be included for release
> >
> > I'd like to aim to get release candidates out as soon as possible, so
> > let's look forward a vigorous discussion and a great release.
> >
> > Thanks for the contributions and for your continued engagement in the
> > Metron community.  If you see something that you don't like in this
> > process, please let me know.  Feedback is very welcome.
> >
> > Best,
> >
> > Casey
> >
>
> --
>
> Jon
>
> Sent from my mobile device
>

Re: [DISCUSS] Next Release (0.3.1) Content

Posted by "Zeolla@GMail.com" <ze...@gmail.com>.
I haven't had a chance to look through the unresolved JIRAs but I did want
to mention a few quick things.

First, when we released 0.3.0 and dropped the BETA flag, one of the things
that was discussed was putting together a method of documenting upgrades
from one version to the next.  As one of the first steps toward making that
a reasonable process, I think we should assemble more detailed release
notes, especially outlining non-backwards compatible changes.  In the
"[DISCUSS] Next Release Name" email thread Kyle Richardson suggested we use
"UPGRADING.md" to do this, and I still agree with that thought, but I'm
open to alternatives.

Separately, a *nice to have* would be *METRON-660*, which was discussed in
the "[PROPOSAL] up-to-date versioned documentation" thread, to give us some
cleaner documentation using the existing READMEs.  I'd be happy to help
with this one, I'm just not sure what the next steps are, aside from the
start that Matt has here
<https://github.com/mattf-horton/incubator-metron/tree/METRON-660>.

My last *nice to have* is *METRON-635*, which I have a *PR open for here
<https://github.com/apache/incubator-metron/pull/411>*.  If I could get
someone else to reproduce the error that I'm seeing I would be happy to
pursue additional testing, troubleshooting, etc.  I've seen others report
the same issue
<https://community.hortonworks.com/search.html?f=&type=question&redirect=search%2Fsearch&sort=relevance&q=scp_if_ssh>
on the HCC boards, so I'm fairly confident that it is not an issue with my
local environment.

Jon

On Thu, Jan 26, 2017 at 6:18 PM Casey Stella <ce...@gmail.com> wrote:

I took the liberty of adding the pull requests since the new year into the
in-progress list in the previous email.  If you own one of these and do not
believe that you can complete the review in the next week or so, let me
know and we can remove.  The only one of these that I see as mandatory is
METRON-650 because the reliance on the opensoc github repo might cause
issues with the release being acceptable (see discussion by Mike Miklavcic
with the mentors a couple weeks ago).

On Thu, Jan 26, 2017 at 6:06 PM, Casey Stella <ce...@gmail.com> wrote:

> Hello Everyone!
>
> It's been almost 2 months since the last major release and I think it
> might be time to do a minor release of 0.3.1. The purpose of this email is
> multifold:
>
>    - to take stock of what we have already committed
>    - figure out what (if anything) is missing to make a release that
>    we're proud of
>    - find volunteers for the missing items
>
> For those who get email alerts on the JIRA changes, it should be no
> surprise that I have gone through and put the committed items in the 0.3.1
> release bucket.  So, let's take stock of what we have.
>
> *What's made it so far into the next release*
>
>
>
>    - METRON-283 Migrate Geo Enrichment outside of MySQL (justinleet)
>    closes apache/incubator-metron#421
>    - METRON-668: Remove the "tickUpdate" profile config and make the
>    "init" phase not reset variables closes apache/incubator-metron#420
>    - METRON-672: SolrIndexingIntegrationTest fails intermittently closes
>    apache/incubator-metron#424
>    - METRON-664: Make the index configuration per-writer with
>    enabled/disabled closes apache/incubator-metron#419
>    - METRON-600: Fix Metron Website (iraghumitra via cestella) closes
>    apache/incubator-metron#399
>    - METRON-666 Fix javadoc doclint errors closes
>    apache/incubator-metron#418
>    - METRON-659 Emulate Sensors in Development Environments (nickwallen)
>    closes apache/incubator-metron#417
>    - METRON-652: Extract indexing config from enrichment config closes
>    apache/incubator-metron#415
>    - METRON-654 Create RPM Installer for Profiler (nickwallen) closes
>    apache/incubator-metron#413
>    - METRON-656: Make Stellar 'in' closer to functioning like python
>    closes apache/incubator-metron#416
>    - METRON-532 Define Profile Period When Calling PROFILE_GET closes
>    apache/incubator-metron#414
>    - METRON-624: Updated Comparison/Equality Evaluations in Stellar
>    closes apache/incubator-metron#404
>    - METRON-644 RPM builds only work with Docker for Mac (kylerichardson)
>    closes apache/incubator-metron#409
>    - METRON-640: Add a Stellar function to compute shannon entropy for
>    strings closes apache/incubator-metron#403
>    - METRON-637: Add a STATS_BIN function to Stellar. closes
>    apache/incubator-metron#401
>    - METRON-622 Create a Metron Docker Compose application  (merrimanr)
>    closes apache/incubator-metron#393
>    - METRON-643: Stellar function documentation needs to be updated
>    closes apache/incubator-metron#407
>    - METRON-645 Unable to Start Fastcapa Test Environment (nickwallen)
>    closes apache/incubator-metron#410
>    - METRON-647 Parser unit test failures due to assumed year values
>    (justinleet) closes apache/incubator-metron#412
>    - METRON-639: The Network Stellar functions need to have better unit
>    testing closes apache/incubator-metron#402
>    - METRON-631 Broken link on fastcapa README (JonZeolla via nickwallen)
>    closes apache/incubator-metron#398
>    - METRON-625: Parser Filters cannot be specified from the sensor
>    config closes apache/incubator-metron#396
>    - METRON-587 Integration tests should use common processor
>    implementations where possible (ottobackwards) closes
>    apache/incubator-metron#374
>    - METRON-616: Added support for float and long literals in Stellar
>    closes apache/incubator-metron#392
>    - METRON-580: Remove hard-coded Metron version from Ambari MPack code
>    (mmiklavc) closes apache/incubator-metron#364
>    - METRON-618 Eliminate Javac Warnings in metron-analytics (justinleet)
>    closes apache/incubator-metron#391
>    - METRON-364: Preserve Type for Arithmetic Expressions in Stellar
>    closes apache/incubator-metron#390
>    - METRON-585 Pcap Replay Fails to Stop (nickwallen) closes
>    apache/incubator-metron#369
>    - METRON-586 STELLAR should have FILL_LEFT and FILL_RIGHT functions
>    (ottobackwards) closes apache/incubator-metron#370
>    - METRON-612 Clean up Error Prone generated warnings (justinleet)
>    closes apache/incubator-metron#389
>    - METRON-610: OnlineStatisticsProvider serialization is broken at
>    random in the REPL closes apache/incubator-metron#388
>    - METRON-596 Eliminate Maven warnings from build (justinleet) closes
>    apache/incubator-metron#378
>    - METRON-604 Mpack installs do not work on clean machines due to
>    missing Elastic Curator repo (justinleet) closes
apache/incubator-metron#385
>    - METRON-607: Enrichment doc improvement and test cleanup (mmiklavc)
>    closes apache/incubator-metron#386
>    - METRON-606 Profiler Overwriting Previously Written Values
>    (nickwallen) closes apache/incubator-metron#387
>    - METRON-591: Make the website in compliance with ASF standards closes
>    apache/incubator-metron#373
>    - METRON-597 Sporadic Failures of Profiler Integration Tests
>    (nickwallen) closes apache/incubator-metron#383
>    - METRON-595 Elasticsearch Writer only uses One IP Address
>    (JonathanRider via dlyle65535) closes apache/incubator-metron#379
>    - METRON-593 Enable an automated static analysis tool in the build
>    (justinleet) closes apache/incubator-metron#376
>    - METRON-592 Change popup to Warning message in Assign Master page for
>    Client check (anandsubbu via dlyle65535) closes
apache/incubator-metron#375
>    - METRON-598 Add Kyle Richardson to committers (kylerichardson) closes
>    apache/incubator-metron#382
>    - METRON-565: apps/metron/enrichment/indexed directory path does not
>    get created for metron cluster deployed via Ambari (mmiklavc) closes
>    apache/incubator-metron#365
>    - METRON-588 Remove ASF License Headers from directly copied Chef
>    Bento Files closes apache/incubator-metron#372
>    - METRON-576 Stellar function resolution takes too long on running
>    cluster (nickwallen) closes apache/incubator-metron#366
>    - METRON-570 Add Profiler Link to README (nickwallen) closes
>    apache/incubator-metron#371
>    - METRON-578 Missing error handling bolts for enrichment and threat
>    intel closes apache/incubator-metron#363
>    - METRON-522 Need to mandate Client installation on Metron Host closes
>    apache/incubator-metron#367
>    - METRON-584 STELLAR should have a TO_LONG Function (ottobackwards)
>    closes apache/incubator-metron#368
>    - METRON-575 State from different profiles can be co-mingled
>    incorrectly (nickwallen) closes apache/incubator-metron#362
>    - METRON-538 Ensure proper shutdown, exceptions of integration test
>    components (ottobackwards) closes apache/incubator-metron#344
>    - METRON-567 Usernames as numerics strings attempted to be parsed and
>    compared as numbers (ottobackwards) closes apache/incubator-metron#360
>    - METRON-562: Add rudimentary statistical outlier detection closes
>    apache/incubator-metron#352
>    - METRON-557 Create Stellar Functions for Kafka (nickwallen) closes
>    apache/incubator-metron#354
>    - METRON-556 Profiler - Refactor 'Group By' Calculation (nickwallen)
>    closes apache/incubator-metron#348
>    - METRON-547 MaasIntegrationTests should integrate with the Metron
>    Integration classes (ottobackwards) closes apache/incubator-metron#350
>    - METRON-561 ShellEditor tests hang if vim is set as EDITOR in your
>    profile (ottobackwards) closes apache/incubator-metron#351
>
> *What else will make it?*
>
> Ok, so now that we know what's in there already, I need to ask your help
> in determining what you want in the next release that is not currently
> being worked on.  In order to do that, please help me by responding to
this
> email and
>
>    - proposing and justifying a JIRA from the list of unresolved JIRAs
>    <
https://issues.apache.org/jira/browse/METRON-469?jql=project%20%3D%20METRON%20AND%20resolution%20%3D%20Unresolved%20AND%20fixVersion%20is%20EMPTY%20ORDER%20BY%20priority%20DESC>
that
>    are not currently scheduled and let me know which ones should be
included.
>    Please include whether the JIRA is "Nice to Have" or "Critical to
Have" and
>    if you volunteer to do it.
>    - proposing and justifying that one of the in-progress tasks
>    <
https://issues.apache.org/jira/browse/METRON-650?jql=project%20%3D%20METRON%20AND%20resolution%20%3D%20Unresolved%20AND%20fixVersion%20%3D%200.3.1%20ORDER%20BY%20priority%20DESC
>
>     should *not* be included for release
>
> I'd like to aim to get release candidates out as soon as possible, so
> let's look forward a vigorous discussion and a great release.
>
> Thanks for the contributions and for your continued engagement in the
> Metron community.  If you see something that you don't like in this
> process, please let me know.  Feedback is very welcome.
>
> Best,
>
> Casey
>

-- 

Jon

Sent from my mobile device

Re: [DISCUSS] Next Release (0.3.1) Content

Posted by Casey Stella <ce...@gmail.com>.
I took the liberty of adding the pull requests since the new year into the
in-progress list in the previous email.  If you own one of these and do not
believe that you can complete the review in the next week or so, let me
know and we can remove.  The only one of these that I see as mandatory is
METRON-650 because the reliance on the opensoc github repo might cause
issues with the release being acceptable (see discussion by Mike Miklavcic
with the mentors a couple weeks ago).

On Thu, Jan 26, 2017 at 6:06 PM, Casey Stella <ce...@gmail.com> wrote:

> Hello Everyone!
>
> It's been almost 2 months since the last major release and I think it
> might be time to do a minor release of 0.3.1. The purpose of this email is
> multifold:
>
>    - to take stock of what we have already committed
>    - figure out what (if anything) is missing to make a release that
>    we're proud of
>    - find volunteers for the missing items
>
> For those who get email alerts on the JIRA changes, it should be no
> surprise that I have gone through and put the committed items in the 0.3.1
> release bucket.  So, let's take stock of what we have.
>
> *What's made it so far into the next release*
>
>
>
>    - METRON-283 Migrate Geo Enrichment outside of MySQL (justinleet)
>    closes apache/incubator-metron#421
>    - METRON-668: Remove the "tickUpdate" profile config and make the
>    "init" phase not reset variables closes apache/incubator-metron#420
>    - METRON-672: SolrIndexingIntegrationTest fails intermittently closes
>    apache/incubator-metron#424
>    - METRON-664: Make the index configuration per-writer with
>    enabled/disabled closes apache/incubator-metron#419
>    - METRON-600: Fix Metron Website (iraghumitra via cestella) closes
>    apache/incubator-metron#399
>    - METRON-666 Fix javadoc doclint errors closes
>    apache/incubator-metron#418
>    - METRON-659 Emulate Sensors in Development Environments (nickwallen)
>    closes apache/incubator-metron#417
>    - METRON-652: Extract indexing config from enrichment config closes
>    apache/incubator-metron#415
>    - METRON-654 Create RPM Installer for Profiler (nickwallen) closes
>    apache/incubator-metron#413
>    - METRON-656: Make Stellar 'in' closer to functioning like python
>    closes apache/incubator-metron#416
>    - METRON-532 Define Profile Period When Calling PROFILE_GET closes
>    apache/incubator-metron#414
>    - METRON-624: Updated Comparison/Equality Evaluations in Stellar
>    closes apache/incubator-metron#404
>    - METRON-644 RPM builds only work with Docker for Mac (kylerichardson)
>    closes apache/incubator-metron#409
>    - METRON-640: Add a Stellar function to compute shannon entropy for
>    strings closes apache/incubator-metron#403
>    - METRON-637: Add a STATS_BIN function to Stellar. closes
>    apache/incubator-metron#401
>    - METRON-622 Create a Metron Docker Compose application  (merrimanr)
>    closes apache/incubator-metron#393
>    - METRON-643: Stellar function documentation needs to be updated
>    closes apache/incubator-metron#407
>    - METRON-645 Unable to Start Fastcapa Test Environment (nickwallen)
>    closes apache/incubator-metron#410
>    - METRON-647 Parser unit test failures due to assumed year values
>    (justinleet) closes apache/incubator-metron#412
>    - METRON-639: The Network Stellar functions need to have better unit
>    testing closes apache/incubator-metron#402
>    - METRON-631 Broken link on fastcapa README (JonZeolla via nickwallen)
>    closes apache/incubator-metron#398
>    - METRON-625: Parser Filters cannot be specified from the sensor
>    config closes apache/incubator-metron#396
>    - METRON-587 Integration tests should use common processor
>    implementations where possible (ottobackwards) closes
>    apache/incubator-metron#374
>    - METRON-616: Added support for float and long literals in Stellar
>    closes apache/incubator-metron#392
>    - METRON-580: Remove hard-coded Metron version from Ambari MPack code
>    (mmiklavc) closes apache/incubator-metron#364
>    - METRON-618 Eliminate Javac Warnings in metron-analytics (justinleet)
>    closes apache/incubator-metron#391
>    - METRON-364: Preserve Type for Arithmetic Expressions in Stellar
>    closes apache/incubator-metron#390
>    - METRON-585 Pcap Replay Fails to Stop (nickwallen) closes
>    apache/incubator-metron#369
>    - METRON-586 STELLAR should have FILL_LEFT and FILL_RIGHT functions
>    (ottobackwards) closes apache/incubator-metron#370
>    - METRON-612 Clean up Error Prone generated warnings (justinleet)
>    closes apache/incubator-metron#389
>    - METRON-610: OnlineStatisticsProvider serialization is broken at
>    random in the REPL closes apache/incubator-metron#388
>    - METRON-596 Eliminate Maven warnings from build (justinleet) closes
>    apache/incubator-metron#378
>    - METRON-604 Mpack installs do not work on clean machines due to
>    missing Elastic Curator repo (justinleet) closes apache/incubator-metron#385
>    - METRON-607: Enrichment doc improvement and test cleanup (mmiklavc)
>    closes apache/incubator-metron#386
>    - METRON-606 Profiler Overwriting Previously Written Values
>    (nickwallen) closes apache/incubator-metron#387
>    - METRON-591: Make the website in compliance with ASF standards closes
>    apache/incubator-metron#373
>    - METRON-597 Sporadic Failures of Profiler Integration Tests
>    (nickwallen) closes apache/incubator-metron#383
>    - METRON-595 Elasticsearch Writer only uses One IP Address
>    (JonathanRider via dlyle65535) closes apache/incubator-metron#379
>    - METRON-593 Enable an automated static analysis tool in the build
>    (justinleet) closes apache/incubator-metron#376
>    - METRON-592 Change popup to Warning message in Assign Master page for
>    Client check (anandsubbu via dlyle65535) closes apache/incubator-metron#375
>    - METRON-598 Add Kyle Richardson to committers (kylerichardson) closes
>    apache/incubator-metron#382
>    - METRON-565: apps/metron/enrichment/indexed directory path does not
>    get created for metron cluster deployed via Ambari (mmiklavc) closes
>    apache/incubator-metron#365
>    - METRON-588 Remove ASF License Headers from directly copied Chef
>    Bento Files closes apache/incubator-metron#372
>    - METRON-576 Stellar function resolution takes too long on running
>    cluster (nickwallen) closes apache/incubator-metron#366
>    - METRON-570 Add Profiler Link to README (nickwallen) closes
>    apache/incubator-metron#371
>    - METRON-578 Missing error handling bolts for enrichment and threat
>    intel closes apache/incubator-metron#363
>    - METRON-522 Need to mandate Client installation on Metron Host closes
>    apache/incubator-metron#367
>    - METRON-584 STELLAR should have a TO_LONG Function (ottobackwards)
>    closes apache/incubator-metron#368
>    - METRON-575 State from different profiles can be co-mingled
>    incorrectly (nickwallen) closes apache/incubator-metron#362
>    - METRON-538 Ensure proper shutdown, exceptions of integration test
>    components (ottobackwards) closes apache/incubator-metron#344
>    - METRON-567 Usernames as numerics strings attempted to be parsed and
>    compared as numbers (ottobackwards) closes apache/incubator-metron#360
>    - METRON-562: Add rudimentary statistical outlier detection closes
>    apache/incubator-metron#352
>    - METRON-557 Create Stellar Functions for Kafka (nickwallen) closes
>    apache/incubator-metron#354
>    - METRON-556 Profiler - Refactor 'Group By' Calculation (nickwallen)
>    closes apache/incubator-metron#348
>    - METRON-547 MaasIntegrationTests should integrate with the Metron
>    Integration classes (ottobackwards) closes apache/incubator-metron#350
>    - METRON-561 ShellEditor tests hang if vim is set as EDITOR in your
>    profile (ottobackwards) closes apache/incubator-metron#351
>
> *What else will make it?*
>
> Ok, so now that we know what's in there already, I need to ask your help
> in determining what you want in the next release that is not currently
> being worked on.  In order to do that, please help me by responding to this
> email and
>
>    - proposing and justifying a JIRA from the list of unresolved JIRAs
>    <https://issues.apache.org/jira/browse/METRON-469?jql=project%20%3D%20METRON%20AND%20resolution%20%3D%20Unresolved%20AND%20fixVersion%20is%20EMPTY%20ORDER%20BY%20priority%20DESC> that
>    are not currently scheduled and let me know which ones should be included.
>    Please include whether the JIRA is "Nice to Have" or "Critical to Have" and
>    if you volunteer to do it.
>    - proposing and justifying that one of the in-progress tasks
>    <https://issues.apache.org/jira/browse/METRON-650?jql=project%20%3D%20METRON%20AND%20resolution%20%3D%20Unresolved%20AND%20fixVersion%20%3D%200.3.1%20ORDER%20BY%20priority%20DESC>
>     should *not* be included for release
>
> I'd like to aim to get release candidates out as soon as possible, so
> let's look forward a vigorous discussion and a great release.
>
> Thanks for the contributions and for your continued engagement in the
> Metron community.  If you see something that you don't like in this
> process, please let me know.  Feedback is very welcome.
>
> Best,
>
> Casey
>