You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@metron.apache.org by justinleet <gi...@git.apache.org> on 2017/02/20 06:23:37 UTC

[GitHub] incubator-metron pull request #459: METRON-726: Clean up mvn site generation

GitHub user justinleet opened a pull request:

    https://github.com/apache/incubator-metron/pull/459

    METRON-726: Clean up mvn site generation

    ### Summary
    An initial attempt to add checkstyle expanded a bit once it turned out that `mvn site` appears to not work right now (and probably hasn't for a long while). More background on some of the issues encountered (and error traces) is in the ticket itself.
    
    A small set of cleanups is included here, and is pretty easy to split out if we want it to be separate PRs (it mostly just got cleaned up while I was trying to fix things one by one, but for example, checkstyle could be included in a separate PR). It's small enough though, that I just left it together since it makes the generated site fairly nice and complete.
    
    ### Changes
    - The major change is to drop EMMA for JaCoCo. As noted in the ticket, it's unlikely to be worth the erffort to get EMMA working, given that it's not actively developed and it seems likely that EMMA just isn't compatible with Java 8 code (not terribly surprising given how old it is). I put in JaCoCo mostly to keep (make?) code coverage available, but I can easily remove it if we'd rather have a discussion on a different tool, or just a separate PR.
    - Added maven-site-plugin to top level pom
    - Refactored most reporting (maven-project-info-reports-plugin, maven-pmd-plugin, and code coverage) to go through top level pom, given that most of our reporting is exactly the same throughout the child poms.
    - Explicitly setup up maven-project-info-report-plugin reports (neither CLI param nor POM param skipped dependency reports correctly. Unsure why that is. If someone knows, I'd love to just change that)
    - Added maven-jxr-plugin to fix warnings in the site build
    - Added names to poms to make sure they show up nicely in site
    - Deploy set to /tmp/metron/site for now. Should eventually be adjusted to be somewhere useful (I believe other components like Hadoop do something nicer with it within Apache, but I'd need to look into it). I'll create a ticket for it, assuming we're good with this PR (mostly) as-is. Otherwise, I'll wait and see what spins out of discussion here.
    - Updated PMD to 3.7
    
    #### Testing
    To run everything, including code coverage up:
    `mvn clean install site:site site:deploy site:stage-deploy`
    
    Open up /tmp/metron/site/index.html in a browser. Everything (other than Javadoc) should be available, and clicking through modules should work because of the `site:stage-deploy`
    
    Skipping tests allows code coverage to be skipped.
    
    I also ran up quick-dev and made sure data passed through topologies. I didn't run up other components, because mostly the reporting section of the poms is touched, so if they built and the reporting built it seemed like enough. I can definitely spin up other components if anybody wants to be more thorough, though.
    
    #### Notes
    - Javadocs are currently broken outside of this ticket. Separate Jira/PR is open for that issue.
    - Reason for `site:stage-deploy` to is get the parent-child pom links working while testing. Otherwise, you can't navigate through the module structure.
    - Currently almost 25k(!) checkstyle issues.
    - It's possible to import a check style file in IntelliJ. I could export this for general use (modified it to use two spaces + possibly things like line length). I tested reformatting all Java files like this, and it clears out several thousand checkstyle issues. And makes the code more consistent. Obviously, that's a bit aggressive and touched things like autogenerated files, but it makes it clear we have more issues than trivially solved by code formatting.
    
    
    
    


You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/justinleet/incubator-metron site_fix

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/incubator-metron/pull/459.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #459
    
----
commit 7028066c3e2cd3823aa15cf7972724e5f61ca05c
Author: justinjleet <ju...@gmail.com>
Date:   2017-02-20T02:00:13Z

    Moving away from emma to jacoco and fixing some other issues

commit f56e8bbbc0863a6a535e663594d049620d53c7d6
Author: justinjleet <ju...@gmail.com>
Date:   2017-02-20T02:44:51Z

    Fixing up and moving PMD upward in POM

----


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron issue #459: METRON-726: Clean up mvn site generation

Posted by justinleet <gi...@git.apache.org>.

Github user justinleet commented on the issue:

https://github.com/apache/incubator-metron/pull/459

@dlyle65535, @mmiklavc, @mattf-horton Update here, I'm interested in your thoughts, given the following.

This has been sitting here awhile and I've been trying to play with it to resolve issues, but I'm honestly not sure I can easily resolve them. Errors are intermittent and inconsistent. To complicate matters, this appears to be the case on master as well.

I've been doing a pretty simple `while true; do date; mvn clean install; sleep 5; done >> ~/log_file 2>&1 ` to just run the build in a loop and see what comes up. It's possible some of this is an artifact of running it in a loop like this, and I can definitely run it up in a Travis repeatedly to see if the same problems show (and I haven't seen Travis issues in a while that appear to be this branch's issue). Having said that, it's worrying that our build is (very) unstable on both this branch and master when run like this.

A lot of these seem to be related to our integration tests with the in memory components. I don't see anything obviously wrong (although I did add some code to make sure ComponentRunners actually get cleaned up in finallys which may help a bit, but didn't resolve it).

I'm not at all sure how to move forward on this, given that the state of master itself makes it almost impossible (unless someone has a clever idea!) to know if each issue is in this branch or not since every issue is intermittent.

---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #459: METRON-726: Clean up mvn site generation

Posted by justinleet <gi...@git.apache.org>.

GitHub user justinleet reopened a pull request:

    https://github.com/apache/incubator-metron/pull/459

    METRON-726: Clean up mvn site generation

    ### Summary
    An initial attempt to add checkstyle expanded a bit once it turned out that `mvn site` appears to not work right now (and probably hasn't for a long while). More background on some of the issues encountered (and error traces) is in the ticket itself.
    
    A small set of cleanups is included here, and is pretty easy to split out if we want it to be separate PRs (it mostly just got cleaned up while I was trying to fix things one by one, but for example, checkstyle could be included in a separate PR). It's small enough though, that I just left it together since it makes the generated site fairly nice and complete.
    
    ### Changes
    - The major change is to drop EMMA for JaCoCo. As noted in the ticket, it's unlikely to be worth the erffort to get EMMA working, given that it's not actively developed and it seems likely that EMMA just isn't compatible with Java 8 code (not terribly surprising given how old it is). I put in JaCoCo mostly to keep (make?) code coverage available, but I can easily remove it if we'd rather have a discussion on a different tool, or just a separate PR.
    - Added maven-site-plugin to top level pom
    - Refactored most reporting (maven-project-info-reports-plugin, maven-pmd-plugin, and code coverage) to go through top level pom, given that most of our reporting is exactly the same throughout the child poms.
    - Explicitly setup up maven-project-info-report-plugin reports (neither CLI param nor POM param skipped dependency reports correctly. Unsure why that is. If someone knows, I'd love to just change that)
    - Added maven-jxr-plugin to fix warnings in the site build
    - Added names to poms to make sure they show up nicely in site
    - Deploy set to /tmp/metron/site for now. Should eventually be adjusted to be somewhere useful (I believe other components like Hadoop do something nicer with it within Apache, but I'd need to look into it). I'll create a ticket for it, assuming we're good with this PR (mostly) as-is. Otherwise, I'll wait and see what spins out of discussion here.
    - Updated PMD to 3.7
    
    #### Testing
    To run everything, including code coverage up:
    `mvn clean install site:site site:deploy site:stage-deploy`
    
    Open up /tmp/metron/site/index.html in a browser. Everything (other than Javadoc) should be available, and clicking through modules should work because of the `site:stage-deploy`
    
    Skipping tests allows code coverage to be skipped.
    
    I also ran up quick-dev and made sure data passed through topologies. I didn't run up other components, because mostly the reporting section of the poms is touched, so if they built and the reporting built it seemed like enough. I can definitely spin up other components if anybody wants to be more thorough, though.
    
    #### Notes
    - Javadocs are currently broken outside of this ticket. Separate Jira/PR is open for that issue.
    - Reason for `site:stage-deploy` to is get the parent-child pom links working while testing. Otherwise, you can't navigate through the module structure.
    - Currently almost 25k(!) checkstyle issues.
    - It's possible to import a check style file in IntelliJ. I could export this for general use (modified it to use two spaces + possibly things like line length). I tested reformatting all Java files like this, and it clears out several thousand checkstyle issues. And makes the code more consistent. Obviously, that's a bit aggressive and touched things like autogenerated files, but it makes it clear we have more issues than trivially solved by code formatting.
    
    
    
    


You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/justinleet/incubator-metron site_fix

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/incubator-metron/pull/459.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #459
    
----
commit 9d219ca54ff3eaeee51759866e993dff4c6d8745
Author: justinjleet <ju...@gmail.com>
Date:   2017-02-20T02:00:13Z

    Moving away from emma to jacoco and fixing some other issues

commit deb6e9587dfa3ad8814202c224dc2b68ef67a288
Author: justinjleet <ju...@gmail.com>
Date:   2017-02-20T02:44:51Z

    Fixing up and moving PMD upward in POM

commit cf585526bb6952cd8848dd475951bc5f093944bf
Author: justinjleet <ju...@gmail.com>
Date:   2017-03-08T14:13:15Z

    Updated metron-interface to not use Emma, and adding Javadoc reporting.  Fixing fatal javadoc error

commit a999f79d18e4f1a3185a91945df2cd278c39b041
Author: justinjleet <ju...@gmail.com>
Date:   2017-03-08T18:07:22Z

    Running JaCoCo's prepare-agent in Travis commands to avoid surefire issues.  Setting surefire version consistently.

commit 448eb70cd0cc7d572a12276d365b63685042f8c5
Author: justinjleet <ju...@gmail.com>
Date:   2017-03-08T20:34:42Z

    Pulling surefire up to top level pom

commit c8954b6f58c1418c46977b1ba710a96221b59ede
Author: justinjleet <ju...@gmail.com>
Date:   2017-03-18T19:43:07Z

    Dropping surefire version because of issues with 2.19.1

commit 1f6d55cb4f0caeeaaf55612e17ae40bd04684677
Author: justinjleet <ju...@gmail.com>
Date:   2017-04-15T15:29:45Z

    clean up from rebase

commit 2f0271a4a197328910b696c98378956a061c8633
Author: justinjleet <ju...@gmail.com>
Date:   2017-04-15T15:38:34Z

    Updating new modules

commit 06c1cbe24698aec602bceaa66547c571f8378a26
Author: justinjleet <ju...@gmail.com>
Date:   2017-04-15T15:43:45Z

    removing ignored maxpermsize

commit 720288768983c58a7b07936882adb3403012269d
Author: justinjleet <ju...@gmail.com>
Date:   2017-04-18T14:04:56Z

    Ensuring ComponentRunners get caught in finallys

commit 770f6b62276f54ab96ef367342c44e8b49171297
Author: justinjleet <ju...@gmail.com>
Date:   2017-04-27T19:40:00Z

    Removing unused Scala import that caused site issues

commit 6ef484b31bff2b909373ad647e3ac82808d9eada
Author: justinjleet <ju...@gmail.com>
Date:   2017-04-29T01:20:25Z

    Correcting URL on child poms

commit 394ec27305aa7651334febe6814854359ac7c928
Author: justinjleet <ju...@gmail.com>
Date:   2017-04-30T11:22:33Z

    Adding generate-md.sh to the pre-site portion of the lifecycle and updating the PR template

----


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron issue #459: METRON-726: Clean up mvn site generation

Posted by mattf-horton <gi...@git.apache.org>.

Github user mattf-horton commented on the issue:

    https://github.com/apache/incubator-metron/pull/459
  
    I've opened METRON-759 to track the need to get javadoc running nightly and integrate with the public site.  Now that METRON-666 and METRON-725 are in 0.3.1, what remains to be fixed, other than the integration work?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron issue #459: METRON-726: Clean up mvn site generation

Posted by justinleet <gi...@git.apache.org>.

Github user justinleet commented on the issue:

    https://github.com/apache/incubator-metron/pull/459
  
    Discussion thread has been created on the dev list, per @cestella's suggestion.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

Re: [GitHub] incubator-metron issue #459: METRON-726: Clean up mvn site generation

Posted by Matt Foley <ma...@apache.org>.

Hi Justin,
If you change the maven invocation to build the site-book, please also change the README at https://github.com/apache/incubator-metron/tree/master/site-book

And is the new invocation done from <base dir>/site-book/ as previously, or from base dir?

Thanks,
--Matt

On 4/30/17, 6:12 AM, "justinleet" <gi...@git.apache.org> wrote:

Github user justinleet commented on the issue:

https://github.com/apache/incubator-metron/pull/459

Ran this up for a little while in down time over the last couple days. These counts include the previous updates (so 2 additional failures and 15 successes, and 1 additional failure and 25 successes for this branch). In both cases, the additional error was the PcapTopologyIntegrationTest test. I'll add in 1 failure for this branch since I had to kick the Travis build.

Master:
6 BUILD FAILURE
23 BUILD SUCCESS

Branch
2 BUILD FAILURE
42 BUILD SUCCESS

At this point, I think this satisfies the criteria of not being worse than master.

I've also run up full dev again (since there've been a few things changed and cleaned up, and so on), and data flowed through as expected.

I did make another change. I was spinning up the site-book, but not running the generate-md.sh. I added it to pre-site portion of the lifecycle. Admittedly, this does mean that running the site itself requires bash, but it seems like a reasonable requirement (particularly since this is only for the site).

At this point, that does change the test run slightly: It's `mvn clean install site site:deploy site:stage-deploy` (instead of site:site, it should be just plain site to ensure site:pre-site is run). Tests can be skipped with -DskipTests and code coverage simply won't run (and won't appear in the reports).

[GitHub] incubator-metron issue #459: METRON-726: Clean up mvn site generation

Posted by justinleet <gi...@git.apache.org>.

Github user justinleet commented on the issue:

https://github.com/apache/incubator-metron/pull/459

Master:
6 BUILD FAILURE
23 BUILD SUCCESS

Branch
2 BUILD FAILURE
42 BUILD SUCCESS

At this point, I think this satisfies the criteria of not being worse than master.

I've also run up full dev again (since there've been a few things changed and cleaned up, and so on), and data flowed through as expected.

[GitHub] incubator-metron issue #459: METRON-726: Clean up mvn site generation

Posted by justinleet <gi...@git.apache.org>.

Github user justinleet commented on the issue:

    https://github.com/apache/incubator-metron/pull/459
  
    @mmiklavc That order is correct, yes.  Updated the main PR with a strikethrough and replace of the original.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron issue #459: METRON-726: Clean up mvn site generation

Posted by mmiklavc <gi...@git.apache.org>.

Github user mmiklavc commented on the issue:

    https://github.com/apache/incubator-metron/pull/459
  
    Running it now. @justinleet can you update your main PR comment to reflect the new command specified in these comments?
    
    > 
    > At this point, that does change the test run slightly: It's mvn clean install site site:deploy site:stage-deploy (instead of site:site, it should be just plain site to ensure site:pre-site is run).
    > 
    > I split out the mvn clean install from mvn site site:stage-deploy site:deploy
    
    Just so we have a record of things, the transformation of the command sequence, in order of first to latest, is the following:
    
    ```
    1. mvn clean install site:site site:deploy site:stage-deploy
    2. mvn clean install site site:stage-deploy site:deploy
    3. mvn clean install && mvn site site:stage-deploy site:deploy
    ```


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron issue #459: METRON-726: Clean up mvn site generation

Posted by justinleet <gi...@git.apache.org>.

Github user justinleet commented on the issue:

    https://github.com/apache/incubator-metron/pull/459
  
    @mmiklavc what maven command did you run?  That looks like that package vs. install issue.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron issue #459: METRON-726: Clean up mvn site generation

Posted by justinleet <gi...@git.apache.org>.

Github user justinleet commented on the issue:

    https://github.com/apache/incubator-metron/pull/459
  
    Updated to (hopefully) not blow up on Travis. Surefire needs jacoco:prepare-agent to resolve the @{argline}.  It's only done in Travis, so it seems reasonable to just call it directly.
    
    Also makes the surefire version a global and sets it up throughout (including in some unversioned spots).  Makes sure @{argLine} is in the appropriate <build> tags. It might also be appropriate in reporting tags, but our management of surefire is pretty variable across the board.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron issue #459: METRON-726: Clean up mvn site generation

Posted by justinleet <gi...@git.apache.org>.

Github user justinleet commented on the issue:

    https://github.com/apache/incubator-metron/pull/459
  
    @mattf-horton Copying your email from the dev list over to here so everything is together on this page.
    
    >If you change the maven invocation to build the site-book, please also change the README at https://github.com/apache/incubator-metron/tree/master/site-book
    >
    >And is the new invocation done from <base dir>/site-book/ as previously, or from base dir?
    
    site-book README is updated to mention that it's part of the standard site process and can be run with a simple mvn site from that directory if you only want the site-book, or from the top level dir if you want the site as a whole (including the book).
    
    The top level readme is updated to add the maven site instructions, since your comment made me realize they only existed in this thread.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron issue #459: METRON-726: Clean up mvn site generation

Posted by mmiklavc <gi...@git.apache.org>.

Github user mmiklavc commented on the issue:

    https://github.com/apache/incubator-metron/pull/459
  
    Still +1. The build works when splitting those phases. I'm still not sure what Maven is doing with our particular build that requires the split. I don't think it's in scope for this PR, and I'm probably just not understanding some aspect of Maven, but we still have some weird things with our build that don't seem to be the norm.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron issue #459: METRON-726: Clean up mvn site generation

Posted by mattf-horton <gi...@git.apache.org>.

Github user mattf-horton commented on the issue:

    https://github.com/apache/incubator-metron/pull/459
  
    @justinleet , I agree with @dlyle65535 that if the intermittent error rate is demonstrably no worse (aprx) than master, that you should proceed with this commit on the reasonable assumption that you're not making the problem worse.  This is a good contribution and I'd like to see it go in.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron issue #459: METRON-726: Clean up mvn site generation

Posted by dlyle65535 <gi...@git.apache.org>.

Github user dlyle65535 commented on the issue:

    https://github.com/apache/incubator-metron/pull/459
  
    I think you're suffering from the same intermittent issues we have on master. Concur that you didn't add anything that obviously would create a problem and you may have helped. Maybe measure the pass/fail rate of N runs on this branch and N runs on master and see if they're similar? What was the fail rate for your current testing, btw?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron issue #459: METRON-726: Clean up mvn site generation

Posted by cestella <gi...@git.apache.org>.

Github user cestella commented on the issue:

    https://github.com/apache/incubator-metron/pull/459
  
    Can we resurrect the thread about enforcing a coding style via checkstyle on the dev list?  If we can import a checkstyle file into intellij, I'm sold.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron issue #459: METRON-726: Clean up mvn site generation

Posted by dlyle65535 <gi...@git.apache.org>.

Github user dlyle65535 commented on the issue:

    https://github.com/apache/incubator-metron/pull/459
  
    +1 on this. Thanks!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron issue #459: METRON-726: Clean up mvn site generation

Posted by mmiklavc <gi...@git.apache.org>.

Github user mmiklavc commented on the issue:

    https://github.com/apache/incubator-metron/pull/459
  
    Pulled down the latest version of this branch:
    ```
    Downloading: https://repo.maven.apache.org/maven2/org/apache/metron/metron-test-utilities/0.4.0/metron-test-utilities-0.4.0.jar
    [INFO] ------------------------------------------------------------------------
    [INFO] Reactor Summary:
    [INFO]
    [INFO] Metron ............................................. FAILURE [ 16.984 s]
    [INFO] metron-analytics ................................... SKIPPED
    [INFO] metron-maas-common ................................. SKIPPED
    [INFO] metron-platform .................................... SKIPPED
    [INFO] metron-test-utilities .............................. SKIPPED
    [INFO] metron-integration-test ............................ SKIPPED
    [INFO] metron-maas-service ................................ SKIPPED
    [INFO] metron-common ...................................... SKIPPED
    [INFO] metron-statistics .................................. SKIPPED
    [INFO] metron-writer ...................................... SKIPPED
    [INFO] metron-storm-kafka ................................. SKIPPED
    [INFO] metron-hbase ....................................... SKIPPED
    [INFO] metron-profiler-common ............................. SKIPPED
    [INFO] metron-profiler-client ............................. SKIPPED
    [INFO] metron-profiler .................................... SKIPPED
    [INFO] metron-enrichment .................................. SKIPPED
    [INFO] metron-indexing .................................... SKIPPED
    [INFO] metron-solr ........................................ SKIPPED
    [INFO] metron-pcap ........................................ SKIPPED
    [INFO] metron-parsers ..................................... SKIPPED
    [INFO] metron-pcap-backend ................................ SKIPPED
    [INFO] metron-data-management ............................. SKIPPED
    [INFO] metron-api ......................................... SKIPPED
    [INFO] metron-management .................................. SKIPPED
    [INFO] elasticsearch-shaded ............................... SKIPPED
    [INFO] metron-elasticsearch ............................... SKIPPED
    [INFO] metron-deployment .................................. SKIPPED
    [INFO] Metron Ambari Management Pack ...................... SKIPPED
    [INFO] metron-docker ...................................... SKIPPED
    [INFO] metron-interface ................................... SKIPPED
    [INFO] metron-config ...................................... SKIPPED
    [INFO] metron-rest-client ................................. SKIPPED
    [INFO] metron-rest ........................................ SKIPPED
    [INFO] site-book .......................................... SKIPPED
    [INFO] ------------------------------------------------------------------------
    [INFO] BUILD FAILURE
    [INFO] ------------------------------------------------------------------------
    [INFO] Total time: 17.458 s
    [INFO] Finished at: 2017-05-02T10:02:36-06:00
    [INFO] Final Memory: 44M/333M
    [INFO] ------------------------------------------------------------------------
    [ERROR] Failed to execute goal org.apache.maven.plugins:maven-site-plugin:3.6:site (default-cli) on project Metron: failed to get report for org.apache.maven.plugins:maven-javadoc-plugin: Failed to execute goal on project metron-integration-test: Could not resolve dependencies for project org.apache.metron:metron-integration-test:jar:0.4.0: Could not find artifact org.apache.metron:metron-test-utilities:jar:0.4.0 in clojars.org (http://clojars.org/repo) -> [Help 1]
    [ERROR]
    [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
    [ERROR] Re-run Maven using the -X switch to enable full debug logging.
    [ERROR]
    [ERROR] For more information about the errors and possible solutions, please read the following articles:
    [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
    ```


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron issue #459: METRON-726: Clean up mvn site generation

Posted by justinleet <gi...@git.apache.org>.

Github user justinleet commented on the issue:

    https://github.com/apache/incubator-metron/pull/459
  
    @mmiklavc @dlyle65535 I'd look to you two, since you gave +1's before master started causing some heartache.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron issue #459: METRON-726: Clean up mvn site generation

Posted by mmiklavc <gi...@git.apache.org>.

Github user mmiklavc commented on the issue:

    https://github.com/apache/incubator-metron/pull/459
  
    +1 : Ran the recommended test script. Site looks good. Thanks for doing this @justinleet. 
    
    Note, I did not run this up in Vagrant.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron issue #459: METRON-726: Clean up mvn site generation

Posted by mattf-horton <gi...@git.apache.org>.

Github user mattf-horton commented on the issue:

    https://github.com/apache/incubator-metron/pull/459
  
    Great, thanks @justinleet !  +1 on that part of it.  I don't feel familiar enough with the other build stuff you modified to evaluate it, so hopefully you can get another +1 too.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

Re: [GitHub] incubator-metron issue #459: METRON-726: Clean up mvn site generation

Posted by Matt Foley <ma...@apache.org>.

The tickets, btw, are METRON-759 and METRON-718.

On 3/8/17, 12:28 PM, "mattf-horton" <gi...@git.apache.org> wrote:

Github user mattf-horton commented on the issue:

https://github.com/apache/incubator-metron/pull/459

@justinleet , +100 that you've added site-book and javadocs to the automated build.
Having site-book in the base build is fine, it only takes seconds to build, and it will keep new README.md files clean. Javadoc takes a little longer, I think, but is also important to keep clean. Is there something like "-DskipSite" that prevents all three (site, site-book, and javadocs) from being built, if a developer wants to skip them?

Regarding the integration, what most projects do is, as part of their Site's documentation area, there is one or more pull-down menus, or a landing page, where you can choose the VERSION you want of:
* Documentation (site-book, in our case)
* Release Notes
* Javadocs

This implies that the Site needs a *cumulative* store of past release Doc builds, and that part of the Release Manager's job is adding each new release to that store. Since it keeps getting larger and larger, storing it in github master (where Site lives) may not be the best thing. Rather, it could go under https://dist.apache.org/repos/dist/release/incubator/metron/ (release -> dev, during votes). The cost of this is a manual step instead of automated, for the Release Manager.

The Site menu would then link into the many doc sets in the store. Given the regular naming of the paths, we could actually have each release contain its own doc set (only) at a standard place, which is consistent with it being part of the build, and then the Site's menu would have a list of links that differ only by a version number. The RM would make the one-line edit to add each new doc set as it is released.

What do you think?

[GitHub] incubator-metron issue #459: METRON-726: Clean up mvn site generation

Posted by mattf-horton <gi...@git.apache.org>.

Github user mattf-horton commented on the issue:

https://github.com/apache/incubator-metron/pull/459

What do you think?

[GitHub] incubator-metron issue #459: METRON-726: Clean up mvn site generation

Posted by justinleet <gi...@git.apache.org>.

Github user justinleet commented on the issue:

    https://github.com/apache/incubator-metron/pull/459
  
    @cestella I wish, I'd have been more than willing to ignore issues if it was.  I'm scraping together a more concise version of the some of the logs and will post them.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #459: METRON-726: Clean up mvn site generation

Posted by justinleet <gi...@git.apache.org>.

Github user justinleet closed the pull request at:

    https://github.com/apache/incubator-metron/pull/459


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron issue #459: METRON-726: Clean up mvn site generation

Posted by justinleet <gi...@git.apache.org>.

Github user justinleet commented on the issue:

    https://github.com/apache/incubator-metron/pull/459
  
    Bumping this.  There was a dev list discussion and it spun out [METRON-746](https://issues.apache.org/jira/browse/METRON-746) and [METRON-747](https://issues.apache.org/jira/browse/METRON-747)
    
    I don't think either of those are blockers to reviewing and pulling in this ticket.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron issue #459: METRON-726: Clean up mvn site generation

Posted by justinleet <gi...@git.apache.org>.

Github user justinleet commented on the issue:

    https://github.com/apache/incubator-metron/pull/459
  
    This PR will need another fix, so I'll update when that's good to go


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron issue #459: METRON-726: Clean up mvn site generation

Posted by justinleet <gi...@git.apache.org>.

Github user justinleet commented on the issue:

    https://github.com/apache/incubator-metron/pull/459
  
    After looking at this some more, I split out the `mvn clean install` from `mvn site site:stage-deploy site:deploy`.  Things got interleaved oddly and gave a similar package vs. install problem.  Running them in sequence solves the problem.
    
    Good catch, @mmiklavc.  Let me know if this fixes your issue.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron issue #459: METRON-726: Clean up mvn site generation

Posted by dlyle65535 <gi...@git.apache.org>.

Github user dlyle65535 commented on the issue:

    https://github.com/apache/incubator-metron/pull/459
  
    Still +1, thanks again!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron issue #459: METRON-726: Clean up mvn site generation

Posted by justinleet <gi...@git.apache.org>.

Github user justinleet commented on the issue:

    https://github.com/apache/incubator-metron/pull/459
  
    To my surprise, I just ran 17 times overnight on the branch with no issues.  I'm going to repeat this over the next couple nights for both master and the branch, because having everything work every time was rather unexpected.  I'll update, probably Monday, with what I find.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron issue #459: METRON-726: Clean up mvn site generation

Posted by justinleet <gi...@git.apache.org>.

Github user justinleet commented on the issue:

    https://github.com/apache/incubator-metron/pull/459
  
    For the errors on master:
    [master_error.txt](https://github.com/apache/incubator-metron/files/958364/master_error.txt)
     
       4 BUILD FAILURE
       8 BUILD SUCCESS
    
    Interestingly, it's two PcapTopologyIntegrationTest tests errors and two intermittent unit test failures.  I'm unsure if there are additional errors if things run longer. I believe the Pcap error has occurred on my branch, but I haven't seen the unit test failures before.
    
    I apparently don't still have a recent log of the branch with errors.  There is one where the branch ran 4 times successfully, but given that I've seen errors, I'm inclined to just run things on the branch overnight again and see where we end up.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron issue #459: METRON-726: Clean up mvn site generation

Posted by justinleet <gi...@git.apache.org>.

Github user justinleet commented on the issue:

https://github.com/apache/incubator-metron/pull/459

@mmiklavc @dlyle65535 I just updated the PR to work with metron-interface (since it came in after this PR). I also merged in master to get some miscellaneous fixes (including in the site-book).

I also added site-book as a component module, so that it gets built at the same time and gets pulled into the site (and can be clicked into when spun up). We may or may not want to leave that, depending on if we want that in the base mvn clean install. Let me know if there's a preference. It's a one line change to revert that.

Finally, I added Javadoc report generation to the top level POM, so it's integrated directly with the site now (feel free to spin it up again and click into Javadocs!)

@mattf-horton This PR now also includes a fatal Javadoc fix. I think at this point, it's mostly the integration (unless something else breaks in the meantime). I think at this point, all the reporting works and gets done in one shot.

I haven't created the ticket for doing something nice with the outputs of this, just because I didn't know if there would be discussion. I'll go ahead and create this, since it seems like people are on board. Matt, if you have any insight into what the appropriate integrations are (e.g. what other projects do), I'd love to add that to the ticket to give whoever picks it up a little extra guidance.

[GitHub] incubator-metron issue #459: METRON-726: Clean up mvn site generation

Posted by cestella <gi...@git.apache.org>.

Github user cestella commented on the issue:

    https://github.com/apache/incubator-metron/pull/459
  
    I did submit a PR that is in flight and somewhat promising at cleaning up the intermittent storm shutdown errors.  Are those what you are seeing?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #459: METRON-726: Clean up mvn site generation

Posted by asfgit <gi...@git.apache.org>.

Github user asfgit closed the pull request at:

    https://github.com/apache/incubator-metron/pull/459


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron issue #459: METRON-726: Clean up mvn site generation

Posted by justinleet <gi...@git.apache.org>.

Github user justinleet commented on the issue:

    https://github.com/apache/incubator-metron/pull/459
  
    @mattf-horton I think this is a pretty good approach and gives us a lot of value especially as we build up more releases.  And there is a `-Dmaven.site.skip=true` flag for maven that can be added to skip all the site stuff.
    
    For right now, I've seen instability in running tests in Jenkins on this branch after pulling in master.  I was hoping to resolve it more quickly, but I might end up having to roll back bits and pieces of the change/integration until I can narrow down what went wrong.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---