You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@ambari.apache.org by Krisztian Kasa <kk...@hortonworks.com> on 2017/11/24 09:21:01 UTC

Review Request 64059: Initial implementation of Schedulable document deletion & archiving for Infra Solr

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/64059/
-----------------------------------------------------------

Review request for Ambari, Attila Magyar, Oliver Szabo, and Robert Nettleton.


Bugs: AMBARI-22514
    https://issues.apache.org/jira/browse/AMBARI-22514


Repository: ambari


Description
-------

Initial implementation of Schedulable document deletion & archiving for Infra Solr
==================================================================================

Spring bacth job for

* exporting solr documents from hadoop_logs collection
* saving the documents to filesystem. All document is converted to json and written into new line.
* compress each file using tar.gz
* job can be started using ambari-infra-manager api and parameters can be overwritten
* s3 upload support (not used yet)


Diffs
-----

  ambari-infra/ambari-infra-manager/pom.xml aa86da82b2 
  ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/CompositeFileAction.java PRE-CREATION 
  ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/Document.java PRE-CREATION 
  ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentDestination.java PRE-CREATION 
  ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentExportConfiguration.java PRE-CREATION 
  ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentExportJobListener.java PRE-CREATION 
  ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentExportProperties.java PRE-CREATION 
  ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentExportStepListener.java PRE-CREATION 
  ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentExporter.java PRE-CREATION 
  ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentItemReader.java PRE-CREATION 
  ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentItemWriter.java PRE-CREATION 
  ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentIterator.java PRE-CREATION 
  ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentSource.java PRE-CREATION 
  ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/FileAction.java PRE-CREATION 
  ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/HadoopLogSource.java PRE-CREATION 
  ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/LocalDocumentItemWriter.java PRE-CREATION 
  ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/S3Properties.java PRE-CREATION 
  ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/S3Uploader.java PRE-CREATION 
  ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/SolrDocumentIterator.java PRE-CREATION 
  ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/SolrDocumentSource.java PRE-CREATION 
  ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/TarGzCompressor.java PRE-CREATION 
  ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/manager/JobManager.java fc0a4f71db 
  ambari-infra/ambari-infra-manager/src/main/resources/infra-manager.properties 81623764da 
  ambari-infra/ambari-infra-manager/src/test/java/org/apache/ambari/infra/job/archive/DocumentExporterTest.java PRE-CREATION 
  ambari-infra/ambari-infra-manager/src/test/java/org/apache/ambari/infra/job/archive/DocumentItemReaderTest.java PRE-CREATION 
  ambari-infra/ambari-infra-manager/src/test/java/org/apache/ambari/infra/job/archive/LocalDocumentItemWriterTest.java PRE-CREATION 
  ambari-infra/pom.xml e4b6a7601f 
  ambari-logsearch/docker/test-config/logfeeder/logfeeder.properties fb7ddf2845 


Diff: https://reviews.apache.org/r/64059/diff/1/


Testing
-------

* unit tests
* manual test using docker-compose


Thanks,

Krisztian Kasa


Re: Review Request 64059: Initial implementation of Schedulable document deletion & archiving for Infra Solr

Posted by Oliver Szabo <os...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/64059/#review191833
-----------------------------------------------------------




ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/HadoopLogSource.java
Lines 27 (patched)
<https://reviews.apache.org/r/64059/#comment269774>

    it can be service log source



ambari-infra/pom.xml
Lines 163 (patched)
<https://reviews.apache.org/r/64059/#comment269773>

    is this really needed? is it becase of you reached the limit of the log file size for long running and it started to rotate? also it is too specific so it can be * * / * . log - * (i just use spaces to the editor not format it)


- Oliver Szabo


On Nov. 24, 2017, 9:31 a.m., Krisztian Kasa wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/64059/
> -----------------------------------------------------------
> 
> (Updated Nov. 24, 2017, 9:31 a.m.)
> 
> 
> Review request for Ambari, Attila Magyar, Oliver Szabo, and Robert Nettleton.
> 
> 
> Bugs: AMBARI-22514
>     https://issues.apache.org/jira/browse/AMBARI-22514
> 
> 
> Repository: ambari
> 
> 
> Description
> -------
> 
> Initial implementation of Schedulable document deletion & archiving for Infra Solr
> ==================================================================================
> 
> Spring bacth job for
> 
> * exporting solr documents from hadoop_logs collection
> * saving the documents to filesystem. All document is converted to json and written into new line.
> * compress each file using tar.gz
> * job can be started using ambari-infra-manager api and parameters can be overwritten
> * s3 upload support (not used yet)
> 
> 
> Diffs
> -----
> 
>   ambari-infra/ambari-infra-manager/pom.xml aa86da82b2 
>   ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/CompositeFileAction.java PRE-CREATION 
>   ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/Document.java PRE-CREATION 
>   ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentDestination.java PRE-CREATION 
>   ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentExportConfiguration.java PRE-CREATION 
>   ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentExportJobListener.java PRE-CREATION 
>   ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentExportProperties.java PRE-CREATION 
>   ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentExportStepListener.java PRE-CREATION 
>   ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentExporter.java PRE-CREATION 
>   ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentItemReader.java PRE-CREATION 
>   ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentItemWriter.java PRE-CREATION 
>   ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentIterator.java PRE-CREATION 
>   ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentSource.java PRE-CREATION 
>   ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/FileAction.java PRE-CREATION 
>   ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/HadoopLogSource.java PRE-CREATION 
>   ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/LocalDocumentItemWriter.java PRE-CREATION 
>   ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/S3Properties.java PRE-CREATION 
>   ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/S3Uploader.java PRE-CREATION 
>   ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/SolrDocumentIterator.java PRE-CREATION 
>   ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/SolrDocumentSource.java PRE-CREATION 
>   ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/TarGzCompressor.java PRE-CREATION 
>   ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/manager/JobManager.java fc0a4f71db 
>   ambari-infra/ambari-infra-manager/src/main/resources/infra-manager.properties 81623764da 
>   ambari-infra/ambari-infra-manager/src/test/java/org/apache/ambari/infra/job/archive/DocumentExporterTest.java PRE-CREATION 
>   ambari-infra/ambari-infra-manager/src/test/java/org/apache/ambari/infra/job/archive/DocumentItemReaderTest.java PRE-CREATION 
>   ambari-infra/ambari-infra-manager/src/test/java/org/apache/ambari/infra/job/archive/LocalDocumentItemWriterTest.java PRE-CREATION 
>   ambari-infra/pom.xml e4b6a7601f 
>   ambari-logsearch/docker/test-config/logfeeder/logfeeder.properties fb7ddf2845 
> 
> 
> Diff: https://reviews.apache.org/r/64059/diff/2/
> 
> 
> Testing
> -------
> 
> * unit tests
> * manual test using docker-compose
> 
> 
> Thanks,
> 
> Krisztian Kasa
> 
>


Re: Review Request 64059: Initial implementation of Schedulable document deletion & archiving for Infra Solr

Posted by Attila Magyar <am...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/64059/#review191827
-----------------------------------------------------------




ambari-infra/ambari-infra-manager/src/test/java/org/apache/ambari/infra/job/archive/LocalDocumentItemWriterTest.java
Lines 85 (patched)
<https://reviews.apache.org/r/64059/#comment269769>

    you can use org.apache.commons.io.FileUtils>>readLines if you have commons-io as a dependency


- Attila Magyar


On Nov. 24, 2017, 9:31 a.m., Krisztian Kasa wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/64059/
> -----------------------------------------------------------
> 
> (Updated Nov. 24, 2017, 9:31 a.m.)
> 
> 
> Review request for Ambari, Attila Magyar, Oliver Szabo, and Robert Nettleton.
> 
> 
> Bugs: AMBARI-22514
>     https://issues.apache.org/jira/browse/AMBARI-22514
> 
> 
> Repository: ambari
> 
> 
> Description
> -------
> 
> Initial implementation of Schedulable document deletion & archiving for Infra Solr
> ==================================================================================
> 
> Spring bacth job for
> 
> * exporting solr documents from hadoop_logs collection
> * saving the documents to filesystem. All document is converted to json and written into new line.
> * compress each file using tar.gz
> * job can be started using ambari-infra-manager api and parameters can be overwritten
> * s3 upload support (not used yet)
> 
> 
> Diffs
> -----
> 
>   ambari-infra/ambari-infra-manager/pom.xml aa86da82b2 
>   ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/CompositeFileAction.java PRE-CREATION 
>   ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/Document.java PRE-CREATION 
>   ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentDestination.java PRE-CREATION 
>   ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentExportConfiguration.java PRE-CREATION 
>   ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentExportJobListener.java PRE-CREATION 
>   ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentExportProperties.java PRE-CREATION 
>   ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentExportStepListener.java PRE-CREATION 
>   ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentExporter.java PRE-CREATION 
>   ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentItemReader.java PRE-CREATION 
>   ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentItemWriter.java PRE-CREATION 
>   ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentIterator.java PRE-CREATION 
>   ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentSource.java PRE-CREATION 
>   ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/FileAction.java PRE-CREATION 
>   ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/HadoopLogSource.java PRE-CREATION 
>   ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/LocalDocumentItemWriter.java PRE-CREATION 
>   ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/S3Properties.java PRE-CREATION 
>   ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/S3Uploader.java PRE-CREATION 
>   ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/SolrDocumentIterator.java PRE-CREATION 
>   ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/SolrDocumentSource.java PRE-CREATION 
>   ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/TarGzCompressor.java PRE-CREATION 
>   ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/manager/JobManager.java fc0a4f71db 
>   ambari-infra/ambari-infra-manager/src/main/resources/infra-manager.properties 81623764da 
>   ambari-infra/ambari-infra-manager/src/test/java/org/apache/ambari/infra/job/archive/DocumentExporterTest.java PRE-CREATION 
>   ambari-infra/ambari-infra-manager/src/test/java/org/apache/ambari/infra/job/archive/DocumentItemReaderTest.java PRE-CREATION 
>   ambari-infra/ambari-infra-manager/src/test/java/org/apache/ambari/infra/job/archive/LocalDocumentItemWriterTest.java PRE-CREATION 
>   ambari-infra/pom.xml e4b6a7601f 
>   ambari-logsearch/docker/test-config/logfeeder/logfeeder.properties fb7ddf2845 
> 
> 
> Diff: https://reviews.apache.org/r/64059/diff/2/
> 
> 
> Testing
> -------
> 
> * unit tests
> * manual test using docker-compose
> 
> 
> Thanks,
> 
> Krisztian Kasa
> 
>


Re: Review Request 64059: Initial implementation of Schedulable document deletion & archiving for Infra Solr

Posted by Oliver Szabo <os...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/64059/#review191897
-----------------------------------------------------------


Ship it!




Ship It!

- Oliver Szabo


On Nov. 27, 2017, 3:39 p.m., Krisztian Kasa wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/64059/
> -----------------------------------------------------------
> 
> (Updated Nov. 27, 2017, 3:39 p.m.)
> 
> 
> Review request for Ambari, Attila Magyar, Oliver Szabo, and Robert Nettleton.
> 
> 
> Bugs: AMBARI-22514
>     https://issues.apache.org/jira/browse/AMBARI-22514
> 
> 
> Repository: ambari
> 
> 
> Description
> -------
> 
> Initial implementation of Schedulable document deletion & archiving for Infra Solr
> ==================================================================================
> 
> Spring bacth job for
> 
> * exporting solr documents from hadoop_logs collection
> * saving the documents to filesystem. All document is converted to json and written into new line.
> * compress each file using tar.gz
> * job can be started using ambari-infra-manager api and parameters can be overwritten
> * s3 upload support (not used yet)
> * job can be configurad to export from any collection: solr query can be defined in properties
> 
> 
> Diffs
> -----
> 
>   ambari-infra/ambari-infra-manager/pom.xml aa86da82b2 
>   ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/CompositeFileAction.java PRE-CREATION 
>   ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/Document.java PRE-CREATION 
>   ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentDestination.java PRE-CREATION 
>   ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentExportConfiguration.java PRE-CREATION 
>   ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentExportJobListener.java PRE-CREATION 
>   ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentExportProperties.java PRE-CREATION 
>   ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentExportStepListener.java PRE-CREATION 
>   ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentExporter.java PRE-CREATION 
>   ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentItemReader.java PRE-CREATION 
>   ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentItemWriter.java PRE-CREATION 
>   ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentIterator.java PRE-CREATION 
>   ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentSource.java PRE-CREATION 
>   ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/FileAction.java PRE-CREATION 
>   ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/LocalDocumentItemWriter.java PRE-CREATION 
>   ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/S3Properties.java PRE-CREATION 
>   ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/S3Uploader.java PRE-CREATION 
>   ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/SolrDocumentIterator.java PRE-CREATION 
>   ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/SolrDocumentSource.java PRE-CREATION 
>   ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/SolrQueryBuilder.java PRE-CREATION 
>   ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/SolrQueryProperties.java PRE-CREATION 
>   ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/TarGzCompressor.java PRE-CREATION 
>   ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/manager/JobManager.java fc0a4f71db 
>   ambari-infra/ambari-infra-manager/src/main/resources/infra-manager.properties 81623764da 
>   ambari-infra/ambari-infra-manager/src/main/resources/log4j2.xml ad1adcdcba 
>   ambari-infra/ambari-infra-manager/src/test/java/org/apache/ambari/infra/job/archive/DocumentExporterTest.java PRE-CREATION 
>   ambari-infra/ambari-infra-manager/src/test/java/org/apache/ambari/infra/job/archive/DocumentItemReaderTest.java PRE-CREATION 
>   ambari-infra/ambari-infra-manager/src/test/java/org/apache/ambari/infra/job/archive/LocalDocumentItemWriterTest.java PRE-CREATION 
>   ambari-infra/ambari-infra-manager/src/test/java/org/apache/ambari/infra/job/archive/SolrQueryBuilderTest.java PRE-CREATION 
>   ambari-logsearch/docker/test-config/logfeeder/logfeeder.properties fb7ddf2845 
> 
> 
> Diff: https://reviews.apache.org/r/64059/diff/3/
> 
> 
> Testing
> -------
> 
> * unit tests
> * manual test using docker-compose
> 
> 
> Thanks,
> 
> Krisztian Kasa
> 
>


Re: Review Request 64059: Initial implementation of Schedulable document deletion & archiving for Infra Solr

Posted by Attila Magyar <am...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/64059/#review192018
-----------------------------------------------------------


Ship it!




Ship It!

- Attila Magyar


On Nov. 27, 2017, 3:39 p.m., Krisztian Kasa wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/64059/
> -----------------------------------------------------------
> 
> (Updated Nov. 27, 2017, 3:39 p.m.)
> 
> 
> Review request for Ambari, Attila Magyar, Oliver Szabo, and Robert Nettleton.
> 
> 
> Bugs: AMBARI-22514
>     https://issues.apache.org/jira/browse/AMBARI-22514
> 
> 
> Repository: ambari
> 
> 
> Description
> -------
> 
> Initial implementation of Schedulable document deletion & archiving for Infra Solr
> ==================================================================================
> 
> Spring bacth job for
> 
> * exporting solr documents from hadoop_logs collection
> * saving the documents to filesystem. All document is converted to json and written into new line.
> * compress each file using tar.gz
> * job can be started using ambari-infra-manager api and parameters can be overwritten
> * s3 upload support (not used yet)
> * job can be configurad to export from any collection: solr query can be defined in properties
> 
> 
> Diffs
> -----
> 
>   ambari-infra/ambari-infra-manager/pom.xml aa86da82b2 
>   ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/CompositeFileAction.java PRE-CREATION 
>   ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/Document.java PRE-CREATION 
>   ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentDestination.java PRE-CREATION 
>   ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentExportConfiguration.java PRE-CREATION 
>   ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentExportJobListener.java PRE-CREATION 
>   ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentExportProperties.java PRE-CREATION 
>   ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentExportStepListener.java PRE-CREATION 
>   ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentExporter.java PRE-CREATION 
>   ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentItemReader.java PRE-CREATION 
>   ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentItemWriter.java PRE-CREATION 
>   ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentIterator.java PRE-CREATION 
>   ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentSource.java PRE-CREATION 
>   ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/FileAction.java PRE-CREATION 
>   ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/LocalDocumentItemWriter.java PRE-CREATION 
>   ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/S3Properties.java PRE-CREATION 
>   ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/S3Uploader.java PRE-CREATION 
>   ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/SolrDocumentIterator.java PRE-CREATION 
>   ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/SolrDocumentSource.java PRE-CREATION 
>   ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/SolrQueryBuilder.java PRE-CREATION 
>   ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/SolrQueryProperties.java PRE-CREATION 
>   ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/TarGzCompressor.java PRE-CREATION 
>   ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/manager/JobManager.java fc0a4f71db 
>   ambari-infra/ambari-infra-manager/src/main/resources/infra-manager.properties 81623764da 
>   ambari-infra/ambari-infra-manager/src/main/resources/log4j2.xml ad1adcdcba 
>   ambari-infra/ambari-infra-manager/src/test/java/org/apache/ambari/infra/job/archive/DocumentExporterTest.java PRE-CREATION 
>   ambari-infra/ambari-infra-manager/src/test/java/org/apache/ambari/infra/job/archive/DocumentItemReaderTest.java PRE-CREATION 
>   ambari-infra/ambari-infra-manager/src/test/java/org/apache/ambari/infra/job/archive/LocalDocumentItemWriterTest.java PRE-CREATION 
>   ambari-infra/ambari-infra-manager/src/test/java/org/apache/ambari/infra/job/archive/SolrQueryBuilderTest.java PRE-CREATION 
>   ambari-logsearch/docker/test-config/logfeeder/logfeeder.properties fb7ddf2845 
> 
> 
> Diff: https://reviews.apache.org/r/64059/diff/3/
> 
> 
> Testing
> -------
> 
> * unit tests
> * manual test using docker-compose
> 
> 
> Thanks,
> 
> Krisztian Kasa
> 
>


Re: Review Request 64059: Initial implementation of Schedulable document deletion & archiving for Infra Solr

Posted by Krisztian Kasa <kk...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/64059/
-----------------------------------------------------------

(Updated Nov. 27, 2017, 3:39 p.m.)


Review request for Ambari, Attila Magyar, Oliver Szabo, and Robert Nettleton.


Bugs: AMBARI-22514
    https://issues.apache.org/jira/browse/AMBARI-22514


Repository: ambari


Description (updated)
-------

Initial implementation of Schedulable document deletion & archiving for Infra Solr
==================================================================================

Spring bacth job for

* exporting solr documents from hadoop_logs collection
* saving the documents to filesystem. All document is converted to json and written into new line.
* compress each file using tar.gz
* job can be started using ambari-infra-manager api and parameters can be overwritten
* s3 upload support (not used yet)
* job can be configurad to export from any collection: solr query can be defined in properties


Diffs (updated)
-----

  ambari-infra/ambari-infra-manager/pom.xml aa86da82b2 
  ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/CompositeFileAction.java PRE-CREATION 
  ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/Document.java PRE-CREATION 
  ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentDestination.java PRE-CREATION 
  ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentExportConfiguration.java PRE-CREATION 
  ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentExportJobListener.java PRE-CREATION 
  ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentExportProperties.java PRE-CREATION 
  ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentExportStepListener.java PRE-CREATION 
  ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentExporter.java PRE-CREATION 
  ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentItemReader.java PRE-CREATION 
  ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentItemWriter.java PRE-CREATION 
  ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentIterator.java PRE-CREATION 
  ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentSource.java PRE-CREATION 
  ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/FileAction.java PRE-CREATION 
  ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/LocalDocumentItemWriter.java PRE-CREATION 
  ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/S3Properties.java PRE-CREATION 
  ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/S3Uploader.java PRE-CREATION 
  ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/SolrDocumentIterator.java PRE-CREATION 
  ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/SolrDocumentSource.java PRE-CREATION 
  ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/SolrQueryBuilder.java PRE-CREATION 
  ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/SolrQueryProperties.java PRE-CREATION 
  ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/TarGzCompressor.java PRE-CREATION 
  ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/manager/JobManager.java fc0a4f71db 
  ambari-infra/ambari-infra-manager/src/main/resources/infra-manager.properties 81623764da 
  ambari-infra/ambari-infra-manager/src/main/resources/log4j2.xml ad1adcdcba 
  ambari-infra/ambari-infra-manager/src/test/java/org/apache/ambari/infra/job/archive/DocumentExporterTest.java PRE-CREATION 
  ambari-infra/ambari-infra-manager/src/test/java/org/apache/ambari/infra/job/archive/DocumentItemReaderTest.java PRE-CREATION 
  ambari-infra/ambari-infra-manager/src/test/java/org/apache/ambari/infra/job/archive/LocalDocumentItemWriterTest.java PRE-CREATION 
  ambari-infra/ambari-infra-manager/src/test/java/org/apache/ambari/infra/job/archive/SolrQueryBuilderTest.java PRE-CREATION 
  ambari-logsearch/docker/test-config/logfeeder/logfeeder.properties fb7ddf2845 


Diff: https://reviews.apache.org/r/64059/diff/3/

Changes: https://reviews.apache.org/r/64059/diff/2-3/


Testing
-------

* unit tests
* manual test using docker-compose


Thanks,

Krisztian Kasa


Re: Review Request 64059: Initial implementation of Schedulable document deletion & archiving for Infra Solr

Posted by Krisztian Kasa <kk...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/64059/
-----------------------------------------------------------

(Updated Nov. 24, 2017, 9:31 a.m.)


Review request for Ambari, Attila Magyar, Oliver Szabo, and Robert Nettleton.


Bugs: AMBARI-22514
    https://issues.apache.org/jira/browse/AMBARI-22514


Repository: ambari


Description
-------

Initial implementation of Schedulable document deletion & archiving for Infra Solr
==================================================================================

Spring bacth job for

* exporting solr documents from hadoop_logs collection
* saving the documents to filesystem. All document is converted to json and written into new line.
* compress each file using tar.gz
* job can be started using ambari-infra-manager api and parameters can be overwritten
* s3 upload support (not used yet)


Diffs (updated)
-----

  ambari-infra/ambari-infra-manager/pom.xml aa86da82b2 
  ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/CompositeFileAction.java PRE-CREATION 
  ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/Document.java PRE-CREATION 
  ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentDestination.java PRE-CREATION 
  ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentExportConfiguration.java PRE-CREATION 
  ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentExportJobListener.java PRE-CREATION 
  ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentExportProperties.java PRE-CREATION 
  ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentExportStepListener.java PRE-CREATION 
  ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentExporter.java PRE-CREATION 
  ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentItemReader.java PRE-CREATION 
  ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentItemWriter.java PRE-CREATION 
  ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentIterator.java PRE-CREATION 
  ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentSource.java PRE-CREATION 
  ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/FileAction.java PRE-CREATION 
  ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/HadoopLogSource.java PRE-CREATION 
  ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/LocalDocumentItemWriter.java PRE-CREATION 
  ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/S3Properties.java PRE-CREATION 
  ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/S3Uploader.java PRE-CREATION 
  ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/SolrDocumentIterator.java PRE-CREATION 
  ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/SolrDocumentSource.java PRE-CREATION 
  ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/TarGzCompressor.java PRE-CREATION 
  ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/manager/JobManager.java fc0a4f71db 
  ambari-infra/ambari-infra-manager/src/main/resources/infra-manager.properties 81623764da 
  ambari-infra/ambari-infra-manager/src/test/java/org/apache/ambari/infra/job/archive/DocumentExporterTest.java PRE-CREATION 
  ambari-infra/ambari-infra-manager/src/test/java/org/apache/ambari/infra/job/archive/DocumentItemReaderTest.java PRE-CREATION 
  ambari-infra/ambari-infra-manager/src/test/java/org/apache/ambari/infra/job/archive/LocalDocumentItemWriterTest.java PRE-CREATION 
  ambari-infra/pom.xml e4b6a7601f 
  ambari-logsearch/docker/test-config/logfeeder/logfeeder.properties fb7ddf2845 


Diff: https://reviews.apache.org/r/64059/diff/2/

Changes: https://reviews.apache.org/r/64059/diff/1-2/


Testing
-------

* unit tests
* manual test using docker-compose


Thanks,

Krisztian Kasa