You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@ambari.apache.org by Krisztian Kasa <kk...@hortonworks.com> on 2017/11/24 09:21:01 UTC
Review Request 64059: Initial implementation of Schedulable document
deletion & archiving for Infra Solr
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/64059/
-----------------------------------------------------------
Review request for Ambari, Attila Magyar, Oliver Szabo, and Robert Nettleton.
Bugs: AMBARI-22514
https://issues.apache.org/jira/browse/AMBARI-22514
Repository: ambari
Description
-------
Initial implementation of Schedulable document deletion & archiving for Infra Solr
==================================================================================
Spring bacth job for
* exporting solr documents from hadoop_logs collection
* saving the documents to filesystem. All document is converted to json and written into new line.
* compress each file using tar.gz
* job can be started using ambari-infra-manager api and parameters can be overwritten
* s3 upload support (not used yet)
Diffs
-----
ambari-infra/ambari-infra-manager/pom.xml aa86da82b2
ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/CompositeFileAction.java PRE-CREATION
ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/Document.java PRE-CREATION
ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentDestination.java PRE-CREATION
ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentExportConfiguration.java PRE-CREATION
ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentExportJobListener.java PRE-CREATION
ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentExportProperties.java PRE-CREATION
ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentExportStepListener.java PRE-CREATION
ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentExporter.java PRE-CREATION
ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentItemReader.java PRE-CREATION
ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentItemWriter.java PRE-CREATION
ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentIterator.java PRE-CREATION
ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentSource.java PRE-CREATION
ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/FileAction.java PRE-CREATION
ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/HadoopLogSource.java PRE-CREATION
ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/LocalDocumentItemWriter.java PRE-CREATION
ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/S3Properties.java PRE-CREATION
ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/S3Uploader.java PRE-CREATION
ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/SolrDocumentIterator.java PRE-CREATION
ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/SolrDocumentSource.java PRE-CREATION
ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/TarGzCompressor.java PRE-CREATION
ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/manager/JobManager.java fc0a4f71db
ambari-infra/ambari-infra-manager/src/main/resources/infra-manager.properties 81623764da
ambari-infra/ambari-infra-manager/src/test/java/org/apache/ambari/infra/job/archive/DocumentExporterTest.java PRE-CREATION
ambari-infra/ambari-infra-manager/src/test/java/org/apache/ambari/infra/job/archive/DocumentItemReaderTest.java PRE-CREATION
ambari-infra/ambari-infra-manager/src/test/java/org/apache/ambari/infra/job/archive/LocalDocumentItemWriterTest.java PRE-CREATION
ambari-infra/pom.xml e4b6a7601f
ambari-logsearch/docker/test-config/logfeeder/logfeeder.properties fb7ddf2845
Diff: https://reviews.apache.org/r/64059/diff/1/
Testing
-------
* unit tests
* manual test using docker-compose
Thanks,
Krisztian Kasa
Re: Review Request 64059: Initial implementation of Schedulable
document deletion & archiving for Infra Solr
Posted by Oliver Szabo <os...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/64059/#review191833
-----------------------------------------------------------
ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/HadoopLogSource.java
Lines 27 (patched)
<https://reviews.apache.org/r/64059/#comment269774>
it can be service log source
ambari-infra/pom.xml
Lines 163 (patched)
<https://reviews.apache.org/r/64059/#comment269773>
is this really needed? is it becase of you reached the limit of the log file size for long running and it started to rotate? also it is too specific so it can be * * / * . log - * (i just use spaces to the editor not format it)
- Oliver Szabo
On Nov. 24, 2017, 9:31 a.m., Krisztian Kasa wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/64059/
> -----------------------------------------------------------
>
> (Updated Nov. 24, 2017, 9:31 a.m.)
>
>
> Review request for Ambari, Attila Magyar, Oliver Szabo, and Robert Nettleton.
>
>
> Bugs: AMBARI-22514
> https://issues.apache.org/jira/browse/AMBARI-22514
>
>
> Repository: ambari
>
>
> Description
> -------
>
> Initial implementation of Schedulable document deletion & archiving for Infra Solr
> ==================================================================================
>
> Spring bacth job for
>
> * exporting solr documents from hadoop_logs collection
> * saving the documents to filesystem. All document is converted to json and written into new line.
> * compress each file using tar.gz
> * job can be started using ambari-infra-manager api and parameters can be overwritten
> * s3 upload support (not used yet)
>
>
> Diffs
> -----
>
> ambari-infra/ambari-infra-manager/pom.xml aa86da82b2
> ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/CompositeFileAction.java PRE-CREATION
> ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/Document.java PRE-CREATION
> ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentDestination.java PRE-CREATION
> ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentExportConfiguration.java PRE-CREATION
> ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentExportJobListener.java PRE-CREATION
> ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentExportProperties.java PRE-CREATION
> ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentExportStepListener.java PRE-CREATION
> ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentExporter.java PRE-CREATION
> ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentItemReader.java PRE-CREATION
> ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentItemWriter.java PRE-CREATION
> ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentIterator.java PRE-CREATION
> ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentSource.java PRE-CREATION
> ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/FileAction.java PRE-CREATION
> ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/HadoopLogSource.java PRE-CREATION
> ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/LocalDocumentItemWriter.java PRE-CREATION
> ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/S3Properties.java PRE-CREATION
> ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/S3Uploader.java PRE-CREATION
> ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/SolrDocumentIterator.java PRE-CREATION
> ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/SolrDocumentSource.java PRE-CREATION
> ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/TarGzCompressor.java PRE-CREATION
> ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/manager/JobManager.java fc0a4f71db
> ambari-infra/ambari-infra-manager/src/main/resources/infra-manager.properties 81623764da
> ambari-infra/ambari-infra-manager/src/test/java/org/apache/ambari/infra/job/archive/DocumentExporterTest.java PRE-CREATION
> ambari-infra/ambari-infra-manager/src/test/java/org/apache/ambari/infra/job/archive/DocumentItemReaderTest.java PRE-CREATION
> ambari-infra/ambari-infra-manager/src/test/java/org/apache/ambari/infra/job/archive/LocalDocumentItemWriterTest.java PRE-CREATION
> ambari-infra/pom.xml e4b6a7601f
> ambari-logsearch/docker/test-config/logfeeder/logfeeder.properties fb7ddf2845
>
>
> Diff: https://reviews.apache.org/r/64059/diff/2/
>
>
> Testing
> -------
>
> * unit tests
> * manual test using docker-compose
>
>
> Thanks,
>
> Krisztian Kasa
>
>
Re: Review Request 64059: Initial implementation of Schedulable
document deletion & archiving for Infra Solr
Posted by Attila Magyar <am...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/64059/#review191827
-----------------------------------------------------------
ambari-infra/ambari-infra-manager/src/test/java/org/apache/ambari/infra/job/archive/LocalDocumentItemWriterTest.java
Lines 85 (patched)
<https://reviews.apache.org/r/64059/#comment269769>
you can use org.apache.commons.io.FileUtils>>readLines if you have commons-io as a dependency
- Attila Magyar
On Nov. 24, 2017, 9:31 a.m., Krisztian Kasa wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/64059/
> -----------------------------------------------------------
>
> (Updated Nov. 24, 2017, 9:31 a.m.)
>
>
> Review request for Ambari, Attila Magyar, Oliver Szabo, and Robert Nettleton.
>
>
> Bugs: AMBARI-22514
> https://issues.apache.org/jira/browse/AMBARI-22514
>
>
> Repository: ambari
>
>
> Description
> -------
>
> Initial implementation of Schedulable document deletion & archiving for Infra Solr
> ==================================================================================
>
> Spring bacth job for
>
> * exporting solr documents from hadoop_logs collection
> * saving the documents to filesystem. All document is converted to json and written into new line.
> * compress each file using tar.gz
> * job can be started using ambari-infra-manager api and parameters can be overwritten
> * s3 upload support (not used yet)
>
>
> Diffs
> -----
>
> ambari-infra/ambari-infra-manager/pom.xml aa86da82b2
> ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/CompositeFileAction.java PRE-CREATION
> ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/Document.java PRE-CREATION
> ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentDestination.java PRE-CREATION
> ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentExportConfiguration.java PRE-CREATION
> ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentExportJobListener.java PRE-CREATION
> ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentExportProperties.java PRE-CREATION
> ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentExportStepListener.java PRE-CREATION
> ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentExporter.java PRE-CREATION
> ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentItemReader.java PRE-CREATION
> ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentItemWriter.java PRE-CREATION
> ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentIterator.java PRE-CREATION
> ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentSource.java PRE-CREATION
> ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/FileAction.java PRE-CREATION
> ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/HadoopLogSource.java PRE-CREATION
> ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/LocalDocumentItemWriter.java PRE-CREATION
> ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/S3Properties.java PRE-CREATION
> ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/S3Uploader.java PRE-CREATION
> ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/SolrDocumentIterator.java PRE-CREATION
> ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/SolrDocumentSource.java PRE-CREATION
> ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/TarGzCompressor.java PRE-CREATION
> ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/manager/JobManager.java fc0a4f71db
> ambari-infra/ambari-infra-manager/src/main/resources/infra-manager.properties 81623764da
> ambari-infra/ambari-infra-manager/src/test/java/org/apache/ambari/infra/job/archive/DocumentExporterTest.java PRE-CREATION
> ambari-infra/ambari-infra-manager/src/test/java/org/apache/ambari/infra/job/archive/DocumentItemReaderTest.java PRE-CREATION
> ambari-infra/ambari-infra-manager/src/test/java/org/apache/ambari/infra/job/archive/LocalDocumentItemWriterTest.java PRE-CREATION
> ambari-infra/pom.xml e4b6a7601f
> ambari-logsearch/docker/test-config/logfeeder/logfeeder.properties fb7ddf2845
>
>
> Diff: https://reviews.apache.org/r/64059/diff/2/
>
>
> Testing
> -------
>
> * unit tests
> * manual test using docker-compose
>
>
> Thanks,
>
> Krisztian Kasa
>
>
Re: Review Request 64059: Initial implementation of Schedulable
document deletion & archiving for Infra Solr
Posted by Oliver Szabo <os...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/64059/#review191897
-----------------------------------------------------------
Ship it!
Ship It!
- Oliver Szabo
On Nov. 27, 2017, 3:39 p.m., Krisztian Kasa wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/64059/
> -----------------------------------------------------------
>
> (Updated Nov. 27, 2017, 3:39 p.m.)
>
>
> Review request for Ambari, Attila Magyar, Oliver Szabo, and Robert Nettleton.
>
>
> Bugs: AMBARI-22514
> https://issues.apache.org/jira/browse/AMBARI-22514
>
>
> Repository: ambari
>
>
> Description
> -------
>
> Initial implementation of Schedulable document deletion & archiving for Infra Solr
> ==================================================================================
>
> Spring bacth job for
>
> * exporting solr documents from hadoop_logs collection
> * saving the documents to filesystem. All document is converted to json and written into new line.
> * compress each file using tar.gz
> * job can be started using ambari-infra-manager api and parameters can be overwritten
> * s3 upload support (not used yet)
> * job can be configurad to export from any collection: solr query can be defined in properties
>
>
> Diffs
> -----
>
> ambari-infra/ambari-infra-manager/pom.xml aa86da82b2
> ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/CompositeFileAction.java PRE-CREATION
> ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/Document.java PRE-CREATION
> ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentDestination.java PRE-CREATION
> ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentExportConfiguration.java PRE-CREATION
> ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentExportJobListener.java PRE-CREATION
> ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentExportProperties.java PRE-CREATION
> ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentExportStepListener.java PRE-CREATION
> ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentExporter.java PRE-CREATION
> ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentItemReader.java PRE-CREATION
> ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentItemWriter.java PRE-CREATION
> ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentIterator.java PRE-CREATION
> ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentSource.java PRE-CREATION
> ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/FileAction.java PRE-CREATION
> ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/LocalDocumentItemWriter.java PRE-CREATION
> ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/S3Properties.java PRE-CREATION
> ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/S3Uploader.java PRE-CREATION
> ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/SolrDocumentIterator.java PRE-CREATION
> ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/SolrDocumentSource.java PRE-CREATION
> ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/SolrQueryBuilder.java PRE-CREATION
> ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/SolrQueryProperties.java PRE-CREATION
> ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/TarGzCompressor.java PRE-CREATION
> ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/manager/JobManager.java fc0a4f71db
> ambari-infra/ambari-infra-manager/src/main/resources/infra-manager.properties 81623764da
> ambari-infra/ambari-infra-manager/src/main/resources/log4j2.xml ad1adcdcba
> ambari-infra/ambari-infra-manager/src/test/java/org/apache/ambari/infra/job/archive/DocumentExporterTest.java PRE-CREATION
> ambari-infra/ambari-infra-manager/src/test/java/org/apache/ambari/infra/job/archive/DocumentItemReaderTest.java PRE-CREATION
> ambari-infra/ambari-infra-manager/src/test/java/org/apache/ambari/infra/job/archive/LocalDocumentItemWriterTest.java PRE-CREATION
> ambari-infra/ambari-infra-manager/src/test/java/org/apache/ambari/infra/job/archive/SolrQueryBuilderTest.java PRE-CREATION
> ambari-logsearch/docker/test-config/logfeeder/logfeeder.properties fb7ddf2845
>
>
> Diff: https://reviews.apache.org/r/64059/diff/3/
>
>
> Testing
> -------
>
> * unit tests
> * manual test using docker-compose
>
>
> Thanks,
>
> Krisztian Kasa
>
>
Re: Review Request 64059: Initial implementation of Schedulable
document deletion & archiving for Infra Solr
Posted by Attila Magyar <am...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/64059/#review192018
-----------------------------------------------------------
Ship it!
Ship It!
- Attila Magyar
On Nov. 27, 2017, 3:39 p.m., Krisztian Kasa wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/64059/
> -----------------------------------------------------------
>
> (Updated Nov. 27, 2017, 3:39 p.m.)
>
>
> Review request for Ambari, Attila Magyar, Oliver Szabo, and Robert Nettleton.
>
>
> Bugs: AMBARI-22514
> https://issues.apache.org/jira/browse/AMBARI-22514
>
>
> Repository: ambari
>
>
> Description
> -------
>
> Initial implementation of Schedulable document deletion & archiving for Infra Solr
> ==================================================================================
>
> Spring bacth job for
>
> * exporting solr documents from hadoop_logs collection
> * saving the documents to filesystem. All document is converted to json and written into new line.
> * compress each file using tar.gz
> * job can be started using ambari-infra-manager api and parameters can be overwritten
> * s3 upload support (not used yet)
> * job can be configurad to export from any collection: solr query can be defined in properties
>
>
> Diffs
> -----
>
> ambari-infra/ambari-infra-manager/pom.xml aa86da82b2
> ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/CompositeFileAction.java PRE-CREATION
> ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/Document.java PRE-CREATION
> ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentDestination.java PRE-CREATION
> ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentExportConfiguration.java PRE-CREATION
> ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentExportJobListener.java PRE-CREATION
> ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentExportProperties.java PRE-CREATION
> ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentExportStepListener.java PRE-CREATION
> ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentExporter.java PRE-CREATION
> ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentItemReader.java PRE-CREATION
> ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentItemWriter.java PRE-CREATION
> ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentIterator.java PRE-CREATION
> ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentSource.java PRE-CREATION
> ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/FileAction.java PRE-CREATION
> ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/LocalDocumentItemWriter.java PRE-CREATION
> ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/S3Properties.java PRE-CREATION
> ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/S3Uploader.java PRE-CREATION
> ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/SolrDocumentIterator.java PRE-CREATION
> ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/SolrDocumentSource.java PRE-CREATION
> ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/SolrQueryBuilder.java PRE-CREATION
> ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/SolrQueryProperties.java PRE-CREATION
> ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/TarGzCompressor.java PRE-CREATION
> ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/manager/JobManager.java fc0a4f71db
> ambari-infra/ambari-infra-manager/src/main/resources/infra-manager.properties 81623764da
> ambari-infra/ambari-infra-manager/src/main/resources/log4j2.xml ad1adcdcba
> ambari-infra/ambari-infra-manager/src/test/java/org/apache/ambari/infra/job/archive/DocumentExporterTest.java PRE-CREATION
> ambari-infra/ambari-infra-manager/src/test/java/org/apache/ambari/infra/job/archive/DocumentItemReaderTest.java PRE-CREATION
> ambari-infra/ambari-infra-manager/src/test/java/org/apache/ambari/infra/job/archive/LocalDocumentItemWriterTest.java PRE-CREATION
> ambari-infra/ambari-infra-manager/src/test/java/org/apache/ambari/infra/job/archive/SolrQueryBuilderTest.java PRE-CREATION
> ambari-logsearch/docker/test-config/logfeeder/logfeeder.properties fb7ddf2845
>
>
> Diff: https://reviews.apache.org/r/64059/diff/3/
>
>
> Testing
> -------
>
> * unit tests
> * manual test using docker-compose
>
>
> Thanks,
>
> Krisztian Kasa
>
>
Re: Review Request 64059: Initial implementation of Schedulable
document deletion & archiving for Infra Solr
Posted by Krisztian Kasa <kk...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/64059/
-----------------------------------------------------------
(Updated Nov. 27, 2017, 3:39 p.m.)
Review request for Ambari, Attila Magyar, Oliver Szabo, and Robert Nettleton.
Bugs: AMBARI-22514
https://issues.apache.org/jira/browse/AMBARI-22514
Repository: ambari
Description (updated)
-------
Initial implementation of Schedulable document deletion & archiving for Infra Solr
==================================================================================
Spring bacth job for
* exporting solr documents from hadoop_logs collection
* saving the documents to filesystem. All document is converted to json and written into new line.
* compress each file using tar.gz
* job can be started using ambari-infra-manager api and parameters can be overwritten
* s3 upload support (not used yet)
* job can be configurad to export from any collection: solr query can be defined in properties
Diffs (updated)
-----
ambari-infra/ambari-infra-manager/pom.xml aa86da82b2
ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/CompositeFileAction.java PRE-CREATION
ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/Document.java PRE-CREATION
ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentDestination.java PRE-CREATION
ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentExportConfiguration.java PRE-CREATION
ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentExportJobListener.java PRE-CREATION
ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentExportProperties.java PRE-CREATION
ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentExportStepListener.java PRE-CREATION
ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentExporter.java PRE-CREATION
ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentItemReader.java PRE-CREATION
ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentItemWriter.java PRE-CREATION
ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentIterator.java PRE-CREATION
ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentSource.java PRE-CREATION
ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/FileAction.java PRE-CREATION
ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/LocalDocumentItemWriter.java PRE-CREATION
ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/S3Properties.java PRE-CREATION
ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/S3Uploader.java PRE-CREATION
ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/SolrDocumentIterator.java PRE-CREATION
ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/SolrDocumentSource.java PRE-CREATION
ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/SolrQueryBuilder.java PRE-CREATION
ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/SolrQueryProperties.java PRE-CREATION
ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/TarGzCompressor.java PRE-CREATION
ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/manager/JobManager.java fc0a4f71db
ambari-infra/ambari-infra-manager/src/main/resources/infra-manager.properties 81623764da
ambari-infra/ambari-infra-manager/src/main/resources/log4j2.xml ad1adcdcba
ambari-infra/ambari-infra-manager/src/test/java/org/apache/ambari/infra/job/archive/DocumentExporterTest.java PRE-CREATION
ambari-infra/ambari-infra-manager/src/test/java/org/apache/ambari/infra/job/archive/DocumentItemReaderTest.java PRE-CREATION
ambari-infra/ambari-infra-manager/src/test/java/org/apache/ambari/infra/job/archive/LocalDocumentItemWriterTest.java PRE-CREATION
ambari-infra/ambari-infra-manager/src/test/java/org/apache/ambari/infra/job/archive/SolrQueryBuilderTest.java PRE-CREATION
ambari-logsearch/docker/test-config/logfeeder/logfeeder.properties fb7ddf2845
Diff: https://reviews.apache.org/r/64059/diff/3/
Changes: https://reviews.apache.org/r/64059/diff/2-3/
Testing
-------
* unit tests
* manual test using docker-compose
Thanks,
Krisztian Kasa
Re: Review Request 64059: Initial implementation of Schedulable
document deletion & archiving for Infra Solr
Posted by Krisztian Kasa <kk...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/64059/
-----------------------------------------------------------
(Updated Nov. 24, 2017, 9:31 a.m.)
Review request for Ambari, Attila Magyar, Oliver Szabo, and Robert Nettleton.
Bugs: AMBARI-22514
https://issues.apache.org/jira/browse/AMBARI-22514
Repository: ambari
Description
-------
Initial implementation of Schedulable document deletion & archiving for Infra Solr
==================================================================================
Spring bacth job for
* exporting solr documents from hadoop_logs collection
* saving the documents to filesystem. All document is converted to json and written into new line.
* compress each file using tar.gz
* job can be started using ambari-infra-manager api and parameters can be overwritten
* s3 upload support (not used yet)
Diffs (updated)
-----
ambari-infra/ambari-infra-manager/pom.xml aa86da82b2
ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/CompositeFileAction.java PRE-CREATION
ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/Document.java PRE-CREATION
ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentDestination.java PRE-CREATION
ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentExportConfiguration.java PRE-CREATION
ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentExportJobListener.java PRE-CREATION
ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentExportProperties.java PRE-CREATION
ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentExportStepListener.java PRE-CREATION
ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentExporter.java PRE-CREATION
ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentItemReader.java PRE-CREATION
ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentItemWriter.java PRE-CREATION
ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentIterator.java PRE-CREATION
ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentSource.java PRE-CREATION
ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/FileAction.java PRE-CREATION
ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/HadoopLogSource.java PRE-CREATION
ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/LocalDocumentItemWriter.java PRE-CREATION
ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/S3Properties.java PRE-CREATION
ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/S3Uploader.java PRE-CREATION
ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/SolrDocumentIterator.java PRE-CREATION
ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/SolrDocumentSource.java PRE-CREATION
ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/TarGzCompressor.java PRE-CREATION
ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/manager/JobManager.java fc0a4f71db
ambari-infra/ambari-infra-manager/src/main/resources/infra-manager.properties 81623764da
ambari-infra/ambari-infra-manager/src/test/java/org/apache/ambari/infra/job/archive/DocumentExporterTest.java PRE-CREATION
ambari-infra/ambari-infra-manager/src/test/java/org/apache/ambari/infra/job/archive/DocumentItemReaderTest.java PRE-CREATION
ambari-infra/ambari-infra-manager/src/test/java/org/apache/ambari/infra/job/archive/LocalDocumentItemWriterTest.java PRE-CREATION
ambari-infra/pom.xml e4b6a7601f
ambari-logsearch/docker/test-config/logfeeder/logfeeder.properties fb7ddf2845
Diff: https://reviews.apache.org/r/64059/diff/2/
Changes: https://reviews.apache.org/r/64059/diff/1-2/
Testing
-------
* unit tests
* manual test using docker-compose
Thanks,
Krisztian Kasa