You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@ambari.apache.org by Krisztian Kasa <kk...@hortonworks.com> on 2017/12/14 12:58:09 UTC
Review Request 64607: Ambari Infra Manager: solr data exporting jobs
and integration test environment
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/64607/
-----------------------------------------------------------
Review request for Ambari, Miklos Gergely, Oliver Szabo, and Sid Wagle.
Bugs: AMBARI-22514 and AMBARI-22653
https://issues.apache.org/jira/browse/AMBARI-22514
https://issues.apache.org/jira/browse/AMBARI-22653
Repository: ambari
Description
-------
* Integration test environment for ambari-infra-manager using docker-compose and jbehave
* test cases for testing data export jobs
* deleting temporary files as soon as the file becomes unnecessary
* configuring solr data export like jobs in property files only, no hardcoded job and step configurations: query can be defined in property file for pulling data from solr
* validate job property values at application startup and before job execution
* s3 upload support
* not just interval end but interval start can be specified when exporting
* interval start and end values can be other type than dateTime
Diffs
-----
ambari-infra/ambari-infra-manager-it/pom.xml PRE-CREATION
ambari-infra/ambari-infra-manager-it/src/test/java/org/apache/ambari/infra/InfraClient.java PRE-CREATION
ambari-infra/ambari-infra-manager-it/src/test/java/org/apache/ambari/infra/InfraManagerStories.java PRE-CREATION
ambari-infra/ambari-infra-manager-it/src/test/java/org/apache/ambari/infra/OffsetDateTimeConverter.java PRE-CREATION
ambari-infra/ambari-infra-manager-it/src/test/java/org/apache/ambari/infra/steps/AbstractInfraSteps.java PRE-CREATION
ambari-infra/ambari-infra-manager-it/src/test/java/org/apache/ambari/infra/steps/ExportJobsSteps.java PRE-CREATION
ambari-infra/ambari-infra-manager-it/src/test/resources/log4j.properties PRE-CREATION
ambari-infra/ambari-infra-manager-it/src/test/resources/stories/infra_api_tests.story PRE-CREATION
ambari-infra/ambari-infra-manager/docker/Dockerfile adb584acec
ambari-infra/ambari-infra-manager/docker/docker-compose.yml PRE-CREATION
ambari-infra/ambari-infra-manager/docker/infra-manager-docker-compose.sh PRE-CREATION
ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/ObjectSource.java PRE-CREATION
ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/AbstractFileAction.java PRE-CREATION
ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/CompositeFileAction.java 84ce16074b
ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/Document.java 84f5ece26f
ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentExportConfiguration.java 69f41d333e
ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentExportJobListener.java f1df46cfcc
ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentExportProperties.java d6301c0c0e
ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentExportPropertyMap.java PRE-CREATION
ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentExportStepListener.java 3bab6d505e
ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentItemReader.java a4378a4340
ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentIterator.java 6232cfc4c6
ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentSource.java c9871a3660
ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/FileAction.java 26a8c6310f
ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/LocalDocumentItemWriter.java 02d898d979
ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/S3Properties.java 495401d633
ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/S3Uploader.java 3214e50e91
ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/SolrDocumentIterator.java db4069bad7
ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/SolrDocumentSource.java 2181ba368d
ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/SolrQueryBuilder.java d0f6d40155
ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/SolrQueryProperties.java 444a15b267
ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/TarGzCompressor.java 8e34ca9b47
ambari-infra/ambari-infra-manager/src/main/resources/infra-manager.properties 7ef70aaf51
ambari-infra/ambari-infra-manager/src/test/java/org/apache/ambari/infra/job/archive/DocumentExportPropertiesTest.java PRE-CREATION
ambari-infra/ambari-infra-manager/src/test/java/org/apache/ambari/infra/job/archive/DocumentItemReaderTest.java 942713f253
ambari-infra/ambari-infra-manager/src/test/java/org/apache/ambari/infra/job/archive/LocalDocumentItemWriterTest.java 6411ff1788
ambari-infra/ambari-infra-manager/src/test/java/org/apache/ambari/infra/job/archive/SolrQueryBuilderTest.java 4d5ebf1144
ambari-infra/ambari-infra-manager/src/test/java/org/apache/ambari/infra/job/archive/SolrQueryPropertiesTest.java PRE-CREATION
ambari-infra/pom.xml e4b6a7601f
Diff: https://reviews.apache.org/r/64607/diff/1/
Testing
-------
* unit tests,
* integration tests using docker containers
* manual testing using docker containers and ide
Thanks,
Krisztian Kasa
Re: Review Request 64607: Ambari Infra Manager: solr data exporting
jobs and integration test environment
Posted by Oliver Szabo <os...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/64607/#review193796
-----------------------------------------------------------
ambari-infra/ambari-infra-manager-it/pom.xml
Lines 111 (patched)
<https://reviews.apache.org/r/64607/#comment272445>
is it skipped by default if you use "mvn integration-test" so skipTests applied here properly (i think it only affects UTs), probably you should pust this plugin usage into a profile (so we can enable it with -Pit or something)
- Oliver Szabo
On Dec. 14, 2017, 12:58 p.m., Krisztian Kasa wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/64607/
> -----------------------------------------------------------
>
> (Updated Dec. 14, 2017, 12:58 p.m.)
>
>
> Review request for Ambari, Miklos Gergely, Oliver Szabo, and Sid Wagle.
>
>
> Bugs: AMBARI-22514 and AMBARI-22653
> https://issues.apache.org/jira/browse/AMBARI-22514
> https://issues.apache.org/jira/browse/AMBARI-22653
>
>
> Repository: ambari
>
>
> Description
> -------
>
> * Integration test environment for ambari-infra-manager using docker-compose and jbehave
> * test cases for testing data export jobs
>
> * deleting temporary files as soon as the file becomes unnecessary
> * configuring solr data export like jobs in property files only, no hardcoded job and step configurations: query can be defined in property file for pulling data from solr
> * validate job property values at application startup and before job execution
> * s3 upload support
> * not just interval end but interval start can be specified when exporting
> * interval start and end values can be other type than dateTime
>
>
> Diffs
> -----
>
> ambari-infra/ambari-infra-manager-it/pom.xml PRE-CREATION
> ambari-infra/ambari-infra-manager-it/src/test/java/org/apache/ambari/infra/InfraClient.java PRE-CREATION
> ambari-infra/ambari-infra-manager-it/src/test/java/org/apache/ambari/infra/InfraManagerStories.java PRE-CREATION
> ambari-infra/ambari-infra-manager-it/src/test/java/org/apache/ambari/infra/OffsetDateTimeConverter.java PRE-CREATION
> ambari-infra/ambari-infra-manager-it/src/test/java/org/apache/ambari/infra/steps/AbstractInfraSteps.java PRE-CREATION
> ambari-infra/ambari-infra-manager-it/src/test/java/org/apache/ambari/infra/steps/ExportJobsSteps.java PRE-CREATION
> ambari-infra/ambari-infra-manager-it/src/test/resources/log4j.properties PRE-CREATION
> ambari-infra/ambari-infra-manager-it/src/test/resources/stories/infra_api_tests.story PRE-CREATION
> ambari-infra/ambari-infra-manager/docker/Dockerfile adb584acec
> ambari-infra/ambari-infra-manager/docker/docker-compose.yml PRE-CREATION
> ambari-infra/ambari-infra-manager/docker/infra-manager-docker-compose.sh PRE-CREATION
> ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/ObjectSource.java PRE-CREATION
> ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/AbstractFileAction.java PRE-CREATION
> ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/CompositeFileAction.java 84ce16074b
> ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/Document.java 84f5ece26f
> ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentExportConfiguration.java 69f41d333e
> ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentExportJobListener.java f1df46cfcc
> ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentExportProperties.java d6301c0c0e
> ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentExportPropertyMap.java PRE-CREATION
> ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentExportStepListener.java 3bab6d505e
> ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentItemReader.java a4378a4340
> ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentIterator.java 6232cfc4c6
> ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentSource.java c9871a3660
> ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/FileAction.java 26a8c6310f
> ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/LocalDocumentItemWriter.java 02d898d979
> ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/S3Properties.java 495401d633
> ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/S3Uploader.java 3214e50e91
> ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/SolrDocumentIterator.java db4069bad7
> ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/SolrDocumentSource.java 2181ba368d
> ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/SolrQueryBuilder.java d0f6d40155
> ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/SolrQueryProperties.java 444a15b267
> ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/TarGzCompressor.java 8e34ca9b47
> ambari-infra/ambari-infra-manager/src/main/resources/infra-manager.properties 7ef70aaf51
> ambari-infra/ambari-infra-manager/src/test/java/org/apache/ambari/infra/job/archive/DocumentExportPropertiesTest.java PRE-CREATION
> ambari-infra/ambari-infra-manager/src/test/java/org/apache/ambari/infra/job/archive/DocumentItemReaderTest.java 942713f253
> ambari-infra/ambari-infra-manager/src/test/java/org/apache/ambari/infra/job/archive/LocalDocumentItemWriterTest.java 6411ff1788
> ambari-infra/ambari-infra-manager/src/test/java/org/apache/ambari/infra/job/archive/SolrQueryBuilderTest.java 4d5ebf1144
> ambari-infra/ambari-infra-manager/src/test/java/org/apache/ambari/infra/job/archive/SolrQueryPropertiesTest.java PRE-CREATION
> ambari-infra/pom.xml e4b6a7601f
>
>
> Diff: https://reviews.apache.org/r/64607/diff/1/
>
>
> Testing
> -------
>
> * unit tests,
> * integration tests using docker containers
> * manual testing using docker containers and ide
>
>
> Thanks,
>
> Krisztian Kasa
>
>
Re: Review Request 64607: Ambari Infra Manager: solr data exporting
jobs and integration test environment
Posted by Sid Wagle <sw...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/64607/#review194364
-----------------------------------------------------------
Ship it!
Ship It!
- Sid Wagle
On Dec. 15, 2017, 8:57 a.m., Krisztian Kasa wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/64607/
> -----------------------------------------------------------
>
> (Updated Dec. 15, 2017, 8:57 a.m.)
>
>
> Review request for Ambari, Miklos Gergely, Oliver Szabo, and Sid Wagle.
>
>
> Bugs: AMBARI-22514 and AMBARI-22653
> https://issues.apache.org/jira/browse/AMBARI-22514
> https://issues.apache.org/jira/browse/AMBARI-22653
>
>
> Repository: ambari
>
>
> Description
> -------
>
> * Integration test environment for ambari-infra-manager using docker-compose and jbehave
> * test cases for testing data export jobs
>
> * deleting temporary files as soon as the file becomes unnecessary
> * configuring solr data export like jobs in property files only, no hardcoded job and step configurations: query can be defined in property file for pulling data from solr
> * validate job property values at application startup and before job execution
> * s3 upload support
> * not just interval end but interval start can be specified when exporting
> * interval start and end values can be other type than dateTime
>
>
> Diffs
> -----
>
> ambari-infra/ambari-infra-manager-it/pom.xml PRE-CREATION
> ambari-infra/ambari-infra-manager-it/src/test/java/org/apache/ambari/infra/InfraClient.java PRE-CREATION
> ambari-infra/ambari-infra-manager-it/src/test/java/org/apache/ambari/infra/InfraManagerStories.java PRE-CREATION
> ambari-infra/ambari-infra-manager-it/src/test/java/org/apache/ambari/infra/OffsetDateTimeConverter.java PRE-CREATION
> ambari-infra/ambari-infra-manager-it/src/test/java/org/apache/ambari/infra/steps/AbstractInfraSteps.java PRE-CREATION
> ambari-infra/ambari-infra-manager-it/src/test/java/org/apache/ambari/infra/steps/ExportJobsSteps.java PRE-CREATION
> ambari-infra/ambari-infra-manager-it/src/test/resources/log4j.properties PRE-CREATION
> ambari-infra/ambari-infra-manager-it/src/test/resources/stories/infra_api_tests.story PRE-CREATION
> ambari-infra/ambari-infra-manager/docker/Dockerfile adb584acec
> ambari-infra/ambari-infra-manager/docker/docker-compose.yml PRE-CREATION
> ambari-infra/ambari-infra-manager/docker/infra-manager-docker-compose.sh PRE-CREATION
> ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/ObjectSource.java PRE-CREATION
> ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/AbstractFileAction.java PRE-CREATION
> ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/CompositeFileAction.java 84ce16074b
> ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/Document.java 84f5ece26f
> ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentExportConfiguration.java 69f41d333e
> ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentExportJobListener.java f1df46cfcc
> ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentExportProperties.java d6301c0c0e
> ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentExportPropertyMap.java PRE-CREATION
> ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentExportStepListener.java 3bab6d505e
> ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentItemReader.java a4378a4340
> ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentIterator.java 6232cfc4c6
> ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentSource.java c9871a3660
> ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/FileAction.java 26a8c6310f
> ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/LocalDocumentItemWriter.java 02d898d979
> ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/S3Properties.java 495401d633
> ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/S3Uploader.java 3214e50e91
> ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/SolrDocumentIterator.java db4069bad7
> ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/SolrDocumentSource.java 2181ba368d
> ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/SolrQueryBuilder.java d0f6d40155
> ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/SolrQueryProperties.java 444a15b267
> ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/TarGzCompressor.java 8e34ca9b47
> ambari-infra/ambari-infra-manager/src/main/resources/infra-manager.properties 7ef70aaf51
> ambari-infra/ambari-infra-manager/src/test/java/org/apache/ambari/infra/job/archive/DocumentExportPropertiesTest.java PRE-CREATION
> ambari-infra/ambari-infra-manager/src/test/java/org/apache/ambari/infra/job/archive/DocumentItemReaderTest.java 942713f253
> ambari-infra/ambari-infra-manager/src/test/java/org/apache/ambari/infra/job/archive/LocalDocumentItemWriterTest.java 6411ff1788
> ambari-infra/ambari-infra-manager/src/test/java/org/apache/ambari/infra/job/archive/SolrQueryBuilderTest.java 4d5ebf1144
> ambari-infra/ambari-infra-manager/src/test/java/org/apache/ambari/infra/job/archive/SolrQueryPropertiesTest.java PRE-CREATION
> ambari-infra/pom.xml e4b6a7601f
>
>
> Diff: https://reviews.apache.org/r/64607/diff/2/
>
>
> Testing
> -------
>
> * unit tests,
> * integration tests using docker containers
> * manual testing using docker containers and ide
>
>
> Thanks,
>
> Krisztian Kasa
>
>
Re: Review Request 64607: Ambari Infra Manager: solr data exporting
jobs and integration test environment
Posted by Oliver Szabo <os...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/64607/#review193910
-----------------------------------------------------------
Ship it!
Shipt It!
- Oliver Szabo
On Dec. 15, 2017, 8:57 a.m., Krisztian Kasa wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/64607/
> -----------------------------------------------------------
>
> (Updated Dec. 15, 2017, 8:57 a.m.)
>
>
> Review request for Ambari, Miklos Gergely, Oliver Szabo, and Sid Wagle.
>
>
> Bugs: AMBARI-22514 and AMBARI-22653
> https://issues.apache.org/jira/browse/AMBARI-22514
> https://issues.apache.org/jira/browse/AMBARI-22653
>
>
> Repository: ambari
>
>
> Description
> -------
>
> * Integration test environment for ambari-infra-manager using docker-compose and jbehave
> * test cases for testing data export jobs
>
> * deleting temporary files as soon as the file becomes unnecessary
> * configuring solr data export like jobs in property files only, no hardcoded job and step configurations: query can be defined in property file for pulling data from solr
> * validate job property values at application startup and before job execution
> * s3 upload support
> * not just interval end but interval start can be specified when exporting
> * interval start and end values can be other type than dateTime
>
>
> Diffs
> -----
>
> ambari-infra/ambari-infra-manager-it/pom.xml PRE-CREATION
> ambari-infra/ambari-infra-manager-it/src/test/java/org/apache/ambari/infra/InfraClient.java PRE-CREATION
> ambari-infra/ambari-infra-manager-it/src/test/java/org/apache/ambari/infra/InfraManagerStories.java PRE-CREATION
> ambari-infra/ambari-infra-manager-it/src/test/java/org/apache/ambari/infra/OffsetDateTimeConverter.java PRE-CREATION
> ambari-infra/ambari-infra-manager-it/src/test/java/org/apache/ambari/infra/steps/AbstractInfraSteps.java PRE-CREATION
> ambari-infra/ambari-infra-manager-it/src/test/java/org/apache/ambari/infra/steps/ExportJobsSteps.java PRE-CREATION
> ambari-infra/ambari-infra-manager-it/src/test/resources/log4j.properties PRE-CREATION
> ambari-infra/ambari-infra-manager-it/src/test/resources/stories/infra_api_tests.story PRE-CREATION
> ambari-infra/ambari-infra-manager/docker/Dockerfile adb584acec
> ambari-infra/ambari-infra-manager/docker/docker-compose.yml PRE-CREATION
> ambari-infra/ambari-infra-manager/docker/infra-manager-docker-compose.sh PRE-CREATION
> ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/ObjectSource.java PRE-CREATION
> ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/AbstractFileAction.java PRE-CREATION
> ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/CompositeFileAction.java 84ce16074b
> ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/Document.java 84f5ece26f
> ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentExportConfiguration.java 69f41d333e
> ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentExportJobListener.java f1df46cfcc
> ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentExportProperties.java d6301c0c0e
> ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentExportPropertyMap.java PRE-CREATION
> ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentExportStepListener.java 3bab6d505e
> ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentItemReader.java a4378a4340
> ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentIterator.java 6232cfc4c6
> ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentSource.java c9871a3660
> ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/FileAction.java 26a8c6310f
> ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/LocalDocumentItemWriter.java 02d898d979
> ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/S3Properties.java 495401d633
> ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/S3Uploader.java 3214e50e91
> ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/SolrDocumentIterator.java db4069bad7
> ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/SolrDocumentSource.java 2181ba368d
> ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/SolrQueryBuilder.java d0f6d40155
> ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/SolrQueryProperties.java 444a15b267
> ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/TarGzCompressor.java 8e34ca9b47
> ambari-infra/ambari-infra-manager/src/main/resources/infra-manager.properties 7ef70aaf51
> ambari-infra/ambari-infra-manager/src/test/java/org/apache/ambari/infra/job/archive/DocumentExportPropertiesTest.java PRE-CREATION
> ambari-infra/ambari-infra-manager/src/test/java/org/apache/ambari/infra/job/archive/DocumentItemReaderTest.java 942713f253
> ambari-infra/ambari-infra-manager/src/test/java/org/apache/ambari/infra/job/archive/LocalDocumentItemWriterTest.java 6411ff1788
> ambari-infra/ambari-infra-manager/src/test/java/org/apache/ambari/infra/job/archive/SolrQueryBuilderTest.java 4d5ebf1144
> ambari-infra/ambari-infra-manager/src/test/java/org/apache/ambari/infra/job/archive/SolrQueryPropertiesTest.java PRE-CREATION
> ambari-infra/pom.xml e4b6a7601f
>
>
> Diff: https://reviews.apache.org/r/64607/diff/2/
>
>
> Testing
> -------
>
> * unit tests,
> * integration tests using docker containers
> * manual testing using docker containers and ide
>
>
> Thanks,
>
> Krisztian Kasa
>
>
Re: Review Request 64607: Ambari Infra Manager: solr data exporting
jobs and integration test environment
Posted by Krisztian Kasa <kk...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/64607/
-----------------------------------------------------------
(Updated Dec. 15, 2017, 8:57 a.m.)
Review request for Ambari, Miklos Gergely, Oliver Szabo, and Sid Wagle.
Changes
-------
* Integration tests plugin was put into a maven profile
* New test case for testing failure while running job and restart
Bugs: AMBARI-22514 and AMBARI-22653
https://issues.apache.org/jira/browse/AMBARI-22514
https://issues.apache.org/jira/browse/AMBARI-22653
Repository: ambari
Description
-------
* Integration test environment for ambari-infra-manager using docker-compose and jbehave
* test cases for testing data export jobs
* deleting temporary files as soon as the file becomes unnecessary
* configuring solr data export like jobs in property files only, no hardcoded job and step configurations: query can be defined in property file for pulling data from solr
* validate job property values at application startup and before job execution
* s3 upload support
* not just interval end but interval start can be specified when exporting
* interval start and end values can be other type than dateTime
Diffs (updated)
-----
ambari-infra/ambari-infra-manager-it/pom.xml PRE-CREATION
ambari-infra/ambari-infra-manager-it/src/test/java/org/apache/ambari/infra/InfraClient.java PRE-CREATION
ambari-infra/ambari-infra-manager-it/src/test/java/org/apache/ambari/infra/InfraManagerStories.java PRE-CREATION
ambari-infra/ambari-infra-manager-it/src/test/java/org/apache/ambari/infra/OffsetDateTimeConverter.java PRE-CREATION
ambari-infra/ambari-infra-manager-it/src/test/java/org/apache/ambari/infra/steps/AbstractInfraSteps.java PRE-CREATION
ambari-infra/ambari-infra-manager-it/src/test/java/org/apache/ambari/infra/steps/ExportJobsSteps.java PRE-CREATION
ambari-infra/ambari-infra-manager-it/src/test/resources/log4j.properties PRE-CREATION
ambari-infra/ambari-infra-manager-it/src/test/resources/stories/infra_api_tests.story PRE-CREATION
ambari-infra/ambari-infra-manager/docker/Dockerfile adb584acec
ambari-infra/ambari-infra-manager/docker/docker-compose.yml PRE-CREATION
ambari-infra/ambari-infra-manager/docker/infra-manager-docker-compose.sh PRE-CREATION
ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/ObjectSource.java PRE-CREATION
ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/AbstractFileAction.java PRE-CREATION
ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/CompositeFileAction.java 84ce16074b
ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/Document.java 84f5ece26f
ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentExportConfiguration.java 69f41d333e
ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentExportJobListener.java f1df46cfcc
ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentExportProperties.java d6301c0c0e
ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentExportPropertyMap.java PRE-CREATION
ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentExportStepListener.java 3bab6d505e
ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentItemReader.java a4378a4340
ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentIterator.java 6232cfc4c6
ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentSource.java c9871a3660
ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/FileAction.java 26a8c6310f
ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/LocalDocumentItemWriter.java 02d898d979
ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/S3Properties.java 495401d633
ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/S3Uploader.java 3214e50e91
ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/SolrDocumentIterator.java db4069bad7
ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/SolrDocumentSource.java 2181ba368d
ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/SolrQueryBuilder.java d0f6d40155
ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/SolrQueryProperties.java 444a15b267
ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/TarGzCompressor.java 8e34ca9b47
ambari-infra/ambari-infra-manager/src/main/resources/infra-manager.properties 7ef70aaf51
ambari-infra/ambari-infra-manager/src/test/java/org/apache/ambari/infra/job/archive/DocumentExportPropertiesTest.java PRE-CREATION
ambari-infra/ambari-infra-manager/src/test/java/org/apache/ambari/infra/job/archive/DocumentItemReaderTest.java 942713f253
ambari-infra/ambari-infra-manager/src/test/java/org/apache/ambari/infra/job/archive/LocalDocumentItemWriterTest.java 6411ff1788
ambari-infra/ambari-infra-manager/src/test/java/org/apache/ambari/infra/job/archive/SolrQueryBuilderTest.java 4d5ebf1144
ambari-infra/ambari-infra-manager/src/test/java/org/apache/ambari/infra/job/archive/SolrQueryPropertiesTest.java PRE-CREATION
ambari-infra/pom.xml e4b6a7601f
Diff: https://reviews.apache.org/r/64607/diff/2/
Changes: https://reviews.apache.org/r/64607/diff/1-2/
Testing
-------
* unit tests,
* integration tests using docker containers
* manual testing using docker containers and ide
Thanks,
Krisztian Kasa