You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@ambari.apache.org by Aravindan Vijayan <av...@hortonworks.com> on 2017/12/06 22:52:30 UTC
Review Request 64393: AMBARI-22607 : Design and implement an AD job
scheduler.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/64393/
-----------------------------------------------------------
Review request for Ambari, Jayush Luniya and Sid Wagle.
Bugs: AMBARI-22607
https://issues.apache.org/jira/browse/AMBARI-22607
Repository: ambari
Description
-------
Design and implement a scheduler for starting and managing AD jobs on Spark.
The Spark runtime configuration will be analyzed and refined in follow up JIRAs.
Diffs
-----
ambari-metrics/ambari-metrics-anomaly-detection-service/conf/unix/config.yaml 85e4004
ambari-metrics/ambari-metrics-anomaly-detection-service/pom.xml 50d7ef6
ambari-metrics/ambari-metrics-anomaly-detection-service/src/main/java/org/apache/ambari/metrics/adservice/prototype/core/MetricSparkConsumer.java addeda7
ambari-metrics/ambari-metrics-anomaly-detection-service/src/main/scala/org/apache/ambari/metrics/adservice/app/AnomalyDetectionApp.scala 2d0dbdf
ambari-metrics/ambari-metrics-anomaly-detection-service/src/main/scala/org/apache/ambari/metrics/adservice/app/AnomalyDetectionAppConfig.scala 58efa97
ambari-metrics/ambari-metrics-anomaly-detection-service/src/main/scala/org/apache/ambari/metrics/adservice/app/AnomalyDetectionAppModule.scala 68e9df9
ambari-metrics/ambari-metrics-anomaly-detection-service/src/main/scala/org/apache/ambari/metrics/adservice/configuration/DetectionServiceConfiguration.scala PRE-CREATION
ambari-metrics/ambari-metrics-anomaly-detection-service/src/main/scala/org/apache/ambari/metrics/adservice/configuration/SparkConfiguration.scala 30efdc7
ambari-metrics/ambari-metrics-anomaly-detection-service/src/main/scala/org/apache/ambari/metrics/adservice/db/PhoenixAnomalyStoreAccessor.scala 53e6dee
ambari-metrics/ambari-metrics-anomaly-detection-service/src/main/scala/org/apache/ambari/metrics/adservice/detection/AdJobManager.scala PRE-CREATION
ambari-metrics/ambari-metrics-anomaly-detection-service/src/main/scala/org/apache/ambari/metrics/adservice/detection/SparkApplicationRunner.scala PRE-CREATION
ambari-metrics/ambari-metrics-anomaly-detection-service/src/main/scala/org/apache/ambari/metrics/adservice/detection/Subsystem.scala PRE-CREATION
ambari-metrics/ambari-metrics-anomaly-detection-service/src/main/scala/org/apache/ambari/metrics/adservice/detection/pointintime/EmaSparkDriver.scala PRE-CREATION
ambari-metrics/ambari-metrics-anomaly-detection-service/src/main/scala/org/apache/ambari/metrics/adservice/detection/pointintime/PointInTimeSubsystem.scala PRE-CREATION
ambari-metrics/ambari-metrics-anomaly-detection-service/src/main/scala/org/apache/ambari/metrics/adservice/metadata/MetricDefinitionService.scala 52ce39e
ambari-metrics/ambari-metrics-anomaly-detection-service/src/main/scala/org/apache/ambari/metrics/adservice/metadata/MetricDefinitionServiceImpl.scala b9b4a7c
ambari-metrics/ambari-metrics-anomaly-detection-service/src/main/scala/org/apache/ambari/metrics/adservice/metadata/MetricSourceDefinition.scala 47b1499
ambari-metrics/ambari-metrics-anomaly-detection-service/src/main/scala/org/apache/ambari/metrics/adservice/resource/DetectionResource.scala PRE-CREATION
ambari-metrics/ambari-metrics-anomaly-detection-service/src/main/scala/org/apache/ambari/metrics/adservice/resource/MetricDefinitionResource.scala 442bf46
ambari-metrics/ambari-metrics-anomaly-detection-service/src/main/scala/org/apache/ambari/metrics/adservice/service/DetectionService.scala PRE-CREATION
ambari-metrics/ambari-metrics-anomaly-detection-service/src/main/scala/org/apache/ambari/metrics/adservice/service/DetectionServiceImpl.scala PRE-CREATION
ambari-metrics/ambari-metrics-anomaly-detection-service/src/main/scala/org/apache/ambari/metrics/adservice/service/MetricDefinitionService.scala PRE-CREATION
ambari-metrics/ambari-metrics-anomaly-detection-service/src/main/scala/org/apache/ambari/metrics/adservice/service/MetricDefinitionServiceImpl.scala PRE-CREATION
ambari-metrics/ambari-metrics-anomaly-detection-service/src/test/resources/config.yaml 6b09499
ambari-metrics/ambari-metrics-anomaly-detection-service/src/test/scala/org/apache/ambari/metrics/adservice/app/AnomalyDetectionAppConfigTest.scala 76391a0
ambari-metrics/ambari-metrics-anomaly-detection-service/src/test/scala/org/apache/ambari/metrics/adservice/metadata/MetricDefinitionServiceTest.scala d3454f2
ambari-server/src/main/resources/common-services/AMBARI_METRICS/0.1.0/configuration/ams-admanager-config.xml 9862f10
ambari-server/src/main/resources/common-services/AMBARI_METRICS/0.1.0/package/scripts/params.py a18b8f6
Diff: https://reviews.apache.org/r/64393/diff/1/
Testing
-------
Manually tested.
Current unit tests pass.
New Unit tests being worked on...
Thanks,
Aravindan Vijayan
Re: Review Request 64393: AMBARI-22607 : Design and implement an AD
job scheduler.
Posted by Aravindan Vijayan <av...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/64393/
-----------------------------------------------------------
(Updated Dec. 8, 2017, 9:27 p.m.)
Review request for Ambari, Jayush Luniya and Sid Wagle.
Bugs: AMBARI-22607
https://issues.apache.org/jira/browse/AMBARI-22607
Repository: ambari
Description
-------
Design and implement a scheduler for starting and managing AD jobs on Spark.
The Spark runtime configuration will be analyzed and refined in follow up JIRAs.
Diffs (updated)
-----
ambari-metrics/ambari-metrics-anomaly-detection-service/conf/unix/config.yaml 85e4004
ambari-metrics/ambari-metrics-anomaly-detection-service/pom.xml 50d7ef6
ambari-metrics/ambari-metrics-anomaly-detection-service/src/main/java/org/apache/ambari/metrics/adservice/prototype/core/MetricSparkConsumer.java addeda7
ambari-metrics/ambari-metrics-anomaly-detection-service/src/main/scala/org/apache/ambari/metrics/adservice/app/AnomalyDetectionApp.scala 2d0dbdf
ambari-metrics/ambari-metrics-anomaly-detection-service/src/main/scala/org/apache/ambari/metrics/adservice/app/AnomalyDetectionAppConfig.scala 58efa97
ambari-metrics/ambari-metrics-anomaly-detection-service/src/main/scala/org/apache/ambari/metrics/adservice/app/AnomalyDetectionAppModule.scala 68e9df9
ambari-metrics/ambari-metrics-anomaly-detection-service/src/main/scala/org/apache/ambari/metrics/adservice/configuration/DetectionServiceConfiguration.scala PRE-CREATION
ambari-metrics/ambari-metrics-anomaly-detection-service/src/main/scala/org/apache/ambari/metrics/adservice/configuration/SparkConfiguration.scala 30efdc7
ambari-metrics/ambari-metrics-anomaly-detection-service/src/main/scala/org/apache/ambari/metrics/adservice/db/PhoenixAnomalyStoreAccessor.scala 53e6dee
ambari-metrics/ambari-metrics-anomaly-detection-service/src/main/scala/org/apache/ambari/metrics/adservice/detection/AdJobManager.scala PRE-CREATION
ambari-metrics/ambari-metrics-anomaly-detection-service/src/main/scala/org/apache/ambari/metrics/adservice/detection/SparkApplicationRunner.scala PRE-CREATION
ambari-metrics/ambari-metrics-anomaly-detection-service/src/main/scala/org/apache/ambari/metrics/adservice/detection/Subsystem.scala PRE-CREATION
ambari-metrics/ambari-metrics-anomaly-detection-service/src/main/scala/org/apache/ambari/metrics/adservice/detection/pointintime/EmaSparkDriver.scala PRE-CREATION
ambari-metrics/ambari-metrics-anomaly-detection-service/src/main/scala/org/apache/ambari/metrics/adservice/detection/pointintime/PointInTimeSubsystem.scala PRE-CREATION
ambari-metrics/ambari-metrics-anomaly-detection-service/src/main/scala/org/apache/ambari/metrics/adservice/metadata/MetricDefinitionService.scala 52ce39e
ambari-metrics/ambari-metrics-anomaly-detection-service/src/main/scala/org/apache/ambari/metrics/adservice/metadata/MetricDefinitionServiceImpl.scala b9b4a7c
ambari-metrics/ambari-metrics-anomaly-detection-service/src/main/scala/org/apache/ambari/metrics/adservice/metadata/MetricSourceDefinition.scala 47b1499
ambari-metrics/ambari-metrics-anomaly-detection-service/src/main/scala/org/apache/ambari/metrics/adservice/resource/DetectionResource.scala PRE-CREATION
ambari-metrics/ambari-metrics-anomaly-detection-service/src/main/scala/org/apache/ambari/metrics/adservice/resource/MetricDefinitionResource.scala 442bf46
ambari-metrics/ambari-metrics-anomaly-detection-service/src/main/scala/org/apache/ambari/metrics/adservice/service/DetectionService.scala PRE-CREATION
ambari-metrics/ambari-metrics-anomaly-detection-service/src/main/scala/org/apache/ambari/metrics/adservice/service/DetectionServiceImpl.scala PRE-CREATION
ambari-metrics/ambari-metrics-anomaly-detection-service/src/main/scala/org/apache/ambari/metrics/adservice/service/MetricDefinitionService.scala PRE-CREATION
ambari-metrics/ambari-metrics-anomaly-detection-service/src/main/scala/org/apache/ambari/metrics/adservice/service/MetricDefinitionServiceImpl.scala PRE-CREATION
ambari-metrics/ambari-metrics-anomaly-detection-service/src/test/resources/config.yaml 6b09499
ambari-metrics/ambari-metrics-anomaly-detection-service/src/test/scala/org/apache/ambari/metrics/adservice/app/AnomalyDetectionAppConfigTest.scala 76391a0
ambari-metrics/ambari-metrics-anomaly-detection-service/src/test/scala/org/apache/ambari/metrics/adservice/metadata/MetricDefinitionServiceTest.scala d3454f2
ambari-server/src/main/resources/common-services/AMBARI_METRICS/0.1.0/configuration/ams-admanager-config.xml 9862f10
ambari-server/src/main/resources/common-services/AMBARI_METRICS/0.1.0/package/scripts/params.py a18b8f6
Diff: https://reviews.apache.org/r/64393/diff/2/
Changes: https://reviews.apache.org/r/64393/diff/1-2/
Testing
-------
Manually tested.
Current unit tests pass.
New Unit tests being worked on...
Thanks,
Aravindan Vijayan
Re: Review Request 64393: AMBARI-22607 : Design and implement an AD
job scheduler.
Posted by Aravindan Vijayan <av...@hortonworks.com>.
> On Dec. 7, 2017, 3:03 a.m., Sid Wagle wrote:
> > ambari-metrics/ambari-metrics-anomaly-detection-service/src/main/scala/org/apache/ambari/metrics/adservice/configuration/DetectionServiceConfiguration.scala
> > Lines 39 (patched)
> > <https://reviews.apache.org/r/64393/diff/1/?file=1909974#file1909974line39>
> >
> > Description / comments?
Will refine this in when we work on EMA application. For this patch, I used EMA as a simple example job.
> On Dec. 7, 2017, 3:03 a.m., Sid Wagle wrote:
> > ambari-metrics/ambari-metrics-anomaly-detection-service/src/main/scala/org/apache/ambari/metrics/adservice/detection/pointintime/EmaSparkDriver.scala
> > Lines 79 (patched)
> > <https://reviews.apache.org/r/64393/diff/1/?file=1909980#file1909980line79>
> >
> > Debugging?
Will refine this in when we work on EMA application. For this patch, I used EMA as a simple example job.
- Aravindan
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/64393/#review193087
-----------------------------------------------------------
On Dec. 6, 2017, 10:52 p.m., Aravindan Vijayan wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/64393/
> -----------------------------------------------------------
>
> (Updated Dec. 6, 2017, 10:52 p.m.)
>
>
> Review request for Ambari, Jayush Luniya and Sid Wagle.
>
>
> Bugs: AMBARI-22607
> https://issues.apache.org/jira/browse/AMBARI-22607
>
>
> Repository: ambari
>
>
> Description
> -------
>
> Design and implement a scheduler for starting and managing AD jobs on Spark.
> The Spark runtime configuration will be analyzed and refined in follow up JIRAs.
>
>
> Diffs
> -----
>
> ambari-metrics/ambari-metrics-anomaly-detection-service/conf/unix/config.yaml 85e4004
> ambari-metrics/ambari-metrics-anomaly-detection-service/pom.xml 50d7ef6
> ambari-metrics/ambari-metrics-anomaly-detection-service/src/main/java/org/apache/ambari/metrics/adservice/prototype/core/MetricSparkConsumer.java addeda7
> ambari-metrics/ambari-metrics-anomaly-detection-service/src/main/scala/org/apache/ambari/metrics/adservice/app/AnomalyDetectionApp.scala 2d0dbdf
> ambari-metrics/ambari-metrics-anomaly-detection-service/src/main/scala/org/apache/ambari/metrics/adservice/app/AnomalyDetectionAppConfig.scala 58efa97
> ambari-metrics/ambari-metrics-anomaly-detection-service/src/main/scala/org/apache/ambari/metrics/adservice/app/AnomalyDetectionAppModule.scala 68e9df9
> ambari-metrics/ambari-metrics-anomaly-detection-service/src/main/scala/org/apache/ambari/metrics/adservice/configuration/DetectionServiceConfiguration.scala PRE-CREATION
> ambari-metrics/ambari-metrics-anomaly-detection-service/src/main/scala/org/apache/ambari/metrics/adservice/configuration/SparkConfiguration.scala 30efdc7
> ambari-metrics/ambari-metrics-anomaly-detection-service/src/main/scala/org/apache/ambari/metrics/adservice/db/PhoenixAnomalyStoreAccessor.scala 53e6dee
> ambari-metrics/ambari-metrics-anomaly-detection-service/src/main/scala/org/apache/ambari/metrics/adservice/detection/AdJobManager.scala PRE-CREATION
> ambari-metrics/ambari-metrics-anomaly-detection-service/src/main/scala/org/apache/ambari/metrics/adservice/detection/SparkApplicationRunner.scala PRE-CREATION
> ambari-metrics/ambari-metrics-anomaly-detection-service/src/main/scala/org/apache/ambari/metrics/adservice/detection/Subsystem.scala PRE-CREATION
> ambari-metrics/ambari-metrics-anomaly-detection-service/src/main/scala/org/apache/ambari/metrics/adservice/detection/pointintime/EmaSparkDriver.scala PRE-CREATION
> ambari-metrics/ambari-metrics-anomaly-detection-service/src/main/scala/org/apache/ambari/metrics/adservice/detection/pointintime/PointInTimeSubsystem.scala PRE-CREATION
> ambari-metrics/ambari-metrics-anomaly-detection-service/src/main/scala/org/apache/ambari/metrics/adservice/metadata/MetricDefinitionService.scala 52ce39e
> ambari-metrics/ambari-metrics-anomaly-detection-service/src/main/scala/org/apache/ambari/metrics/adservice/metadata/MetricDefinitionServiceImpl.scala b9b4a7c
> ambari-metrics/ambari-metrics-anomaly-detection-service/src/main/scala/org/apache/ambari/metrics/adservice/metadata/MetricSourceDefinition.scala 47b1499
> ambari-metrics/ambari-metrics-anomaly-detection-service/src/main/scala/org/apache/ambari/metrics/adservice/resource/DetectionResource.scala PRE-CREATION
> ambari-metrics/ambari-metrics-anomaly-detection-service/src/main/scala/org/apache/ambari/metrics/adservice/resource/MetricDefinitionResource.scala 442bf46
> ambari-metrics/ambari-metrics-anomaly-detection-service/src/main/scala/org/apache/ambari/metrics/adservice/service/DetectionService.scala PRE-CREATION
> ambari-metrics/ambari-metrics-anomaly-detection-service/src/main/scala/org/apache/ambari/metrics/adservice/service/DetectionServiceImpl.scala PRE-CREATION
> ambari-metrics/ambari-metrics-anomaly-detection-service/src/main/scala/org/apache/ambari/metrics/adservice/service/MetricDefinitionService.scala PRE-CREATION
> ambari-metrics/ambari-metrics-anomaly-detection-service/src/main/scala/org/apache/ambari/metrics/adservice/service/MetricDefinitionServiceImpl.scala PRE-CREATION
> ambari-metrics/ambari-metrics-anomaly-detection-service/src/test/resources/config.yaml 6b09499
> ambari-metrics/ambari-metrics-anomaly-detection-service/src/test/scala/org/apache/ambari/metrics/adservice/app/AnomalyDetectionAppConfigTest.scala 76391a0
> ambari-metrics/ambari-metrics-anomaly-detection-service/src/test/scala/org/apache/ambari/metrics/adservice/metadata/MetricDefinitionServiceTest.scala d3454f2
> ambari-server/src/main/resources/common-services/AMBARI_METRICS/0.1.0/configuration/ams-admanager-config.xml 9862f10
> ambari-server/src/main/resources/common-services/AMBARI_METRICS/0.1.0/package/scripts/params.py a18b8f6
>
>
> Diff: https://reviews.apache.org/r/64393/diff/1/
>
>
> Testing
> -------
>
> Manually tested.
> Current unit tests pass.
> New Unit tests being worked on...
>
>
> Thanks,
>
> Aravindan Vijayan
>
>
Re: Review Request 64393: AMBARI-22607 : Design and implement an AD
job scheduler.
Posted by Aravindan Vijayan <av...@hortonworks.com>.
> On Dec. 7, 2017, 3:03 a.m., Sid Wagle wrote:
> > ambari-metrics/ambari-metrics-anomaly-detection-service/src/main/java/org/apache/ambari/metrics/adservice/prototype/core/MetricSparkConsumer.java
> > Line 155 (original), 154 (patched)
> > <https://reviews.apache.org/r/64393/diff/1/?file=1909970#file1909970line155>
> >
> > Can move this to a unit test?
This is just prototype code that will be removed or ported to Scala when we work on the EMA application.
- Aravindan
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/64393/#review193087
-----------------------------------------------------------
On Dec. 6, 2017, 10:52 p.m., Aravindan Vijayan wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/64393/
> -----------------------------------------------------------
>
> (Updated Dec. 6, 2017, 10:52 p.m.)
>
>
> Review request for Ambari, Jayush Luniya and Sid Wagle.
>
>
> Bugs: AMBARI-22607
> https://issues.apache.org/jira/browse/AMBARI-22607
>
>
> Repository: ambari
>
>
> Description
> -------
>
> Design and implement a scheduler for starting and managing AD jobs on Spark.
> The Spark runtime configuration will be analyzed and refined in follow up JIRAs.
>
>
> Diffs
> -----
>
> ambari-metrics/ambari-metrics-anomaly-detection-service/conf/unix/config.yaml 85e4004
> ambari-metrics/ambari-metrics-anomaly-detection-service/pom.xml 50d7ef6
> ambari-metrics/ambari-metrics-anomaly-detection-service/src/main/java/org/apache/ambari/metrics/adservice/prototype/core/MetricSparkConsumer.java addeda7
> ambari-metrics/ambari-metrics-anomaly-detection-service/src/main/scala/org/apache/ambari/metrics/adservice/app/AnomalyDetectionApp.scala 2d0dbdf
> ambari-metrics/ambari-metrics-anomaly-detection-service/src/main/scala/org/apache/ambari/metrics/adservice/app/AnomalyDetectionAppConfig.scala 58efa97
> ambari-metrics/ambari-metrics-anomaly-detection-service/src/main/scala/org/apache/ambari/metrics/adservice/app/AnomalyDetectionAppModule.scala 68e9df9
> ambari-metrics/ambari-metrics-anomaly-detection-service/src/main/scala/org/apache/ambari/metrics/adservice/configuration/DetectionServiceConfiguration.scala PRE-CREATION
> ambari-metrics/ambari-metrics-anomaly-detection-service/src/main/scala/org/apache/ambari/metrics/adservice/configuration/SparkConfiguration.scala 30efdc7
> ambari-metrics/ambari-metrics-anomaly-detection-service/src/main/scala/org/apache/ambari/metrics/adservice/db/PhoenixAnomalyStoreAccessor.scala 53e6dee
> ambari-metrics/ambari-metrics-anomaly-detection-service/src/main/scala/org/apache/ambari/metrics/adservice/detection/AdJobManager.scala PRE-CREATION
> ambari-metrics/ambari-metrics-anomaly-detection-service/src/main/scala/org/apache/ambari/metrics/adservice/detection/SparkApplicationRunner.scala PRE-CREATION
> ambari-metrics/ambari-metrics-anomaly-detection-service/src/main/scala/org/apache/ambari/metrics/adservice/detection/Subsystem.scala PRE-CREATION
> ambari-metrics/ambari-metrics-anomaly-detection-service/src/main/scala/org/apache/ambari/metrics/adservice/detection/pointintime/EmaSparkDriver.scala PRE-CREATION
> ambari-metrics/ambari-metrics-anomaly-detection-service/src/main/scala/org/apache/ambari/metrics/adservice/detection/pointintime/PointInTimeSubsystem.scala PRE-CREATION
> ambari-metrics/ambari-metrics-anomaly-detection-service/src/main/scala/org/apache/ambari/metrics/adservice/metadata/MetricDefinitionService.scala 52ce39e
> ambari-metrics/ambari-metrics-anomaly-detection-service/src/main/scala/org/apache/ambari/metrics/adservice/metadata/MetricDefinitionServiceImpl.scala b9b4a7c
> ambari-metrics/ambari-metrics-anomaly-detection-service/src/main/scala/org/apache/ambari/metrics/adservice/metadata/MetricSourceDefinition.scala 47b1499
> ambari-metrics/ambari-metrics-anomaly-detection-service/src/main/scala/org/apache/ambari/metrics/adservice/resource/DetectionResource.scala PRE-CREATION
> ambari-metrics/ambari-metrics-anomaly-detection-service/src/main/scala/org/apache/ambari/metrics/adservice/resource/MetricDefinitionResource.scala 442bf46
> ambari-metrics/ambari-metrics-anomaly-detection-service/src/main/scala/org/apache/ambari/metrics/adservice/service/DetectionService.scala PRE-CREATION
> ambari-metrics/ambari-metrics-anomaly-detection-service/src/main/scala/org/apache/ambari/metrics/adservice/service/DetectionServiceImpl.scala PRE-CREATION
> ambari-metrics/ambari-metrics-anomaly-detection-service/src/main/scala/org/apache/ambari/metrics/adservice/service/MetricDefinitionService.scala PRE-CREATION
> ambari-metrics/ambari-metrics-anomaly-detection-service/src/main/scala/org/apache/ambari/metrics/adservice/service/MetricDefinitionServiceImpl.scala PRE-CREATION
> ambari-metrics/ambari-metrics-anomaly-detection-service/src/test/resources/config.yaml 6b09499
> ambari-metrics/ambari-metrics-anomaly-detection-service/src/test/scala/org/apache/ambari/metrics/adservice/app/AnomalyDetectionAppConfigTest.scala 76391a0
> ambari-metrics/ambari-metrics-anomaly-detection-service/src/test/scala/org/apache/ambari/metrics/adservice/metadata/MetricDefinitionServiceTest.scala d3454f2
> ambari-server/src/main/resources/common-services/AMBARI_METRICS/0.1.0/configuration/ams-admanager-config.xml 9862f10
> ambari-server/src/main/resources/common-services/AMBARI_METRICS/0.1.0/package/scripts/params.py a18b8f6
>
>
> Diff: https://reviews.apache.org/r/64393/diff/1/
>
>
> Testing
> -------
>
> Manually tested.
> Current unit tests pass.
> New Unit tests being worked on...
>
>
> Thanks,
>
> Aravindan Vijayan
>
>
Re: Review Request 64393: AMBARI-22607 : Design and implement an AD
job scheduler.
Posted by Aravindan Vijayan <av...@hortonworks.com>.
> On Dec. 7, 2017, 3:03 a.m., Sid Wagle wrote:
> > ambari-metrics/ambari-metrics-anomaly-detection-service/src/main/scala/org/apache/ambari/metrics/adservice/detection/pointintime/PointInTimeSubsystem.scala
> > Lines 61 (patched)
> > <https://reviews.apache.org/r/64393/diff/1/?file=1909981#file1909981line61>
> >
> > Better model will be to add a listener and handle events.
Will handle in subsequent refinements.
- Aravindan
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/64393/#review193087
-----------------------------------------------------------
On Dec. 8, 2017, 9:27 p.m., Aravindan Vijayan wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/64393/
> -----------------------------------------------------------
>
> (Updated Dec. 8, 2017, 9:27 p.m.)
>
>
> Review request for Ambari, Jayush Luniya and Sid Wagle.
>
>
> Bugs: AMBARI-22607
> https://issues.apache.org/jira/browse/AMBARI-22607
>
>
> Repository: ambari
>
>
> Description
> -------
>
> Design and implement a scheduler for starting and managing AD jobs on Spark.
> The Spark runtime configuration will be analyzed and refined in follow up JIRAs.
>
>
> Diffs
> -----
>
> ambari-metrics/ambari-metrics-anomaly-detection-service/conf/unix/config.yaml 85e4004
> ambari-metrics/ambari-metrics-anomaly-detection-service/pom.xml 50d7ef6
> ambari-metrics/ambari-metrics-anomaly-detection-service/src/main/java/org/apache/ambari/metrics/adservice/prototype/core/MetricSparkConsumer.java addeda7
> ambari-metrics/ambari-metrics-anomaly-detection-service/src/main/scala/org/apache/ambari/metrics/adservice/app/AnomalyDetectionApp.scala 2d0dbdf
> ambari-metrics/ambari-metrics-anomaly-detection-service/src/main/scala/org/apache/ambari/metrics/adservice/app/AnomalyDetectionAppConfig.scala 58efa97
> ambari-metrics/ambari-metrics-anomaly-detection-service/src/main/scala/org/apache/ambari/metrics/adservice/app/AnomalyDetectionAppModule.scala 68e9df9
> ambari-metrics/ambari-metrics-anomaly-detection-service/src/main/scala/org/apache/ambari/metrics/adservice/configuration/DetectionServiceConfiguration.scala PRE-CREATION
> ambari-metrics/ambari-metrics-anomaly-detection-service/src/main/scala/org/apache/ambari/metrics/adservice/configuration/SparkConfiguration.scala 30efdc7
> ambari-metrics/ambari-metrics-anomaly-detection-service/src/main/scala/org/apache/ambari/metrics/adservice/db/PhoenixAnomalyStoreAccessor.scala 53e6dee
> ambari-metrics/ambari-metrics-anomaly-detection-service/src/main/scala/org/apache/ambari/metrics/adservice/detection/AdJobManager.scala PRE-CREATION
> ambari-metrics/ambari-metrics-anomaly-detection-service/src/main/scala/org/apache/ambari/metrics/adservice/detection/SparkApplicationRunner.scala PRE-CREATION
> ambari-metrics/ambari-metrics-anomaly-detection-service/src/main/scala/org/apache/ambari/metrics/adservice/detection/Subsystem.scala PRE-CREATION
> ambari-metrics/ambari-metrics-anomaly-detection-service/src/main/scala/org/apache/ambari/metrics/adservice/detection/pointintime/EmaSparkDriver.scala PRE-CREATION
> ambari-metrics/ambari-metrics-anomaly-detection-service/src/main/scala/org/apache/ambari/metrics/adservice/detection/pointintime/PointInTimeSubsystem.scala PRE-CREATION
> ambari-metrics/ambari-metrics-anomaly-detection-service/src/main/scala/org/apache/ambari/metrics/adservice/metadata/MetricDefinitionService.scala 52ce39e
> ambari-metrics/ambari-metrics-anomaly-detection-service/src/main/scala/org/apache/ambari/metrics/adservice/metadata/MetricDefinitionServiceImpl.scala b9b4a7c
> ambari-metrics/ambari-metrics-anomaly-detection-service/src/main/scala/org/apache/ambari/metrics/adservice/metadata/MetricSourceDefinition.scala 47b1499
> ambari-metrics/ambari-metrics-anomaly-detection-service/src/main/scala/org/apache/ambari/metrics/adservice/resource/DetectionResource.scala PRE-CREATION
> ambari-metrics/ambari-metrics-anomaly-detection-service/src/main/scala/org/apache/ambari/metrics/adservice/resource/MetricDefinitionResource.scala 442bf46
> ambari-metrics/ambari-metrics-anomaly-detection-service/src/main/scala/org/apache/ambari/metrics/adservice/service/DetectionService.scala PRE-CREATION
> ambari-metrics/ambari-metrics-anomaly-detection-service/src/main/scala/org/apache/ambari/metrics/adservice/service/DetectionServiceImpl.scala PRE-CREATION
> ambari-metrics/ambari-metrics-anomaly-detection-service/src/main/scala/org/apache/ambari/metrics/adservice/service/MetricDefinitionService.scala PRE-CREATION
> ambari-metrics/ambari-metrics-anomaly-detection-service/src/main/scala/org/apache/ambari/metrics/adservice/service/MetricDefinitionServiceImpl.scala PRE-CREATION
> ambari-metrics/ambari-metrics-anomaly-detection-service/src/test/resources/config.yaml 6b09499
> ambari-metrics/ambari-metrics-anomaly-detection-service/src/test/scala/org/apache/ambari/metrics/adservice/app/AnomalyDetectionAppConfigTest.scala 76391a0
> ambari-metrics/ambari-metrics-anomaly-detection-service/src/test/scala/org/apache/ambari/metrics/adservice/metadata/MetricDefinitionServiceTest.scala d3454f2
> ambari-server/src/main/resources/common-services/AMBARI_METRICS/0.1.0/configuration/ams-admanager-config.xml 9862f10
> ambari-server/src/main/resources/common-services/AMBARI_METRICS/0.1.0/package/scripts/params.py a18b8f6
>
>
> Diff: https://reviews.apache.org/r/64393/diff/2/
>
>
> Testing
> -------
>
> Manually tested.
> Current unit tests pass.
> New Unit tests being worked on...
>
>
> Thanks,
>
> Aravindan Vijayan
>
>
Re: Review Request 64393: AMBARI-22607 : Design and implement an AD
job scheduler.
Posted by Sid Wagle <sw...@hortonworks.com>.
> On Dec. 7, 2017, 3:03 a.m., Sid Wagle wrote:
> > ambari-metrics/ambari-metrics-anomaly-detection-service/src/main/scala/org/apache/ambari/metrics/adservice/configuration/DetectionServiceConfiguration.scala
> > Lines 39 (patched)
> > <https://reviews.apache.org/r/64393/diff/1/?file=1909974#file1909974line39>
> >
> > Description / comments?
>
> Aravindan Vijayan wrote:
> Will refine this in when we work on EMA application. For this patch, I used EMA as a simple example job.
Sure go ahead and close issues that will be addresses later on, just want to make sure the point is raised so we don't forget.
- Sid
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/64393/#review193087
-----------------------------------------------------------
On Dec. 6, 2017, 10:52 p.m., Aravindan Vijayan wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/64393/
> -----------------------------------------------------------
>
> (Updated Dec. 6, 2017, 10:52 p.m.)
>
>
> Review request for Ambari, Jayush Luniya and Sid Wagle.
>
>
> Bugs: AMBARI-22607
> https://issues.apache.org/jira/browse/AMBARI-22607
>
>
> Repository: ambari
>
>
> Description
> -------
>
> Design and implement a scheduler for starting and managing AD jobs on Spark.
> The Spark runtime configuration will be analyzed and refined in follow up JIRAs.
>
>
> Diffs
> -----
>
> ambari-metrics/ambari-metrics-anomaly-detection-service/conf/unix/config.yaml 85e4004
> ambari-metrics/ambari-metrics-anomaly-detection-service/pom.xml 50d7ef6
> ambari-metrics/ambari-metrics-anomaly-detection-service/src/main/java/org/apache/ambari/metrics/adservice/prototype/core/MetricSparkConsumer.java addeda7
> ambari-metrics/ambari-metrics-anomaly-detection-service/src/main/scala/org/apache/ambari/metrics/adservice/app/AnomalyDetectionApp.scala 2d0dbdf
> ambari-metrics/ambari-metrics-anomaly-detection-service/src/main/scala/org/apache/ambari/metrics/adservice/app/AnomalyDetectionAppConfig.scala 58efa97
> ambari-metrics/ambari-metrics-anomaly-detection-service/src/main/scala/org/apache/ambari/metrics/adservice/app/AnomalyDetectionAppModule.scala 68e9df9
> ambari-metrics/ambari-metrics-anomaly-detection-service/src/main/scala/org/apache/ambari/metrics/adservice/configuration/DetectionServiceConfiguration.scala PRE-CREATION
> ambari-metrics/ambari-metrics-anomaly-detection-service/src/main/scala/org/apache/ambari/metrics/adservice/configuration/SparkConfiguration.scala 30efdc7
> ambari-metrics/ambari-metrics-anomaly-detection-service/src/main/scala/org/apache/ambari/metrics/adservice/db/PhoenixAnomalyStoreAccessor.scala 53e6dee
> ambari-metrics/ambari-metrics-anomaly-detection-service/src/main/scala/org/apache/ambari/metrics/adservice/detection/AdJobManager.scala PRE-CREATION
> ambari-metrics/ambari-metrics-anomaly-detection-service/src/main/scala/org/apache/ambari/metrics/adservice/detection/SparkApplicationRunner.scala PRE-CREATION
> ambari-metrics/ambari-metrics-anomaly-detection-service/src/main/scala/org/apache/ambari/metrics/adservice/detection/Subsystem.scala PRE-CREATION
> ambari-metrics/ambari-metrics-anomaly-detection-service/src/main/scala/org/apache/ambari/metrics/adservice/detection/pointintime/EmaSparkDriver.scala PRE-CREATION
> ambari-metrics/ambari-metrics-anomaly-detection-service/src/main/scala/org/apache/ambari/metrics/adservice/detection/pointintime/PointInTimeSubsystem.scala PRE-CREATION
> ambari-metrics/ambari-metrics-anomaly-detection-service/src/main/scala/org/apache/ambari/metrics/adservice/metadata/MetricDefinitionService.scala 52ce39e
> ambari-metrics/ambari-metrics-anomaly-detection-service/src/main/scala/org/apache/ambari/metrics/adservice/metadata/MetricDefinitionServiceImpl.scala b9b4a7c
> ambari-metrics/ambari-metrics-anomaly-detection-service/src/main/scala/org/apache/ambari/metrics/adservice/metadata/MetricSourceDefinition.scala 47b1499
> ambari-metrics/ambari-metrics-anomaly-detection-service/src/main/scala/org/apache/ambari/metrics/adservice/resource/DetectionResource.scala PRE-CREATION
> ambari-metrics/ambari-metrics-anomaly-detection-service/src/main/scala/org/apache/ambari/metrics/adservice/resource/MetricDefinitionResource.scala 442bf46
> ambari-metrics/ambari-metrics-anomaly-detection-service/src/main/scala/org/apache/ambari/metrics/adservice/service/DetectionService.scala PRE-CREATION
> ambari-metrics/ambari-metrics-anomaly-detection-service/src/main/scala/org/apache/ambari/metrics/adservice/service/DetectionServiceImpl.scala PRE-CREATION
> ambari-metrics/ambari-metrics-anomaly-detection-service/src/main/scala/org/apache/ambari/metrics/adservice/service/MetricDefinitionService.scala PRE-CREATION
> ambari-metrics/ambari-metrics-anomaly-detection-service/src/main/scala/org/apache/ambari/metrics/adservice/service/MetricDefinitionServiceImpl.scala PRE-CREATION
> ambari-metrics/ambari-metrics-anomaly-detection-service/src/test/resources/config.yaml 6b09499
> ambari-metrics/ambari-metrics-anomaly-detection-service/src/test/scala/org/apache/ambari/metrics/adservice/app/AnomalyDetectionAppConfigTest.scala 76391a0
> ambari-metrics/ambari-metrics-anomaly-detection-service/src/test/scala/org/apache/ambari/metrics/adservice/metadata/MetricDefinitionServiceTest.scala d3454f2
> ambari-server/src/main/resources/common-services/AMBARI_METRICS/0.1.0/configuration/ams-admanager-config.xml 9862f10
> ambari-server/src/main/resources/common-services/AMBARI_METRICS/0.1.0/package/scripts/params.py a18b8f6
>
>
> Diff: https://reviews.apache.org/r/64393/diff/1/
>
>
> Testing
> -------
>
> Manually tested.
> Current unit tests pass.
> New Unit tests being worked on...
>
>
> Thanks,
>
> Aravindan Vijayan
>
>
Re: Review Request 64393: AMBARI-22607 : Design and implement an AD
job scheduler.
Posted by Sid Wagle <sw...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/64393/#review193087
-----------------------------------------------------------
Fix it, then Ship it!
ambari-metrics/ambari-metrics-anomaly-detection-service/pom.xml
Lines 244 (patched)
<https://reviews.apache.org/r/64393/#comment271574>
2.11 is binary verison - parameterize
ambari-metrics/ambari-metrics-anomaly-detection-service/src/main/java/org/apache/ambari/metrics/adservice/prototype/core/MetricSparkConsumer.java
Line 155 (original), 154 (patched)
<https://reviews.apache.org/r/64393/#comment271575>
Can move this to a unit test?
ambari-metrics/ambari-metrics-anomaly-detection-service/src/main/scala/org/apache/ambari/metrics/adservice/configuration/DetectionServiceConfiguration.scala
Lines 39 (patched)
<https://reviews.apache.org/r/64393/#comment271576>
Description / comments?
ambari-metrics/ambari-metrics-anomaly-detection-service/src/main/scala/org/apache/ambari/metrics/adservice/detection/AdJobManager.scala
Lines 80 (patched)
<https://reviews.apache.org/r/64393/#comment271577>
Nice !!
ambari-metrics/ambari-metrics-anomaly-detection-service/src/main/scala/org/apache/ambari/metrics/adservice/detection/pointintime/EmaSparkDriver.scala
Lines 79 (patched)
<https://reviews.apache.org/r/64393/#comment271579>
Debugging?
ambari-metrics/ambari-metrics-anomaly-detection-service/src/main/scala/org/apache/ambari/metrics/adservice/detection/pointintime/PointInTimeSubsystem.scala
Lines 61 (patched)
<https://reviews.apache.org/r/64393/#comment271580>
Better model will be to add a listener and handle events.
- Sid Wagle
On Dec. 6, 2017, 10:52 p.m., Aravindan Vijayan wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/64393/
> -----------------------------------------------------------
>
> (Updated Dec. 6, 2017, 10:52 p.m.)
>
>
> Review request for Ambari, Jayush Luniya and Sid Wagle.
>
>
> Bugs: AMBARI-22607
> https://issues.apache.org/jira/browse/AMBARI-22607
>
>
> Repository: ambari
>
>
> Description
> -------
>
> Design and implement a scheduler for starting and managing AD jobs on Spark.
> The Spark runtime configuration will be analyzed and refined in follow up JIRAs.
>
>
> Diffs
> -----
>
> ambari-metrics/ambari-metrics-anomaly-detection-service/conf/unix/config.yaml 85e4004
> ambari-metrics/ambari-metrics-anomaly-detection-service/pom.xml 50d7ef6
> ambari-metrics/ambari-metrics-anomaly-detection-service/src/main/java/org/apache/ambari/metrics/adservice/prototype/core/MetricSparkConsumer.java addeda7
> ambari-metrics/ambari-metrics-anomaly-detection-service/src/main/scala/org/apache/ambari/metrics/adservice/app/AnomalyDetectionApp.scala 2d0dbdf
> ambari-metrics/ambari-metrics-anomaly-detection-service/src/main/scala/org/apache/ambari/metrics/adservice/app/AnomalyDetectionAppConfig.scala 58efa97
> ambari-metrics/ambari-metrics-anomaly-detection-service/src/main/scala/org/apache/ambari/metrics/adservice/app/AnomalyDetectionAppModule.scala 68e9df9
> ambari-metrics/ambari-metrics-anomaly-detection-service/src/main/scala/org/apache/ambari/metrics/adservice/configuration/DetectionServiceConfiguration.scala PRE-CREATION
> ambari-metrics/ambari-metrics-anomaly-detection-service/src/main/scala/org/apache/ambari/metrics/adservice/configuration/SparkConfiguration.scala 30efdc7
> ambari-metrics/ambari-metrics-anomaly-detection-service/src/main/scala/org/apache/ambari/metrics/adservice/db/PhoenixAnomalyStoreAccessor.scala 53e6dee
> ambari-metrics/ambari-metrics-anomaly-detection-service/src/main/scala/org/apache/ambari/metrics/adservice/detection/AdJobManager.scala PRE-CREATION
> ambari-metrics/ambari-metrics-anomaly-detection-service/src/main/scala/org/apache/ambari/metrics/adservice/detection/SparkApplicationRunner.scala PRE-CREATION
> ambari-metrics/ambari-metrics-anomaly-detection-service/src/main/scala/org/apache/ambari/metrics/adservice/detection/Subsystem.scala PRE-CREATION
> ambari-metrics/ambari-metrics-anomaly-detection-service/src/main/scala/org/apache/ambari/metrics/adservice/detection/pointintime/EmaSparkDriver.scala PRE-CREATION
> ambari-metrics/ambari-metrics-anomaly-detection-service/src/main/scala/org/apache/ambari/metrics/adservice/detection/pointintime/PointInTimeSubsystem.scala PRE-CREATION
> ambari-metrics/ambari-metrics-anomaly-detection-service/src/main/scala/org/apache/ambari/metrics/adservice/metadata/MetricDefinitionService.scala 52ce39e
> ambari-metrics/ambari-metrics-anomaly-detection-service/src/main/scala/org/apache/ambari/metrics/adservice/metadata/MetricDefinitionServiceImpl.scala b9b4a7c
> ambari-metrics/ambari-metrics-anomaly-detection-service/src/main/scala/org/apache/ambari/metrics/adservice/metadata/MetricSourceDefinition.scala 47b1499
> ambari-metrics/ambari-metrics-anomaly-detection-service/src/main/scala/org/apache/ambari/metrics/adservice/resource/DetectionResource.scala PRE-CREATION
> ambari-metrics/ambari-metrics-anomaly-detection-service/src/main/scala/org/apache/ambari/metrics/adservice/resource/MetricDefinitionResource.scala 442bf46
> ambari-metrics/ambari-metrics-anomaly-detection-service/src/main/scala/org/apache/ambari/metrics/adservice/service/DetectionService.scala PRE-CREATION
> ambari-metrics/ambari-metrics-anomaly-detection-service/src/main/scala/org/apache/ambari/metrics/adservice/service/DetectionServiceImpl.scala PRE-CREATION
> ambari-metrics/ambari-metrics-anomaly-detection-service/src/main/scala/org/apache/ambari/metrics/adservice/service/MetricDefinitionService.scala PRE-CREATION
> ambari-metrics/ambari-metrics-anomaly-detection-service/src/main/scala/org/apache/ambari/metrics/adservice/service/MetricDefinitionServiceImpl.scala PRE-CREATION
> ambari-metrics/ambari-metrics-anomaly-detection-service/src/test/resources/config.yaml 6b09499
> ambari-metrics/ambari-metrics-anomaly-detection-service/src/test/scala/org/apache/ambari/metrics/adservice/app/AnomalyDetectionAppConfigTest.scala 76391a0
> ambari-metrics/ambari-metrics-anomaly-detection-service/src/test/scala/org/apache/ambari/metrics/adservice/metadata/MetricDefinitionServiceTest.scala d3454f2
> ambari-server/src/main/resources/common-services/AMBARI_METRICS/0.1.0/configuration/ams-admanager-config.xml 9862f10
> ambari-server/src/main/resources/common-services/AMBARI_METRICS/0.1.0/package/scripts/params.py a18b8f6
>
>
> Diff: https://reviews.apache.org/r/64393/diff/1/
>
>
> Testing
> -------
>
> Manually tested.
> Current unit tests pass.
> New Unit tests being worked on...
>
>
> Thanks,
>
> Aravindan Vijayan
>
>