You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@samza.apache.org by "Shadi A. Noghabi" <ab...@illinois.edu> on 2015/08/14 20:21:03 UTC
Re: Review Request 36471: Autoscaling for samza (work in progress)
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/36471/
-----------------------------------------------------------
(Updated Aug. 14, 2015, 6:20 p.m.)
Review request for samza and Navina Ramesh.
Changes
-------
Separated config for auto-scaling, and added stat collection into csv files.
Repository: samza
Description
-------
This work is for SAMZA-719. Currently, a fixed number of containers is assigned to a job as an input configuration parameter. However, with this design jobs can fail due to lack of enough resources (such as memory), or they can become a bottleneck in a workflow containing many jobs. While auto-scaling is much broader term, the goal of this project will be to enable a Samza job to automatically scale its containers such that there is improved job performance.
Based on the design, we need a profiler, analyzer, optimizer and deployer module.
-currently, all components added based on memory usage and the loop seems to work in inital test
-tests not added for all components, and further testing is needed.
Diffs (updated)
-----
build.gradle 0852adc4e8e0c2816afd1ebf433f1af6b44852f7
checkstyle/import-control.xml 6654319392929857bb861d77763afd8a5ea7674c
docs/learn/documentation/versioned/jobs/configuration-table.html ea73b40a054002a8f4b0e1fca9e271aae59148ab
gradle/dependency-versions.gradle fb06e8ed393d1a38abfa1a48fe5244fc7f6c7339
samza-core/src/main/java/org/apache/samza/autoScaling/AutoScalingConfig.java PRE-CREATION
samza-core/src/main/java/org/apache/samza/autoScaling/AutoScalingMode.java PRE-CREATION
samza-core/src/main/java/org/apache/samza/autoScaling/AutoScalingSystem.java PRE-CREATION
samza-core/src/main/java/org/apache/samza/autoScaling/analyzer/Analyzer.java PRE-CREATION
samza-core/src/main/java/org/apache/samza/autoScaling/analyzer/MemoryAnalyzer.java PRE-CREATION
samza-core/src/main/java/org/apache/samza/autoScaling/deployer/ConfigManager.java PRE-CREATION
samza-core/src/main/java/org/apache/samza/autoScaling/deployer/Deployer.java PRE-CREATION
samza-core/src/main/java/org/apache/samza/autoScaling/deployer/RestartingJobDeployer.java PRE-CREATION
samza-core/src/main/java/org/apache/samza/autoScaling/metrics/MemoryMetrics.java PRE-CREATION
samza-core/src/main/java/org/apache/samza/autoScaling/metrics/MetricsHistory.java PRE-CREATION
samza-core/src/main/java/org/apache/samza/autoScaling/metrics/SlidingWindowMetric.java PRE-CREATION
samza-core/src/main/java/org/apache/samza/autoScaling/metrics/StatCollector.java PRE-CREATION
samza-core/src/main/java/org/apache/samza/autoScaling/optimizer/CapacityBasedJobModel.java PRE-CREATION
samza-core/src/main/java/org/apache/samza/autoScaling/optimizer/Optimizer.java PRE-CREATION
samza-core/src/main/java/org/apache/samza/autoScaling/optimizer/PerContainerMemoryOptimizer.java PRE-CREATION
samza-core/src/main/java/org/apache/samza/autoScaling/optimizer/PerContainerOptimizer.java PRE-CREATION
samza-core/src/main/java/org/apache/samza/autoScaling/profiler/Profiler.java PRE-CREATION
samza-core/src/main/java/org/apache/samza/autoScaling/profiler/SnapshotReporterProfiler.java PRE-CREATION
samza-core/src/main/java/org/apache/samza/autoScaling/stream/AutoScalingMetricsSystemConsumer.java PRE-CREATION
samza-core/src/main/java/org/apache/samza/coordinator/stream/CoordinatorStreamSystemConsumer.java b1078bdf7bddd16c9ccc6559b9efd40ca5ae67bc
samza-core/src/main/scala/org/apache/samza/autoScaling/AutoScalingCommandLine.scala PRE-CREATION
samza-core/src/main/scala/org/apache/samza/autoScaling/stream/AutoScalingMetricsSystemFactory.scala PRE-CREATION
samza-core/src/main/scala/org/apache/samza/config/AutoScalingConfigRewriter.scala PRE-CREATION
samza-core/src/main/scala/org/apache/samza/config/JobConfig.scala e4b14f4da6649eb78753ba3b3f529373b6f2dbe4
samza-core/src/main/scala/org/apache/samza/config/MetricsConfig.scala f7a11c575807c9d47e441bb8c6b0616142700d37
samza-core/src/main/scala/org/apache/samza/container/SamzaContainer.scala 27b2517048ad5730762506426ee7578c66181db8
samza-core/src/main/scala/org/apache/samza/container/grouper/task/GroupByContainerCountFactory.scala eca6215a1b84c6dc0a4aa1e9e0f8d9f8bff1b467
samza-core/src/main/scala/org/apache/samza/container/grouper/task/GroupByJobModel.scala PRE-CREATION
samza-core/src/main/scala/org/apache/samza/container/grouper/task/GroupByjobModelFactory.scala PRE-CREATION
samza-core/src/main/scala/org/apache/samza/coordinator/JobCoordinator.scala 0dbf14b8bbb8236a8cdbe037a99dec91c5e63965
samza-core/src/main/scala/org/apache/samza/job/JobRunner.scala 1c178a661e449c6bdfc4ce431aef9bb2d261a6c2
samza-core/src/main/scala/org/apache/samza/metrics/JvmMetrics.scala 64daa0f0bb7c6350a57e06705aef41b781779416
samza-core/src/main/scala/org/apache/samza/metrics/reporter/MetricsSnapshotReporter.scala b6696f823e11a0c2134fc830178e47c3ee857378
samza-core/src/main/scala/org/apache/samza/metrics/reporter/MetricsSnapshotReporterFactory.scala dcf0435351cb5deffc7dc4a0cd438013939d54c0
samza-core/src/main/scala/org/apache/samza/util/Util.scala 419452c0d48d8faa84279fcf02a03e87309821d8
samza-core/src/test/java/org/apache/samza/autoScaling/AutoScalingConfigTest.java PRE-CREATION
samza-core/src/test/java/org/apache/samza/autoScaling/SlidingWindowMetricTest.java PRE-CREATION
samza-core/src/test/java/org/apache/samza/autoScaling/analyzer/MockAnalyzer.java PRE-CREATION
samza-core/src/test/java/org/apache/samza/autoScaling/optimizer/CapacityBasedJobModelTest.java PRE-CREATION
samza-core/src/test/java/org/apache/samza/autoScaling/optimizer/MemoryOptimizerTest.java PRE-CREATION
samza-core/src/test/java/org/apache/samza/autoScaling/optimizer/PerContainerMemoryOptimizerTest.java PRE-CREATION
samza-core/src/test/scala/org/apache/samza/container/grouper/task/TestGroupByJobModel.scala PRE-CREATION
samza-shell/src/main/bash/run-auto-scaling.sh PRE-CREATION
samza-shell/src/main/bash/run-config-manager.sh PRE-CREATION
samza-test/src/test/scala/org/apache/samza/test/integration/TestStatefulTask.scala ea702a919348305ff95ce0b4ca1996a13aff04ec
samza-yarn/src/main/scala/org/apache/samza/job/yarn/SamzaAppMaster.scala af42c6a6636953a95f79837fe372e0dbd735df70
samza-yarn/src/test/scala/org/apache/samza/job/yarn/TestSamzaAppMaster.scala 7b7d86a43c69e72c47eaa91f68be24e0f4022891
Diff: https://reviews.apache.org/r/36471/diff/
Testing
-------
tested with hello samza
Thanks,
Shadi A. Noghabi