You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@samza.apache.org by GitBox <gi...@apache.org> on 2020/07/31 16:56:52 UTC
[GitHub] [samza] PawasChhokra opened a new pull request #1406: SAMZA-2561: Add config map to MetricsHeader
PawasChhokra opened a new pull request #1406:
URL: https://github.com/apache/samza/pull/1406
Issues: Currently, the MetricsHeader object emitted by the SamzaContainer does not include basic job level information.
Changes: Added the config of the job to be emitted by the MetricsHeader
API Changes: With this change, the MetricsHeader class will emit all the properties of the job stored in its config.
Upgrade instructions: None
Usage instructions: None
Tests: Modified existing tests to work with this change.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [samza] rmatharu commented on a change in pull request #1406: SAMZA-2561: Add config map to MetricsHeader
Posted by GitBox <gi...@apache.org>.
rmatharu commented on a change in pull request #1406:
URL: https://github.com/apache/samza/pull/1406#discussion_r463898479
##########
File path: samza-core/src/main/scala/org/apache/samza/diagnostics/DiagnosticsStreamMessage.java
##########
@@ -228,6 +239,14 @@ public Boolean getAutosizingEnabled() {
return (Boolean) getFromMetricsMessage(GROUP_NAME_FOR_DIAGNOSTICS_MANAGER, AUTOSIZING_ENABLED_METRIC_NAME);
}
+ /**
+ * This method gets the config of the job from the MetricsMessage.
+ * @return the config of the job.
+ */
+ public Config getConfig() {
+ return (Config) getFromMetricsMessage(GROUP_NAME_FOR_DIAGNOSTICS_MANAGER, CONFIG);
Review comment:
Since we will serialize as Map<String, String>
getFromMetricsMessage will return a Map<String, String>
that needs to be converted to Config like this:
new MapConfig(getFromMetricsMessage(GROUP_NAME_FOR_DIAGNOSTICS_MANAGER, CONFIG));
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [samza] rmatharu commented on a change in pull request #1406: SAMZA-2561: Add config map to MetricsHeader
Posted by GitBox <gi...@apache.org>.
rmatharu commented on a change in pull request #1406:
URL: https://github.com/apache/samza/pull/1406#discussion_r463893096
##########
File path: samza-core/src/main/scala/org/apache/samza/diagnostics/DiagnosticsStreamMessage.java
##########
@@ -60,6 +62,7 @@
private static final String CONTAINER_THREAD_POOL_SIZE_METRIC_NAME = "containerThreadPoolSize";
private static final String CONTAINER_MODELS_METRIC_NAME = "containerModels";
private static final String AUTOSIZING_ENABLED_METRIC_NAME = "autosizingEnabled";
+ private static final String CONFIG = "config";
Review comment:
Nit: CONFIG_METRICGROUP_NAME
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [samza] rmatharu commented on a change in pull request #1406: SAMZA-2561: Add config map to MetricsHeader
Posted by GitBox <gi...@apache.org>.
rmatharu commented on a change in pull request #1406:
URL: https://github.com/apache/samza/pull/1406#discussion_r463856744
##########
File path: samza-core/src/main/scala/org/apache/samza/diagnostics/DiagnosticsStreamMessage.java
##########
@@ -228,6 +235,10 @@ public Boolean getAutosizingEnabled() {
return (Boolean) getFromMetricsMessage(GROUP_NAME_FOR_DIAGNOSTICS_MANAGER, AUTOSIZING_ENABLED_METRIC_NAME);
}
+ public Config getConfig() {
Review comment:
Javadoc?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [samza] abhishekshivanna commented on a change in pull request #1406: SAMZA-2561: Add config map to MetricsHeader
Posted by GitBox <gi...@apache.org>.
abhishekshivanna commented on a change in pull request #1406:
URL: https://github.com/apache/samza/pull/1406#discussion_r463877183
##########
File path: samza-core/src/main/java/org/apache/samza/util/DiagnosticsUtil.java
##########
@@ -70,7 +70,7 @@ public static void writeMetadataFile(String jobName, String jobId, String contai
MetricsHeader metricsHeader =
new MetricsHeader(jobName, jobId, "samza-container-" + containerId, execEnvContainerId.orElse(""),
LocalContainerRunner.class.getName(), Util.getTaskClassVersion(config), Util.getSamzaVersion(),
- Util.getLocalHost().getHostName(), System.currentTimeMillis(), System.currentTimeMillis());
+ Util.getLocalHost().getHostName(), System.currentTimeMillis(), System.currentTimeMillis(), config);
Review comment:
+1 I agree we should only pick the fields we are interested in. I suggest we create a versioned model (JSON serializable) that holds the fields we are interested in.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [samza] rmatharu commented on a change in pull request #1406: SAMZA-2561: Add config map to MetricsHeader
Posted by GitBox <gi...@apache.org>.
rmatharu commented on a change in pull request #1406:
URL: https://github.com/apache/samza/pull/1406#discussion_r463858838
##########
File path: samza-core/src/main/scala/org/apache/samza/metrics/reporter/MetricsHeader.scala
##########
@@ -66,6 +68,7 @@ class MetricsHeader(
map.put("host", host)
map.put("time", time: java.lang.Long)
map.put("reset-time", resetTime: java.lang.Long)
+ map.put("config", config)
Review comment:
remove
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [samza] rmatharu commented on a change in pull request #1406: SAMZA-2561: Add config map to MetricsHeader
Posted by GitBox <gi...@apache.org>.
rmatharu commented on a change in pull request #1406:
URL: https://github.com/apache/samza/pull/1406#discussion_r463898362
##########
File path: samza-core/src/main/scala/org/apache/samza/diagnostics/DiagnosticsStreamMessage.java
##########
@@ -155,6 +158,14 @@ public void addProcessorStopEvents(List<ProcessorStopEvent> stopEventList) {
}
}
+ /**
+ * Add the job's config to the message.
+ * @param config the config to add.
+ */
+ public void addConfig(Config config) {
+ addToMetricsMessage(GROUP_NAME_FOR_DIAGNOSTICS_MANAGER, CONFIG, config);
Review comment:
config is a non-primitive object, cast to Map<String, String> here before serializing and adding into the map; similar to how containerModels are added above.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [samza] rmatharu commented on a change in pull request #1406: SAMZA-2561: Add config map to MetricsHeader
Posted by GitBox <gi...@apache.org>.
rmatharu commented on a change in pull request #1406:
URL: https://github.com/apache/samza/pull/1406#discussion_r463857036
##########
File path: samza-core/src/main/scala/org/apache/samza/diagnostics/DiagnosticsStreamMessage.java
##########
@@ -60,17 +62,18 @@
private static final String CONTAINER_THREAD_POOL_SIZE_METRIC_NAME = "containerThreadPoolSize";
private static final String CONTAINER_MODELS_METRIC_NAME = "containerModels";
private static final String AUTOSIZING_ENABLED_METRIC_NAME = "autosizingEnabled";
+ private static final String CONFIG = "config";
private final MetricsHeader metricsHeader;
private final Map<String, Map<String, Object>> metricsMessage;
public DiagnosticsStreamMessage(String jobName, String jobId, String containerName, String executionEnvContainerId,
- String taskClassVersion, String samzaVersion, String hostname, long timestamp, long resetTimestamp) {
+ String taskClassVersion, String samzaVersion, String hostname, long timestamp, long resetTimestamp, Map<String, String> config) {
Review comment:
Because we dont need to add it to the header, we dont need to pass it in here.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [samza] PawasChhokra commented on a change in pull request #1406: SAMZA-2561: Add config map to MetricsHeader
Posted by GitBox <gi...@apache.org>.
PawasChhokra commented on a change in pull request #1406:
URL: https://github.com/apache/samza/pull/1406#discussion_r463892769
##########
File path: samza-core/src/main/scala/org/apache/samza/diagnostics/DiagnosticsStreamMessage.java
##########
@@ -228,6 +235,10 @@ public Boolean getAutosizingEnabled() {
return (Boolean) getFromMetricsMessage(GROUP_NAME_FOR_DIAGNOSTICS_MANAGER, AUTOSIZING_ENABLED_METRIC_NAME);
}
+ public Config getConfig() {
Review comment:
Added.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [samza] PawasChhokra commented on a change in pull request #1406: SAMZA-2561: Add config map to MetricsHeader
Posted by GitBox <gi...@apache.org>.
PawasChhokra commented on a change in pull request #1406:
URL: https://github.com/apache/samza/pull/1406#discussion_r465377876
##########
File path: samza-core/src/test/java/org/apache/samza/diagnostics/TestDiagnosticsManager.java
##########
@@ -272,6 +278,7 @@ private void validateOutgoingMessageEnvelope(OutgoingMessageEnvelope outgoingMes
Assert.assertEquals(containerNumCores, diagnosticsStreamMessage.getContainerNumCores().intValue());
Assert.assertEquals(numPersistentStores, diagnosticsStreamMessage.getNumPersistentStores().intValue());
Assert.assertEquals(autosizingEnabled, diagnosticsStreamMessage.getAutosizingEnabled());
+ Assert.assertEquals(config, diagnosticsStreamMessage.getConfig());
Review comment:
This invokes the equals method of MapConfig class.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [samza] rmatharu commented on a change in pull request #1406: SAMZA-2561: Add config map to MetricsHeader
Posted by GitBox <gi...@apache.org>.
rmatharu commented on a change in pull request #1406:
URL: https://github.com/apache/samza/pull/1406#discussion_r463856253
##########
File path: samza-core/src/main/scala/org/apache/samza/diagnostics/DiagnosticsStreamMessage.java
##########
@@ -60,17 +62,18 @@
private static final String CONTAINER_THREAD_POOL_SIZE_METRIC_NAME = "containerThreadPoolSize";
private static final String CONTAINER_MODELS_METRIC_NAME = "containerModels";
private static final String AUTOSIZING_ENABLED_METRIC_NAME = "autosizingEnabled";
+ private static final String CONFIG = "config";
private final MetricsHeader metricsHeader;
private final Map<String, Map<String, Object>> metricsMessage;
public DiagnosticsStreamMessage(String jobName, String jobId, String containerName, String executionEnvContainerId,
- String taskClassVersion, String samzaVersion, String hostname, long timestamp, long resetTimestamp) {
+ String taskClassVersion, String samzaVersion, String hostname, long timestamp, long resetTimestamp, Map<String, String> config) {
// Create the metricHeader
metricsHeader =
new MetricsHeader(jobName, jobId, containerName, executionEnvContainerId, DiagnosticsManager.class.getName(),
- taskClassVersion, samzaVersion, hostname, timestamp, resetTimestamp);
+ taskClassVersion, samzaVersion, hostname, timestamp, resetTimestamp, config);
Review comment:
See comment above,
config is not required to be in the header.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [samza] rmatharu commented on a change in pull request #1406: SAMZA-2561: Add config map to MetricsHeader
Posted by GitBox <gi...@apache.org>.
rmatharu commented on a change in pull request #1406:
URL: https://github.com/apache/samza/pull/1406#discussion_r463897962
##########
File path: samza-core/src/test/scala/org/apache/samza/serializers/TestMetricsSnapshotSerde.scala
##########
@@ -43,6 +39,9 @@ class TestMetricsSnapshotSerde {
val snapshot = new MetricsSnapshot(header, metrics)
val serde = new MetricsSnapshotSerde()
val bytes = serde.toBytes(snapshot)
- assertTrue(serde.fromBytes(bytes).equals(metrics))
+ val deserialized = serde.fromBytes(bytes)
+ assertTrue(deserialized.getHeader.jobName.equals(snapshot.getHeader.jobName))
+ assertTrue(deserialized.getHeader.jobId.equals(snapshot.getHeader.jobId))
+ assertTrue(deserialized.getMetrics.immutableMetrics.keySet().equals(snapshot.getMetrics.immutableMetrics.keySet()))
Review comment:
Remove?
This code branch and test need not be updated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [samza] rmatharu commented on a change in pull request #1406: SAMZA-2561: Add config map to MetricsHeader
Posted by GitBox <gi...@apache.org>.
rmatharu commented on a change in pull request #1406:
URL: https://github.com/apache/samza/pull/1406#discussion_r463898578
##########
File path: samza-core/src/test/java/org/apache/samza/diagnostics/TestDiagnosticsManager.java
##########
@@ -272,6 +278,7 @@ private void validateOutgoingMessageEnvelope(OutgoingMessageEnvelope outgoingMes
Assert.assertEquals(containerNumCores, diagnosticsStreamMessage.getContainerNumCores().intValue());
Assert.assertEquals(numPersistentStores, diagnosticsStreamMessage.getNumPersistentStores().intValue());
Assert.assertEquals(autosizingEnabled, diagnosticsStreamMessage.getAutosizingEnabled());
+ Assert.assertEquals(config, diagnosticsStreamMessage.getConfig());
Review comment:
Config does not define an equals method, so not sure how this test is working.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [samza] PawasChhokra commented on a change in pull request #1406: SAMZA-2561: Add config map to MetricsHeader
Posted by GitBox <gi...@apache.org>.
PawasChhokra commented on a change in pull request #1406:
URL: https://github.com/apache/samza/pull/1406#discussion_r463892701
##########
File path: samza-core/src/main/scala/org/apache/samza/diagnostics/DiagnosticsStreamMessage.java
##########
@@ -60,17 +62,18 @@
private static final String CONTAINER_THREAD_POOL_SIZE_METRIC_NAME = "containerThreadPoolSize";
private static final String CONTAINER_MODELS_METRIC_NAME = "containerModels";
private static final String AUTOSIZING_ENABLED_METRIC_NAME = "autosizingEnabled";
+ private static final String CONFIG = "config";
private final MetricsHeader metricsHeader;
private final Map<String, Map<String, Object>> metricsMessage;
public DiagnosticsStreamMessage(String jobName, String jobId, String containerName, String executionEnvContainerId,
- String taskClassVersion, String samzaVersion, String hostname, long timestamp, long resetTimestamp) {
+ String taskClassVersion, String samzaVersion, String hostname, long timestamp, long resetTimestamp, Map<String, String> config) {
// Create the metricHeader
metricsHeader =
new MetricsHeader(jobName, jobId, containerName, executionEnvContainerId, DiagnosticsManager.class.getName(),
- taskClassVersion, samzaVersion, hostname, timestamp, resetTimestamp);
+ taskClassVersion, samzaVersion, hostname, timestamp, resetTimestamp, config);
Review comment:
Removed.
##########
File path: samza-core/src/main/scala/org/apache/samza/diagnostics/DiagnosticsStreamMessage.java
##########
@@ -155,6 +158,10 @@ public void addProcessorStopEvents(List<ProcessorStopEvent> stopEventList) {
}
}
+ public void addConfig(Config config) {
Review comment:
Added.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [samza] PawasChhokra commented on a change in pull request #1406: SAMZA-2561: Add config map to MetricsHeader
Posted by GitBox <gi...@apache.org>.
PawasChhokra commented on a change in pull request #1406:
URL: https://github.com/apache/samza/pull/1406#discussion_r463892674
##########
File path: samza-core/src/main/scala/org/apache/samza/diagnostics/DiagnosticsStreamMessage.java
##########
@@ -60,17 +62,18 @@
private static final String CONTAINER_THREAD_POOL_SIZE_METRIC_NAME = "containerThreadPoolSize";
private static final String CONTAINER_MODELS_METRIC_NAME = "containerModels";
private static final String AUTOSIZING_ENABLED_METRIC_NAME = "autosizingEnabled";
+ private static final String CONFIG = "config";
private final MetricsHeader metricsHeader;
private final Map<String, Map<String, Object>> metricsMessage;
public DiagnosticsStreamMessage(String jobName, String jobId, String containerName, String executionEnvContainerId,
- String taskClassVersion, String samzaVersion, String hostname, long timestamp, long resetTimestamp) {
+ String taskClassVersion, String samzaVersion, String hostname, long timestamp, long resetTimestamp, Map<String, String> config) {
Review comment:
Removed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [samza] rmatharu commented on a change in pull request #1406: SAMZA-2561: Add config map to MetricsHeader
Posted by GitBox <gi...@apache.org>.
rmatharu commented on a change in pull request #1406:
URL: https://github.com/apache/samza/pull/1406#discussion_r463897616
##########
File path: samza-core/src/test/java/org/apache/samza/serializers/model/serializers/TestMetricsSnapshotSerdeV2.java
##########
@@ -37,8 +37,8 @@
@Test
public void testSerde() {
MetricsHeader metricsHeader =
- new MetricsHeader("jobName", "i001", "container 0", "test container ID", "source", "300.14.25.1", "1", "1", 1,
- 1);
+ new MetricsHeader("jobName", "i001", "container 0", "test container ID",
+ "source", "300.14.25.1", "1", "1", 1, 1);
Review comment:
Remove?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [samza] PawasChhokra commented on a change in pull request #1406: SAMZA-2561: Add config map to MetricsHeader
Posted by GitBox <gi...@apache.org>.
PawasChhokra commented on a change in pull request #1406:
URL: https://github.com/apache/samza/pull/1406#discussion_r463898108
##########
File path: samza-core/src/test/java/org/apache/samza/diagnostics/TestDiagnosticsManager.java
##########
@@ -59,6 +62,7 @@
private int numPersistentStores = 2;
private int containerNumCores = 2;
private boolean autosizingEnabled = false;
+ private Config config = new MapConfig(ImmutableMap.of("job.name", jobName, "job.id", jobId));
Review comment:
Done.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [samza] rmatharu commented on a change in pull request #1406: SAMZA-2561: Add config map to MetricsHeader
Posted by GitBox <gi...@apache.org>.
rmatharu commented on a change in pull request #1406:
URL: https://github.com/apache/samza/pull/1406#discussion_r463893096
##########
File path: samza-core/src/main/scala/org/apache/samza/diagnostics/DiagnosticsStreamMessage.java
##########
@@ -60,6 +62,7 @@
private static final String CONTAINER_THREAD_POOL_SIZE_METRIC_NAME = "containerThreadPoolSize";
private static final String CONTAINER_MODELS_METRIC_NAME = "containerModels";
private static final String AUTOSIZING_ENABLED_METRIC_NAME = "autosizingEnabled";
+ private static final String CONFIG = "config";
Review comment:
Nit: CONFIG_METRIC_NAME
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [samza] rmatharu commented on a change in pull request #1406: SAMZA-2561: Add config map to MetricsHeader
Posted by GitBox <gi...@apache.org>.
rmatharu commented on a change in pull request #1406:
URL: https://github.com/apache/samza/pull/1406#discussion_r463806408
##########
File path: samza-core/src/main/java/org/apache/samza/util/DiagnosticsUtil.java
##########
@@ -70,7 +70,7 @@ public static void writeMetadataFile(String jobName, String jobId, String contai
MetricsHeader metricsHeader =
new MetricsHeader(jobName, jobId, "samza-container-" + containerId, execEnvContainerId.orElse(""),
LocalContainerRunner.class.getName(), Util.getTaskClassVersion(config), Util.getSamzaVersion(),
- Util.getLocalHost().getHostName(), System.currentTimeMillis(), System.currentTimeMillis());
+ Util.getLocalHost().getHostName(), System.currentTimeMillis(), System.currentTimeMillis(), config);
Review comment:
Why do we want to add the config to the metric-header?
That way the entire config map (potentially 10s of MB of data) will be emitted in every message.
This is NOT what we want to do right?
Instead we want to emit it in the MetricsMessage, but exactly once at container start (first message) not in every message.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [samza] PawasChhokra commented on a change in pull request #1406: SAMZA-2561: Add config map to MetricsHeader
Posted by GitBox <gi...@apache.org>.
PawasChhokra commented on a change in pull request #1406:
URL: https://github.com/apache/samza/pull/1406#discussion_r465379110
##########
File path: samza-core/src/main/scala/org/apache/samza/diagnostics/DiagnosticsStreamMessage.java
##########
@@ -228,6 +239,14 @@ public Boolean getAutosizingEnabled() {
return (Boolean) getFromMetricsMessage(GROUP_NAME_FOR_DIAGNOSTICS_MANAGER, AUTOSIZING_ENABLED_METRIC_NAME);
}
+ /**
+ * This method gets the config of the job from the MetricsMessage.
+ * @return the config of the job.
+ */
+ public Config getConfig() {
+ return (Config) getFromMetricsMessage(GROUP_NAME_FOR_DIAGNOSTICS_MANAGER, CONFIG);
Review comment:
Done.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [samza] PawasChhokra commented on a change in pull request #1406: SAMZA-2561: Add config map to DiagnosticsManager
Posted by GitBox <gi...@apache.org>.
PawasChhokra commented on a change in pull request #1406:
URL: https://github.com/apache/samza/pull/1406#discussion_r465891747
##########
File path: samza-core/src/main/java/org/apache/samza/util/DiagnosticsUtil.java
##########
@@ -70,7 +70,7 @@ public static void writeMetadataFile(String jobName, String jobId, String contai
MetricsHeader metricsHeader =
new MetricsHeader(jobName, jobId, "samza-container-" + containerId, execEnvContainerId.orElse(""),
LocalContainerRunner.class.getName(), Util.getTaskClassVersion(config), Util.getSamzaVersion(),
- Util.getLocalHost().getHostName(), System.currentTimeMillis(), System.currentTimeMillis());
+ Util.getLocalHost().getHostName(), System.currentTimeMillis(), System.currentTimeMillis(), config);
Review comment:
Removed the config from the MetricsHeader.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [samza] rmatharu commented on a change in pull request #1406: SAMZA-2561: Add config map to MetricsHeader
Posted by GitBox <gi...@apache.org>.
rmatharu commented on a change in pull request #1406:
URL: https://github.com/apache/samza/pull/1406#discussion_r463858926
##########
File path: samza-core/src/main/scala/org/apache/samza/metrics/reporter/MetricsSnapshotReporter.scala
##########
@@ -148,7 +149,7 @@ class MetricsSnapshotReporter(
// publish to Kafka only if the metricsMsg carries any metrics
if (!metricsMsg.isEmpty) {
- val header = new MetricsHeader(jobName, jobId, containerName, execEnvironmentContainerId, source, version, samzaVersion, host, clock(), resetTime)
+ val header = new MetricsHeader(jobName, jobId, containerName, execEnvironmentContainerId, source, version, samzaVersion, host, clock(), resetTime, config)
Review comment:
see comment above.
##########
File path: samza-core/src/main/scala/org/apache/samza/metrics/reporter/MetricsSnapshotReporterFactory.scala
##########
@@ -124,7 +124,9 @@ class MetricsSnapshotReporterFactory extends MetricsReporterFactory with Logging
Util.getTaskClassVersion(config),
Util.getSamzaVersion,
Util.getLocalHost.getHostName,
- serde, blacklist)
+ serde,
+ blacklist,
+ config)
Review comment:
remove
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [samza] rmatharu commented on a change in pull request #1406: SAMZA-2561: Add config map to MetricsHeader
Posted by GitBox <gi...@apache.org>.
rmatharu commented on a change in pull request #1406:
URL: https://github.com/apache/samza/pull/1406#discussion_r463897962
##########
File path: samza-core/src/test/scala/org/apache/samza/serializers/TestMetricsSnapshotSerde.scala
##########
@@ -43,6 +39,9 @@ class TestMetricsSnapshotSerde {
val snapshot = new MetricsSnapshot(header, metrics)
val serde = new MetricsSnapshotSerde()
val bytes = serde.toBytes(snapshot)
- assertTrue(serde.fromBytes(bytes).equals(metrics))
+ val deserialized = serde.fromBytes(bytes)
+ assertTrue(deserialized.getHeader.jobName.equals(snapshot.getHeader.jobName))
+ assertTrue(deserialized.getHeader.jobId.equals(snapshot.getHeader.jobId))
+ assertTrue(deserialized.getMetrics.immutableMetrics.keySet().equals(snapshot.getMetrics.immutableMetrics.keySet()))
Review comment:
Remove?
This code branch and test need not be updated.
To test we need to test equality on the header-map and the metric map, not just their keyset, and specific keys (jobname, jobid)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [samza] rmatharu commented on a change in pull request #1406: SAMZA-2561: Add config map to MetricsHeader
Posted by GitBox <gi...@apache.org>.
rmatharu commented on a change in pull request #1406:
URL: https://github.com/apache/samza/pull/1406#discussion_r463898078
##########
File path: samza-core/src/test/java/org/apache/samza/metrics/TestMetricsSnapshotReporter.java
##########
@@ -35,10 +35,7 @@
import scala.Some;
import scala.runtime.AbstractFunction0;
-import static org.mockito.Mockito.mock;
-import static org.mockito.Mockito.verify;
-import static org.mockito.Mockito.times;
-import static org.mockito.Mockito.eq;
+import static org.mockito.Mockito.*;
Review comment:
remove?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [samza] rmatharu commented on a change in pull request #1406: SAMZA-2561: Add config map to MetricsHeader
Posted by GitBox <gi...@apache.org>.
rmatharu commented on a change in pull request #1406:
URL: https://github.com/apache/samza/pull/1406#discussion_r463856116
##########
File path: samza-core/src/main/scala/org/apache/samza/diagnostics/DiagnosticsStreamMessage.java
##########
@@ -155,6 +158,10 @@ public void addProcessorStopEvents(List<ProcessorStopEvent> stopEventList) {
}
}
+ public void addConfig(Config config) {
Review comment:
Javadoc?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [samza] rmatharu commented on a change in pull request #1406: SAMZA-2561: Add config map to DiagnosticsManager
Posted by GitBox <gi...@apache.org>.
rmatharu commented on a change in pull request #1406:
URL: https://github.com/apache/samza/pull/1406#discussion_r465383565
##########
File path: samza-core/src/main/scala/org/apache/samza/diagnostics/DiagnosticsStreamMessage.java
##########
@@ -155,6 +158,14 @@ public void addProcessorStopEvents(List<ProcessorStopEvent> stopEventList) {
}
}
+ /**
+ * Add the job's config to the message.
+ * @param config the config to add.
+ */
+ public void addConfig(Map<String, String> config) {
Review comment:
Nit: The method definition could be
`public void addConfig(Config config) {
addToMetricsMessage(GROUP_NAME_FOR_DIAGNOSTICS_MANAGER, CONFIG_METRIC_NAME, (Map<String, String>) config);
}`
So that the cast-to-map logic is encapsulated in this class; and not visible to callers.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [samza] PawasChhokra commented on a change in pull request #1406: SAMZA-2561: Add config map to MetricsHeader
Posted by GitBox <gi...@apache.org>.
PawasChhokra commented on a change in pull request #1406:
URL: https://github.com/apache/samza/pull/1406#discussion_r465377578
##########
File path: samza-core/src/test/scala/org/apache/samza/serializers/TestMetricsSnapshotSerde.scala
##########
@@ -43,6 +39,9 @@ class TestMetricsSnapshotSerde {
val snapshot = new MetricsSnapshot(header, metrics)
val serde = new MetricsSnapshotSerde()
val bytes = serde.toBytes(snapshot)
- assertTrue(serde.fromBytes(bytes).equals(metrics))
+ val deserialized = serde.fromBytes(bytes)
+ assertTrue(deserialized.getHeader.jobName.equals(snapshot.getHeader.jobName))
+ assertTrue(deserialized.getHeader.jobId.equals(snapshot.getHeader.jobId))
+ assertTrue(deserialized.getMetrics.immutableMetrics.keySet().equals(snapshot.getMetrics.immutableMetrics.keySet()))
Review comment:
Removed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [samza] rmatharu commented on a change in pull request #1406: SAMZA-2561: Add config map to MetricsHeader
Posted by GitBox <gi...@apache.org>.
rmatharu commented on a change in pull request #1406:
URL: https://github.com/apache/samza/pull/1406#discussion_r463860833
##########
File path: samza-core/src/test/java/org/apache/samza/diagnostics/TestDiagnosticsManager.java
##########
@@ -59,6 +62,7 @@
private int numPersistentStores = 2;
private int containerNumCores = 2;
private boolean autosizingEnabled = false;
+ private Config config = new MapConfig(ImmutableMap.of("job.name", jobName, "job.id", jobId));
Review comment:
Please add additional configs to make it somewhat representative of the configs that we want to emit.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [samza] rmatharu merged pull request #1406: SAMZA-2561: Add config map to DiagnosticsManager
Posted by GitBox <gi...@apache.org>.
rmatharu merged pull request #1406:
URL: https://github.com/apache/samza/pull/1406
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [samza] rmatharu commented on a change in pull request #1406: SAMZA-2561: Add config map to DiagnosticsManager
Posted by GitBox <gi...@apache.org>.
rmatharu commented on a change in pull request #1406:
URL: https://github.com/apache/samza/pull/1406#discussion_r465383565
##########
File path: samza-core/src/main/scala/org/apache/samza/diagnostics/DiagnosticsStreamMessage.java
##########
@@ -155,6 +158,14 @@ public void addProcessorStopEvents(List<ProcessorStopEvent> stopEventList) {
}
}
+ /**
+ * Add the job's config to the message.
+ * @param config the config to add.
+ */
+ public void addConfig(Map<String, String> config) {
Review comment:
Nit: The method definition could be
`public void addConfig(Config config) {
addToMetricsMessage(GROUP_NAME_FOR_DIAGNOSTICS_MANAGER, CONFIG_METRIC_NAME, (Map<String, String>) config);
}`
So that the cast-to-map logic is encapsulated in this class; and not visible to callers.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [samza] PawasChhokra commented on a change in pull request #1406: SAMZA-2561: Add config map to MetricsHeader
Posted by GitBox <gi...@apache.org>.
PawasChhokra commented on a change in pull request #1406:
URL: https://github.com/apache/samza/pull/1406#discussion_r465377657
##########
File path: samza-core/src/test/java/org/apache/samza/serializers/model/serializers/TestMetricsSnapshotSerdeV2.java
##########
@@ -37,8 +37,8 @@
@Test
public void testSerde() {
MetricsHeader metricsHeader =
- new MetricsHeader("jobName", "i001", "container 0", "test container ID", "source", "300.14.25.1", "1", "1", 1,
- 1);
+ new MetricsHeader("jobName", "i001", "container 0", "test container ID",
+ "source", "300.14.25.1", "1", "1", 1, 1);
Review comment:
Removed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [samza] rmatharu commented on a change in pull request #1406: SAMZA-2561: Add config map to MetricsHeader
Posted by GitBox <gi...@apache.org>.
rmatharu commented on a change in pull request #1406:
URL: https://github.com/apache/samza/pull/1406#discussion_r463857273
##########
File path: samza-core/src/main/scala/org/apache/samza/diagnostics/DiagnosticsStreamMessage.java
##########
@@ -237,7 +248,7 @@ public static DiagnosticsStreamMessage convertToDiagnosticsStreamMessage(Metrics
metricsSnapshot.getHeader().getContainerName(), metricsSnapshot.getHeader().getExecEnvironmentContainerId(),
metricsSnapshot.getHeader().getVersion(), metricsSnapshot.getHeader().getSamzaVersion(),
metricsSnapshot.getHeader().getHost(), metricsSnapshot.getHeader().getTime(),
- metricsSnapshot.getHeader().getResetTime());
+ metricsSnapshot.getHeader().getResetTime(), metricsSnapshot.getHeader().getConfig());
Review comment:
Not needed
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org