You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@beam.apache.org by "Aljoscha Krettek (JIRA)" <ji...@apache.org> on 2017/04/23 10:39:04 UTC
[jira] [Created] (BEAM-2056) Add tests for exporting Beam Metrics
to Flink Metrics
Aljoscha Krettek created BEAM-2056:
--------------------------------------
Summary: Add tests for exporting Beam Metrics to Flink Metrics
Key: BEAM-2056
URL: https://issues.apache.org/jira/browse/BEAM-2056
Project: Beam
Issue Type: Improvement
Components: runner-flink
Reporter: Aljoscha Krettek
Assignee: Aljoscha Krettek
Fix For: First stable release
There are currently no tests that verify that metrics that are reported using the Beam Metrics API are forwarded to Flink and a {{MetricReporter}}.
A test for this would have to manually configure a Flink "Mini Cluster", as in
{code}
// start also a re-usable Flink mini cluster
flink = new LocalFlinkMiniCluster(getFlinkConfiguration(), false);
flink.start();
flinkPort = flink.getLeaderRPCPort();
{code}
with {{getFlinkConfiguration()}}:
{code}
protected static Configuration getFlinkConfiguration() {
Configuration flinkConfig = new Configuration();
flinkConfig.setInteger(ConfigConstants.LOCAL_NUMBER_TASK_MANAGER, 1);
flinkConfig.setInteger(ConfigConstants.TASK_MANAGER_NUM_TASK_SLOTS, 8);
flinkConfig.setInteger(ConfigConstants.TASK_MANAGER_MEMORY_SIZE_KEY, 16);
flinkConfig.setString(ConfigConstants.RESTART_STRATEGY_FIXED_DELAY_DELAY, "0 s");
flinkConfig.setString(ConfigConstants.METRICS_REPORTERS_LIST, "my_reporter");
flinkConfig.setString(ConfigConstants.METRICS_REPORTER_PREFIX + "my_reporter." + ConfigConstants.METRICS_REPORTER_CLASS_SUFFIX, MyTestReporter.class.getName());
return flinkConfig;
}
{code}
where {{MyTestReporter}} is a {{MetricReporter}} that stores metrics being reported to it so we can verify that they are there after the job finishes.
Running a Pipeline on the mini cluster should be possible by specifying "localhost" and the port we received as a cluster endpoint.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)