You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@apex.apache.org by davidyan74 <gi...@git.apache.org> on 2015/12/29 23:12:03 UTC
[GitHub] incubator-apex-core pull request: APEXCORE-267 made METRICS_TRANSP...
GitHub user davidyan74 opened a pull request:
https://github.com/apache/incubator-apex-core/pull/193
APEXCORE-267 made METRICS_TRANSPORT pluggable
@tweise please review and merge
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/davidyan74/incubator-apex-core APEXCORE-276
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-apex-core/pull/193.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #193
----
commit d6202d3c7c67114898764923d74d95eea96eb6e9
Author: David Yan <da...@datatorrent.com>
Date: 2015-12-29T21:46:58Z
APEXCORE-267 made METRICS_TRANSPORT pluggable
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-apex-core pull request: APEXCORE-276 made METRICS_TRANSP...
Posted by chandnisingh <gi...@git.apache.org>.
Github user chandnisingh commented on the pull request:
https://github.com/apache/incubator-apex-core/pull/193#issuecomment-167954861
A recommendation is to add a test which sets a custom metrics transport and see if that gets picked up
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-apex-core pull request: APEXCORE-276 made METRICS_TRANSP...
Posted by davidyan74 <gi...@git.apache.org>.
Github user davidyan74 commented on the pull request:
https://github.com/apache/incubator-apex-core/pull/193#issuecomment-168100563
Addressed all PR comments and rebased.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-apex-core pull request: APEXCORE-276 made METRICS_TRANSP...
Posted by davidyan74 <gi...@git.apache.org>.
Github user davidyan74 commented on the pull request:
https://github.com/apache/incubator-apex-core/pull/193#issuecomment-168076912
I just implemented what Thomas suggested, but I kept the METRICS_TRANSPORT attribute since it's not currently widely used but I can change it if people have concerns.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-apex-core pull request: APEXCORE-276 made METRICS_TRANSP...
Posted by davidyan74 <gi...@git.apache.org>.
Github user davidyan74 commented on the pull request:
https://github.com/apache/incubator-apex-core/pull/193#issuecomment-168049780
test is added
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-apex-core pull request: APEXCORE-276 made METRICS_TRANSP...
Posted by chandnisingh <gi...@git.apache.org>.
Github user chandnisingh commented on a diff in the pull request:
https://github.com/apache/incubator-apex-core/pull/193#discussion_r48619961
--- Diff: api/src/main/java/com/datatorrent/api/Context.java ---
@@ -335,10 +335,16 @@
Attribute<String> APPLICATION_DATA_LINK = new Attribute<String>(new String2String());
/**
* Transport to push the stats and the metrics, "builtin:{topic}" if STRAM should push the data directly
- * using websocket with the given topic
+ * using websocket with the given topic.
+ * If using a custom transport, please use Object2String codec to specify the transport object. The object must
+ * be from a class that implements the AutoMetric.Transport interface
*/
Attribute<String> METRICS_TRANSPORT = new Attribute<String>(new String2String());
--- End diff --
So what it means here is that we are asking the user to convert the object to string using Object2String codec before setting this attribute?
Why don't we deprecate this and add another attribute which is of type Attribute<AutoMetric.Transport> which is consistent with rest of the complex attributes?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-apex-core pull request: APEXCORE-276 made METRICS_TRANSP...
Posted by chandnisingh <gi...@git.apache.org>.
Github user chandnisingh commented on the pull request:
https://github.com/apache/incubator-apex-core/pull/193#issuecomment-168101819
There is no change made to japicmp config to exclude the attribute from semver, so just was wondering why it didn't complain. I don't have any problem with this change just curious to know if japicmp doesn't check for parameterized types
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-apex-core pull request: APEXCORE-276 made METRICS_TRANSP...
Posted by tweise <gi...@git.apache.org>.
Github user tweise commented on a diff in the pull request:
https://github.com/apache/incubator-apex-core/pull/193#discussion_r48622829
--- Diff: api/src/main/java/com/datatorrent/api/Context.java ---
@@ -335,10 +335,16 @@
Attribute<String> APPLICATION_DATA_LINK = new Attribute<String>(new String2String());
/**
* Transport to push the stats and the metrics, "builtin:{topic}" if STRAM should push the data directly
- * using websocket with the given topic
+ * using websocket with the given topic.
+ * If using a custom transport, please use Object2String codec to specify the transport object. The object must
+ * be from a class that implements the AutoMetric.Transport interface
*/
Attribute<String> METRICS_TRANSPORT = new Attribute<String>(new String2String());
/**
+ * The interval in milliseconds to resend metrics schema. Default value is 10,000 (10 seconds)
+ */
+ Attribute<Long> METRICS_SCHEMA_RESEND_INTERVAL = new Attribute<Long>(10000L);
--- End diff --
Shouldn't the resend interval be part of the transport as the need to do that is driven by the properties of the transport, i.e. need to resend if messages could be lost?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-apex-core pull request: APEXCORE-276 made METRICS_TRANSP...
Posted by tweise <gi...@git.apache.org>.
Github user tweise commented on the pull request:
https://github.com/apache/incubator-apex-core/pull/193#issuecomment-168046724
This is why is is important to not assume things as stable right of the bat. Is this attribute in its present form used anywhere? I don't think an end user would even be able to understand what to do with it? If so, we should consider it a bug and fix it.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-apex-core pull request: APEXCORE-276 made METRICS_TRANSP...
Posted by chandnisingh <gi...@git.apache.org>.
Github user chandnisingh commented on a diff in the pull request:
https://github.com/apache/incubator-apex-core/pull/193#discussion_r48590583
--- Diff: api/src/main/java/com/datatorrent/api/Context.java ---
@@ -335,10 +335,16 @@
Attribute<String> APPLICATION_DATA_LINK = new Attribute<String>(new String2String());
/**
* Transport to push the stats and the metrics, "builtin:{topic}" if STRAM should push the data directly
- * using websocket with the given topic
+ * using websocket with the given topic.
+ * If using a custom transport, please use Object2String codec to specify the transport object. The object must
+ * be from a class that implements the AutoMetric.Transport interface
*/
Attribute<String> METRICS_TRANSPORT = new Attribute<String>(new String2String());
--- End diff --
In the javadoc for the attribute Object2String codec is mentioned but in the declaration it is String2String
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-apex-core pull request: APEXCORE-276 made METRICS_TRANSP...
Posted by davidyan74 <gi...@git.apache.org>.
Github user davidyan74 commented on a diff in the pull request:
https://github.com/apache/incubator-apex-core/pull/193#discussion_r48619057
--- Diff: api/src/main/java/com/datatorrent/api/Context.java ---
@@ -335,10 +335,16 @@
Attribute<String> APPLICATION_DATA_LINK = new Attribute<String>(new String2String());
/**
* Transport to push the stats and the metrics, "builtin:{topic}" if STRAM should push the data directly
- * using websocket with the given topic
+ * using websocket with the given topic.
+ * If using a custom transport, please use Object2String codec to specify the transport object. The object must
+ * be from a class that implements the AutoMetric.Transport interface
*/
Attribute<String> METRICS_TRANSPORT = new Attribute<String>(new String2String());
--- End diff --
@chandnisingh this is for backward compatibility for the value "builtin:{topic}"
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-apex-core pull request: APEXCORE-276 made METRICS_TRANSP...
Posted by davidyan74 <gi...@git.apache.org>.
Github user davidyan74 commented on a diff in the pull request:
https://github.com/apache/incubator-apex-core/pull/193#discussion_r48621808
--- Diff: api/src/main/java/com/datatorrent/api/Context.java ---
@@ -335,10 +335,16 @@
Attribute<String> APPLICATION_DATA_LINK = new Attribute<String>(new String2String());
/**
* Transport to push the stats and the metrics, "builtin:{topic}" if STRAM should push the data directly
- * using websocket with the given topic
+ * using websocket with the given topic.
+ * If using a custom transport, please use Object2String codec to specify the transport object. The object must
+ * be from a class that implements the AutoMetric.Transport interface
*/
Attribute<String> METRICS_TRANSPORT = new Attribute<String>(new String2String());
--- End diff --
I originally wanted to do this, but how do we support this while supporting the built-in websocket transport at the same time? Looking for suggestions on how to do this a better way.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-apex-core pull request: APEXCORE-276 made METRICS_TRANSP...
Posted by tweise <gi...@git.apache.org>.
Github user tweise commented on a diff in the pull request:
https://github.com/apache/incubator-apex-core/pull/193#discussion_r48622986
--- Diff: api/src/main/java/com/datatorrent/api/Context.java ---
@@ -335,10 +335,16 @@
Attribute<String> APPLICATION_DATA_LINK = new Attribute<String>(new String2String());
/**
* Transport to push the stats and the metrics, "builtin:{topic}" if STRAM should push the data directly
- * using websocket with the given topic
+ * using websocket with the given topic.
+ * If using a custom transport, please use Object2String codec to specify the transport object. The object must
+ * be from a class that implements the AutoMetric.Transport interface
*/
Attribute<String> METRICS_TRANSPORT = new Attribute<String>(new String2String());
--- End diff --
As discussed, maybe look at the other option again to use a placeholder object for the "built-in" transport and swap it out for the engine internal implementation when wiring the dependencies in the AM?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-apex-core pull request: APEXCORE-276 made METRICS_TRANSP...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/incubator-apex-core/pull/193
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-apex-core pull request: APEXCORE-276 made METRICS_TRANSP...
Posted by tweise <gi...@git.apache.org>.
Github user tweise commented on a diff in the pull request:
https://github.com/apache/incubator-apex-core/pull/193#discussion_r48641557
--- Diff: engine/src/test/java/com/datatorrent/stram/StreamingContainerManagerTest.java ---
@@ -970,4 +969,48 @@ public void onClose(int closeCode, String message)
server.stop();
}
}
+
+ public static class TestMetricTransport implements AutoMetric.Transport, Serializable
+ {
+ private String prefix;
+ private static List<String> messages = new ArrayList<>();
+
+ public TestMetricTransport(String prefix)
+ {
+ this.prefix = prefix;
+ }
+
+ @Override
+ public void push(String jsonData) throws IOException
+ {
+ messages.add(prefix + ":" + jsonData);
+ }
+
+ @Override
+ public long getSchemaResendInterval()
+ {
+ return 0;
+ }
+ }
+
+ @Test
+ public void testCustomMetricsTransport() throws Exception
+ {
+ TestGeneratorInputOperator o1 = dag.addOperator("o1", TestGeneratorInputOperator.class);
+ GenericTestOperator o2 = dag.addOperator("o2", GenericTestOperator.class);
+ dag.addStream("o1.outport", o1.outport, o2.inport1);
+ dag.setAttribute(LogicalPlan.METRICS_TRANSPORT, new TestMetricTransport("xyz"));
+ StramLocalCluster lc = new StramLocalCluster(dag);
+ StreamingContainerManager dnmgr = lc.dnmgr;
+ StramAppContext appContext = new StramTestSupport.TestAppContext();
+
+ AppDataPushAgent pushAgent = new AppDataPushAgent(dnmgr, appContext);
+ pushAgent.init();
+ pushAgent.pushData();
+ Assert.assertTrue(TestMetricTransport.messages.size() > 0);
+ pushAgent.close();
+ String msg = TestMetricTransport.messages.get(0);
+ System.out.println("Got this message: " + msg);
--- End diff --
Not needed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-apex-core pull request: APEXCORE-276 made METRICS_TRANSP...
Posted by davidyan74 <gi...@git.apache.org>.
Github user davidyan74 commented on the pull request:
https://github.com/apache/incubator-apex-core/pull/193#issuecomment-167889209
@chandnisingh please review
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---