You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@apex.apache.org by "Ganelin, Ilya" <Il...@capitalone.com> on 2015/09/02 19:47:16 UTC
Visualize custom metrics in the dashboard
Hi all – I’m attempting to surface some output directly to the existing Apex dashboard. Based on the Twitter example here:
https://github.com/DataTorrent/Malhar/blob/release-2.0.0/demos/twitter/src/main/java/com/datatorrent/demos/twitter/TwitterTrendingHashtagsApplication.java
With the API here:
https://www.datatorrent.com/docs/guides/DTGatewayAPISpecification.html#h.gsxbm0ivxe5y
I am defining a PubSub socket as such:
public static Operator.InputPort<Object> getWebOutput(DAG dag,
String operatorName) {
ConsoleOutputOperator operator = dag.addOperator(operatorName, new ConsoleOutputOperator());
operator.setStringFormat(operatorName + ": %s");
return operator.input;
}
public static Operator.InputPort<Object> getWebOutput(DAG dag,
String operatorName,
String topic) {
String gatewayAddress = dag.getValue(DAG.GATEWAY_CONNECT_ADDRESS);
if (!StringUtils.isEmpty(gatewayAddress)) {
URI uri = URI.create("ws://" + gatewayAddress + "/pubsub");
LOG.info("WebSocket with gateway at: " + gatewayAddress);
PubSubWebSocketOutputOperator<Object> wsOut = dag.addOperator(operatorName,
new PubSubWebSocketOutputOperator<>());
wsOut.setUri(uri);
wsOut.setTopic(topic);
return wsOut.input;
}
return getWebOutput(dag, operatorName);
}
I however see the following error:
Traceback (most recent call last):
File "<string>", line 4, in <module>
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/__init__.py", line 338, in loads
return _default_decoder.decode(s)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/decoder.py", line 365, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/decoder.py", line 383, in raw_decode
raise ValueError("No JSON object could be decoded")
ValueError: No JSON object could be decoded
An example of the JSON object I’m outputting is below:
{"type":"data", "topic":"test1", "data":["Elapsed":"1234", "Max (30s)":"5678ms", "Avg (30s)":"2000.00"ms, "Msgs/sec":"5000"]}
Any thoughts on what’s going on here or how I can debug this? The job does not ever appear to even launch. Thanks!
________________________________________________________
The information contained in this e-mail is confidential and/or proprietary to Capital One and/or its affiliates and may only be used solely in performance of work or services for Capital One. The information transmitted herewith is intended only for use by the individual or entity to which it is addressed. If the reader of this message is not the intended recipient, you are hereby notified that any review, retransmission, dissemination, distribution, copying or other use of, or taking of any action in reliance upon this information is strictly prohibited. If you have received this communication in error, please contact the sender and delete the material from your computer.
Re: Visualize custom metrics in the dashboard
Posted by "Ganelin, Ilya" <Il...@capitalone.com>.
Got it - thanks for the in-depth explanation. I’ll work through this and
let you know how it goes. Thanks!
On 9/3/15, 1:26 PM, "Timothy Farkas" <ti...@datatorrent.com> wrote:
>Hey Ilya,
>
>We can discuss the simple AppData operators present in Malhar in this
>thread. Operators which can perform aggregations over time buckets, store
>results to hdfs using hdht, and make the aggregations available for
>visualization are not open source and will have to be discussed offline if
>you want to use them.
>
>There are three components in enabling an application for visualization:
>
>- A Query operator which receives queries from the UI
>- A Store operator which receives queries from the Query operator and has
>the data required to respond to the query.
>- A Result operator which receives the data to reply to a query from the
>Store operator and send the result to the ui
>
>The sub-DAG of an application enabled application looks like this
>
>(Query Operator) -> (Store Operator) -> (Result Operator)
>
>- The Query operator to use is PubSubWebSocketAppDataQuery, The topic and
>uri need to be set on this like in your code example.
>- The Store operator to use is AppDataSnapshotServerMap. This operator
>accepts a list of Map<String,Object> objects on its input port. The latest
>list of maps is what is served in response to a query. Each Map in the
>list
>represents a row in a table. The key of a map corresponds to a column
>name,
>and the value corresponds to the value of that column for that row. The
>Store requires a schema to be set on it. The schema defines the columns of
>the table that the operator holds, and the types of the columns. An
>example
>of a schema is the following:
>
>{
> "values": [{"name": "hashtag", "type": "string"},
> {"name": "count", "type": "integer"}]
>}
>
>In this example the store has two columns hashtag and count of types
>string
>and integer respectively. When the List of Maps is sent to the store, the
>store will be expecting each map to have a "hashtag" key with a string
>value and a "name" key with an integer value.
>
>- The Result operator to use is PubSubWebSocketAppDataResult. The topic
>and
>uri need to be set on this like in your code example.
>
>The twitter demo in 3.0 is doing this:
>https://github.com/apache/incubator-apex-malhar/blob/master/demos/twitter/
>src/main/java/com/datatorrent/demos/twitter/TwitterTopCounterApplication.j
>ava
>
>
> URI uri = URI.create("ws://" + gatewayAddress + "/pubsub");
>
> AppDataSnapshotServerMap snapshotServer = dag.addOperator("Snapshot
>Server", new AppDataSnapshotServerMap());
>
> //Dont Worry about this conversion Map for your use case
> Map<String, String> conversionMap = Maps.newHashMap();
> conversionMap.put(alias, WindowedTopCounter.FIELD_TYPE);
> //This gets the json which is a schema that defines the types of the
>data that the operator expects to receive
> String snapshotServerJSON =
>SchemaUtils.jarResourceFileToString(schemaFile);
>
> //Set the schema on the store so that it knows the type of data to
>expect
> snapshotServer.setSnapshotSchemaJSON(snapshotServerJSON);
> //Don't set the conversion map for your use case
> snapshotServer.setTableFieldToMapField(conversionMap);
>
> PubSubWebSocketAppDataQuery wsQuery = dag.addOperator("Query", new
>PubSubWebSocketAppDataQuery());
> wsQuery.setUri(uri);
> Operator.OutputPort<String> queryPort = wsQuery.outputPort;
> PubSubWebSocketAppDataResult wsResult =
>dag.addOperator("QueryResult", new PubSubWebSocketAppDataResult());
> wsResult.setUri(uri);
> Operator.InputPort<String> queryResultPort = wsResult.input;
>
> dag.addStream("MapProvider", topCount, snapshotServer.input);
> dag.addStream("Query", queryPort, snapshotServer.query);
> dag.addStream("Result", snapshotServer.queryResult,
>queryResultPort);
>
>
>Once you have this portion of the Dag built you just need to have an
>upstream operator periodically send the AppDataSnapshotServer (on its
>input
>port) a List of Maps containing the data that you want made available to
>query. In the case of twitter, the Top 10 hashtags, urls, or words are
>sent
>to the snapshot server at the end of each application window. The operator
>which does this is this:
>
>https://github.com/apache/incubator-apex-malhar/blob/master/demos/twitter/
>src/main/java/com/datatorrent/demos/twitter/WindowedTopCounter.java
>
>On line 145 you'll be able to see the List of Maps being built and sent
>out
>to the Snapshot Server.
>
>Once you have your application built, you can launch it. After launching
>you can click on the Application's name, near the top of the screen there
>will be a Visualize button, which you can click. If you click generate
>dashboard, a page should pop up and start displaying charts automatically.
>
>Please let us know how it goes,
>Tim
>
>
>On Thu, Sep 3, 2015 at 7:07 AM, Ganelin, Ilya
><Il...@capitalone.com>
>wrote:
>
>> Correct
>>
>>
>>
>> Thank you,
>> Ilya Ganelin
>>
>>
>>
>> -----Original Message-----
>> From: Timothy Farkas [tim@datatorrent.com<ma...@datatorrent.com>]
>> Sent: Thursday, September 03, 2015 03:41 AM Eastern Standard Time
>> To: dev@apex.incubator.apache.org
>> Subject: Re: Visualize custom metrics in the dashboard
>>
>>
>> Cool just to check though do you have the enterprise license for 3.0?
>>2.0
>> does not include the visualization tools.
>>
>> Thanks,
>> Tim
>>
>> On Wed, Sep 2, 2015 at 11:57 PM, Ganelin, Ilya <
>> Ilya.Ganelin@capitalone.com>
>> wrote:
>>
>> > Hi Timothy - we do actually have an enterprise license and would
>> > appreciate more details on how to use those operators. Thanks!
>> >
>> >
>> >
>> > Thank you,
>> > Ilya Ganelin
>> >
>> >
>> >
>> > -----Original Message-----
>> > From: Timothy Farkas [tim@datatorrent.com<ma...@datatorrent.com>]
>> > Sent: Thursday, September 03, 2015 12:15 AM Eastern Standard Time
>> > To: dev@apex.incubator.apache.org
>> > Subject: Re: Visualize custom metrics in the dashboard
>> >
>> >
>> > Hi Ilya,
>> >
>> > I believe the overall approach is correct. Is {"type":"data",
>> > "topic":"c1.citadel.latencies", "data":[{"Elapsed":"3987"}, {"Max
>> > (30s)":"3987ms"}, {"Avg (30s)":"257.00ms"}, {"Msgs/sec":"48"}]} the
>>JSON
>> > that you are seeing output by the PubSubWebsocketOutputOperator? Also
>>is
>> > the receiver of the message sending a {"type":"subscribe",
>> > "topic":"c1.citadel.latencies"} message?
>> >
>> > A quick way to debug whether your messages can be sent or received
>> properly
>> > is to use Simple Websocket
>> >
>> >
>>
>>https://chrome.google.com/webstore/detail/simple-websocket-client/pfdhobl
>>ngboilpfeibdedpjgfnlcodoo?hl=en
>> >
>> > -You can use it to connect to ws://your.gateway/pubsub
>> > -Then enter {"type":"subscribe", "topic":"c1.citadel.latencies"}
>> > -Then enter {"type":"data", "topic":"c1.citadel.latencies",
>> > "data":[{"Elapsed":"3987"},
>> > {"Max (30s)":"3987ms"}, {"Avg (30s)":"257.00ms"} This will not echo
>>the
>> > result back to you
>> > - If you enter {"type":"publish", "topic":"c1.citadel.latencies",
>> > "data":[{"Elapsed":"3987"},
>> > {"Max (30s)":"3987ms"}, {"Avg (30s)":"257.00ms"} This will echo the
>> result
>> > back to you
>> >
>> > Note that if you have a fresh connection and you don't send the
>>subscribe
>> > message, you will not get the result echoed back to you.
>> >
>> > There is an easier way. Datatorrent already has operators which serve
>> data
>> > to the UI and Widgets which display the data in the UI, but that is in
>> the
>> > Enterprise edition.
>> >
>> > Hope this helps, please keep us posted
>> > Tim
>> >
>> > On Wed, Sep 2, 2015 at 8:17 PM, Ganelin, Ilya <
>> Ilya.Ganelin@capitalone.com
>> > >
>> > wrote:
>> >
>> > > Hi Timothy - the idea was to surface our own metrics to the
>>DataTorrent
>> > > UI. We have been rolling with this approach since it was
>>recommended to
>> > us
>> > > by DT folks. Is there a better way to visualize metrics in the UI?
>> > >
>> > >
>> > >
>> > > Thank you,
>> > > Ilya Ganelin
>> > >
>> > >
>> > >
>> > > -----Original Message-----
>> > > From: Timothy Farkas
>>[tim@datatorrent.com<ma...@datatorrent.com>]
>> > > Sent: Wednesday, September 02, 2015 10:24 PM Eastern Standard Time
>> > > To: dev@apex.incubator.apache.org
>> > > Subject: Re: Visualize custom metrics in the dashboard
>> > >
>> > >
>> > > Hi Ilya,
>> > >
>> > > It looks like you are using Gateway as your websocket server. The
>> > protocol
>> > > to send messages to gateway is to use "publish" as you "type" not
>> "data".
>> > > Also whoever is receiving data from the gateway needs to send a
>> subscribe
>> > > message like this to gateway first
>> > {"type":"subscribe","topic":"myTopic"},
>> > > before they will receive any data from a topic.
>> > >
>> > > Tim
>> > >
>> > > On Wed, Sep 2, 2015 at 6:18 PM, Ganelin, Ilya <
>> > Ilya.Ganelin@capitalone.com
>> > > >
>> > > wrote:
>> > >
>> > > > Chetan, thanks! I fixed that error, and validated the JSON shown
>> below,
>> > > > any further ideas?
>> > > >
>> > > > {"type":"data", "topic":"c1.citadel.latencies",
>> > > > "data":[{"Elapsed":"3987"}, {"Max (30s)":"3987ms"}, {"Avg
>> > > > (30s)":"257.00ms"}, {"Msgs/sec":"48"}]}
>> > > >
>> > > > On 9/2/15, 12:28 PM, "Chetan Narsude" <ch...@datatorrent.com>
>> wrote:
>> > > >
>> > > > >On Wed, Sep 2, 2015 at 10:47 AM, Ganelin, Ilya
>> > > > ><Il...@capitalone.com>
>> > > > >wrote:
>> > > > >
>> > > > >> Hi all I¹m attempting to surface some output directly to the
>> > > existing
>> > > > >> Apex dashboard. Based on the Twitter example here:
>> > > > >>
>> > > > >>
>> > > > >>
>> > > >
>> > >
>> >
>>
>>https://github.com/DataTorrent/Malhar/blob/release-2.0.0/demos/twitter/sr
>> > > >
>> > >
>> >
>>
>>>>c/main/java/com/datatorrent/demos/twitter/TwitterTrendingHashtagsApplic
>>>>at
>> > > > >>ion.java
>> > > > >> [snip]
>> > > > >>
>> > > > >>
>> > > > >> An example of the JSON object I¹m outputting is below:
>> > > > >> {"type":"data", "topic":"test1", "data":["Elapsed":"1234", "Max
>> > > > >> (30s)":"5678ms", "Avg (30s)":*"2000.00"ms*, "Msgs/sec":"5000"]}
>> > > > >>
>> > > > >>
>> > > > >The highlighted portion (*"2000.00"ms*) above looks pretty wrong
>>to
>> me
>> > > :-)
>> > > > >
>> > > > >--
>> > > > >Chetan
>> > > > >
>> > > > >
>> > > > >
>> > > > >>
>> > > > >> Any thoughts on what¹s going on here or how I can debug this?
>>The
>> > job
>> > > > >>does
>> > > > >> not ever appear to even launch. Thanks!
>> > > > >> ________________________________________________________
>> > > > >>
>> > > > >> The information contained in this e-mail is confidential and/or
>> > > > >> proprietary to Capital One and/or its affiliates and may only
>>be
>> > used
>> > > > >> solely in performance of work or services for Capital One. The
>> > > > >>information
>> > > > >> transmitted herewith is intended only for use by the
>>individual or
>> > > > >>entity
>> > > > >> to which it is addressed. If the reader of this message is not
>>the
>> > > > >>intended
>> > > > >> recipient, you are hereby notified that any review,
>> retransmission,
>> > > > >> dissemination, distribution, copying or other use of, or
>>taking of
>> > any
>> > > > >> action in reliance upon this information is strictly
>>prohibited.
>> If
>> > > you
>> > > > >> have received this communication in error, please contact the
>> sender
>> > > and
>> > > > >> delete the material from your computer.
>> > > > >>
>> > > >
>> > > > ________________________________________________________
>> > > >
>> > > > The information contained in this e-mail is confidential and/or
>> > > > proprietary to Capital One and/or its affiliates and may only be
>>used
>> > > > solely in performance of work or services for Capital One. The
>> > > information
>> > > > transmitted herewith is intended only for use by the individual or
>> > entity
>> > > > to which it is addressed. If the reader of this message is not the
>> > > intended
>> > > > recipient, you are hereby notified that any review,
>>retransmission,
>> > > > dissemination, distribution, copying or other use of, or taking of
>> any
>> > > > action in reliance upon this information is strictly prohibited.
>>If
>> you
>> > > > have received this communication in error, please contact the
>>sender
>> > and
>> > > > delete the material from your computer.
>> > > >
>> > > >
>> > > ________________________________________________________
>> > >
>> > > The information contained in this e-mail is confidential and/or
>> > > proprietary to Capital One and/or its affiliates and may only be
>>used
>> > > solely in performance of work or services for Capital One. The
>> > information
>> > > transmitted herewith is intended only for use by the individual or
>> entity
>> > > to which it is addressed. If the reader of this message is not the
>> > intended
>> > > recipient, you are hereby notified that any review, retransmission,
>> > > dissemination, distribution, copying or other use of, or taking of
>>any
>> > > action in reliance upon this information is strictly prohibited. If
>>you
>> > > have received this communication in error, please contact the sender
>> and
>> > > delete the material from your computer.
>> > >
>> > ________________________________________________________
>> >
>> > The information contained in this e-mail is confidential and/or
>> > proprietary to Capital One and/or its affiliates and may only be used
>> > solely in performance of work or services for Capital One. The
>> information
>> > transmitted herewith is intended only for use by the individual or
>>entity
>> > to which it is addressed. If the reader of this message is not the
>> intended
>> > recipient, you are hereby notified that any review, retransmission,
>> > dissemination, distribution, copying or other use of, or taking of any
>> > action in reliance upon this information is strictly prohibited. If
>>you
>> > have received this communication in error, please contact the sender
>>and
>> > delete the material from your computer.
>> >
>> ________________________________________________________
>>
>> The information contained in this e-mail is confidential and/or
>> proprietary to Capital One and/or its affiliates and may only be used
>> solely in performance of work or services for Capital One. The
>>information
>> transmitted herewith is intended only for use by the individual or
>>entity
>> to which it is addressed. If the reader of this message is not the
>>intended
>> recipient, you are hereby notified that any review, retransmission,
>> dissemination, distribution, copying or other use of, or taking of any
>> action in reliance upon this information is strictly prohibited. If you
>> have received this communication in error, please contact the sender and
>> delete the material from your computer.
>>
________________________________________________________
The information contained in this e-mail is confidential and/or proprietary to Capital One and/or its affiliates and may only be used solely in performance of work or services for Capital One. The information transmitted herewith is intended only for use by the individual or entity to which it is addressed. If the reader of this message is not the intended recipient, you are hereby notified that any review, retransmission, dissemination, distribution, copying or other use of, or taking of any action in reliance upon this information is strictly prohibited. If you have received this communication in error, please contact the sender and delete the material from your computer.
Re: Visualize custom metrics in the dashboard
Posted by Timothy Farkas <ti...@datatorrent.com>.
Hey Ilya,
We can discuss the simple AppData operators present in Malhar in this
thread. Operators which can perform aggregations over time buckets, store
results to hdfs using hdht, and make the aggregations available for
visualization are not open source and will have to be discussed offline if
you want to use them.
There are three components in enabling an application for visualization:
- A Query operator which receives queries from the UI
- A Store operator which receives queries from the Query operator and has
the data required to respond to the query.
- A Result operator which receives the data to reply to a query from the
Store operator and send the result to the ui
The sub-DAG of an application enabled application looks like this
(Query Operator) -> (Store Operator) -> (Result Operator)
- The Query operator to use is PubSubWebSocketAppDataQuery, The topic and
uri need to be set on this like in your code example.
- The Store operator to use is AppDataSnapshotServerMap. This operator
accepts a list of Map<String,Object> objects on its input port. The latest
list of maps is what is served in response to a query. Each Map in the list
represents a row in a table. The key of a map corresponds to a column name,
and the value corresponds to the value of that column for that row. The
Store requires a schema to be set on it. The schema defines the columns of
the table that the operator holds, and the types of the columns. An example
of a schema is the following:
{
"values": [{"name": "hashtag", "type": "string"},
{"name": "count", "type": "integer"}]
}
In this example the store has two columns hashtag and count of types string
and integer respectively. When the List of Maps is sent to the store, the
store will be expecting each map to have a "hashtag" key with a string
value and a "name" key with an integer value.
- The Result operator to use is PubSubWebSocketAppDataResult. The topic and
uri need to be set on this like in your code example.
The twitter demo in 3.0 is doing this:
https://github.com/apache/incubator-apex-malhar/blob/master/demos/twitter/src/main/java/com/datatorrent/demos/twitter/TwitterTopCounterApplication.java
URI uri = URI.create("ws://" + gatewayAddress + "/pubsub");
AppDataSnapshotServerMap snapshotServer = dag.addOperator("Snapshot
Server", new AppDataSnapshotServerMap());
//Dont Worry about this conversion Map for your use case
Map<String, String> conversionMap = Maps.newHashMap();
conversionMap.put(alias, WindowedTopCounter.FIELD_TYPE);
//This gets the json which is a schema that defines the types of the
data that the operator expects to receive
String snapshotServerJSON =
SchemaUtils.jarResourceFileToString(schemaFile);
//Set the schema on the store so that it knows the type of data to
expect
snapshotServer.setSnapshotSchemaJSON(snapshotServerJSON);
//Don't set the conversion map for your use case
snapshotServer.setTableFieldToMapField(conversionMap);
PubSubWebSocketAppDataQuery wsQuery = dag.addOperator("Query", new
PubSubWebSocketAppDataQuery());
wsQuery.setUri(uri);
Operator.OutputPort<String> queryPort = wsQuery.outputPort;
PubSubWebSocketAppDataResult wsResult =
dag.addOperator("QueryResult", new PubSubWebSocketAppDataResult());
wsResult.setUri(uri);
Operator.InputPort<String> queryResultPort = wsResult.input;
dag.addStream("MapProvider", topCount, snapshotServer.input);
dag.addStream("Query", queryPort, snapshotServer.query);
dag.addStream("Result", snapshotServer.queryResult, queryResultPort);
Once you have this portion of the Dag built you just need to have an
upstream operator periodically send the AppDataSnapshotServer (on its input
port) a List of Maps containing the data that you want made available to
query. In the case of twitter, the Top 10 hashtags, urls, or words are sent
to the snapshot server at the end of each application window. The operator
which does this is this:
https://github.com/apache/incubator-apex-malhar/blob/master/demos/twitter/src/main/java/com/datatorrent/demos/twitter/WindowedTopCounter.java
On line 145 you'll be able to see the List of Maps being built and sent out
to the Snapshot Server.
Once you have your application built, you can launch it. After launching
you can click on the Application's name, near the top of the screen there
will be a Visualize button, which you can click. If you click generate
dashboard, a page should pop up and start displaying charts automatically.
Please let us know how it goes,
Tim
On Thu, Sep 3, 2015 at 7:07 AM, Ganelin, Ilya <Il...@capitalone.com>
wrote:
> Correct
>
>
>
> Thank you,
> Ilya Ganelin
>
>
>
> -----Original Message-----
> From: Timothy Farkas [tim@datatorrent.com<ma...@datatorrent.com>]
> Sent: Thursday, September 03, 2015 03:41 AM Eastern Standard Time
> To: dev@apex.incubator.apache.org
> Subject: Re: Visualize custom metrics in the dashboard
>
>
> Cool just to check though do you have the enterprise license for 3.0? 2.0
> does not include the visualization tools.
>
> Thanks,
> Tim
>
> On Wed, Sep 2, 2015 at 11:57 PM, Ganelin, Ilya <
> Ilya.Ganelin@capitalone.com>
> wrote:
>
> > Hi Timothy - we do actually have an enterprise license and would
> > appreciate more details on how to use those operators. Thanks!
> >
> >
> >
> > Thank you,
> > Ilya Ganelin
> >
> >
> >
> > -----Original Message-----
> > From: Timothy Farkas [tim@datatorrent.com<ma...@datatorrent.com>]
> > Sent: Thursday, September 03, 2015 12:15 AM Eastern Standard Time
> > To: dev@apex.incubator.apache.org
> > Subject: Re: Visualize custom metrics in the dashboard
> >
> >
> > Hi Ilya,
> >
> > I believe the overall approach is correct. Is {"type":"data",
> > "topic":"c1.citadel.latencies", "data":[{"Elapsed":"3987"}, {"Max
> > (30s)":"3987ms"}, {"Avg (30s)":"257.00ms"}, {"Msgs/sec":"48"}]} the JSON
> > that you are seeing output by the PubSubWebsocketOutputOperator? Also is
> > the receiver of the message sending a {"type":"subscribe",
> > "topic":"c1.citadel.latencies"} message?
> >
> > A quick way to debug whether your messages can be sent or received
> properly
> > is to use Simple Websocket
> >
> >
> https://chrome.google.com/webstore/detail/simple-websocket-client/pfdhoblngboilpfeibdedpjgfnlcodoo?hl=en
> >
> > -You can use it to connect to ws://your.gateway/pubsub
> > -Then enter {"type":"subscribe", "topic":"c1.citadel.latencies"}
> > -Then enter {"type":"data", "topic":"c1.citadel.latencies",
> > "data":[{"Elapsed":"3987"},
> > {"Max (30s)":"3987ms"}, {"Avg (30s)":"257.00ms"} This will not echo the
> > result back to you
> > - If you enter {"type":"publish", "topic":"c1.citadel.latencies",
> > "data":[{"Elapsed":"3987"},
> > {"Max (30s)":"3987ms"}, {"Avg (30s)":"257.00ms"} This will echo the
> result
> > back to you
> >
> > Note that if you have a fresh connection and you don't send the subscribe
> > message, you will not get the result echoed back to you.
> >
> > There is an easier way. Datatorrent already has operators which serve
> data
> > to the UI and Widgets which display the data in the UI, but that is in
> the
> > Enterprise edition.
> >
> > Hope this helps, please keep us posted
> > Tim
> >
> > On Wed, Sep 2, 2015 at 8:17 PM, Ganelin, Ilya <
> Ilya.Ganelin@capitalone.com
> > >
> > wrote:
> >
> > > Hi Timothy - the idea was to surface our own metrics to the DataTorrent
> > > UI. We have been rolling with this approach since it was recommended to
> > us
> > > by DT folks. Is there a better way to visualize metrics in the UI?
> > >
> > >
> > >
> > > Thank you,
> > > Ilya Ganelin
> > >
> > >
> > >
> > > -----Original Message-----
> > > From: Timothy Farkas [tim@datatorrent.com<ma...@datatorrent.com>]
> > > Sent: Wednesday, September 02, 2015 10:24 PM Eastern Standard Time
> > > To: dev@apex.incubator.apache.org
> > > Subject: Re: Visualize custom metrics in the dashboard
> > >
> > >
> > > Hi Ilya,
> > >
> > > It looks like you are using Gateway as your websocket server. The
> > protocol
> > > to send messages to gateway is to use "publish" as you "type" not
> "data".
> > > Also whoever is receiving data from the gateway needs to send a
> subscribe
> > > message like this to gateway first
> > {"type":"subscribe","topic":"myTopic"},
> > > before they will receive any data from a topic.
> > >
> > > Tim
> > >
> > > On Wed, Sep 2, 2015 at 6:18 PM, Ganelin, Ilya <
> > Ilya.Ganelin@capitalone.com
> > > >
> > > wrote:
> > >
> > > > Chetan, thanks! I fixed that error, and validated the JSON shown
> below,
> > > > any further ideas?
> > > >
> > > > {"type":"data", "topic":"c1.citadel.latencies",
> > > > "data":[{"Elapsed":"3987"}, {"Max (30s)":"3987ms"}, {"Avg
> > > > (30s)":"257.00ms"}, {"Msgs/sec":"48"}]}
> > > >
> > > > On 9/2/15, 12:28 PM, "Chetan Narsude" <ch...@datatorrent.com>
> wrote:
> > > >
> > > > >On Wed, Sep 2, 2015 at 10:47 AM, Ganelin, Ilya
> > > > ><Il...@capitalone.com>
> > > > >wrote:
> > > > >
> > > > >> Hi all I¹m attempting to surface some output directly to the
> > > existing
> > > > >> Apex dashboard. Based on the Twitter example here:
> > > > >>
> > > > >>
> > > > >>
> > > >
> > >
> >
> https://github.com/DataTorrent/Malhar/blob/release-2.0.0/demos/twitter/sr
> > > >
> > >
> >
> >>c/main/java/com/datatorrent/demos/twitter/TwitterTrendingHashtagsApplicat
> > > > >>ion.java
> > > > >> [snip]
> > > > >>
> > > > >>
> > > > >> An example of the JSON object I¹m outputting is below:
> > > > >> {"type":"data", "topic":"test1", "data":["Elapsed":"1234", "Max
> > > > >> (30s)":"5678ms", "Avg (30s)":*"2000.00"ms*, "Msgs/sec":"5000"]}
> > > > >>
> > > > >>
> > > > >The highlighted portion (*"2000.00"ms*) above looks pretty wrong to
> me
> > > :-)
> > > > >
> > > > >--
> > > > >Chetan
> > > > >
> > > > >
> > > > >
> > > > >>
> > > > >> Any thoughts on what¹s going on here or how I can debug this? The
> > job
> > > > >>does
> > > > >> not ever appear to even launch. Thanks!
> > > > >> ________________________________________________________
> > > > >>
> > > > >> The information contained in this e-mail is confidential and/or
> > > > >> proprietary to Capital One and/or its affiliates and may only be
> > used
> > > > >> solely in performance of work or services for Capital One. The
> > > > >>information
> > > > >> transmitted herewith is intended only for use by the individual or
> > > > >>entity
> > > > >> to which it is addressed. If the reader of this message is not the
> > > > >>intended
> > > > >> recipient, you are hereby notified that any review,
> retransmission,
> > > > >> dissemination, distribution, copying or other use of, or taking of
> > any
> > > > >> action in reliance upon this information is strictly prohibited.
> If
> > > you
> > > > >> have received this communication in error, please contact the
> sender
> > > and
> > > > >> delete the material from your computer.
> > > > >>
> > > >
> > > > ________________________________________________________
> > > >
> > > > The information contained in this e-mail is confidential and/or
> > > > proprietary to Capital One and/or its affiliates and may only be used
> > > > solely in performance of work or services for Capital One. The
> > > information
> > > > transmitted herewith is intended only for use by the individual or
> > entity
> > > > to which it is addressed. If the reader of this message is not the
> > > intended
> > > > recipient, you are hereby notified that any review, retransmission,
> > > > dissemination, distribution, copying or other use of, or taking of
> any
> > > > action in reliance upon this information is strictly prohibited. If
> you
> > > > have received this communication in error, please contact the sender
> > and
> > > > delete the material from your computer.
> > > >
> > > >
> > > ________________________________________________________
> > >
> > > The information contained in this e-mail is confidential and/or
> > > proprietary to Capital One and/or its affiliates and may only be used
> > > solely in performance of work or services for Capital One. The
> > information
> > > transmitted herewith is intended only for use by the individual or
> entity
> > > to which it is addressed. If the reader of this message is not the
> > intended
> > > recipient, you are hereby notified that any review, retransmission,
> > > dissemination, distribution, copying or other use of, or taking of any
> > > action in reliance upon this information is strictly prohibited. If you
> > > have received this communication in error, please contact the sender
> and
> > > delete the material from your computer.
> > >
> > ________________________________________________________
> >
> > The information contained in this e-mail is confidential and/or
> > proprietary to Capital One and/or its affiliates and may only be used
> > solely in performance of work or services for Capital One. The
> information
> > transmitted herewith is intended only for use by the individual or entity
> > to which it is addressed. If the reader of this message is not the
> intended
> > recipient, you are hereby notified that any review, retransmission,
> > dissemination, distribution, copying or other use of, or taking of any
> > action in reliance upon this information is strictly prohibited. If you
> > have received this communication in error, please contact the sender and
> > delete the material from your computer.
> >
> ________________________________________________________
>
> The information contained in this e-mail is confidential and/or
> proprietary to Capital One and/or its affiliates and may only be used
> solely in performance of work or services for Capital One. The information
> transmitted herewith is intended only for use by the individual or entity
> to which it is addressed. If the reader of this message is not the intended
> recipient, you are hereby notified that any review, retransmission,
> dissemination, distribution, copying or other use of, or taking of any
> action in reliance upon this information is strictly prohibited. If you
> have received this communication in error, please contact the sender and
> delete the material from your computer.
>
RE: Visualize custom metrics in the dashboard
Posted by "Ganelin, Ilya" <Il...@capitalone.com>.
Correct
Thank you,
Ilya Ganelin
-----Original Message-----
From: Timothy Farkas [tim@datatorrent.com<ma...@datatorrent.com>]
Sent: Thursday, September 03, 2015 03:41 AM Eastern Standard Time
To: dev@apex.incubator.apache.org
Subject: Re: Visualize custom metrics in the dashboard
Cool just to check though do you have the enterprise license for 3.0? 2.0
does not include the visualization tools.
Thanks,
Tim
On Wed, Sep 2, 2015 at 11:57 PM, Ganelin, Ilya <Il...@capitalone.com>
wrote:
> Hi Timothy - we do actually have an enterprise license and would
> appreciate more details on how to use those operators. Thanks!
>
>
>
> Thank you,
> Ilya Ganelin
>
>
>
> -----Original Message-----
> From: Timothy Farkas [tim@datatorrent.com<ma...@datatorrent.com>]
> Sent: Thursday, September 03, 2015 12:15 AM Eastern Standard Time
> To: dev@apex.incubator.apache.org
> Subject: Re: Visualize custom metrics in the dashboard
>
>
> Hi Ilya,
>
> I believe the overall approach is correct. Is {"type":"data",
> "topic":"c1.citadel.latencies", "data":[{"Elapsed":"3987"}, {"Max
> (30s)":"3987ms"}, {"Avg (30s)":"257.00ms"}, {"Msgs/sec":"48"}]} the JSON
> that you are seeing output by the PubSubWebsocketOutputOperator? Also is
> the receiver of the message sending a {"type":"subscribe",
> "topic":"c1.citadel.latencies"} message?
>
> A quick way to debug whether your messages can be sent or received properly
> is to use Simple Websocket
>
> https://chrome.google.com/webstore/detail/simple-websocket-client/pfdhoblngboilpfeibdedpjgfnlcodoo?hl=en
>
> -You can use it to connect to ws://your.gateway/pubsub
> -Then enter {"type":"subscribe", "topic":"c1.citadel.latencies"}
> -Then enter {"type":"data", "topic":"c1.citadel.latencies",
> "data":[{"Elapsed":"3987"},
> {"Max (30s)":"3987ms"}, {"Avg (30s)":"257.00ms"} This will not echo the
> result back to you
> - If you enter {"type":"publish", "topic":"c1.citadel.latencies",
> "data":[{"Elapsed":"3987"},
> {"Max (30s)":"3987ms"}, {"Avg (30s)":"257.00ms"} This will echo the result
> back to you
>
> Note that if you have a fresh connection and you don't send the subscribe
> message, you will not get the result echoed back to you.
>
> There is an easier way. Datatorrent already has operators which serve data
> to the UI and Widgets which display the data in the UI, but that is in the
> Enterprise edition.
>
> Hope this helps, please keep us posted
> Tim
>
> On Wed, Sep 2, 2015 at 8:17 PM, Ganelin, Ilya <Ilya.Ganelin@capitalone.com
> >
> wrote:
>
> > Hi Timothy - the idea was to surface our own metrics to the DataTorrent
> > UI. We have been rolling with this approach since it was recommended to
> us
> > by DT folks. Is there a better way to visualize metrics in the UI?
> >
> >
> >
> > Thank you,
> > Ilya Ganelin
> >
> >
> >
> > -----Original Message-----
> > From: Timothy Farkas [tim@datatorrent.com<ma...@datatorrent.com>]
> > Sent: Wednesday, September 02, 2015 10:24 PM Eastern Standard Time
> > To: dev@apex.incubator.apache.org
> > Subject: Re: Visualize custom metrics in the dashboard
> >
> >
> > Hi Ilya,
> >
> > It looks like you are using Gateway as your websocket server. The
> protocol
> > to send messages to gateway is to use "publish" as you "type" not "data".
> > Also whoever is receiving data from the gateway needs to send a subscribe
> > message like this to gateway first
> {"type":"subscribe","topic":"myTopic"},
> > before they will receive any data from a topic.
> >
> > Tim
> >
> > On Wed, Sep 2, 2015 at 6:18 PM, Ganelin, Ilya <
> Ilya.Ganelin@capitalone.com
> > >
> > wrote:
> >
> > > Chetan, thanks! I fixed that error, and validated the JSON shown below,
> > > any further ideas?
> > >
> > > {"type":"data", "topic":"c1.citadel.latencies",
> > > "data":[{"Elapsed":"3987"}, {"Max (30s)":"3987ms"}, {"Avg
> > > (30s)":"257.00ms"}, {"Msgs/sec":"48"}]}
> > >
> > > On 9/2/15, 12:28 PM, "Chetan Narsude" <ch...@datatorrent.com> wrote:
> > >
> > > >On Wed, Sep 2, 2015 at 10:47 AM, Ganelin, Ilya
> > > ><Il...@capitalone.com>
> > > >wrote:
> > > >
> > > >> Hi all I¹m attempting to surface some output directly to the
> > existing
> > > >> Apex dashboard. Based on the Twitter example here:
> > > >>
> > > >>
> > > >>
> > >
> >
> https://github.com/DataTorrent/Malhar/blob/release-2.0.0/demos/twitter/sr
> > >
> >
> >>c/main/java/com/datatorrent/demos/twitter/TwitterTrendingHashtagsApplicat
> > > >>ion.java
> > > >> [snip]
> > > >>
> > > >>
> > > >> An example of the JSON object I¹m outputting is below:
> > > >> {"type":"data", "topic":"test1", "data":["Elapsed":"1234", "Max
> > > >> (30s)":"5678ms", "Avg (30s)":*"2000.00"ms*, "Msgs/sec":"5000"]}
> > > >>
> > > >>
> > > >The highlighted portion (*"2000.00"ms*) above looks pretty wrong to me
> > :-)
> > > >
> > > >--
> > > >Chetan
> > > >
> > > >
> > > >
> > > >>
> > > >> Any thoughts on what¹s going on here or how I can debug this? The
> job
> > > >>does
> > > >> not ever appear to even launch. Thanks!
> > > >> ________________________________________________________
> > > >>
> > > >> The information contained in this e-mail is confidential and/or
> > > >> proprietary to Capital One and/or its affiliates and may only be
> used
> > > >> solely in performance of work or services for Capital One. The
> > > >>information
> > > >> transmitted herewith is intended only for use by the individual or
> > > >>entity
> > > >> to which it is addressed. If the reader of this message is not the
> > > >>intended
> > > >> recipient, you are hereby notified that any review, retransmission,
> > > >> dissemination, distribution, copying or other use of, or taking of
> any
> > > >> action in reliance upon this information is strictly prohibited. If
> > you
> > > >> have received this communication in error, please contact the sender
> > and
> > > >> delete the material from your computer.
> > > >>
> > >
> > > ________________________________________________________
> > >
> > > The information contained in this e-mail is confidential and/or
> > > proprietary to Capital One and/or its affiliates and may only be used
> > > solely in performance of work or services for Capital One. The
> > information
> > > transmitted herewith is intended only for use by the individual or
> entity
> > > to which it is addressed. If the reader of this message is not the
> > intended
> > > recipient, you are hereby notified that any review, retransmission,
> > > dissemination, distribution, copying or other use of, or taking of any
> > > action in reliance upon this information is strictly prohibited. If you
> > > have received this communication in error, please contact the sender
> and
> > > delete the material from your computer.
> > >
> > >
> > ________________________________________________________
> >
> > The information contained in this e-mail is confidential and/or
> > proprietary to Capital One and/or its affiliates and may only be used
> > solely in performance of work or services for Capital One. The
> information
> > transmitted herewith is intended only for use by the individual or entity
> > to which it is addressed. If the reader of this message is not the
> intended
> > recipient, you are hereby notified that any review, retransmission,
> > dissemination, distribution, copying or other use of, or taking of any
> > action in reliance upon this information is strictly prohibited. If you
> > have received this communication in error, please contact the sender and
> > delete the material from your computer.
> >
> ________________________________________________________
>
> The information contained in this e-mail is confidential and/or
> proprietary to Capital One and/or its affiliates and may only be used
> solely in performance of work or services for Capital One. The information
> transmitted herewith is intended only for use by the individual or entity
> to which it is addressed. If the reader of this message is not the intended
> recipient, you are hereby notified that any review, retransmission,
> dissemination, distribution, copying or other use of, or taking of any
> action in reliance upon this information is strictly prohibited. If you
> have received this communication in error, please contact the sender and
> delete the material from your computer.
>
________________________________________________________
The information contained in this e-mail is confidential and/or proprietary to Capital One and/or its affiliates and may only be used solely in performance of work or services for Capital One. The information transmitted herewith is intended only for use by the individual or entity to which it is addressed. If the reader of this message is not the intended recipient, you are hereby notified that any review, retransmission, dissemination, distribution, copying or other use of, or taking of any action in reliance upon this information is strictly prohibited. If you have received this communication in error, please contact the sender and delete the material from your computer.
Re: Visualize custom metrics in the dashboard
Posted by Timothy Farkas <ti...@datatorrent.com>.
Cool just to check though do you have the enterprise license for 3.0? 2.0
does not include the visualization tools.
Thanks,
Tim
On Wed, Sep 2, 2015 at 11:57 PM, Ganelin, Ilya <Il...@capitalone.com>
wrote:
> Hi Timothy - we do actually have an enterprise license and would
> appreciate more details on how to use those operators. Thanks!
>
>
>
> Thank you,
> Ilya Ganelin
>
>
>
> -----Original Message-----
> From: Timothy Farkas [tim@datatorrent.com<ma...@datatorrent.com>]
> Sent: Thursday, September 03, 2015 12:15 AM Eastern Standard Time
> To: dev@apex.incubator.apache.org
> Subject: Re: Visualize custom metrics in the dashboard
>
>
> Hi Ilya,
>
> I believe the overall approach is correct. Is {"type":"data",
> "topic":"c1.citadel.latencies", "data":[{"Elapsed":"3987"}, {"Max
> (30s)":"3987ms"}, {"Avg (30s)":"257.00ms"}, {"Msgs/sec":"48"}]} the JSON
> that you are seeing output by the PubSubWebsocketOutputOperator? Also is
> the receiver of the message sending a {"type":"subscribe",
> "topic":"c1.citadel.latencies"} message?
>
> A quick way to debug whether your messages can be sent or received properly
> is to use Simple Websocket
>
> https://chrome.google.com/webstore/detail/simple-websocket-client/pfdhoblngboilpfeibdedpjgfnlcodoo?hl=en
>
> -You can use it to connect to ws://your.gateway/pubsub
> -Then enter {"type":"subscribe", "topic":"c1.citadel.latencies"}
> -Then enter {"type":"data", "topic":"c1.citadel.latencies",
> "data":[{"Elapsed":"3987"},
> {"Max (30s)":"3987ms"}, {"Avg (30s)":"257.00ms"} This will not echo the
> result back to you
> - If you enter {"type":"publish", "topic":"c1.citadel.latencies",
> "data":[{"Elapsed":"3987"},
> {"Max (30s)":"3987ms"}, {"Avg (30s)":"257.00ms"} This will echo the result
> back to you
>
> Note that if you have a fresh connection and you don't send the subscribe
> message, you will not get the result echoed back to you.
>
> There is an easier way. Datatorrent already has operators which serve data
> to the UI and Widgets which display the data in the UI, but that is in the
> Enterprise edition.
>
> Hope this helps, please keep us posted
> Tim
>
> On Wed, Sep 2, 2015 at 8:17 PM, Ganelin, Ilya <Ilya.Ganelin@capitalone.com
> >
> wrote:
>
> > Hi Timothy - the idea was to surface our own metrics to the DataTorrent
> > UI. We have been rolling with this approach since it was recommended to
> us
> > by DT folks. Is there a better way to visualize metrics in the UI?
> >
> >
> >
> > Thank you,
> > Ilya Ganelin
> >
> >
> >
> > -----Original Message-----
> > From: Timothy Farkas [tim@datatorrent.com<ma...@datatorrent.com>]
> > Sent: Wednesday, September 02, 2015 10:24 PM Eastern Standard Time
> > To: dev@apex.incubator.apache.org
> > Subject: Re: Visualize custom metrics in the dashboard
> >
> >
> > Hi Ilya,
> >
> > It looks like you are using Gateway as your websocket server. The
> protocol
> > to send messages to gateway is to use "publish" as you "type" not "data".
> > Also whoever is receiving data from the gateway needs to send a subscribe
> > message like this to gateway first
> {"type":"subscribe","topic":"myTopic"},
> > before they will receive any data from a topic.
> >
> > Tim
> >
> > On Wed, Sep 2, 2015 at 6:18 PM, Ganelin, Ilya <
> Ilya.Ganelin@capitalone.com
> > >
> > wrote:
> >
> > > Chetan, thanks! I fixed that error, and validated the JSON shown below,
> > > any further ideas?
> > >
> > > {"type":"data", "topic":"c1.citadel.latencies",
> > > "data":[{"Elapsed":"3987"}, {"Max (30s)":"3987ms"}, {"Avg
> > > (30s)":"257.00ms"}, {"Msgs/sec":"48"}]}
> > >
> > > On 9/2/15, 12:28 PM, "Chetan Narsude" <ch...@datatorrent.com> wrote:
> > >
> > > >On Wed, Sep 2, 2015 at 10:47 AM, Ganelin, Ilya
> > > ><Il...@capitalone.com>
> > > >wrote:
> > > >
> > > >> Hi all I¹m attempting to surface some output directly to the
> > existing
> > > >> Apex dashboard. Based on the Twitter example here:
> > > >>
> > > >>
> > > >>
> > >
> >
> https://github.com/DataTorrent/Malhar/blob/release-2.0.0/demos/twitter/sr
> > >
> >
> >>c/main/java/com/datatorrent/demos/twitter/TwitterTrendingHashtagsApplicat
> > > >>ion.java
> > > >> [snip]
> > > >>
> > > >>
> > > >> An example of the JSON object I¹m outputting is below:
> > > >> {"type":"data", "topic":"test1", "data":["Elapsed":"1234", "Max
> > > >> (30s)":"5678ms", "Avg (30s)":*"2000.00"ms*, "Msgs/sec":"5000"]}
> > > >>
> > > >>
> > > >The highlighted portion (*"2000.00"ms*) above looks pretty wrong to me
> > :-)
> > > >
> > > >--
> > > >Chetan
> > > >
> > > >
> > > >
> > > >>
> > > >> Any thoughts on what¹s going on here or how I can debug this? The
> job
> > > >>does
> > > >> not ever appear to even launch. Thanks!
> > > >> ________________________________________________________
> > > >>
> > > >> The information contained in this e-mail is confidential and/or
> > > >> proprietary to Capital One and/or its affiliates and may only be
> used
> > > >> solely in performance of work or services for Capital One. The
> > > >>information
> > > >> transmitted herewith is intended only for use by the individual or
> > > >>entity
> > > >> to which it is addressed. If the reader of this message is not the
> > > >>intended
> > > >> recipient, you are hereby notified that any review, retransmission,
> > > >> dissemination, distribution, copying or other use of, or taking of
> any
> > > >> action in reliance upon this information is strictly prohibited. If
> > you
> > > >> have received this communication in error, please contact the sender
> > and
> > > >> delete the material from your computer.
> > > >>
> > >
> > > ________________________________________________________
> > >
> > > The information contained in this e-mail is confidential and/or
> > > proprietary to Capital One and/or its affiliates and may only be used
> > > solely in performance of work or services for Capital One. The
> > information
> > > transmitted herewith is intended only for use by the individual or
> entity
> > > to which it is addressed. If the reader of this message is not the
> > intended
> > > recipient, you are hereby notified that any review, retransmission,
> > > dissemination, distribution, copying or other use of, or taking of any
> > > action in reliance upon this information is strictly prohibited. If you
> > > have received this communication in error, please contact the sender
> and
> > > delete the material from your computer.
> > >
> > >
> > ________________________________________________________
> >
> > The information contained in this e-mail is confidential and/or
> > proprietary to Capital One and/or its affiliates and may only be used
> > solely in performance of work or services for Capital One. The
> information
> > transmitted herewith is intended only for use by the individual or entity
> > to which it is addressed. If the reader of this message is not the
> intended
> > recipient, you are hereby notified that any review, retransmission,
> > dissemination, distribution, copying or other use of, or taking of any
> > action in reliance upon this information is strictly prohibited. If you
> > have received this communication in error, please contact the sender and
> > delete the material from your computer.
> >
> ________________________________________________________
>
> The information contained in this e-mail is confidential and/or
> proprietary to Capital One and/or its affiliates and may only be used
> solely in performance of work or services for Capital One. The information
> transmitted herewith is intended only for use by the individual or entity
> to which it is addressed. If the reader of this message is not the intended
> recipient, you are hereby notified that any review, retransmission,
> dissemination, distribution, copying or other use of, or taking of any
> action in reliance upon this information is strictly prohibited. If you
> have received this communication in error, please contact the sender and
> delete the material from your computer.
>
RE: Visualize custom metrics in the dashboard
Posted by "Ganelin, Ilya" <Il...@capitalone.com>.
Hi Timothy - we do actually have an enterprise license and would appreciate more details on how to use those operators. Thanks!
Thank you,
Ilya Ganelin
-----Original Message-----
From: Timothy Farkas [tim@datatorrent.com<ma...@datatorrent.com>]
Sent: Thursday, September 03, 2015 12:15 AM Eastern Standard Time
To: dev@apex.incubator.apache.org
Subject: Re: Visualize custom metrics in the dashboard
Hi Ilya,
I believe the overall approach is correct. Is {"type":"data",
"topic":"c1.citadel.latencies", "data":[{"Elapsed":"3987"}, {"Max
(30s)":"3987ms"}, {"Avg (30s)":"257.00ms"}, {"Msgs/sec":"48"}]} the JSON
that you are seeing output by the PubSubWebsocketOutputOperator? Also is
the receiver of the message sending a {"type":"subscribe",
"topic":"c1.citadel.latencies"} message?
A quick way to debug whether your messages can be sent or received properly
is to use Simple Websocket
https://chrome.google.com/webstore/detail/simple-websocket-client/pfdhoblngboilpfeibdedpjgfnlcodoo?hl=en
-You can use it to connect to ws://your.gateway/pubsub
-Then enter {"type":"subscribe", "topic":"c1.citadel.latencies"}
-Then enter {"type":"data", "topic":"c1.citadel.latencies",
"data":[{"Elapsed":"3987"},
{"Max (30s)":"3987ms"}, {"Avg (30s)":"257.00ms"} This will not echo the
result back to you
- If you enter {"type":"publish", "topic":"c1.citadel.latencies",
"data":[{"Elapsed":"3987"},
{"Max (30s)":"3987ms"}, {"Avg (30s)":"257.00ms"} This will echo the result
back to you
Note that if you have a fresh connection and you don't send the subscribe
message, you will not get the result echoed back to you.
There is an easier way. Datatorrent already has operators which serve data
to the UI and Widgets which display the data in the UI, but that is in the
Enterprise edition.
Hope this helps, please keep us posted
Tim
On Wed, Sep 2, 2015 at 8:17 PM, Ganelin, Ilya <Il...@capitalone.com>
wrote:
> Hi Timothy - the idea was to surface our own metrics to the DataTorrent
> UI. We have been rolling with this approach since it was recommended to us
> by DT folks. Is there a better way to visualize metrics in the UI?
>
>
>
> Thank you,
> Ilya Ganelin
>
>
>
> -----Original Message-----
> From: Timothy Farkas [tim@datatorrent.com<ma...@datatorrent.com>]
> Sent: Wednesday, September 02, 2015 10:24 PM Eastern Standard Time
> To: dev@apex.incubator.apache.org
> Subject: Re: Visualize custom metrics in the dashboard
>
>
> Hi Ilya,
>
> It looks like you are using Gateway as your websocket server. The protocol
> to send messages to gateway is to use "publish" as you "type" not "data".
> Also whoever is receiving data from the gateway needs to send a subscribe
> message like this to gateway first {"type":"subscribe","topic":"myTopic"},
> before they will receive any data from a topic.
>
> Tim
>
> On Wed, Sep 2, 2015 at 6:18 PM, Ganelin, Ilya <Ilya.Ganelin@capitalone.com
> >
> wrote:
>
> > Chetan, thanks! I fixed that error, and validated the JSON shown below,
> > any further ideas?
> >
> > {"type":"data", "topic":"c1.citadel.latencies",
> > "data":[{"Elapsed":"3987"}, {"Max (30s)":"3987ms"}, {"Avg
> > (30s)":"257.00ms"}, {"Msgs/sec":"48"}]}
> >
> > On 9/2/15, 12:28 PM, "Chetan Narsude" <ch...@datatorrent.com> wrote:
> >
> > >On Wed, Sep 2, 2015 at 10:47 AM, Ganelin, Ilya
> > ><Il...@capitalone.com>
> > >wrote:
> > >
> > >> Hi all I¹m attempting to surface some output directly to the
> existing
> > >> Apex dashboard. Based on the Twitter example here:
> > >>
> > >>
> > >>
> >
> https://github.com/DataTorrent/Malhar/blob/release-2.0.0/demos/twitter/sr
> >
> >>c/main/java/com/datatorrent/demos/twitter/TwitterTrendingHashtagsApplicat
> > >>ion.java
> > >> [snip]
> > >>
> > >>
> > >> An example of the JSON object I¹m outputting is below:
> > >> {"type":"data", "topic":"test1", "data":["Elapsed":"1234", "Max
> > >> (30s)":"5678ms", "Avg (30s)":*"2000.00"ms*, "Msgs/sec":"5000"]}
> > >>
> > >>
> > >The highlighted portion (*"2000.00"ms*) above looks pretty wrong to me
> :-)
> > >
> > >--
> > >Chetan
> > >
> > >
> > >
> > >>
> > >> Any thoughts on what¹s going on here or how I can debug this? The job
> > >>does
> > >> not ever appear to even launch. Thanks!
> > >> ________________________________________________________
> > >>
> > >> The information contained in this e-mail is confidential and/or
> > >> proprietary to Capital One and/or its affiliates and may only be used
> > >> solely in performance of work or services for Capital One. The
> > >>information
> > >> transmitted herewith is intended only for use by the individual or
> > >>entity
> > >> to which it is addressed. If the reader of this message is not the
> > >>intended
> > >> recipient, you are hereby notified that any review, retransmission,
> > >> dissemination, distribution, copying or other use of, or taking of any
> > >> action in reliance upon this information is strictly prohibited. If
> you
> > >> have received this communication in error, please contact the sender
> and
> > >> delete the material from your computer.
> > >>
> >
> > ________________________________________________________
> >
> > The information contained in this e-mail is confidential and/or
> > proprietary to Capital One and/or its affiliates and may only be used
> > solely in performance of work or services for Capital One. The
> information
> > transmitted herewith is intended only for use by the individual or entity
> > to which it is addressed. If the reader of this message is not the
> intended
> > recipient, you are hereby notified that any review, retransmission,
> > dissemination, distribution, copying or other use of, or taking of any
> > action in reliance upon this information is strictly prohibited. If you
> > have received this communication in error, please contact the sender and
> > delete the material from your computer.
> >
> >
> ________________________________________________________
>
> The information contained in this e-mail is confidential and/or
> proprietary to Capital One and/or its affiliates and may only be used
> solely in performance of work or services for Capital One. The information
> transmitted herewith is intended only for use by the individual or entity
> to which it is addressed. If the reader of this message is not the intended
> recipient, you are hereby notified that any review, retransmission,
> dissemination, distribution, copying or other use of, or taking of any
> action in reliance upon this information is strictly prohibited. If you
> have received this communication in error, please contact the sender and
> delete the material from your computer.
>
________________________________________________________
The information contained in this e-mail is confidential and/or proprietary to Capital One and/or its affiliates and may only be used solely in performance of work or services for Capital One. The information transmitted herewith is intended only for use by the individual or entity to which it is addressed. If the reader of this message is not the intended recipient, you are hereby notified that any review, retransmission, dissemination, distribution, copying or other use of, or taking of any action in reliance upon this information is strictly prohibited. If you have received this communication in error, please contact the sender and delete the material from your computer.
Re: Visualize custom metrics in the dashboard
Posted by Timothy Farkas <ti...@datatorrent.com>.
Hi Ilya,
I believe the overall approach is correct. Is {"type":"data",
"topic":"c1.citadel.latencies", "data":[{"Elapsed":"3987"}, {"Max
(30s)":"3987ms"}, {"Avg (30s)":"257.00ms"}, {"Msgs/sec":"48"}]} the JSON
that you are seeing output by the PubSubWebsocketOutputOperator? Also is
the receiver of the message sending a {"type":"subscribe",
"topic":"c1.citadel.latencies"} message?
A quick way to debug whether your messages can be sent or received properly
is to use Simple Websocket
https://chrome.google.com/webstore/detail/simple-websocket-client/pfdhoblngboilpfeibdedpjgfnlcodoo?hl=en
-You can use it to connect to ws://your.gateway/pubsub
-Then enter {"type":"subscribe", "topic":"c1.citadel.latencies"}
-Then enter {"type":"data", "topic":"c1.citadel.latencies",
"data":[{"Elapsed":"3987"},
{"Max (30s)":"3987ms"}, {"Avg (30s)":"257.00ms"} This will not echo the
result back to you
- If you enter {"type":"publish", "topic":"c1.citadel.latencies",
"data":[{"Elapsed":"3987"},
{"Max (30s)":"3987ms"}, {"Avg (30s)":"257.00ms"} This will echo the result
back to you
Note that if you have a fresh connection and you don't send the subscribe
message, you will not get the result echoed back to you.
There is an easier way. Datatorrent already has operators which serve data
to the UI and Widgets which display the data in the UI, but that is in the
Enterprise edition.
Hope this helps, please keep us posted
Tim
On Wed, Sep 2, 2015 at 8:17 PM, Ganelin, Ilya <Il...@capitalone.com>
wrote:
> Hi Timothy - the idea was to surface our own metrics to the DataTorrent
> UI. We have been rolling with this approach since it was recommended to us
> by DT folks. Is there a better way to visualize metrics in the UI?
>
>
>
> Thank you,
> Ilya Ganelin
>
>
>
> -----Original Message-----
> From: Timothy Farkas [tim@datatorrent.com<ma...@datatorrent.com>]
> Sent: Wednesday, September 02, 2015 10:24 PM Eastern Standard Time
> To: dev@apex.incubator.apache.org
> Subject: Re: Visualize custom metrics in the dashboard
>
>
> Hi Ilya,
>
> It looks like you are using Gateway as your websocket server. The protocol
> to send messages to gateway is to use "publish" as you "type" not "data".
> Also whoever is receiving data from the gateway needs to send a subscribe
> message like this to gateway first {"type":"subscribe","topic":"myTopic"},
> before they will receive any data from a topic.
>
> Tim
>
> On Wed, Sep 2, 2015 at 6:18 PM, Ganelin, Ilya <Ilya.Ganelin@capitalone.com
> >
> wrote:
>
> > Chetan, thanks! I fixed that error, and validated the JSON shown below,
> > any further ideas?
> >
> > {"type":"data", "topic":"c1.citadel.latencies",
> > "data":[{"Elapsed":"3987"}, {"Max (30s)":"3987ms"}, {"Avg
> > (30s)":"257.00ms"}, {"Msgs/sec":"48"}]}
> >
> > On 9/2/15, 12:28 PM, "Chetan Narsude" <ch...@datatorrent.com> wrote:
> >
> > >On Wed, Sep 2, 2015 at 10:47 AM, Ganelin, Ilya
> > ><Il...@capitalone.com>
> > >wrote:
> > >
> > >> Hi all I¹m attempting to surface some output directly to the
> existing
> > >> Apex dashboard. Based on the Twitter example here:
> > >>
> > >>
> > >>
> >
> https://github.com/DataTorrent/Malhar/blob/release-2.0.0/demos/twitter/sr
> >
> >>c/main/java/com/datatorrent/demos/twitter/TwitterTrendingHashtagsApplicat
> > >>ion.java
> > >> [snip]
> > >>
> > >>
> > >> An example of the JSON object I¹m outputting is below:
> > >> {"type":"data", "topic":"test1", "data":["Elapsed":"1234", "Max
> > >> (30s)":"5678ms", "Avg (30s)":*"2000.00"ms*, "Msgs/sec":"5000"]}
> > >>
> > >>
> > >The highlighted portion (*"2000.00"ms*) above looks pretty wrong to me
> :-)
> > >
> > >--
> > >Chetan
> > >
> > >
> > >
> > >>
> > >> Any thoughts on what¹s going on here or how I can debug this? The job
> > >>does
> > >> not ever appear to even launch. Thanks!
> > >> ________________________________________________________
> > >>
> > >> The information contained in this e-mail is confidential and/or
> > >> proprietary to Capital One and/or its affiliates and may only be used
> > >> solely in performance of work or services for Capital One. The
> > >>information
> > >> transmitted herewith is intended only for use by the individual or
> > >>entity
> > >> to which it is addressed. If the reader of this message is not the
> > >>intended
> > >> recipient, you are hereby notified that any review, retransmission,
> > >> dissemination, distribution, copying or other use of, or taking of any
> > >> action in reliance upon this information is strictly prohibited. If
> you
> > >> have received this communication in error, please contact the sender
> and
> > >> delete the material from your computer.
> > >>
> >
> > ________________________________________________________
> >
> > The information contained in this e-mail is confidential and/or
> > proprietary to Capital One and/or its affiliates and may only be used
> > solely in performance of work or services for Capital One. The
> information
> > transmitted herewith is intended only for use by the individual or entity
> > to which it is addressed. If the reader of this message is not the
> intended
> > recipient, you are hereby notified that any review, retransmission,
> > dissemination, distribution, copying or other use of, or taking of any
> > action in reliance upon this information is strictly prohibited. If you
> > have received this communication in error, please contact the sender and
> > delete the material from your computer.
> >
> >
> ________________________________________________________
>
> The information contained in this e-mail is confidential and/or
> proprietary to Capital One and/or its affiliates and may only be used
> solely in performance of work or services for Capital One. The information
> transmitted herewith is intended only for use by the individual or entity
> to which it is addressed. If the reader of this message is not the intended
> recipient, you are hereby notified that any review, retransmission,
> dissemination, distribution, copying or other use of, or taking of any
> action in reliance upon this information is strictly prohibited. If you
> have received this communication in error, please contact the sender and
> delete the material from your computer.
>
RE: Visualize custom metrics in the dashboard
Posted by "Ganelin, Ilya" <Il...@capitalone.com>.
Hi Timothy - the idea was to surface our own metrics to the DataTorrent UI. We have been rolling with this approach since it was recommended to us by DT folks. Is there a better way to visualize metrics in the UI?
Thank you,
Ilya Ganelin
-----Original Message-----
From: Timothy Farkas [tim@datatorrent.com<ma...@datatorrent.com>]
Sent: Wednesday, September 02, 2015 10:24 PM Eastern Standard Time
To: dev@apex.incubator.apache.org
Subject: Re: Visualize custom metrics in the dashboard
Hi Ilya,
It looks like you are using Gateway as your websocket server. The protocol
to send messages to gateway is to use "publish" as you "type" not "data".
Also whoever is receiving data from the gateway needs to send a subscribe
message like this to gateway first {"type":"subscribe","topic":"myTopic"},
before they will receive any data from a topic.
Tim
On Wed, Sep 2, 2015 at 6:18 PM, Ganelin, Ilya <Il...@capitalone.com>
wrote:
> Chetan, thanks! I fixed that error, and validated the JSON shown below,
> any further ideas?
>
> {"type":"data", "topic":"c1.citadel.latencies",
> "data":[{"Elapsed":"3987"}, {"Max (30s)":"3987ms"}, {"Avg
> (30s)":"257.00ms"}, {"Msgs/sec":"48"}]}
>
> On 9/2/15, 12:28 PM, "Chetan Narsude" <ch...@datatorrent.com> wrote:
>
> >On Wed, Sep 2, 2015 at 10:47 AM, Ganelin, Ilya
> ><Il...@capitalone.com>
> >wrote:
> >
> >> Hi all I¹m attempting to surface some output directly to the existing
> >> Apex dashboard. Based on the Twitter example here:
> >>
> >>
> >>
> https://github.com/DataTorrent/Malhar/blob/release-2.0.0/demos/twitter/sr
> >>c/main/java/com/datatorrent/demos/twitter/TwitterTrendingHashtagsApplicat
> >>ion.java
> >> [snip]
> >>
> >>
> >> An example of the JSON object I¹m outputting is below:
> >> {"type":"data", "topic":"test1", "data":["Elapsed":"1234", "Max
> >> (30s)":"5678ms", "Avg (30s)":*"2000.00"ms*, "Msgs/sec":"5000"]}
> >>
> >>
> >The highlighted portion (*"2000.00"ms*) above looks pretty wrong to me :-)
> >
> >--
> >Chetan
> >
> >
> >
> >>
> >> Any thoughts on what¹s going on here or how I can debug this? The job
> >>does
> >> not ever appear to even launch. Thanks!
> >> ________________________________________________________
> >>
> >> The information contained in this e-mail is confidential and/or
> >> proprietary to Capital One and/or its affiliates and may only be used
> >> solely in performance of work or services for Capital One. The
> >>information
> >> transmitted herewith is intended only for use by the individual or
> >>entity
> >> to which it is addressed. If the reader of this message is not the
> >>intended
> >> recipient, you are hereby notified that any review, retransmission,
> >> dissemination, distribution, copying or other use of, or taking of any
> >> action in reliance upon this information is strictly prohibited. If you
> >> have received this communication in error, please contact the sender and
> >> delete the material from your computer.
> >>
>
> ________________________________________________________
>
> The information contained in this e-mail is confidential and/or
> proprietary to Capital One and/or its affiliates and may only be used
> solely in performance of work or services for Capital One. The information
> transmitted herewith is intended only for use by the individual or entity
> to which it is addressed. If the reader of this message is not the intended
> recipient, you are hereby notified that any review, retransmission,
> dissemination, distribution, copying or other use of, or taking of any
> action in reliance upon this information is strictly prohibited. If you
> have received this communication in error, please contact the sender and
> delete the material from your computer.
>
>
________________________________________________________
The information contained in this e-mail is confidential and/or proprietary to Capital One and/or its affiliates and may only be used solely in performance of work or services for Capital One. The information transmitted herewith is intended only for use by the individual or entity to which it is addressed. If the reader of this message is not the intended recipient, you are hereby notified that any review, retransmission, dissemination, distribution, copying or other use of, or taking of any action in reliance upon this information is strictly prohibited. If you have received this communication in error, please contact the sender and delete the material from your computer.
Re: Visualize custom metrics in the dashboard
Posted by Timothy Farkas <ti...@datatorrent.com>.
Hi Ilya,
It looks like you are using Gateway as your websocket server. The protocol
to send messages to gateway is to use "publish" as you "type" not "data".
Also whoever is receiving data from the gateway needs to send a subscribe
message like this to gateway first {"type":"subscribe","topic":"myTopic"},
before they will receive any data from a topic.
Tim
On Wed, Sep 2, 2015 at 6:18 PM, Ganelin, Ilya <Il...@capitalone.com>
wrote:
> Chetan, thanks! I fixed that error, and validated the JSON shown below,
> any further ideas?
>
> {"type":"data", "topic":"c1.citadel.latencies",
> "data":[{"Elapsed":"3987"}, {"Max (30s)":"3987ms"}, {"Avg
> (30s)":"257.00ms"}, {"Msgs/sec":"48"}]}
>
> On 9/2/15, 12:28 PM, "Chetan Narsude" <ch...@datatorrent.com> wrote:
>
> >On Wed, Sep 2, 2015 at 10:47 AM, Ganelin, Ilya
> ><Il...@capitalone.com>
> >wrote:
> >
> >> Hi all I¹m attempting to surface some output directly to the existing
> >> Apex dashboard. Based on the Twitter example here:
> >>
> >>
> >>
> https://github.com/DataTorrent/Malhar/blob/release-2.0.0/demos/twitter/sr
> >>c/main/java/com/datatorrent/demos/twitter/TwitterTrendingHashtagsApplicat
> >>ion.java
> >> [snip]
> >>
> >>
> >> An example of the JSON object I¹m outputting is below:
> >> {"type":"data", "topic":"test1", "data":["Elapsed":"1234", "Max
> >> (30s)":"5678ms", "Avg (30s)":*"2000.00"ms*, "Msgs/sec":"5000"]}
> >>
> >>
> >The highlighted portion (*"2000.00"ms*) above looks pretty wrong to me :-)
> >
> >--
> >Chetan
> >
> >
> >
> >>
> >> Any thoughts on what¹s going on here or how I can debug this? The job
> >>does
> >> not ever appear to even launch. Thanks!
> >> ________________________________________________________
> >>
> >> The information contained in this e-mail is confidential and/or
> >> proprietary to Capital One and/or its affiliates and may only be used
> >> solely in performance of work or services for Capital One. The
> >>information
> >> transmitted herewith is intended only for use by the individual or
> >>entity
> >> to which it is addressed. If the reader of this message is not the
> >>intended
> >> recipient, you are hereby notified that any review, retransmission,
> >> dissemination, distribution, copying or other use of, or taking of any
> >> action in reliance upon this information is strictly prohibited. If you
> >> have received this communication in error, please contact the sender and
> >> delete the material from your computer.
> >>
>
> ________________________________________________________
>
> The information contained in this e-mail is confidential and/or
> proprietary to Capital One and/or its affiliates and may only be used
> solely in performance of work or services for Capital One. The information
> transmitted herewith is intended only for use by the individual or entity
> to which it is addressed. If the reader of this message is not the intended
> recipient, you are hereby notified that any review, retransmission,
> dissemination, distribution, copying or other use of, or taking of any
> action in reliance upon this information is strictly prohibited. If you
> have received this communication in error, please contact the sender and
> delete the material from your computer.
>
>
Re: Visualize custom metrics in the dashboard
Posted by "Ganelin, Ilya" <Il...@capitalone.com>.
Chetan, thanks! I fixed that error, and validated the JSON shown below,
any further ideas?
{"type":"data", "topic":"c1.citadel.latencies",
"data":[{"Elapsed":"3987"}, {"Max (30s)":"3987ms"}, {"Avg
(30s)":"257.00ms"}, {"Msgs/sec":"48"}]}
On 9/2/15, 12:28 PM, "Chetan Narsude" <ch...@datatorrent.com> wrote:
>On Wed, Sep 2, 2015 at 10:47 AM, Ganelin, Ilya
><Il...@capitalone.com>
>wrote:
>
>> Hi all I¹m attempting to surface some output directly to the existing
>> Apex dashboard. Based on the Twitter example here:
>>
>>
>>https://github.com/DataTorrent/Malhar/blob/release-2.0.0/demos/twitter/sr
>>c/main/java/com/datatorrent/demos/twitter/TwitterTrendingHashtagsApplicat
>>ion.java
>> [snip]
>>
>>
>> An example of the JSON object I¹m outputting is below:
>> {"type":"data", "topic":"test1", "data":["Elapsed":"1234", "Max
>> (30s)":"5678ms", "Avg (30s)":*"2000.00"ms*, "Msgs/sec":"5000"]}
>>
>>
>The highlighted portion (*"2000.00"ms*) above looks pretty wrong to me :-)
>
>--
>Chetan
>
>
>
>>
>> Any thoughts on what¹s going on here or how I can debug this? The job
>>does
>> not ever appear to even launch. Thanks!
>> ________________________________________________________
>>
>> The information contained in this e-mail is confidential and/or
>> proprietary to Capital One and/or its affiliates and may only be used
>> solely in performance of work or services for Capital One. The
>>information
>> transmitted herewith is intended only for use by the individual or
>>entity
>> to which it is addressed. If the reader of this message is not the
>>intended
>> recipient, you are hereby notified that any review, retransmission,
>> dissemination, distribution, copying or other use of, or taking of any
>> action in reliance upon this information is strictly prohibited. If you
>> have received this communication in error, please contact the sender and
>> delete the material from your computer.
>>
________________________________________________________
The information contained in this e-mail is confidential and/or proprietary to Capital One and/or its affiliates and may only be used solely in performance of work or services for Capital One. The information transmitted herewith is intended only for use by the individual or entity to which it is addressed. If the reader of this message is not the intended recipient, you are hereby notified that any review, retransmission, dissemination, distribution, copying or other use of, or taking of any action in reliance upon this information is strictly prohibited. If you have received this communication in error, please contact the sender and delete the material from your computer.
Re: Visualize custom metrics in the dashboard
Posted by Chetan Narsude <ch...@datatorrent.com>.
On Wed, Sep 2, 2015 at 10:47 AM, Ganelin, Ilya <Il...@capitalone.com>
wrote:
> Hi all – I’m attempting to surface some output directly to the existing
> Apex dashboard. Based on the Twitter example here:
>
> https://github.com/DataTorrent/Malhar/blob/release-2.0.0/demos/twitter/src/main/java/com/datatorrent/demos/twitter/TwitterTrendingHashtagsApplication.java
> [snip]
>
>
> An example of the JSON object I’m outputting is below:
> {"type":"data", "topic":"test1", "data":["Elapsed":"1234", "Max
> (30s)":"5678ms", "Avg (30s)":*"2000.00"ms*, "Msgs/sec":"5000"]}
>
>
The highlighted portion (*"2000.00"ms*) above looks pretty wrong to me :-)
--
Chetan
>
> Any thoughts on what’s going on here or how I can debug this? The job does
> not ever appear to even launch. Thanks!
> ________________________________________________________
>
> The information contained in this e-mail is confidential and/or
> proprietary to Capital One and/or its affiliates and may only be used
> solely in performance of work or services for Capital One. The information
> transmitted herewith is intended only for use by the individual or entity
> to which it is addressed. If the reader of this message is not the intended
> recipient, you are hereby notified that any review, retransmission,
> dissemination, distribution, copying or other use of, or taking of any
> action in reliance upon this information is strictly prohibited. If you
> have received this communication in error, please contact the sender and
> delete the material from your computer.
>