You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@storm.apache.org by Yuheng Du <yu...@clemson.edu> on 2014/11/05 20:24:32 UTC
question about spout emitted tuples
Hi guys,
I am using rabbitmq-spout offered by https://github.com/ppat/storm-rabbitmq
to read data from rabbitmq. Here is what I see in the console when I run
the topology locally:
101703 [Thread-16-storm-obser-spout] INFO backtype.storm.daemon.executor -
Acking message 33
113099 [Thread-16-storm-obser-spout] INFO backtype.storm.daemon.task -
Emitting: storm-obser-spout default
[{"readings":[0.0,0.0,0.0,88.007,0.0,0.0,0.0,17.83,64.0,0.0,0.0,0.0,0.0,11.071,0.2,1.69,0.0,0.0,-0.23,88.007,0.0,0.0,0.0,0.0,0.0,0.0],"observationId":"f45ef06f-88dc-4d50-b28e-24937dda5240","deploymentId":"
http://www.xxxx.org/resource/deployment#aiken_8","deviceId":"0","observationDateTime":"11-05-2014
19:05:15 UTC"},
io.latent.storm.rabbitmq.RabbitMQMessageScheme$Envelope@2a8ec790,
io.latent.storm.rabbitmq.RabbitMQMessageScheme$Properties@3cfd4c4]
113099 [Thread-16-storm-obser-spout] INFO backtype.storm.daemon.task -
Emitting: storm-obser-spout __ack_init [-3424274568001459772 0 2]
Can anybody tell me where does these two lines come from:
"io.latent.storm.rabbitmq.RabbitMQMessageScheme$Envelope@2a8ec790,
io.latent.storm.rabbitmq.RabbitMQMessageScheme$Properties@3cfd4c4" ?
How can I parse each field in my JSON format messages like "readings" and
"observationDateTime"?
I attached my topology source code.
Thanks.
Re: question about spout emitted tuples
Posted by Yuheng Du <yu...@gmail.com>.
Got it! Thanks!
On Thu, Nov 6, 2014 at 4:12 PM, Stephen Armstrong <
stephen.armstrong@linqia.com> wrote:
> Hi Yuheng,
>
> Don't worry about Storm, just use Gson in standard Java.
>
> String json = ...
>
> // If you don't know the structure of the data, you can do this
> JsonArray array = new Gson().fromJson(json, JsonArray.class);
> JsonObject object = new Gson().fromJson(json, JsonObject.class);
>
> // What is normally done:
> public static class MyObj {
> int a;
> String b;
> }
> MyObj deserializedObject = new Gson().fromJson(json, MyObj.class);
>
> If you want to send the MyObj instance from bolt to bolt, then just make
> sure it implements Serializable, and you can emit it as part of a tuple.
>
> Steve
>
> On Thu, Nov 6, 2014 at 11:11 AM, Yuheng Du <yu...@gmail.com>
> wrote:
>
>> Hi Stephen,
>>
>> Can you give a tiny example to explain how can I use gson to do that? I
>> am quite new to storm serialization/deserialization.
>>
>> I need to deserialize a json format string from the spout into jsonobject
>> or jsonarray.
>>
>> Thanks.
>>
>>
>>
>> On Thu, Nov 6, 2014 at 1:25 PM, Stephen Armstrong <
>> stephen.armstrong@linqia.com> wrote:
>>
>>> I'm currently doing json serialize/deserialize using gson-2.3 on
>>> storm-0.9.2-incubating and it seems to be working fine.
>>>
>>> On Wed, Nov 5, 2014 at 5:18 PM, Yuheng Du <yu...@clemson.edu> wrote:
>>>
>>>> Does anyone implemented a JSON serialization scheme for Strom version
>>>> >= 0.9.0?
>>>>
>>>> On Wed, Nov 5, 2014 at 2:24 PM, Yuheng Du <yu...@clemson.edu> wrote:
>>>>
>>>>> Hi guys,
>>>>>
>>>>> I am using rabbitmq-spout offered by
>>>>> https://github.com/ppat/storm-rabbitmq to read data from rabbitmq.
>>>>> Here is what I see in the console when I run the topology locally:
>>>>>
>>>>> 101703 [Thread-16-storm-obser-spout] INFO
>>>>> backtype.storm.daemon.executor - Acking message 33
>>>>> 113099 [Thread-16-storm-obser-spout] INFO backtype.storm.daemon.task
>>>>> - Emitting: storm-obser-spout default
>>>>> [{"readings":[0.0,0.0,0.0,88.007,0.0,0.0,0.0,17.83,64.0,0.0,0.0,0.0,0.0,11.071,0.2,1.69,0.0,0.0,-0.23,88.007,0.0,0.0,0.0,0.0,0.0,0.0],"observationId":"f45ef06f-88dc-4d50-b28e-24937dda5240","deploymentId":"
>>>>> http://www.xxxx.org/resource/deployment#aiken_8","deviceId":"0","observationDateTime":"11-05-2014
>>>>> 19:05:15 UTC"},
>>>>> io.latent.storm.rabbitmq.RabbitMQMessageScheme$Envelope@2a8ec790,
>>>>> io.latent.storm.rabbitmq.RabbitMQMessageScheme$Properties@3cfd4c4]
>>>>> 113099 [Thread-16-storm-obser-spout] INFO backtype.storm.daemon.task
>>>>> - Emitting: storm-obser-spout __ack_init [-3424274568001459772 0 2]
>>>>>
>>>>> Can anybody tell me where does these two lines come from:
>>>>> "io.latent.storm.rabbitmq.RabbitMQMessageScheme$Envelope@2a8ec790,
>>>>> io.latent.storm.rabbitmq.RabbitMQMessageScheme$Properties@3cfd4c4" ?
>>>>>
>>>>> How can I parse each field in my JSON format messages like "readings"
>>>>> and "observationDateTime"?
>>>>>
>>>>> I attached my topology source code.
>>>>>
>>>>> Thanks.
>>>>>
>>>>>
>>>>
>>>
>>
>
Re: question about spout emitted tuples
Posted by Stephen Armstrong <st...@linqia.com>.
Hi Yuheng,
Don't worry about Storm, just use Gson in standard Java.
String json = ...
// If you don't know the structure of the data, you can do this
JsonArray array = new Gson().fromJson(json, JsonArray.class);
JsonObject object = new Gson().fromJson(json, JsonObject.class);
// What is normally done:
public static class MyObj {
int a;
String b;
}
MyObj deserializedObject = new Gson().fromJson(json, MyObj.class);
If you want to send the MyObj instance from bolt to bolt, then just make
sure it implements Serializable, and you can emit it as part of a tuple.
Steve
On Thu, Nov 6, 2014 at 11:11 AM, Yuheng Du <yu...@gmail.com> wrote:
> Hi Stephen,
>
> Can you give a tiny example to explain how can I use gson to do that? I am
> quite new to storm serialization/deserialization.
>
> I need to deserialize a json format string from the spout into jsonobject
> or jsonarray.
>
> Thanks.
>
>
>
> On Thu, Nov 6, 2014 at 1:25 PM, Stephen Armstrong <
> stephen.armstrong@linqia.com> wrote:
>
>> I'm currently doing json serialize/deserialize using gson-2.3 on
>> storm-0.9.2-incubating and it seems to be working fine.
>>
>> On Wed, Nov 5, 2014 at 5:18 PM, Yuheng Du <yu...@clemson.edu> wrote:
>>
>>> Does anyone implemented a JSON serialization scheme for Strom version >=
>>> 0.9.0?
>>>
>>> On Wed, Nov 5, 2014 at 2:24 PM, Yuheng Du <yu...@clemson.edu> wrote:
>>>
>>>> Hi guys,
>>>>
>>>> I am using rabbitmq-spout offered by
>>>> https://github.com/ppat/storm-rabbitmq to read data from rabbitmq.
>>>> Here is what I see in the console when I run the topology locally:
>>>>
>>>> 101703 [Thread-16-storm-obser-spout] INFO
>>>> backtype.storm.daemon.executor - Acking message 33
>>>> 113099 [Thread-16-storm-obser-spout] INFO backtype.storm.daemon.task -
>>>> Emitting: storm-obser-spout default
>>>> [{"readings":[0.0,0.0,0.0,88.007,0.0,0.0,0.0,17.83,64.0,0.0,0.0,0.0,0.0,11.071,0.2,1.69,0.0,0.0,-0.23,88.007,0.0,0.0,0.0,0.0,0.0,0.0],"observationId":"f45ef06f-88dc-4d50-b28e-24937dda5240","deploymentId":"
>>>> http://www.xxxx.org/resource/deployment#aiken_8","deviceId":"0","observationDateTime":"11-05-2014
>>>> 19:05:15 UTC"},
>>>> io.latent.storm.rabbitmq.RabbitMQMessageScheme$Envelope@2a8ec790,
>>>> io.latent.storm.rabbitmq.RabbitMQMessageScheme$Properties@3cfd4c4]
>>>> 113099 [Thread-16-storm-obser-spout] INFO backtype.storm.daemon.task -
>>>> Emitting: storm-obser-spout __ack_init [-3424274568001459772 0 2]
>>>>
>>>> Can anybody tell me where does these two lines come from:
>>>> "io.latent.storm.rabbitmq.RabbitMQMessageScheme$Envelope@2a8ec790,
>>>> io.latent.storm.rabbitmq.RabbitMQMessageScheme$Properties@3cfd4c4" ?
>>>>
>>>> How can I parse each field in my JSON format messages like "readings"
>>>> and "observationDateTime"?
>>>>
>>>> I attached my topology source code.
>>>>
>>>> Thanks.
>>>>
>>>>
>>>
>>
>
Re: question about spout emitted tuples
Posted by Yuheng Du <yu...@gmail.com>.
Hi Stephen,
Can you give a tiny example to explain how can I use gson to do that? I am
quite new to storm serialization/deserialization.
I need to deserialize a json format string from the spout into jsonobject
or jsonarray.
Thanks.
On Thu, Nov 6, 2014 at 1:25 PM, Stephen Armstrong <
stephen.armstrong@linqia.com> wrote:
> I'm currently doing json serialize/deserialize using gson-2.3 on
> storm-0.9.2-incubating and it seems to be working fine.
>
> On Wed, Nov 5, 2014 at 5:18 PM, Yuheng Du <yu...@clemson.edu> wrote:
>
>> Does anyone implemented a JSON serialization scheme for Strom version >=
>> 0.9.0?
>>
>> On Wed, Nov 5, 2014 at 2:24 PM, Yuheng Du <yu...@clemson.edu> wrote:
>>
>>> Hi guys,
>>>
>>> I am using rabbitmq-spout offered by
>>> https://github.com/ppat/storm-rabbitmq to read data from rabbitmq. Here
>>> is what I see in the console when I run the topology locally:
>>>
>>> 101703 [Thread-16-storm-obser-spout] INFO
>>> backtype.storm.daemon.executor - Acking message 33
>>> 113099 [Thread-16-storm-obser-spout] INFO backtype.storm.daemon.task -
>>> Emitting: storm-obser-spout default
>>> [{"readings":[0.0,0.0,0.0,88.007,0.0,0.0,0.0,17.83,64.0,0.0,0.0,0.0,0.0,11.071,0.2,1.69,0.0,0.0,-0.23,88.007,0.0,0.0,0.0,0.0,0.0,0.0],"observationId":"f45ef06f-88dc-4d50-b28e-24937dda5240","deploymentId":"
>>> http://www.xxxx.org/resource/deployment#aiken_8","deviceId":"0","observationDateTime":"11-05-2014
>>> 19:05:15 UTC"},
>>> io.latent.storm.rabbitmq.RabbitMQMessageScheme$Envelope@2a8ec790,
>>> io.latent.storm.rabbitmq.RabbitMQMessageScheme$Properties@3cfd4c4]
>>> 113099 [Thread-16-storm-obser-spout] INFO backtype.storm.daemon.task -
>>> Emitting: storm-obser-spout __ack_init [-3424274568001459772 0 2]
>>>
>>> Can anybody tell me where does these two lines come from:
>>> "io.latent.storm.rabbitmq.RabbitMQMessageScheme$Envelope@2a8ec790,
>>> io.latent.storm.rabbitmq.RabbitMQMessageScheme$Properties@3cfd4c4" ?
>>>
>>> How can I parse each field in my JSON format messages like "readings"
>>> and "observationDateTime"?
>>>
>>> I attached my topology source code.
>>>
>>> Thanks.
>>>
>>>
>>
>
Re: question about spout emitted tuples
Posted by Stephen Armstrong <st...@linqia.com>.
I'm currently doing json serialize/deserialize using gson-2.3 on
storm-0.9.2-incubating and it seems to be working fine.
On Wed, Nov 5, 2014 at 5:18 PM, Yuheng Du <yu...@clemson.edu> wrote:
> Does anyone implemented a JSON serialization scheme for Strom version >=
> 0.9.0?
>
> On Wed, Nov 5, 2014 at 2:24 PM, Yuheng Du <yu...@clemson.edu> wrote:
>
>> Hi guys,
>>
>> I am using rabbitmq-spout offered by
>> https://github.com/ppat/storm-rabbitmq to read data from rabbitmq. Here
>> is what I see in the console when I run the topology locally:
>>
>> 101703 [Thread-16-storm-obser-spout] INFO backtype.storm.daemon.executor
>> - Acking message 33
>> 113099 [Thread-16-storm-obser-spout] INFO backtype.storm.daemon.task -
>> Emitting: storm-obser-spout default
>> [{"readings":[0.0,0.0,0.0,88.007,0.0,0.0,0.0,17.83,64.0,0.0,0.0,0.0,0.0,11.071,0.2,1.69,0.0,0.0,-0.23,88.007,0.0,0.0,0.0,0.0,0.0,0.0],"observationId":"f45ef06f-88dc-4d50-b28e-24937dda5240","deploymentId":"
>> http://www.xxxx.org/resource/deployment#aiken_8","deviceId":"0","observationDateTime":"11-05-2014
>> 19:05:15 UTC"},
>> io.latent.storm.rabbitmq.RabbitMQMessageScheme$Envelope@2a8ec790,
>> io.latent.storm.rabbitmq.RabbitMQMessageScheme$Properties@3cfd4c4]
>> 113099 [Thread-16-storm-obser-spout] INFO backtype.storm.daemon.task -
>> Emitting: storm-obser-spout __ack_init [-3424274568001459772 0 2]
>>
>> Can anybody tell me where does these two lines come from:
>> "io.latent.storm.rabbitmq.RabbitMQMessageScheme$Envelope@2a8ec790,
>> io.latent.storm.rabbitmq.RabbitMQMessageScheme$Properties@3cfd4c4" ?
>>
>> How can I parse each field in my JSON format messages like "readings" and
>> "observationDateTime"?
>>
>> I attached my topology source code.
>>
>> Thanks.
>>
>>
>
Re: question about spout emitted tuples
Posted by Yuheng Du <yu...@clemson.edu>.
Does anyone implemented a JSON serialization scheme for Strom version >=
0.9.0?
On Wed, Nov 5, 2014 at 2:24 PM, Yuheng Du <yu...@clemson.edu> wrote:
> Hi guys,
>
> I am using rabbitmq-spout offered by
> https://github.com/ppat/storm-rabbitmq to read data from rabbitmq. Here
> is what I see in the console when I run the topology locally:
>
> 101703 [Thread-16-storm-obser-spout] INFO backtype.storm.daemon.executor
> - Acking message 33
> 113099 [Thread-16-storm-obser-spout] INFO backtype.storm.daemon.task -
> Emitting: storm-obser-spout default
> [{"readings":[0.0,0.0,0.0,88.007,0.0,0.0,0.0,17.83,64.0,0.0,0.0,0.0,0.0,11.071,0.2,1.69,0.0,0.0,-0.23,88.007,0.0,0.0,0.0,0.0,0.0,0.0],"observationId":"f45ef06f-88dc-4d50-b28e-24937dda5240","deploymentId":"
> http://www.xxxx.org/resource/deployment#aiken_8","deviceId":"0","observationDateTime":"11-05-2014
> 19:05:15 UTC"},
> io.latent.storm.rabbitmq.RabbitMQMessageScheme$Envelope@2a8ec790,
> io.latent.storm.rabbitmq.RabbitMQMessageScheme$Properties@3cfd4c4]
> 113099 [Thread-16-storm-obser-spout] INFO backtype.storm.daemon.task -
> Emitting: storm-obser-spout __ack_init [-3424274568001459772 0 2]
>
> Can anybody tell me where does these two lines come from:
> "io.latent.storm.rabbitmq.RabbitMQMessageScheme$Envelope@2a8ec790,
> io.latent.storm.rabbitmq.RabbitMQMessageScheme$Properties@3cfd4c4" ?
>
> How can I parse each field in my JSON format messages like "readings" and
> "observationDateTime"?
>
> I attached my topology source code.
>
> Thanks.
>
>