You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@storm.apache.org by Software Dev <st...@gmail.com> on 2014/03/29 20:06:06 UTC
Working with JSON
Say we are receiving tuples of JSON from a spout. Should we just keep
passing around the JSON string and deserialize it in each bolt or Is
it best to break apart the JSON object into a bunch of fields that can
be passed around.
I'm thinking in terms of performance the latter may be "better"
although it will slightly make the rest of the topology more complex.
Also, what is a good JSON library to work with?
Thanks
Re: Working with JSON
Posted by Software Dev <st...@gmail.com>.
Michael our input is actually in the form of Avro (Avro Camus encoded
actually). We were thinking about having a bolt that deserializes the
camus encoded record (GenericData.Record) to JSON and then pass that
around. From what I gather from this thread, passing around JSON and
deserializing is probably not the best. Should we just be passing
around the GenericData.Record? If so, is there any special
serialization we need to do? Is this the best approach?
On Mon, Mar 31, 2014 at 11:33 AM, Michael G. Noll
<mi...@michael-noll.com> wrote:
> On 03/31/2014 07:31 PM, Michael Rose wrote:
>> It's much more efficient to deserialize it once then pass around POJOs.
>> JSON serialization is slow compared to Kryo. Our topologies tend to take
>> in JSON, then emit JSON to external systems at later phases, but all
>> intermediate stages are POJOs.
>
> +1 to what Michael (Rose) said. Don't forget to register the required
> Kryo serializers for your pojos though.
>
> Also, consider the use of Avro. You can turn Avro into JSON once you
> actually do need JSON. Tools like Twitter Bijection and Twitter Chill
> make this as simple as it can get (Avro also ships with JSON support,
> but IMHO Bijection/Chill are preferable).
>
> Michael
>
>
>
Re: Working with JSON
Posted by "Michael G. Noll" <mi...@michael-noll.com>.
On 03/31/2014 07:31 PM, Michael Rose wrote:
> It's much more efficient to deserialize it once then pass around POJOs.
> JSON serialization is slow compared to Kryo. Our topologies tend to take
> in JSON, then emit JSON to external systems at later phases, but all
> intermediate stages are POJOs.
+1 to what Michael (Rose) said. Don't forget to register the required
Kryo serializers for your pojos though.
Also, consider the use of Avro. You can turn Avro into JSON once you
actually do need JSON. Tools like Twitter Bijection and Twitter Chill
make this as simple as it can get (Avro also ships with JSON support,
but IMHO Bijection/Chill are preferable).
Michael
Re: Working with JSON
Posted by Michael Rose <mi...@fullcontact.com>.
It's much more efficient to deserialize it once then pass around POJOs.
JSON serialization is slow compared to Kryo. Our topologies tend to take in
JSON, then emit JSON to external systems at later phases, but all
intermediate stages are POJOs.
Michael Rose (@Xorlev <https://twitter.com/xorlev>)
Senior Platform Engineer, FullContact <http://www.fullcontact.com/>
michael@fullcontact.com
On Mon, Mar 31, 2014 at 11:18 AM, Cody A. Ray <co...@gmail.com> wrote:
> We have one spout which always emits JSON of the same form (this is a
> metric represented as a JSON object with timestamp, name, and value
> fields). Note this is the OpaqueTridentKafkaSpout from storm-kafka-0.8-plus
> which just emits a "bytes" array field, so we first convert it to a string.
>
> Stream stream = topology.newStream("spout1", spout)
> .each(new Fields("bytes"), new *BinaryToString*(), new Fields(
> "string"))
> .each(new Fields("string"), new *MetricJsonParser*(), new Fields(
> "timestamp", "metric", "value"))
> ...
>
> Each of these two functions are pretty simple:
> https://gist.github.com/codyaray/9897217. I'm using Codehaus' Jettison<http://jettison.codehaus.org/>to parse the JSON.
>
> -Cody
>
>
> On Sat, Mar 29, 2014 at 5:30 PM, Tyson Norris <tn...@adobe.com> wrote:
>
>> We have a similar setup, and based on routing needs we plan to pass the
>> original JSON, plus some extra fields that will simplify routing via
>> fieldsGrouping.
>> e.g.
>> spout -> content (JSONObject)
>> bolt1 -> receives all spout tuples; execute() uses JSONObject ->
>> emit(JSONObject, String, String) (String values are parsed out of
>> JSONObject)
>> bolt2 -> receives bolt1 tuples based on fieldsGrouping; execute uses
>> JSONObject, String, String to perform some operation -> emit(JSONObject,
>> String) (String values are based on some logic)
>>
>> So while you could go either extreme (continually pass JSONObject value
>> as tuple, or parse the JSONObject and pass only decomposed values as
>> tuple), you can also do both, which would allow you to use fieldGrouping,
>> in case that is important (it is important for our case).
>>
>> Tyson
>>
>>
>> On Mar 29, 2014, at 1:44 PM, Software Dev <st...@gmail.com>
>> wrote:
>>
>> > We actually have a spout that emits just 1 JSON string per tuple.
>> > Wondering what should be down downstream after we have the JSON string
>> >
>> > On Sat, Mar 29, 2014 at 12:20 PM, Andrew Neilson <ar...@gmail.com>
>> wrote:
>> >> my team's project has successfully used Jackson
>> >> (https://github.com/FasterXML/jackson) to deserialize a spout of JSON
>> arrays
>> >> into tuples, and I can recommend it. Though I'll warn you that it
>> takes a
>> >> little bit of work beyond the most basic usage (i.e.
>> mapper.readValue(json,
>> >> List.class)) to avoid dealing with type ambiguity.
>> >>
>> >>
>> >> On Sat, Mar 29, 2014 at 12:06 PM, Software Dev <
>> static.void.dev@gmail.com>
>> >> wrote:
>> >>>
>> >>> Say we are receiving tuples of JSON from a spout. Should we just keep
>> >>> passing around the JSON string and deserialize it in each bolt or Is
>> >>> it best to break apart the JSON object into a bunch of fields that can
>> >>> be passed around.
>> >>>
>> >>> I'm thinking in terms of performance the latter may be "better"
>> >>> although it will slightly make the rest of the topology more complex.
>> >>>
>> >>> Also, what is a good JSON library to work with?
>> >>>
>> >>> Thanks
>> >>
>> >>
>>
>>
>
>
> --
> Cody A. Ray, LEED AP
> cody.a.ray@gmail.com
> 215.501.7891
>
Re: Working with JSON
Posted by "Cody A. Ray" <co...@gmail.com>.
We have one spout which always emits JSON of the same form (this is a
metric represented as a JSON object with timestamp, name, and value
fields). Note this is the OpaqueTridentKafkaSpout from storm-kafka-0.8-plus
which just emits a "bytes" array field, so we first convert it to a string.
Stream stream = topology.newStream("spout1", spout)
.each(new Fields("bytes"), new *BinaryToString*(), new Fields(
"string"))
.each(new Fields("string"), new *MetricJsonParser*(), new Fields(
"timestamp", "metric", "value"))
...
Each of these two functions are pretty simple:
https://gist.github.com/codyaray/9897217. I'm using Codehaus'
Jettison<http://jettison.codehaus.org/>to parse the JSON.
-Cody
On Sat, Mar 29, 2014 at 5:30 PM, Tyson Norris <tn...@adobe.com> wrote:
> We have a similar setup, and based on routing needs we plan to pass the
> original JSON, plus some extra fields that will simplify routing via
> fieldsGrouping.
> e.g.
> spout -> content (JSONObject)
> bolt1 -> receives all spout tuples; execute() uses JSONObject ->
> emit(JSONObject, String, String) (String values are parsed out of
> JSONObject)
> bolt2 -> receives bolt1 tuples based on fieldsGrouping; execute uses
> JSONObject, String, String to perform some operation -> emit(JSONObject,
> String) (String values are based on some logic)
>
> So while you could go either extreme (continually pass JSONObject value as
> tuple, or parse the JSONObject and pass only decomposed values as tuple),
> you can also do both, which would allow you to use fieldGrouping, in case
> that is important (it is important for our case).
>
> Tyson
>
>
> On Mar 29, 2014, at 1:44 PM, Software Dev <st...@gmail.com>
> wrote:
>
> > We actually have a spout that emits just 1 JSON string per tuple.
> > Wondering what should be down downstream after we have the JSON string
> >
> > On Sat, Mar 29, 2014 at 12:20 PM, Andrew Neilson <ar...@gmail.com>
> wrote:
> >> my team's project has successfully used Jackson
> >> (https://github.com/FasterXML/jackson) to deserialize a spout of JSON
> arrays
> >> into tuples, and I can recommend it. Though I'll warn you that it takes
> a
> >> little bit of work beyond the most basic usage (i.e.
> mapper.readValue(json,
> >> List.class)) to avoid dealing with type ambiguity.
> >>
> >>
> >> On Sat, Mar 29, 2014 at 12:06 PM, Software Dev <
> static.void.dev@gmail.com>
> >> wrote:
> >>>
> >>> Say we are receiving tuples of JSON from a spout. Should we just keep
> >>> passing around the JSON string and deserialize it in each bolt or Is
> >>> it best to break apart the JSON object into a bunch of fields that can
> >>> be passed around.
> >>>
> >>> I'm thinking in terms of performance the latter may be "better"
> >>> although it will slightly make the rest of the topology more complex.
> >>>
> >>> Also, what is a good JSON library to work with?
> >>>
> >>> Thanks
> >>
> >>
>
>
--
Cody A. Ray, LEED AP
cody.a.ray@gmail.com
215.501.7891
Re: Working with JSON
Posted by Tyson Norris <tn...@adobe.com>.
We have a similar setup, and based on routing needs we plan to pass the original JSON, plus some extra fields that will simplify routing via fieldsGrouping.
e.g.
spout -> content (JSONObject)
bolt1 -> receives all spout tuples; execute() uses JSONObject -> emit(JSONObject, String, String) (String values are parsed out of JSONObject)
bolt2 -> receives bolt1 tuples based on fieldsGrouping; execute uses JSONObject, String, String to perform some operation -> emit(JSONObject, String) (String values are based on some logic)
So while you could go either extreme (continually pass JSONObject value as tuple, or parse the JSONObject and pass only decomposed values as tuple), you can also do both, which would allow you to use fieldGrouping, in case that is important (it is important for our case).
Tyson
On Mar 29, 2014, at 1:44 PM, Software Dev <st...@gmail.com> wrote:
> We actually have a spout that emits just 1 JSON string per tuple.
> Wondering what should be down downstream after we have the JSON string
>
> On Sat, Mar 29, 2014 at 12:20 PM, Andrew Neilson <ar...@gmail.com> wrote:
>> my team's project has successfully used Jackson
>> (https://github.com/FasterXML/jackson) to deserialize a spout of JSON arrays
>> into tuples, and I can recommend it. Though I'll warn you that it takes a
>> little bit of work beyond the most basic usage (i.e. mapper.readValue(json,
>> List.class)) to avoid dealing with type ambiguity.
>>
>>
>> On Sat, Mar 29, 2014 at 12:06 PM, Software Dev <st...@gmail.com>
>> wrote:
>>>
>>> Say we are receiving tuples of JSON from a spout. Should we just keep
>>> passing around the JSON string and deserialize it in each bolt or Is
>>> it best to break apart the JSON object into a bunch of fields that can
>>> be passed around.
>>>
>>> I'm thinking in terms of performance the latter may be "better"
>>> although it will slightly make the rest of the topology more complex.
>>>
>>> Also, what is a good JSON library to work with?
>>>
>>> Thanks
>>
>>
Re: Working with JSON
Posted by Software Dev <st...@gmail.com>.
We actually have a spout that emits just 1 JSON string per tuple.
Wondering what should be down downstream after we have the JSON string
On Sat, Mar 29, 2014 at 12:20 PM, Andrew Neilson <ar...@gmail.com> wrote:
> my team's project has successfully used Jackson
> (https://github.com/FasterXML/jackson) to deserialize a spout of JSON arrays
> into tuples, and I can recommend it. Though I'll warn you that it takes a
> little bit of work beyond the most basic usage (i.e. mapper.readValue(json,
> List.class)) to avoid dealing with type ambiguity.
>
>
> On Sat, Mar 29, 2014 at 12:06 PM, Software Dev <st...@gmail.com>
> wrote:
>>
>> Say we are receiving tuples of JSON from a spout. Should we just keep
>> passing around the JSON string and deserialize it in each bolt or Is
>> it best to break apart the JSON object into a bunch of fields that can
>> be passed around.
>>
>> I'm thinking in terms of performance the latter may be "better"
>> although it will slightly make the rest of the topology more complex.
>>
>> Also, what is a good JSON library to work with?
>>
>> Thanks
>
>
Re: Working with JSON
Posted by Andrew Neilson <ar...@gmail.com>.
my team's project has successfully used Jackson (
https://github.com/FasterXML/jackson) to deserialize a spout of JSON arrays
into tuples, and I can recommend it. Though I'll warn you that it takes a
little bit of work beyond the most basic usage (i.e. mapper.readValue(json,
List.class)) to avoid dealing with type ambiguity.
On Sat, Mar 29, 2014 at 12:06 PM, Software Dev <st...@gmail.com>wrote:
> Say we are receiving tuples of JSON from a spout. Should we just keep
> passing around the JSON string and deserialize it in each bolt or Is
> it best to break apart the JSON object into a bunch of fields that can
> be passed around.
>
> I'm thinking in terms of performance the latter may be "better"
> although it will slightly make the rest of the topology more complex.
>
> Also, what is a good JSON library to work with?
>
> Thanks
>