You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@storm.apache.org by Pete Carlson <pe...@tetraconcepts.com> on 2013/12/19 16:18:02 UTC
Example of parsing JSON for a field in a Storm bolt
Hi,
Can anyone point me to an example of a Storm bolt that is parsing JSON for
a specific field type? I'vetried googling and looking through the storm
examples with no luck.
For example, say I'm receiving two JSON strings from a JMS queue spout:
[{"Company":"GM", "Facility":"Manufacturing Line", "Make":"Chevy",
"Model":"Cobalt", "Type":"Parts_Received", "Data":"<CSV file of parts
received>"}]
and
[{"Company":"GM", "Facility":"Manufacturing Line", "Make":"Chevy",
"Model":"Cobalt", "Type":"Car_Recalls", "Data":"<CSV file of cars
recalled>"}]
but I only want to emit the JSON of type "Car_Recalls" from my bolt.
How do I parse for the Type field and then use that to emit the JSON?
Pete
Re: Example of parsing JSON for a field in a Storm bolt
Posted by Pete Carlson <pe...@tetraconcepts.com>.
Btw, the JPF is a JsonParserFactory that is extending the JsonFactory from
the Jackson library
On Thu, Dec 19, 2013 at 10:39 AM, Pete Carlson <pe...@tetraconcepts.com>wrote:
> Basically I have some existing code that is a Node parser of JSON:
>
> // Split each sub-event
> try (NodeParser nodes = new NodeParser(json)) {
> for (TreeNode node : nodes) {
> try {
> String obj = JPF.nodeToString(node);
> LOG.info("Node: {} => {}", node, obj);
> Values output = new Values(new JsonHolder(obj, node));
> collector.emit(output);
> } catch (IOException iox) {
> LOG.error("Error processing event: {}", node);
> }
> }
> } catch (Exception jpe) {
> LOG.error("Error parsing data: " + data, jpe);
> collector.fail(input);
> return;
> }
>
> So, it sounds like I just need to put the output in a hashmap and then
> only emit the type I want?
>
>
> On Thu, Dec 19, 2013 at 10:34 AM, Ricardo Teixeira <ri...@rteixeira.eu>wrote:
>
>> Oh, and BTW, I'm my experience with XML, converting the json to an
>> hashmap instead of an complex object like CarRecalls or PartsReceived is
>> much faster.
>>
>> I would say that the same would be true for JSON.
>>
>> Ricardo Teixeira
>> http://www.rteixeira.eu
>> tel: +351 938655255
>> skype: rjsteixeira
>>
>>
>> On Thu, Dec 19, 2013 at 3:26 PM, Nathan Leung <nc...@gmail.com> wrote:
>>
>>> To add a little to Ricardo's response there are several json parsers
>>> that are very easy to use such as Jackson and gson.
>>> On Dec 19, 2013 10:25 AM, "Ricardo Teixeira" <ri...@rteixeira.eu>
>>> wrote:
>>>
>>>> Deserialize the data to a hashmap (or something like that) and
>>>> get("Type") from that hashmap.
>>>> Then emit Data IF Type == "Car_Recalls".
>>>>
>>>>
>>>>
>>>> Ricardo Teixeira
>>>> http://www.rteixeira.eu
>>>> tel: +351 938655255
>>>> skype: rjsteixeira
>>>>
>>>>
>>>> On Thu, Dec 19, 2013 at 3:18 PM, Pete Carlson <pe...@tetraconcepts.com>wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> Can anyone point me to an example of a Storm bolt that is parsing JSON
>>>>> for a specific field type? I'vetried googling and looking through the
>>>>> storm examples with no luck.
>>>>>
>>>>> For example, say I'm receiving two JSON strings from a JMS queue spout:
>>>>>
>>>>> [{"Company":"GM", "Facility":"Manufacturing Line", "Make":"Chevy",
>>>>> "Model":"Cobalt", "Type":"Parts_Received", "Data":"<CSV file of parts
>>>>> received>"}]
>>>>>
>>>>> and
>>>>>
>>>>> [{"Company":"GM", "Facility":"Manufacturing Line", "Make":"Chevy",
>>>>> "Model":"Cobalt", "Type":"Car_Recalls", "Data":"<CSV file of cars
>>>>> recalled>"}]
>>>>>
>>>>> but I only want to emit the JSON of type "Car_Recalls" from my bolt.
>>>>>
>>>>> How do I parse for the Type field and then use that to emit the JSON?
>>>>>
>>>>>
>>>>> Pete
>>>>>
>>>>
>>>>
>>
>
>
> --
> Pete Carlson
> Software Developer
> Tetra Concepts LLC
> Cell: 301-526-0681
>
--
Pete Carlson
Software Developer
Tetra Concepts LLC
Cell: 301-526-0681
Re: Example of parsing JSON for a field in a Storm bolt
Posted by Pete Carlson <pe...@tetraconcepts.com>.
Basically I have some existing code that is a Node parser of JSON:
// Split each sub-event
try (NodeParser nodes = new NodeParser(json)) {
for (TreeNode node : nodes) {
try {
String obj = JPF.nodeToString(node);
LOG.info("Node: {} => {}", node, obj);
Values output = new Values(new JsonHolder(obj, node));
collector.emit(output);
} catch (IOException iox) {
LOG.error("Error processing event: {}", node);
}
}
} catch (Exception jpe) {
LOG.error("Error parsing data: " + data, jpe);
collector.fail(input);
return;
}
So, it sounds like I just need to put the output in a hashmap and then only
emit the type I want?
On Thu, Dec 19, 2013 at 10:34 AM, Ricardo Teixeira <ri...@rteixeira.eu>wrote:
> Oh, and BTW, I'm my experience with XML, converting the json to an hashmap
> instead of an complex object like CarRecalls or PartsReceived is much
> faster.
>
> I would say that the same would be true for JSON.
>
> Ricardo Teixeira
> http://www.rteixeira.eu
> tel: +351 938655255
> skype: rjsteixeira
>
>
> On Thu, Dec 19, 2013 at 3:26 PM, Nathan Leung <nc...@gmail.com> wrote:
>
>> To add a little to Ricardo's response there are several json parsers that
>> are very easy to use such as Jackson and gson.
>> On Dec 19, 2013 10:25 AM, "Ricardo Teixeira" <ri...@rteixeira.eu>
>> wrote:
>>
>>> Deserialize the data to a hashmap (or something like that) and
>>> get("Type") from that hashmap.
>>> Then emit Data IF Type == "Car_Recalls".
>>>
>>>
>>>
>>> Ricardo Teixeira
>>> http://www.rteixeira.eu
>>> tel: +351 938655255
>>> skype: rjsteixeira
>>>
>>>
>>> On Thu, Dec 19, 2013 at 3:18 PM, Pete Carlson <pe...@tetraconcepts.com>wrote:
>>>
>>>> Hi,
>>>>
>>>> Can anyone point me to an example of a Storm bolt that is parsing JSON
>>>> for a specific field type? I'vetried googling and looking through the
>>>> storm examples with no luck.
>>>>
>>>> For example, say I'm receiving two JSON strings from a JMS queue spout:
>>>>
>>>> [{"Company":"GM", "Facility":"Manufacturing Line", "Make":"Chevy",
>>>> "Model":"Cobalt", "Type":"Parts_Received", "Data":"<CSV file of parts
>>>> received>"}]
>>>>
>>>> and
>>>>
>>>> [{"Company":"GM", "Facility":"Manufacturing Line", "Make":"Chevy",
>>>> "Model":"Cobalt", "Type":"Car_Recalls", "Data":"<CSV file of cars
>>>> recalled>"}]
>>>>
>>>> but I only want to emit the JSON of type "Car_Recalls" from my bolt.
>>>>
>>>> How do I parse for the Type field and then use that to emit the JSON?
>>>>
>>>>
>>>> Pete
>>>>
>>>
>>>
>
--
Pete Carlson
Software Developer
Tetra Concepts LLC
Cell: 301-526-0681
Re: Example of parsing JSON for a field in a Storm bolt
Posted by Ricardo Teixeira <ri...@rteixeira.eu>.
Oh, and BTW, I'm my experience with XML, converting the json to an hashmap
instead of an complex object like CarRecalls or PartsReceived is much
faster.
I would say that the same would be true for JSON.
Ricardo Teixeira
http://www.rteixeira.eu
tel: +351 938655255
skype: rjsteixeira
On Thu, Dec 19, 2013 at 3:26 PM, Nathan Leung <nc...@gmail.com> wrote:
> To add a little to Ricardo's response there are several json parsers that
> are very easy to use such as Jackson and gson.
> On Dec 19, 2013 10:25 AM, "Ricardo Teixeira" <ri...@rteixeira.eu> wrote:
>
>> Deserialize the data to a hashmap (or something like that) and
>> get("Type") from that hashmap.
>> Then emit Data IF Type == "Car_Recalls".
>>
>>
>>
>> Ricardo Teixeira
>> http://www.rteixeira.eu
>> tel: +351 938655255
>> skype: rjsteixeira
>>
>>
>> On Thu, Dec 19, 2013 at 3:18 PM, Pete Carlson <pe...@tetraconcepts.com>wrote:
>>
>>> Hi,
>>>
>>> Can anyone point me to an example of a Storm bolt that is parsing JSON
>>> for a specific field type? I'vetried googling and looking through the
>>> storm examples with no luck.
>>>
>>> For example, say I'm receiving two JSON strings from a JMS queue spout:
>>>
>>> [{"Company":"GM", "Facility":"Manufacturing Line", "Make":"Chevy",
>>> "Model":"Cobalt", "Type":"Parts_Received", "Data":"<CSV file of parts
>>> received>"}]
>>>
>>> and
>>>
>>> [{"Company":"GM", "Facility":"Manufacturing Line", "Make":"Chevy",
>>> "Model":"Cobalt", "Type":"Car_Recalls", "Data":"<CSV file of cars
>>> recalled>"}]
>>>
>>> but I only want to emit the JSON of type "Car_Recalls" from my bolt.
>>>
>>> How do I parse for the Type field and then use that to emit the JSON?
>>>
>>>
>>> Pete
>>>
>>
>>
Re: Example of parsing JSON for a field in a Storm bolt
Posted by Nathan Leung <nc...@gmail.com>.
To add a little to Ricardo's response there are several json parsers that
are very easy to use such as Jackson and gson.
On Dec 19, 2013 10:25 AM, "Ricardo Teixeira" <ri...@rteixeira.eu> wrote:
> Deserialize the data to a hashmap (or something like that) and get("Type")
> from that hashmap.
> Then emit Data IF Type == "Car_Recalls".
>
>
>
> Ricardo Teixeira
> http://www.rteixeira.eu
> tel: +351 938655255
> skype: rjsteixeira
>
>
> On Thu, Dec 19, 2013 at 3:18 PM, Pete Carlson <pe...@tetraconcepts.com>wrote:
>
>> Hi,
>>
>> Can anyone point me to an example of a Storm bolt that is parsing JSON
>> for a specific field type? I'vetried googling and looking through the
>> storm examples with no luck.
>>
>> For example, say I'm receiving two JSON strings from a JMS queue spout:
>>
>> [{"Company":"GM", "Facility":"Manufacturing Line", "Make":"Chevy",
>> "Model":"Cobalt", "Type":"Parts_Received", "Data":"<CSV file of parts
>> received>"}]
>>
>> and
>>
>> [{"Company":"GM", "Facility":"Manufacturing Line", "Make":"Chevy",
>> "Model":"Cobalt", "Type":"Car_Recalls", "Data":"<CSV file of cars
>> recalled>"}]
>>
>> but I only want to emit the JSON of type "Car_Recalls" from my bolt.
>>
>> How do I parse for the Type field and then use that to emit the JSON?
>>
>>
>> Pete
>>
>
>
Re: Example of parsing JSON for a field in a Storm bolt
Posted by Ricardo Teixeira <ri...@rteixeira.eu>.
Deserialize the data to a hashmap (or something like that) and get("Type")
from that hashmap.
Then emit Data IF Type == "Car_Recalls".
Ricardo Teixeira
http://www.rteixeira.eu
tel: +351 938655255
skype: rjsteixeira
On Thu, Dec 19, 2013 at 3:18 PM, Pete Carlson <pe...@tetraconcepts.com>wrote:
> Hi,
>
> Can anyone point me to an example of a Storm bolt that is parsing JSON for
> a specific field type? I'vetried googling and looking through the storm
> examples with no luck.
>
> For example, say I'm receiving two JSON strings from a JMS queue spout:
>
> [{"Company":"GM", "Facility":"Manufacturing Line", "Make":"Chevy",
> "Model":"Cobalt", "Type":"Parts_Received", "Data":"<CSV file of parts
> received>"}]
>
> and
>
> [{"Company":"GM", "Facility":"Manufacturing Line", "Make":"Chevy",
> "Model":"Cobalt", "Type":"Car_Recalls", "Data":"<CSV file of cars
> recalled>"}]
>
> but I only want to emit the JSON of type "Car_Recalls" from my bolt.
>
> How do I parse for the Type field and then use that to emit the JSON?
>
>
> Pete
>