You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@storm.apache.org by Nathan Leung <nc...@gmail.com> on 2016/05/15 18:04:00 UTC

Re: Split Kafka JSON String

Easiest way is to map your job to a pojo and use Jackson or gson to convert
the json.
On May 15, 2016 1:48 PM, "Daniela S" <da...@gmx.at> wrote:

> Hi
>
> I am receiving Strings of JSON from Kafka. I would like to split the
> string to get each field from the JSON object to store it into Redis.
> How can I split the string and assign the parts to variables?
>
> Thank you in advance.
>
> Regards,
> Daniela
>

Re: Split Kafka JSON String

Posted by Henry Hottelet <ho...@gmail.com>.

HERE is the JSON to POJO IntelliJ plugin

https://plugins.jetbrains.com/plugin/7678?pr=

— 
Henry Hottelet
646-543-6104 (Google Voice: cell/sms)
hottelet@gmail.com <ma...@gmail.com> (Email)
http://www.linkedin.com/in/hottelet <http://www.linkedin.com/in/hottelet> (LinkedIn)
https://technologyventureslimited.appointlet.com <https://technologyventureslimited.appointlet.com/> (Schedule appointment)

This transmission may contain information that is privileged,
confidential, legally privileged, and/or exempt from disclosure
under applicable law. If you are not the intended recipient, you
are hereby notified that any disclosure, copying, distribution, or
use of the information contained herein (including any reliance
thereon) is STRICTLY PROHIBITED. Although this transmission and any
attachments are believed to be free of any virus or other defect
that might affect any computer system into which it is received and
opened, it is the responsibility of the recipient to ensure that it
is virus free and no responsibility is accepted by Henry Hottelet
as applicable, for any loss or damage arising in any way from its use.
If you received this transmission in error, please immediately contact
the sender and destroy the material in its entirety, whether in
electronic or hard copy format. Thank you.

> On May 15, 2016, at 2:04 PM, Nathan Leung <nc...@gmail.com> wrote:
> 
> Map your json to a pojo. Sorry auto correct got me.
> 
> On May 15, 2016 2:04 PM, "Nathan Leung" <ncleung@gmail.com <ma...@gmail.com>> wrote:
> Easiest way is to map your job to a pojo and use Jackson or gson to convert the json.
> 
> On May 15, 2016 1:48 PM, "Daniela S" <daniela_4444@gmx.at <ma...@gmx.at>> wrote:
> Hi
>  
> I am receiving Strings of JSON from Kafka. I would like to split the string to get each field from the JSON object to store it into Redis.
> How can I split the string and assign the parts to variables?
>  
> Thank you in advance.
>  
> Regards,
> Daniela

Re: Split Kafka JSON String

Posted by Nathan Leung <nc...@gmail.com>.

Map your json to a pojo. Sorry auto correct got me.
On May 15, 2016 2:04 PM, "Nathan Leung" <nc...@gmail.com> wrote:

> Easiest way is to map your job to a pojo and use Jackson or gson to
> convert the json.
> On May 15, 2016 1:48 PM, "Daniela S" <da...@gmx.at> wrote:
>
>> Hi
>>
>> I am receiving Strings of JSON from Kafka. I would like to split the
>> string to get each field from the JSON object to store it into Redis.
>> How can I split the string and assign the parts to variables?
>>
>> Thank you in advance.
>>
>> Regards,
>> Daniela
>>
>

Re: Aw: Re: Split Kafka JSON String

Posted by Sai Dilip Reddy Kiralam <dk...@aadhya-analytics.com>.

Hi,

use json parsing in the bolt code.get the string  json value of 0; use
following link for clear understanding  !

https://azure.microsoft.com/en-in/documentation/articles/hdinsight-storm-develop-csharp-event-hub-topology/



*Best regards,*

*K.Sai Dilip Reddy.*

On Mon, May 16, 2016 at 1:37 AM, Henry Hottelet <ho...@gmail.com> wrote:

> Daniella,
>
> I am curious about your architecture with Kafka and storm.  Can you send
> the github link you used for that project?
>
> I would like to be able to see your example. I use IntelliJ.
>
> Henry
>
> Sent from my iPad
>
> On May 15, 2016, at 2:14 PM, Daniela S <da...@gmx.at> wrote:
>
> Hi,
>
> thank you very much for your answers. But unfortunately I do not have a
> JSON object yet, I only have a String, containing a JSON object. I receive
> the Strings from Kafka but I do not know how to get the information out of
> the String.
>
> example: {"timestamp": 1463335828342, "ID":570,"command":"START"}
>
> I would like to extract the three fields out of the String. Then I would
> like to store it into Redis.
> Afterwards I would like to pull all the stored (active) devices out of
> Redis and to build a sum over all the devices every minute. For building
> the sum I have to add another value for every device. I would like to use
> Redis as a kind of cache.
>
> Regards,
> Daniela
>
> *Gesendet:* Sonntag, 15. Mai 2016 um 20:06 Uhr
> *Von:* "Henry Hottelet" <ho...@gmail.com>
> *An:* user@storm.apache.org
> *Betreff:* Re: Split Kafka JSON String
>
> Try using intellij plug in that converts json to pojo classes
> On May 15, 2016 2:04 PM, "Nathan Leung" <nc...@gmail.com> wrote:
>>
>> Easiest way is to map your job to a pojo and use Jackson or gson to
>> convert the json.
>> On May 15, 2016 1:48 PM, "Daniela S" <da...@gmx.at> wrote:
>>>
>>> Hi
>>>
>>> I am receiving Strings of JSON from Kafka. I would like to split the
>>> string to get each field from the JSON object to store it into Redis.
>>> How can I split the string and assign the parts to variables?
>>>
>>> Thank you in advance.
>>>
>>> Regards,
>>> Daniela
>>>
>>

RE: Re: Re: Split Kafka JSON String

Posted by "Sinnema, Remon" <re...@emc.com>.

Import org.json.JSONObject;

JSONObject obj = new JSONObject(jsonString);
String command = obj.getString(“command”);

From: Daniela S [mailto:daniela_4444@gmx.at]
Sent: dinsdag 24 mei 2016 18:08
To: user@storm.apache.org
Subject: Aw: Re: Re: Split Kafka JSON String

Hi,

unfortunately it does not work. Do I need this step? Or is it also possible to extract the value of a specific field out of my JSON string?
So my JSON string is for example
  {"timestamp": "23462626", "ID1": "1", "ID2": "100", "command": "START"}

I would like to use a filter bolt to filter for START and END. So I have to access the command field within the JSON string.

Afterwards I would like to store the START JSONs in Redis. The key should be ID1 and ID2. Is that possible? What would be the best way to store it in Redis and to define two keys for each value?

The END JSONs should be deleted from Redis.

Thank you in advance.

Regards,
Daniela

Gesendet: Montag, 16. Mai 2016 um 14:08 Uhr
Von: "Henry Hottelet" <ho...@gmail.com>>
An: user@storm.apache.org<ma...@storm.apache.org>
Betreff: Re: Re: Split Kafka JSON String

https://code.google.com/archive/p/json-simple/

import org.json.simple.JSONObject;

void main() {

String yourString;

String key;

JSONObject j = yourString;

Redis.put(key,j);

}

—
Henry Hottelet
646-543-6104 (Google Voice: cell/sms)
hottelet@gmail.com (Email)
http://www.linkedin.com/in/hottelet (LinkedIn)
https://technologyventureslimited.appointlet.com (Schedule appointment)

This transmission may contain information that is privileged,
confidential, legally privileged, and/or exempt from disclosure
under applicable law. If you are not the intended recipient, you
are hereby notified that any disclosure, copying, distribution, or
use of the information contained herein (including any reliance
thereon) is STRICTLY PROHIBITED. Although this transmission and any
attachments are believed to be free of any virus or other defect
that might affect any computer system into which it is received and
opened, it is the responsibility of the recipient to ensure that it
is virus free and no responsibility is accepted by Henry Hottelet
as applicable, for any loss or damage arising in any way from its use.
If you received this transmission in error, please immediately contact
the sender and destroy the material in its entirety, whether in
electronic or hard copy format. Thank you.

On May 16, 2016, at 4:11 AM, Daniela S <da...@gmx.at> wrote:

Henry,

thank you for your help. I did not use any github link for my project. I tried to build the project on my own, with the help of some books and available code parts from the Internet. I try to explain what I am doing:

1. I have generated random JSON data with a JSON data generator from the Internet
    the JSON data looks as follows:
    {"timestamp": "nowTimestamp()",
      "ID1": "integer(1,1000)",
      "ID2": "integer(1,100)",
      "command": "random('START','END')"
   }
   The JSON generator simulates a data stream, sending the data to a specific Kafka topic as String

2. I would like to have a KafkaSpout to fetch the data from the Kafka topic.

3. Then I would like to store all the events containing "START" into Redis and to delete all the events containing "END" from Redis. The key should be a combination of ID1 and ID2.

4. In the next step I would like to pull all the entries stored in Redis and to add a specific value for every entry (this values is located in a database and I have to look it up for every entry). Then I would like to use these values to build a sum over all entries. This should be repeated every minute, as there might have been new start or end events. I thought I could use a sliding window for that.

But unfortunately I have no idea how to proceed with the JSON String, I receive from Kafka. I already read a lot about the possibility to store JSON Strings in Redis, but I have no idea how to build the key with both IDs and how to access the fields in the JSON String.

Hope you can help me.

Thank you so much in advance!

Regards,
Daniela

Gesendet: Sonntag, 15. Mai 2016 um 22:07 Uhr
Von: "Henry Hottelet" <ho...@gmail.com>
An: user@storm.apache.org
Betreff: Re: Aw: Re: Split Kafka JSON String
Daniella,

I am curious about your architecture with Kafka and storm.  Can you send the github link you used for that project?

I would like to be able to see your example. I use IntelliJ.

Henry

Sent from my iPad

On May 15, 2016, at 2:14 PM, Daniela S <da...@gmx.at>> wrote:

Hi,

thank you very much for your answers. But unfortunately I do not have a JSON object yet, I only have a String, containing a JSON object. I receive the Strings from Kafka but I do not know how to get the information out of the String.

example: {"timestamp": 1463335828342, "ID":570,"command":"START"}

I would like to extract the three fields out of the String. Then I would like to store it into Redis.
Afterwards I would like to pull all the stored (active) devices out of Redis and to build a sum over all the devices every minute. For building the sum I have to add another value for every device. I would like to use Redis as a kind of cache.

Regards,
Daniela

Gesendet: Sonntag, 15. Mai 2016 um 20:06 Uhr
Von: "Henry Hottelet" <ho...@gmail.com>>
An: user@storm.apache.org<ma...@storm.apache.org>
Betreff: Re: Split Kafka JSON String

Try using intellij plug in that converts json to pojo classes
On May 15, 2016 2:04 PM, "Nathan Leung" <nc...@gmail.com>> wrote:

Easiest way is to map your job to a pojo and use Jackson or gson to convert the json.
On May 15, 2016 1:48 PM, "Daniela S" <da...@gmx.at>> wrote:
Hi

I am receiving Strings of JSON from Kafka. I would like to split the string to get each field from the JSON object to store it into Redis.
How can I split the string and assign the parts to variables?

Thank you in advance.

Regards,
Daniela

Re: Re: Split Kafka JSON String

Posted by Henry Hottelet <ho...@gmail.com>.

https://code.google.com/archive/p/json-simple/ <https://code.google.com/archive/p/json-simple/>

import org.json.simple.JSONObject;

void main() {
String yourString;
String key;
JSONObject j = yourString;

Redis.put(key,j);


}


— 
Henry Hottelet
646-543-6104 (Google Voice: cell/sms)
hottelet@gmail.com <ma...@gmail.com> (Email)
http://www.linkedin.com/in/hottelet <http://www.linkedin.com/in/hottelet> (LinkedIn)
https://technologyventureslimited.appointlet.com <https://technologyventureslimited.appointlet.com/> (Schedule appointment)

This transmission may contain information that is privileged,
confidential, legally privileged, and/or exempt from disclosure
under applicable law. If you are not the intended recipient, you
are hereby notified that any disclosure, copying, distribution, or
use of the information contained herein (including any reliance
thereon) is STRICTLY PROHIBITED. Although this transmission and any
attachments are believed to be free of any virus or other defect
that might affect any computer system into which it is received and
opened, it is the responsibility of the recipient to ensure that it
is virus free and no responsibility is accepted by Henry Hottelet
as applicable, for any loss or damage arising in any way from its use.
If you received this transmission in error, please immediately contact
the sender and destroy the material in its entirety, whether in
electronic or hard copy format. Thank you.

> On May 16, 2016, at 4:11 AM, Daniela S <da...@gmx.at> wrote:
> 
> Henry,
>  
> thank you for your help. I did not use any github link for my project. I tried to build the project on my own, with the help of some books and available code parts from the Internet. I try to explain what I am doing:
>  
> 1. I have generated random JSON data with a JSON data generator from the Internet
>     the JSON data looks as follows:
>     {"timestamp": "nowTimestamp()",
>       "ID1": "integer(1,1000)",
>       "ID2": "integer(1,100)",
>       "command": "random('START','END')"
>    }
>    The JSON generator simulates a data stream, sending the data to a specific Kafka topic as String
>  
> 2. I would like to have a KafkaSpout to fetch the data from the Kafka topic.
>  
> 3. Then I would like to store all the events containing "START" into Redis and to delete all the events containing "END" from Redis. The key should be a combination of ID1 and ID2.
>  
> 4. In the next step I would like to pull all the entries stored in Redis and to add a specific value for every entry (this values is located in a database and I have to look it up for every entry). Then I would like to use these values to build a sum over all entries. This should be repeated every minute, as there might have been new start or end events. I thought I could use a sliding window for that.
>  
> But unfortunately I have no idea how to proceed with the JSON String, I receive from Kafka. I already read a lot about the possibility to store JSON Strings in Redis, but I have no idea how to build the key with both IDs and how to access the fields in the JSON String.
>  
> Hope you can help me.
>  
> Thank you so much in advance!
>  
> Regards,
> Daniela 
>  
>  
>  
> Gesendet: Sonntag, 15. Mai 2016 um 22:07 Uhr
> Von: "Henry Hottelet" <ho...@gmail.com>
> An: user@storm.apache.org
> Betreff: Re: Aw: Re: Split Kafka JSON String
> Daniella,
>  
> I am curious about your architecture with Kafka and storm.  Can you send the github link you used for that project?
>  
> I would like to be able to see your example. I use IntelliJ.
>  
> Henry
> 
> Sent from my iPad
> 
> On May 15, 2016, at 2:14 PM, Daniela S <daniela_4444@gmx.at <x-...@gmx.at>> wrote:
>  
> Hi,
>  
> thank you very much for your answers. But unfortunately I do not have a JSON object yet, I only have a String, containing a JSON object. I receive the Strings from Kafka but I do not know how to get the information out of the String.
>  
> example: {"timestamp": 1463335828342, "ID":570,"command":"START"}
>  
> I would like to extract the three fields out of the String. Then I would like to store it into Redis.
> Afterwards I would like to pull all the stored (active) devices out of Redis and to build a sum over all the devices every minute. For building the sum I have to add another value for every device. I would like to use Redis as a kind of cache.
>  
> Regards,
> Daniela
>  
> Gesendet: Sonntag, 15. Mai 2016 um 20:06 Uhr
> Von: "Henry Hottelet" <hottelet@gmail.com <x-...@gmail.com>>
> An: user@storm.apache.org <x-...@storm.apache.org>
> Betreff: Re: Split Kafka JSON String
> Try using intellij plug in that converts json to pojo classes
> 
> On May 15, 2016 2:04 PM, "Nathan Leung" <ncleung@gmail.com <>> wrote:
> Easiest way is to map your job to a pojo and use Jackson or gson to convert the json.
> 
> On May 15, 2016 1:48 PM, "Daniela S" <daniela_4444@gmx.at <>> wrote:
> Hi
>  
> I am receiving Strings of JSON from Kafka. I would like to split the string to get each field from the JSON object to store it into Redis.
> How can I split the string and assign the parts to variables?
>  
> Thank you in advance.
>  
> Regards,
> Daniela

Re: Aw: Re: Split Kafka JSON String

Posted by Henry Hottelet <ho...@gmail.com>.

Daniella,

I am curious about your architecture with Kafka and storm.  Can you send the github link you used for that project?

I would like to be able to see your example. I use IntelliJ.

Henry

Sent from my iPad

> On May 15, 2016, at 2:14 PM, Daniela S <da...@gmx.at> wrote:
> 
> Hi,
>  
> thank you very much for your answers. But unfortunately I do not have a JSON object yet, I only have a String, containing a JSON object. I receive the Strings from Kafka but I do not know how to get the information out of the String.
>  
> example: {"timestamp": 1463335828342, "ID":570,"command":"START"}
>  
> I would like to extract the three fields out of the String. Then I would like to store it into Redis.
> Afterwards I would like to pull all the stored (active) devices out of Redis and to build a sum over all the devices every minute. For building the sum I have to add another value for every device. I would like to use Redis as a kind of cache.
>  
> Regards,
> Daniela
>  
> Gesendet: Sonntag, 15. Mai 2016 um 20:06 Uhr
> Von: "Henry Hottelet" <ho...@gmail.com>
> An: user@storm.apache.org
> Betreff: Re: Split Kafka JSON String
> Try using intellij plug in that converts json to pojo classes
> 
>> On May 15, 2016 2:04 PM, "Nathan Leung" <nc...@gmail.com> wrote:
>> Easiest way is to map your job to a pojo and use Jackson or gson to convert the json.
>> 
>>> On May 15, 2016 1:48 PM, "Daniela S" <da...@gmx.at> wrote:
>>> Hi
>>>  
>>> I am receiving Strings of JSON from Kafka. I would like to split the string to get each field from the JSON object to store it into Redis.
>>> How can I split the string and assign the parts to variables?
>>>  
>>> Thank you in advance.
>>>  
>>> Regards,
>>> Daniela

Re: Aw: Re: Split Kafka JSON String

Posted by Henry Hottelet <ho...@gmail.com>.

Daniela,

You Have To Use The Jackson mapper. Search for an example on stackoverflow.

Henry
On May 15, 2016 2:14 PM, "Daniela S" <da...@gmx.at> wrote:

> Hi,
>
> thank you very much for your answers. But unfortunately I do not have a
> JSON object yet, I only have a String, containing a JSON object. I receive
> the Strings from Kafka but I do not know how to get the information out of
> the String.
>
> example: {"timestamp": 1463335828342, "ID":570,"command":"START"}
>
> I would like to extract the three fields out of the String. Then I would
> like to store it into Redis.
> Afterwards I would like to pull all the stored (active) devices out of
> Redis and to build a sum over all the devices every minute. For building
> the sum I have to add another value for every device. I would like to use
> Redis as a kind of cache.
>
> Regards,
> Daniela
>
> *Gesendet:* Sonntag, 15. Mai 2016 um 20:06 Uhr
> *Von:* "Henry Hottelet" <ho...@gmail.com>
> *An:* user@storm.apache.org
> *Betreff:* Re: Split Kafka JSON String
>
> Try using intellij plug in that converts json to pojo classes
> On May 15, 2016 2:04 PM, "Nathan Leung" <nc...@gmail.com> wrote:
>>
>> Easiest way is to map your job to a pojo and use Jackson or gson to
>> convert the json.
>> On May 15, 2016 1:48 PM, "Daniela S" <da...@gmx.at> wrote:
>>>
>>> Hi
>>>
>>> I am receiving Strings of JSON from Kafka. I would like to split the
>>> string to get each field from the JSON object to store it into Redis.
>>> How can I split the string and assign the parts to variables?
>>>
>>> Thank you in advance.
>>>
>>> Regards,
>>> Daniela
>>>
>>

Re: Split Kafka JSON String

Posted by Henry Hottelet <ho...@gmail.com>.

Try using intellij plug in that converts json to pojo classes
On May 15, 2016 2:04 PM, "Nathan Leung" <nc...@gmail.com> wrote:

> Easiest way is to map your job to a pojo and use Jackson or gson to
> convert the json.
> On May 15, 2016 1:48 PM, "Daniela S" <da...@gmx.at> wrote:
>
>> Hi
>>
>> I am receiving Strings of JSON from Kafka. I would like to split the
>> string to get each field from the JSON object to store it into Redis.
>> How can I split the string and assign the parts to variables?
>>
>> Thank you in advance.
>>
>> Regards,
>> Daniela
>>
>