You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@samza.apache.org by Raj raj <ra...@gmail.com> on 2016/11/05 15:09:40 UTC
Using json serde with KeyValueStore
Hi,
I am starting out with Samza and my test program is able to read a
message from Kafka. My next task is to write the message to KeyValue
store. The message is formatted as Json in Kafka and I am using json
serde as my msg serde. The incoming message is available to me as a
LinkedHashMap when I call envelope.getMessage(). i.e. serde
automatically converted the json string to a LinkedHashMap.
Now I want to store this to KeyValueStore. Will using a json serde for
stores.test-container.msg.serde automatically convert the
LinkedHashMap to a format suitable for storing in the KeyValueStore?
i.e. the reverse process of what it did when I called
envelope.getMessage().
If it is not possible what is the best practice of storing a
LinkedHashMap in KeyStoreValue?
I am pasting the current code I am working with in pastebin for
reference, there isn't much in the code, just the boilerplate stuff.
test-container.properties : http://pastebin.com/dkbnjtNh
TestContainer.java : http://pastebin.com/93ZBgeWa
Thanks and Regards,
Raj
Re: Using json serde with KeyValueStore
Posted by Jagadish Venkatraman <ja...@gmail.com>.
Good to know!
Thanks,
Jagadish
On Tue, Nov 8, 2016 at 12:01 AM, Raj raj <ra...@gmail.com> wrote:
> Thanks!
>
> It is working as expected.
>
> raj
> Thanks and Regards,
>
> Raj
>
>
> On Sat, Nov 5, 2016 at 11:24 PM, Jagadish Venkatraman
> <ja...@gmail.com> wrote:
> > You can write a small test job and verify this! I believe the serde
> should
> > probably be able to serialize it correctly.
> >
> > You can define the KeyValueStore to be typed as <String,Object>.
> >
> > - You can put the LHM directly into the store.
> > - When you do a get on the store, you should probably be able re-cast it
> > back to an LHM.
> >
> > String/json serialization is not necessarily performant for all
> > cases. There maybe other efficient serialization formats like protobuffs/
> > avro depending on the use case. Should you want to write your own
> > serialization have a look at the SerdeFactory interface.
> >
> > Thanks
> > Jag
> >
> > On Saturday, November 5, 2016, Raj raj <ra...@gmail.com> wrote:
> >
> >> Hi,
> >>
> >> I am starting out with Samza and my test program is able to read a
> >> message from Kafka. My next task is to write the message to KeyValue
> >> store. The message is formatted as Json in Kafka and I am using json
> >> serde as my msg serde. The incoming message is available to me as a
> >> LinkedHashMap when I call envelope.getMessage(). i.e. serde
> >> automatically converted the json string to a LinkedHashMap.
> >>
> >> Now I want to store this to KeyValueStore. Will using a json serde for
> >> stores.test-container.msg.serde automatically convert the
> >> LinkedHashMap to a format suitable for storing in the KeyValueStore?
> >> i.e. the reverse process of what it did when I called
> >> envelope.getMessage().
> >>
> >> If it is not possible what is the best practice of storing a
> >> LinkedHashMap in KeyStoreValue?
> >>
> >> I am pasting the current code I am working with in pastebin for
> >> reference, there isn't much in the code, just the boilerplate stuff.
> >>
> >> test-container.properties : http://pastebin.com/dkbnjtNh
> >> TestContainer.java : http://pastebin.com/93ZBgeWa
> >>
> >> Thanks and Regards,
> >>
> >> Raj
> >>
> >
> >
> > --
> > Sent from my iphone.
>
--
Jagadish V,
Graduate Student,
Department of Computer Science,
Stanford University
Re: Using json serde with KeyValueStore
Posted by Raj raj <ra...@gmail.com>.
Thanks!
It is working as expected.
raj
Thanks and Regards,
Raj
On Sat, Nov 5, 2016 at 11:24 PM, Jagadish Venkatraman
<ja...@gmail.com> wrote:
> You can write a small test job and verify this! I believe the serde should
> probably be able to serialize it correctly.
>
> You can define the KeyValueStore to be typed as <String,Object>.
>
> - You can put the LHM directly into the store.
> - When you do a get on the store, you should probably be able re-cast it
> back to an LHM.
>
> String/json serialization is not necessarily performant for all
> cases. There maybe other efficient serialization formats like protobuffs/
> avro depending on the use case. Should you want to write your own
> serialization have a look at the SerdeFactory interface.
>
> Thanks
> Jag
>
> On Saturday, November 5, 2016, Raj raj <ra...@gmail.com> wrote:
>
>> Hi,
>>
>> I am starting out with Samza and my test program is able to read a
>> message from Kafka. My next task is to write the message to KeyValue
>> store. The message is formatted as Json in Kafka and I am using json
>> serde as my msg serde. The incoming message is available to me as a
>> LinkedHashMap when I call envelope.getMessage(). i.e. serde
>> automatically converted the json string to a LinkedHashMap.
>>
>> Now I want to store this to KeyValueStore. Will using a json serde for
>> stores.test-container.msg.serde automatically convert the
>> LinkedHashMap to a format suitable for storing in the KeyValueStore?
>> i.e. the reverse process of what it did when I called
>> envelope.getMessage().
>>
>> If it is not possible what is the best practice of storing a
>> LinkedHashMap in KeyStoreValue?
>>
>> I am pasting the current code I am working with in pastebin for
>> reference, there isn't much in the code, just the boilerplate stuff.
>>
>> test-container.properties : http://pastebin.com/dkbnjtNh
>> TestContainer.java : http://pastebin.com/93ZBgeWa
>>
>> Thanks and Regards,
>>
>> Raj
>>
>
>
> --
> Sent from my iphone.
Re: Using json serde with KeyValueStore
Posted by Jagadish Venkatraman <ja...@gmail.com>.
You can write a small test job and verify this! I believe the serde should
probably be able to serialize it correctly.
You can define the KeyValueStore to be typed as <String,Object>.
- You can put the LHM directly into the store.
- When you do a get on the store, you should probably be able re-cast it
back to an LHM.
String/json serialization is not necessarily performant for all
cases. There maybe other efficient serialization formats like protobuffs/
avro depending on the use case. Should you want to write your own
serialization have a look at the SerdeFactory interface.
Thanks
Jag
On Saturday, November 5, 2016, Raj raj <ra...@gmail.com> wrote:
> Hi,
>
> I am starting out with Samza and my test program is able to read a
> message from Kafka. My next task is to write the message to KeyValue
> store. The message is formatted as Json in Kafka and I am using json
> serde as my msg serde. The incoming message is available to me as a
> LinkedHashMap when I call envelope.getMessage(). i.e. serde
> automatically converted the json string to a LinkedHashMap.
>
> Now I want to store this to KeyValueStore. Will using a json serde for
> stores.test-container.msg.serde automatically convert the
> LinkedHashMap to a format suitable for storing in the KeyValueStore?
> i.e. the reverse process of what it did when I called
> envelope.getMessage().
>
> If it is not possible what is the best practice of storing a
> LinkedHashMap in KeyStoreValue?
>
> I am pasting the current code I am working with in pastebin for
> reference, there isn't much in the code, just the boilerplate stuff.
>
> test-container.properties : http://pastebin.com/dkbnjtNh
> TestContainer.java : http://pastebin.com/93ZBgeWa
>
> Thanks and Regards,
>
> Raj
>
--
Sent from my iphone.