You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@kafka.apache.org by Jiayi Hoffman <ji...@gmail.com> on 2018/11/28 07:11:41 UTC

best practice of storing JSON that is searchable

Hi,

We have an application that uses MySQL to store devices and its metrics. We are thinking of migrating all data to Kafka, and using Kafka as the storage for the benefit of scaling, event souring and searchability in JSON.   I am looking for some advice on the topic design for my following use case: 

We currently have "device" data stored in a table. "device" has "device type", and we support hundreds of device types. Each type of device has its unique attributes. We keep “type" specific attributes as a JSON, and store the JSON in a CLOB column. The JSON schema is different for different “device types". If we store “device” data in Kafka, what is the best practice on designing the topic in this use case. We want to be able to search within the JSON.

Thanks,
Jiayi

Re: best practice of storing JSON that is searchable

Posted by Jiayi Hoffman <ji...@gmail.com>.

Ok. thanks.

> On Nov 27, 2018, at 11:31 PM, Jörn Franke <jo...@gmail.com> wrote:
> 
> Why not put it in Solr or similar ?
> 
>> Am 28.11.2018 um 08:11 schrieb Jiayi Hoffman <ji...@gmail.com>:
>> 
>> Hi,
>> 
>> We have an application that uses MySQL to store devices and its metrics. We are thinking of migrating all data to Kafka, and using Kafka as the storage for the benefit of scaling, event souring and searchability in JSON.   I am looking for some advice on the topic design for my following use case: 
>> 
>> We currently have "device" data stored in a table. "device" has "device type", and we support hundreds of device types. Each type of device has its unique attributes. We keep “type" specific attributes as a JSON, and store the JSON in a CLOB column. The JSON schema is different for different “device types". If we store “device” data in Kafka, what is the best practice on designing the topic in this use case. We want to be able to search within the JSON.
>> 
>> Thanks,
>> Jiayi
>>

Re: best practice of storing JSON that is searchable

Posted by Jörn Franke <jo...@gmail.com>.

Why not put it in Solr or similar ?

> Am 28.11.2018 um 08:11 schrieb Jiayi Hoffman <ji...@gmail.com>:
> 
> Hi,
> 
> We have an application that uses MySQL to store devices and its metrics. We are thinking of migrating all data to Kafka, and using Kafka as the storage for the benefit of scaling, event souring and searchability in JSON.   I am looking for some advice on the topic design for my following use case: 
> 
> We currently have "device" data stored in a table. "device" has "device type", and we support hundreds of device types. Each type of device has its unique attributes. We keep “type" specific attributes as a JSON, and store the JSON in a CLOB column. The JSON schema is different for different “device types". If we store “device” data in Kafka, what is the best practice on designing the topic in this use case. We want to be able to search within the JSON.
> 
> Thanks,
> Jiayi
>