You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by SenthilKumar K <se...@gmail.com> on 2017/08/18 15:45:20 UTC
Different Data Types under same topic
Hi Users , We have planned to use Kafka for one of the use to collect data
from different server and persist into Message Bus ..
Flow Would Be :
Source --> Kafka --> Streaming Engine --> Reports
We like to store different types of data in the same topic , same time data
should be accessed easily ..
here is sample data :
{"code" :"100" , "type" : "a" , "data" : " hello"}
{"code" :"100" , "type" : "b" , "data" : " hello"}
{"code" :"100" , "type" : "c" , "data" : " hello"}
This case we want to create topic called : *topic_100* and store all data
but the access pattern is using type.
Example :
1) Read only *Type : "a"* data
There is an Option to Partition the data using type so that all types goes
to same partition. The problem is the data is not distributed across
cluster.
What is the preferred approach to Use Same Topic but different types of
data ?
--Senthil
Re: Different Data Types under same topic
Posted by SenthilKumar K <se...@gmail.com>.
+ dev experts for inputs.
--Senthil
On Fri, Aug 18, 2017 at 9:15 PM, SenthilKumar K <se...@gmail.com>
wrote:
> Hi Users , We have planned to use Kafka for one of the use to collect data
> from different server and persist into Message Bus ..
>
> Flow Would Be :
> Source --> Kafka --> Streaming Engine --> Reports
>
> We like to store different types of data in the same topic , same time
> data should be accessed easily ..
>
> here is sample data :
> {"code" :"100" , "type" : "a" , "data" : " hello"}
> {"code" :"100" , "type" : "b" , "data" : " hello"}
> {"code" :"100" , "type" : "c" , "data" : " hello"}
>
> This case we want to create topic called : *topic_100* and store all
> data but the access pattern is using type.
>
> Example :
> 1) Read only *Type : "a"* data
>
>
> There is an Option to Partition the data using type so that all types goes
> to same partition. The problem is the data is not distributed across
> cluster.
>
> What is the preferred approach to Use Same Topic but different types of
> data ?
>
>
> --Senthil
>
Re: Different Data Types under same topic
Posted by SenthilKumar K <se...@gmail.com>.
+ dev experts for inputs.
--Senthil
On Fri, Aug 18, 2017 at 9:15 PM, SenthilKumar K <se...@gmail.com>
wrote:
> Hi Users , We have planned to use Kafka for one of the use to collect data
> from different server and persist into Message Bus ..
>
> Flow Would Be :
> Source --> Kafka --> Streaming Engine --> Reports
>
> We like to store different types of data in the same topic , same time
> data should be accessed easily ..
>
> here is sample data :
> {"code" :"100" , "type" : "a" , "data" : " hello"}
> {"code" :"100" , "type" : "b" , "data" : " hello"}
> {"code" :"100" , "type" : "c" , "data" : " hello"}
>
> This case we want to create topic called : *topic_100* and store all
> data but the access pattern is using type.
>
> Example :
> 1) Read only *Type : "a"* data
>
>
> There is an Option to Partition the data using type so that all types goes
> to same partition. The problem is the data is not distributed across
> cluster.
>
> What is the preferred approach to Use Same Topic but different types of
> data ?
>
>
> --Senthil
>