You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Mina Aslani <as...@gmail.com> on 2017/06/02 14:59:22 UTC
Data in kafka topic in Json format
Hi.
Is there any way that I get the data into a Kafka topic in Json format?
The source that I ingest the data from have the data in Json format,
however when I look that data in the kafka topic, schema and payload fields
are added and data is not in json format.
I want to avoid implementing a transformer to get the data from the topic
and publishes Json in another one.
Your input is appreciated.
Best regards,
Mina
Re: Data in kafka topic in Json format
Posted by Hans Jespersen <ha...@confluent.io>.
You have shared the Kafka connect properties but not the source connector config.
Which source connector are you using? Does it override the default settings you provided?
Are you running the connector in standalone mode or distributed mode?
Also what are you using to consume the messages and see the message format?
-hans
> On Jun 2, 2017, at 9:10 AM, Mina Aslani <as...@gmail.com> wrote:
>
> Hi Hans,
>
> Thank you for your quick response, appreciate it.
>
> In *kafka-connect* docker, I see below settings in
> *kafka-connect.properties* file in *kafka-connect *directory:
>
> key.converter.schemas.enable=false
> key.converter.schema.registry.url=http://kafka-schema-registry:<PORT>
> value.converter.schema.registry.url=http://kafka-schema-registry:<PORT>
>
> key.converter=org.apache.kafka.connect.json.JsonConverter
> value.converter=org.apache.kafka.connect.json.JsonConverter
>
> internal.value.converter.schemas.enable=false
> rest.advertised.host.name=kafka-connect
> internal.value.converter=org.apache.kafka.connect.json.JsonConverter
> internal.key.converter=org.apache.kafka.connect.json.JsonConverter
> And the settings in *schema-registry *directory of *kafka-connect *docker
> are as
>
> https://github.com/confluentinc/schema-registry/tree/master/config
>
> Should I consider any other settings for *kafka-connect* or
> *schema-registry* to get the real json object NOT string
> formatted/stringified json which has extra "\" and is not json any more?
>
> Best regards,
> Mina
>
> On Fri, Jun 2, 2017 at 11:18 AM, Hans Jespersen <ha...@confluent.io> wrote:
>
>>
>> My earlier comment still applies but in Kafka Connect the equivalent of a
>> serializer/deserializer (serdes) is called a “converter”.
>> Check which converter you have configured for your source connector and if
>> it is overriding whatever the default converter is configured for the
>> connect worker it is running in.
>>
>> -hans
>>
>>
>>
>>
>>> On Jun 2, 2017, at 8:12 AM, Mina Aslani <as...@gmail.com> wrote:
>>>
>>> Hi,
>>>
>>> I would like to add that I use kafka-connect and schema-registery
>> version `
>>> 3.2.1-6`.
>>>
>>> Best regards,
>>> Mina
>>>
>>> On Fri, Jun 2, 2017 at 10:59 AM, Mina Aslani <as...@gmail.com>
>> wrote:
>>>
>>>> Hi.
>>>>
>>>> Is there any way that I get the data into a Kafka topic in Json format?
>>>> The source that I ingest the data from have the data in Json format,
>>>> however when I look that data in the kafka topic, schema and payload
>> fields
>>>> are added and data is not in json format.
>>>>
>>>> I want to avoid implementing a transformer to get the data from the
>> topic
>>>> and publishes Json in another one.
>>>>
>>>> Your input is appreciated.
>>>>
>>>> Best regards,
>>>> Mina
>>>>
>>
>>
Re: Data in kafka topic in Json format
Posted by Mina Aslani <as...@gmail.com>.
Hi Hans,
Thank you for your quick response, appreciate it.
In *kafka-connect* docker, I see below settings in
*kafka-connect.properties* file in *kafka-connect *directory:
key.converter.schemas.enable=false
key.converter.schema.registry.url=http://kafka-schema-registry:<PORT>
value.converter.schema.registry.url=http://kafka-schema-registry:<PORT>
key.converter=org.apache.kafka.connect.json.JsonConverter
value.converter=org.apache.kafka.connect.json.JsonConverter
internal.value.converter.schemas.enable=false
rest.advertised.host.name=kafka-connect
internal.value.converter=org.apache.kafka.connect.json.JsonConverter
internal.key.converter=org.apache.kafka.connect.json.JsonConverter
And the settings in *schema-registry *directory of *kafka-connect *docker
are as
https://github.com/confluentinc/schema-registry/tree/master/config
Should I consider any other settings for *kafka-connect* or
*schema-registry* to get the real json object NOT string
formatted/stringified json which has extra "\" and is not json any more?
Best regards,
Mina
On Fri, Jun 2, 2017 at 11:18 AM, Hans Jespersen <ha...@confluent.io> wrote:
>
> My earlier comment still applies but in Kafka Connect the equivalent of a
> serializer/deserializer (serdes) is called a “converter”.
> Check which converter you have configured for your source connector and if
> it is overriding whatever the default converter is configured for the
> connect worker it is running in.
>
> -hans
>
>
>
>
> > On Jun 2, 2017, at 8:12 AM, Mina Aslani <as...@gmail.com> wrote:
> >
> > Hi,
> >
> > I would like to add that I use kafka-connect and schema-registery
> version `
> > 3.2.1-6`.
> >
> > Best regards,
> > Mina
> >
> > On Fri, Jun 2, 2017 at 10:59 AM, Mina Aslani <as...@gmail.com>
> wrote:
> >
> >> Hi.
> >>
> >> Is there any way that I get the data into a Kafka topic in Json format?
> >> The source that I ingest the data from have the data in Json format,
> >> however when I look that data in the kafka topic, schema and payload
> fields
> >> are added and data is not in json format.
> >>
> >> I want to avoid implementing a transformer to get the data from the
> topic
> >> and publishes Json in another one.
> >>
> >> Your input is appreciated.
> >>
> >> Best regards,
> >> Mina
> >>
>
>
Re: Data in kafka topic in Json format
Posted by Hans Jespersen <ha...@confluent.io>.
My earlier comment still applies but in Kafka Connect the equivalent of a serializer/deserializer (serdes) is called a “converter”.
Check which converter you have configured for your source connector and if it is overriding whatever the default converter is configured for the connect worker it is running in.
-hans
> On Jun 2, 2017, at 8:12 AM, Mina Aslani <as...@gmail.com> wrote:
>
> Hi,
>
> I would like to add that I use kafka-connect and schema-registery version `
> 3.2.1-6`.
>
> Best regards,
> Mina
>
> On Fri, Jun 2, 2017 at 10:59 AM, Mina Aslani <as...@gmail.com> wrote:
>
>> Hi.
>>
>> Is there any way that I get the data into a Kafka topic in Json format?
>> The source that I ingest the data from have the data in Json format,
>> however when I look that data in the kafka topic, schema and payload fields
>> are added and data is not in json format.
>>
>> I want to avoid implementing a transformer to get the data from the topic
>> and publishes Json in another one.
>>
>> Your input is appreciated.
>>
>> Best regards,
>> Mina
>>
Re: Data in kafka topic in Json format
Posted by Mina Aslani <as...@gmail.com>.
Hi,
I would like to add that I use kafka-connect and schema-registery version `
3.2.1-6`.
Best regards,
Mina
On Fri, Jun 2, 2017 at 10:59 AM, Mina Aslani <as...@gmail.com> wrote:
> Hi.
>
> Is there any way that I get the data into a Kafka topic in Json format?
> The source that I ingest the data from have the data in Json format,
> however when I look that data in the kafka topic, schema and payload fields
> are added and data is not in json format.
>
> I want to avoid implementing a transformer to get the data from the topic
> and publishes Json in another one.
>
> Your input is appreciated.
>
> Best regards,
> Mina
>
Re: Data in kafka topic in Json format
Posted by Hans Jespersen <ha...@confluent.io>.
Check which serializer you have configured in your producer. You are probably using an Avro serializer which will add the schema and modify the payload to avro data. You can use a String serializer or a ByteArray serializer and the data will either be Base64 encoded or not encoded at all.
-hans
> On Jun 2, 2017, at 7:59 AM, Mina Aslani <as...@gmail.com> wrote:
>
> Hi.
>
> Is there any way that I get the data into a Kafka topic in Json format?
> The source that I ingest the data from have the data in Json format,
> however when I look that data in the kafka topic, schema and payload fields
> are added and data is not in json format.
>
> I want to avoid implementing a transformer to get the data from the topic
> and publishes Json in another one.
>
> Your input is appreciated.
>
> Best regards,
> Mina