You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Swapnil Chougule <th...@gmail.com> on 2016/09/05 13:50:00 UTC
RawSchema as deserialization schema
I am using Kafka consumer in flink 1.1.1 with Kafka 0.8.2. I want to read
byte array as it is in datastream. I tried to use RawSchema as
desrialization schema but couldn't find same 1.1.1.
I want to know whether I have to write my custom implementation for same ?
Can somebody help me to sort out same ?
Also passing byte[] to next operator is supported as far as serialization
is concerned ?
Re: RawSchema as deserialization schema
Posted by Swapnil Chougule <th...@gmail.com>.
Okay. Thanks for the update Robert.
On Mon, Sep 26, 2016 at 3:08 PM, Robert Metzger <rm...@apache.org> wrote:
> The RawSchema was once part of Flink's Kafka connector. I've removed it
> because its implementation is trivial and I didn't expect that there are
> many people who need the schema (also, I think I saw people using a map()
> operator after the consumer to deserialize the byte[] into their formats).
>
> As Stephan mentioned, I'm going to add some tooling so that people can
> directly use Kafka's Serialization classes, and they provide a
> "ByteArraySerializer" which is doing exactly the same as the "RawSchema"
>
>
> On Mon, Sep 26, 2016 at 11:27 AM, Swapnil Chougule <
> the.swapnil09@gmail.com> wrote:
>
>> Hi Robert, May I know your inputs on same ?
>>
>> Thanks,
>> Swapnil
>>
>> On Thu, Sep 22, 2016 at 7:12 PM, Stephan Ewen <se...@apache.org> wrote:
>>
>>> /cc Robert, he is looking into extending the Kafka Connectors to support
>>> more of Kafka's direct utilities
>>>
>>> On Thu, Sep 22, 2016 at 3:17 PM, Swapnil Chougule <
>>> the.swapnil09@gmail.com> wrote:
>>>
>>>> It will be good to have RawSchema as one of the deserialization schema
>>>> in streaming framework (like SimpleStringSchema).
>>>> Many use cases needs data in byte array format after reading from
>>>> source like kafka.
>>>>
>>>> Any inputs for same ?
>>>>
>>>> On Mon, Sep 12, 2016 at 11:42 AM, Swapnil Chougule <
>>>> the.swapnil09@gmail.com> wrote:
>>>>
>>>>> Thanks Maximilian. I implemented same & it worked for me. I was under
>>>>> impression that RawSchema is available from flink.
>>>>>
>>>>> Regards,
>>>>> Swapnil
>>>>>
>>>>> On Mon, Sep 5, 2016 at 8:48 PM, Maximilian Michels <mx...@apache.org>
>>>>> wrote:
>>>>>
>>>>>> Just implement DeserializationSchema and return the byte array from
>>>>>> Kafka. Byte array serialization poses no problem to the Flink
>>>>>> serialization.
>>>>>>
>>>>>> On Mon, Sep 5, 2016 at 3:50 PM, Swapnil Chougule
>>>>>> <th...@gmail.com> wrote:
>>>>>> > I am using Kafka consumer in flink 1.1.1 with Kafka 0.8.2. I want
>>>>>> to read
>>>>>> > byte array as it is in datastream. I tried to use RawSchema as
>>>>>> > desrialization schema but couldn't find same 1.1.1.
>>>>>> > I want to know whether I have to write my custom implementation
>>>>>> for same ?
>>>>>> > Can somebody help me to sort out same ?
>>>>>> >
>>>>>> > Also passing byte[] to next operator is supported as far as
>>>>>> serialization is
>>>>>> > concerned ?
>>>>>>
>>>>>
>>>>>
>>>>
>>>
>>
>
Re: RawSchema as deserialization schema
Posted by Robert Metzger <rm...@apache.org>.
The RawSchema was once part of Flink's Kafka connector. I've removed it
because its implementation is trivial and I didn't expect that there are
many people who need the schema (also, I think I saw people using a map()
operator after the consumer to deserialize the byte[] into their formats).
As Stephan mentioned, I'm going to add some tooling so that people can
directly use Kafka's Serialization classes, and they provide a
"ByteArraySerializer" which is doing exactly the same as the "RawSchema"
On Mon, Sep 26, 2016 at 11:27 AM, Swapnil Chougule <th...@gmail.com>
wrote:
> Hi Robert, May I know your inputs on same ?
>
> Thanks,
> Swapnil
>
> On Thu, Sep 22, 2016 at 7:12 PM, Stephan Ewen <se...@apache.org> wrote:
>
>> /cc Robert, he is looking into extending the Kafka Connectors to support
>> more of Kafka's direct utilities
>>
>> On Thu, Sep 22, 2016 at 3:17 PM, Swapnil Chougule <
>> the.swapnil09@gmail.com> wrote:
>>
>>> It will be good to have RawSchema as one of the deserialization schema
>>> in streaming framework (like SimpleStringSchema).
>>> Many use cases needs data in byte array format after reading from source
>>> like kafka.
>>>
>>> Any inputs for same ?
>>>
>>> On Mon, Sep 12, 2016 at 11:42 AM, Swapnil Chougule <
>>> the.swapnil09@gmail.com> wrote:
>>>
>>>> Thanks Maximilian. I implemented same & it worked for me. I was under
>>>> impression that RawSchema is available from flink.
>>>>
>>>> Regards,
>>>> Swapnil
>>>>
>>>> On Mon, Sep 5, 2016 at 8:48 PM, Maximilian Michels <mx...@apache.org>
>>>> wrote:
>>>>
>>>>> Just implement DeserializationSchema and return the byte array from
>>>>> Kafka. Byte array serialization poses no problem to the Flink
>>>>> serialization.
>>>>>
>>>>> On Mon, Sep 5, 2016 at 3:50 PM, Swapnil Chougule
>>>>> <th...@gmail.com> wrote:
>>>>> > I am using Kafka consumer in flink 1.1.1 with Kafka 0.8.2. I want to
>>>>> read
>>>>> > byte array as it is in datastream. I tried to use RawSchema as
>>>>> > desrialization schema but couldn't find same 1.1.1.
>>>>> > I want to know whether I have to write my custom implementation for
>>>>> same ?
>>>>> > Can somebody help me to sort out same ?
>>>>> >
>>>>> > Also passing byte[] to next operator is supported as far as
>>>>> serialization is
>>>>> > concerned ?
>>>>>
>>>>
>>>>
>>>
>>
>
Re: RawSchema as deserialization schema
Posted by Swapnil Chougule <th...@gmail.com>.
Hi Robert, May I know your inputs on same ?
Thanks,
Swapnil
On Thu, Sep 22, 2016 at 7:12 PM, Stephan Ewen <se...@apache.org> wrote:
> /cc Robert, he is looking into extending the Kafka Connectors to support
> more of Kafka's direct utilities
>
> On Thu, Sep 22, 2016 at 3:17 PM, Swapnil Chougule <the.swapnil09@gmail.com
> > wrote:
>
>> It will be good to have RawSchema as one of the deserialization schema in
>> streaming framework (like SimpleStringSchema).
>> Many use cases needs data in byte array format after reading from source
>> like kafka.
>>
>> Any inputs for same ?
>>
>> On Mon, Sep 12, 2016 at 11:42 AM, Swapnil Chougule <
>> the.swapnil09@gmail.com> wrote:
>>
>>> Thanks Maximilian. I implemented same & it worked for me. I was under
>>> impression that RawSchema is available from flink.
>>>
>>> Regards,
>>> Swapnil
>>>
>>> On Mon, Sep 5, 2016 at 8:48 PM, Maximilian Michels <mx...@apache.org>
>>> wrote:
>>>
>>>> Just implement DeserializationSchema and return the byte array from
>>>> Kafka. Byte array serialization poses no problem to the Flink
>>>> serialization.
>>>>
>>>> On Mon, Sep 5, 2016 at 3:50 PM, Swapnil Chougule
>>>> <th...@gmail.com> wrote:
>>>> > I am using Kafka consumer in flink 1.1.1 with Kafka 0.8.2. I want to
>>>> read
>>>> > byte array as it is in datastream. I tried to use RawSchema as
>>>> > desrialization schema but couldn't find same 1.1.1.
>>>> > I want to know whether I have to write my custom implementation for
>>>> same ?
>>>> > Can somebody help me to sort out same ?
>>>> >
>>>> > Also passing byte[] to next operator is supported as far as
>>>> serialization is
>>>> > concerned ?
>>>>
>>>
>>>
>>
>
Re: RawSchema as deserialization schema
Posted by Stephan Ewen <se...@apache.org>.
/cc Robert, he is looking into extending the Kafka Connectors to support
more of Kafka's direct utilities
On Thu, Sep 22, 2016 at 3:17 PM, Swapnil Chougule <th...@gmail.com>
wrote:
> It will be good to have RawSchema as one of the deserialization schema in
> streaming framework (like SimpleStringSchema).
> Many use cases needs data in byte array format after reading from source
> like kafka.
>
> Any inputs for same ?
>
> On Mon, Sep 12, 2016 at 11:42 AM, Swapnil Chougule <
> the.swapnil09@gmail.com> wrote:
>
>> Thanks Maximilian. I implemented same & it worked for me. I was under
>> impression that RawSchema is available from flink.
>>
>> Regards,
>> Swapnil
>>
>> On Mon, Sep 5, 2016 at 8:48 PM, Maximilian Michels <mx...@apache.org>
>> wrote:
>>
>>> Just implement DeserializationSchema and return the byte array from
>>> Kafka. Byte array serialization poses no problem to the Flink
>>> serialization.
>>>
>>> On Mon, Sep 5, 2016 at 3:50 PM, Swapnil Chougule
>>> <th...@gmail.com> wrote:
>>> > I am using Kafka consumer in flink 1.1.1 with Kafka 0.8.2. I want to
>>> read
>>> > byte array as it is in datastream. I tried to use RawSchema as
>>> > desrialization schema but couldn't find same 1.1.1.
>>> > I want to know whether I have to write my custom implementation for
>>> same ?
>>> > Can somebody help me to sort out same ?
>>> >
>>> > Also passing byte[] to next operator is supported as far as
>>> serialization is
>>> > concerned ?
>>>
>>
>>
>
Re: RawSchema as deserialization schema
Posted by Swapnil Chougule <th...@gmail.com>.
It will be good to have RawSchema as one of the deserialization schema in
streaming framework (like SimpleStringSchema).
Many use cases needs data in byte array format after reading from source
like kafka.
Any inputs for same ?
On Mon, Sep 12, 2016 at 11:42 AM, Swapnil Chougule <th...@gmail.com>
wrote:
> Thanks Maximilian. I implemented same & it worked for me. I was under
> impression that RawSchema is available from flink.
>
> Regards,
> Swapnil
>
> On Mon, Sep 5, 2016 at 8:48 PM, Maximilian Michels <mx...@apache.org> wrote:
>
>> Just implement DeserializationSchema and return the byte array from
>> Kafka. Byte array serialization poses no problem to the Flink
>> serialization.
>>
>> On Mon, Sep 5, 2016 at 3:50 PM, Swapnil Chougule
>> <th...@gmail.com> wrote:
>> > I am using Kafka consumer in flink 1.1.1 with Kafka 0.8.2. I want to
>> read
>> > byte array as it is in datastream. I tried to use RawSchema as
>> > desrialization schema but couldn't find same 1.1.1.
>> > I want to know whether I have to write my custom implementation for
>> same ?
>> > Can somebody help me to sort out same ?
>> >
>> > Also passing byte[] to next operator is supported as far as
>> serialization is
>> > concerned ?
>>
>
>
Re: RawSchema as deserialization schema
Posted by Swapnil Chougule <th...@gmail.com>.
Thanks Maximilian. I implemented same & it worked for me. I was under
impression that RawSchema is available from flink.
Regards,
Swapnil
On Mon, Sep 5, 2016 at 8:48 PM, Maximilian Michels <mx...@apache.org> wrote:
> Just implement DeserializationSchema and return the byte array from
> Kafka. Byte array serialization poses no problem to the Flink
> serialization.
>
> On Mon, Sep 5, 2016 at 3:50 PM, Swapnil Chougule
> <th...@gmail.com> wrote:
> > I am using Kafka consumer in flink 1.1.1 with Kafka 0.8.2. I want to read
> > byte array as it is in datastream. I tried to use RawSchema as
> > desrialization schema but couldn't find same 1.1.1.
> > I want to know whether I have to write my custom implementation for
> same ?
> > Can somebody help me to sort out same ?
> >
> > Also passing byte[] to next operator is supported as far as
> serialization is
> > concerned ?
>
Re: RawSchema as deserialization schema
Posted by Maximilian Michels <mx...@apache.org>.
Just implement DeserializationSchema and return the byte array from
Kafka. Byte array serialization poses no problem to the Flink
serialization.
On Mon, Sep 5, 2016 at 3:50 PM, Swapnil Chougule
<th...@gmail.com> wrote:
> I am using Kafka consumer in flink 1.1.1 with Kafka 0.8.2. I want to read
> byte array as it is in datastream. I tried to use RawSchema as
> desrialization schema but couldn't find same 1.1.1.
> I want to know whether I have to write my custom implementation for same ?
> Can somebody help me to sort out same ?
>
> Also passing byte[] to next operator is supported as far as serialization is
> concerned ?