You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by so...@accenture.com on 2014/08/26 02:14:49 UTC

EBCDIC support

Hey all,

This might seem like a silly question, but does kafka have support for EBCDIC? Say I had to read data from an IBM mainframe via a TCP/IP socket where the data resides in EBCDIC format, can Kafka read that directly?

Thanks,
Sonali

________________________________

This message is for the designated recipient only and may contain privileged, proprietary, or otherwise confidential information. If you have received it in error, please notify the sender immediately and delete the original. Any other use of the e-mail by you is prohibited. Where allowed by local law, electronic communications with Accenture and its affiliates, including e-mail and instant messaging (including content), may be scanned by our systems for the purposes of information security and assessment of internal compliance with Accenture policy.
______________________________________________________________________________________

www.accenture.com

Re: EBCDIC support

Posted by Christian Csar <ca...@gmail.com>.
Having been spared any EBCDIC experience whatsoever (ie from a positio
of thorough ignorance), if you are transmitting text or things with a
designated textual form (presumably) I would recommend that your
conversion be to unicode rather than ascii if you don't already have
consumers expecting a given conversion. That way you will avoid losing
information, particularly if you expect any of your conversion tools to
be of more general use.

Christian

On 08/25/2014 05:36 PM, Gwen Shapira wrote:
> Personally, I like converting data before writing to Kafka, so I can
> easily support many consumers who don't know about EBCDIC.
> 
> A third option is to have a consumer that reads EBCDIC data from one
> Kafka topic and writes ASCII to another Kafka topic. This has the
> benefits of preserving the raw data in Kafka, in case you need it for
> troubleshooting, and also supporting non-EBCDIC consumers.
> 
> The cost is a more complex architecture, but if you already have a
> stream processing system around (Storm, Samza, Spark), it can be an
> easy addition.
> 
> 
> On Mon, Aug 25, 2014 at 5:28 PM,  <so...@accenture.com> wrote:
>> Thanks Gwen! makes sense. So I'll have to weigh the pros and cons of doing an EBCDIC to ASCII conversion before sending to Kafka Vs. using an ebcdic library after in the consumer
>>
>> Thanks!
>> S
>>
>> -----Original Message-----
>> From: Gwen Shapira [mailto:gshapira@cloudera.com]
>> Sent: Monday, August 25, 2014 5:22 PM
>> To: users@kafka.apache.org
>> Subject: Re: EBCDIC support
>>
>> Hi Sonali,
>>
>> Kafka doesn't really care about EBCDIC or any other format -  for Kafka bits are just bits. So they are all supported.
>>
>> Kafka does not "read" data from a socket though. Well, it does, but the data has to be sent by a Kafka producer. Most likely you'll need to implement a producer that will get the data from the socket and send it as a message to Kafka. The content of the message can be anything, including EBCDIC -.
>>
>> Then  you'll need a consumer to read the data from Kafka and do something with this - the consumer will need to know what to do with a message that contains EBCDIC data. Perhaps you have EBCDIC libraries you can reuse there.
>>
>> Hope this helps.
>>
>> Gwen
>>
>> On Mon, Aug 25, 2014 at 5:14 PM,  <so...@accenture.com> wrote:
>>> Hey all,
>>>
>>> This might seem like a silly question, but does kafka have support for EBCDIC? Say I had to read data from an IBM mainframe via a TCP/IP socket where the data resides in EBCDIC format, can Kafka read that directly?
>>>
>>> Thanks,
>>> Sonali
>>>
>>> ________________________________
>>>
>>> This message is for the designated recipient only and may contain privileged, proprietary, or otherwise confidential information. If you have received it in error, please notify the sender immediately and delete the original. Any other use of the e-mail by you is prohibited. Where allowed by local law, electronic communications with Accenture and its affiliates, including e-mail and instant messaging (including content), may be scanned by our systems for the purposes of information security and assessment of internal compliance with Accenture policy.
>>> ______________________________________________________________________
>>> ________________
>>>
>>> www.accenture.com



Re: EBCDIC support

Posted by Robert Hodges <be...@gmail.com>.
Hi Gwen,

I like the approach of converting to general forms in most cases, which is
an early bound design.

You can also take a late bound approach of leaving data in its original
form but adding metadata to enable translation if needed at a later time.
 This is necessary if you have homogeneous consumers downstream and the
translation is "lossy" in any sense. Just as one example floating point
numbers can lose precision on conversion.

Cheers, Robert


On Mon, Aug 25, 2014 at 5:36 PM, Gwen Shapira <gs...@cloudera.com> wrote:

> Personally, I like converting data before writing to Kafka, so I can
> easily support many consumers who don't know about EBCDIC.
>
> A third option is to have a consumer that reads EBCDIC data from one
> Kafka topic and writes ASCII to another Kafka topic. This has the
> benefits of preserving the raw data in Kafka, in case you need it for
> troubleshooting, and also supporting non-EBCDIC consumers.
>
> The cost is a more complex architecture, but if you already have a
> stream processing system around (Storm, Samza, Spark), it can be an
> easy addition.
>
>
> On Mon, Aug 25, 2014 at 5:28 PM,  <so...@accenture.com>
> wrote:
> > Thanks Gwen! makes sense. So I'll have to weigh the pros and cons of
> doing an EBCDIC to ASCII conversion before sending to Kafka Vs. using an
> ebcdic library after in the consumer
> >
> > Thanks!
> > S
> >
> > -----Original Message-----
> > From: Gwen Shapira [mailto:gshapira@cloudera.com]
> > Sent: Monday, August 25, 2014 5:22 PM
> > To: users@kafka.apache.org
> > Subject: Re: EBCDIC support
> >
> > Hi Sonali,
> >
> > Kafka doesn't really care about EBCDIC or any other format -  for Kafka
> bits are just bits. So they are all supported.
> >
> > Kafka does not "read" data from a socket though. Well, it does, but the
> data has to be sent by a Kafka producer. Most likely you'll need to
> implement a producer that will get the data from the socket and send it as
> a message to Kafka. The content of the message can be anything, including
> EBCDIC -.
> >
> > Then  you'll need a consumer to read the data from Kafka and do
> something with this - the consumer will need to know what to do with a
> message that contains EBCDIC data. Perhaps you have EBCDIC libraries you
> can reuse there.
> >
> > Hope this helps.
> >
> > Gwen
> >
> > On Mon, Aug 25, 2014 at 5:14 PM,  <so...@accenture.com>
> wrote:
> >> Hey all,
> >>
> >> This might seem like a silly question, but does kafka have support for
> EBCDIC? Say I had to read data from an IBM mainframe via a TCP/IP socket
> where the data resides in EBCDIC format, can Kafka read that directly?
> >>
> >> Thanks,
> >> Sonali
> >>
> >> ________________________________
> >>
> >> This message is for the designated recipient only and may contain
> privileged, proprietary, or otherwise confidential information. If you have
> received it in error, please notify the sender immediately and delete the
> original. Any other use of the e-mail by you is prohibited. Where allowed
> by local law, electronic communications with Accenture and its affiliates,
> including e-mail and instant messaging (including content), may be scanned
> by our systems for the purposes of information security and assessment of
> internal compliance with Accenture policy.
> >> ______________________________________________________________________
> >> ________________
> >>
> >> www.accenture.com
>

Re: EBCDIC support

Posted by Gwen Shapira <gs...@cloudera.com>.
Personally, I like converting data before writing to Kafka, so I can
easily support many consumers who don't know about EBCDIC.

A third option is to have a consumer that reads EBCDIC data from one
Kafka topic and writes ASCII to another Kafka topic. This has the
benefits of preserving the raw data in Kafka, in case you need it for
troubleshooting, and also supporting non-EBCDIC consumers.

The cost is a more complex architecture, but if you already have a
stream processing system around (Storm, Samza, Spark), it can be an
easy addition.


On Mon, Aug 25, 2014 at 5:28 PM,  <so...@accenture.com> wrote:
> Thanks Gwen! makes sense. So I'll have to weigh the pros and cons of doing an EBCDIC to ASCII conversion before sending to Kafka Vs. using an ebcdic library after in the consumer
>
> Thanks!
> S
>
> -----Original Message-----
> From: Gwen Shapira [mailto:gshapira@cloudera.com]
> Sent: Monday, August 25, 2014 5:22 PM
> To: users@kafka.apache.org
> Subject: Re: EBCDIC support
>
> Hi Sonali,
>
> Kafka doesn't really care about EBCDIC or any other format -  for Kafka bits are just bits. So they are all supported.
>
> Kafka does not "read" data from a socket though. Well, it does, but the data has to be sent by a Kafka producer. Most likely you'll need to implement a producer that will get the data from the socket and send it as a message to Kafka. The content of the message can be anything, including EBCDIC -.
>
> Then  you'll need a consumer to read the data from Kafka and do something with this - the consumer will need to know what to do with a message that contains EBCDIC data. Perhaps you have EBCDIC libraries you can reuse there.
>
> Hope this helps.
>
> Gwen
>
> On Mon, Aug 25, 2014 at 5:14 PM,  <so...@accenture.com> wrote:
>> Hey all,
>>
>> This might seem like a silly question, but does kafka have support for EBCDIC? Say I had to read data from an IBM mainframe via a TCP/IP socket where the data resides in EBCDIC format, can Kafka read that directly?
>>
>> Thanks,
>> Sonali
>>
>> ________________________________
>>
>> This message is for the designated recipient only and may contain privileged, proprietary, or otherwise confidential information. If you have received it in error, please notify the sender immediately and delete the original. Any other use of the e-mail by you is prohibited. Where allowed by local law, electronic communications with Accenture and its affiliates, including e-mail and instant messaging (including content), may be scanned by our systems for the purposes of information security and assessment of internal compliance with Accenture policy.
>> ______________________________________________________________________
>> ________________
>>
>> www.accenture.com

RE: EBCDIC support

Posted by so...@accenture.com.
Thanks Gwen! makes sense. So I'll have to weigh the pros and cons of doing an EBCDIC to ASCII conversion before sending to Kafka Vs. using an ebcdic library after in the consumer

Thanks!
S

-----Original Message-----
From: Gwen Shapira [mailto:gshapira@cloudera.com] 
Sent: Monday, August 25, 2014 5:22 PM
To: users@kafka.apache.org
Subject: Re: EBCDIC support

Hi Sonali,

Kafka doesn't really care about EBCDIC or any other format -  for Kafka bits are just bits. So they are all supported.

Kafka does not "read" data from a socket though. Well, it does, but the data has to be sent by a Kafka producer. Most likely you'll need to implement a producer that will get the data from the socket and send it as a message to Kafka. The content of the message can be anything, including EBCDIC -.

Then  you'll need a consumer to read the data from Kafka and do something with this - the consumer will need to know what to do with a message that contains EBCDIC data. Perhaps you have EBCDIC libraries you can reuse there.

Hope this helps.

Gwen

On Mon, Aug 25, 2014 at 5:14 PM,  <so...@accenture.com> wrote:
> Hey all,
>
> This might seem like a silly question, but does kafka have support for EBCDIC? Say I had to read data from an IBM mainframe via a TCP/IP socket where the data resides in EBCDIC format, can Kafka read that directly?
>
> Thanks,
> Sonali
>
> ________________________________
>
> This message is for the designated recipient only and may contain privileged, proprietary, or otherwise confidential information. If you have received it in error, please notify the sender immediately and delete the original. Any other use of the e-mail by you is prohibited. Where allowed by local law, electronic communications with Accenture and its affiliates, including e-mail and instant messaging (including content), may be scanned by our systems for the purposes of information security and assessment of internal compliance with Accenture policy.
> ______________________________________________________________________
> ________________
>
> www.accenture.com

Re: EBCDIC support

Posted by Gwen Shapira <gs...@cloudera.com>.
Hi Sonali,

Kafka doesn't really care about EBCDIC or any other format -  for
Kafka bits are just bits. So they are all supported.

Kafka does not "read" data from a socket though. Well, it does, but
the data has to be sent by a Kafka producer. Most likely you'll need
to implement a producer that will get the data from the socket and
send it as a message to Kafka. The content of the message can be
anything, including EBCDIC -.

Then  you'll need a consumer to read the data from Kafka and do
something with this - the consumer will need to know what to do with a
message that contains EBCDIC data. Perhaps you have EBCDIC libraries
you can reuse there.

Hope this helps.

Gwen

On Mon, Aug 25, 2014 at 5:14 PM,  <so...@accenture.com> wrote:
> Hey all,
>
> This might seem like a silly question, but does kafka have support for EBCDIC? Say I had to read data from an IBM mainframe via a TCP/IP socket where the data resides in EBCDIC format, can Kafka read that directly?
>
> Thanks,
> Sonali
>
> ________________________________
>
> This message is for the designated recipient only and may contain privileged, proprietary, or otherwise confidential information. If you have received it in error, please notify the sender immediately and delete the original. Any other use of the e-mail by you is prohibited. Where allowed by local law, electronic communications with Accenture and its affiliates, including e-mail and instant messaging (including content), may be scanned by our systems for the purposes of information security and assessment of internal compliance with Accenture policy.
> ______________________________________________________________________________________
>
> www.accenture.com