You are viewing a plain text version of this content. The canonical link for it is here.

Posted to users@kafka.apache.org by "Hopson, Stephen" <st...@gb.unisys.com> on 2016/03/03 11:47:37 UTC

Writing a Producer from Scratch

Hi,
Not sure if this is the right forum for this question, but if it not I'm sure someone will direct me to the proper one.
Also, I am new to Kafka (but not new to computers).

I want to write a kafka producer client for a Unisys OS 2200 mainframe. I need to write it in C, and since I have no access to Windows / Unix / Linux libraries, I have to develop the interface at the lowest level.

So far, I have downloaded a kafka server with associated zookeeper (kafka _2.10-0.8.2.2). Note I have downloaded the Windows version and have it running on my laptop, successfully tested on the same laptop with the provided provider and consumer clients.

I have developed code to open a TCP session to the kafka server which appears to work and I have attempted to send a metadata request which does not appear to work. When I say it does not appear to work, I mean that I send the message and then I sit on a retrieve, which eventually times out ( I do seem to get one character in the receive buffer of 0235 octal). The message format I am using is the one described by the excellent document by Jay Creps / Gwen Shapira at https://cwiki.apache.org/confluence/display/KAFKA/A+Guide+To+The+Kafka+Protocol However, it is not clear what level of kafka these message formats are applicable for.

Can anybody offer me any advice or suggestions as to how to progress?

PS is the CRC mandatory in the Producer messages?
Many thanks in advance.

[unisys_logo]<http://www.unisys.com/>

THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all devices.

RE: Writing a Producer from Scratch

Posted by "Hopson, Stephen" <st...@gb.unisys.com>.

Thanks James.
I will repost my question on dev mailing list and checkout the forum you mention. HTTP is a possibility but not the preferred option.

Cheers.
Steve

Stephen Hopson | Infrastructure Architect | Enterprise Solutions
Unisys | +44 (0)1908 805010 | +44 (0)7557 303321 | stephen.hopson@gb.unisys.com 



THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all devices.

-----Original Message-----
From: James Cheng [mailto:jcheng@tivo.com] 
Sent: 04 March 2016 06:55
To: users@kafka.apache.org
Subject: Re: Writing a Producer from Scratch

Stephen,

There is a mailing list for kafka client developers that you may find useful: https://groups.google.com/forum/#!forum/kafka-clients

The dev@kafka.apache.org<ma...@kafka.apache.org> mailing list might also be a good resource: http://kafka.apache.org/contact.html

Lastly, do you have any way to do HTTP calls on your platform? There exist some REST servers that you speak HTTP to and then they will produce to Kafka on your behalf. Here is one: http://docs.confluent.io/2.0.1/kafka-rest/docs/index.html

-James

On Mar 3, 2016, at 2:47 AM, Hopson, Stephen <st...@gb.unisys.com>> wrote:

Hi,
Not sure if this is the right forum for this question, but if it not I’m sure someone will direct me to the proper one.
Also, I am new to Kafka (but not new to computers).

I want to write a kafka producer client for a Unisys OS 2200 mainframe. I need to write it in C, and since I have no access to Windows / Unix / Linux libraries, I have to develop the interface at the lowest level.

So far, I have downloaded a kafka server with associated zookeeper (kafka _2.10-0.8.2.2). Note I have downloaded the Windows version and have it running on my laptop, successfully tested on the same laptop with the provided provider and consumer clients.

I have developed code to open a TCP session to the kafka server which appears to work and I have attempted to send a metadata request which does not appear to work. When I say it does not appear to work, I mean that I send the message and then I sit on a retrieve, which eventually times out ( I do seem to get one character in the receive buffer of 0235 octal). The message format I am using is the one described by the excellent document by Jay Creps / Gwen Shapira athttps://cwiki.apache.org/confluence/display/KAFKA/A+Guide+To+The+Kafka+Protocol  However, it is not clear what level of kafka these message formats are applicable for.

Can anybody offer me any advice or suggestions as to how to progress?

PS is the CRC mandatory in the Producer messages?
Many thanks in advance.

Stephen Hopson | Infrastructure Architect | Enterprise Solutions
Unisys | +44 (0)1908 805010 | +44 (0)7557 303321 | stephen.hopson@gb.unisys.com<ma...@gb.unisys.com>

<http://www.unisys.com/>

THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all devices.


________________________________

This email and any attachments may contain confidential and privileged material for the sole use of the intended recipient. Any review, copying, or distribution of this email (or any attachments) by others is prohibited. If you are not the intended recipient, please contact the sender immediately and permanently delete this email and any attachments. No employee or agent of TiVo Inc. is authorized to conclude any binding agreement on behalf of TiVo Inc. by email. Binding agreements with TiVo Inc. may only be made by a signed written agreement.

Re: Writing a Producer from Scratch

Posted by James Cheng <jc...@tivo.com>.

Stephen,

There is a mailing list for kafka client developers that you may find useful: https://groups.google.com/forum/#!forum/kafka-clients

The dev@kafka.apache.org<ma...@kafka.apache.org> mailing list might also be a good resource: http://kafka.apache.org/contact.html

Lastly, do you have any way to do HTTP calls on your platform? There exist some REST servers that you speak HTTP to and then they will produce to Kafka on your behalf. Here is one: http://docs.confluent.io/2.0.1/kafka-rest/docs/index.html

-James

On Mar 3, 2016, at 2:47 AM, Hopson, Stephen <st...@gb.unisys.com>> wrote:

Hi,
Not sure if this is the right forum for this question, but if it not I’m sure someone will direct me to the proper one.
Also, I am new to Kafka (but not new to computers).

I have developed code to open a TCP session to the kafka server which appears to work and I have attempted to send a metadata request which does not appear to work. When I say it does not appear to work, I mean that I send the message and then I sit on a retrieve, which eventually times out ( I do seem to get one character in the receive buffer of 0235 octal). The message format I am using is the one described by the excellent document by Jay Creps / Gwen Shapira athttps://cwiki.apache.org/confluence/display/KAFKA/A+Guide+To+The+Kafka+Protocol However, it is not clear what level of kafka these message formats are applicable for.

Can anybody offer me any advice or suggestions as to how to progress?

PS is the CRC mandatory in the Producer messages?
Many thanks in advance.

<http://www.unisys.com/>

________________________________

This email and any attachments may contain confidential and privileged material for the sole use of the intended recipient. Any review, copying, or distribution of this email (or any attachments) by others is prohibited. If you are not the intended recipient, please contact the sender immediately and permanently delete this email and any attachments. No employee or agent of TiVo Inc. is authorized to conclude any binding agreement on behalf of TiVo Inc. by email. Binding agreements with TiVo Inc. may only be made by a signed written agreement.

RE: Writing a Producer from Scratch

Posted by "Hopson, Stephen" <st...@gb.unisys.com>.

Hi Ben,
 Yes I have looked at that code. It relies heavily on either Windows or Linux / Unix libraries and even the C libraries are not compatible with the C compiler on my system (my mainframe is 36 bit WORD based) so there are a number of issues with sizes that I am aware of and have to deal with.

Thanks for the suggestion

Steve

Stephen Hopson | Infrastructure Architect | Enterprise Solutions
Unisys | +44 (0)1908 805010 | +44 (0)7557 303321 | stephen.hopson@gb.unisys.com 

THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all devices.

-----Original Message-----
From: Ben Davison [mailto:ben.davison@7digital.com] 
Sent: 03 March 2016 16:19
To: users@kafka.apache.org
Subject: Re: Writing a Producer from Scratch

Hi Stephen,

Have you checked out https://github.com/edenhill/librdkafka ? It might be
what you need (I don't do C, so it might not be right for you)

Regards,

Ben

On Thu, Mar 3, 2016 at 10:47 AM, Hopson, Stephen <
stephen.hopson@gb.unisys.com> wrote:

> Hi,
>
> Not sure if this is the right forum for this question, but if it not I’m
> sure someone will direct me to the proper one.
>
> Also, I am new to Kafka (but not new to computers).
>
>
>
> I want to write a kafka producer client for a Unisys OS 2200 mainframe. I
> need to write it in C, and since I have no access to Windows / Unix / Linux
> libraries, I have to develop the interface at the lowest level.
>
>
>
> So far, I have downloaded a kafka server with associated zookeeper (kafka
> _2.10-0.8.2.2). Note I have downloaded the Windows version and have it
> running on my laptop, successfully tested on the same laptop with the
> provided provider and consumer clients.
>
>
>
> I have developed code to open a TCP session to the kafka server which
> appears to work and I have attempted to send a metadata request which does
> not appear to work. When I say it does not appear to work, I mean that I
> send the message and then I sit on a retrieve, which eventually times out (
> I do seem to get one character in the receive buffer of 0235 octal). The
> message format I am using is the one described by the excellent document by
> Jay Creps / Gwen Shapira at
> https://cwiki.apache.org/confluence/display/KAFKA/A+Guide+To+The+Kafka+Protocol
> However, it is not clear what level of kafka these message formats are
> applicable for.
>
>
>
> Can anybody offer me any advice or suggestions as to how to progress?
>
>
>
> PS is the CRC mandatory in the Producer messages?
>
> Many thanks in advance.
>
>
>
> *Stephen Hopson* | Infrastructure Architect | Enterprise Solutions
>
> Unisys | +44 (0)1908 805010 | +44 (0)7557 303321 |
> stephen.hopson@gb.unisys.com
>
>
>
> [image: unisys_logo] <http://www.unisys.com/>
>
>
>
> THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY
> MATERIAL and is for use only by the intended recipient. If you received
> this in error, please contact the sender and delete the e-mail and its
> attachments from all devices.
>
>
>

-- 

This email, including attachments, is private and confidential. If you have 
received this email in error please notify the sender and delete it from 
your system. Emails are not secure and may contain viruses. No liability 
can be accepted for viruses that might be transferred by this email or any 
attachment. Any unauthorised copying of this message or unauthorised 
distribution and publication of the information contained herein are 
prohibited.

7digital Limited. Registered office: 69 Wilson Street, London EC2A 2BB.
Registered in England and Wales. Registered No. 04843573.

Re: Writing a Producer from Scratch

Posted by Ben Davison <be...@7digital.com>.

Hi Stephen,

Have you checked out https://github.com/edenhill/librdkafka ? It might be
what you need (I don't do C, so it might not be right for you)

Regards,

Ben

On Thu, Mar 3, 2016 at 10:47 AM, Hopson, Stephen <
stephen.hopson@gb.unisys.com> wrote:

> Hi,
>
> Not sure if this is the right forum for this question, but if it not I’m
> sure someone will direct me to the proper one.
>
> Also, I am new to Kafka (but not new to computers).
>
>
>
> I want to write a kafka producer client for a Unisys OS 2200 mainframe. I
> need to write it in C, and since I have no access to Windows / Unix / Linux
> libraries, I have to develop the interface at the lowest level.
>
>
>
> So far, I have downloaded a kafka server with associated zookeeper (kafka
> _2.10-0.8.2.2). Note I have downloaded the Windows version and have it
> running on my laptop, successfully tested on the same laptop with the
> provided provider and consumer clients.
>
>
>
> I have developed code to open a TCP session to the kafka server which
> appears to work and I have attempted to send a metadata request which does
> not appear to work. When I say it does not appear to work, I mean that I
> send the message and then I sit on a retrieve, which eventually times out (
> I do seem to get one character in the receive buffer of 0235 octal). The
> message format I am using is the one described by the excellent document by
> Jay Creps / Gwen Shapira at
> https://cwiki.apache.org/confluence/display/KAFKA/A+Guide+To+The+Kafka+Protocol
> However, it is not clear what level of kafka these message formats are
> applicable for.
>
>
>
> Can anybody offer me any advice or suggestions as to how to progress?
>
>
>
> PS is the CRC mandatory in the Producer messages?
>
> Many thanks in advance.
>
>
>
> *Stephen Hopson* | Infrastructure Architect | Enterprise Solutions
>
> Unisys | +44 (0)1908 805010 | +44 (0)7557 303321 |
> stephen.hopson@gb.unisys.com
>
>
>
> [image: unisys_logo] <http://www.unisys.com/>
>
>
>
> THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY
> MATERIAL and is for use only by the intended recipient. If you received
> this in error, please contact the sender and delete the e-mail and its
> attachments from all devices.
>
>
>

-- 


This email, including attachments, is private and confidential. If you have 
received this email in error please notify the sender and delete it from 
your system. Emails are not secure and may contain viruses. No liability 
can be accepted for viruses that might be transferred by this email or any 
attachment. Any unauthorised copying of this message or unauthorised 
distribution and publication of the information contained herein are 
prohibited.

7digital Limited. Registered office: 69 Wilson Street, London EC2A 2BB.
Registered in England and Wales. Registered No. 04843573.

Re: Writing a Producer from Scratch

Posted by Péricé Robin <pe...@gmail.com>.

Oups didn't see your response.

Sorry

2016-03-03 17:23 GMT+01:00 Péricé Robin <pe...@gmail.com>:

> Hi,
>
> Maybe you should look at this : https://github.com/edenhill/librdkafka
>
> Regards,
>
> Robin
>
> 2016-03-03 11:47 GMT+01:00 Hopson, Stephen <st...@gb.unisys.com>:
>
>> Hi,
>>
>> Not sure if this is the right forum for this question, but if it not I’m
>> sure someone will direct me to the proper one.
>>
>> Also, I am new to Kafka (but not new to computers).
>>
>>
>>
>> I want to write a kafka producer client for a Unisys OS 2200 mainframe. I
>> need to write it in C, and since I have no access to Windows / Unix / Linux
>> libraries, I have to develop the interface at the lowest level.
>>
>>
>>
>> So far, I have downloaded a kafka server with associated zookeeper (kafka
>> _2.10-0.8.2.2). Note I have downloaded the Windows version and have it
>> running on my laptop, successfully tested on the same laptop with the
>> provided provider and consumer clients.
>>
>>
>>
>> I have developed code to open a TCP session to the kafka server which
>> appears to work and I have attempted to send a metadata request which does
>> not appear to work. When I say it does not appear to work, I mean that I
>> send the message and then I sit on a retrieve, which eventually times out (
>> I do seem to get one character in the receive buffer of 0235 octal). The
>> message format I am using is the one described by the excellent document by
>> Jay Creps / Gwen Shapira at
>> https://cwiki.apache.org/confluence/display/KAFKA/A+Guide+To+The+Kafka+Protocol
>> However, it is not clear what level of kafka these message formats are
>> applicable for.
>>
>>
>>
>> Can anybody offer me any advice or suggestions as to how to progress?
>>
>>
>>
>> PS is the CRC mandatory in the Producer messages?
>>
>> Many thanks in advance.
>>
>>
>>
>> *Stephen Hopson* | Infrastructure Architect | Enterprise Solutions
>>
>> Unisys | +44 (0)1908 805010 | +44 (0)7557 303321 |
>> stephen.hopson@gb.unisys.com
>>
>>
>>
>> [image: unisys_logo] <http://www.unisys.com/>
>>
>>
>>
>> THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY
>> MATERIAL and is for use only by the intended recipient. If you received
>> this in error, please contact the sender and delete the e-mail and its
>> attachments from all devices.
>>
>>
>>
>
>

Re: Writing a Producer from Scratch

Posted by Péricé Robin <pe...@gmail.com>.

Hi,

Maybe you should look at this : https://github.com/edenhill/librdkafka

Regards,

Robin

2016-03-03 11:47 GMT+01:00 Hopson, Stephen <st...@gb.unisys.com>:

> Hi,
>
> Not sure if this is the right forum for this question, but if it not I’m
> sure someone will direct me to the proper one.
>
> Also, I am new to Kafka (but not new to computers).
>
>
>
> I want to write a kafka producer client for a Unisys OS 2200 mainframe. I
> need to write it in C, and since I have no access to Windows / Unix / Linux
> libraries, I have to develop the interface at the lowest level.
>
>
>
> So far, I have downloaded a kafka server with associated zookeeper (kafka
> _2.10-0.8.2.2). Note I have downloaded the Windows version and have it
> running on my laptop, successfully tested on the same laptop with the
> provided provider and consumer clients.
>
>
>
> I have developed code to open a TCP session to the kafka server which
> appears to work and I have attempted to send a metadata request which does
> not appear to work. When I say it does not appear to work, I mean that I
> send the message and then I sit on a retrieve, which eventually times out (
> I do seem to get one character in the receive buffer of 0235 octal). The
> message format I am using is the one described by the excellent document by
> Jay Creps / Gwen Shapira at
> https://cwiki.apache.org/confluence/display/KAFKA/A+Guide+To+The+Kafka+Protocol
> However, it is not clear what level of kafka these message formats are
> applicable for.
>
>
>
> Can anybody offer me any advice or suggestions as to how to progress?
>
>
>
> PS is the CRC mandatory in the Producer messages?
>
> Many thanks in advance.
>
>
>
> *Stephen Hopson* | Infrastructure Architect | Enterprise Solutions
>
> Unisys | +44 (0)1908 805010 | +44 (0)7557 303321 |
> stephen.hopson@gb.unisys.com
>
>
>
> [image: unisys_logo] <http://www.unisys.com/>
>
>
>
> THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY
> MATERIAL and is for use only by the intended recipient. If you received
> this in error, please contact the sender and delete the e-mail and its
> attachments from all devices.
>
>
>