You are viewing a plain text version of this content. The canonical link for it is here.

Posted to users@kafka.apache.org by "Doyle, Keith" <Ke...@greenwayhealth.com> on 2016/12/05 23:50:53 UTC

Some general questions...

We're beginning to make use of Kafka, and it is encouraging. But there are a couple of questions I've had a hard time finding answers for.

We're using the rdkafka-dotnet client on the consumer side and it's straightforward as far as it goes. However, documentation seems to be scant-the Wiki points to a FAQ which has, like, two questions neither of which are the questions we have. And I can't find a mailing list, forum, blog, or other community where questions can be asked. I found some indication in the Git repository that there may be some API docs, but it's not at all clear exactly where those are.

So I'm posting that question here because I can't find anywhere else that might be even remotely relevant to post it-where can I find out more info about rdkafka and particularly rdkafka-dotnet, and some way to ask questions that aren't answered in the documentation?

And second, my current question about rdkafka-dotnet, is the example consumers both seem to read an entire message into memory. We don't want to presume any particular message size, and may not want to cache the entire message in memory while processing it. Is there an interface where we can consume messages via a stream, so that we can read chunks of a message and process them based on some kind of batch size that will allow us better control over memory usage?

Thanks,

[Greenway_Health_PNG_large_NO_tag]

Keith Doyle | Senior Software Engineer
Greenway Health | 4301 W. Boy Scout Blvd., Suite 800, Tampa, FL 33607
(702) 256-9911 office | GreenwayHealth.com<http://www.greenwayhealth.com/>
[cid:image003.png@01D04086.868CBCB0]<https://www.linkedin.com/company/greenway-health>[cid:image004.png@01D04086.868CBCB0]<https://www.facebook.com/GreenwayHealthLLC>[cid:image005.png@01D04086.868CBCB0]<https://twitter.com/greenway>

Re: Some general questions...

Posted by Gwen Shapira <gw...@confluent.io>.

Confluent already supports a C client (the famous librdkafka). We are
indeed going to support a C# client, based on rdkafka-dotnet - we are
currently busy modifying the API a bit to fit our taste better :)



On Mon, Dec 5, 2016 at 6:34 PM, Tauzell, Dave
<Da...@surescripts.com> wrote:
> I don't know if any API to stream a message.  I don't suggest putting lots of large messages onto Kafka.
>
> As far as documentation I hear that confluent is going to support a C and C# client so you could try asking questions on the confluent mailing list.
>
> Dave
>
> On Dec 5, 2016, at 17:51, Doyle, Keith <Ke...@greenwayhealth.com>> wrote:
>
>
> We're beginning to make use of Kafka, and it is encouraging.  But there are a couple of questions I've had a hard time finding answers for.
>
> We're using the rdkafka-dotnet client on the consumer side and it's straightforward as far as it goes.  However, documentation seems to be scant-the Wiki points to a FAQ which has, like, two questions neither of which are the questions we have.   And I can't find a mailing list, forum, blog, or other community where questions can be asked.  I found some indication in the Git repository that there may be some API docs, but it's not at all clear exactly where those are.
>
> So I'm posting that question here because I can't find anywhere else that might be even remotely relevant to post it-where can I find out more info about rdkafka and particularly rdkafka-dotnet, and some way to ask questions that aren't answered in the documentation?
>
> And second, my current question about rdkafka-dotnet, is the example consumers both seem to read an entire message into memory.   We don't want to presume any particular message size, and may not want to cache the entire message in memory while processing it.   Is there an interface where we can consume messages via a stream, so that we can read chunks of a message and process them based on some kind of batch size that will allow us better control over memory usage?
>
>
> Thanks,
>
>
> --
>
> [Greenway_Health_PNG_large_NO_tag]
>
> Keith Doyle  |  Senior Software Engineer
> Greenway Health  |  4301 W. Boy Scout Blvd., Suite 800, Tampa, FL 33607
> (702) 256-9911 office  |  GreenwayHealth.com<http://www.greenwayhealth.com/>
> [cid:image003.png@01D04086.868CBCB0]<https://www.linkedin.com/company/greenway-health>[cid:image004.png@01D04086.868CBCB0]<https://www.facebook.com/GreenwayHealthLLC>[cid:image005.png@01D04086.868CBCB0]<https://twitter.com/greenway>
>
> NOTICE: This e-mail message and all attachments transmitted with it may contain legally privileged and confidential information intended solely for the use of the addressee. If the reader of this message is not the intended recipient, you are hereby notified that any reading, dissemination, distribution, copying, or other use of this message or its attachments is strictly prohibited. If you have received this message in error, please notify the sender immediately by electronic mail and delete this message and all copies and backups thereof. Thank you. Greenway Health.
> This e-mail and any files transmitted with it are confidential, may contain sensitive information, and are intended solely for the use of the individual or entity to whom they are addressed. If you have received this e-mail in error, please notify the sender by reply e-mail immediately and destroy all copies of the e-mail and any attachments.



-- 
Gwen Shapira
Product Manager | Confluent
650.450.2760 | @gwenshap
Follow us: Twitter | blog

Re: Some general questions...

Posted by "Tauzell, Dave" <Da...@surescripts.com>.

I don't know if any API to stream a message. I don't suggest putting lots of large messages onto Kafka.

As far as documentation I hear that confluent is going to support a C and C# client so you could try asking questions on the confluent mailing list.

Dave

On Dec 5, 2016, at 17:51, Doyle, Keith <Ke...@greenwayhealth.com>> wrote:

We're beginning to make use of Kafka, and it is encouraging. But there are a couple of questions I've had a hard time finding answers for.

Thanks,

[Greenway_Health_PNG_large_NO_tag]

NOTICE: This e-mail message and all attachments transmitted with it may contain legally privileged and confidential information intended solely for the use of the addressee. If the reader of this message is not the intended recipient, you are hereby notified that any reading, dissemination, distribution, copying, or other use of this message or its attachments is strictly prohibited. If you have received this message in error, please notify the sender immediately by electronic mail and delete this message and all copies and backups thereof. Thank you. Greenway Health.
This e-mail and any files transmitted with it are confidential, may contain sensitive information, and are intended solely for the use of the individual or entity to whom they are addressed. If you have received this e-mail in error, please notify the sender by reply e-mail immediately and destroy all copies of the e-mail and any attachments.

Re: Some general questions...

Posted by Gwen Shapira <gw...@confluent.io>.

Yeah, that's a good point - Kafka on Windows has few quirks because
most core Kafka developers are not windows experts and the big
deployments are almost all on Linux.

We discovered that most our .NET users actually run Kafka on Linux.
Turns out that installing few VMs with Linux and running Kafka is
fairly easy, but a programming language is not something you can
easily change.
Fortunately, thanks to librdkafka we can implement a good .NET client
without worrying about windows internals :)

On Tue, Dec 6, 2016 at 7:07 AM, Harald Kirsch <ha...@raytion.com> wrote:
> This sounds like you might want to run the Kafka broker on Windows. Have a
> look at https://issues.apache.org/jira/browse/KAFKA-1194 for possible issues
> with regard to log cleaning.
>
> Regards,
> Harald.
>
>
>
> On 06.12.2016 00:50, Doyle, Keith wrote:
>>
>>
>>
>> We’re beginning to make use of Kafka, and it is encouraging.  But there
>> are a couple of questions I’ve had a hard time finding answers for.
>>
>>
>>
>> We’re using the rdkafka-dotnet client on the consumer side and it’s
>> straightforward as far as it goes.  However, documentation seems to be
>> scant—the Wiki points to a FAQ which has, like, two questions neither of
>> which are the questions we have.   And I can’t find a mailing list,
>> forum, blog, or other community where questions can be asked.  I found
>> some indication in the Git repository that there may be some API docs,
>> but it’s not at all clear exactly where those are.
>>
>>
>>
>> So I’m posting that question here because I can’t find anywhere else
>> that might be even remotely relevant to post it—where can I find out
>> more info about rdkafka and particularly rdkafka-dotnet, and some way to
>> ask questions that aren’t answered in the documentation?
>>
>>
>>
>> And second, my current question about rdkafka-dotnet, is the example
>> consumers both seem to read an entire message into memory.   We don’t
>> want to presume any particular message size, and may not want to cache
>> the entire message in memory while processing it.   Is there an
>> interface where we can consume messages via a stream, so that we can
>> read chunks of a message and process them based on some kind of batch
>> size that will allow us better control over memory usage?
>>
>>
>>
>



-- 
Gwen Shapira
Product Manager | Confluent
650.450.2760 | @gwenshap
Follow us: Twitter | blog

Re: Some general questions...

Posted by Harald Kirsch <ha...@raytion.com>.

This sounds like you might want to run the Kafka broker on Windows. Have 
a look at https://issues.apache.org/jira/browse/KAFKA-1194 for possible 
issues with regard to log cleaning.

Regards,
Harald.


On 06.12.2016 00:50, Doyle, Keith wrote:
>
>
> Were beginning to make use of Kafka, and it is encouraging.  But there
> are a couple of questions Ive had a hard time finding answers for.
>
>
>
> Were using the rdkafka-dotnet client on the consumer side and its
> straightforward as far as it goes.  However, documentation seems to be
> scantthe Wiki points to a FAQ which has, like, two questions neither of
> which are the questions we have.   And I cant find a mailing list,
> forum, blog, or other community where questions can be asked.  I found
> some indication in the Git repository that there may be some API docs,
> but its not at all clear exactly where those are.
>
>
>
> So Im posting that question here because I cant find anywhere else
> that might be even remotely relevant to post itwhere can I find out
> more info about rdkafka and particularly rdkafka-dotnet, and some way to
> ask questions that arent answered in the documentation?
>
>
>
> And second, my current question about rdkafka-dotnet, is the example
> consumers both seem to read an entire message into memory.   We dont
> want to presume any particular message size, and may not want to cache
> the entire message in memory while processing it.   Is there an
> interface where we can consume messages via a stream, so that we can
> read chunks of a message and process them based on some kind of batch
> size that will allow us better control over memory usage?
>
>
>