You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Bhuvana Baskar <bh...@gmail.com> on 2015/02/23 06:05:17 UTC
How to find the number of message in the topic
Hi,
Please let me know how to find the total number of messages in a particular
topic.
Regards,
Bhuvana
Re: How to find the number of message in the topic
Posted by Stuart Reynolds <st...@stureynolds.com>.
At a fixed number. I think my problem was that I assuming that there
was a single partition in my code ... still working on this.
On Tue, Feb 24, 2015 at 11:06 AM, Zakee <kz...@netzero.net> wrote:
> Does that count get frozen on a fixed number or any random number?
>
> -Zakee
>
> On Mon, Feb 23, 2015 at 9:48 AM, Stuart Reynolds <st...@stureynolds.com>
> wrote:
>
>> See SimpleConsumer. getOffsetsBefore
>> and the .... getLastOffset example here:
>>
>> https://cwiki.apache.org/confluence/display/KAFKA/0.8.0+SimpleConsumer+Example
>> whichTime=-1 or -2 (depending whether you want the latest or earliest
>> offset.
>>
>> However, I've been banging my head against the wall with this API --
>> it seems that, even with a single broker and a single host, sometimes
>> the latest offset gets frozen as new data comes in. :-/
>>
>>
>>
>> public static long getLastOffset(SimpleConsumer consumer, String
>> topic, int partition,
>> long whichTime, String clientName) {
>> TopicAndPartition topicAndPartition = new
>> TopicAndPartition(topic, partition);
>> Map<TopicAndPartition, PartitionOffsetRequestInfo> requestInfo
>> = new HashMap<TopicAndPartition, PartitionOffsetRequestInfo>();
>> requestInfo.put(topicAndPartition, new
>> PartitionOffsetRequestInfo(whichTime, 1));
>> kafka.javaapi.OffsetRequest request = new
>> kafka.javaapi.OffsetRequest(requestInfo,
>> kafka.api.OffsetRequest.CurrentVersion(),clientName);
>> OffsetResponse response = consumer.getOffsetsBefore(request);
>>
>> if (response.hasError()) {
>> System.out.println("Error fetching data Offset Data the
>> Broker. Reason: " + response.errorCode(topic, partition) );
>> return 0;
>> }
>> long[] offsets = response.offsets(topic, partition);
>> return offsets[0];
>> }
>>
>> On Sun, Feb 22, 2015 at 9:05 PM, Bhuvana Baskar
>> <bh...@gmail.com> wrote:
>> > Hi,
>> >
>> > Please let me know how to find the total number of messages in a
>> particular
>> > topic.
>> >
>> > Regards,
>> > Bhuvana
>> ____________________________________________________________
>> Heavy rains mean flooding
>> Anywhere it rains it can flood. Learn your risk. Get flood insurance.
>> http://thirdpartyoffers.netzero.net/TGL3255/54eb6db072a8f6db07624mp13duc
>>
>>
Re: How to find the number of message in the topic
Posted by Zakee <kz...@netzero.net>.
Does that count get frozen on a fixed number or any random number?
-Zakee
On Mon, Feb 23, 2015 at 9:48 AM, Stuart Reynolds <st...@stureynolds.com>
wrote:
> See SimpleConsumer. getOffsetsBefore
> and the .... getLastOffset example here:
>
> https://cwiki.apache.org/confluence/display/KAFKA/0.8.0+SimpleConsumer+Example
> whichTime=-1 or -2 (depending whether you want the latest or earliest
> offset.
>
> However, I've been banging my head against the wall with this API --
> it seems that, even with a single broker and a single host, sometimes
> the latest offset gets frozen as new data comes in. :-/
>
>
>
> public static long getLastOffset(SimpleConsumer consumer, String
> topic, int partition,
> long whichTime, String clientName) {
> TopicAndPartition topicAndPartition = new
> TopicAndPartition(topic, partition);
> Map<TopicAndPartition, PartitionOffsetRequestInfo> requestInfo
> = new HashMap<TopicAndPartition, PartitionOffsetRequestInfo>();
> requestInfo.put(topicAndPartition, new
> PartitionOffsetRequestInfo(whichTime, 1));
> kafka.javaapi.OffsetRequest request = new
> kafka.javaapi.OffsetRequest(requestInfo,
> kafka.api.OffsetRequest.CurrentVersion(),clientName);
> OffsetResponse response = consumer.getOffsetsBefore(request);
>
> if (response.hasError()) {
> System.out.println("Error fetching data Offset Data the
> Broker. Reason: " + response.errorCode(topic, partition) );
> return 0;
> }
> long[] offsets = response.offsets(topic, partition);
> return offsets[0];
> }
>
> On Sun, Feb 22, 2015 at 9:05 PM, Bhuvana Baskar
> <bh...@gmail.com> wrote:
> > Hi,
> >
> > Please let me know how to find the total number of messages in a
> particular
> > topic.
> >
> > Regards,
> > Bhuvana
> ____________________________________________________________
> Heavy rains mean flooding
> Anywhere it rains it can flood. Learn your risk. Get flood insurance.
> http://thirdpartyoffers.netzero.net/TGL3255/54eb6db072a8f6db07624mp13duc
>
>
Re: How to find the number of message in the topic
Posted by Stuart Reynolds <st...@stureynolds.com>.
See SimpleConsumer. getOffsetsBefore
and the .... getLastOffset example here:
https://cwiki.apache.org/confluence/display/KAFKA/0.8.0+SimpleConsumer+Example
whichTime=-1 or -2 (depending whether you want the latest or earliest offset.
However, I've been banging my head against the wall with this API --
it seems that, even with a single broker and a single host, sometimes
the latest offset gets frozen as new data comes in. :-/
public static long getLastOffset(SimpleConsumer consumer, String
topic, int partition,
long whichTime, String clientName) {
TopicAndPartition topicAndPartition = new
TopicAndPartition(topic, partition);
Map<TopicAndPartition, PartitionOffsetRequestInfo> requestInfo
= new HashMap<TopicAndPartition, PartitionOffsetRequestInfo>();
requestInfo.put(topicAndPartition, new
PartitionOffsetRequestInfo(whichTime, 1));
kafka.javaapi.OffsetRequest request = new
kafka.javaapi.OffsetRequest(requestInfo,
kafka.api.OffsetRequest.CurrentVersion(),clientName);
OffsetResponse response = consumer.getOffsetsBefore(request);
if (response.hasError()) {
System.out.println("Error fetching data Offset Data the
Broker. Reason: " + response.errorCode(topic, partition) );
return 0;
}
long[] offsets = response.offsets(topic, partition);
return offsets[0];
}
On Sun, Feb 22, 2015 at 9:05 PM, Bhuvana Baskar
<bh...@gmail.com> wrote:
> Hi,
>
> Please let me know how to find the total number of messages in a particular
> topic.
>
> Regards,
> Bhuvana