You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kafka.apache.org by Bhavesh Mistry <mi...@gmail.com> on 2014/10/06 00:30:58 UTC
[Java New Producer Kafka Trunk ] Need a State Check API Method
HI Kafka Dev,
I would like to request state check state so I can manage the Life Cycle
of Producer better. I you guys agree I will file Jira request. I just
give state of producer can be I would like mange or start (create new
instance of producer) or restart or close based on state. I just gave
example, you may add or remove states.
/***
* API TO CHECK STATE OF PRODUCER
* @Return
STATE.INIT_IN_PROGRESS
STATE.INIT_DONE
STATE.RUNNING
STATE.CLOSE_REQUESTED
STATE.CLOSE_IN_PROGRESS
STATE.CLOSED
*/
public State getCurrentState();
Thanks,
Bhavesh
Re: [Java New Producer Kafka Trunk ] Need a State Check API Method
Posted by Jay Kreps <ja...@gmail.com>.
Hey Bhavesh,
But isn't the problem here that you are trying to send messages after
closing the producer?
I think what I am saying is that since calling close is something the user
initiates we don't need an api for this--you can keep track of this
yourself, right?
-Jay
On Mon, Oct 6, 2014 at 11:11 PM, Bhavesh Mistry <mi...@gmail.com>
wrote:
> I agree with that statement that if producer is closed and try to send
> message it will give close. What we have done is wrap the NEW Producer API
> with Old Producer API. So when I use same code with OLD I do not get this
> issue. It is only problem with NEW Producer. Regardless of close, state I
> think it will be good to have API to check the state of producer (at least
> isClosed() API).
>
> If you agree, I can file a Jira Request for STATE check API and let me know
> which flavor of State Check API you prefer.
>
>
> Thanks,
>
> Bhavesh
>
> On Mon, Oct 6, 2014 at 9:34 AM, Jay Kreps <ja...@gmail.com> wrote:
>
> > Hey Bhavesh,
> >
> > This is a sanity check. If you send a message after calling close on the
> > producer you should get this error. It sounds like you have multiple
> > threads sending, and you close the producer in the middle of this, then
> you
> > get this error. This is expected.
> >
> > Perhaps I am misunderstanding?
> >
> > I think tracking the state (i.e. whether you have called close or not)
> can
> > be done just as easily in your code, right?
> >
> > -Jay
> >
> > On Sun, Oct 5, 2014 at 7:32 PM, Bhavesh Mistry <
> mistry.p.bhavesh@gmail.com
> > >
> > wrote:
> >
> > > Hi Kafka Dev Team,
> > >
> > > *java.lang.*
> > > *IllegalStateException: Cannot send after the producer is closed.*
> > >
> > > The above seems to bug. If the ProducerRecord is in flight within send
> > > method is execute and another thread seems to shutdown in the middle of
> > > flight will get error.
> > >
> > > Thanks,
> > >
> > > Bhavesh
> > >
> > > On Sun, Oct 5, 2014 at 7:15 PM, Bhavesh Mistry <
> > mistry.p.bhavesh@gmail.com
> > > >
> > > wrote:
> > >
> > > > Hi Kafka Dev Team,
> > > >
> > > > The use case is that we need to know producer state in background
> > Threads
> > > > and so we can submit the message.
> > > >
> > > > This seems to a bug in trunk code. I have notice that KafkaProducer
> > > > itself does not have close state and inflight message will encounter
> > > > following issues. Should I file bug for this issue ?
> > > >
> > > > java.lang.IllegalStateException: Cannot send after the producer is
> > > closed.
> > > > at
> > > >
> > >
> >
> org.apache.kafka.clients.producer.internals.RecordAccumulator.append(RecordAccumulator.java:136)
> > > > at
> > > >
> > >
> >
> org.apache.kafka.clients.producer.KafkaProducer.send(KafkaProducer.java:237)
> > > > .................................................
> > > > at java.util.TimerThread.mainLoop(Timer.java:555)
> > > > at java.util.TimerThread.run(Timer.java:505)
> > > >
> > > > Thanks,
> > > >
> > > > Bhavesh
> > > >
> > > > On Sun, Oct 5, 2014 at 3:30 PM, Bhavesh Mistry <
> > > mistry.p.bhavesh@gmail.com
> > > > > wrote:
> > > >
> > > >> HI Kafka Dev,
> > > >>
> > > >> I would like to request state check state so I can manage the Life
> > > Cycle
> > > >> of Producer better. I you guys agree I will file Jira request. I
> > just
> > > >> give state of producer can be I would like mange or start (create
> new
> > > >> instance of producer) or restart or close based on state. I just
> > gave
> > > >> example, you may add or remove states.
> > > >>
> > > >> /***
> > > >>
> > > >> * API TO CHECK STATE OF PRODUCER
> > > >>
> > > >> * @Return
> > > >>
> > > >>
> > > >>
> > > >> STATE.INIT_IN_PROGRESS
> > > >>
> > > >> STATE.INIT_DONE
> > > >>
> > > >> STATE.RUNNING
> > > >>
> > > >> STATE.CLOSE_REQUESTED
> > > >>
> > > >> STATE.CLOSE_IN_PROGRESS
> > > >>
> > > >> STATE.CLOSED
> > > >>
> > > >> */
> > > >>
> > > >> public State getCurrentState();
> > > >>
> > > >> Thanks,
> > > >>
> > > >> Bhavesh
> > > >>
> > > >
> > > >
> > >
> >
>
Re: [Java New Producer Kafka Trunk ] Need a State Check API Method
Posted by Bhavesh Mistry <mi...@gmail.com>.
I agree with that statement that if producer is closed and try to send
message it will give close. What we have done is wrap the NEW Producer API
with Old Producer API. So when I use same code with OLD I do not get this
issue. It is only problem with NEW Producer. Regardless of close, state I
think it will be good to have API to check the state of producer (at least
isClosed() API).
If you agree, I can file a Jira Request for STATE check API and let me know
which flavor of State Check API you prefer.
Thanks,
Bhavesh
On Mon, Oct 6, 2014 at 9:34 AM, Jay Kreps <ja...@gmail.com> wrote:
> Hey Bhavesh,
>
> This is a sanity check. If you send a message after calling close on the
> producer you should get this error. It sounds like you have multiple
> threads sending, and you close the producer in the middle of this, then you
> get this error. This is expected.
>
> Perhaps I am misunderstanding?
>
> I think tracking the state (i.e. whether you have called close or not) can
> be done just as easily in your code, right?
>
> -Jay
>
> On Sun, Oct 5, 2014 at 7:32 PM, Bhavesh Mistry <mistry.p.bhavesh@gmail.com
> >
> wrote:
>
> > Hi Kafka Dev Team,
> >
> > *java.lang.*
> > *IllegalStateException: Cannot send after the producer is closed.*
> >
> > The above seems to bug. If the ProducerRecord is in flight within send
> > method is execute and another thread seems to shutdown in the middle of
> > flight will get error.
> >
> > Thanks,
> >
> > Bhavesh
> >
> > On Sun, Oct 5, 2014 at 7:15 PM, Bhavesh Mistry <
> mistry.p.bhavesh@gmail.com
> > >
> > wrote:
> >
> > > Hi Kafka Dev Team,
> > >
> > > The use case is that we need to know producer state in background
> Threads
> > > and so we can submit the message.
> > >
> > > This seems to a bug in trunk code. I have notice that KafkaProducer
> > > itself does not have close state and inflight message will encounter
> > > following issues. Should I file bug for this issue ?
> > >
> > > java.lang.IllegalStateException: Cannot send after the producer is
> > closed.
> > > at
> > >
> >
> org.apache.kafka.clients.producer.internals.RecordAccumulator.append(RecordAccumulator.java:136)
> > > at
> > >
> >
> org.apache.kafka.clients.producer.KafkaProducer.send(KafkaProducer.java:237)
> > > .................................................
> > > at java.util.TimerThread.mainLoop(Timer.java:555)
> > > at java.util.TimerThread.run(Timer.java:505)
> > >
> > > Thanks,
> > >
> > > Bhavesh
> > >
> > > On Sun, Oct 5, 2014 at 3:30 PM, Bhavesh Mistry <
> > mistry.p.bhavesh@gmail.com
> > > > wrote:
> > >
> > >> HI Kafka Dev,
> > >>
> > >> I would like to request state check state so I can manage the Life
> > Cycle
> > >> of Producer better. I you guys agree I will file Jira request. I
> just
> > >> give state of producer can be I would like mange or start (create new
> > >> instance of producer) or restart or close based on state. I just
> gave
> > >> example, you may add or remove states.
> > >>
> > >> /***
> > >>
> > >> * API TO CHECK STATE OF PRODUCER
> > >>
> > >> * @Return
> > >>
> > >>
> > >>
> > >> STATE.INIT_IN_PROGRESS
> > >>
> > >> STATE.INIT_DONE
> > >>
> > >> STATE.RUNNING
> > >>
> > >> STATE.CLOSE_REQUESTED
> > >>
> > >> STATE.CLOSE_IN_PROGRESS
> > >>
> > >> STATE.CLOSED
> > >>
> > >> */
> > >>
> > >> public State getCurrentState();
> > >>
> > >> Thanks,
> > >>
> > >> Bhavesh
> > >>
> > >
> > >
> >
>
Re: [Java New Producer Kafka Trunk ] Need a State Check API Method
Posted by Jay Kreps <ja...@gmail.com>.
Hey Bhavesh,
This is a sanity check. If you send a message after calling close on the
producer you should get this error. It sounds like you have multiple
threads sending, and you close the producer in the middle of this, then you
get this error. This is expected.
Perhaps I am misunderstanding?
I think tracking the state (i.e. whether you have called close or not) can
be done just as easily in your code, right?
-Jay
On Sun, Oct 5, 2014 at 7:32 PM, Bhavesh Mistry <mi...@gmail.com>
wrote:
> Hi Kafka Dev Team,
>
> *java.lang.*
> *IllegalStateException: Cannot send after the producer is closed.*
>
> The above seems to bug. If the ProducerRecord is in flight within send
> method is execute and another thread seems to shutdown in the middle of
> flight will get error.
>
> Thanks,
>
> Bhavesh
>
> On Sun, Oct 5, 2014 at 7:15 PM, Bhavesh Mistry <mistry.p.bhavesh@gmail.com
> >
> wrote:
>
> > Hi Kafka Dev Team,
> >
> > The use case is that we need to know producer state in background Threads
> > and so we can submit the message.
> >
> > This seems to a bug in trunk code. I have notice that KafkaProducer
> > itself does not have close state and inflight message will encounter
> > following issues. Should I file bug for this issue ?
> >
> > java.lang.IllegalStateException: Cannot send after the producer is
> closed.
> > at
> >
> org.apache.kafka.clients.producer.internals.RecordAccumulator.append(RecordAccumulator.java:136)
> > at
> >
> org.apache.kafka.clients.producer.KafkaProducer.send(KafkaProducer.java:237)
> > .................................................
> > at java.util.TimerThread.mainLoop(Timer.java:555)
> > at java.util.TimerThread.run(Timer.java:505)
> >
> > Thanks,
> >
> > Bhavesh
> >
> > On Sun, Oct 5, 2014 at 3:30 PM, Bhavesh Mistry <
> mistry.p.bhavesh@gmail.com
> > > wrote:
> >
> >> HI Kafka Dev,
> >>
> >> I would like to request state check state so I can manage the Life
> Cycle
> >> of Producer better. I you guys agree I will file Jira request. I just
> >> give state of producer can be I would like mange or start (create new
> >> instance of producer) or restart or close based on state. I just gave
> >> example, you may add or remove states.
> >>
> >> /***
> >>
> >> * API TO CHECK STATE OF PRODUCER
> >>
> >> * @Return
> >>
> >>
> >>
> >> STATE.INIT_IN_PROGRESS
> >>
> >> STATE.INIT_DONE
> >>
> >> STATE.RUNNING
> >>
> >> STATE.CLOSE_REQUESTED
> >>
> >> STATE.CLOSE_IN_PROGRESS
> >>
> >> STATE.CLOSED
> >>
> >> */
> >>
> >> public State getCurrentState();
> >>
> >> Thanks,
> >>
> >> Bhavesh
> >>
> >
> >
>
Re: [Java New Producer Kafka Trunk ] Need a State Check API Method
Posted by Bhavesh Mistry <mi...@gmail.com>.
Hi Kafka Dev Team,
*java.lang.*
*IllegalStateException: Cannot send after the producer is closed.*
The above seems to bug. If the ProducerRecord is in flight within send
method is execute and another thread seems to shutdown in the middle of
flight will get error.
Thanks,
Bhavesh
On Sun, Oct 5, 2014 at 7:15 PM, Bhavesh Mistry <mi...@gmail.com>
wrote:
> Hi Kafka Dev Team,
>
> The use case is that we need to know producer state in background Threads
> and so we can submit the message.
>
> This seems to a bug in trunk code. I have notice that KafkaProducer
> itself does not have close state and inflight message will encounter
> following issues. Should I file bug for this issue ?
>
> java.lang.IllegalStateException: Cannot send after the producer is closed.
> at
> org.apache.kafka.clients.producer.internals.RecordAccumulator.append(RecordAccumulator.java:136)
> at
> org.apache.kafka.clients.producer.KafkaProducer.send(KafkaProducer.java:237)
> .................................................
> at java.util.TimerThread.mainLoop(Timer.java:555)
> at java.util.TimerThread.run(Timer.java:505)
>
> Thanks,
>
> Bhavesh
>
> On Sun, Oct 5, 2014 at 3:30 PM, Bhavesh Mistry <mistry.p.bhavesh@gmail.com
> > wrote:
>
>> HI Kafka Dev,
>>
>> I would like to request state check state so I can manage the Life Cycle
>> of Producer better. I you guys agree I will file Jira request. I just
>> give state of producer can be I would like mange or start (create new
>> instance of producer) or restart or close based on state. I just gave
>> example, you may add or remove states.
>>
>> /***
>>
>> * API TO CHECK STATE OF PRODUCER
>>
>> * @Return
>>
>>
>>
>> STATE.INIT_IN_PROGRESS
>>
>> STATE.INIT_DONE
>>
>> STATE.RUNNING
>>
>> STATE.CLOSE_REQUESTED
>>
>> STATE.CLOSE_IN_PROGRESS
>>
>> STATE.CLOSED
>>
>> */
>>
>> public State getCurrentState();
>>
>> Thanks,
>>
>> Bhavesh
>>
>
>
Re: [Java New Producer Kafka Trunk ] Need a State Check API Method
Posted by Bhavesh Mistry <mi...@gmail.com>.
Hi Kafka Dev Team,
The use case is that we need to know producer state in background Threads
and so we can submit the message.
This seems to a bug in trunk code. I have notice that KafkaProducer itself
does not have close state and inflight message will encounter following
issues. Should I file bug for this issue ?
java.lang.IllegalStateException: Cannot send after the producer is closed.
at
org.apache.kafka.clients.producer.internals.RecordAccumulator.append(RecordAccumulator.java:136)
at
org.apache.kafka.clients.producer.KafkaProducer.send(KafkaProducer.java:237)
.................................................
at java.util.TimerThread.mainLoop(Timer.java:555)
at java.util.TimerThread.run(Timer.java:505)
Thanks,
Bhavesh
On Sun, Oct 5, 2014 at 3:30 PM, Bhavesh Mistry <mi...@gmail.com>
wrote:
> HI Kafka Dev,
>
> I would like to request state check state so I can manage the Life Cycle
> of Producer better. I you guys agree I will file Jira request. I just
> give state of producer can be I would like mange or start (create new
> instance of producer) or restart or close based on state. I just gave
> example, you may add or remove states.
>
> /***
>
> * API TO CHECK STATE OF PRODUCER
>
> * @Return
>
>
>
> STATE.INIT_IN_PROGRESS
>
> STATE.INIT_DONE
>
> STATE.RUNNING
>
> STATE.CLOSE_REQUESTED
>
> STATE.CLOSE_IN_PROGRESS
>
> STATE.CLOSED
>
> */
>
> public State getCurrentState();
>
> Thanks,
>
> Bhavesh
>