Kafka offset problem when using Spark Streaming...

[Pena Quijada Alexander <a....@reply.it>]

Hi all,

We're facing some problems with ours Spark Streaming jobs, from yesterday we have got the following error into our logs when the jobs fail:

java.lang.AssertionError: assertion failed: Beginning offset 562747 is after the ending offset 562743 for topic elk-topic partition 0.

Any help about this issue will be appreciated!

Many thanks in advance for the kind cooperation.

Best regards,

Alex



________________________________

--
The information transmitted is intended for the person or entity to which it is addressed and may contain confidential and/or privileged material. Any review, retransmission, dissemination or other use of, or taking of any action in reliance upon, this information by persons or entities other than the intended recipient is prohibited. If you received this in error, please contact the sender and delete the material from any computer.

Re: Kafka offset problem when using Spark Streaming...

["Matthias J. Sax" <ma...@confluent.io>]

Hard to say. Might be a Spark issue though...

On 5/9/18 3:42 AM, Pena Quijada Alexander wrote:
> Hi all,
> 
> We're facing some problems with ours Spark Streaming jobs, from yesterday we have got the following error into our logs when the jobs fail:
> 
> java.lang.AssertionError: assertion failed: Beginning offset 562747 is after the ending offset 562743 for topic elk-topic partition 0.
> 
> Any help about this issue will be appreciated!
> 
> Many thanks in advance for the kind cooperation.
> 
> Best regards,
> 
> Alex
> 
> 
> 
> ________________________________
> 
> --
> The information transmitted is intended for the person or entity to which it is addressed and may contain confidential and/or privileged material. Any review, retransmission, dissemination or other use of, or taking of any action in reliance upon, this information by persons or entities other than the intended recipient is prohibited. If you received this in error, please contact the sender and delete the material from any computer.
> 

R: Kafka offset problem when using Spark Streaming...

[Pena Quijada Alexander <a....@reply.it>]

Hi all,

sorry for delay answer, I was a bit busy in the last few days at work. We have found the root cause of our problem, we have had a network problem with one of our kafka brokers, in fact, in that broker there weren't any active partitions anymore, all the partitions were in the other brokers. We solved the network problem and we rebalanced the Kafka partitions, after that, the problem evidenced in my last email has been fixed!

We're using Spark 2.0 and Kafka 0.9.

Thanks a lot for the kind cooperation!

Best regards,

Alex

-----Messaggio originale-----
Da: Ted Yu <yu...@gmail.com>
Inviato: mercoledì 9 maggio 2018 17:58
A: users@kafka.apache.org
Oggetto: Re: Kafka offset problem when using Spark Streaming...

Can you give us more information:

release of Spark and Kafka you're using

anything interesting from broker / client logs - feel free to use pastebin to pass snippet if needed

Thanks

On Wed, May 9, 2018 at 3:42 AM, Pena Quijada Alexander < a.penaquijada@reply.it> wrote:

> Hi all,
>
> We're facing some problems with ours Spark Streaming jobs, from
> yesterday we have got the following error into our logs when the jobs fail:
>
> java.lang.AssertionError: assertion failed: Beginning offset 562747 is
> after the ending offset 562743 for topic elk-topic partition 0.
>
> Any help about this issue will be appreciated!
>
> Many thanks in advance for the kind cooperation.
>
> Best regards,
>
> Alex
>
>
>
> ________________________________
>
> --
> The information transmitted is intended for the person or entity to
> which it is addressed and may contain confidential and/or privileged material.
> Any review, retransmission, dissemination or other use of, or taking
> of any action in reliance upon, this information by persons or
> entities other than the intended recipient is prohibited. If you
> received this in error, please contact the sender and delete the material from any computer.
>



________________________________

--
The information transmitted is intended for the person or entity to which it is addressed and may contain confidential and/or privileged material. Any review, retransmission, dissemination or other use of, or taking of any action in reliance upon, this information by persons or entities other than the intended recipient is prohibited. If you received this in error, please contact the sender and delete the material from any computer.

Re: Kafka offset problem when using Spark Streaming...

[Ted Yu <yu...@gmail.com>]

Can you give us more information:

release of Spark and Kafka you're using

anything interesting from broker / client logs - feel free to use pastebin
to pass snippet if needed

Thanks

On Wed, May 9, 2018 at 3:42 AM, Pena Quijada Alexander <
a.penaquijada@reply.it> wrote:

> Hi all,
>
> We're facing some problems with ours Spark Streaming jobs, from yesterday
> we have got the following error into our logs when the jobs fail:
>
> java.lang.AssertionError: assertion failed: Beginning offset 562747 is
> after the ending offset 562743 for topic elk-topic partition 0.
>
> Any help about this issue will be appreciated!
>
> Many thanks in advance for the kind cooperation.
>
> Best regards,
>
> Alex
>
>
>
> ________________________________
>
> --
> The information transmitted is intended for the person or entity to which
> it is addressed and may contain confidential and/or privileged material.
> Any review, retransmission, dissemination or other use of, or taking of any
> action in reliance upon, this information by persons or entities other than
> the intended recipient is prohibited. If you received this in error, please
> contact the sender and delete the material from any computer.
>