You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Julian Keppel <ju...@gmail.com> on 2016/11/18 10:38:45 UTC
Kafka direct approach,App UI shows wrong input rate
Hello,
I use Spark 2.0.2 with Kafka integration 0-8. The Kafka version is 0.10.0.1
(Scala 2.11). I read data from Kafka with the direct approach. The complete
infrastructure runs on Google Container Engine.
I wonder why the corresponding application UI says the input rate is zero
records per second. This is definitely wrong. I checked it while I printed
out the incoming records to the driver console. All other metrics seem to
be correct (at least they are realistic).
What is going on here? Do you have any idea? Thanks for you help.
Julian
Re: Kafka direct approach,App UI shows wrong input rate
Posted by Julian Keppel <ju...@gmail.com>.
Oh, sorry. I made a mistake... It's spark version 2.0.1, not 2.0.2.
When I wrote the initial message I built my app with 2.0.2 and deployed it
on a cluster with 2.0.1. So I thought this could be the problem. But now I
changed it and build my app with 2.0.1 but the problem still remains.
2016-11-19 18:06 GMT+01:00 Cody Koeninger <co...@koeninger.org>:
> There have definitely been issues with UI reporting for the direct
> stream in the past, but I'm not able to reproduce this with 2.0.2 and
> 0.8. See below:
>
> https://i.imgsafe.org/086019ae57.png
>
>
>
> On Fri, Nov 18, 2016 at 4:38 AM, Julian Keppel
> <ju...@gmail.com> wrote:
> > Hello,
> >
> > I use Spark 2.0.2 with Kafka integration 0-8. The Kafka version is
> 0.10.0.1
> > (Scala 2.11). I read data from Kafka with the direct approach. The
> complete
> > infrastructure runs on Google Container Engine.
> >
> > I wonder why the corresponding application UI says the input rate is zero
> > records per second. This is definitely wrong. I checked it while I
> printed
> > out the incoming records to the driver console. All other metrics seem
> to be
> > correct (at least they are realistic).
> >
> > What is going on here? Do you have any idea? Thanks for you help.
> >
> > Julian
>
Re: Kafka direct approach,App UI shows wrong input rate
Posted by Cody Koeninger <co...@koeninger.org>.
There have definitely been issues with UI reporting for the direct
stream in the past, but I'm not able to reproduce this with 2.0.2 and
0.8. See below:
https://i.imgsafe.org/086019ae57.png
On Fri, Nov 18, 2016 at 4:38 AM, Julian Keppel
<ju...@gmail.com> wrote:
> Hello,
>
> I use Spark 2.0.2 with Kafka integration 0-8. The Kafka version is 0.10.0.1
> (Scala 2.11). I read data from Kafka with the direct approach. The complete
> infrastructure runs on Google Container Engine.
>
> I wonder why the corresponding application UI says the input rate is zero
> records per second. This is definitely wrong. I checked it while I printed
> out the incoming records to the driver console. All other metrics seem to be
> correct (at least they are realistic).
>
> What is going on here? Do you have any idea? Thanks for you help.
>
> Julian
---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscribe@spark.apache.org