Web UI for Spark Streaming app lists jobs incorrectly

[Jon Chase <jo...@gmail.com>]

I've got a Spark 1.4.1 (also observed on 1.3.1) Streaming app running.  I
have the scheduler mode set to "fair":

conf.set("spark.scheduler.mode", "FAIR");
JavaSparkContext sc = new JavaSparkContext(conf);

I set up a streaming context and start it:

...
dStream.foreachRDD(myFunc);
...

public static void startStreaming(JavaStreamingContext ssc, boolean
awaitTermination, long awaitTerminationTimeoutMs) {
        ...
        ssc.start();
        ...
}

After doing this, I then have a separate thread that runs some batches at
predefined intervals.  I expect to see those batches in the Spark UI with
their associated stages/tasks, but instead I only get references to either
"foreachRDD" or "start", and the line numbers associated with them
reference back to those two methods instead of my (non-streaming) batch
transformations.

When I run the app with Spark Streaming disabled (no StreamingContext), the
UI works fine.

[image: Inline image 2]


The DAG visualizations I get are also very strange when streaming is
enabled:

[image: Inline image 3]

This issue appears both when running in Yarn and running locally on my
laptop (via IDE).

What am I doing wrong?

Re: Web UI for Spark Streaming app lists jobs incorrectly

[Tathagata Das <td...@databricks.com>]

This is a known issue in the way we were passing on the callsite
information. Internally ThreadLocal variables are used in a way that has
weird behavior.

The related JIRA is this - https://issues.apache.org/jira/browse/SPARK-10649
This has been fixed few days ago, and will be available in 1.5.1 and 1.6.0

In the meantime, if you start the thread before streamingContext.start(),
you would not see the callsite being the same on as the streaming ones.


On Sun, Sep 20, 2015 at 6:28 PM, Jon Chase <jo...@gmail.com> wrote:

> I've got a Spark 1.4.1 (also observed on 1.3.1) Streaming app running.  I
> have the scheduler mode set to "fair":
>
> conf.set("spark.scheduler.mode", "FAIR");
> JavaSparkContext sc = new JavaSparkContext(conf);
>
> I set up a streaming context and start it:
>
> ...
> dStream.foreachRDD(myFunc);
> ...
>
> public static void startStreaming(JavaStreamingContext ssc, boolean
> awaitTermination, long awaitTerminationTimeoutMs) {
>         ...
>         ssc.start();
>         ...
> }
>
> After doing this, I then have a separate thread that runs some batches at
> predefined intervals.  I expect to see those batches in the Spark UI with
> their associated stages/tasks, but instead I only get references to either
> "foreachRDD" or "start", and the line numbers associated with them
> reference back to those two methods instead of my (non-streaming) batch
> transformations.
>
> When I run the app with Spark Streaming disabled (no StreamingContext),
> the UI works fine.
>
> [image: Inline image 2]
>
>
> The DAG visualizations I get are also very strange when streaming is
> enabled:
>
> [image: Inline image 3]
>
> This issue appears both when running in Yarn and running locally on my
> laptop (via IDE).
>
> What am I doing wrong?
>