SparkLauncher not receiving events

[Mariano Semelman <ma...@despegar.com>]

Hello,

I'm having problems to receive events from the submited app. The app
succesfuly submits, but the listener I'm passing to SparkLauncher is not
receiving events.

Spark Version: 1.6.1 (both client app and master)

here are the relevant snippets I'm using in my code:
https://gist.github.com/msemelman/d9d2b54ce0c01af8952bc298058dc0d5

BTW the logger is correctly configured if you ask for that. I even debugged
the spark code and put some breakpoints
in org.apache.spark.launcher.LauncherServer.ServerConnection#handle to
check if any message were coming but nothing.

Here it is the verbose output of the submission:
https://gist.github.com/msemelman/2367fe4899886e2f2e38c87edfe4e9a9
And what the master shows:
https://www.dropbox.com/s/afmhnn3ra9wndca/Screenshot%20Master.png?dl=1


Any help would be much appreciated, even if you told me where to debug, I'm
a bit lost here.

Thanks in advance

Re: SparkLauncher not receiving events

[Mariano Semelman <ma...@despegar.com>]

Solved,
tl;dr
I was using port: 6066 instead of 7077,


I got confused because of this message in the log when I submit to the
legacy port:

[info] - org.apache.spark.launcher.app.ActivitiesSortingAggregateJob -
16/09/26 11:43:27 WARN RestSubmissionClient: Unable to connect to server
spark://ponyo:7077.
[info] - org.apache.spark.launcher.app.ActivitiesSortingAggregateJob -
Warning: Master endpoint spark://ponyo:7077 was not a REST server. Falling
back to legacy submission gateway instead.

That's why I started using the 6066 (rest), that was a some time ago.

But it seems that for some unknown reason to me sparkLauncher listeners
don't work with the rest port, absolutely all the logs (ie: client, master,
worker, driver) don't show anything at all.

Well, I hope I didn't make anyone spend too much time on this and saved
some other soul a few days with this.
​​
​Mariano


On 26 September 2016 at 11:37, Mariano Semelman <
mariano.semelman@despegar.com> wrote:

> Hello,
>
> I'm having problems to receive events from the submited app. The app
> succesfuly submits, but the listener I'm passing to SparkLauncher is not
> receiving events.
>
> Spark Version: 1.6.1 (both client app and master)
>
> here are the relevant snippets I'm using in my code:
> https://gist.github.com/msemelman/d9d2b54ce0c01af8952bc298058dc0d5
>
> BTW the logger is correctly configured if you ask for that. I even
> debugged the spark code and put some breakpoints
> in org.apache.spark.launcher.LauncherServer.ServerConnection#handle to
> check if any message were coming but nothing.
>
> Here it is the verbose output of the submission:
> https://gist.github.com/msemelman/2367fe4899886e2f2e38c87edfe4e9a9
> And what the master shows:
> https://www.dropbox.com/s/afmhnn3ra9wndca/Screenshot%20Master.png?dl=1
>
>
> Any help would be much appreciated, even if you told me where to debug,
> I'm a bit lost here.
>
> Thanks in advance
>