stdout in workers

[Jim Carroll <ji...@gmail.com>]

I'm experimenting with a few things trying to understand how it's working. I
took the JavaSparkPi example as a starting point and added a few System.out
lines.

I added a system.out to the main body of the driver program (not inside of
any Functions).
I added another to the mapper.
I added another to the reducer.

I set up a simple "standalone" distribution with a 1 master (no zookeeper)
and a 1 worker.

The main println and the reducer println print out from the driver program.

The mapper one doesn't print in the shell I started the worker in, nor does
it appear in any logs (in case stdout was redirected).

However, the program executes (and the mapper executes on the worker) since
I can see the job run in the logs and I get an answer.

Where should I be looking?

Thanks. Apologies if this is a dumb question - I searched for answer but
only found a reference to another list posting where the user thought his
driver print statement should have printed on the worker.

Jim




--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/stdout-in-workers-tp4537.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

Re: stdout in workers

[Daniel Darabos <da...@lynxanalytics.com>]

On Mon, Apr 21, 2014 at 7:59 PM, Jim Carroll <ji...@gmail.com> wrote:

>
> I'm experimenting with a few things trying to understand how it's working.
> I
> took the JavaSparkPi example as a starting point and added a few System.out
> lines.
>
> I added a system.out to the main body of the driver program (not inside of
> any Functions).
> I added another to the mapper.
> I added another to the reducer.
>
> I set up a simple "standalone" distribution with a 1 master (no zookeeper)
> and a 1 worker.
>
> The main println and the reducer println print out from the driver program.
>
> The mapper one doesn't print in the shell I started the worker in, nor does
> it appear in any logs (in case stdout was redirected).
>
> However, the program executes (and the mapper executes on the worker) since
> I can see the job run in the logs and I get an answer.
>
> Where should I be looking?
>

It is in the executor logs. The executor is a process started by the worker
for your application, which runs your custom code. So the logs are on the
worker machines. The default location is within the Spark directory under
the "work" subdirectory. For example
~/spark-0.9.0-incubating/work/app-20140418164503-0025/0/stdout.

It is often much easier to access this log from the UI. Click on the
application ID on the master, or on a worker, and then you see the
stdout/stderr links.

Thanks. Apologies if this is a dumb question - I searched for answer but
> only found a reference to another list posting where the user thought his
> driver print statement should have printed on the worker.
>

I think it's a perfectly good question, and you might want to make a
suggestion for adding it to the documentation somewhere.

Jim
>
>
>
>
> --
> View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/stdout-in-workers-tp4537.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>