You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Stephen Boesch <ja...@gmail.com> on 2016/11/24 03:36:31 UTC
Invalid log directory running pyspark job
For a pyspark job with 54 executors all of the task outputs have a single
line in both the stderr and stdout similar to:
Error: invalid log directory /shared/sparkmaven/work/app-20161119222540-0000/0/
Note: the directory /shared/sparkmaven/work exists and is owned by the same
user running the job. There are plenty of other app-*** subdirectories that
do have contents in the stdout/stderr files.
$ls -lrta /shared/sparkmaven/work
total 0
drwxr-xr-x 59 steve staff 2006 Nov 23 05:01 ..
drwxr-xr-x 41 steve staff 1394 Nov 23 18:22 app-20161123050122-0002
drwxr-xr-x 6 steve staff 204 Nov 23 18:22 app-20161123182031-0005
drwxr-xr-x 6 steve staff 204 Nov 23 18:44 app-20161123184349-0006
drwxr-xr-x 6 steve staff 204 Nov 23 18:46 app-20161123184613-0007
drwxr-xr-x 3 steve staff 102 Nov 23 19:20 app-20161123192048-0008
Here is a sample of the contents
/shared/sparkmaven/work/app-20161123184613-0007/2:
total 16
-rw-r--r-- 1 steve staff 0 Nov 23 18:46 stdout
drwxr-xr-x 4 steve staff 136 Nov 23 18:46 .
-rw-r--r-- 1 steve staff 4830 Nov 23 18:46 stderr
drwxr-xr-x 6 steve staff 204 Nov 23 18:46 ..
/shared/sparkmaven/work/app-20161123184613-0007/3:
total 16
-rw-r--r-- 1 steve staff 0 Nov 23 18:46 stdout
drwxr-xr-x 6 steve staff 204 Nov 23 18:46 ..
drwxr-xr-x 4 steve staff 136 Nov 23 18:46 .
-rw-r--r-- 1 steve staff 4830 Nov 23 18:46 stderr
Note also: the *SparkPI* program does run succesfully - which validates
the basic spark installation/functionality.
Re: Invalid log directory running pyspark job
Posted by Stephen Boesch <ja...@gmail.com>.
This problem appears to be a regression on HEAD/master: when running
against 2.0.2 the pyspark job completes successfully including running
predictions.
2016-11-23 19:36 GMT-08:00 Stephen Boesch <ja...@gmail.com>:
>
> For a pyspark job with 54 executors all of the task outputs have a single
> line in both the stderr and stdout similar to:
>
> Error: invalid log directory /shared/sparkmaven/work/app-20161119222540-0000/0/
>
>
> Note: the directory /shared/sparkmaven/work exists and is owned by the
> same user running the job. There are plenty of other app-*** subdirectories
> that do have contents in the stdout/stderr files.
>
>
> $ls -lrta /shared/sparkmaven/work
> total 0
> drwxr-xr-x 59 steve staff 2006 Nov 23 05:01 ..
> drwxr-xr-x 41 steve staff 1394 Nov 23 18:22 app-20161123050122-0002
> drwxr-xr-x 6 steve staff 204 Nov 23 18:22 app-20161123182031-0005
> drwxr-xr-x 6 steve staff 204 Nov 23 18:44 app-20161123184349-0006
> drwxr-xr-x 6 steve staff 204 Nov 23 18:46 app-20161123184613-0007
> drwxr-xr-x 3 steve staff 102 Nov 23 19:20 app-20161123192048-0008
>
>
>
> Here is a sample of the contents
>
> /shared/sparkmaven/work/app-20161123184613-0007/2:
> total 16
> -rw-r--r-- 1 steve staff 0 Nov 23 18:46 stdout
> drwxr-xr-x 4 steve staff 136 Nov 23 18:46 .
> -rw-r--r-- 1 steve staff 4830 Nov 23 18:46 stderr
> drwxr-xr-x 6 steve staff 204 Nov 23 18:46 ..
>
> /shared/sparkmaven/work/app-20161123184613-0007/3:
> total 16
> -rw-r--r-- 1 steve staff 0 Nov 23 18:46 stdout
> drwxr-xr-x 6 steve staff 204 Nov 23 18:46 ..
> drwxr-xr-x 4 steve staff 136 Nov 23 18:46 .
> -rw-r--r-- 1 steve staff 4830 Nov 23 18:46 stderr
>
>
> Note also: the *SparkPI* program does run succesfully - which validates
> the basic spark installation/functionality.
>
>