Livy jobs keeps runing forever

[Joaquín Silva <jo...@akainix.com>]

Hello,

 

I'm new in Livy, and I have this problem.

 

When I run a batch job like this:

curl -X POST --data '{"file": "/user/spark/program.jar", "className": "my.program"}' -H "Content-Type: application/json" LIVY-HOST:8998/batches

 

The program keeps runing for ever.

Searching in the logs I found the reason:

 

17/07/25 14:21:46 ERROR yarn.ApplicationMaster: User class threw exception: java.lang.OutOfMemoryError: PermGen space

 

So in order to solve this issue I increased the executor and driver memory: "driverMemory":"15g","executorMemory":"15g". But I still seen this error.

 

 

So I tried running this job directly  using spark-submit and it worked perfectly, no memory error.

 

spark-submit --class my.program  --deploy-mode cluster --conf spark.master=yarn hdfs://HDFS_NN:8020/user/spark/program.jar

 

I'm using Livy 0.4.0-SNAPSHOT on Spark 2.1.0. 

 

 

Joaquín Silva | Pentagon Security & AKAINIX

Av. Kennedy 4.700, Piso 10, Of. 1002, Edificio New Century, Vitacura | Código Postal (ZIP Code) 7561127

Cel: (56-9) 6304 2498 

 

Re: Livy jobs keeps runing forever

[Meisam Fathi <me...@gmail.com>]

Livy garbage collects jobs. These setting in conf/livy.com are available to
control how Livy garbage collects sessions.
livy.server.session.timeout-check
ivy.server.session.timeout
livy.server.session.state-retain.sec

In our clusters, we have seen sometimes batch jobs fail to start and remain
in the starting state forever. With the current logic for garbage
collection in Livy, such jobs are never garbage collected. We modified the
garbage collection logic so that all jobs are eventually garbage collected
unless they are in running state.

Thanks,
Meisam

On Tue, Jul 25, 2017 at 11:00 AM Marcelo Vanzin <va...@cloudera.com> wrote:

> On Tue, Jul 25, 2017 at 8:35 AM, Joaquín Silva <jo...@akainix.com>
> wrote:
> > 17/07/25 14:21:46 ERROR yarn.ApplicationMaster: User class threw
> exception:
> > java.lang.OutOfMemoryError: PermGen space
> >
> > So in order to solve this issue I increased the executor and driver
> memory:
> > "driverMemory":"15g","executorMemory":"15g". But I still seen this error.
>
> That error won't be fixed by adding more memory; you need to set
> "XX:MaxPermSize=blah" to fix it, or use Java 8.
>
> Still it shouldn't cause the app to just hang, it should eventually
> fail. So perhaps there's a bug in Livy's error handling path
> somewhere.
>
> --
> Marcelo
>

Re: Livy jobs keeps runing forever

[Marcelo Vanzin <va...@cloudera.com>]

On Tue, Jul 25, 2017 at 8:35 AM, Joaquín Silva <jo...@akainix.com> wrote:
> 17/07/25 14:21:46 ERROR yarn.ApplicationMaster: User class threw exception:
> java.lang.OutOfMemoryError: PermGen space
>
> So in order to solve this issue I increased the executor and driver memory:
> "driverMemory":"15g","executorMemory":"15g". But I still seen this error.

That error won't be fixed by adding more memory; you need to set
"XX:MaxPermSize=blah" to fix it, or use Java 8.

Still it shouldn't cause the app to just hang, it should eventually
fail. So perhaps there's a bug in Livy's error handling path
somewhere.

-- 
Marcelo