You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@livy.apache.org by Wing Yew Poon <wy...@cloudera.com> on 2019/09/27 03:59:17 UTC

How to set Spark conf when using beeline to connect to Livy Thrift server

Hi,
I am trying out the Thrift sever functionality in Livy 0.6.
I use the beeline shipped with Hive to connect to the Livy Thrift
server. My Spark runs on YARN.
I am able to run simple queries on my Hive tables. However, I need to
know how to control configs for the Spark application that the Livy
session starts. In particular, how do I set configs such as
spark.driver.memory and spark.executor.memory? In my case, a query I
want to run dies because the AM exits due to exceeding the physical
memory limits (YARN kills the AM). The AM has the default 1.5 GB of
memory (1 GB + 384 MB, to the nearest 0.5 GB increment).
I tried

beeline -u 'jdbc:hive2://<livy_server>:<thrift_port>/<db>;?spark.driver.memory=4g'

but that doesn't seem to have the desired effect.
- Wing Yew