You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Night Wolf <ni...@gmail.com> on 2015/05/01 07:56:30 UTC
Spark SQL ThriftServer Impersonation Support
Hi guys,
Trying to use the SparkSQL Thriftserver with hive metastore. It seems that
hive meta impersonation works fine (when running Hive tasks). However
spinning up SparkSQL thrift server, impersonation doesn't seem to work...
What settings do I need to enable impersonation?
I've copied the same config as in my hive-site. Here is my launch command
for the spark thrift server;
--hiveconf hive.server2.enable.impersonation=true --hiveconf
hive.server2.enable.doAs=true --hiveconf hive.metastore.execute.setugi=true
Here is my full run script:
export HIVE_SERVER2_THRIFT_BIND_HOST=0.0.0.0
export HIVE_SERVER2_THRIFT_PORT=10000
export HIVE_CONF_DIR=/opt/mapr/hive/hive-0.13/conf/
export HIVE_HOME=/opt/mapr/hive/hive-0.13/
export HADOOP_HOME=/opt/mapr/hadoop/hadoop-2.5.1/
export HADOOP_CONF_DIR=/opt/mapr/hadoop/hadoop-2.5.1/etc/hadoop
export EXECUTOR_MEMORY=30g
export DRIVER_MEMORY=4g
export EXECUTOR_CORES=15
export NUM_EXECUTORS=20
export KRYO_BUFFER=512
export SPARK_DRIVER_MAXRESULTSIZE=4096
export HIVE_METASTORE_URIS=thrift://localhost:9083
export HIVE_METASTORE_WAREHOUSE_DIR=/user/hive/warehouse
export
SPARK_DIST_CLASSPATH=/opt/mapr/lib/*:/opt/mapr/hadoop/hadoop-2.5.1/share/hadoop/yarn/*:/opt/mapr/hadoop/hadoop-2.5.1/share/hadoop/common/lib/*:/opt/mapr/hive/hive-current/lib/*
export SPARK_LOG_DIR=/tmp/spark-log
export
SPARK_HISTORY_OPTS=-Dspark.history.fs.logDirectory=hdfs:///log/spark-events
export SPARK_CONF_DIR=/apps/spark/global-conf/
export SPARK_HOME=/apps/spark/spark-1.3.1-bin-mapr4.0.2_yarn_j6_2.10
export SPARK_LIBRARY_PATH=/opt/mapr/lib/*
export SPARK_JAVA_OPTS=-Djava.library.path=/opt/mapr/lib
$SPARK_HOME/*sbin/start-thriftserver.sh* --master yarn-client --jars
/opt/mapr/lib/libjpam.so --executor-memory $EXECUTOR_MEMORY --driver-memory
$DRIVER_MEMORY --executor-cores $EXECUTOR_CORES --num-executors
$NUM_EXECUTORS --conf spark.scheduler.mode=FAIR --conf
spark.kryoserializer.buffer.mb=$KRYO_BUFFER --conf
spark.serializer=org.apache.spark.serializer.KryoSerializer --conf
spark.files.useFetchCache=false --conf
spark.driver.maxResultSize=$SPARK_DRIVER_MAXRESULTSIZE --hiveconf
hive.metastore.uris=$HIVE_METASTORE_URIS --hiveconf
hive.metastore.warehouse.dir=$HIVE_METASTORE_WAREHOUSE_DIR --hiveconf
hive.server2.enable.impersonation=true --hiveconf
hive.server2.enable.doAs=true --hiveconf hive.metastore.execute.setugi=true
Cheers,
N
Re: Spark SQL ThriftServer Impersonation Support
Posted by Night Wolf <ni...@gmail.com>.
Thanks Andrew. What version of HS2 is the SparkSQL thrift server using?
What would be involved in updating? Is it a simple case of increasing the
deep version in one of the project POMs?
Cheers,
~N
On Sat, May 2, 2015 at 11:38 AM, Andrew Lee <al...@hotmail.com> wrote:
> Hi N,
>
> See: https://issues.apache.org/jira/browse/SPARK-5159
>
> I don't think it is yet supported until the HS2 code base is updated in
> Spark hive-thriftserver project.
>
> ------------------------------
> Date: Fri, 1 May 2015 15:56:30 +1000
> Subject: Spark SQL ThriftServer Impersonation Support
> From: nightwolfzor@gmail.com
> To: user@spark.apache.org
>
>
> Hi guys,
>
>
> Trying to use the SparkSQL Thriftserver with hive metastore. It seems that
> hive meta impersonation works fine (when running Hive tasks). However
> spinning up SparkSQL thrift server, impersonation doesn't seem to work...
>
> What settings do I need to enable impersonation?
>
> I've copied the same config as in my hive-site. Here is my launch command
> for the spark thrift server;
>
> --hiveconf hive.server2.enable.impersonation=true --hiveconf
> hive.server2.enable.doAs=true --hiveconf hive.metastore.execute.setugi=true
>
> Here is my full run script:
>
> export HIVE_SERVER2_THRIFT_BIND_HOST=0.0.0.0
> export HIVE_SERVER2_THRIFT_PORT=10000
>
> export HIVE_CONF_DIR=/opt/mapr/hive/hive-0.13/conf/
> export HIVE_HOME=/opt/mapr/hive/hive-0.13/
> export HADOOP_HOME=/opt/mapr/hadoop/hadoop-2.5.1/
> export HADOOP_CONF_DIR=/opt/mapr/hadoop/hadoop-2.5.1/etc/hadoop
>
> export EXECUTOR_MEMORY=30g
> export DRIVER_MEMORY=4g
> export EXECUTOR_CORES=15
> export NUM_EXECUTORS=20
> export KRYO_BUFFER=512
> export SPARK_DRIVER_MAXRESULTSIZE=4096
>
> export HIVE_METASTORE_URIS=thrift://localhost:9083
> export HIVE_METASTORE_WAREHOUSE_DIR=/user/hive/warehouse
>
> export
> SPARK_DIST_CLASSPATH=/opt/mapr/lib/*:/opt/mapr/hadoop/hadoop-2.5.1/share/hadoop/yarn/*:/opt/mapr/hadoop/hadoop-2.5.1/share/hadoop/common/lib/*:/opt/mapr/hive/hive-current/lib/*
> export SPARK_LOG_DIR=/tmp/spark-log
> export
> SPARK_HISTORY_OPTS=-Dspark.history.fs.logDirectory=hdfs:///log/spark-events
> export SPARK_CONF_DIR=/apps/spark/global-conf/
>
> export SPARK_HOME=/apps/spark/spark-1.3.1-bin-mapr4.0.2_yarn_j6_2.10
>
> export SPARK_LIBRARY_PATH=/opt/mapr/lib/*
> export SPARK_JAVA_OPTS=-Djava.library.path=/opt/mapr/lib
>
>
> $SPARK_HOME/*sbin/start-thriftserver.sh* --master yarn-client --jars
> /opt/mapr/lib/libjpam.so --executor-memory $EXECUTOR_MEMORY --driver-memory
> $DRIVER_MEMORY --executor-cores $EXECUTOR_CORES --num-executors
> $NUM_EXECUTORS --conf spark.scheduler.mode=FAIR --conf
> spark.kryoserializer.buffer.mb=$KRYO_BUFFER --conf
> spark.serializer=org.apache.spark.serializer.KryoSerializer --conf
> spark.files.useFetchCache=false --conf
> spark.driver.maxResultSize=$SPARK_DRIVER_MAXRESULTSIZE --hiveconf
> hive.metastore.uris=$HIVE_METASTORE_URIS --hiveconf
> hive.metastore.warehouse.dir=$HIVE_METASTORE_WAREHOUSE_DIR --hiveconf
> hive.server2.enable.impersonation=true --hiveconf
> hive.server2.enable.doAs=true --hiveconf hive.metastore.execute.setugi=true
>
>
> Cheers,
> N
>