You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@livy.apache.org by js...@apache.org on 2017/06/27 06:39:06 UTC
[25/50] [abbrv] incubator-livy git commit: LIVY-90. Update conf dir
to use templates (#301)
LIVY-90. Update conf dir to use templates (#301)
- Added instructions to livy-client template.
- Filled out livy-client.conf.template.
- Updated conf naming to all camelCase, switched conf files to templates and updated .gitignore.
Project: http://git-wip-us.apache.org/repos/asf/incubator-livy/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-livy/commit/76d969e5
Tree: http://git-wip-us.apache.org/repos/asf/incubator-livy/tree/76d969e5
Diff: http://git-wip-us.apache.org/repos/asf/incubator-livy/diff/76d969e5
Branch: refs/heads/master
Commit: 76d969e58586f0fe2e707cc0bf81fdf595b01792
Parents: d74d5a9
Author: Alex Bozarth <aj...@us.ibm.com>
Authored: Fri Mar 3 16:51:58 2017 -0800
Committer: Alex Man <al...@users.noreply.github.com>
Committed: Fri Mar 3 16:51:58 2017 -0800
----------------------------------------------------------------------
.gitignore | 4 ++
conf/livy-client.conf.template | 86 +++++++++++++++++++++++++++++
conf/livy-env.sh | 33 ------------
conf/livy-env.sh.template | 33 ++++++++++++
conf/livy.conf | 96 ---------------------------------
conf/livy.conf.template | 96 +++++++++++++++++++++++++++++++++
conf/log4j.properties | 25 ---------
conf/log4j.properties.template | 25 +++++++++
conf/spark-blacklist.conf | 19 -------
conf/spark-blacklist.conf.template | 19 +++++++
10 files changed, 263 insertions(+), 173 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-livy/blob/76d969e5/.gitignore
----------------------------------------------------------------------
diff --git a/.gitignore b/.gitignore
index 798c94f..b5ad546 100644
--- a/.gitignore
+++ b/.gitignore
@@ -11,7 +11,11 @@
.project
.classpath
.scala_dependencies
+conf/*.conf
+conf/*.properties
+conf/*.sh
lib_managed/
+logs/
src_managed/
target/
reports/
http://git-wip-us.apache.org/repos/asf/incubator-livy/blob/76d969e5/conf/livy-client.conf.template
----------------------------------------------------------------------
diff --git a/conf/livy-client.conf.template b/conf/livy-client.conf.template
new file mode 100644
index 0000000..c0adb30
--- /dev/null
+++ b/conf/livy-client.conf.template
@@ -0,0 +1,86 @@
+#
+# Configurations for a Livy Client, any configurations set here will override any
+# livy or spark-default configurations.
+#
+# Before a Livy Client is able to load these configurations the folder containing
+# this file must be added to the application classpath
+#
+
+#
+# Configurations for Livy HTTPClient
+#
+
+# HTTP Request configurations
+# How long before a request times out
+# livy.client.http.connection.timeout = 10s
+# How long between data packets before a request times out
+# livy.client.http.connection.socket.timeout = 5m
+# Whether content is compressed
+# livy.client.http.content.compress.enable = true
+
+# How long before idle connections are closed
+# livy.client.http.connection.idle.timeout = 10m
+
+# Initial interval before polling for Job results
+# livy.client.http.job.initial_poll_interval = 100ms
+# Maximum interval between successive polls
+# livy.client.http.job.max_poll_interval = 5s
+
+#
+# Configurations for Livy RSCClient
+#
+
+# Configurations for registering a client with the rpc server
+# Unique client id for connections to the rpc server
+# livy.rsc.client.auth.id =
+# Secret value for authenticating client connections with server
+# livy.rsc.client.auth.secret =
+
+# Timeout when stopping a rsc client
+# livy.rsc.client.shutdown_timeout = 10s
+
+# Class of the rsc driver to use
+# livy.rsc.driver_class =
+# The kind of rsc session. Examples: pyspark or sparkr
+# livy.rsc.session.kind =
+
+# Comma-separated list of Livy RSC jars. By default Livy will upload jars from its installation
+# directory every time a session is started. By caching these files in HDFS, for example, startup
+# time of sessions on YARN can be reduced.
+# livy.rsc.jars =
+# Location of the SparkR package for running sparkr
+# livy.rsc.sparkr.package =
+# Location of the PySpark package for running pyspark
+# livy.rsc.pyspark.archives =
+
+# Address for the RSC driver to connect back with it's connection info.
+# livy.rsc.launcher.address =
+# livy.rsc.launcher.port = -1
+
+# How long will the RSC wait for a connection for a Livy server before shutting itself down.
+# livy.rsc.server.idle_timeout = 10m
+
+# The user that should be impersonated when requesting a Livy session
+# livy.rsc.proxy_user =
+
+# Host or IP adress of the rpc server
+# livy.rsc.rpc.server.address =
+# How long the rsc client will wait when attempting to connect to the Livy server
+# livy.rsc.server.connect.timeout = 90s
+
+# The logging level for the rpc channel. Possible values: TRACE, DEBUG, INFO, WARN, or ERROR
+# livy.rsc.channel.log.level =
+
+# SASL configurations for authentication
+# SASL mechanism used for authentication
+# livy.rsc.rpc.sasl.mechanisms = DIGEST-MD5
+# SASL qop used for authentication
+# livy.rsc.rpc.sasl.qop =
+
+# Time between status checks for cancelled a Job
+# livy.rsc.job_cancel.trigger_interval = 100ms
+# Time before a cancelled a Job is forced into a Cancelled state
+# livy.rsc.job_cancel.timeout = 30s
+
+# Number of statements kept in driver's memory
+# livy.rsc.retained_statements = 100
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/incubator-livy/blob/76d969e5/conf/livy-env.sh
----------------------------------------------------------------------
diff --git a/conf/livy-env.sh b/conf/livy-env.sh
deleted file mode 100644
index b348935..0000000
--- a/conf/livy-env.sh
+++ /dev/null
@@ -1,33 +0,0 @@
-#!/usr/bin/env bash
-#
-# Licensed to Cloudera, Inc. under one
-# or more contributor license agreements. See the NOTICE file
-# distributed with this work for additional information
-# regarding copyright ownership. Cloudera, Inc. licenses this file
-# to you under the Apache License, Version 2.0 (the
-# "License"); you may not use this file except in compliance
-# with the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-# LIVY ENVIRONMENT VARIABLES
-#
-# - JAVA_HOME Java runtime to use. By default use "java" from PATH.
-# - HADOOP_CONF_DIR Directory containing the Hadoop / YARN configuration to use.
-# - SPARK_HOME Spark which you would like to use in Livy.
-# - SPARK_CONF_DIR Optional directory where the Spark configuration lives.
-# (Default: $SPARK_HOME/conf)
-# - LIVY_LOG_DIR Where log files are stored. (Default: ${LIVY_HOME}/logs)
-# - LIVY_PID_DIR Where the pid file is stored. (Default: /tmp)
-# - LIVY_SERVER_JAVA_OPTS Java Opts for running livy server (You can set jvm related setting here,
-# like jvm memory/gc algorithm and etc.)
-# - LIVY_IDENT_STRING A name that identifies the Livy server instance, used to generate log file
-# names. (Default: name of the user starting Livy).
-# - LIVY_MAX_LOG_FILES Max number of log file to keep in the log directory. (Default: 5.)
-# - LIVY_NICENESS Niceness of the Livy server process when running in the background. (Default: 0.)
http://git-wip-us.apache.org/repos/asf/incubator-livy/blob/76d969e5/conf/livy-env.sh.template
----------------------------------------------------------------------
diff --git a/conf/livy-env.sh.template b/conf/livy-env.sh.template
new file mode 100644
index 0000000..b348935
--- /dev/null
+++ b/conf/livy-env.sh.template
@@ -0,0 +1,33 @@
+#!/usr/bin/env bash
+#
+# Licensed to Cloudera, Inc. under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. Cloudera, Inc. licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+# LIVY ENVIRONMENT VARIABLES
+#
+# - JAVA_HOME Java runtime to use. By default use "java" from PATH.
+# - HADOOP_CONF_DIR Directory containing the Hadoop / YARN configuration to use.
+# - SPARK_HOME Spark which you would like to use in Livy.
+# - SPARK_CONF_DIR Optional directory where the Spark configuration lives.
+# (Default: $SPARK_HOME/conf)
+# - LIVY_LOG_DIR Where log files are stored. (Default: ${LIVY_HOME}/logs)
+# - LIVY_PID_DIR Where the pid file is stored. (Default: /tmp)
+# - LIVY_SERVER_JAVA_OPTS Java Opts for running livy server (You can set jvm related setting here,
+# like jvm memory/gc algorithm and etc.)
+# - LIVY_IDENT_STRING A name that identifies the Livy server instance, used to generate log file
+# names. (Default: name of the user starting Livy).
+# - LIVY_MAX_LOG_FILES Max number of log file to keep in the log directory. (Default: 5.)
+# - LIVY_NICENESS Niceness of the Livy server process when running in the background. (Default: 0.)
http://git-wip-us.apache.org/repos/asf/incubator-livy/blob/76d969e5/conf/livy.conf
----------------------------------------------------------------------
diff --git a/conf/livy.conf b/conf/livy.conf
deleted file mode 100644
index c23aab2..0000000
--- a/conf/livy.conf
+++ /dev/null
@@ -1,96 +0,0 @@
-# Use this keystore for the SSL certificate and key.
-# livy.keystore =
-
-# Specify the keystore password.
-# livy.keystore.password =
-
-# What host address to start the server on. By default, Livy will bind to all network interfaces.
-# livy.server.host = 0.0.0.0
-
-# What port to start the server on.
-# livy.server.port = 8998
-
-# What spark master Livy sessions should use.
-# livy.spark.master = local
-
-# What spark deploy mode Livy sessions should use.
-# livy.spark.deployMode =
-
-# Enabled to check whether timeout Livy sessions should be stopped.
-# livy.server.session.timeout-check = true
-
-# Time in milliseconds on how long Livy will wait before timing out an idle session.
-# livy.server.session.timeout = 1h
-#
-# How long a finished session state should be kept in LivyServer for query.
-# livy.server.session.state-retain.sec = 600s
-
-# If livy should impersonate the requesting users when creating a new session.
-# livy.impersonation.enabled = true
-
-# Comma-separated list of Livy RSC jars. By default Livy will upload jars from its installation
-# directory every time a session is started. By caching these files in HDFS, for example, startup
-# time of sessions on YARN can be reduced.
-# livy.rsc.jars =
-
-# Comma-separated list of Livy REPL jars. By default Livy will upload jars from its installation
-# directory every time a session is started. By caching these files in HDFS, for example, startup
-# time of sessions on YARN can be reduced. Please list all the repl dependencies including
-# livy-repl_2.10 and livy-repl_2.11 jars, Livy will automatically pick the right dependencies in
-# session creation.
-# livy.repl.jars =
-
-# Location of PySpark archives. By default Livy will upload the file from SPARK_HOME, but
-# by caching the file in HDFS, startup time of PySpark sessions on YARN can be reduced.
-# livy.pyspark.archives =
-
-# Location of the SparkR package. By default Livy will upload the file from SPARK_HOME, but
-# by caching the file in HDFS, startup time of R sessions on YARN can be reduced.
-# livy.sparkr.package =
-
-# List of local directories from where files are allowed to be added to user sessions. By
-# default it's empty, meaning users can only reference remote URIs when starting their
-# sessions.
-# livy.file.local-dir-whitelist =
-
-# Whether to enable csrf protection, by default it is false. If it is enabled, client should add
-# http-header "X-Requested-By" in request if the http method is POST/DELETE/PUT/PATCH.
-# livy.server.csrf_protection.enabled =
-
-# Whether to enable HiveContext in livy interpreter, if it is true hive-site.xml will be detected
-# on user request and then livy server classpath automatically.
-# livy.repl.enableHiveContext =
-
-# Recovery mode of Livy. Possible values:
-# off: Default. Turn off recovery. Every time Livy shuts down, it stops and forgets all sessions.
-# recovery: Livy persists session info to the state store. When Livy restarts, it recovers
-# previous sessions from the state store.
-# Must set livy.server.recovery.state-store and livy.server.recovery.state-store.url to
-# configure the state store.
-# livy.server.recovery.mode = off
-
-# Where Livy should store state to for recovery. Possible values:
-# <empty>: Default. State store disabled.
-# filesystem: Store state on a file system.
-# zookeeper: Store state in a Zookeeper instance.
-# livy.server.recovery.state-store =
-
-# For filesystem state store, the path of the state store directory. Please don't use a filesystem
-# that doesn't support atomic rename (e.g. S3). e.g. file:///tmp/livy or hdfs:///.
-# For zookeeper, the address to the Zookeeper servers. e.g. host1:port1,host2:port2
-# livy.server.recovery.state-store.url =
-
-# If Livy can't find the yarn app within this time, consider it lost.
-# livy.server.yarn.app-lookup-timeout = 60s
-# When the cluster is busy, we may fail to launch yarn app in app-lookup-timeout, then it would
-# cause session leakage, so we need to check session leakage.
-# How long to check livy session leakage
-# livy.server.yarn.app-leakage.check_timeout = 600s
-# how often to check livy session leakage
-# livy.server.yarn.app-leakage.check_interval = 60s
-
-# How often Livy polls YARN to refresh YARN app state.
-# livy.server.yarn.poll-interval = 1s
-#
-# Days to keep Livy server request logs.
-# livy.server.request-log-retain.days = 5
http://git-wip-us.apache.org/repos/asf/incubator-livy/blob/76d969e5/conf/livy.conf.template
----------------------------------------------------------------------
diff --git a/conf/livy.conf.template b/conf/livy.conf.template
new file mode 100644
index 0000000..c23aab2
--- /dev/null
+++ b/conf/livy.conf.template
@@ -0,0 +1,96 @@
+# Use this keystore for the SSL certificate and key.
+# livy.keystore =
+
+# Specify the keystore password.
+# livy.keystore.password =
+
+# What host address to start the server on. By default, Livy will bind to all network interfaces.
+# livy.server.host = 0.0.0.0
+
+# What port to start the server on.
+# livy.server.port = 8998
+
+# What spark master Livy sessions should use.
+# livy.spark.master = local
+
+# What spark deploy mode Livy sessions should use.
+# livy.spark.deployMode =
+
+# Enabled to check whether timeout Livy sessions should be stopped.
+# livy.server.session.timeout-check = true
+
+# Time in milliseconds on how long Livy will wait before timing out an idle session.
+# livy.server.session.timeout = 1h
+#
+# How long a finished session state should be kept in LivyServer for query.
+# livy.server.session.state-retain.sec = 600s
+
+# If livy should impersonate the requesting users when creating a new session.
+# livy.impersonation.enabled = true
+
+# Comma-separated list of Livy RSC jars. By default Livy will upload jars from its installation
+# directory every time a session is started. By caching these files in HDFS, for example, startup
+# time of sessions on YARN can be reduced.
+# livy.rsc.jars =
+
+# Comma-separated list of Livy REPL jars. By default Livy will upload jars from its installation
+# directory every time a session is started. By caching these files in HDFS, for example, startup
+# time of sessions on YARN can be reduced. Please list all the repl dependencies including
+# livy-repl_2.10 and livy-repl_2.11 jars, Livy will automatically pick the right dependencies in
+# session creation.
+# livy.repl.jars =
+
+# Location of PySpark archives. By default Livy will upload the file from SPARK_HOME, but
+# by caching the file in HDFS, startup time of PySpark sessions on YARN can be reduced.
+# livy.pyspark.archives =
+
+# Location of the SparkR package. By default Livy will upload the file from SPARK_HOME, but
+# by caching the file in HDFS, startup time of R sessions on YARN can be reduced.
+# livy.sparkr.package =
+
+# List of local directories from where files are allowed to be added to user sessions. By
+# default it's empty, meaning users can only reference remote URIs when starting their
+# sessions.
+# livy.file.local-dir-whitelist =
+
+# Whether to enable csrf protection, by default it is false. If it is enabled, client should add
+# http-header "X-Requested-By" in request if the http method is POST/DELETE/PUT/PATCH.
+# livy.server.csrf_protection.enabled =
+
+# Whether to enable HiveContext in livy interpreter, if it is true hive-site.xml will be detected
+# on user request and then livy server classpath automatically.
+# livy.repl.enableHiveContext =
+
+# Recovery mode of Livy. Possible values:
+# off: Default. Turn off recovery. Every time Livy shuts down, it stops and forgets all sessions.
+# recovery: Livy persists session info to the state store. When Livy restarts, it recovers
+# previous sessions from the state store.
+# Must set livy.server.recovery.state-store and livy.server.recovery.state-store.url to
+# configure the state store.
+# livy.server.recovery.mode = off
+
+# Where Livy should store state to for recovery. Possible values:
+# <empty>: Default. State store disabled.
+# filesystem: Store state on a file system.
+# zookeeper: Store state in a Zookeeper instance.
+# livy.server.recovery.state-store =
+
+# For filesystem state store, the path of the state store directory. Please don't use a filesystem
+# that doesn't support atomic rename (e.g. S3). e.g. file:///tmp/livy or hdfs:///.
+# For zookeeper, the address to the Zookeeper servers. e.g. host1:port1,host2:port2
+# livy.server.recovery.state-store.url =
+
+# If Livy can't find the yarn app within this time, consider it lost.
+# livy.server.yarn.app-lookup-timeout = 60s
+# When the cluster is busy, we may fail to launch yarn app in app-lookup-timeout, then it would
+# cause session leakage, so we need to check session leakage.
+# How long to check livy session leakage
+# livy.server.yarn.app-leakage.check_timeout = 600s
+# how often to check livy session leakage
+# livy.server.yarn.app-leakage.check_interval = 60s
+
+# How often Livy polls YARN to refresh YARN app state.
+# livy.server.yarn.poll-interval = 1s
+#
+# Days to keep Livy server request logs.
+# livy.server.request-log-retain.days = 5
http://git-wip-us.apache.org/repos/asf/incubator-livy/blob/76d969e5/conf/log4j.properties
----------------------------------------------------------------------
diff --git a/conf/log4j.properties b/conf/log4j.properties
deleted file mode 100644
index f855899..0000000
--- a/conf/log4j.properties
+++ /dev/null
@@ -1,25 +0,0 @@
-#
-# Licensed to Cloudera, Inc. under one
-# or more contributor license agreements. See the NOTICE file
-# distributed with this work for additional information
-# regarding copyright ownership. Cloudera, Inc. licenses this file
-# to you under the Apache License, Version 2.0 (the
-# "License"); you may not use this file except in compliance
-# with the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-# The default Livy logging configuration.
-log4j.rootCategory=INFO, console
-log4j.appender.console=org.apache.log4j.ConsoleAppender
-log4j.appender.console.target=System.err
-log4j.appender.console.layout=org.apache.log4j.PatternLayout
-log4j.appender.console.layout.ConversionPattern=%d{yy/MM/dd HH:mm:ss} %p %c{1}: %m%n
-
-log4j.logger.org.eclipse.jetty=WARN
http://git-wip-us.apache.org/repos/asf/incubator-livy/blob/76d969e5/conf/log4j.properties.template
----------------------------------------------------------------------
diff --git a/conf/log4j.properties.template b/conf/log4j.properties.template
new file mode 100644
index 0000000..f855899
--- /dev/null
+++ b/conf/log4j.properties.template
@@ -0,0 +1,25 @@
+#
+# Licensed to Cloudera, Inc. under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. Cloudera, Inc. licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+# The default Livy logging configuration.
+log4j.rootCategory=INFO, console
+log4j.appender.console=org.apache.log4j.ConsoleAppender
+log4j.appender.console.target=System.err
+log4j.appender.console.layout=org.apache.log4j.PatternLayout
+log4j.appender.console.layout.ConversionPattern=%d{yy/MM/dd HH:mm:ss} %p %c{1}: %m%n
+
+log4j.logger.org.eclipse.jetty=WARN
http://git-wip-us.apache.org/repos/asf/incubator-livy/blob/76d969e5/conf/spark-blacklist.conf
----------------------------------------------------------------------
diff --git a/conf/spark-blacklist.conf b/conf/spark-blacklist.conf
deleted file mode 100644
index f0919b0..0000000
--- a/conf/spark-blacklist.conf
+++ /dev/null
@@ -1,19 +0,0 @@
-#
-# Configuration override / blacklist. Defines a list of properties that users are not allowed
-# to override when starting Spark sessions.
-#
-# This file takes a list of property names (one per line). Empty lines and lines starting with "#"
-# are ignored.
-#
-
-# Disallow overriding the master and the deploy mode.
-spark.master
-spark.submit.deployMode
-
-# Disallow overriding the location of Spark cached jars.
-spark.yarn.jar
-spark.yarn.jars
-spark.yarn.archive
-
-# Don't allow users to override the RSC timeout.
-livy.rsc.server.idle_timeout
http://git-wip-us.apache.org/repos/asf/incubator-livy/blob/76d969e5/conf/spark-blacklist.conf.template
----------------------------------------------------------------------
diff --git a/conf/spark-blacklist.conf.template b/conf/spark-blacklist.conf.template
new file mode 100644
index 0000000..f0919b0
--- /dev/null
+++ b/conf/spark-blacklist.conf.template
@@ -0,0 +1,19 @@
+#
+# Configuration override / blacklist. Defines a list of properties that users are not allowed
+# to override when starting Spark sessions.
+#
+# This file takes a list of property names (one per line). Empty lines and lines starting with "#"
+# are ignored.
+#
+
+# Disallow overriding the master and the deploy mode.
+spark.master
+spark.submit.deployMode
+
+# Disallow overriding the location of Spark cached jars.
+spark.yarn.jar
+spark.yarn.jars
+spark.yarn.archive
+
+# Don't allow users to override the RSC timeout.
+livy.rsc.server.idle_timeout