You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@impala.apache.org by ta...@apache.org on 2019/05/14 22:43:09 UTC
[impala] branch master updated (5a23bac -> d12675a)
This is an automated email from the ASF dual-hosted git repository.
tarmstrong pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/impala.git.
from 5a23bac IMPALA-8528: Refactor authorization check in AnalysisContext
new 559f19a fe: set classpath using maven dependency resolution
new d12675a IMPALA-8072: addendum: don't require fe rebuild for config
The 2 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
bin/set-classpath.sh | 18 ++++++++----------
bin/start-impala-cluster.py | 2 +-
docker/setup_build_context.py | 14 +++++++++-----
fe/pom.xml | 20 +++++++++++++++++++-
4 files changed, 37 insertions(+), 17 deletions(-)
[impala] 01/02: fe: set classpath using maven dependency resolution
Posted by ta...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
tarmstrong pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/impala.git
commit 559f19a5be95d686a71e616517ad4ae916c85824
Author: Todd Lipcon <to...@apache.org>
AuthorDate: Mon Apr 29 20:34:38 2019 -0700
fe: set classpath using maven dependency resolution
This changes the FE pom to generate a build classpath file in the
target/ directory. Then, bin/set-classpath.sh uses this file to generate
the classpath to start the cluster. This replaces the former approach of
including all of the jars found in target/dependency/
The advantage of this is that a clean build is no longer required when
switching artifact versions. Prior to this patch, if you changed an
artifact version and rebuilt, both the old and new artifact would be
left in the target/dependency/ directory and pollute the classpath.
This doesn't fully remove the target/dependency/ directory, because its
existence is likely important for downstream packaging of Impala. We can
likely assume that such packaging always does a clean build.
This also changes the set-classpath script to no longer load jars from
testdata/target/dependency/ since it appears that directory doesn't
actually get created during the build.
Change-Id: I103a1da10a54c7525ba7fb584d942ba1cb9fcb94
Reviewed-on: http://gerrit.cloudera.org:8080/13185
Tested-by: Impala Public Jenkins <im...@cloudera.com>
Reviewed-by: Todd Lipcon <to...@apache.org>
---
bin/set-classpath.sh | 18 ++++++++----------
docker/setup_build_context.py | 14 +++++++++-----
fe/pom.xml | 20 +++++++++++++++++++-
3 files changed, 36 insertions(+), 16 deletions(-)
diff --git a/bin/set-classpath.sh b/bin/set-classpath.sh
index 68aec2b..0f98533 100755
--- a/bin/set-classpath.sh
+++ b/bin/set-classpath.sh
@@ -31,16 +31,14 @@ CLASSPATH=\
"${HIVE_HOME}"/lib/datanucleus-core-3.2.2.jar:\
"${HIVE_HOME}"/lib/datanucleus-rdbms-3.2.1.jar:
-for jar in "${IMPALA_HOME}"/fe/target/dependency/*.jar; do
- if [ -e "$jar" ] ; then
- CLASSPATH="${CLASSPATH}:$jar"
- fi
-done
+FE_CP_FILE="$IMPALA_HOME/fe/target/build-classpath.txt"
-for jar in "${IMPALA_HOME}"/testdata/target/dependency/*.jar; do
- if [ -e "$jar" ] ; then
- CLASSPATH="${CLASSPATH}:$jar"
- fi
-done
+if [ ! -s "$FE_CP_FILE" ]; then
+ >&2 echo FE classpath file $FE_CP_FILE missing.
+ >&2 echo Build the front-end first.
+ exit 1
+fi
+
+CLASSPATH=$(cat "$IMPALA_HOME"/fe/target/build-classpath.txt):"$CLASSPATH"
export CLASSPATH
diff --git a/docker/setup_build_context.py b/docker/setup_build_context.py
index 0b50fb6..817263e 100755
--- a/docker/setup_build_context.py
+++ b/docker/setup_build_context.py
@@ -60,11 +60,15 @@ for lib in ["libstdc++", "libgcc"]:
symlink_file_into_dir(so, LIB_DIR)
os.symlink(os.environ["IMPALA_KUDU_HOME"], os.path.join(OUTPUT_DIR, "kudu"))
-# Impala jars and dependencies.
-for glob_pattern in [os.path.join(IMPALA_HOME, "fe/target/dependency/*.jar"),
- os.path.join(IMPALA_HOME, "fe/target/impala-frontend-*.jar")]:
- for jar in glob.glob(glob_pattern):
- symlink_file_into_dir(jar, LIB_DIR)
+# Impala dependencies.
+dep_classpath = file(os.path.join(IMPALA_HOME, "fe/target/build-classpath.txt")).read()
+for jar in dep_classpath.split(":"):
+ assert os.path.exists(jar), "missing jar from classpath: {0}".format(jar)
+ symlink_file_into_dir(jar, LIB_DIR)
+
+# Impala jars.
+for jar in glob.glob(os.path.join(IMPALA_HOME, "fe/target/impala-frontend-*.jar")):
+ symlink_file_into_dir(jar, LIB_DIR)
# Templates for debug web pages.
os.symlink(os.path.join(IMPALA_HOME, "www"), os.path.join(OUTPUT_DIR, "www"))
diff --git a/fe/pom.xml b/fe/pom.xml
index bdab1b7..62fc500 100644
--- a/fe/pom.xml
+++ b/fe/pom.xml
@@ -492,8 +492,11 @@ under the License.
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-dependency-plugin</artifactId>
- <version>2.10</version>
+ <version>3.1.1</version>
<executions>
+ <!-- TODO(todd): consider removing this execution or moving it to
+ some kind of 'dist' profile. No need to copy all of these jars
+ on every build of the FE! -->
<execution>
<id>copy-dependencies</id>
<phase>package</phase>
@@ -506,6 +509,21 @@ under the License.
<silent>true</silent>
</configuration>
</execution>
+ <!--
+ Write the runtime classpath to a file in the target directory
+ so it can be picked up by bin/set-classpath.sh
+ -->
+ <execution>
+ <id>write-classpath</id>
+ <goals>
+ <goal>build-classpath</goal>
+ </goals>
+ <configuration>
+ <outputFile>${project.build.directory}/build-classpath.txt</outputFile>
+ <includeScope>runtime</includeScope>
+ <excludeTypes>pom</excludeTypes>
+ </configuration>
+ </execution>
</executions>
</plugin>
[impala] 02/02: IMPALA-8072: addendum: don't require fe rebuild for
config
Posted by ta...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
tarmstrong pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/impala.git
commit d12675af59f2ac74db4fae09d41b720bfd72fe4b
Author: Tim Armstrong <ta...@cloudera.com>
AuthorDate: Wed May 8 16:55:11 2019 -0700
IMPALA-8072: addendum: don't require fe rebuild for config
Previously config changes wouldn't be picked up by containers until
maven copied the files from fe/src/test/resources to
fe/target/test-classes. This makes it more convenient - after running
./bin/create-test-configuration.sh new configs are picked up by
any newly-run containers.
Change-Id: I18f9f90667b1d16cf97d3e3f9fac400980d5b733
Reviewed-on: http://gerrit.cloudera.org:8080/13288
Reviewed-by: Impala Public Jenkins <im...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
---
bin/start-impala-cluster.py | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/bin/start-impala-cluster.py b/bin/start-impala-cluster.py
index b7df29b..2ac46b1 100755
--- a/bin/start-impala-cluster.py
+++ b/bin/start-impala-cluster.py
@@ -554,7 +554,7 @@ class DockerMiniClusterOperations(object):
container_name = self.__gen_container_name__(daemon, instance)
# Mount configuration into container so that we don't need to rebuild container
# for config changes to take effect.
- conf_dir = os.path.join(IMPALA_HOME, "fe/target/test-classes")
+ conf_dir = os.path.join(IMPALA_HOME, "fe/src/test/resources/")
mount_args = ["--mount", "type=bind,src={0},dst=/opt/impala/conf".format(conf_dir)]
# Allow loading LZO plugin, if built.