[GitHub] [hudi] stym06 commented on issue #3747: [SUPPORT] Hive Sync process stuck and unable to exit

[GitBox <gi...@apache.org>]

stym06 commented on issue #3747:
URL: https://github.com/apache/hudi/issues/3747#issuecomment-940130218


   1. Apache Hive
   2. Attaching the file (Have altered the provided sh file to include all Hive libraries as it was not working with Hive 3.1.2)
   3. I will check this and revert.
   
   
   ###hudi-cli_run_sync.sh
   ```
   #!/usr/bin/env bash
   
   #  Licensed to the Apache Software Foundation (ASF) under one
   #  or more contributor license agreements.  See the NOTICE file
   #  distributed with this work for additional information
   #  regarding copyright ownership.  The ASF licenses this file
   #  to you under the Apache License, Version 2.0 (the
   #  "License"); you may not use this file except in compliance
   #  with the License.  You may obtain a copy of the License at
   #
   #      http://www.apache.org/licenses/LICENSE-2.0
   #
   #  Unless required by applicable law or agreed to in writing, software
   #  distributed under the License is distributed on an "AS IS" BASIS,
   #  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
   #  See the License for the specific language governing permissions and
   # limitations under the License.
   
   function error_exit {
       echo "$1" >&2   ## Send message to stderr. Exclude >&2 if you don't want it that way.
       exit "${2:-1}"  ## Return a code specified by $2 or 1 by default.
   }
   
   if [ -z "${HADOOP_HOME}" ]; then
     error_exit "Please make sure the environment variable HADOOP_HOME is setup"
   fi
   
   if [ -z "${HIVE_HOME}" ]; then
     error_exit "Please make sure the environment variable HIVE_HOME is setup"
   fi
   
   DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"
   #Ensure we pick the right jar even for hive11 builds
   HUDI_HIVE_UBER_JAR=`ls -c $DIR/hudi-hive-sync-*.jar | grep -v source | head -1`
   
   if [ -z "$HADOOP_CONF_DIR" ]; then
     echo "setting hadoop conf dir"
     HADOOP_CONF_DIR="${HADOOP_HOME}/etc/hadoop"
   fi
   
   ## Include only specific packages from HIVE_HOME/lib to avoid version mismatches
   HIVE_EXEC=`ls ${HIVE_HOME}/lib/hive-exec-*.jar | tr '\n' ':'`
   HIVE_SERVICE=`ls ${HIVE_HOME}/lib/hive-service-*.jar | grep -v rpc | tr '\n' ':'`
   HIVE_METASTORE=`ls ${HIVE_HOME}/lib/hive-metastore-*.jar | tr '\n' ':'`
   HIVE_JDBC=`ls ${HIVE_HOME}/lib/hive-jdbc-*.jar | tr '\n' ':'`
   if [ -z "${HIVE_JDBC}" ]; then
     HIVE_JDBC=`ls ${HIVE_HOME}/lib/hive-jdbc-*.jar | grep -v handler | tr '\n' ':'`
   fi
   HIVE_JACKSON=`ls ${HIVE_HOME}/lib/jackson-*.jar | tr '\n' ':'`
   HIVE_JARS=$HIVE_METASTORE:$HIVE_SERVICE:$HIVE_EXEC:$HIVE_JDBC:$HIVE_JACKSON
   
   HIVE_CALCITE=`ls ${HIVE_HOME}/lib/calcite-*.jar | tr '\n' ':'`
   if [ -n "$HIVE_CALCITE" ]; then
       HIVE_JARS=$HIVE_JARS:$HIVE_CALCITE
   fi
   HIVE_LIBFB303=`ls ${HIVE_HOME}/lib/libfb303-*.jar | tr '\n' ':'`
   if [ -n "$HIVE_LIBFB303" ]; then
       HIVE_JARS=$HIVE_JARS:$HIVE_LIBFB303
   fi
   HIVE_DATANUCLEUS=`ls ${HIVE_HOME}/lib/datanucleus-*.jar | tr '\n' ':'`
   HIVE_HIKARI=`ls ${HIVE_HOME}/lib/HikariCP-2.6.1.jar | tr '\n' ':'`
   HIVE_JARS=$HIVE_JARS:$HIVE_DATANUCLEUS:$HIVE_HIKARI
   
   HUDI_MR_BUNDLE=`ls ${HIVE_HOME}/lib/hudi-hadoop-*.jar | tr '\n' ':'`
   HIVE_JARS=$HIVE_JARS:$HUDI_MR_BUNDLE
   
   HADOOP_HIVE_JARS=${HIVE_JARS}:${HADOOP_HOME}/share/hadoop/common/*:${HADOOP_HOME}/share/hadoop/mapreduce/*:${HADOOP_HOME}/share/hadoop/hdfs/*:${HADOOP_HOME}/share/hadoop/common/lib/*:${HADOOP_HOME}/share/hadoop/hdfs/lib/*
   
   echo "Running Command : java -cp ${HADOOP_HIVE_JARS}:${HADOOP_CONF_DIR}:$HUDI_HIVE_UBER_JAR org.apache.hudi.hive.HiveSyncTool $@"
   java -cp $HUDI_HIVE_UBER_JAR:${HADOOP_HIVE_JARS}:${HADOOP_CONF_DIR}:${HIVE_HOME}/lib/*: org.apache.hudi.hive.HiveSyncTool "$@"
   
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org