You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2021/10/04 03:32:40 UTC
[GitHub] [hudi] stym06 opened a new issue #3747: [SUPPORT] Hive Sync process stuck and unable to exit
stym06 opened a new issue #3747:
URL: https://github.com/apache/hudi/issues/3747
**Describe the problem you faced**
I'm running the hive sync process to sync the Hudi data stored in the Azure blob storage, written by the Deltastreamer tool.
The problem is that this process is unable to exit after the sync completes and partitions etc are added.
**To Reproduce**
Steps to reproduce the behavior:
1. Run Hive sync independently with Hudi data written to Blob
**Expected behavior**
Hive sync process should complete as expected and exit.
**Environment Description**
* Hudi version : 0.9.0-SNAPSHOT
* Spark version : 2.4.4
* Hive version : 3.1.2
* Hadoop version : 3.2.0
* Storage (HDFS/S3/GCS..) : Azure Blob
* Running on Docker? (yes/no) : K8s
**Additional context**
I'm running the hive metastore and hive server as separate services on a K8s cluster, using the below repo images: https://github.com/joshuarobinson/trino-on-k8s/blob/master/hive_metastore/Dockerfile
**Stacktrace**
```
2021-09-22 14:56:18,313 INFO [main] utils.MetaStoreUtils (MetaStoreUtils.java:updateTableStatsSlow(705)) - Updated size of table hmi_quectel_battery_packet_v1_test to 5784679
2021-09-22 14:56:18,341 INFO [main] hive.HiveSyncTool (HiveSyncTool.java:syncHoodieTable(197)) - Sync complete for hmi_quectel_battery_packet_v1_test
2021-09-22 14:56:18,342 INFO [main] metastore.HiveMetaStore (HiveMetaStore.java:logInfo(895)) - 0: Cleaning up thread local RawStore...
2021-09-22 14:56:18,342 INFO [main] HiveMetaStore.audit (HiveMetaStore.java:logAuditEvent(347)) - ugi=root ip=unknown-ip-addr cmd=Cleaning up thread local RawStore...
2021-09-22 14:56:18,343 INFO [main] metastore.HiveMetaStore (HiveMetaStore.java:logInfo(895)) - 0: Done cleaning up thread local RawStore
2021-09-22 14:56:18,343 INFO [main] HiveMetaStore.audit (HiveMetaStore.java:logAuditEvent(347)) - ugi=root ip=unknown-ip-addr cmd=Done cleaning up thread local RawStore
2021-09-22 14:56:18,343 INFO [main] metastore.HiveMetaStore (HiveMetaStore.java:logInfo(895)) - 0: Done cleaning up thread local RawStore
2021-09-22 14:56:18,343 INFO [main] HiveMetaStore.audit (HiveMetaStore.java:logAuditEvent(347)) - ugi=root ip=unknown-ip-addr cmd=Done cleaning up thread local RawStore
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] codope commented on issue #3747: [SUPPORT] Hive Sync process stuck and unable to exit
Posted by GitBox <gi...@apache.org>.
codope commented on issue #3747:
URL: https://github.com/apache/hudi/issues/3747#issuecomment-939890785
@stym06 I am not able to reproduce this issue locally or with S3 storage. Can you please clarify a couple of things:
1. Is this with Apache Hive and Apache Spark?
2. Can you share the `hudi-cli_run_sync` script? Typically, we use `run_sync_tool` script. I am wondering what's different.
3. Did you try hive sync with Hive 2.3.x ? If not, can you please give it a shot?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] stym06 commented on issue #3747: [SUPPORT] Hive Sync process stuck and unable to exit
Posted by GitBox <gi...@apache.org>.
stym06 commented on issue #3747:
URL: https://github.com/apache/hudi/issues/3747#issuecomment-1018677357
@nsivabalan we have a usecase where we needed to do hudisync for a lot of tables, and when we were using sync in the deltastreamer, every job ended up creating jdbc connection pool with hive metastore that caused lots of connection timeout errors. so, i ended up rewriting a tool to do the sync for all tables in a loop that used the thrift client instead of jdbc.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on issue #3747: [SUPPORT] Hive Sync process stuck and unable to exit
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on issue #3747:
URL: https://github.com/apache/hudi/issues/3747#issuecomment-997073235
@stym06 : are you still stuck at the issue. Can you enable debug logs while doing the hive sync and provide us the same. Is it just that your process is stuck, but all data is synced in good shape or do you see issues w/ sync as well ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on issue #3747: [SUPPORT] Hive Sync process stuck and unable to exit
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on issue #3747:
URL: https://github.com/apache/hudi/issues/3747#issuecomment-997073516
we don't have azure wasb FS atm. need to set up one to repro.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] stym06 commented on issue #3747: [SUPPORT] Hive Sync process stuck and unable to exit
Posted by GitBox <gi...@apache.org>.
stym06 commented on issue #3747:
URL: https://github.com/apache/hudi/issues/3747#issuecomment-1020991975
How can I enable debug logs in the run_sync_tool.sh ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xushiyan commented on issue #3747: [SUPPORT] Hive Sync process stuck and unable to exit
Posted by GitBox <gi...@apache.org>.
xushiyan commented on issue #3747:
URL: https://github.com/apache/hudi/issues/3747#issuecomment-948227548
@stym06 noticed that you used 0.9.0-SNAPSHOT is this a custom build version? is it close to release-0.9.0 ? please let use know how it goes with hive 2.3.x, since current Hudi version is built against hive 2.3.1 and spark 2.4.4. thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] stym06 commented on issue #3747: [SUPPORT] Hive Sync process stuck and unable to exit
Posted by GitBox <gi...@apache.org>.
stym06 commented on issue #3747:
URL: https://github.com/apache/hudi/issues/3747#issuecomment-940130218
1. Apache Hive
2. Attaching the file (Have altered the provided sh file to include all Hive libraries as it was not working with Hive 3.1.2)
3. I will check this and revert.
###hudi-cli_run_sync.sh
```
#!/usr/bin/env bash
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
function error_exit {
echo "$1" >&2 ## Send message to stderr. Exclude >&2 if you don't want it that way.
exit "${2:-1}" ## Return a code specified by $2 or 1 by default.
}
if [ -z "${HADOOP_HOME}" ]; then
error_exit "Please make sure the environment variable HADOOP_HOME is setup"
fi
if [ -z "${HIVE_HOME}" ]; then
error_exit "Please make sure the environment variable HIVE_HOME is setup"
fi
DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"
#Ensure we pick the right jar even for hive11 builds
HUDI_HIVE_UBER_JAR=`ls -c $DIR/hudi-hive-sync-*.jar | grep -v source | head -1`
if [ -z "$HADOOP_CONF_DIR" ]; then
echo "setting hadoop conf dir"
HADOOP_CONF_DIR="${HADOOP_HOME}/etc/hadoop"
fi
## Include only specific packages from HIVE_HOME/lib to avoid version mismatches
HIVE_EXEC=`ls ${HIVE_HOME}/lib/hive-exec-*.jar | tr '\n' ':'`
HIVE_SERVICE=`ls ${HIVE_HOME}/lib/hive-service-*.jar | grep -v rpc | tr '\n' ':'`
HIVE_METASTORE=`ls ${HIVE_HOME}/lib/hive-metastore-*.jar | tr '\n' ':'`
HIVE_JDBC=`ls ${HIVE_HOME}/lib/hive-jdbc-*.jar | tr '\n' ':'`
if [ -z "${HIVE_JDBC}" ]; then
HIVE_JDBC=`ls ${HIVE_HOME}/lib/hive-jdbc-*.jar | grep -v handler | tr '\n' ':'`
fi
HIVE_JACKSON=`ls ${HIVE_HOME}/lib/jackson-*.jar | tr '\n' ':'`
HIVE_JARS=$HIVE_METASTORE:$HIVE_SERVICE:$HIVE_EXEC:$HIVE_JDBC:$HIVE_JACKSON
HIVE_CALCITE=`ls ${HIVE_HOME}/lib/calcite-*.jar | tr '\n' ':'`
if [ -n "$HIVE_CALCITE" ]; then
HIVE_JARS=$HIVE_JARS:$HIVE_CALCITE
fi
HIVE_LIBFB303=`ls ${HIVE_HOME}/lib/libfb303-*.jar | tr '\n' ':'`
if [ -n "$HIVE_LIBFB303" ]; then
HIVE_JARS=$HIVE_JARS:$HIVE_LIBFB303
fi
HIVE_DATANUCLEUS=`ls ${HIVE_HOME}/lib/datanucleus-*.jar | tr '\n' ':'`
HIVE_HIKARI=`ls ${HIVE_HOME}/lib/HikariCP-2.6.1.jar | tr '\n' ':'`
HIVE_JARS=$HIVE_JARS:$HIVE_DATANUCLEUS:$HIVE_HIKARI
HUDI_MR_BUNDLE=`ls ${HIVE_HOME}/lib/hudi-hadoop-*.jar | tr '\n' ':'`
HIVE_JARS=$HIVE_JARS:$HUDI_MR_BUNDLE
HADOOP_HIVE_JARS=${HIVE_JARS}:${HADOOP_HOME}/share/hadoop/common/*:${HADOOP_HOME}/share/hadoop/mapreduce/*:${HADOOP_HOME}/share/hadoop/hdfs/*:${HADOOP_HOME}/share/hadoop/common/lib/*:${HADOOP_HOME}/share/hadoop/hdfs/lib/*
echo "Running Command : java -cp ${HADOOP_HIVE_JARS}:${HADOOP_CONF_DIR}:$HUDI_HIVE_UBER_JAR org.apache.hudi.hive.HiveSyncTool $@"
java -cp $HUDI_HIVE_UBER_JAR:${HADOOP_HIVE_JARS}:${HADOOP_CONF_DIR}:${HIVE_HOME}/lib/*: org.apache.hudi.hive.HiveSyncTool "$@"
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] stym06 commented on issue #3747: [SUPPORT] Hive Sync process stuck and unable to exit
Posted by GitBox <gi...@apache.org>.
stym06 commented on issue #3747:
URL: https://github.com/apache/hudi/issues/3747#issuecomment-1020991975
How can I enable debug logs in the run_sync_tool.sh ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xushiyan closed issue #3747: [SUPPORT] Hive Sync process stuck and unable to exit
Posted by GitBox <gi...@apache.org>.
xushiyan closed issue #3747:
URL: https://github.com/apache/hudi/issues/3747
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on issue #3747: [SUPPORT] Hive Sync process stuck and unable to exit
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on issue #3747:
URL: https://github.com/apache/hudi/issues/3747#issuecomment-1020597982
got it. Can you enable debug logs and share with us.
wrt thrift client, I did saw your patch. appreciative of you. will review it by this week.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on issue #3747: [SUPPORT] Hive Sync process stuck and unable to exit
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on issue #3747:
URL: https://github.com/apache/hudi/issues/3747#issuecomment-1018632506
also, interested to know, why not enable hive sync along w/ deltastreamer configs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on issue #3747: [SUPPORT] Hive Sync process stuck and unable to exit
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on issue #3747:
URL: https://github.com/apache/hudi/issues/3747#issuecomment-1020597982
got it. Can you enable debug logs and share with us.
wrt thrift client, I did saw your patch. appreciative of you. will review it by this week.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on issue #3747: [SUPPORT] Hive Sync process stuck and unable to exit
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on issue #3747:
URL: https://github.com/apache/hudi/issues/3747#issuecomment-935136426
@codope can you follow up on this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xushiyan commented on issue #3747: [SUPPORT] Hive Sync process stuck and unable to exit
Posted by GitBox <gi...@apache.org>.
xushiyan commented on issue #3747:
URL: https://github.com/apache/hudi/issues/3747#issuecomment-1025303976
@stym06 thanks for filing https://issues.apache.org/jira/browse/HUDI-2733 and the patch. Let's continue collaborating from there.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on issue #3747: [SUPPORT] Hive Sync process stuck and unable to exit
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on issue #3747:
URL: https://github.com/apache/hudi/issues/3747#issuecomment-1018631854
@stym06 : can we have any updates on this regard please.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] stym06 commented on issue #3747: [SUPPORT] Hive Sync process stuck and unable to exit
Posted by GitBox <gi...@apache.org>.
stym06 commented on issue #3747:
URL: https://github.com/apache/hudi/issues/3747#issuecomment-948283325
@xushiyan thanks for reverting. I have set up a standalone hive metastore (https://repo1.maven.org/maven2/org/apache/hive/hive-standalone-metastore/3.1.2/hive-standalone-metastore-3.1.2-bin.tar.gz) and am trying to run this sync job in a separate container in a K8s environment.
Can you please let me know the right way to sync this Hoodie table? The base path is Azure wasb filesystem
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] stym06 edited a comment on issue #3747: [SUPPORT] Hive Sync process stuck and unable to exit
Posted by GitBox <gi...@apache.org>.
stym06 edited a comment on issue #3747:
URL: https://github.com/apache/hudi/issues/3747#issuecomment-940130218
1. Apache Hive
2. Attaching the file (Have altered the provided sh file to include all Hive libraries as it was not working with Hive 3.1.2)
3. I will check this and revert.
### `hudi-cli_run_sync.sh`
```
#!/usr/bin/env bash
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
function error_exit {
echo "$1" >&2 ## Send message to stderr. Exclude >&2 if you don't want it that way.
exit "${2:-1}" ## Return a code specified by $2 or 1 by default.
}
if [ -z "${HADOOP_HOME}" ]; then
error_exit "Please make sure the environment variable HADOOP_HOME is setup"
fi
if [ -z "${HIVE_HOME}" ]; then
error_exit "Please make sure the environment variable HIVE_HOME is setup"
fi
DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"
#Ensure we pick the right jar even for hive11 builds
HUDI_HIVE_UBER_JAR=`ls -c $DIR/hudi-hive-sync-*.jar | grep -v source | head -1`
if [ -z "$HADOOP_CONF_DIR" ]; then
echo "setting hadoop conf dir"
HADOOP_CONF_DIR="${HADOOP_HOME}/etc/hadoop"
fi
## Include only specific packages from HIVE_HOME/lib to avoid version mismatches
HIVE_EXEC=`ls ${HIVE_HOME}/lib/hive-exec-*.jar | tr '\n' ':'`
HIVE_SERVICE=`ls ${HIVE_HOME}/lib/hive-service-*.jar | grep -v rpc | tr '\n' ':'`
HIVE_METASTORE=`ls ${HIVE_HOME}/lib/hive-metastore-*.jar | tr '\n' ':'`
HIVE_JDBC=`ls ${HIVE_HOME}/lib/hive-jdbc-*.jar | tr '\n' ':'`
if [ -z "${HIVE_JDBC}" ]; then
HIVE_JDBC=`ls ${HIVE_HOME}/lib/hive-jdbc-*.jar | grep -v handler | tr '\n' ':'`
fi
HIVE_JACKSON=`ls ${HIVE_HOME}/lib/jackson-*.jar | tr '\n' ':'`
HIVE_JARS=$HIVE_METASTORE:$HIVE_SERVICE:$HIVE_EXEC:$HIVE_JDBC:$HIVE_JACKSON
HIVE_CALCITE=`ls ${HIVE_HOME}/lib/calcite-*.jar | tr '\n' ':'`
if [ -n "$HIVE_CALCITE" ]; then
HIVE_JARS=$HIVE_JARS:$HIVE_CALCITE
fi
HIVE_LIBFB303=`ls ${HIVE_HOME}/lib/libfb303-*.jar | tr '\n' ':'`
if [ -n "$HIVE_LIBFB303" ]; then
HIVE_JARS=$HIVE_JARS:$HIVE_LIBFB303
fi
HIVE_DATANUCLEUS=`ls ${HIVE_HOME}/lib/datanucleus-*.jar | tr '\n' ':'`
HIVE_HIKARI=`ls ${HIVE_HOME}/lib/HikariCP-2.6.1.jar | tr '\n' ':'`
HIVE_JARS=$HIVE_JARS:$HIVE_DATANUCLEUS:$HIVE_HIKARI
HUDI_MR_BUNDLE=`ls ${HIVE_HOME}/lib/hudi-hadoop-*.jar | tr '\n' ':'`
HIVE_JARS=$HIVE_JARS:$HUDI_MR_BUNDLE
HADOOP_HIVE_JARS=${HIVE_JARS}:${HADOOP_HOME}/share/hadoop/common/*:${HADOOP_HOME}/share/hadoop/mapreduce/*:${HADOOP_HOME}/share/hadoop/hdfs/*:${HADOOP_HOME}/share/hadoop/common/lib/*:${HADOOP_HOME}/share/hadoop/hdfs/lib/*
echo "Running Command : java -cp ${HADOOP_HIVE_JARS}:${HADOOP_CONF_DIR}:$HUDI_HIVE_UBER_JAR org.apache.hudi.hive.HiveSyncTool $@"
java -cp $HUDI_HIVE_UBER_JAR:${HADOOP_HIVE_JARS}:${HADOOP_CONF_DIR}:${HIVE_HOME}/lib/*: org.apache.hudi.hive.HiveSyncTool "$@"
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] stym06 edited a comment on issue #3747: [SUPPORT] Hive Sync process stuck and unable to exit
Posted by GitBox <gi...@apache.org>.
stym06 edited a comment on issue #3747:
URL: https://github.com/apache/hudi/issues/3747#issuecomment-1018677357
@nsivabalan we have a usecase where we needed to do hudisync for a lot of tables, and when we were using sync in the deltastreamer, every job ended up creating jdbc connection pool with hive metastore that caused lots of connection timeout errors. so, i ended up rewriting a tool to do the sync for all tables in a loop that used the thrift client instead of jdbc. it currently runs on cron
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org