You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@dolphinscheduler.apache.org by GitBox <gi...@apache.org> on 2022/03/17 02:42:23 UTC
[GitHub] [dolphinscheduler] lordk911 opened a new issue #8942: [Bug] zookeeper EndOfStreamException when using 2.0.5
lordk911 opened a new issue #8942:
URL: https://github.com/apache/dolphinscheduler/issues/8942
### Search before asking
- [X] I had searched in the [issues](https://github.com/apache/dolphinscheduler/issues?q=is%3Aissue) and found no similar issues.
### What happened
**I have many etl task by calling hive beeline in dolphin shell task:**
**in dolphin 2.0.0 all works well. after upgrade to 2.0.5 , many task of this kind hung up by zookeeper's EndOfStreamException:**
```
[INFO] 2022-03-17 05:00:47.669 TaskLogLogger-class org.apache.dolphinscheduler.plugin.task.shell.ShellTask:[83] - shell task params {"resourceList":[],"localParams":[],"rawScript":"#!/bin/sh\n\nbeeline -u 'jdbc:hive2://worker-10-0-161-20:2181,worker-10-0-161-21:2181,worker-10-0-161-22:2181/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver-prd' -e \"load data inpath '/tmp/datax/tmp/ods/ods_io_dm_lp_cloud_lock_his_ratio' overwrite into table ods.ods_io_dm_lp_cloud_lock_his_ratio\"","dependence":{},"conditionResult":{"successNode":[],"failedNode":[]},"waitStartTimeout":{},"switchResult":{}}
[INFO] 2022-03-17 05:00:47.671 TaskLogLogger-class org.apache.dolphinscheduler.plugin.task.shell.ShellTask:[137] - raw script : #!/bin/sh
beeline -u 'jdbc:hive2://worker-10-0-161-20:2181,worker-10-0-161-21:2181,worker-10-0-161-22:2181/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver-prd' -e "load data inpath '/tmp/datax/tmp/ods/ods_io_dm_lp_cloud_lock_his_ratio' overwrite into table ods.ods_io_dm_lp_cloud_lock_his_ratio"
[INFO] 2022-03-17 05:00:47.672 TaskLogLogger-class org.apache.dolphinscheduler.plugin.task.shell.ShellTask:[138] - task execute path : /tmp/dolphinscheduler/exec/process/3755309710848/4008378301440_4/36722/150650
[INFO] 2022-03-17 05:00:47.672 TaskLogLogger-class org.apache.dolphinscheduler.plugin.task.shell.ShellTask:[86] - tenantCode user:bigtop, task dir:36722_150650
[INFO] 2022-03-17 05:00:47.672 TaskLogLogger-class org.apache.dolphinscheduler.plugin.task.shell.ShellTask:[91] - create command file:/tmp/dolphinscheduler/exec/process/3755309710848/4008378301440_4/36722/150650/36722_150650.command
[INFO] 2022-03-17 05:00:47.673 TaskLogLogger-class org.apache.dolphinscheduler.plugin.task.shell.ShellTask:[117] - command : #!/bin/sh
BASEDIR=$(cd `dirname $0`; pwd)
cd $BASEDIR
source /data/soft/dolphinscheduler-2.0.5/conf/env/dolphinscheduler_env.sh
/tmp/dolphinscheduler/exec/process/3755309710848/4008378301440_4/36722/150650/36722_150650_node.sh
[INFO] 2022-03-17 05:00:47.678 TaskLogLogger-class org.apache.dolphinscheduler.plugin.task.shell.ShellTask:[287] - task run command: sudo -u bigtop sh /tmp/dolphinscheduler/exec/process/3755309710848/4008378301440_4/36722/150650/36722_150650.command
[INFO] 2022-03-17 05:00:47.679 TaskLogLogger-class org.apache.dolphinscheduler.plugin.task.shell.ShellTask:[178] - process start, process id is: 36353
[INFO] 2022-03-17 05:00:48.680 TaskLogLogger-class org.apache.dolphinscheduler.plugin.task.shell.ShellTask:[66] - -> welcome to use bigdata scheduling system...
Connecting to jdbc:hive2://worker-10-0-161-20:2181,worker-10-0-161-21:2181,worker-10-0-161-22:2181/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver-prd
22/03/17 05:00:48 INFO jdbc.Utils: Supplied authorities: worker-10-0-161-20:2181,worker-10-0-161-21:2181,worker-10-0-161-22:2181
[INFO] 2022-03-17 05:00:49.681 TaskLogLogger-class org.apache.dolphinscheduler.plugin.task.shell.ShellTask:[66] - -> 22/03/17 05:00:48 INFO imps.CuratorFrameworkImpl: Starting
22/03/17 05:00:48 INFO zookeeper.ZooKeeper: Client environment:zookeeper.version=3.6.2--803c7f1a12f85978cb049af5e4ef23bd8b688715, built on 09/04/2020 12:44 GMT
22/03/17 05:00:48 INFO zookeeper.ZooKeeper: Client environment:host.name=client-10-0-161-28
22/03/17 05:00:48 INFO zookeeper.ZooKeeper: Client environment:java.version=1.8.0_281
22/03/17 05:00:48 INFO zookeeper.ZooKeeper: Client environment:java.vendor=Oracle Corporation
22/03/17 05:00:48 INFO zookeeper.ZooKeeper: Client environment:java.home=/usr/java/jdk1.8.0_281-amd64/jre
22/03/17 05:00:48 INFO zookeeper.ZooKeeper: Client environment:java.class.path=/data/soft/spark3/conf/:/data/soft/spark3/jars/HikariCP-2.5.1.jar:/data/soft/spark3/jars/JLargeArrays-1.5.jar:/data/soft/spark3/jars/JTransforms-3.1.jar:/data/soft/spark3/jars/RoaringBitmap-0.9.0.jar:/data/soft/spark3/jars/ST4-4.0.4.jar:/data/soft/spark3/jars/activation-1.1.1.jar:/data/soft/spark3/jars/aircompressor-0.21.jar:/data/soft/spark3/jars/algebra_2.12-2.0.1.jar:/data/soft/spark3/jars/annotations-17.0.0.jar:/data/soft/spark3/jars/antlr-runtime-3.5.2.jar:/data/soft/spark3/jars/antlr4-runtime-4.8.jar:/data/soft/spark3/jars/aopalliance-repackaged-2.6.1.jar:/data/soft/spark3/jars/arpack-2.2.1.jar:/data/soft/spark3/jars/arpack_combined_all-0.1.jar:/data/soft/spark3/jars/arrow-format-2.0.0.jar:/data/soft/spark3/jars/arrow-memory-core-2.0.0.jar:/data/soft/spark3/jars/arrow-memory-netty-2.0.0.jar:/data/soft/spark3/jars/arrow-vector-2.0.0.jar:/data/soft/spark3/jars/audience-annotations-0.5.0.jar:/data/s
oft/spark3/jars/automaton-1.11-8.jar:/data/soft/spark3/jars/avro-1.10.2.jar:/data/soft/spark3/jars/avro-ipc-1.10.2.jar:/data/soft/spark3/jars/avro-mapred-1.10.2.jar:/data/soft/spark3/jars/blas-2.2.1.jar:/data/soft/spark3/jars/bonecp-0.8.0.RELEASE.jar:/data/soft/spark3/jars/breeze-macros_2.12-1.2.jar:/data/soft/spark3/jars/breeze_2.12-1.2.jar:/data/soft/spark3/jars/cats-kernel_2.12-2.1.1.jar:/data/soft/spark3/jars/chill-java-0.10.0.jar:/data/soft/spark3/jars/chill_2.12-0.10.0.jar:/data/soft/spark3/jars/commons-cli-1.2.jar:/data/soft/spark3/jars/commons-codec-1.15.jar:/data/soft/spark3/jars/commons-collections-3.2.2.jar:/data/soft/spark3/jars/commons-compiler-3.0.16.jar:/data/soft/spark3/jars/commons-compress-1.21.jar:/data/soft/spark3/jars/commons-crypto-1.1.0.jar:/data/soft/spark3/jars/commons-dbcp-1.4.jar:/data/soft/spark3/jars/commons-io-2.8.0.jar:/data/soft/spark3/jars/commons-lang-2.6.jar:/data/soft/spark3/jars/commons-lang3-3.12.0.jar:/data/soft/spark3/jars/commons-logging-1.1.
3.jar:/data/soft/spark3/jars/commons-math3-3.4.1.jar:/data/soft/spark3/jars/commons-net-3.1.jar:/data/soft/spark3/jars/commons-pool-1.5.4.jar:/data/soft/spark3/jars/commons-text-1.6.jar:/data/soft/spark3/jars/compress-lzf-1.0.3.jar:/data/soft/spark3/jars/core-1.1.2.jar:/data/soft/spark3/jars/curator-client-2.13.0.jar:/data/soft/spark3/jars/curator-framework-2.13.0.jar:/data/soft/spark3/jars/curator-recipes-2.13.0.jar:/data/soft/spark3/jars/datanucleus-api-jdo-4.2.4.jar:/data/soft/spark3/jars/datanucleus-core-4.1.17.jar:/data/soft/spark3/jars/datanucleus-rdbms-4.1.19.jar:/data/soft/spark3/jars/derby-10.14.2.0.jar:/data/soft/spark3/jars/dropwizard-metrics-hadoop-metrics2-reporter-0.1.2.jar:/data/soft/spark3/jars/flatbuffers-java-1.9.0.jar:/data/soft/spark3/jars/generex-1.0.2.jar:/data/soft/spark3/jars/gson-2.2.4.jar:/data/soft/spark3/jars/guava-14.0.1.jar:/data/soft/spark3/jars/hadoop-client-api-3.3.1.jar:/data/soft/spark3/jars/hadoop-client-runtime-3.3.1.jar:/data/soft/spark3/jars/ha
doop-shaded-guava-1.1.1.jar:/data/soft/spark3/jars/hadoop-yarn-server-web-proxy-3.3.1.jar:/data/soft/spark3/jars/hive-beeline-2.3.9.jar:/data/soft/spark3/jars/hive-cli-2.3.9.jar:/data/soft/spark3/jars/hive-common-2.3.9.jar:/data/soft/spark3/jars/hive-exec-2.3.9-core.jar:/data/soft/spark3/jars/hive-jdbc-2.3.9.jar:/data/soft/spark3/jars/hive-llap-common-2.3.9.jar:/data/soft/spark3/jars/hive-metastore-2.3.9.jar:/data/soft/spark3/jars/hive-serde-2.3.9.jar:/data/soft/spark3/jars/hive-service-rpc-3.1.2.jar:/data/soft/spark3/jars/hive-shims-0.23-2.3.9.jar:/data/soft/spark3/jars/hive-shims-2.3.9.jar:/data/soft/spark3/jars/hive-shims-common-2.3.9.jar:/data/soft/spark3/jars/hive-shims-scheduler-2.3.9.jar:/data/soft/spark3/jars/hive-storage-api-2.7.2.jar:/data/soft/spark3/jars/hive-vector-code-gen-2.3.9.jar:/data/soft/spark3/jars/hk2-api-2.6.1.jar:/data/soft/spark3/jars/hk2-locator-2.6.1.jar:/data/soft/spark3/jars/hk2-utils-2.6.1.jar:/data/soft/spark3/jars/htrace-core4-4.1.0-incubating.jar:/da
ta/soft/spark3/jars/httpclient-4.5.13.jar:/data/soft/spark3/jars/httpcore-4.4.14.jar:/data/soft/spark3/jars/istack-commons-runtime-3.0.8.jar:/data/soft/spark3/jars/ivy-2.5.0.jar:/data/soft/spark3/jars/jackson-annotations-2.12.3.jar:/data/soft/spark3/jars/jackson-core-2.12.3.jar:/data/soft/spark3/jars/jackson-core-asl-1.9.13.jar:/data/soft/spark3/jars/jackson-databind-2.12.3.jar:/data/soft/spark3/jars/jackson-dataformat-yaml-2.12.3.jar:/data/soft/spark3/jars/jackson-datatype-jsr310-2.11.2.jar:/data/soft/spark3/jars/jackson-mapper-asl-1.9.13.jar:/data/soft/spark3/jars/jackson-module-scala_2.12-2.12.3.jar:/data/soft/spark3/jars/jakarta.annotation-api-1.3.5.jar:/data/soft/spark3/jars/jakarta.inject-2.6.1.jar:/data/soft/spark3/jars/jakarta.servlet-api-4.0.3.jar:/data/soft/spark3/jars/jakarta.validation-api-2.0.2.jar:/data/soft/spark3/jars/jakarta.ws.rs-api-2.1.6.jar:/data/soft/spark3/jars/jakarta.xml.bind-api-2.3.2.jar:/data/soft/spark3/jars/janino-3.0.16.jar:/data/soft/spark3/jars/javas
sist-3.25.0-GA.jar:/data/soft/spark3/jars/javax.jdo-3.2.0-m3.jar:/data/soft/spark3/jars/javolution-5.5.1.jar:/data/soft/spark3/jars/jaxb-api-2.2.11.jar:/data/soft/spark3/jars/jaxb-runtime-2.3.2.jar:/data/soft/spark3/jars/jcl-over-slf4j-1.7.30.jar:/data/soft/spark3/jars/jdo-api-3.0.1.jar:/data/soft/spark3/jars/jersey-client-2.34.jar:/data/soft/spark3/jars/jersey-common-2.34.jar:/data/soft/spark3/jars/jersey-container-servlet-2.34.jar:/data/soft/spark3/jars/jersey-container-servlet-core-2.34.jar:/data/soft/spark3/jars/jersey-hk2-2.34.jar:/data/soft/spark3/jars/jersey-server-2.34.jar:/data/soft/spark3/jars/jline-2.14.6.jar:/data/soft/spark3/jars/joda-time-2.10.10.jar:/data/soft/spark3/jars/jodd-core-3.5.2.jar:/data/soft/spark3/jars/jpam-1.1.jar:/data/soft/spark3/jars/json-1.8.jar:/data/soft/spark3/jars/json4s-ast_2.12-3.7.0-M11.jar:/data/soft/spark3/jars/json4s-core_2.12-3.7.0-M11.jar:/data/soft/spark3/jars/json4s-jackson_2.12-3.7.0-M11.jar:/data/soft/spark3/jars/json4s-scalap_2.12-3.7
.0-M11.jar:/data/soft/spark3/jars/jsr305-3.0.0.jar:/data/soft/spark3/jars/jta-1.1.jar:/data/soft/spark3/jars/jul-to-slf4j-1.7.30.jar:/data/soft/spark3/jars/kryo-shaded-4.0.2.jar:/data/soft/spark3/jars/kubernetes-client-5.4.1.jar:/data/soft/spark3/jars/kubernetes-model-admissionregistration-5.4.1.jar:/data/soft/spark3/jars/kubernetes-model-apiextensions-5.4.1.jar:/data/soft/spark3/jars/kubernetes-model-apps-5.4.1.jar:/data/soft/spark3/jars/kubernetes-model-autoscaling-5.4.1.jar:/data/soft/spark3/jars/kubernetes-model-batch-5.4.1.jar:/data/soft/spark3/jars/kubernetes-model-certificates-5.4.1.jar:/data/soft/spark3/jars/kubernetes-model-common-5.4.1.jar:/data/soft/spark3/jars/kubernetes-model-coordination-5.4.1.jar:/data/soft/spark3/jars/kubernetes-model-core-5.4.1.jar:/data/soft/spark3/jars/kubernetes-model-discovery-5.4.1.jar:/data/soft/spark3/jars/kubernetes-model-events-5.4.1.jar:/data/soft/spark3/jars/kubernetes-model-extensions-5.4.1.jar:/data/soft/spark3/jars/kubernetes-model-flo
wcontrol-5.4.1.jar:/data/soft/spark3/jars/kubernetes-model-metrics-5.4.1.jar:/data/soft/spark3/jars/kubernetes-model-networking-5.4.1.jar:/data/soft/spark3/jars/kubernetes-model-node-5.4.1.jar:/data/soft/spark3/jars/kubernetes-model-policy-5.4.1.jar:/data/soft/spark3/jars/kubernetes-model-rbac-5.4.1.jar:/data/soft/spark3/jars/kubernetes-model-scheduling-5.4.1.jar:/data/soft/spark3/jars/kubernetes-model-storageclass-5.4.1.jar:/data/soft/spark3/jars/lapack-2.2.1.jar:/data/soft/spark3/jars/leveldbjni-all-1.8.jar:/data/soft/spark3/jars/libfb303-0.9.3.jar:/data/soft/spark3/jars/libthrift-0.12.0.jar:/data/soft/spark3/jars/log4j-1.2.17.jar:/data/soft/spark3/jars/logging-interceptor-3.12.12.jar:/data/soft/spark3/jars/lz4-java-1.7.1.jar:/data/soft/spark3/jars/macro-compat_2.12-1.1.1.jar:/data/soft/spark3/jars/mesos-1.4.0-shaded-protobuf.jar:/data/soft/spark3/jars/metrics-core-4.2.0.jar:/data/soft/spark3/jars/metrics-graphite-4.2.0.jar:/data/soft/spark3/jars/metrics-jmx-4.2.0.jar:/data/soft/s
park3/jars/metrics-json-4.2.0.jar:/data/soft/spark3/jars/metrics-jvm-4.2.0.jar:/data/soft/spark3/jars/minlog-1.3.0.jar:/data/soft/spark3/jars/netty-all-4.1.68.Final.jar:/data/soft/spark3/jars/objenesis-2.6.jar:/data/soft/spark3/jars/okhttp-3.12.12.jar:/data/soft/spark3/jars/okio-1.14.0.jar:/data/soft/spark3/jars/opencsv-2.3.jar:/data/soft/spark3/jars/orc-core-1.6.12.jar:/data/soft/spark3/jars/orc-mapreduce-1.6.12.jar:/data/soft/spark3/jars/orc-shims-1.6.12.jar:/data/soft/spark3/jars/oro-2.0.8.jar:/data/soft/spark3/jars/osgi-resource-locator-1.0.3.jar:/data/soft/spark3/jars/paranamer-2.8.jar:/data/soft/spark3/jars/parquet-column-1.12.2.jar:/data/soft/spark3/jars/parquet-common-1.12.2.jar:/data/soft/spark3/jars/parquet-encoding-1.12.2.jar:/data/soft/spark3/jars/parquet-format-structures-1.12.2.jar:/data/soft/spark3/jars/parquet-hadoop-1.12.2.jar:/data/soft/spark3/jars/parquet-jackson-1.12.2.jar:/data/soft/spark3/jars/protobuf-java-2.5.0.jar:/data/soft/spark3/jars/py4j-0.10.9.3.jar:/da
ta/soft/spark3/jars/pyrolite-4.30.jar:/data/soft/spark3/jars/rocksdbjni-6.20.3.jar:/data/soft/spark3/jars/scala-collection-compat_2.12-2.1.1.jar:/data/soft/spark3/jars/scala-compiler-2.12.15.jar:/data/soft/spark3/jars/scala-library-2.12.15.jar:/data/soft/spark3/jars/scala-parser-combinators_2.12-1.1.2.jar:/data/soft/spark3/jars/scala-reflect-2.12.15.jar:/data/soft/spark3/jars/scala-xml_2.12-1.2.0.jar:/data/soft/spark3/jars/shapeless_2.12-2.3.3.jar:/data/soft/spark3/jars/shims-0.9.0.jar:/data/soft/spark3/jars/slf4j-api-1.7.30.jar:/data/soft/spark3/jars/slf4j-log4j12-1.7.30.jar:/data/soft/spark3/jars/snakeyaml-1.27.jar:/data/soft/spark3/jars/snappy-java-1.1.8.4.jar:/data/soft/spark3/jars/spark-catalyst_2.12-3.2.1.jar:/data/soft/spark3/jars/spark-core_2.12-3.2.1.jar:/data/soft/spark3/jars/stax-api-1.0.1.jar:/data/soft/spark3/jars/spark-graphx_2.12-3.2.1.jar:/data/soft/spark3/jars/spark-hive-thriftserver_2.12-3.2.1.jar:/data/soft/spark3/jars/spark-hive_2.12-3.2.1.jar:/data/soft/spark3/j
ars/spark-kubernetes_2.12-3.2.1.jar:/data/soft/spark3/jars/spark-kvstore_2.12-3.2.1.jar:/data/soft/spark3/jars/spark-launcher_2.12-3.2.1.jar:/data/soft/spark3/jars/spark-mesos_2.12-3.2.1.jar:/data/soft/spark3/jars/spark-mllib-local_2.12-3.2.1.jar:/data/soft/spark3/jars/spark-mllib_2.12-3.2.1.jar:/data/soft/spark3/jars/spark-network-common_2.12-3.2.1.jar:/data/soft/spark3/jars/spark-network-shuffle_2.12-3.2.1.jar:/data/soft/spark3/jars/spark-repl_2.12-3.2.1.jar:/data/soft/spark3/jars/spark-sketch_2.12-3.2.1.jar:/data/soft/spark3/jars/spark-sql_2.12-3.2.1.jar:/data/soft/spark3/jars/spark-streaming_2.12-3.2.1.jar:/data/soft/spark3/jars/spark-tags_2.12-3.2.1-tests.jar:/data/soft/spark3/jars/spark-tags_2.12-3.2.1.jar:/data/soft/spark3/jars/spark-unsafe_2.12-3.2.1.jar:/data/soft/spark3/jars/spark-yarn_2.12-3.2.1.jar:/data/soft/spark3/jars/spire-macros_2.12-0.17.0.jar:/data/soft/spark3/jars/spire-platform_2.12-0.17.0.jar:/data/soft/spark3/jars/spire-util_2.12-0.17.0.jar:/data/soft/spark3/j
ars/spire_2.12-0.17.0.jar:/data/soft/spark3/jars/stream-2.9.6.jar:/data/soft/spark3/jars/super-csv-2.2.0.jar:/data/soft/spark3/jars/threeten-extra-1.5.0.jar:/data/soft/spark3/jars/tink-1.6.0.jar:/data/soft/spark3/jars/transaction-api-1.1.jar:/data/soft/spark3/jars/univocity-parsers-2.9.1.jar:/data/soft/spark3/jars/velocity-1.5.jar:/data/soft/spark3/jars/xbean-asm9-shaded-4.20.jar:/data/soft/spark3/jars/xz-1.8.jar:/data/soft/spark3/jars/zjsonpatch-0.3.0.jar:/data/soft/spark3/jars/zookeeper-3.6.2.jar:/data/soft/spark3/jars/zookeeper-jute-3.6.2.jar:/data/soft/spark3/jars/zstd-jni-1.5.0-4.jar:/data/soft/spark3/jars/submarine-spark-security-spark32-ranger12.jar:/data/soft/spark3/jars/ojdbc8-19.6.0.0.jar:/data/soft/spark3/jars/iceberg-spark-runtime-3.2_2.12-0.13.1.jar:/data/soft/spark3/jars/kyuubi-extension-spark-3-2_2.12-1.4.1-incubating.jar:/etc/hadoop/conf/
22/03/17 05:00:48 INFO zookeeper.ZooKeeper: Client environment:java.library.path=:/usr/hdp/3.1.4.0-315/hadoop/lib/native:/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
22/03/17 05:00:48 INFO zookeeper.ZooKeeper: Client environment:java.io.tmpdir=/tmp
22/03/17 05:00:48 INFO zookeeper.ZooKeeper: Client environment:java.compiler=<NA>
22/03/17 05:00:48 INFO zookeeper.ZooKeeper: Client environment:os.name=Linux
22/03/17 05:00:48 INFO zookeeper.ZooKeeper: Client environment:os.arch=amd64
22/03/17 05:00:48 INFO zookeeper.ZooKeeper: Client environment:os.version=3.10.0-1127.el7.x86_64
22/03/17 05:00:48 INFO zookeeper.ZooKeeper: Client environment:user.name=bigtop
22/03/17 05:00:48 INFO zookeeper.ZooKeeper: Client environment:user.home=/home/bigtop
22/03/17 05:00:48 INFO zookeeper.ZooKeeper: Client environment:user.dir=/tmp/dolphinscheduler/exec/process/3755309710848/4008378301440_4/36722/150650
22/03/17 05:00:48 INFO zookeeper.ZooKeeper: Client environment:os.memory.free=884MB
22/03/17 05:00:48 INFO zookeeper.ZooKeeper: Client environment:os.memory.max=981MB
22/03/17 05:00:48 INFO zookeeper.ZooKeeper: Client environment:os.memory.total=981MB
22/03/17 05:00:48 INFO zookeeper.ZooKeeper: Initiating client connection, connectString=worker-10-0-161-20:2181,worker-10-0-161-21:2181,worker-10-0-161-22:2181 sessionTimeout=60000 watcher=org.apache.curator.ConnectionState@1ddf84b8
22/03/17 05:00:48 INFO common.X509Util: Setting -D jdk.tls.rejectClientInitiatedRenegotiation=true to disable client-initiated TLS renegotiation
22/03/17 05:00:48 INFO zookeeper.ClientCnxnSocket: jute.maxbuffer value is 1048575 Bytes
22/03/17 05:00:48 INFO zookeeper.ClientCnxn: zookeeper.request.timeout value is 0. feature enabled=false
22/03/17 05:00:48 INFO zookeeper.ClientCnxn: Opening socket connection to server worker-10-0-161-21/10.0.161.21:2181.
22/03/17 05:00:48 INFO zookeeper.ClientCnxn: SASL config status: Will not attempt to authenticate using SASL (unknown error)
22/03/17 05:00:48 INFO zookeeper.ClientCnxn: Socket connection established, initiating session, client: /10.0.161.28:58538, server: worker-10-0-161-21/10.0.161.21:2181
22/03/17 05:00:48 INFO zookeeper.ClientCnxn: Session establishment complete on server worker-10-0-161-21/10.0.161.21:2181, session id = 0x37d3196dfaac391, negotiated timeout = 60000
22/03/17 05:00:48 INFO state.ConnectionStateManager: State change: CONNECTED
22/03/17 05:00:48 INFO imps.CuratorFrameworkImpl: backgroundOperationsLoop exiting
22/03/17 05:00:48 WARN zookeeper.ClientCnxn: An exception was thrown while closing send thread for session 0x37d3196dfaac391.
EndOfStreamException: Unable to read additional data from server sessionid 0x37d3196dfaac391, likely server has closed socket
at org.apache.zookeeper.ClientCnxnSocketNIO.doIO(ClientCnxnSocketNIO.java:77)
at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1275)
22/03/17 05:00:48 INFO zookeeper.ZooKeeper: Session: 0x37d3196dfaac391 closed
22/03/17 05:00:48 INFO zookeeper.ClientCnxn: EventThread shut down for session: 0x37d3196dfaac391
22/03/17 05:00:48 INFO jdbc.Utils: Resolved authority: worker-10-0-161-21:10000
22/03/17 05:00:49 INFO jdbc.HiveConnection: Connected to worker-10-0-161-21:10000
Connected to: Apache Hive (version 3.1.0.3.1.4.0-315)
Driver: Hive JDBC (version 2.3.9)
Transaction isolation: TRANSACTION_REPEATABLE_READ
[INFO] 2022-03-17 07:08:05.125 TaskLogLogger-class org.apache.dolphinscheduler.plugin.task.shell.ShellTask:[66] - -> . . . . . . . . . . . . . . . . . . . . . . .> /tmp/dolphinscheduler/exec/process/3755309710848/4008378301440_4/36722/150650/36722_150650_node.sh: line 3: 36362 Killed beeline -u 'jdbc:hive2://worker-10-0-161-20:2181,worker-10-0-161-21:2181,worker-10-0-161-22:2181/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver-prd' -e "load data inpath '/tmp/datax/tmp/ods/ods_io_dm_lp_cloud_lock_his_ratio' overwrite into table ods.ods_io_dm_lp_cloud_lock_his_ratio"
[INFO] 2022-03-17 07:08:05.096 TaskLogLogger-class org.apache.dolphinscheduler.plugin.task.shell.ShellTask:[202] - process has exited, execute path:/tmp/dolphinscheduler/exec/process/3755309710848/4008378301440_4/36722/150650, processId:36353 ,exitStatusCode:137 ,processWaitForStatus:true ,processExitValue:137
[INFO] 2022-03-17 07:08:05.128 TaskLogLogger-class org.apache.dolphinscheduler.plugin.task.shell.ShellTask:[60] - FINALIZE_SESSION
```
**from the log we know task was hung up untill I kill it**
### What you expected to happen
1、I need some clue to find out the reason.
2、Is it because dolphin give to much pressure to zookeeper?
3、my dolphinscheduler deployed in cluster mode , if I build a new zookeeper cluster , and make dolphinscheduler to point the new zookeeper cluster , need I migration data between zookeeper?
### How to reproduce
I'm not sure this problem is because of dolphinscheduler, But it real happen ofter a upgrade, I will make some test.
### Anything else
_No response_
### Version
2.0.5
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] github-actions[bot] commented on issue #8942: [Bug] zookeeper EndOfStreamException when using 2.0.5
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on issue #8942:
URL: https://github.com/apache/dolphinscheduler/issues/8942#issuecomment-1069955804
Hi:
* Thank you for your feedback, we have received your issue, Please wait patiently for a reply.
* In order for us to understand your request as soon as possible, please provide detailed information、version or pictures.
* If you haven't received a reply for a long time, you can subscribe to the developer's email，Mail subscription steps reference https://dolphinscheduler.apache.org/en-us/community/development/subscribe.html ,Then write the issue URL in the email content and send question to dev@dolphinscheduler.apache.org.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] lordk911 commented on issue #8942: [Bug] zookeeper EndOfStreamException when using 2.0.5
Posted by GitBox <gi...@apache.org>.
lordk911 commented on issue #8942:
URL: https://github.com/apache/dolphinscheduler/issues/8942#issuecomment-1070330044
my env problem.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] lordk911 closed issue #8942: [Bug] zookeeper EndOfStreamException when using 2.0.5
Posted by GitBox <gi...@apache.org>.
lordk911 closed issue #8942:
URL: https://github.com/apache/dolphinscheduler/issues/8942
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org