You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hudi.apache.org by selvaraj periyasamy <se...@gmail.com> on 2020/03/01 21:09:31 UTC
Hudi 0.5.0 -> Hive JDBC call fails
I am using Hudi 0.5.0 and then write using sparkwriter.
My spark version is 2.3.0
Scala version 2.11.8
Hive version 1.2.2
Write is success but hive call is failing. When checked some google
reference, It seems to be an hive client is higher version the server.
Since Hudi is built on hive 2.3.1, Is there a way to use 1.2.2?
2020-03-01 12:16:50 WARN HoodieSparkSqlWriter$:110 - hoodie dataset at
hdfs://localhost:9000/projects/cdp/data/attunity_poc/attunity_rep_base
already exists. Deleting existing data & overwriting with new data.
[Stage 111:============================>
2020-03-01
12:16:51 ERROR HiveConnection:697 - Error opening session
org.apache.thrift.TApplicationException: Required field 'client_protocol'
is unset! Struct:TOpenSessionReq(client_protocol:null,
configuration:{set:hiveconf:hive.server2.thrift.resultset.default.fetch.size=1000,
use:database=default})
at
org.apache.thrift.TApplicationException.read(TApplicationException.java:111)
at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:79)
at
org.apache.hudi.org.apache.hive.service.rpc.thrift.TCLIService$Client.recv_OpenSession(TCLIService.java:168)
at
org.apache.hudi.org.apache.hive.service.rpc.thrift.TCLIService$Client.OpenSession(TCLIService.java:155)
Thanks,
Selva
Re: Hudi 0.5.0 -> Hive JDBC call fails
Posted by selvaraj periyasamy <se...@gmail.com>.
I opened Jira to track it .
https://github.com/apache/hudi/issues/1679
Thanks,
Selva
On Tue, May 26, 2020 at 9:52 AM selvaraj periyasamy <
selvaraj.periyasamy1983@gmail.com> wrote:
> Hi Vinoth,
>
>
>
> I see the below comment in Hudi code. How can I start using metastore
> client for hive registrations? is there a way to disable useJdbc flag?
>
>
>
> *// Support both JDBC and metastore based implementations for backwards
> compatiblity. Future users should// disable jdbc and depend on metastore
> client for all hive registrations*
>
>
>
> Below is my log. It makes hive JDBC connection and failing due to method
> not available error.
>
>
> 20/05/26 15:38:15 INFO HoodieSparkSqlWriter$: Syncing to Hive Metastore
> (URL: jdbc:hive2://server1.visa.com:2181,server2.visa.com:2181,
> server3.visa.com:2181/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2
> )
>
> 20/05/26 15:38:15 INFO FSUtils: Hadoop Configuration: fs.defaultFS:
> [hdfs://oprhqanameservice], Config:[Configuration: core-default.xml,
> core-site.xml, mapred-default.xml, mapred-site.xml, yarn-default.xml,
> yarn-site.xml, hdfs-default.xml, hdfs-site.xml, __spark_hadoop_conf__.xml,
> file:/etc/spark2/2.6.5.179-4/0/hive-site.xml], FileSystem:
> [DFS[DFSClient[clientName=DFSClient_NONMAPREDUCE_1153590032_1, ugi=
> svchdc36q@VISA.COM (auth:KERBEROS)]]]
>
> 20/05/26 15:38:15 INFO HiveConf: Found configuration file
> file:/etc/spark2/2.6.5.179-4/0/hive-site.xml
>
> 20/05/26 15:38:16 INFO HoodieTableMetaClient: Loading
> HoodieTableMetaClient from /projects/cdp/data/cdp_reporting/trr
>
> 20/05/26 15:38:16 INFO FSUtils: Hadoop Configuration: fs.defaultFS:
> [hdfs://oprhqanameservice], Config:[Configuration: core-default.xml,
> core-site.xml, mapred-default.xml, mapred-site.xml, yarn-default.xml,
> yarn-site.xml, hdfs-default.xml, hdfs-site.xml, __spark_hadoop_conf__.xml,
> file:/etc/spark2/2.6.5.179-4/0/hive-site.xml], FileSystem:
> [DFS[DFSClient[clientName=DFSClient_NONMAPREDUCE_1153590032_1, ugi=
> svchdc36q@VISA.COM (auth:KERBEROS)]]]
>
> 20/05/26 15:38:16 INFO HoodieTableConfig: Loading dataset properties from
> /projects/cdp/data/cdp_reporting/trr/.hoodie/hoodie.properties
>
> 20/05/26 15:38:16 INFO HoodieTableMetaClient: Finished Loading Table of
> type COPY_ON_WRITE from /projects/cdp/data/cdp_reporting/trr
>
> 20/05/26 15:38:16 INFO HoodieTableMetaClient: Loading Active commit
> timeline for /projects/cdp/data/cdp_reporting/trr
>
> 20/05/26 15:38:16 INFO HoodieActiveTimeline: Loaded instants
> java.util.stream.ReferencePipeline$Head@a1fca5a
>
> 20/05/26 15:38:16 INFO HoodieHiveClient: Creating hive connection
> jdbc:hive2://server1.visa.com:2181,server2.visa.com:2181,
> server3.visa.com:2181/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2
>
> 20/05/26 15:38:16 INFO Utils: Supplied authorities: server1.visa.com:2181,
> server2.visa.com:2181,server3.visa.com:2181
>
> 20/05/26 15:38:16 INFO CuratorFrameworkImpl: Starting
>
> 20/05/26 15:38:16 INFO ZooKeeper: Client
> environment:zookeeper.version=3.4.6-4--1, built on 08/09/2019 23:18 GMT
>
> 20/05/26 15:38:16 INFO ZooKeeper: Client environment:host.name=
> server4.visa.com
>
> 20/05/26 15:38:16 INFO ZooKeeper: Client environment:java.version=1.8.0_241
>
> 20/05/26 15:38:16 INFO ZooKeeper: Client environment:java.vendor=Oracle
> Corporation
>
> 20/05/26 15:38:16 INFO ZooKeeper: Client
> environment:java.home=/usr/java/jdk1.8.0_241-amd64/jre
>
> 20/05/26 15:38:16 INFO ZooKeeper: Client
> environment:java.class.path=/usr/hdp/current/spark2-client/conf/:/usr/hdp/current/spark2-client/jars/hk2-api-2.4.0-b34.jar:/usr/hdp/current/spark2-client/jars/JavaEWAH-0.3.2.jar:/usr/hdp/current/spark2-client/jars/commons-pool-1.5.4.jar:/usr/hdp/current/spark2-client/jars/RoaringBitmap-0.5.11.jar:/usr/hdp/current/spark2-client/jars/hk2-locator-2.4.0-b34.jar:/usr/hdp/current/spark2-client/jars/ST4-4.0.4.jar:/usr/hdp/current/spark2-client/jars/compress-lzf-1.0.3.jar:/usr/hdp/current/spark2-client/jars/activation-1.1.1.jar:/usr/hdp/current/spark2-client/jars/core-1.1.2.jar:/usr/hdp/current/spark2-client/jars/aircompressor-0.8.jar:/usr/hdp/current/spark2-client/jars/hk2-utils-2.4.0-b34.jar:/usr/hdp/current/spark2-client/jars/antlr-2.7.7.jar:/usr/hdp/current/spark2-client/jars/curator-client-2.7.1.jar:/usr/hdp/current/spark2-client/jars/antlr-runtime-3.4.jar:/usr/hdp/current/spark2-client/jars/curator-framework-2.7.1.jar:/usr/hdp/current/spark2-client/jars/antlr4-runtime-4.7.jar:/usr/hdp/current/spark2-client/jars/ivy-2.4.0.jar:/usr/hdp/current/spark2-client/jars/aopalliance-1.0.jar:/usr/hdp/current/spark2-client/jars/commons-io-2.4.jar:/usr/hdp/current/spark2-client/jars/janino-3.0.8.jar:/usr/hdp/current/spark2-client/jars/aopalliance-repackaged-2.4.0-b34.jar:/usr/hdp/current/spark2-client/jars/commons-collections-3.2.2.jar:/usr/hdp/current/spark2-client/jars/apache-log4j-extras-1.2.17.jar:/usr/hdp/current/spark2-client/jars/curator-recipes-2.7.1.jar:/usr/hdp/current/spark2-client/jars/apacheds-i18n-2.0.0-M15.jar:/usr/hdp/current/spark2-client/jars/commons-cli-1.2.jar:/usr/hdp/current/spark2-client/jars/javax.inject-1.jar:/usr/hdp/current/spark2-client/jars/apacheds-kerberos-codec-2.0.0-M15.jar:/usr/hdp/current/spark2-client/jars/datanucleus-api-jdo-3.2.6.jar:/usr/hdp/current/spark2-client/jars/api-asn1-api-1.0.0-M20.jar:/usr/hdp/current/spark2-client/jars/datanucleus-core-3.2.10.jar:/usr/hdp/current/spark2-client/jars/api-util-1.0.0-M20.jar:/usr/hdp/current/spark2-client/jars/datanucleus-rdbms-3.2.9.jar:/usr/hdp/current/spark2-client/jars/arpack_combined_all-0.1.jar:/usr/hdp/current/spark2-client/jars/derby-10.12.1.1.jar:/usr/hdp/current/spark2-client/jars/arrow-format-0.8.0.jar:/usr/hdp/current/spark2-client/jars/eigenbase-properties-1.1.5.jar:/usr/hdp/current/spark2-client/jars/arrow-memory-0.8.0.jar:/usr/hdp/current/spark2-client/jars/flatbuffers-1.2.0-3f79e055.jar:/usr/hdp/current/spark2-client/jars/arrow-vector-0.8.0.jar:/usr/hdp/current/spark2-client/jars/hppc-0.7.2.jar:/usr/hdp/current/spark2-client/jars/avro-1.7.7.jar:/usr/hdp/current/spark2-client/jars/httpclient-4.5.2.jar:/usr/hdp/current/spark2-client/jars/avro-ipc-1.7.7.jar:/usr/hdp/current/spark2-client/jars/commons-compiler-3.0.8.jar:/usr/hdp/current/spark2-client/jars/avro-mapred-1.7.7-hadoop2.jar:/usr/hdp/current/spark2-client/jars/commons-compress-1.4.1.jar:/usr/hdp/current/spark2-client/jars/aws-java-sdk-core-1.10.6.jar:/usr/hdp/current/spark2-client/jars/guava-14.0.1.jar:/usr/hdp/current/spark2-client/jars/aws-java-sdk-kms-1.10.6.jar:/usr/hdp/current/spark2-client/jars/gson-2.2.4.jar:/usr/hdp/current/spark2-client/jars/aws-java-sdk-s3-1.10.6.jar:/usr/hdp/current/spark2-client/jars/commons-configuration-1.6.jar:/usr/hdp/current/spark2-client/jars/azure-data-lake-store-sdk-2.1.4.jar:/usr/hdp/current/spark2-client/jars/commons-lang-2.6.jar:/usr/hdp/current/spark2-client/jars/jpam-1.1.jar:/usr/hdp/current/spark2-client/jars/azure-keyvault-core-0.8.0.jar:/usr/hdp/current/spark2-client/jars/guice-3.0.jar:/usr/hdp/current/spark2-client/jars/azure-storage-5.4.0.jar:/usr/hdp/current/spark2-client/jars/httpcore-4.4.4.jar:/usr/hdp/current/spark2-client/jars/base64-2.3.8.jar:/usr/hdp/current/spark2-client/jars/guice-servlet-3.0.jar:/usr/hdp/current/spark2-client/jars/bcprov-jdk15on-1.58.jar:/usr/hdp/current/spark2-client/jars/hadoop-aws-2.7.3.2.6.5.179-4.jar:/usr/hdp/current/spark2-client/jars/bonecp-0.8.0.RELEASE.jar:/usr/hdp/current/spark2-client/jars/commons-lang3-3.5.jar:/usr/hdp/current/spark2-client/jars/breeze-macros_2.11-0.13.2.jar:/usr/hdp/current/spark2-client/jars/hadoop-auth-2.7.3.2.6.5.179-4.jar:/usr/hdp/current/spark2-client/jars/breeze_2.11-0.13.2.jar:/usr/hdp/current/spark2-client/jars/commons-codec-1.10.jar:/usr/hdp/current/spark2-client/jars/jta-1.1.jar:/usr/hdp/current/spark2-client/jars/calcite-avatica-1.2.0-incubating.jar:/usr/hdp/current/spark2-client/jars/commons-logging-1.1.3.jar:/usr/hdp/current/spark2-client/jars/calcite-core-1.2.0-incubating.jar:/usr/hdp/current/spark2-client/jars/commons-math3-3.4.1.jar:/usr/hdp/current/spark2-client/jars/calcite-linq4j-1.2.0-incubating.jar:/usr/hdp/current/spark2-client/jars/hadoop-azure-2.7.3.2.6.5.179-4.jar:/usr/hdp/current/spark2-client/jars/chill-java-0.8.4.jar:/usr/hdp/current/spark2-client/jars/hadoop-client-2.7.3.2.6.5.179-4.jar:/usr/hdp/current/spark2-client/jars/chill_2.11-0.8.4.jar:/usr/hdp/current/spark2-client/jars/hadoop-common-2.7.3.2.6.5.179-4.jar:/usr/hdp/current/spark2-client/jars/commons-beanutils-1.7.0.jar:/usr/hdp/current/spark2-client/jars/gcs-connector-1.8.1.2.6.5.179-4-shaded.jar:/usr/hdp/current/spark2-client/jars/commons-beanutils-core-1.8.0.jar:/usr/hdp/current/spark2-client/jars/commons-crypto-1.0.0.jar:/usr/hdp/current/spark2-client/jars/hadoop-hdfs-2.7.3.2.6.5.179-4.jar:/usr/hdp/current/spark2-client/jars/commons-dbcp-1.4.jar:/usr/hdp/current/spark2-client/jars/hive-beeline-1.21.2.2.6.5.179-4.jar:/usr/hdp/current/spark2-client/jars/commons-digester-1.8.jar:/usr/hdp/current/spark2-client/jars/hive-cli-1.21.2.2.6.5.179-4.jar:/usr/hdp/current/spark2-client/jars/commons-httpclient-3.1.jar:/usr/hdp/current/spark2-client/jars/jackson-core-2.6.7.jar:/usr/hdp/current/spark2-client/jars/commons-net-2.2.jar:/usr/hdp/current/spark2-client/jars/javolution-5.5.1.jar:/usr/hdp/current/spark2-client/jars/jersey-server-2.22.2.jar:/usr/hdp/current/spark2-client/jars/xz-1.0.jar:/usr/hdp/current/spark2-client/jars/hadoop-annotations-2.7.3.2.6.5.179-4.jar:/usr/hdp/current/spark2-client/jars/jets3t-0.9.4.jar:/usr/hdp/current/spark2-client/jars/okhttp-2.7.5.jar:/usr/hdp/current/spark2-client/jars/hadoop-azure-datalake-2.7.3.2.6.5.179-4.jar:/usr/hdp/current/spark2-client/jars/jersey-container-servlet-2.22.2.jar:/usr/hdp/current/spark2-client/jars/hadoop-mapreduce-client-app-2.7.3.2.6.5.179-4.jar:/usr/hdp/current/spark2-client/jars/jdo-api-3.0.1.jar:/usr/hdp/current/spark2-client/jars/libfb303-0.9.3.jar:/usr/hdp/current/spark2-client/jars/hadoop-mapreduce-client-common-2.7.3.2.6.5.179-4.jar:/usr/hdp/current/spark2-client/jars/jersey-container-servlet-core-2.22.2.jar:/usr/hdp/current/spark2-client/jars/hadoop-mapreduce-client-core-2.7.3.2.6.5.179-4.jar:/usr/hdp/current/spark2-client/jars/jersey-client-2.22.2.jar:/usr/hdp/current/spark2-client/jars/netty-3.9.9.Final.jar:/usr/hdp/current/spark2-client/jars/hadoop-mapreduce-client-jobclient-2.7.3.2.6.5.179-4.jar:/usr/hdp/current/spark2-client/jars/jersey-common-2.22.2.jar:/usr/hdp/current/spark2-client/jars/netty-all-4.1.17.Final.jar:/usr/hdp/current/spark2-client/jars/hadoop-mapreduce-client-shuffle-2.7.3.2.6.5.179-4.jar:/usr/hdp/current/spark2-client/jars/jetty-6.1.26.hwx.jar:/usr/hdp/current/spark2-client/jars/okio-1.6.0.jar:/usr/hdp/current/spark2-client/jars/hadoop-openstack-2.7.3.2.6.5.179-4.jar:/usr/hdp/current/spark2-client/jars/jetty-sslengine-6.1.26.hwx.jar:/usr/hdp/current/spark2-client/jars/hadoop-yarn-api-2.7.3.2.6.5.179-4.jar:/usr/hdp/current/spark2-client/jars/jetty-util-6.1.26.hwx.jar:/usr/hdp/current/spark2-client/jars/hadoop-yarn-client-2.7.3.2.6.5.179-4.jar:/usr/hdp/current/spark2-client/jars/jline-2.12.1.jar:/usr/hdp/current/spark2-client/jars/opencsv-2.3.jar:/usr/hdp/current/spark2-client/jars/hadoop-yarn-common-2.7.3.2.6.5.179-4.jar:/usr/hdp/current/spark2-client/jars/joda-time-2.9.3.jar:/usr/hdp/current/spark2-client/jars/oro-2.0.8.jar:/usr/hdp/current/spark2-client/jars/hadoop-yarn-registry-2.7.3.2.6.5.179-4.jar:/usr/hdp/current/spark2-client/jars/jersey-guava-2.22.2.jar:/usr/hdp/current/spark2-client/jars/paranamer-2.8.jar:/usr/hdp/current/spark2-client/jars/hadoop-yarn-server-common-2.7.3.2.6.5.179-4.jar:/usr/hdp/current/spark2-client/jars/jersey-media-jaxb-2.22.2.jar:/usr/hdp/current/spark2-client/jars/py4j-0.10.6.jar:/usr/hdp/current/spark2-client/jars/hadoop-yarn-server-web-proxy-2.7.3.2.6.5.179-4.jar:/usr/hdp/current/spark2-client/jars/json4s-ast_2.11-3.2.11.jar:/usr/hdp/current/spark2-client/jars/hive-exec-1.21.2.2.6.5.179-4.jar:/usr/hdp/current/spark2-client/jars/json4s-core_2.11-3.2.11.jar:/usr/hdp/current/spark2-client/jars/hive-jdbc-1.21.2.2.6.5.179-4.jar:/usr/hdp/current/spark2-client/jars/jodd-core-3.5.2.jar:/usr/hdp/current/spark2-client/jars/pyrolite-4.13.jar:/usr/hdp/current/spark2-client/jars/hive-metastore-1.21.2.2.6.5.179-4.jar:/usr/hdp/current/spark2-client/jars/json4s-jackson_2.11-3.2.11.jar:/usr/hdp/current/spark2-client/jars/htrace-core-3.1.0-incubating.jar:/usr/hdp/current/spark2-client/jars/jsp-api-2.1.jar:/usr/hdp/current/spark2-client/jars/jackson-annotations-2.6.7.jar:/usr/hdp/current/spark2-client/jars/libthrift-0.9.3.jar:/usr/hdp/current/spark2-client/jars/jackson-core-asl-1.9.13.jar:/usr/hdp/current/spark2-client/jars/jsr305-1.3.9.jar:/usr/hdp/current/spark2-client/jars/jackson-databind-2.6.7.1.jar:/usr/hdp/current/spark2-client/jars/jtransforms-2.4.0.jar:/usr/hdp/current/spark2-client/jars/jackson-dataformat-cbor-2.6.7.jar:/usr/hdp/current/spark2-client/jars/log4j-1.2.17.jar:/usr/hdp/current/spark2-client/jars/jackson-jaxrs-1.9.13.jar:/usr/hdp/current/spark2-client/jars/jul-to-slf4j-1.7.16.jar:/usr/hdp/current/spark2-client/jars/jackson-mapper-asl-1.9.13.jar:/usr/hdp/current/spark2-client/jars/kryo-shaded-3.0.3.jar:/usr/hdp/current/spark2-client/jars/jackson-module-paranamer-2.7.9.jar:/usr/hdp/current/spark2-client/jars/json-smart-1.3.1.jar:/usr/hdp/current/spark2-client/jars/scalap-2.11.8.jar:/usr/hdp/current/spark2-client/jars/jackson-module-scala_2.11-2.6.7.1.jar:/usr/hdp/current/spark2-client/jars/lz4-java-1.4.0.jar:/usr/hdp/current/spark2-client/jars/jackson-xc-1.9.13.jar:/usr/hdp/current/spark2-client/jars/machinist_2.11-0.6.1.jar:/usr/hdp/current/spark2-client/jars/java-xmlbuilder-1.1.jar:/usr/hdp/current/spark2-client/jars/macro-compat_2.11-1.1.1.jar:/usr/hdp/current/spark2-client/jars/javassist-3.18.1-GA.jar:/usr/hdp/current/spark2-client/jars/leveldbjni-all-1.8.jar:/usr/hdp/current/spark2-client/jars/javax.annotation-api-1.2.jar:/usr/hdp/current/spark2-client/jars/metrics-core-3.1.5.jar:/usr/hdp/current/spark2-client/jars/javax.inject-2.4.0-b34.jar:/usr/hdp/current/spark2-client/jars/metrics-graphite-3.1.5.jar:/usr/hdp/current/spark2-client/jars/javax.servlet-api-3.1.0.jar:/usr/hdp/current/spark2-client/jars/metrics-json-3.1.5.jar:/usr/hdp/current/spark2-client/jars/javax.ws.rs-api-2.0.1.jar:/usr/hdp/current/spark2-client/jars/nimbus-jose-jwt-4.41.1.jar:/usr/hdp/current/spark2-client/jars/jaxb-api-2.2.2.jar:/usr/hdp/current/spark2-client/jars/metrics-jvm-3.1.5.jar:/usr/hdp/current/spark2-client/jars/jcip-annotations-1.0-1.jar:/usr/hdp/current/spark2-client/jars/minlog-1.3.0.jar:/usr/hdp/current/spark2-client/jars/jcl-over-slf4j-1.7.16.jar:/usr/hdp/current/spark2-client/jars/parquet-column-1.8.2.jar:/usr/hdp/current/spark2-client/jars/objenesis-2.1.jar:/usr/hdp/current/spark2-client/jars/spire_2.11-0.13.0.jar:/usr/hdp/current/spark2-client/jars/orc-core-1.4.3.2.6.5.179-4-nohive.jar:/usr/hdp/current/spark2-client/jars/stax-api-1.0-2.jar:/usr/hdp/current/spark2-client/jars/orc-mapreduce-1.4.3.2.6.5.179-4-nohive.jar:/usr/hdp/current/spark2-client/jars/osgi-resource-locator-1.0.1.jar:/usr/hdp/current/spark2-client/jars/parquet-common-1.8.2.jar:/usr/hdp/current/spark2-client/jars/parquet-encoding-1.8.2.jar:/usr/hdp/current/spark2-client/jars/parquet-format-2.3.1.jar:/usr/hdp/current/spark2-client/jars/parquet-hadoop-1.8.2.jar:/usr/hdp/current/spark2-client/jars/parquet-hadoop-bundle-1.6.0.jar:/usr/hdp/current/spark2-client/jars/parquet-jackson-1.8.2.jar:/usr/hdp/current/spark2-client/jars/protobuf-java-2.5.0.jar:/usr/hdp/current/spark2-client/jars/scala-compiler-2.11.8.jar:/usr/hdp/current/spark2-client/jars/scala-library-2.11.8.jar:/usr/hdp/current/spark2-client/jars/stax-api-1.0.1.jar:/usr/hdp/current/spark2-client/jars/scala-parser-combinators_2.11-1.0.4.jar:/usr/hdp/current/spark2-client/jars/scala-reflect-2.11.8.jar:/usr/hdp/current/spark2-client/jars/scala-xml_2.11-1.0.5.jar:/usr/hdp/current/spark2-client/jars/shapeless_2.11-2.3.2.jar:/usr/hdp/current/spark2-client/jars/slf4j-api-1.7.16.jar:/usr/hdp/current/spark2-client/jars/slf4j-log4j12-1.7.16.jar:/usr/hdp/current/spark2-client/jars/snappy-0.2.jar:/usr/hdp/current/spark2-client/jars/snappy-java-1.1.2.6.jar:/usr/hdp/current/spark2-client/jars/stream-2.7.0.jar:/usr/hdp/current/spark2-client/jars/spark-catalyst_2.11-2.3.0.2.6.5.179-4.jar:/usr/hdp/current/spark2-client/jars/stringtemplate-3.2.1.jar:/usr/hdp/current/spark2-client/jars/spark-cloud_2.11-2.3.0.2.6.5.179-4.jar:/usr/hdp/current/spark2-client/jars/super-csv-2.2.0.jar:/usr/hdp/current/spark2-client/jars/spark-core_2.11-2.3.0.2.6.5.179-4.jar:/usr/hdp/current/spark2-client/jars/univocity-parsers-2.5.9.jar:/usr/hdp/current/spark2-client/jars/spark-graphx_2.11-2.3.0.2.6.5.179-4.jar:/usr/hdp/current/spark2-client/jars/spark-unsafe_2.11-2.3.0.2.6.5.179-4.jar:/usr/hdp/current/spark2-client/jars/spark-hadoop-cloud_2.11-2.3.0.2.6.5.179-4.jar:/usr/hdp/current/spark2-client/jars/spark-tags_2.11-2.3.0.2.6.5.179-4.jar:/usr/hdp/current/spark2-client/jars/spark-hive-thriftserver_2.11-2.3.0.2.6.5.179-4.jar:/usr/hdp/current/spark2-client/jars/validation-api-1.1.0.Final.jar:/usr/hdp/current/spark2-client/jars/spark-hive_2.11-2.3.0.2.6.5.179-4.jar:/usr/hdp/current/spark2-client/jars/xbean-asm5-shaded-4.4.jar:/usr/hdp/current/spark2-client/jars/spark-kvstore_2.11-2.3.0.2.6.5.179-4.jar:/usr/hdp/current/spark2-client/jars/xercesImpl-2.9.1.jar:/usr/hdp/current/spark2-client/jars/spark-launcher_2.11-2.3.0.2.6.5.179-4.jar:/usr/hdp/current/spark2-client/jars/spark-mllib-local_2.11-2.3.0.2.6.5.179-4.jar:/usr/hdp/current/spark2-client/jars/xmlenc-0.52.jar:/usr/hdp/current/spark2-client/jars/spark-mllib_2.11-2.3.0.2.6.5.179-4.jar:/usr/hdp/current/spark2-client/jars/spark-yarn_2.11-2.3.0.2.6.5.179-4.jar:/usr/hdp/current/spark2-client/jars/spark-network-common_2.11-2.3.0.2.6.5.179-4.jar:/usr/hdp/current/spark2-client/jars/spire-macros_2.11-0.13.0.jar:/usr/hdp/current/spark2-client/jars/spark-network-shuffle_2.11-2.3.0.2.6.5.179-4.jar:/usr/hdp/current/spark2-client/jars/zookeeper-3.4.6.2.6.5.179-4.jar:/usr/hdp/current/spark2-client/jars/spark-repl_2.11-2.3.0.2.6.5.179-4.jar:/usr/hdp/current/spark2-client/jars/zstd-jni-1.3.2-2.jar:/usr/hdp/current/spark2-client/jars/spark-sketch_2.11-2.3.0.2.6.5.179-4.jar:/usr/hdp/current/spark2-client/jars/spark-sql_2.11-2.3.0.2.6.5.179-4.jar:/usr/hdp/current/spark2-client/jars/spark-streaming_2.11-2.3.0.2.6.5.179-4.jar:/usr/hdp/2.6.5.179-4/hadoop/conf/
>
> 20/05/26 15:38:16 INFO ZooKeeper: Client
> environment:java.library.path=:/export/home/sobla/oracle_client/instantclient_19_5:/usr/hdp/current/hadoop-client/lib/native:/usr/hdp/current/hadoop-client/lib/native/Linux-amd64-64:/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
>
> 20/05/26 15:38:16 INFO ZooKeeper: Client environment:java.io.tmpdir=/tmp
>
> 20/05/26 15:38:16 INFO ZooKeeper: Client environment:java.compiler=<NA>
>
> 20/05/26 15:38:16 INFO ZooKeeper: Client environment:os.name=Linux
>
> 20/05/26 15:38:16 INFO ZooKeeper: Client environment:os.arch=amd64
>
> 20/05/26 15:38:16 INFO ZooKeeper: Client
> environment:os.version=3.10.0-1062.9.1.el7.x86_64
>
> 20/05/26 15:38:16 INFO ZooKeeper: Client environment:user.name=svchdc36q
>
> 20/05/26 15:38:16 INFO ZooKeeper: Client
> environment:user.home=/home/svchdc36q
>
> 20/05/26 15:38:16 INFO ZooKeeper: Client
> environment:user.dir=/home/svchdc36q
>
> *20/05/26 15:38:16 INFO ZooKeeper: Initiating client connection,
> connectString=server1.visa.com:2181
> <http://server1.visa.com:2181>,server2.visa.com:2181
> <http://server2.visa.com:2181>,server3.visa.com:2181
> <http://server3.visa.com:2181> sessionTimeout=60000
> watcher=org.apache.curator.ConnectionState@4ed31bc9*
>
> *20/05/26 15:38:16 INFO ClientCnxn: Opening socket connection to server
> server2.visa.com/x.x.x.x:2181 <http://server2.visa.com/x.x.x.x:2181>. Will
> not attempt to authenticate using SASL (unknown error)*
>
> *20/05/26 15:38:16 INFO ClientCnxn: Socket connection established,
> initiating session, client: /x.x.x.x:36938, server:
> server2.visa.com/x.x.x.x:2181 <http://server2.visa.com/x.x.x.x:2181>*
>
> *20/05/26 15:38:16 INFO ClientCnxn: Session establishment complete on
> server server2.visa.com/x.x.x.x:2181
> <http://server2.visa.com/x.x.x.x:2181>, sessionid = 0x27234630fb51f5b,
> negotiated timeout = 40000*
>
> 20/05/26 15:38:16 INFO ConnectionStateManager: State change: CONNECTED
>
> 20/05/26 15:38:17 INFO ZooKeeper: Session: 0x27234630fb51f5b closed
>
> 20/05/26 15:38:17 INFO ClientCnxn: EventThread shut down
>
> 20/05/26 15:38:17 INFO Utils: Resolved authority: server2.visa.com:10000
>
> 20/05/26 15:38:17 INFO HiveConnection: Will try to open client transport
> with JDBC Uri: jdbc:hive2://
> server2.visa.com:10000/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2
>
> 20/05/26 15:38:19 INFO HoodieHiveClient: Successfully established Hive
> connection to jdbc:hive2://server1.visa.com:2181,server2.visa.com:2181,
> server3.visa.com:2181/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2
>
> 20/05/26 15:38:19 INFO metastore: Trying to connect to metastore with URI
> thrift://server2.visa.com:9083
>
> 20/05/26 15:38:19 INFO metastore: Opened a connection to metastore,
> current connections: 1
>
> 20/05/26 15:38:19 INFO metastore: Connected to metastore.
>
> 20/05/26 15:38:19 INFO HiveSyncTool: Trying to sync hoodie table trr with
> base path /projects/cdp/data/cdp_reporting/trr of type COPY_ON_WRITE
>
> 20/05/26 15:38:19 ERROR DBUtil$: [App] *********************** Exception
> occurred in baseTableWrite for trr : Failed to check if table exists trr
>
> org.apache.hudi.hive.HoodieHiveSyncException: Failed to check if table
> exists trr
>
> at
> org.apache.hudi.hive.HoodieHiveClient.doesTableExist(HoodieHiveClient.java:459)
>
> at org.apache.hudi.hive.HiveSyncTool.syncHoodieTable(HiveSyncTool.java:91)
>
> at org.apache.hudi.hive.HiveSyncTool.syncHoodieTable(HiveSyncTool.java:67)
>
> at
> org.apache.hudi.HoodieSparkSqlWriter$.syncHive(HoodieSparkSqlWriter.scala:235)
>
> at
> org.apache.hudi.HoodieSparkSqlWriter$.write(HoodieSparkSqlWriter.scala:169)
>
> at org.apache.hudi.DefaultSource.createRelation(DefaultSource.scala:91)
>
> at
> org.apache.spark.sql.execution.datasources.SaveIntoDataSourceCommand.run(SaveIntoDataSourceCommand.scala:46)
>
> at
> org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult$lzycompute(commands.scala:70)
>
> at
> org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult(commands.scala:68)
>
> at
> org.apache.spark.sql.execution.command.ExecutedCommandExec.doExecute(commands.scala:86)
>
> at
> org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:131)
>
> at
> org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:127)
>
> at
> org.apache.spark.sql.execution.SparkPlan$$anonfun$executeQuery$1.apply(SparkPlan.scala:155)
>
> at
> org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
>
> at
> org.apache.spark.sql.execution.SparkPlan.executeQuery(SparkPlan.scala:152)
>
> at org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:127)
>
> at
> org.apache.spark.sql.execution.QueryExecution.toRdd$lzycompute(QueryExecution.scala:80)
>
> at
> org.apache.spark.sql.execution.QueryExecution.toRdd(QueryExecution.scala:80)
>
> at
> org.apache.spark.sql.DataFrameWriter$$anonfun$runCommand$1.apply(DataFrameWriter.scala:654)
>
> at
> org.apache.spark.sql.DataFrameWriter$$anonfun$runCommand$1.apply(DataFrameWriter.scala:654)
>
> at
> org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:77)
>
> at
> org.apache.spark.sql.DataFrameWriter.runCommand(DataFrameWriter.scala:654)
>
> at
> org.apache.spark.sql.DataFrameWriter.saveToV1Source(DataFrameWriter.scala:273)
>
> at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:267)
>
> at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:225)
>
> at
> com.cybs.cdp.reporting.trr.DBUtil$.transactionTableWrite(DBUtil.scala:62)
>
> at com.cybs.cdp.reporting.trr.TRREngine$.startEngine(TRREngine.scala:45)
>
> at com.cybs.cdp.reporting.trr.TRREngine$.main(TRREngine.scala:23)
>
> at com.cybs.cdp.reporting.trr.TRREngine.main(TRREngine.scala)
>
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>
> at java.lang.reflect.Method.invoke(Method.java:498)
>
> at
> org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
>
> at
> org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:906)
>
> at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:197)
>
> at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:227)
>
> at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:136)
>
> at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
>
> Caused by: org.apache.thrift.TApplicationException: Invalid method name:
> 'get_table_req'
>
> at
> org.apache.thrift.TApplicationException.read(TApplicationException.java:111)
>
> at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:79)
>
> at
> org.apache.hudi.org.apache.hadoop_hive.metastore.api.ThriftHiveMetastore$Client.recv_get_table_req(ThriftHiveMetastore.java:1563)
>
> at
> org.apache.hudi.org.apache.hadoop_hive.metastore.api.ThriftHiveMetastore$Client.get_table_req(ThriftHiveMetastore.java:1550)
>
> at
> org.apache.hudi.org.apache.hadoop_hive.metastore.HiveMetaStoreClient.tableExists(HiveMetaStoreClient.java:1443)
>
> at
> org.apache.hudi.hive.HoodieHiveClient.doesTableExist(HoodieHiveClient.java:457)
>
> ... 38 more
>
> Exception in thread "main" java.lang.Exception: Failed to check if table
> exists trr
>
> at
> com.cybs.cdp.reporting.trr.DBUtil$.transactionTableWrite(DBUtil.scala:69)
>
> at com.cybs.cdp.reporting.trr.TRREngine$.startEngine(TRREngine.scala:45)
>
> at com.cybs.cdp.reporting.trr.TRREngine$.main(TRREngine.scala:23)
>
> at com.cybs.cdp.reporting.trr.TRREngine.main(TRREngine.scala)
>
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>
> at java.lang.reflect.Method.invoke(Method.java:498)
>
> at
> org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
>
> at
> org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:906)
>
> at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:197)
>
> at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:227)
>
> at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:136)
>
>
>
> Thanks,
>
> Selva
>
>
>
> On Mon, Mar 2, 2020 at 10:02 AM Vinoth Chandar <vi...@apache.org> wrote:
>
>> Hi Selva,
>>
>> See if this helps.
>>
>> https://lists.apache.org/thread.html/e1fd539ac438276dd7feb2bc813bf85f84a95f7f25b638488eb2e110%40%3Cdev.hudi.apache.org%3E
>>
>> Its long thread, but you can probably skim to the last few conversations
>> around Hive 1.x
>>
>> Thanks
>> Vinoth
>>
>> On Sun, Mar 1, 2020 at 5:26 PM selvaraj periyasamy <
>> selvaraj.periyasamy1983@gmail.com> wrote:
>>
>> > Thanks Vinoth. We do have plan to move hive 2.x version in near future.
>> > Can I get any info on the workaround for hive 1.x versions?
>> >
>> > Thanks,
>> > Selva
>> >
>> > On Sun, Mar 1, 2020 at 3:19 PM Vinoth Chandar <vi...@apache.org>
>> wrote:
>> >
>> > > We have dropped support for Hive 1.x, a while back. Would you be able
>> to
>> > > move to Hive 2.x?
>> > >
>> > > IIRC there were some workarounds discussed on this thread before. But,
>> > > given the push towards Hive 3.x, its good to be on 2.x atleast ..
>> > > Let me know and we can go from there :)
>> > >
>> > > On Sun, Mar 1, 2020 at 1:09 PM selvaraj periyasamy <
>> > > selvaraj.periyasamy1983@gmail.com> wrote:
>> > >
>> > > > I am using Hudi 0.5.0 and then write using sparkwriter.
>> > > >
>> > > > My spark version is 2.3.0
>> > > > Scala version 2.11.8
>> > > > Hive version 1.2.2
>> > > >
>> > > > Write is success but hive call is failing. When checked some google
>> > > > reference, It seems to be an hive client is higher version the
>> server.
>> > > > Since Hudi is built on hive 2.3.1, Is there a way to use 1.2.2?
>> > > >
>> > > > 2020-03-01 12:16:50 WARN HoodieSparkSqlWriter$:110 - hoodie
>> dataset at
>> > > >
>> hdfs://localhost:9000/projects/cdp/data/attunity_poc/attunity_rep_base
>> > > > already exists. Deleting existing data & overwriting with new data.
>> > > > [Stage 111:============================>
>> > > > 2020-03-01
>> > > > 12:16:51 ERROR HiveConnection:697 - Error opening session
>> > > > org.apache.thrift.TApplicationException: Required field
>> > 'client_protocol'
>> > > > is unset! Struct:TOpenSessionReq(client_protocol:null,
>> > > >
>> > > >
>> > >
>> >
>> configuration:{set:hiveconf:hive.server2.thrift.resultset.default.fetch.size=1000,
>> > > > use:database=default})
>> > > > at
>> > > >
>> > > >
>> > >
>> >
>> org.apache.thrift.TApplicationException.read(TApplicationException.java:111)
>> > > > at
>> org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:79)
>> > > > at
>> > > >
>> > > >
>> > >
>> >
>> org.apache.hudi.org.apache.hive.service.rpc.thrift.TCLIService$Client.recv_OpenSession(TCLIService.java:168)
>> > > > at
>> > > >
>> > > >
>> > >
>> >
>> org.apache.hudi.org.apache.hive.service.rpc.thrift.TCLIService$Client.OpenSession(TCLIService.java:155)
>> > > >
>> > > >
>> > > > Thanks,
>> > > > Selva
>> > > >
>> > >
>> >
>>
>
Re: Hudi 0.5.0 -> Hive JDBC call fails
Posted by selvaraj periyasamy <se...@gmail.com>.
Hi Vinoth,
I see the below comment in Hudi code. How can I start using metastore
client for hive registrations? is there a way to disable useJdbc flag?
*// Support both JDBC and metastore based implementations for backwards
compatiblity. Future users should// disable jdbc and depend on metastore
client for all hive registrations*
Below is my log. It makes hive JDBC connection and failing due to method
not available error.
20/05/26 15:38:15 INFO HoodieSparkSqlWriter$: Syncing to Hive Metastore
(URL: jdbc:hive2://server1.visa.com:2181,server2.visa.com:2181,
server3.visa.com:2181/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2
)
20/05/26 15:38:15 INFO FSUtils: Hadoop Configuration: fs.defaultFS:
[hdfs://oprhqanameservice], Config:[Configuration: core-default.xml,
core-site.xml, mapred-default.xml, mapred-site.xml, yarn-default.xml,
yarn-site.xml, hdfs-default.xml, hdfs-site.xml, __spark_hadoop_conf__.xml,
file:/etc/spark2/2.6.5.179-4/0/hive-site.xml], FileSystem:
[DFS[DFSClient[clientName=DFSClient_NONMAPREDUCE_1153590032_1, ugi=
svchdc36q@VISA.COM (auth:KERBEROS)]]]
20/05/26 15:38:15 INFO HiveConf: Found configuration file
file:/etc/spark2/2.6.5.179-4/0/hive-site.xml
20/05/26 15:38:16 INFO HoodieTableMetaClient: Loading HoodieTableMetaClient
from /projects/cdp/data/cdp_reporting/trr
20/05/26 15:38:16 INFO FSUtils: Hadoop Configuration: fs.defaultFS:
[hdfs://oprhqanameservice], Config:[Configuration: core-default.xml,
core-site.xml, mapred-default.xml, mapred-site.xml, yarn-default.xml,
yarn-site.xml, hdfs-default.xml, hdfs-site.xml, __spark_hadoop_conf__.xml,
file:/etc/spark2/2.6.5.179-4/0/hive-site.xml], FileSystem:
[DFS[DFSClient[clientName=DFSClient_NONMAPREDUCE_1153590032_1, ugi=
svchdc36q@VISA.COM (auth:KERBEROS)]]]
20/05/26 15:38:16 INFO HoodieTableConfig: Loading dataset properties from
/projects/cdp/data/cdp_reporting/trr/.hoodie/hoodie.properties
20/05/26 15:38:16 INFO HoodieTableMetaClient: Finished Loading Table of
type COPY_ON_WRITE from /projects/cdp/data/cdp_reporting/trr
20/05/26 15:38:16 INFO HoodieTableMetaClient: Loading Active commit
timeline for /projects/cdp/data/cdp_reporting/trr
20/05/26 15:38:16 INFO HoodieActiveTimeline: Loaded instants
java.util.stream.ReferencePipeline$Head@a1fca5a
20/05/26 15:38:16 INFO HoodieHiveClient: Creating hive connection
jdbc:hive2://server1.visa.com:2181,server2.visa.com:2181,
server3.visa.com:2181/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2
20/05/26 15:38:16 INFO Utils: Supplied authorities: server1.visa.com:2181,
server2.visa.com:2181,server3.visa.com:2181
20/05/26 15:38:16 INFO CuratorFrameworkImpl: Starting
20/05/26 15:38:16 INFO ZooKeeper: Client
environment:zookeeper.version=3.4.6-4--1, built on 08/09/2019 23:18 GMT
20/05/26 15:38:16 INFO ZooKeeper: Client environment:host.name=
server4.visa.com
20/05/26 15:38:16 INFO ZooKeeper: Client environment:java.version=1.8.0_241
20/05/26 15:38:16 INFO ZooKeeper: Client environment:java.vendor=Oracle
Corporation
20/05/26 15:38:16 INFO ZooKeeper: Client
environment:java.home=/usr/java/jdk1.8.0_241-amd64/jre
20/05/26 15:38:16 INFO ZooKeeper: Client
environment:java.class.path=/usr/hdp/current/spark2-client/conf/:/usr/hdp/current/spark2-client/jars/hk2-api-2.4.0-b34.jar:/usr/hdp/current/spark2-client/jars/JavaEWAH-0.3.2.jar:/usr/hdp/current/spark2-client/jars/commons-pool-1.5.4.jar:/usr/hdp/current/spark2-client/jars/RoaringBitmap-0.5.11.jar:/usr/hdp/current/spark2-client/jars/hk2-locator-2.4.0-b34.jar:/usr/hdp/current/spark2-client/jars/ST4-4.0.4.jar:/usr/hdp/current/spark2-client/jars/compress-lzf-1.0.3.jar:/usr/hdp/current/spark2-client/jars/activation-1.1.1.jar:/usr/hdp/current/spark2-client/jars/core-1.1.2.jar:/usr/hdp/current/spark2-client/jars/aircompressor-0.8.jar:/usr/hdp/current/spark2-client/jars/hk2-utils-2.4.0-b34.jar:/usr/hdp/current/spark2-client/jars/antlr-2.7.7.jar:/usr/hdp/current/spark2-client/jars/curator-client-2.7.1.jar:/usr/hdp/current/spark2-client/jars/antlr-runtime-3.4.jar:/usr/hdp/current/spark2-client/jars/curator-framework-2.7.1.jar:/usr/hdp/current/spark2-client/jars/antlr4-runtime-4.7.jar:/usr/hdp/current/spark2-client/jars/ivy-2.4.0.jar:/usr/hdp/current/spark2-client/jars/aopalliance-1.0.jar:/usr/hdp/current/spark2-client/jars/commons-io-2.4.jar:/usr/hdp/current/spark2-client/jars/janino-3.0.8.jar:/usr/hdp/current/spark2-client/jars/aopalliance-repackaged-2.4.0-b34.jar:/usr/hdp/current/spark2-client/jars/commons-collections-3.2.2.jar:/usr/hdp/current/spark2-client/jars/apache-log4j-extras-1.2.17.jar:/usr/hdp/current/spark2-client/jars/curator-recipes-2.7.1.jar:/usr/hdp/current/spark2-client/jars/apacheds-i18n-2.0.0-M15.jar:/usr/hdp/current/spark2-client/jars/commons-cli-1.2.jar:/usr/hdp/current/spark2-client/jars/javax.inject-1.jar:/usr/hdp/current/spark2-client/jars/apacheds-kerberos-codec-2.0.0-M15.jar:/usr/hdp/current/spark2-client/jars/datanucleus-api-jdo-3.2.6.jar:/usr/hdp/current/spark2-client/jars/api-asn1-api-1.0.0-M20.jar:/usr/hdp/current/spark2-client/jars/datanucleus-core-3.2.10.jar:/usr/hdp/current/spark2-client/jars/api-util-1.0.0-M20.jar:/usr/hdp/current/spark2-client/jars/datanucleus-rdbms-3.2.9.jar:/usr/hdp/current/spark2-client/jars/arpack_combined_all-0.1.jar:/usr/hdp/current/spark2-client/jars/derby-10.12.1.1.jar:/usr/hdp/current/spark2-client/jars/arrow-format-0.8.0.jar:/usr/hdp/current/spark2-client/jars/eigenbase-properties-1.1.5.jar:/usr/hdp/current/spark2-client/jars/arrow-memory-0.8.0.jar:/usr/hdp/current/spark2-client/jars/flatbuffers-1.2.0-3f79e055.jar:/usr/hdp/current/spark2-client/jars/arrow-vector-0.8.0.jar:/usr/hdp/current/spark2-client/jars/hppc-0.7.2.jar:/usr/hdp/current/spark2-client/jars/avro-1.7.7.jar:/usr/hdp/current/spark2-client/jars/httpclient-4.5.2.jar:/usr/hdp/current/spark2-client/jars/avro-ipc-1.7.7.jar:/usr/hdp/current/spark2-client/jars/commons-compiler-3.0.8.jar:/usr/hdp/current/spark2-client/jars/avro-mapred-1.7.7-hadoop2.jar:/usr/hdp/current/spark2-client/jars/commons-compress-1.4.1.jar:/usr/hdp/current/spark2-client/jars/aws-java-sdk-core-1.10.6.jar:/usr/hdp/current/spark2-client/jars/guava-14.0.1.jar:/usr/hdp/current/spark2-client/jars/aws-java-sdk-kms-1.10.6.jar:/usr/hdp/current/spark2-client/jars/gson-2.2.4.jar:/usr/hdp/current/spark2-client/jars/aws-java-sdk-s3-1.10.6.jar:/usr/hdp/current/spark2-client/jars/commons-configuration-1.6.jar:/usr/hdp/current/spark2-client/jars/azure-data-lake-store-sdk-2.1.4.jar:/usr/hdp/current/spark2-client/jars/commons-lang-2.6.jar:/usr/hdp/current/spark2-client/jars/jpam-1.1.jar:/usr/hdp/current/spark2-client/jars/azure-keyvault-core-0.8.0.jar:/usr/hdp/current/spark2-client/jars/guice-3.0.jar:/usr/hdp/current/spark2-client/jars/azure-storage-5.4.0.jar:/usr/hdp/current/spark2-client/jars/httpcore-4.4.4.jar:/usr/hdp/current/spark2-client/jars/base64-2.3.8.jar:/usr/hdp/current/spark2-client/jars/guice-servlet-3.0.jar:/usr/hdp/current/spark2-client/jars/bcprov-jdk15on-1.58.jar:/usr/hdp/current/spark2-client/jars/hadoop-aws-2.7.3.2.6.5.179-4.jar:/usr/hdp/current/spark2-client/jars/bonecp-0.8.0.RELEASE.jar:/usr/hdp/current/spark2-client/jars/commons-lang3-3.5.jar:/usr/hdp/current/spark2-client/jars/breeze-macros_2.11-0.13.2.jar:/usr/hdp/current/spark2-client/jars/hadoop-auth-2.7.3.2.6.5.179-4.jar:/usr/hdp/current/spark2-client/jars/breeze_2.11-0.13.2.jar:/usr/hdp/current/spark2-client/jars/commons-codec-1.10.jar:/usr/hdp/current/spark2-client/jars/jta-1.1.jar:/usr/hdp/current/spark2-client/jars/calcite-avatica-1.2.0-incubating.jar:/usr/hdp/current/spark2-client/jars/commons-logging-1.1.3.jar:/usr/hdp/current/spark2-client/jars/calcite-core-1.2.0-incubating.jar:/usr/hdp/current/spark2-client/jars/commons-math3-3.4.1.jar:/usr/hdp/current/spark2-client/jars/calcite-linq4j-1.2.0-incubating.jar:/usr/hdp/current/spark2-client/jars/hadoop-azure-2.7.3.2.6.5.179-4.jar:/usr/hdp/current/spark2-client/jars/chill-java-0.8.4.jar:/usr/hdp/current/spark2-client/jars/hadoop-client-2.7.3.2.6.5.179-4.jar:/usr/hdp/current/spark2-client/jars/chill_2.11-0.8.4.jar:/usr/hdp/current/spark2-client/jars/hadoop-common-2.7.3.2.6.5.179-4.jar:/usr/hdp/current/spark2-client/jars/commons-beanutils-1.7.0.jar:/usr/hdp/current/spark2-client/jars/gcs-connector-1.8.1.2.6.5.179-4-shaded.jar:/usr/hdp/current/spark2-client/jars/commons-beanutils-core-1.8.0.jar:/usr/hdp/current/spark2-client/jars/commons-crypto-1.0.0.jar:/usr/hdp/current/spark2-client/jars/hadoop-hdfs-2.7.3.2.6.5.179-4.jar:/usr/hdp/current/spark2-client/jars/commons-dbcp-1.4.jar:/usr/hdp/current/spark2-client/jars/hive-beeline-1.21.2.2.6.5.179-4.jar:/usr/hdp/current/spark2-client/jars/commons-digester-1.8.jar:/usr/hdp/current/spark2-client/jars/hive-cli-1.21.2.2.6.5.179-4.jar:/usr/hdp/current/spark2-client/jars/commons-httpclient-3.1.jar:/usr/hdp/current/spark2-client/jars/jackson-core-2.6.7.jar:/usr/hdp/current/spark2-client/jars/commons-net-2.2.jar:/usr/hdp/current/spark2-client/jars/javolution-5.5.1.jar:/usr/hdp/current/spark2-client/jars/jersey-server-2.22.2.jar:/usr/hdp/current/spark2-client/jars/xz-1.0.jar:/usr/hdp/current/spark2-client/jars/hadoop-annotations-2.7.3.2.6.5.179-4.jar:/usr/hdp/current/spark2-client/jars/jets3t-0.9.4.jar:/usr/hdp/current/spark2-client/jars/okhttp-2.7.5.jar:/usr/hdp/current/spark2-client/jars/hadoop-azure-datalake-2.7.3.2.6.5.179-4.jar:/usr/hdp/current/spark2-client/jars/jersey-container-servlet-2.22.2.jar:/usr/hdp/current/spark2-client/jars/hadoop-mapreduce-client-app-2.7.3.2.6.5.179-4.jar:/usr/hdp/current/spark2-client/jars/jdo-api-3.0.1.jar:/usr/hdp/current/spark2-client/jars/libfb303-0.9.3.jar:/usr/hdp/current/spark2-client/jars/hadoop-mapreduce-client-common-2.7.3.2.6.5.179-4.jar:/usr/hdp/current/spark2-client/jars/jersey-container-servlet-core-2.22.2.jar:/usr/hdp/current/spark2-client/jars/hadoop-mapreduce-client-core-2.7.3.2.6.5.179-4.jar:/usr/hdp/current/spark2-client/jars/jersey-client-2.22.2.jar:/usr/hdp/current/spark2-client/jars/netty-3.9.9.Final.jar:/usr/hdp/current/spark2-client/jars/hadoop-mapreduce-client-jobclient-2.7.3.2.6.5.179-4.jar:/usr/hdp/current/spark2-client/jars/jersey-common-2.22.2.jar:/usr/hdp/current/spark2-client/jars/netty-all-4.1.17.Final.jar:/usr/hdp/current/spark2-client/jars/hadoop-mapreduce-client-shuffle-2.7.3.2.6.5.179-4.jar:/usr/hdp/current/spark2-client/jars/jetty-6.1.26.hwx.jar:/usr/hdp/current/spark2-client/jars/okio-1.6.0.jar:/usr/hdp/current/spark2-client/jars/hadoop-openstack-2.7.3.2.6.5.179-4.jar:/usr/hdp/current/spark2-client/jars/jetty-sslengine-6.1.26.hwx.jar:/usr/hdp/current/spark2-client/jars/hadoop-yarn-api-2.7.3.2.6.5.179-4.jar:/usr/hdp/current/spark2-client/jars/jetty-util-6.1.26.hwx.jar:/usr/hdp/current/spark2-client/jars/hadoop-yarn-client-2.7.3.2.6.5.179-4.jar:/usr/hdp/current/spark2-client/jars/jline-2.12.1.jar:/usr/hdp/current/spark2-client/jars/opencsv-2.3.jar:/usr/hdp/current/spark2-client/jars/hadoop-yarn-common-2.7.3.2.6.5.179-4.jar:/usr/hdp/current/spark2-client/jars/joda-time-2.9.3.jar:/usr/hdp/current/spark2-client/jars/oro-2.0.8.jar:/usr/hdp/current/spark2-client/jars/hadoop-yarn-registry-2.7.3.2.6.5.179-4.jar:/usr/hdp/current/spark2-client/jars/jersey-guava-2.22.2.jar:/usr/hdp/current/spark2-client/jars/paranamer-2.8.jar:/usr/hdp/current/spark2-client/jars/hadoop-yarn-server-common-2.7.3.2.6.5.179-4.jar:/usr/hdp/current/spark2-client/jars/jersey-media-jaxb-2.22.2.jar:/usr/hdp/current/spark2-client/jars/py4j-0.10.6.jar:/usr/hdp/current/spark2-client/jars/hadoop-yarn-server-web-proxy-2.7.3.2.6.5.179-4.jar:/usr/hdp/current/spark2-client/jars/json4s-ast_2.11-3.2.11.jar:/usr/hdp/current/spark2-client/jars/hive-exec-1.21.2.2.6.5.179-4.jar:/usr/hdp/current/spark2-client/jars/json4s-core_2.11-3.2.11.jar:/usr/hdp/current/spark2-client/jars/hive-jdbc-1.21.2.2.6.5.179-4.jar:/usr/hdp/current/spark2-client/jars/jodd-core-3.5.2.jar:/usr/hdp/current/spark2-client/jars/pyrolite-4.13.jar:/usr/hdp/current/spark2-client/jars/hive-metastore-1.21.2.2.6.5.179-4.jar:/usr/hdp/current/spark2-client/jars/json4s-jackson_2.11-3.2.11.jar:/usr/hdp/current/spark2-client/jars/htrace-core-3.1.0-incubating.jar:/usr/hdp/current/spark2-client/jars/jsp-api-2.1.jar:/usr/hdp/current/spark2-client/jars/jackson-annotations-2.6.7.jar:/usr/hdp/current/spark2-client/jars/libthrift-0.9.3.jar:/usr/hdp/current/spark2-client/jars/jackson-core-asl-1.9.13.jar:/usr/hdp/current/spark2-client/jars/jsr305-1.3.9.jar:/usr/hdp/current/spark2-client/jars/jackson-databind-2.6.7.1.jar:/usr/hdp/current/spark2-client/jars/jtransforms-2.4.0.jar:/usr/hdp/current/spark2-client/jars/jackson-dataformat-cbor-2.6.7.jar:/usr/hdp/current/spark2-client/jars/log4j-1.2.17.jar:/usr/hdp/current/spark2-client/jars/jackson-jaxrs-1.9.13.jar:/usr/hdp/current/spark2-client/jars/jul-to-slf4j-1.7.16.jar:/usr/hdp/current/spark2-client/jars/jackson-mapper-asl-1.9.13.jar:/usr/hdp/current/spark2-client/jars/kryo-shaded-3.0.3.jar:/usr/hdp/current/spark2-client/jars/jackson-module-paranamer-2.7.9.jar:/usr/hdp/current/spark2-client/jars/json-smart-1.3.1.jar:/usr/hdp/current/spark2-client/jars/scalap-2.11.8.jar:/usr/hdp/current/spark2-client/jars/jackson-module-scala_2.11-2.6.7.1.jar:/usr/hdp/current/spark2-client/jars/lz4-java-1.4.0.jar:/usr/hdp/current/spark2-client/jars/jackson-xc-1.9.13.jar:/usr/hdp/current/spark2-client/jars/machinist_2.11-0.6.1.jar:/usr/hdp/current/spark2-client/jars/java-xmlbuilder-1.1.jar:/usr/hdp/current/spark2-client/jars/macro-compat_2.11-1.1.1.jar:/usr/hdp/current/spark2-client/jars/javassist-3.18.1-GA.jar:/usr/hdp/current/spark2-client/jars/leveldbjni-all-1.8.jar:/usr/hdp/current/spark2-client/jars/javax.annotation-api-1.2.jar:/usr/hdp/current/spark2-client/jars/metrics-core-3.1.5.jar:/usr/hdp/current/spark2-client/jars/javax.inject-2.4.0-b34.jar:/usr/hdp/current/spark2-client/jars/metrics-graphite-3.1.5.jar:/usr/hdp/current/spark2-client/jars/javax.servlet-api-3.1.0.jar:/usr/hdp/current/spark2-client/jars/metrics-json-3.1.5.jar:/usr/hdp/current/spark2-client/jars/javax.ws.rs-api-2.0.1.jar:/usr/hdp/current/spark2-client/jars/nimbus-jose-jwt-4.41.1.jar:/usr/hdp/current/spark2-client/jars/jaxb-api-2.2.2.jar:/usr/hdp/current/spark2-client/jars/metrics-jvm-3.1.5.jar:/usr/hdp/current/spark2-client/jars/jcip-annotations-1.0-1.jar:/usr/hdp/current/spark2-client/jars/minlog-1.3.0.jar:/usr/hdp/current/spark2-client/jars/jcl-over-slf4j-1.7.16.jar:/usr/hdp/current/spark2-client/jars/parquet-column-1.8.2.jar:/usr/hdp/current/spark2-client/jars/objenesis-2.1.jar:/usr/hdp/current/spark2-client/jars/spire_2.11-0.13.0.jar:/usr/hdp/current/spark2-client/jars/orc-core-1.4.3.2.6.5.179-4-nohive.jar:/usr/hdp/current/spark2-client/jars/stax-api-1.0-2.jar:/usr/hdp/current/spark2-client/jars/orc-mapreduce-1.4.3.2.6.5.179-4-nohive.jar:/usr/hdp/current/spark2-client/jars/osgi-resource-locator-1.0.1.jar:/usr/hdp/current/spark2-client/jars/parquet-common-1.8.2.jar:/usr/hdp/current/spark2-client/jars/parquet-encoding-1.8.2.jar:/usr/hdp/current/spark2-client/jars/parquet-format-2.3.1.jar:/usr/hdp/current/spark2-client/jars/parquet-hadoop-1.8.2.jar:/usr/hdp/current/spark2-client/jars/parquet-hadoop-bundle-1.6.0.jar:/usr/hdp/current/spark2-client/jars/parquet-jackson-1.8.2.jar:/usr/hdp/current/spark2-client/jars/protobuf-java-2.5.0.jar:/usr/hdp/current/spark2-client/jars/scala-compiler-2.11.8.jar:/usr/hdp/current/spark2-client/jars/scala-library-2.11.8.jar:/usr/hdp/current/spark2-client/jars/stax-api-1.0.1.jar:/usr/hdp/current/spark2-client/jars/scala-parser-combinators_2.11-1.0.4.jar:/usr/hdp/current/spark2-client/jars/scala-reflect-2.11.8.jar:/usr/hdp/current/spark2-client/jars/scala-xml_2.11-1.0.5.jar:/usr/hdp/current/spark2-client/jars/shapeless_2.11-2.3.2.jar:/usr/hdp/current/spark2-client/jars/slf4j-api-1.7.16.jar:/usr/hdp/current/spark2-client/jars/slf4j-log4j12-1.7.16.jar:/usr/hdp/current/spark2-client/jars/snappy-0.2.jar:/usr/hdp/current/spark2-client/jars/snappy-java-1.1.2.6.jar:/usr/hdp/current/spark2-client/jars/stream-2.7.0.jar:/usr/hdp/current/spark2-client/jars/spark-catalyst_2.11-2.3.0.2.6.5.179-4.jar:/usr/hdp/current/spark2-client/jars/stringtemplate-3.2.1.jar:/usr/hdp/current/spark2-client/jars/spark-cloud_2.11-2.3.0.2.6.5.179-4.jar:/usr/hdp/current/spark2-client/jars/super-csv-2.2.0.jar:/usr/hdp/current/spark2-client/jars/spark-core_2.11-2.3.0.2.6.5.179-4.jar:/usr/hdp/current/spark2-client/jars/univocity-parsers-2.5.9.jar:/usr/hdp/current/spark2-client/jars/spark-graphx_2.11-2.3.0.2.6.5.179-4.jar:/usr/hdp/current/spark2-client/jars/spark-unsafe_2.11-2.3.0.2.6.5.179-4.jar:/usr/hdp/current/spark2-client/jars/spark-hadoop-cloud_2.11-2.3.0.2.6.5.179-4.jar:/usr/hdp/current/spark2-client/jars/spark-tags_2.11-2.3.0.2.6.5.179-4.jar:/usr/hdp/current/spark2-client/jars/spark-hive-thriftserver_2.11-2.3.0.2.6.5.179-4.jar:/usr/hdp/current/spark2-client/jars/validation-api-1.1.0.Final.jar:/usr/hdp/current/spark2-client/jars/spark-hive_2.11-2.3.0.2.6.5.179-4.jar:/usr/hdp/current/spark2-client/jars/xbean-asm5-shaded-4.4.jar:/usr/hdp/current/spark2-client/jars/spark-kvstore_2.11-2.3.0.2.6.5.179-4.jar:/usr/hdp/current/spark2-client/jars/xercesImpl-2.9.1.jar:/usr/hdp/current/spark2-client/jars/spark-launcher_2.11-2.3.0.2.6.5.179-4.jar:/usr/hdp/current/spark2-client/jars/spark-mllib-local_2.11-2.3.0.2.6.5.179-4.jar:/usr/hdp/current/spark2-client/jars/xmlenc-0.52.jar:/usr/hdp/current/spark2-client/jars/spark-mllib_2.11-2.3.0.2.6.5.179-4.jar:/usr/hdp/current/spark2-client/jars/spark-yarn_2.11-2.3.0.2.6.5.179-4.jar:/usr/hdp/current/spark2-client/jars/spark-network-common_2.11-2.3.0.2.6.5.179-4.jar:/usr/hdp/current/spark2-client/jars/spire-macros_2.11-0.13.0.jar:/usr/hdp/current/spark2-client/jars/spark-network-shuffle_2.11-2.3.0.2.6.5.179-4.jar:/usr/hdp/current/spark2-client/jars/zookeeper-3.4.6.2.6.5.179-4.jar:/usr/hdp/current/spark2-client/jars/spark-repl_2.11-2.3.0.2.6.5.179-4.jar:/usr/hdp/current/spark2-client/jars/zstd-jni-1.3.2-2.jar:/usr/hdp/current/spark2-client/jars/spark-sketch_2.11-2.3.0.2.6.5.179-4.jar:/usr/hdp/current/spark2-client/jars/spark-sql_2.11-2.3.0.2.6.5.179-4.jar:/usr/hdp/current/spark2-client/jars/spark-streaming_2.11-2.3.0.2.6.5.179-4.jar:/usr/hdp/2.6.5.179-4/hadoop/conf/
20/05/26 15:38:16 INFO ZooKeeper: Client
environment:java.library.path=:/export/home/sobla/oracle_client/instantclient_19_5:/usr/hdp/current/hadoop-client/lib/native:/usr/hdp/current/hadoop-client/lib/native/Linux-amd64-64:/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
20/05/26 15:38:16 INFO ZooKeeper: Client environment:java.io.tmpdir=/tmp
20/05/26 15:38:16 INFO ZooKeeper: Client environment:java.compiler=<NA>
20/05/26 15:38:16 INFO ZooKeeper: Client environment:os.name=Linux
20/05/26 15:38:16 INFO ZooKeeper: Client environment:os.arch=amd64
20/05/26 15:38:16 INFO ZooKeeper: Client
environment:os.version=3.10.0-1062.9.1.el7.x86_64
20/05/26 15:38:16 INFO ZooKeeper: Client environment:user.name=svchdc36q
20/05/26 15:38:16 INFO ZooKeeper: Client
environment:user.home=/home/svchdc36q
20/05/26 15:38:16 INFO ZooKeeper: Client
environment:user.dir=/home/svchdc36q
*20/05/26 15:38:16 INFO ZooKeeper: Initiating client connection,
connectString=server1.visa.com:2181
<http://server1.visa.com:2181>,server2.visa.com:2181
<http://server2.visa.com:2181>,server3.visa.com:2181
<http://server3.visa.com:2181> sessionTimeout=60000
watcher=org.apache.curator.ConnectionState@4ed31bc9*
*20/05/26 15:38:16 INFO ClientCnxn: Opening socket connection to server
server2.visa.com/x.x.x.x:2181 <http://server2.visa.com/x.x.x.x:2181>. Will
not attempt to authenticate using SASL (unknown error)*
*20/05/26 15:38:16 INFO ClientCnxn: Socket connection established,
initiating session, client: /x.x.x.x:36938, server:
server2.visa.com/x.x.x.x:2181 <http://server2.visa.com/x.x.x.x:2181>*
*20/05/26 15:38:16 INFO ClientCnxn: Session establishment complete on
server server2.visa.com/x.x.x.x:2181
<http://server2.visa.com/x.x.x.x:2181>, sessionid = 0x27234630fb51f5b,
negotiated timeout = 40000*
20/05/26 15:38:16 INFO ConnectionStateManager: State change: CONNECTED
20/05/26 15:38:17 INFO ZooKeeper: Session: 0x27234630fb51f5b closed
20/05/26 15:38:17 INFO ClientCnxn: EventThread shut down
20/05/26 15:38:17 INFO Utils: Resolved authority: server2.visa.com:10000
20/05/26 15:38:17 INFO HiveConnection: Will try to open client transport
with JDBC Uri: jdbc:hive2://
server2.visa.com:10000/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2
20/05/26 15:38:19 INFO HoodieHiveClient: Successfully established Hive
connection to jdbc:hive2://server1.visa.com:2181,server2.visa.com:2181,
server3.visa.com:2181/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2
20/05/26 15:38:19 INFO metastore: Trying to connect to metastore with URI
thrift://server2.visa.com:9083
20/05/26 15:38:19 INFO metastore: Opened a connection to metastore, current
connections: 1
20/05/26 15:38:19 INFO metastore: Connected to metastore.
20/05/26 15:38:19 INFO HiveSyncTool: Trying to sync hoodie table trr with
base path /projects/cdp/data/cdp_reporting/trr of type COPY_ON_WRITE
20/05/26 15:38:19 ERROR DBUtil$: [App] *********************** Exception
occurred in baseTableWrite for trr : Failed to check if table exists trr
org.apache.hudi.hive.HoodieHiveSyncException: Failed to check if table
exists trr
at
org.apache.hudi.hive.HoodieHiveClient.doesTableExist(HoodieHiveClient.java:459)
at org.apache.hudi.hive.HiveSyncTool.syncHoodieTable(HiveSyncTool.java:91)
at org.apache.hudi.hive.HiveSyncTool.syncHoodieTable(HiveSyncTool.java:67)
at
org.apache.hudi.HoodieSparkSqlWriter$.syncHive(HoodieSparkSqlWriter.scala:235)
at
org.apache.hudi.HoodieSparkSqlWriter$.write(HoodieSparkSqlWriter.scala:169)
at org.apache.hudi.DefaultSource.createRelation(DefaultSource.scala:91)
at
org.apache.spark.sql.execution.datasources.SaveIntoDataSourceCommand.run(SaveIntoDataSourceCommand.scala:46)
at
org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult$lzycompute(commands.scala:70)
at
org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult(commands.scala:68)
at
org.apache.spark.sql.execution.command.ExecutedCommandExec.doExecute(commands.scala:86)
at
org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:131)
at
org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:127)
at
org.apache.spark.sql.execution.SparkPlan$$anonfun$executeQuery$1.apply(SparkPlan.scala:155)
at
org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
at
org.apache.spark.sql.execution.SparkPlan.executeQuery(SparkPlan.scala:152)
at org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:127)
at
org.apache.spark.sql.execution.QueryExecution.toRdd$lzycompute(QueryExecution.scala:80)
at
org.apache.spark.sql.execution.QueryExecution.toRdd(QueryExecution.scala:80)
at
org.apache.spark.sql.DataFrameWriter$$anonfun$runCommand$1.apply(DataFrameWriter.scala:654)
at
org.apache.spark.sql.DataFrameWriter$$anonfun$runCommand$1.apply(DataFrameWriter.scala:654)
at
org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:77)
at
org.apache.spark.sql.DataFrameWriter.runCommand(DataFrameWriter.scala:654)
at
org.apache.spark.sql.DataFrameWriter.saveToV1Source(DataFrameWriter.scala:273)
at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:267)
at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:225)
at com.cybs.cdp.reporting.trr.DBUtil$.transactionTableWrite(DBUtil.scala:62)
at com.cybs.cdp.reporting.trr.TRREngine$.startEngine(TRREngine.scala:45)
at com.cybs.cdp.reporting.trr.TRREngine$.main(TRREngine.scala:23)
at com.cybs.cdp.reporting.trr.TRREngine.main(TRREngine.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
at
org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:906)
at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:197)
at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:227)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:136)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: org.apache.thrift.TApplicationException: Invalid method name:
'get_table_req'
at
org.apache.thrift.TApplicationException.read(TApplicationException.java:111)
at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:79)
at
org.apache.hudi.org.apache.hadoop_hive.metastore.api.ThriftHiveMetastore$Client.recv_get_table_req(ThriftHiveMetastore.java:1563)
at
org.apache.hudi.org.apache.hadoop_hive.metastore.api.ThriftHiveMetastore$Client.get_table_req(ThriftHiveMetastore.java:1550)
at
org.apache.hudi.org.apache.hadoop_hive.metastore.HiveMetaStoreClient.tableExists(HiveMetaStoreClient.java:1443)
at
org.apache.hudi.hive.HoodieHiveClient.doesTableExist(HoodieHiveClient.java:457)
... 38 more
Exception in thread "main" java.lang.Exception: Failed to check if table
exists trr
at com.cybs.cdp.reporting.trr.DBUtil$.transactionTableWrite(DBUtil.scala:69)
at com.cybs.cdp.reporting.trr.TRREngine$.startEngine(TRREngine.scala:45)
at com.cybs.cdp.reporting.trr.TRREngine$.main(TRREngine.scala:23)
at com.cybs.cdp.reporting.trr.TRREngine.main(TRREngine.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
at
org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:906)
at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:197)
at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:227)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:136)
Thanks,
Selva
On Mon, Mar 2, 2020 at 10:02 AM Vinoth Chandar <vi...@apache.org> wrote:
> Hi Selva,
>
> See if this helps.
>
> https://lists.apache.org/thread.html/e1fd539ac438276dd7feb2bc813bf85f84a95f7f25b638488eb2e110%40%3Cdev.hudi.apache.org%3E
>
> Its long thread, but you can probably skim to the last few conversations
> around Hive 1.x
>
> Thanks
> Vinoth
>
> On Sun, Mar 1, 2020 at 5:26 PM selvaraj periyasamy <
> selvaraj.periyasamy1983@gmail.com> wrote:
>
> > Thanks Vinoth. We do have plan to move hive 2.x version in near future.
> > Can I get any info on the workaround for hive 1.x versions?
> >
> > Thanks,
> > Selva
> >
> > On Sun, Mar 1, 2020 at 3:19 PM Vinoth Chandar <vi...@apache.org> wrote:
> >
> > > We have dropped support for Hive 1.x, a while back. Would you be able
> to
> > > move to Hive 2.x?
> > >
> > > IIRC there were some workarounds discussed on this thread before. But,
> > > given the push towards Hive 3.x, its good to be on 2.x atleast ..
> > > Let me know and we can go from there :)
> > >
> > > On Sun, Mar 1, 2020 at 1:09 PM selvaraj periyasamy <
> > > selvaraj.periyasamy1983@gmail.com> wrote:
> > >
> > > > I am using Hudi 0.5.0 and then write using sparkwriter.
> > > >
> > > > My spark version is 2.3.0
> > > > Scala version 2.11.8
> > > > Hive version 1.2.2
> > > >
> > > > Write is success but hive call is failing. When checked some google
> > > > reference, It seems to be an hive client is higher version the
> server.
> > > > Since Hudi is built on hive 2.3.1, Is there a way to use 1.2.2?
> > > >
> > > > 2020-03-01 12:16:50 WARN HoodieSparkSqlWriter$:110 - hoodie dataset
> at
> > > >
> hdfs://localhost:9000/projects/cdp/data/attunity_poc/attunity_rep_base
> > > > already exists. Deleting existing data & overwriting with new data.
> > > > [Stage 111:============================>
> > > > 2020-03-01
> > > > 12:16:51 ERROR HiveConnection:697 - Error opening session
> > > > org.apache.thrift.TApplicationException: Required field
> > 'client_protocol'
> > > > is unset! Struct:TOpenSessionReq(client_protocol:null,
> > > >
> > > >
> > >
> >
> configuration:{set:hiveconf:hive.server2.thrift.resultset.default.fetch.size=1000,
> > > > use:database=default})
> > > > at
> > > >
> > > >
> > >
> >
> org.apache.thrift.TApplicationException.read(TApplicationException.java:111)
> > > > at
> org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:79)
> > > > at
> > > >
> > > >
> > >
> >
> org.apache.hudi.org.apache.hive.service.rpc.thrift.TCLIService$Client.recv_OpenSession(TCLIService.java:168)
> > > > at
> > > >
> > > >
> > >
> >
> org.apache.hudi.org.apache.hive.service.rpc.thrift.TCLIService$Client.OpenSession(TCLIService.java:155)
> > > >
> > > >
> > > > Thanks,
> > > > Selva
> > > >
> > >
> >
>
Re: Hudi 0.5.0 -> Hive JDBC call fails
Posted by Vinoth Chandar <vi...@apache.org>.
Hi Selva,
See if this helps.
https://lists.apache.org/thread.html/e1fd539ac438276dd7feb2bc813bf85f84a95f7f25b638488eb2e110%40%3Cdev.hudi.apache.org%3E
Its long thread, but you can probably skim to the last few conversations
around Hive 1.x
Thanks
Vinoth
On Sun, Mar 1, 2020 at 5:26 PM selvaraj periyasamy <
selvaraj.periyasamy1983@gmail.com> wrote:
> Thanks Vinoth. We do have plan to move hive 2.x version in near future.
> Can I get any info on the workaround for hive 1.x versions?
>
> Thanks,
> Selva
>
> On Sun, Mar 1, 2020 at 3:19 PM Vinoth Chandar <vi...@apache.org> wrote:
>
> > We have dropped support for Hive 1.x, a while back. Would you be able to
> > move to Hive 2.x?
> >
> > IIRC there were some workarounds discussed on this thread before. But,
> > given the push towards Hive 3.x, its good to be on 2.x atleast ..
> > Let me know and we can go from there :)
> >
> > On Sun, Mar 1, 2020 at 1:09 PM selvaraj periyasamy <
> > selvaraj.periyasamy1983@gmail.com> wrote:
> >
> > > I am using Hudi 0.5.0 and then write using sparkwriter.
> > >
> > > My spark version is 2.3.0
> > > Scala version 2.11.8
> > > Hive version 1.2.2
> > >
> > > Write is success but hive call is failing. When checked some google
> > > reference, It seems to be an hive client is higher version the server.
> > > Since Hudi is built on hive 2.3.1, Is there a way to use 1.2.2?
> > >
> > > 2020-03-01 12:16:50 WARN HoodieSparkSqlWriter$:110 - hoodie dataset at
> > > hdfs://localhost:9000/projects/cdp/data/attunity_poc/attunity_rep_base
> > > already exists. Deleting existing data & overwriting with new data.
> > > [Stage 111:============================>
> > > 2020-03-01
> > > 12:16:51 ERROR HiveConnection:697 - Error opening session
> > > org.apache.thrift.TApplicationException: Required field
> 'client_protocol'
> > > is unset! Struct:TOpenSessionReq(client_protocol:null,
> > >
> > >
> >
> configuration:{set:hiveconf:hive.server2.thrift.resultset.default.fetch.size=1000,
> > > use:database=default})
> > > at
> > >
> > >
> >
> org.apache.thrift.TApplicationException.read(TApplicationException.java:111)
> > > at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:79)
> > > at
> > >
> > >
> >
> org.apache.hudi.org.apache.hive.service.rpc.thrift.TCLIService$Client.recv_OpenSession(TCLIService.java:168)
> > > at
> > >
> > >
> >
> org.apache.hudi.org.apache.hive.service.rpc.thrift.TCLIService$Client.OpenSession(TCLIService.java:155)
> > >
> > >
> > > Thanks,
> > > Selva
> > >
> >
>
Re: Hudi 0.5.0 -> Hive JDBC call fails
Posted by selvaraj periyasamy <se...@gmail.com>.
Thanks Vinoth. We do have plan to move hive 2.x version in near future.
Can I get any info on the workaround for hive 1.x versions?
Thanks,
Selva
On Sun, Mar 1, 2020 at 3:19 PM Vinoth Chandar <vi...@apache.org> wrote:
> We have dropped support for Hive 1.x, a while back. Would you be able to
> move to Hive 2.x?
>
> IIRC there were some workarounds discussed on this thread before. But,
> given the push towards Hive 3.x, its good to be on 2.x atleast ..
> Let me know and we can go from there :)
>
> On Sun, Mar 1, 2020 at 1:09 PM selvaraj periyasamy <
> selvaraj.periyasamy1983@gmail.com> wrote:
>
> > I am using Hudi 0.5.0 and then write using sparkwriter.
> >
> > My spark version is 2.3.0
> > Scala version 2.11.8
> > Hive version 1.2.2
> >
> > Write is success but hive call is failing. When checked some google
> > reference, It seems to be an hive client is higher version the server.
> > Since Hudi is built on hive 2.3.1, Is there a way to use 1.2.2?
> >
> > 2020-03-01 12:16:50 WARN HoodieSparkSqlWriter$:110 - hoodie dataset at
> > hdfs://localhost:9000/projects/cdp/data/attunity_poc/attunity_rep_base
> > already exists. Deleting existing data & overwriting with new data.
> > [Stage 111:============================>
> > 2020-03-01
> > 12:16:51 ERROR HiveConnection:697 - Error opening session
> > org.apache.thrift.TApplicationException: Required field 'client_protocol'
> > is unset! Struct:TOpenSessionReq(client_protocol:null,
> >
> >
> configuration:{set:hiveconf:hive.server2.thrift.resultset.default.fetch.size=1000,
> > use:database=default})
> > at
> >
> >
> org.apache.thrift.TApplicationException.read(TApplicationException.java:111)
> > at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:79)
> > at
> >
> >
> org.apache.hudi.org.apache.hive.service.rpc.thrift.TCLIService$Client.recv_OpenSession(TCLIService.java:168)
> > at
> >
> >
> org.apache.hudi.org.apache.hive.service.rpc.thrift.TCLIService$Client.OpenSession(TCLIService.java:155)
> >
> >
> > Thanks,
> > Selva
> >
>
Re: Hudi 0.5.0 -> Hive JDBC call fails
Posted by Vinoth Chandar <vi...@apache.org>.
We have dropped support for Hive 1.x, a while back. Would you be able to
move to Hive 2.x?
IIRC there were some workarounds discussed on this thread before. But,
given the push towards Hive 3.x, its good to be on 2.x atleast ..
Let me know and we can go from there :)
On Sun, Mar 1, 2020 at 1:09 PM selvaraj periyasamy <
selvaraj.periyasamy1983@gmail.com> wrote:
> I am using Hudi 0.5.0 and then write using sparkwriter.
>
> My spark version is 2.3.0
> Scala version 2.11.8
> Hive version 1.2.2
>
> Write is success but hive call is failing. When checked some google
> reference, It seems to be an hive client is higher version the server.
> Since Hudi is built on hive 2.3.1, Is there a way to use 1.2.2?
>
> 2020-03-01 12:16:50 WARN HoodieSparkSqlWriter$:110 - hoodie dataset at
> hdfs://localhost:9000/projects/cdp/data/attunity_poc/attunity_rep_base
> already exists. Deleting existing data & overwriting with new data.
> [Stage 111:============================>
> 2020-03-01
> 12:16:51 ERROR HiveConnection:697 - Error opening session
> org.apache.thrift.TApplicationException: Required field 'client_protocol'
> is unset! Struct:TOpenSessionReq(client_protocol:null,
>
> configuration:{set:hiveconf:hive.server2.thrift.resultset.default.fetch.size=1000,
> use:database=default})
> at
>
> org.apache.thrift.TApplicationException.read(TApplicationException.java:111)
> at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:79)
> at
>
> org.apache.hudi.org.apache.hive.service.rpc.thrift.TCLIService$Client.recv_OpenSession(TCLIService.java:168)
> at
>
> org.apache.hudi.org.apache.hive.service.rpc.thrift.TCLIService$Client.OpenSession(TCLIService.java:155)
>
>
> Thanks,
> Selva
>