You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@seatunnel.apache.org by "successok (via GitHub)" <gi...@apache.org> on 2023/07/28 03:26:06 UTC

[GitHub] [seatunnel] successok opened a new issue, #5174: [Bug] spark engine can not read hive source

successok opened a new issue, #5174:
URL: https://github.com/apache/seatunnel/issues/5174

   ### Search before asking
   
   - [X] I had searched in the [issues](https://github.com/apache/seatunnel/issues?q=is%3Aissue+label%3A%22bug%22) and found no similar issues.
   
   
   ### What happened
   
   23/07/28 10:45:06 INFO discovery.AbstractPluginDiscovery: Discovery plugin jar: Hive at: file:/bigdata/local/seatunnel-2.3.1/connectors/seatunnel/connector-hive-2.3.1.jar
   23/07/28 10:45:06 INFO discovery.AbstractPluginDiscovery: Load plugin: PluginIdentifier{engineType='seatunnel', pluginType='source', pluginName='Hive'} from classpath
   23/07/28 10:45:07 WARN conf.HiveConf: HiveConf of name hive.masking.algo does not exist
   23/07/28 10:45:07 INFO util.FileSystemUtils: Start Kerberos authentication using principal xxxx. and keytab xxxx.keytab
   23/07/28 10:45:07 INFO security.UserGroupInformation: Login successful for user xxxx.COM using keytab file xxxxx..keytab. Keytab auto renewal enabled : false
   23/07/28 10:45:07 INFO util.FileSystemUtils: Kerberos authentication successful
   Exception in thread "main" java.lang.NoSuchMethodError: org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(Lorg/apache/hadoop/hive/conf/HiveConf;)V
   	at org.apache.seatunnel.connectors.seatunnel.hive.utils.HiveMetaStoreProxy.<init>(HiveMetaStoreProxy.java:58)
   	at org.apache.seatunnel.connectors.seatunnel.hive.utils.HiveMetaStoreProxy.getInstance(HiveMetaStoreProxy.java:74)
   	at org.apache.seatunnel.connectors.seatunnel.hive.config.HiveConfig.getTableInfo(HiveConfig.java:59)
   	at org.apache.seatunnel.connectors.seatunnel.hive.source.HiveSource.prepare(HiveSource.java:123)
   	at org.apache.seatunnel.core.starter.spark.execution.SourceExecuteProcessor.initializePlugins(SourceExecuteProcessor.java:104)
   	at org.apache.seatunnel.core.starter.spark.execution.SparkAbstractPluginExecuteProcessor.<init>(SparkAbstractPluginExecuteProcessor.java:49)
   	at org.apache.seatunnel.core.starter.spark.execution.SourceExecuteProcessor.<init>(SourceExecuteProcessor.java:51)
   	at org.apache.seatunnel.core.starter.spark.execution.SparkExecution.<init>(SparkExecution.java:57)
   	at org.apache.seatunnel.core.starter.spark.command.SparkTaskExecuteCommand.execute(SparkTaskExecuteCommand.java:59)
   	at org.apache.seatunnel.core.starter.SeaTunnel.run(SeaTunnel.java:40)
   	at org.apache.seatunnel.core.starter.spark.SeaTunnelSpark.main(SeaTunnelSpark.java:35)
   	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
   	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   	at java.lang.reflect.Method.invoke(Method.java:498)
   	at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
   	at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:847)
   	at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:161)
   	at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:184)
   	at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:86)
   	at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:922)
   	at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:931)
   	at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
   23/07/28 10:45:07 INFO spark.SparkContext: Invoking stop() from shutdown hook
   
   
   ### SeaTunnel Version
   
   seatunnel2.3.1
   
   ### SeaTunnel Config
   
   ```conf
   env {
     spark.app.name = "apache-seatunnel-2.3.1_hive_to_dr"
     spark.yarn.queue = "root.hive"
     spark.executor.instances = 2
     spark.executor.cores = 4
     spark.driver.memory = "3g"
     spark.executor.memory = "4g"
     spark.ui.port = 1300
     spark.sql.catalogImplementation = "hive"
     spark.hadoop.hive.exec.dynamic.partition = "true"
     spark.hadoop.hive.exec.dynamic.partition.mode = "nonstrict"
     spark.network.timeout = "1200s"
     spark.sql.sources.partitionOverwriteMode = "dynamic"
     spark.yarn.executor.memoryOverhead = 800m
     spark.kryoserializer.buffer.max = 512m
     spark.task.maxFailures=0
     spark.executor.extraJavaOptions = "-Dfile.encoding=UTF-8"
     spark.driver.extraJavaOptions = "-Dfile.encoding=UTF-8"
     job.name = "apache-seatunnel-2.3.1_hive_to_dr"
   }
   source {
     Hive {
       table_name = "test.xxxx"
       metastore_uri = "thrift://xxxx:9083"
       result_table_name = "dwd_xxxx_t1"
       kerberos_principal = "xxxx"
       kerberos_keytab_path = "xxxx"
       hdfs_site_path = "/etc/hadoop/conf/hdfs-site.xml"
     }
   }
   transform {
    sql {
         query = "xxxxx"
         source_table_name = "xxx"
         result_table_name = "xxxx"
    }
   }
   sink {
     console {
       source_table_name = "xxxx"
     }
   }
   ```
   
   
   ### Running Command
   
   ```shell
   bin/start-seatunnel-spark-2-connector-v2.sh -m local[*] -e client -c ./config/example02.conf
   ```
   
   
   ### Error Exception
   
   ```log
   Exception in thread "main" java.lang.NoSuchMethodError: org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(Lorg/apache/hadoop/hive/conf/HiveConf;)V
   	at org.apache.seatunnel.connectors.seatunnel.hive.utils.HiveMetaStoreProxy.<init>(HiveMetaStoreProxy.java:58)
   
   I have checked the directory of Spark and confirmed that there is a `hive-exe.jar` in it. Additionally, the local spark-shell is able to read Hive files correctly.I also copied `hive-exe.jar` to `seatunnel-2.3.1/lib`  and  `seatunnel-2.3.1/connectors/seatunnel/`
   ```
   
   
   ### Zeta or Flink or Spark Version
   
   spark 2.4.7 by cloudera CDP 7.1.6
   
   
   
   ### Java or Scala Version
   
   java 8
   
   ### Screenshots
   
   ![image](https://github.com/apache/seatunnel/assets/50657384/32927c1b-20cf-4ecc-b217-3242aa6e92b8)
   
   
   ### Are you willing to submit PR?
   
   - [X] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [seatunnel] github-actions[bot] commented on issue #5174: [Bug] spark engine can not read hive source

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #5174:
URL: https://github.com/apache/seatunnel/issues/5174#issuecomment-1742254543

   This issue has been closed because it has not received response for too long time. You could reopen it if you encountered similar problems in the future.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Bug] spark engine can not read hive source [seatunnel]

Posted by "Light-Towers (via GitHub)" <gi...@apache.org>.
Light-Towers commented on issue #5174:
URL: https://github.com/apache/seatunnel/issues/5174#issuecomment-1859866895

   @successok  the problem has been solved now?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [seatunnel] CheneyYin commented on issue #5174: [Bug] spark engine can not read hive source

Posted by "CheneyYin (via GitHub)" <gi...@apache.org>.
CheneyYin commented on issue #5174:
URL: https://github.com/apache/seatunnel/issues/5174#issuecomment-1657400454

   Your environment used [Apache Hive : 3.1.3000.7.1.6.0-297](https://docs.cloudera.com/cdp-private-cloud-base/7.1.6/runtime-release-notes/topics/rt-pvc-runtime-component-versions.html). But `connector-hive` just compatible with `Hive 2`. 
   The constructor of  `HiveMetaStoreClient` in `Hive 2.3.9` is different from `Hive 3.1.3`.
   
   ```java
   // Hive 2.3.9
   public HiveMetaStoreClient(HiveConf conf) throws MetaException {
   ...
   }
   
   public HiveMetaStoreClient(HiveConf conf, HiveMetaHookLoader hookLoader, Boolean allowEmbedded) {
   ...
   }
   ```
   
   ```java
   // Hive 3.1.3
   public HiveMetaStoreClient(Configuration conf) throws MetaException {
   ...
   }
   
   public HiveMetaStoreClient(Configuration conf, HiveMetaHookLoader hookLoader) throws MetaException {
   ...
   }
   
   public HiveMetaStoreClient(Configuration conf, HiveMetaHookLoader hookLoader, Boolean allowEmbedded) {
   ...
   }
   ```
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [seatunnel] github-actions[bot] closed issue #5174: [Bug] spark engine can not read hive source

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed issue #5174: [Bug]  spark engine can not  read hive source 
URL: https://github.com/apache/seatunnel/issues/5174


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [seatunnel] sh1shan commented on issue #5174: [Bug] spark engine can not read hive source

Posted by "sh1shan (via GitHub)" <gi...@apache.org>.
sh1shan commented on issue #5174:
URL: https://github.com/apache/seatunnel/issues/5174#issuecomment-1671066120

   Get the same error,How do I troubleshoot this?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [seatunnel] github-actions[bot] commented on issue #5174: [Bug] spark engine can not read hive source

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #5174:
URL: https://github.com/apache/seatunnel/issues/5174#issuecomment-1732717020

   This issue has been automatically marked as stale because it has not had recent activity for 30 days. It will be closed in next 7 days if no further activity occurs.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org