You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@seatunnel.apache.org by GitBox <gi...@apache.org> on 2022/08/02 09:53:50 UTC
[GitHub] [incubator-seatunnel] weichao93 opened a new issue, #2340: hdfs to clickhouse 报错:java.lang.NoSuchMethodError: org.apache.spark.sql.UDFRegistration.register
weichao93 opened a new issue, #2340:
URL: https://github.com/apache/incubator-seatunnel/issues/2340
### Search before asking
- [X] I had searched in the [issues](https://github.com/apache/incubator-seatunnel/issues?q=is%3Aissue+label%3A%22bug%22) and found no similar issues.
### What happened
将hdfs的数据同步至clickhouse时报错，找不到自定义函数：java.lang.NoSuchMethodError: org.apache.spark.sql.UDFRegistration.register
### SeaTunnel Version
无
### SeaTunnel Config
```conf
无
```
### Running Command
```shell
../bin/start-waterdrop.sh --master yarn --deploy-mode client --config ../config/user_label_test.conf
```
### Error Exception
```log
22/08/02 17:48:54 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@1eb9bf60{/SQL/execution/json,null,AVAILABLE,@Spark}
22/08/02 17:48:54 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@4cc8d70d{/static/sql,null,AVAILABLE,@Spark}
Exception in thread "main" java.lang.NoSuchMethodError: org.apache.spark.sql.UDFRegistration.register(Ljava/lang/String;Lorg/apache/spark/sql/expressions/UserDefinedFunction;)Lorg/apache/spark/sql/expressions/UserDefinedFunction;
at io.github.interestinglab.waterdrop.filter.UdfRegister$$anonfun$findAndRegisterUdfs$1$$anonfun$apply$1.apply(UdfRegister.scala:26)
at io.github.interestinglab.waterdrop.filter.UdfRegister$$anonfun$findAndRegisterUdfs$1$$anonfun$apply$1.apply(UdfRegister.scala:24)
at scala.collection.immutable.List.foreach(List.scala:381)
at io.github.interestinglab.waterdrop.filter.UdfRegister$$anonfun$findAndRegisterUdfs$1.apply(UdfRegister.scala:24)
at io.github.interestinglab.waterdrop.filter.UdfRegister$$anonfun$findAndRegisterUdfs$1.apply(UdfRegister.scala:21)
at scala.collection.Iterator$class.foreach(Iterator.scala:893)
at scala.collection.AbstractIterator.foreach(Iterator.scala:1336)
at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
at io.github.interestinglab.waterdrop.filter.UdfRegister$.findAndRegisterUdfs(UdfRegister.scala:21)
at io.github.interestinglab.waterdrop.Waterdrop$.io$github$interestinglab$waterdrop$Waterdrop$$entrypoint(Waterdrop.scala:108)
at io.github.interestinglab.waterdrop.Waterdrop$$anonfun$1.apply$mcV$sp(Waterdrop.scala:38)
at io.github.interestinglab.waterdrop.Waterdrop$$anonfun$1.apply(Waterdrop.scala:38)
at io.github.interestinglab.waterdrop.Waterdrop$$anonfun$1.apply(Waterdrop.scala:38)
at scala.util.Try$.apply(Try.scala:192)
at io.github.interestinglab.waterdrop.Waterdrop$.main(Waterdrop.scala:38)
at io.github.interestinglab.waterdrop.Waterdrop.main(Waterdrop.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:744)
at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:187)
at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:212)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:126)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
```
### Flink or Spark Version
spark：spark-2.1.2-SNAPSHOT-bin-2.6.0-cdh5.5.0
### Java or Scala Version
java version "1.8.0_73"
### Screenshots
_No response_
### Are you willing to submit PR?
- [X] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] Hisoka-X commented on issue #2340: hdfs to clickhouse 报错:java.lang.NoSuchMethodError: org.apache.spark.sql.UDFRegistration.register
Posted by GitBox <gi...@apache.org>.
Hisoka-X commented on issue #2340:
URL: https://github.com/apache/incubator-seatunnel/issues/2340#issuecomment-1206168490
Which spark version you used? I think this maybe spark version not match caused.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] github-actions[bot] commented on issue #2340: hdfs to clickhouse 报错:java.lang.NoSuchMethodError: org.apache.spark.sql.UDFRegistration.register
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #2340:
URL: https://github.com/apache/incubator-seatunnel/issues/2340#issuecomment-1499785948
This issue has been closed because it has not received response for too long time. You could reopen it if you encountered similar problems in the future.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] weichao93 commented on issue #2340: hdfs to clickhouse 报错:java.lang.NoSuchMethodError: org.apache.spark.sql.UDFRegistration.register
Posted by GitBox <gi...@apache.org>.
weichao93 commented on issue #2340:
URL: https://github.com/apache/incubator-seatunnel/issues/2340#issuecomment-1207305933
spark version：spark-2.1.2-SNAPSHOT-bin-2.6.0-cdh5.5.0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] github-actions[bot] commented on issue #2340: hdfs to clickhouse 报错:java.lang.NoSuchMethodError: org.apache.spark.sql.UDFRegistration.register
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #2340:
URL: https://github.com/apache/incubator-seatunnel/issues/2340#issuecomment-1489515617
This issue has been automatically marked as stale because it has not had recent activity for 30 days. It will be closed in next 7 days if no further activity occurs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] github-actions[bot] closed issue #2340: hdfs to clickhouse 报错:java.lang.NoSuchMethodError: org.apache.spark.sql.UDFRegistration.register
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed issue #2340: hdfs to clickhouse 报错:java.lang.NoSuchMethodError: org.apache.spark.sql.UDFRegistration.register
URL: https://github.com/apache/incubator-seatunnel/issues/2340
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org