You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@seatunnel.apache.org by GitBox <gi...@apache.org> on 2022/02/22 08:44:10 UTC
[GitHub] [incubator-seatunnel] yuangjiang opened a new issue #1311: IDE run project no working
yuangjiang opened a new issue #1311:
URL: https://github.com/apache/incubator-seatunnel/issues/1311
### Search before asking
- [X] I had searched in the [issues](https://github.com/apache/incubator-seatunnel/issues?q=is%3Aissue+label%3A%22bug%22) and found no similar issues.
### What happened
710 [main] INFO org.apache.spark.SecurityManager - Changing modify acls to: Admin,hdfs
2713 [main] INFO org.apache.spark.SecurityManager - Changing view acls groups to:
2715 [main] INFO org.apache.spark.SecurityManager - Changing modify acls groups to:
2717 [main] INFO org.apache.spark.SecurityManager - SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(Admin, hdfs); groups with view permissions: Set(); users with modify permissions: Set(Admin, hdfs); groups with modify permissions: Set()
Exception in thread "main" java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.intellij.rt.execution.CommandLineWrapper.main(CommandLineWrapper.java:64)
Caused by: java.lang.NoSuchMethodError: io.netty.buffer.PooledByteBufAllocator.defaultNumHeapArena()I
at org.apache.spark.network.util.NettyUtils.createPooledByteBufAllocator(NettyUtils.java:113)
at org.apache.spark.network.client.TransportClientFactory.<init>(TransportClientFactory.java:106)
at org.apache.spark.network.TransportContext.createClientFactory(TransportContext.java:99)
at org.apache.spark.rpc.netty.NettyRpcEnv.<init>(NettyRpcEnv.scala:71)
at org.apache.spark.rpc.netty.NettyRpcEnvFactory.create(NettyRpcEnv.scala:461)
at org.apache.spark.rpc.RpcEnv$.create(RpcEnv.scala:57)
at org.apache.spark.SparkEnv$.create(SparkEnv.scala:249)
at org.apache.spark.SparkEnv$.createDriverEnv(SparkEnv.scala:175)
at org.apache.spark.SparkContext.createSparkEnv(SparkContext.scala:257)
at org.apache.spark.SparkContext.<init>(SparkContext.scala:424)
at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2520)
at org.apache.spark.sql.SparkSession$Builder$$anonfun$7.apply(SparkSession.scala:935)
at org.apache.spark.sql.SparkSession$Builder$$anonfun$7.apply(SparkSession.scala:926)
at scala.Option.getOrElse(Option.scala:121)
at org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:926)
at org.apache.seatunnel.spark.SparkEnvironment.prepare(SparkEnvironment.java:59)
at org.apache.seatunnel.spark.SparkEnvironment.prepare(SparkEnvironment.java:31)
at org.apache.seatunnel.config.ConfigBuilder.createEnv(ConfigBuilder.java:196)
at org.apache.seatunnel.config.ConfigBuilder.<init>(ConfigBuilder.java:66)
at org.apache.seatunnel.Seatunnel.entryPoint(Seatunnel.java:94)
at org.apache.seatunnel.Seatunnel.run(Seatunnel.java:61)
at org.apache.seatunnel.SeatunnelSpark.main(SeatunnelSpark.java:29)
... 5 more
### SeaTunnel Version
dev
### SeaTunnel Config
```conf
#
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#
######
###### This config file is a demonstration of batch processing in seatunnel config
######
env {
# You can set spark configuration here
# see available properties defined by spark: https://spark.apache.org/docs/latest/configuration.html#available-properties
spark.master = "local[2]"
spark.app.name = "SeaTunnel"
spark.executor.instances = 2
spark.executor.cores = 1
spark.executor.memory = "1g"
spark.streaming.batchDuration = 5
}
source {
# This is a example input plugin **only for test and demonstrate the feature input plugin**
fakeStream {
content = ["Hello World, SeaTunnel"]
}
# You can also use other input plugins, such as file
# file {
# result_table_name = "accesslog"
# path = "hdfs://hadoop-cluster-01/nginx/accesslog"
# format = "json"
# }
# If you would like to get more information about how to configure seatunnel and see full list of input plugins,
# please go to https://seatunnel.apache.org/docs/spark/configuration/source-plugins/FakeStream
}
transform {
split {
fields = ["msg", "name"]
delimiter = ","
}
# you can also use other filter plugins, such as sql
# Sql {
# sql = "select * from accesslog where request_time > 1000"
# }
# If you would like to get more information about how to configure seatunnel and see full list of filter plugins,
# please go to https://seatunnel.apache.org/docs/spark/configuration/transform-plugins/Split
}
sink {
# choose stdout output plugin to output data to console
Console {}
# you can also use other output plugins, such as hdfs
# hdfs {
# path = "hdfs://hadoop-cluster-01/nginx/accesslog_processed"
# save_mode = "append"
# }
# If you would like to get more information about how to configure seatunnel and see full list of output plugins,
# please go to https://seatunnel.apache.org/docs/spark/configuration/sink-plugins/Console
}
```
### Running Command
```shell
--config D:\incubator-seatunnel\config\spark.streaming.conf --deploy-mode client --master local
```
### Error Exception
```log
Exception in thread "main" java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.intellij.rt.execution.CommandLineWrapper.main(CommandLineWrapper.java:64)
Caused by: java.lang.NoSuchMethodError: io.netty.buffer.PooledByteBufAllocator.defaultNumHeapArena()I
at org.apache.spark.network.util.NettyUtils.createPooledByteBufAllocator(NettyUtils.java:113)
at org.apache.spark.network.client.TransportClientFactory.<init>(TransportClientFactory.java:106)
at org.apache.spark.network.TransportContext.createClientFactory(TransportContext.java:99)
at org.apache.spark.rpc.netty.NettyRpcEnv.<init>(NettyRpcEnv.scala:71)
at org.apache.spark.rpc.netty.NettyRpcEnvFactory.create(NettyRpcEnv.scala:461)
```
### Flink or Spark Version
spark 2.4.0
### Java or Scala Version
jdk 1.8
### Screenshots
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] CalvinKirs commented on issue #1311: IDE run project no working
Posted by GitBox <gi...@apache.org>.
CalvinKirs commented on issue #1311:
URL: https://github.com/apache/incubator-seatunnel/issues/1311#issuecomment-1047633132
@felix-thinkingdata , Hi, can you take a look?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] yuangjiang commented on issue #1311: IDE run project no working
Posted by GitBox <gi...@apache.org>.
yuangjiang commented on issue #1311:
URL: https://github.com/apache/incubator-seatunnel/issues/1311#issuecomment-1056943626
IDE run project
0 [main] INFO org.apache.seatunnel.config.ConfigBuilder - Loading config file: D:\IdeaProjects\incubator-seatunnel\config\spark.streaming.conf
391 [main] INFO org.apache.seatunnel.config.ConfigBuilder - parsed config file: {
"env" : {
"spark.app.name" : "SeaTunnel",
"spark.executor.cores" : 1,
"spark.executor.instances" : 2,
"spark.executor.memory" : "1g",
"spark.master" : "local[2]",
"spark.streaming.batchDuration" : 5
},
"sink" : [
{
"plugin_name" : "Console"
}
],
"source" : [
{
"content" : [
"Hello World, SeaTunnel"
],
"plugin_name" : "fakeStream"
}
],
"transform" : [
{
"delimiter" : ",",
"fields" : [
"msg",
"name"
],
"plugin_name" : "split"
}
]
}
2587 [main] INFO org.apache.spark.SparkContext - Running Spark version 2.4.0
3549 [main] INFO org.apache.spark.SparkContext - Submitted application: SeaTunnel
3835 [main] INFO org.apache.spark.SecurityManager - Changing view acls to: Admin
3837 [main] INFO org.apache.spark.SecurityManager - Changing modify acls to: Admin
3839 [main] INFO org.apache.spark.SecurityManager - Changing view acls groups to:
3843 [main] INFO org.apache.spark.SecurityManager - Changing modify acls groups to:
3845 [main] INFO org.apache.spark.SecurityManager - SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(Admin); groups with view permissions: Set(); users with modify permissions: Set(Admin); groups with modify permissions: Set()
Exception in thread "main" java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.intellij.rt.execution.CommandLineWrapper.main(CommandLineWrapper.java:64)
Caused by: java.lang.NoSuchMethodError: io.netty.buffer.PooledByteBufAllocator.defaultNumHeapArena()I
at org.apache.spark.network.util.NettyUtils.createPooledByteBufAllocator(NettyUtils.java:113)
at org.apache.spark.network.client.TransportClientFactory.<init>(TransportClientFactory.java:106)
at org.apache.spark.network.TransportContext.createClientFactory(TransportContext.java:99)
at org.apache.spark.rpc.netty.NettyRpcEnv.<init>(NettyRpcEnv.scala:71)
at org.apache.spark.rpc.netty.NettyRpcEnvFactory.create(NettyRpcEnv.scala:461)
at org.apache.spark.rpc.RpcEnv$.create(RpcEnv.scala:57)
at org.apache.spark.SparkEnv$.create(SparkEnv.scala:249)
at org.apache.spark.SparkEnv$.createDriverEnv(SparkEnv.scala:175)
at org.apache.spark.SparkContext.createSparkEnv(SparkContext.scala:257)
at org.apache.spark.SparkContext.<init>(SparkContext.scala:424)
at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2520)
at org.apache.spark.sql.SparkSession$Builder$$anonfun$7.apply(SparkSession.scala:935)
at org.apache.spark.sql.SparkSession$Builder$$anonfun$7.apply(SparkSession.scala:926)
at scala.Option.getOrElse(Option.scala:121)
at org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:926)
at org.apache.seatunnel.spark.SparkEnvironment.prepare(SparkEnvironment.java:59)
at org.apache.seatunnel.spark.SparkEnvironment.prepare(SparkEnvironment.java:31)
at org.apache.seatunnel.config.ConfigBuilder.createEnv(ConfigBuilder.java:196)
at org.apache.seatunnel.config.ConfigBuilder.<init>(ConfigBuilder.java:66)
at org.apache.seatunnel.Seatunnel.entryPoint(Seatunnel.java:94)
at org.apache.seatunnel.Seatunnel.run(Seatunnel.java:61)
at org.apache.seatunnel.SeatunnelSpark.main(SeatunnelSpark.java:29)
... 5 more
------------------ 原始邮件 ------------------
发件人: "apache/incubator-seatunnel" ***@***.***>;
发送时间: 2022年3月2日(星期三) 晚上9:27
***@***.***>;
***@***.******@***.***>;
主题: Re: [apache/incubator-seatunnel] IDE run project no working (Issue #1311)
Can you share more detail? Including full shell command and which shell script you run ( shell from source code or release version). Let me reproduce this bug and fix it.
+1 I didn't have this problem in my local test, so I need more details from you.
—
Reply to this email directly, view it on GitHub, or unsubscribe.
Triage notifications on the go with GitHub Mobile for iOS or Android.
You are receiving this because you authored the thread.Message ID: ***@***.***>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] CalvinKirs commented on issue #1311: IDE run project no working
Posted by GitBox <gi...@apache.org>.
CalvinKirs commented on issue #1311:
URL: https://github.com/apache/incubator-seatunnel/issues/1311#issuecomment-1056931271
> Can you share more detail? Including full shell command and which shell script you run ( shell from source code or release version). Let me reproduce this bug and fix it.
+1 I didn't have this problem in my local test, so I need more details from you.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org