You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@seatunnel.apache.org by GitBox <gi...@apache.org> on 2022/01/12 10:01:46 UTC
[GitHub] [incubator-seatunnel] xyhc opened a new issue #1034: start-seatunnel-structured-streaming.sh ERROR ERROR
xyhc opened a new issue #1034:
URL: https://github.com/apache/incubator-seatunnel/issues/1034
seatunnel version: 1.5.7
spark version: 2.4.8
run shell:
./bin/start-seatunnel-structured-streaming.sh --master local[4] --deploy-mode client --config ./config/realtimeaggr.conf
happen:
Exception in thread "main" java.lang.ArrayIndexOutOfBoundsException: 1
at io.github.interestinglab.waterdrop.config.ExposeSparkConf.main(ExposeSparkConf.java:43)
[INFO] spark conf: [ERROR] config file does not exists or cannot be parsed due to invalid format
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/home/test/spark-2.4.8/jars/slf4j-log4j12-1.7.16.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/local/hadoop-client/hadoop-2.6.0-cdh5.8.3/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
1 [main] WARN org.apache.spark.deploy.DependencyUtils - Local jar /home/test/seatunnel/seatunnel-1.5.7/[ERROR] does not exist, skipping.
3 [main] WARN org.apache.spark.deploy.SparkSubmit$$anon$2 - Failed to load io.github.interestinglab.waterdrop.WaterdropStructuredStreaming.
java.lang.ClassNotFoundException: io.github.interestinglab.waterdrop.WaterdropStructuredStreaming
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:348)
at org.apache.spark.util.Utils$.classForName(Utils.scala:238)
at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:816)
at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:161)
at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:184)
at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:86)
at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:930)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:939)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
7 [Thread-1] INFO org.apache.spark.util.ShutdownHookManager - Shutdown hook called
8 [Thread-1] INFO org.apache.spark.util.ShutdownHookManager - Deleting directory /tmp/spark-0f9e0a6d-650a-4bb0-86b9-ac433961fa60
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] RickyHuo commented on issue #1034: [Bug]start-seatunnel-structured-streaming.sh ERROR
Posted by GitBox <gi...@apache.org>.
RickyHuo commented on issue #1034:
URL: https://github.com/apache/incubator-seatunnel/issues/1034#issuecomment-1013866502
@xyhc There is a bug in `seatunnel-structured-streaming.sh`. You can copy the bash from https://github.com/apache/incubator-seatunnel/blob/master/bin/start-seatunnel-structured-streaming.sh to fix it or wait for the next release.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] xyhc edited a comment on issue #1034: start-seatunnel-structured-streaming.sh ERROR ERROR
Posted by GitBox <gi...@apache.org>.
xyhc edited a comment on issue #1034:
URL: https://github.com/apache/incubator-seatunnel/issues/1034#issuecomment-1010871489
seatunnel config(realtimeaggr.conf):
spark {
spark.app.name = "seatunnel"
spark.executor.instances = 2
spark.executor.cores = 1
spark.executor.memory = "1g"
}
input {
kafkaStream {
consumer.bootstrap.servers = "127.0.0.1:9092"
topics = "seatunnel_input"
consumer.group.id = "seatunnel_input_group_id"
result_table_name = "seatunnel_message"
}
}
filter {
sql {
sql = "select string(value) as raw_message from seatunnel_message"
}
json {
source_field = "raw_message"
target_field = "jsonMap"
schema_file = "demo.json"
result_table_name = "flatten_msg"
}
}
output {
stdout {
streaming_output_mode = "Append"
}
}
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] xyhc commented on issue #1034: start-seatunnel-structured-streaming.sh ERROR ERROR
Posted by GitBox <gi...@apache.org>.
xyhc commented on issue #1034:
URL: https://github.com/apache/incubator-seatunnel/issues/1034#issuecomment-1010871489
seatunnel config:
spark {
# You can set spark configuration here
# see available properties defined by spark: https://spark.apache.org/docs/latest/configuration.html#available-properties
spark.app.name = "seatunnel"
spark.executor.instances = 2
spark.executor.cores = 1
spark.executor.memory = "1g"
}
input {
# This is a example input plugin **only for test and demonstrate the feature input plugin**
kafkaStream {
consumer.bootstrap.servers = "127.0.0.1:9092"
topics = "seatunnel_input"
consumer.group.id = "seatunnel_input_group_id"
result_table_name = "seatunnel_message"
}
}
filter {
sql {
sql = "select string(value) as raw_message from seatunnel_message"
}
json {
source_field = "raw_message"
target_field = "jsonMap"
schema_file = "demo.json"
result_table_name = "flatten_msg"
}
// sql {
// sql = "select name, count(name), date_format(window.end, 'MMM-dd HH:mm') as time from flatten_msg group by window('datetime', '1 minutes', '1 seconds'), name"
// }
// Watermark {
// delay_threshold = "5 minutes"
// time_field = "tf"
// time_type = "UNIX"
// watermark_field = "wm"
// }
// sql {
// table_name = "res"
// sql = "select name, count(name), date_format(dt, 'MM-dd HH:mm:00') as time from res group by dt, name"
// }
# If you would like to get more information about how to configure seatunnel and see full list of filter plugins,
# please go to https://interestinglab.github.io/seatunnel-docs/#/zh-cn/v1/configuration/base
}
output {
stdout {
streaming_output_mode = "Append"
# trigger_mode = "ProcessingTime"
# interval = "5 seconds"
}
# If you would like to get more information about how to configure seatunnel and see full list of output plugins,
# please go to https://interestinglab.github.io/seatunnel-docs/#/zh-cn/v1/configuration/base
}
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] xyhc edited a comment on issue #1034: [Bug]start-seatunnel-structured-streaming.sh ERROR
Posted by GitBox <gi...@apache.org>.
xyhc edited a comment on issue #1034:
URL: https://github.com/apache/incubator-seatunnel/issues/1034#issuecomment-1010871489
seatunnel config(realtimeaggr.conf):
```
spark {
spark.app.name = "seatunnel"
spark.executor.instances = 2
spark.executor.cores = 1
spark.executor.memory = "1g"
}
input {
kafkaStream {
consumer.bootstrap.servers = "127.0.0.1:9092"
topics = "seatunnel_input"
consumer.group.id = "seatunnel_input_group_id"
result_table_name = "seatunnel_message"
}
}
filter {
sql {
sql = "select string(value) as raw_message from seatunnel_message"
}
json {
source_field = "raw_message"
target_field = "jsonMap"
schema_file = "demo.json"
result_table_name = "flatten_msg"
}
}
output {
stdout {
streaming_output_mode = "Append"
}
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] RickyHuo closed issue #1034: [Bug]start-seatunnel-structured-streaming.sh ERROR
Posted by GitBox <gi...@apache.org>.
RickyHuo closed issue #1034:
URL: https://github.com/apache/incubator-seatunnel/issues/1034
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org