You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@seatunnel.apache.org by GitBox <gi...@apache.org> on 2022/01/12 10:01:46 UTC

[GitHub] [incubator-seatunnel] xyhc opened a new issue #1034: start-seatunnel-structured-streaming.sh ERROR ERROR

xyhc opened a new issue #1034:
URL: https://github.com/apache/incubator-seatunnel/issues/1034


   seatunnel version: 1.5.7
   spark version: 2.4.8
   
   run shell: 
   ./bin/start-seatunnel-structured-streaming.sh --master local[4] --deploy-mode client --config ./config/realtimeaggr.conf
   
   happen:
   Exception in thread "main" java.lang.ArrayIndexOutOfBoundsException: 1
   	at io.github.interestinglab.waterdrop.config.ExposeSparkConf.main(ExposeSparkConf.java:43)
   [INFO] spark conf: [ERROR] config file does not exists or cannot be parsed due to invalid format
   SLF4J: Class path contains multiple SLF4J bindings.
   SLF4J: Found binding in [jar:file:/home/test/spark-2.4.8/jars/slf4j-log4j12-1.7.16.jar!/org/slf4j/impl/StaticLoggerBinder.class]
   SLF4J: Found binding in [jar:file:/usr/local/hadoop-client/hadoop-2.6.0-cdh5.8.3/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
   SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
   SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
   1    [main] WARN  org.apache.spark.deploy.DependencyUtils  - Local jar /home/test/seatunnel/seatunnel-1.5.7/[ERROR] does not exist, skipping.
   3    [main] WARN  org.apache.spark.deploy.SparkSubmit$$anon$2  - Failed to load io.github.interestinglab.waterdrop.WaterdropStructuredStreaming.
   java.lang.ClassNotFoundException: io.github.interestinglab.waterdrop.WaterdropStructuredStreaming
   	at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
   	at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
   	at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
   	at java.lang.Class.forName0(Native Method)
   	at java.lang.Class.forName(Class.java:348)
   	at org.apache.spark.util.Utils$.classForName(Utils.scala:238)
   	at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:816)
   	at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:161)
   	at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:184)
   	at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:86)
   	at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:930)
   	at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:939)
   	at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
   7    [Thread-1] INFO  org.apache.spark.util.ShutdownHookManager  - Shutdown hook called
   8    [Thread-1] INFO  org.apache.spark.util.ShutdownHookManager  - Deleting directory /tmp/spark-0f9e0a6d-650a-4bb0-86b9-ac433961fa60


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [incubator-seatunnel] RickyHuo commented on issue #1034: [Bug]start-seatunnel-structured-streaming.sh ERROR

Posted by GitBox <gi...@apache.org>.
RickyHuo commented on issue #1034:
URL: https://github.com/apache/incubator-seatunnel/issues/1034#issuecomment-1013866502


   @xyhc There is a bug in `seatunnel-structured-streaming.sh`. You can copy the bash from https://github.com/apache/incubator-seatunnel/blob/master/bin/start-seatunnel-structured-streaming.sh to fix it or wait for the next release.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [incubator-seatunnel] xyhc edited a comment on issue #1034: start-seatunnel-structured-streaming.sh ERROR ERROR

Posted by GitBox <gi...@apache.org>.
xyhc edited a comment on issue #1034:
URL: https://github.com/apache/incubator-seatunnel/issues/1034#issuecomment-1010871489


   seatunnel config(realtimeaggr.conf):
   
   
   spark {
     spark.app.name = "seatunnel"
     spark.executor.instances = 2
     spark.executor.cores = 1
     spark.executor.memory = "1g"
   }
   
   input {
   
     kafkaStream {
       consumer.bootstrap.servers = "127.0.0.1:9092"
       topics = "seatunnel_input"
       consumer.group.id = "seatunnel_input_group_id"
       result_table_name = "seatunnel_message"
     }
   }
   
   filter {
   
     sql {
         sql = "select string(value) as raw_message from seatunnel_message"
     }
   
     json {
       source_field = "raw_message"
       target_field = "jsonMap"
       schema_file = "demo.json"
       result_table_name = "flatten_msg"
     }
   
   }
   
   output {
     stdout {
       streaming_output_mode = "Append"
     }
   }


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [incubator-seatunnel] xyhc commented on issue #1034: start-seatunnel-structured-streaming.sh ERROR ERROR

Posted by GitBox <gi...@apache.org>.
xyhc commented on issue #1034:
URL: https://github.com/apache/incubator-seatunnel/issues/1034#issuecomment-1010871489


   seatunnel config:
   
   
   spark {
     # You can set spark configuration here
   
     # see available properties defined by spark: https://spark.apache.org/docs/latest/configuration.html#available-properties
     spark.app.name = "seatunnel"
     spark.executor.instances = 2
     spark.executor.cores = 1
     spark.executor.memory = "1g"
   }
   
   input {
   
     # This is a example input plugin **only for test and demonstrate the feature input plugin**
     kafkaStream {
       consumer.bootstrap.servers = "127.0.0.1:9092"
       topics = "seatunnel_input"
       consumer.group.id = "seatunnel_input_group_id"
       result_table_name = "seatunnel_message"
     }
   }
   
   filter {
   
     sql {
         sql = "select string(value) as raw_message from seatunnel_message"
     }
   
     json {
       source_field = "raw_message"
       target_field = "jsonMap"
       schema_file = "demo.json"
       result_table_name = "flatten_msg"
     }
   
     // sql {
     //   sql = "select name, count(name), date_format(window.end, 'MMM-dd HH:mm') as time from flatten_msg group by window('datetime', '1 minutes', '1 seconds'), name"
     // }
     // Watermark {
     //      delay_threshold = "5 minutes"
     //      time_field = "tf"
     //      time_type = "UNIX"
     //      watermark_field = "wm"
     // }
   
     // sql {
     //    table_name = "res"
     //    sql = "select name, count(name), date_format(dt, 'MM-dd HH:mm:00') as time from res group by dt, name"
     // }
     # If you would like to get more information about how to configure seatunnel and see full list of filter plugins,
     # please go to https://interestinglab.github.io/seatunnel-docs/#/zh-cn/v1/configuration/base
   }
   
   output {
     stdout {
       streaming_output_mode = "Append"
       # trigger_mode = "ProcessingTime"
       # interval = "5 seconds"
     }
   
   
     # If you would like to get more information about how to configure seatunnel and see full list of output plugins,
     # please go to https://interestinglab.github.io/seatunnel-docs/#/zh-cn/v1/configuration/base
   }


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [incubator-seatunnel] xyhc edited a comment on issue #1034: [Bug]start-seatunnel-structured-streaming.sh ERROR

Posted by GitBox <gi...@apache.org>.
xyhc edited a comment on issue #1034:
URL: https://github.com/apache/incubator-seatunnel/issues/1034#issuecomment-1010871489


   seatunnel config(realtimeaggr.conf):
   
   ```
   spark {
     spark.app.name = "seatunnel"
     spark.executor.instances = 2
     spark.executor.cores = 1
     spark.executor.memory = "1g"
   }
   
   input {
   
     kafkaStream {
       consumer.bootstrap.servers = "127.0.0.1:9092"
       topics = "seatunnel_input"
       consumer.group.id = "seatunnel_input_group_id"
       result_table_name = "seatunnel_message"
     }
   }
   
   filter {
   
     sql {
         sql = "select string(value) as raw_message from seatunnel_message"
     }
   
     json {
       source_field = "raw_message"
       target_field = "jsonMap"
       schema_file = "demo.json"
       result_table_name = "flatten_msg"
     }
   
   }
   
   output {
     stdout {
       streaming_output_mode = "Append"
     }
   }
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [incubator-seatunnel] RickyHuo closed issue #1034: [Bug]start-seatunnel-structured-streaming.sh ERROR

Posted by GitBox <gi...@apache.org>.
RickyHuo closed issue #1034:
URL: https://github.com/apache/incubator-seatunnel/issues/1034


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org