You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@seatunnel.apache.org by GitBox <gi...@apache.org> on 2022/06/30 05:48:08 UTC
[GitHub] [incubator-seatunnel] mjp27258 opened a new issue, #2092: Http Source
mjp27258 opened a new issue, #2092:
URL: https://github.com/apache/incubator-seatunnel/issues/2092
### Search before asking
- [X] I had searched in the [issues](https://github.com/apache/incubator-seatunnel/issues?q=is%3Aissue+label%3A%22bug%22) and found no similar issues.
### What happened
from Source Http to Kafka by spark
### SeaTunnel Version
2.1.2
### SeaTunnel Config
```conf
env {
spark.app.name = "SeaTunnel"
# spark.executor.instances = 2
spark.executor.cores = 1
spark.executor.memory = "1g"
# spark.streaming.batchDuration = 5
}
source {
Http {
url = "http://localhost:8088/send-kafka/check"
method = "get"
request_params = {"startDate":"${startDate}","endDate":"${endDate}"}
result_table_name= "message"
}
}
transform {
json {
source_field = "message"
}
}
sink {
kafka {
topic = "seatunnel-http2kafka"
producer.request.timeout.ms = 60000
semantic = "exactly_once"
producer.bootstrap.servers = "localhost:9092"
}
}
```
### Running Command
```shell
../bin/start-seatunnel-spark.sh --master local[4] --deploy-mode client --config http2kafka.conf --variable "startDate=2022-06-01/00:00:03" --variable "endDate=2022-06-01/00:00:03"
```
### Error Exception
```log
Exception in thread "main" java.lang.RuntimeException: Execute Spark task error
at org.apache.seatunnel.core.spark.command.SparkTaskExecuteCommand.execute(SparkTaskExecuteCommand.java:79)
at org.apache.seatunnel.core.base.Seatunnel.run(Seatunnel.java:39)
at org.apache.seatunnel.core.spark.SeatunnelSpark.main(SeatunnelSpark.java:32)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:851)
at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:167)
at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:195)
at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:86)
at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:926)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:935)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: org.apache.seatunnel.shade.com.typesafe.config.ConfigException$WrongType: http2kafka.conf: 13: request_params has type OBJECT rather than STRING
at org.apache.seatunnel.shade.com.typesafe.config.impl.SimpleConfig.findKeyOrNull(SimpleConfig.java:163)
at org.apache.seatunnel.shade.com.typesafe.config.impl.SimpleConfig.findOrNull(SimpleConfig.java:174)
at org.apache.seatunnel.shade.com.typesafe.config.impl.SimpleConfig.find(SimpleConfig.java:188)
at org.apache.seatunnel.shade.com.typesafe.config.impl.SimpleConfig.find(SimpleConfig.java:193)
at org.apache.seatunnel.shade.com.typesafe.config.impl.SimpleConfig.getString(SimpleConfig.java:250)
at org.apache.seatunnel.common.config.TypesafeConfigUtils.getConfig(TypesafeConfigUtils.java:106)
at org.apache.seatunnel.spark.http.source.Http.getData(Http.java:71)
at org.apache.seatunnel.spark.http.source.Http.getData(Http.java:48)
at org.apache.seatunnel.spark.SparkEnvironment.registerInputTempView(SparkEnvironment.java:141)
at org.apache.seatunnel.spark.batch.SparkBatchExecution.lambda$start$0(SparkBatchExecution.java:45)
at java.util.ArrayList.forEach(ArrayList.java:1257)
at org.apache.seatunnel.spark.batch.SparkBatchExecution.start(SparkBatchExecution.java:45)
at org.apache.seatunnel.core.spark.command.SparkTaskExecuteCommand.execute(SparkTaskExecuteCommand.java:76)
... 14 more
```
### Flink or Spark Version
spark-2.4
### Java or Scala Version
_No response_
### Screenshots
_No response_
### Are you willing to submit PR?
- [X] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] mjp27258 commented on issue #2092: Http Source
Posted by GitBox <gi...@apache.org>.
mjp27258 commented on issue #2092:
URL: https://github.com/apache/incubator-seatunnel/issues/2092#issuecomment-1173103478
3ks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] Hisoka-X commented on issue #2092: Http Source
Posted by GitBox <gi...@apache.org>.
Hisoka-X commented on issue #2092:
URL: https://github.com/apache/incubator-seatunnel/issues/2092#issuecomment-1172146726
`request_params` should be string. please use string with escapes to save json map.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] mjp27258 closed issue #2092: Http Source
Posted by GitBox <gi...@apache.org>.
mjp27258 closed issue #2092: Http Source
URL: https://github.com/apache/incubator-seatunnel/issues/2092
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org