You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@seatunnel.apache.org by GitBox <gi...@apache.org> on 2022/10/09 11:42:36 UTC
[GitHub] [incubator-seatunnel] woozhijun opened a new issue, #3042: When using cluster deploy mode for spark, Unable to get variable value
woozhijun opened a new issue, #3042:
URL: https://github.com/apache/incubator-seatunnel/issues/3042
### Search before asking
- [X] I had searched in the [issues](https://github.com/apache/incubator-seatunnel/issues?q=is%3Aissue+label%3A%22bug%22) and found no similar issues.
### What happened
When using cluster deploy mode for spark, Unable to get variable value.
However, looking at the yarn log is that the variable already exists。eg: -Dtarget_date,-Dstart,-Dend
> command:
LD_LIBRARY_PATH=\"/opt/Bigdata/FusionInsight_HD_8.1.0.1/install/FusionInsight-Hadoop-3.1.1/hadoop/lib/native:$LD_LIBRARY_PATH\" \
{{JAVA_HOME}}/bin/java \
-server \
-Xmx2048m \
'-Dtarget_date=2022-09-26' \
'-Dstart=1664121600' \
'-Dend=1664207999' \
-Djava.io.tmpdir={{PWD}}/tmp \
-DSPARK_APP_NAME=seatunnel_ck_http_connection_details \
-DSPARK_APP_ID=application_1661328709020_330725 \
'-Dspark.network.crypto.keyLength=256' \
'-Dspark.driver.port=22804' \
'-Dspark.authenticate=false' \
'-Dspark.rpc.askTimeout=120s' \
I am not sure if there is a problem ?
### SeaTunnel Version
2.1.3
### SeaTunnel Config
```conf
env {
spark.app.name="seatunnel_ck_http_connection_details"
spark.yarn.queue="default"
spark.executor.memory=2g
}
source {
Http {
url = "http://xxxx:9xxx/api/v1/query_range?query=ClickHouseMetrics_HTTPConnection%7Bproject%3D%22clickhouse%22%7D&start="${start}"&end="${end}"&step=60s"
method="GET"
result_table_name= "response_body"
}
}
transform {
sql{
sql = "select '"${target_date}"' as d, project, cluster_name, instance, metric_name, t.arr_vaules[1] as metric_value, cast(t.arr_vaules[0] as bigint) as timestamp from (select metrics.metric.project as project, metrics.metric.cluster as cluster_name, metrics.metric.instance as instance, metrics.metric.__name__ as metric_name, explode(metrics.values) as arr_vaules from (select explode(data.result) as metrics from response_body)) t"
}
}
sink {
console {
}
}
```
### Running Command
```shell
~/just/seatunnel/apache-seatunnel-incubating-2.1.3/bin/start-seatunnel-spark.sh \
--master yarn \
--deploy-mode cluster \
--config /home/data/just/seatunnel/sea_config/online/$config_file \
-i target_date=$target_date \
-i start=${start_ts} \
-i end=${end_ts}
```
### Error Exception
```log
Load plugin: PluginIdentifier{engineType='spark', pluginType='sink', pluginName='console'} from classpath | org.apache.seatunnel.plugin.discovery.AbstractPluginDiscovery.createPluginInstance(AbstractPluginDiscovery.java:101)
2022-10-08 11:42:56,799 | ERROR | [Driver] | Plugin[org.apache.seatunnel.spark.http.source.Http] contains invalid config, error: need to Config#resolve(), see the API docs for Config#resolve(); substitution not resolved: ConfigConcatenation("http://xxxx:9xxx/api/v1/query_range?query=ClickHouseMetrics_HTTPConnection%7Bproject%3D%22clickhouse%22%7D&start="${start}"&end="${end}"&step=60s")
| org.apache.seatunnel.core.base.command.BaseTaskExecuteCommand.pluginCheck(BaseTaskExecuteCommand.java:120)
2022-10-08 11:42:56,888 | INFO | [shutdown-hook-0] | Invoking stop() from shutdown hook | org.apache.spark.SparkContext.logInfo(Logging.scala:54)
```
### Flink or Spark Version
spark 2.4.5
### Java or Scala Version
1.8
### Screenshots
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] github-actions[bot] commented on issue #3042: When using cluster deploy mode for spark, Unable to get variable value
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on issue #3042:
URL: https://github.com/apache/incubator-seatunnel/issues/3042#issuecomment-1317869161
This issue has been closed because it has not received response for too long time. You could reopen it if you encountered similar problems in the future.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] github-actions[bot] commented on issue #3042: When using cluster deploy mode for spark, Unable to get variable value
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on issue #3042:
URL: https://github.com/apache/incubator-seatunnel/issues/3042#issuecomment-1308023866
This issue has been automatically marked as stale because it has not had recent activity for 30 days. It will be closed in next 7 days if no further activity occurs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] github-actions[bot] closed issue #3042: When using cluster deploy mode for spark, Unable to get variable value
Posted by GitBox <gi...@apache.org>.
github-actions[bot] closed issue #3042: When using cluster deploy mode for spark, Unable to get variable value
URL: https://github.com/apache/incubator-seatunnel/issues/3042
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org