You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@seatunnel.apache.org by GitBox <gi...@apache.org> on 2022/04/25 12:40:13 UTC
[GitHub] [incubator-seatunnel] whb-bigdata opened a new issue, #1743: [Bug] [seatunnel-core-spark] spark on yarn cluster mode can not get --config/-c file
whb-bigdata opened a new issue, #1743:
URL: https://github.com/apache/incubator-seatunnel/issues/1743
### Search before asking
- [X] I had searched in the [issues](https://github.com/apache/incubator-seatunnel/issues?q=is%3Aissue+label%3A%22bug%22) and found no similar issues.
### What happened
when use spark-submit yarn cluster deploy-mode ，spark can not get config file on yarn
### SeaTunnel Version
2.1.1-prepare
### SeaTunnel Config
```conf
env {
spark.master = yarn
spark.app.name = "20220315"
spark.executor.instances = 1
spark.executor.cores = 1
spark.executor.memory = "4g"
spark.driver.memory = "4g"
spark.yarn.keytab = "xxxx.keytab"
spark.yarn.principal = "xxxxxx"
spark.yarn.queue = "xxxxxx"
}
source{
jdbc {
driver = "com.mysql.jdbc.Driver"
url = "xxxxxxxxxx"
table = "xxxxxxxxxxx"
result_table_name = "xxxxxxxxxxx"
user = "root"
password = "xxxxxxxxxxx"
}
}
transform{
sql {
sql = "select * from xxxxxx"
result_table_name = "xxxxx"
}
}
sink{
jdbc {
driver = "com.mysql.jdbc.Driver"
source_table_name = "xxxxx"
saveMode = "append"
url = "xxxxx"
user = "root"
password = "xxxxx"
dbTable = "xxxxx"
}
}
```
### Running Command
```shell
bin/start-seatunnel-spark.sh --master yarn --deploy-mode cluster -c config/application.conf
```
### Error Exception
```log
22/04/24 14:27:19 WARN conf.HiveConf: HiveConf of name hive.strict.checks.cartesian.product does not exist
22/04/24 14:27:20 INFO security.AMCredentialRenewer: Scheduling login from keytab in 18.0 h.
22/04/24 14:27:20 INFO yarn.ApplicationMaster: Preparing Local resources
22/04/24 14:27:20 INFO yarn.ApplicationMaster: ApplicationAttemptId: appattempt_1647333129725_263600_000001
22/04/24 14:27:20 INFO yarn.ApplicationMaster: Starting the user application in a separate Thread
22/04/24 14:27:20 INFO yarn.ApplicationMaster: Waiting for spark context initialization...
22/04/24 14:27:20 INFO config.ConfigBuilder: Loading config file: ./config/application.conf
22/04/24 14:27:21 INFO config.ConfigBuilder: parsed config file: {}
22/04/24 14:27:21 ERROR seatunnel.Seatunnel:
===============================================================================
22/04/24 14:27:21 ERROR seatunnel.Seatunnel: Fatal Error,
22/04/24 14:27:21 ERROR seatunnel.Seatunnel: Please submit bug report in https://github.com/apache/incubator-seatunnel/issues
22/04/24 14:27:21 ERROR seatunnel.Seatunnel: Reason:No configuration setting found for key 'env'
22/04/24 14:27:21 ERROR seatunnel.Seatunnel: Exception StackTrace:org.apache.seatunnel.shade.com.typesafe.config.ConfigException$Missing: No configuration setting found for key 'env'
at org.apache.seatunnel.shade.com.typesafe.config.impl.SimpleConfig.findKeyOrNull(SimpleConfig.java:156)
at org.apache.seatunnel.shade.com.typesafe.config.impl.SimpleConfig.findOrNull(SimpleConfig.java:174)
at org.apache.seatunnel.shade.com.typesafe.config.impl.SimpleConfig.find(SimpleConfig.java:188)
at org.apache.seatunnel.shade.com.typesafe.config.impl.SimpleConfig.find(SimpleConfig.java:193)
at org.apache.seatunnel.shade.com.typesafe.config.impl.SimpleConfig.getObject(SimpleConfig.java:268)
at org.apache.seatunnel.shade.com.typesafe.config.impl.SimpleConfig.getConfig(SimpleConfig.java:274)
at org.apache.seatunnel.shade.com.typesafe.config.impl.SimpleConfig.getConfig(SimpleConfig.java:41)
at org.apache.seatunnel.config.EnvironmentFactory.getEnvironment(EnvironmentFactory.java:47)
at org.apache.seatunnel.config.ExecutionContext.<init>(ExecutionContext.java:49)
at org.apache.seatunnel.command.spark.SparkTaskExecuteCommand.execute(SparkTaskExecuteCommand.java:44)
at org.apache.seatunnel.command.spark.SparkTaskExecuteCommand.execute(SparkTaskExecuteCommand.java:36)
at org.apache.seatunnel.Seatunnel.run(Seatunnel.java:48)
at org.apache.seatunnel.SeatunnelSpark.main(SeatunnelSpark.java:27)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:685)
```
### Flink or Spark Version
_No response_
### Java or Scala Version
_No response_
### Screenshots
it can not get file so ERROR seatunnel.Seatunnel: Reason:No configuration setting found for key 'env'
### Are you willing to submit PR?
- [X] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] yx91490 commented on issue #1743: [Bug] [seatunnel-core-spark] spark on yarn cluster mode can not get --config/-c file
Posted by GitBox <gi...@apache.org>.
yx91490 commented on issue #1743:
URL: https://github.com/apache/incubator-seatunnel/issues/1743#issuecomment-1109201773
can you rerun command with `bash -x ./bin/start-seatunnel-spark.sh...` and paste log here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] quanzhian commented on issue #1743: [Bug] [seatunnel-core-spark] spark on yarn cluster mode can not get --config/-c file
Posted by GitBox <gi...@apache.org>.
quanzhian commented on issue #1743:
URL: https://github.com/apache/incubator-seatunnel/issues/1743#issuecomment-1110455358
Just add check to the command to solve the problem
@whb-bigdata
Running Command
`bin/start-seatunnel-spark.sh --master yarn --deploy-mode cluster --check -c config/application.conf`
Here are the details of my test
`[xxxxxx@bigdata-app03 apache-seatunnel-incubating-2.1.1-SNAPSHOT]# ./bin/start-seatunnel-spark.sh --master yarn --deploy-mode cluster --check --config /mnt/services/seatunnel/spark_batch.conf
22/04/27 09:49:00 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
22/04/27 09:49:01 WARN DomainSocketFactory: The short-circuit local reads feature cannot be used because libhadoop cannot be loaded.
22/04/27 09:49:01 INFO EsServiceCredentialProvider: Loaded EsServiceCredentialProvider
22/04/27 09:49:01 INFO Client: Requesting a new application from cluster with 5 NodeManagers
22/04/27 09:49:01 INFO Configuration: found resource resource-types.xml at file:/etc/hadoop/3.1.4.0-315/0/resource-types.xml
22/04/27 09:49:01 INFO Client: Verifying our application has not requested more than the maximum memory capability of the cluster (6144 MB per container)
22/04/27 09:49:01 INFO Client: Will allocate AM container, with 1408 MB memory including 384 MB overhead
22/04/27 09:49:01 INFO Client: Setting up container launch context for our AM
22/04/27 09:49:01 INFO Client: Setting up the launch environment for our AM container
22/04/27 09:49:01 INFO Client: Preparing resources for our AM container
22/04/27 09:49:02 INFO EsServiceCredentialProvider: Hadoop Security Enabled = [false]
22/04/27 09:49:02 INFO EsServiceCredentialProvider: ES Auth Method = [SIMPLE]
22/04/27 09:49:02 INFO EsServiceCredentialProvider: Are creds required = [false]
22/04/27 09:49:02 INFO Client: Source and destination file systems are the same. Not copying hdfs:/hdp/apps/3.1.4.0-315/spark2/spark2-hdp-yarn-archive.tar.gz
22/04/27 09:49:02 INFO Client: Uploading resource file:/mnt/services/seatunnel/apache-seatunnel-incubating-2.1.1-SNAPSHOT/lib/seatunnel-core-spark.jar -> hdfs://nameservice1/user/xxx_user/.sparkStaging/application_1643094720025_42346/seatunnel-core-spark.jar
22/04/27 09:49:03 INFO Client: Uploading resource file:/mnt/services/seatunnel/apache-seatunnel-incubating-2.1.1-SNAPSHOT/plugins.tar.gz -> hdfs://nameservice1/user/xxx_user/.sparkStaging/application_1643094720025_42346/plugins.tar.gz
22/04/27 09:49:03 INFO Client: Uploading resource file:/mnt/services/seatunnel/spark_batch.conf -> hdfs://nameservice1/user/xxx_user/.sparkStaging/application_1643094720025_42346/spark_batch.conf
22/04/27 09:49:03 INFO Client: Uploading resource file:/tmp/spark-9bbe4cf2-c001-4c0c-a918-7a552a6e5b5b/__spark_conf__1828107881235032523.zip -> hdfs://nameservice1/user/xxx_user/.sparkStaging/application_1643094720025_42346/__spark_conf__.zip
22/04/27 09:49:03 INFO SecurityManager: Changing view acls to: xxxxxx,xxx_user
22/04/27 09:49:03 INFO SecurityManager: Changing modify acls to: xxxxxx,xxx_user
22/04/27 09:49:03 INFO SecurityManager: Changing view acls groups to:
22/04/27 09:49:03 INFO SecurityManager: Changing modify acls groups to:
22/04/27 09:49:03 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(xxxxxx, xxx_user); groups with view permissions: Set(); users with modify permissions: Set(xxxxxx, xxx_user); groups with modify permissions: Set()
22/04/27 09:49:03 INFO Client: Submitting application application_1643094720025_42346 to ResourceManager
22/04/27 09:49:03 INFO YarnClientImpl: Submitted application application_1643094720025_42346
22/04/27 09:49:04 INFO Client: Application report for application_1643094720025_42346 (state: ACCEPTED)
22/04/27 09:49:04 INFO Client:
client token: N/A
diagnostics: AM container is launched, waiting for AM container to Register with RM
ApplicationMaster host: N/A
ApplicationMaster RPC port: -1
queue: default
start time: 1651024143796
final status: UNDEFINED
tracking URL: http://bigdata-master01:8088/proxy/application_1643094720025_42346/
user: xxx_user
22/04/27 09:49:05 INFO Client: Application report for application_1643094720025_42346 (state: ACCEPTED)
22/04/27 09:49:06 INFO Client: Application report for application_1643094720025_42346 (state: ACCEPTED)
22/04/27 09:49:07 INFO Client: Application report for application_1643094720025_42346 (state: RUNNING)
22/04/27 09:49:07 INFO Client:
client token: N/A
diagnostics: N/A
ApplicationMaster host: 172.18.247.18
ApplicationMaster RPC port: 0
queue: default
start time: 1651024143796
final status: UNDEFINED
tracking URL: http://bigdata-master01:8088/proxy/application_1643094720025_42346/
user: xxx_user
22/04/27 09:49:08 INFO Client: Application report for application_1643094720025_42346 (state: RUNNING)
22/04/27 09:49:09 INFO Client: Application report for application_1643094720025_42346 (state: RUNNING)
22/04/27 09:49:10 INFO Client: Application report for application_1643094720025_42346 (state: RUNNING)
22/04/27 09:49:11 INFO Client: Application report for application_1643094720025_42346 (state: FINISHED)
22/04/27 09:49:11 INFO Client:
client token: N/A
diagnostics: N/A
ApplicationMaster host: 172.18.247.18
ApplicationMaster RPC port: 0
queue: default
start time: 1651024143796
final status: SUCCEEDED
tracking URL: http://bigdata-master01:8088/proxy/application_1643094720025_42346/
user: xxx_user
22/04/27 09:49:11 INFO ShutdownHookManager: Shutdown hook called
22/04/27 09:49:11 INFO ShutdownHookManager: Deleting directory /tmp/spark-11894826-40fd-484e-aac6-ec360c1f02c2
22/04/27 09:49:11 INFO ShutdownHookManager: Deleting directory /tmp/spark-9bbe4cf2-c001-4c0c-a918-7a552a6e5b5b
`
Code bug fixes
org.apache.seatunnel.command.spark.SparkTaskExecuteCommand
add Code
`
if (DeployMode.CLUSTER.equals(sparkCommandArgs.getDeployMode())) {
confFile = Paths.get(sparkCommandArgs.getConfigFile()).getFileName().toString();
}
`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] ruanwenjun closed issue #1743: [Bug] [seatunnel-core-spark] spark on yarn cluster mode can not get --config/-c file
Posted by GitBox <gi...@apache.org>.
ruanwenjun closed issue #1743: [Bug] [seatunnel-core-spark] spark on yarn cluster mode can not get --config/-c file
URL: https://github.com/apache/incubator-seatunnel/issues/1743
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] whb-bigdata commented on issue #1743: [Bug] [seatunnel-core-spark] spark on yarn cluster mode can not get --config/-c file
Posted by GitBox <gi...@apache.org>.
whb-bigdata commented on issue #1743:
URL: https://github.com/apache/incubator-seatunnel/issues/1743#issuecomment-1111021051
@quanzhian Thank you for your reply. If adding -- check is only to check the creation environment, not to execute the positive circulation, so it can succeed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] whb-bigdata commented on issue #1743: [Bug] [seatunnel-core-spark] spark on yarn cluster mode can not get --config/-c file
Posted by GitBox <gi...@apache.org>.
whb-bigdata commented on issue #1743:
URL: https://github.com/apache/incubator-seatunnel/issues/1743#issuecomment-1109234988
+ CMD='${SPARK_HOME}/bin/spark-submit --class "org.apache.seatunnel.SeatunnelSpark" --name "xxxxx" --master "yarn" --deploy-mode "cluster" --files "plugins.tar.gz,config/xxxxxx" --conf "spark.executor.memory=4g" --conf "spark.driver.memory=4g" --conf "spark.master=yarn" --conf "spark.yarn.keytab=xxxxx" --conf "spark.app.name=20220315" --conf "spark.executor.cores=1" --conf "spark.executor.instances=1" --conf "spark.yarn.principal=xxxxx" --conf "spark.yarn.queue=xxxxx" lib/seatunnel-core-spark.jar --master yarn --deploy-mode cluster -c config/xxxxxx'
this is the submit command，when run on yarn ,-c config/xxxxxx file Will not be found
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org