You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@seatunnel.apache.org by "itsallsame (via GitHub)" <gi...@apache.org> on 2023/04/21 10:02:23 UTC
[GitHub] [incubator-seatunnel] itsallsame opened a new issue, #4643: [Bug] [connetor-hive] when use spark2 transfer data from hive to console, get a error "java.lang.IllegalStateException: User did not initialize spark context! User did not initialize spark context!"
itsallsame opened a new issue, #4643:
URL: https://github.com/apache/incubator-seatunnel/issues/4643
### Search before asking
- [X] I had searched in the [issues](https://github.com/apache/incubator-seatunnel/issues?q=is%3Aissue+label%3A%22bug%22) and found no similar issues.
### What happened
when use spark2 transfer data from hive to doris, get a error "java.lang.IllegalStateException: User did not initialize spark context! User did not initialize spark context!"
btw, we use huawei cloud, and hive version is 3.1.0, data storage in Huawei Cloud Object Storage(OBS).
### SeaTunnel Version
2.3.1
### SeaTunnel Config
```conf
{
"env": {
"job.mode": "BATCH",
"spark.yarn.queue": "data",
"execution.parallelism": 1
},
"source": [
{
"plugin_name": "Hive",
"metastore_uri": "thrift://xxx:9083,thrift://xxx:9083",
"table_name": "tmp.test_table"
}
],
"sink": [
{
"plugin_name": "Console"
}
]
}
```
### Running Command
```shell
apache-seatunnel-incubating-2.3.1/bin/start-seatunnel-spark-2-connector-v2.sh \
--name test_seatunnle \
--check \
--deploy-mode cluster \
--master yarn \
--config config_test.json
```
### Error Exception
```log
2023-04-21 17:48:56,479 | INFO | [main] | Starting the user application in a separate Thread | org.apache.spark.deploy.yarn.ApplicationMaster.logInfo(Logging.scala:54)
2023-04-21 17:48:56,506 | INFO | [main] | Waiting for spark context initialization... | org.apache.spark.deploy.yarn.ApplicationMaster.logInfo(Logging.scala:54)
2023-04-21 17:48:56,774 | INFO | [Driver] | Final app status: FAILED, exitCode: 13 | org.apache.spark.deploy.yarn.ApplicationMaster.logInfo(Logging.scala:54)
2023-04-21 17:48:56,778 | ERROR | [main] | Uncaught exception: | org.apache.spark.deploy.yarn.ApplicationMaster.logError(Logging.scala:91)
java.lang.IllegalStateException: User did not initialize spark context!
at org.apache.spark.deploy.yarn.ApplicationMaster.runDriver(ApplicationMaster.scala:492)
at org.apache.spark.deploy.yarn.ApplicationMaster.org$apache$spark$deploy$yarn$ApplicationMaster$$runImpl(ApplicationMaster.scala:307)
at org.apache.spark.deploy.yarn.ApplicationMaster$$anonfun$run$1.apply$mcV$sp(ApplicationMaster.scala:247)
at org.apache.spark.deploy.yarn.ApplicationMaster$$anonfun$run$1.apply(ApplicationMaster.scala:247)
at org.apache.spark.deploy.yarn.ApplicationMaster$$anonfun$run$1.apply(ApplicationMaster.scala:247)
at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$3.run(ApplicationMaster.scala:802)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1761)
at org.apache.spark.deploy.yarn.ApplicationMaster.doAsUser(ApplicationMaster.scala:801)
at org.apache.spark.deploy.yarn.ApplicationMaster.run(ApplicationMaster.scala:246)
at org.apache.spark.deploy.yarn.ApplicationMaster$.main(ApplicationMaster.scala:828)
at org.apache.spark.deploy.yarn.ApplicationMaster.main(ApplicationMaster.scala)
```
### Flink or Spark Version
spark2.4.5
### Java or Scala Version
jdk1.8
### Screenshots
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [seatunnel] github-actions[bot] closed issue #4643: [Bug] [connetor-hive] when use spark2 transfer data from hive to console, get a error "java.lang.IllegalStateException: User did not initialize spark context! User did not initialize spark context!"
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed issue #4643: [Bug] [connetor-hive] when use spark2 transfer data from hive to console, get a error "java.lang.IllegalStateException: User did not initialize spark context! User did not initialize spark context!"
URL: https://github.com/apache/seatunnel/issues/4643
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [seatunnel] github-actions[bot] commented on issue #4643: [Bug] [connetor-hive] when use spark2 transfer data from hive to console, get a error "java.lang.IllegalStateException: User did not initialize spark context! User did not initialize spark context!"
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #4643:
URL: https://github.com/apache/seatunnel/issues/4643#issuecomment-1556346257
This issue has been automatically marked as stale because it has not had recent activity for 30 days. It will be closed in next 7 days if no further activity occurs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [seatunnel] github-actions[bot] commented on issue #4643: [Bug] [connetor-hive] when use spark2 transfer data from hive to console, get a error "java.lang.IllegalStateException: User did not initialize spark context! User did not initialize spark context!"
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #4643:
URL: https://github.com/apache/seatunnel/issues/4643#issuecomment-1574485827
This issue has been closed because it has not received response for too long time. You could reopen it if you encountered similar problems in the future.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org