You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@seatunnel.apache.org by "Xuzhengz (via GitHub)" <gi...@apache.org> on 2024/07/26 02:01:01 UTC
[I] [Bug] [connector-file-s3] Lack of dependencies [seatunnel]
Xuzhengz opened a new issue, #7270:
URL: https://github.com/apache/seatunnel/issues/7270
### Search before asking
- [X] I had searched in the [issues](https://github.com/apache/seatunnel/issues?q=is%3Aissue+label%3A%22bug%22) and found no similar issues.
### What happened
After the dev branch finishes packaging, run the s3 file plugin with an error message
### SeaTunnel Version
dev
### SeaTunnel Config
```conf
{
"env": {
"job.name": "文本文件输出测试",
"job.mode": "batch"
},
"source": [
{
"plugin_name": "FakeSource",
"resourceType": "nosql",
"row.num": 100,
"schema": {
"table": "ottomi",
"fields": {
"id": "string",
"name": "string"
}
},
"resources": [
"ottomi"
],
"result_table_name": "ot_547a5338dd494638a8cfc2157f4fd06f",
"ottomi_node_type": "FakeSource"
}
],
"transform": [
],
"sink": [
{
"file_format_type": "text",
"field_delimiter": ",",
"custom_filename": true,
"file_name_expression": "text_test",
"is_enable_transaction": false,
"batch_size": 1000000000,
"plugin_name": "S3File",
"bucket": "s3a://ottomi",
"fs.s3a.endpoint": "******",
"fs.s3a.aws.credentials.provider": "org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider",
"access_key": "P7HpCVa0Eth8X3dM",
"secret_key": "xxM9cJL5QKqq9M7jvwkQ5LJgsQSVvIay",
"tmp_path": "/tmp/ottomi",
"path": "/file-node/download/1793520703906430978/text/",
"result_table_name": "ot_8e1b8b2321db40cc8b62667b0ac794f1",
"source_table_name": [
"ot_547a5338dd494638a8cfc2157f4fd06f"
]
}
]
}
```
### Running Command
```shell
bin/seatunnel.sh -c demo.json
```
### Error Exception
```log
Caused by: java.io.IOException: org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider instantiation exception: java.lang.NoSuchMethodError: com.google.common.base.Preconditions.checkArgument(ZLjava/lang/String;Ljava/lang/Object;Ljava/lang/Object;)V
at org.apache.hadoop.fs.s3a.S3AUtils.createAWSCredentialProvider(S3AUtils.java:678)
at org.apache.hadoop.fs.s3a.S3AUtils.createAWSCredentialProviderSet(S3AUtils.java:566)
at org.apache.hadoop.fs.s3a.DefaultS3ClientFactory.createS3Client(DefaultS3ClientFactory.java:52)
at org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:256)
at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3303)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:476)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:227)
at org.apache.seatunnel.connectors.seatunnel.file.hadoop.HadoopFileSystemProxy.initialize(HadoopFileSystemProxy.java:222)
at org.apache.seatunnel.connectors.seatunnel.file.hadoop.HadoopFileSystemProxy.getFileSystem(HadoopFileSystemProxy.java:171)
at org.apache.seatunnel.connectors.seatunnel.file.hadoop.HadoopFileSystemProxy.fileExist(HadoopFileSystemProxy.java:61)
at org.apache.seatunnel.connectors.seatunnel.file.s3.catalog.S3FileCatalog.tableExists(S3FileCatalog.java:90)
at org.apache.seatunnel.api.sink.DefaultSaveModeHandler.tableExists(DefaultSaveModeHandler.java:138)
at org.apache.seatunnel.api.sink.DefaultSaveModeHandler.createSchemaWhenNotExist(DefaultSaveModeHandler.java:107)
at org.apache.seatunnel.api.sink.DefaultSaveModeHandler.handleSchemaSaveMode(DefaultSaveModeHandler.java:69)
at org.apache.seatunnel.api.sink.SaveModeHandler.handleSaveMode(SaveModeHandler.java:38)
at org.apache.seatunnel.api.sink.SaveModeExecuteWrapper.execute(SaveModeExecuteWrapper.java:36)
at org.apache.seatunnel.engine.server.master.JobMaster.handleSaveMode(JobMaster.java:373)
... 20 more
Caused by: java.lang.NoSuchMethodError: com.google.common.base.Preconditions.checkArgument(ZLjava/lang/String;Ljava/lang/Object;Ljava/lang/Object;)V
at org.apache.hadoop.fs.s3a.S3AUtils.lookupPassword(S3AUtils.java:742)
at org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider.<init>(SimpleAWSCredentialsProvider.java:59)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.apache.hadoop.fs.s3a.S3AUtils.createAWSCredentialProvider(S3AUtils.java:635)
... 36 more
at org.apache.seatunnel.engine.server.CoordinatorService.lambda$submitJob$4(CoordinatorService.java:506)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
```
### Zeta or Flink or Spark Version
_No response_
### Java or Scala Version
1.8
### Screenshots


### Are you willing to submit PR?
- [X] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] [connector-file-s3] Lack of dependencies [seatunnel]
Posted by "Xuzhengz (via GitHub)" <gi...@apache.org>.
Xuzhengz commented on issue #7270:
URL: https://github.com/apache/seatunnel/issues/7270#issuecomment-2251890981
Due to a dependency conflict with the Hive driver, it has been resolved
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] [connector-file-s3] Lack of dependencies [seatunnel]
Posted by "Xuzhengz (via GitHub)" <gi...@apache.org>.
Xuzhengz closed issue #7270: [Bug] [connector-file-s3] Lack of dependencies
URL: https://github.com/apache/seatunnel/issues/7270
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org