You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "michael1991 (via GitHub)" <gi...@apache.org> on 2023/06/08 07:33:08 UTC
[GitHub] [doris] michael1991 opened a new issue, #20589: [Bug] S3 load fails when access GCS
michael1991 opened a new issue, #20589:
URL: https://github.com/apache/doris/issues/20589
### Search before asking
- [X] I had searched in the [issues](https://github.com/apache/doris/issues?q=is%3Aissue) and found no similar issues.
### Version
1.2.4.1
### What's Wrong?
When I use S3 load or S3 tvf to load parquet files stored in GCS, I get errors as below:
```
2023-06-08 07:14:51,734 ERROR (mysql-nio-pool-3539|101239) [S3Storage.list():372] errors while get file status
org.apache.doris.common.UserException: errCode = 2, detailMessage = Failed to get S3 FileSystem for bucket is null/empty
at org.apache.doris.backup.S3Storage.getFileSystem(S3Storage.java:142) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.backup.S3Storage.list(S3Storage.java:356) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.common.util.BrokerUtil.parseFile(BrokerUtil.java:90) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.tablefunction.ExternalFileTableValuedFunction.parseFile(ExternalFileTableValuedFunction.java:142) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.tablefunction.S3TableValuedFunction.<init>(S3TableValuedFunction.java:127) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.tablefunction.TableValuedFunctionIf.getTableFunction(TableValuedFunctionIf.java:49) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.analysis.TableValuedFunctionRef.<init>(TableValuedFunctionRef.java:40) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.analysis.CUP$SqlParser$actions.case817(SqlParser.java:28107) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.analysis.CUP$SqlParser$actions.CUP$SqlParser$do_action(SqlParser.java:8620) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.analysis.SqlParser.do_action(SqlParser.java:2291) ~[doris-fe.jar:1.2-SNAPSHOT]
at java_cup.runtime.lr_parser.parse(lr_parser.java:584) ~[jflex-1.4.3.jar:?]
at org.apache.doris.common.util.SqlParserUtils.getMultiStmts(SqlParserUtils.java:60) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.qe.ConnectProcessor.parse(ConnectProcessor.java:388) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.qe.ConnectProcessor.handleQuery(ConnectProcessor.java:285) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.qe.ConnectProcessor.dispatch(ConnectProcessor.java:473) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.qe.ConnectProcessor.processOnce(ConnectProcessor.java:700) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.mysql.ReadListener.lambda$handleEvent$0(ReadListener.java:52) ~[doris-fe.jar:1.2-SNAPSHOT]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_372]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_372]
at java.lang.Thread.run(Thread.java:750) ~[?:1.8.0_372]
Caused by: java.lang.IllegalArgumentException: bucket is null/empty
at org.apache.hadoop.thirdparty.com.google.common.base.Preconditions.checkArgument(Preconditions.java:144) ~[hadoop-shaded-guava-1.1.1.jar:1.1.1]
at org.apache.hadoop.fs.s3a.S3AUtils.propagateBucketOptions(S3AUtils.java:1162) ~[hadoop-aws-3.3.3.jar:?]
at org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:426) ~[hadoop-aws-3.3.3.jar:?]
at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3469) ~[hadoop-common-3.3.3.jar:?]
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:537) ~[hadoop-common-3.3.3.jar:?]
at org.apache.doris.backup.S3Storage.getFileSystem(S3Storage.java:140) ~[doris-fe.jar:1.2-SNAPSHOT]
... 19 more
```
Load SQL
```sql
LOAD LABEL label_gcs_test
(
DATA INFILE("s3://bucket/tables_prod/parquet_log_report/part*")
INTO TABLE load_test
FORMAT AS "parquet"
)
WITH S3
(
"AWS_ENDPOINT" = "https://storage.googleapis.com",
"AWS_ACCESS_KEY" = "ak",
"AWS_SECRET_KEY" = "sk",
"AWS_REGION" = "us"
)
PROPERTIES ( "timeout" = "3600" );
LOAD LABEL label_gcs_test
(
DATA INFILE("https://storage.googleapis.com/bucket/tables_prod/parquet_log_report/part*")
INTO TABLE load_test
FORMAT AS "parquet"
)
WITH S3
(
"AWS_ENDPOINT" = "https://storage.googleapis.com",
"AWS_ACCESS_KEY" = "ak",
"AWS_SECRET_KEY" = "sk",
"AWS_REGION" = "us",
"use_path_style" = "true"
)
PROPERTIES ( "timeout" = "3600" );
```
Whatever which method I used, I always got same error. Then I tried in code:
```java
URI uri = new URI("s3://bucket/tables_prod/parquet_log_report/part*");
```
Then bucket ( uri.getHost() ) is null.
### What You Expected?
S3 load can load data stored in GCS successfully.
### How to Reproduce?
_No response_
### Anything Else?
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] michael1991 commented on issue #20589: [Bug] S3 load fails when access GCS
Posted by "michael1991 (via GitHub)" <gi...@apache.org>.
michael1991 commented on issue #20589:
URL: https://github.com/apache/doris/issues/20589#issuecomment-1582112145
I've tested bucket name without underline, it works with "use_path_style" = "true" now, close it now.
Apologize here, thanks again.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] michael1991 closed issue #20589: [Bug] S3 load fails when access GCS
Posted by "michael1991 (via GitHub)" <gi...@apache.org>.
michael1991 closed issue #20589: [Bug] S3 load fails when access GCS
URL: https://github.com/apache/doris/issues/20589
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] michael1991 commented on issue #20589: [Bug] S3 load fails when access GCS
Posted by "michael1991 (via GitHub)" <gi...@apache.org>.
michael1991 commented on issue #20589:
URL: https://github.com/apache/doris/issues/20589#issuecomment-1582076284
seems coz of underline in bucket name, i will try bucket name without underline now
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org