You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "ranryl (via GitHub)" <gi...@apache.org> on 2023/11/20 04:14:05 UTC
[I] [Bug] 配置tencent cos resources，写入数据报SignatureDoesNotMatch，The Signature you specified is invalid [doris]
ranryl opened a new issue, #27266:
URL: https://github.com/apache/doris/issues/27266
### Search before asking
- [X] I had searched in the [issues](https://github.com/apache/doris/issues?q=is%3Aissue) and found no similar issues.
### Version
2.0.2
### What's Wrong?
CREATE RESOURCE "tc_cos"
PROPERTIES
(
"type" = "s3",
"s3.endpoint" = "https://cos.ap-beijing.myqcloud.com",
"s3.region" = "ap-beijing",
"s3.bucket" = "xxx",
"s3.root.path" = "/doris",
"s3.access_key" = "xxx",
"s3.secret_key" = "xxx",
"s3.connection.maximum" = "50",
"s3.connection.request.timeout" = "3000",
"s3.connection.timeout" = "1000",
"use_path_style" = "true"
);
be log
W1120 04:09:47.819643 1658 file_system.cpp:56] [IO_ERROR]failed to delete objects: (endpoint: http://cos.ap-beijing.myqcloud.com, bucket: xxxx, key:/doris/data/10215/020000000029d919594ea78c20b75277bc698e15e444038d_0.dat, SignatureDoesNotMatch), The Signature you specified is invalid.
### What You Expected?
能正常读写cos数据
### How to Reproduce?
_No response_
### Anything Else?
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [I] [Bug] 配置tencent cos resources，写入数据报SignatureDoesNotMatch，The Signature you specified is invalid [doris]
Posted by "swapkh91 (via GitHub)" <gi...@apache.org>.
swapkh91 commented on issue #27266:
URL: https://github.com/apache/doris/issues/27266#issuecomment-1820337538
+1
to reproduce, I think this is what happened
1. I created a resource for remote GCS bucket for cold-hot separation
2. Data was tiered to the remote bucket
3. Accidently deleted the folder in remote GCS bucket
```
CREATE RESOURCE "remote_gcs"
PROPERTIES
(
"type" = "s3",
"s3.endpoint" = "storage.googleapis.com",
"s3.region" = "asia-south1",
"s3.bucket" = "<gcs bucket name>",
"s3.root.path" = "doris_dump",
"s3.access_key" = "",
"s3.secret_key" = "",
"s3.connection.maximum" = "50",
"s3.connection.request.timeout" = "3000",
"s3.connection.timeout" = "1000"
);
```
I accidently deleted the `doris_dump` folder created at GCS bucket
Now getting the following error
```
[IO_ERROR]failed to delete objects: (endpoint: http://storage.googleapis.com, bucket: <gcs bucket name>, key:doris_dump/data/10120/02000000000000586b4e7843288f3e9b437318c552cc4dbc_0.dat, NotImplemented), Unable to parse ExceptionName: NotImplemented Message: A header or query you provided requested a function that is not implemented.
```
```
failed to delete remote rowset. err=[IO_ERROR]failed to delete objects: (endpoint: http://storage.googleapis.com, bucket: <gcs bucket name>, key:doris_dump/data/10120/02000000000000586b4e7843288f3e9b437318c552cc4dbc_0.dat, NotImplemented), Unable to parse ExceptionName: NotImplemented Message: A header or query you provided requested a function that is not implemented.
```
I ran the below commands to clear everything but still getting these errors in log
```
DROP table table1;
DROP STORAGE POLICY test_policy;
DROP RESOURCE 'remote_gcs';
select * from information_schema.rowsets where ROWSET_ID = "02000000000000556b4e7843288f3e9b437318c552cc4dbc";
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [I] [Bug] 配置tencent cos resources，写入数据报SignatureDoesNotMatch，The Signature you specified is invalid [doris]
Posted by "ranryl (via GitHub)" <gi...@apache.org>.
ranryl commented on issue #27266:
URL: https://github.com/apache/doris/issues/27266#issuecomment-1820381047
@swapkh91 I think you can try restart be node.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [I] [Bug] 配置tencent cos resources，写入数据报SignatureDoesNotMatch，The Signature you specified is invalid [doris]
Posted by "ranryl (via GitHub)" <gi...@apache.org>.
ranryl closed issue #27266: [Bug] 配置tencent cos resources，写入数据报SignatureDoesNotMatch，The Signature you specified is invalid
URL: https://github.com/apache/doris/issues/27266
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [I] [Bug] 配置tencent cos resources，写入数据报SignatureDoesNotMatch，The Signature you specified is invalid [doris]
Posted by "ranryl (via GitHub)" <gi...@apache.org>.
ranryl commented on issue #27266:
URL: https://github.com/apache/doris/issues/27266#issuecomment-1820295170
> @ranryl how to reproduce? what operation you did with this resource?
table storage_policy，cold hot separation
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [I] [Bug] 配置tencent cos resources，写入数据报SignatureDoesNotMatch，The Signature you specified is invalid [doris]
Posted by "Yulei-Yang (via GitHub)" <gi...@apache.org>.
Yulei-Yang commented on issue #27266:
URL: https://github.com/apache/doris/issues/27266#issuecomment-1820174355
I cannot reproduce this error when use this resource in Cold hot separation. And when creating resource, s3.endpoint only support non-secure protocal, in this case its' value should be "http://cos.ap-guangzhou.myqcloud.com"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [I] [Bug] 配置tencent cos resources，写入数据报SignatureDoesNotMatch，The Signature you specified is invalid [doris]
Posted by "swapkh91 (via GitHub)" <gi...@apache.org>.
swapkh91 commented on issue #27266:
URL: https://github.com/apache/doris/issues/27266#issuecomment-1820390633
@ranryl I think you meant **enable_storage_policy=true** in FE conf. I have added that and also restarted BE node
right now not seeing the log, lets see. I doesn't always come all the time
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [I] [Bug] 配置tencent cos resources，写入数据报SignatureDoesNotMatch，The Signature you specified is invalid [doris]
Posted by "ranryl (via GitHub)" <gi...@apache.org>.
ranryl commented on issue #27266:
URL: https://github.com/apache/doris/issues/27266#issuecomment-1820359785
> +1
>
> to reproduce, I think this is what happened
>
> 1. I created a resource for remote GCS bucket for cold-hot separation
> 2. Data was tiered to the remote bucket
> 3. Accidently deleted the folder in remote GCS bucket
>
> ```
> CREATE RESOURCE "remote_gcs"
> PROPERTIES
> (
> "type" = "s3",
> "s3.endpoint" = "storage.googleapis.com",
> "s3.region" = "asia-south1",
> "s3.bucket" = "<gcs bucket name>",
> "s3.root.path" = "doris_dump",
> "s3.access_key" = "",
> "s3.secret_key" = "",
> "s3.connection.maximum" = "50",
> "s3.connection.request.timeout" = "3000",
> "s3.connection.timeout" = "1000"
> );
> ```
>
> I accidently deleted the `doris_dump` folder created at GCS bucket Now getting the following error
>
> ```
> [IO_ERROR]failed to delete objects: (endpoint: http://storage.googleapis.com, bucket: <gcs bucket name>, key:doris_dump/data/10120/02000000000000586b4e7843288f3e9b437318c552cc4dbc_0.dat, NotImplemented), Unable to parse ExceptionName: NotImplemented Message: A header or query you provided requested a function that is not implemented.
> ```
>
> ```
> failed to delete remote rowset. err=[IO_ERROR]failed to delete objects: (endpoint: http://storage.googleapis.com, bucket: <gcs bucket name>, key:doris_dump/data/10120/02000000000000586b4e7843288f3e9b437318c552cc4dbc_0.dat, NotImplemented), Unable to parse ExceptionName: NotImplemented Message: A header or query you provided requested a function that is not implemented.
> ```
>
> I ran the below commands to clear everything but still getting these errors in log
>
> ```
> DROP table table1;
> DROP STORAGE POLICY test_policy;
> DROP RESOURCE 'remote_gcs';
>
> select * from information_schema.rowsets where ROWSET_ID = "02000000000000556b4e7843288f3e9b437318c552cc4dbc";
>
> <this command shows 0 rows>
> ```
I think try restart be
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [I] [Bug] 配置tencent cos resources，写入数据报SignatureDoesNotMatch，The Signature you specified is invalid [doris]
Posted by "ranryl (via GitHub)" <gi...@apache.org>.
ranryl commented on issue #27266:
URL: https://github.com/apache/doris/issues/27266#issuecomment-1820352893
I have resolved it，I added field **enable_storage_policy=true** to be.conf
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [I] [Bug] 配置tencent cos resources，写入数据报SignatureDoesNotMatch，The Signature you specified is invalid [doris]
Posted by "ranryl (via GitHub)" <gi...@apache.org>.
ranryl commented on issue #27266:
URL: https://github.com/apache/doris/issues/27266#issuecomment-1820295903
CREATE RESOURCE "remote_cos"
PROPERTIES
(
"type" = "s3",
"s3.endpoint" = "cos.ap-beijing.myqcloud.com",
"s3.region" = "ap-beijing",
"s3.bucket" = "xxx",
"s3.root.path" = "/doris",
"s3.access_key" = "xxx",
"s3.secret_key" = "xxx",
"s3.connection.maximum" = "50",
"s3.connection.request.timeout" = "3000",
"s3.connection.timeout" = "1000",
"use_path_style" = "true"
);
CREATE STORAGE POLICY log_remote_policy
PROPERTIES(
"storage_resource" = "remote_cos",
"cooldown_ttl" = "1h"
);
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [I] [Bug] 配置tencent cos resources，写入数据报SignatureDoesNotMatch，The Signature you specified is invalid [doris]
Posted by "Yulei-Yang (via GitHub)" <gi...@apache.org>.
Yulei-Yang commented on issue #27266:
URL: https://github.com/apache/doris/issues/27266#issuecomment-1820332242
> I cannot reproduce this error when use this resource in Cold hot separation. And when creating resource, s3.endpoint only support non-secure protocal, in this case its' value should be "http://cos.ap-guangzhou.myqcloud.com"
@ranryl check this post
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [I] [Bug] 配置tencent cos resources，写入数据报SignatureDoesNotMatch，The Signature you specified is invalid [doris]
Posted by "Yulei-Yang (via GitHub)" <gi...@apache.org>.
Yulei-Yang commented on issue #27266:
URL: https://github.com/apache/doris/issues/27266#issuecomment-1820165883
@ranryl how to reproduce? what operation you did with this resource?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org