You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by GitBox <gi...@apache.org> on 2021/09/06 09:36:17 UTC
[GitHub] [incubator-doris] 345362231 opened a new issue #6574: [Bug] doris 0.14 backup/restore问题
345362231 opened a new issue #6574:
URL: https://github.com/apache/incubator-doris/issues/6574
### Search before asking
- [X] I had searched in the [issues](https://github.com/apache/incubator-doris/issues?q=is%3Aissue) and found no similar issues.
### Version
0.14.0-release-4c1e52261
### What's Wrong?
使用minio作为s3的存储源，按照官方的备份恢复步骤进行操作。
第一步，创建s3的仓库，仓库名为s3_repo，创建成功。
第二步，创建备份，通过命令BACKUP SNAPSHOT kimi112.b1 TO s3_repo on (hash_k1_replace5_bak)；创建备份成成功。
这里有个问题就是生成了2个__palo_repository_s3_repo文件件并在不同的目录中，这里有点怪。
第三步，查看备份信息，show snapshot on s3_repo可查看到
第四步，恢复快照，通过命令restore SNAPSHOT kimi112.b1 from s3_repo on (`hash_k1_replace5_bak`) PROPERTIES ("backup_timestamp"="2021-09-06-17-18-32" );执行命令成功。
第五步，查看恢复状态，进入到目标库，show restore;
这里downloading卡住，这里应该算是报错：get nothing from remote path: s3://velero/velero/__palo_repository_s3_repo/__ss_b1/__ss_content/__db_664542/__tbl_941686/__part_941685/__idx_941687/__941688
无法表恢复。
这里应该是备份路径不统一，应该是个bug。
在进行恢复的时候，不能获取到远程路径的数据，也应该是个bug。
### What You Expected?
按照help cmd中的操作帮助，我不认为我操作有问题，那怎么解决这个备份还原问题，希望能得到帮助。谢谢
### How to Reproduce?
_No response_
### Anything Else?
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] yangzhg commented on issue #6574: [Bug] doris 0.14 backup/restore问题
Posted by GitBox <gi...@apache.org>.
yangzhg commented on issue #6574:
URL: https://github.com/apache/incubator-doris/issues/6574#issuecomment-914890958
> 使用他这个方法，build新的palo_be，替换后，测试备份还原是可以使用的
你可以通过 export MINIO_DOMAIN=mydomain.com 使minio 支持VirtualHost style， path style 因为安全问题 已经被大多数云储存厂商逐渐废弃了
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] 345362231 commented on issue #6574: [Bug] doris 0.14 backup/restore问题
Posted by GitBox <gi...@apache.org>.
345362231 commented on issue #6574:
URL: https://github.com/apache/incubator-doris/issues/6574#issuecomment-913508818
目前只用了s3，没有其他存储方式
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] ChPi commented on issue #6574: [Bug] doris 0.14 backup/restore问题
Posted by GitBox <gi...@apache.org>.
ChPi commented on issue #6574:
URL: https://github.com/apache/incubator-doris/issues/6574#issuecomment-958599678
resolved #6962
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] ChPi edited a comment on issue #6574: [Bug] doris 0.14 backup/restore问题
Posted by GitBox <gi...@apache.org>.
ChPi edited a comment on issue #6574:
URL: https://github.com/apache/incubator-doris/issues/6574#issuecomment-914007214
这个问题是因为client参数链接你的s3存储有问题。be代码s3_util.cpp改为下面就可以了，把useVirtualAdressing改为false。
new Aws::S3::S3Client(aws_cred, aws_config, Aws::Client::AWSAuthV4Signer::PayloadSigningPolicy::Never, false)
be配置可以加一个useVirtualAdressing参数控制这个
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] morningman commented on issue #6574: [Bug] doris 0.14 backup/restore问题
Posted by GitBox <gi...@apache.org>.
morningman commented on issue #6574:
URL: https://github.com/apache/incubator-doris/issues/6574#issuecomment-913958894
你重新创建一个 repo，使用这个路径：s3://velero/velero/ 最后加个 / ，先试试，不是很确定
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] ChPi edited a comment on issue #6574: [Bug] doris 0.14 backup/restore问题
Posted by GitBox <gi...@apache.org>.
ChPi edited a comment on issue #6574:
URL: https://github.com/apache/incubator-doris/issues/6574#issuecomment-914007214
这个问题是因为client参数链接你的s3存储有问题, s3路径有两种，默认的解析可能有问题。
be代码s3_util.cpp改为下面另外一种，把useVirtualAdressing改为false。
new Aws::S3::S3Client(aws_cred, aws_config, Aws::Client::AWSAuthV4Signer::PayloadSigningPolicy::Never, false)
be配置可以加一个useVirtualAdressing参数控制这个
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] morningman commented on issue #6574: [Bug] doris 0.14 backup/restore问题
Posted by GitBox <gi...@apache.org>.
morningman commented on issue #6574:
URL: https://github.com/apache/incubator-doris/issues/6574#issuecomment-913969209
> 这个尝试过，没有任何变化
这个只能等我们有时间找个minio的环境试试了，不同系统似乎对s3协议中路径的解析规则不太一样
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] morningman commented on issue #6574: [Bug] doris 0.14 backup/restore问题
Posted by GitBox <gi...@apache.org>.
morningman commented on issue #6574:
URL: https://github.com/apache/incubator-doris/issues/6574#issuecomment-913553781
感觉可能是minio 对s3的兼容语义和其他对象存储不太一样。。
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] morningman commented on issue #6574: [Bug] doris 0.14 backup/restore问题
Posted by GitBox <gi...@apache.org>.
morningman commented on issue #6574:
URL: https://github.com/apache/incubator-doris/issues/6574#issuecomment-913936150
> 就是说这个问题 没有其他解决方法？只能用云厂商的s3存储？
我们没有测试过minio，先确认下：s3://velero/velero/__palo_repository_s3_repo/__ss_b1/__ss_content/__db_664542/__tbl_941686/__part_941685/__idx_941687/__941688 这路径下有文件么？如果没有的话，实际文件是到哪个路径下了？
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] ChPi commented on issue #6574: [Bug] doris 0.14 backup/restore问题
Posted by GitBox <gi...@apache.org>.
ChPi commented on issue #6574:
URL: https://github.com/apache/incubator-doris/issues/6574#issuecomment-958599678
resolved #6962
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] ChPi commented on issue #6574: [Bug] doris 0.14 backup/restore问题
Posted by GitBox <gi...@apache.org>.
ChPi commented on issue #6574:
URL: https://github.com/apache/incubator-doris/issues/6574#issuecomment-914007214
这个问题是因为client参数不兼容。be代码s3_util.cpp改为下面就可以了，把useVirtualAdressing改为false。
new Aws::S3::S3Client(aws_cred, aws_config, Aws::Client::AWSAuthV4Signer::PayloadSigningPolicy::Never, false)
be配置可以加一个useVirtualAdressing参数控制这个
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] yangzhg commented on issue #6574: [Bug] doris 0.14 backup/restore问题
Posted by GitBox <gi...@apache.org>.
yangzhg commented on issue #6574:
URL: https://github.com/apache/incubator-doris/issues/6574#issuecomment-914889454
> 这个问题是因为client参数链接你的s3存储有问题, s3路径有两种，默认的解析可能有问题。
> be代码s3_util.cpp改为下面另外一种，把useVirtualAdressing改为false。
> new Aws::S3::S3Client(aws_cred, aws_config, Aws::Client::AWSAuthV4Signer::PayloadSigningPolicy::Never, false)
>
> be配置可以加一个useVirtualAdressing参数控制这个
增加参数是不行的，一个be 接收的请求是多个，可能有的支持useVirtualAdressing 有的不支持，目前绝大部分支持s3 协议的对象存储都支持VirtualHost style， path style 支持的越来越少了，所以现在统一成了 VirtualHost style
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] yangzhg edited a comment on issue #6574: [Bug] doris 0.14 backup/restore问题
Posted by GitBox <gi...@apache.org>.
yangzhg edited a comment on issue #6574:
URL: https://github.com/apache/incubator-doris/issues/6574#issuecomment-914890958
> 使用他这个方法，build新的palo_be，替换后，测试备份还原是可以使用的
他的方式 在只使用一种对象存储时可用，但是大多数时候是不合适的
你可以通过 export MINIO_DOMAIN=mydomain.com 使minio 支持VirtualHost style， path style 因为安全问题 已经被大多数云储存厂商逐渐废弃了
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] 345362231 commented on issue #6574: [Bug] doris 0.14 backup/restore问题
Posted by GitBox <gi...@apache.org>.
345362231 commented on issue #6574:
URL: https://github.com/apache/incubator-doris/issues/6574#issuecomment-913505053
CREATE REPOSITORY `s3_repo` WITH S3 ON LOCATION "s3://velero/velero" PROPERTIES ("AWS_ENDPOINT" = "http://test-minio.XXX.XX:30542","AWS_ACCESS_KEY"="XXXX","AWS_SE[DCRET_KEY"="XXXX","AWS_REGION"="velero");
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] 345362231 commented on issue #6574: [Bug] doris 0.14 backup/restore问题
Posted by GitBox <gi...@apache.org>.
345362231 commented on issue #6574:
URL: https://github.com/apache/incubator-doris/issues/6574#issuecomment-913937761
s3://velero/velero/__palo_repository_s3_repo/__ss_b1/__ss_content/__db_664542/__tbl_941686/__part_941685/__idx_941687/__941688 这个路径下是没有的
实际路径是s3://velero/__palo_repository_s3_repo/__ss_b1/__ss_content/__db_664542/__tbl_941686/__part_941685/__idx_941687/__941688
前面少了个velero
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] 345362231 commented on issue #6574: [Bug] doris 0.14 backup/restore问题
Posted by GitBox <gi...@apache.org>.
345362231 commented on issue #6574:
URL: https://github.com/apache/incubator-doris/issues/6574#issuecomment-914890459
> 这个问题是因为client参数链接你的s3存储有问题, s3路径有两种，默认的解析可能有问题。
> be代码s3_util.cpp改为下面另外一种，把useVirtualAdressing改为false。
> new Aws::S3::S3Client(aws_cred, aws_config, Aws::Client::AWSAuthV4Signer::PayloadSigningPolicy::Never, false)
>
> be配置可以加一个useVirtualAdressing参数控制这个
使用他这个方法，build新的palo_be，替换后，测试备份还原是可以使用的
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] 345362231 commented on issue #6574: [Bug] doris 0.14 backup/restore问题
Posted by GitBox <gi...@apache.org>.
345362231 commented on issue #6574:
URL: https://github.com/apache/incubator-doris/issues/6574#issuecomment-913920972
就是说这个问题 没有其他解决方法？只能用云厂商的s3存储？
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] 345362231 commented on issue #6574: [Bug] doris 0.14 backup/restore问题
Posted by GitBox <gi...@apache.org>.
345362231 commented on issue #6574:
URL: https://github.com/apache/incubator-doris/issues/6574#issuecomment-913946870
也有这个目录
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] 345362231 commented on issue #6574: [Bug] doris 0.14 backup/restore问题
Posted by GitBox <gi...@apache.org>.
345362231 commented on issue #6574:
URL: https://github.com/apache/incubator-doris/issues/6574#issuecomment-913959094
这个尝试过，没有任何变化
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] morningman commented on issue #6574: [Bug] doris 0.14 backup/restore问题
Posted by GitBox <gi...@apache.org>.
morningman commented on issue #6574:
URL: https://github.com/apache/incubator-doris/issues/6574#issuecomment-913942084
s3://velero/velero/__palo_repository_s3_repo/ 这个目录有么？
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] yangzhg edited a comment on issue #6574: [Bug] doris 0.14 backup/restore问题
Posted by GitBox <gi...@apache.org>.
yangzhg edited a comment on issue #6574:
URL: https://github.com/apache/incubator-doris/issues/6574#issuecomment-914890958
> 使用他这个方法，build新的palo_be，替换后，测试备份还原是可以使用的
他的方式 在只使用一种对象存储时可用，但是大多数时候是不合适的
你可以通过 export MINIO_DOMAIN=mydomain.com 使minio 支持VirtualHost style， path style 因为安全问题 已经被大多数云储存厂商逐渐废弃了
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] morningman commented on issue #6574: [Bug] doris 0.14 backup/restore问题
Posted by GitBox <gi...@apache.org>.
morningman commented on issue #6574:
URL: https://github.com/apache/incubator-doris/issues/6574#issuecomment-913503155
what is your `create repository` statement?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] xinghuayu007 commented on issue #6574: [Bug] doris 0.14 backup/restore问题
Posted by GitBox <gi...@apache.org>.
xinghuayu007 commented on issue #6574:
URL: https://github.com/apache/incubator-doris/issues/6574#issuecomment-913505922
除了s3，其他存储能浮现吗？
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] ChPi commented on issue #6574: [Bug] doris 0.14 backup/restore问题
Posted by GitBox <gi...@apache.org>.
ChPi commented on issue #6574:
URL: https://github.com/apache/incubator-doris/issues/6574#issuecomment-958599678
resolved #6962
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org