You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by GitBox <gi...@apache.org> on 2021/01/20 09:34:38 UTC
[GitHub] [ozone] elek opened a new pull request #1825: Debug utility to export container data
elek opened a new pull request #1825:
URL: https://github.com/apache/ozone/pull/1825
## What changes were proposed in this pull request?
A new `ozone debug export-container --container=1` command which exports the container to a tar file.
The code path is the same as the closed container replication.
Can be useful for:
1. Measure speed of export part of the closed container replication
2. Debug container related problems locally
## How was this patch tested?
Executed on real Ozone cluster and measured the performance (and found HDDS-4687)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] bshashikant commented on pull request #1825: HDDS-4726. Debug utility to export container data
Posted by GitBox <gi...@apache.org>.
bshashikant commented on pull request #1825:
URL: https://github.com/apache/ozone/pull/1825#issuecomment-771543712
> > an we extend the tool to work against a datanode UUID as well a specific volume path to probably read all containers from these. This will be useful to simulate container replication in case a disk goes bad/node goes down scenario and verify the replication performance?
>
> I am not fully sure what do you mean. This tool reads the data directly from the disk using the same layout as the datanode. It's always user only one datanode UUID, and all the configured disks are used. You can configure any sub-set of the available disks if you would like to use less disks.
>
> end2end container replication can be used with https://github.com/apache/ozone/blob/master/hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/freon/ClosedContainerReplicator.java
Thanks @elek . I understand that, this can only be run on datanode and it makes sense. +1
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] bshashikant commented on pull request #1825: HDDS-4726. Debug utility to export container data
Posted by GitBox <gi...@apache.org>.
bshashikant commented on pull request #1825:
URL: https://github.com/apache/ozone/pull/1825#issuecomment-764538991
Thanks @elek for the patch. The changes look good. Can we extend the tool to work against a datanode UUID as well a specific volume path to probably read all containers from these. This will be useful to simulate container replication in case a disk goes bad/node goes down scenario and verify the replication performance?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] bshashikant commented on pull request #1825: HDDS-4726. Debug utility to export container data
Posted by GitBox <gi...@apache.org>.
bshashikant commented on pull request #1825:
URL: https://github.com/apache/ozone/pull/1825#issuecomment-764538991
Thanks @elek for the patch. The changes look good. Can we extend the tool to work against a datanode UUID as well a specific volume path to probably read all containers from these. This will be useful to simulate container replication in case a disk goes bad/node goes down scenario and verify the replication performance?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] elek commented on pull request #1825: HDDS-4726. Debug utility to export container data
Posted by GitBox <gi...@apache.org>.
elek commented on pull request #1825:
URL: https://github.com/apache/ozone/pull/1825#issuecomment-770873070
> an we extend the tool to work against a datanode UUID as well a specific volume path to probably read all containers from these. This will be useful to simulate container replication in case a disk goes bad/node goes down scenario and verify the replication performance?
I am not fully sure what do you mean. This tool reads the data directly from the disk using the same layout as the datanode. It's always user only one datanode UUID, and all the configured disks are used. You can configure any sub-set of the available disks if you would like to use less disks.
end2end container replication can be used with https://github.com/apache/ozone/blob/master/hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/freon/ClosedContainerReplicator.java
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] elek commented on pull request #1825: HDDS-4726. Debug utility to export container data
Posted by GitBox <gi...@apache.org>.
elek commented on pull request #1825:
URL: https://github.com/apache/ozone/pull/1825#issuecomment-775795048
Thanks the review @bshashikant, merging it based on your +1
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] elek merged pull request #1825: HDDS-4726. Debug utility to export container data
Posted by GitBox <gi...@apache.org>.
elek merged pull request #1825:
URL: https://github.com/apache/ozone/pull/1825
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] bshashikant commented on pull request #1825: HDDS-4726. Debug utility to export container data
Posted by GitBox <gi...@apache.org>.
bshashikant commented on pull request #1825:
URL: https://github.com/apache/ozone/pull/1825#issuecomment-771543712
> > an we extend the tool to work against a datanode UUID as well a specific volume path to probably read all containers from these. This will be useful to simulate container replication in case a disk goes bad/node goes down scenario and verify the replication performance?
>
> I am not fully sure what do you mean. This tool reads the data directly from the disk using the same layout as the datanode. It's always user only one datanode UUID, and all the configured disks are used. You can configure any sub-set of the available disks if you would like to use less disks.
>
> end2end container replication can be used with https://github.com/apache/ozone/blob/master/hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/freon/ClosedContainerReplicator.java
Thanks @elek . I understand that, this can only be run on datanode and it makes sense. +1
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org