You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "Marton Elek (Jira)" <ji...@apache.org> on 2020/11/13 14:48:00 UTC
[jira] [Updated] (HDDS-4459) Improve closed container replication
[ https://issues.apache.org/jira/browse/HDDS-4459?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Marton Elek updated HDDS-4459:
------------------------------
Description:
Closed are immutable therefore the replication is different from Ratis.
Closed containers are downloaded from other datanodes over a specific GRPC channel.
While the first implementation works as intended it has many spaces for improvements.
1. the container replica is compressed with gzip. A more efficient compression codec like zstd might be better (reported Wei-Chiu)
2. Current downloader uses only one thread (reported Wei-Chiu)
3. Temporary space is configurable, but by default uses /tmp. This means we are bottlenecked by the root disk, even though we have 48 disks on each DN. There should be an option to store the temp file under the destination disk instead. (reported Wei-Chiu)
4. Buffer size is hard coded (1MB) and may not be the ideal
5. There are no metrics which clearly shows the performance of the replication
6. Separate client and server2server RPC listeners
was:
Closed are immutable therefore the replication is different from Ratis.
Closed containers are downloaded from other datanodes over a specific GRPC channel.
While the first implementation works as intended it has many spaces for improvements.
1. the container replica is compressed with gzip. A more efficient compression codec like zstd might be better (reported Wei-Chiu)
2. Current downloader uses only one thread (reported Wei-Chiu)
3. Temporary space is configurable, but by default uses /tmp. This means we are bottlenecked by the root disk, even though we have 48 disks on each DN. There should be an option to store the temp file under the destination disk instead. (reported Wei-Chiu)
4. Buffer size is hard coded (1MB) and may not be the ideal
5. There are no metrics which clearly shows the performance of the replication
> Improve closed container replication
> ------------------------------------
>
> Key: HDDS-4459
> URL: https://issues.apache.org/jira/browse/HDDS-4459
> Project: Hadoop Distributed Data Store
> Issue Type: Improvement
> Reporter: Marton Elek
> Assignee: Marton Elek
> Priority: Major
>
> Closed are immutable therefore the replication is different from Ratis.
> Closed containers are downloaded from other datanodes over a specific GRPC channel.
> While the first implementation works as intended it has many spaces for improvements.
> 1. the container replica is compressed with gzip. A more efficient compression codec like zstd might be better (reported Wei-Chiu)
> 2. Current downloader uses only one thread (reported Wei-Chiu)
> 3. Temporary space is configurable, but by default uses /tmp. This means we are bottlenecked by the root disk, even though we have 48 disks on each DN. There should be an option to store the temp file under the destination disk instead. (reported Wei-Chiu)
> 4. Buffer size is hard coded (1MB) and may not be the ideal
> 5. There are no metrics which clearly shows the performance of the replication
> 6. Separate client and server2server RPC listeners
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org