You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "Ethan Rose (Jira)" <ji...@apache.org> on 2021/10/20 20:36:07 UTC
[jira] [Updated] (HDDS-4687) Disable compression for
closed-container replication
[ https://issues.apache.org/jira/browse/HDDS-4687?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ethan Rose updated HDDS-4687:
-----------------------------
Target Version/s: 1.3.0 (was: 1.2.0)
I am managing the 1.2.0 release and we currently have more than 600 issues targeted for 1.2.0. I am moving the target field to 1.3.0.
If you are actively working on this jira and believe this should be targeted for the 1.2.0 release, Please reach out to me via Apache email or Slack.
> Disable compression for closed-container replication
> ----------------------------------------------------
>
> Key: HDDS-4687
> URL: https://issues.apache.org/jira/browse/HDDS-4687
> Project: Apache Ozone
> Issue Type: Improvement
> Reporter: Marton Elek
> Assignee: Marton Elek
> Priority: Critical
> Labels: pull-request-available
> Attachments: HDDS-4687.patch
>
>
> During the measurement of closed container replication I found that the biggest bottleneck is the read side. 5 Gb container is replicated under ~3 minutes but ~2:30 was the downloading part.
> Closed containers are replicated via GRPC. The source side creates an OutputStream on-the-fly (OnDemandContainerReplicationSource.java) and stream all the container content as a "tar.gz" archive to the client.
> It turned out that the compression (the .gz part) is quite expensive:
> I created a CLI tool to export containers to tar files (same logic as the replication but without streaming via GRPC, just saving to a file).
> I have seen the 2:30 time to create the archive:
> {code}
> 2021-01-13 05:51:25,302 [main] INFO debug.ExportContainer: Preparation is done
> 2021-01-13 05:53:53,472 [main] INFO debug.ExportContainer: Container is exported to /tmp/container-3.tar.gz
> {code}
> But when I removed the compression in TarContainerPacker.java, the speed was significant better (25 sec instead of the 150 sec)
> {code}
> 2021-01-13 06:11:46,254 [main] INFO debug.ExportContainer: Preparation is done
> 2021-01-13 06:12:11,512 [main] INFO debug.ExportContainer: Container is exported to /tmp/container-3.tar
> {code}
> As a result I suggest turning off the compression for closed container replication.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org