You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by GitBox <gi...@apache.org> on 2020/01/21 14:34:11 UTC
[GitHub] [hadoop-ozone] adoroszlai opened a new pull request #468:
HDDS-2717. Handle chunk increments in datanode
adoroszlai opened a new pull request #468: HDDS-2717. Handle chunk increments in datanode
URL: https://github.com/apache/hadoop-ozone/pull/468
## What changes were proposed in this pull request?
Work-in-progress change for handling incremental chunks in datanode.
https://issues.apache.org/jira/browse/HDDS-2717
## How was this patch tested?
Configured buffer flush size to 1MB:
```
OZONE-SITE.XML_ozone.client.bytes.per.checksum=1MB
OZONE-SITE.XML_ozone.client.stream.buffer.flush.size=8MB
OZONE-SITE.XML_ozone.client.stream.buffer.max.size=16MB
```
Verified that `put`/`get` works for >1MB text file:
```
$ ozone freon ockg -n 1 -t 1 -p warmup
...
$ cat /etc/services /etc/services > /tmp/services2
$ ls -lah /tmp/services2
-rw-r--r-- 1 hadoop users 1.3M Jan 20 14:25 /tmp/services2
$ ozone sh key put /vol1/bucket1/services /tmp/services2
$ ozone sh key get /vol1/bucket1/services /tmp/services2.get
$ diff -q /tmp/services2 /tmp/services2.get; echo $?
0
$ ls -lh /data/hdds/hdds/*/current/containerDir0/2/chunks
total 1.3M
-rw-r--r-- 1 hadoop users 1.3M Jan 20 14:25 103516100478959618_chunk
```
Also tested with `freon ockg` (key size = 32MB, threads = 2), albeit with not too great results:
```
mean rate = 1.40 calls/second
```
vs.
```
mean rate = 1.86 calls/second
```
on master.
https://github.com/adoroszlai/hadoop-ozone/runs/400871105
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] adoroszlai closed pull request #468: HDDS-2717.
Handle chunk increments in datanode
Posted by GitBox <gi...@apache.org>.
adoroszlai closed pull request #468: HDDS-2717. Handle chunk increments in datanode
URL: https://github.com/apache/hadoop-ozone/pull/468
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org