You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by GitBox <gi...@apache.org> on 2022/08/11 03:04:30 UTC
[GitHub] [ozone] szetszwo opened a new pull request, #3674: HDDS-7117. Consider reading chunk files using MappedByteBuffer.
szetszwo opened a new pull request, #3674:
URL: https://github.com/apache/ozone/pull/3674
See https://issues.apache.org/jira/browse/HDDS-7117
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-7117. Consider reading chunk files using MappedByteBuffer. [ozone]
Posted by "ChenSammi (via GitHub)" <gi...@apache.org>.
ChenSammi commented on PR #3674:
URL: https://github.com/apache/ozone/pull/3674#issuecomment-1840265436
@szetszwo , there is another your PR #3730 which aims to improve the read performance too. I didn't have time to closely look at #3730 yet. Not sure is there overlap between this one and #3730?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] szetszwo commented on pull request #3674: HDDS-7117. Consider reading chunk files using MappedByteBuffer.
Posted by GitBox <gi...@apache.org>.
szetszwo commented on PR #3674:
URL: https://github.com/apache/ozone/pull/3674#issuecomment-1229038269
@jojochuang , sure. Will wait for your test results. Thanks a lot!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-7117. Consider reading chunk files using MappedByteBuffer. [ozone]
Posted by "szetszwo (via GitHub)" <gi...@apache.org>.
szetszwo commented on PR #3674:
URL: https://github.com/apache/ozone/pull/3674#issuecomment-1841409074
@ChenSammi , this PR is to change reading the local files using java `MappedByteBuffer` while https://github.com/apache/ozone/pull/3730 uses Netty `ChunkedNioFile`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] szetszwo commented on pull request #3674: HDDS-7117. Consider reading chunk files using MappedByteBuffer.
Posted by GitBox <gi...@apache.org>.
szetszwo commented on PR #3674:
URL: https://github.com/apache/ozone/pull/3674#issuecomment-1219786404
@jojochuang , thanks a lot for testing this!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-7117. Consider reading chunk files using MappedByteBuffer. [ozone]
Posted by "ChenSammi (via GitHub)" <gi...@apache.org>.
ChenSammi commented on PR #3674:
URL: https://github.com/apache/ozone/pull/3674#issuecomment-1838121303
@szetszwo , thanks for working on this. Do you have time to do a patch rebase?
I did a micro performance test, comparing the on heap buffer, off heap buffer and memory mapped buffer, here is the result.
By default, DN will use 1MB(bytes.per.checksum) as the buffer size. So using memory mapped buffer can increase at least micro level performance a lot.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] swamirishi commented on pull request #3674: HDDS-7117. Consider reading chunk files using MappedByteBuffer.
Posted by "swamirishi (via GitHub)" <gi...@apache.org>.
swamirishi commented on PR #3674:
URL: https://github.com/apache/ozone/pull/3674#issuecomment-1406920727
@jojochuang Can this be merged? Are the test results out?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-7117. Consider reading chunk files using MappedByteBuffer. [ozone]
Posted by "szetszwo (via GitHub)" <gi...@apache.org>.
szetszwo commented on PR #3674:
URL: https://github.com/apache/ozone/pull/3674#issuecomment-1839112527
@ChenSammi , thanks a lot for running the benchmark! When the file size is large, the performance is expected to be better for `MappedByteBuffer `. Your benchmark confirms the expectation.
Let me update this change.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] jojochuang commented on pull request #3674: HDDS-7117. Consider reading chunk files using MappedByteBuffer.
Posted by GitBox <gi...@apache.org>.
jojochuang commented on PR #3674:
URL: https://github.com/apache/ozone/pull/3674#issuecomment-1228878926
Please hold the merge as I am still doing a few more perf benchmarks. Thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-7117. Consider reading chunk files using MappedByteBuffer. [ozone]
Posted by "szetszwo (via GitHub)" <gi...@apache.org>.
szetszwo commented on PR #3674:
URL: https://github.com/apache/ozone/pull/3674#issuecomment-1843130831
@ChenSammi , thanks a lot for reviewing this!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-7117. Consider reading chunk files using MappedByteBuffer. [ozone]
Posted by "szetszwo (via GitHub)" <gi...@apache.org>.
szetszwo merged PR #3674:
URL: https://github.com/apache/ozone/pull/3674
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org