You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by "lirui-apache (via GitHub)" <gi...@apache.org> on 2026/04/28 11:01:21 UTC
[PR] HDFS-17914. Prevent DFSInputStream from issuing 0-byte reads [hadoop]
lirui-apache opened a new pull request, #8463:
URL: https://github.com/apache/hadoop/pull/8463
<!--
Thanks for sending a pull request!
1. If this is your first time, please read our contributor guidelines: https://cwiki.apache.org/confluence/display/HADOOP/How+To+Contribute
2. Make sure your PR title starts with JIRA issue id, e.g., 'HADOOP-17799. Your PR title ...'.
-->
### Description of PR
Avoid empty buffer read in DFSInputStream#actualGetFromOneDataNode by terminating the loop once the buffer is filled.
### How was this patch tested?
Added new test
### For code changes:
- [x] Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?
- [ ] Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation?
- [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] If applicable, have you updated the `LICENSE`, `LICENSE-binary`, `NOTICE-binary` files?
### AI Tooling
If an AI tool was used:
- [ ] The PR includes the phrase "Contains content generated by <tool>"
where <tool> is the name of the AI tool used.
- [ ] My use of AI contributions follows the ASF legal policy
https://www.apache.org/legal/generative-tooling.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HDFS-17914. Prevent DFSInputStream from issuing 0-byte reads [hadoop]
Posted by "lirui-apache (via GitHub)" <gi...@apache.org>.
lirui-apache commented on code in PR #8463:
URL: https://github.com/apache/hadoop/pull/8463#discussion_r3167465565
##########
hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSInputStream.java:
##########
@@ -1230,7 +1230,9 @@ void actualGetFromOneDataNode(final DNAddrPair datanode, final long startInBlk,
long beginReadMS = Time.monotonicNow();
int nread = 0;
int ret;
- while (true) {
+ // Stop once the slice is filled; an extra read with remaining()==0
+ // can trigger wasted slow-lane I/O in SCR.
+ while (tmp.hasRemaining()) {
Review Comment:
Thanks for the review. I hit the issue when I was testing short circuit read. If we call `BlockReaderLocal.read` with `tmp.remaining() == 0`, it will hit the slow lane via `BlockReaderLocal::readWithBounceBuffer::fillDataBuf`, which will actually allocate and read into the bounce buffer. But since user buffer is filled, we cannot drain into it and the read data is thrown away and wasted.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HDFS-17914. Prevent DFSInputStream from issuing 0-byte reads [hadoop]
Posted by "lirui-apache (via GitHub)" <gi...@apache.org>.
lirui-apache commented on code in PR #8463:
URL: https://github.com/apache/hadoop/pull/8463#discussion_r3256142315
##########
hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSInputStream.java:
##########
@@ -1230,7 +1230,9 @@ void actualGetFromOneDataNode(final DNAddrPair datanode, final long startInBlk,
long beginReadMS = Time.monotonicNow();
int nread = 0;
int ret;
- while (true) {
+ // Stop once the slice is filled; an extra read with remaining()==0
+ // can trigger wasted slow-lane I/O in SCR.
+ while (tmp.hasRemaining()) {
Review Comment:
I was doing aligned positional reads with a DirectByteBuffer. Ideally I should only hit the fast lane and avoid the bounce buffer. But I noticed some cost from the slow lane and found the extra read. I believe the issue is pretty easy to trigger, but whether it brings real perf regression depends on the BlockReader implementations.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HDFS-17914. Prevent DFSInputStream from issuing 0-byte reads [hadoop]
Posted by "Hexiaoqiao (via GitHub)" <gi...@apache.org>.
Hexiaoqiao commented on code in PR #8463:
URL: https://github.com/apache/hadoop/pull/8463#discussion_r3245949619
##########
hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSInputStream.java:
##########
@@ -1230,7 +1230,9 @@ void actualGetFromOneDataNode(final DNAddrPair datanode, final long startInBlk,
long beginReadMS = Time.monotonicNow();
int nread = 0;
int ret;
- while (true) {
+ // Stop once the slice is filled; an extra read with remaining()==0
+ // can trigger wasted slow-lane I/O in SCR.
+ while (tmp.hasRemaining()) {
Review Comment:
Great catch. Make sense to me.
Just wonder how you hit this corner case, and why `tmp.hasRemaining==0` here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HDFS-17914. Prevent DFSInputStream from issuing 0-byte reads [hadoop]
Posted by "Hexiaoqiao (via GitHub)" <gi...@apache.org>.
Hexiaoqiao commented on code in PR #8463:
URL: https://github.com/apache/hadoop/pull/8463#discussion_r3165554638
##########
hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSInputStream.java:
##########
@@ -1230,7 +1230,9 @@ void actualGetFromOneDataNode(final DNAddrPair datanode, final long startInBlk,
long beginReadMS = Time.monotonicNow();
int nread = 0;
int ret;
- while (true) {
+ // Stop once the slice is filled; an extra read with remaining()==0
+ // can trigger wasted slow-lane I/O in SCR.
+ while (tmp.hasRemaining()) {
Review Comment:
Thanks for your contribution first. Just one nit concern, is it only one difference that return at L1235(after PR) or return at L1236(before PR)? It will perform a zero-byte transfer in-memory and return immediately if tmp.remaining()=0, So it is OK for me before PR. welcome discussion if I miss something. Thanks again.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HDFS-17914. Prevent DFSInputStream from issuing 0-byte reads [hadoop]
Posted by "Hexiaoqiao (via GitHub)" <gi...@apache.org>.
Hexiaoqiao commented on code in PR #8463:
URL: https://github.com/apache/hadoop/pull/8463#discussion_r3245949619
##########
hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSInputStream.java:
##########
@@ -1230,7 +1230,9 @@ void actualGetFromOneDataNode(final DNAddrPair datanode, final long startInBlk,
long beginReadMS = Time.monotonicNow();
int nread = 0;
int ret;
- while (true) {
+ // Stop once the slice is filled; an extra read with remaining()==0
+ // can trigger wasted slow-lane I/O in SCR.
+ while (tmp.hasRemaining()) {
Review Comment:
Great cache. Make sense to me.
Just wonder how you hit this corner case, and why `tmp.hasRemaining==0` here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org