You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by GitBox <gi...@apache.org> on 2020/11/16 05:03:59 UTC
[GitHub] [hadoop] bilaharith opened a new pull request #2464: Draft PR for read optimizations
bilaharith opened a new pull request #2464:
URL: https://github.com/apache/hadoop/pull/2464
Draft PR for read optimizations
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] bilaharith commented on a change in pull request #2464: HADOOP-17347. ABFS: Read optimizations
Posted by GitBox <gi...@apache.org>.
bilaharith commented on a change in pull request #2464:
URL: https://github.com/apache/hadoop/pull/2464#discussion_r540353397
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsInputStream.java
##########
@@ -137,7 +142,13 @@ public synchronized int read(final byte[] b, final int off, final int len) throw
}
incrementReadOps();
do {
- lastReadBytes = readOneBlock(b, currentOff, currentLen);
+ if (shouldReadFully()) {
Review comment:
Done
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] bilaharith commented on a change in pull request #2464: HADOOP-17347. ABFS: Read optimizations
Posted by GitBox <gi...@apache.org>.
bilaharith commented on a change in pull request #2464:
URL: https://github.com/apache/hadoop/pull/2464#discussion_r549302272
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsInputStream.java
##########
@@ -211,6 +235,59 @@ private int readOneBlock(final byte[] b, final int off, final int len) throws IO
return bytesToRead;
}
+ private boolean shouldReadFully() {
+ return this.firstRead && this.context.readSmallFilesCompletely()
+ && this.contentLength <= this.bufferSize;
+ }
+
+ private boolean shouldReadLastBlock(int len) {
+ return this.firstRead && this.context.optimizeFooterRead()
+ && len == FOOTER_SIZE
+ && this.fCursor == this.contentLength - FOOTER_SIZE;
+ }
+
+ private int readFileCompletely(final byte[] b, final int off, final int len) throws IOException {
+ int validation = validate(b, off, len);
+ if (validation < 1) {
+ return validation;
+ }
+
+ buffer = new byte[bufferSize];
+ bCursor = (int) fCursor;
+ int bytesRead = readInternal(0, buffer, 0, (int) contentLength, true);
+ firstRead = false;
Review comment:
Test cases added for the same
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] bilaharith commented on a change in pull request #2464: HADOOP-17347. ABFS: Read optimizations
Posted by GitBox <gi...@apache.org>.
bilaharith commented on a change in pull request #2464:
URL: https://github.com/apache/hadoop/pull/2464#discussion_r544182784
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsInputStream.java
##########
@@ -38,6 +38,10 @@
import org.apache.hadoop.fs.azurebfs.contracts.exceptions.AzureBlobFileSystemException;
import org.apache.hadoop.fs.azurebfs.utils.CachedSASToken;
+import static java.lang.Math.max;
Review comment:
When static import is used, I can simply write min(a,b) instead of Math.min(a, b)
Also I don't want to import the whole class statically for just these 2 methods.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] snvijaya commented on a change in pull request #2464: HADOOP-17347. ABFS: Read optimizations
Posted by GitBox <gi...@apache.org>.
snvijaya commented on a change in pull request #2464:
URL: https://github.com/apache/hadoop/pull/2464#discussion_r543919524
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsInputStream.java
##########
@@ -206,11 +231,127 @@ private int readOneBlock(final byte[] b, final int off, final int len) throws IO
fCursor += bytesRead;
fCursorAfterLastRead = fCursor;
}
+ return copyToUserBuffer(b, off, len);
+ }
+ private int readFileCompletely(final byte[] b, final int off, final int len)
+ throws IOException {
+ if (len == 0) {
+ return 0;
+ }
+ if (!validate(b, off, len)) {
+ return -1;
+ }
+ savePointerState();
+
+ buffer = new byte[bufferSize];
+ // data need to be copied to user buffer from index bCursor, bCursor has
+ // to be the current fCusor
+ bCursor = (int) fCursor;
+ return optimisedRead(b, off, len, 0, contentLength);
+ }
+
+ private int readLastBlock(final byte[] b, final int off, final int len)
+ throws IOException {
+ if (len == 0) {
+ return 0;
+ }
+ if (!validate(b, off, len)) {
+ return -1;
+ }
+ savePointerState();
+
+ buffer = new byte[bufferSize];
+ // data need to be copied to user buffer from index bCursor,
+ // AbfsInutStream buffer is going p contain dta from footer start. In
+ // that case bCursor will be set to footerStart - fCursor
+ long lastBlockStart = max(0, contentLength - bufferSize);
+ bCursor = (int) (fCursor - lastBlockStart);
+ // read API call is considered 1 single operation in reality server could
+ // return partial data and client has to retry untill the last full block
+ // is read. So setting the fCursorAfterLastRead before the possible
+ // multiple server calls
+ fCursorAfterLastRead = fCursor;
Review comment:
why is this done here ? Why cant this be updated as the read progresses ?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] vinaysbadami commented on a change in pull request #2464: Draft PR for read optimizations
Posted by GitBox <gi...@apache.org>.
vinaysbadami commented on a change in pull request #2464:
URL: https://github.com/apache/hadoop/pull/2464#discussion_r523911156
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsInputStream.java
##########
@@ -128,7 +138,7 @@ public synchronized int read(final byte[] b, final int off, final int len) throw
}
incrementReadOps();
do {
- lastReadBytes = readOneBlock(b, currentOff, currentLen);
+ lastReadBytes = readToUserBuffer(b, currentOff, currentLen);
Review comment:
why the change? Please revert if not require
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] bilaharith commented on a change in pull request #2464: HADOOP-17347. ABFS: Read optimizations
Posted by GitBox <gi...@apache.org>.
bilaharith commented on a change in pull request #2464:
URL: https://github.com/apache/hadoop/pull/2464#discussion_r543157634
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsInputStream.java
##########
@@ -206,11 +229,120 @@ private int readOneBlock(final byte[] b, final int off, final int len) throws IO
fCursor += bytesRead;
fCursorAfterLastRead = fCursor;
}
+ return copyToUserBuffer(b, off, len);
+ }
+
+ private int readFileCompletely(final byte[] b, final int off, final int len)
+ throws IOException {
+ if (len == 0) {
+ return 0;
+ }
+ if (!validate(b, off, len)) {
+ return -1;
+ }
+ savePointerState();
+
+ buffer = new byte[bufferSize];
+ // data need to be copied to user buffer from index bCursor, bCursor has
+ // to be the current fCusor
+ bCursor = (int) fCursor;
+ return optimisedRead(b, off, len, 0, contentLength);
+ }
+
+ private int readLastBlock(final byte[] b, final int off, final int len)
+ throws IOException {
+ if (len == 0) {
+ return 0;
+ }
+ if (!validate(b, off, len)) {
+ return -1;
+ }
+ savePointerState();
+
+ buffer = new byte[bufferSize];
+ // data need to be copied to user buffer from index bCursor,
+ // AbfsInutStream buffer is going p contain dta from footer start. In
+ // that case bCursor will be set to footerStart - fCursor
+ long lastBlockStart = max(0, contentLength - bufferSize);
+ bCursor = (int) (fCursor - lastBlockStart);
+ // read API call is considered 1 single operation in reality server could
+ // return partial data and client has to retry untill the last full block
+ // is read. So setting the fCursorAfterLastRead before the possible
+ // multiple server calls
+ fCursorAfterLastRead = fCursor;
+ // 0 if contentlength is < buffersize
+ long actualLenToRead = min(bufferSize, contentLength);
+ return optimisedRead(b, off, len, lastBlockStart, actualLenToRead);
+ }
+
+ private int optimisedRead(final byte[] b, final int off, final int len,
+ final long readFrom, final long actualLen) throws IOException {
+ int totalBytesRead = 0;
+ fCursor = readFrom;
+ for (int i = 0; i < 2 && fCursor < contentLength; i++) {
+ int bytesRead = readInternal(fCursor, buffer, limit,
Review comment:
Done
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] vinaysbadami commented on a change in pull request #2464: Draft PR for read optimizations
Posted by GitBox <gi...@apache.org>.
vinaysbadami commented on a change in pull request #2464:
URL: https://github.com/apache/hadoop/pull/2464#discussion_r523921761
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsInputStream.java
##########
@@ -162,14 +172,86 @@ private int readOneBlock(final byte[] b, final int off, final int len) throws IO
throw new IndexOutOfBoundsException();
}
- //If buffer is empty, then fill the buffer.
- if (bCursor == limit) {
+ if (firstRead && this.readSmallFilesCompletely
+ && contentLength <= bufferSize) { // Read small files completely
+ if (readFileCompletely() == -1) {
+ return -1;
+ }
+ } else if (firstRead && this.optimizeFooterRead && fCursor
+ == contentLength - FOOTER_DELTA) { // Read the last one block if the
+ // read is for the footer
+ if (readLastBlock() == -1) {
+ return -1;
+ }
+ } else {
+ if (readOneBlock(b) == -1) {
+ return -1;
+ }
+ }
+ fCursorAfterLastRead = fCursor;
+
+ //If there is anything in the buffer, then return lesser of (requested bytes) and (bytes in buffer)
+ //(bytes returned may be less than requested)
+ int bytesRemaining = limit - bCursor;
+ int bytesToRead = Math.min(len, bytesRemaining);
+ System.arraycopy(buffer, bCursor, b, off, bytesToRead);
+ bCursor += bytesToRead;
+ if (statistics != null) {
+ statistics.incrementBytesRead(bytesToRead);
+ }
+ if (streamStatistics != null) {
+ // Bytes read from the local buffer.
+ streamStatistics.bytesReadFromBuffer(bytesToRead);
+ streamStatistics.bytesRead(bytesToRead);
+ }
+ return bytesToRead;
+ }
+
+ private long readFileCompletely() throws IOException {
+ bCursor = (int) getPos();
Review comment:
check this - should be the offset into buffer passed into read
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] snvijaya commented on a change in pull request #2464: HADOOP-17347. ABFS: Read optimizations
Posted by GitBox <gi...@apache.org>.
snvijaya commented on a change in pull request #2464:
URL: https://github.com/apache/hadoop/pull/2464#discussion_r543916656
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsInputStream.java
##########
@@ -206,11 +231,127 @@ private int readOneBlock(final byte[] b, final int off, final int len) throws IO
fCursor += bytesRead;
fCursorAfterLastRead = fCursor;
}
+ return copyToUserBuffer(b, off, len);
+ }
+ private int readFileCompletely(final byte[] b, final int off, final int len)
+ throws IOException {
+ if (len == 0) {
+ return 0;
+ }
+ if (!validate(b, off, len)) {
+ return -1;
+ }
+ savePointerState();
+
+ buffer = new byte[bufferSize];
+ // data need to be copied to user buffer from index bCursor, bCursor has
+ // to be the current fCusor
+ bCursor = (int) fCursor;
+ return optimisedRead(b, off, len, 0, contentLength);
+ }
+
+ private int readLastBlock(final byte[] b, final int off, final int len)
+ throws IOException {
+ if (len == 0) {
+ return 0;
+ }
+ if (!validate(b, off, len)) {
+ return -1;
+ }
+ savePointerState();
+
+ buffer = new byte[bufferSize];
+ // data need to be copied to user buffer from index bCursor,
+ // AbfsInutStream buffer is going p contain dta from footer start. In
+ // that case bCursor will be set to footerStart - fCursor
+ long lastBlockStart = max(0, contentLength - bufferSize);
+ bCursor = (int) (fCursor - lastBlockStart);
+ // read API call is considered 1 single operation in reality server could
+ // return partial data and client has to retry untill the last full block
+ // is read. So setting the fCursorAfterLastRead before the possible
+ // multiple server calls
+ fCursorAfterLastRead = fCursor;
+ // 0 if contentlength is < buffersize
+ long actualLenToRead = min(bufferSize, contentLength);
+ return optimisedRead(b, off, len, lastBlockStart, actualLenToRead);
+ }
+
+ private int optimisedRead(final byte[] b, final int off, final int len,
+ final long readFrom, final long actualLen) throws IOException {
+ int totalBytesRead = 0;
+ fCursor = readFrom;
+ try {
+ for (int i = 0;
+ i < OPTIMIZATION_ATTEMPTS && fCursor < contentLength; i++) {
+ int bytesRead = readInternal(fCursor, buffer, limit,
+ (int) actualLen - limit, true);
+ if (bytesRead > 0) {
+ totalBytesRead += bytesRead;
+ limit += bytesRead;
+ fCursor += bytesRead;
+ }
Review comment:
What about fCursorAfterLastRead ?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] bilaharith commented on a change in pull request #2464: HADOOP-17347. ABFS: Read optimizations
Posted by GitBox <gi...@apache.org>.
bilaharith commented on a change in pull request #2464:
URL: https://github.com/apache/hadoop/pull/2464#discussion_r548604646
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsInputStream.java
##########
@@ -206,11 +231,121 @@ private int readOneBlock(final byte[] b, final int off, final int len) throws IO
fCursor += bytesRead;
fCursorAfterLastRead = fCursor;
}
+ return copyToUserBuffer(b, off, len);
+ }
+
+ private int readFileCompletely(final byte[] b, final int off, final int len)
+ throws IOException {
+ if (len == 0) {
+ return 0;
+ }
+ if (!validate(b, off, len)) {
+ return -1;
+ }
+ savePointerState();
+ // data need to be copied to user buffer from index bCursor, bCursor has
+ // to be the current fCusor
+ bCursor = (int) fCursor;
+ return optimisedRead(b, off, len, 0, contentLength);
+ }
+
+ private int readLastBlock(final byte[] b, final int off, final int len)
+ throws IOException {
+ if (len == 0) {
+ return 0;
+ }
+ if (!validate(b, off, len)) {
+ return -1;
+ }
+ savePointerState();
+ // data need to be copied to user buffer from index bCursor,
+ // AbfsInutStream buffer is going to contain data from last block start. In
+ // that case bCursor will be set to fCursor - lastBlockStart
+ long lastBlockStart = max(0, contentLength - bufferSize);
+ bCursor = (int) (fCursor - lastBlockStart);
+ // 0 if contentlength is < buffersize
+ long actualLenToRead = min(bufferSize, contentLength);
+ return optimisedRead(b, off, len, lastBlockStart, actualLenToRead);
+ }
+
+ private int optimisedRead(final byte[] b, final int off, final int len,
+ final long readFrom, final long actualLen) throws IOException {
+ fCursor = readFrom;
+ int totalBytesRead = 0;
+ int lastBytesRead = 0;
+ try {
+ buffer = new byte[bufferSize];
+ for (int i = 0;
+ i < MAX_OPTIMIZED_READ_ATTEMPTS && fCursor < contentLength; i++) {
+ lastBytesRead = readInternal(fCursor, buffer, limit,
+ (int) actualLen - limit, true);
+ if (lastBytesRead > 0) {
+ totalBytesRead += lastBytesRead;
+ limit += lastBytesRead;
+ fCursor += lastBytesRead;
+ fCursorAfterLastRead = fCursor;
+ }
+ }
+ } catch (IOException e) {
+ LOG.debug("Optimized read failed. Defaulting to readOneBlock {}", e);
+ restorePointerState();
+ return readOneBlock(b, off, len);
+ }
+ firstRead = false;
+ if (totalBytesRead < 1) {
+ return lastBytesRead;
Review comment:
Done
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] snvijaya commented on a change in pull request #2464: HADOOP-17347. ABFS: Read optimizations
Posted by GitBox <gi...@apache.org>.
snvijaya commented on a change in pull request #2464:
URL: https://github.com/apache/hadoop/pull/2464#discussion_r543101567
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsInputStream.java
##########
@@ -46,6 +50,7 @@
public class AbfsInputStream extends FSInputStream implements CanUnbuffer,
StreamCapabilities {
private static final Logger LOG = LoggerFactory.getLogger(AbfsInputStream.class);
+ public static final int FOOTER_SIZE = 16 * ONE_KB;
Review comment:
Add a comment that footer size is set to qualify for both ORC and parquet files.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] bilaharith commented on a change in pull request #2464: HADOOP-17347. ABFS: Read optimizations
Posted by GitBox <gi...@apache.org>.
bilaharith commented on a change in pull request #2464:
URL: https://github.com/apache/hadoop/pull/2464#discussion_r540353048
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsInputStream.java
##########
@@ -211,6 +235,59 @@ private int readOneBlock(final byte[] b, final int off, final int len) throws IO
return bytesToRead;
}
+ private boolean shouldReadFully() {
+ return this.firstRead && this.context.readSmallFilesCompletely()
+ && this.contentLength <= this.bufferSize;
+ }
+
+ private boolean shouldReadLastBlock(int len) {
+ return this.firstRead && this.context.optimizeFooterRead()
+ && len == FOOTER_SIZE
+ && this.fCursor == this.contentLength - FOOTER_SIZE;
+ }
+
+ private int readFileCompletely(final byte[] b, final int off, final int len) throws IOException {
+ int validation = validate(b, off, len);
+ if (validation < 1) {
+ return validation;
+ }
+
+ buffer = new byte[bufferSize];
+ bCursor = (int) fCursor;
+ int bytesRead = readInternal(0, buffer, 0, (int) contentLength, true);
+ firstRead = false;
+ if (bytesRead == -1) {
+ return -1;
+ }
+ fCursorAfterLastRead = fCursor;
+ limit = bytesRead;
+ fCursor = bytesRead;
+ return copyToUserBuffer(b, off, len);
+ }
+
+ private int readLastBlock(final byte[] b, final int off, final int len) throws IOException {
+ int validation = validate(b, off, len);
+ if (validation < 1) {
+ return validation;
+ }
+
+ buffer = new byte[bufferSize];
+ bCursor = (int) (((contentLength < bufferSize) ? contentLength : bufferSize)
+ - FOOTER_SIZE);
+ long lastBlockStartPos = (contentLength < bufferSize)
+ ? 0
+ : contentLength - bufferSize;
+ int bytesRead = readInternal(lastBlockStartPos, buffer, 0, bufferSize,
Review comment:
Fixes are done for partial read scenarios
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #2464: HADOOP-17347. ABFS: Read optimizations
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on pull request #2464:
URL: https://github.com/apache/hadoop/pull/2464#issuecomment-746167436
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 34s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | | 0m 0s | [test4tests](test4tests) | The patch appears to include 2 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 33m 41s | | trunk passed |
| +1 :green_heart: | compile | 0m 38s | | trunk passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 |
| +1 :green_heart: | compile | 0m 34s | | trunk passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| +1 :green_heart: | checkstyle | 0m 28s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 32s | | trunk passed |
| +1 :green_heart: | shadedclient | 16m 42s | | branch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 30s | | trunk passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 |
| +1 :green_heart: | javadoc | 0m 29s | | trunk passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| +0 :ok: | spotbugs | 0m 59s | | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 0m 55s | | trunk passed |
| -0 :warning: | patch | 1m 15s | | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 31s | | the patch passed |
| +1 :green_heart: | compile | 0m 31s | | the patch passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 |
| +1 :green_heart: | javac | 0m 31s | | the patch passed |
| +1 :green_heart: | compile | 0m 27s | | the patch passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| +1 :green_heart: | javac | 0m 27s | | the patch passed |
| -0 :warning: | checkstyle | 0m 16s | [/diff-checkstyle-hadoop-tools_hadoop-azure.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2464/17/artifact/out/diff-checkstyle-hadoop-tools_hadoop-azure.txt) | hadoop-tools/hadoop-azure: The patch generated 2 new + 2 unchanged - 0 fixed = 4 total (was 2) |
| +1 :green_heart: | mvnsite | 0m 29s | | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | | The patch has no whitespace issues. |
| +1 :green_heart: | shadedclient | 14m 45s | | patch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 28s | | the patch passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 |
| +1 :green_heart: | javadoc | 0m 23s | | the patch passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| +1 :green_heart: | findbugs | 1m 1s | | the patch passed |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 1m 25s | | hadoop-azure in the patch passed. |
| +1 :green_heart: | asflicense | 0m 33s | | The patch does not generate ASF License warnings. |
| | | 78m 23s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2464/17/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/2464 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 65bf37a93bd8 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / df35c7f5199 |
| Default Java | Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2464/17/testReport/ |
| Max. process+thread count | 679 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2464/17/console |
| versions | git=2.17.1 maven=3.6.0 findbugs=4.0.6 |
| Powered by | Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] bilaharith commented on a change in pull request #2464: HADOOP-17347. ABFS: Read optimizations
Posted by GitBox <gi...@apache.org>.
bilaharith commented on a change in pull request #2464:
URL: https://github.com/apache/hadoop/pull/2464#discussion_r539027402
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsInputStream.java
##########
@@ -211,6 +235,59 @@ private int readOneBlock(final byte[] b, final int off, final int len) throws IO
return bytesToRead;
}
+ private boolean shouldReadFully() {
+ return this.firstRead && this.context.readSmallFilesCompletely()
+ && this.contentLength <= this.bufferSize;
+ }
+
+ private boolean shouldReadLastBlock(int len) {
+ return this.firstRead && this.context.optimizeFooterRead()
+ && len == FOOTER_SIZE
+ && this.fCursor == this.contentLength - FOOTER_SIZE;
+ }
+
+ private int readFileCompletely(final byte[] b, final int off, final int len) throws IOException {
+ int validation = validate(b, off, len);
+ if (validation < 1) {
+ return validation;
+ }
+
+ buffer = new byte[bufferSize];
+ bCursor = (int) fCursor;
+ int bytesRead = readInternal(0, buffer, 0, (int) contentLength, true);
+ firstRead = false;
+ if (bytesRead == -1) {
+ return -1;
+ }
+ fCursorAfterLastRead = fCursor;
+ limit = bytesRead;
+ fCursor = bytesRead;
+ return copyToUserBuffer(b, off, len);
+ }
+
+ private int readLastBlock(final byte[] b, final int off, final int len) throws IOException {
+ int validation = validate(b, off, len);
+ if (validation < 1) {
+ return validation;
+ }
+
+ buffer = new byte[bufferSize];
+ bCursor = (int) (((contentLength < bufferSize) ? contentLength : bufferSize)
+ - FOOTER_SIZE);
+ long lastBlockStartPos = (contentLength < bufferSize)
+ ? 0
+ : contentLength - bufferSize;
+ int bytesRead = readInternal(lastBlockStartPos, buffer, 0, bufferSize,
Review comment:
Done
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] bilaharith commented on a change in pull request #2464: HADOOP-17347. ABFS: Read optimizations
Posted by GitBox <gi...@apache.org>.
bilaharith commented on a change in pull request #2464:
URL: https://github.com/apache/hadoop/pull/2464#discussion_r535145966
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsInputStream.java
##########
@@ -161,6 +174,14 @@ private int readOneBlock(final byte[] b, final int off, final int len) throws IO
if (off < 0 || len < 0 || len > b.length - off) {
Review comment:
Done
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] snvijaya commented on a change in pull request #2464: HADOOP-17347. ABFS: Read optimizations
Posted by GitBox <gi...@apache.org>.
snvijaya commented on a change in pull request #2464:
URL: https://github.com/apache/hadoop/pull/2464#discussion_r543108714
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsInputStream.java
##########
@@ -206,11 +229,120 @@ private int readOneBlock(final byte[] b, final int off, final int len) throws IO
fCursor += bytesRead;
fCursorAfterLastRead = fCursor;
}
+ return copyToUserBuffer(b, off, len);
+ }
+
+ private int readFileCompletely(final byte[] b, final int off, final int len)
+ throws IOException {
+ if (len == 0) {
+ return 0;
+ }
+ if (!validate(b, off, len)) {
+ return -1;
+ }
+ savePointerState();
+
+ buffer = new byte[bufferSize];
+ // data need to be copied to user buffer from index bCursor, bCursor has
+ // to be the current fCusor
+ bCursor = (int) fCursor;
+ return optimisedRead(b, off, len, 0, contentLength);
+ }
+
+ private int readLastBlock(final byte[] b, final int off, final int len)
+ throws IOException {
+ if (len == 0) {
+ return 0;
+ }
+ if (!validate(b, off, len)) {
+ return -1;
+ }
+ savePointerState();
+
+ buffer = new byte[bufferSize];
+ // data need to be copied to user buffer from index bCursor,
+ // AbfsInutStream buffer is going p contain dta from footer start. In
+ // that case bCursor will be set to footerStart - fCursor
+ long lastBlockStart = max(0, contentLength - bufferSize);
+ bCursor = (int) (fCursor - lastBlockStart);
+ // read API call is considered 1 single operation in reality server could
+ // return partial data and client has to retry untill the last full block
+ // is read. So setting the fCursorAfterLastRead before the possible
+ // multiple server calls
+ fCursorAfterLastRead = fCursor;
+ // 0 if contentlength is < buffersize
+ long actualLenToRead = min(bufferSize, contentLength);
+ return optimisedRead(b, off, len, lastBlockStart, actualLenToRead);
+ }
+
+ private int optimisedRead(final byte[] b, final int off, final int len,
+ final long readFrom, final long actualLen) throws IOException {
+ int totalBytesRead = 0;
+ fCursor = readFrom;
+ for (int i = 0; i < 2 && fCursor < contentLength; i++) {
+ int bytesRead = readInternal(fCursor, buffer, limit,
Review comment:
In failure cases, file pointers can be in incorrect position. Need proper restore.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] snvijaya commented on a change in pull request #2464: HADOOP-17347. ABFS: Read optimizations
Posted by GitBox <gi...@apache.org>.
snvijaya commented on a change in pull request #2464:
URL: https://github.com/apache/hadoop/pull/2464#discussion_r544816563
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsInputStream.java
##########
@@ -206,11 +231,127 @@ private int readOneBlock(final byte[] b, final int off, final int len) throws IO
fCursor += bytesRead;
fCursorAfterLastRead = fCursor;
}
+ return copyToUserBuffer(b, off, len);
+ }
+ private int readFileCompletely(final byte[] b, final int off, final int len)
+ throws IOException {
+ if (len == 0) {
+ return 0;
+ }
+ if (!validate(b, off, len)) {
+ return -1;
+ }
+ savePointerState();
+
+ buffer = new byte[bufferSize];
+ // data need to be copied to user buffer from index bCursor, bCursor has
+ // to be the current fCusor
+ bCursor = (int) fCursor;
+ return optimisedRead(b, off, len, 0, contentLength);
+ }
+
+ private int readLastBlock(final byte[] b, final int off, final int len)
+ throws IOException {
+ if (len == 0) {
+ return 0;
+ }
+ if (!validate(b, off, len)) {
+ return -1;
+ }
+ savePointerState();
+
+ buffer = new byte[bufferSize];
+ // data need to be copied to user buffer from index bCursor,
+ // AbfsInutStream buffer is going p contain dta from footer start. In
+ // that case bCursor will be set to footerStart - fCursor
+ long lastBlockStart = max(0, contentLength - bufferSize);
+ bCursor = (int) (fCursor - lastBlockStart);
+ // read API call is considered 1 single operation in reality server could
+ // return partial data and client has to retry untill the last full block
+ // is read. So setting the fCursorAfterLastRead before the possible
+ // multiple server calls
+ fCursorAfterLastRead = fCursor;
+ // 0 if contentlength is < buffersize
+ long actualLenToRead = min(bufferSize, contentLength);
+ return optimisedRead(b, off, len, lastBlockStart, actualLenToRead);
+ }
+
+ private int optimisedRead(final byte[] b, final int off, final int len,
+ final long readFrom, final long actualLen) throws IOException {
+ int totalBytesRead = 0;
+ fCursor = readFrom;
+ try {
+ for (int i = 0;
+ i < OPTIMIZATION_ATTEMPTS && fCursor < contentLength; i++) {
+ int bytesRead = readInternal(fCursor, buffer, limit,
+ (int) actualLen - limit, true);
+ if (bytesRead > 0) {
+ totalBytesRead += bytesRead;
+ limit += bytesRead;
+ fCursor += bytesRead;
+ }
+ }
+ } catch (IOException e) {
+ LOG.debug("Optimized read failed. alling back to readOneBlock {}", e);
Review comment:
Please check again. Change not reflecting.
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsInputStream.java
##########
@@ -206,11 +231,127 @@ private int readOneBlock(final byte[] b, final int off, final int len) throws IO
fCursor += bytesRead;
fCursorAfterLastRead = fCursor;
}
+ return copyToUserBuffer(b, off, len);
+ }
+ private int readFileCompletely(final byte[] b, final int off, final int len)
+ throws IOException {
+ if (len == 0) {
+ return 0;
+ }
+ if (!validate(b, off, len)) {
+ return -1;
+ }
+ savePointerState();
+
+ buffer = new byte[bufferSize];
+ // data need to be copied to user buffer from index bCursor, bCursor has
+ // to be the current fCusor
+ bCursor = (int) fCursor;
+ return optimisedRead(b, off, len, 0, contentLength);
+ }
+
+ private int readLastBlock(final byte[] b, final int off, final int len)
+ throws IOException {
+ if (len == 0) {
+ return 0;
+ }
+ if (!validate(b, off, len)) {
+ return -1;
+ }
+ savePointerState();
+
+ buffer = new byte[bufferSize];
+ // data need to be copied to user buffer from index bCursor,
+ // AbfsInutStream buffer is going p contain dta from footer start. In
+ // that case bCursor will be set to footerStart - fCursor
+ long lastBlockStart = max(0, contentLength - bufferSize);
+ bCursor = (int) (fCursor - lastBlockStart);
+ // read API call is considered 1 single operation in reality server could
+ // return partial data and client has to retry untill the last full block
+ // is read. So setting the fCursorAfterLastRead before the possible
+ // multiple server calls
+ fCursorAfterLastRead = fCursor;
+ // 0 if contentlength is < buffersize
+ long actualLenToRead = min(bufferSize, contentLength);
+ return optimisedRead(b, off, len, lastBlockStart, actualLenToRead);
+ }
+
+ private int optimisedRead(final byte[] b, final int off, final int len,
+ final long readFrom, final long actualLen) throws IOException {
+ int totalBytesRead = 0;
+ fCursor = readFrom;
+ try {
+ for (int i = 0;
+ i < OPTIMIZATION_ATTEMPTS && fCursor < contentLength; i++) {
Review comment:
Please check again. Change not reflecting.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] bilaharith commented on a change in pull request #2464: HADOOP-17347. ABFS: Read optimizations
Posted by GitBox <gi...@apache.org>.
bilaharith commented on a change in pull request #2464:
URL: https://github.com/apache/hadoop/pull/2464#discussion_r544148005
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsInputStream.java
##########
@@ -206,11 +231,127 @@ private int readOneBlock(final byte[] b, final int off, final int len) throws IO
fCursor += bytesRead;
fCursorAfterLastRead = fCursor;
}
+ return copyToUserBuffer(b, off, len);
+ }
+ private int readFileCompletely(final byte[] b, final int off, final int len)
+ throws IOException {
+ if (len == 0) {
+ return 0;
+ }
+ if (!validate(b, off, len)) {
+ return -1;
+ }
+ savePointerState();
+
+ buffer = new byte[bufferSize];
+ // data need to be copied to user buffer from index bCursor, bCursor has
+ // to be the current fCusor
+ bCursor = (int) fCursor;
+ return optimisedRead(b, off, len, 0, contentLength);
+ }
+
+ private int readLastBlock(final byte[] b, final int off, final int len)
+ throws IOException {
+ if (len == 0) {
+ return 0;
+ }
+ if (!validate(b, off, len)) {
+ return -1;
+ }
+ savePointerState();
+
+ buffer = new byte[bufferSize];
+ // data need to be copied to user buffer from index bCursor,
+ // AbfsInutStream buffer is going p contain dta from footer start. In
+ // that case bCursor will be set to footerStart - fCursor
+ long lastBlockStart = max(0, contentLength - bufferSize);
+ bCursor = (int) (fCursor - lastBlockStart);
+ // read API call is considered 1 single operation in reality server could
+ // return partial data and client has to retry untill the last full block
+ // is read. So setting the fCursorAfterLastRead before the possible
+ // multiple server calls
+ fCursorAfterLastRead = fCursor;
+ // 0 if contentlength is < buffersize
+ long actualLenToRead = min(bufferSize, contentLength);
+ return optimisedRead(b, off, len, lastBlockStart, actualLenToRead);
+ }
+
+ private int optimisedRead(final byte[] b, final int off, final int len,
+ final long readFrom, final long actualLen) throws IOException {
+ int totalBytesRead = 0;
+ fCursor = readFrom;
+ try {
+ for (int i = 0;
+ i < OPTIMIZATION_ATTEMPTS && fCursor < contentLength; i++) {
+ int bytesRead = readInternal(fCursor, buffer, limit,
+ (int) actualLen - limit, true);
+ if (bytesRead > 0) {
+ totalBytesRead += bytesRead;
+ limit += bytesRead;
+ fCursor += bytesRead;
+ }
Review comment:
Done
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] snvijaya commented on a change in pull request #2464: Draft PR for read optimizations
Posted by GitBox <gi...@apache.org>.
snvijaya commented on a change in pull request #2464:
URL: https://github.com/apache/hadoop/pull/2464#discussion_r531364075
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AzureBlobFileSystem.java
##########
@@ -1242,7 +1242,7 @@ boolean getIsNamespaceEnabled() throws AzureBlobFileSystemException {
}
@VisibleForTesting
- Map<String, Long> getInstrumentationMap() {
+ public Map<String, Long> getInstrumentationMap() {
Review comment:
Same as above.
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsInputStream.java
##########
@@ -47,6 +47,10 @@
StreamCapabilities {
private static final Logger LOG = LoggerFactory.getLogger(AbfsInputStream.class);
+ public static final int FOOTER_SIZE = 8;
+
Review comment:
static finals together, non-static together .. no need for new lines above and below for every new field
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsInputStream.java
##########
@@ -211,6 +235,59 @@ private int readOneBlock(final byte[] b, final int off, final int len) throws IO
return bytesToRead;
}
+ private boolean shouldReadFully() {
+ return this.firstRead && this.context.readSmallFilesCompletely()
+ && this.contentLength <= this.bufferSize;
+ }
+
+ private boolean shouldReadLastBlock(int len) {
+ return this.firstRead && this.context.optimizeFooterRead()
+ && len == FOOTER_SIZE
+ && this.fCursor == this.contentLength - FOOTER_SIZE;
+ }
+
+ private int readFileCompletely(final byte[] b, final int off, final int len) throws IOException {
+ int validation = validate(b, off, len);
+ if (validation < 1) {
Review comment:
Refer to validate method comments
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsInputStream.java
##########
@@ -211,6 +235,59 @@ private int readOneBlock(final byte[] b, final int off, final int len) throws IO
return bytesToRead;
}
+ private boolean shouldReadFully() {
+ return this.firstRead && this.context.readSmallFilesCompletely()
+ && this.contentLength <= this.bufferSize;
+ }
+
+ private boolean shouldReadLastBlock(int len) {
+ return this.firstRead && this.context.optimizeFooterRead()
+ && len == FOOTER_SIZE
+ && this.fCursor == this.contentLength - FOOTER_SIZE;
+ }
+
+ private int readFileCompletely(final byte[] b, final int off, final int len) throws IOException {
+ int validation = validate(b, off, len);
+ if (validation < 1) {
+ return validation;
+ }
+
+ buffer = new byte[bufferSize];
+ bCursor = (int) fCursor;
+ int bytesRead = readInternal(0, buffer, 0, (int) contentLength, true);
+ firstRead = false;
+ if (bytesRead == -1) {
+ return -1;
+ }
+ fCursorAfterLastRead = fCursor;
+ limit = bytesRead;
+ fCursor = bytesRead;
+ return copyToUserBuffer(b, off, len);
+ }
+
+ private int readLastBlock(final byte[] b, final int off, final int len) throws IOException {
+ int validation = validate(b, off, len);
+ if (validation < 1) {
+ return validation;
+ }
+
+ buffer = new byte[bufferSize];
+ bCursor = (int) (((contentLength < bufferSize) ? contentLength : bufferSize)
+ - FOOTER_SIZE);
+ long lastBlockStartPos = (contentLength < bufferSize)
+ ? 0
+ : contentLength - bufferSize;
+ int bytesRead = readInternal(lastBlockStartPos, buffer, 0, bufferSize,
Review comment:
So readInternal will go directly to server to fetch the data and server returning partial data is a valid case. Expectation is that the client will loop over and continue calling read until it has fetched all the data it needs.
Take a case of client requesting last 8 bytes of a file which here will translate to a read of last 4 MB. ReadInternal returned 3MB data, and so we havent got the last 8 bytes that needs to be returned to client app. bCursor on the other hand has already been set to 4MB - 8 bytes location. This will end up returning what ever is present in the allocated buffer which is corrupt data.
bytesRead has to be of bufferSize else the whole logic falls apart. As server can return partial data, respective handling is needed.
##########
File path: hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/services/ITestAbfsInputStreamReadFooter.java
##########
@@ -0,0 +1,199 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.azurebfs.services;
+
+import java.io.IOException;
+import java.util.Map;
+import java.util.Random;
+
+import org.junit.Test;
+
+import org.apache.hadoop.fs.FSDataInputStream;
+import org.apache.hadoop.fs.FSDataOutputStream;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.fs.azurebfs.AbfsConfiguration;
+import org.apache.hadoop.fs.azurebfs.AbstractAbfsIntegrationTest;
+import org.apache.hadoop.fs.azurebfs.AzureBlobFileSystem;
+
+import static org.apache.hadoop.fs.azurebfs.AbfsStatistic.CONNECTIONS_MADE;
+import static org.apache.hadoop.fs.azurebfs.constants.FileSystemConfigurations.ONE_KB;
+import static org.apache.hadoop.fs.azurebfs.constants.FileSystemConfigurations.ONE_MB;
+
+public class ITestAbfsInputStreamReadFooter
+ extends AbstractAbfsIntegrationTest {
+
+ private static final int TEN = 10;
+ private static final int TWENTY = 20;
+
+ public ITestAbfsInputStreamReadFooter() throws Exception {
+ }
+
+ @Test
+ public void testOnlyOneServerCallIsMadeWhenTheConfIsTrue() throws Exception {
+ testNumBackendCalls(true);
+ }
+
+ @Test
+ public void testMultipleServerCallsAreMadeWhenTheConfIsFalse()
+ throws Exception {
+ testNumBackendCalls(false);
+ }
+
+ private void testNumBackendCalls(boolean optimizeFooterRead)
+ throws Exception {
+ final AzureBlobFileSystem fs = getFileSystem(optimizeFooterRead);
+ for (int i = 1; i <= 4; i++) {
+ String fileName = methodName.getMethodName() + i;
+ int fileSize = i * ONE_MB;
+ byte[] fileContent = getRandomBytesArray(fileSize);
+ Path testFilePath = createFileWithContent(fs, fileName, fileContent);
+ int length = AbfsInputStream.FOOTER_SIZE;
+ try (FSDataInputStream iStream = fs.open(testFilePath)) {
+ byte[] buffer = new byte[length];
+
+ Map<String, Long> metricMap = fs.getInstrumentationMap();
+ long requestsMadeBeforeTest = metricMap
+ .get(CONNECTIONS_MADE.getStatName());
+
+ iStream.seek(fileSize - 8);
+ iStream.read(buffer, 0, length);
+
+ iStream.seek(fileSize - (TEN * ONE_KB));
+ iStream.read(buffer, 0, length);
+
+ iStream.seek(fileSize - (TWENTY * ONE_KB));
+ iStream.read(buffer, 0, length);
+
+ metricMap = fs.getInstrumentationMap();
+ long requestsMadeAfterTest = metricMap
+ .get(CONNECTIONS_MADE.getStatName());
+
+ if (optimizeFooterRead) {
+ assertEquals(1, requestsMadeAfterTest - requestsMadeBeforeTest);
+ } else {
+ assertEquals(3, requestsMadeAfterTest - requestsMadeBeforeTest);
+ }
+ }
+ }
+ }
+
+ @Test
+ public void testSeekToEndAndReadWithConfTrue() throws Exception {
+ testSeekToEndAndReadWithConf(true);
+ }
+
+ @Test
+ public void testSeekToEndAndReadWithConfFalse() throws Exception {
+ testSeekToEndAndReadWithConf(false);
+ }
+
+ private void testSeekToEndAndReadWithConf(boolean optimizeFooterRead) throws Exception {
+ final AzureBlobFileSystem fs = getFileSystem(optimizeFooterRead);
+ for (int i = 5; i <= 10; i++) {
+ String fileName = methodName.getMethodName() + i;
+ int fileSize = i * ONE_MB;
+ byte[] fileContent = getRandomBytesArray(fileSize);
+ Path testFilePath = createFileWithContent(fs, fileName, fileContent);
+ seekReadAndTest(fs, testFilePath, fileSize - AbfsInputStream.FOOTER_SIZE,
+ AbfsInputStream.FOOTER_SIZE, fileContent);
+ }
+ }
+
+ private AzureBlobFileSystem getFileSystem(boolean optimizeFooterRead)
+ throws IOException {
+ final AzureBlobFileSystem fs = getFileSystem();
+ fs.getAbfsStore().getAbfsConfiguration()
+ .setOptimizeFooterRead(optimizeFooterRead);
+ fs.getAbfsStore().getAbfsConfiguration()
+ .setReadSmallFilesCompletely(false);
Review comment:
This shouldnt be enforced by config. As mentioned in earlier comments, align buffer size and file size for it to become eligible for footer read optimization.
We need the test to validate that it goes to footer checks even when the other config is on.
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AzureBlobFileSystem.java
##########
@@ -1218,7 +1218,7 @@ public boolean failed() {
}
@VisibleForTesting
- AzureBlobFileSystemStore getAbfsStore() {
+ public AzureBlobFileSystemStore getAbfsStore() {
Review comment:
Making methods public for test purposes is not a good idea, esp. for AzureBlobFileSystem class. Find alternative.
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsInputStream.java
##########
@@ -161,6 +174,14 @@ private int readOneBlock(final byte[] b, final int off, final int len) throws IO
if (off < 0 || len < 0 || len > b.length - off) {
Review comment:
Validation for read beyond EOF - fCursor > contentLength - can also be added here
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsInputStream.java
##########
@@ -161,6 +174,14 @@ private int readOneBlock(final byte[] b, final int off, final int len) throws IO
if (off < 0 || len < 0 || len > b.length - off) {
throw new IndexOutOfBoundsException();
}
+ return 1; // 1 indicate success
+ }
+
+ private int readOneBlock(final byte[] b, final int off, final int len) throws IOException {
+ int validation = validate(b, off, len);
+ if (validation < 1) {
+ return validation;
Review comment:
validate should return a boolean , true if validation is successful and false if not.
For cases it needs to throw exception is already present, so the if check wont be hit. Change to:
if (!validate(b, off, len)) { return -1; }
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsInputStream.java
##########
@@ -141,7 +154,7 @@ public synchronized int read(final byte[] b, final int off, final int len) throw
return totalReadBytes > 0 ? totalReadBytes : lastReadBytes;
}
- private int readOneBlock(final byte[] b, final int off, final int len) throws IOException {
+ private int validate(byte[] b, int off, int len) throws IOException {
Review comment:
Why does validate return an int ? There is a return from what validate does in below method, even if its for error case, its a wrong case.
readOneBlock is supposed to written size of the data read, validate as such does no data read.
Change to return boolean true for success here.
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsInputStream.java
##########
@@ -211,6 +235,59 @@ private int readOneBlock(final byte[] b, final int off, final int len) throws IO
return bytesToRead;
}
+ private boolean shouldReadFully() {
+ return this.firstRead && this.context.readSmallFilesCompletely()
+ && this.contentLength <= this.bufferSize;
+ }
+
+ private boolean shouldReadLastBlock(int len) {
+ return this.firstRead && this.context.optimizeFooterRead()
+ && len == FOOTER_SIZE
+ && this.fCursor == this.contentLength - FOOTER_SIZE;
+ }
+
+ private int readFileCompletely(final byte[] b, final int off, final int len) throws IOException {
+ int validation = validate(b, off, len);
+ if (validation < 1) {
+ return validation;
+ }
+
+ buffer = new byte[bufferSize];
+ bCursor = (int) fCursor;
+ int bytesRead = readInternal(0, buffer, 0, (int) contentLength, true);
+ firstRead = false;
Review comment:
If read failed due to some reason, say throttling. Will this line still be hit ? Is there a scenario to handle if it doesnt ?
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsInputStream.java
##########
@@ -211,6 +235,59 @@ private int readOneBlock(final byte[] b, final int off, final int len) throws IO
return bytesToRead;
}
+ private boolean shouldReadFully() {
+ return this.firstRead && this.context.readSmallFilesCompletely()
+ && this.contentLength <= this.bufferSize;
+ }
+
+ private boolean shouldReadLastBlock(int len) {
+ return this.firstRead && this.context.optimizeFooterRead()
+ && len == FOOTER_SIZE
+ && this.fCursor == this.contentLength - FOOTER_SIZE;
+ }
+
+ private int readFileCompletely(final byte[] b, final int off, final int len) throws IOException {
+ int validation = validate(b, off, len);
+ if (validation < 1) {
+ return validation;
+ }
+
+ buffer = new byte[bufferSize];
+ bCursor = (int) fCursor;
+ int bytesRead = readInternal(0, buffer, 0, (int) contentLength, true);
+ firstRead = false;
+ if (bytesRead == -1) {
+ return -1;
+ }
+ fCursorAfterLastRead = fCursor;
+ limit = bytesRead;
+ fCursor = bytesRead;
+ return copyToUserBuffer(b, off, len);
+ }
+
+ private int readLastBlock(final byte[] b, final int off, final int len) throws IOException {
+ int validation = validate(b, off, len);
+ if (validation < 1) {
Review comment:
Refer to comments in validate.
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsInputStream.java
##########
@@ -211,6 +235,59 @@ private int readOneBlock(final byte[] b, final int off, final int len) throws IO
return bytesToRead;
}
+ private boolean shouldReadFully() {
+ return this.firstRead && this.context.readSmallFilesCompletely()
+ && this.contentLength <= this.bufferSize;
+ }
+
+ private boolean shouldReadLastBlock(int len) {
+ return this.firstRead && this.context.optimizeFooterRead()
+ && len == FOOTER_SIZE
+ && this.fCursor == this.contentLength - FOOTER_SIZE;
+ }
+
+ private int readFileCompletely(final byte[] b, final int off, final int len) throws IOException {
+ int validation = validate(b, off, len);
+ if (validation < 1) {
+ return validation;
+ }
+
+ buffer = new byte[bufferSize];
+ bCursor = (int) fCursor;
+ int bytesRead = readInternal(0, buffer, 0, (int) contentLength, true);
+ firstRead = false;
+ if (bytesRead == -1) {
+ return -1;
+ }
+ fCursorAfterLastRead = fCursor;
+ limit = bytesRead;
+ fCursor = bytesRead;
+ return copyToUserBuffer(b, off, len);
+ }
+
+ private int readLastBlock(final byte[] b, final int off, final int len) throws IOException {
+ int validation = validate(b, off, len);
+ if (validation < 1) {
+ return validation;
+ }
+
+ buffer = new byte[bufferSize];
+ bCursor = (int) (((contentLength < bufferSize) ? contentLength : bufferSize)
+ - FOOTER_SIZE);
+ long lastBlockStartPos = (contentLength < bufferSize)
+ ? 0
+ : contentLength - bufferSize;
+ int bytesRead = readInternal(lastBlockStartPos, buffer, 0, bufferSize,
+ true);
Review comment:
In both the new handling, we are bypassing readAheads.
(Unless readAhead is disabled) if read.request.size is configured to 100MB, data read from store would still be in 4MB chunks as that is the readAhead buffer size.
If any of these logics determines that more data from an earlier offset needs to be read, it is triggering reads by bypassing readAheads. So wont it end up in a 100MB direct read to store ?
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsInputStream.java
##########
@@ -211,6 +235,59 @@ private int readOneBlock(final byte[] b, final int off, final int len) throws IO
return bytesToRead;
}
+ private boolean shouldReadFully() {
+ return this.firstRead && this.context.readSmallFilesCompletely()
+ && this.contentLength <= this.bufferSize;
+ }
+
+ private boolean shouldReadLastBlock(int len) {
+ return this.firstRead && this.context.optimizeFooterRead()
+ && len == FOOTER_SIZE
+ && this.fCursor == this.contentLength - FOOTER_SIZE;
+ }
+
+ private int readFileCompletely(final byte[] b, final int off, final int len) throws IOException {
+ int validation = validate(b, off, len);
+ if (validation < 1) {
+ return validation;
+ }
+
+ buffer = new byte[bufferSize];
+ bCursor = (int) fCursor;
+ int bytesRead = readInternal(0, buffer, 0, (int) contentLength, true);
+ firstRead = false;
+ if (bytesRead == -1) {
+ return -1;
+ }
+ fCursorAfterLastRead = fCursor;
+ limit = bytesRead;
+ fCursor = bytesRead;
+ return copyToUserBuffer(b, off, len);
+ }
+
+ private int readLastBlock(final byte[] b, final int off, final int len) throws IOException {
+ int validation = validate(b, off, len);
+ if (validation < 1) {
+ return validation;
+ }
+
+ buffer = new byte[bufferSize];
+ bCursor = (int) (((contentLength < bufferSize) ? contentLength : bufferSize)
+ - FOOTER_SIZE);
+ long lastBlockStartPos = (contentLength < bufferSize)
+ ? 0
+ : contentLength - bufferSize;
+ int bytesRead = readInternal(lastBlockStartPos, buffer, 0, bufferSize,
+ true);
+ firstRead = false;
+ if (bytesRead == -1) {
+ return -1;
+ }
+ fCursorAfterLastRead = fCursor;
+ limit = bytesRead;
+ fCursor = lastBlockStartPos + bytesRead;
Review comment:
Shouldnt fCursorAfterLastRead be set after this line ?
##########
File path: hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/services/ITestAbfsInputStreamReadFooter.java
##########
@@ -0,0 +1,199 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.azurebfs.services;
+
+import java.io.IOException;
+import java.util.Map;
+import java.util.Random;
+
+import org.junit.Test;
+
+import org.apache.hadoop.fs.FSDataInputStream;
+import org.apache.hadoop.fs.FSDataOutputStream;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.fs.azurebfs.AbfsConfiguration;
+import org.apache.hadoop.fs.azurebfs.AbstractAbfsIntegrationTest;
+import org.apache.hadoop.fs.azurebfs.AzureBlobFileSystem;
+
+import static org.apache.hadoop.fs.azurebfs.AbfsStatistic.CONNECTIONS_MADE;
+import static org.apache.hadoop.fs.azurebfs.constants.FileSystemConfigurations.ONE_KB;
+import static org.apache.hadoop.fs.azurebfs.constants.FileSystemConfigurations.ONE_MB;
+
+public class ITestAbfsInputStreamReadFooter
+ extends AbstractAbfsIntegrationTest {
+
+ private static final int TEN = 10;
+ private static final int TWENTY = 20;
+
+ public ITestAbfsInputStreamReadFooter() throws Exception {
+ }
+
+ @Test
+ public void testOnlyOneServerCallIsMadeWhenTheConfIsTrue() throws Exception {
+ testNumBackendCalls(true);
+ }
+
+ @Test
+ public void testMultipleServerCallsAreMadeWhenTheConfIsFalse()
+ throws Exception {
+ testNumBackendCalls(false);
+ }
+
+ private void testNumBackendCalls(boolean optimizeFooterRead)
+ throws Exception {
+ final AzureBlobFileSystem fs = getFileSystem(optimizeFooterRead);
+ for (int i = 1; i <= 4; i++) {
+ String fileName = methodName.getMethodName() + i;
+ int fileSize = i * ONE_MB;
Review comment:
For testing of footer read optimized, better to have a file that is atleast 3 or 4 times buffer size. For testing purposes, filesize of 1 mb is good, set the buffer to a much smaller size over config.
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AbfsConfiguration.java
##########
@@ -517,6 +527,14 @@ public int getWriteBufferSize() {
return this.writeBufferSize;
}
+ public boolean readSmallFilesCompletely() {
Review comment:
I recall it was discussed that small file criteria will be capped at 4Mb file size. I dont see that in the review. Was it scratched back to small file being one less than buffer size ?
##########
File path: hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/services/ITestAbfsInputStreamReadFooter.java
##########
@@ -0,0 +1,199 @@
+/**
Review comment:
Run read tests in particular and contract test on a whole multiple times to check for any random failures.
##########
File path: hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/services/ITestAbfsInputStreamReadFooter.java
##########
@@ -0,0 +1,199 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.azurebfs.services;
+
+import java.io.IOException;
+import java.util.Map;
+import java.util.Random;
+
+import org.junit.Test;
+
+import org.apache.hadoop.fs.FSDataInputStream;
+import org.apache.hadoop.fs.FSDataOutputStream;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.fs.azurebfs.AbfsConfiguration;
+import org.apache.hadoop.fs.azurebfs.AbstractAbfsIntegrationTest;
+import org.apache.hadoop.fs.azurebfs.AzureBlobFileSystem;
+
+import static org.apache.hadoop.fs.azurebfs.AbfsStatistic.CONNECTIONS_MADE;
+import static org.apache.hadoop.fs.azurebfs.constants.FileSystemConfigurations.ONE_KB;
+import static org.apache.hadoop.fs.azurebfs.constants.FileSystemConfigurations.ONE_MB;
+
+public class ITestAbfsInputStreamReadFooter
+ extends AbstractAbfsIntegrationTest {
+
+ private static final int TEN = 10;
+ private static final int TWENTY = 20;
+
+ public ITestAbfsInputStreamReadFooter() throws Exception {
+ }
+
+ @Test
+ public void testOnlyOneServerCallIsMadeWhenTheConfIsTrue() throws Exception {
+ testNumBackendCalls(true);
+ }
+
+ @Test
+ public void testMultipleServerCallsAreMadeWhenTheConfIsFalse()
+ throws Exception {
+ testNumBackendCalls(false);
+ }
+
+ private void testNumBackendCalls(boolean optimizeFooterRead)
+ throws Exception {
+ final AzureBlobFileSystem fs = getFileSystem(optimizeFooterRead);
+ for (int i = 1; i <= 4; i++) {
+ String fileName = methodName.getMethodName() + i;
+ int fileSize = i * ONE_MB;
+ byte[] fileContent = getRandomBytesArray(fileSize);
+ Path testFilePath = createFileWithContent(fs, fileName, fileContent);
+ int length = AbfsInputStream.FOOTER_SIZE;
+ try (FSDataInputStream iStream = fs.open(testFilePath)) {
+ byte[] buffer = new byte[length];
+
+ Map<String, Long> metricMap = fs.getInstrumentationMap();
+ long requestsMadeBeforeTest = metricMap
+ .get(CONNECTIONS_MADE.getStatName());
+
+ iStream.seek(fileSize - 8);
+ iStream.read(buffer, 0, length);
+
+ iStream.seek(fileSize - (TEN * ONE_KB));
+ iStream.read(buffer, 0, length);
+
+ iStream.seek(fileSize - (TWENTY * ONE_KB));
+ iStream.read(buffer, 0, length);
+
+ metricMap = fs.getInstrumentationMap();
+ long requestsMadeAfterTest = metricMap
+ .get(CONNECTIONS_MADE.getStatName());
+
+ if (optimizeFooterRead) {
+ assertEquals(1, requestsMadeAfterTest - requestsMadeBeforeTest);
+ } else {
+ assertEquals(3, requestsMadeAfterTest - requestsMadeBeforeTest);
+ }
+ }
+ }
+ }
+
+ @Test
+ public void testSeekToEndAndReadWithConfTrue() throws Exception {
+ testSeekToEndAndReadWithConf(true);
+ }
+
+ @Test
+ public void testSeekToEndAndReadWithConfFalse() throws Exception {
+ testSeekToEndAndReadWithConf(false);
+ }
+
+ private void testSeekToEndAndReadWithConf(boolean optimizeFooterRead) throws Exception {
+ final AzureBlobFileSystem fs = getFileSystem(optimizeFooterRead);
+ for (int i = 5; i <= 10; i++) {
+ String fileName = methodName.getMethodName() + i;
+ int fileSize = i * ONE_MB;
+ byte[] fileContent = getRandomBytesArray(fileSize);
+ Path testFilePath = createFileWithContent(fs, fileName, fileContent);
+ seekReadAndTest(fs, testFilePath, fileSize - AbfsInputStream.FOOTER_SIZE,
+ AbfsInputStream.FOOTER_SIZE, fileContent);
+ }
+ }
+
+ private AzureBlobFileSystem getFileSystem(boolean optimizeFooterRead)
+ throws IOException {
+ final AzureBlobFileSystem fs = getFileSystem();
+ fs.getAbfsStore().getAbfsConfiguration()
+ .setOptimizeFooterRead(optimizeFooterRead);
+ fs.getAbfsStore().getAbfsConfiguration()
+ .setReadSmallFilesCompletely(false);
+ return fs;
+ }
+
+ private Path createFileWithContent(FileSystem fs, String fileName,
+ byte[] fileContent) throws IOException {
+ Path testFilePath = path(fileName);
+ try (FSDataOutputStream oStream = fs.create(testFilePath)) {
+ oStream.write(fileContent);
+ oStream.flush();
+ }
+ return testFilePath;
+ }
+
+ private void seekReadAndTest(final FileSystem fs, final Path testFilePath,
+ final int seekPos, final int length, final byte[] fileContent) throws IOException {
+ try (FSDataInputStream iStream = fs.open(testFilePath)) {
+ iStream.seek(seekPos);
+ byte[] buffer = new byte[length];
+ iStream.read(buffer, 0, length);
+ assertSuccessfulRead(fileContent, seekPos, length, buffer);
+ AbfsInputStream abfsInputStream = (AbfsInputStream) iStream
+ .getWrappedStream();
+
+ AzureBlobFileSystem abfs = (AzureBlobFileSystem) fs;
+ AbfsConfiguration conf = abfs.getAbfsStore().getAbfsConfiguration();
+
+ int expectedFCursor = fileContent.length;
+ int expectedLimit;
+ int expectedBCursor;
+ if (conf.optimizeFooterRead()) {
+ expectedBCursor = ((conf.getReadBufferSize() < fileContent.length)
+ ? conf.getReadBufferSize()
+ : fileContent.length);
+ expectedLimit = (conf.getReadBufferSize() < fileContent.length)
+ ? conf.getReadBufferSize()
+ : fileContent.length;
+ } else {
+ expectedBCursor = length;
+ expectedLimit = length;
+ }
+ assertSuccessfulRead(fileContent, abfsInputStream.getBuffer(),
+ conf, length);
+ assertEquals(expectedFCursor, abfsInputStream.getFCursor());
+ assertEquals(expectedBCursor, abfsInputStream.getBCursor());
+ assertEquals(expectedLimit, abfsInputStream.getLimit());
+ }
+ }
+
+ private void assertSuccessfulRead(byte[] actualFileContent,
+ byte[] contentRead, AbfsConfiguration conf, int len) {
+ int buffersize = conf.getReadBufferSize();
+ int actualContentSize = actualFileContent.length;
+ if (conf.optimizeFooterRead()) {
+ len = (actualContentSize < buffersize)
+ ? actualContentSize
+ : buffersize;
+ }
+ assertSuccessfulRead(actualFileContent, actualContentSize - len, len,
Review comment:
File content asserts are needed on the data that is returned through the client provided buffer as well.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] vinaysbadami commented on a change in pull request #2464: Draft PR for read optimizations
Posted by GitBox <gi...@apache.org>.
vinaysbadami commented on a change in pull request #2464:
URL: https://github.com/apache/hadoop/pull/2464#discussion_r523917070
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsInputStream.java
##########
@@ -92,6 +99,9 @@ public AbfsInputStream(
this.cachedSasToken = new CachedSASToken(
abfsInputStreamContext.getSasTokenRenewPeriodForStreamsInSeconds());
this.streamStatistics = abfsInputStreamContext.getStreamStatistics();
+ this.readSmallFilesCompletely = abfsInputStreamContext
+ .readSmallFilesCompletely();
Review comment:
we are passed in an AbfsInputStreamContext. We should hold onto that instance instead of creating member variables for each of the members of AbfsInputStreamContext.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] bilaharith commented on a change in pull request #2464: Draft PR for read optimizations
Posted by GitBox <gi...@apache.org>.
bilaharith commented on a change in pull request #2464:
URL: https://github.com/apache/hadoop/pull/2464#discussion_r528631501
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/constants/ConfigurationKeys.java
##########
@@ -56,6 +56,8 @@
public static final String AZURE_WRITE_MAX_REQUESTS_TO_QUEUE = "fs.azure.write.max.requests.to.queue";
public static final String AZURE_WRITE_BUFFER_SIZE = "fs.azure.write.request.size";
public static final String AZURE_READ_BUFFER_SIZE = "fs.azure.read.request.size";
Review comment:
Done
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] bilaharith commented on a change in pull request #2464: Draft PR: HADOOP-17347. ABFS: Read optimizations
Posted by GitBox <gi...@apache.org>.
bilaharith commented on a change in pull request #2464:
URL: https://github.com/apache/hadoop/pull/2464#discussion_r533551588
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsInputStream.java
##########
@@ -141,7 +154,7 @@ public synchronized int read(final byte[] b, final int off, final int len) throw
return totalReadBytes > 0 ? totalReadBytes : lastReadBytes;
}
- private int readOneBlock(final byte[] b, final int off, final int len) throws IOException {
+ private int validate(byte[] b, int off, int len) throws IOException {
Review comment:
That would alter the existing flow. In certain cases it has to return 0 and -1 in some other cases, exceptions in case validations failed for len, off values.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] DadanielZ merged pull request #2464: HADOOP-17347. ABFS: Read optimizations
Posted by GitBox <gi...@apache.org>.
DadanielZ merged pull request #2464:
URL: https://github.com/apache/hadoop/pull/2464
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #2464: HADOOP-17347. ABFS: Read optimizations
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on pull request #2464:
URL: https://github.com/apache/hadoop/pull/2464#issuecomment-742287182
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 30s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | | 0m 0s | [test4tests](test4tests) | The patch appears to include 2 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 32m 50s | | trunk passed |
| +1 :green_heart: | compile | 0m 39s | | trunk passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 |
| +1 :green_heart: | compile | 0m 35s | | trunk passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| +1 :green_heart: | checkstyle | 0m 26s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 39s | | trunk passed |
| +1 :green_heart: | shadedclient | 16m 10s | | branch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 32s | | trunk passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 |
| +1 :green_heart: | javadoc | 0m 28s | | trunk passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| +0 :ok: | spotbugs | 0m 58s | | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 0m 57s | | trunk passed |
| -0 :warning: | patch | 1m 17s | | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 29s | | the patch passed |
| +1 :green_heart: | compile | 0m 29s | | the patch passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 |
| +1 :green_heart: | javac | 0m 29s | | the patch passed |
| +1 :green_heart: | compile | 0m 27s | | the patch passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| +1 :green_heart: | javac | 0m 27s | | the patch passed |
| +1 :green_heart: | checkstyle | 0m 17s | | the patch passed |
| +1 :green_heart: | mvnsite | 0m 29s | | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | | The patch has no whitespace issues. |
| +1 :green_heart: | shadedclient | 14m 41s | | patch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 26s | | the patch passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 |
| +1 :green_heart: | javadoc | 0m 24s | | the patch passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| +1 :green_heart: | findbugs | 0m 58s | | the patch passed |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 1m 31s | | hadoop-azure in the patch passed. |
| +1 :green_heart: | asflicense | 0m 33s | | The patch does not generate ASF License warnings. |
| | | 76m 40s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2464/11/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/2464 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux f78a5be3f96f 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / c2cecfc9b95 |
| Default Java | Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2464/11/testReport/ |
| Max. process+thread count | 699 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2464/11/console |
| versions | git=2.17.1 maven=3.6.0 findbugs=4.0.6 |
| Powered by | Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] bilaharith commented on a change in pull request #2464: HADOOP-17347. ABFS: Read optimizations
Posted by GitBox <gi...@apache.org>.
bilaharith commented on a change in pull request #2464:
URL: https://github.com/apache/hadoop/pull/2464#discussion_r540354018
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsInputStream.java
##########
@@ -224,6 +240,123 @@ private int readOneBlock(final byte[] b, final int off, final int len) throws IO
return bytesToRead;
}
+ private boolean shouldReadFully() {
+ return this.firstRead && this.context.readSmallFilesCompletely()
+ && this.contentLength <= this.bufferSize;
+ }
+
+ private boolean shouldReadLastBlock(int len) {
+ return this.firstRead && this.context.optimizeFooterRead()
+ && len == FOOTER_SIZE
+ && this.fCursor == this.contentLength - FOOTER_SIZE;
+ }
+
+ private int readFileCompletely(final byte[] b, final int off, final int len)
+ throws IOException {
+ if (closed) {
+ throw new IOException(FSExceptionMessages.STREAM_IS_CLOSED);
+ }
+
+ Preconditions.checkNotNull(b);
+ LOG.debug("read one block requested b.length = {} off {} len {}", b.length,
+ off, len);
+
+ if (len == 0) {
+ return 0;
+ }
+
+ if (this.available() == 0) {
+ return -1;
+ }
+
+ if (off < 0 || len < 0 || len > b.length - off) {
+ throw new IndexOutOfBoundsException();
+ }
+
+ buffer = new byte[bufferSize];
+ // data need to be copied to user buffer from index bCursor, bCursor has
+ // to be the current fCusor
+ bCursor = (int) fCursor;
+ fCursorAfterLastRead = fCursor;
+ int totalBytesRead = 0;
+ int loopCount = 0;
+ // Read from begining
+ fCursor = 0;
+ while (fCursor < contentLength) {
+ int bytesRead = readInternal(fCursor, buffer, limit,
+ (int) contentLength - limit, true);
+ if (bytesRead > 0) {
+ totalBytesRead += bytesRead;
+ limit += bytesRead;
+ fCursor += bytesRead;
+ }
+ if (loopCount++ >= 10) {
Review comment:
Made changes the following way
1. The available data is returned
2. Done
3. Done
4. Done
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsInputStream.java
##########
@@ -224,6 +240,123 @@ private int readOneBlock(final byte[] b, final int off, final int len) throws IO
return bytesToRead;
}
+ private boolean shouldReadFully() {
+ return this.firstRead && this.context.readSmallFilesCompletely()
+ && this.contentLength <= this.bufferSize;
+ }
+
+ private boolean shouldReadLastBlock(int len) {
+ return this.firstRead && this.context.optimizeFooterRead()
+ && len == FOOTER_SIZE
+ && this.fCursor == this.contentLength - FOOTER_SIZE;
+ }
+
+ private int readFileCompletely(final byte[] b, final int off, final int len)
+ throws IOException {
+ if (closed) {
+ throw new IOException(FSExceptionMessages.STREAM_IS_CLOSED);
+ }
+
+ Preconditions.checkNotNull(b);
+ LOG.debug("read one block requested b.length = {} off {} len {}", b.length,
+ off, len);
+
+ if (len == 0) {
+ return 0;
+ }
+
+ if (this.available() == 0) {
+ return -1;
+ }
+
+ if (off < 0 || len < 0 || len > b.length - off) {
+ throw new IndexOutOfBoundsException();
+ }
+
+ buffer = new byte[bufferSize];
+ // data need to be copied to user buffer from index bCursor, bCursor has
+ // to be the current fCusor
+ bCursor = (int) fCursor;
+ fCursorAfterLastRead = fCursor;
+ int totalBytesRead = 0;
+ int loopCount = 0;
+ // Read from begining
+ fCursor = 0;
+ while (fCursor < contentLength) {
+ int bytesRead = readInternal(fCursor, buffer, limit,
+ (int) contentLength - limit, true);
+ if (bytesRead > 0) {
+ totalBytesRead += bytesRead;
+ limit += bytesRead;
+ fCursor += bytesRead;
+ }
+ if (loopCount++ >= 10) {
+ throw new IOException(
+ "Too many attempts in reading whole file " + path);
+ }
+ }
+ firstRead = false;
+ if (totalBytesRead == -1) {
+ return -1;
+ }
+ return copyToUserBuffer(b, off, len);
+ }
+
+ private int readLastBlock(final byte[] b, final int off, final int len)
+ throws IOException {
+ if (closed) {
+ throw new IOException(FSExceptionMessages.STREAM_IS_CLOSED);
+ }
+
+ Preconditions.checkNotNull(b);
+ LOG.debug("read one block requested b.length = {} off {} len {}", b.length,
+ off, len);
+
+ if (len == 0) {
+ return 0;
+ }
+
+ if (this.available() == 0) {
+ return -1;
+ }
+
+ if (off < 0 || len < 0 || len > b.length - off) {
+ throw new IndexOutOfBoundsException();
+ }
+
+ buffer = new byte[bufferSize];
+ // data need to be copied to user buffer from index bCursor, for small
+ // files the bCursor will be contentlength - footer size,
+ // otherwise buffersize - footer size
+ bCursor = (int) (Math.min(contentLength, bufferSize) - FOOTER_SIZE);
+ // read API call is considered 1 single operation in reality server could
+ // return partial data and client has to retry untill the last full block
+ // is read. So setting the fCursorAfterLastRead before the possible
+ // multiple server calls
+ fCursorAfterLastRead = fCursor;
+ // 0 if contentlength is < buffersize
+ fCursor = Math.max(0, contentLength - bufferSize);
+ int totalBytesRead = 0;
+ int loopCount = 0;
+ while (fCursor < contentLength) {
+ int bytesRead = readInternal(fCursor, buffer, limit, bufferSize - limit,
+ true);
+ if (bytesRead > 0) {
+ totalBytesRead += bytesRead;
+ limit += bytesRead;
+ fCursor += bytesRead;
+ }
+ if (loopCount++ >= 10) {
Review comment:
Done
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] bilaharith commented on a change in pull request #2464: Draft PR: HADOOP-17347. ABFS: Read optimizations
Posted by GitBox <gi...@apache.org>.
bilaharith commented on a change in pull request #2464:
URL: https://github.com/apache/hadoop/pull/2464#discussion_r533560221
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AzureBlobFileSystem.java
##########
@@ -1218,7 +1218,7 @@ public boolean failed() {
}
@VisibleForTesting
- AzureBlobFileSystemStore getAbfsStore() {
+ public AzureBlobFileSystemStore getAbfsStore() {
Review comment:
Done
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AzureBlobFileSystem.java
##########
@@ -1242,7 +1242,7 @@ boolean getIsNamespaceEnabled() throws AzureBlobFileSystemException {
}
@VisibleForTesting
- Map<String, Long> getInstrumentationMap() {
+ public Map<String, Long> getInstrumentationMap() {
Review comment:
Done
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsInputStream.java
##########
@@ -47,6 +47,10 @@
StreamCapabilities {
private static final Logger LOG = LoggerFactory.getLogger(AbfsInputStream.class);
+ public static final int FOOTER_SIZE = 8;
+
Review comment:
Done
##########
File path: hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/services/ITestAbfsInputStreamReadFooter.java
##########
@@ -0,0 +1,199 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.azurebfs.services;
+
+import java.io.IOException;
+import java.util.Map;
+import java.util.Random;
+
+import org.junit.Test;
+
+import org.apache.hadoop.fs.FSDataInputStream;
+import org.apache.hadoop.fs.FSDataOutputStream;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.fs.azurebfs.AbfsConfiguration;
+import org.apache.hadoop.fs.azurebfs.AbstractAbfsIntegrationTest;
+import org.apache.hadoop.fs.azurebfs.AzureBlobFileSystem;
+
+import static org.apache.hadoop.fs.azurebfs.AbfsStatistic.CONNECTIONS_MADE;
+import static org.apache.hadoop.fs.azurebfs.constants.FileSystemConfigurations.ONE_KB;
+import static org.apache.hadoop.fs.azurebfs.constants.FileSystemConfigurations.ONE_MB;
+
+public class ITestAbfsInputStreamReadFooter
+ extends AbstractAbfsIntegrationTest {
+
+ private static final int TEN = 10;
+ private static final int TWENTY = 20;
+
+ public ITestAbfsInputStreamReadFooter() throws Exception {
+ }
+
+ @Test
+ public void testOnlyOneServerCallIsMadeWhenTheConfIsTrue() throws Exception {
+ testNumBackendCalls(true);
+ }
+
+ @Test
+ public void testMultipleServerCallsAreMadeWhenTheConfIsFalse()
+ throws Exception {
+ testNumBackendCalls(false);
+ }
+
+ private void testNumBackendCalls(boolean optimizeFooterRead)
+ throws Exception {
+ final AzureBlobFileSystem fs = getFileSystem(optimizeFooterRead);
+ for (int i = 1; i <= 4; i++) {
+ String fileName = methodName.getMethodName() + i;
+ int fileSize = i * ONE_MB;
+ byte[] fileContent = getRandomBytesArray(fileSize);
+ Path testFilePath = createFileWithContent(fs, fileName, fileContent);
+ int length = AbfsInputStream.FOOTER_SIZE;
+ try (FSDataInputStream iStream = fs.open(testFilePath)) {
+ byte[] buffer = new byte[length];
+
+ Map<String, Long> metricMap = fs.getInstrumentationMap();
+ long requestsMadeBeforeTest = metricMap
+ .get(CONNECTIONS_MADE.getStatName());
+
+ iStream.seek(fileSize - 8);
+ iStream.read(buffer, 0, length);
+
+ iStream.seek(fileSize - (TEN * ONE_KB));
+ iStream.read(buffer, 0, length);
+
+ iStream.seek(fileSize - (TWENTY * ONE_KB));
+ iStream.read(buffer, 0, length);
+
+ metricMap = fs.getInstrumentationMap();
+ long requestsMadeAfterTest = metricMap
+ .get(CONNECTIONS_MADE.getStatName());
+
+ if (optimizeFooterRead) {
+ assertEquals(1, requestsMadeAfterTest - requestsMadeBeforeTest);
+ } else {
+ assertEquals(3, requestsMadeAfterTest - requestsMadeBeforeTest);
+ }
+ }
+ }
+ }
+
+ @Test
+ public void testSeekToEndAndReadWithConfTrue() throws Exception {
+ testSeekToEndAndReadWithConf(true);
+ }
+
+ @Test
+ public void testSeekToEndAndReadWithConfFalse() throws Exception {
+ testSeekToEndAndReadWithConf(false);
+ }
+
+ private void testSeekToEndAndReadWithConf(boolean optimizeFooterRead) throws Exception {
+ final AzureBlobFileSystem fs = getFileSystem(optimizeFooterRead);
+ for (int i = 5; i <= 10; i++) {
+ String fileName = methodName.getMethodName() + i;
+ int fileSize = i * ONE_MB;
+ byte[] fileContent = getRandomBytesArray(fileSize);
+ Path testFilePath = createFileWithContent(fs, fileName, fileContent);
+ seekReadAndTest(fs, testFilePath, fileSize - AbfsInputStream.FOOTER_SIZE,
+ AbfsInputStream.FOOTER_SIZE, fileContent);
+ }
+ }
+
+ private AzureBlobFileSystem getFileSystem(boolean optimizeFooterRead)
+ throws IOException {
+ final AzureBlobFileSystem fs = getFileSystem();
+ fs.getAbfsStore().getAbfsConfiguration()
+ .setOptimizeFooterRead(optimizeFooterRead);
+ fs.getAbfsStore().getAbfsConfiguration()
+ .setReadSmallFilesCompletely(false);
Review comment:
Done
##########
File path: hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/services/ITestAbfsInputStreamReadFooter.java
##########
@@ -0,0 +1,199 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.azurebfs.services;
+
+import java.io.IOException;
+import java.util.Map;
+import java.util.Random;
+
+import org.junit.Test;
+
+import org.apache.hadoop.fs.FSDataInputStream;
+import org.apache.hadoop.fs.FSDataOutputStream;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.fs.azurebfs.AbfsConfiguration;
+import org.apache.hadoop.fs.azurebfs.AbstractAbfsIntegrationTest;
+import org.apache.hadoop.fs.azurebfs.AzureBlobFileSystem;
+
+import static org.apache.hadoop.fs.azurebfs.AbfsStatistic.CONNECTIONS_MADE;
+import static org.apache.hadoop.fs.azurebfs.constants.FileSystemConfigurations.ONE_KB;
+import static org.apache.hadoop.fs.azurebfs.constants.FileSystemConfigurations.ONE_MB;
+
+public class ITestAbfsInputStreamReadFooter
+ extends AbstractAbfsIntegrationTest {
+
+ private static final int TEN = 10;
+ private static final int TWENTY = 20;
+
+ public ITestAbfsInputStreamReadFooter() throws Exception {
+ }
+
+ @Test
+ public void testOnlyOneServerCallIsMadeWhenTheConfIsTrue() throws Exception {
+ testNumBackendCalls(true);
+ }
+
+ @Test
+ public void testMultipleServerCallsAreMadeWhenTheConfIsFalse()
+ throws Exception {
+ testNumBackendCalls(false);
+ }
+
+ private void testNumBackendCalls(boolean optimizeFooterRead)
+ throws Exception {
+ final AzureBlobFileSystem fs = getFileSystem(optimizeFooterRead);
+ for (int i = 1; i <= 4; i++) {
+ String fileName = methodName.getMethodName() + i;
+ int fileSize = i * ONE_MB;
+ byte[] fileContent = getRandomBytesArray(fileSize);
+ Path testFilePath = createFileWithContent(fs, fileName, fileContent);
+ int length = AbfsInputStream.FOOTER_SIZE;
+ try (FSDataInputStream iStream = fs.open(testFilePath)) {
+ byte[] buffer = new byte[length];
+
+ Map<String, Long> metricMap = fs.getInstrumentationMap();
+ long requestsMadeBeforeTest = metricMap
+ .get(CONNECTIONS_MADE.getStatName());
+
+ iStream.seek(fileSize - 8);
+ iStream.read(buffer, 0, length);
+
+ iStream.seek(fileSize - (TEN * ONE_KB));
+ iStream.read(buffer, 0, length);
+
+ iStream.seek(fileSize - (TWENTY * ONE_KB));
+ iStream.read(buffer, 0, length);
+
+ metricMap = fs.getInstrumentationMap();
+ long requestsMadeAfterTest = metricMap
+ .get(CONNECTIONS_MADE.getStatName());
+
+ if (optimizeFooterRead) {
+ assertEquals(1, requestsMadeAfterTest - requestsMadeBeforeTest);
+ } else {
+ assertEquals(3, requestsMadeAfterTest - requestsMadeBeforeTest);
+ }
+ }
+ }
+ }
+
+ @Test
+ public void testSeekToEndAndReadWithConfTrue() throws Exception {
+ testSeekToEndAndReadWithConf(true);
+ }
+
+ @Test
+ public void testSeekToEndAndReadWithConfFalse() throws Exception {
+ testSeekToEndAndReadWithConf(false);
+ }
+
+ private void testSeekToEndAndReadWithConf(boolean optimizeFooterRead) throws Exception {
+ final AzureBlobFileSystem fs = getFileSystem(optimizeFooterRead);
+ for (int i = 5; i <= 10; i++) {
+ String fileName = methodName.getMethodName() + i;
+ int fileSize = i * ONE_MB;
+ byte[] fileContent = getRandomBytesArray(fileSize);
+ Path testFilePath = createFileWithContent(fs, fileName, fileContent);
+ seekReadAndTest(fs, testFilePath, fileSize - AbfsInputStream.FOOTER_SIZE,
+ AbfsInputStream.FOOTER_SIZE, fileContent);
+ }
+ }
+
+ private AzureBlobFileSystem getFileSystem(boolean optimizeFooterRead)
+ throws IOException {
+ final AzureBlobFileSystem fs = getFileSystem();
+ fs.getAbfsStore().getAbfsConfiguration()
+ .setOptimizeFooterRead(optimizeFooterRead);
+ fs.getAbfsStore().getAbfsConfiguration()
+ .setReadSmallFilesCompletely(false);
+ return fs;
+ }
+
+ private Path createFileWithContent(FileSystem fs, String fileName,
+ byte[] fileContent) throws IOException {
+ Path testFilePath = path(fileName);
+ try (FSDataOutputStream oStream = fs.create(testFilePath)) {
+ oStream.write(fileContent);
+ oStream.flush();
+ }
+ return testFilePath;
+ }
+
+ private void seekReadAndTest(final FileSystem fs, final Path testFilePath,
+ final int seekPos, final int length, final byte[] fileContent) throws IOException {
+ try (FSDataInputStream iStream = fs.open(testFilePath)) {
+ iStream.seek(seekPos);
+ byte[] buffer = new byte[length];
+ iStream.read(buffer, 0, length);
+ assertSuccessfulRead(fileContent, seekPos, length, buffer);
+ AbfsInputStream abfsInputStream = (AbfsInputStream) iStream
+ .getWrappedStream();
+
+ AzureBlobFileSystem abfs = (AzureBlobFileSystem) fs;
+ AbfsConfiguration conf = abfs.getAbfsStore().getAbfsConfiguration();
+
+ int expectedFCursor = fileContent.length;
+ int expectedLimit;
+ int expectedBCursor;
+ if (conf.optimizeFooterRead()) {
+ expectedBCursor = ((conf.getReadBufferSize() < fileContent.length)
+ ? conf.getReadBufferSize()
+ : fileContent.length);
+ expectedLimit = (conf.getReadBufferSize() < fileContent.length)
+ ? conf.getReadBufferSize()
+ : fileContent.length;
+ } else {
+ expectedBCursor = length;
+ expectedLimit = length;
+ }
+ assertSuccessfulRead(fileContent, abfsInputStream.getBuffer(),
+ conf, length);
+ assertEquals(expectedFCursor, abfsInputStream.getFCursor());
+ assertEquals(expectedBCursor, abfsInputStream.getBCursor());
+ assertEquals(expectedLimit, abfsInputStream.getLimit());
+ }
+ }
+
+ private void assertSuccessfulRead(byte[] actualFileContent,
+ byte[] contentRead, AbfsConfiguration conf, int len) {
+ int buffersize = conf.getReadBufferSize();
+ int actualContentSize = actualFileContent.length;
+ if (conf.optimizeFooterRead()) {
+ len = (actualContentSize < buffersize)
+ ? actualContentSize
+ : buffersize;
+ }
+ assertSuccessfulRead(actualFileContent, actualContentSize - len, len,
Review comment:
Done
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] bilaharith commented on a change in pull request #2464: Draft PR for read optimizations
Posted by GitBox <gi...@apache.org>.
bilaharith commented on a change in pull request #2464:
URL: https://github.com/apache/hadoop/pull/2464#discussion_r526588179
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsInputStream.java
##########
@@ -162,14 +172,86 @@ private int readOneBlock(final byte[] b, final int off, final int len) throws IO
throw new IndexOutOfBoundsException();
}
- //If buffer is empty, then fill the buffer.
- if (bCursor == limit) {
+ if (firstRead && this.readSmallFilesCompletely
+ && contentLength <= bufferSize) { // Read small files completely
+ if (readFileCompletely() == -1) {
+ return -1;
+ }
+ } else if (firstRead && this.optimizeFooterRead && fCursor
+ == contentLength - FOOTER_DELTA) { // Read the last one block if the
Review comment:
Done
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus removed a comment on pull request #2464: HADOOP-17347. ABFS: Read optimizations
Posted by GitBox <gi...@apache.org>.
hadoop-yetus removed a comment on pull request #2464:
URL: https://github.com/apache/hadoop/pull/2464#issuecomment-746214660
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] bilaharith commented on a change in pull request #2464: HADOOP-17347. ABFS: Read optimizations
Posted by GitBox <gi...@apache.org>.
bilaharith commented on a change in pull request #2464:
URL: https://github.com/apache/hadoop/pull/2464#discussion_r543157560
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsInputStream.java
##########
@@ -206,11 +229,120 @@ private int readOneBlock(final byte[] b, final int off, final int len) throws IO
fCursor += bytesRead;
fCursorAfterLastRead = fCursor;
}
+ return copyToUserBuffer(b, off, len);
+ }
+
+ private int readFileCompletely(final byte[] b, final int off, final int len)
+ throws IOException {
+ if (len == 0) {
+ return 0;
+ }
+ if (!validate(b, off, len)) {
+ return -1;
+ }
+ savePointerState();
+
+ buffer = new byte[bufferSize];
+ // data need to be copied to user buffer from index bCursor, bCursor has
+ // to be the current fCusor
+ bCursor = (int) fCursor;
+ return optimisedRead(b, off, len, 0, contentLength);
+ }
+
+ private int readLastBlock(final byte[] b, final int off, final int len)
+ throws IOException {
+ if (len == 0) {
+ return 0;
+ }
+ if (!validate(b, off, len)) {
+ return -1;
+ }
+ savePointerState();
+
+ buffer = new byte[bufferSize];
+ // data need to be copied to user buffer from index bCursor,
+ // AbfsInutStream buffer is going p contain dta from footer start. In
+ // that case bCursor will be set to footerStart - fCursor
+ long lastBlockStart = max(0, contentLength - bufferSize);
+ bCursor = (int) (fCursor - lastBlockStart);
+ // read API call is considered 1 single operation in reality server could
+ // return partial data and client has to retry untill the last full block
+ // is read. So setting the fCursorAfterLastRead before the possible
+ // multiple server calls
+ fCursorAfterLastRead = fCursor;
+ // 0 if contentlength is < buffersize
+ long actualLenToRead = min(bufferSize, contentLength);
+ return optimisedRead(b, off, len, lastBlockStart, actualLenToRead);
+ }
+
+ private int optimisedRead(final byte[] b, final int off, final int len,
+ final long readFrom, final long actualLen) throws IOException {
+ int totalBytesRead = 0;
+ fCursor = readFrom;
+ for (int i = 0; i < 2 && fCursor < contentLength; i++) {
+ int bytesRead = readInternal(fCursor, buffer, limit,
+ (int) actualLen - limit, true);
+ if (bytesRead > 0) {
+ totalBytesRead += bytesRead;
+ limit += bytesRead;
+ fCursor += bytesRead;
+ }
+ }
+ // if the read was not success and the user requested part of data has
Review comment:
Done
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #2464: HADOOP-17347. ABFS: Read optimizations
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on pull request #2464:
URL: https://github.com/apache/hadoop/pull/2464#issuecomment-745124861
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 1m 18s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | | 0m 0s | [test4tests](test4tests) | The patch appears to include 2 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 33m 37s | | trunk passed |
| +1 :green_heart: | compile | 0m 35s | | trunk passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 |
| +1 :green_heart: | compile | 0m 30s | | trunk passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| +1 :green_heart: | checkstyle | 0m 24s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 34s | | trunk passed |
| +1 :green_heart: | shadedclient | 16m 8s | | branch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 30s | | trunk passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 |
| +1 :green_heart: | javadoc | 0m 27s | | trunk passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| +0 :ok: | spotbugs | 0m 57s | | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 0m 55s | | trunk passed |
| -0 :warning: | patch | 1m 14s | | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 29s | | the patch passed |
| +1 :green_heart: | compile | 0m 29s | | the patch passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 |
| +1 :green_heart: | javac | 0m 29s | | the patch passed |
| +1 :green_heart: | compile | 0m 25s | | the patch passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| +1 :green_heart: | javac | 0m 25s | | the patch passed |
| -0 :warning: | checkstyle | 0m 17s | [/diff-checkstyle-hadoop-tools_hadoop-azure.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2464/15/artifact/out/diff-checkstyle-hadoop-tools_hadoop-azure.txt) | hadoop-tools/hadoop-azure: The patch generated 2 new + 2 unchanged - 0 fixed = 4 total (was 2) |
| +1 :green_heart: | mvnsite | 0m 29s | | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | | The patch has no whitespace issues. |
| +1 :green_heart: | shadedclient | 14m 44s | | patch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 25s | | the patch passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 |
| +1 :green_heart: | javadoc | 0m 25s | | the patch passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| +1 :green_heart: | findbugs | 0m 58s | | the patch passed |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 1m 29s | | hadoop-azure in the patch passed. |
| +1 :green_heart: | asflicense | 0m 34s | | The patch does not generate ASF License warnings. |
| | | 77m 47s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2464/15/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/2464 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 461d550a07da 4.15.0-65-generic #74-Ubuntu SMP Tue Sep 17 17:06:04 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 5bf977e6b16 |
| Default Java | Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2464/15/testReport/ |
| Max. process+thread count | 685 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2464/15/console |
| versions | git=2.17.1 maven=3.6.0 findbugs=4.0.6 |
| Powered by | Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] bilaharith commented on a change in pull request #2464: HADOOP-17347. ABFS: Read optimizations
Posted by GitBox <gi...@apache.org>.
bilaharith commented on a change in pull request #2464:
URL: https://github.com/apache/hadoop/pull/2464#discussion_r538340101
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsInputStream.java
##########
@@ -211,6 +235,59 @@ private int readOneBlock(final byte[] b, final int off, final int len) throws IO
return bytesToRead;
}
+ private boolean shouldReadFully() {
+ return this.firstRead && this.context.readSmallFilesCompletely()
+ && this.contentLength <= this.bufferSize;
+ }
+
+ private boolean shouldReadLastBlock(int len) {
+ return this.firstRead && this.context.optimizeFooterRead()
+ && len == FOOTER_SIZE
+ && this.fCursor == this.contentLength - FOOTER_SIZE;
+ }
+
+ private int readFileCompletely(final byte[] b, final int off, final int len) throws IOException {
+ int validation = validate(b, off, len);
+ if (validation < 1) {
+ return validation;
+ }
+
+ buffer = new byte[bufferSize];
+ bCursor = (int) fCursor;
+ int bytesRead = readInternal(0, buffer, 0, (int) contentLength, true);
+ firstRead = false;
+ if (bytesRead == -1) {
+ return -1;
+ }
+ fCursorAfterLastRead = fCursor;
+ limit = bytesRead;
+ fCursor = bytesRead;
+ return copyToUserBuffer(b, off, len);
+ }
+
+ private int readLastBlock(final byte[] b, final int off, final int len) throws IOException {
+ int validation = validate(b, off, len);
+ if (validation < 1) {
+ return validation;
+ }
+
+ buffer = new byte[bufferSize];
+ bCursor = (int) (((contentLength < bufferSize) ? contentLength : bufferSize)
+ - FOOTER_SIZE);
+ long lastBlockStartPos = (contentLength < bufferSize)
+ ? 0
+ : contentLength - bufferSize;
+ int bytesRead = readInternal(lastBlockStartPos, buffer, 0, bufferSize,
+ true);
Review comment:
Yes, it will read 100MB.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] snvijaya commented on a change in pull request #2464: HADOOP-17347. ABFS: Read optimizations
Posted by GitBox <gi...@apache.org>.
snvijaya commented on a change in pull request #2464:
URL: https://github.com/apache/hadoop/pull/2464#discussion_r543887310
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsInputStream.java
##########
@@ -206,11 +231,127 @@ private int readOneBlock(final byte[] b, final int off, final int len) throws IO
fCursor += bytesRead;
fCursorAfterLastRead = fCursor;
}
+ return copyToUserBuffer(b, off, len);
+ }
+ private int readFileCompletely(final byte[] b, final int off, final int len)
+ throws IOException {
+ if (len == 0) {
+ return 0;
+ }
+ if (!validate(b, off, len)) {
+ return -1;
+ }
+ savePointerState();
+
+ buffer = new byte[bufferSize];
+ // data need to be copied to user buffer from index bCursor, bCursor has
+ // to be the current fCusor
+ bCursor = (int) fCursor;
+ return optimisedRead(b, off, len, 0, contentLength);
+ }
+
+ private int readLastBlock(final byte[] b, final int off, final int len)
+ throws IOException {
+ if (len == 0) {
+ return 0;
+ }
+ if (!validate(b, off, len)) {
+ return -1;
+ }
+ savePointerState();
+
+ buffer = new byte[bufferSize];
+ // data need to be copied to user buffer from index bCursor,
+ // AbfsInutStream buffer is going p contain dta from footer start. In
+ // that case bCursor will be set to footerStart - fCursor
+ long lastBlockStart = max(0, contentLength - bufferSize);
+ bCursor = (int) (fCursor - lastBlockStart);
+ // read API call is considered 1 single operation in reality server could
+ // return partial data and client has to retry untill the last full block
+ // is read. So setting the fCursorAfterLastRead before the possible
+ // multiple server calls
+ fCursorAfterLastRead = fCursor;
+ // 0 if contentlength is < buffersize
+ long actualLenToRead = min(bufferSize, contentLength);
+ return optimisedRead(b, off, len, lastBlockStart, actualLenToRead);
+ }
+
+ private int optimisedRead(final byte[] b, final int off, final int len,
+ final long readFrom, final long actualLen) throws IOException {
Review comment:
Rename actualLen to fileSize
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #2464: HADOOP-17347. ABFS: Read optimizations
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on pull request #2464:
URL: https://github.com/apache/hadoop/pull/2464#issuecomment-745199325
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 1m 12s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | | 0m 0s | [test4tests](test4tests) | The patch appears to include 2 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 37m 40s | | trunk passed |
| +1 :green_heart: | compile | 0m 40s | | trunk passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 |
| +1 :green_heart: | compile | 0m 37s | | trunk passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| +1 :green_heart: | checkstyle | 0m 27s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 43s | | trunk passed |
| +1 :green_heart: | shadedclient | 20m 46s | | branch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 32s | | trunk passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 |
| +1 :green_heart: | javadoc | 0m 29s | | trunk passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| +0 :ok: | spotbugs | 1m 9s | | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 1m 6s | | trunk passed |
| -0 :warning: | patch | 1m 26s | | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 33s | | the patch passed |
| +1 :green_heart: | compile | 0m 35s | | the patch passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 |
| +1 :green_heart: | javac | 0m 35s | | the patch passed |
| +1 :green_heart: | compile | 0m 28s | | the patch passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| +1 :green_heart: | javac | 0m 28s | | the patch passed |
| -0 :warning: | checkstyle | 0m 18s | [/diff-checkstyle-hadoop-tools_hadoop-azure.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2464/16/artifact/out/diff-checkstyle-hadoop-tools_hadoop-azure.txt) | hadoop-tools/hadoop-azure: The patch generated 2 new + 2 unchanged - 0 fixed = 4 total (was 2) |
| +1 :green_heart: | mvnsite | 0m 32s | | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | | The patch has no whitespace issues. |
| +1 :green_heart: | shadedclient | 19m 19s | | patch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 30s | | the patch passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 |
| +1 :green_heart: | javadoc | 0m 25s | | the patch passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| +1 :green_heart: | findbugs | 1m 13s | | the patch passed |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 1m 35s | | hadoop-azure in the patch passed. |
| +1 :green_heart: | asflicense | 0m 38s | | The patch does not generate ASF License warnings. |
| | | 92m 23s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2464/16/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/2464 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 0a3de331a1aa 4.15.0-65-generic #74-Ubuntu SMP Tue Sep 17 17:06:04 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 5bf977e6b16 |
| Default Java | Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2464/16/testReport/ |
| Max. process+thread count | 677 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2464/16/console |
| versions | git=2.17.1 maven=3.6.0 findbugs=4.0.6 |
| Powered by | Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] vinaysbadami commented on a change in pull request #2464: Draft PR for read optimizations
Posted by GitBox <gi...@apache.org>.
vinaysbadami commented on a change in pull request #2464:
URL: https://github.com/apache/hadoop/pull/2464#discussion_r523918190
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsInputStream.java
##########
@@ -162,14 +172,86 @@ private int readOneBlock(final byte[] b, final int off, final int len) throws IO
throw new IndexOutOfBoundsException();
}
- //If buffer is empty, then fill the buffer.
- if (bCursor == limit) {
+ if (firstRead && this.readSmallFilesCompletely
+ && contentLength <= bufferSize) { // Read small files completely
+ if (readFileCompletely() == -1) {
+ return -1;
+ }
+ } else if (firstRead && this.optimizeFooterRead && fCursor
+ == contentLength - FOOTER_DELTA) { // Read the last one block if the
Review comment:
&& len == FOOTER_DELTA
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] bilaharith commented on a change in pull request #2464: Draft PR for read optimizations
Posted by GitBox <gi...@apache.org>.
bilaharith commented on a change in pull request #2464:
URL: https://github.com/apache/hadoop/pull/2464#discussion_r526586639
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsInputStream.java
##########
@@ -92,6 +99,9 @@ public AbfsInputStream(
this.cachedSasToken = new CachedSASToken(
abfsInputStreamContext.getSasTokenRenewPeriodForStreamsInSeconds());
this.streamStatistics = abfsInputStreamContext.getStreamStatistics();
+ this.readSmallFilesCompletely = abfsInputStreamContext
+ .readSmallFilesCompletely();
Review comment:
Done
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] bilaharith commented on a change in pull request #2464: HADOOP-17347. ABFS: Read optimizations
Posted by GitBox <gi...@apache.org>.
bilaharith commented on a change in pull request #2464:
URL: https://github.com/apache/hadoop/pull/2464#discussion_r539027581
##########
File path: hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/services/ITestAbfsInputStreamReadFooter.java
##########
@@ -0,0 +1,199 @@
+/**
Review comment:
Yes, if 100 MB is set as buffer size, current logic is forcing a single server read request for 100 MB.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on pull request #2464: HADOOP-17347. ABFS: Read optimizations
Posted by GitBox <gi...@apache.org>.
steveloughran commented on pull request #2464:
URL: https://github.com/apache/hadoop/pull/2464#issuecomment-756759257
It's moot now that the PR is merged, but can I remind everyone
## Add a meaningful message to assertTrue/assertFalse asserts
Imagine that you are trying to debug a test run from an automated build. All you have is that an assert failed on a given line. Does that provide enough information to diagnose the problem? Or would you need extra information? If so: what information should be included?
It's OK to use assertJ's assertThat, which is where new tests are going. It's a bit more verbose but its assertions are very informative and easily extensible. If you haven't used the library yet -it's on the classpath, try using its assertions in new test suites.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] bilaharith commented on a change in pull request #2464: HADOOP-17347. ABFS: Read optimizations
Posted by GitBox <gi...@apache.org>.
bilaharith commented on a change in pull request #2464:
URL: https://github.com/apache/hadoop/pull/2464#discussion_r539025293
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AbfsConfiguration.java
##########
@@ -517,6 +527,14 @@ public int getWriteBufferSize() {
return this.writeBufferSize;
}
+ public boolean readSmallFilesCompletely() {
Review comment:
Yes, if 100 MB is set as buffer size, current logic is forcing a single server read request for 100 MB.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] bilaharith commented on a change in pull request #2464: HADOOP-17347. ABFS: Read optimizations
Posted by GitBox <gi...@apache.org>.
bilaharith commented on a change in pull request #2464:
URL: https://github.com/apache/hadoop/pull/2464#discussion_r538340690
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsInputStream.java
##########
@@ -211,6 +235,59 @@ private int readOneBlock(final byte[] b, final int off, final int len) throws IO
return bytesToRead;
}
+ private boolean shouldReadFully() {
+ return this.firstRead && this.context.readSmallFilesCompletely()
+ && this.contentLength <= this.bufferSize;
+ }
+
+ private boolean shouldReadLastBlock(int len) {
+ return this.firstRead && this.context.optimizeFooterRead()
+ && len == FOOTER_SIZE
+ && this.fCursor == this.contentLength - FOOTER_SIZE;
+ }
+
+ private int readFileCompletely(final byte[] b, final int off, final int len) throws IOException {
+ int validation = validate(b, off, len);
+ if (validation < 1) {
+ return validation;
+ }
+
+ buffer = new byte[bufferSize];
+ bCursor = (int) fCursor;
+ int bytesRead = readInternal(0, buffer, 0, (int) contentLength, true);
+ firstRead = false;
+ if (bytesRead == -1) {
+ return -1;
+ }
+ fCursorAfterLastRead = fCursor;
+ limit = bytesRead;
+ fCursor = bytesRead;
+ return copyToUserBuffer(b, off, len);
+ }
+
+ private int readLastBlock(final byte[] b, final int off, final int len) throws IOException {
+ int validation = validate(b, off, len);
+ if (validation < 1) {
+ return validation;
+ }
+
+ buffer = new byte[bufferSize];
+ bCursor = (int) (((contentLength < bufferSize) ? contentLength : bufferSize)
+ - FOOTER_SIZE);
+ long lastBlockStartPos = (contentLength < bufferSize)
+ ? 0
+ : contentLength - bufferSize;
+ int bytesRead = readInternal(lastBlockStartPos, buffer, 0, bufferSize,
+ true);
+ firstRead = false;
+ if (bytesRead == -1) {
+ return -1;
+ }
+ fCursorAfterLastRead = fCursor;
+ limit = bytesRead;
+ fCursor = lastBlockStartPos + bytesRead;
Review comment:
fCursor is moved after the read is complete and the current fCursor is set to fCursorAfterLastRead. SO this is correct.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] bilaharith commented on a change in pull request #2464: HADOOP-17347. ABFS: Read optimizations
Posted by GitBox <gi...@apache.org>.
bilaharith commented on a change in pull request #2464:
URL: https://github.com/apache/hadoop/pull/2464#discussion_r544177687
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsInputStream.java
##########
@@ -206,11 +231,127 @@ private int readOneBlock(final byte[] b, final int off, final int len) throws IO
fCursor += bytesRead;
fCursorAfterLastRead = fCursor;
}
+ return copyToUserBuffer(b, off, len);
+ }
+ private int readFileCompletely(final byte[] b, final int off, final int len)
+ throws IOException {
+ if (len == 0) {
+ return 0;
+ }
+ if (!validate(b, off, len)) {
+ return -1;
+ }
+ savePointerState();
+
+ buffer = new byte[bufferSize];
+ // data need to be copied to user buffer from index bCursor, bCursor has
+ // to be the current fCusor
+ bCursor = (int) fCursor;
+ return optimisedRead(b, off, len, 0, contentLength);
+ }
+
+ private int readLastBlock(final byte[] b, final int off, final int len)
+ throws IOException {
+ if (len == 0) {
+ return 0;
+ }
+ if (!validate(b, off, len)) {
+ return -1;
+ }
+ savePointerState();
+
+ buffer = new byte[bufferSize];
+ // data need to be copied to user buffer from index bCursor,
+ // AbfsInutStream buffer is going p contain dta from footer start. In
Review comment:
Done
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] bilaharith commented on a change in pull request #2464: Draft PR for read optimizations
Posted by GitBox <gi...@apache.org>.
bilaharith commented on a change in pull request #2464:
URL: https://github.com/apache/hadoop/pull/2464#discussion_r528642440
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsInputStream.java
##########
@@ -162,14 +172,86 @@ private int readOneBlock(final byte[] b, final int off, final int len) throws IO
throw new IndexOutOfBoundsException();
}
- //If buffer is empty, then fill the buffer.
- if (bCursor == limit) {
+ if (firstRead && this.readSmallFilesCompletely
Review comment:
corrected
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus removed a comment on pull request #2464: HADOOP-17347. ABFS: Read optimizations
Posted by GitBox <gi...@apache.org>.
hadoop-yetus removed a comment on pull request #2464:
URL: https://github.com/apache/hadoop/pull/2464#issuecomment-746214660
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] snvijaya commented on a change in pull request #2464: HADOOP-17347. ABFS: Read optimizations
Posted by GitBox <gi...@apache.org>.
snvijaya commented on a change in pull request #2464:
URL: https://github.com/apache/hadoop/pull/2464#discussion_r543881568
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsInputStream.java
##########
@@ -206,11 +231,127 @@ private int readOneBlock(final byte[] b, final int off, final int len) throws IO
fCursor += bytesRead;
fCursorAfterLastRead = fCursor;
}
+ return copyToUserBuffer(b, off, len);
+ }
+ private int readFileCompletely(final byte[] b, final int off, final int len)
+ throws IOException {
+ if (len == 0) {
+ return 0;
+ }
+ if (!validate(b, off, len)) {
+ return -1;
+ }
+ savePointerState();
+
+ buffer = new byte[bufferSize];
+ // data need to be copied to user buffer from index bCursor, bCursor has
+ // to be the current fCusor
+ bCursor = (int) fCursor;
+ return optimisedRead(b, off, len, 0, contentLength);
+ }
+
+ private int readLastBlock(final byte[] b, final int off, final int len)
+ throws IOException {
+ if (len == 0) {
+ return 0;
+ }
+ if (!validate(b, off, len)) {
+ return -1;
+ }
+ savePointerState();
+
+ buffer = new byte[bufferSize];
+ // data need to be copied to user buffer from index bCursor,
+ // AbfsInutStream buffer is going p contain dta from footer start. In
Review comment:
Fix comments
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] snvijaya commented on a change in pull request #2464: HADOOP-17347. ABFS: Read optimizations
Posted by GitBox <gi...@apache.org>.
snvijaya commented on a change in pull request #2464:
URL: https://github.com/apache/hadoop/pull/2464#discussion_r549294264
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsInputStream.java
##########
@@ -211,6 +235,59 @@ private int readOneBlock(final byte[] b, final int off, final int len) throws IO
return bytesToRead;
}
+ private boolean shouldReadFully() {
+ return this.firstRead && this.context.readSmallFilesCompletely()
+ && this.contentLength <= this.bufferSize;
+ }
+
+ private boolean shouldReadLastBlock(int len) {
+ return this.firstRead && this.context.optimizeFooterRead()
+ && len == FOOTER_SIZE
+ && this.fCursor == this.contentLength - FOOTER_SIZE;
+ }
+
+ private int readFileCompletely(final byte[] b, final int off, final int len) throws IOException {
+ int validation = validate(b, off, len);
+ if (validation < 1) {
+ return validation;
+ }
+
+ buffer = new byte[bufferSize];
+ bCursor = (int) fCursor;
+ int bytesRead = readInternal(0, buffer, 0, (int) contentLength, true);
+ firstRead = false;
+ if (bytesRead == -1) {
+ return -1;
+ }
+ fCursorAfterLastRead = fCursor;
+ limit = bytesRead;
+ fCursor = bytesRead;
+ return copyToUserBuffer(b, off, len);
+ }
+
+ private int readLastBlock(final byte[] b, final int off, final int len) throws IOException {
+ int validation = validate(b, off, len);
+ if (validation < 1) {
+ return validation;
+ }
+
+ buffer = new byte[bufferSize];
+ bCursor = (int) (((contentLength < bufferSize) ? contentLength : bufferSize)
+ - FOOTER_SIZE);
+ long lastBlockStartPos = (contentLength < bufferSize)
+ ? 0
+ : contentLength - bufferSize;
+ int bytesRead = readInternal(lastBlockStartPos, buffer, 0, bufferSize,
+ true);
+ firstRead = false;
+ if (bytesRead == -1) {
+ return -1;
+ }
+ fCursorAfterLastRead = fCursor;
+ limit = bytesRead;
+ fCursor = lastBlockStartPos + bytesRead;
Review comment:
Dont recall the context here. But the code is completely different now and this doesnt hold ?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] snvijaya commented on a change in pull request #2464: HADOOP-17347. ABFS: Read optimizations
Posted by GitBox <gi...@apache.org>.
snvijaya commented on a change in pull request #2464:
URL: https://github.com/apache/hadoop/pull/2464#discussion_r543887310
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsInputStream.java
##########
@@ -206,11 +231,127 @@ private int readOneBlock(final byte[] b, final int off, final int len) throws IO
fCursor += bytesRead;
fCursorAfterLastRead = fCursor;
}
+ return copyToUserBuffer(b, off, len);
+ }
+ private int readFileCompletely(final byte[] b, final int off, final int len)
+ throws IOException {
+ if (len == 0) {
+ return 0;
+ }
+ if (!validate(b, off, len)) {
+ return -1;
+ }
+ savePointerState();
+
+ buffer = new byte[bufferSize];
+ // data need to be copied to user buffer from index bCursor, bCursor has
+ // to be the current fCusor
+ bCursor = (int) fCursor;
+ return optimisedRead(b, off, len, 0, contentLength);
+ }
+
+ private int readLastBlock(final byte[] b, final int off, final int len)
+ throws IOException {
+ if (len == 0) {
+ return 0;
+ }
+ if (!validate(b, off, len)) {
+ return -1;
+ }
+ savePointerState();
+
+ buffer = new byte[bufferSize];
+ // data need to be copied to user buffer from index bCursor,
+ // AbfsInutStream buffer is going p contain dta from footer start. In
+ // that case bCursor will be set to footerStart - fCursor
+ long lastBlockStart = max(0, contentLength - bufferSize);
+ bCursor = (int) (fCursor - lastBlockStart);
+ // read API call is considered 1 single operation in reality server could
+ // return partial data and client has to retry untill the last full block
+ // is read. So setting the fCursorAfterLastRead before the possible
+ // multiple server calls
+ fCursorAfterLastRead = fCursor;
+ // 0 if contentlength is < buffersize
+ long actualLenToRead = min(bufferSize, contentLength);
+ return optimisedRead(b, off, len, lastBlockStart, actualLenToRead);
+ }
+
+ private int optimisedRead(final byte[] b, final int off, final int len,
+ final long readFrom, final long actualLen) throws IOException {
Review comment:
Rename actualLen to fileSize
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #2464: HADOOP-17347. ABFS: Read optimizations
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on pull request #2464:
URL: https://github.com/apache/hadoop/pull/2464#issuecomment-750935468
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 1m 8s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | | 0m 0s | [test4tests](test4tests) | The patch appears to include 3 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 32m 59s | | trunk passed |
| +1 :green_heart: | compile | 0m 40s | | trunk passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 |
| +1 :green_heart: | compile | 0m 35s | | trunk passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| +1 :green_heart: | checkstyle | 0m 27s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 40s | | trunk passed |
| +1 :green_heart: | shadedclient | 16m 15s | | branch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 33s | | trunk passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 |
| +1 :green_heart: | javadoc | 0m 28s | | trunk passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| +0 :ok: | spotbugs | 0m 59s | | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 0m 58s | | trunk passed |
| -0 :warning: | patch | 1m 18s | | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 32s | | the patch passed |
| +1 :green_heart: | compile | 0m 29s | | the patch passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 |
| +1 :green_heart: | javac | 0m 29s | | the patch passed |
| +1 :green_heart: | compile | 0m 25s | | the patch passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| +1 :green_heart: | javac | 0m 25s | | the patch passed |
| +1 :green_heart: | checkstyle | 0m 17s | | the patch passed |
| +1 :green_heart: | mvnsite | 0m 29s | | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | | The patch has no whitespace issues. |
| +1 :green_heart: | shadedclient | 14m 48s | | patch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 26s | | the patch passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 |
| +1 :green_heart: | javadoc | 0m 24s | | the patch passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| +1 :green_heart: | findbugs | 0m 58s | | the patch passed |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 1m 29s | | hadoop-azure in the patch passed. |
| +1 :green_heart: | asflicense | 0m 32s | | The patch does not generate ASF License warnings. |
| | | 77m 50s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2464/20/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/2464 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux e1f25b1d30b6 4.15.0-65-generic #74-Ubuntu SMP Tue Sep 17 17:06:04 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 005b854f6ba |
| Default Java | Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2464/20/testReport/ |
| Max. process+thread count | 535 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2464/20/console |
| versions | git=2.17.1 maven=3.6.0 findbugs=4.0.6 |
| Powered by | Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #2464: HADOOP-17347. ABFS: Read optimizations
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on pull request #2464:
URL: https://github.com/apache/hadoop/pull/2464#issuecomment-743317123
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 30s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | | 0m 0s | [test4tests](test4tests) | The patch appears to include 2 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 32m 48s | | trunk passed |
| +1 :green_heart: | compile | 0m 38s | | trunk passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 |
| +1 :green_heart: | compile | 0m 34s | | trunk passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| +1 :green_heart: | checkstyle | 0m 27s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 39s | | trunk passed |
| +1 :green_heart: | shadedclient | 16m 20s | | branch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 31s | | trunk passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 |
| +1 :green_heart: | javadoc | 0m 29s | | trunk passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| +0 :ok: | spotbugs | 0m 58s | | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 0m 57s | | trunk passed |
| -0 :warning: | patch | 1m 17s | | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 31s | | the patch passed |
| +1 :green_heart: | compile | 0m 29s | | the patch passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 |
| +1 :green_heart: | javac | 0m 29s | | the patch passed |
| +1 :green_heart: | compile | 0m 26s | | the patch passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| +1 :green_heart: | javac | 0m 26s | | the patch passed |
| +1 :green_heart: | checkstyle | 0m 17s | | the patch passed |
| +1 :green_heart: | mvnsite | 0m 30s | | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | | The patch has no whitespace issues. |
| +1 :green_heart: | shadedclient | 14m 45s | | patch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 25s | | the patch passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 |
| +1 :green_heart: | javadoc | 0m 25s | | the patch passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| +1 :green_heart: | findbugs | 0m 58s | | the patch passed |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 1m 30s | | hadoop-azure in the patch passed. |
| +1 :green_heart: | asflicense | 0m 32s | | The patch does not generate ASF License warnings. |
| | | 77m 6s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2464/14/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/2464 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux ca618b2b5990 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 9bd3c9bc506 |
| Default Java | Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2464/14/testReport/ |
| Max. process+thread count | 684 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2464/14/console |
| versions | git=2.17.1 maven=3.6.0 findbugs=4.0.6 |
| Powered by | Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] snvijaya commented on a change in pull request #2464: HADOOP-17347. ABFS: Read optimizations
Posted by GitBox <gi...@apache.org>.
snvijaya commented on a change in pull request #2464:
URL: https://github.com/apache/hadoop/pull/2464#discussion_r540082701
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsInputStream.java
##########
@@ -224,6 +240,123 @@ private int readOneBlock(final byte[] b, final int off, final int len) throws IO
return bytesToRead;
}
+ private boolean shouldReadFully() {
+ return this.firstRead && this.context.readSmallFilesCompletely()
+ && this.contentLength <= this.bufferSize;
+ }
+
+ private boolean shouldReadLastBlock(int len) {
+ return this.firstRead && this.context.optimizeFooterRead()
+ && len == FOOTER_SIZE
+ && this.fCursor == this.contentLength - FOOTER_SIZE;
+ }
+
+ private int readFileCompletely(final byte[] b, final int off, final int len)
+ throws IOException {
+ if (closed) {
+ throw new IOException(FSExceptionMessages.STREAM_IS_CLOSED);
+ }
+
+ Preconditions.checkNotNull(b);
+ LOG.debug("read one block requested b.length = {} off {} len {}", b.length,
+ off, len);
+
+ if (len == 0) {
+ return 0;
+ }
+
+ if (this.available() == 0) {
+ return -1;
+ }
+
+ if (off < 0 || len < 0 || len > b.length - off) {
+ throw new IndexOutOfBoundsException();
+ }
+
+ buffer = new byte[bufferSize];
+ // data need to be copied to user buffer from index bCursor, bCursor has
+ // to be the current fCusor
+ bCursor = (int) fCursor;
+ fCursorAfterLastRead = fCursor;
+ int totalBytesRead = 0;
+ int loopCount = 0;
+ // Read from begining
+ fCursor = 0;
+ while (fCursor < contentLength) {
+ int bytesRead = readInternal(fCursor, buffer, limit,
+ (int) contentLength - limit, true);
+ if (bytesRead > 0) {
+ totalBytesRead += bytesRead;
+ limit += bytesRead;
+ fCursor += bytesRead;
+ }
+ if (loopCount++ >= 10) {
Review comment:
As discussed, this logic needs to be relooked for
1. What if the requested data is already available in the partial read done ?
2. Reduce the loopCount as the retry logic on ABFS driver can make the client read overall expensive. Fail faster with just 1 or 2 tries.
3. Never fail the read request because optimization code failed to read full file. Fail fast and send read for client requested position.
4. All pointer fields need to be in valid state (bCursor, fCursor, fCursorAfterLastRead). In failure case currently fCursor could be in a different position that last seek done.
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsInputStream.java
##########
@@ -137,7 +142,13 @@ public synchronized int read(final byte[] b, final int off, final int len) throw
}
incrementReadOps();
do {
- lastReadBytes = readOneBlock(b, currentOff, currentLen);
+ if (shouldReadFully()) {
Review comment:
validate logic should be grouped into a method and be called from here so that code duplication can be avoided in the 3 flows below.
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsInputStream.java
##########
@@ -224,6 +240,123 @@ private int readOneBlock(final byte[] b, final int off, final int len) throws IO
return bytesToRead;
}
+ private boolean shouldReadFully() {
+ return this.firstRead && this.context.readSmallFilesCompletely()
+ && this.contentLength <= this.bufferSize;
+ }
+
+ private boolean shouldReadLastBlock(int len) {
+ return this.firstRead && this.context.optimizeFooterRead()
+ && len == FOOTER_SIZE
+ && this.fCursor == this.contentLength - FOOTER_SIZE;
+ }
+
+ private int readFileCompletely(final byte[] b, final int off, final int len)
+ throws IOException {
+ if (closed) {
+ throw new IOException(FSExceptionMessages.STREAM_IS_CLOSED);
+ }
+
+ Preconditions.checkNotNull(b);
+ LOG.debug("read one block requested b.length = {} off {} len {}", b.length,
+ off, len);
+
+ if (len == 0) {
+ return 0;
+ }
+
+ if (this.available() == 0) {
+ return -1;
+ }
+
+ if (off < 0 || len < 0 || len > b.length - off) {
+ throw new IndexOutOfBoundsException();
+ }
+
+ buffer = new byte[bufferSize];
+ // data need to be copied to user buffer from index bCursor, bCursor has
+ // to be the current fCusor
+ bCursor = (int) fCursor;
+ fCursorAfterLastRead = fCursor;
+ int totalBytesRead = 0;
+ int loopCount = 0;
+ // Read from begining
+ fCursor = 0;
+ while (fCursor < contentLength) {
+ int bytesRead = readInternal(fCursor, buffer, limit,
+ (int) contentLength - limit, true);
+ if (bytesRead > 0) {
+ totalBytesRead += bytesRead;
+ limit += bytesRead;
+ fCursor += bytesRead;
+ }
+ if (loopCount++ >= 10) {
+ throw new IOException(
+ "Too many attempts in reading whole file " + path);
+ }
+ }
+ firstRead = false;
+ if (totalBytesRead == -1) {
+ return -1;
+ }
+ return copyToUserBuffer(b, off, len);
+ }
+
+ private int readLastBlock(final byte[] b, final int off, final int len)
+ throws IOException {
+ if (closed) {
+ throw new IOException(FSExceptionMessages.STREAM_IS_CLOSED);
+ }
+
+ Preconditions.checkNotNull(b);
+ LOG.debug("read one block requested b.length = {} off {} len {}", b.length,
+ off, len);
+
+ if (len == 0) {
+ return 0;
+ }
+
+ if (this.available() == 0) {
+ return -1;
+ }
+
+ if (off < 0 || len < 0 || len > b.length - off) {
+ throw new IndexOutOfBoundsException();
+ }
+
+ buffer = new byte[bufferSize];
+ // data need to be copied to user buffer from index bCursor, for small
+ // files the bCursor will be contentlength - footer size,
+ // otherwise buffersize - footer size
+ bCursor = (int) (Math.min(contentLength, bufferSize) - FOOTER_SIZE);
+ // read API call is considered 1 single operation in reality server could
+ // return partial data and client has to retry untill the last full block
+ // is read. So setting the fCursorAfterLastRead before the possible
+ // multiple server calls
+ fCursorAfterLastRead = fCursor;
+ // 0 if contentlength is < buffersize
+ fCursor = Math.max(0, contentLength - bufferSize);
+ int totalBytesRead = 0;
+ int loopCount = 0;
+ while (fCursor < contentLength) {
+ int bytesRead = readInternal(fCursor, buffer, limit, bufferSize - limit,
+ true);
+ if (bytesRead > 0) {
+ totalBytesRead += bytesRead;
+ limit += bytesRead;
+ fCursor += bytesRead;
+ }
+ if (loopCount++ >= 10) {
Review comment:
Comments added in readSmallFileCompletely will apply here too.
Also, failure state recovery might lead to throwing away the data that was retried till then. Using readAhead threads to read will help to save the partial data read and be of use for next read call.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] bilaharith commented on pull request #2464: Draft PR: HADOOP-17347. ABFS: Read optimizations
Posted by GitBox <gi...@apache.org>.
bilaharith commented on pull request #2464:
URL: https://github.com/apache/hadoop/pull/2464#issuecomment-737683882
Driver test results against test accounts in Central India region
HNS-OAuth
========================
[INFO] Results:
[INFO]
[INFO] Tests run: 90, Failures: 0, Errors: 0, Skipped: 0
[INFO] Results:
[INFO]
[WARNING] Tests run: 478, Failures: 0, Errors: 0, Skipped: 66
[INFO] Results:
[INFO]
[WARNING] Tests run: 208, Failures: 0, Errors: 0, Skipped: 24
HNS-SharedKey
========================
[INFO] Results:
[INFO]
[INFO] Tests run: 90, Failures: 0, Errors: 0, Skipped: 0
[INFO] Results:
[INFO]
[WARNING] Tests run: 478, Failures: 0, Errors: 0, Skipped: 24
[INFO] Results:
[INFO]
[WARNING] Tests run: 208, Failures: 0, Errors: 0, Skipped: 16
NonHNS-SharedKey
========================
[INFO] Results:
[INFO]
[INFO] Tests run: 90, Failures: 0, Errors: 0, Skipped: 0
[INFO] Results:
[INFO]
[WARNING] Tests run: 478, Failures: 0, Errors: 0, Skipped: 247
[INFO] Results:
[INFO]
[WARNING] Tests run: 208, Failures: 0, Errors: 0, Skipped: 16
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #2464: Draft PR for read optimizations
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on pull request #2464:
URL: https://github.com/apache/hadoop/pull/2464#issuecomment-733482728
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 26m 9s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files found. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | | 0m 0s | [test4tests](test4tests) | The patch appears to include 2 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 32m 30s | | trunk passed |
| +1 :green_heart: | compile | 0m 38s | | trunk passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 |
| +1 :green_heart: | compile | 0m 35s | | trunk passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| +1 :green_heart: | checkstyle | 0m 28s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 40s | | trunk passed |
| +1 :green_heart: | shadedclient | 16m 14s | | branch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 33s | | trunk passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 |
| +1 :green_heart: | javadoc | 0m 29s | | trunk passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| +0 :ok: | spotbugs | 0m 59s | | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 0m 57s | | trunk passed |
| -0 :warning: | patch | 1m 17s | | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 30s | | the patch passed |
| +1 :green_heart: | compile | 0m 30s | | the patch passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 |
| +1 :green_heart: | javac | 0m 30s | | the patch passed |
| +1 :green_heart: | compile | 0m 26s | | the patch passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| +1 :green_heart: | javac | 0m 26s | | the patch passed |
| +1 :green_heart: | checkstyle | 0m 19s | | the patch passed |
| +1 :green_heart: | mvnsite | 0m 29s | | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | | The patch has no whitespace issues. |
| +1 :green_heart: | shadedclient | 14m 39s | | patch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 26s | | the patch passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 |
| +1 :green_heart: | javadoc | 0m 25s | | the patch passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| +1 :green_heart: | findbugs | 0m 58s | | the patch passed |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 1m 31s | | hadoop-azure in the patch passed. |
| +1 :green_heart: | asflicense | 0m 32s | | The patch does not generate ASF License warnings. |
| | | 102m 9s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2464/4/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/2464 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux ed52fb6b755d 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 08b2e285dbd |
| Default Java | Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2464/4/testReport/ |
| Max. process+thread count | 685 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2464/4/console |
| versions | git=2.17.1 maven=3.6.0 findbugs=4.0.6 |
| Powered by | Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #2464: HADOOP-17347. ABFS: Read optimizations
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on pull request #2464:
URL: https://github.com/apache/hadoop/pull/2464#issuecomment-741581624
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 30s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | | 0m 0s | [test4tests](test4tests) | The patch appears to include 2 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 32m 59s | | trunk passed |
| +1 :green_heart: | compile | 0m 37s | | trunk passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 |
| +1 :green_heart: | compile | 0m 35s | | trunk passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| +1 :green_heart: | checkstyle | 0m 28s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 39s | | trunk passed |
| +1 :green_heart: | shadedclient | 16m 39s | | branch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 31s | | trunk passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 |
| +1 :green_heart: | javadoc | 0m 30s | | trunk passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| +0 :ok: | spotbugs | 0m 59s | | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 0m 57s | | trunk passed |
| -0 :warning: | patch | 1m 17s | | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 30s | | the patch passed |
| +1 :green_heart: | compile | 0m 29s | | the patch passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 |
| +1 :green_heart: | javac | 0m 29s | | the patch passed |
| +1 :green_heart: | compile | 0m 26s | | the patch passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| +1 :green_heart: | javac | 0m 26s | | the patch passed |
| +1 :green_heart: | checkstyle | 0m 16s | | the patch passed |
| +1 :green_heart: | mvnsite | 0m 28s | | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | | The patch has no whitespace issues. |
| +1 :green_heart: | shadedclient | 14m 47s | | patch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 26s | | the patch passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 |
| +1 :green_heart: | javadoc | 0m 23s | | the patch passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| +1 :green_heart: | findbugs | 0m 57s | | the patch passed |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 1m 28s | | hadoop-azure in the patch passed. |
| +1 :green_heart: | asflicense | 0m 33s | | The patch does not generate ASF License warnings. |
| | | 77m 25s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2464/10/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/2464 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 20549d74de16 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / aaf9e3d320a |
| Default Java | Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2464/10/testReport/ |
| Max. process+thread count | 536 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2464/10/console |
| versions | git=2.17.1 maven=3.6.0 findbugs=4.0.6 |
| Powered by | Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] bilaharith commented on a change in pull request #2464: Draft PR for read optimizations
Posted by GitBox <gi...@apache.org>.
bilaharith commented on a change in pull request #2464:
URL: https://github.com/apache/hadoop/pull/2464#discussion_r526588153
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsInputStream.java
##########
@@ -162,14 +172,86 @@ private int readOneBlock(final byte[] b, final int off, final int len) throws IO
throw new IndexOutOfBoundsException();
}
- //If buffer is empty, then fill the buffer.
- if (bCursor == limit) {
+ if (firstRead && this.readSmallFilesCompletely
+ && contentLength <= bufferSize) { // Read small files completely
+ if (readFileCompletely() == -1) {
+ return -1;
+ }
+ } else if (firstRead && this.optimizeFooterRead && fCursor
+ == contentLength - FOOTER_DELTA) { // Read the last one block if the
+ // read is for the footer
+ if (readLastBlock() == -1) {
+ return -1;
+ }
+ } else {
+ if (readOneBlock(b) == -1) {
+ return -1;
+ }
+ }
+ fCursorAfterLastRead = fCursor;
+
+ //If there is anything in the buffer, then return lesser of (requested bytes) and (bytes in buffer)
+ //(bytes returned may be less than requested)
+ int bytesRemaining = limit - bCursor;
+ int bytesToRead = Math.min(len, bytesRemaining);
+ System.arraycopy(buffer, bCursor, b, off, bytesToRead);
+ bCursor += bytesToRead;
+ if (statistics != null) {
+ statistics.incrementBytesRead(bytesToRead);
+ }
+ if (streamStatistics != null) {
+ // Bytes read from the local buffer.
+ streamStatistics.bytesReadFromBuffer(bytesToRead);
+ streamStatistics.bytesRead(bytesToRead);
+ }
+ return bytesToRead;
+ }
+
+ private long readFileCompletely() throws IOException {
+ bCursor = (int) getPos();
+ // Read full file in case the file size <= buffer size
+ buffer = new byte[bufferSize];
+ long bytesRead = readInternal(0, buffer, 0, (int) contentLength, true);
+ firstRead = false;
+
+ if (bytesRead == -1) {
+ return -1;
+ }
+
+ limit = (int) bytesRead;
+ fCursor = bytesRead;
+ return bytesRead;
+ }
+
+ private long readLastBlock() throws IOException {
+ bCursor = (int) (
+ ((contentLength < bufferSize) ? contentLength : bufferSize)
+ - FOOTER_DELTA);
+ buffer = new byte[bufferSize];
+ long startPos = (contentLength < bufferSize)
+ ? 0
+ : contentLength - bufferSize;
+ long bytesRead = readInternal(startPos, buffer, 0, bufferSize, true);
+ firstRead = false;
+
+ if (bytesRead == -1) {
+ return -1;
+ }
+
+ limit += (int) bytesRead;
Review comment:
Done
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] bilaharith commented on a change in pull request #2464: HADOOP-17347. ABFS: Read optimizations
Posted by GitBox <gi...@apache.org>.
bilaharith commented on a change in pull request #2464:
URL: https://github.com/apache/hadoop/pull/2464#discussion_r540268718
##########
File path: hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/services/ITestAbfsInputStreamReadFooter.java
##########
@@ -0,0 +1,199 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.azurebfs.services;
+
+import java.io.IOException;
+import java.util.Map;
+import java.util.Random;
+
+import org.junit.Test;
+
+import org.apache.hadoop.fs.FSDataInputStream;
+import org.apache.hadoop.fs.FSDataOutputStream;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.fs.azurebfs.AbfsConfiguration;
+import org.apache.hadoop.fs.azurebfs.AbstractAbfsIntegrationTest;
+import org.apache.hadoop.fs.azurebfs.AzureBlobFileSystem;
+
+import static org.apache.hadoop.fs.azurebfs.AbfsStatistic.CONNECTIONS_MADE;
+import static org.apache.hadoop.fs.azurebfs.constants.FileSystemConfigurations.ONE_KB;
+import static org.apache.hadoop.fs.azurebfs.constants.FileSystemConfigurations.ONE_MB;
+
+public class ITestAbfsInputStreamReadFooter
+ extends AbstractAbfsIntegrationTest {
+
+ private static final int TEN = 10;
+ private static final int TWENTY = 20;
+
+ public ITestAbfsInputStreamReadFooter() throws Exception {
+ }
+
+ @Test
+ public void testOnlyOneServerCallIsMadeWhenTheConfIsTrue() throws Exception {
+ testNumBackendCalls(true);
+ }
+
+ @Test
+ public void testMultipleServerCallsAreMadeWhenTheConfIsFalse()
+ throws Exception {
+ testNumBackendCalls(false);
+ }
+
+ private void testNumBackendCalls(boolean optimizeFooterRead)
+ throws Exception {
+ final AzureBlobFileSystem fs = getFileSystem(optimizeFooterRead);
+ for (int i = 1; i <= 4; i++) {
+ String fileName = methodName.getMethodName() + i;
+ int fileSize = i * ONE_MB;
+ byte[] fileContent = getRandomBytesArray(fileSize);
+ Path testFilePath = createFileWithContent(fs, fileName, fileContent);
+ int length = AbfsInputStream.FOOTER_SIZE;
+ try (FSDataInputStream iStream = fs.open(testFilePath)) {
+ byte[] buffer = new byte[length];
+
+ Map<String, Long> metricMap = fs.getInstrumentationMap();
+ long requestsMadeBeforeTest = metricMap
+ .get(CONNECTIONS_MADE.getStatName());
+
+ iStream.seek(fileSize - 8);
+ iStream.read(buffer, 0, length);
+
+ iStream.seek(fileSize - (TEN * ONE_KB));
+ iStream.read(buffer, 0, length);
+
+ iStream.seek(fileSize - (TWENTY * ONE_KB));
+ iStream.read(buffer, 0, length);
+
+ metricMap = fs.getInstrumentationMap();
+ long requestsMadeAfterTest = metricMap
+ .get(CONNECTIONS_MADE.getStatName());
+
+ if (optimizeFooterRead) {
+ assertEquals(1, requestsMadeAfterTest - requestsMadeBeforeTest);
+ } else {
+ assertEquals(3, requestsMadeAfterTest - requestsMadeBeforeTest);
+ }
+ }
+ }
+ }
+
+ @Test
+ public void testSeekToEndAndReadWithConfTrue() throws Exception {
+ testSeekToEndAndReadWithConf(true);
+ }
+
+ @Test
+ public void testSeekToEndAndReadWithConfFalse() throws Exception {
+ testSeekToEndAndReadWithConf(false);
+ }
+
+ private void testSeekToEndAndReadWithConf(boolean optimizeFooterRead) throws Exception {
+ final AzureBlobFileSystem fs = getFileSystem(optimizeFooterRead);
+ for (int i = 5; i <= 10; i++) {
+ String fileName = methodName.getMethodName() + i;
+ int fileSize = i * ONE_MB;
+ byte[] fileContent = getRandomBytesArray(fileSize);
+ Path testFilePath = createFileWithContent(fs, fileName, fileContent);
+ seekReadAndTest(fs, testFilePath, fileSize - AbfsInputStream.FOOTER_SIZE,
+ AbfsInputStream.FOOTER_SIZE, fileContent);
+ }
+ }
+
+ private AzureBlobFileSystem getFileSystem(boolean optimizeFooterRead)
+ throws IOException {
+ final AzureBlobFileSystem fs = getFileSystem();
+ fs.getAbfsStore().getAbfsConfiguration()
+ .setOptimizeFooterRead(optimizeFooterRead);
+ fs.getAbfsStore().getAbfsConfiguration()
+ .setReadSmallFilesCompletely(false);
Review comment:
For small files this needs to be test in order to test the footer read.
For big files this is not set and expected to skip the smallfile read if condition.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] snvijaya commented on a change in pull request #2464: HADOOP-17347. ABFS: Read optimizations
Posted by GitBox <gi...@apache.org>.
snvijaya commented on a change in pull request #2464:
URL: https://github.com/apache/hadoop/pull/2464#discussion_r543900358
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsInputStream.java
##########
@@ -206,11 +231,127 @@ private int readOneBlock(final byte[] b, final int off, final int len) throws IO
fCursor += bytesRead;
fCursorAfterLastRead = fCursor;
}
+ return copyToUserBuffer(b, off, len);
+ }
+ private int readFileCompletely(final byte[] b, final int off, final int len)
+ throws IOException {
+ if (len == 0) {
+ return 0;
+ }
+ if (!validate(b, off, len)) {
+ return -1;
+ }
+ savePointerState();
+
+ buffer = new byte[bufferSize];
+ // data need to be copied to user buffer from index bCursor, bCursor has
+ // to be the current fCusor
+ bCursor = (int) fCursor;
+ return optimisedRead(b, off, len, 0, contentLength);
+ }
+
+ private int readLastBlock(final byte[] b, final int off, final int len)
+ throws IOException {
+ if (len == 0) {
+ return 0;
+ }
+ if (!validate(b, off, len)) {
+ return -1;
+ }
+ savePointerState();
+
+ buffer = new byte[bufferSize];
+ // data need to be copied to user buffer from index bCursor,
+ // AbfsInutStream buffer is going p contain dta from footer start. In
+ // that case bCursor will be set to footerStart - fCursor
+ long lastBlockStart = max(0, contentLength - bufferSize);
+ bCursor = (int) (fCursor - lastBlockStart);
+ // read API call is considered 1 single operation in reality server could
+ // return partial data and client has to retry untill the last full block
+ // is read. So setting the fCursorAfterLastRead before the possible
+ // multiple server calls
+ fCursorAfterLastRead = fCursor;
+ // 0 if contentlength is < buffersize
+ long actualLenToRead = min(bufferSize, contentLength);
+ return optimisedRead(b, off, len, lastBlockStart, actualLenToRead);
+ }
+
+ private int optimisedRead(final byte[] b, final int off, final int len,
+ final long readFrom, final long actualLen) throws IOException {
+ int totalBytesRead = 0;
+ fCursor = readFrom;
+ try {
Review comment:
Initialize buffer here
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #2464: Draft PR: HADOOP-17347. ABFS: Read optimizations
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on pull request #2464:
URL: https://github.com/apache/hadoop/pull/2464#issuecomment-736683730
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 1m 8s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | | 0m 0s | [test4tests](test4tests) | The patch appears to include 2 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 35m 17s | | trunk passed |
| +1 :green_heart: | compile | 0m 34s | | trunk passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 |
| +1 :green_heart: | compile | 0m 29s | | trunk passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| +1 :green_heart: | checkstyle | 0m 22s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 33s | | trunk passed |
| +1 :green_heart: | shadedclient | 17m 50s | | branch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 26s | | trunk passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 |
| +1 :green_heart: | javadoc | 0m 24s | | trunk passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| +0 :ok: | spotbugs | 0m 59s | | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 0m 57s | | trunk passed |
| -0 :warning: | patch | 1m 15s | | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 29s | | the patch passed |
| +1 :green_heart: | compile | 0m 28s | | the patch passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 |
| +1 :green_heart: | javac | 0m 28s | | the patch passed |
| +1 :green_heart: | compile | 0m 24s | | the patch passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| +1 :green_heart: | javac | 0m 24s | | the patch passed |
| +1 :green_heart: | checkstyle | 0m 15s | | the patch passed |
| +1 :green_heart: | mvnsite | 0m 28s | | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | | The patch has no whitespace issues. |
| +1 :green_heart: | shadedclient | 16m 57s | | patch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 23s | | the patch passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 |
| +1 :green_heart: | javadoc | 0m 21s | | the patch passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| +1 :green_heart: | findbugs | 0m 56s | | the patch passed |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 1m 26s | | hadoop-azure in the patch passed. |
| +1 :green_heart: | asflicense | 0m 29s | | The patch does not generate ASF License warnings. |
| | | 82m 43s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2464/8/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/2464 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux e13a46fa7bd4 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 6a1d7d9ed25 |
| Default Java | Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2464/8/testReport/ |
| Max. process+thread count | 572 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2464/8/console |
| versions | git=2.17.1 maven=3.6.0 findbugs=4.0.6 |
| Powered by | Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] bilaharith commented on a change in pull request #2464: HADOOP-17347. ABFS: Read optimizations
Posted by GitBox <gi...@apache.org>.
bilaharith commented on a change in pull request #2464:
URL: https://github.com/apache/hadoop/pull/2464#discussion_r544176128
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsInputStream.java
##########
@@ -206,11 +231,127 @@ private int readOneBlock(final byte[] b, final int off, final int len) throws IO
fCursor += bytesRead;
fCursorAfterLastRead = fCursor;
}
+ return copyToUserBuffer(b, off, len);
+ }
+ private int readFileCompletely(final byte[] b, final int off, final int len)
+ throws IOException {
+ if (len == 0) {
+ return 0;
+ }
+ if (!validate(b, off, len)) {
+ return -1;
+ }
+ savePointerState();
+
+ buffer = new byte[bufferSize];
+ // data need to be copied to user buffer from index bCursor, bCursor has
+ // to be the current fCusor
+ bCursor = (int) fCursor;
+ return optimisedRead(b, off, len, 0, contentLength);
+ }
+
+ private int readLastBlock(final byte[] b, final int off, final int len)
+ throws IOException {
+ if (len == 0) {
+ return 0;
+ }
+ if (!validate(b, off, len)) {
+ return -1;
+ }
+ savePointerState();
+
+ buffer = new byte[bufferSize];
+ // data need to be copied to user buffer from index bCursor,
+ // AbfsInutStream buffer is going p contain dta from footer start. In
+ // that case bCursor will be set to footerStart - fCursor
+ long lastBlockStart = max(0, contentLength - bufferSize);
+ bCursor = (int) (fCursor - lastBlockStart);
+ // read API call is considered 1 single operation in reality server could
+ // return partial data and client has to retry untill the last full block
+ // is read. So setting the fCursorAfterLastRead before the possible
+ // multiple server calls
+ fCursorAfterLastRead = fCursor;
+ // 0 if contentlength is < buffersize
+ long actualLenToRead = min(bufferSize, contentLength);
+ return optimisedRead(b, off, len, lastBlockStart, actualLenToRead);
+ }
+
+ private int optimisedRead(final byte[] b, final int off, final int len,
+ final long readFrom, final long actualLen) throws IOException {
+ int totalBytesRead = 0;
+ fCursor = readFrom;
+ try {
+ for (int i = 0;
+ i < OPTIMIZATION_ATTEMPTS && fCursor < contentLength; i++) {
+ int bytesRead = readInternal(fCursor, buffer, limit,
+ (int) actualLen - limit, true);
+ if (bytesRead > 0) {
+ totalBytesRead += bytesRead;
+ limit += bytesRead;
+ fCursor += bytesRead;
+ }
+ }
+ } catch (IOException e) {
+ LOG.debug("Optimized read failed. alling back to readOneBlock {}", e);
Review comment:
Done
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] bilaharith commented on a change in pull request #2464: HADOOP-17347. ABFS: Read optimizations
Posted by GitBox <gi...@apache.org>.
bilaharith commented on a change in pull request #2464:
URL: https://github.com/apache/hadoop/pull/2464#discussion_r539062144
##########
File path: hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/services/ITestAbfsInputStreamReadFooter.java
##########
@@ -0,0 +1,199 @@
+/**
Review comment:
Sure
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] snvijaya commented on a change in pull request #2464: HADOOP-17347. ABFS: Read optimizations
Posted by GitBox <gi...@apache.org>.
snvijaya commented on a change in pull request #2464:
URL: https://github.com/apache/hadoop/pull/2464#discussion_r543887863
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsInputStream.java
##########
@@ -206,11 +231,127 @@ private int readOneBlock(final byte[] b, final int off, final int len) throws IO
fCursor += bytesRead;
fCursorAfterLastRead = fCursor;
}
+ return copyToUserBuffer(b, off, len);
+ }
+ private int readFileCompletely(final byte[] b, final int off, final int len)
+ throws IOException {
+ if (len == 0) {
+ return 0;
+ }
+ if (!validate(b, off, len)) {
+ return -1;
+ }
+ savePointerState();
+
+ buffer = new byte[bufferSize];
+ // data need to be copied to user buffer from index bCursor, bCursor has
+ // to be the current fCusor
+ bCursor = (int) fCursor;
+ return optimisedRead(b, off, len, 0, contentLength);
+ }
+
+ private int readLastBlock(final byte[] b, final int off, final int len)
+ throws IOException {
+ if (len == 0) {
+ return 0;
+ }
+ if (!validate(b, off, len)) {
+ return -1;
+ }
+ savePointerState();
+
+ buffer = new byte[bufferSize];
+ // data need to be copied to user buffer from index bCursor,
+ // AbfsInutStream buffer is going p contain dta from footer start. In
+ // that case bCursor will be set to footerStart - fCursor
+ long lastBlockStart = max(0, contentLength - bufferSize);
+ bCursor = (int) (fCursor - lastBlockStart);
+ // read API call is considered 1 single operation in reality server could
+ // return partial data and client has to retry untill the last full block
+ // is read. So setting the fCursorAfterLastRead before the possible
+ // multiple server calls
+ fCursorAfterLastRead = fCursor;
+ // 0 if contentlength is < buffersize
+ long actualLenToRead = min(bufferSize, contentLength);
+ return optimisedRead(b, off, len, lastBlockStart, actualLenToRead);
+ }
+
+ private int optimisedRead(final byte[] b, final int off, final int len,
+ final long readFrom, final long actualLen) throws IOException {
+ int totalBytesRead = 0;
+ fCursor = readFrom;
+ try {
+ for (int i = 0;
+ i < OPTIMIZATION_ATTEMPTS && fCursor < contentLength; i++) {
Review comment:
Rename OPTIMIZATION_ATTEMPTS to MAX_OPTIMIZED_READ_ATTEMPTS
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] snvijaya commented on a change in pull request #2464: HADOOP-17347. ABFS: Read optimizations
Posted by GitBox <gi...@apache.org>.
snvijaya commented on a change in pull request #2464:
URL: https://github.com/apache/hadoop/pull/2464#discussion_r535129578
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsInputStream.java
##########
@@ -141,7 +154,7 @@ public synchronized int read(final byte[] b, final int off, final int len) throw
return totalReadBytes > 0 ? totalReadBytes : lastReadBytes;
}
- private int readOneBlock(final byte[] b, final int off, final int len) throws IOException {
+ private int validate(byte[] b, int off, int len) throws IOException {
Review comment:
Returning int from validate which is used by caller as bytes read is wrong. Any failure in validate should return Boolean false and the caller in turn can convert it to -1 for bytes read.
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AbfsConfiguration.java
##########
@@ -517,6 +527,14 @@ public int getWriteBufferSize() {
return this.writeBufferSize;
}
+ public boolean readSmallFilesCompletely() {
Review comment:
In that case, skipping readAhead wont be ideal when reads are done by means of this optimizations. if 100 MB is set as buffer size, current logic is forcing a single server read request for 100 MB.
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsInputStream.java
##########
@@ -161,6 +174,14 @@ private int readOneBlock(final byte[] b, final int off, final int len) throws IO
if (off < 0 || len < 0 || len > b.length - off) {
throw new IndexOutOfBoundsException();
}
+ return 1; // 1 indicate success
+ }
+
+ private int readOneBlock(final byte[] b, final int off, final int len) throws IOException {
+ int validation = validate(b, off, len);
+ if (validation < 1) {
+ return validation;
Review comment:
see comment above
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] bilaharith commented on a change in pull request #2464: Draft PR: HADOOP-17347. ABFS: Read optimizations
Posted by GitBox <gi...@apache.org>.
bilaharith commented on a change in pull request #2464:
URL: https://github.com/apache/hadoop/pull/2464#discussion_r533561059
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AbfsConfiguration.java
##########
@@ -517,6 +527,14 @@ public int getWriteBufferSize() {
return this.writeBufferSize;
}
+ public boolean readSmallFilesCompletely() {
Review comment:
It was finalised as the criteria to consider a file small as the size smaller than the buffer size.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #2464: HADOOP-17347. ABFS: Read optimizations
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on pull request #2464:
URL: https://github.com/apache/hadoop/pull/2464#issuecomment-742713971
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 32s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | | 0m 0s | [test4tests](test4tests) | The patch appears to include 2 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 32m 35s | | trunk passed |
| +1 :green_heart: | compile | 0m 38s | | trunk passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 |
| +1 :green_heart: | compile | 0m 34s | | trunk passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| +1 :green_heart: | checkstyle | 0m 27s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 39s | | trunk passed |
| +1 :green_heart: | shadedclient | 16m 29s | | branch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 31s | | trunk passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 |
| +1 :green_heart: | javadoc | 0m 30s | | trunk passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| +0 :ok: | spotbugs | 0m 59s | | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 0m 58s | | trunk passed |
| -0 :warning: | patch | 1m 17s | | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 30s | | the patch passed |
| +1 :green_heart: | compile | 0m 29s | | the patch passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 |
| +1 :green_heart: | javac | 0m 29s | | the patch passed |
| +1 :green_heart: | compile | 0m 25s | | the patch passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| +1 :green_heart: | javac | 0m 25s | | the patch passed |
| +1 :green_heart: | checkstyle | 0m 17s | | the patch passed |
| +1 :green_heart: | mvnsite | 0m 29s | | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | | The patch has no whitespace issues. |
| +1 :green_heart: | shadedclient | 14m 59s | | patch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 25s | | the patch passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 |
| +1 :green_heart: | javadoc | 0m 25s | | the patch passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| +1 :green_heart: | findbugs | 0m 59s | | the patch passed |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 1m 30s | | hadoop-azure in the patch passed. |
| +1 :green_heart: | asflicense | 0m 34s | | The patch does not generate ASF License warnings. |
| | | 77m 7s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2464/12/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/2464 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 323b485c5a8a 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / c2cecfc9b95 |
| Default Java | Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2464/12/testReport/ |
| Max. process+thread count | 729 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2464/12/console |
| versions | git=2.17.1 maven=3.6.0 findbugs=4.0.6 |
| Powered by | Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #2464: HADOOP-17347. ABFS: Read optimizations
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on pull request #2464:
URL: https://github.com/apache/hadoop/pull/2464#issuecomment-743264188
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 26m 13s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | | 0m 0s | [test4tests](test4tests) | The patch appears to include 2 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 32m 43s | | trunk passed |
| +1 :green_heart: | compile | 0m 37s | | trunk passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 |
| +1 :green_heart: | compile | 0m 34s | | trunk passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| +1 :green_heart: | checkstyle | 0m 26s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 39s | | trunk passed |
| +1 :green_heart: | shadedclient | 16m 16s | | branch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 32s | | trunk passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 |
| +1 :green_heart: | javadoc | 0m 29s | | trunk passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| +0 :ok: | spotbugs | 1m 0s | | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 0m 57s | | trunk passed |
| -0 :warning: | patch | 1m 15s | | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 30s | | the patch passed |
| +1 :green_heart: | compile | 0m 30s | | the patch passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 |
| +1 :green_heart: | javac | 0m 30s | | the patch passed |
| +1 :green_heart: | compile | 0m 26s | | the patch passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| +1 :green_heart: | javac | 0m 26s | | the patch passed |
| -0 :warning: | checkstyle | 0m 16s | [/diff-checkstyle-hadoop-tools_hadoop-azure.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2464/13/artifact/out/diff-checkstyle-hadoop-tools_hadoop-azure.txt) | hadoop-tools/hadoop-azure: The patch generated 4 new + 2 unchanged - 0 fixed = 6 total (was 2) |
| +1 :green_heart: | mvnsite | 0m 30s | | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | | The patch has no whitespace issues. |
| +1 :green_heart: | shadedclient | 18m 17s | | patch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 29s | | the patch passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 |
| +1 :green_heart: | javadoc | 0m 27s | | the patch passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| +1 :green_heart: | findbugs | 1m 22s | | the patch passed |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 1m 44s | | hadoop-azure in the patch passed. |
| +1 :green_heart: | asflicense | 0m 33s | | The patch does not generate ASF License warnings. |
| | | 106m 50s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2464/13/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/2464 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 509ebfb7c1e8 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 9bd3c9bc506 |
| Default Java | Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2464/13/testReport/ |
| Max. process+thread count | 671 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2464/13/console |
| versions | git=2.17.1 maven=3.6.0 findbugs=4.0.6 |
| Powered by | Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] bilaharith commented on a change in pull request #2464: Draft PR: HADOOP-17347. ABFS: Read optimizations
Posted by GitBox <gi...@apache.org>.
bilaharith commented on a change in pull request #2464:
URL: https://github.com/apache/hadoop/pull/2464#discussion_r533552825
##########
File path: hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/services/ITestAbfsInputStreamReadFooter.java
##########
@@ -0,0 +1,199 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.azurebfs.services;
+
+import java.io.IOException;
+import java.util.Map;
+import java.util.Random;
+
+import org.junit.Test;
+
+import org.apache.hadoop.fs.FSDataInputStream;
+import org.apache.hadoop.fs.FSDataOutputStream;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.fs.azurebfs.AbfsConfiguration;
+import org.apache.hadoop.fs.azurebfs.AbstractAbfsIntegrationTest;
+import org.apache.hadoop.fs.azurebfs.AzureBlobFileSystem;
+
+import static org.apache.hadoop.fs.azurebfs.AbfsStatistic.CONNECTIONS_MADE;
+import static org.apache.hadoop.fs.azurebfs.constants.FileSystemConfigurations.ONE_KB;
+import static org.apache.hadoop.fs.azurebfs.constants.FileSystemConfigurations.ONE_MB;
+
+public class ITestAbfsInputStreamReadFooter
+ extends AbstractAbfsIntegrationTest {
+
+ private static final int TEN = 10;
+ private static final int TWENTY = 20;
+
+ public ITestAbfsInputStreamReadFooter() throws Exception {
+ }
+
+ @Test
+ public void testOnlyOneServerCallIsMadeWhenTheConfIsTrue() throws Exception {
+ testNumBackendCalls(true);
+ }
+
+ @Test
+ public void testMultipleServerCallsAreMadeWhenTheConfIsFalse()
+ throws Exception {
+ testNumBackendCalls(false);
+ }
+
+ private void testNumBackendCalls(boolean optimizeFooterRead)
+ throws Exception {
+ final AzureBlobFileSystem fs = getFileSystem(optimizeFooterRead);
+ for (int i = 1; i <= 4; i++) {
+ String fileName = methodName.getMethodName() + i;
+ int fileSize = i * ONE_MB;
Review comment:
File sizes are now 2 MB to 6MB
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] snvijaya commented on a change in pull request #2464: HADOOP-17347. ABFS: Read optimizations
Posted by GitBox <gi...@apache.org>.
snvijaya commented on a change in pull request #2464:
URL: https://github.com/apache/hadoop/pull/2464#discussion_r544816563
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsInputStream.java
##########
@@ -206,11 +231,127 @@ private int readOneBlock(final byte[] b, final int off, final int len) throws IO
fCursor += bytesRead;
fCursorAfterLastRead = fCursor;
}
+ return copyToUserBuffer(b, off, len);
+ }
+ private int readFileCompletely(final byte[] b, final int off, final int len)
+ throws IOException {
+ if (len == 0) {
+ return 0;
+ }
+ if (!validate(b, off, len)) {
+ return -1;
+ }
+ savePointerState();
+
+ buffer = new byte[bufferSize];
+ // data need to be copied to user buffer from index bCursor, bCursor has
+ // to be the current fCusor
+ bCursor = (int) fCursor;
+ return optimisedRead(b, off, len, 0, contentLength);
+ }
+
+ private int readLastBlock(final byte[] b, final int off, final int len)
+ throws IOException {
+ if (len == 0) {
+ return 0;
+ }
+ if (!validate(b, off, len)) {
+ return -1;
+ }
+ savePointerState();
+
+ buffer = new byte[bufferSize];
+ // data need to be copied to user buffer from index bCursor,
+ // AbfsInutStream buffer is going p contain dta from footer start. In
+ // that case bCursor will be set to footerStart - fCursor
+ long lastBlockStart = max(0, contentLength - bufferSize);
+ bCursor = (int) (fCursor - lastBlockStart);
+ // read API call is considered 1 single operation in reality server could
+ // return partial data and client has to retry untill the last full block
+ // is read. So setting the fCursorAfterLastRead before the possible
+ // multiple server calls
+ fCursorAfterLastRead = fCursor;
+ // 0 if contentlength is < buffersize
+ long actualLenToRead = min(bufferSize, contentLength);
+ return optimisedRead(b, off, len, lastBlockStart, actualLenToRead);
+ }
+
+ private int optimisedRead(final byte[] b, final int off, final int len,
+ final long readFrom, final long actualLen) throws IOException {
+ int totalBytesRead = 0;
+ fCursor = readFrom;
+ try {
+ for (int i = 0;
+ i < OPTIMIZATION_ATTEMPTS && fCursor < contentLength; i++) {
+ int bytesRead = readInternal(fCursor, buffer, limit,
+ (int) actualLen - limit, true);
+ if (bytesRead > 0) {
+ totalBytesRead += bytesRead;
+ limit += bytesRead;
+ fCursor += bytesRead;
+ }
+ }
+ } catch (IOException e) {
+ LOG.debug("Optimized read failed. alling back to readOneBlock {}", e);
Review comment:
Please check again. Change not reflecting.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] bilaharith commented on pull request #2464: HADOOP-17347. ABFS: Read optimizations
Posted by GitBox <gi...@apache.org>.
bilaharith commented on pull request #2464:
URL: https://github.com/apache/hadoop/pull/2464#issuecomment-741548744
HNS-OAuth
========================
[INFO] Results:
[INFO]
[INFO] Tests run: 90, Failures: 0, Errors: 0, Skipped: 0
[INFO] Results:
[INFO]
[WARNING] Tests run: 478, Failures: 0, Errors: 0, Skipped: 66
[INFO] Results:
[INFO]
[WARNING] Tests run: 208, Failures: 0, Errors: 0, Skipped: 24
HNS-SharedKey
========================
[INFO] Results:
[INFO]
[INFO] Tests run: 90, Failures: 0, Errors: 0, Skipped: 0
[INFO] Results:
[INFO]
[WARNING] Tests run: 478, Failures: 0, Errors: 0, Skipped: 24
[INFO] Results:
[INFO]
[WARNING] Tests run: 208, Failures: 0, Errors: 0, Skipped: 16
NonHNS-SharedKey
========================
[INFO] Results:
[INFO]
[INFO] Tests run: 90, Failures: 0, Errors: 0, Skipped: 0
[INFO] Results:
[INFO]
[WARNING] Tests run: 478, Failures: 0, Errors: 0, Skipped: 247
[INFO] Results:
[INFO]
[WARNING] Tests run: 208, Failures: 0, Errors: 0, Skipped: 16
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] snvijaya commented on a change in pull request #2464: HADOOP-17347. ABFS: Read optimizations
Posted by GitBox <gi...@apache.org>.
snvijaya commented on a change in pull request #2464:
URL: https://github.com/apache/hadoop/pull/2464#discussion_r544827644
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsInputStream.java
##########
@@ -211,6 +235,59 @@ private int readOneBlock(final byte[] b, final int off, final int len) throws IO
return bytesToRead;
}
+ private boolean shouldReadFully() {
+ return this.firstRead && this.context.readSmallFilesCompletely()
+ && this.contentLength <= this.bufferSize;
+ }
+
+ private boolean shouldReadLastBlock(int len) {
+ return this.firstRead && this.context.optimizeFooterRead()
+ && len == FOOTER_SIZE
+ && this.fCursor == this.contentLength - FOOTER_SIZE;
+ }
+
+ private int readFileCompletely(final byte[] b, final int off, final int len) throws IOException {
+ int validation = validate(b, off, len);
+ if (validation < 1) {
+ return validation;
+ }
+
+ buffer = new byte[bufferSize];
+ bCursor = (int) fCursor;
+ int bytesRead = readInternal(0, buffer, 0, (int) contentLength, true);
+ firstRead = false;
Review comment:
Testcases needed for fault handling.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] bilaharith commented on a change in pull request #2464: Draft PR for read optimizations
Posted by GitBox <gi...@apache.org>.
bilaharith commented on a change in pull request #2464:
URL: https://github.com/apache/hadoop/pull/2464#discussion_r526588250
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsInputStream.java
##########
@@ -162,14 +172,86 @@ private int readOneBlock(final byte[] b, final int off, final int len) throws IO
throw new IndexOutOfBoundsException();
}
- //If buffer is empty, then fill the buffer.
- if (bCursor == limit) {
+ if (firstRead && this.readSmallFilesCompletely
+ && contentLength <= bufferSize) { // Read small files completely
+ if (readFileCompletely() == -1) {
+ return -1;
+ }
+ } else if (firstRead && this.optimizeFooterRead && fCursor
+ == contentLength - FOOTER_DELTA) { // Read the last one block if the
+ // read is for the footer
+ if (readLastBlock() == -1) {
+ return -1;
+ }
+ } else {
+ if (readOneBlock(b) == -1) {
+ return -1;
+ }
+ }
+ fCursorAfterLastRead = fCursor;
+
+ //If there is anything in the buffer, then return lesser of (requested bytes) and (bytes in buffer)
+ //(bytes returned may be less than requested)
+ int bytesRemaining = limit - bCursor;
+ int bytesToRead = Math.min(len, bytesRemaining);
+ System.arraycopy(buffer, bCursor, b, off, bytesToRead);
+ bCursor += bytesToRead;
+ if (statistics != null) {
+ statistics.incrementBytesRead(bytesToRead);
+ }
+ if (streamStatistics != null) {
+ // Bytes read from the local buffer.
+ streamStatistics.bytesReadFromBuffer(bytesToRead);
+ streamStatistics.bytesRead(bytesToRead);
+ }
+ return bytesToRead;
+ }
+
+ private long readFileCompletely() throws IOException {
+ bCursor = (int) getPos();
Review comment:
Done
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] bilaharith commented on a change in pull request #2464: Draft PR for read optimizations
Posted by GitBox <gi...@apache.org>.
bilaharith commented on a change in pull request #2464:
URL: https://github.com/apache/hadoop/pull/2464#discussion_r529216077
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsInputStream.java
##########
@@ -162,14 +169,87 @@ private int readOneBlock(final byte[] b, final int off, final int len) throws IO
throw new IndexOutOfBoundsException();
}
- //If buffer is empty, then fill the buffer.
- if (bCursor == limit) {
+ int bytesRead = 0;
+ if (shouldReadFully()) {
+ bytesRead = readFileCompletely();
+ } else if (shouldReadLastBlock(len)) {
+ bytesRead = readLastBlock();
+ } else {
+ bytesRead = readOneBlock(b);
+ }
+ if (bytesRead == -1) {
+ return -1;
+ }
+ fCursorAfterLastRead = fCursor;
Review comment:
move above the if check
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] bilaharith commented on a change in pull request #2464: HADOOP-17347. ABFS: Read optimizations
Posted by GitBox <gi...@apache.org>.
bilaharith commented on a change in pull request #2464:
URL: https://github.com/apache/hadoop/pull/2464#discussion_r543156538
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsInputStream.java
##########
@@ -206,11 +229,120 @@ private int readOneBlock(final byte[] b, final int off, final int len) throws IO
fCursor += bytesRead;
fCursorAfterLastRead = fCursor;
}
+ return copyToUserBuffer(b, off, len);
+ }
+
+ private int readFileCompletely(final byte[] b, final int off, final int len)
+ throws IOException {
+ if (len == 0) {
+ return 0;
+ }
+ if (!validate(b, off, len)) {
+ return -1;
+ }
+ savePointerState();
+
+ buffer = new byte[bufferSize];
+ // data need to be copied to user buffer from index bCursor, bCursor has
+ // to be the current fCusor
+ bCursor = (int) fCursor;
+ return optimisedRead(b, off, len, 0, contentLength);
+ }
+
+ private int readLastBlock(final byte[] b, final int off, final int len)
+ throws IOException {
+ if (len == 0) {
+ return 0;
+ }
+ if (!validate(b, off, len)) {
+ return -1;
+ }
+ savePointerState();
+
+ buffer = new byte[bufferSize];
+ // data need to be copied to user buffer from index bCursor,
+ // AbfsInutStream buffer is going p contain dta from footer start. In
+ // that case bCursor will be set to footerStart - fCursor
+ long lastBlockStart = max(0, contentLength - bufferSize);
+ bCursor = (int) (fCursor - lastBlockStart);
+ // read API call is considered 1 single operation in reality server could
+ // return partial data and client has to retry untill the last full block
+ // is read. So setting the fCursorAfterLastRead before the possible
+ // multiple server calls
+ fCursorAfterLastRead = fCursor;
+ // 0 if contentlength is < buffersize
+ long actualLenToRead = min(bufferSize, contentLength);
+ return optimisedRead(b, off, len, lastBlockStart, actualLenToRead);
+ }
+
+ private int optimisedRead(final byte[] b, final int off, final int len,
+ final long readFrom, final long actualLen) throws IOException {
+ int totalBytesRead = 0;
+ fCursor = readFrom;
+ for (int i = 0; i < 2 && fCursor < contentLength; i++) {
Review comment:
Done
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] snvijaya commented on a change in pull request #2464: HADOOP-17347. ABFS: Read optimizations
Posted by GitBox <gi...@apache.org>.
snvijaya commented on a change in pull request #2464:
URL: https://github.com/apache/hadoop/pull/2464#discussion_r544816586
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsInputStream.java
##########
@@ -206,11 +231,127 @@ private int readOneBlock(final byte[] b, final int off, final int len) throws IO
fCursor += bytesRead;
fCursorAfterLastRead = fCursor;
}
+ return copyToUserBuffer(b, off, len);
+ }
+ private int readFileCompletely(final byte[] b, final int off, final int len)
+ throws IOException {
+ if (len == 0) {
+ return 0;
+ }
+ if (!validate(b, off, len)) {
+ return -1;
+ }
+ savePointerState();
+
+ buffer = new byte[bufferSize];
+ // data need to be copied to user buffer from index bCursor, bCursor has
+ // to be the current fCusor
+ bCursor = (int) fCursor;
+ return optimisedRead(b, off, len, 0, contentLength);
+ }
+
+ private int readLastBlock(final byte[] b, final int off, final int len)
+ throws IOException {
+ if (len == 0) {
+ return 0;
+ }
+ if (!validate(b, off, len)) {
+ return -1;
+ }
+ savePointerState();
+
+ buffer = new byte[bufferSize];
+ // data need to be copied to user buffer from index bCursor,
+ // AbfsInutStream buffer is going p contain dta from footer start. In
+ // that case bCursor will be set to footerStart - fCursor
+ long lastBlockStart = max(0, contentLength - bufferSize);
+ bCursor = (int) (fCursor - lastBlockStart);
+ // read API call is considered 1 single operation in reality server could
+ // return partial data and client has to retry untill the last full block
+ // is read. So setting the fCursorAfterLastRead before the possible
+ // multiple server calls
+ fCursorAfterLastRead = fCursor;
+ // 0 if contentlength is < buffersize
+ long actualLenToRead = min(bufferSize, contentLength);
+ return optimisedRead(b, off, len, lastBlockStart, actualLenToRead);
+ }
+
+ private int optimisedRead(final byte[] b, final int off, final int len,
+ final long readFrom, final long actualLen) throws IOException {
+ int totalBytesRead = 0;
+ fCursor = readFrom;
+ try {
+ for (int i = 0;
+ i < OPTIMIZATION_ATTEMPTS && fCursor < contentLength; i++) {
Review comment:
Please check again. Change not reflecting.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] bilaharith commented on a change in pull request #2464: HADOOP-17347. ABFS: Read optimizations
Posted by GitBox <gi...@apache.org>.
bilaharith commented on a change in pull request #2464:
URL: https://github.com/apache/hadoop/pull/2464#discussion_r540268718
##########
File path: hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/services/ITestAbfsInputStreamReadFooter.java
##########
@@ -0,0 +1,199 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.azurebfs.services;
+
+import java.io.IOException;
+import java.util.Map;
+import java.util.Random;
+
+import org.junit.Test;
+
+import org.apache.hadoop.fs.FSDataInputStream;
+import org.apache.hadoop.fs.FSDataOutputStream;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.fs.azurebfs.AbfsConfiguration;
+import org.apache.hadoop.fs.azurebfs.AbstractAbfsIntegrationTest;
+import org.apache.hadoop.fs.azurebfs.AzureBlobFileSystem;
+
+import static org.apache.hadoop.fs.azurebfs.AbfsStatistic.CONNECTIONS_MADE;
+import static org.apache.hadoop.fs.azurebfs.constants.FileSystemConfigurations.ONE_KB;
+import static org.apache.hadoop.fs.azurebfs.constants.FileSystemConfigurations.ONE_MB;
+
+public class ITestAbfsInputStreamReadFooter
+ extends AbstractAbfsIntegrationTest {
+
+ private static final int TEN = 10;
+ private static final int TWENTY = 20;
+
+ public ITestAbfsInputStreamReadFooter() throws Exception {
+ }
+
+ @Test
+ public void testOnlyOneServerCallIsMadeWhenTheConfIsTrue() throws Exception {
+ testNumBackendCalls(true);
+ }
+
+ @Test
+ public void testMultipleServerCallsAreMadeWhenTheConfIsFalse()
+ throws Exception {
+ testNumBackendCalls(false);
+ }
+
+ private void testNumBackendCalls(boolean optimizeFooterRead)
+ throws Exception {
+ final AzureBlobFileSystem fs = getFileSystem(optimizeFooterRead);
+ for (int i = 1; i <= 4; i++) {
+ String fileName = methodName.getMethodName() + i;
+ int fileSize = i * ONE_MB;
+ byte[] fileContent = getRandomBytesArray(fileSize);
+ Path testFilePath = createFileWithContent(fs, fileName, fileContent);
+ int length = AbfsInputStream.FOOTER_SIZE;
+ try (FSDataInputStream iStream = fs.open(testFilePath)) {
+ byte[] buffer = new byte[length];
+
+ Map<String, Long> metricMap = fs.getInstrumentationMap();
+ long requestsMadeBeforeTest = metricMap
+ .get(CONNECTIONS_MADE.getStatName());
+
+ iStream.seek(fileSize - 8);
+ iStream.read(buffer, 0, length);
+
+ iStream.seek(fileSize - (TEN * ONE_KB));
+ iStream.read(buffer, 0, length);
+
+ iStream.seek(fileSize - (TWENTY * ONE_KB));
+ iStream.read(buffer, 0, length);
+
+ metricMap = fs.getInstrumentationMap();
+ long requestsMadeAfterTest = metricMap
+ .get(CONNECTIONS_MADE.getStatName());
+
+ if (optimizeFooterRead) {
+ assertEquals(1, requestsMadeAfterTest - requestsMadeBeforeTest);
+ } else {
+ assertEquals(3, requestsMadeAfterTest - requestsMadeBeforeTest);
+ }
+ }
+ }
+ }
+
+ @Test
+ public void testSeekToEndAndReadWithConfTrue() throws Exception {
+ testSeekToEndAndReadWithConf(true);
+ }
+
+ @Test
+ public void testSeekToEndAndReadWithConfFalse() throws Exception {
+ testSeekToEndAndReadWithConf(false);
+ }
+
+ private void testSeekToEndAndReadWithConf(boolean optimizeFooterRead) throws Exception {
+ final AzureBlobFileSystem fs = getFileSystem(optimizeFooterRead);
+ for (int i = 5; i <= 10; i++) {
+ String fileName = methodName.getMethodName() + i;
+ int fileSize = i * ONE_MB;
+ byte[] fileContent = getRandomBytesArray(fileSize);
+ Path testFilePath = createFileWithContent(fs, fileName, fileContent);
+ seekReadAndTest(fs, testFilePath, fileSize - AbfsInputStream.FOOTER_SIZE,
+ AbfsInputStream.FOOTER_SIZE, fileContent);
+ }
+ }
+
+ private AzureBlobFileSystem getFileSystem(boolean optimizeFooterRead)
+ throws IOException {
+ final AzureBlobFileSystem fs = getFileSystem();
+ fs.getAbfsStore().getAbfsConfiguration()
+ .setOptimizeFooterRead(optimizeFooterRead);
+ fs.getAbfsStore().getAbfsConfiguration()
+ .setReadSmallFilesCompletely(false);
Review comment:
For small files this needs to be test in order to test the footer read.
For big files this is not set and expected to skip the smallfile read if condition.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] snvijaya commented on a change in pull request #2464: HADOOP-17347. ABFS: Read optimizations
Posted by GitBox <gi...@apache.org>.
snvijaya commented on a change in pull request #2464:
URL: https://github.com/apache/hadoop/pull/2464#discussion_r544826406
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsInputStream.java
##########
@@ -206,11 +231,121 @@ private int readOneBlock(final byte[] b, final int off, final int len) throws IO
fCursor += bytesRead;
fCursorAfterLastRead = fCursor;
}
+ return copyToUserBuffer(b, off, len);
+ }
+
+ private int readFileCompletely(final byte[] b, final int off, final int len)
+ throws IOException {
+ if (len == 0) {
+ return 0;
+ }
+ if (!validate(b, off, len)) {
+ return -1;
+ }
+ savePointerState();
+ // data need to be copied to user buffer from index bCursor, bCursor has
+ // to be the current fCusor
+ bCursor = (int) fCursor;
+ return optimisedRead(b, off, len, 0, contentLength);
+ }
+
+ private int readLastBlock(final byte[] b, final int off, final int len)
+ throws IOException {
+ if (len == 0) {
+ return 0;
+ }
+ if (!validate(b, off, len)) {
+ return -1;
+ }
+ savePointerState();
+ // data need to be copied to user buffer from index bCursor,
+ // AbfsInutStream buffer is going to contain data from last block start. In
+ // that case bCursor will be set to fCursor - lastBlockStart
+ long lastBlockStart = max(0, contentLength - bufferSize);
+ bCursor = (int) (fCursor - lastBlockStart);
+ // 0 if contentlength is < buffersize
+ long actualLenToRead = min(bufferSize, contentLength);
+ return optimisedRead(b, off, len, lastBlockStart, actualLenToRead);
+ }
+
+ private int optimisedRead(final byte[] b, final int off, final int len,
+ final long readFrom, final long actualLen) throws IOException {
+ fCursor = readFrom;
+ int totalBytesRead = 0;
+ int lastBytesRead = 0;
+ try {
+ buffer = new byte[bufferSize];
+ for (int i = 0;
+ i < MAX_OPTIMIZED_READ_ATTEMPTS && fCursor < contentLength; i++) {
+ lastBytesRead = readInternal(fCursor, buffer, limit,
+ (int) actualLen - limit, true);
+ if (lastBytesRead > 0) {
+ totalBytesRead += lastBytesRead;
+ limit += lastBytesRead;
+ fCursor += lastBytesRead;
+ fCursorAfterLastRead = fCursor;
+ }
+ }
+ } catch (IOException e) {
+ LOG.debug("Optimized read failed. Defaulting to readOneBlock {}", e);
+ restorePointerState();
+ return readOneBlock(b, off, len);
+ }
+ firstRead = false;
+ if (totalBytesRead < 1) {
+ return lastBytesRead;
Review comment:
Restore needed in case of no read to get the file pointer to where app had set it earlier. Also return explicit -1 as it is a no data read case.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] vinaysbadami commented on a change in pull request #2464: Draft PR for read optimizations
Posted by GitBox <gi...@apache.org>.
vinaysbadami commented on a change in pull request #2464:
URL: https://github.com/apache/hadoop/pull/2464#discussion_r523919070
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsInputStream.java
##########
@@ -162,14 +172,86 @@ private int readOneBlock(final byte[] b, final int off, final int len) throws IO
throw new IndexOutOfBoundsException();
}
- //If buffer is empty, then fill the buffer.
- if (bCursor == limit) {
+ if (firstRead && this.readSmallFilesCompletely
Review comment:
if (firstRead) {
if (smallfilecase)
{
res = readfilecompletelty;
firstread = false;
}
else if (readfootercase)
{
res = readfooter
firstread = false
}
}
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #2464: Draft PR: HADOOP-17347. ABFS: Read optimizations
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on pull request #2464:
URL: https://github.com/apache/hadoop/pull/2464#issuecomment-736749031
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 49m 57s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | | 0m 0s | [test4tests](test4tests) | The patch appears to include 2 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 34m 9s | | trunk passed |
| +1 :green_heart: | compile | 0m 38s | | trunk passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 |
| +1 :green_heart: | compile | 0m 34s | | trunk passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| +1 :green_heart: | checkstyle | 0m 27s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 39s | | trunk passed |
| +1 :green_heart: | shadedclient | 16m 20s | | branch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 31s | | trunk passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 |
| +1 :green_heart: | javadoc | 0m 29s | | trunk passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| +0 :ok: | spotbugs | 0m 58s | | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 0m 56s | | trunk passed |
| -0 :warning: | patch | 1m 15s | | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 30s | | the patch passed |
| +1 :green_heart: | compile | 0m 30s | | the patch passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 |
| +1 :green_heart: | javac | 0m 30s | | the patch passed |
| +1 :green_heart: | compile | 0m 26s | | the patch passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| +1 :green_heart: | javac | 0m 26s | | the patch passed |
| +1 :green_heart: | checkstyle | 0m 17s | | the patch passed |
| +1 :green_heart: | mvnsite | 0m 29s | | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | | The patch has no whitespace issues. |
| +1 :green_heart: | shadedclient | 14m 51s | | patch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 26s | | the patch passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 |
| +1 :green_heart: | javadoc | 0m 24s | | the patch passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| +1 :green_heart: | findbugs | 1m 0s | | the patch passed |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 1m 28s | | hadoop-azure in the patch passed. |
| +1 :green_heart: | asflicense | 0m 33s | | The patch does not generate ASF License warnings. |
| | | 127m 50s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2464/9/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/2464 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux bc7b1e4ca565 4.15.0-65-generic #74-Ubuntu SMP Tue Sep 17 17:06:04 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 6a1d7d9ed25 |
| Default Java | Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2464/9/testReport/ |
| Max. process+thread count | 579 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2464/9/console |
| versions | git=2.17.1 maven=3.6.0 findbugs=4.0.6 |
| Powered by | Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] vinaysbadami commented on a change in pull request #2464: HADOOP-17347. ABFS: Read optimizations
Posted by GitBox <gi...@apache.org>.
vinaysbadami commented on a change in pull request #2464:
URL: https://github.com/apache/hadoop/pull/2464#discussion_r544133985
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsInputStream.java
##########
@@ -38,6 +38,10 @@
import org.apache.hadoop.fs.azurebfs.contracts.exceptions.AzureBlobFileSystemException;
import org.apache.hadoop.fs.azurebfs.utils.CachedSASToken;
+import static java.lang.Math.max;
Review comment:
any reason for not impoting java.lang.math?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] bilaharith commented on a change in pull request #2464: HADOOP-17347. ABFS: Read optimizations
Posted by GitBox <gi...@apache.org>.
bilaharith commented on a change in pull request #2464:
URL: https://github.com/apache/hadoop/pull/2464#discussion_r544175765
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsInputStream.java
##########
@@ -206,11 +231,127 @@ private int readOneBlock(final byte[] b, final int off, final int len) throws IO
fCursor += bytesRead;
fCursorAfterLastRead = fCursor;
}
+ return copyToUserBuffer(b, off, len);
+ }
+ private int readFileCompletely(final byte[] b, final int off, final int len)
+ throws IOException {
+ if (len == 0) {
+ return 0;
+ }
+ if (!validate(b, off, len)) {
+ return -1;
+ }
+ savePointerState();
+
+ buffer = new byte[bufferSize];
+ // data need to be copied to user buffer from index bCursor, bCursor has
+ // to be the current fCusor
+ bCursor = (int) fCursor;
+ return optimisedRead(b, off, len, 0, contentLength);
+ }
+
+ private int readLastBlock(final byte[] b, final int off, final int len)
+ throws IOException {
+ if (len == 0) {
+ return 0;
+ }
+ if (!validate(b, off, len)) {
+ return -1;
+ }
+ savePointerState();
+
+ buffer = new byte[bufferSize];
+ // data need to be copied to user buffer from index bCursor,
+ // AbfsInutStream buffer is going p contain dta from footer start. In
+ // that case bCursor will be set to footerStart - fCursor
+ long lastBlockStart = max(0, contentLength - bufferSize);
+ bCursor = (int) (fCursor - lastBlockStart);
+ // read API call is considered 1 single operation in reality server could
+ // return partial data and client has to retry untill the last full block
+ // is read. So setting the fCursorAfterLastRead before the possible
+ // multiple server calls
+ fCursorAfterLastRead = fCursor;
+ // 0 if contentlength is < buffersize
+ long actualLenToRead = min(bufferSize, contentLength);
+ return optimisedRead(b, off, len, lastBlockStart, actualLenToRead);
+ }
+
+ private int optimisedRead(final byte[] b, final int off, final int len,
+ final long readFrom, final long actualLen) throws IOException {
+ int totalBytesRead = 0;
+ fCursor = readFrom;
+ try {
Review comment:
Done
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] bilaharith commented on a change in pull request #2464: Draft PR for read optimizations
Posted by GitBox <gi...@apache.org>.
bilaharith commented on a change in pull request #2464:
URL: https://github.com/apache/hadoop/pull/2464#discussion_r526583785
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/constants/FileSystemConfigurations.java
##########
@@ -50,13 +50,15 @@
public static final int DEFAULT_AZURE_OAUTH_TOKEN_FETCH_RETRY_MAX_BACKOFF_INTERVAL = SIXTY_SECONDS;
public static final int DEFAULT_AZURE_OAUTH_TOKEN_FETCH_RETRY_DELTA_BACKOFF = 2;
- private static final int ONE_KB = 1024;
- private static final int ONE_MB = ONE_KB * ONE_KB;
+ public static final int ONE_KB = 1024;
+ public static final int ONE_MB = ONE_KB * ONE_KB;
// Default upload and download buffer size
public static final int DEFAULT_WRITE_BUFFER_SIZE = 8 * ONE_MB; // 8 MB
public static final int APPENDBLOB_MAX_WRITE_BUFFER_SIZE = 4 * ONE_MB; // 4 MB
public static final int DEFAULT_READ_BUFFER_SIZE = 4 * ONE_MB; // 4 MB
+ public static final boolean DEFAULT_READ_SMALL_FILES_COMPLETELY = true;
+ public static final boolean DEFAULT_OPTIMIZE_FOOTER_READ = true;
Review comment:
Done
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #2464: HADOOP-17347. ABFS: Read optimizations
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on pull request #2464:
URL: https://github.com/apache/hadoop/pull/2464#issuecomment-750752035
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 36s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | | 0m 0s | [test4tests](test4tests) | The patch appears to include 3 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 34m 32s | | trunk passed |
| +1 :green_heart: | compile | 0m 36s | | trunk passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 |
| +1 :green_heart: | compile | 0m 32s | | trunk passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| +1 :green_heart: | checkstyle | 0m 24s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 38s | | trunk passed |
| +1 :green_heart: | shadedclient | 16m 19s | | branch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 34s | | trunk passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 |
| +1 :green_heart: | javadoc | 0m 27s | | trunk passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| +0 :ok: | spotbugs | 1m 1s | | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 0m 59s | | trunk passed |
| -0 :warning: | patch | 1m 20s | | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 31s | | the patch passed |
| +1 :green_heart: | compile | 0m 30s | | the patch passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 |
| +1 :green_heart: | javac | 0m 30s | | the patch passed |
| +1 :green_heart: | compile | 0m 28s | | the patch passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| +1 :green_heart: | javac | 0m 28s | | the patch passed |
| +1 :green_heart: | checkstyle | 0m 18s | | the patch passed |
| +1 :green_heart: | mvnsite | 0m 30s | | the patch passed |
| -1 :x: | whitespace | 0m 0s | [/whitespace-eol.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2464/19/artifact/out/whitespace-eol.txt) | The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply |
| +1 :green_heart: | shadedclient | 14m 36s | | patch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 27s | | the patch passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 |
| +1 :green_heart: | javadoc | 0m 22s | | the patch passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| +1 :green_heart: | findbugs | 1m 3s | | the patch passed |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 1m 28s | | hadoop-azure in the patch passed. |
| +1 :green_heart: | asflicense | 0m 33s | | The patch does not generate ASF License warnings. |
| | | 78m 33s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2464/19/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/2464 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 3f39b5953a53 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 16a20503cac |
| Default Java | Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2464/19/testReport/ |
| Max. process+thread count | 689 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2464/19/console |
| versions | git=2.17.1 maven=3.6.0 findbugs=4.0.6 |
| Powered by | Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] bilaharith commented on a change in pull request #2464: Draft PR: HADOOP-17347. ABFS: Read optimizations
Posted by GitBox <gi...@apache.org>.
bilaharith commented on a change in pull request #2464:
URL: https://github.com/apache/hadoop/pull/2464#discussion_r533551769
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsInputStream.java
##########
@@ -161,6 +174,14 @@ private int readOneBlock(final byte[] b, final int off, final int len) throws IO
if (off < 0 || len < 0 || len > b.length - off) {
throw new IndexOutOfBoundsException();
}
+ return 1; // 1 indicate success
+ }
+
+ private int readOneBlock(final byte[] b, final int off, final int len) throws IOException {
+ int validation = validate(b, off, len);
+ if (validation < 1) {
+ return validation;
Review comment:
That would alter the existing flow. In certain cases it has to return 0 and -1 in some other cases, exceptions in case validations failed for len, off values.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] bilaharith commented on a change in pull request #2464: HADOOP-17347. ABFS: Read optimizations
Posted by GitBox <gi...@apache.org>.
bilaharith commented on a change in pull request #2464:
URL: https://github.com/apache/hadoop/pull/2464#discussion_r543156111
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsInputStream.java
##########
@@ -46,6 +50,7 @@
public class AbfsInputStream extends FSInputStream implements CanUnbuffer,
StreamCapabilities {
private static final Logger LOG = LoggerFactory.getLogger(AbfsInputStream.class);
+ public static final int FOOTER_SIZE = 16 * ONE_KB;
Review comment:
Done
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] bilaharith commented on a change in pull request #2464: Draft PR for read optimizations
Posted by GitBox <gi...@apache.org>.
bilaharith commented on a change in pull request #2464:
URL: https://github.com/apache/hadoop/pull/2464#discussion_r526582998
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsInputStream.java
##########
@@ -128,7 +138,7 @@ public synchronized int read(final byte[] b, final int off, final int len) throw
}
incrementReadOps();
do {
- lastReadBytes = readOneBlock(b, currentOff, currentLen);
+ lastReadBytes = readToUserBuffer(b, currentOff, currentLen);
Review comment:
I have forked 3 different methods from the existing ethod readOneBlock, readFileCompletely. readLastBlock and readOneBlock. There are common parts from the existing method readOneBlock is used from for all the 3 forked methods. In that case renaming the existing method to readToUserBuffer would make sense.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] bilaharith commented on a change in pull request #2464: HADOOP-17347. ABFS: Read optimizations
Posted by GitBox <gi...@apache.org>.
bilaharith commented on a change in pull request #2464:
URL: https://github.com/apache/hadoop/pull/2464#discussion_r544176541
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsInputStream.java
##########
@@ -206,11 +231,127 @@ private int readOneBlock(final byte[] b, final int off, final int len) throws IO
fCursor += bytesRead;
fCursorAfterLastRead = fCursor;
}
+ return copyToUserBuffer(b, off, len);
+ }
+ private int readFileCompletely(final byte[] b, final int off, final int len)
+ throws IOException {
+ if (len == 0) {
+ return 0;
+ }
+ if (!validate(b, off, len)) {
+ return -1;
+ }
+ savePointerState();
+
+ buffer = new byte[bufferSize];
+ // data need to be copied to user buffer from index bCursor, bCursor has
+ // to be the current fCusor
+ bCursor = (int) fCursor;
+ return optimisedRead(b, off, len, 0, contentLength);
+ }
+
+ private int readLastBlock(final byte[] b, final int off, final int len)
+ throws IOException {
+ if (len == 0) {
+ return 0;
+ }
+ if (!validate(b, off, len)) {
+ return -1;
+ }
+ savePointerState();
+
+ buffer = new byte[bufferSize];
+ // data need to be copied to user buffer from index bCursor,
+ // AbfsInutStream buffer is going p contain dta from footer start. In
+ // that case bCursor will be set to footerStart - fCursor
+ long lastBlockStart = max(0, contentLength - bufferSize);
+ bCursor = (int) (fCursor - lastBlockStart);
+ // read API call is considered 1 single operation in reality server could
+ // return partial data and client has to retry untill the last full block
+ // is read. So setting the fCursorAfterLastRead before the possible
+ // multiple server calls
+ fCursorAfterLastRead = fCursor;
+ // 0 if contentlength is < buffersize
+ long actualLenToRead = min(bufferSize, contentLength);
+ return optimisedRead(b, off, len, lastBlockStart, actualLenToRead);
+ }
+
+ private int optimisedRead(final byte[] b, final int off, final int len,
+ final long readFrom, final long actualLen) throws IOException {
+ int totalBytesRead = 0;
+ fCursor = readFrom;
+ try {
+ for (int i = 0;
+ i < OPTIMIZATION_ATTEMPTS && fCursor < contentLength; i++) {
Review comment:
Done
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] snvijaya commented on a change in pull request #2464: HADOOP-17347. ABFS: Read optimizations
Posted by GitBox <gi...@apache.org>.
snvijaya commented on a change in pull request #2464:
URL: https://github.com/apache/hadoop/pull/2464#discussion_r543899390
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsInputStream.java
##########
@@ -206,11 +231,127 @@ private int readOneBlock(final byte[] b, final int off, final int len) throws IO
fCursor += bytesRead;
fCursorAfterLastRead = fCursor;
}
+ return copyToUserBuffer(b, off, len);
+ }
+ private int readFileCompletely(final byte[] b, final int off, final int len)
+ throws IOException {
+ if (len == 0) {
+ return 0;
+ }
+ if (!validate(b, off, len)) {
+ return -1;
+ }
+ savePointerState();
+
+ buffer = new byte[bufferSize];
+ // data need to be copied to user buffer from index bCursor, bCursor has
+ // to be the current fCusor
+ bCursor = (int) fCursor;
+ return optimisedRead(b, off, len, 0, contentLength);
+ }
+
+ private int readLastBlock(final byte[] b, final int off, final int len)
+ throws IOException {
+ if (len == 0) {
+ return 0;
+ }
+ if (!validate(b, off, len)) {
+ return -1;
+ }
+ savePointerState();
+
+ buffer = new byte[bufferSize];
+ // data need to be copied to user buffer from index bCursor,
+ // AbfsInutStream buffer is going p contain dta from footer start. In
+ // that case bCursor will be set to footerStart - fCursor
+ long lastBlockStart = max(0, contentLength - bufferSize);
+ bCursor = (int) (fCursor - lastBlockStart);
+ // read API call is considered 1 single operation in reality server could
+ // return partial data and client has to retry untill the last full block
+ // is read. So setting the fCursorAfterLastRead before the possible
+ // multiple server calls
+ fCursorAfterLastRead = fCursor;
+ // 0 if contentlength is < buffersize
+ long actualLenToRead = min(bufferSize, contentLength);
+ return optimisedRead(b, off, len, lastBlockStart, actualLenToRead);
+ }
+
+ private int optimisedRead(final byte[] b, final int off, final int len,
+ final long readFrom, final long actualLen) throws IOException {
+ int totalBytesRead = 0;
+ fCursor = readFrom;
+ try {
+ for (int i = 0;
+ i < OPTIMIZATION_ATTEMPTS && fCursor < contentLength; i++) {
+ int bytesRead = readInternal(fCursor, buffer, limit,
+ (int) actualLen - limit, true);
+ if (bytesRead > 0) {
+ totalBytesRead += bytesRead;
+ limit += bytesRead;
+ fCursor += bytesRead;
+ }
+ }
+ } catch (IOException e) {
+ LOG.debug("Optimized read failed. alling back to readOneBlock {}", e);
Review comment:
"Optimized read failed. Defaulting to readOneBlock"
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] vinaysbadami commented on a change in pull request #2464: Draft PR for read optimizations
Posted by GitBox <gi...@apache.org>.
vinaysbadami commented on a change in pull request #2464:
URL: https://github.com/apache/hadoop/pull/2464#discussion_r523922208
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsInputStream.java
##########
@@ -162,14 +172,86 @@ private int readOneBlock(final byte[] b, final int off, final int len) throws IO
throw new IndexOutOfBoundsException();
}
- //If buffer is empty, then fill the buffer.
- if (bCursor == limit) {
+ if (firstRead && this.readSmallFilesCompletely
+ && contentLength <= bufferSize) { // Read small files completely
+ if (readFileCompletely() == -1) {
+ return -1;
+ }
+ } else if (firstRead && this.optimizeFooterRead && fCursor
+ == contentLength - FOOTER_DELTA) { // Read the last one block if the
+ // read is for the footer
+ if (readLastBlock() == -1) {
+ return -1;
+ }
+ } else {
+ if (readOneBlock(b) == -1) {
+ return -1;
+ }
+ }
+ fCursorAfterLastRead = fCursor;
+
+ //If there is anything in the buffer, then return lesser of (requested bytes) and (bytes in buffer)
+ //(bytes returned may be less than requested)
+ int bytesRemaining = limit - bCursor;
+ int bytesToRead = Math.min(len, bytesRemaining);
+ System.arraycopy(buffer, bCursor, b, off, bytesToRead);
+ bCursor += bytesToRead;
+ if (statistics != null) {
+ statistics.incrementBytesRead(bytesToRead);
+ }
+ if (streamStatistics != null) {
+ // Bytes read from the local buffer.
+ streamStatistics.bytesReadFromBuffer(bytesToRead);
+ streamStatistics.bytesRead(bytesToRead);
+ }
+ return bytesToRead;
+ }
+
+ private long readFileCompletely() throws IOException {
+ bCursor = (int) getPos();
+ // Read full file in case the file size <= buffer size
+ buffer = new byte[bufferSize];
+ long bytesRead = readInternal(0, buffer, 0, (int) contentLength, true);
+ firstRead = false;
+
+ if (bytesRead == -1) {
+ return -1;
+ }
+
+ limit = (int) bytesRead;
+ fCursor = bytesRead;
Review comment:
is this correct?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #2464: HADOOP-17347. ABFS: Read optimizations
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on pull request #2464:
URL: https://github.com/apache/hadoop/pull/2464#issuecomment-746214660
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 36s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | | 0m 0s | [test4tests](test4tests) | The patch appears to include 2 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 33m 4s | | trunk passed |
| +1 :green_heart: | compile | 0m 42s | | trunk passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 |
| +1 :green_heart: | compile | 0m 34s | | trunk passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| +1 :green_heart: | checkstyle | 0m 28s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 40s | | trunk passed |
| +1 :green_heart: | shadedclient | 16m 9s | | branch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 33s | | trunk passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 |
| +1 :green_heart: | javadoc | 0m 30s | | trunk passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| +0 :ok: | spotbugs | 0m 57s | | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 0m 56s | | trunk passed |
| -0 :warning: | patch | 1m 16s | | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 31s | | the patch passed |
| +1 :green_heart: | compile | 0m 30s | | the patch passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 |
| -1 :x: | javac | 0m 30s | [/diff-compile-javac-hadoop-tools_hadoop-azure-jdkUbuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2464/18/artifact/out/diff-compile-javac-hadoop-tools_hadoop-azure-jdkUbuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04.txt) | hadoop-tools_hadoop-azure-jdkUbuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 generated 2 new + 13 unchanged - 2 fixed = 15 total (was 15) |
| +1 :green_heart: | compile | 0m 25s | | the patch passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| -1 :x: | javac | 0m 25s | [/diff-compile-javac-hadoop-tools_hadoop-azure-jdkPrivateBuild-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2464/18/artifact/out/diff-compile-javac-hadoop-tools_hadoop-azure-jdkPrivateBuild-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01.txt) | hadoop-tools_hadoop-azure-jdkPrivateBuild-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 generated 2 new + 7 unchanged - 2 fixed = 9 total (was 9) |
| -0 :warning: | checkstyle | 0m 16s | [/diff-checkstyle-hadoop-tools_hadoop-azure.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2464/18/artifact/out/diff-checkstyle-hadoop-tools_hadoop-azure.txt) | hadoop-tools/hadoop-azure: The patch generated 2 new + 2 unchanged - 0 fixed = 4 total (was 2) |
| +1 :green_heart: | mvnsite | 0m 29s | | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | | The patch has no whitespace issues. |
| +1 :green_heart: | shadedclient | 14m 42s | | patch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 28s | | the patch passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 |
| +1 :green_heart: | javadoc | 0m 25s | | the patch passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| +1 :green_heart: | findbugs | 0m 59s | | the patch passed |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 1m 26s | | hadoop-azure in the patch passed. |
| +1 :green_heart: | asflicense | 0m 34s | | The patch does not generate ASF License warnings. |
| | | 77m 14s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2464/18/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/2464 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 97b751902643 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / df35c7f5199 |
| Default Java | Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2464/18/testReport/ |
| Max. process+thread count | 539 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2464/18/console |
| versions | git=2.17.1 maven=3.6.0 findbugs=4.0.6 |
| Powered by | Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #2464: Draft PR for read optimizations
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on pull request #2464:
URL: https://github.com/apache/hadoop/pull/2464#issuecomment-731950516
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 31s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | | 0m 0s | [test4tests](test4tests) | The patch appears to include 2 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 34m 16s | | trunk passed |
| +1 :green_heart: | compile | 0m 40s | | trunk passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 |
| +1 :green_heart: | compile | 0m 35s | | trunk passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| +1 :green_heart: | checkstyle | 0m 28s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 39s | | trunk passed |
| +1 :green_heart: | shadedclient | 16m 36s | | branch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 34s | | trunk passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 |
| +1 :green_heart: | javadoc | 0m 28s | | trunk passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| +0 :ok: | spotbugs | 1m 3s | | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 1m 0s | | trunk passed |
| -0 :warning: | patch | 1m 21s | | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 29s | | the patch passed |
| +1 :green_heart: | compile | 0m 30s | | the patch passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 |
| +1 :green_heart: | javac | 0m 30s | | the patch passed |
| +1 :green_heart: | compile | 0m 27s | | the patch passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| +1 :green_heart: | javac | 0m 27s | | the patch passed |
| -0 :warning: | checkstyle | 0m 16s | [/diff-checkstyle-hadoop-tools_hadoop-azure.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2464/2/artifact/out/diff-checkstyle-hadoop-tools_hadoop-azure.txt) | hadoop-tools/hadoop-azure: The patch generated 3 new + 4 unchanged - 0 fixed = 7 total (was 4) |
| +1 :green_heart: | mvnsite | 0m 29s | | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | | The patch has no whitespace issues. |
| +1 :green_heart: | shadedclient | 14m 55s | | patch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 27s | | the patch passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 |
| +1 :green_heart: | javadoc | 0m 24s | | the patch passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| +1 :green_heart: | findbugs | 0m 58s | | the patch passed |
|||| _ Other Tests _ |
| -1 :x: | unit | 1m 29s | [/patch-unit-hadoop-tools_hadoop-azure.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2464/2/artifact/out/patch-unit-hadoop-tools_hadoop-azure.txt) | hadoop-azure in the patch passed. |
| +1 :green_heart: | asflicense | 0m 35s | | The patch does not generate ASF License warnings. |
| | | 79m 4s | | |
| Reason | Tests |
|-------:|:------|
| Failed junit tests | hadoop.fs.azure.TestWasbFsck |
| | hadoop.fs.azure.TestNativeAzureFileSystemConcurrency |
| | hadoop.fs.azure.TestNativeAzureFileSystemOperationsMocked |
| | hadoop.fs.azure.TestNativeAzureFileSystemFileNameCheck |
| | hadoop.fs.azure.TestOutOfBandAzureBlobOperations |
| | hadoop.fs.azure.TestNativeAzureFileSystemMocked |
| | hadoop.fs.azure.TestNativeAzureFileSystemContractMocked |
| | hadoop.fs.azure.TestBlobMetadata |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2464/2/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/2464 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 6a1c07fc0f8d 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 641d8856d20 |
| Default Java | Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2464/2/testReport/ |
| Max. process+thread count | 689 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2464/2/console |
| versions | git=2.17.1 maven=3.6.0 findbugs=4.0.6 |
| Powered by | Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on pull request #2464: HADOOP-17347. ABFS: Read optimizations
Posted by GitBox <gi...@apache.org>.
steveloughran commented on pull request #2464:
URL: https://github.com/apache/hadoop/pull/2464#issuecomment-756759257
It's moot now that the PR is merged, but can I remind everyone
## Add a meaningful message to assertTrue/assertFalse asserts
Imagine that you are trying to debug a test run from an automated build. All you have is that an assert failed on a given line. Does that provide enough information to diagnose the problem? Or would you need extra information? If so: what information should be included?
It's OK to use assertJ's assertThat, which is where new tests are going. It's a bit more verbose but its assertions are very informative and easily extensible. If you haven't used the library yet -it's on the classpath, try using its assertions in new test suites.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] snvijaya commented on a change in pull request #2464: HADOOP-17347. ABFS: Read optimizations
Posted by GitBox <gi...@apache.org>.
snvijaya commented on a change in pull request #2464:
URL: https://github.com/apache/hadoop/pull/2464#discussion_r543916656
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsInputStream.java
##########
@@ -206,11 +231,127 @@ private int readOneBlock(final byte[] b, final int off, final int len) throws IO
fCursor += bytesRead;
fCursorAfterLastRead = fCursor;
}
+ return copyToUserBuffer(b, off, len);
+ }
+ private int readFileCompletely(final byte[] b, final int off, final int len)
+ throws IOException {
+ if (len == 0) {
+ return 0;
+ }
+ if (!validate(b, off, len)) {
+ return -1;
+ }
+ savePointerState();
+
+ buffer = new byte[bufferSize];
+ // data need to be copied to user buffer from index bCursor, bCursor has
+ // to be the current fCusor
+ bCursor = (int) fCursor;
+ return optimisedRead(b, off, len, 0, contentLength);
+ }
+
+ private int readLastBlock(final byte[] b, final int off, final int len)
+ throws IOException {
+ if (len == 0) {
+ return 0;
+ }
+ if (!validate(b, off, len)) {
+ return -1;
+ }
+ savePointerState();
+
+ buffer = new byte[bufferSize];
+ // data need to be copied to user buffer from index bCursor,
+ // AbfsInutStream buffer is going p contain dta from footer start. In
+ // that case bCursor will be set to footerStart - fCursor
+ long lastBlockStart = max(0, contentLength - bufferSize);
+ bCursor = (int) (fCursor - lastBlockStart);
+ // read API call is considered 1 single operation in reality server could
+ // return partial data and client has to retry untill the last full block
+ // is read. So setting the fCursorAfterLastRead before the possible
+ // multiple server calls
+ fCursorAfterLastRead = fCursor;
+ // 0 if contentlength is < buffersize
+ long actualLenToRead = min(bufferSize, contentLength);
+ return optimisedRead(b, off, len, lastBlockStart, actualLenToRead);
+ }
+
+ private int optimisedRead(final byte[] b, final int off, final int len,
+ final long readFrom, final long actualLen) throws IOException {
+ int totalBytesRead = 0;
+ fCursor = readFrom;
+ try {
+ for (int i = 0;
+ i < OPTIMIZATION_ATTEMPTS && fCursor < contentLength; i++) {
+ int bytesRead = readInternal(fCursor, buffer, limit,
+ (int) actualLen - limit, true);
+ if (bytesRead > 0) {
+ totalBytesRead += bytesRead;
+ limit += bytesRead;
+ fCursor += bytesRead;
+ }
Review comment:
What about fCursorAfterLastRead in readSmallFileCompletely case ? Other case too not clear why this should be set anywhere but here.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] snvijaya commented on a change in pull request #2464: HADOOP-17347. ABFS: Read optimizations
Posted by GitBox <gi...@apache.org>.
snvijaya commented on a change in pull request #2464:
URL: https://github.com/apache/hadoop/pull/2464#discussion_r543903666
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsInputStream.java
##########
@@ -206,11 +231,127 @@ private int readOneBlock(final byte[] b, final int off, final int len) throws IO
fCursor += bytesRead;
fCursorAfterLastRead = fCursor;
}
+ return copyToUserBuffer(b, off, len);
+ }
+ private int readFileCompletely(final byte[] b, final int off, final int len)
+ throws IOException {
+ if (len == 0) {
+ return 0;
+ }
+ if (!validate(b, off, len)) {
+ return -1;
+ }
+ savePointerState();
+
+ buffer = new byte[bufferSize];
+ // data need to be copied to user buffer from index bCursor, bCursor has
+ // to be the current fCusor
+ bCursor = (int) fCursor;
+ return optimisedRead(b, off, len, 0, contentLength);
+ }
+
+ private int readLastBlock(final byte[] b, final int off, final int len)
+ throws IOException {
+ if (len == 0) {
+ return 0;
+ }
+ if (!validate(b, off, len)) {
+ return -1;
+ }
+ savePointerState();
+
+ buffer = new byte[bufferSize];
+ // data need to be copied to user buffer from index bCursor,
+ // AbfsInutStream buffer is going p contain dta from footer start. In
+ // that case bCursor will be set to footerStart - fCursor
+ long lastBlockStart = max(0, contentLength - bufferSize);
+ bCursor = (int) (fCursor - lastBlockStart);
+ // read API call is considered 1 single operation in reality server could
+ // return partial data and client has to retry untill the last full block
+ // is read. So setting the fCursorAfterLastRead before the possible
+ // multiple server calls
+ fCursorAfterLastRead = fCursor;
+ // 0 if contentlength is < buffersize
+ long actualLenToRead = min(bufferSize, contentLength);
+ return optimisedRead(b, off, len, lastBlockStart, actualLenToRead);
+ }
+
+ private int optimisedRead(final byte[] b, final int off, final int len,
+ final long readFrom, final long actualLen) throws IOException {
+ int totalBytesRead = 0;
+ fCursor = readFrom;
+ try {
+ for (int i = 0;
+ i < OPTIMIZATION_ATTEMPTS && fCursor < contentLength; i++) {
+ int bytesRead = readInternal(fCursor, buffer, limit,
+ (int) actualLen - limit, true);
+ if (bytesRead > 0) {
+ totalBytesRead += bytesRead;
+ limit += bytesRead;
+ fCursor += bytesRead;
+ }
+ }
+ } catch (IOException e) {
+ LOG.debug("Optimized read failed. alling back to readOneBlock {}", e);
+ restorePointerState();
+ return readOneBlock(b, off, len);
+ }
+ // If the read was partial and the user requested part of data has
+ // not read then fallback to readoneblock. When limit is smaller than
+ // bCursor that means the user requested data has not been read.
+ if (fCursor < contentLength && bCursor > limit) {
+ restorePointerState();
+ return readOneBlock(b, off, len);
+ }
+ firstRead = false;
+ if (totalBytesRead == -1) {
Review comment:
Shouldnt this happen before the partial read check block ?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] bilaharith commented on pull request #2464: HADOOP-17347. ABFS: Read optimizations
Posted by GitBox <gi...@apache.org>.
bilaharith commented on pull request #2464:
URL: https://github.com/apache/hadoop/pull/2464#issuecomment-743274463
HNS-OAuth
========================
[INFO] Results:
[INFO]
[INFO] Tests run: 90, Failures: 0, Errors: 0, Skipped: 0
[INFO] Results:
[INFO]
[WARNING] Tests run: 484, Failures: 0, Errors: 0, Skipped: 68
[INFO] Results:
[INFO]
[WARNING] Tests run: 208, Failures: 0, Errors: 0, Skipped: 24
HNS-SharedKey
========================
[INFO] Results:
[INFO]
[INFO] Tests run: 90, Failures: 0, Errors: 0, Skipped: 0
[INFO] Results:
[INFO]
[WARNING] Tests run: 484, Failures: 0, Errors: 0, Skipped: 24
[INFO] Results:
[INFO]
[WARNING] Tests run: 208, Failures: 0, Errors: 0, Skipped: 16
NonHNS-SharedKey
========================
[INFO] Results:
[INFO]
[INFO] Tests run: 90, Failures: 0, Errors: 0, Skipped: 0
[INFO] Results:
[INFO]
[WARNING] Tests run: 484, Failures: 0, Errors: 0, Skipped: 249
[INFO] Results:
[INFO]
[WARNING] Tests run: 208, Failures: 0, Errors: 0, Skipped: 16
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] bilaharith commented on a change in pull request #2464: HADOOP-17347. ABFS: Read optimizations
Posted by GitBox <gi...@apache.org>.
bilaharith commented on a change in pull request #2464:
URL: https://github.com/apache/hadoop/pull/2464#discussion_r544151030
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsInputStream.java
##########
@@ -206,11 +231,127 @@ private int readOneBlock(final byte[] b, final int off, final int len) throws IO
fCursor += bytesRead;
fCursorAfterLastRead = fCursor;
}
+ return copyToUserBuffer(b, off, len);
+ }
+ private int readFileCompletely(final byte[] b, final int off, final int len)
+ throws IOException {
+ if (len == 0) {
+ return 0;
+ }
+ if (!validate(b, off, len)) {
+ return -1;
+ }
+ savePointerState();
+
+ buffer = new byte[bufferSize];
+ // data need to be copied to user buffer from index bCursor, bCursor has
+ // to be the current fCusor
+ bCursor = (int) fCursor;
+ return optimisedRead(b, off, len, 0, contentLength);
+ }
+
+ private int readLastBlock(final byte[] b, final int off, final int len)
+ throws IOException {
+ if (len == 0) {
+ return 0;
+ }
+ if (!validate(b, off, len)) {
+ return -1;
+ }
+ savePointerState();
+
+ buffer = new byte[bufferSize];
+ // data need to be copied to user buffer from index bCursor,
+ // AbfsInutStream buffer is going p contain dta from footer start. In
+ // that case bCursor will be set to footerStart - fCursor
+ long lastBlockStart = max(0, contentLength - bufferSize);
+ bCursor = (int) (fCursor - lastBlockStart);
+ // read API call is considered 1 single operation in reality server could
+ // return partial data and client has to retry untill the last full block
+ // is read. So setting the fCursorAfterLastRead before the possible
+ // multiple server calls
+ fCursorAfterLastRead = fCursor;
+ // 0 if contentlength is < buffersize
+ long actualLenToRead = min(bufferSize, contentLength);
+ return optimisedRead(b, off, len, lastBlockStart, actualLenToRead);
+ }
+
+ private int optimisedRead(final byte[] b, final int off, final int len,
+ final long readFrom, final long actualLen) throws IOException {
+ int totalBytesRead = 0;
+ fCursor = readFrom;
+ try {
+ for (int i = 0;
+ i < OPTIMIZATION_ATTEMPTS && fCursor < contentLength; i++) {
+ int bytesRead = readInternal(fCursor, buffer, limit,
+ (int) actualLen - limit, true);
+ if (bytesRead > 0) {
+ totalBytesRead += bytesRead;
+ limit += bytesRead;
+ fCursor += bytesRead;
+ }
+ }
+ } catch (IOException e) {
+ LOG.debug("Optimized read failed. alling back to readOneBlock {}", e);
+ restorePointerState();
+ return readOneBlock(b, off, len);
+ }
+ // If the read was partial and the user requested part of data has
+ // not read then fallback to readoneblock. When limit is smaller than
+ // bCursor that means the user requested data has not been read.
+ if (fCursor < contentLength && bCursor > limit) {
+ restorePointerState();
+ return readOneBlock(b, off, len);
+ }
+ firstRead = false;
+ if (totalBytesRead == -1) {
Review comment:
Yes, done.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] bilaharith commented on a change in pull request #2464: HADOOP-17347. ABFS: Read optimizations
Posted by GitBox <gi...@apache.org>.
bilaharith commented on a change in pull request #2464:
URL: https://github.com/apache/hadoop/pull/2464#discussion_r544147541
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsInputStream.java
##########
@@ -206,11 +231,127 @@ private int readOneBlock(final byte[] b, final int off, final int len) throws IO
fCursor += bytesRead;
fCursorAfterLastRead = fCursor;
}
+ return copyToUserBuffer(b, off, len);
+ }
+ private int readFileCompletely(final byte[] b, final int off, final int len)
+ throws IOException {
+ if (len == 0) {
+ return 0;
+ }
+ if (!validate(b, off, len)) {
+ return -1;
+ }
+ savePointerState();
+
+ buffer = new byte[bufferSize];
+ // data need to be copied to user buffer from index bCursor, bCursor has
+ // to be the current fCusor
+ bCursor = (int) fCursor;
+ return optimisedRead(b, off, len, 0, contentLength);
+ }
+
+ private int readLastBlock(final byte[] b, final int off, final int len)
+ throws IOException {
+ if (len == 0) {
+ return 0;
+ }
+ if (!validate(b, off, len)) {
+ return -1;
+ }
+ savePointerState();
+
+ buffer = new byte[bufferSize];
+ // data need to be copied to user buffer from index bCursor,
+ // AbfsInutStream buffer is going p contain dta from footer start. In
+ // that case bCursor will be set to footerStart - fCursor
+ long lastBlockStart = max(0, contentLength - bufferSize);
+ bCursor = (int) (fCursor - lastBlockStart);
+ // read API call is considered 1 single operation in reality server could
+ // return partial data and client has to retry untill the last full block
+ // is read. So setting the fCursorAfterLastRead before the possible
+ // multiple server calls
+ fCursorAfterLastRead = fCursor;
Review comment:
Done
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] bilaharith commented on a change in pull request #2464: HADOOP-17347. ABFS: Read optimizations
Posted by GitBox <gi...@apache.org>.
bilaharith commented on a change in pull request #2464:
URL: https://github.com/apache/hadoop/pull/2464#discussion_r541050129
##########
File path: hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/services/ITestAbfsInputStreamReadFooter.java
##########
@@ -0,0 +1,199 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.azurebfs.services;
+
+import java.io.IOException;
+import java.util.Map;
+import java.util.Random;
+
+import org.junit.Test;
+
+import org.apache.hadoop.fs.FSDataInputStream;
+import org.apache.hadoop.fs.FSDataOutputStream;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.fs.azurebfs.AbfsConfiguration;
+import org.apache.hadoop.fs.azurebfs.AbstractAbfsIntegrationTest;
+import org.apache.hadoop.fs.azurebfs.AzureBlobFileSystem;
+
+import static org.apache.hadoop.fs.azurebfs.AbfsStatistic.CONNECTIONS_MADE;
+import static org.apache.hadoop.fs.azurebfs.constants.FileSystemConfigurations.ONE_KB;
+import static org.apache.hadoop.fs.azurebfs.constants.FileSystemConfigurations.ONE_MB;
+
+public class ITestAbfsInputStreamReadFooter
+ extends AbstractAbfsIntegrationTest {
+
+ private static final int TEN = 10;
+ private static final int TWENTY = 20;
+
+ public ITestAbfsInputStreamReadFooter() throws Exception {
+ }
+
+ @Test
+ public void testOnlyOneServerCallIsMadeWhenTheConfIsTrue() throws Exception {
+ testNumBackendCalls(true);
+ }
+
+ @Test
+ public void testMultipleServerCallsAreMadeWhenTheConfIsFalse()
+ throws Exception {
+ testNumBackendCalls(false);
+ }
+
+ private void testNumBackendCalls(boolean optimizeFooterRead)
+ throws Exception {
+ final AzureBlobFileSystem fs = getFileSystem(optimizeFooterRead);
+ for (int i = 1; i <= 4; i++) {
+ String fileName = methodName.getMethodName() + i;
+ int fileSize = i * ONE_MB;
+ byte[] fileContent = getRandomBytesArray(fileSize);
+ Path testFilePath = createFileWithContent(fs, fileName, fileContent);
+ int length = AbfsInputStream.FOOTER_SIZE;
+ try (FSDataInputStream iStream = fs.open(testFilePath)) {
+ byte[] buffer = new byte[length];
+
+ Map<String, Long> metricMap = fs.getInstrumentationMap();
+ long requestsMadeBeforeTest = metricMap
+ .get(CONNECTIONS_MADE.getStatName());
+
+ iStream.seek(fileSize - 8);
+ iStream.read(buffer, 0, length);
+
+ iStream.seek(fileSize - (TEN * ONE_KB));
+ iStream.read(buffer, 0, length);
+
+ iStream.seek(fileSize - (TWENTY * ONE_KB));
+ iStream.read(buffer, 0, length);
+
+ metricMap = fs.getInstrumentationMap();
+ long requestsMadeAfterTest = metricMap
+ .get(CONNECTIONS_MADE.getStatName());
+
+ if (optimizeFooterRead) {
+ assertEquals(1, requestsMadeAfterTest - requestsMadeBeforeTest);
+ } else {
+ assertEquals(3, requestsMadeAfterTest - requestsMadeBeforeTest);
+ }
+ }
+ }
+ }
+
+ @Test
+ public void testSeekToEndAndReadWithConfTrue() throws Exception {
+ testSeekToEndAndReadWithConf(true);
+ }
+
+ @Test
+ public void testSeekToEndAndReadWithConfFalse() throws Exception {
+ testSeekToEndAndReadWithConf(false);
+ }
+
+ private void testSeekToEndAndReadWithConf(boolean optimizeFooterRead) throws Exception {
+ final AzureBlobFileSystem fs = getFileSystem(optimizeFooterRead);
+ for (int i = 5; i <= 10; i++) {
+ String fileName = methodName.getMethodName() + i;
+ int fileSize = i * ONE_MB;
+ byte[] fileContent = getRandomBytesArray(fileSize);
+ Path testFilePath = createFileWithContent(fs, fileName, fileContent);
+ seekReadAndTest(fs, testFilePath, fileSize - AbfsInputStream.FOOTER_SIZE,
+ AbfsInputStream.FOOTER_SIZE, fileContent);
+ }
+ }
+
+ private AzureBlobFileSystem getFileSystem(boolean optimizeFooterRead)
+ throws IOException {
+ final AzureBlobFileSystem fs = getFileSystem();
+ fs.getAbfsStore().getAbfsConfiguration()
+ .setOptimizeFooterRead(optimizeFooterRead);
+ fs.getAbfsStore().getAbfsConfiguration()
+ .setReadSmallFilesCompletely(false);
Review comment:
For small files this setting is required for testing the footer read.
For big files this is not set and expected to skip the smallfile read if condition.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] vinaysbadami commented on a change in pull request #2464: Draft PR for read optimizations
Posted by GitBox <gi...@apache.org>.
vinaysbadami commented on a change in pull request #2464:
URL: https://github.com/apache/hadoop/pull/2464#discussion_r523908784
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/constants/ConfigurationKeys.java
##########
@@ -56,6 +56,8 @@
public static final String AZURE_WRITE_MAX_REQUESTS_TO_QUEUE = "fs.azure.write.max.requests.to.queue";
public static final String AZURE_WRITE_BUFFER_SIZE = "fs.azure.write.request.size";
public static final String AZURE_READ_BUFFER_SIZE = "fs.azure.read.request.size";
Review comment:
smallfilescompletely
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] vinaysbadami commented on a change in pull request #2464: Draft PR for read optimizations
Posted by GitBox <gi...@apache.org>.
vinaysbadami commented on a change in pull request #2464:
URL: https://github.com/apache/hadoop/pull/2464#discussion_r523908784
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/constants/ConfigurationKeys.java
##########
@@ -56,6 +56,8 @@
public static final String AZURE_WRITE_MAX_REQUESTS_TO_QUEUE = "fs.azure.write.max.requests.to.queue";
public static final String AZURE_WRITE_BUFFER_SIZE = "fs.azure.write.request.size";
public static final String AZURE_READ_BUFFER_SIZE = "fs.azure.read.request.size";
Review comment:
smallfilescompletely
optimizefooterread
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] vinaysbadami commented on a change in pull request #2464: Draft PR for read optimizations
Posted by GitBox <gi...@apache.org>.
vinaysbadami commented on a change in pull request #2464:
URL: https://github.com/apache/hadoop/pull/2464#discussion_r523919820
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsInputStream.java
##########
@@ -162,14 +172,86 @@ private int readOneBlock(final byte[] b, final int off, final int len) throws IO
throw new IndexOutOfBoundsException();
}
- //If buffer is empty, then fill the buffer.
- if (bCursor == limit) {
+ if (firstRead && this.readSmallFilesCompletely
+ && contentLength <= bufferSize) { // Read small files completely
+ if (readFileCompletely() == -1) {
+ return -1;
+ }
+ } else if (firstRead && this.optimizeFooterRead && fCursor
+ == contentLength - FOOTER_DELTA) { // Read the last one block if the
+ // read is for the footer
+ if (readLastBlock() == -1) {
+ return -1;
+ }
+ } else {
+ if (readOneBlock(b) == -1) {
+ return -1;
+ }
+ }
+ fCursorAfterLastRead = fCursor;
+
+ //If there is anything in the buffer, then return lesser of (requested bytes) and (bytes in buffer)
+ //(bytes returned may be less than requested)
+ int bytesRemaining = limit - bCursor;
+ int bytesToRead = Math.min(len, bytesRemaining);
+ System.arraycopy(buffer, bCursor, b, off, bytesToRead);
+ bCursor += bytesToRead;
+ if (statistics != null) {
+ statistics.incrementBytesRead(bytesToRead);
+ }
+ if (streamStatistics != null) {
+ // Bytes read from the local buffer.
+ streamStatistics.bytesReadFromBuffer(bytesToRead);
+ streamStatistics.bytesRead(bytesToRead);
+ }
+ return bytesToRead;
+ }
+
+ private long readFileCompletely() throws IOException {
+ bCursor = (int) getPos();
+ // Read full file in case the file size <= buffer size
+ buffer = new byte[bufferSize];
+ long bytesRead = readInternal(0, buffer, 0, (int) contentLength, true);
+ firstRead = false;
+
+ if (bytesRead == -1) {
+ return -1;
+ }
+
+ limit = (int) bytesRead;
+ fCursor = bytesRead;
+ return bytesRead;
+ }
+
+ private long readLastBlock() throws IOException {
+ bCursor = (int) (
+ ((contentLength < bufferSize) ? contentLength : bufferSize)
+ - FOOTER_DELTA);
+ buffer = new byte[bufferSize];
+ long startPos = (contentLength < bufferSize)
+ ? 0
+ : contentLength - bufferSize;
+ long bytesRead = readInternal(startPos, buffer, 0, bufferSize, true);
+ firstRead = false;
+
+ if (bytesRead == -1) {
+ return -1;
+ }
+
+ limit += (int) bytesRead;
Review comment:
limit = bytesread
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] vinaysbadami commented on a change in pull request #2464: Draft PR for read optimizations
Posted by GitBox <gi...@apache.org>.
vinaysbadami commented on a change in pull request #2464:
URL: https://github.com/apache/hadoop/pull/2464#discussion_r523916467
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/constants/FileSystemConfigurations.java
##########
@@ -50,13 +50,15 @@
public static final int DEFAULT_AZURE_OAUTH_TOKEN_FETCH_RETRY_MAX_BACKOFF_INTERVAL = SIXTY_SECONDS;
public static final int DEFAULT_AZURE_OAUTH_TOKEN_FETCH_RETRY_DELTA_BACKOFF = 2;
- private static final int ONE_KB = 1024;
- private static final int ONE_MB = ONE_KB * ONE_KB;
+ public static final int ONE_KB = 1024;
+ public static final int ONE_MB = ONE_KB * ONE_KB;
// Default upload and download buffer size
public static final int DEFAULT_WRITE_BUFFER_SIZE = 8 * ONE_MB; // 8 MB
public static final int APPENDBLOB_MAX_WRITE_BUFFER_SIZE = 4 * ONE_MB; // 4 MB
public static final int DEFAULT_READ_BUFFER_SIZE = 4 * ONE_MB; // 4 MB
+ public static final boolean DEFAULT_READ_SMALL_FILES_COMPLETELY = true;
+ public static final boolean DEFAULT_OPTIMIZE_FOOTER_READ = true;
Review comment:
default false
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #2464: Draft PR for read optimizations
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on pull request #2464:
URL: https://github.com/apache/hadoop/pull/2464#issuecomment-733480056
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 1m 40s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files found. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | | 0m 0s | [test4tests](test4tests) | The patch appears to include 2 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 40m 53s | | trunk passed |
| +1 :green_heart: | compile | 0m 44s | | trunk passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 |
| +1 :green_heart: | compile | 0m 36s | | trunk passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| +1 :green_heart: | checkstyle | 0m 27s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 41s | | trunk passed |
| +1 :green_heart: | shadedclient | 20m 46s | | branch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 30s | | trunk passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 |
| +1 :green_heart: | javadoc | 0m 29s | | trunk passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| +0 :ok: | spotbugs | 1m 12s | | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 1m 10s | | trunk passed |
| -0 :warning: | patch | 1m 29s | | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 35s | | the patch passed |
| +1 :green_heart: | compile | 0m 36s | | the patch passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 |
| +1 :green_heart: | javac | 0m 36s | | the patch passed |
| +1 :green_heart: | compile | 0m 33s | | the patch passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| +1 :green_heart: | javac | 0m 33s | | the patch passed |
| +1 :green_heart: | checkstyle | 0m 20s | | the patch passed |
| +1 :green_heart: | mvnsite | 0m 33s | | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | | The patch has no whitespace issues. |
| +1 :green_heart: | shadedclient | 20m 14s | | patch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 29s | | the patch passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 |
| +1 :green_heart: | javadoc | 0m 26s | | the patch passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| +1 :green_heart: | findbugs | 1m 20s | | the patch passed |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 1m 45s | | hadoop-azure in the patch passed. |
| +1 :green_heart: | asflicense | 0m 36s | | The patch does not generate ASF License warnings. |
| | | 97m 44s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2464/3/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/2464 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux bae31219bc44 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 08b2e285dbd |
| Default Java | Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2464/3/testReport/ |
| Max. process+thread count | 511 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2464/3/console |
| versions | git=2.17.1 maven=3.6.0 findbugs=4.0.6 |
| Powered by | Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] snvijaya commented on a change in pull request #2464: HADOOP-17347. ABFS: Read optimizations
Posted by GitBox <gi...@apache.org>.
snvijaya commented on a change in pull request #2464:
URL: https://github.com/apache/hadoop/pull/2464#discussion_r543108306
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsInputStream.java
##########
@@ -206,11 +229,120 @@ private int readOneBlock(final byte[] b, final int off, final int len) throws IO
fCursor += bytesRead;
fCursorAfterLastRead = fCursor;
}
+ return copyToUserBuffer(b, off, len);
+ }
+
+ private int readFileCompletely(final byte[] b, final int off, final int len)
+ throws IOException {
+ if (len == 0) {
+ return 0;
+ }
+ if (!validate(b, off, len)) {
+ return -1;
+ }
+ savePointerState();
+
+ buffer = new byte[bufferSize];
+ // data need to be copied to user buffer from index bCursor, bCursor has
+ // to be the current fCusor
+ bCursor = (int) fCursor;
+ return optimisedRead(b, off, len, 0, contentLength);
+ }
+
+ private int readLastBlock(final byte[] b, final int off, final int len)
+ throws IOException {
+ if (len == 0) {
+ return 0;
+ }
+ if (!validate(b, off, len)) {
+ return -1;
+ }
+ savePointerState();
+
+ buffer = new byte[bufferSize];
+ // data need to be copied to user buffer from index bCursor,
+ // AbfsInutStream buffer is going p contain dta from footer start. In
+ // that case bCursor will be set to footerStart - fCursor
+ long lastBlockStart = max(0, contentLength - bufferSize);
+ bCursor = (int) (fCursor - lastBlockStart);
+ // read API call is considered 1 single operation in reality server could
+ // return partial data and client has to retry untill the last full block
+ // is read. So setting the fCursorAfterLastRead before the possible
+ // multiple server calls
+ fCursorAfterLastRead = fCursor;
+ // 0 if contentlength is < buffersize
+ long actualLenToRead = min(bufferSize, contentLength);
+ return optimisedRead(b, off, len, lastBlockStart, actualLenToRead);
+ }
+
+ private int optimisedRead(final byte[] b, final int off, final int len,
+ final long readFrom, final long actualLen) throws IOException {
+ int totalBytesRead = 0;
+ fCursor = readFrom;
+ for (int i = 0; i < 2 && fCursor < contentLength; i++) {
Review comment:
2 => field name
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsInputStream.java
##########
@@ -206,11 +229,120 @@ private int readOneBlock(final byte[] b, final int off, final int len) throws IO
fCursor += bytesRead;
fCursorAfterLastRead = fCursor;
}
+ return copyToUserBuffer(b, off, len);
+ }
+
+ private int readFileCompletely(final byte[] b, final int off, final int len)
+ throws IOException {
+ if (len == 0) {
+ return 0;
+ }
+ if (!validate(b, off, len)) {
+ return -1;
+ }
+ savePointerState();
+
+ buffer = new byte[bufferSize];
+ // data need to be copied to user buffer from index bCursor, bCursor has
+ // to be the current fCusor
+ bCursor = (int) fCursor;
+ return optimisedRead(b, off, len, 0, contentLength);
+ }
+
+ private int readLastBlock(final byte[] b, final int off, final int len)
+ throws IOException {
+ if (len == 0) {
+ return 0;
+ }
+ if (!validate(b, off, len)) {
+ return -1;
+ }
+ savePointerState();
+
+ buffer = new byte[bufferSize];
+ // data need to be copied to user buffer from index bCursor,
+ // AbfsInutStream buffer is going p contain dta from footer start. In
+ // that case bCursor will be set to footerStart - fCursor
+ long lastBlockStart = max(0, contentLength - bufferSize);
+ bCursor = (int) (fCursor - lastBlockStart);
+ // read API call is considered 1 single operation in reality server could
+ // return partial data and client has to retry untill the last full block
+ // is read. So setting the fCursorAfterLastRead before the possible
+ // multiple server calls
+ fCursorAfterLastRead = fCursor;
+ // 0 if contentlength is < buffersize
+ long actualLenToRead = min(bufferSize, contentLength);
+ return optimisedRead(b, off, len, lastBlockStart, actualLenToRead);
+ }
+
+ private int optimisedRead(final byte[] b, final int off, final int len,
+ final long readFrom, final long actualLen) throws IOException {
+ int totalBytesRead = 0;
+ fCursor = readFrom;
+ for (int i = 0; i < 2 && fCursor < contentLength; i++) {
+ int bytesRead = readInternal(fCursor, buffer, limit,
+ (int) actualLen - limit, true);
+ if (bytesRead > 0) {
+ totalBytesRead += bytesRead;
+ limit += bytesRead;
+ fCursor += bytesRead;
+ }
+ }
+ // if the read was not success and the user requested part of data has
Review comment:
re-write the comment
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #2464: Draft PR for read optimizations
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on pull request #2464:
URL: https://github.com/apache/hadoop/pull/2464#issuecomment-727774104
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 26m 9s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | | 0m 0s | [test4tests](test4tests) | The patch appears to include 2 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 34m 4s | | trunk passed |
| +1 :green_heart: | compile | 0m 38s | | trunk passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 |
| +1 :green_heart: | compile | 0m 34s | | trunk passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| +1 :green_heart: | checkstyle | 0m 28s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 39s | | trunk passed |
| +1 :green_heart: | shadedclient | 16m 17s | | branch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 32s | | trunk passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 |
| +1 :green_heart: | javadoc | 0m 29s | | trunk passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| +0 :ok: | spotbugs | 0m 58s | | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 0m 57s | | trunk passed |
| -0 :warning: | patch | 1m 17s | | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 29s | | the patch passed |
| +1 :green_heart: | compile | 0m 29s | | the patch passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 |
| +1 :green_heart: | javac | 0m 29s | | the patch passed |
| +1 :green_heart: | compile | 0m 26s | | the patch passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| +1 :green_heart: | javac | 0m 26s | | the patch passed |
| +1 :green_heart: | checkstyle | 0m 16s | | the patch passed |
| +1 :green_heart: | mvnsite | 0m 30s | | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | | The patch has no whitespace issues. |
| +1 :green_heart: | shadedclient | 14m 53s | | patch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 25s | | the patch passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 |
| +1 :green_heart: | javadoc | 0m 24s | | the patch passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| +1 :green_heart: | findbugs | 0m 59s | | the patch passed |
|||| _ Other Tests _ |
| -1 :x: | unit | 1m 31s | [/patch-unit-hadoop-tools_hadoop-azure.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2464/1/artifact/out/patch-unit-hadoop-tools_hadoop-azure.txt) | hadoop-azure in the patch passed. |
| +1 :green_heart: | asflicense | 0m 32s | | The patch does not generate ASF License warnings. |
| | | 103m 38s | | |
| Reason | Tests |
|-------:|:------|
| Failed junit tests | hadoop.fs.azure.TestWasbFsck |
| | hadoop.fs.azure.TestNativeAzureFileSystemConcurrency |
| | hadoop.fs.azure.TestNativeAzureFileSystemOperationsMocked |
| | hadoop.fs.azure.TestNativeAzureFileSystemFileNameCheck |
| | hadoop.fs.azure.TestOutOfBandAzureBlobOperations |
| | hadoop.fs.azure.TestNativeAzureFileSystemMocked |
| | hadoop.fs.azure.TestNativeAzureFileSystemContractMocked |
| | hadoop.fs.azure.TestBlobMetadata |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2464/1/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/2464 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 04f229091069 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / ebe1d1fbf7a |
| Default Java | Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2464/1/testReport/ |
| Max. process+thread count | 535 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2464/1/console |
| versions | git=2.17.1 maven=3.6.0 findbugs=4.0.6 |
| Powered by | Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] bilaharith commented on a change in pull request #2464: HADOOP-17347. ABFS: Read optimizations
Posted by GitBox <gi...@apache.org>.
bilaharith commented on a change in pull request #2464:
URL: https://github.com/apache/hadoop/pull/2464#discussion_r538339310
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsInputStream.java
##########
@@ -141,7 +154,7 @@ public synchronized int read(final byte[] b, final int off, final int len) throw
return totalReadBytes > 0 ? totalReadBytes : lastReadBytes;
}
- private int readOneBlock(final byte[] b, final int off, final int len) throws IOException {
+ private int validate(byte[] b, int off, int len) throws IOException {
Review comment:
Done
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsInputStream.java
##########
@@ -161,6 +174,14 @@ private int readOneBlock(final byte[] b, final int off, final int len) throws IO
if (off < 0 || len < 0 || len > b.length - off) {
throw new IndexOutOfBoundsException();
}
+ return 1; // 1 indicate success
+ }
+
+ private int readOneBlock(final byte[] b, final int off, final int len) throws IOException {
+ int validation = validate(b, off, len);
+ if (validation < 1) {
+ return validation;
Review comment:
Done
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsInputStream.java
##########
@@ -211,6 +235,59 @@ private int readOneBlock(final byte[] b, final int off, final int len) throws IO
return bytesToRead;
}
+ private boolean shouldReadFully() {
+ return this.firstRead && this.context.readSmallFilesCompletely()
+ && this.contentLength <= this.bufferSize;
+ }
+
+ private boolean shouldReadLastBlock(int len) {
+ return this.firstRead && this.context.optimizeFooterRead()
+ && len == FOOTER_SIZE
+ && this.fCursor == this.contentLength - FOOTER_SIZE;
+ }
+
+ private int readFileCompletely(final byte[] b, final int off, final int len) throws IOException {
+ int validation = validate(b, off, len);
+ if (validation < 1) {
Review comment:
Done
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsInputStream.java
##########
@@ -211,6 +235,59 @@ private int readOneBlock(final byte[] b, final int off, final int len) throws IO
return bytesToRead;
}
+ private boolean shouldReadFully() {
+ return this.firstRead && this.context.readSmallFilesCompletely()
+ && this.contentLength <= this.bufferSize;
+ }
+
+ private boolean shouldReadLastBlock(int len) {
+ return this.firstRead && this.context.optimizeFooterRead()
+ && len == FOOTER_SIZE
+ && this.fCursor == this.contentLength - FOOTER_SIZE;
+ }
+
+ private int readFileCompletely(final byte[] b, final int off, final int len) throws IOException {
+ int validation = validate(b, off, len);
+ if (validation < 1) {
+ return validation;
+ }
+
+ buffer = new byte[bufferSize];
+ bCursor = (int) fCursor;
+ int bytesRead = readInternal(0, buffer, 0, (int) contentLength, true);
+ firstRead = false;
+ if (bytesRead == -1) {
+ return -1;
+ }
+ fCursorAfterLastRead = fCursor;
+ limit = bytesRead;
+ fCursor = bytesRead;
+ return copyToUserBuffer(b, off, len);
+ }
+
+ private int readLastBlock(final byte[] b, final int off, final int len) throws IOException {
+ int validation = validate(b, off, len);
+ if (validation < 1) {
Review comment:
Done
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] bilaharith commented on a change in pull request #2464: Draft PR for read optimizations
Posted by GitBox <gi...@apache.org>.
bilaharith commented on a change in pull request #2464:
URL: https://github.com/apache/hadoop/pull/2464#discussion_r528642700
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsInputStream.java
##########
@@ -162,14 +172,86 @@ private int readOneBlock(final byte[] b, final int off, final int len) throws IO
throw new IndexOutOfBoundsException();
}
- //If buffer is empty, then fill the buffer.
- if (bCursor == limit) {
+ if (firstRead && this.readSmallFilesCompletely
+ && contentLength <= bufferSize) { // Read small files completely
+ if (readFileCompletely() == -1) {
+ return -1;
+ }
+ } else if (firstRead && this.optimizeFooterRead && fCursor
+ == contentLength - FOOTER_DELTA) { // Read the last one block if the
+ // read is for the footer
+ if (readLastBlock() == -1) {
+ return -1;
+ }
+ } else {
+ if (readOneBlock(b) == -1) {
+ return -1;
+ }
+ }
+ fCursorAfterLastRead = fCursor;
+
+ //If there is anything in the buffer, then return lesser of (requested bytes) and (bytes in buffer)
+ //(bytes returned may be less than requested)
+ int bytesRemaining = limit - bCursor;
+ int bytesToRead = Math.min(len, bytesRemaining);
+ System.arraycopy(buffer, bCursor, b, off, bytesToRead);
+ bCursor += bytesToRead;
+ if (statistics != null) {
+ statistics.incrementBytesRead(bytesToRead);
+ }
+ if (streamStatistics != null) {
+ // Bytes read from the local buffer.
+ streamStatistics.bytesReadFromBuffer(bytesToRead);
+ streamStatistics.bytesRead(bytesToRead);
+ }
+ return bytesToRead;
+ }
+
+ private long readFileCompletely() throws IOException {
+ bCursor = (int) getPos();
+ // Read full file in case the file size <= buffer size
+ buffer = new byte[bufferSize];
+ long bytesRead = readInternal(0, buffer, 0, (int) contentLength, true);
+ firstRead = false;
+
+ if (bytesRead == -1) {
+ return -1;
+ }
+
+ limit = (int) bytesRead;
+ fCursor = bytesRead;
Review comment:
corrected
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] bilaharith commented on a change in pull request #2464: HADOOP-17347. ABFS: Read optimizations
Posted by GitBox <gi...@apache.org>.
bilaharith commented on a change in pull request #2464:
URL: https://github.com/apache/hadoop/pull/2464#discussion_r539101796
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsInputStream.java
##########
@@ -211,6 +235,59 @@ private int readOneBlock(final byte[] b, final int off, final int len) throws IO
return bytesToRead;
}
+ private boolean shouldReadFully() {
+ return this.firstRead && this.context.readSmallFilesCompletely()
+ && this.contentLength <= this.bufferSize;
+ }
+
+ private boolean shouldReadLastBlock(int len) {
+ return this.firstRead && this.context.optimizeFooterRead()
+ && len == FOOTER_SIZE
+ && this.fCursor == this.contentLength - FOOTER_SIZE;
+ }
+
+ private int readFileCompletely(final byte[] b, final int off, final int len) throws IOException {
+ int validation = validate(b, off, len);
+ if (validation < 1) {
+ return validation;
+ }
+
+ buffer = new byte[bufferSize];
+ bCursor = (int) fCursor;
+ int bytesRead = readInternal(0, buffer, 0, (int) contentLength, true);
+ firstRead = false;
Review comment:
in case of 429, exception is thrown and the line ( firstRead = false;) won't get executed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org