You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by GitBox <gi...@apache.org> on 2020/01/03 10:09:56 UTC
[GitHub] [hadoop] ishaniahuja opened a new pull request #1790: changes for combined append+flush calls for blockblob(traditionan) an…
ishaniahuja opened a new pull request #1790: changes for combined append+flush calls for blockblob(traditionan) an…
URL: https://github.com/apache/hadoop/pull/1790
…d support for appendblob
## NOTICE
Please create an issue in ASF JIRA before opening a pull request,
and you need to set the title of the pull request which starts with
the corresponding JIRA issue number. (e.g. HADOOP-XXXXX. Fix a typo in YYY.)
For more details, please see https://cwiki.apache.org/confluence/display/HADOOP/How+To+Contribute
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] snvijaya commented on a change in pull request #1790: changes for combined append+flush calls for blockblob(traditionan) an…
Posted by GitBox <gi...@apache.org>.
snvijaya commented on a change in pull request #1790: changes for combined append+flush calls for blockblob(traditionan) an…
URL: https://github.com/apache/hadoop/pull/1790#discussion_r363152120
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsOutputStream.java
##########
@@ -257,17 +265,16 @@ public synchronized void close() throws IOException {
private synchronized void flushInternal(boolean isClose) throws IOException {
maybeThrowLastError();
- writeCurrentBufferToService();
- flushWrittenBytesToService(isClose);
+ writeAndFlushWrittenBytesToService(isClose);
}
private synchronized void flushInternalAsync() throws IOException {
maybeThrowLastError();
- writeCurrentBufferToService();
+ writeCurrentBufferToService(true, false);
flushWrittenBytesToServiceAsync();
}
- private synchronized void writeCurrentBufferToService() throws IOException {
+ private synchronized void writeCurrentBufferToService(final boolean flush, final boolean isClose) throws IOException {
Review comment:
As mentioned in the comments for AbfsConfiguration.java, dont see flush in use inside the method. Also applies to the new isClose argument.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] snvijaya commented on a change in pull request #1790: changes for combined append+flush calls for blockblob(traditionan) an…
Posted by GitBox <gi...@apache.org>.
snvijaya commented on a change in pull request #1790: changes for combined append+flush calls for blockblob(traditionan) an…
URL: https://github.com/apache/hadoop/pull/1790#discussion_r365171372
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AbfsConfiguration.java
##########
@@ -146,6 +150,10 @@
DefaultValue = DEFAULT_ENABLE_FLUSH)
private boolean enableFlush;
+ @BooleanConfigurationValidatorAnnotation(ConfigurationKey = FS_AZURE_ENABLE_APPEND_WITH_FLUSH,
+ DefaultValue = DEFAULT_ENABLE_APPEND_WITH_FLUSH)
+ private boolean enableAppendWithFlush;
Review comment:
Last iteration added has bought in the missing code piece.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] snvijaya commented on a change in pull request #1790: changes for combined append+flush calls for blockblob(traditionan) an…
Posted by GitBox <gi...@apache.org>.
snvijaya commented on a change in pull request #1790: changes for combined append+flush calls for blockblob(traditionan) an…
URL: https://github.com/apache/hadoop/pull/1790#discussion_r363154151
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsClient.java
##########
@@ -278,7 +282,8 @@ public AbfsRestOperation renamePath(final String source, final String destinatio
}
public AbfsRestOperation append(final String path, final long position, final byte[] buffer, final int offset,
- final int length) throws AzureBlobFileSystemException {
+ final int length, boolean flush, boolean isClose)
Review comment:
This method is supposed to be called from AbfsOutputStream.java -> writeCurrentBufferToService(), but there is no change for call to this API within that method. How did the compilation pass.
And, (probably a repeat question of config query posted above, still) can you please explain why flush and close needs to be added to Append method.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] bilaharith commented on a change in pull request #1790: changes for combined append+flush calls for blockblob(traditionan) an…
Posted by GitBox <gi...@apache.org>.
bilaharith commented on a change in pull request #1790: changes for combined append+flush calls for blockblob(traditionan) an…
URL: https://github.com/apache/hadoop/pull/1790#discussion_r363223682
##########
File path: hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/services/TestAbfsOutputStream.java
##########
@@ -0,0 +1,268 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.azurebfs.services;
+import java.util.Arrays;
+import java.util.HashSet;
+import java.util.Random;
+import org.junit.Assert;
+import org.junit.Test;
+
+import org.mockito.ArgumentCaptor;
+import static org.mockito.Mockito.*;
+/**
+ * Test useragent of abfs client.
+ *
+ */
+public final class TestAbfsOutputStream {
+
+ private static int bufferSize = 4096;
+ private static int writeSize = 1000;
+ private static String path = "~/testpath";
+
+ @Test
+ public void verifyShortWriteRequest() throws Exception {
+
+ AbfsClient client = mock(AbfsClient.class);
+ AbfsOutputStream out = new AbfsOutputStream(client, path, 0, bufferSize, true, true, false);
+ final byte[] b = new byte[writeSize];
+ new Random().nextBytes(b);
+ out.write(b);
+ out.hsync();
+ ArgumentCaptor<String> acString = ArgumentCaptor.forClass(String.class);
+ ArgumentCaptor<Long> acLong = ArgumentCaptor.forClass(Long.class);
+ ArgumentCaptor<Integer> acInt = ArgumentCaptor.forClass(Integer.class);
+ ArgumentCaptor<Boolean> acBool = ArgumentCaptor.forClass(Boolean.class);
+ ArgumentCaptor<byte[]> acByteArray = ArgumentCaptor.forClass(byte[].class);
+
+ final byte[] b1 = new byte[2*writeSize];
+ new Random().nextBytes(b1);
+ out.write(b1);
+ out.flush();
+ out.hflush();
+
+ out.hsync();
+
+ verify(client, times(2)).append(acString.capture(), acLong.capture(), acByteArray.capture(), acInt.capture(), acInt.capture(), acBool.capture(), acBool.capture());
+ Assert.assertEquals(Arrays.asList(path, path) , acString.getAllValues());
+ Assert.assertEquals(Arrays.asList(Long.valueOf(0), Long.valueOf(writeSize)), acLong.getAllValues());
+ //flush=true, close=false, flush=true, close=false
+ Assert.assertEquals(Arrays.asList(true, false, true, false), acBool.getAllValues());
+ Assert.assertEquals(Arrays.asList(0,writeSize, 0, 2*writeSize), acInt.getAllValues());
+
+ verifyNoMoreInteractions(client);
+
+ }
+
+ @Test
+ public void verifyWriteRequest() throws Exception {
+
+ AbfsClient client = mock(AbfsClient.class);
+ AbfsOutputStream out = new AbfsOutputStream(client, path, 0, bufferSize, true, true, false);
+ final byte[] b = new byte[writeSize];
+ new Random().nextBytes(b);
+
+ for (int i = 0; i < 5; i++) {
+ out.write(b);
+ }
+ out.close();
+
+ ArgumentCaptor<String> acString = ArgumentCaptor.forClass(String.class);
+ ArgumentCaptor<Long> acLong = ArgumentCaptor.forClass(Long.class);
+ ArgumentCaptor<Integer> acInt = ArgumentCaptor.forClass(Integer.class);
+ ArgumentCaptor<Boolean> acBool = ArgumentCaptor.forClass(Boolean.class);
+ ArgumentCaptor<byte[]> acByteArray = ArgumentCaptor.forClass(byte[].class);
+
+ verify(client, times(2)).append(acString.capture(), acLong.capture(), acByteArray.capture(), acInt.capture(), acInt.capture(), acBool.capture(), acBool.capture());
+ Assert.assertEquals(Arrays.asList(path, path) , acString.getAllValues());
+ Assert.assertEquals(Arrays.asList(Long.valueOf(0), Long.valueOf(bufferSize)), acLong.getAllValues());
+ //flush=false,close=false, flush=true,close=true
+ Assert.assertEquals(Arrays.asList(false, false, true, true), acBool.getAllValues());
+ Assert.assertEquals(Arrays.asList(0, bufferSize, 0, 5*writeSize-bufferSize), acInt.getAllValues());
+
+ verifyNoMoreInteractions(client);
+ }
+
+ @Test
+ public void verifyWriteRequestOfBufferSizeAndClose() throws Exception {
+
+ AbfsClient client = mock(AbfsClient.class);
+ AbfsOutputStream out = new AbfsOutputStream(client, path, 0, bufferSize, true, true, false);
+ final byte[] b = new byte[bufferSize];
+ new Random().nextBytes(b);
+
+ for (int i = 0; i < 2; i++) {
+ out.write(b);
+ }
+ out.close();
+
+ ArgumentCaptor<String> acString = ArgumentCaptor.forClass(String.class);
+ ArgumentCaptor<Long> acLong = ArgumentCaptor.forClass(Long.class);
+ ArgumentCaptor<Integer> acInt = ArgumentCaptor.forClass(Integer.class);
+ ArgumentCaptor<Boolean> acBool = ArgumentCaptor.forClass(Boolean.class);
+ ArgumentCaptor<byte[]> acByteArray = ArgumentCaptor.forClass(byte[].class);
+
+ verify(client, times(2)).append(acString.capture(), acLong.capture(), acByteArray.capture(), acInt.capture(), acInt.capture(), acBool.capture(), acBool.capture());
+ Assert.assertEquals(Arrays.asList(path, path) , acString.getAllValues());
+ Assert.assertEquals(new HashSet<Long>(Arrays.asList(Long.valueOf(0), Long.valueOf(bufferSize))), new HashSet<Long>(acLong.getAllValues()));
+ //flush=false, close=false, flush=false, close=false
+ Assert.assertEquals(Arrays.asList(false, false, false, false), acBool.getAllValues());
+ Assert.assertEquals(Arrays.asList(0, bufferSize, 0, bufferSize), acInt.getAllValues());
+
+ ArgumentCaptor<String> acFlushString = ArgumentCaptor.forClass(String.class);
+ ArgumentCaptor<Long> acFlushLong = ArgumentCaptor.forClass(Long.class);
+ ArgumentCaptor<Boolean> acFlushBool = ArgumentCaptor.forClass(Boolean.class);
+
+ verify(client, times(1)).flush(acFlushString.capture(), acFlushLong.capture(), acFlushBool.capture(), acFlushBool.capture());
+ Assert.assertEquals(Arrays.asList(path) , acFlushString.getAllValues());
+ Assert.assertEquals(Arrays.asList(Long.valueOf(2*bufferSize)), acFlushLong.getAllValues());
+ Assert.assertEquals(Arrays.asList(false, true), acFlushBool.getAllValues());
+
+ verifyNoMoreInteractions(client);
+ }
+
+ @Test
+ public void verifyWriteRequestOfBufferSize() throws Exception {
+
+ AbfsClient client = mock(AbfsClient.class);
+ AbfsOutputStream out = new AbfsOutputStream(client, path, 0, bufferSize, true, true, false);
+ final byte[] b = new byte[bufferSize];
+ new Random().nextBytes(b);
+
+ for (int i = 0; i < 2; i++) {
+ out.write(b);
+ }
+ Thread.sleep(1000);
+
+ ArgumentCaptor<String> acString = ArgumentCaptor.forClass(String.class);
+ ArgumentCaptor<Long> acLong = ArgumentCaptor.forClass(Long.class);
+ ArgumentCaptor<Integer> acInt = ArgumentCaptor.forClass(Integer.class);
+ ArgumentCaptor<Boolean> acBool = ArgumentCaptor.forClass(Boolean.class);
+ ArgumentCaptor<byte[]> acByteArray = ArgumentCaptor.forClass(byte[].class);
+
+ verify(client, times(2)).append(acString.capture(), acLong.capture(), acByteArray.capture(), acInt.capture(), acInt.capture(), acBool.capture(), acBool.capture());
+ Assert.assertEquals(Arrays.asList(path, path) , acString.getAllValues());
+ Assert.assertEquals(new HashSet<Long>(Arrays.asList(Long.valueOf(0), Long.valueOf(bufferSize))), new HashSet<Long>(acLong.getAllValues()));
+ //flush=false, close=false, flush=false, close=false
+ Assert.assertEquals(Arrays.asList(false, false, false, false), acBool.getAllValues());
+ Assert.assertEquals(Arrays.asList(0, bufferSize, 0, bufferSize), acInt.getAllValues());
+
+ verifyNoMoreInteractions(client);
+ }
+
+ @Test
+ public void verifyWriteRequestOfBufferSizeWithAppendBlob() throws Exception {
+
+ AbfsClient client = mock(AbfsClient.class);
+ AbfsOutputStream out = new AbfsOutputStream(client, path, 0, bufferSize, true, true, true);
+ final byte[] b = new byte[bufferSize];
+ new Random().nextBytes(b);
+
+ for (int i = 0; i < 2; i++) {
+ out.write(b);
+ }
+ Thread.sleep(1000);
+
+ ArgumentCaptor<String> acString = ArgumentCaptor.forClass(String.class);
+ ArgumentCaptor<Long> acLong = ArgumentCaptor.forClass(Long.class);
+ ArgumentCaptor<Integer> acInt = ArgumentCaptor.forClass(Integer.class);
+ ArgumentCaptor<Boolean> acBool = ArgumentCaptor.forClass(Boolean.class);
+ ArgumentCaptor<byte[]> acByteArray = ArgumentCaptor.forClass(byte[].class);
Review comment:
can the lines 171-185 be made a separate method and reuse in the test methods
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on issue #1790: [HADOOP-16818]
ABFS: Combine append+flush calls for blockblob & appendblob
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on issue #1790: [HADOOP-16818] ABFS: Combine append+flush calls for blockblob & appendblob
URL: https://github.com/apache/hadoop/pull/1790#issuecomment-590474367
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 33m 8s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +0 :ok: | markdownlint | 0m 1s | markdownlint was not available. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 1 new or modified test files. |
||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 23m 46s | trunk passed |
| +1 :green_heart: | compile | 0m 32s | trunk passed |
| +1 :green_heart: | checkstyle | 0m 23s | trunk passed |
| +1 :green_heart: | mvnsite | 0m 37s | trunk passed |
| +1 :green_heart: | shadedclient | 17m 5s | branch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 23s | trunk passed |
| +0 :ok: | spotbugs | 1m 0s | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 0m 58s | trunk passed |
||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 33s | the patch passed |
| +1 :green_heart: | compile | 0m 23s | the patch passed |
| +1 :green_heart: | javac | 0m 23s | the patch passed |
| -0 :warning: | checkstyle | 0m 17s | hadoop-tools/hadoop-azure: The patch generated 23 new + 5 unchanged - 0 fixed = 28 total (was 5) |
| +1 :green_heart: | mvnsite | 0m 29s | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | xml | 0m 1s | The patch has no ill-formed XML file. |
| +1 :green_heart: | shadedclient | 16m 23s | patch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 20s | the patch passed |
| -1 :x: | findbugs | 0m 57s | hadoop-tools/hadoop-azure generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 1m 28s | hadoop-azure in the patch passed. |
| +1 :green_heart: | asflicense | 0m 28s | The patch does not generate ASF License warnings. |
| | | 99m 38s | |
| Reason | Tests |
|-------:|:------|
| FindBugs | module:hadoop-tools/hadoop-azure |
| | There is an apparent infinite recursive loop in org.apache.hadoop.fs.azurebfs.services.AbfsClient.createDefaultHeaders() At AbfsClient.java:recursive loop in org.apache.hadoop.fs.azurebfs.services.AbfsClient.createDefaultHeaders() At AbfsClient.java:[line 124] |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | Client=19.03.6 Server=19.03.6 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1790/12/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/1790 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient xml findbugs checkstyle markdownlint |
| uname | Linux f37996238dfa 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19 08:06:28 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | personality/hadoop.sh |
| git revision | trunk / 42dfd27 |
| Default Java | 1.8.0_242 |
| checkstyle | https://builds.apache.org/job/hadoop-multibranch/job/PR-1790/12/artifact/out/diff-checkstyle-hadoop-tools_hadoop-azure.txt |
| findbugs | https://builds.apache.org/job/hadoop-multibranch/job/PR-1790/12/artifact/out/new-findbugs-hadoop-tools_hadoop-azure.html |
| Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1790/12/testReport/ |
| Max. process+thread count | 311 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1790/12/console |
| versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on a change in pull request #1790:
[HADOOP-16818] ABFS: Combine append+flush calls for blockblob & appendblob
Posted by GitBox <gi...@apache.org>.
steveloughran commented on a change in pull request #1790: [HADOOP-16818] ABFS: Combine append+flush calls for blockblob & appendblob
URL: https://github.com/apache/hadoop/pull/1790#discussion_r386481585
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/constants/ConfigurationKeys.java
##########
@@ -60,6 +61,12 @@
* documentation does not have such expectations of data being persisted.
* Default value of this config is true. **/
public static final String FS_AZURE_DISABLE_OUTPUTSTREAM_FLUSH = "fs.azure.disable.outputstream.flush";
+ public static final String FS_AZURE_ENABLE_APPEND_WITH_FLUSH = "fs.azure.enable.appendwithflush";
+ /** Provides a config control to disable or enable OutputStream Flush API
Review comment:
javadocs should be above the new option
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus removed a comment on issue #1790:
[HADOOP-16818] ABFS: Combine append+flush calls for blockblob & appendblob
Posted by GitBox <gi...@apache.org>.
hadoop-yetus removed a comment on issue #1790: [HADOOP-16818] ABFS: Combine append+flush calls for blockblob & appendblob
URL: https://github.com/apache/hadoop/pull/1790#issuecomment-585720440
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 0m 33s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | No case conflicting files found. |
| +0 :ok: | markdownlint | 0m 0s | markdownlint was not available. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 1 new or modified test files. |
||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 18m 56s | trunk passed |
| +1 :green_heart: | compile | 0m 31s | trunk passed |
| +1 :green_heart: | checkstyle | 0m 25s | trunk passed |
| +1 :green_heart: | mvnsite | 0m 34s | trunk passed |
| +1 :green_heart: | shadedclient | 14m 46s | branch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 25s | trunk passed |
| +0 :ok: | spotbugs | 0m 51s | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 0m 49s | trunk passed |
||| _ Patch Compile Tests _ |
| -1 :x: | mvninstall | 0m 14s | hadoop-azure in the patch failed. |
| -1 :x: | compile | 0m 15s | hadoop-azure in the patch failed. |
| -1 :x: | javac | 0m 15s | hadoop-azure in the patch failed. |
| -0 :warning: | checkstyle | 0m 12s | The patch fails to run checkstyle in hadoop-azure |
| -1 :x: | mvnsite | 0m 16s | hadoop-azure in the patch failed. |
| -1 :x: | whitespace | 0m 0s | The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply |
| +1 :green_heart: | xml | 0m 2s | The patch has no ill-formed XML file. |
| +1 :green_heart: | shadedclient | 13m 51s | patch has no errors when building and testing our client artifacts. |
| -1 :x: | javadoc | 0m 19s | hadoop-azure in the patch failed. |
| -1 :x: | findbugs | 0m 18s | hadoop-azure in the patch failed. |
||| _ Other Tests _ |
| -1 :x: | unit | 0m 18s | hadoop-azure in the patch failed. |
| +1 :green_heart: | asflicense | 0m 32s | The patch does not generate ASF License warnings. |
| | | 54m 48s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | Client=19.03.6 Server=19.03.6 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1790/7/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/1790 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient xml findbugs checkstyle markdownlint |
| uname | Linux 2d8ef296c6b8 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | personality/hadoop.sh |
| git revision | trunk / fe7d67a |
| Default Java | 1.8.0_242 |
| mvninstall | https://builds.apache.org/job/hadoop-multibranch/job/PR-1790/7/artifact/out/patch-mvninstall-hadoop-tools_hadoop-azure.txt |
| compile | https://builds.apache.org/job/hadoop-multibranch/job/PR-1790/7/artifact/out/patch-compile-hadoop-tools_hadoop-azure.txt |
| javac | https://builds.apache.org/job/hadoop-multibranch/job/PR-1790/7/artifact/out/patch-compile-hadoop-tools_hadoop-azure.txt |
| checkstyle | https://builds.apache.org/job/hadoop-multibranch/job/PR-1790/7/artifact/out//home/jenkins/jenkins-slave/workspace/hadoop-multibranch_PR-1790/out/maven-patch-checkstyle-hadoop-tools_hadoop-azure.txt |
| mvnsite | https://builds.apache.org/job/hadoop-multibranch/job/PR-1790/7/artifact/out/patch-mvnsite-hadoop-tools_hadoop-azure.txt |
| whitespace | https://builds.apache.org/job/hadoop-multibranch/job/PR-1790/7/artifact/out/whitespace-eol.txt |
| javadoc | https://builds.apache.org/job/hadoop-multibranch/job/PR-1790/7/artifact/out/patch-javadoc-hadoop-tools_hadoop-azure.txt |
| findbugs | https://builds.apache.org/job/hadoop-multibranch/job/PR-1790/7/artifact/out/patch-findbugs-hadoop-tools_hadoop-azure.txt |
| unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-1790/7/artifact/out/patch-unit-hadoop-tools_hadoop-azure.txt |
| Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1790/7/testReport/ |
| Max. process+thread count | 413 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1790/7/console |
| versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] ishaniahuja commented on issue #1790: [HADOOP-16818] ABFS:
Combine append+flush calls for blockblob & appendblob
Posted by GitBox <gi...@apache.org>.
ishaniahuja commented on issue #1790: [HADOOP-16818] ABFS: Combine append+flush calls for blockblob & appendblob
URL: https://github.com/apache/hadoop/pull/1790#issuecomment-597457995
> going to add @bgaborg as a reviewer.
>
> Now, something fundamental: what are blockblobs and appendblobs?
@steveloughran Some documentation on blockblob and appendblob: https://docs.microsoft.com/en-us/rest/api/storageservices/understanding-block-blobs--append-blobs--and-page-blobs
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] ishaniahuja commented on a change in pull request #1790:
[HADOOP-16818] ABFS: Combine append+flush calls for blockblob & appendblob
Posted by GitBox <gi...@apache.org>.
ishaniahuja commented on a change in pull request #1790: [HADOOP-16818] ABFS: Combine append+flush calls for blockblob & appendblob
URL: https://github.com/apache/hadoop/pull/1790#discussion_r378767249
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AbfsConfiguration.java
##########
@@ -158,6 +162,14 @@
DefaultValue = DEFAULT_DISABLE_OUTPUTSTREAM_FLUSH)
private boolean disableOutputStreamFlush;
+ @BooleanConfigurationValidatorAnnotation(ConfigurationKey = FS_AZURE_ENABLE_APPEND_WITH_FLUSH,
+ DefaultValue = DEFAULT_ENABLE_APPEND_WITH_FLUSH)
+ private boolean enableAppendWithFlush;
+
+ @BooleanConfigurationValidatorAnnotation(ConfigurationKey = FS_AZURE_DISABLE_OUTPUTSTREAM_FLUSH,
Review comment:
there was a conflict for which I had to add this.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] snvijaya commented on a change in pull request #1790: changes for combined append+flush calls for blockblob(traditionan) an…
Posted by GitBox <gi...@apache.org>.
snvijaya commented on a change in pull request #1790: changes for combined append+flush calls for blockblob(traditionan) an…
URL: https://github.com/apache/hadoop/pull/1790#discussion_r363151735
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AbfsConfiguration.java
##########
@@ -411,6 +419,10 @@ public String getAzureAtomicRenameDirs() {
return this.azureAtomicDirs;
}
+ public String getAppendBlobDirs() {
Review comment:
Why not return a HashSet as all callers are going to need it so.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] DadanielZ commented on a change in pull request #1790:
[HADOOP-16818] ABFS: Combine append+flush calls for blockblob & appendblob
Posted by GitBox <gi...@apache.org>.
DadanielZ commented on a change in pull request #1790: [HADOOP-16818] ABFS: Combine append+flush calls for blockblob & appendblob
URL: https://github.com/apache/hadoop/pull/1790#discussion_r370478260
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AzureBlobFileSystemStore.java
##########
@@ -344,40 +366,53 @@ public void deleteFilesystem() throws AzureBlobFileSystemException {
public OutputStream createFile(final Path path, final boolean overwrite, final FsPermission permission,
final FsPermission umask) throws AzureBlobFileSystemException {
- boolean isNamespaceEnabled = getIsNamespaceEnabled();
- LOG.debug("createFile filesystem: {} path: {} overwrite: {} permission: {} umask: {} isNamespaceEnabled: {}",
- client.getFileSystem(),
- path,
- overwrite,
- permission.toString(),
- umask.toString(),
- isNamespaceEnabled);
-
- client.createPath(AbfsHttpConstants.FORWARD_SLASH + getRelativePath(path), true, overwrite,
- isNamespaceEnabled ? getOctalNotation(permission) : null,
- isNamespaceEnabled ? getOctalNotation(umask) : null);
+ try (AbfsPerfInfo perfInfo = startTracking("createFile", "createPath")) {
+ boolean isNamespaceEnabled = getIsNamespaceEnabled();
+ LOG.debug("createFile filesystem: {} path: {} overwrite: {} permission: {} umask: {} isNamespaceEnabled: {}",
+ client.getFileSystem(),
+ path,
+ overwrite,
+ permission.toString(),
+ umask.toString(),
+ isNamespaceEnabled);
+
+ boolean appendBlob = false;
+ if (isAppendBlobKey(path.toString())) {
+ appendBlob = true;
+ }
- return new AbfsOutputStream(
- client,
- AbfsHttpConstants.FORWARD_SLASH + getRelativePath(path),
- 0,
- abfsConfiguration.getWriteBufferSize(),
- abfsConfiguration.isFlushEnabled());
+ client.createPath(AbfsHttpConstants.FORWARD_SLASH + getRelativePath(path), true, overwrite,
+ isNamespaceEnabled ? getOctalNotation(permission) : null,
+ isNamespaceEnabled ? getOctalNotation(umask) : null,
+ appendBlob);
+
+ return new AbfsOutputStream(
+ client,
+ AbfsHttpConstants.FORWARD_SLASH + getRelativePath(path),
+ 0,
+ abfsConfiguration.getWriteBufferSize(),
+ abfsConfiguration.isFlushEnabled(),
+ abfsConfiguration.isAppendWithFlushEnabled(),
+ appendBlob);
+ }
}
public void createDirectory(final Path path, final FsPermission permission, final FsPermission umask)
throws AzureBlobFileSystemException {
- boolean isNamespaceEnabled = getIsNamespaceEnabled();
- LOG.debug("createDirectory filesystem: {} path: {} permission: {} umask: {} isNamespaceEnabled: {}",
- client.getFileSystem(),
- path,
- permission,
- umask,
- isNamespaceEnabled);
-
- client.createPath(AbfsHttpConstants.FORWARD_SLASH + getRelativePath(path), false, true,
- isNamespaceEnabled ? getOctalNotation(permission) : null,
- isNamespaceEnabled ? getOctalNotation(umask) : null);
+ try (AbfsPerfInfo perfInfo = startTracking("createDirectory", "createPath")) {
Review comment:
Looks like there are some issue in this PR, this try logic is already in:
https://github.com/apache/hadoop/blob/978c487672edd9f18d8e2c9a1da063ae789bd774/hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AzureBlobFileSystemStore.java#L423
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] DadanielZ commented on a change in pull request #1790:
[HADOOP-16818] ABFS: Combine append+flush calls for blockblob & appendblob
Posted by GitBox <gi...@apache.org>.
DadanielZ commented on a change in pull request #1790: [HADOOP-16818] ABFS: Combine append+flush calls for blockblob & appendblob
URL: https://github.com/apache/hadoop/pull/1790#discussion_r390514693
##########
File path: hadoop-tools/hadoop-azure/pom.xml
##########
@@ -260,6 +260,7 @@
<dependency>
<groupId>org.mockito</groupId>
<artifactId>mockito-core</artifactId>
+ <version>3.3.0</version>
Review comment:
version number should not be defined here, please do not add this change.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] bilaharith commented on a change in pull request #1790:
[HADOOP-16818] ABFS: Combine append+flush calls for blockblob & appendblob
Posted by GitBox <gi...@apache.org>.
bilaharith commented on a change in pull request #1790: [HADOOP-16818] ABFS: Combine append+flush calls for blockblob & appendblob
URL: https://github.com/apache/hadoop/pull/1790#discussion_r392655195
##########
File path: hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/services/TestAbfsOutputStream.java
##########
@@ -0,0 +1,355 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.azurebfs.services;
+import java.util.Arrays;
+import java.util.HashSet;
+import java.util.Random;
+import org.junit.Assert;
+import org.junit.Test;
+import org.mockito.ArgumentCaptor;
+import org.apache.hadoop.fs.azurebfs.AbfsConfiguration;
+import org.apache.hadoop.conf.Configuration;
+
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.when;
+import static org.mockito.Mockito.times;
+import static org.mockito.Mockito.verify;
+import static org.mockito.Mockito.verifyNoMoreInteractions;
+import static org.mockito.Mockito.*;
+
+public final class TestAbfsOutputStream {
+
+ private static int bufferSize = 4096;
+ private static int writeSize = 1000;
+ private static String path = "~/testpath";
+ private final String globalKey = "fs.azure.configuration";
+ private final String accountName1 = "account1";
+ private final String accountKey1 = globalKey + "." + accountName1;
+ private final String accountValue1 = "one";
+
+ @Test
+ /**
+ * The test verifies OutputStream shortwrite case(2000bytes write followed by flush, hflush, hsync) is making correct HTTP calls to the server
+ */
+ public void verifyShortWriteRequest() throws Exception {
+
+ AbfsClient client = mock(AbfsClient.class);
+ AbfsRestOperation op = mock(AbfsRestOperation.class);
+ AbfsConfiguration abfsConf;
+ final Configuration conf = new Configuration();
+ conf.set(accountKey1, accountValue1);
+ abfsConf = new AbfsConfiguration(conf, accountName1);
+ AbfsPerfTracker tracker = new AbfsPerfTracker("test", accountName1, abfsConf);
+ when(client.getAbfsPerfTracker()).thenReturn(tracker);
+ when(client.append(anyString(), anyLong(), any(byte[].class), anyInt(), anyInt(), anyBoolean(), anyBoolean())).thenReturn(op);
+
+ AbfsOutputStream out = new AbfsOutputStream(client, path, 0, bufferSize, true, false, true, false);
+ final byte[] b = new byte[writeSize];
+ new Random().nextBytes(b);
+ out.write(b);
+ out.hsync();
+ ArgumentCaptor<String> acString = ArgumentCaptor.forClass(String.class);
+ ArgumentCaptor<Long> acLong = ArgumentCaptor.forClass(Long.class);
+ ArgumentCaptor<Integer> acInt = ArgumentCaptor.forClass(Integer.class);
+ ArgumentCaptor<Boolean> acBool = ArgumentCaptor.forClass(Boolean.class);
+ ArgumentCaptor<byte[]> acByteArray = ArgumentCaptor.forClass(byte[].class);
+
+ final byte[] b1 = new byte[2*writeSize];
+ new Random().nextBytes(b1);
+ out.write(b1);
+ out.flush();
+ out.hflush();
+
+ out.hsync();
+
+ verify(client, times(2)).append(acString.capture(), acLong.capture(), acByteArray.capture(), acInt.capture(), acInt.capture(), acBool.capture(), acBool.capture());
+ Assert.assertEquals(Arrays.asList(path, path) , acString.getAllValues());
+ Assert.assertEquals(Arrays.asList(Long.valueOf(0), Long.valueOf(writeSize)), acLong.getAllValues());
+ //flush=true, close=false, flush=true, close=false
+ Assert.assertEquals(Arrays.asList(true, false, true, false), acBool.getAllValues());
+ Assert.assertEquals(Arrays.asList(0,writeSize, 0, 2*writeSize), acInt.getAllValues());
+
+ }
+
+ @Test
+ /**
+ * The test verifies OutputStream Write of writeSize(1000 bytes) followed by a close is making correct HTTP calls to the server
+ */
+ public void verifyWriteRequest() throws Exception {
+
+ AbfsClient client = mock(AbfsClient.class);
+ AbfsRestOperation op = mock(AbfsRestOperation.class);
+ AbfsConfiguration abfsConf;
+ final Configuration conf = new Configuration();
+ conf.set(accountKey1, accountValue1);
+ abfsConf = new AbfsConfiguration(conf, accountName1);
+ AbfsPerfTracker tracker = new AbfsPerfTracker("test", accountName1, abfsConf);
+
+ when(client.getAbfsPerfTracker()).thenReturn(tracker);
+ when(client.append(anyString(), anyLong(), any(byte[].class), anyInt(), anyInt(), anyBoolean(), anyBoolean())).thenReturn(op);
+
+ AbfsOutputStream out = new AbfsOutputStream(client, path, 0, bufferSize, true, false, true, false);
+ final byte[] b = new byte[writeSize];
+ new Random().nextBytes(b);
+
+ for (int i = 0; i < 5; i++) {
+ out.write(b);
+ }
+ out.close();
+
+ ArgumentCaptor<String> acString = ArgumentCaptor.forClass(String.class);
+ ArgumentCaptor<Long> acLong = ArgumentCaptor.forClass(Long.class);
+ ArgumentCaptor<Integer> acInt = ArgumentCaptor.forClass(Integer.class);
+ ArgumentCaptor<Boolean> acBool = ArgumentCaptor.forClass(Boolean.class);
+ ArgumentCaptor<byte[]> acByteArray = ArgumentCaptor.forClass(byte[].class);
+
+ verify(client, times(2)).append(acString.capture(), acLong.capture(), acByteArray.capture(), acInt.capture(), acInt.capture(), acBool.capture(), acBool.capture());
+ Assert.assertEquals(Arrays.asList(path, path) , acString.getAllValues());
+ Assert.assertEquals(Arrays.asList(Long.valueOf(0), Long.valueOf(bufferSize)), acLong.getAllValues());
+ //flush=false,close=false, flush=true,close=true
+ Assert.assertEquals(Arrays.asList(false, false, true, true), acBool.getAllValues());
+ Assert.assertEquals(Arrays.asList(0, bufferSize, 0, 5*writeSize-bufferSize), acInt.getAllValues());
+
+ }
+
+ @Test
+ /**
+ * The test verifies OutputStream Write of bufferSize(4KB) followed by a close is making correct HTTP calls to the server
+ */
+ public void verifyWriteRequestOfBufferSizeAndClose() throws Exception {
+
+ AbfsClient client = mock(AbfsClient.class);
+ AbfsRestOperation op = mock(AbfsRestOperation.class);
+ AbfsConfiguration abfsConf;
+ final Configuration conf = new Configuration();
+ conf.set(accountKey1, accountValue1);
+ abfsConf = new AbfsConfiguration(conf, accountName1);
+ AbfsPerfTracker tracker = new AbfsPerfTracker("test", accountName1, abfsConf);
+
+ when(client.getAbfsPerfTracker()).thenReturn(tracker);
+ when(client.append(anyString(), anyLong(), any(byte[].class), anyInt(), anyInt(), anyBoolean(), anyBoolean())).thenReturn(op);
+ when(client.flush(anyString(), anyLong(), anyBoolean(), anyBoolean())).thenReturn(op);
+
+ AbfsOutputStream out = new AbfsOutputStream(client, path, 0, bufferSize, true, false, true, false);
+ final byte[] b = new byte[bufferSize];
+ new Random().nextBytes(b);
+
+ for (int i = 0; i < 2; i++) {
+ out.write(b);
+ }
+ out.close();
+
+ ArgumentCaptor<String> acString = ArgumentCaptor.forClass(String.class);
+ ArgumentCaptor<Long> acLong = ArgumentCaptor.forClass(Long.class);
+ ArgumentCaptor<Integer> acInt = ArgumentCaptor.forClass(Integer.class);
+ ArgumentCaptor<Boolean> acBool = ArgumentCaptor.forClass(Boolean.class);
+ ArgumentCaptor<byte[]> acByteArray = ArgumentCaptor.forClass(byte[].class);
+
+ verify(client, times(2)).append(acString.capture(), acLong.capture(), acByteArray.capture(), acInt.capture(), acInt.capture(), acBool.capture(), acBool.capture());
+ Assert.assertEquals(Arrays.asList(path, path) , acString.getAllValues());
+ Assert.assertEquals(new HashSet<Long>(Arrays.asList(Long.valueOf(0), Long.valueOf(bufferSize))), new HashSet<Long>(acLong.getAllValues()));
+ //flush=false, close=false, flush=false, close=false
+ Assert.assertEquals(Arrays.asList(false, false, false, false), acBool.getAllValues());
+ Assert.assertEquals(Arrays.asList(0, bufferSize, 0, bufferSize), acInt.getAllValues());
+
+ ArgumentCaptor<String> acFlushString = ArgumentCaptor.forClass(String.class);
+ ArgumentCaptor<Long> acFlushLong = ArgumentCaptor.forClass(Long.class);
+ ArgumentCaptor<Boolean> acFlushBool = ArgumentCaptor.forClass(Boolean.class);
+
+ verify(client, times(1)).flush(acFlushString.capture(), acFlushLong.capture(), acFlushBool.capture(), acFlushBool.capture());
+ Assert.assertEquals(Arrays.asList(path) , acFlushString.getAllValues());
+ Assert.assertEquals(Arrays.asList(Long.valueOf(2*bufferSize)), acFlushLong.getAllValues());
+ Assert.assertEquals(Arrays.asList(false, true), acFlushBool.getAllValues());
+
+ }
+
+ @Test
+ /**
+ * The test verifies OutputStream Write of bufferSize(4KB) is making correct HTTP calls to the server
+ */
+ public void verifyWriteRequestOfBufferSize() throws Exception {
+
+ AbfsClient client = mock(AbfsClient.class);
+ AbfsRestOperation op = mock(AbfsRestOperation.class);
+ AbfsConfiguration abfsConf;
+ final Configuration conf = new Configuration();
+ conf.set(accountKey1, accountValue1);
+ abfsConf = new AbfsConfiguration(conf, accountName1);
+ AbfsPerfTracker tracker = new AbfsPerfTracker("test", accountName1, abfsConf);
+
+ when(client.getAbfsPerfTracker()).thenReturn(tracker);
+ when(client.append(anyString(), anyLong(), any(byte[].class), anyInt(), anyInt(), anyBoolean(), anyBoolean())).thenReturn(op);
+
+ AbfsOutputStream out = new AbfsOutputStream(client, path, 0, bufferSize, true, false, true, false);
+ final byte[] b = new byte[bufferSize];
+ new Random().nextBytes(b);
+
+ for (int i = 0; i < 2; i++) {
+ out.write(b);
+ }
+ Thread.sleep(1000);
+
+ ArgumentCaptor<String> acString = ArgumentCaptor.forClass(String.class);
+ ArgumentCaptor<Long> acLong = ArgumentCaptor.forClass(Long.class);
+ ArgumentCaptor<Integer> acInt = ArgumentCaptor.forClass(Integer.class);
+ ArgumentCaptor<Boolean> acBool = ArgumentCaptor.forClass(Boolean.class);
+ ArgumentCaptor<byte[]> acByteArray = ArgumentCaptor.forClass(byte[].class);
+
+ verify(client, times(2)).append(acString.capture(), acLong.capture(), acByteArray.capture(), acInt.capture(), acInt.capture(), acBool.capture(), acBool.capture());
+ Assert.assertEquals(Arrays.asList(path, path) , acString.getAllValues());
+ Assert.assertEquals(new HashSet<Long>(Arrays.asList(Long.valueOf(0), Long.valueOf(bufferSize))), new HashSet<Long>(acLong.getAllValues()));
+ //flush=false, close=false, flush=false, close=false
+ Assert.assertEquals(Arrays.asList(false, false, false, false), acBool.getAllValues());
+ Assert.assertEquals(Arrays.asList(0, bufferSize, 0, bufferSize), acInt.getAllValues());
+
+ }
+
+ @Test
+ /**
+ * The test verifies OutputStream Write of bufferSize(4KB) on a AppendBlob based stream is making correct HTTP calls to the server
+ */
+ public void verifyWriteRequestOfBufferSizeWithAppendBlob() throws Exception {
+
+ AbfsClient client = mock(AbfsClient.class);
+ AbfsRestOperation op = mock(AbfsRestOperation.class);
+ AbfsConfiguration abfsConf;
+ final Configuration conf = new Configuration();
+ conf.set(accountKey1, accountValue1);
+ abfsConf = new AbfsConfiguration(conf, accountName1);
+ AbfsPerfTracker tracker = new AbfsPerfTracker("test", accountName1, abfsConf);
+
+ when(client.getAbfsPerfTracker()).thenReturn(tracker);
+ when(client.append(anyString(), anyLong(), any(byte[].class), anyInt(), anyInt(), anyBoolean(), anyBoolean())).thenReturn(op);
+
+ AbfsOutputStream out = new AbfsOutputStream(client, path, 0, bufferSize, true, false, true, false);
+ final byte[] b = new byte[bufferSize];
+ new Random().nextBytes(b);
+
+ for (int i = 0; i < 2; i++) {
+ out.write(b);
+ }
+ Thread.sleep(1000);
+
+ ArgumentCaptor<String> acString = ArgumentCaptor.forClass(String.class);
+ ArgumentCaptor<Long> acLong = ArgumentCaptor.forClass(Long.class);
+ ArgumentCaptor<Integer> acInt = ArgumentCaptor.forClass(Integer.class);
+ ArgumentCaptor<Boolean> acBool = ArgumentCaptor.forClass(Boolean.class);
+ ArgumentCaptor<byte[]> acByteArray = ArgumentCaptor.forClass(byte[].class);
+
+ verify(client, times(2)).append(acString.capture(), acLong.capture(), acByteArray.capture(), acInt.capture(), acInt.capture(), acBool.capture(), acBool.capture());
+ Assert.assertEquals(Arrays.asList(path, path) , acString.getAllValues());
+ Assert.assertEquals(Arrays.asList(Long.valueOf(0), Long.valueOf(bufferSize)), acLong.getAllValues());
+ //flush=false, close=false, flush=false, close=false
+ Assert.assertEquals(Arrays.asList(false, false, false, false), acBool.getAllValues());
+ Assert.assertEquals(Arrays.asList(0, bufferSize, 0, bufferSize), acInt.getAllValues());
+
+ }
+
+ @Test
+ /**
+ * The test verifies OutputStream Write of bufferSize(4KB) followed by a hflush call is making correct HTTP calls to the server
+ */
+ public void verifyWriteRequestOfBufferSizeAndHFlush() throws Exception {
+
+ AbfsClient client = mock(AbfsClient.class);
+ AbfsRestOperation op = mock(AbfsRestOperation.class);
+ AbfsConfiguration abfsConf;
+ final Configuration conf = new Configuration();
+ conf.set(accountKey1, accountValue1);
+ abfsConf = new AbfsConfiguration(conf, accountName1);
+ AbfsPerfTracker tracker = new AbfsPerfTracker("test", accountName1, abfsConf);
+
+ when(client.getAbfsPerfTracker()).thenReturn(tracker);
+ when(client.append(anyString(), anyLong(), any(byte[].class), anyInt(), anyInt(), anyBoolean(), anyBoolean())).thenReturn(op);
+ when(client.flush(anyString(), anyLong(), anyBoolean(), anyBoolean())).thenReturn(op);
+
+ AbfsOutputStream out = new AbfsOutputStream(client, path, 0, bufferSize, true, false, true, false);
+ final byte[] b = new byte[bufferSize];
+ new Random().nextBytes(b);
+
+ for (int i = 0; i < 2; i++) {
+ out.write(b);
+ }
+ out.hflush();
+
+ ArgumentCaptor<String> acString = ArgumentCaptor.forClass(String.class);
+ ArgumentCaptor<Long> acLong = ArgumentCaptor.forClass(Long.class);
+ ArgumentCaptor<Integer> acInt = ArgumentCaptor.forClass(Integer.class);
+ ArgumentCaptor<Boolean> acBool = ArgumentCaptor.forClass(Boolean.class);
+ ArgumentCaptor<byte[]> acByteArray = ArgumentCaptor.forClass(byte[].class);
+
+ verify(client, times(2)).append(acString.capture(), acLong.capture(), acByteArray.capture(), acInt.capture(), acInt.capture(), acBool.capture(), acBool.capture());
+ Assert.assertEquals(Arrays.asList(path, path) , acString.getAllValues());
+ Assert.assertEquals(new HashSet<Long>(Arrays.asList(Long.valueOf(0), Long.valueOf(bufferSize))), new HashSet<Long>(acLong.getAllValues()));
+ //flush=false, close=false, flush=false, close=false
+ Assert.assertEquals(Arrays.asList(false, false, false, false), acBool.getAllValues());
+ Assert.assertEquals(Arrays.asList(0, bufferSize, 0, bufferSize), acInt.getAllValues());
+
+ ArgumentCaptor<String> acFlushString = ArgumentCaptor.forClass(String.class);
+ ArgumentCaptor<Long> acFlushLong = ArgumentCaptor.forClass(Long.class);
+ ArgumentCaptor<Boolean> acFlushBool = ArgumentCaptor.forClass(Boolean.class);
+
+ verify(client, times(1)).flush(acFlushString.capture(), acFlushLong.capture(), acFlushBool.capture(), acFlushBool.capture());
+ Assert.assertEquals(Arrays.asList(path) , acFlushString.getAllValues());
+ Assert.assertEquals(Arrays.asList(Long.valueOf(2*bufferSize)), acFlushLong.getAllValues());
+ Assert.assertEquals(Arrays.asList(false, false), acFlushBool.getAllValues());
+
+ }
+
+ @Test
+ /**
Review comment:
Have the comment above the @Test annotation
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] DadanielZ commented on a change in pull request #1790:
[HADOOP-16818] ABFS: Combine append+flush calls for blockblob & appendblob
Posted by GitBox <gi...@apache.org>.
DadanielZ commented on a change in pull request #1790: [HADOOP-16818] ABFS: Combine append+flush calls for blockblob & appendblob
URL: https://github.com/apache/hadoop/pull/1790#discussion_r370474451
##########
File path: hadoop-tools/hadoop-azure/pom.xml
##########
@@ -194,7 +203,8 @@
<dependency>
<groupId>org.wildfly.openssl</groupId>
<artifactId>wildfly-openssl</artifactId>
- <scope>runtime</scope>
+ <version>1.0.9.Final</version>
+ <!--<scope>runtime</scope>-->
Review comment:
why commented?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] snvijaya commented on a change in pull request #1790: changes for combined append+flush calls for blockblob(traditionan) an…
Posted by GitBox <gi...@apache.org>.
snvijaya commented on a change in pull request #1790: changes for combined append+flush calls for blockblob(traditionan) an…
URL: https://github.com/apache/hadoop/pull/1790#discussion_r363151653
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AbfsConfiguration.java
##########
@@ -130,6 +130,10 @@
DefaultValue = DEFAULT_FS_AZURE_ATOMIC_RENAME_DIRECTORIES)
private String azureAtomicDirs;
+ @StringConfigurationValidatorAnnotation(ConfigurationKey = FS_AZURE_APPEND_BLOB_KEY,
+ DefaultValue = DEFAULT_FS_AZURE_APPEND_BLOB_DIRECTORIES)
+ private String azureAppendBlobDirs;
Review comment:
New configs need documentation. For reference, recent documentation done for flush related config : https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-azure/src/site/markdown/abfs.md#-flush-options
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on issue #1790: [HADOOP-16818]
ABFS: Combine append+flush calls for blockblob & appendblob
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on issue #1790: [HADOOP-16818] ABFS: Combine append+flush calls for blockblob & appendblob
URL: https://github.com/apache/hadoop/pull/1790#issuecomment-587440341
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 30m 22s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +0 :ok: | markdownlint | 0m 1s | markdownlint was not available. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 1 new or modified test files. |
||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 22m 21s | trunk passed |
| +1 :green_heart: | compile | 0m 28s | trunk passed |
| +1 :green_heart: | checkstyle | 0m 24s | trunk passed |
| +1 :green_heart: | mvnsite | 0m 31s | trunk passed |
| +1 :green_heart: | shadedclient | 16m 22s | branch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 23s | trunk passed |
| +0 :ok: | spotbugs | 0m 53s | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 0m 51s | trunk passed |
||| _ Patch Compile Tests _ |
| -1 :x: | mvninstall | 0m 23s | hadoop-azure in the patch failed. |
| -1 :x: | compile | 0m 22s | hadoop-azure in the patch failed. |
| -1 :x: | javac | 0m 22s | hadoop-azure in the patch failed. |
| -0 :warning: | checkstyle | 0m 15s | hadoop-tools/hadoop-azure: The patch generated 22 new + 5 unchanged - 0 fixed = 27 total (was 5) |
| -1 :x: | mvnsite | 0m 23s | hadoop-azure in the patch failed. |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | xml | 0m 1s | The patch has no ill-formed XML file. |
| +1 :green_heart: | shadedclient | 15m 52s | patch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 21s | the patch passed |
| -1 :x: | findbugs | 0m 27s | hadoop-azure in the patch failed. |
||| _ Other Tests _ |
| -1 :x: | unit | 0m 28s | hadoop-azure in the patch failed. |
| +1 :green_heart: | asflicense | 0m 29s | The patch does not generate ASF License warnings. |
| | | 92m 10s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | Client=19.03.6 Server=19.03.6 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1790/8/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/1790 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient xml findbugs checkstyle markdownlint |
| uname | Linux 9c898717ba62 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19 08:06:28 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | personality/hadoop.sh |
| git revision | trunk / a562942 |
| Default Java | 1.8.0_242 |
| mvninstall | https://builds.apache.org/job/hadoop-multibranch/job/PR-1790/8/artifact/out/patch-mvninstall-hadoop-tools_hadoop-azure.txt |
| compile | https://builds.apache.org/job/hadoop-multibranch/job/PR-1790/8/artifact/out/patch-compile-hadoop-tools_hadoop-azure.txt |
| javac | https://builds.apache.org/job/hadoop-multibranch/job/PR-1790/8/artifact/out/patch-compile-hadoop-tools_hadoop-azure.txt |
| checkstyle | https://builds.apache.org/job/hadoop-multibranch/job/PR-1790/8/artifact/out/diff-checkstyle-hadoop-tools_hadoop-azure.txt |
| mvnsite | https://builds.apache.org/job/hadoop-multibranch/job/PR-1790/8/artifact/out/patch-mvnsite-hadoop-tools_hadoop-azure.txt |
| findbugs | https://builds.apache.org/job/hadoop-multibranch/job/PR-1790/8/artifact/out/patch-findbugs-hadoop-tools_hadoop-azure.txt |
| unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-1790/8/artifact/out/patch-unit-hadoop-tools_hadoop-azure.txt |
| Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1790/8/testReport/ |
| Max. process+thread count | 307 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1790/8/console |
| versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] bilaharith commented on a change in pull request #1790: changes for combined append+flush calls for blockblob(traditionan) an…
Posted by GitBox <gi...@apache.org>.
bilaharith commented on a change in pull request #1790: changes for combined append+flush calls for blockblob(traditionan) an…
URL: https://github.com/apache/hadoop/pull/1790#discussion_r363225156
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AzureBlobFileSystemStore.java
##########
@@ -164,6 +169,23 @@ public AzureBlobFileSystemStore(URI uri, boolean isSecureScheme, Configuration c
boolean useHttps = (usingOauth || abfsConfiguration.isHttpsAlwaysUsed()) ? true : isSecureScheme;
initializeClient(uri, fileSystemName, accountName, useHttps);
this.identityTransformer = new IdentityTransformer(abfsConfiguration.getRawConfiguration());
+
+ // Extract the directories that should contain page blobs
Review comment:
Shall we have these operations as a separate private method and call that from this constructor. This constructor is becoming too long.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on issue #1790: [HADOOP-16818]
ABFS: Combine append+flush calls for blockblob & appendblob
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on issue #1790: [HADOOP-16818] ABFS: Combine append+flush calls for blockblob & appendblob
URL: https://github.com/apache/hadoop/pull/1790#issuecomment-583353780
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 1m 22s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +0 :ok: | markdownlint | 0m 1s | markdownlint was not available. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 1 new or modified test files. |
||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 23m 23s | trunk passed |
| +1 :green_heart: | compile | 0m 28s | trunk passed |
| +1 :green_heart: | checkstyle | 0m 22s | trunk passed |
| +1 :green_heart: | mvnsite | 0m 32s | trunk passed |
| +1 :green_heart: | shadedclient | 16m 35s | branch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 23s | trunk passed |
| +0 :ok: | spotbugs | 1m 1s | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 0m 59s | trunk passed |
||| _ Patch Compile Tests _ |
| -1 :x: | mvninstall | 0m 13s | hadoop-azure in the patch failed. |
| -1 :x: | compile | 0m 14s | hadoop-azure in the patch failed. |
| -1 :x: | javac | 0m 14s | hadoop-azure in the patch failed. |
| -0 :warning: | checkstyle | 0m 11s | The patch fails to run checkstyle in hadoop-azure |
| -1 :x: | mvnsite | 0m 16s | hadoop-azure in the patch failed. |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | xml | 0m 1s | The patch has no ill-formed XML file. |
| +1 :green_heart: | shadedclient | 15m 52s | patch has no errors when building and testing our client artifacts. |
| -1 :x: | javadoc | 0m 16s | hadoop-azure in the patch failed. |
| -1 :x: | findbugs | 0m 16s | hadoop-azure in the patch failed. |
||| _ Other Tests _ |
| -1 :x: | unit | 0m 15s | hadoop-azure in the patch failed. |
| +1 :green_heart: | asflicense | 0m 29s | The patch does not generate ASF License warnings. |
| | | 63m 36s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | Client=19.03.5 Server=19.03.5 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1790/4/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/1790 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient xml findbugs checkstyle markdownlint |
| uname | Linux 73f05f3de826 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19 08:06:28 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | personality/hadoop.sh |
| git revision | trunk / fafe78f |
| Default Java | 1.8.0_232 |
| mvninstall | https://builds.apache.org/job/hadoop-multibranch/job/PR-1790/4/artifact/out/patch-mvninstall-hadoop-tools_hadoop-azure.txt |
| compile | https://builds.apache.org/job/hadoop-multibranch/job/PR-1790/4/artifact/out/patch-compile-hadoop-tools_hadoop-azure.txt |
| javac | https://builds.apache.org/job/hadoop-multibranch/job/PR-1790/4/artifact/out/patch-compile-hadoop-tools_hadoop-azure.txt |
| checkstyle | https://builds.apache.org/job/hadoop-multibranch/job/PR-1790/4/artifact/out//home/jenkins/jenkins-slave/workspace/hadoop-multibranch_PR-1790/out/maven-patch-checkstyle-hadoop-tools_hadoop-azure.txt |
| mvnsite | https://builds.apache.org/job/hadoop-multibranch/job/PR-1790/4/artifact/out/patch-mvnsite-hadoop-tools_hadoop-azure.txt |
| javadoc | https://builds.apache.org/job/hadoop-multibranch/job/PR-1790/4/artifact/out/patch-javadoc-hadoop-tools_hadoop-azure.txt |
| findbugs | https://builds.apache.org/job/hadoop-multibranch/job/PR-1790/4/artifact/out/patch-findbugs-hadoop-tools_hadoop-azure.txt |
| unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-1790/4/artifact/out/patch-unit-hadoop-tools_hadoop-azure.txt |
| Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1790/4/testReport/ |
| Max. process+thread count | 311 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1790/4/console |
| versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] ishaniahuja commented on a change in pull request #1790:
[HADOOP-16818] ABFS: Combine append+flush calls for blockblob & appendblob
Posted by GitBox <gi...@apache.org>.
ishaniahuja commented on a change in pull request #1790: [HADOOP-16818] ABFS: Combine append+flush calls for blockblob & appendblob
URL: https://github.com/apache/hadoop/pull/1790#discussion_r393423470
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsOutputStream.java
##########
@@ -55,6 +55,8 @@
private boolean closed;
private boolean supportFlush;
private boolean disableOutputStreamFlush;
+ private boolean supportAppendWithFlush;
+ private boolean appendBlob;
Review comment:
it is the same as for other booleans in the file.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] snvijaya commented on issue #1790: [HADOOP-16818] ABFS:
Combine append+flush calls for blockblob & appendblob
Posted by GitBox <gi...@apache.org>.
snvijaya commented on issue #1790: [HADOOP-16818] ABFS: Combine append+flush calls for blockblob & appendblob
URL: https://github.com/apache/hadoop/pull/1790#issuecomment-585636885
Yetus has given a -1. Please check that too.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] snvijaya commented on a change in pull request #1790:
[HADOOP-16818] ABFS: Combine append+flush calls for blockblob & appendblob
Posted by GitBox <gi...@apache.org>.
snvijaya commented on a change in pull request #1790: [HADOOP-16818] ABFS: Combine append+flush calls for blockblob & appendblob
URL: https://github.com/apache/hadoop/pull/1790#discussion_r388085796
##########
File path: hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/ITestAzureBlobFileSystemE2E.java
##########
@@ -209,9 +209,10 @@ public void testFlushWithFileNotFoundException() throws Exception {
fs.delete(testFilePath, true);
assertFalse(fs.exists(testFilePath));
+ AbfsConfiguration configuration = this.getConfiguration();
- intercept(FileNotFoundException.class,
- () -> stream.close());
+ //With the new code, it would not trigger a call to the backend
Review comment:
Could you please point me to the check in close() call that prevents server call ?
Also have the tests been run on these different configurations:
1. With appenblob feature enabled
- With and Without Namespace enabled account : fs.azure.enable.appendwithflush true and false
2. With appenblob feature disabled
- With and Without Namespace enabled account : fs.azure.enable.appendwithflush true and false
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] bilaharith commented on a change in pull request #1790:
[HADOOP-16818] ABFS: Combine append+flush calls for blockblob & appendblob
Posted by GitBox <gi...@apache.org>.
bilaharith commented on a change in pull request #1790: [HADOOP-16818] ABFS: Combine append+flush calls for blockblob & appendblob
URL: https://github.com/apache/hadoop/pull/1790#discussion_r392938515
##########
File path: hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/services/TestAbfsOutputStream.java
##########
@@ -0,0 +1,355 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.azurebfs.services;
+import java.util.Arrays;
+import java.util.HashSet;
+import java.util.Random;
+import org.junit.Assert;
+import org.junit.Test;
+import org.mockito.ArgumentCaptor;
+import org.apache.hadoop.fs.azurebfs.AbfsConfiguration;
+import org.apache.hadoop.conf.Configuration;
+
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.when;
+import static org.mockito.Mockito.times;
+import static org.mockito.Mockito.verify;
+import static org.mockito.Mockito.verifyNoMoreInteractions;
+import static org.mockito.Mockito.*;
+
+public final class TestAbfsOutputStream {
+
+ private static int bufferSize = 4096;
+ private static int writeSize = 1000;
+ private static String path = "~/testpath";
+ private final String globalKey = "fs.azure.configuration";
+ private final String accountName1 = "account1";
+ private final String accountKey1 = globalKey + "." + accountName1;
+ private final String accountValue1 = "one";
+
+ @Test
+ /**
+ * The test verifies OutputStream shortwrite case(2000bytes write followed by flush, hflush, hsync) is making correct HTTP calls to the server
+ */
+ public void verifyShortWriteRequest() throws Exception {
+
+ AbfsClient client = mock(AbfsClient.class);
+ AbfsRestOperation op = mock(AbfsRestOperation.class);
+ AbfsConfiguration abfsConf;
+ final Configuration conf = new Configuration();
+ conf.set(accountKey1, accountValue1);
+ abfsConf = new AbfsConfiguration(conf, accountName1);
+ AbfsPerfTracker tracker = new AbfsPerfTracker("test", accountName1, abfsConf);
+ when(client.getAbfsPerfTracker()).thenReturn(tracker);
+ when(client.append(anyString(), anyLong(), any(byte[].class), anyInt(), anyInt(), anyBoolean(), anyBoolean())).thenReturn(op);
+
+ AbfsOutputStream out = new AbfsOutputStream(client, path, 0, bufferSize, true, false, true, false);
+ final byte[] b = new byte[writeSize];
+ new Random().nextBytes(b);
+ out.write(b);
+ out.hsync();
+ ArgumentCaptor<String> acString = ArgumentCaptor.forClass(String.class);
+ ArgumentCaptor<Long> acLong = ArgumentCaptor.forClass(Long.class);
+ ArgumentCaptor<Integer> acInt = ArgumentCaptor.forClass(Integer.class);
+ ArgumentCaptor<Boolean> acBool = ArgumentCaptor.forClass(Boolean.class);
+ ArgumentCaptor<byte[]> acByteArray = ArgumentCaptor.forClass(byte[].class);
+
+ final byte[] b1 = new byte[2*writeSize];
+ new Random().nextBytes(b1);
+ out.write(b1);
+ out.flush();
+ out.hflush();
+
+ out.hsync();
+
+ verify(client, times(2)).append(acString.capture(), acLong.capture(), acByteArray.capture(), acInt.capture(), acInt.capture(), acBool.capture(), acBool.capture());
+ Assert.assertEquals(Arrays.asList(path, path) , acString.getAllValues());
Review comment:
Nit: ',' is preceded with whitespace
Take care of the same in the entire file
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on issue #1790: [HADOOP-16818]
ABFS: Combine append+flush calls for blockblob & appendblob
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on issue #1790: [HADOOP-16818] ABFS: Combine append+flush calls for blockblob & appendblob
URL: https://github.com/apache/hadoop/pull/1790#issuecomment-578686362
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 0m 0s | Docker mode activated. |
| -1 :x: | patch | 0m 8s | https://github.com/apache/hadoop/pull/1790 does not apply to trunk. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. |
| Subsystem | Report/Notes |
|----------:|:-------------|
| GITHUB PR | https://github.com/apache/hadoop/pull/1790 |
| Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1790/1/console |
| versions | git=2.17.1 |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] bilaharith commented on a change in pull request #1790:
[HADOOP-16818] ABFS: Combine append+flush calls for blockblob & appendblob
Posted by GitBox <gi...@apache.org>.
bilaharith commented on a change in pull request #1790: [HADOOP-16818] ABFS: Combine append+flush calls for blockblob & appendblob
URL: https://github.com/apache/hadoop/pull/1790#discussion_r392655852
##########
File path: hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/services/TestAbfsOutputStream.java
##########
@@ -0,0 +1,355 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.azurebfs.services;
+import java.util.Arrays;
Review comment:
Pls add new line before the first import
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] DadanielZ commented on a change in pull request #1790:
[HADOOP-16818] ABFS: Combine append+flush calls for blockblob & appendblob
Posted by GitBox <gi...@apache.org>.
DadanielZ commented on a change in pull request #1790: [HADOOP-16818] ABFS: Combine append+flush calls for blockblob & appendblob
URL: https://github.com/apache/hadoop/pull/1790#discussion_r370474339
##########
File path: hadoop-tools/hadoop-azure/pom.xml
##########
@@ -194,7 +203,8 @@
<dependency>
<groupId>org.wildfly.openssl</groupId>
<artifactId>wildfly-openssl</artifactId>
- <scope>runtime</scope>
+ <version>1.0.9.Final</version>
Review comment:
we are using 1.0.7.final: https://github.com/apache/hadoop/blob/978c487672edd9f18d8e2c9a1da063ae789bd774/hadoop-project/pom.xml#L199
any special reason to upgrade?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] DadanielZ commented on a change in pull request #1790:
[HADOOP-16818] ABFS: Combine append+flush calls for blockblob & appendblob
Posted by GitBox <gi...@apache.org>.
DadanielZ commented on a change in pull request #1790: [HADOOP-16818] ABFS: Combine append+flush calls for blockblob & appendblob
URL: https://github.com/apache/hadoop/pull/1790#discussion_r370473503
##########
File path: hadoop-tools/hadoop-azure/pom.xml
##########
@@ -143,6 +143,15 @@
<!-- see hadoop-project/pom.xml for version number declarations -->
<dependencies>
+<!--
+ <dependency>
+ <groupId>org.mockito</groupId>
+ <artifactId>mockito-all</artifactId>
+ <version>1.8.4</version>
+ <scope>test</scope>
+ </dependency>
+-->
Review comment:
removed this commented section if not used
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus removed a comment on issue #1790:
[HADOOP-16818] ABFS: Combine append+flush calls for blockblob & appendblob
Posted by GitBox <gi...@apache.org>.
hadoop-yetus removed a comment on issue #1790: [HADOOP-16818] ABFS: Combine append+flush calls for blockblob & appendblob
URL: https://github.com/apache/hadoop/pull/1790#issuecomment-583380344
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 1m 24s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | No case conflicting files found. |
| +0 :ok: | markdownlint | 0m 0s | markdownlint was not available. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 1 new or modified test files. |
||| _ trunk Compile Tests _ |
| -1 :x: | mvninstall | 27m 52s | root in trunk failed. |
| +1 :green_heart: | compile | 0m 35s | trunk passed |
| +1 :green_heart: | checkstyle | 0m 35s | trunk passed |
| +1 :green_heart: | mvnsite | 0m 50s | trunk passed |
| +1 :green_heart: | shadedclient | 21m 13s | branch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 25s | trunk passed |
| +0 :ok: | spotbugs | 0m 50s | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 0m 47s | trunk passed |
||| _ Patch Compile Tests _ |
| -1 :x: | mvninstall | 0m 15s | hadoop-azure in the patch failed. |
| -1 :x: | compile | 0m 15s | hadoop-azure in the patch failed. |
| -1 :x: | javac | 0m 15s | hadoop-azure in the patch failed. |
| -0 :warning: | checkstyle | 0m 14s | The patch fails to run checkstyle in hadoop-azure |
| -1 :x: | mvnsite | 0m 16s | hadoop-azure in the patch failed. |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | xml | 0m 2s | The patch has no ill-formed XML file. |
| +1 :green_heart: | shadedclient | 13m 38s | patch has no errors when building and testing our client artifacts. |
| -1 :x: | javadoc | 0m 19s | hadoop-azure in the patch failed. |
| -1 :x: | findbugs | 0m 18s | hadoop-azure in the patch failed. |
||| _ Other Tests _ |
| -1 :x: | unit | 0m 18s | hadoop-azure in the patch failed. |
| +1 :green_heart: | asflicense | 0m 32s | The patch does not generate ASF License warnings. |
| | | 71m 19s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | Client=19.03.5 Server=19.03.5 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1790/5/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/1790 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient xml findbugs checkstyle markdownlint |
| uname | Linux 7745ce42e0ec 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | personality/hadoop.sh |
| git revision | trunk / fafe78f |
| Default Java | 1.8.0_242 |
| mvninstall | https://builds.apache.org/job/hadoop-multibranch/job/PR-1790/5/artifact/out/branch-mvninstall-root.txt |
| mvninstall | https://builds.apache.org/job/hadoop-multibranch/job/PR-1790/5/artifact/out/patch-mvninstall-hadoop-tools_hadoop-azure.txt |
| compile | https://builds.apache.org/job/hadoop-multibranch/job/PR-1790/5/artifact/out/patch-compile-hadoop-tools_hadoop-azure.txt |
| javac | https://builds.apache.org/job/hadoop-multibranch/job/PR-1790/5/artifact/out/patch-compile-hadoop-tools_hadoop-azure.txt |
| checkstyle | https://builds.apache.org/job/hadoop-multibranch/job/PR-1790/5/artifact/out//home/jenkins/jenkins-slave/workspace/hadoop-multibranch_PR-1790/out/maven-patch-checkstyle-hadoop-tools_hadoop-azure.txt |
| mvnsite | https://builds.apache.org/job/hadoop-multibranch/job/PR-1790/5/artifact/out/patch-mvnsite-hadoop-tools_hadoop-azure.txt |
| javadoc | https://builds.apache.org/job/hadoop-multibranch/job/PR-1790/5/artifact/out/patch-javadoc-hadoop-tools_hadoop-azure.txt |
| findbugs | https://builds.apache.org/job/hadoop-multibranch/job/PR-1790/5/artifact/out/patch-findbugs-hadoop-tools_hadoop-azure.txt |
| unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-1790/5/artifact/out/patch-unit-hadoop-tools_hadoop-azure.txt |
| Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1790/5/testReport/ |
| Max. process+thread count | 413 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1790/5/console |
| versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] snvijaya commented on a change in pull request #1790:
[HADOOP-16818] ABFS: Combine append+flush calls for blockblob & appendblob
Posted by GitBox <gi...@apache.org>.
snvijaya commented on a change in pull request #1790: [HADOOP-16818] ABFS: Combine append+flush calls for blockblob & appendblob
URL: https://github.com/apache/hadoop/pull/1790#discussion_r378400446
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsOutputStream.java
##########
@@ -307,10 +318,18 @@ public Void call() throws Exception {
perfInfo.registerSuccess(true);
return null;
}
+ if (flush) {
+ while(lastTotalAppendOffset < lastFlushOffset);
Review comment:
Could you explain the logic here. Please add to code comments.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on issue #1790: [HADOOP-16818]
ABFS: Combine append+flush calls for blockblob & appendblob
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on issue #1790: [HADOOP-16818] ABFS: Combine append+flush calls for blockblob & appendblob
URL: https://github.com/apache/hadoop/pull/1790#issuecomment-594470718
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 3m 13s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | No case conflicting files found. |
| +0 :ok: | markdownlint | 0m 0s | markdownlint was not available. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 2 new or modified test files. |
||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 23m 27s | trunk passed |
| +1 :green_heart: | compile | 0m 28s | trunk passed |
| +1 :green_heart: | checkstyle | 0m 21s | trunk passed |
| +1 :green_heart: | mvnsite | 0m 32s | trunk passed |
| +1 :green_heart: | shadedclient | 16m 24s | branch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 23s | trunk passed |
| +0 :ok: | spotbugs | 0m 55s | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 0m 53s | trunk passed |
||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 26s | the patch passed |
| +1 :green_heart: | compile | 0m 23s | the patch passed |
| +1 :green_heart: | javac | 0m 23s | the patch passed |
| -0 :warning: | checkstyle | 0m 15s | hadoop-tools/hadoop-azure: The patch generated 23 new + 5 unchanged - 0 fixed = 28 total (was 5) |
| +1 :green_heart: | mvnsite | 0m 25s | the patch passed |
| -1 :x: | whitespace | 0m 0s | The patch has 3 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply |
| +1 :green_heart: | xml | 0m 2s | The patch has no ill-formed XML file. |
| +1 :green_heart: | shadedclient | 16m 18s | patch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 22s | the patch passed |
| +1 :green_heart: | findbugs | 1m 2s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 1m 32s | hadoop-azure in the patch passed. |
| +1 :green_heart: | asflicense | 0m 31s | The patch does not generate ASF License warnings. |
| | | 68m 46s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | Client=19.03.7 Server=19.03.7 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1790/15/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/1790 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient xml findbugs checkstyle markdownlint |
| uname | Linux 8292e4c6a320 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19 08:06:28 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | personality/hadoop.sh |
| git revision | trunk / bbd704b |
| Default Java | 1.8.0_242 |
| checkstyle | https://builds.apache.org/job/hadoop-multibranch/job/PR-1790/15/artifact/out/diff-checkstyle-hadoop-tools_hadoop-azure.txt |
| whitespace | https://builds.apache.org/job/hadoop-multibranch/job/PR-1790/15/artifact/out/whitespace-eol.txt |
| Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1790/15/testReport/ |
| Max. process+thread count | 307 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1790/15/console |
| versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] DadanielZ commented on a change in pull request #1790:
[HADOOP-16818] ABFS: Combine append+flush calls for blockblob & appendblob
Posted by GitBox <gi...@apache.org>.
DadanielZ commented on a change in pull request #1790: [HADOOP-16818] ABFS: Combine append+flush calls for blockblob & appendblob
URL: https://github.com/apache/hadoop/pull/1790#discussion_r370476831
##########
File path: hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/services/TestAbfsOutputStream.java
##########
@@ -0,0 +1,268 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.azurebfs.services;
+import java.util.Arrays;
+import java.util.HashSet;
+import java.util.Random;
+import org.junit.Assert;
+import org.junit.Test;
+
+import org.mockito.ArgumentCaptor;
+import static org.mockito.Mockito.*;
Review comment:
remove the *, import the actual class
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on issue #1790: [HADOOP-16818]
ABFS: Combine append+flush calls for blockblob & appendblob
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on issue #1790: [HADOOP-16818] ABFS: Combine append+flush calls for blockblob & appendblob
URL: https://github.com/apache/hadoop/pull/1790#issuecomment-578722809
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 27m 48s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 1 new or modified test files. |
||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 18m 36s | trunk passed |
| +1 :green_heart: | compile | 0m 31s | trunk passed |
| +1 :green_heart: | checkstyle | 0m 24s | trunk passed |
| +1 :green_heart: | mvnsite | 0m 35s | trunk passed |
| +1 :green_heart: | shadedclient | 13m 32s | branch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 26s | trunk passed |
| +0 :ok: | spotbugs | 0m 50s | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 0m 49s | trunk passed |
||| _ Patch Compile Tests _ |
| -1 :x: | mvninstall | 0m 18s | hadoop-azure in the patch failed. |
| -1 :x: | compile | 0m 15s | hadoop-azure in the patch failed. |
| -1 :x: | javac | 0m 15s | hadoop-azure in the patch failed. |
| -0 :warning: | checkstyle | 0m 13s | The patch fails to run checkstyle in hadoop-azure |
| -1 :x: | mvnsite | 0m 17s | hadoop-azure in the patch failed. |
| -1 :x: | whitespace | 0m 0s | The patch 6 line(s) with tabs. |
| +1 :green_heart: | xml | 0m 1s | The patch has no ill-formed XML file. |
| +1 :green_heart: | shadedclient | 13m 32s | patch has no errors when building and testing our client artifacts. |
| -1 :x: | javadoc | 0m 20s | hadoop-azure in the patch failed. |
| -1 :x: | findbugs | 0m 19s | hadoop-azure in the patch failed. |
||| _ Other Tests _ |
| -1 :x: | unit | 0m 19s | hadoop-azure in the patch failed. |
| +1 :green_heart: | asflicense | 0m 31s | The patch does not generate ASF License warnings. |
| | | 80m 15s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | Client=19.03.5 Server=19.03.5 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1790/2/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/1790 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient xml findbugs checkstyle |
| uname | Linux 8c0ffd23132b 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | personality/hadoop.sh |
| git revision | trunk / 7f40e66 |
| Default Java | 1.8.0_232 |
| mvninstall | https://builds.apache.org/job/hadoop-multibranch/job/PR-1790/2/artifact/out/patch-mvninstall-hadoop-tools_hadoop-azure.txt |
| compile | https://builds.apache.org/job/hadoop-multibranch/job/PR-1790/2/artifact/out/patch-compile-hadoop-tools_hadoop-azure.txt |
| javac | https://builds.apache.org/job/hadoop-multibranch/job/PR-1790/2/artifact/out/patch-compile-hadoop-tools_hadoop-azure.txt |
| checkstyle | https://builds.apache.org/job/hadoop-multibranch/job/PR-1790/2/artifact/out//home/jenkins/jenkins-slave/workspace/hadoop-multibranch_PR-1790/out/maven-patch-checkstyle-hadoop-tools_hadoop-azure.txt |
| mvnsite | https://builds.apache.org/job/hadoop-multibranch/job/PR-1790/2/artifact/out/patch-mvnsite-hadoop-tools_hadoop-azure.txt |
| whitespace | https://builds.apache.org/job/hadoop-multibranch/job/PR-1790/2/artifact/out/whitespace-tabs.txt |
| javadoc | https://builds.apache.org/job/hadoop-multibranch/job/PR-1790/2/artifact/out/patch-javadoc-hadoop-tools_hadoop-azure.txt |
| findbugs | https://builds.apache.org/job/hadoop-multibranch/job/PR-1790/2/artifact/out/patch-findbugs-hadoop-tools_hadoop-azure.txt |
| unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-1790/2/artifact/out/patch-unit-hadoop-tools_hadoop-azure.txt |
| Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1790/2/testReport/ |
| Max. process+thread count | 418 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1790/2/console |
| versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] ishaniahuja commented on a change in pull request #1790: changes for combined append+flush calls for blockblob(traditionan) an…
Posted by GitBox <gi...@apache.org>.
ishaniahuja commented on a change in pull request #1790: changes for combined append+flush calls for blockblob(traditionan) an…
URL: https://github.com/apache/hadoop/pull/1790#discussion_r365109068
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AbfsConfiguration.java
##########
@@ -130,6 +130,10 @@
DefaultValue = DEFAULT_FS_AZURE_ATOMIC_RENAME_DIRECTORIES)
private String azureAtomicDirs;
+ @StringConfigurationValidatorAnnotation(ConfigurationKey = FS_AZURE_APPEND_BLOB_KEY,
+ DefaultValue = DEFAULT_FS_AZURE_APPEND_BLOB_DIRECTORIES)
+ private String azureAppendBlobDirs;
Review comment:
what is the procedure for the config edit? who are the owners of this config edit? do I need to email someone for this?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] bilaharith commented on a change in pull request #1790:
[HADOOP-16818] ABFS: Combine append+flush calls for blockblob & appendblob
Posted by GitBox <gi...@apache.org>.
bilaharith commented on a change in pull request #1790: [HADOOP-16818] ABFS: Combine append+flush calls for blockblob & appendblob
URL: https://github.com/apache/hadoop/pull/1790#discussion_r392938996
##########
File path: hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/services/TestAbfsOutputStream.java
##########
@@ -0,0 +1,355 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.azurebfs.services;
+import java.util.Arrays;
+import java.util.HashSet;
+import java.util.Random;
+import org.junit.Assert;
+import org.junit.Test;
+import org.mockito.ArgumentCaptor;
+import org.apache.hadoop.fs.azurebfs.AbfsConfiguration;
+import org.apache.hadoop.conf.Configuration;
+
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.when;
+import static org.mockito.Mockito.times;
+import static org.mockito.Mockito.verify;
+import static org.mockito.Mockito.verifyNoMoreInteractions;
+import static org.mockito.Mockito.*;
+
+public final class TestAbfsOutputStream {
+
+ private static int bufferSize = 4096;
+ private static int writeSize = 1000;
+ private static String path = "~/testpath";
+ private final String globalKey = "fs.azure.configuration";
+ private final String accountName1 = "account1";
+ private final String accountKey1 = globalKey + "." + accountName1;
+ private final String accountValue1 = "one";
+
+ @Test
+ /**
+ * The test verifies OutputStream shortwrite case(2000bytes write followed by flush, hflush, hsync) is making correct HTTP calls to the server
+ */
+ public void verifyShortWriteRequest() throws Exception {
+
+ AbfsClient client = mock(AbfsClient.class);
+ AbfsRestOperation op = mock(AbfsRestOperation.class);
+ AbfsConfiguration abfsConf;
+ final Configuration conf = new Configuration();
+ conf.set(accountKey1, accountValue1);
+ abfsConf = new AbfsConfiguration(conf, accountName1);
+ AbfsPerfTracker tracker = new AbfsPerfTracker("test", accountName1, abfsConf);
+ when(client.getAbfsPerfTracker()).thenReturn(tracker);
+ when(client.append(anyString(), anyLong(), any(byte[].class), anyInt(), anyInt(), anyBoolean(), anyBoolean())).thenReturn(op);
+
+ AbfsOutputStream out = new AbfsOutputStream(client, path, 0, bufferSize, true, false, true, false);
+ final byte[] b = new byte[writeSize];
+ new Random().nextBytes(b);
+ out.write(b);
+ out.hsync();
+ ArgumentCaptor<String> acString = ArgumentCaptor.forClass(String.class);
+ ArgumentCaptor<Long> acLong = ArgumentCaptor.forClass(Long.class);
+ ArgumentCaptor<Integer> acInt = ArgumentCaptor.forClass(Integer.class);
+ ArgumentCaptor<Boolean> acBool = ArgumentCaptor.forClass(Boolean.class);
+ ArgumentCaptor<byte[]> acByteArray = ArgumentCaptor.forClass(byte[].class);
+
+ final byte[] b1 = new byte[2*writeSize];
+ new Random().nextBytes(b1);
+ out.write(b1);
+ out.flush();
+ out.hflush();
+
+ out.hsync();
+
+ verify(client, times(2)).append(acString.capture(), acLong.capture(), acByteArray.capture(), acInt.capture(), acInt.capture(), acBool.capture(), acBool.capture());
+ Assert.assertEquals(Arrays.asList(path, path) , acString.getAllValues());
+ Assert.assertEquals(Arrays.asList(Long.valueOf(0), Long.valueOf(writeSize)), acLong.getAllValues());
+ //flush=true, close=false, flush=true, close=false
+ Assert.assertEquals(Arrays.asList(true, false, true, false), acBool.getAllValues());
+ Assert.assertEquals(Arrays.asList(0,writeSize, 0, 2*writeSize), acInt.getAllValues());
Review comment:
Nit: ',' is not followed by whitespace. space before and after *
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] snvijaya commented on a change in pull request #1790: changes for combined append+flush calls for blockblob(traditionan) an…
Posted by GitBox <gi...@apache.org>.
snvijaya commented on a change in pull request #1790: changes for combined append+flush calls for blockblob(traditionan) an…
URL: https://github.com/apache/hadoop/pull/1790#discussion_r365177009
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsClient.java
##########
@@ -55,7 +55,7 @@
public static final Logger LOG = LoggerFactory.getLogger(AbfsClient.class);
private final URL baseUrl;
private final SharedKeyCredentials sharedKeyCredentials;
- private final String xMsVersion = "2018-11-09";
+ private final String xMsVersion = "2019-12-12";
Review comment:
Server behaviour is controlled with this API version. So until server is ready, we can not bump up the api version support in driver.
This is not recommended to be a config control either, to prevent clients from changing api versions to any older one.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on issue #1790: [HADOOP-16818]
ABFS: Combine append+flush calls for blockblob & appendblob
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on issue #1790: [HADOOP-16818] ABFS: Combine append+flush calls for blockblob & appendblob
URL: https://github.com/apache/hadoop/pull/1790#issuecomment-583380344
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 1m 24s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | No case conflicting files found. |
| +0 :ok: | markdownlint | 0m 0s | markdownlint was not available. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 1 new or modified test files. |
||| _ trunk Compile Tests _ |
| -1 :x: | mvninstall | 27m 52s | root in trunk failed. |
| +1 :green_heart: | compile | 0m 35s | trunk passed |
| +1 :green_heart: | checkstyle | 0m 35s | trunk passed |
| +1 :green_heart: | mvnsite | 0m 50s | trunk passed |
| +1 :green_heart: | shadedclient | 21m 13s | branch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 25s | trunk passed |
| +0 :ok: | spotbugs | 0m 50s | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 0m 47s | trunk passed |
||| _ Patch Compile Tests _ |
| -1 :x: | mvninstall | 0m 15s | hadoop-azure in the patch failed. |
| -1 :x: | compile | 0m 15s | hadoop-azure in the patch failed. |
| -1 :x: | javac | 0m 15s | hadoop-azure in the patch failed. |
| -0 :warning: | checkstyle | 0m 14s | The patch fails to run checkstyle in hadoop-azure |
| -1 :x: | mvnsite | 0m 16s | hadoop-azure in the patch failed. |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | xml | 0m 2s | The patch has no ill-formed XML file. |
| +1 :green_heart: | shadedclient | 13m 38s | patch has no errors when building and testing our client artifacts. |
| -1 :x: | javadoc | 0m 19s | hadoop-azure in the patch failed. |
| -1 :x: | findbugs | 0m 18s | hadoop-azure in the patch failed. |
||| _ Other Tests _ |
| -1 :x: | unit | 0m 18s | hadoop-azure in the patch failed. |
| +1 :green_heart: | asflicense | 0m 32s | The patch does not generate ASF License warnings. |
| | | 71m 19s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | Client=19.03.5 Server=19.03.5 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1790/5/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/1790 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient xml findbugs checkstyle markdownlint |
| uname | Linux 7745ce42e0ec 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | personality/hadoop.sh |
| git revision | trunk / fafe78f |
| Default Java | 1.8.0_242 |
| mvninstall | https://builds.apache.org/job/hadoop-multibranch/job/PR-1790/5/artifact/out/branch-mvninstall-root.txt |
| mvninstall | https://builds.apache.org/job/hadoop-multibranch/job/PR-1790/5/artifact/out/patch-mvninstall-hadoop-tools_hadoop-azure.txt |
| compile | https://builds.apache.org/job/hadoop-multibranch/job/PR-1790/5/artifact/out/patch-compile-hadoop-tools_hadoop-azure.txt |
| javac | https://builds.apache.org/job/hadoop-multibranch/job/PR-1790/5/artifact/out/patch-compile-hadoop-tools_hadoop-azure.txt |
| checkstyle | https://builds.apache.org/job/hadoop-multibranch/job/PR-1790/5/artifact/out//home/jenkins/jenkins-slave/workspace/hadoop-multibranch_PR-1790/out/maven-patch-checkstyle-hadoop-tools_hadoop-azure.txt |
| mvnsite | https://builds.apache.org/job/hadoop-multibranch/job/PR-1790/5/artifact/out/patch-mvnsite-hadoop-tools_hadoop-azure.txt |
| javadoc | https://builds.apache.org/job/hadoop-multibranch/job/PR-1790/5/artifact/out/patch-javadoc-hadoop-tools_hadoop-azure.txt |
| findbugs | https://builds.apache.org/job/hadoop-multibranch/job/PR-1790/5/artifact/out/patch-findbugs-hadoop-tools_hadoop-azure.txt |
| unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-1790/5/artifact/out/patch-unit-hadoop-tools_hadoop-azure.txt |
| Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1790/5/testReport/ |
| Max. process+thread count | 413 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1790/5/console |
| versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on issue #1790: [HADOOP-16818]
ABFS: Combine append+flush calls for blockblob & appendblob
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on issue #1790: [HADOOP-16818] ABFS: Combine append+flush calls for blockblob & appendblob
URL: https://github.com/apache/hadoop/pull/1790#issuecomment-594413615
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 0m 0s | Docker mode activated. |
| -1 :x: | patch | 0m 9s | https://github.com/apache/hadoop/pull/1790 does not apply to trunk. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. |
| Subsystem | Report/Notes |
|----------:|:-------------|
| GITHUB PR | https://github.com/apache/hadoop/pull/1790 |
| Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1790/14/console |
| versions | git=2.17.1 |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] ishaniahuja commented on a change in pull request #1790:
[HADOOP-16818] ABFS: Combine append+flush calls for blockblob & appendblob
Posted by GitBox <gi...@apache.org>.
ishaniahuja commented on a change in pull request #1790: [HADOOP-16818] ABFS: Combine append+flush calls for blockblob & appendblob
URL: https://github.com/apache/hadoop/pull/1790#discussion_r371723337
##########
File path: hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/services/TestAbfsOutputStream.java
##########
@@ -0,0 +1,268 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.azurebfs.services;
+import java.util.Arrays;
+import java.util.HashSet;
+import java.util.Random;
+import org.junit.Assert;
+import org.junit.Test;
+
+import org.mockito.ArgumentCaptor;
+import static org.mockito.Mockito.*;
+/**
+ * Test useragent of abfs client.
+ *
+ */
+public final class TestAbfsOutputStream {
+
+ private static int bufferSize = 4096;
+ private static int writeSize = 1000;
+ private static String path = "~/testpath";
+
+ @Test
+ public void verifyShortWriteRequest() throws Exception {
+
+ AbfsClient client = mock(AbfsClient.class);
+ AbfsOutputStream out = new AbfsOutputStream(client, path, 0, bufferSize, true, true, false);
+ final byte[] b = new byte[writeSize];
+ new Random().nextBytes(b);
+ out.write(b);
+ out.hsync();
+ ArgumentCaptor<String> acString = ArgumentCaptor.forClass(String.class);
+ ArgumentCaptor<Long> acLong = ArgumentCaptor.forClass(Long.class);
+ ArgumentCaptor<Integer> acInt = ArgumentCaptor.forClass(Integer.class);
+ ArgumentCaptor<Boolean> acBool = ArgumentCaptor.forClass(Boolean.class);
+ ArgumentCaptor<byte[]> acByteArray = ArgumentCaptor.forClass(byte[].class);
+
+ final byte[] b1 = new byte[2*writeSize];
+ new Random().nextBytes(b1);
+ out.write(b1);
+ out.flush();
+ out.hflush();
+
+ out.hsync();
+
+ verify(client, times(2)).append(acString.capture(), acLong.capture(), acByteArray.capture(), acInt.capture(), acInt.capture(), acBool.capture(), acBool.capture());
+ Assert.assertEquals(Arrays.asList(path, path) , acString.getAllValues());
+ Assert.assertEquals(Arrays.asList(Long.valueOf(0), Long.valueOf(writeSize)), acLong.getAllValues());
+ //flush=true, close=false, flush=true, close=false
+ Assert.assertEquals(Arrays.asList(true, false, true, false), acBool.getAllValues());
+ Assert.assertEquals(Arrays.asList(0,writeSize, 0, 2*writeSize), acInt.getAllValues());
+
+ verifyNoMoreInteractions(client);
+
+ }
+
+ @Test
+ public void verifyWriteRequest() throws Exception {
+
+ AbfsClient client = mock(AbfsClient.class);
+ AbfsOutputStream out = new AbfsOutputStream(client, path, 0, bufferSize, true, true, false);
+ final byte[] b = new byte[writeSize];
+ new Random().nextBytes(b);
+
+ for (int i = 0; i < 5; i++) {
+ out.write(b);
+ }
+ out.close();
+
+ ArgumentCaptor<String> acString = ArgumentCaptor.forClass(String.class);
+ ArgumentCaptor<Long> acLong = ArgumentCaptor.forClass(Long.class);
+ ArgumentCaptor<Integer> acInt = ArgumentCaptor.forClass(Integer.class);
+ ArgumentCaptor<Boolean> acBool = ArgumentCaptor.forClass(Boolean.class);
+ ArgumentCaptor<byte[]> acByteArray = ArgumentCaptor.forClass(byte[].class);
+
+ verify(client, times(2)).append(acString.capture(), acLong.capture(), acByteArray.capture(), acInt.capture(), acInt.capture(), acBool.capture(), acBool.capture());
+ Assert.assertEquals(Arrays.asList(path, path) , acString.getAllValues());
+ Assert.assertEquals(Arrays.asList(Long.valueOf(0), Long.valueOf(bufferSize)), acLong.getAllValues());
+ //flush=false,close=false, flush=true,close=true
+ Assert.assertEquals(Arrays.asList(false, false, true, true), acBool.getAllValues());
+ Assert.assertEquals(Arrays.asList(0, bufferSize, 0, 5*writeSize-bufferSize), acInt.getAllValues());
+
+ verifyNoMoreInteractions(client);
+ }
+
+ @Test
+ public void verifyWriteRequestOfBufferSizeAndClose() throws Exception {
+
+ AbfsClient client = mock(AbfsClient.class);
+ AbfsOutputStream out = new AbfsOutputStream(client, path, 0, bufferSize, true, true, false);
+ final byte[] b = new byte[bufferSize];
+ new Random().nextBytes(b);
+
+ for (int i = 0; i < 2; i++) {
+ out.write(b);
+ }
+ out.close();
+
+ ArgumentCaptor<String> acString = ArgumentCaptor.forClass(String.class);
+ ArgumentCaptor<Long> acLong = ArgumentCaptor.forClass(Long.class);
+ ArgumentCaptor<Integer> acInt = ArgumentCaptor.forClass(Integer.class);
+ ArgumentCaptor<Boolean> acBool = ArgumentCaptor.forClass(Boolean.class);
+ ArgumentCaptor<byte[]> acByteArray = ArgumentCaptor.forClass(byte[].class);
+
+ verify(client, times(2)).append(acString.capture(), acLong.capture(), acByteArray.capture(), acInt.capture(), acInt.capture(), acBool.capture(), acBool.capture());
+ Assert.assertEquals(Arrays.asList(path, path) , acString.getAllValues());
+ Assert.assertEquals(new HashSet<Long>(Arrays.asList(Long.valueOf(0), Long.valueOf(bufferSize))), new HashSet<Long>(acLong.getAllValues()));
+ //flush=false, close=false, flush=false, close=false
+ Assert.assertEquals(Arrays.asList(false, false, false, false), acBool.getAllValues());
+ Assert.assertEquals(Arrays.asList(0, bufferSize, 0, bufferSize), acInt.getAllValues());
+
+ ArgumentCaptor<String> acFlushString = ArgumentCaptor.forClass(String.class);
+ ArgumentCaptor<Long> acFlushLong = ArgumentCaptor.forClass(Long.class);
+ ArgumentCaptor<Boolean> acFlushBool = ArgumentCaptor.forClass(Boolean.class);
+
+ verify(client, times(1)).flush(acFlushString.capture(), acFlushLong.capture(), acFlushBool.capture(), acFlushBool.capture());
+ Assert.assertEquals(Arrays.asList(path) , acFlushString.getAllValues());
+ Assert.assertEquals(Arrays.asList(Long.valueOf(2*bufferSize)), acFlushLong.getAllValues());
+ Assert.assertEquals(Arrays.asList(false, true), acFlushBool.getAllValues());
+
+ verifyNoMoreInteractions(client);
+ }
+
+ @Test
+ public void verifyWriteRequestOfBufferSize() throws Exception {
+
+ AbfsClient client = mock(AbfsClient.class);
+ AbfsOutputStream out = new AbfsOutputStream(client, path, 0, bufferSize, true, true, false);
+ final byte[] b = new byte[bufferSize];
+ new Random().nextBytes(b);
+
+ for (int i = 0; i < 2; i++) {
+ out.write(b);
+ }
+ Thread.sleep(1000);
+
+ ArgumentCaptor<String> acString = ArgumentCaptor.forClass(String.class);
+ ArgumentCaptor<Long> acLong = ArgumentCaptor.forClass(Long.class);
+ ArgumentCaptor<Integer> acInt = ArgumentCaptor.forClass(Integer.class);
+ ArgumentCaptor<Boolean> acBool = ArgumentCaptor.forClass(Boolean.class);
+ ArgumentCaptor<byte[]> acByteArray = ArgumentCaptor.forClass(byte[].class);
+
+ verify(client, times(2)).append(acString.capture(), acLong.capture(), acByteArray.capture(), acInt.capture(), acInt.capture(), acBool.capture(), acBool.capture());
+ Assert.assertEquals(Arrays.asList(path, path) , acString.getAllValues());
+ Assert.assertEquals(new HashSet<Long>(Arrays.asList(Long.valueOf(0), Long.valueOf(bufferSize))), new HashSet<Long>(acLong.getAllValues()));
+ //flush=false, close=false, flush=false, close=false
+ Assert.assertEquals(Arrays.asList(false, false, false, false), acBool.getAllValues());
+ Assert.assertEquals(Arrays.asList(0, bufferSize, 0, bufferSize), acInt.getAllValues());
+
+ verifyNoMoreInteractions(client);
+ }
+
+ @Test
+ public void verifyWriteRequestOfBufferSizeWithAppendBlob() throws Exception {
+
+ AbfsClient client = mock(AbfsClient.class);
+ AbfsOutputStream out = new AbfsOutputStream(client, path, 0, bufferSize, true, true, true);
+ final byte[] b = new byte[bufferSize];
+ new Random().nextBytes(b);
+
+ for (int i = 0; i < 2; i++) {
+ out.write(b);
+ }
+ Thread.sleep(1000);
+
+ ArgumentCaptor<String> acString = ArgumentCaptor.forClass(String.class);
+ ArgumentCaptor<Long> acLong = ArgumentCaptor.forClass(Long.class);
+ ArgumentCaptor<Integer> acInt = ArgumentCaptor.forClass(Integer.class);
+ ArgumentCaptor<Boolean> acBool = ArgumentCaptor.forClass(Boolean.class);
+ ArgumentCaptor<byte[]> acByteArray = ArgumentCaptor.forClass(byte[].class);
Review comment:
these are just inits.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on issue #1790: [HADOOP-16818]
ABFS: Combine append+flush calls for blockblob & appendblob
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on issue #1790: [HADOOP-16818] ABFS: Combine append+flush calls for blockblob & appendblob
URL: https://github.com/apache/hadoop/pull/1790#issuecomment-588062177
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 0m 50s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +0 :ok: | markdownlint | 0m 0s | markdownlint was not available. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 1 new or modified test files. |
||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 23m 49s | trunk passed |
| +1 :green_heart: | compile | 0m 32s | trunk passed |
| +1 :green_heart: | checkstyle | 0m 26s | trunk passed |
| +1 :green_heart: | mvnsite | 0m 36s | trunk passed |
| +1 :green_heart: | shadedclient | 17m 56s | branch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 29s | trunk passed |
| +0 :ok: | spotbugs | 0m 58s | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 0m 56s | trunk passed |
||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 32s | the patch passed |
| +1 :green_heart: | compile | 0m 25s | the patch passed |
| +1 :green_heart: | javac | 0m 25s | the patch passed |
| -0 :warning: | checkstyle | 0m 18s | hadoop-tools/hadoop-azure: The patch generated 22 new + 5 unchanged - 0 fixed = 27 total (was 5) |
| +1 :green_heart: | mvnsite | 0m 28s | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | xml | 0m 1s | The patch has no ill-formed XML file. |
| +1 :green_heart: | shadedclient | 16m 28s | patch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 22s | the patch passed |
| -1 :x: | findbugs | 1m 1s | hadoop-tools/hadoop-azure generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) |
||| _ Other Tests _ |
| -1 :x: | unit | 1m 27s | hadoop-azure in the patch passed. |
| +1 :green_heart: | asflicense | 0m 32s | The patch does not generate ASF License warnings. |
| | | 68m 57s | |
| Reason | Tests |
|-------:|:------|
| FindBugs | module:hadoop-tools/hadoop-azure |
| | There is an apparent infinite recursive loop in org.apache.hadoop.fs.azurebfs.services.AbfsClient.createDefaultHeaders() At AbfsClient.java:recursive loop in org.apache.hadoop.fs.azurebfs.services.AbfsClient.createDefaultHeaders() At AbfsClient.java:[line 124] |
| Failed junit tests | hadoop.fs.azurebfs.services.TestAbfsOutputStream |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | Client=19.03.6 Server=19.03.6 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1790/9/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/1790 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient xml findbugs checkstyle markdownlint |
| uname | Linux 41026eb974e4 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | personality/hadoop.sh |
| git revision | trunk / ac4b556 |
| Default Java | 1.8.0_242 |
| checkstyle | https://builds.apache.org/job/hadoop-multibranch/job/PR-1790/9/artifact/out/diff-checkstyle-hadoop-tools_hadoop-azure.txt |
| findbugs | https://builds.apache.org/job/hadoop-multibranch/job/PR-1790/9/artifact/out/new-findbugs-hadoop-tools_hadoop-azure.html |
| unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-1790/9/artifact/out/patch-unit-hadoop-tools_hadoop-azure.txt |
| Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1790/9/testReport/ |
| Max. process+thread count | 454 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1790/9/console |
| versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] bilaharith commented on a change in pull request #1790:
[HADOOP-16818] ABFS: Combine append+flush calls for blockblob & appendblob
Posted by GitBox <gi...@apache.org>.
bilaharith commented on a change in pull request #1790: [HADOOP-16818] ABFS: Combine append+flush calls for blockblob & appendblob
URL: https://github.com/apache/hadoop/pull/1790#discussion_r393500489
##########
File path: hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/services/TestAbfsOutputStream.java
##########
@@ -0,0 +1,355 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.azurebfs.services;
+import java.util.Arrays;
+import java.util.HashSet;
+import java.util.Random;
+import org.junit.Assert;
+import org.junit.Test;
+import org.mockito.ArgumentCaptor;
+import org.apache.hadoop.fs.azurebfs.AbfsConfiguration;
+import org.apache.hadoop.conf.Configuration;
+
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.when;
+import static org.mockito.Mockito.times;
+import static org.mockito.Mockito.verify;
+import static org.mockito.Mockito.verifyNoMoreInteractions;
+import static org.mockito.Mockito.*;
+
+public final class TestAbfsOutputStream {
+
+ private static int bufferSize = 4096;
+ private static int writeSize = 1000;
+ private static String path = "~/testpath";
+ private final String globalKey = "fs.azure.configuration";
+ private final String accountName1 = "account1";
+ private final String accountKey1 = globalKey + "." + accountName1;
+ private final String accountValue1 = "one";
+
+ @Test
+ /**
+ * The test verifies OutputStream shortwrite case(2000bytes write followed by flush, hflush, hsync) is making correct HTTP calls to the server
Review comment:
Need to move @Test below the comments
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus removed a comment on issue #1790:
[HADOOP-16818] ABFS: Combine append+flush calls for blockblob & appendblob
Posted by GitBox <gi...@apache.org>.
hadoop-yetus removed a comment on issue #1790: [HADOOP-16818] ABFS: Combine append+flush calls for blockblob & appendblob
URL: https://github.com/apache/hadoop/pull/1790#issuecomment-583353780
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 1m 22s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +0 :ok: | markdownlint | 0m 1s | markdownlint was not available. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 1 new or modified test files. |
||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 23m 23s | trunk passed |
| +1 :green_heart: | compile | 0m 28s | trunk passed |
| +1 :green_heart: | checkstyle | 0m 22s | trunk passed |
| +1 :green_heart: | mvnsite | 0m 32s | trunk passed |
| +1 :green_heart: | shadedclient | 16m 35s | branch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 23s | trunk passed |
| +0 :ok: | spotbugs | 1m 1s | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 0m 59s | trunk passed |
||| _ Patch Compile Tests _ |
| -1 :x: | mvninstall | 0m 13s | hadoop-azure in the patch failed. |
| -1 :x: | compile | 0m 14s | hadoop-azure in the patch failed. |
| -1 :x: | javac | 0m 14s | hadoop-azure in the patch failed. |
| -0 :warning: | checkstyle | 0m 11s | The patch fails to run checkstyle in hadoop-azure |
| -1 :x: | mvnsite | 0m 16s | hadoop-azure in the patch failed. |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | xml | 0m 1s | The patch has no ill-formed XML file. |
| +1 :green_heart: | shadedclient | 15m 52s | patch has no errors when building and testing our client artifacts. |
| -1 :x: | javadoc | 0m 16s | hadoop-azure in the patch failed. |
| -1 :x: | findbugs | 0m 16s | hadoop-azure in the patch failed. |
||| _ Other Tests _ |
| -1 :x: | unit | 0m 15s | hadoop-azure in the patch failed. |
| +1 :green_heart: | asflicense | 0m 29s | The patch does not generate ASF License warnings. |
| | | 63m 36s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | Client=19.03.5 Server=19.03.5 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1790/4/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/1790 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient xml findbugs checkstyle markdownlint |
| uname | Linux 73f05f3de826 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19 08:06:28 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | personality/hadoop.sh |
| git revision | trunk / fafe78f |
| Default Java | 1.8.0_232 |
| mvninstall | https://builds.apache.org/job/hadoop-multibranch/job/PR-1790/4/artifact/out/patch-mvninstall-hadoop-tools_hadoop-azure.txt |
| compile | https://builds.apache.org/job/hadoop-multibranch/job/PR-1790/4/artifact/out/patch-compile-hadoop-tools_hadoop-azure.txt |
| javac | https://builds.apache.org/job/hadoop-multibranch/job/PR-1790/4/artifact/out/patch-compile-hadoop-tools_hadoop-azure.txt |
| checkstyle | https://builds.apache.org/job/hadoop-multibranch/job/PR-1790/4/artifact/out//home/jenkins/jenkins-slave/workspace/hadoop-multibranch_PR-1790/out/maven-patch-checkstyle-hadoop-tools_hadoop-azure.txt |
| mvnsite | https://builds.apache.org/job/hadoop-multibranch/job/PR-1790/4/artifact/out/patch-mvnsite-hadoop-tools_hadoop-azure.txt |
| javadoc | https://builds.apache.org/job/hadoop-multibranch/job/PR-1790/4/artifact/out/patch-javadoc-hadoop-tools_hadoop-azure.txt |
| findbugs | https://builds.apache.org/job/hadoop-multibranch/job/PR-1790/4/artifact/out/patch-findbugs-hadoop-tools_hadoop-azure.txt |
| unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-1790/4/artifact/out/patch-unit-hadoop-tools_hadoop-azure.txt |
| Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1790/4/testReport/ |
| Max. process+thread count | 311 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1790/4/console |
| versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] bilaharith commented on a change in pull request #1790:
[HADOOP-16818] ABFS: Combine append+flush calls for blockblob & appendblob
Posted by GitBox <gi...@apache.org>.
bilaharith commented on a change in pull request #1790: [HADOOP-16818] ABFS: Combine append+flush calls for blockblob & appendblob
URL: https://github.com/apache/hadoop/pull/1790#discussion_r392654692
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/constants/ConfigurationKeys.java
##########
@@ -60,6 +61,15 @@
* documentation does not have such expectations of data being persisted.
* Default value of this config is true. **/
public static final String FS_AZURE_DISABLE_OUTPUTSTREAM_FLUSH = "fs.azure.disable.outputstream.flush";
+ /** Provides a config control to enable OutputStream AppendWithFlush API
+ * operations in AbfsOutputStream.
+ * Default value of this config is true. **/
+ public static final String FS_AZURE_ENABLE_APPEND_WITH_FLUSH = "fs.azure.enable.appendwithflush";
+ /** Provides a config control to disable or enable OutputStream Flush API
Review comment:
This comment looks misplaced
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] ishaniahuja commented on a change in pull request #1790:
[HADOOP-16818] ABFS: Combine append+flush calls for blockblob & appendblob
Posted by GitBox <gi...@apache.org>.
ishaniahuja commented on a change in pull request #1790: [HADOOP-16818] ABFS: Combine append+flush calls for blockblob & appendblob
URL: https://github.com/apache/hadoop/pull/1790#discussion_r371720954
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AzureBlobFileSystemStore.java
##########
@@ -164,6 +169,23 @@ public AzureBlobFileSystemStore(URI uri, boolean isSecureScheme, Configuration c
boolean useHttps = (usingOauth || abfsConfiguration.isHttpsAlwaysUsed()) ? true : isSecureScheme;
initializeClient(uri, fileSystemName, accountName, useHttps);
this.identityTransformer = new IdentityTransformer(abfsConfiguration.getRawConfiguration());
+
+ // Extract the directories that should contain page blobs
Review comment:
I don't think that should be done as a part of this PR.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus removed a comment on issue #1790:
[HADOOP-16818] ABFS: Combine append+flush calls for blockblob & appendblob
Posted by GitBox <gi...@apache.org>.
hadoop-yetus removed a comment on issue #1790: [HADOOP-16818] ABFS: Combine append+flush calls for blockblob & appendblob
URL: https://github.com/apache/hadoop/pull/1790#issuecomment-587440341
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 30m 22s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +0 :ok: | markdownlint | 0m 1s | markdownlint was not available. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 1 new or modified test files. |
||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 22m 21s | trunk passed |
| +1 :green_heart: | compile | 0m 28s | trunk passed |
| +1 :green_heart: | checkstyle | 0m 24s | trunk passed |
| +1 :green_heart: | mvnsite | 0m 31s | trunk passed |
| +1 :green_heart: | shadedclient | 16m 22s | branch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 23s | trunk passed |
| +0 :ok: | spotbugs | 0m 53s | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 0m 51s | trunk passed |
||| _ Patch Compile Tests _ |
| -1 :x: | mvninstall | 0m 23s | hadoop-azure in the patch failed. |
| -1 :x: | compile | 0m 22s | hadoop-azure in the patch failed. |
| -1 :x: | javac | 0m 22s | hadoop-azure in the patch failed. |
| -0 :warning: | checkstyle | 0m 15s | hadoop-tools/hadoop-azure: The patch generated 22 new + 5 unchanged - 0 fixed = 27 total (was 5) |
| -1 :x: | mvnsite | 0m 23s | hadoop-azure in the patch failed. |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | xml | 0m 1s | The patch has no ill-formed XML file. |
| +1 :green_heart: | shadedclient | 15m 52s | patch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 21s | the patch passed |
| -1 :x: | findbugs | 0m 27s | hadoop-azure in the patch failed. |
||| _ Other Tests _ |
| -1 :x: | unit | 0m 28s | hadoop-azure in the patch failed. |
| +1 :green_heart: | asflicense | 0m 29s | The patch does not generate ASF License warnings. |
| | | 92m 10s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | Client=19.03.6 Server=19.03.6 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1790/8/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/1790 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient xml findbugs checkstyle markdownlint |
| uname | Linux 9c898717ba62 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19 08:06:28 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | personality/hadoop.sh |
| git revision | trunk / a562942 |
| Default Java | 1.8.0_242 |
| mvninstall | https://builds.apache.org/job/hadoop-multibranch/job/PR-1790/8/artifact/out/patch-mvninstall-hadoop-tools_hadoop-azure.txt |
| compile | https://builds.apache.org/job/hadoop-multibranch/job/PR-1790/8/artifact/out/patch-compile-hadoop-tools_hadoop-azure.txt |
| javac | https://builds.apache.org/job/hadoop-multibranch/job/PR-1790/8/artifact/out/patch-compile-hadoop-tools_hadoop-azure.txt |
| checkstyle | https://builds.apache.org/job/hadoop-multibranch/job/PR-1790/8/artifact/out/diff-checkstyle-hadoop-tools_hadoop-azure.txt |
| mvnsite | https://builds.apache.org/job/hadoop-multibranch/job/PR-1790/8/artifact/out/patch-mvnsite-hadoop-tools_hadoop-azure.txt |
| findbugs | https://builds.apache.org/job/hadoop-multibranch/job/PR-1790/8/artifact/out/patch-findbugs-hadoop-tools_hadoop-azure.txt |
| unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-1790/8/artifact/out/patch-unit-hadoop-tools_hadoop-azure.txt |
| Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1790/8/testReport/ |
| Max. process+thread count | 307 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1790/8/console |
| versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran merged pull request #1790: [HADOOP-16818]
ABFS: Combine append+flush calls for blockblob & appendblob
Posted by GitBox <gi...@apache.org>.
steveloughran merged pull request #1790: [HADOOP-16818] ABFS: Combine append+flush calls for blockblob & appendblob
URL: https://github.com/apache/hadoop/pull/1790
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] ishaniahuja closed pull request #1790: changes for combined append+flush calls for blockblob(traditionan) an…
Posted by GitBox <gi...@apache.org>.
ishaniahuja closed pull request #1790: changes for combined append+flush calls for blockblob(traditionan) an…
URL: https://github.com/apache/hadoop/pull/1790
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] ishaniahuja commented on a change in pull request #1790:
[HADOOP-16818] ABFS: Combine append+flush calls for blockblob & appendblob
Posted by GitBox <gi...@apache.org>.
ishaniahuja commented on a change in pull request #1790: [HADOOP-16818] ABFS: Combine append+flush calls for blockblob & appendblob
URL: https://github.com/apache/hadoop/pull/1790#discussion_r379260603
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsOutputStream.java
##########
@@ -307,10 +318,18 @@ public Void call() throws Exception {
perfInfo.registerSuccess(true);
return null;
}
+ if (flush) {
+ while(lastTotalAppendOffset < lastFlushOffset);
Review comment:
I have added comment in the code. This code ensures that the append+flush (new API is called only once all the append and flush have finished and there are no pending appends in the driver.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on a change in pull request #1790:
[HADOOP-16818] ABFS: Combine append+flush calls for blockblob & appendblob
Posted by GitBox <gi...@apache.org>.
steveloughran commented on a change in pull request #1790: [HADOOP-16818] ABFS: Combine append+flush calls for blockblob & appendblob
URL: https://github.com/apache/hadoop/pull/1790#discussion_r386488058
##########
File path: hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/services/TestAbfsOutputStream.java
##########
@@ -0,0 +1,349 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.azurebfs.services;
+import org.apache.hadoop.fs.azurebfs.AbfsConfiguration;
+import org.apache.hadoop.conf.Configuration;
+import java.util.Arrays;
+import java.util.HashSet;
+import java.util.Random;
+import org.junit.Assert;
+import org.junit.Test;
+
+import org.mockito.ArgumentCaptor;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.when;
+import static org.mockito.Mockito.times;
+import static org.mockito.Mockito.verify;
+import static org.mockito.Mockito.verifyNoMoreInteractions;
+import static org.mockito.Mockito.*;
+/**
+ * Test useragent of abfs client.
+ *
+ */
+public final class TestAbfsOutputStream {
Review comment:
Mockito tests are always a maintenance pain because they are so brittle and sho hard to understand what is going on -for example, here I couldn't really understand any of the tests. Could you add some more detail as a comment for each test -at least to level of what each test case is looking for.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus removed a comment on issue #1790:
[HADOOP-16818] ABFS: Combine append+flush calls for blockblob & appendblob
Posted by GitBox <gi...@apache.org>.
hadoop-yetus removed a comment on issue #1790: [HADOOP-16818] ABFS: Combine append+flush calls for blockblob & appendblob
URL: https://github.com/apache/hadoop/pull/1790#issuecomment-578722809
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 27m 48s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 1 new or modified test files. |
||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 18m 36s | trunk passed |
| +1 :green_heart: | compile | 0m 31s | trunk passed |
| +1 :green_heart: | checkstyle | 0m 24s | trunk passed |
| +1 :green_heart: | mvnsite | 0m 35s | trunk passed |
| +1 :green_heart: | shadedclient | 13m 32s | branch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 26s | trunk passed |
| +0 :ok: | spotbugs | 0m 50s | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 0m 49s | trunk passed |
||| _ Patch Compile Tests _ |
| -1 :x: | mvninstall | 0m 18s | hadoop-azure in the patch failed. |
| -1 :x: | compile | 0m 15s | hadoop-azure in the patch failed. |
| -1 :x: | javac | 0m 15s | hadoop-azure in the patch failed. |
| -0 :warning: | checkstyle | 0m 13s | The patch fails to run checkstyle in hadoop-azure |
| -1 :x: | mvnsite | 0m 17s | hadoop-azure in the patch failed. |
| -1 :x: | whitespace | 0m 0s | The patch 6 line(s) with tabs. |
| +1 :green_heart: | xml | 0m 1s | The patch has no ill-formed XML file. |
| +1 :green_heart: | shadedclient | 13m 32s | patch has no errors when building and testing our client artifacts. |
| -1 :x: | javadoc | 0m 20s | hadoop-azure in the patch failed. |
| -1 :x: | findbugs | 0m 19s | hadoop-azure in the patch failed. |
||| _ Other Tests _ |
| -1 :x: | unit | 0m 19s | hadoop-azure in the patch failed. |
| +1 :green_heart: | asflicense | 0m 31s | The patch does not generate ASF License warnings. |
| | | 80m 15s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | Client=19.03.5 Server=19.03.5 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1790/2/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/1790 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient xml findbugs checkstyle |
| uname | Linux 8c0ffd23132b 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | personality/hadoop.sh |
| git revision | trunk / 7f40e66 |
| Default Java | 1.8.0_232 |
| mvninstall | https://builds.apache.org/job/hadoop-multibranch/job/PR-1790/2/artifact/out/patch-mvninstall-hadoop-tools_hadoop-azure.txt |
| compile | https://builds.apache.org/job/hadoop-multibranch/job/PR-1790/2/artifact/out/patch-compile-hadoop-tools_hadoop-azure.txt |
| javac | https://builds.apache.org/job/hadoop-multibranch/job/PR-1790/2/artifact/out/patch-compile-hadoop-tools_hadoop-azure.txt |
| checkstyle | https://builds.apache.org/job/hadoop-multibranch/job/PR-1790/2/artifact/out//home/jenkins/jenkins-slave/workspace/hadoop-multibranch_PR-1790/out/maven-patch-checkstyle-hadoop-tools_hadoop-azure.txt |
| mvnsite | https://builds.apache.org/job/hadoop-multibranch/job/PR-1790/2/artifact/out/patch-mvnsite-hadoop-tools_hadoop-azure.txt |
| whitespace | https://builds.apache.org/job/hadoop-multibranch/job/PR-1790/2/artifact/out/whitespace-tabs.txt |
| javadoc | https://builds.apache.org/job/hadoop-multibranch/job/PR-1790/2/artifact/out/patch-javadoc-hadoop-tools_hadoop-azure.txt |
| findbugs | https://builds.apache.org/job/hadoop-multibranch/job/PR-1790/2/artifact/out/patch-findbugs-hadoop-tools_hadoop-azure.txt |
| unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-1790/2/artifact/out/patch-unit-hadoop-tools_hadoop-azure.txt |
| Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1790/2/testReport/ |
| Max. process+thread count | 418 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1790/2/console |
| versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on issue #1790: [HADOOP-16818]
ABFS: Combine append+flush calls for blockblob & appendblob
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on issue #1790: [HADOOP-16818] ABFS: Combine append+flush calls for blockblob & appendblob
URL: https://github.com/apache/hadoop/pull/1790#issuecomment-594598989
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 0m 38s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | No case conflicting files found. |
| +0 :ok: | markdownlint | 0m 0s | markdownlint was not available. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 2 new or modified test files. |
||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 19m 39s | trunk passed |
| +1 :green_heart: | compile | 0m 30s | trunk passed |
| +1 :green_heart: | checkstyle | 0m 22s | trunk passed |
| +1 :green_heart: | mvnsite | 0m 34s | trunk passed |
| +1 :green_heart: | shadedclient | 15m 7s | branch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 26s | trunk passed |
| +0 :ok: | spotbugs | 0m 52s | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 0m 49s | trunk passed |
||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 29s | the patch passed |
| +1 :green_heart: | compile | 0m 23s | the patch passed |
| +1 :green_heart: | javac | 0m 23s | the patch passed |
| -0 :warning: | checkstyle | 0m 17s | hadoop-tools/hadoop-azure: The patch generated 23 new + 5 unchanged - 0 fixed = 28 total (was 5) |
| +1 :green_heart: | mvnsite | 0m 27s | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | xml | 0m 2s | The patch has no ill-formed XML file. |
| +1 :green_heart: | shadedclient | 13m 56s | patch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 22s | the patch passed |
| +1 :green_heart: | findbugs | 0m 53s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 1m 23s | hadoop-azure in the patch passed. |
| +1 :green_heart: | asflicense | 0m 31s | The patch does not generate ASF License warnings. |
| | | 58m 27s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | Client=19.03.7 Server=19.03.7 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1790/17/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/1790 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient xml findbugs checkstyle markdownlint |
| uname | Linux 48fe9120a0cb 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | personality/hadoop.sh |
| git revision | trunk / bbd704b |
| Default Java | 1.8.0_242 |
| checkstyle | https://builds.apache.org/job/hadoop-multibranch/job/PR-1790/17/artifact/out/diff-checkstyle-hadoop-tools_hadoop-azure.txt |
| Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1790/17/testReport/ |
| Max. process+thread count | 423 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1790/17/console |
| versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on issue #1790: [HADOOP-16818]
ABFS: Combine append+flush calls for blockblob & appendblob
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on issue #1790: [HADOOP-16818] ABFS: Combine append+flush calls for blockblob & appendblob
URL: https://github.com/apache/hadoop/pull/1790#issuecomment-585102571
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 1m 33s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +0 :ok: | markdownlint | 0m 1s | markdownlint was not available. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 1 new or modified test files. |
||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 22m 40s | trunk passed |
| +1 :green_heart: | compile | 0m 40s | trunk passed |
| +1 :green_heart: | checkstyle | 0m 24s | trunk passed |
| +1 :green_heart: | mvnsite | 0m 35s | trunk passed |
| +1 :green_heart: | shadedclient | 16m 42s | branch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 23s | trunk passed |
| +0 :ok: | spotbugs | 0m 51s | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 0m 49s | trunk passed |
||| _ Patch Compile Tests _ |
| -1 :x: | mvninstall | 0m 13s | hadoop-azure in the patch failed. |
| -1 :x: | compile | 0m 13s | hadoop-azure in the patch failed. |
| -1 :x: | javac | 0m 13s | hadoop-azure in the patch failed. |
| -0 :warning: | checkstyle | 0m 12s | The patch fails to run checkstyle in hadoop-azure |
| -1 :x: | mvnsite | 0m 14s | hadoop-azure in the patch failed. |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | xml | 0m 1s | The patch has no ill-formed XML file. |
| +1 :green_heart: | shadedclient | 15m 27s | patch has no errors when building and testing our client artifacts. |
| -1 :x: | javadoc | 0m 16s | hadoop-azure in the patch failed. |
| -1 :x: | findbugs | 0m 14s | hadoop-azure in the patch failed. |
||| _ Other Tests _ |
| -1 :x: | unit | 0m 15s | hadoop-azure in the patch failed. |
| +1 :green_heart: | asflicense | 0m 29s | The patch does not generate ASF License warnings. |
| | | 62m 38s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | Client=19.03.5 Server=19.03.5 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1790/6/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/1790 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient xml findbugs checkstyle markdownlint |
| uname | Linux b5e26bda998d 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19 08:06:28 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | personality/hadoop.sh |
| git revision | trunk / 9709afe |
| Default Java | 1.8.0_242 |
| mvninstall | https://builds.apache.org/job/hadoop-multibranch/job/PR-1790/6/artifact/out/patch-mvninstall-hadoop-tools_hadoop-azure.txt |
| compile | https://builds.apache.org/job/hadoop-multibranch/job/PR-1790/6/artifact/out/patch-compile-hadoop-tools_hadoop-azure.txt |
| javac | https://builds.apache.org/job/hadoop-multibranch/job/PR-1790/6/artifact/out/patch-compile-hadoop-tools_hadoop-azure.txt |
| checkstyle | https://builds.apache.org/job/hadoop-multibranch/job/PR-1790/6/artifact/out//home/jenkins/jenkins-slave/workspace/hadoop-multibranch_PR-1790/out/maven-patch-checkstyle-hadoop-tools_hadoop-azure.txt |
| mvnsite | https://builds.apache.org/job/hadoop-multibranch/job/PR-1790/6/artifact/out/patch-mvnsite-hadoop-tools_hadoop-azure.txt |
| javadoc | https://builds.apache.org/job/hadoop-multibranch/job/PR-1790/6/artifact/out/patch-javadoc-hadoop-tools_hadoop-azure.txt |
| findbugs | https://builds.apache.org/job/hadoop-multibranch/job/PR-1790/6/artifact/out/patch-findbugs-hadoop-tools_hadoop-azure.txt |
| unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-1790/6/artifact/out/patch-unit-hadoop-tools_hadoop-azure.txt |
| Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1790/6/testReport/ |
| Max. process+thread count | 334 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1790/6/console |
| versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] ishaniahuja commented on a change in pull request #1790:
[HADOOP-16818] ABFS: Combine append+flush calls for blockblob & appendblob
Posted by GitBox <gi...@apache.org>.
ishaniahuja commented on a change in pull request #1790: [HADOOP-16818] ABFS: Combine append+flush calls for blockblob & appendblob
URL: https://github.com/apache/hadoop/pull/1790#discussion_r389527177
##########
File path: hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/ITestAzureBlobFileSystemE2E.java
##########
@@ -209,9 +209,10 @@ public void testFlushWithFileNotFoundException() throws Exception {
fs.delete(testFilePath, true);
assertFalse(fs.exists(testFilePath));
+ AbfsConfiguration configuration = this.getConfiguration();
- intercept(FileNotFoundException.class,
- () -> stream.close());
+ //With the new code, it would not trigger a call to the backend
Review comment:
I updated the comment with more details. You can also look at the code to understand the behavior. AbfsOutputStream close(). Yes I have tested the different configurations (appendblob, blockblob, append+flush enabled, append+flush disabled, with and without namespace disabled account.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus removed a comment on issue #1790:
[HADOOP-16818] ABFS: Combine append+flush calls for blockblob & appendblob
Posted by GitBox <gi...@apache.org>.
hadoop-yetus removed a comment on issue #1790: [HADOOP-16818] ABFS: Combine append+flush calls for blockblob & appendblob
URL: https://github.com/apache/hadoop/pull/1790#issuecomment-590257579
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 0m 38s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | No case conflicting files found. |
| +0 :ok: | markdownlint | 0m 0s | markdownlint was not available. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 1 new or modified test files. |
||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 19m 54s | trunk passed |
| +1 :green_heart: | compile | 0m 29s | trunk passed |
| +1 :green_heart: | checkstyle | 0m 21s | trunk passed |
| +1 :green_heart: | mvnsite | 0m 35s | trunk passed |
| +1 :green_heart: | shadedclient | 15m 21s | branch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 22s | trunk passed |
| +0 :ok: | spotbugs | 0m 53s | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 0m 52s | trunk passed |
||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 31s | the patch passed |
| +1 :green_heart: | compile | 0m 23s | the patch passed |
| +1 :green_heart: | javac | 0m 23s | the patch passed |
| -0 :warning: | checkstyle | 0m 16s | hadoop-tools/hadoop-azure: The patch generated 23 new + 5 unchanged - 0 fixed = 28 total (was 5) |
| +1 :green_heart: | mvnsite | 0m 28s | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | xml | 0m 1s | The patch has no ill-formed XML file. |
| +1 :green_heart: | shadedclient | 14m 1s | patch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 21s | the patch passed |
| -1 :x: | findbugs | 0m 58s | hadoop-tools/hadoop-azure generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) |
||| _ Other Tests _ |
| -1 :x: | unit | 1m 22s | hadoop-azure in the patch passed. |
| +1 :green_heart: | asflicense | 0m 28s | The patch does not generate ASF License warnings. |
| | | 58m 54s | |
| Reason | Tests |
|-------:|:------|
| FindBugs | module:hadoop-tools/hadoop-azure |
| | There is an apparent infinite recursive loop in org.apache.hadoop.fs.azurebfs.services.AbfsClient.createDefaultHeaders() At AbfsClient.java:recursive loop in org.apache.hadoop.fs.azurebfs.services.AbfsClient.createDefaultHeaders() At AbfsClient.java:[line 124] |
| Failed junit tests | hadoop.fs.azurebfs.services.TestAbfsOutputStream |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | Client=19.03.6 Server=19.03.6 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1790/10/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/1790 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient xml findbugs checkstyle markdownlint |
| uname | Linux cc0baa985310 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | personality/hadoop.sh |
| git revision | trunk / b5698e0 |
| Default Java | 1.8.0_242 |
| checkstyle | https://builds.apache.org/job/hadoop-multibranch/job/PR-1790/10/artifact/out/diff-checkstyle-hadoop-tools_hadoop-azure.txt |
| findbugs | https://builds.apache.org/job/hadoop-multibranch/job/PR-1790/10/artifact/out/new-findbugs-hadoop-tools_hadoop-azure.html |
| unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-1790/10/artifact/out/patch-unit-hadoop-tools_hadoop-azure.txt |
| Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1790/10/testReport/ |
| Max. process+thread count | 414 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1790/10/console |
| versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] snvijaya commented on a change in pull request #1790: changes for combined append+flush calls for blockblob(traditionan) an…
Posted by GitBox <gi...@apache.org>.
snvijaya commented on a change in pull request #1790: changes for combined append+flush calls for blockblob(traditionan) an…
URL: https://github.com/apache/hadoop/pull/1790#discussion_r363154151
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsClient.java
##########
@@ -278,7 +282,8 @@ public AbfsRestOperation renamePath(final String source, final String destinatio
}
public AbfsRestOperation append(final String path, final long position, final byte[] buffer, final int offset,
- final int length) throws AzureBlobFileSystemException {
+ final int length, boolean flush, boolean isClose)
Review comment:
This method is supposed to be called from AbfsOutputStream.java -> writeCurrentBufferToService(), but there is no change for call to this API within that method. How did the compilation pass.
And, (probably a repeat question of config query posted above, still) can you please why add flush and close needs to be added to Append method.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] ishaniahuja commented on a change in pull request #1790:
[HADOOP-16818] ABFS: Combine append+flush calls for blockblob & appendblob
Posted by GitBox <gi...@apache.org>.
ishaniahuja commented on a change in pull request #1790: [HADOOP-16818] ABFS: Combine append+flush calls for blockblob & appendblob
URL: https://github.com/apache/hadoop/pull/1790#discussion_r371721850
##########
File path: hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/services/TestAbfsOutputStream.java
##########
@@ -0,0 +1,268 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.azurebfs.services;
+import java.util.Arrays;
+import java.util.HashSet;
+import java.util.Random;
+import org.junit.Assert;
+import org.junit.Test;
+
+import org.mockito.ArgumentCaptor;
+import static org.mockito.Mockito.*;
+/**
+ * Test useragent of abfs client.
+ *
+ */
+public final class TestAbfsOutputStream {
+
+ private static int bufferSize = 4096;
+ private static int writeSize = 1000;
+ private static String path = "~/testpath";
+
+ @Test
+ public void verifyShortWriteRequest() throws Exception {
+
+ AbfsClient client = mock(AbfsClient.class);
+ AbfsOutputStream out = new AbfsOutputStream(client, path, 0, bufferSize, true, true, false);
+ final byte[] b = new byte[writeSize];
+ new Random().nextBytes(b);
+ out.write(b);
+ out.hsync();
+ ArgumentCaptor<String> acString = ArgumentCaptor.forClass(String.class);
+ ArgumentCaptor<Long> acLong = ArgumentCaptor.forClass(Long.class);
+ ArgumentCaptor<Integer> acInt = ArgumentCaptor.forClass(Integer.class);
+ ArgumentCaptor<Boolean> acBool = ArgumentCaptor.forClass(Boolean.class);
+ ArgumentCaptor<byte[]> acByteArray = ArgumentCaptor.forClass(byte[].class);
+
+ final byte[] b1 = new byte[2*writeSize];
+ new Random().nextBytes(b1);
+ out.write(b1);
+ out.flush();
+ out.hflush();
+
+ out.hsync();
+
+ verify(client, times(2)).append(acString.capture(), acLong.capture(), acByteArray.capture(), acInt.capture(), acInt.capture(), acBool.capture(), acBool.capture());
+ Assert.assertEquals(Arrays.asList(path, path) , acString.getAllValues());
+ Assert.assertEquals(Arrays.asList(Long.valueOf(0), Long.valueOf(writeSize)), acLong.getAllValues());
+ //flush=true, close=false, flush=true, close=false
+ Assert.assertEquals(Arrays.asList(true, false, true, false), acBool.getAllValues());
+ Assert.assertEquals(Arrays.asList(0,writeSize, 0, 2*writeSize), acInt.getAllValues());
+
+ verifyNoMoreInteractions(client);
+
+ }
+
+ @Test
+ public void verifyWriteRequest() throws Exception {
+
+ AbfsClient client = mock(AbfsClient.class);
+ AbfsOutputStream out = new AbfsOutputStream(client, path, 0, bufferSize, true, true, false);
+ final byte[] b = new byte[writeSize];
+ new Random().nextBytes(b);
+
+ for (int i = 0; i < 5; i++) {
+ out.write(b);
+ }
+ out.close();
+
+ ArgumentCaptor<String> acString = ArgumentCaptor.forClass(String.class);
+ ArgumentCaptor<Long> acLong = ArgumentCaptor.forClass(Long.class);
+ ArgumentCaptor<Integer> acInt = ArgumentCaptor.forClass(Integer.class);
+ ArgumentCaptor<Boolean> acBool = ArgumentCaptor.forClass(Boolean.class);
+ ArgumentCaptor<byte[]> acByteArray = ArgumentCaptor.forClass(byte[].class);
+
+ verify(client, times(2)).append(acString.capture(), acLong.capture(), acByteArray.capture(), acInt.capture(), acInt.capture(), acBool.capture(), acBool.capture());
+ Assert.assertEquals(Arrays.asList(path, path) , acString.getAllValues());
+ Assert.assertEquals(Arrays.asList(Long.valueOf(0), Long.valueOf(bufferSize)), acLong.getAllValues());
+ //flush=false,close=false, flush=true,close=true
+ Assert.assertEquals(Arrays.asList(false, false, true, true), acBool.getAllValues());
+ Assert.assertEquals(Arrays.asList(0, bufferSize, 0, 5*writeSize-bufferSize), acInt.getAllValues());
+
+ verifyNoMoreInteractions(client);
+ }
+
+ @Test
+ public void verifyWriteRequestOfBufferSizeAndClose() throws Exception {
+
+ AbfsClient client = mock(AbfsClient.class);
+ AbfsOutputStream out = new AbfsOutputStream(client, path, 0, bufferSize, true, true, false);
+ final byte[] b = new byte[bufferSize];
+ new Random().nextBytes(b);
+
+ for (int i = 0; i < 2; i++) {
+ out.write(b);
+ }
+ out.close();
+
+ ArgumentCaptor<String> acString = ArgumentCaptor.forClass(String.class);
+ ArgumentCaptor<Long> acLong = ArgumentCaptor.forClass(Long.class);
+ ArgumentCaptor<Integer> acInt = ArgumentCaptor.forClass(Integer.class);
+ ArgumentCaptor<Boolean> acBool = ArgumentCaptor.forClass(Boolean.class);
+ ArgumentCaptor<byte[]> acByteArray = ArgumentCaptor.forClass(byte[].class);
+
+ verify(client, times(2)).append(acString.capture(), acLong.capture(), acByteArray.capture(), acInt.capture(), acInt.capture(), acBool.capture(), acBool.capture());
+ Assert.assertEquals(Arrays.asList(path, path) , acString.getAllValues());
+ Assert.assertEquals(new HashSet<Long>(Arrays.asList(Long.valueOf(0), Long.valueOf(bufferSize))), new HashSet<Long>(acLong.getAllValues()));
+ //flush=false, close=false, flush=false, close=false
+ Assert.assertEquals(Arrays.asList(false, false, false, false), acBool.getAllValues());
+ Assert.assertEquals(Arrays.asList(0, bufferSize, 0, bufferSize), acInt.getAllValues());
+
+ ArgumentCaptor<String> acFlushString = ArgumentCaptor.forClass(String.class);
+ ArgumentCaptor<Long> acFlushLong = ArgumentCaptor.forClass(Long.class);
+ ArgumentCaptor<Boolean> acFlushBool = ArgumentCaptor.forClass(Boolean.class);
+
+ verify(client, times(1)).flush(acFlushString.capture(), acFlushLong.capture(), acFlushBool.capture(), acFlushBool.capture());
+ Assert.assertEquals(Arrays.asList(path) , acFlushString.getAllValues());
+ Assert.assertEquals(Arrays.asList(Long.valueOf(2*bufferSize)), acFlushLong.getAllValues());
+ Assert.assertEquals(Arrays.asList(false, true), acFlushBool.getAllValues());
+
+ verifyNoMoreInteractions(client);
+ }
+
+ @Test
+ public void verifyWriteRequestOfBufferSize() throws Exception {
+
+ AbfsClient client = mock(AbfsClient.class);
+ AbfsOutputStream out = new AbfsOutputStream(client, path, 0, bufferSize, true, true, false);
+ final byte[] b = new byte[bufferSize];
+ new Random().nextBytes(b);
+
+ for (int i = 0; i < 2; i++) {
+ out.write(b);
+ }
+ Thread.sleep(1000);
+
+ ArgumentCaptor<String> acString = ArgumentCaptor.forClass(String.class);
+ ArgumentCaptor<Long> acLong = ArgumentCaptor.forClass(Long.class);
+ ArgumentCaptor<Integer> acInt = ArgumentCaptor.forClass(Integer.class);
+ ArgumentCaptor<Boolean> acBool = ArgumentCaptor.forClass(Boolean.class);
+ ArgumentCaptor<byte[]> acByteArray = ArgumentCaptor.forClass(byte[].class);
+
+ verify(client, times(2)).append(acString.capture(), acLong.capture(), acByteArray.capture(), acInt.capture(), acInt.capture(), acBool.capture(), acBool.capture());
+ Assert.assertEquals(Arrays.asList(path, path) , acString.getAllValues());
+ Assert.assertEquals(new HashSet<Long>(Arrays.asList(Long.valueOf(0), Long.valueOf(bufferSize))), new HashSet<Long>(acLong.getAllValues()));
+ //flush=false, close=false, flush=false, close=false
+ Assert.assertEquals(Arrays.asList(false, false, false, false), acBool.getAllValues());
+ Assert.assertEquals(Arrays.asList(0, bufferSize, 0, bufferSize), acInt.getAllValues());
+
+ verifyNoMoreInteractions(client);
+ }
+
+ @Test
+ public void verifyWriteRequestOfBufferSizeWithAppendBlob() throws Exception {
+
+ AbfsClient client = mock(AbfsClient.class);
+ AbfsOutputStream out = new AbfsOutputStream(client, path, 0, bufferSize, true, true, true);
+ final byte[] b = new byte[bufferSize];
+ new Random().nextBytes(b);
+
+ for (int i = 0; i < 2; i++) {
+ out.write(b);
+ }
+ Thread.sleep(1000);
+
+ ArgumentCaptor<String> acString = ArgumentCaptor.forClass(String.class);
+ ArgumentCaptor<Long> acLong = ArgumentCaptor.forClass(Long.class);
+ ArgumentCaptor<Integer> acInt = ArgumentCaptor.forClass(Integer.class);
+ ArgumentCaptor<Boolean> acBool = ArgumentCaptor.forClass(Boolean.class);
+ ArgumentCaptor<byte[]> acByteArray = ArgumentCaptor.forClass(byte[].class);
Review comment:
these are just initializations.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on issue #1790: [HADOOP-16818]
ABFS: Combine append+flush calls for blockblob & appendblob
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on issue #1790: [HADOOP-16818] ABFS: Combine append+flush calls for blockblob & appendblob
URL: https://github.com/apache/hadoop/pull/1790#issuecomment-596461050
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 1m 36s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +0 :ok: | markdownlint | 0m 0s | markdownlint was not available. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 2 new or modified test files. |
||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 22m 52s | trunk passed |
| +1 :green_heart: | compile | 0m 30s | trunk passed |
| +1 :green_heart: | checkstyle | 0m 22s | trunk passed |
| +1 :green_heart: | mvnsite | 0m 33s | trunk passed |
| +1 :green_heart: | shadedclient | 16m 30s | branch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 24s | trunk passed |
| +0 :ok: | spotbugs | 0m 54s | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 0m 50s | trunk passed |
||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 29s | the patch passed |
| +1 :green_heart: | compile | 0m 23s | the patch passed |
| +1 :green_heart: | javac | 0m 23s | the patch passed |
| -0 :warning: | checkstyle | 0m 15s | hadoop-tools/hadoop-azure: The patch generated 23 new + 5 unchanged - 0 fixed = 28 total (was 5) |
| +1 :green_heart: | mvnsite | 0m 27s | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | xml | 0m 2s | The patch has no ill-formed XML file. |
| +1 :green_heart: | shadedclient | 15m 22s | patch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 21s | the patch passed |
| +1 :green_heart: | findbugs | 0m 55s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 1m 25s | hadoop-azure in the patch passed. |
| +1 :green_heart: | asflicense | 0m 29s | The patch does not generate ASF License warnings. |
| | | 65m 14s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | Client=19.03.7 Server=19.03.7 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1790/19/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/1790 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient xml findbugs checkstyle markdownlint |
| uname | Linux 98c431ab038a 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19 08:06:28 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | personality/hadoop.sh |
| git revision | trunk / 6dfe00c |
| Default Java | 1.8.0_242 |
| checkstyle | https://builds.apache.org/job/hadoop-multibranch/job/PR-1790/19/artifact/out/diff-checkstyle-hadoop-tools_hadoop-azure.txt |
| Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1790/19/testReport/ |
| Max. process+thread count | 307 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1790/19/console |
| versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on a change in pull request #1790:
[HADOOP-16818] ABFS: Combine append+flush calls for blockblob & appendblob
Posted by GitBox <gi...@apache.org>.
steveloughran commented on a change in pull request #1790: [HADOOP-16818] ABFS: Combine append+flush calls for blockblob & appendblob
URL: https://github.com/apache/hadoop/pull/1790#discussion_r393302657
##########
File path: hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/services/TestAbfsOutputStream.java
##########
@@ -0,0 +1,355 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.azurebfs.services;
+import java.util.Arrays;
+import java.util.HashSet;
+import java.util.Random;
+import org.junit.Assert;
+import org.junit.Test;
+import org.mockito.ArgumentCaptor;
+import org.apache.hadoop.fs.azurebfs.AbfsConfiguration;
+import org.apache.hadoop.conf.Configuration;
+
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.when;
+import static org.mockito.Mockito.times;
+import static org.mockito.Mockito.verify;
+import static org.mockito.Mockito.verifyNoMoreInteractions;
+import static org.mockito.Mockito.*;
+
+public final class TestAbfsOutputStream {
+
+ private static int bufferSize = 4096;
+ private static int writeSize = 1000;
+ private static String path = "~/testpath";
+ private final String globalKey = "fs.azure.configuration";
+ private final String accountName1 = "account1";
+ private final String accountKey1 = globalKey + "." + accountName1;
+ private final String accountValue1 = "one";
+
+ @Test
+ /**
+ * The test verifies OutputStream shortwrite case(2000bytes write followed by flush, hflush, hsync) is making correct HTTP calls to the server
+ */
+ public void verifyShortWriteRequest() throws Exception {
+
+ AbfsClient client = mock(AbfsClient.class);
+ AbfsRestOperation op = mock(AbfsRestOperation.class);
+ AbfsConfiguration abfsConf;
+ final Configuration conf = new Configuration();
+ conf.set(accountKey1, accountValue1);
+ abfsConf = new AbfsConfiguration(conf, accountName1);
+ AbfsPerfTracker tracker = new AbfsPerfTracker("test", accountName1, abfsConf);
+ when(client.getAbfsPerfTracker()).thenReturn(tracker);
+ when(client.append(anyString(), anyLong(), any(byte[].class), anyInt(), anyInt(), anyBoolean(), anyBoolean())).thenReturn(op);
+
+ AbfsOutputStream out = new AbfsOutputStream(client, path, 0, bufferSize, true, false, true, false);
+ final byte[] b = new byte[writeSize];
+ new Random().nextBytes(b);
+ out.write(b);
+ out.hsync();
+ ArgumentCaptor<String> acString = ArgumentCaptor.forClass(String.class);
+ ArgumentCaptor<Long> acLong = ArgumentCaptor.forClass(Long.class);
+ ArgumentCaptor<Integer> acInt = ArgumentCaptor.forClass(Integer.class);
+ ArgumentCaptor<Boolean> acBool = ArgumentCaptor.forClass(Boolean.class);
+ ArgumentCaptor<byte[]> acByteArray = ArgumentCaptor.forClass(byte[].class);
+
+ final byte[] b1 = new byte[2*writeSize];
+ new Random().nextBytes(b1);
+ out.write(b1);
+ out.flush();
+ out.hflush();
+
+ out.hsync();
+
+ verify(client, times(2)).append(acString.capture(), acLong.capture(), acByteArray.capture(), acInt.capture(), acInt.capture(), acBool.capture(), acBool.capture());
+ Assert.assertEquals(Arrays.asList(path, path) , acString.getAllValues());
+ Assert.assertEquals(Arrays.asList(Long.valueOf(0), Long.valueOf(writeSize)), acLong.getAllValues());
+ //flush=true, close=false, flush=true, close=false
+ Assert.assertEquals(Arrays.asList(true, false, true, false), acBool.getAllValues());
+ Assert.assertEquals(Arrays.asList(0,writeSize, 0, 2*writeSize), acInt.getAllValues());
+
+ }
+
+ @Test
+ /**
+ * The test verifies OutputStream Write of writeSize(1000 bytes) followed by a close is making correct HTTP calls to the server
+ */
+ public void verifyWriteRequest() throws Exception {
+
+ AbfsClient client = mock(AbfsClient.class);
+ AbfsRestOperation op = mock(AbfsRestOperation.class);
+ AbfsConfiguration abfsConf;
+ final Configuration conf = new Configuration();
+ conf.set(accountKey1, accountValue1);
+ abfsConf = new AbfsConfiguration(conf, accountName1);
+ AbfsPerfTracker tracker = new AbfsPerfTracker("test", accountName1, abfsConf);
+
+ when(client.getAbfsPerfTracker()).thenReturn(tracker);
+ when(client.append(anyString(), anyLong(), any(byte[].class), anyInt(), anyInt(), anyBoolean(), anyBoolean())).thenReturn(op);
+
+ AbfsOutputStream out = new AbfsOutputStream(client, path, 0, bufferSize, true, false, true, false);
+ final byte[] b = new byte[writeSize];
+ new Random().nextBytes(b);
+
+ for (int i = 0; i < 5; i++) {
+ out.write(b);
+ }
+ out.close();
+
+ ArgumentCaptor<String> acString = ArgumentCaptor.forClass(String.class);
+ ArgumentCaptor<Long> acLong = ArgumentCaptor.forClass(Long.class);
+ ArgumentCaptor<Integer> acInt = ArgumentCaptor.forClass(Integer.class);
+ ArgumentCaptor<Boolean> acBool = ArgumentCaptor.forClass(Boolean.class);
+ ArgumentCaptor<byte[]> acByteArray = ArgumentCaptor.forClass(byte[].class);
+
+ verify(client, times(2)).append(acString.capture(), acLong.capture(), acByteArray.capture(), acInt.capture(), acInt.capture(), acBool.capture(), acBool.capture());
+ Assert.assertEquals(Arrays.asList(path, path) , acString.getAllValues());
+ Assert.assertEquals(Arrays.asList(Long.valueOf(0), Long.valueOf(bufferSize)), acLong.getAllValues());
+ //flush=false,close=false, flush=true,close=true
+ Assert.assertEquals(Arrays.asList(false, false, true, true), acBool.getAllValues());
+ Assert.assertEquals(Arrays.asList(0, bufferSize, 0, 5*writeSize-bufferSize), acInt.getAllValues());
+
+ }
+
+ @Test
+ /**
+ * The test verifies OutputStream Write of bufferSize(4KB) followed by a close is making correct HTTP calls to the server
+ */
+ public void verifyWriteRequestOfBufferSizeAndClose() throws Exception {
+
+ AbfsClient client = mock(AbfsClient.class);
+ AbfsRestOperation op = mock(AbfsRestOperation.class);
+ AbfsConfiguration abfsConf;
+ final Configuration conf = new Configuration();
+ conf.set(accountKey1, accountValue1);
+ abfsConf = new AbfsConfiguration(conf, accountName1);
+ AbfsPerfTracker tracker = new AbfsPerfTracker("test", accountName1, abfsConf);
+
+ when(client.getAbfsPerfTracker()).thenReturn(tracker);
+ when(client.append(anyString(), anyLong(), any(byte[].class), anyInt(), anyInt(), anyBoolean(), anyBoolean())).thenReturn(op);
+ when(client.flush(anyString(), anyLong(), anyBoolean(), anyBoolean())).thenReturn(op);
+
+ AbfsOutputStream out = new AbfsOutputStream(client, path, 0, bufferSize, true, false, true, false);
+ final byte[] b = new byte[bufferSize];
+ new Random().nextBytes(b);
+
+ for (int i = 0; i < 2; i++) {
+ out.write(b);
+ }
+ out.close();
+
+ ArgumentCaptor<String> acString = ArgumentCaptor.forClass(String.class);
+ ArgumentCaptor<Long> acLong = ArgumentCaptor.forClass(Long.class);
+ ArgumentCaptor<Integer> acInt = ArgumentCaptor.forClass(Integer.class);
+ ArgumentCaptor<Boolean> acBool = ArgumentCaptor.forClass(Boolean.class);
+ ArgumentCaptor<byte[]> acByteArray = ArgumentCaptor.forClass(byte[].class);
+
+ verify(client, times(2)).append(acString.capture(), acLong.capture(), acByteArray.capture(), acInt.capture(), acInt.capture(), acBool.capture(), acBool.capture());
+ Assert.assertEquals(Arrays.asList(path, path) , acString.getAllValues());
+ Assert.assertEquals(new HashSet<Long>(Arrays.asList(Long.valueOf(0), Long.valueOf(bufferSize))), new HashSet<Long>(acLong.getAllValues()));
+ //flush=false, close=false, flush=false, close=false
+ Assert.assertEquals(Arrays.asList(false, false, false, false), acBool.getAllValues());
+ Assert.assertEquals(Arrays.asList(0, bufferSize, 0, bufferSize), acInt.getAllValues());
+
+ ArgumentCaptor<String> acFlushString = ArgumentCaptor.forClass(String.class);
+ ArgumentCaptor<Long> acFlushLong = ArgumentCaptor.forClass(Long.class);
+ ArgumentCaptor<Boolean> acFlushBool = ArgumentCaptor.forClass(Boolean.class);
+
+ verify(client, times(1)).flush(acFlushString.capture(), acFlushLong.capture(), acFlushBool.capture(), acFlushBool.capture());
+ Assert.assertEquals(Arrays.asList(path) , acFlushString.getAllValues());
+ Assert.assertEquals(Arrays.asList(Long.valueOf(2*bufferSize)), acFlushLong.getAllValues());
+ Assert.assertEquals(Arrays.asList(false, true), acFlushBool.getAllValues());
+
+ }
+
+ @Test
+ /**
+ * The test verifies OutputStream Write of bufferSize(4KB) is making correct HTTP calls to the server
+ */
+ public void verifyWriteRequestOfBufferSize() throws Exception {
+
+ AbfsClient client = mock(AbfsClient.class);
+ AbfsRestOperation op = mock(AbfsRestOperation.class);
+ AbfsConfiguration abfsConf;
+ final Configuration conf = new Configuration();
+ conf.set(accountKey1, accountValue1);
+ abfsConf = new AbfsConfiguration(conf, accountName1);
+ AbfsPerfTracker tracker = new AbfsPerfTracker("test", accountName1, abfsConf);
+
+ when(client.getAbfsPerfTracker()).thenReturn(tracker);
+ when(client.append(anyString(), anyLong(), any(byte[].class), anyInt(), anyInt(), anyBoolean(), anyBoolean())).thenReturn(op);
+
+ AbfsOutputStream out = new AbfsOutputStream(client, path, 0, bufferSize, true, false, true, false);
+ final byte[] b = new byte[bufferSize];
+ new Random().nextBytes(b);
+
+ for (int i = 0; i < 2; i++) {
+ out.write(b);
+ }
+ Thread.sleep(1000);
+
+ ArgumentCaptor<String> acString = ArgumentCaptor.forClass(String.class);
+ ArgumentCaptor<Long> acLong = ArgumentCaptor.forClass(Long.class);
+ ArgumentCaptor<Integer> acInt = ArgumentCaptor.forClass(Integer.class);
+ ArgumentCaptor<Boolean> acBool = ArgumentCaptor.forClass(Boolean.class);
+ ArgumentCaptor<byte[]> acByteArray = ArgumentCaptor.forClass(byte[].class);
+
+ verify(client, times(2)).append(acString.capture(), acLong.capture(), acByteArray.capture(), acInt.capture(), acInt.capture(), acBool.capture(), acBool.capture());
+ Assert.assertEquals(Arrays.asList(path, path) , acString.getAllValues());
+ Assert.assertEquals(new HashSet<Long>(Arrays.asList(Long.valueOf(0), Long.valueOf(bufferSize))), new HashSet<Long>(acLong.getAllValues()));
+ //flush=false, close=false, flush=false, close=false
+ Assert.assertEquals(Arrays.asList(false, false, false, false), acBool.getAllValues());
+ Assert.assertEquals(Arrays.asList(0, bufferSize, 0, bufferSize), acInt.getAllValues());
+
+ }
+
+ @Test
+ /**
+ * The test verifies OutputStream Write of bufferSize(4KB) on a AppendBlob based stream is making correct HTTP calls to the server
+ */
+ public void verifyWriteRequestOfBufferSizeWithAppendBlob() throws Exception {
+
+ AbfsClient client = mock(AbfsClient.class);
+ AbfsRestOperation op = mock(AbfsRestOperation.class);
+ AbfsConfiguration abfsConf;
+ final Configuration conf = new Configuration();
+ conf.set(accountKey1, accountValue1);
+ abfsConf = new AbfsConfiguration(conf, accountName1);
+ AbfsPerfTracker tracker = new AbfsPerfTracker("test", accountName1, abfsConf);
+
+ when(client.getAbfsPerfTracker()).thenReturn(tracker);
+ when(client.append(anyString(), anyLong(), any(byte[].class), anyInt(), anyInt(), anyBoolean(), anyBoolean())).thenReturn(op);
+
+ AbfsOutputStream out = new AbfsOutputStream(client, path, 0, bufferSize, true, false, true, false);
+ final byte[] b = new byte[bufferSize];
+ new Random().nextBytes(b);
+
+ for (int i = 0; i < 2; i++) {
+ out.write(b);
+ }
+ Thread.sleep(1000);
+
+ ArgumentCaptor<String> acString = ArgumentCaptor.forClass(String.class);
+ ArgumentCaptor<Long> acLong = ArgumentCaptor.forClass(Long.class);
+ ArgumentCaptor<Integer> acInt = ArgumentCaptor.forClass(Integer.class);
+ ArgumentCaptor<Boolean> acBool = ArgumentCaptor.forClass(Boolean.class);
+ ArgumentCaptor<byte[]> acByteArray = ArgumentCaptor.forClass(byte[].class);
+
+ verify(client, times(2)).append(acString.capture(), acLong.capture(), acByteArray.capture(), acInt.capture(), acInt.capture(), acBool.capture(), acBool.capture());
+ Assert.assertEquals(Arrays.asList(path, path) , acString.getAllValues());
+ Assert.assertEquals(Arrays.asList(Long.valueOf(0), Long.valueOf(bufferSize)), acLong.getAllValues());
+ //flush=false, close=false, flush=false, close=false
+ Assert.assertEquals(Arrays.asList(false, false, false, false), acBool.getAllValues());
+ Assert.assertEquals(Arrays.asList(0, bufferSize, 0, bufferSize), acInt.getAllValues());
+
+ }
+
+ @Test
+ /**
+ * The test verifies OutputStream Write of bufferSize(4KB) followed by a hflush call is making correct HTTP calls to the server
+ */
+ public void verifyWriteRequestOfBufferSizeAndHFlush() throws Exception {
+
+ AbfsClient client = mock(AbfsClient.class);
+ AbfsRestOperation op = mock(AbfsRestOperation.class);
+ AbfsConfiguration abfsConf;
+ final Configuration conf = new Configuration();
+ conf.set(accountKey1, accountValue1);
+ abfsConf = new AbfsConfiguration(conf, accountName1);
+ AbfsPerfTracker tracker = new AbfsPerfTracker("test", accountName1, abfsConf);
+
+ when(client.getAbfsPerfTracker()).thenReturn(tracker);
+ when(client.append(anyString(), anyLong(), any(byte[].class), anyInt(), anyInt(), anyBoolean(), anyBoolean())).thenReturn(op);
+ when(client.flush(anyString(), anyLong(), anyBoolean(), anyBoolean())).thenReturn(op);
+
+ AbfsOutputStream out = new AbfsOutputStream(client, path, 0, bufferSize, true, false, true, false);
+ final byte[] b = new byte[bufferSize];
+ new Random().nextBytes(b);
+
+ for (int i = 0; i < 2; i++) {
+ out.write(b);
+ }
+ out.hflush();
+
+ ArgumentCaptor<String> acString = ArgumentCaptor.forClass(String.class);
+ ArgumentCaptor<Long> acLong = ArgumentCaptor.forClass(Long.class);
+ ArgumentCaptor<Integer> acInt = ArgumentCaptor.forClass(Integer.class);
+ ArgumentCaptor<Boolean> acBool = ArgumentCaptor.forClass(Boolean.class);
+ ArgumentCaptor<byte[]> acByteArray = ArgumentCaptor.forClass(byte[].class);
+
+ verify(client, times(2)).append(acString.capture(), acLong.capture(), acByteArray.capture(), acInt.capture(), acInt.capture(), acBool.capture(), acBool.capture());
+ Assert.assertEquals(Arrays.asList(path, path) , acString.getAllValues());
+ Assert.assertEquals(new HashSet<Long>(Arrays.asList(Long.valueOf(0), Long.valueOf(bufferSize))), new HashSet<Long>(acLong.getAllValues()));
+ //flush=false, close=false, flush=false, close=false
+ Assert.assertEquals(Arrays.asList(false, false, false, false), acBool.getAllValues());
+ Assert.assertEquals(Arrays.asList(0, bufferSize, 0, bufferSize), acInt.getAllValues());
+
+ ArgumentCaptor<String> acFlushString = ArgumentCaptor.forClass(String.class);
+ ArgumentCaptor<Long> acFlushLong = ArgumentCaptor.forClass(Long.class);
+ ArgumentCaptor<Boolean> acFlushBool = ArgumentCaptor.forClass(Boolean.class);
+
+ verify(client, times(1)).flush(acFlushString.capture(), acFlushLong.capture(), acFlushBool.capture(), acFlushBool.capture());
+ Assert.assertEquals(Arrays.asList(path) , acFlushString.getAllValues());
+ Assert.assertEquals(Arrays.asList(Long.valueOf(2*bufferSize)), acFlushLong.getAllValues());
+ Assert.assertEquals(Arrays.asList(false, false), acFlushBool.getAllValues());
+
+ }
+
+ @Test
+ /**
+ * The test verifies OutputStream Write of bufferSize(4KB) followed by a flush call is making correct HTTP calls to the server
+ */
+ public void verifyWriteRequestOfBufferSizeAndFlush() throws Exception {
+
+ AbfsClient client = mock(AbfsClient.class);
+ AbfsRestOperation op = mock(AbfsRestOperation.class);
+ AbfsConfiguration abfsConf;
+ final Configuration conf = new Configuration();
+ conf.set(accountKey1, accountValue1);
+ abfsConf = new AbfsConfiguration(conf, accountName1);
+ AbfsPerfTracker tracker = new AbfsPerfTracker("test", accountName1, abfsConf);
+ when(client.getAbfsPerfTracker()).thenReturn(tracker);
+ when(client.append(anyString(), anyLong(), any(byte[].class), anyInt(), anyInt(), anyBoolean(), anyBoolean())).thenReturn(op);
+
+ AbfsOutputStream out = new AbfsOutputStream(client, path, 0, bufferSize, true, false, true, false);
+ final byte[] b = new byte[bufferSize];
+ new Random().nextBytes(b);
+
+ for (int i = 0; i < 2; i++) {
+ out.write(b);
+ }
+ out.flush();
+ Thread.sleep(1000);
+
+ ArgumentCaptor<String> acString = ArgumentCaptor.forClass(String.class);
+ ArgumentCaptor<Long> acLong = ArgumentCaptor.forClass(Long.class);
+ ArgumentCaptor<Integer> acInt = ArgumentCaptor.forClass(Integer.class);
+ ArgumentCaptor<Boolean> acBool = ArgumentCaptor.forClass(Boolean.class);
+ ArgumentCaptor<byte[]> acByteArray = ArgumentCaptor.forClass(byte[].class);
+
+ verify(client, times(2)).append(acString.capture(), acLong.capture(), acByteArray.capture(), acInt.capture(), acInt.capture(), acBool.capture(), acBool.capture());
+ Assert.assertEquals(Arrays.asList(path, path) , acString.getAllValues());
Review comment:
* Can you look at moving to assertJ here; these declarations are complex enough they need one
* and add error strings via .describedAs()
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] bilaharith commented on a change in pull request #1790:
[HADOOP-16818] ABFS: Combine append+flush calls for blockblob & appendblob
Posted by GitBox <gi...@apache.org>.
bilaharith commented on a change in pull request #1790: [HADOOP-16818] ABFS: Combine append+flush calls for blockblob & appendblob
URL: https://github.com/apache/hadoop/pull/1790#discussion_r392937874
##########
File path: hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/services/TestAbfsOutputStream.java
##########
@@ -0,0 +1,355 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.azurebfs.services;
+import java.util.Arrays;
+import java.util.HashSet;
+import java.util.Random;
+import org.junit.Assert;
+import org.junit.Test;
+import org.mockito.ArgumentCaptor;
+import org.apache.hadoop.fs.azurebfs.AbfsConfiguration;
+import org.apache.hadoop.conf.Configuration;
+
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.when;
+import static org.mockito.Mockito.times;
+import static org.mockito.Mockito.verify;
+import static org.mockito.Mockito.verifyNoMoreInteractions;
+import static org.mockito.Mockito.*;
+
+public final class TestAbfsOutputStream {
+
+ private static int bufferSize = 4096;
+ private static int writeSize = 1000;
+ private static String path = "~/testpath";
+ private final String globalKey = "fs.azure.configuration";
+ private final String accountName1 = "account1";
+ private final String accountKey1 = globalKey + "." + accountName1;
+ private final String accountValue1 = "one";
+
+ @Test
+ /**
+ * The test verifies OutputStream shortwrite case(2000bytes write followed by flush, hflush, hsync) is making correct HTTP calls to the server
+ */
+ public void verifyShortWriteRequest() throws Exception {
+
+ AbfsClient client = mock(AbfsClient.class);
+ AbfsRestOperation op = mock(AbfsRestOperation.class);
+ AbfsConfiguration abfsConf;
+ final Configuration conf = new Configuration();
+ conf.set(accountKey1, accountValue1);
+ abfsConf = new AbfsConfiguration(conf, accountName1);
+ AbfsPerfTracker tracker = new AbfsPerfTracker("test", accountName1, abfsConf);
+ when(client.getAbfsPerfTracker()).thenReturn(tracker);
+ when(client.append(anyString(), anyLong(), any(byte[].class), anyInt(), anyInt(), anyBoolean(), anyBoolean())).thenReturn(op);
+
+ AbfsOutputStream out = new AbfsOutputStream(client, path, 0, bufferSize, true, false, true, false);
+ final byte[] b = new byte[writeSize];
+ new Random().nextBytes(b);
+ out.write(b);
+ out.hsync();
+ ArgumentCaptor<String> acString = ArgumentCaptor.forClass(String.class);
+ ArgumentCaptor<Long> acLong = ArgumentCaptor.forClass(Long.class);
+ ArgumentCaptor<Integer> acInt = ArgumentCaptor.forClass(Integer.class);
+ ArgumentCaptor<Boolean> acBool = ArgumentCaptor.forClass(Boolean.class);
+ ArgumentCaptor<byte[]> acByteArray = ArgumentCaptor.forClass(byte[].class);
+
+ final byte[] b1 = new byte[2*writeSize];
+ new Random().nextBytes(b1);
+ out.write(b1);
+ out.flush();
+ out.hflush();
+
+ out.hsync();
+
+ verify(client, times(2)).append(acString.capture(), acLong.capture(), acByteArray.capture(), acInt.capture(), acInt.capture(), acBool.capture(), acBool.capture());
Review comment:
Nit: Line is longer than 160 characters
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] ishaniahuja commented on a change in pull request #1790: changes for combined append+flush calls for blockblob(traditionan) an…
Posted by GitBox <gi...@apache.org>.
ishaniahuja commented on a change in pull request #1790: changes for combined append+flush calls for blockblob(traditionan) an…
URL: https://github.com/apache/hadoop/pull/1790#discussion_r365108093
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsClient.java
##########
@@ -55,7 +55,7 @@
public static final Logger LOG = LoggerFactory.getLogger(AbfsClient.class);
private final URL baseUrl;
private final SharedKeyCredentials sharedKeyCredentials;
- private final String xMsVersion = "2018-11-09";
+ private final String xMsVersion = "2019-12-12";
Review comment:
nopes actually. how can we go about using the new version with the new API only? is there a way to do this in the driver?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] snvijaya commented on a change in pull request #1790: changes for combined append+flush calls for blockblob(traditionan) an…
Posted by GitBox <gi...@apache.org>.
snvijaya commented on a change in pull request #1790: changes for combined append+flush calls for blockblob(traditionan) an…
URL: https://github.com/apache/hadoop/pull/1790#discussion_r363151516
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AbfsConfiguration.java
##########
@@ -146,6 +150,10 @@
DefaultValue = DEFAULT_ENABLE_FLUSH)
private boolean enableFlush;
+ @BooleanConfigurationValidatorAnnotation(ConfigurationKey = FS_AZURE_ENABLE_APPEND_WITH_FLUSH,
+ DefaultValue = DEFAULT_ENABLE_APPEND_WITH_FLUSH)
+ private boolean enableAppendWithFlush;
Review comment:
This config value is passed until AbfsOutputStream.java -> writeCurrentBufferToService() as the first argument. but no specific action is seen inside writeCurrentBufferToService() if the value is true or false.
What is intended of this config ? is it required ?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] ishaniahuja commented on a change in pull request #1790: changes for combined append+flush calls for blockblob(traditionan) an…
Posted by GitBox <gi...@apache.org>.
ishaniahuja commented on a change in pull request #1790: changes for combined append+flush calls for blockblob(traditionan) an…
URL: https://github.com/apache/hadoop/pull/1790#discussion_r365108476
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AbfsConfiguration.java
##########
@@ -411,6 +419,10 @@ public String getAzureAtomicRenameDirs() {
return this.azureAtomicDirs;
}
+ public String getAppendBlobDirs() {
Review comment:
this is just the way it is done for other config parameter for dirs. see: getAzureAtomicRenameDirs
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on a change in pull request #1790:
[HADOOP-16818] ABFS: Combine append+flush calls for blockblob & appendblob
Posted by GitBox <gi...@apache.org>.
steveloughran commented on a change in pull request #1790: [HADOOP-16818] ABFS: Combine append+flush calls for blockblob & appendblob
URL: https://github.com/apache/hadoop/pull/1790#discussion_r386486689
##########
File path: hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/services/TestAbfsOutputStream.java
##########
@@ -0,0 +1,349 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.azurebfs.services;
+import org.apache.hadoop.fs.azurebfs.AbfsConfiguration;
+import org.apache.hadoop.conf.Configuration;
+import java.util.Arrays;
+import java.util.HashSet;
+import java.util.Random;
+import org.junit.Assert;
+import org.junit.Test;
+
+import org.mockito.ArgumentCaptor;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.when;
+import static org.mockito.Mockito.times;
+import static org.mockito.Mockito.verify;
+import static org.mockito.Mockito.verifyNoMoreInteractions;
+import static org.mockito.Mockito.*;
+/**
+ * Test useragent of abfs client.
+ *
+ */
+public final class TestAbfsOutputStream {
+
+ private static int bufferSize = 4096;
+ private static int writeSize = 1000;
+ private static String path = "~/testpath";
+ private final String globalKey = "fs.azure.configuration";
+ private final String accountName1 = "account1";
+ private final String accountKey1 = globalKey + "." + accountName1;
+ private final String accountValue1 = "one";
+
+ @Test
+ public void verifyShortWriteRequest() throws Exception {
+
+ AbfsClient client = mock(AbfsClient.class);
+ AbfsRestOperation op = mock(AbfsRestOperation.class);
+ AbfsConfiguration abfsConf;
+ final Configuration conf = new Configuration();
+ conf.set(accountKey1, accountValue1);
+ abfsConf = new AbfsConfiguration(conf, accountName1);
+ AbfsPerfTracker tracker = new AbfsPerfTracker("test", accountName1, abfsConf);
+ when(client.getAbfsPerfTracker()).thenReturn(tracker);
+ when(client.append(anyString(), anyLong(), any(byte[].class), anyInt(), anyInt(), anyBoolean(), anyBoolean())).thenReturn(op);
+
+ AbfsOutputStream out = new AbfsOutputStream(client, path, 0, bufferSize, true, false, true, false);
+ final byte[] b = new byte[writeSize];
+ new Random().nextBytes(b);
+ out.write(b);
+ out.hsync();
+ ArgumentCaptor<String> acString = ArgumentCaptor.forClass(String.class);
+ ArgumentCaptor<Long> acLong = ArgumentCaptor.forClass(Long.class);
+ ArgumentCaptor<Integer> acInt = ArgumentCaptor.forClass(Integer.class);
+ ArgumentCaptor<Boolean> acBool = ArgumentCaptor.forClass(Boolean.class);
+ ArgumentCaptor<byte[]> acByteArray = ArgumentCaptor.forClass(byte[].class);
+
+ final byte[] b1 = new byte[2*writeSize];
+ new Random().nextBytes(b1);
+ out.write(b1);
+ out.flush();
+ out.hflush();
+
+ out.hsync();
+
+ verify(client, times(2)).append(acString.capture(), acLong.capture(), acByteArray.capture(), acInt.capture(), acInt.capture(), acBool.capture(), acBool.capture());
+ Assert.assertEquals(Arrays.asList(path, path) , acString.getAllValues());
+ Assert.assertEquals(Arrays.asList(Long.valueOf(0), Long.valueOf(writeSize)), acLong.getAllValues());
+ //flush=true, close=false, flush=true, close=false
+ Assert.assertEquals(Arrays.asList(true, false, true, false), acBool.getAllValues());
+ Assert.assertEquals(Arrays.asList(0,writeSize, 0, 2*writeSize), acInt.getAllValues());
+
+ //verifyNoMoreInteractions(client);
+
+ }
+
+ @Test
+ public void verifyWriteRequest() throws Exception {
+
+ AbfsClient client = mock(AbfsClient.class);
+ AbfsRestOperation op = mock(AbfsRestOperation.class);
+ AbfsConfiguration abfsConf;
+ final Configuration conf = new Configuration();
+ conf.set(accountKey1, accountValue1);
+ abfsConf = new AbfsConfiguration(conf, accountName1);
+ AbfsPerfTracker tracker = new AbfsPerfTracker("test", accountName1, abfsConf);
+
+ when(client.getAbfsPerfTracker()).thenReturn(tracker);
+ when(client.append(anyString(), anyLong(), any(byte[].class), anyInt(), anyInt(), anyBoolean(), anyBoolean())).thenReturn(op);
+
+ AbfsOutputStream out = new AbfsOutputStream(client, path, 0, bufferSize, true, false, true, false);
+ final byte[] b = new byte[writeSize];
+ new Random().nextBytes(b);
+
+ for (int i = 0; i < 5; i++) {
+ out.write(b);
+ }
+ out.close();
+
+ ArgumentCaptor<String> acString = ArgumentCaptor.forClass(String.class);
+ ArgumentCaptor<Long> acLong = ArgumentCaptor.forClass(Long.class);
+ ArgumentCaptor<Integer> acInt = ArgumentCaptor.forClass(Integer.class);
+ ArgumentCaptor<Boolean> acBool = ArgumentCaptor.forClass(Boolean.class);
+ ArgumentCaptor<byte[]> acByteArray = ArgumentCaptor.forClass(byte[].class);
+
+ verify(client, times(2)).append(acString.capture(), acLong.capture(), acByteArray.capture(), acInt.capture(), acInt.capture(), acBool.capture(), acBool.capture());
+ Assert.assertEquals(Arrays.asList(path, path) , acString.getAllValues());
+ Assert.assertEquals(Arrays.asList(Long.valueOf(0), Long.valueOf(bufferSize)), acLong.getAllValues());
+ //flush=false,close=false, flush=true,close=true
+ Assert.assertEquals(Arrays.asList(false, false, true, true), acBool.getAllValues());
+ Assert.assertEquals(Arrays.asList(0, bufferSize, 0, 5*writeSize-bufferSize), acInt.getAllValues());
+
+ //verifyNoMoreInteractions(client);
+ }
+
+ @Test
+ public void verifyWriteRequestOfBufferSizeAndClose() throws Exception {
+
+ AbfsClient client = mock(AbfsClient.class);
+ AbfsRestOperation op = mock(AbfsRestOperation.class);
+ AbfsConfiguration abfsConf;
+ final Configuration conf = new Configuration();
+ conf.set(accountKey1, accountValue1);
+ abfsConf = new AbfsConfiguration(conf, accountName1);
+ AbfsPerfTracker tracker = new AbfsPerfTracker("test", accountName1, abfsConf);
+
+ when(client.getAbfsPerfTracker()).thenReturn(tracker);
+ when(client.append(anyString(), anyLong(), any(byte[].class), anyInt(), anyInt(), anyBoolean(), anyBoolean())).thenReturn(op);
+ when(client.flush(anyString(), anyLong(), anyBoolean(), anyBoolean())).thenReturn(op);
+
+ AbfsOutputStream out = new AbfsOutputStream(client, path, 0, bufferSize, true, false, true, false);
+ final byte[] b = new byte[bufferSize];
+ new Random().nextBytes(b);
+
+ for (int i = 0; i < 2; i++) {
+ out.write(b);
+ }
+ out.close();
+
+ ArgumentCaptor<String> acString = ArgumentCaptor.forClass(String.class);
+ ArgumentCaptor<Long> acLong = ArgumentCaptor.forClass(Long.class);
+ ArgumentCaptor<Integer> acInt = ArgumentCaptor.forClass(Integer.class);
+ ArgumentCaptor<Boolean> acBool = ArgumentCaptor.forClass(Boolean.class);
+ ArgumentCaptor<byte[]> acByteArray = ArgumentCaptor.forClass(byte[].class);
+
+ verify(client, times(2)).append(acString.capture(), acLong.capture(), acByteArray.capture(), acInt.capture(), acInt.capture(), acBool.capture(), acBool.capture());
+ Assert.assertEquals(Arrays.asList(path, path) , acString.getAllValues());
+ Assert.assertEquals(new HashSet<Long>(Arrays.asList(Long.valueOf(0), Long.valueOf(bufferSize))), new HashSet<Long>(acLong.getAllValues()));
+ //flush=false, close=false, flush=false, close=false
+ Assert.assertEquals(Arrays.asList(false, false, false, false), acBool.getAllValues());
+ Assert.assertEquals(Arrays.asList(0, bufferSize, 0, bufferSize), acInt.getAllValues());
+
+ ArgumentCaptor<String> acFlushString = ArgumentCaptor.forClass(String.class);
+ ArgumentCaptor<Long> acFlushLong = ArgumentCaptor.forClass(Long.class);
+ ArgumentCaptor<Boolean> acFlushBool = ArgumentCaptor.forClass(Boolean.class);
+
+ verify(client, times(1)).flush(acFlushString.capture(), acFlushLong.capture(), acFlushBool.capture(), acFlushBool.capture());
+ Assert.assertEquals(Arrays.asList(path) , acFlushString.getAllValues());
+ Assert.assertEquals(Arrays.asList(Long.valueOf(2*bufferSize)), acFlushLong.getAllValues());
+ Assert.assertEquals(Arrays.asList(false, true), acFlushBool.getAllValues());
+
+// verifyNoMoreInteractions(client);
+ }
+
+ @Test
+ public void verifyWriteRequestOfBufferSize() throws Exception {
+
+ AbfsClient client = mock(AbfsClient.class);
+ AbfsRestOperation op = mock(AbfsRestOperation.class);
+ AbfsConfiguration abfsConf;
+ final Configuration conf = new Configuration();
+ conf.set(accountKey1, accountValue1);
+ abfsConf = new AbfsConfiguration(conf, accountName1);
+ AbfsPerfTracker tracker = new AbfsPerfTracker("test", accountName1, abfsConf);
+
+ when(client.getAbfsPerfTracker()).thenReturn(tracker);
+ when(client.append(anyString(), anyLong(), any(byte[].class), anyInt(), anyInt(), anyBoolean(), anyBoolean())).thenReturn(op);
+
+ AbfsOutputStream out = new AbfsOutputStream(client, path, 0, bufferSize, true, false, true, false);
+ final byte[] b = new byte[bufferSize];
+ new Random().nextBytes(b);
+
+ for (int i = 0; i < 2; i++) {
+ out.write(b);
+ }
+ Thread.sleep(1000);
+
+ ArgumentCaptor<String> acString = ArgumentCaptor.forClass(String.class);
+ ArgumentCaptor<Long> acLong = ArgumentCaptor.forClass(Long.class);
+ ArgumentCaptor<Integer> acInt = ArgumentCaptor.forClass(Integer.class);
+ ArgumentCaptor<Boolean> acBool = ArgumentCaptor.forClass(Boolean.class);
+ ArgumentCaptor<byte[]> acByteArray = ArgumentCaptor.forClass(byte[].class);
+
+ verify(client, times(2)).append(acString.capture(), acLong.capture(), acByteArray.capture(), acInt.capture(), acInt.capture(), acBool.capture(), acBool.capture());
+ Assert.assertEquals(Arrays.asList(path, path) , acString.getAllValues());
+ Assert.assertEquals(new HashSet<Long>(Arrays.asList(Long.valueOf(0), Long.valueOf(bufferSize))), new HashSet<Long>(acLong.getAllValues()));
+ //flush=false, close=false, flush=false, close=false
+ Assert.assertEquals(Arrays.asList(false, false, false, false), acBool.getAllValues());
+ Assert.assertEquals(Arrays.asList(0, bufferSize, 0, bufferSize), acInt.getAllValues());
+
+ // verifyNoMoreInteractions(client);
+ }
+
+ @Test
+ public void verifyWriteRequestOfBufferSizeWithAppendBlob() throws Exception {
+
+ AbfsClient client = mock(AbfsClient.class);
+ AbfsRestOperation op = mock(AbfsRestOperation.class);
+ AbfsConfiguration abfsConf;
+ final Configuration conf = new Configuration();
+ conf.set(accountKey1, accountValue1);
+ abfsConf = new AbfsConfiguration(conf, accountName1);
+ AbfsPerfTracker tracker = new AbfsPerfTracker("test", accountName1, abfsConf);
+
+ when(client.getAbfsPerfTracker()).thenReturn(tracker);
+ when(client.append(anyString(), anyLong(), any(byte[].class), anyInt(), anyInt(), anyBoolean(), anyBoolean())).thenReturn(op);
+
+ AbfsOutputStream out = new AbfsOutputStream(client, path, 0, bufferSize, true, false, true, false);
+ final byte[] b = new byte[bufferSize];
+ new Random().nextBytes(b);
+
+ for (int i = 0; i < 2; i++) {
+ out.write(b);
+ }
+ Thread.sleep(1000);
+
+ ArgumentCaptor<String> acString = ArgumentCaptor.forClass(String.class);
+ ArgumentCaptor<Long> acLong = ArgumentCaptor.forClass(Long.class);
+ ArgumentCaptor<Integer> acInt = ArgumentCaptor.forClass(Integer.class);
+ ArgumentCaptor<Boolean> acBool = ArgumentCaptor.forClass(Boolean.class);
+ ArgumentCaptor<byte[]> acByteArray = ArgumentCaptor.forClass(byte[].class);
+
+ verify(client, times(2)).append(acString.capture(), acLong.capture(), acByteArray.capture(), acInt.capture(), acInt.capture(), acBool.capture(), acBool.capture());
+ Assert.assertEquals(Arrays.asList(path, path) , acString.getAllValues());
+ Assert.assertEquals(Arrays.asList(Long.valueOf(0), Long.valueOf(bufferSize)), acLong.getAllValues());
+ //flush=false, close=false, flush=false, close=false
+ Assert.assertEquals(Arrays.asList(false, false, false, false), acBool.getAllValues());
+ Assert.assertEquals(Arrays.asList(0, bufferSize, 0, bufferSize), acInt.getAllValues());
+
+ //verifyNoMoreInteractions(client);
+ }
+
+ @Test
+ public void verifyWriteRequestOfBufferSizeAndHFlush() throws Exception {
+
+ AbfsClient client = mock(AbfsClient.class);
+ AbfsRestOperation op = mock(AbfsRestOperation.class);
+ AbfsConfiguration abfsConf;
+ final Configuration conf = new Configuration();
+ conf.set(accountKey1, accountValue1);
+ abfsConf = new AbfsConfiguration(conf, accountName1);
+ AbfsPerfTracker tracker = new AbfsPerfTracker("test", accountName1, abfsConf);
+
+ when(client.getAbfsPerfTracker()).thenReturn(tracker);
+ when(client.append(anyString(), anyLong(), any(byte[].class), anyInt(), anyInt(), anyBoolean(), anyBoolean())).thenReturn(op);
+ when(client.flush(anyString(), anyLong(), anyBoolean(), anyBoolean())).thenReturn(op);
+
+ AbfsOutputStream out = new AbfsOutputStream(client, path, 0, bufferSize, true, false, true, false);
+ final byte[] b = new byte[bufferSize];
+ new Random().nextBytes(b);
+
+ for (int i = 0; i < 2; i++) {
+ out.write(b);
+ }
+ out.hflush();
+
+ ArgumentCaptor<String> acString = ArgumentCaptor.forClass(String.class);
+ ArgumentCaptor<Long> acLong = ArgumentCaptor.forClass(Long.class);
+ ArgumentCaptor<Integer> acInt = ArgumentCaptor.forClass(Integer.class);
+ ArgumentCaptor<Boolean> acBool = ArgumentCaptor.forClass(Boolean.class);
+ ArgumentCaptor<byte[]> acByteArray = ArgumentCaptor.forClass(byte[].class);
+
+ verify(client, times(2)).append(acString.capture(), acLong.capture(), acByteArray.capture(), acInt.capture(), acInt.capture(), acBool.capture(), acBool.capture());
+ Assert.assertEquals(Arrays.asList(path, path) , acString.getAllValues());
+ Assert.assertEquals(new HashSet<Long>(Arrays.asList(Long.valueOf(0), Long.valueOf(bufferSize))), new HashSet<Long>(acLong.getAllValues()));
+ //flush=false, close=false, flush=false, close=false
+ Assert.assertEquals(Arrays.asList(false, false, false, false), acBool.getAllValues());
+ Assert.assertEquals(Arrays.asList(0, bufferSize, 0, bufferSize), acInt.getAllValues());
+
+ ArgumentCaptor<String> acFlushString = ArgumentCaptor.forClass(String.class);
+ ArgumentCaptor<Long> acFlushLong = ArgumentCaptor.forClass(Long.class);
+ ArgumentCaptor<Boolean> acFlushBool = ArgumentCaptor.forClass(Boolean.class);
+
+ verify(client, times(1)).flush(acFlushString.capture(), acFlushLong.capture(), acFlushBool.capture(), acFlushBool.capture());
+ Assert.assertEquals(Arrays.asList(path) , acFlushString.getAllValues());
+ Assert.assertEquals(Arrays.asList(Long.valueOf(2*bufferSize)), acFlushLong.getAllValues());
+ Assert.assertEquals(Arrays.asList(false, false), acFlushBool.getAllValues());
+
+ //verifyNoMoreInteractions(client);
+ }
+
+ @Test
+ public void verifyWriteRequestOfBufferSizeAndFlush() throws Exception {
+
+ AbfsClient client = mock(AbfsClient.class);
+ AbfsRestOperation op = mock(AbfsRestOperation.class);
+ AbfsConfiguration abfsConf;
+ final Configuration conf = new Configuration();
+ conf.set(accountKey1, accountValue1);
+ abfsConf = new AbfsConfiguration(conf, accountName1);
+ AbfsPerfTracker tracker = new AbfsPerfTracker("test", accountName1, abfsConf);
+ when(client.getAbfsPerfTracker()).thenReturn(tracker);
+ when(client.append(anyString(), anyLong(), any(byte[].class), anyInt(), anyInt(), anyBoolean(), anyBoolean())).thenReturn(op);
+
+ AbfsOutputStream out = new AbfsOutputStream(client, path, 0, bufferSize, true, false, true, false);
+ final byte[] b = new byte[bufferSize];
+ new Random().nextBytes(b);
+
+ for (int i = 0; i < 2; i++) {
+ out.write(b);
+ }
+ out.flush();
+ Thread.sleep(1000);
+
+ ArgumentCaptor<String> acString = ArgumentCaptor.forClass(String.class);
+ ArgumentCaptor<Long> acLong = ArgumentCaptor.forClass(Long.class);
+ ArgumentCaptor<Integer> acInt = ArgumentCaptor.forClass(Integer.class);
+ ArgumentCaptor<Boolean> acBool = ArgumentCaptor.forClass(Boolean.class);
+ ArgumentCaptor<byte[]> acByteArray = ArgumentCaptor.forClass(byte[].class);
+
+ verify(client, times(2)).append(acString.capture(), acLong.capture(), acByteArray.capture(), acInt.capture(), acInt.capture(), acBool.capture(), acBool.capture());
+ Assert.assertEquals(Arrays.asList(path, path) , acString.getAllValues());
+ Assert.assertEquals(new HashSet<Long>(Arrays.asList(Long.valueOf(0), Long.valueOf(bufferSize))), new HashSet<Long>(acLong.getAllValues()));
+ //flush=false, close=false, flush=false, close=false
+ Assert.assertEquals(Arrays.asList(false, false, false, false), acBool.getAllValues());
+ Assert.assertEquals(Arrays.asList(0, bufferSize, 0, bufferSize), acInt.getAllValues());
+
+ ArgumentCaptor<String> acFlushString = ArgumentCaptor.forClass(String.class);
Review comment:
are these used
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] ishaniahuja commented on a change in pull request #1790:
[HADOOP-16818] ABFS: Combine append+flush calls for blockblob & appendblob
Posted by GitBox <gi...@apache.org>.
ishaniahuja commented on a change in pull request #1790: [HADOOP-16818] ABFS: Combine append+flush calls for blockblob & appendblob
URL: https://github.com/apache/hadoop/pull/1790#discussion_r379260387
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsClient.java
##########
@@ -254,6 +273,9 @@ public AbfsRestOperation createPath(final String path, final boolean isFile, fin
final AbfsUriQueryBuilder abfsUriQueryBuilder = createDefaultUriQueryBuilder();
abfsUriQueryBuilder.addQuery(QUERY_PARAM_RESOURCE, isFile ? FILE : DIRECTORY);
+ if (appendBlob) {
Review comment:
tested the changes in a test tenant, with rest version older than Dec 12.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on a change in pull request #1790:
[HADOOP-16818] ABFS: Combine append+flush calls for blockblob & appendblob
Posted by GitBox <gi...@apache.org>.
steveloughran commented on a change in pull request #1790: [HADOOP-16818] ABFS: Combine append+flush calls for blockblob & appendblob
URL: https://github.com/apache/hadoop/pull/1790#discussion_r387726574
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AzureBlobFileSystemStore.java
##########
@@ -404,18 +425,25 @@ public OutputStream createFile(final Path path, final boolean overwrite, final F
umask.toString(),
isNamespaceEnabled);
- final AbfsRestOperation op = client.createPath(AbfsHttpConstants.FORWARD_SLASH + getRelativePath(path), true, overwrite,
- isNamespaceEnabled ? getOctalNotation(permission) : null,
- isNamespaceEnabled ? getOctalNotation(umask) : null);
- perfInfo.registerResult(op.getResult()).registerSuccess(true);
+ boolean appendBlob = false;
+ if (isAppendBlobKey(path.toString())) {
+ appendBlob = true;
+ }
+
+ client.createPath(AbfsHttpConstants.FORWARD_SLASH + getRelativePath(path), true, overwrite,
+ isNamespaceEnabled ? getOctalNotation(permission) : null,
+ isNamespaceEnabled ? getOctalNotation(umask) : null,
+ appendBlob);
return new AbfsOutputStream(
- client,
- AbfsHttpConstants.FORWARD_SLASH + getRelativePath(path),
- 0,
- abfsConfiguration.getWriteBufferSize(),
- abfsConfiguration.isFlushEnabled(),
- abfsConfiguration.isOutputStreamFlushDisabled());
+ client,
+ AbfsHttpConstants.FORWARD_SLASH + getRelativePath(path),
+ Long.valueOf(0),
Review comment:
nit: why not just 0?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] snvijaya commented on a change in pull request #1790: changes for combined append+flush calls for blockblob(traditionan) an…
Posted by GitBox <gi...@apache.org>.
snvijaya commented on a change in pull request #1790: changes for combined append+flush calls for blockblob(traditionan) an…
URL: https://github.com/apache/hadoop/pull/1790#discussion_r365196475
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsOutputStream.java
##########
@@ -287,13 +297,13 @@ private synchronized void writeCurrentBufferToService() throws IOException {
@Override
public Void call() throws Exception {
client.append(path, offset, bytes, 0,
Review comment:
In the trunk, client calls have AbfsPerfTracker tracking all the calls. Not sure why the PR view is failing to show it as a delta from trunk. But looks like there is some issue here with merge from trunk. Please do a file to file compare from PR branch to trunk.
example: from trunk:
` public Void call() throws Exception {
AbfsPerfTracker tracker = client.getAbfsPerfTracker();
try (AbfsPerfInfo perfInfo = new AbfsPerfInfo(tracker,
"writeCurrentBufferToService", "append")) {
AbfsRestOperation op = client.append(path, offset, bytes, 0,
bytesLength);
perfInfo.registerResult(op.getResult());
byteBufferPool.putBuffer(ByteBuffer.wrap(bytes));
perfInfo.registerSuccess(true);
return null;
}`
(this is not the only delta)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] snvijaya commented on issue #1790: [HADOOP-16818] ABFS:
Combine append+flush calls for blockblob & appendblob
Posted by GitBox <gi...@apache.org>.
snvijaya commented on issue #1790: [HADOOP-16818] ABFS: Combine append+flush calls for blockblob & appendblob
URL: https://github.com/apache/hadoop/pull/1790#issuecomment-590679751
Changes look good to me.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] ishaniahuja commented on a change in pull request #1790:
[HADOOP-16818] ABFS: Combine append+flush calls for blockblob & appendblob
Posted by GitBox <gi...@apache.org>.
ishaniahuja commented on a change in pull request #1790: [HADOOP-16818] ABFS: Combine append+flush calls for blockblob & appendblob
URL: https://github.com/apache/hadoop/pull/1790#discussion_r371080998
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsOutputStream.java
##########
@@ -80,20 +82,29 @@ public AbfsOutputStream(
final String path,
final long position,
final int bufferSize,
- final boolean supportFlush) {
+ final boolean supportFlush,
+ final boolean disableOutputStreamFlush,
+ final boolean supportAppendWithFlush,
+ final boolean appendBlob) {
this.client = client;
this.path = path;
this.position = position;
this.closed = false;
+ this.disableOutputStreamFlush = disableOutputStreamFlush;
this.supportFlush = supportFlush;
+ this.supportAppendWithFlush = supportAppendWithFlush;
this.lastError = null;
this.lastFlushOffset = 0;
this.bufferSize = bufferSize;
this.buffer = byteBufferPool.getBuffer(false, bufferSize).array();
this.bufferIndex = 0;
this.writeOperations = new ConcurrentLinkedDeque<>();
- this.maxConcurrentRequestCount = 4 * Runtime.getRuntime().availableProcessors();
+ if (appendBlob) {
+ this.maxConcurrentRequestCount = 1;
Review comment:
AppendBlob for an HDFS file write should not allow parallel upload because that would lead to random writes in the file.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] ishaniahuja commented on a change in pull request #1790:
[HADOOP-16818] ABFS: Combine append+flush calls for blockblob & appendblob
Posted by GitBox <gi...@apache.org>.
ishaniahuja commented on a change in pull request #1790: [HADOOP-16818] ABFS: Combine append+flush calls for blockblob & appendblob
URL: https://github.com/apache/hadoop/pull/1790#discussion_r371681234
##########
File path: hadoop-tools/hadoop-azure/pom.xml
##########
@@ -194,7 +203,8 @@
<dependency>
<groupId>org.wildfly.openssl</groupId>
<artifactId>wildfly-openssl</artifactId>
- <scope>runtime</scope>
+ <version>1.0.9.Final</version>
Review comment:
I was seeing compilation failure with 1.0.7.Final
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] DadanielZ commented on a change in pull request #1790:
[HADOOP-16818] ABFS: Combine append+flush calls for blockblob & appendblob
Posted by GitBox <gi...@apache.org>.
DadanielZ commented on a change in pull request #1790: [HADOOP-16818] ABFS: Combine append+flush calls for blockblob & appendblob
URL: https://github.com/apache/hadoop/pull/1790#discussion_r390514693
##########
File path: hadoop-tools/hadoop-azure/pom.xml
##########
@@ -260,6 +260,7 @@
<dependency>
<groupId>org.mockito</groupId>
<artifactId>mockito-core</artifactId>
+ <version>3.3.0</version>
Review comment:
version number should not be defined here, please do not add this change unless you have to.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] bilaharith commented on a change in pull request #1790:
[HADOOP-16818] ABFS: Combine append+flush calls for blockblob & appendblob
Posted by GitBox <gi...@apache.org>.
bilaharith commented on a change in pull request #1790: [HADOOP-16818] ABFS: Combine append+flush calls for blockblob & appendblob
URL: https://github.com/apache/hadoop/pull/1790#discussion_r392937874
##########
File path: hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/services/TestAbfsOutputStream.java
##########
@@ -0,0 +1,355 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.azurebfs.services;
+import java.util.Arrays;
+import java.util.HashSet;
+import java.util.Random;
+import org.junit.Assert;
+import org.junit.Test;
+import org.mockito.ArgumentCaptor;
+import org.apache.hadoop.fs.azurebfs.AbfsConfiguration;
+import org.apache.hadoop.conf.Configuration;
+
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.when;
+import static org.mockito.Mockito.times;
+import static org.mockito.Mockito.verify;
+import static org.mockito.Mockito.verifyNoMoreInteractions;
+import static org.mockito.Mockito.*;
+
+public final class TestAbfsOutputStream {
+
+ private static int bufferSize = 4096;
+ private static int writeSize = 1000;
+ private static String path = "~/testpath";
+ private final String globalKey = "fs.azure.configuration";
+ private final String accountName1 = "account1";
+ private final String accountKey1 = globalKey + "." + accountName1;
+ private final String accountValue1 = "one";
+
+ @Test
+ /**
+ * The test verifies OutputStream shortwrite case(2000bytes write followed by flush, hflush, hsync) is making correct HTTP calls to the server
+ */
+ public void verifyShortWriteRequest() throws Exception {
+
+ AbfsClient client = mock(AbfsClient.class);
+ AbfsRestOperation op = mock(AbfsRestOperation.class);
+ AbfsConfiguration abfsConf;
+ final Configuration conf = new Configuration();
+ conf.set(accountKey1, accountValue1);
+ abfsConf = new AbfsConfiguration(conf, accountName1);
+ AbfsPerfTracker tracker = new AbfsPerfTracker("test", accountName1, abfsConf);
+ when(client.getAbfsPerfTracker()).thenReturn(tracker);
+ when(client.append(anyString(), anyLong(), any(byte[].class), anyInt(), anyInt(), anyBoolean(), anyBoolean())).thenReturn(op);
+
+ AbfsOutputStream out = new AbfsOutputStream(client, path, 0, bufferSize, true, false, true, false);
+ final byte[] b = new byte[writeSize];
+ new Random().nextBytes(b);
+ out.write(b);
+ out.hsync();
+ ArgumentCaptor<String> acString = ArgumentCaptor.forClass(String.class);
+ ArgumentCaptor<Long> acLong = ArgumentCaptor.forClass(Long.class);
+ ArgumentCaptor<Integer> acInt = ArgumentCaptor.forClass(Integer.class);
+ ArgumentCaptor<Boolean> acBool = ArgumentCaptor.forClass(Boolean.class);
+ ArgumentCaptor<byte[]> acByteArray = ArgumentCaptor.forClass(byte[].class);
+
+ final byte[] b1 = new byte[2*writeSize];
+ new Random().nextBytes(b1);
+ out.write(b1);
+ out.flush();
+ out.hflush();
+
+ out.hsync();
+
+ verify(client, times(2)).append(acString.capture(), acLong.capture(), acByteArray.capture(), acInt.capture(), acInt.capture(), acBool.capture(), acBool.capture());
Review comment:
Nit: Line is longer than 160 characters
Take care of the same in the entire file
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] ishaniahuja commented on a change in pull request #1790:
[HADOOP-16818] ABFS: Combine append+flush calls for blockblob & appendblob
Posted by GitBox <gi...@apache.org>.
ishaniahuja commented on a change in pull request #1790: [HADOOP-16818] ABFS: Combine append+flush calls for blockblob & appendblob
URL: https://github.com/apache/hadoop/pull/1790#discussion_r371080998
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsOutputStream.java
##########
@@ -80,20 +82,29 @@ public AbfsOutputStream(
final String path,
final long position,
final int bufferSize,
- final boolean supportFlush) {
+ final boolean supportFlush,
+ final boolean disableOutputStreamFlush,
+ final boolean supportAppendWithFlush,
+ final boolean appendBlob) {
this.client = client;
this.path = path;
this.position = position;
this.closed = false;
+ this.disableOutputStreamFlush = disableOutputStreamFlush;
this.supportFlush = supportFlush;
+ this.supportAppendWithFlush = supportAppendWithFlush;
this.lastError = null;
this.lastFlushOffset = 0;
this.bufferSize = bufferSize;
this.buffer = byteBufferPool.getBuffer(false, bufferSize).array();
this.bufferIndex = 0;
this.writeOperations = new ConcurrentLinkedDeque<>();
- this.maxConcurrentRequestCount = 4 * Runtime.getRuntime().availableProcessors();
+ if (appendBlob) {
+ this.maxConcurrentRequestCount = 1;
Review comment:
AppendBlob for an HDFS file write should not allow parallel upload because that would lead to random file writes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus removed a comment on issue #1790:
[HADOOP-16818] ABFS: Combine append+flush calls for blockblob & appendblob
Posted by GitBox <gi...@apache.org>.
hadoop-yetus removed a comment on issue #1790: [HADOOP-16818] ABFS: Combine append+flush calls for blockblob & appendblob
URL: https://github.com/apache/hadoop/pull/1790#issuecomment-578686362
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 0m 0s | Docker mode activated. |
| -1 :x: | patch | 0m 8s | https://github.com/apache/hadoop/pull/1790 does not apply to trunk. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. |
| Subsystem | Report/Notes |
|----------:|:-------------|
| GITHUB PR | https://github.com/apache/hadoop/pull/1790 |
| Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1790/1/console |
| versions | git=2.17.1 |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] bilaharith commented on a change in pull request #1790:
[HADOOP-16818] ABFS: Combine append+flush calls for blockblob & appendblob
Posted by GitBox <gi...@apache.org>.
bilaharith commented on a change in pull request #1790: [HADOOP-16818] ABFS: Combine append+flush calls for blockblob & appendblob
URL: https://github.com/apache/hadoop/pull/1790#discussion_r392936093
##########
File path: hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/services/TestAbfsOutputStream.java
##########
@@ -0,0 +1,355 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.azurebfs.services;
+import java.util.Arrays;
+import java.util.HashSet;
+import java.util.Random;
+import org.junit.Assert;
+import org.junit.Test;
+import org.mockito.ArgumentCaptor;
+import org.apache.hadoop.fs.azurebfs.AbfsConfiguration;
+import org.apache.hadoop.conf.Configuration;
+
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.when;
+import static org.mockito.Mockito.times;
+import static org.mockito.Mockito.verify;
+import static org.mockito.Mockito.verifyNoMoreInteractions;
Review comment:
Remove unused import
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] bilaharith commented on a change in pull request #1790: changes for combined append+flush calls for blockblob(traditionan) an…
Posted by GitBox <gi...@apache.org>.
bilaharith commented on a change in pull request #1790: changes for combined append+flush calls for blockblob(traditionan) an…
URL: https://github.com/apache/hadoop/pull/1790#discussion_r363222173
##########
File path: hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/services/TestAbfsOutputStream.java
##########
@@ -0,0 +1,268 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.azurebfs.services;
+import java.util.Arrays;
+import java.util.HashSet;
+import java.util.Random;
+import org.junit.Assert;
+import org.junit.Test;
+
+import org.mockito.ArgumentCaptor;
+import static org.mockito.Mockito.*;
+/**
+ * Test useragent of abfs client.
Review comment:
useragent?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] DadanielZ commented on a change in pull request #1790:
[HADOOP-16818] ABFS: Combine append+flush calls for blockblob & appendblob
Posted by GitBox <gi...@apache.org>.
DadanielZ commented on a change in pull request #1790: [HADOOP-16818] ABFS: Combine append+flush calls for blockblob & appendblob
URL: https://github.com/apache/hadoop/pull/1790#discussion_r370478672
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsOutputStream.java
##########
@@ -80,20 +82,29 @@ public AbfsOutputStream(
final String path,
final long position,
final int bufferSize,
- final boolean supportFlush) {
+ final boolean supportFlush,
+ final boolean disableOutputStreamFlush,
+ final boolean supportAppendWithFlush,
+ final boolean appendBlob) {
this.client = client;
this.path = path;
this.position = position;
this.closed = false;
+ this.disableOutputStreamFlush = disableOutputStreamFlush;
this.supportFlush = supportFlush;
+ this.supportAppendWithFlush = supportAppendWithFlush;
this.lastError = null;
this.lastFlushOffset = 0;
this.bufferSize = bufferSize;
this.buffer = byteBufferPool.getBuffer(false, bufferSize).array();
this.bufferIndex = 0;
this.writeOperations = new ConcurrentLinkedDeque<>();
- this.maxConcurrentRequestCount = 4 * Runtime.getRuntime().availableProcessors();
+ if (appendBlob) {
+ this.maxConcurrentRequestCount = 1;
Review comment:
could you elaborate here a little bit more why for appendblob the max concurrent requests is 1?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on issue #1790: [HADOOP-16818]
ABFS: Combine append+flush calls for blockblob & appendblob
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on issue #1790: [HADOOP-16818] ABFS: Combine append+flush calls for blockblob & appendblob
URL: https://github.com/apache/hadoop/pull/1790#issuecomment-579239147
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 42m 7s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | No case conflicting files found. |
| +0 :ok: | markdownlint | 0m 0s | markdownlint was not available. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 1 new or modified test files. |
||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 21m 32s | trunk passed |
| +1 :green_heart: | compile | 0m 31s | trunk passed |
| +1 :green_heart: | checkstyle | 0m 25s | trunk passed |
| +1 :green_heart: | mvnsite | 0m 36s | trunk passed |
| +1 :green_heart: | shadedclient | 16m 7s | branch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 28s | trunk passed |
| +0 :ok: | spotbugs | 1m 7s | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 1m 6s | trunk passed |
||| _ Patch Compile Tests _ |
| -1 :x: | mvninstall | 0m 17s | hadoop-azure in the patch failed. |
| -1 :x: | compile | 0m 13s | hadoop-azure in the patch failed. |
| -1 :x: | javac | 0m 13s | hadoop-azure in the patch failed. |
| -0 :warning: | checkstyle | 0m 13s | The patch fails to run checkstyle in hadoop-azure |
| -1 :x: | mvnsite | 0m 18s | hadoop-azure in the patch failed. |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | xml | 0m 1s | The patch has no ill-formed XML file. |
| +1 :green_heart: | shadedclient | 15m 8s | patch has no errors when building and testing our client artifacts. |
| -1 :x: | javadoc | 0m 16s | hadoop-azure in the patch failed. |
| -1 :x: | findbugs | 0m 15s | hadoop-azure in the patch failed. |
||| _ Other Tests _ |
| -1 :x: | unit | 0m 15s | hadoop-azure in the patch failed. |
| +1 :green_heart: | asflicense | 0m 28s | The patch does not generate ASF License warnings. |
| | | 101m 43s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | Client=19.03.5 Server=19.03.5 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1790/3/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/1790 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient xml findbugs checkstyle markdownlint |
| uname | Linux fb8d88d7c55a 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19 08:06:28 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | personality/hadoop.sh |
| git revision | trunk / 3f01c48 |
| Default Java | 1.8.0_232 |
| mvninstall | https://builds.apache.org/job/hadoop-multibranch/job/PR-1790/3/artifact/out/patch-mvninstall-hadoop-tools_hadoop-azure.txt |
| compile | https://builds.apache.org/job/hadoop-multibranch/job/PR-1790/3/artifact/out/patch-compile-hadoop-tools_hadoop-azure.txt |
| javac | https://builds.apache.org/job/hadoop-multibranch/job/PR-1790/3/artifact/out/patch-compile-hadoop-tools_hadoop-azure.txt |
| checkstyle | https://builds.apache.org/job/hadoop-multibranch/job/PR-1790/3/artifact/out//home/jenkins/jenkins-slave/workspace/hadoop-multibranch_PR-1790/out/maven-patch-checkstyle-hadoop-tools_hadoop-azure.txt |
| mvnsite | https://builds.apache.org/job/hadoop-multibranch/job/PR-1790/3/artifact/out/patch-mvnsite-hadoop-tools_hadoop-azure.txt |
| javadoc | https://builds.apache.org/job/hadoop-multibranch/job/PR-1790/3/artifact/out/patch-javadoc-hadoop-tools_hadoop-azure.txt |
| findbugs | https://builds.apache.org/job/hadoop-multibranch/job/PR-1790/3/artifact/out/patch-findbugs-hadoop-tools_hadoop-azure.txt |
| unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-1790/3/artifact/out/patch-unit-hadoop-tools_hadoop-azure.txt |
| Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1790/3/testReport/ |
| Max. process+thread count | 306 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1790/3/console |
| versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] bilaharith commented on a change in pull request #1790:
[HADOOP-16818] ABFS: Combine append+flush calls for blockblob & appendblob
Posted by GitBox <gi...@apache.org>.
bilaharith commented on a change in pull request #1790: [HADOOP-16818] ABFS: Combine append+flush calls for blockblob & appendblob
URL: https://github.com/apache/hadoop/pull/1790#discussion_r392936434
##########
File path: hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/services/TestAbfsOutputStream.java
##########
@@ -0,0 +1,355 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.azurebfs.services;
+import java.util.Arrays;
+import java.util.HashSet;
+import java.util.Random;
+import org.junit.Assert;
+import org.junit.Test;
+import org.mockito.ArgumentCaptor;
+import org.apache.hadoop.fs.azurebfs.AbfsConfiguration;
+import org.apache.hadoop.conf.Configuration;
+
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.when;
+import static org.mockito.Mockito.times;
+import static org.mockito.Mockito.verify;
+import static org.mockito.Mockito.verifyNoMoreInteractions;
+import static org.mockito.Mockito.*;
Review comment:
Nit: Using the '.*' form of import should be avoided
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on a change in pull request #1790:
[HADOOP-16818] ABFS: Combine append+flush calls for blockblob & appendblob
Posted by GitBox <gi...@apache.org>.
steveloughran commented on a change in pull request #1790: [HADOOP-16818] ABFS: Combine append+flush calls for blockblob & appendblob
URL: https://github.com/apache/hadoop/pull/1790#discussion_r386483373
##########
File path: hadoop-tools/hadoop-azure/src/site/markdown/abfs.md
##########
@@ -643,6 +643,10 @@ Consult the javadocs for `org.apache.hadoop.fs.azurebfs.constants.ConfigurationK
`org.apache.hadoop.fs.azurebfs.AbfsConfiguration` for the full list
of configuration options and their default values.
+### <a name="appendblobkeyconfigoptions"></a> Append Blob Directories Options
+### <a name="appendblobkeyconfigoptions"></a> Config `fs.azure.appendblob.key` provides
Review comment:
should be nested (i.e. ####); duplicate name will confuse link generation. Just cut the "a name" tag from the second line
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] snvijaya commented on a change in pull request #1790: changes for combined append+flush calls for blockblob(traditionan) an…
Posted by GitBox <gi...@apache.org>.
snvijaya commented on a change in pull request #1790: changes for combined append+flush calls for blockblob(traditionan) an…
URL: https://github.com/apache/hadoop/pull/1790#discussion_r365174656
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AzureBlobFileSystemStore.java
##########
@@ -344,40 +366,53 @@ public void deleteFilesystem() throws AzureBlobFileSystemException {
public OutputStream createFile(final Path path, final boolean overwrite, final FsPermission permission,
final FsPermission umask) throws AzureBlobFileSystemException {
- boolean isNamespaceEnabled = getIsNamespaceEnabled();
- LOG.debug("createFile filesystem: {} path: {} overwrite: {} permission: {} umask: {} isNamespaceEnabled: {}",
- client.getFileSystem(),
- path,
- overwrite,
- permission.toString(),
- umask.toString(),
- isNamespaceEnabled);
-
- client.createPath(AbfsHttpConstants.FORWARD_SLASH + getRelativePath(path), true, overwrite,
- isNamespaceEnabled ? getOctalNotation(permission) : null,
- isNamespaceEnabled ? getOctalNotation(umask) : null);
+ try (AbfsPerfInfo perfInfo = startTracking("createFile", "createPath")) {
+ boolean isNamespaceEnabled = getIsNamespaceEnabled();
+ LOG.debug("createFile filesystem: {} path: {} overwrite: {} permission: {} umask: {} isNamespaceEnabled: {}",
+ client.getFileSystem(),
+ path,
+ overwrite,
+ permission.toString(),
+ umask.toString(),
+ isNamespaceEnabled);
+
+ boolean appendBlob = false;
+ if (isAppendBlobKey(path.toString())) {
+ appendBlob = true;
+ }
- return new AbfsOutputStream(
- client,
- AbfsHttpConstants.FORWARD_SLASH + getRelativePath(path),
- 0,
- abfsConfiguration.getWriteBufferSize(),
- abfsConfiguration.isFlushEnabled());
+ client.createPath(AbfsHttpConstants.FORWARD_SLASH + getRelativePath(path), true, overwrite,
Review comment:
Is support for AppendBlob over queryparam "blobtype" also enabled with new Dec-2019 version ? If yes, add a config control for AppendBlob as well and have the config off, which can be turned on when the API version is upgraded in driver.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] ishaniahuja commented on a change in pull request #1790: changes for combined append+flush calls for blockblob(traditionan) an…
Posted by GitBox <gi...@apache.org>.
ishaniahuja commented on a change in pull request #1790: changes for combined append+flush calls for blockblob(traditionan) an…
URL: https://github.com/apache/hadoop/pull/1790#discussion_r365103591
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsOutputStream.java
##########
@@ -309,15 +316,23 @@ private synchronized void flushWrittenBytesToService(boolean isClose) throws IOE
throw new FileNotFoundException(ex.getMessage());
}
}
-
if (ex.getCause() instanceof AzureBlobFileSystemException) {
ex = (AzureBlobFileSystemException) ex.getCause();
}
lastError = new IOException(ex);
throw lastError;
}
}
- flushWrittenBytesToServiceInternal(position, false, isClose);
+ shrinkWriteOperationQueue();
Review comment:
as discussed offline, please look at new REST API. the code is same for both the appendBlob and BlockBlob. THe difference at the OutputStream Code is to use a single upload thread for appendBlob and multiple upload threads for block Blob. Its a lot of code rewrite, hence the full file read might be more useful.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] bilaharith commented on a change in pull request #1790:
[HADOOP-16818] ABFS: Combine append+flush calls for blockblob & appendblob
Posted by GitBox <gi...@apache.org>.
bilaharith commented on a change in pull request #1790: [HADOOP-16818] ABFS: Combine append+flush calls for blockblob & appendblob
URL: https://github.com/apache/hadoop/pull/1790#discussion_r392938996
##########
File path: hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/services/TestAbfsOutputStream.java
##########
@@ -0,0 +1,355 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.azurebfs.services;
+import java.util.Arrays;
+import java.util.HashSet;
+import java.util.Random;
+import org.junit.Assert;
+import org.junit.Test;
+import org.mockito.ArgumentCaptor;
+import org.apache.hadoop.fs.azurebfs.AbfsConfiguration;
+import org.apache.hadoop.conf.Configuration;
+
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.when;
+import static org.mockito.Mockito.times;
+import static org.mockito.Mockito.verify;
+import static org.mockito.Mockito.verifyNoMoreInteractions;
+import static org.mockito.Mockito.*;
+
+public final class TestAbfsOutputStream {
+
+ private static int bufferSize = 4096;
+ private static int writeSize = 1000;
+ private static String path = "~/testpath";
+ private final String globalKey = "fs.azure.configuration";
+ private final String accountName1 = "account1";
+ private final String accountKey1 = globalKey + "." + accountName1;
+ private final String accountValue1 = "one";
+
+ @Test
+ /**
+ * The test verifies OutputStream shortwrite case(2000bytes write followed by flush, hflush, hsync) is making correct HTTP calls to the server
+ */
+ public void verifyShortWriteRequest() throws Exception {
+
+ AbfsClient client = mock(AbfsClient.class);
+ AbfsRestOperation op = mock(AbfsRestOperation.class);
+ AbfsConfiguration abfsConf;
+ final Configuration conf = new Configuration();
+ conf.set(accountKey1, accountValue1);
+ abfsConf = new AbfsConfiguration(conf, accountName1);
+ AbfsPerfTracker tracker = new AbfsPerfTracker("test", accountName1, abfsConf);
+ when(client.getAbfsPerfTracker()).thenReturn(tracker);
+ when(client.append(anyString(), anyLong(), any(byte[].class), anyInt(), anyInt(), anyBoolean(), anyBoolean())).thenReturn(op);
+
+ AbfsOutputStream out = new AbfsOutputStream(client, path, 0, bufferSize, true, false, true, false);
+ final byte[] b = new byte[writeSize];
+ new Random().nextBytes(b);
+ out.write(b);
+ out.hsync();
+ ArgumentCaptor<String> acString = ArgumentCaptor.forClass(String.class);
+ ArgumentCaptor<Long> acLong = ArgumentCaptor.forClass(Long.class);
+ ArgumentCaptor<Integer> acInt = ArgumentCaptor.forClass(Integer.class);
+ ArgumentCaptor<Boolean> acBool = ArgumentCaptor.forClass(Boolean.class);
+ ArgumentCaptor<byte[]> acByteArray = ArgumentCaptor.forClass(byte[].class);
+
+ final byte[] b1 = new byte[2*writeSize];
+ new Random().nextBytes(b1);
+ out.write(b1);
+ out.flush();
+ out.hflush();
+
+ out.hsync();
+
+ verify(client, times(2)).append(acString.capture(), acLong.capture(), acByteArray.capture(), acInt.capture(), acInt.capture(), acBool.capture(), acBool.capture());
+ Assert.assertEquals(Arrays.asList(path, path) , acString.getAllValues());
+ Assert.assertEquals(Arrays.asList(Long.valueOf(0), Long.valueOf(writeSize)), acLong.getAllValues());
+ //flush=true, close=false, flush=true, close=false
+ Assert.assertEquals(Arrays.asList(true, false, true, false), acBool.getAllValues());
+ Assert.assertEquals(Arrays.asList(0,writeSize, 0, 2*writeSize), acInt.getAllValues());
Review comment:
Nit: ',' is not followed by whitespace. space before and after *
Take care of the same in the entire file
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on a change in pull request #1790:
[HADOOP-16818] ABFS: Combine append+flush calls for blockblob & appendblob
Posted by GitBox <gi...@apache.org>.
steveloughran commented on a change in pull request #1790: [HADOOP-16818] ABFS: Combine append+flush calls for blockblob & appendblob
URL: https://github.com/apache/hadoop/pull/1790#discussion_r386488149
##########
File path: hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/services/TestAbfsOutputStream.java
##########
@@ -0,0 +1,349 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.azurebfs.services;
+import org.apache.hadoop.fs.azurebfs.AbfsConfiguration;
+import org.apache.hadoop.conf.Configuration;
+import java.util.Arrays;
+import java.util.HashSet;
+import java.util.Random;
+import org.junit.Assert;
+import org.junit.Test;
+
+import org.mockito.ArgumentCaptor;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.when;
+import static org.mockito.Mockito.times;
+import static org.mockito.Mockito.verify;
+import static org.mockito.Mockito.verifyNoMoreInteractions;
+import static org.mockito.Mockito.*;
+/**
+ * Test useragent of abfs client.
Review comment:
check this
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] snvijaya commented on a change in pull request #1790:
[HADOOP-16818] ABFS: Combine append+flush calls for blockblob & appendblob
Posted by GitBox <gi...@apache.org>.
snvijaya commented on a change in pull request #1790: [HADOOP-16818] ABFS: Combine append+flush calls for blockblob & appendblob
URL: https://github.com/apache/hadoop/pull/1790#discussion_r378394347
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsClient.java
##########
@@ -254,6 +273,9 @@ public AbfsRestOperation createPath(final String path, final boolean isFile, fin
final AbfsUriQueryBuilder abfsUriQueryBuilder = createDefaultUriQueryBuilder();
abfsUriQueryBuilder.addQuery(QUERY_PARAM_RESOURCE, isFile ? FILE : DIRECTORY);
+ if (appendBlob) {
Review comment:
Test the change against an account in prod tenant which still doesnt have Dec-12 bits .
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] snvijaya commented on a change in pull request #1790:
[HADOOP-16818] ABFS: Combine append+flush calls for blockblob & appendblob
Posted by GitBox <gi...@apache.org>.
snvijaya commented on a change in pull request #1790: [HADOOP-16818] ABFS: Combine append+flush calls for blockblob & appendblob
URL: https://github.com/apache/hadoop/pull/1790#discussion_r378389461
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AbfsConfiguration.java
##########
@@ -158,6 +162,14 @@
DefaultValue = DEFAULT_DISABLE_OUTPUTSTREAM_FLUSH)
private boolean disableOutputStreamFlush;
+ @BooleanConfigurationValidatorAnnotation(ConfigurationKey = FS_AZURE_ENABLE_APPEND_WITH_FLUSH,
+ DefaultValue = DEFAULT_ENABLE_APPEND_WITH_FLUSH)
+ private boolean enableAppendWithFlush;
+
+ @BooleanConfigurationValidatorAnnotation(ConfigurationKey = FS_AZURE_DISABLE_OUTPUTSTREAM_FLUSH,
Review comment:
Are the trunk merge issues resolved ? This is an existing code in trunk but shown as new change here.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] snvijaya commented on a change in pull request #1790: changes for combined append+flush calls for blockblob(traditionan) an…
Posted by GitBox <gi...@apache.org>.
snvijaya commented on a change in pull request #1790: changes for combined append+flush calls for blockblob(traditionan) an…
URL: https://github.com/apache/hadoop/pull/1790#discussion_r365177009
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsClient.java
##########
@@ -55,7 +55,7 @@
public static final Logger LOG = LoggerFactory.getLogger(AbfsClient.class);
private final URL baseUrl;
private final SharedKeyCredentials sharedKeyCredentials;
- private final String xMsVersion = "2018-11-09";
+ private final String xMsVersion = "2019-12-12";
Review comment:
Server behaviour is controlled with this API version. So until server is ready, we can not bump up the api version support in driver.
This is not recommended to be a config control either, to prevent clients from changing api versions to any older one.
All dev work for new features is done with a feature config control that will turn off the feature. When the api version is incremented, the feature config will be turned on too.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] DadanielZ commented on a change in pull request #1790:
[HADOOP-16818] ABFS: Combine append+flush calls for blockblob & appendblob
Posted by GitBox <gi...@apache.org>.
DadanielZ commented on a change in pull request #1790: [HADOOP-16818] ABFS: Combine append+flush calls for blockblob & appendblob
URL: https://github.com/apache/hadoop/pull/1790#discussion_r377860636
##########
File path: hadoop-tools/hadoop-azure/pom.xml
##########
@@ -675,6 +679,7 @@
</execution>
</executions>
</plugin>
+-->
Review comment:
It looks like there is no need to update this pom file, if that is true can you reset the changes in this pom?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] DadanielZ removed a comment on issue #1790: [HADOOP-16818]
ABFS: Combine append+flush calls for blockblob & appendblob
Posted by GitBox <gi...@apache.org>.
DadanielZ removed a comment on issue #1790: [HADOOP-16818] ABFS: Combine append+flush calls for blockblob & appendblob
URL: https://github.com/apache/hadoop/pull/1790#issuecomment-597231945
LGTM +1, Hey @steveloughran do you still have the pending requests for changes? I will help to commit this one if it looks good to you now.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] bilaharith commented on a change in pull request #1790:
[HADOOP-16818] ABFS: Combine append+flush calls for blockblob & appendblob
Posted by GitBox <gi...@apache.org>.
bilaharith commented on a change in pull request #1790: [HADOOP-16818] ABFS: Combine append+flush calls for blockblob & appendblob
URL: https://github.com/apache/hadoop/pull/1790#discussion_r392655993
##########
File path: hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/services/TestAbfsOutputStream.java
##########
@@ -0,0 +1,355 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.azurebfs.services;
+import java.util.Arrays;
+import java.util.HashSet;
+import java.util.Random;
+import org.junit.Assert;
Review comment:
Order and group the non-static imports in the order:
java*
any non org.apache imports
org.apache imports
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] bilaharith commented on a change in pull request #1790: changes for combined append+flush calls for blockblob(traditionan) an…
Posted by GitBox <gi...@apache.org>.
bilaharith commented on a change in pull request #1790: changes for combined append+flush calls for blockblob(traditionan) an…
URL: https://github.com/apache/hadoop/pull/1790#discussion_r363222173
##########
File path: hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/services/TestAbfsOutputStream.java
##########
@@ -0,0 +1,268 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.azurebfs.services;
+import java.util.Arrays;
+import java.util.HashSet;
+import java.util.Random;
+import org.junit.Assert;
+import org.junit.Test;
+
+import org.mockito.ArgumentCaptor;
+import static org.mockito.Mockito.*;
+/**
+ * Test useragent of abfs client.
Review comment:
useragent?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] DadanielZ commented on a change in pull request #1790:
[HADOOP-16818] ABFS: Combine append+flush calls for blockblob & appendblob
Posted by GitBox <gi...@apache.org>.
DadanielZ commented on a change in pull request #1790: [HADOOP-16818] ABFS: Combine append+flush calls for blockblob & appendblob
URL: https://github.com/apache/hadoop/pull/1790#discussion_r377860636
##########
File path: hadoop-tools/hadoop-azure/pom.xml
##########
@@ -675,6 +679,7 @@
</execution>
</executions>
</plugin>
+-->
Review comment:
It looks like there is no need to update this pom file, if that is true can you reset the changes in this pom?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on issue #1790: [HADOOP-16818]
ABFS: Combine append+flush calls for blockblob & appendblob
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on issue #1790: [HADOOP-16818] ABFS: Combine append+flush calls for blockblob & appendblob
URL: https://github.com/apache/hadoop/pull/1790#issuecomment-594519889
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 0m 33s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +0 :ok: | markdownlint | 0m 0s | markdownlint was not available. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 2 new or modified test files. |
||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 19m 0s | trunk passed |
| +1 :green_heart: | compile | 0m 31s | trunk passed |
| +1 :green_heart: | checkstyle | 0m 24s | trunk passed |
| +1 :green_heart: | mvnsite | 0m 33s | trunk passed |
| +1 :green_heart: | shadedclient | 14m 55s | branch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 26s | trunk passed |
| +0 :ok: | spotbugs | 0m 51s | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 0m 50s | trunk passed |
||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 28s | the patch passed |
| +1 :green_heart: | compile | 0m 23s | the patch passed |
| +1 :green_heart: | javac | 0m 23s | the patch passed |
| -0 :warning: | checkstyle | 0m 17s | hadoop-tools/hadoop-azure: The patch generated 23 new + 5 unchanged - 0 fixed = 28 total (was 5) |
| +1 :green_heart: | mvnsite | 0m 28s | the patch passed |
| -1 :x: | whitespace | 0m 0s | The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply |
| +1 :green_heart: | xml | 0m 2s | The patch has no ill-formed XML file. |
| +1 :green_heart: | shadedclient | 13m 49s | patch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 22s | the patch passed |
| +1 :green_heart: | findbugs | 0m 53s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 1m 22s | hadoop-azure in the patch passed. |
| +1 :green_heart: | asflicense | 0m 31s | The patch does not generate ASF License warnings. |
| | | 57m 24s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | Client=19.03.7 Server=19.03.7 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1790/16/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/1790 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient xml findbugs checkstyle markdownlint |
| uname | Linux edf58e3bad4f 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | personality/hadoop.sh |
| git revision | trunk / bbd704b |
| Default Java | 1.8.0_242 |
| checkstyle | https://builds.apache.org/job/hadoop-multibranch/job/PR-1790/16/artifact/out/diff-checkstyle-hadoop-tools_hadoop-azure.txt |
| whitespace | https://builds.apache.org/job/hadoop-multibranch/job/PR-1790/16/artifact/out/whitespace-eol.txt |
| Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1790/16/testReport/ |
| Max. process+thread count | 455 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1790/16/console |
| versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] bilaharith commented on a change in pull request #1790:
[HADOOP-16818] ABFS: Combine append+flush calls for blockblob & appendblob
Posted by GitBox <gi...@apache.org>.
bilaharith commented on a change in pull request #1790: [HADOOP-16818] ABFS: Combine append+flush calls for blockblob & appendblob
URL: https://github.com/apache/hadoop/pull/1790#discussion_r392656044
##########
File path: hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/services/TestAbfsOutputStream.java
##########
@@ -0,0 +1,355 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.azurebfs.services;
+import java.util.Arrays;
+import java.util.HashSet;
+import java.util.Random;
+import org.junit.Assert;
+import org.junit.Test;
+import org.mockito.ArgumentCaptor;
+import org.apache.hadoop.fs.azurebfs.AbfsConfiguration;
+import org.apache.hadoop.conf.Configuration;
+
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.when;
+import static org.mockito.Mockito.times;
+import static org.mockito.Mockito.verify;
+import static org.mockito.Mockito.verifyNoMoreInteractions;
+import static org.mockito.Mockito.*;
+
+public final class TestAbfsOutputStream {
+
+ private static int bufferSize = 4096;
+ private static int writeSize = 1000;
+ private static String path = "~/testpath";
+ private final String globalKey = "fs.azure.configuration";
+ private final String accountName1 = "account1";
+ private final String accountKey1 = globalKey + "." + accountName1;
+ private final String accountValue1 = "one";
+
+ @Test
+ /**
+ * The test verifies OutputStream shortwrite case(2000bytes write followed by flush, hflush, hsync) is making correct HTTP calls to the server
Review comment:
Keep the comment above the @Test annotation for all the tests
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] bilaharith commented on a change in pull request #1790:
[HADOOP-16818] ABFS: Combine append+flush calls for blockblob & appendblob
Posted by GitBox <gi...@apache.org>.
bilaharith commented on a change in pull request #1790: [HADOOP-16818] ABFS: Combine append+flush calls for blockblob & appendblob
URL: https://github.com/apache/hadoop/pull/1790#discussion_r393499796
##########
File path: hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/services/TestAbfsOutputStream.java
##########
@@ -0,0 +1,355 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.azurebfs.services;
+import java.util.Arrays;
+import java.util.HashSet;
+import java.util.Random;
+import org.junit.Assert;
Review comment:
Import order is still not fixed
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] snvijaya commented on a change in pull request #1790: changes for combined append+flush calls for blockblob(traditionan) an…
Posted by GitBox <gi...@apache.org>.
snvijaya commented on a change in pull request #1790: changes for combined append+flush calls for blockblob(traditionan) an…
URL: https://github.com/apache/hadoop/pull/1790#discussion_r365179445
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AbfsConfiguration.java
##########
@@ -427,6 +443,14 @@ public boolean isFlushEnabled() {
return this.enableFlush;
}
+ public boolean isAppendWithFlushEnabled() {
+ return this.enableAppendWithFlush;
+ }
+
+ public boolean isOutputStreamFlushDisabled() {
Review comment:
Many changes from trunk merge is showing up as new updates even though not added as part of this PR.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on issue #1790: [HADOOP-16818]
ABFS: Combine append+flush calls for blockblob & appendblob
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on issue #1790: [HADOOP-16818] ABFS: Combine append+flush calls for blockblob & appendblob
URL: https://github.com/apache/hadoop/pull/1790#issuecomment-590257579
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 0m 38s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | No case conflicting files found. |
| +0 :ok: | markdownlint | 0m 0s | markdownlint was not available. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 1 new or modified test files. |
||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 19m 54s | trunk passed |
| +1 :green_heart: | compile | 0m 29s | trunk passed |
| +1 :green_heart: | checkstyle | 0m 21s | trunk passed |
| +1 :green_heart: | mvnsite | 0m 35s | trunk passed |
| +1 :green_heart: | shadedclient | 15m 21s | branch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 22s | trunk passed |
| +0 :ok: | spotbugs | 0m 53s | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 0m 52s | trunk passed |
||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 31s | the patch passed |
| +1 :green_heart: | compile | 0m 23s | the patch passed |
| +1 :green_heart: | javac | 0m 23s | the patch passed |
| -0 :warning: | checkstyle | 0m 16s | hadoop-tools/hadoop-azure: The patch generated 23 new + 5 unchanged - 0 fixed = 28 total (was 5) |
| +1 :green_heart: | mvnsite | 0m 28s | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | xml | 0m 1s | The patch has no ill-formed XML file. |
| +1 :green_heart: | shadedclient | 14m 1s | patch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 21s | the patch passed |
| -1 :x: | findbugs | 0m 58s | hadoop-tools/hadoop-azure generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) |
||| _ Other Tests _ |
| -1 :x: | unit | 1m 22s | hadoop-azure in the patch passed. |
| +1 :green_heart: | asflicense | 0m 28s | The patch does not generate ASF License warnings. |
| | | 58m 54s | |
| Reason | Tests |
|-------:|:------|
| FindBugs | module:hadoop-tools/hadoop-azure |
| | There is an apparent infinite recursive loop in org.apache.hadoop.fs.azurebfs.services.AbfsClient.createDefaultHeaders() At AbfsClient.java:recursive loop in org.apache.hadoop.fs.azurebfs.services.AbfsClient.createDefaultHeaders() At AbfsClient.java:[line 124] |
| Failed junit tests | hadoop.fs.azurebfs.services.TestAbfsOutputStream |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | Client=19.03.6 Server=19.03.6 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1790/10/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/1790 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient xml findbugs checkstyle markdownlint |
| uname | Linux cc0baa985310 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | personality/hadoop.sh |
| git revision | trunk / b5698e0 |
| Default Java | 1.8.0_242 |
| checkstyle | https://builds.apache.org/job/hadoop-multibranch/job/PR-1790/10/artifact/out/diff-checkstyle-hadoop-tools_hadoop-azure.txt |
| findbugs | https://builds.apache.org/job/hadoop-multibranch/job/PR-1790/10/artifact/out/new-findbugs-hadoop-tools_hadoop-azure.html |
| unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-1790/10/artifact/out/patch-unit-hadoop-tools_hadoop-azure.txt |
| Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1790/10/testReport/ |
| Max. process+thread count | 414 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1790/10/console |
| versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] bilaharith commented on a change in pull request #1790:
[HADOOP-16818] ABFS: Combine append+flush calls for blockblob & appendblob
Posted by GitBox <gi...@apache.org>.
bilaharith commented on a change in pull request #1790: [HADOOP-16818] ABFS: Combine append+flush calls for blockblob & appendblob
URL: https://github.com/apache/hadoop/pull/1790#discussion_r392936093
##########
File path: hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/services/TestAbfsOutputStream.java
##########
@@ -0,0 +1,355 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.azurebfs.services;
+import java.util.Arrays;
+import java.util.HashSet;
+import java.util.Random;
+import org.junit.Assert;
+import org.junit.Test;
+import org.mockito.ArgumentCaptor;
+import org.apache.hadoop.fs.azurebfs.AbfsConfiguration;
+import org.apache.hadoop.conf.Configuration;
+
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.when;
+import static org.mockito.Mockito.times;
+import static org.mockito.Mockito.verify;
+import static org.mockito.Mockito.verifyNoMoreInteractions;
Review comment:
Nit: Remove unused import
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on a change in pull request #1790:
[HADOOP-16818] ABFS: Combine append+flush calls for blockblob & appendblob
Posted by GitBox <gi...@apache.org>.
steveloughran commented on a change in pull request #1790: [HADOOP-16818] ABFS: Combine append+flush calls for blockblob & appendblob
URL: https://github.com/apache/hadoop/pull/1790#discussion_r386485772
##########
File path: hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/services/TestAbfsOutputStream.java
##########
@@ -0,0 +1,268 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.azurebfs.services;
+import java.util.Arrays;
+import java.util.HashSet;
+import java.util.Random;
+import org.junit.Assert;
+import org.junit.Test;
+
+import org.mockito.ArgumentCaptor;
+import static org.mockito.Mockito.*;
Review comment:
ideally, yes, though we are a bit more relaxed about static imports...
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] ishaniahuja opened a new pull request #1790: changes for combined append+flush calls for blockblob(traditionan) an…
Posted by GitBox <gi...@apache.org>.
ishaniahuja opened a new pull request #1790: changes for combined append+flush calls for blockblob(traditionan) an…
URL: https://github.com/apache/hadoop/pull/1790
…d support for appendblob
## NOTICE
Please create an issue in ASF JIRA before opening a pull request,
and you need to set the title of the pull request which starts with
the corresponding JIRA issue number. (e.g. HADOOP-XXXXX. Fix a typo in YYY.)
For more details, please see https://cwiki.apache.org/confluence/display/HADOOP/How+To+Contribute
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] snvijaya commented on a change in pull request #1790: changes for combined append+flush calls for blockblob(traditionan) an…
Posted by GitBox <gi...@apache.org>.
snvijaya commented on a change in pull request #1790: changes for combined append+flush calls for blockblob(traditionan) an…
URL: https://github.com/apache/hadoop/pull/1790#discussion_r365170305
##########
File path: hadoop-tools/hadoop-azure/pom.xml
##########
@@ -194,7 +203,8 @@
<dependency>
<groupId>org.wildfly.openssl</groupId>
<artifactId>wildfly-openssl</artifactId>
- <scope>runtime</scope>
+ <version>1.0.9.Final</version>
Review comment:
Wildfly jar version is picked from the parent project. Retain the original config.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] ishaniahuja commented on a change in pull request #1790:
[HADOOP-16818] ABFS: Combine append+flush calls for blockblob & appendblob
Posted by GitBox <gi...@apache.org>.
ishaniahuja commented on a change in pull request #1790: [HADOOP-16818] ABFS: Combine append+flush calls for blockblob & appendblob
URL: https://github.com/apache/hadoop/pull/1790#discussion_r379260268
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AzureBlobFileSystemStore.java
##########
@@ -344,40 +366,53 @@ public void deleteFilesystem() throws AzureBlobFileSystemException {
public OutputStream createFile(final Path path, final boolean overwrite, final FsPermission permission,
final FsPermission umask) throws AzureBlobFileSystemException {
- boolean isNamespaceEnabled = getIsNamespaceEnabled();
- LOG.debug("createFile filesystem: {} path: {} overwrite: {} permission: {} umask: {} isNamespaceEnabled: {}",
- client.getFileSystem(),
- path,
- overwrite,
- permission.toString(),
- umask.toString(),
- isNamespaceEnabled);
-
- client.createPath(AbfsHttpConstants.FORWARD_SLASH + getRelativePath(path), true, overwrite,
- isNamespaceEnabled ? getOctalNotation(permission) : null,
- isNamespaceEnabled ? getOctalNotation(umask) : null);
+ try (AbfsPerfInfo perfInfo = startTracking("createFile", "createPath")) {
+ boolean isNamespaceEnabled = getIsNamespaceEnabled();
+ LOG.debug("createFile filesystem: {} path: {} overwrite: {} permission: {} umask: {} isNamespaceEnabled: {}",
+ client.getFileSystem(),
+ path,
+ overwrite,
+ permission.toString(),
+ umask.toString(),
+ isNamespaceEnabled);
+
+ boolean appendBlob = false;
+ if (isAppendBlobKey(path.toString())) {
+ appendBlob = true;
+ }
- return new AbfsOutputStream(
- client,
- AbfsHttpConstants.FORWARD_SLASH + getRelativePath(path),
- 0,
- abfsConfiguration.getWriteBufferSize(),
- abfsConfiguration.isFlushEnabled());
+ client.createPath(AbfsHttpConstants.FORWARD_SLASH + getRelativePath(path), true, overwrite,
Review comment:
its added.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus removed a comment on issue #1790:
[HADOOP-16818] ABFS: Combine append+flush calls for blockblob & appendblob
Posted by GitBox <gi...@apache.org>.
hadoop-yetus removed a comment on issue #1790: [HADOOP-16818] ABFS: Combine append+flush calls for blockblob & appendblob
URL: https://github.com/apache/hadoop/pull/1790#issuecomment-590325640
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 1m 3s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | No case conflicting files found. |
| +0 :ok: | markdownlint | 0m 0s | markdownlint was not available. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 1 new or modified test files. |
||| _ trunk Compile Tests _ |
| -1 :x: | mvninstall | 27m 16s | root in trunk failed. |
| +1 :green_heart: | compile | 0m 32s | trunk passed |
| +1 :green_heart: | checkstyle | 0m 25s | trunk passed |
| +1 :green_heart: | mvnsite | 0m 36s | trunk passed |
| +1 :green_heart: | shadedclient | 14m 53s | branch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 27s | trunk passed |
| +0 :ok: | spotbugs | 0m 51s | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 0m 49s | trunk passed |
||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 33s | the patch passed |
| +1 :green_heart: | compile | 0m 24s | the patch passed |
| +1 :green_heart: | javac | 0m 24s | the patch passed |
| -0 :warning: | checkstyle | 0m 17s | hadoop-tools/hadoop-azure: The patch generated 23 new + 5 unchanged - 0 fixed = 28 total (was 5) |
| +1 :green_heart: | mvnsite | 0m 26s | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | xml | 0m 2s | The patch has no ill-formed XML file. |
| +1 :green_heart: | shadedclient | 13m 47s | patch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 23s | the patch passed |
| -1 :x: | findbugs | 0m 55s | hadoop-tools/hadoop-azure generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) |
||| _ Other Tests _ |
| -1 :x: | unit | 1m 22s | hadoop-azure in the patch passed. |
| +1 :green_heart: | asflicense | 0m 32s | The patch does not generate ASF License warnings. |
| | | 66m 42s | |
| Reason | Tests |
|-------:|:------|
| FindBugs | module:hadoop-tools/hadoop-azure |
| | There is an apparent infinite recursive loop in org.apache.hadoop.fs.azurebfs.services.AbfsClient.createDefaultHeaders() At AbfsClient.java:recursive loop in org.apache.hadoop.fs.azurebfs.services.AbfsClient.createDefaultHeaders() At AbfsClient.java:[line 124] |
| Failed junit tests | hadoop.fs.azurebfs.services.TestAbfsOutputStream |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | Client=19.03.6 Server=19.03.6 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1790/11/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/1790 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient xml findbugs checkstyle markdownlint |
| uname | Linux 2ae0881ac296 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | personality/hadoop.sh |
| git revision | trunk / b5698e0 |
| Default Java | 1.8.0_242 |
| mvninstall | https://builds.apache.org/job/hadoop-multibranch/job/PR-1790/11/artifact/out/branch-mvninstall-root.txt |
| checkstyle | https://builds.apache.org/job/hadoop-multibranch/job/PR-1790/11/artifact/out/diff-checkstyle-hadoop-tools_hadoop-azure.txt |
| findbugs | https://builds.apache.org/job/hadoop-multibranch/job/PR-1790/11/artifact/out/new-findbugs-hadoop-tools_hadoop-azure.html |
| unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-1790/11/artifact/out/patch-unit-hadoop-tools_hadoop-azure.txt |
| Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1790/11/testReport/ |
| Max. process+thread count | 467 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1790/11/console |
| versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on issue #1790: [HADOOP-16818]
ABFS: Combine append+flush calls for blockblob & appendblob
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on issue #1790: [HADOOP-16818] ABFS: Combine append+flush calls for blockblob & appendblob
URL: https://github.com/apache/hadoop/pull/1790#issuecomment-590325640
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 1m 3s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | No case conflicting files found. |
| +0 :ok: | markdownlint | 0m 0s | markdownlint was not available. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 1 new or modified test files. |
||| _ trunk Compile Tests _ |
| -1 :x: | mvninstall | 27m 16s | root in trunk failed. |
| +1 :green_heart: | compile | 0m 32s | trunk passed |
| +1 :green_heart: | checkstyle | 0m 25s | trunk passed |
| +1 :green_heart: | mvnsite | 0m 36s | trunk passed |
| +1 :green_heart: | shadedclient | 14m 53s | branch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 27s | trunk passed |
| +0 :ok: | spotbugs | 0m 51s | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 0m 49s | trunk passed |
||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 33s | the patch passed |
| +1 :green_heart: | compile | 0m 24s | the patch passed |
| +1 :green_heart: | javac | 0m 24s | the patch passed |
| -0 :warning: | checkstyle | 0m 17s | hadoop-tools/hadoop-azure: The patch generated 23 new + 5 unchanged - 0 fixed = 28 total (was 5) |
| +1 :green_heart: | mvnsite | 0m 26s | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | xml | 0m 2s | The patch has no ill-formed XML file. |
| +1 :green_heart: | shadedclient | 13m 47s | patch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 23s | the patch passed |
| -1 :x: | findbugs | 0m 55s | hadoop-tools/hadoop-azure generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) |
||| _ Other Tests _ |
| -1 :x: | unit | 1m 22s | hadoop-azure in the patch passed. |
| +1 :green_heart: | asflicense | 0m 32s | The patch does not generate ASF License warnings. |
| | | 66m 42s | |
| Reason | Tests |
|-------:|:------|
| FindBugs | module:hadoop-tools/hadoop-azure |
| | There is an apparent infinite recursive loop in org.apache.hadoop.fs.azurebfs.services.AbfsClient.createDefaultHeaders() At AbfsClient.java:recursive loop in org.apache.hadoop.fs.azurebfs.services.AbfsClient.createDefaultHeaders() At AbfsClient.java:[line 124] |
| Failed junit tests | hadoop.fs.azurebfs.services.TestAbfsOutputStream |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | Client=19.03.6 Server=19.03.6 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1790/11/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/1790 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient xml findbugs checkstyle markdownlint |
| uname | Linux 2ae0881ac296 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | personality/hadoop.sh |
| git revision | trunk / b5698e0 |
| Default Java | 1.8.0_242 |
| mvninstall | https://builds.apache.org/job/hadoop-multibranch/job/PR-1790/11/artifact/out/branch-mvninstall-root.txt |
| checkstyle | https://builds.apache.org/job/hadoop-multibranch/job/PR-1790/11/artifact/out/diff-checkstyle-hadoop-tools_hadoop-azure.txt |
| findbugs | https://builds.apache.org/job/hadoop-multibranch/job/PR-1790/11/artifact/out/new-findbugs-hadoop-tools_hadoop-azure.html |
| unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-1790/11/artifact/out/patch-unit-hadoop-tools_hadoop-azure.txt |
| Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1790/11/testReport/ |
| Max. process+thread count | 467 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1790/11/console |
| versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] ishaniahuja commented on issue #1790: [HADOOP-16818] ABFS:
Combine append+flush calls for blockblob & appendblob
Posted by GitBox <gi...@apache.org>.
ishaniahuja commented on issue #1790: [HADOOP-16818] ABFS: Combine append+flush calls for blockblob & appendblob
URL: https://github.com/apache/hadoop/pull/1790#issuecomment-594391697
mvn test results:
[out.txt](https://github.com/apache/hadoop/files/4286169/out.txt)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] bilaharith commented on a change in pull request #1790:
[HADOOP-16818] ABFS: Combine append+flush calls for blockblob & appendblob
Posted by GitBox <gi...@apache.org>.
bilaharith commented on a change in pull request #1790: [HADOOP-16818] ABFS: Combine append+flush calls for blockblob & appendblob
URL: https://github.com/apache/hadoop/pull/1790#discussion_r392654902
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsOutputStream.java
##########
@@ -55,6 +55,8 @@
private boolean closed;
private boolean supportFlush;
private boolean disableOutputStreamFlush;
+ private boolean supportAppendWithFlush;
+ private boolean appendBlob;
Review comment:
Shouldn't boolean be named with the prefix "is"
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus removed a comment on issue #1790:
[HADOOP-16818] ABFS: Combine append+flush calls for blockblob & appendblob
Posted by GitBox <gi...@apache.org>.
hadoop-yetus removed a comment on issue #1790: [HADOOP-16818] ABFS: Combine append+flush calls for blockblob & appendblob
URL: https://github.com/apache/hadoop/pull/1790#issuecomment-590474367
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 33m 8s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +0 :ok: | markdownlint | 0m 1s | markdownlint was not available. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 1 new or modified test files. |
||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 23m 46s | trunk passed |
| +1 :green_heart: | compile | 0m 32s | trunk passed |
| +1 :green_heart: | checkstyle | 0m 23s | trunk passed |
| +1 :green_heart: | mvnsite | 0m 37s | trunk passed |
| +1 :green_heart: | shadedclient | 17m 5s | branch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 23s | trunk passed |
| +0 :ok: | spotbugs | 1m 0s | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 0m 58s | trunk passed |
||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 33s | the patch passed |
| +1 :green_heart: | compile | 0m 23s | the patch passed |
| +1 :green_heart: | javac | 0m 23s | the patch passed |
| -0 :warning: | checkstyle | 0m 17s | hadoop-tools/hadoop-azure: The patch generated 23 new + 5 unchanged - 0 fixed = 28 total (was 5) |
| +1 :green_heart: | mvnsite | 0m 29s | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | xml | 0m 1s | The patch has no ill-formed XML file. |
| +1 :green_heart: | shadedclient | 16m 23s | patch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 20s | the patch passed |
| -1 :x: | findbugs | 0m 57s | hadoop-tools/hadoop-azure generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 1m 28s | hadoop-azure in the patch passed. |
| +1 :green_heart: | asflicense | 0m 28s | The patch does not generate ASF License warnings. |
| | | 99m 38s | |
| Reason | Tests |
|-------:|:------|
| FindBugs | module:hadoop-tools/hadoop-azure |
| | There is an apparent infinite recursive loop in org.apache.hadoop.fs.azurebfs.services.AbfsClient.createDefaultHeaders() At AbfsClient.java:recursive loop in org.apache.hadoop.fs.azurebfs.services.AbfsClient.createDefaultHeaders() At AbfsClient.java:[line 124] |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | Client=19.03.6 Server=19.03.6 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1790/12/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/1790 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient xml findbugs checkstyle markdownlint |
| uname | Linux f37996238dfa 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19 08:06:28 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | personality/hadoop.sh |
| git revision | trunk / 42dfd27 |
| Default Java | 1.8.0_242 |
| checkstyle | https://builds.apache.org/job/hadoop-multibranch/job/PR-1790/12/artifact/out/diff-checkstyle-hadoop-tools_hadoop-azure.txt |
| findbugs | https://builds.apache.org/job/hadoop-multibranch/job/PR-1790/12/artifact/out/new-findbugs-hadoop-tools_hadoop-azure.html |
| Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1790/12/testReport/ |
| Max. process+thread count | 311 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1790/12/console |
| versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on issue #1790: [HADOOP-16818]
ABFS: Combine append+flush calls for blockblob & appendblob
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on issue #1790: [HADOOP-16818] ABFS: Combine append+flush calls for blockblob & appendblob
URL: https://github.com/apache/hadoop/pull/1790#issuecomment-590574917
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 1m 53s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +0 :ok: | markdownlint | 0m 0s | markdownlint was not available. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 1 new or modified test files. |
||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 22m 28s | trunk passed |
| +1 :green_heart: | compile | 0m 28s | trunk passed |
| +1 :green_heart: | checkstyle | 0m 20s | trunk passed |
| +1 :green_heart: | mvnsite | 0m 31s | trunk passed |
| +1 :green_heart: | shadedclient | 16m 10s | branch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 22s | trunk passed |
| +0 :ok: | spotbugs | 0m 53s | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 0m 51s | trunk passed |
||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 28s | the patch passed |
| +1 :green_heart: | compile | 0m 22s | the patch passed |
| +1 :green_heart: | javac | 0m 22s | the patch passed |
| -0 :warning: | checkstyle | 0m 15s | hadoop-tools/hadoop-azure: The patch generated 23 new + 5 unchanged - 0 fixed = 28 total (was 5) |
| +1 :green_heart: | mvnsite | 0m 26s | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | xml | 0m 1s | The patch has no ill-formed XML file. |
| +1 :green_heart: | shadedclient | 15m 17s | patch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 20s | the patch passed |
| +1 :green_heart: | findbugs | 0m 53s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 1m 25s | hadoop-azure in the patch passed. |
| +1 :green_heart: | asflicense | 0m 27s | The patch does not generate ASF License warnings. |
| | | 64m 34s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | Client=19.03.6 Server=19.03.6 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1790/13/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/1790 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient xml findbugs checkstyle markdownlint |
| uname | Linux f3041d945d40 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19 08:06:28 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | personality/hadoop.sh |
| git revision | trunk / 72fa9c7 |
| Default Java | 1.8.0_242 |
| checkstyle | https://builds.apache.org/job/hadoop-multibranch/job/PR-1790/13/artifact/out/diff-checkstyle-hadoop-tools_hadoop-azure.txt |
| Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1790/13/testReport/ |
| Max. process+thread count | 308 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1790/13/console |
| versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] ishaniahuja commented on a change in pull request #1790: changes for combined append+flush calls for blockblob(traditionan) an…
Posted by GitBox <gi...@apache.org>.
ishaniahuja commented on a change in pull request #1790: changes for combined append+flush calls for blockblob(traditionan) an…
URL: https://github.com/apache/hadoop/pull/1790#discussion_r365109068
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AbfsConfiguration.java
##########
@@ -130,6 +130,10 @@
DefaultValue = DEFAULT_FS_AZURE_ATOMIC_RENAME_DIRECTORIES)
private String azureAtomicDirs;
+ @StringConfigurationValidatorAnnotation(ConfigurationKey = FS_AZURE_APPEND_BLOB_KEY,
+ DefaultValue = DEFAULT_FS_AZURE_APPEND_BLOB_DIRECTORIES)
+ private String azureAppendBlobDirs;
Review comment:
what is the procedure for the documentation edit? who are the owners of this documentation edit? do I need to email someone for this?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] bilaharith commented on a change in pull request #1790:
[HADOOP-16818] ABFS: Combine append+flush calls for blockblob & appendblob
Posted by GitBox <gi...@apache.org>.
bilaharith commented on a change in pull request #1790: [HADOOP-16818] ABFS: Combine append+flush calls for blockblob & appendblob
URL: https://github.com/apache/hadoop/pull/1790#discussion_r392937467
##########
File path: hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/services/TestAbfsOutputStream.java
##########
@@ -0,0 +1,355 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.azurebfs.services;
+import java.util.Arrays;
+import java.util.HashSet;
+import java.util.Random;
+import org.junit.Assert;
+import org.junit.Test;
+import org.mockito.ArgumentCaptor;
+import org.apache.hadoop.fs.azurebfs.AbfsConfiguration;
+import org.apache.hadoop.conf.Configuration;
+
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.when;
+import static org.mockito.Mockito.times;
+import static org.mockito.Mockito.verify;
+import static org.mockito.Mockito.verifyNoMoreInteractions;
+import static org.mockito.Mockito.*;
+
+public final class TestAbfsOutputStream {
+
+ private static int bufferSize = 4096;
Review comment:
Nit: These 2 can also be made constants
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] snvijaya commented on a change in pull request #1790: changes for combined append+flush calls for blockblob(traditionan) an…
Posted by GitBox <gi...@apache.org>.
snvijaya commented on a change in pull request #1790: changes for combined append+flush calls for blockblob(traditionan) an…
URL: https://github.com/apache/hadoop/pull/1790#discussion_r365177280
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsClient.java
##########
@@ -278,7 +282,8 @@ public AbfsRestOperation renamePath(final String source, final String destinatio
}
public AbfsRestOperation append(final String path, final long position, final byte[] buffer, final int offset,
- final int length) throws AzureBlobFileSystemException {
+ final int length, boolean flush, boolean isClose)
Review comment:
New iteration has the caller updated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] snvijaya commented on a change in pull request #1790: changes for combined append+flush calls for blockblob(traditionan) an…
Posted by GitBox <gi...@apache.org>.
snvijaya commented on a change in pull request #1790: changes for combined append+flush calls for blockblob(traditionan) an…
URL: https://github.com/apache/hadoop/pull/1790#discussion_r365171120
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AbfsConfiguration.java
##########
@@ -130,6 +130,10 @@
DefaultValue = DEFAULT_FS_AZURE_ATOMIC_RENAME_DIRECTORIES)
private String azureAtomicDirs;
+ @StringConfigurationValidatorAnnotation(ConfigurationKey = FS_AZURE_APPEND_BLOB_KEY,
+ DefaultValue = DEFAULT_FS_AZURE_APPEND_BLOB_DIRECTORIES)
+ private String azureAppendBlobDirs;
Review comment:
Dev introducing new config needs to add the documentation for it. You need to edit the abfs.md file mentioned in above comment. Follow format already present in it.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] bilaharith commented on a change in pull request #1790:
[HADOOP-16818] ABFS: Combine append+flush calls for blockblob & appendblob
Posted by GitBox <gi...@apache.org>.
bilaharith commented on a change in pull request #1790: [HADOOP-16818] ABFS: Combine append+flush calls for blockblob & appendblob
URL: https://github.com/apache/hadoop/pull/1790#discussion_r393476488
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/constants/ConfigurationKeys.java
##########
@@ -60,6 +61,15 @@
* documentation does not have such expectations of data being persisted.
* Default value of this config is true. **/
public static final String FS_AZURE_DISABLE_OUTPUTSTREAM_FLUSH = "fs.azure.disable.outputstream.flush";
+ /** Provides a config control to enable OutputStream AppendWithFlush API
+ * operations in AbfsOutputStream.
+ * Default value of this config is true. **/
+ public static final String FS_AZURE_ENABLE_APPEND_WITH_FLUSH = "fs.azure.enable.appendwithflush";
+ /** Provides a config control to disable or enable OutputStream Flush API
Review comment:
@snvijaya Doesn't this look misplaced?
@ishaniahuja Check with @snvijaya and then resolve this one
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus removed a comment on issue #1790:
[HADOOP-16818] ABFS: Combine append+flush calls for blockblob & appendblob
Posted by GitBox <gi...@apache.org>.
hadoop-yetus removed a comment on issue #1790: [HADOOP-16818] ABFS: Combine append+flush calls for blockblob & appendblob
URL: https://github.com/apache/hadoop/pull/1790#issuecomment-579239147
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 42m 7s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | No case conflicting files found. |
| +0 :ok: | markdownlint | 0m 0s | markdownlint was not available. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 1 new or modified test files. |
||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 21m 32s | trunk passed |
| +1 :green_heart: | compile | 0m 31s | trunk passed |
| +1 :green_heart: | checkstyle | 0m 25s | trunk passed |
| +1 :green_heart: | mvnsite | 0m 36s | trunk passed |
| +1 :green_heart: | shadedclient | 16m 7s | branch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 28s | trunk passed |
| +0 :ok: | spotbugs | 1m 7s | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 1m 6s | trunk passed |
||| _ Patch Compile Tests _ |
| -1 :x: | mvninstall | 0m 17s | hadoop-azure in the patch failed. |
| -1 :x: | compile | 0m 13s | hadoop-azure in the patch failed. |
| -1 :x: | javac | 0m 13s | hadoop-azure in the patch failed. |
| -0 :warning: | checkstyle | 0m 13s | The patch fails to run checkstyle in hadoop-azure |
| -1 :x: | mvnsite | 0m 18s | hadoop-azure in the patch failed. |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | xml | 0m 1s | The patch has no ill-formed XML file. |
| +1 :green_heart: | shadedclient | 15m 8s | patch has no errors when building and testing our client artifacts. |
| -1 :x: | javadoc | 0m 16s | hadoop-azure in the patch failed. |
| -1 :x: | findbugs | 0m 15s | hadoop-azure in the patch failed. |
||| _ Other Tests _ |
| -1 :x: | unit | 0m 15s | hadoop-azure in the patch failed. |
| +1 :green_heart: | asflicense | 0m 28s | The patch does not generate ASF License warnings. |
| | | 101m 43s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | Client=19.03.5 Server=19.03.5 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1790/3/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/1790 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient xml findbugs checkstyle markdownlint |
| uname | Linux fb8d88d7c55a 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19 08:06:28 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | personality/hadoop.sh |
| git revision | trunk / 3f01c48 |
| Default Java | 1.8.0_232 |
| mvninstall | https://builds.apache.org/job/hadoop-multibranch/job/PR-1790/3/artifact/out/patch-mvninstall-hadoop-tools_hadoop-azure.txt |
| compile | https://builds.apache.org/job/hadoop-multibranch/job/PR-1790/3/artifact/out/patch-compile-hadoop-tools_hadoop-azure.txt |
| javac | https://builds.apache.org/job/hadoop-multibranch/job/PR-1790/3/artifact/out/patch-compile-hadoop-tools_hadoop-azure.txt |
| checkstyle | https://builds.apache.org/job/hadoop-multibranch/job/PR-1790/3/artifact/out//home/jenkins/jenkins-slave/workspace/hadoop-multibranch_PR-1790/out/maven-patch-checkstyle-hadoop-tools_hadoop-azure.txt |
| mvnsite | https://builds.apache.org/job/hadoop-multibranch/job/PR-1790/3/artifact/out/patch-mvnsite-hadoop-tools_hadoop-azure.txt |
| javadoc | https://builds.apache.org/job/hadoop-multibranch/job/PR-1790/3/artifact/out/patch-javadoc-hadoop-tools_hadoop-azure.txt |
| findbugs | https://builds.apache.org/job/hadoop-multibranch/job/PR-1790/3/artifact/out/patch-findbugs-hadoop-tools_hadoop-azure.txt |
| unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-1790/3/artifact/out/patch-unit-hadoop-tools_hadoop-azure.txt |
| Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1790/3/testReport/ |
| Max. process+thread count | 306 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1790/3/console |
| versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] bilaharith commented on a change in pull request #1790: changes for combined append+flush calls for blockblob(traditionan) an…
Posted by GitBox <gi...@apache.org>.
bilaharith commented on a change in pull request #1790: changes for combined append+flush calls for blockblob(traditionan) an…
URL: https://github.com/apache/hadoop/pull/1790#discussion_r363225156
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AzureBlobFileSystemStore.java
##########
@@ -164,6 +169,23 @@ public AzureBlobFileSystemStore(URI uri, boolean isSecureScheme, Configuration c
boolean useHttps = (usingOauth || abfsConfiguration.isHttpsAlwaysUsed()) ? true : isSecureScheme;
initializeClient(uri, fileSystemName, accountName, useHttps);
this.identityTransformer = new IdentityTransformer(abfsConfiguration.getRawConfiguration());
+
+ // Extract the directories that should contain page blobs
Review comment:
Shall we have these operations as a separate private method and call that from this constructor. This constructor is becoming lengthy.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] ishaniahuja commented on a change in pull request #1790: changes for combined append+flush calls for blockblob(traditionan) an…
Posted by GitBox <gi...@apache.org>.
ishaniahuja commented on a change in pull request #1790: changes for combined append+flush calls for blockblob(traditionan) an…
URL: https://github.com/apache/hadoop/pull/1790#discussion_r365104492
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsOutputStream.java
##########
@@ -257,17 +265,16 @@ public synchronized void close() throws IOException {
private synchronized void flushInternal(boolean isClose) throws IOException {
maybeThrowLastError();
- writeCurrentBufferToService();
- flushWrittenBytesToService(isClose);
+ writeAndFlushWrittenBytesToService(isClose);
}
private synchronized void flushInternalAsync() throws IOException {
maybeThrowLastError();
- writeCurrentBufferToService();
+ writeCurrentBufferToService(true, false);
flushWrittenBytesToServiceAsync();
}
- private synchronized void writeCurrentBufferToService() throws IOException {
+ private synchronized void writeCurrentBufferToService(final boolean flush, final boolean isClose) throws IOException {
Review comment:
it does not call flush. but passes flush and close flag in the append method in AbfsClient.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] snvijaya commented on a change in pull request #1790: changes for combined append+flush calls for blockblob(traditionan) an…
Posted by GitBox <gi...@apache.org>.
snvijaya commented on a change in pull request #1790: changes for combined append+flush calls for blockblob(traditionan) an…
URL: https://github.com/apache/hadoop/pull/1790#discussion_r365177842
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsOutputStream.java
##########
@@ -257,17 +265,16 @@ public synchronized void close() throws IOException {
private synchronized void flushInternal(boolean isClose) throws IOException {
maybeThrowLastError();
- writeCurrentBufferToService();
- flushWrittenBytesToService(isClose);
+ writeAndFlushWrittenBytesToService(isClose);
}
private synchronized void flushInternalAsync() throws IOException {
maybeThrowLastError();
- writeCurrentBufferToService();
+ writeCurrentBufferToService(true, false);
flushWrittenBytesToServiceAsync();
}
- private synchronized void writeCurrentBufferToService() throws IOException {
+ private synchronized void writeCurrentBufferToService(final boolean flush, final boolean isClose) throws IOException {
Review comment:
Param passing was missing earlier iteration. Can see in new update.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on issue #1790: [HADOOP-16818]
ABFS: Combine append+flush calls for blockblob & appendblob
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on issue #1790: [HADOOP-16818] ABFS: Combine append+flush calls for blockblob & appendblob
URL: https://github.com/apache/hadoop/pull/1790#issuecomment-585720440
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 0m 33s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | No case conflicting files found. |
| +0 :ok: | markdownlint | 0m 0s | markdownlint was not available. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 1 new or modified test files. |
||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 18m 56s | trunk passed |
| +1 :green_heart: | compile | 0m 31s | trunk passed |
| +1 :green_heart: | checkstyle | 0m 25s | trunk passed |
| +1 :green_heart: | mvnsite | 0m 34s | trunk passed |
| +1 :green_heart: | shadedclient | 14m 46s | branch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 25s | trunk passed |
| +0 :ok: | spotbugs | 0m 51s | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 0m 49s | trunk passed |
||| _ Patch Compile Tests _ |
| -1 :x: | mvninstall | 0m 14s | hadoop-azure in the patch failed. |
| -1 :x: | compile | 0m 15s | hadoop-azure in the patch failed. |
| -1 :x: | javac | 0m 15s | hadoop-azure in the patch failed. |
| -0 :warning: | checkstyle | 0m 12s | The patch fails to run checkstyle in hadoop-azure |
| -1 :x: | mvnsite | 0m 16s | hadoop-azure in the patch failed. |
| -1 :x: | whitespace | 0m 0s | The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply |
| +1 :green_heart: | xml | 0m 2s | The patch has no ill-formed XML file. |
| +1 :green_heart: | shadedclient | 13m 51s | patch has no errors when building and testing our client artifacts. |
| -1 :x: | javadoc | 0m 19s | hadoop-azure in the patch failed. |
| -1 :x: | findbugs | 0m 18s | hadoop-azure in the patch failed. |
||| _ Other Tests _ |
| -1 :x: | unit | 0m 18s | hadoop-azure in the patch failed. |
| +1 :green_heart: | asflicense | 0m 32s | The patch does not generate ASF License warnings. |
| | | 54m 48s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | Client=19.03.6 Server=19.03.6 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1790/7/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/1790 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient xml findbugs checkstyle markdownlint |
| uname | Linux 2d8ef296c6b8 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | personality/hadoop.sh |
| git revision | trunk / fe7d67a |
| Default Java | 1.8.0_242 |
| mvninstall | https://builds.apache.org/job/hadoop-multibranch/job/PR-1790/7/artifact/out/patch-mvninstall-hadoop-tools_hadoop-azure.txt |
| compile | https://builds.apache.org/job/hadoop-multibranch/job/PR-1790/7/artifact/out/patch-compile-hadoop-tools_hadoop-azure.txt |
| javac | https://builds.apache.org/job/hadoop-multibranch/job/PR-1790/7/artifact/out/patch-compile-hadoop-tools_hadoop-azure.txt |
| checkstyle | https://builds.apache.org/job/hadoop-multibranch/job/PR-1790/7/artifact/out//home/jenkins/jenkins-slave/workspace/hadoop-multibranch_PR-1790/out/maven-patch-checkstyle-hadoop-tools_hadoop-azure.txt |
| mvnsite | https://builds.apache.org/job/hadoop-multibranch/job/PR-1790/7/artifact/out/patch-mvnsite-hadoop-tools_hadoop-azure.txt |
| whitespace | https://builds.apache.org/job/hadoop-multibranch/job/PR-1790/7/artifact/out/whitespace-eol.txt |
| javadoc | https://builds.apache.org/job/hadoop-multibranch/job/PR-1790/7/artifact/out/patch-javadoc-hadoop-tools_hadoop-azure.txt |
| findbugs | https://builds.apache.org/job/hadoop-multibranch/job/PR-1790/7/artifact/out/patch-findbugs-hadoop-tools_hadoop-azure.txt |
| unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-1790/7/artifact/out/patch-unit-hadoop-tools_hadoop-azure.txt |
| Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1790/7/testReport/ |
| Max. process+thread count | 413 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1790/7/console |
| versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] bilaharith commented on a change in pull request #1790:
[HADOOP-16818] ABFS: Combine append+flush calls for blockblob & appendblob
Posted by GitBox <gi...@apache.org>.
bilaharith commented on a change in pull request #1790: [HADOOP-16818] ABFS: Combine append+flush calls for blockblob & appendblob
URL: https://github.com/apache/hadoop/pull/1790#discussion_r392938515
##########
File path: hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/services/TestAbfsOutputStream.java
##########
@@ -0,0 +1,355 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.azurebfs.services;
+import java.util.Arrays;
+import java.util.HashSet;
+import java.util.Random;
+import org.junit.Assert;
+import org.junit.Test;
+import org.mockito.ArgumentCaptor;
+import org.apache.hadoop.fs.azurebfs.AbfsConfiguration;
+import org.apache.hadoop.conf.Configuration;
+
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.when;
+import static org.mockito.Mockito.times;
+import static org.mockito.Mockito.verify;
+import static org.mockito.Mockito.verifyNoMoreInteractions;
+import static org.mockito.Mockito.*;
+
+public final class TestAbfsOutputStream {
+
+ private static int bufferSize = 4096;
+ private static int writeSize = 1000;
+ private static String path = "~/testpath";
+ private final String globalKey = "fs.azure.configuration";
+ private final String accountName1 = "account1";
+ private final String accountKey1 = globalKey + "." + accountName1;
+ private final String accountValue1 = "one";
+
+ @Test
+ /**
+ * The test verifies OutputStream shortwrite case(2000bytes write followed by flush, hflush, hsync) is making correct HTTP calls to the server
+ */
+ public void verifyShortWriteRequest() throws Exception {
+
+ AbfsClient client = mock(AbfsClient.class);
+ AbfsRestOperation op = mock(AbfsRestOperation.class);
+ AbfsConfiguration abfsConf;
+ final Configuration conf = new Configuration();
+ conf.set(accountKey1, accountValue1);
+ abfsConf = new AbfsConfiguration(conf, accountName1);
+ AbfsPerfTracker tracker = new AbfsPerfTracker("test", accountName1, abfsConf);
+ when(client.getAbfsPerfTracker()).thenReturn(tracker);
+ when(client.append(anyString(), anyLong(), any(byte[].class), anyInt(), anyInt(), anyBoolean(), anyBoolean())).thenReturn(op);
+
+ AbfsOutputStream out = new AbfsOutputStream(client, path, 0, bufferSize, true, false, true, false);
+ final byte[] b = new byte[writeSize];
+ new Random().nextBytes(b);
+ out.write(b);
+ out.hsync();
+ ArgumentCaptor<String> acString = ArgumentCaptor.forClass(String.class);
+ ArgumentCaptor<Long> acLong = ArgumentCaptor.forClass(Long.class);
+ ArgumentCaptor<Integer> acInt = ArgumentCaptor.forClass(Integer.class);
+ ArgumentCaptor<Boolean> acBool = ArgumentCaptor.forClass(Boolean.class);
+ ArgumentCaptor<byte[]> acByteArray = ArgumentCaptor.forClass(byte[].class);
+
+ final byte[] b1 = new byte[2*writeSize];
+ new Random().nextBytes(b1);
+ out.write(b1);
+ out.flush();
+ out.hflush();
+
+ out.hsync();
+
+ verify(client, times(2)).append(acString.capture(), acLong.capture(), acByteArray.capture(), acInt.capture(), acInt.capture(), acBool.capture(), acBool.capture());
+ Assert.assertEquals(Arrays.asList(path, path) , acString.getAllValues());
Review comment:
Nit: ',' is preceded with whitespace
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] bilaharith commented on a change in pull request #1790:
[HADOOP-16818] ABFS: Combine append+flush calls for blockblob & appendblob
Posted by GitBox <gi...@apache.org>.
bilaharith commented on a change in pull request #1790: [HADOOP-16818] ABFS: Combine append+flush calls for blockblob & appendblob
URL: https://github.com/apache/hadoop/pull/1790#discussion_r392656011
##########
File path: hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/services/TestAbfsOutputStream.java
##########
@@ -0,0 +1,355 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.azurebfs.services;
+import java.util.Arrays;
+import java.util.HashSet;
+import java.util.Random;
+import org.junit.Assert;
+import org.junit.Test;
+import org.mockito.ArgumentCaptor;
+import org.apache.hadoop.fs.azurebfs.AbfsConfiguration;
+import org.apache.hadoop.conf.Configuration;
+
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.when;
Review comment:
Same for static imports
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] snvijaya commented on a change in pull request #1790: changes for combined append+flush calls for blockblob(traditionan) an…
Posted by GitBox <gi...@apache.org>.
snvijaya commented on a change in pull request #1790: changes for combined append+flush calls for blockblob(traditionan) an…
URL: https://github.com/apache/hadoop/pull/1790#discussion_r365170489
##########
File path: hadoop-tools/hadoop-azure/pom.xml
##########
@@ -291,6 +301,7 @@
</activation>
<build>
<plugins>
+<!--
Review comment:
Undo unintended config changes
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] bilaharith commented on a change in pull request #1790: changes for combined append+flush calls for blockblob(traditionan) an…
Posted by GitBox <gi...@apache.org>.
bilaharith commented on a change in pull request #1790: changes for combined append+flush calls for blockblob(traditionan) an…
URL: https://github.com/apache/hadoop/pull/1790#discussion_r363219914
##########
File path: hadoop-tools/hadoop-azure/pom.xml
##########
@@ -143,6 +143,14 @@
<!-- see hadoop-project/pom.xml for version number declarations -->
<dependencies>
+ <!-- https://mvnrepository.com/artifact/org.mockito/mockito-all -->
+ <dependency>
+ <groupId>org.mockito</groupId>
+ <artifactId>mockito-all</artifactId>
+ <version>1.8.4</version>
Review comment:
We specify dependency versions in the parent pom
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] DadanielZ commented on issue #1790: [HADOOP-16818] ABFS:
Combine append+flush calls for blockblob & appendblob
Posted by GitBox <gi...@apache.org>.
DadanielZ commented on issue #1790: [HADOOP-16818] ABFS: Combine append+flush calls for blockblob & appendblob
URL: https://github.com/apache/hadoop/pull/1790#issuecomment-597231945
LGTM +1, Hey @steveloughran do you still have the pending requests for changes? I will help to commit this one if it looks good to you now.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] snvijaya commented on a change in pull request #1790: changes for combined append+flush calls for blockblob(traditionan) an…
Posted by GitBox <gi...@apache.org>.
snvijaya commented on a change in pull request #1790: changes for combined append+flush calls for blockblob(traditionan) an…
URL: https://github.com/apache/hadoop/pull/1790#discussion_r363151946
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsClient.java
##########
@@ -55,7 +55,7 @@
public static final Logger LOG = LoggerFactory.getLogger(AbfsClient.class);
private final URL baseUrl;
private final SharedKeyCredentials sharedKeyCredentials;
- private final String xMsVersion = "2018-11-09";
+ private final String xMsVersion = "2019-12-12";
Review comment:
Is this version present on all the production clusters already ? If not, this change should go in only after the support is available.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] snvijaya commented on a change in pull request #1790: changes for combined append+flush calls for blockblob(traditionan) an…
Posted by GitBox <gi...@apache.org>.
snvijaya commented on a change in pull request #1790: changes for combined append+flush calls for blockblob(traditionan) an…
URL: https://github.com/apache/hadoop/pull/1790#discussion_r363153206
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsOutputStream.java
##########
@@ -309,15 +316,23 @@ private synchronized void flushWrittenBytesToService(boolean isClose) throws IOE
throw new FileNotFoundException(ex.getMessage());
}
}
-
if (ex.getCause() instanceof AzureBlobFileSystemException) {
ex = (AzureBlobFileSystemException) ex.getCause();
}
lastError = new IOException(ex);
throw lastError;
}
}
- flushWrittenBytesToServiceInternal(position, false, isClose);
+ shrinkWriteOperationQueue();
Review comment:
AbfsOutputStream.close() call flow is:
close() -> flushInternal() -> flushWrittenBytesToService() **-> flushWrittenBytesToServiceInternal()**
Within flushWrittenBytesToServiceInternal() is where service "Flush" gets called.
Above change removes call to flushWrittenBytesToServiceInternal() and replaces it with shrinkWriteOperationQueue(). This is going to prevent flush for AbfsOutputStream.close() calls.
This might be ok for AppendBlob, but BlockBlob behaviour needs to be retained where AbfsClient->Flush() will get triggered.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on issue #1790: [HADOOP-16818]
ABFS: Combine append+flush calls for blockblob & appendblob
Posted by GitBox <gi...@apache.org>.
steveloughran commented on issue #1790: [HADOOP-16818] ABFS: Combine append+flush calls for blockblob & appendblob
URL: https://github.com/apache/hadoop/pull/1790#issuecomment-594585020
going to add @bgaborg as a reviewer.
Now, something fundamental: what are blockblobs and appendblobs?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on a change in pull request #1790:
[HADOOP-16818] ABFS: Combine append+flush calls for blockblob & appendblob
Posted by GitBox <gi...@apache.org>.
steveloughran commented on a change in pull request #1790: [HADOOP-16818] ABFS: Combine append+flush calls for blockblob & appendblob
URL: https://github.com/apache/hadoop/pull/1790#discussion_r386485353
##########
File path: hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/services/TestAbfsOutputStream.java
##########
@@ -0,0 +1,349 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.azurebfs.services;
+import org.apache.hadoop.fs.azurebfs.AbfsConfiguration;
Review comment:
org.apache imports need to go into their own block just above any static imports, ordering imports
```
java., javax.
--
other
---
org.apache
---
static
```
This is to try and keep cherry-picking under control.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on issue #1790: [HADOOP-16818]
ABFS: Combine append+flush calls for blockblob & appendblob
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on issue #1790: [HADOOP-16818] ABFS: Combine append+flush calls for blockblob & appendblob
URL: https://github.com/apache/hadoop/pull/1790#issuecomment-596432484
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 0m 34s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +0 :ok: | markdownlint | 0m 0s | markdownlint was not available. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 2 new or modified test files. |
||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 20m 31s | trunk passed |
| +1 :green_heart: | compile | 0m 32s | trunk passed |
| +1 :green_heart: | checkstyle | 0m 25s | trunk passed |
| +1 :green_heart: | mvnsite | 0m 35s | trunk passed |
| +1 :green_heart: | shadedclient | 14m 55s | branch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 26s | trunk passed |
| +0 :ok: | spotbugs | 0m 52s | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 0m 50s | trunk passed |
||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 28s | the patch passed |
| +1 :green_heart: | compile | 0m 24s | the patch passed |
| +1 :green_heart: | javac | 0m 24s | the patch passed |
| -0 :warning: | checkstyle | 0m 17s | hadoop-tools/hadoop-azure: The patch generated 23 new + 5 unchanged - 0 fixed = 28 total (was 5) |
| +1 :green_heart: | mvnsite | 0m 27s | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | xml | 0m 2s | The patch has no ill-formed XML file. |
| +1 :green_heart: | shadedclient | 13m 41s | patch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 23s | the patch passed |
| +1 :green_heart: | findbugs | 0m 54s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 1m 22s | hadoop-azure in the patch passed. |
| +1 :green_heart: | asflicense | 0m 32s | The patch does not generate ASF License warnings. |
| | | 58m 51s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | Client=19.03.7 Server=19.03.7 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1790/18/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/1790 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient xml findbugs checkstyle markdownlint |
| uname | Linux 64fd29760367 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | personality/hadoop.sh |
| git revision | trunk / 6dfe00c |
| Default Java | 1.8.0_242 |
| checkstyle | https://builds.apache.org/job/hadoop-multibranch/job/PR-1790/18/artifact/out/diff-checkstyle-hadoop-tools_hadoop-azure.txt |
| Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1790/18/testReport/ |
| Max. process+thread count | 431 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1790/18/console |
| versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org