You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by GitBox <gi...@apache.org> on 2020/12/04 11:26:26 UTC
[GitHub] [hadoop] sumangala-patki opened a new pull request #2520: HADOOP-17290. ABFS: Add Identifiers to Client Request Header
sumangala-patki opened a new pull request #2520:
URL: https://github.com/apache/hadoop/pull/2520
Adding a set of identifiers to the `X_MS_CLIENT_REQUEST_ID` header to help correlate requests. This header contains IDs concatenated into a string, and appears in the storage diagnostic logs.
The clientRequestId guid used to identify requests is uniquely generated for each HTTP request. The additional identifiers will help group and/or filter requests for analysis. Of these, the clientCorrelationId is a unique identifier that can be provided by the user. The rest are generated by the driver.
Two configs introduced:
1. `fs.azure.client.correlation.id` - takes String for clientCorrelationId (alphanumeric characters/hyphens, max length = 72)
2. `fs.azure.tracingcontext.format` - option for selecting format of IDs to be included in header [select enum from `SINGLE_ID_FORMAT` (existing format), `ALL_ID_FORMAT` (new default), `TWO_ID_FORMAT`].
Tests to check format of header and validate the identifiers have been added. The tests also ensure that an invalid config input does not result in request failure.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] sumangala-patki commented on a change in pull request #2520: HADOOP-17290. ABFS: Add Identifiers to Client Request Header
Posted by GitBox <gi...@apache.org>.
sumangala-patki commented on a change in pull request #2520:
URL: https://github.com/apache/hadoop/pull/2520#discussion_r660084742
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/utils/TracingContext.java
##########
@@ -0,0 +1,170 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.azurebfs.utils;
+
+import java.util.UUID;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import org.apache.hadoop.fs.azurebfs.constants.FSOperationType;
+import org.apache.hadoop.fs.azurebfs.services.AbfsClient;
+
+import static org.apache.hadoop.fs.azurebfs.constants.AbfsHttpConstants.EMPTY_STRING;
+
+/**
+ * The TracingContext class to correlate Store requests using unique
+ * identifiers and resources common to requests (e.g. filesystem, stream)
+ *
+ * Implementing new HDFS method:
+ * Create TracingContext instance in method of outer layer of
+ * ABFS driver (AzureBlobFileSystem/AbfsInputStream/AbfsOutputStream), to be
+ * passed through ABFS layers up to AbfsRestOperation.
+ *
+ * Add new operations to HdfsOperationConstants file.
+ *
+ * PrimaryRequestId can be enabled for individual HDFS API that invoke
+ * multiple Store calls.
+ *
+ * Testing:
+ * Pass an instance of TracingHeaderValidator to registerListener() of ABFS
+ * filesystem/stream class before calling the API in tests.
+ */
+
+public class TracingContext {
+ private final String clientCorrelationID;
+ private final String fileSystemID;
+ private String clientRequestId = EMPTY_STRING;
+ private String primaryRequestID;
+ private String streamID;
+ private int retryCount;
+ private FSOperationType hadoopOpName;
+ private final TracingHeaderFormat format;
+ private Listener listener = null;
+
+ private static final Logger LOG = LoggerFactory.getLogger(AbfsClient.class);
+ public static final int MAX_CLIENT_CORRELATION_ID_LENGTH = 72;
+ public static final String CLIENT_CORRELATION_ID_PATTERN = "[a-zA-Z0-9-]*";
+
+ /**
+ * Initialize TracingContext
+ * @param clientCorrelationID Provided over config by client
+ * @param fileSystemID Unique guid for AzureBlobFileSystem instance
+ * @param hadoopOpName Code indicating the high-level Hadoop operation that
+ * triggered the current Store request
+ * @param tracingHeaderFormat Format of IDs to be printed in header and logs
+ * @param listener Holds instance of TracingHeaderValidator during testing,
+ * null otherwise
+ */
+ public TracingContext(String clientCorrelationID, String fileSystemID,
+ FSOperationType hadoopOpName, TracingHeaderFormat tracingHeaderFormat,
+ Listener listener) {
+ this.fileSystemID = fileSystemID;
+ this.hadoopOpName = hadoopOpName;
+ this.clientCorrelationID = validateClientCorrelationID(clientCorrelationID);
+ streamID = EMPTY_STRING;
+ retryCount = 0;
+ primaryRequestID = EMPTY_STRING;
+ format = tracingHeaderFormat;
+ this.listener = listener;
+ }
+
+ public TracingContext(String clientCorrelationID, String fileSystemID,
+ FSOperationType hadoopOpName, boolean needsPrimaryReqId,
+ TracingHeaderFormat tracingHeaderFormat, Listener listener) {
+ this(clientCorrelationID, fileSystemID, hadoopOpName, tracingHeaderFormat,
+ listener);
+ primaryRequestID = needsPrimaryReqId ? UUID.randomUUID().toString() : "";
+ if (listener != null) {
+ listener.updatePrimaryRequestID(primaryRequestID);
+ }
+ }
+
+ public TracingContext(TracingContext originalTracingContext) {
+ this.fileSystemID = originalTracingContext.fileSystemID;
+ this.streamID = originalTracingContext.streamID;
+ this.clientCorrelationID = originalTracingContext.clientCorrelationID;
+ this.hadoopOpName = originalTracingContext.hadoopOpName;
+ this.retryCount = 0;
+ this.primaryRequestID = originalTracingContext.primaryRequestID;
+ this.format = originalTracingContext.format;
+ if (originalTracingContext.listener != null) {
+ this.listener = originalTracingContext.listener.getClone();
+ }
+ }
+
+ public String validateClientCorrelationID(String clientCorrelationID) {
+ if ((clientCorrelationID.length() > MAX_CLIENT_CORRELATION_ID_LENGTH)
+ || (!clientCorrelationID.matches(CLIENT_CORRELATION_ID_PATTERN))) {
+ LOG.debug(
+ "Invalid config provided; correlation id not included in header.");
+ return EMPTY_STRING;
+ }
+ return clientCorrelationID;
+ }
+
+ public void generateClientRequestId() {
+ clientRequestId = UUID.randomUUID().toString();
+ }
+
+ public void setPrimaryRequestID() {
+ primaryRequestID = UUID.randomUUID().toString();
+ if (listener != null) {
+ listener.updatePrimaryRequestID(primaryRequestID);
+ }
+ }
+
+ public void setStreamID(String stream) {
+ streamID = stream;
+ }
+
+ public void setOperation(FSOperationType operation) {
+ this.hadoopOpName = operation;
+ }
+
+ public void setRetryCount(int retryCount) {
+ this.retryCount = retryCount;
+ }
+
+ public void setListener(Listener listener) {
+ this.listener = listener;
+ }
+
+ public String constructHeader() {
+ String header;
+ switch (format) {
+ case ALL_ID_FORMAT:
+ header =
Review comment:
Added description to member declaration and added note in the fn
Yes the number of separators (:) is kept constant for parsing
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/utils/TracingHeaderFormat.java
##########
@@ -0,0 +1,28 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.azurebfs.utils;
+
+public enum TracingHeaderFormat {
Review comment:
switched enum, but keeping correlationId at the start would be easier to distinguish it as a custom id of variable length
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] sumangala-patki commented on a change in pull request #2520: HADOOP-17290. ABFS: Add Identifiers to Client Request Header
Posted by GitBox <gi...@apache.org>.
sumangala-patki commented on a change in pull request #2520:
URL: https://github.com/apache/hadoop/pull/2520#discussion_r658212519
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AzureBlobFileSystem.java
##########
@@ -1283,6 +1373,11 @@ public String getCanonicalServiceName() {
return this.statistics;
}
+ @VisibleForTesting
+ void setListenerOperation(String operation) {
+ listener.setOperation(operation);
Review comment:
this is to reset and verify the 2-letter operation type (along with other IDs) when header tests are triggered using callback after header construction
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/constants/ConfigurationKeys.java
##########
@@ -109,6 +109,12 @@
* Default value of this config is true. **/
public static final String FS_AZURE_DISABLE_OUTPUTSTREAM_FLUSH = "fs.azure.disable.outputstream.flush";
public static final String FS_AZURE_USER_AGENT_PREFIX_KEY = "fs.azure.user.agent.prefix";
+ /**
+ * The client correlation ID provided over config that will be added to
+ * x-ms-client-request-Id header. Defaults to empty string if the length and
+ * character constraints are not satisfied. **/
+ public static final String FS_AZURE_CLIENT_CORRELATIONID = "fs.azure.client.correlationid";
+ public static final String FS_AZURE_TRACINGCONTEXT_FORMAT = "fs.azure.tracingcontext.format";
Review comment:
Yes, renamed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] sumangala-patki commented on pull request #2520: HADOOP-17290. ABFS: Add Identifiers to Client Request Header
Posted by GitBox <gi...@apache.org>.
sumangala-patki commented on pull request #2520:
URL: https://github.com/apache/hadoop/pull/2520#issuecomment-789484324
TEST RESULTS
HNS Account Location: East US 2
NonHNS Account Location: East US 2, Central US
```
HNS OAuth
[WARNING] Tests run: 96, Failures: 0, Errors: 0, Skipped: 1
[WARNING] Tests run: 454, Failures: 0, Errors: 0, Skipped: 68
[WARNING] Tests run: 261, Failures: 0, Errors: 0, Skipped: 48
HNS SharedKey
[WARNING] Tests run: 96, Failures: 0, Errors: 0, Skipped: 2
[WARNING] Tests run: 504, Failures: 0, Errors: 0, Skipped: 24
[WARNING] Tests run: 261, Failures: 0, Errors: 0, Skipped: 40
Non-HNS SharedKey
[WARNING] Tests run: 96, Failures: 0, Errors: 0, Skipped: 2
[WARNING] Tests run: 504, Failures: 0, Errors: 0, Skipped: 249
[WARNING] Tests run: 261, Failures: 0, Errors: 0, Skipped: 40
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #2520: HADOOP-17290. ABFS: Add Identifiers to Client Request Header
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on pull request #2520:
URL: https://github.com/apache/hadoop/pull/2520#issuecomment-738764426
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 33s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files found. |
| +0 :ok: | markdownlint | 0m 0s | | markdownlint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | | 0m 0s | [test4tests](test4tests) | The patch appears to include 29 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 32m 43s | | trunk passed |
| +1 :green_heart: | compile | 0m 39s | | trunk passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 |
| +1 :green_heart: | compile | 0m 34s | | trunk passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| +1 :green_heart: | checkstyle | 0m 27s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 39s | | trunk passed |
| +1 :green_heart: | shadedclient | 16m 20s | | branch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 33s | | trunk passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 |
| +1 :green_heart: | javadoc | 0m 29s | | trunk passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| +0 :ok: | spotbugs | 0m 58s | | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 0m 57s | | trunk passed |
| -0 :warning: | patch | 1m 17s | | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 30s | | the patch passed |
| +1 :green_heart: | compile | 0m 29s | | the patch passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 |
| +1 :green_heart: | javac | 0m 29s | | the patch passed |
| +1 :green_heart: | compile | 0m 25s | | the patch passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| +1 :green_heart: | javac | 0m 25s | | the patch passed |
| -0 :warning: | checkstyle | 0m 17s | [/diff-checkstyle-hadoop-tools_hadoop-azure.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2520/1/artifact/out/diff-checkstyle-hadoop-tools_hadoop-azure.txt) | hadoop-tools/hadoop-azure: The patch generated 34 new + 17 unchanged - 0 fixed = 51 total (was 17) |
| +1 :green_heart: | mvnsite | 0m 28s | | the patch passed |
| -1 :x: | whitespace | 0m 0s | [/whitespace-eol.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2520/1/artifact/out/whitespace-eol.txt) | The patch has 2 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply |
| +1 :green_heart: | shadedclient | 14m 39s | | patch has no errors when building and testing our client artifacts. |
| -1 :x: | javadoc | 0m 26s | [/diff-javadoc-javadoc-hadoop-tools_hadoop-azure-jdkUbuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2520/1/artifact/out/diff-javadoc-javadoc-hadoop-tools_hadoop-azure-jdkUbuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04.txt) | hadoop-tools_hadoop-azure-jdkUbuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 generated 5 new + 17 unchanged - 0 fixed = 22 total (was 17) |
| -1 :x: | javadoc | 0m 24s | [/diff-javadoc-javadoc-hadoop-tools_hadoop-azure-jdkPrivateBuild-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2520/1/artifact/out/diff-javadoc-javadoc-hadoop-tools_hadoop-azure-jdkPrivateBuild-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01.txt) | hadoop-tools_hadoop-azure-jdkPrivateBuild-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 generated 5 new + 17 unchanged - 0 fixed = 22 total (was 17) |
| -1 :x: | findbugs | 1m 0s | [/new-findbugs-hadoop-tools_hadoop-azure.html](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2520/1/artifact/out/new-findbugs-hadoop-tools_hadoop-azure.html) | hadoop-tools/hadoop-azure generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 1m 30s | | hadoop-azure in the patch passed. |
| -1 :x: | asflicense | 0m 34s | [/patch-asflicense-problems.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2520/1/artifact/out/patch-asflicense-problems.txt) | The patch generated 5 ASF License warnings. |
| | | 77m 13s | | |
| Reason | Tests |
|-------:|:------|
| FindBugs | module:hadoop-tools/hadoop-azure |
| | Switch statement found in org.apache.hadoop.fs.azurebfs.utils.TracingContext.toString() where default case is missing At TracingContext.java:where default case is missing At TracingContext.java:[lines 118-125] |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2520/1/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/2520 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle markdownlint |
| uname | Linux da6fd7bf2334 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / e2c1268ebd5 |
| Default Java | Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2520/1/testReport/ |
| Max. process+thread count | 536 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2520/1/console |
| versions | git=2.17.1 maven=3.6.0 findbugs=4.0.6 |
| Powered by | Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] surendralilhore commented on a change in pull request #2520: HADOOP-17290. ABFS: Add Identifiers to Client Request Header
Posted by GitBox <gi...@apache.org>.
surendralilhore commented on a change in pull request #2520:
URL: https://github.com/apache/hadoop/pull/2520#discussion_r650787085
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsInputStream.java
##########
@@ -142,6 +156,10 @@ public String getPath() {
return path;
}
+ private String getInputStreamID() {
+ return StringUtils.right(UUID.randomUUID().toString(), STREAM_ID_LEN);
Review comment:
Why this applicable for only stream id, not for filesystem id ? ?
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/constants/ConfigurationKeys.java
##########
@@ -109,6 +109,12 @@
* Default value of this config is true. **/
public static final String FS_AZURE_DISABLE_OUTPUTSTREAM_FLUSH = "fs.azure.disable.outputstream.flush";
public static final String FS_AZURE_USER_AGENT_PREFIX_KEY = "fs.azure.user.agent.prefix";
+ /**
+ * The client correlation ID provided over config that will be added to
+ * x-ms-client-request-Id header. Defaults to empty string if the length and
+ * character constraints are not satisfied. **/
+ public static final String FS_AZURE_CLIENT_CORRELATIONID = "fs.azure.client.correlationid";
Review comment:
Can you give me the example of possible value ?
What if in same jvm two FS objects are there ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] surendralilhore commented on pull request #2520: HADOOP-17290. ABFS: Add Identifiers to Client Request Header
Posted by GitBox <gi...@apache.org>.
surendralilhore commented on pull request #2520:
URL: https://github.com/apache/hadoop/pull/2520#issuecomment-873007684
+1
Thanks @anoopsjohn for review.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus removed a comment on pull request #2520: HADOOP-17290. ABFS: Add Identifiers to Client Request Header
Posted by GitBox <gi...@apache.org>.
hadoop-yetus removed a comment on pull request #2520:
URL: https://github.com/apache/hadoop/pull/2520#issuecomment-870058533
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] sumangala-patki commented on a change in pull request #2520: HADOOP-17290. ABFS: Add Identifiers to Client Request Header
Posted by GitBox <gi...@apache.org>.
sumangala-patki commented on a change in pull request #2520:
URL: https://github.com/apache/hadoop/pull/2520#discussion_r658211341
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AzureBlobFileSystem.java
##########
@@ -1049,8 +1130,11 @@ public boolean exists(Path f) throws IOException {
throws IOException {
LOG.debug("AzureBlobFileSystem.listStatusIterator path : {}", path);
if (abfsStore.getAbfsConfiguration().enableAbfsListIterator()) {
+ TracingContext tracingContext = new TracingContext(clientCorrelationID,
+ fileSystemID, HdfsOperationConstants.LISTSTATUS, true,
+ tracingContextFormat, listener);
AbfsListStatusRemoteIterator abfsLsItr =
- new AbfsListStatusRemoteIterator(getFileStatus(path), abfsStore);
+ new AbfsListStatusRemoteIterator(getFileStatus(path), abfsStore, tracingContext);
return RemoteIterators.typeCastingRemoteIterator(abfsLsItr);
} else {
return super.listStatusIterator(path);
Review comment:
the else block eventually calls ABFS liststatus method, which generates its own context. It is equivalent to generating and passing object from here as there is no action (request) before/after the super method call
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] surendralilhore merged pull request #2520: HADOOP-17290. ABFS: Add Identifiers to Client Request Header
Posted by GitBox <gi...@apache.org>.
surendralilhore merged pull request #2520:
URL: https://github.com/apache/hadoop/pull/2520
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] sumangala-patki commented on pull request #2520: HADOOP-17290. ABFS: Add Identifiers to Client Request Header
Posted by GitBox <gi...@apache.org>.
sumangala-patki commented on pull request #2520:
URL: https://github.com/apache/hadoop/pull/2520#issuecomment-799930385
TEST RESULTS
HNS Account Location: East US 2
NonHNS Account Location: East US 2, Central US
```
HNS OAuth
[WARNING] Tests run: 96, Failures: 0, Errors: 0, Skipped: 1
[WARNING] Tests run: 506, Failures: 0, Errors: 0, Skipped: 70
[WARNING] Tests run: 261, Failures: 0, Errors: 0, Skipped: 48
HNS SharedKey
[WARNING] Tests run: 96, Failures: 0, Errors: 0, Skipped: 2
[WARNING] Tests run: 506, Failures: 0, Errors: 0, Skipped: 26
[WARNING] Tests run: 261, Failures: 0, Errors: 0, Skipped: 40
Non-HNS SharedKey
[WARNING] Tests run: 96, Failures: 0, Errors: 0, Skipped: 2
[WARNING] Tests run: 506, Failures: 0, Errors: 0, Skipped: 250
[WARNING] Tests run: 261, Failures: 0, Errors: 0, Skipped: 40
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] sumangala-patki commented on a change in pull request #2520: HADOOP-17290. ABFS: Add Identifiers to Client Request Header
Posted by GitBox <gi...@apache.org>.
sumangala-patki commented on a change in pull request #2520:
URL: https://github.com/apache/hadoop/pull/2520#discussion_r658210556
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AzureBlobFileSystem.java
##########
@@ -1049,8 +1130,11 @@ public boolean exists(Path f) throws IOException {
throws IOException {
LOG.debug("AzureBlobFileSystem.listStatusIterator path : {}", path);
if (abfsStore.getAbfsConfiguration().enableAbfsListIterator()) {
+ TracingContext tracingContext = new TracingContext(clientCorrelationID,
+ fileSystemID, HdfsOperationConstants.LISTSTATUS, true,
+ tracingContextFormat, listener);
AbfsListStatusRemoteIterator abfsLsItr =
- new AbfsListStatusRemoteIterator(getFileStatus(path), abfsStore);
+ new AbfsListStatusRemoteIterator(getFileStatus(path), abfsStore, tracingContext);
Review comment:
modified getFileStatus to take tracingContext arg
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] sumangala-patki commented on a change in pull request #2520: HADOOP-17290. ABFS: Add Identifiers to Client Request Header
Posted by GitBox <gi...@apache.org>.
sumangala-patki commented on a change in pull request #2520:
URL: https://github.com/apache/hadoop/pull/2520#discussion_r660083974
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsOutputStream.java
##########
@@ -160,6 +170,14 @@ public AbfsOutputStream(
if (outputStreamStatistics != null) {
this.ioStatistics = outputStreamStatistics.getIOStatistics();
}
+ this.outputStreamId = getOutputStreamId();
+ this.tracingContext = new TracingContext(tracingContext);
+ this.tracingContext.setStreamID(outputStreamId);
+ this.tracingContext.setOperation(FSOperationType.WRITE);
+ }
+
+ private String getOutputStreamId() {
Review comment:
done
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsOutputStream.java
##########
@@ -385,7 +412,9 @@ private void writeAppendBlobCurrentBufferToService() throws IOException {
"writeCurrentBufferToService", "append")) {
AppendRequestParameters reqParams = new AppendRequestParameters(offset, 0,
bytesLength, APPEND_MODE, true, leaseId);
- AbfsRestOperation op = client.append(path, bytes, reqParams, cachedSasToken.get());
+ AbfsRestOperation op = client
+ .append(path, bytes, reqParams, cachedSasToken.get(),
+ new TracingContext(tracingContext));
Review comment:
yes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] sumangala-patki commented on a change in pull request #2520: HADOOP-17290. ABFS: Add Identifiers to Client Request Header
Posted by GitBox <gi...@apache.org>.
sumangala-patki commented on a change in pull request #2520:
URL: https://github.com/apache/hadoop/pull/2520#discussion_r658211715
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AzureBlobFileSystem.java
##########
@@ -335,7 +361,10 @@ public boolean rename(final Path src, final Path dst) throws IOException {
}
// Non-HNS account need to check dst status on driver side.
- if (!abfsStore.getIsNamespaceEnabled() && dstFileStatus == null) {
+ TracingContext tracingContext = new TracingContext(clientCorrelationID,
+ fileSystemID, HdfsOperationConstants.RENAME, true, tracingContextFormat,
+ listener);
+ if (!abfsStore.getIsNamespaceEnabled(tracingContext) && dstFileStatus == null) {
Review comment:
refactored the getFileStatus methods in AzureBlobFileSystem class to accommodate these. Changes done:
- overload getFileStatus to use context passed by other methods
- tryGetFileStatus to take context as arg
- pvt method createFileSystem signature change to avoid overloading tryGetFileStatus
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #2520: HADOOP-17290. ABFS: Add Identifiers to Client Request Header
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on pull request #2520:
URL: https://github.com/apache/hadoop/pull/2520#issuecomment-871296601
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 36s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | markdownlint | 0m 1s | | markdownlint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 37 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 30m 42s | | trunk passed |
| +1 :green_heart: | compile | 0m 42s | | trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 0m 36s | | trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | checkstyle | 0m 31s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 42s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 34s | | trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 0m 31s | | trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 1m 1s | | trunk passed |
| +1 :green_heart: | shadedclient | 14m 0s | | branch has no errors when building and testing our client artifacts. |
| -0 :warning: | patch | 14m 20s | | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 31s | | the patch passed |
| +1 :green_heart: | compile | 0m 32s | | the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javac | 0m 32s | | the patch passed |
| +1 :green_heart: | compile | 0m 28s | | the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | javac | 0m 28s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| -0 :warning: | checkstyle | 0m 18s | [/results-checkstyle-hadoop-tools_hadoop-azure.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2520/22/artifact/out/results-checkstyle-hadoop-tools_hadoop-azure.txt) | hadoop-tools/hadoop-azure: The patch generated 2 new + 17 unchanged - 0 fixed = 19 total (was 17) |
| +1 :green_heart: | mvnsite | 0m 31s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 23s | | the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 0m 22s | | the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 1m 1s | | the patch passed |
| +1 :green_heart: | shadedclient | 13m 42s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 1m 56s | | hadoop-azure in the patch passed. |
| +1 :green_heart: | asflicense | 0m 34s | | The patch does not generate ASF License warnings. |
| | | 71m 56s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2520/22/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/2520 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell markdownlint |
| uname | Linux 8699d6db5930 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / b69d9e209e0f6646cd1f1ff6a1a84ac4e7f3bfc8 |
| Default Java | Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2520/22/testReport/ |
| Max. process+thread count | 544 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2520/22/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #2520: HADOOP-17290. ABFS: Add Identifiers to Client Request Header
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on pull request #2520:
URL: https://github.com/apache/hadoop/pull/2520#issuecomment-750778958
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 39m 28s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files found. |
| +0 :ok: | markdownlint | 0m 0s | | markdownlint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | | 0m 0s | [test4tests](test4tests) | The patch appears to include 29 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 35m 38s | | trunk passed |
| +1 :green_heart: | compile | 0m 34s | | trunk passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 |
| +1 :green_heart: | compile | 0m 32s | | trunk passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| +1 :green_heart: | checkstyle | 0m 23s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 37s | | trunk passed |
| +1 :green_heart: | shadedclient | 17m 43s | | branch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 27s | | trunk passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 |
| +1 :green_heart: | javadoc | 0m 25s | | trunk passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| +0 :ok: | spotbugs | 0m 58s | | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 0m 55s | | trunk passed |
| -0 :warning: | patch | 1m 14s | | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 28s | | the patch passed |
| +1 :green_heart: | compile | 0m 30s | | the patch passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 |
| +1 :green_heart: | javac | 0m 30s | | the patch passed |
| +1 :green_heart: | compile | 0m 26s | | the patch passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| +1 :green_heart: | javac | 0m 26s | | the patch passed |
| -0 :warning: | checkstyle | 0m 15s | [/diff-checkstyle-hadoop-tools_hadoop-azure.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2520/4/artifact/out/diff-checkstyle-hadoop-tools_hadoop-azure.txt) | hadoop-tools/hadoop-azure: The patch generated 1 new + 17 unchanged - 0 fixed = 18 total (was 17) |
| +1 :green_heart: | mvnsite | 0m 28s | | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | | The patch has no whitespace issues. |
| +1 :green_heart: | shadedclient | 16m 35s | | patch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 23s | | the patch passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 |
| +1 :green_heart: | javadoc | 0m 21s | | the patch passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| +1 :green_heart: | findbugs | 1m 1s | | the patch passed |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 1m 29s | | hadoop-azure in the patch passed. |
| +1 :green_heart: | asflicense | 0m 30s | | The patch does not generate ASF License warnings. |
| | | 121m 25s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2520/4/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/2520 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle markdownlint |
| uname | Linux 451b1005c615 4.15.0-101-generic #102-Ubuntu SMP Mon May 11 10:07:26 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 16a20503cac |
| Default Java | Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2520/4/testReport/ |
| Max. process+thread count | 516 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2520/4/console |
| versions | git=2.17.1 maven=3.6.0 findbugs=4.0.6 |
| Powered by | Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] sumangala-patki commented on a change in pull request #2520: HADOOP-17290. ABFS: Add Identifiers to Client Request Header
Posted by GitBox <gi...@apache.org>.
sumangala-patki commented on a change in pull request #2520:
URL: https://github.com/apache/hadoop/pull/2520#discussion_r660083909
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsLease.java
##########
@@ -114,13 +119,15 @@ public AbfsLease(AbfsClient client, String path, int acquireMaxRetries,
LOG.debug("Acquired lease {} on {}", leaseID, path);
}
- private void acquireLease(RetryPolicy retryPolicy, int numRetries, int retryInterval, long delay)
+ private void acquireLease(RetryPolicy retryPolicy, int numRetries,
+ int retryInterval, long delay)
throws LeaseException {
LOG.debug("Attempting to acquire lease on {}, retry {}", path, numRetries);
if (future != null && !future.isDone()) {
throw new LeaseException(ERR_LEASE_FUTURE_EXISTS);
}
- future = client.schedule(() -> client.acquireLease(path, INFINITE_LEASE_DURATION),
+ future = client.schedule(() -> client.acquireLease(path,
+ INFINITE_LEASE_DURATION, new TracingContext(tracingContext)),
Review comment:
not needed; moved to calling method (constructor) where it is cloned only once, instead of per retry
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] sumangala-patki commented on a change in pull request #2520: HADOOP-17290. ABFS: Add Identifiers to Client Request Header
Posted by GitBox <gi...@apache.org>.
sumangala-patki commented on a change in pull request #2520:
URL: https://github.com/apache/hadoop/pull/2520#discussion_r660084233
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsRestOperation.java
##########
@@ -221,17 +226,27 @@ private void completeExecute() throws AzureBlobFileSystemException {
LOG.trace("{} REST operation complete", operationType);
}
+ private void updateClientRequestHeader(AbfsHttpOperation httpOperation,
+ TracingContext tracingContext) {
+ tracingContext.generateClientRequestId();
+ httpOperation.getConnection()
+ .setRequestProperty(HttpHeaderConfigurations.X_MS_CLIENT_REQUEST_ID,
+ tracingContext.constructHeader());
Review comment:
- constructHeader() may be called simply to display IDs, in which case it should not regenerate new clientReqId each time
- callee setting it would mean adding code to all methods to call this, and retry case would not be handled
Can be done internally within TC: have a generateClientRequestId function to create guid, and call it in constructor as well as the setRetryCount which is invoked per retry. However, had avoided this since we are populating ID variables only as and when they pass through the corresponding ABFS layers.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] anoopsjohn commented on a change in pull request #2520: HADOOP-17290. ABFS: Add Identifiers to Client Request Header
Posted by GitBox <gi...@apache.org>.
anoopsjohn commented on a change in pull request #2520:
URL: https://github.com/apache/hadoop/pull/2520#discussion_r658613610
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsLease.java
##########
@@ -114,13 +119,15 @@ public AbfsLease(AbfsClient client, String path, int acquireMaxRetries,
LOG.debug("Acquired lease {} on {}", leaseID, path);
}
- private void acquireLease(RetryPolicy retryPolicy, int numRetries, int retryInterval, long delay)
+ private void acquireLease(RetryPolicy retryPolicy, int numRetries,
+ int retryInterval, long delay)
throws LeaseException {
LOG.debug("Attempting to acquire lease on {}, retry {}", path, numRetries);
if (future != null && !future.isDone()) {
throw new LeaseException(ERR_LEASE_FUTURE_EXISTS);
}
- future = client.schedule(() -> client.acquireLease(path, INFINITE_LEASE_DURATION),
+ future = client.schedule(() -> client.acquireLease(path,
+ INFINITE_LEASE_DURATION, new TracingContext(tracingContext)),
Review comment:
This clone of Context needed here? What gets changed?
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsOutputStream.java
##########
@@ -160,6 +170,14 @@ public AbfsOutputStream(
if (outputStreamStatistics != null) {
this.ioStatistics = outputStreamStatistics.getIOStatistics();
}
+ this.outputStreamId = getOutputStreamId();
+ this.tracingContext = new TracingContext(tracingContext);
+ this.tracingContext.setStreamID(outputStreamId);
+ this.tracingContext.setOperation(FSOperationType.WRITE);
+ }
+
+ private String getOutputStreamId() {
Review comment:
getOutputStreamId() and getStreamID() -> Both create some confusion. Normally the Getter just return a already available value. getStreamID() make sense.
You can use createOutputStreamId() instead?
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsOutputStream.java
##########
@@ -385,7 +412,9 @@ private void writeAppendBlobCurrentBufferToService() throws IOException {
"writeCurrentBufferToService", "append")) {
AppendRequestParameters reqParams = new AppendRequestParameters(offset, 0,
bytesLength, APPEND_MODE, true, leaseId);
- AbfsRestOperation op = client.append(path, bytes, reqParams, cachedSasToken.get());
+ AbfsRestOperation op = client
+ .append(path, bytes, reqParams, cachedSasToken.get(),
+ new TracingContext(tracingContext));
Review comment:
Ok the context been passed here might get changed at least wrt the retryCount. that is why been cloned here?
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsRestOperation.java
##########
@@ -202,9 +206,10 @@ private void completeExecute() throws AzureBlobFileSystemException {
retryCount = 0;
LOG.debug("First execution of REST operation - {}", operationType);
- while (!executeHttpOperation(retryCount)) {
+ while (!executeHttpOperation(retryCount, tracingContext)) {
try {
++retryCount;
+ tracingContext.setRetryCount(retryCount);
Review comment:
Yaaa here.
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsOutputStream.java
##########
@@ -160,6 +170,14 @@ public AbfsOutputStream(
if (outputStreamStatistics != null) {
this.ioStatistics = outputStreamStatistics.getIOStatistics();
}
+ this.outputStreamId = getOutputStreamId();
+ this.tracingContext = new TracingContext(tracingContext);
+ this.tracingContext.setStreamID(outputStreamId);
+ this.tracingContext.setOperation(FSOperationType.WRITE);
+ }
+
+ private String getOutputStreamId() {
Review comment:
The same thing in ABFSInputStream also
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/utils/TracingContext.java
##########
@@ -0,0 +1,170 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.azurebfs.utils;
+
+import java.util.UUID;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import org.apache.hadoop.fs.azurebfs.constants.FSOperationType;
+import org.apache.hadoop.fs.azurebfs.services.AbfsClient;
+
+import static org.apache.hadoop.fs.azurebfs.constants.AbfsHttpConstants.EMPTY_STRING;
+
+/**
+ * The TracingContext class to correlate Store requests using unique
+ * identifiers and resources common to requests (e.g. filesystem, stream)
+ *
+ * Implementing new HDFS method:
+ * Create TracingContext instance in method of outer layer of
+ * ABFS driver (AzureBlobFileSystem/AbfsInputStream/AbfsOutputStream), to be
+ * passed through ABFS layers up to AbfsRestOperation.
+ *
+ * Add new operations to HdfsOperationConstants file.
+ *
+ * PrimaryRequestId can be enabled for individual HDFS API that invoke
+ * multiple Store calls.
+ *
+ * Testing:
+ * Pass an instance of TracingHeaderValidator to registerListener() of ABFS
+ * filesystem/stream class before calling the API in tests.
+ */
+
+public class TracingContext {
+ private final String clientCorrelationID;
+ private final String fileSystemID;
+ private String clientRequestId = EMPTY_STRING;
+ private String primaryRequestID;
Review comment:
primaryRequestID will be used in case of Read ahead? Worth adding come comments about its use here
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/utils/TracingHeaderFormat.java
##########
@@ -0,0 +1,28 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.azurebfs.utils;
+
+public enum TracingHeaderFormat {
Review comment:
Will the below way will be more consistent?
SINGLE_ID_FORMAT, // client-req-id
TWO_ID_FORMAT; // client-req-id:client-correlation-id
ALL_ID_FORMAT, // client-req-id:client-correlation-id:filesystem-id:primary-req-id:stream-id:hdfs-operation:retry-count
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsRestOperation.java
##########
@@ -221,17 +226,27 @@ private void completeExecute() throws AzureBlobFileSystemException {
LOG.trace("{} REST operation complete", operationType);
}
+ private void updateClientRequestHeader(AbfsHttpOperation httpOperation,
+ TracingContext tracingContext) {
+ tracingContext.generateClientRequestId();
+ httpOperation.getConnection()
+ .setRequestProperty(HttpHeaderConfigurations.X_MS_CLIENT_REQUEST_ID,
+ tracingContext.constructHeader());
Review comment:
generateClientRequestId() can be done internally within constructHeader()? Else its upto callee to set it proper before. Actually the UUID clientReqId generation should be an internal responsibility of this tracingContext right?
All other methods are like setter.
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsRestOperation.java
##########
@@ -221,17 +226,27 @@ private void completeExecute() throws AzureBlobFileSystemException {
LOG.trace("{} REST operation complete", operationType);
}
+ private void updateClientRequestHeader(AbfsHttpOperation httpOperation,
+ TracingContext tracingContext) {
+ tracingContext.generateClientRequestId();
+ httpOperation.getConnection()
+ .setRequestProperty(HttpHeaderConfigurations.X_MS_CLIENT_REQUEST_ID,
+ tracingContext.constructHeader());
+ }
+
/**
* Executes a single HTTP operation to complete the REST operation. If it
* fails, there may be a retry. The retryCount is incremented with each
* attempt.
*/
- private boolean executeHttpOperation(final int retryCount) throws AzureBlobFileSystemException {
+ private boolean executeHttpOperation(final int retryCount,
+ TracingContext tracingContext) throws AzureBlobFileSystemException {
AbfsHttpOperation httpOperation = null;
try {
// initialize the HTTP request and open the connection
httpOperation = new AbfsHttpOperation(url, method, requestHeaders);
incrementCounter(AbfsStatistic.CONNECTIONS_MADE, 1);
+ updateClientRequestHeader(httpOperation, tracingContext);
Review comment:
Its actually a fresh AbfsHttpOperation operation and here by we set the header. Can we call it setXXX than updateXXX?
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/utils/Listener.java
##########
@@ -0,0 +1,33 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.azurebfs.utils;
+
+import org.apache.hadoop.fs.azurebfs.constants.FSOperationType;
+
+/**
+ * Interface for testing identifiers tracked via TracingContext
+ * Implemented in TracingHeaderValidator
+ */
+
+public interface Listener {
Review comment:
This is for testability only?
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/utils/TracingContext.java
##########
@@ -0,0 +1,170 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.azurebfs.utils;
+
+import java.util.UUID;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import org.apache.hadoop.fs.azurebfs.constants.FSOperationType;
+import org.apache.hadoop.fs.azurebfs.services.AbfsClient;
+
+import static org.apache.hadoop.fs.azurebfs.constants.AbfsHttpConstants.EMPTY_STRING;
+
+/**
+ * The TracingContext class to correlate Store requests using unique
+ * identifiers and resources common to requests (e.g. filesystem, stream)
+ *
+ * Implementing new HDFS method:
+ * Create TracingContext instance in method of outer layer of
+ * ABFS driver (AzureBlobFileSystem/AbfsInputStream/AbfsOutputStream), to be
+ * passed through ABFS layers up to AbfsRestOperation.
+ *
+ * Add new operations to HdfsOperationConstants file.
+ *
+ * PrimaryRequestId can be enabled for individual HDFS API that invoke
+ * multiple Store calls.
+ *
+ * Testing:
+ * Pass an instance of TracingHeaderValidator to registerListener() of ABFS
+ * filesystem/stream class before calling the API in tests.
+ */
+
+public class TracingContext {
+ private final String clientCorrelationID;
+ private final String fileSystemID;
+ private String clientRequestId = EMPTY_STRING;
+ private String primaryRequestID;
+ private String streamID;
+ private int retryCount;
+ private FSOperationType hadoopOpName;
+ private final TracingHeaderFormat format;
+ private Listener listener = null;
+
+ private static final Logger LOG = LoggerFactory.getLogger(AbfsClient.class);
+ public static final int MAX_CLIENT_CORRELATION_ID_LENGTH = 72;
+ public static final String CLIENT_CORRELATION_ID_PATTERN = "[a-zA-Z0-9-]*";
+
+ /**
+ * Initialize TracingContext
+ * @param clientCorrelationID Provided over config by client
+ * @param fileSystemID Unique guid for AzureBlobFileSystem instance
+ * @param hadoopOpName Code indicating the high-level Hadoop operation that
+ * triggered the current Store request
+ * @param tracingHeaderFormat Format of IDs to be printed in header and logs
+ * @param listener Holds instance of TracingHeaderValidator during testing,
+ * null otherwise
+ */
+ public TracingContext(String clientCorrelationID, String fileSystemID,
+ FSOperationType hadoopOpName, TracingHeaderFormat tracingHeaderFormat,
+ Listener listener) {
+ this.fileSystemID = fileSystemID;
+ this.hadoopOpName = hadoopOpName;
Review comment:
call it opType only?
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/utils/TracingContext.java
##########
@@ -0,0 +1,170 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.azurebfs.utils;
+
+import java.util.UUID;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import org.apache.hadoop.fs.azurebfs.constants.FSOperationType;
+import org.apache.hadoop.fs.azurebfs.services.AbfsClient;
+
+import static org.apache.hadoop.fs.azurebfs.constants.AbfsHttpConstants.EMPTY_STRING;
+
+/**
+ * The TracingContext class to correlate Store requests using unique
+ * identifiers and resources common to requests (e.g. filesystem, stream)
+ *
+ * Implementing new HDFS method:
+ * Create TracingContext instance in method of outer layer of
+ * ABFS driver (AzureBlobFileSystem/AbfsInputStream/AbfsOutputStream), to be
+ * passed through ABFS layers up to AbfsRestOperation.
+ *
+ * Add new operations to HdfsOperationConstants file.
+ *
+ * PrimaryRequestId can be enabled for individual HDFS API that invoke
+ * multiple Store calls.
+ *
+ * Testing:
+ * Pass an instance of TracingHeaderValidator to registerListener() of ABFS
+ * filesystem/stream class before calling the API in tests.
+ */
+
+public class TracingContext {
+ private final String clientCorrelationID;
+ private final String fileSystemID;
+ private String clientRequestId = EMPTY_STRING;
+ private String primaryRequestID;
+ private String streamID;
+ private int retryCount;
+ private FSOperationType hadoopOpName;
+ private final TracingHeaderFormat format;
+ private Listener listener = null;
+
+ private static final Logger LOG = LoggerFactory.getLogger(AbfsClient.class);
+ public static final int MAX_CLIENT_CORRELATION_ID_LENGTH = 72;
+ public static final String CLIENT_CORRELATION_ID_PATTERN = "[a-zA-Z0-9-]*";
+
+ /**
+ * Initialize TracingContext
+ * @param clientCorrelationID Provided over config by client
+ * @param fileSystemID Unique guid for AzureBlobFileSystem instance
+ * @param hadoopOpName Code indicating the high-level Hadoop operation that
+ * triggered the current Store request
+ * @param tracingHeaderFormat Format of IDs to be printed in header and logs
+ * @param listener Holds instance of TracingHeaderValidator during testing,
+ * null otherwise
+ */
+ public TracingContext(String clientCorrelationID, String fileSystemID,
+ FSOperationType hadoopOpName, TracingHeaderFormat tracingHeaderFormat,
+ Listener listener) {
+ this.fileSystemID = fileSystemID;
+ this.hadoopOpName = hadoopOpName;
+ this.clientCorrelationID = validateClientCorrelationID(clientCorrelationID);
+ streamID = EMPTY_STRING;
+ retryCount = 0;
+ primaryRequestID = EMPTY_STRING;
+ format = tracingHeaderFormat;
+ this.listener = listener;
+ }
+
+ public TracingContext(String clientCorrelationID, String fileSystemID,
+ FSOperationType hadoopOpName, boolean needsPrimaryReqId,
+ TracingHeaderFormat tracingHeaderFormat, Listener listener) {
+ this(clientCorrelationID, fileSystemID, hadoopOpName, tracingHeaderFormat,
+ listener);
+ primaryRequestID = needsPrimaryReqId ? UUID.randomUUID().toString() : "";
+ if (listener != null) {
+ listener.updatePrimaryRequestID(primaryRequestID);
+ }
+ }
+
+ public TracingContext(TracingContext originalTracingContext) {
+ this.fileSystemID = originalTracingContext.fileSystemID;
+ this.streamID = originalTracingContext.streamID;
+ this.clientCorrelationID = originalTracingContext.clientCorrelationID;
+ this.hadoopOpName = originalTracingContext.hadoopOpName;
+ this.retryCount = 0;
+ this.primaryRequestID = originalTracingContext.primaryRequestID;
+ this.format = originalTracingContext.format;
+ if (originalTracingContext.listener != null) {
+ this.listener = originalTracingContext.listener.getClone();
+ }
+ }
+
+ public String validateClientCorrelationID(String clientCorrelationID) {
+ if ((clientCorrelationID.length() > MAX_CLIENT_CORRELATION_ID_LENGTH)
+ || (!clientCorrelationID.matches(CLIENT_CORRELATION_ID_PATTERN))) {
Review comment:
A regex matching call is not that cheap. We will end up calling this for every object creation of this TracingContext. Can we limit this check only at the time of FS instantiation?
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/utils/TracingContext.java
##########
@@ -0,0 +1,170 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.azurebfs.utils;
+
+import java.util.UUID;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import org.apache.hadoop.fs.azurebfs.constants.FSOperationType;
+import org.apache.hadoop.fs.azurebfs.services.AbfsClient;
+
+import static org.apache.hadoop.fs.azurebfs.constants.AbfsHttpConstants.EMPTY_STRING;
+
+/**
+ * The TracingContext class to correlate Store requests using unique
+ * identifiers and resources common to requests (e.g. filesystem, stream)
+ *
+ * Implementing new HDFS method:
+ * Create TracingContext instance in method of outer layer of
+ * ABFS driver (AzureBlobFileSystem/AbfsInputStream/AbfsOutputStream), to be
+ * passed through ABFS layers up to AbfsRestOperation.
+ *
+ * Add new operations to HdfsOperationConstants file.
+ *
+ * PrimaryRequestId can be enabled for individual HDFS API that invoke
+ * multiple Store calls.
+ *
+ * Testing:
+ * Pass an instance of TracingHeaderValidator to registerListener() of ABFS
+ * filesystem/stream class before calling the API in tests.
+ */
+
+public class TracingContext {
+ private final String clientCorrelationID;
+ private final String fileSystemID;
+ private String clientRequestId = EMPTY_STRING;
+ private String primaryRequestID;
+ private String streamID;
+ private int retryCount;
+ private FSOperationType hadoopOpName;
+ private final TracingHeaderFormat format;
+ private Listener listener = null;
+
+ private static final Logger LOG = LoggerFactory.getLogger(AbfsClient.class);
+ public static final int MAX_CLIENT_CORRELATION_ID_LENGTH = 72;
+ public static final String CLIENT_CORRELATION_ID_PATTERN = "[a-zA-Z0-9-]*";
+
+ /**
+ * Initialize TracingContext
+ * @param clientCorrelationID Provided over config by client
+ * @param fileSystemID Unique guid for AzureBlobFileSystem instance
+ * @param hadoopOpName Code indicating the high-level Hadoop operation that
+ * triggered the current Store request
+ * @param tracingHeaderFormat Format of IDs to be printed in header and logs
+ * @param listener Holds instance of TracingHeaderValidator during testing,
+ * null otherwise
+ */
+ public TracingContext(String clientCorrelationID, String fileSystemID,
+ FSOperationType hadoopOpName, TracingHeaderFormat tracingHeaderFormat,
+ Listener listener) {
+ this.fileSystemID = fileSystemID;
+ this.hadoopOpName = hadoopOpName;
+ this.clientCorrelationID = validateClientCorrelationID(clientCorrelationID);
+ streamID = EMPTY_STRING;
+ retryCount = 0;
+ primaryRequestID = EMPTY_STRING;
+ format = tracingHeaderFormat;
+ this.listener = listener;
+ }
+
+ public TracingContext(String clientCorrelationID, String fileSystemID,
+ FSOperationType hadoopOpName, boolean needsPrimaryReqId,
+ TracingHeaderFormat tracingHeaderFormat, Listener listener) {
+ this(clientCorrelationID, fileSystemID, hadoopOpName, tracingHeaderFormat,
+ listener);
+ primaryRequestID = needsPrimaryReqId ? UUID.randomUUID().toString() : "";
+ if (listener != null) {
+ listener.updatePrimaryRequestID(primaryRequestID);
+ }
+ }
+
+ public TracingContext(TracingContext originalTracingContext) {
+ this.fileSystemID = originalTracingContext.fileSystemID;
+ this.streamID = originalTracingContext.streamID;
+ this.clientCorrelationID = originalTracingContext.clientCorrelationID;
+ this.hadoopOpName = originalTracingContext.hadoopOpName;
+ this.retryCount = 0;
+ this.primaryRequestID = originalTracingContext.primaryRequestID;
+ this.format = originalTracingContext.format;
+ if (originalTracingContext.listener != null) {
+ this.listener = originalTracingContext.listener.getClone();
+ }
+ }
+
+ public String validateClientCorrelationID(String clientCorrelationID) {
+ if ((clientCorrelationID.length() > MAX_CLIENT_CORRELATION_ID_LENGTH)
+ || (!clientCorrelationID.matches(CLIENT_CORRELATION_ID_PATTERN))) {
+ LOG.debug(
+ "Invalid config provided; correlation id not included in header.");
+ return EMPTY_STRING;
+ }
+ return clientCorrelationID;
+ }
+
+ public void generateClientRequestId() {
+ clientRequestId = UUID.randomUUID().toString();
+ }
+
+ public void setPrimaryRequestID() {
+ primaryRequestID = UUID.randomUUID().toString();
+ if (listener != null) {
+ listener.updatePrimaryRequestID(primaryRequestID);
+ }
+ }
+
+ public void setStreamID(String stream) {
+ streamID = stream;
+ }
+
+ public void setOperation(FSOperationType operation) {
+ this.hadoopOpName = operation;
+ }
+
+ public void setRetryCount(int retryCount) {
+ this.retryCount = retryCount;
+ }
+
+ public void setListener(Listener listener) {
+ this.listener = listener;
+ }
+
+ public String constructHeader() {
+ String header;
+ switch (format) {
+ case ALL_ID_FORMAT:
+ header =
Review comment:
In any kind of op clientCorrelationID , clientRequestId , fileSystemID , hadoopOpName and retryCount will be present
Optional things are primaryRequestID and streamID . Correct?
Worth detailing in some comments here.
When these 2 are not there it will come like <file system ID>:::<op type>:... ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #2520: HADOOP-17290. ABFS: Add Identifiers to Client Request Header
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on pull request #2520:
URL: https://github.com/apache/hadoop/pull/2520#issuecomment-872117658
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 39s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | markdownlint | 0m 0s | | markdownlint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 37 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 30m 45s | | trunk passed |
| +1 :green_heart: | compile | 0m 40s | | trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 0m 36s | | trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | checkstyle | 0m 29s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 41s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 34s | | trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 0m 31s | | trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 1m 1s | | trunk passed |
| +1 :green_heart: | shadedclient | 14m 11s | | branch has no errors when building and testing our client artifacts. |
| -0 :warning: | patch | 14m 29s | | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 32s | | the patch passed |
| +1 :green_heart: | compile | 0m 31s | | the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javac | 0m 31s | | the patch passed |
| +1 :green_heart: | compile | 0m 28s | | the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | javac | 0m 28s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| +1 :green_heart: | checkstyle | 0m 15s | | the patch passed |
| +1 :green_heart: | mvnsite | 0m 31s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 22s | | the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 0m 22s | | the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 1m 2s | | the patch passed |
| +1 :green_heart: | shadedclient | 14m 4s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 2m 11s | | hadoop-azure in the patch passed. |
| +1 :green_heart: | asflicense | 0m 29s | | The patch does not generate ASF License warnings. |
| | | 72m 30s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2520/23/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/2520 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell markdownlint |
| uname | Linux 093ca10a13ea 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 8cf0fe32c89351a55c475cdd91261335591129f8 |
| Default Java | Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2520/23/testReport/ |
| Max. process+thread count | 543 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2520/23/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] sumangala-patki commented on a change in pull request #2520: HADOOP-17290. ABFS: Add Identifiers to Client Request Header
Posted by GitBox <gi...@apache.org>.
sumangala-patki commented on a change in pull request #2520:
URL: https://github.com/apache/hadoop/pull/2520#discussion_r660084305
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsRestOperation.java
##########
@@ -221,17 +226,27 @@ private void completeExecute() throws AzureBlobFileSystemException {
LOG.trace("{} REST operation complete", operationType);
}
+ private void updateClientRequestHeader(AbfsHttpOperation httpOperation,
+ TracingContext tracingContext) {
+ tracingContext.generateClientRequestId();
+ httpOperation.getConnection()
+ .setRequestProperty(HttpHeaderConfigurations.X_MS_CLIENT_REQUEST_ID,
+ tracingContext.constructHeader());
+ }
+
/**
* Executes a single HTTP operation to complete the REST operation. If it
* fails, there may be a retry. The retryCount is incremented with each
* attempt.
*/
- private boolean executeHttpOperation(final int retryCount) throws AzureBlobFileSystemException {
+ private boolean executeHttpOperation(final int retryCount,
+ TracingContext tracingContext) throws AzureBlobFileSystemException {
AbfsHttpOperation httpOperation = null;
try {
// initialize the HTTP request and open the connection
httpOperation = new AbfsHttpOperation(url, method, requestHeaders);
incrementCounter(AbfsStatistic.CONNECTIONS_MADE, 1);
+ updateClientRequestHeader(httpOperation, tracingContext);
Review comment:
done
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/utils/Listener.java
##########
@@ -0,0 +1,33 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.azurebfs.utils;
+
+import org.apache.hadoop.fs.azurebfs.constants.FSOperationType;
+
+/**
+ * Interface for testing identifiers tracked via TracingContext
+ * Implemented in TracingHeaderValidator
+ */
+
+public interface Listener {
Review comment:
yes, it's an interface to trigger header tests through callback when header is constructed. The tests are run only when listener is registered (not null), which is done across existing tests for different methods
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #2520: HADOOP-17290. ABFS: Add Identifiers to Client Request Header
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on pull request #2520:
URL: https://github.com/apache/hadoop/pull/2520#issuecomment-826808330
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 35s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 2s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | markdownlint | 0m 0s | | markdownlint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 36 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 34m 21s | | trunk passed |
| +1 :green_heart: | compile | 0m 38s | | trunk passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | compile | 0m 34s | | trunk passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | checkstyle | 0m 27s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 40s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 32s | | trunk passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 0m 30s | | trunk passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | spotbugs | 1m 0s | | trunk passed |
| +1 :green_heart: | shadedclient | 14m 3s | | branch has no errors when building and testing our client artifacts. |
| -0 :warning: | patch | 14m 22s | | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 28s | | the patch passed |
| +1 :green_heart: | compile | 0m 31s | | the patch passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javac | 0m 31s | | the patch passed |
| +1 :green_heart: | compile | 0m 26s | | the patch passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | javac | 0m 26s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| +1 :green_heart: | checkstyle | 0m 18s | | the patch passed |
| +1 :green_heart: | mvnsite | 0m 28s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 22s | | the patch passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 0m 21s | | the patch passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | spotbugs | 1m 0s | | the patch passed |
| +1 :green_heart: | shadedclient | 13m 55s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 2m 8s | | hadoop-azure in the patch passed. |
| +1 :green_heart: | asflicense | 0m 32s | | The patch does not generate ASF License warnings. |
| | | 75m 28s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2520/16/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/2520 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell markdownlint |
| uname | Linux f3d72e73dd5c 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / d52ee795da47e98178b2498357fb9f55a4e78ef1 |
| Default Java | Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2520/16/testReport/ |
| Max. process+thread count | 696 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2520/16/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #2520: HADOOP-17290. ABFS: Add Identifiers to Client Request Header
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on pull request #2520:
URL: https://github.com/apache/hadoop/pull/2520#issuecomment-739145117
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 32s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files found. |
| +0 :ok: | markdownlint | 0m 1s | | markdownlint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | | 0m 0s | [test4tests](test4tests) | The patch appears to include 29 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 32m 49s | | trunk passed |
| +1 :green_heart: | compile | 0m 38s | | trunk passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 |
| +1 :green_heart: | compile | 0m 34s | | trunk passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| +1 :green_heart: | checkstyle | 0m 28s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 39s | | trunk passed |
| +1 :green_heart: | shadedclient | 16m 17s | | branch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 32s | | trunk passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 |
| +1 :green_heart: | javadoc | 0m 29s | | trunk passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| +0 :ok: | spotbugs | 0m 58s | | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 0m 56s | | trunk passed |
| -0 :warning: | patch | 1m 15s | | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 32s | | the patch passed |
| +1 :green_heart: | compile | 0m 29s | | the patch passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 |
| +1 :green_heart: | javac | 0m 29s | | the patch passed |
| +1 :green_heart: | compile | 0m 26s | | the patch passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| +1 :green_heart: | javac | 0m 26s | | the patch passed |
| -0 :warning: | checkstyle | 0m 18s | [/diff-checkstyle-hadoop-tools_hadoop-azure.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2520/2/artifact/out/diff-checkstyle-hadoop-tools_hadoop-azure.txt) | hadoop-tools/hadoop-azure: The patch generated 1 new + 17 unchanged - 0 fixed = 18 total (was 17) |
| +1 :green_heart: | mvnsite | 0m 29s | | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | | The patch has no whitespace issues. |
| +1 :green_heart: | shadedclient | 15m 7s | | patch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 27s | | the patch passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 |
| +1 :green_heart: | javadoc | 0m 24s | | the patch passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| +1 :green_heart: | findbugs | 0m 59s | | the patch passed |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 1m 36s | | hadoop-azure in the patch passed. |
| +1 :green_heart: | asflicense | 0m 33s | | The patch does not generate ASF License warnings. |
| | | 77m 44s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2520/2/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/2520 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle markdownlint |
| uname | Linux bc79199cbef7 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 7dda804a1a7 |
| Default Java | Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2520/2/testReport/ |
| Max. process+thread count | 694 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2520/2/console |
| versions | git=2.17.1 maven=3.6.0 findbugs=4.0.6 |
| Powered by | Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus removed a comment on pull request #2520: HADOOP-17290. ABFS: Add Identifiers to Client Request Header
Posted by GitBox <gi...@apache.org>.
hadoop-yetus removed a comment on pull request #2520:
URL: https://github.com/apache/hadoop/pull/2520#issuecomment-872117658
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 39s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | markdownlint | 0m 0s | | markdownlint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 37 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 30m 45s | | trunk passed |
| +1 :green_heart: | compile | 0m 40s | | trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 0m 36s | | trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | checkstyle | 0m 29s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 41s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 34s | | trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 0m 31s | | trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 1m 1s | | trunk passed |
| +1 :green_heart: | shadedclient | 14m 11s | | branch has no errors when building and testing our client artifacts. |
| -0 :warning: | patch | 14m 29s | | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 32s | | the patch passed |
| +1 :green_heart: | compile | 0m 31s | | the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javac | 0m 31s | | the patch passed |
| +1 :green_heart: | compile | 0m 28s | | the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | javac | 0m 28s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| +1 :green_heart: | checkstyle | 0m 15s | | the patch passed |
| +1 :green_heart: | mvnsite | 0m 31s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 22s | | the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 0m 22s | | the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 1m 2s | | the patch passed |
| +1 :green_heart: | shadedclient | 14m 4s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 2m 11s | | hadoop-azure in the patch passed. |
| +1 :green_heart: | asflicense | 0m 29s | | The patch does not generate ASF License warnings. |
| | | 72m 30s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2520/23/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/2520 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell markdownlint |
| uname | Linux 093ca10a13ea 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 8cf0fe32c89351a55c475cdd91261335591129f8 |
| Default Java | Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2520/23/testReport/ |
| Max. process+thread count | 543 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2520/23/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] sumangala-patki commented on a change in pull request #2520: HADOOP-17290. ABFS: Add Identifiers to Client Request Header
Posted by GitBox <gi...@apache.org>.
sumangala-patki commented on a change in pull request #2520:
URL: https://github.com/apache/hadoop/pull/2520#discussion_r548384518
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsInputStream.java
##########
@@ -107,6 +122,15 @@ public String getPath() {
return path;
}
+ private String getInputStreamID() {
+ return StringUtils.right(UUID.randomUUID().toString(), STREAM_ID_LEN);
+ }
+
+ public void registerListener(Listener listener1) {
Review comment:
Can not make it private since the test classes using it are in a different package. Have moved to the end of file where other VisibleForTesting methods are written (which are also public due to the same reason)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] sumangala-patki commented on a change in pull request #2520: HADOOP-17290. ABFS: Add Identifiers to Client Request Header
Posted by GitBox <gi...@apache.org>.
sumangala-patki commented on a change in pull request #2520:
URL: https://github.com/apache/hadoop/pull/2520#discussion_r658212696
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/constants/HdfsOperationConstants.java
##########
@@ -0,0 +1,50 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.azurebfs.constants;
+
+public final class HdfsOperationConstants {
Review comment:
switched to enum; renamed class to FSOperationType
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsHttpOperation.java
##########
@@ -166,6 +166,11 @@ public String getResponseHeader(String httpHeader) {
return connection.getHeaderField(httpHeader);
}
+ @VisibleForTesting
+ public String getRequestHeader(String httpHeader) {
+ return connection.getRequestProperties().get(httpHeader).toString();
Review comment:
method not required, removed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] sumangala-patki commented on a change in pull request #2520: HADOOP-17290. ABFS: Add Identifiers to Client Request Header
Posted by GitBox <gi...@apache.org>.
sumangala-patki commented on a change in pull request #2520:
URL: https://github.com/apache/hadoop/pull/2520#discussion_r660083909
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsLease.java
##########
@@ -114,13 +119,15 @@ public AbfsLease(AbfsClient client, String path, int acquireMaxRetries,
LOG.debug("Acquired lease {} on {}", leaseID, path);
}
- private void acquireLease(RetryPolicy retryPolicy, int numRetries, int retryInterval, long delay)
+ private void acquireLease(RetryPolicy retryPolicy, int numRetries,
+ int retryInterval, long delay)
throws LeaseException {
LOG.debug("Attempting to acquire lease on {}, retry {}", path, numRetries);
if (future != null && !future.isDone()) {
throw new LeaseException(ERR_LEASE_FUTURE_EXISTS);
}
- future = client.schedule(() -> client.acquireLease(path, INFINITE_LEASE_DURATION),
+ future = client.schedule(() -> client.acquireLease(path,
+ INFINITE_LEASE_DURATION, new TracingContext(tracingContext)),
Review comment:
not needed; moved to calling method (constructor) where it is cloned only once, instead of per retry
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsOutputStream.java
##########
@@ -160,6 +170,14 @@ public AbfsOutputStream(
if (outputStreamStatistics != null) {
this.ioStatistics = outputStreamStatistics.getIOStatistics();
}
+ this.outputStreamId = getOutputStreamId();
+ this.tracingContext = new TracingContext(tracingContext);
+ this.tracingContext.setStreamID(outputStreamId);
+ this.tracingContext.setOperation(FSOperationType.WRITE);
+ }
+
+ private String getOutputStreamId() {
Review comment:
done
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsOutputStream.java
##########
@@ -385,7 +412,9 @@ private void writeAppendBlobCurrentBufferToService() throws IOException {
"writeCurrentBufferToService", "append")) {
AppendRequestParameters reqParams = new AppendRequestParameters(offset, 0,
bytesLength, APPEND_MODE, true, leaseId);
- AbfsRestOperation op = client.append(path, bytes, reqParams, cachedSasToken.get());
+ AbfsRestOperation op = client
+ .append(path, bytes, reqParams, cachedSasToken.get(),
+ new TracingContext(tracingContext));
Review comment:
yes
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsRestOperation.java
##########
@@ -221,17 +226,27 @@ private void completeExecute() throws AzureBlobFileSystemException {
LOG.trace("{} REST operation complete", operationType);
}
+ private void updateClientRequestHeader(AbfsHttpOperation httpOperation,
+ TracingContext tracingContext) {
+ tracingContext.generateClientRequestId();
+ httpOperation.getConnection()
+ .setRequestProperty(HttpHeaderConfigurations.X_MS_CLIENT_REQUEST_ID,
+ tracingContext.constructHeader());
Review comment:
- constructHeader() may be called simply to display IDs, in which case it should not regenerate new clientReqId each time
- callee setting it would mean adding code to all methods to call this, and retry case would not be handled
Can be done internally within TC: have a generateClientRequestId function to create guid, and call it in constructor as well as the setRetryCount which is invoked per retry. However, had avoided this since we are populating ID variables only as and when they pass through the corresponding ABFS layers.
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsRestOperation.java
##########
@@ -221,17 +226,27 @@ private void completeExecute() throws AzureBlobFileSystemException {
LOG.trace("{} REST operation complete", operationType);
}
+ private void updateClientRequestHeader(AbfsHttpOperation httpOperation,
+ TracingContext tracingContext) {
+ tracingContext.generateClientRequestId();
+ httpOperation.getConnection()
+ .setRequestProperty(HttpHeaderConfigurations.X_MS_CLIENT_REQUEST_ID,
+ tracingContext.constructHeader());
+ }
+
/**
* Executes a single HTTP operation to complete the REST operation. If it
* fails, there may be a retry. The retryCount is incremented with each
* attempt.
*/
- private boolean executeHttpOperation(final int retryCount) throws AzureBlobFileSystemException {
+ private boolean executeHttpOperation(final int retryCount,
+ TracingContext tracingContext) throws AzureBlobFileSystemException {
AbfsHttpOperation httpOperation = null;
try {
// initialize the HTTP request and open the connection
httpOperation = new AbfsHttpOperation(url, method, requestHeaders);
incrementCounter(AbfsStatistic.CONNECTIONS_MADE, 1);
+ updateClientRequestHeader(httpOperation, tracingContext);
Review comment:
done
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/utils/Listener.java
##########
@@ -0,0 +1,33 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.azurebfs.utils;
+
+import org.apache.hadoop.fs.azurebfs.constants.FSOperationType;
+
+/**
+ * Interface for testing identifiers tracked via TracingContext
+ * Implemented in TracingHeaderValidator
+ */
+
+public interface Listener {
Review comment:
yes, it's an interface to trigger header tests through callback when header is constructed. The tests are run only when listener is registered (not null), which is done across existing tests for different methods
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/utils/TracingContext.java
##########
@@ -0,0 +1,170 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.azurebfs.utils;
+
+import java.util.UUID;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import org.apache.hadoop.fs.azurebfs.constants.FSOperationType;
+import org.apache.hadoop.fs.azurebfs.services.AbfsClient;
+
+import static org.apache.hadoop.fs.azurebfs.constants.AbfsHttpConstants.EMPTY_STRING;
+
+/**
+ * The TracingContext class to correlate Store requests using unique
+ * identifiers and resources common to requests (e.g. filesystem, stream)
+ *
+ * Implementing new HDFS method:
+ * Create TracingContext instance in method of outer layer of
+ * ABFS driver (AzureBlobFileSystem/AbfsInputStream/AbfsOutputStream), to be
+ * passed through ABFS layers up to AbfsRestOperation.
+ *
+ * Add new operations to HdfsOperationConstants file.
+ *
+ * PrimaryRequestId can be enabled for individual HDFS API that invoke
+ * multiple Store calls.
+ *
+ * Testing:
+ * Pass an instance of TracingHeaderValidator to registerListener() of ABFS
+ * filesystem/stream class before calling the API in tests.
+ */
+
+public class TracingContext {
+ private final String clientCorrelationID;
+ private final String fileSystemID;
+ private String clientRequestId = EMPTY_STRING;
+ private String primaryRequestID;
Review comment:
primaryRequestId is applicable for any method call that triggers more than one http request. For example, methods using continuation logic like listStatus and rename
Have added comments explaining use of tracingContext and its members at the beginning of file
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/utils/TracingContext.java
##########
@@ -0,0 +1,170 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.azurebfs.utils;
+
+import java.util.UUID;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import org.apache.hadoop.fs.azurebfs.constants.FSOperationType;
+import org.apache.hadoop.fs.azurebfs.services.AbfsClient;
+
+import static org.apache.hadoop.fs.azurebfs.constants.AbfsHttpConstants.EMPTY_STRING;
+
+/**
+ * The TracingContext class to correlate Store requests using unique
+ * identifiers and resources common to requests (e.g. filesystem, stream)
+ *
+ * Implementing new HDFS method:
+ * Create TracingContext instance in method of outer layer of
+ * ABFS driver (AzureBlobFileSystem/AbfsInputStream/AbfsOutputStream), to be
+ * passed through ABFS layers up to AbfsRestOperation.
+ *
+ * Add new operations to HdfsOperationConstants file.
+ *
+ * PrimaryRequestId can be enabled for individual HDFS API that invoke
+ * multiple Store calls.
+ *
+ * Testing:
+ * Pass an instance of TracingHeaderValidator to registerListener() of ABFS
+ * filesystem/stream class before calling the API in tests.
+ */
+
+public class TracingContext {
+ private final String clientCorrelationID;
+ private final String fileSystemID;
+ private String clientRequestId = EMPTY_STRING;
+ private String primaryRequestID;
+ private String streamID;
+ private int retryCount;
+ private FSOperationType hadoopOpName;
+ private final TracingHeaderFormat format;
+ private Listener listener = null;
+
+ private static final Logger LOG = LoggerFactory.getLogger(AbfsClient.class);
+ public static final int MAX_CLIENT_CORRELATION_ID_LENGTH = 72;
+ public static final String CLIENT_CORRELATION_ID_PATTERN = "[a-zA-Z0-9-]*";
+
+ /**
+ * Initialize TracingContext
+ * @param clientCorrelationID Provided over config by client
+ * @param fileSystemID Unique guid for AzureBlobFileSystem instance
+ * @param hadoopOpName Code indicating the high-level Hadoop operation that
+ * triggered the current Store request
+ * @param tracingHeaderFormat Format of IDs to be printed in header and logs
+ * @param listener Holds instance of TracingHeaderValidator during testing,
+ * null otherwise
+ */
+ public TracingContext(String clientCorrelationID, String fileSystemID,
+ FSOperationType hadoopOpName, TracingHeaderFormat tracingHeaderFormat,
+ Listener listener) {
+ this.fileSystemID = fileSystemID;
+ this.hadoopOpName = hadoopOpName;
Review comment:
done
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/utils/TracingContext.java
##########
@@ -0,0 +1,170 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.azurebfs.utils;
+
+import java.util.UUID;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import org.apache.hadoop.fs.azurebfs.constants.FSOperationType;
+import org.apache.hadoop.fs.azurebfs.services.AbfsClient;
+
+import static org.apache.hadoop.fs.azurebfs.constants.AbfsHttpConstants.EMPTY_STRING;
+
+/**
+ * The TracingContext class to correlate Store requests using unique
+ * identifiers and resources common to requests (e.g. filesystem, stream)
+ *
+ * Implementing new HDFS method:
+ * Create TracingContext instance in method of outer layer of
+ * ABFS driver (AzureBlobFileSystem/AbfsInputStream/AbfsOutputStream), to be
+ * passed through ABFS layers up to AbfsRestOperation.
+ *
+ * Add new operations to HdfsOperationConstants file.
+ *
+ * PrimaryRequestId can be enabled for individual HDFS API that invoke
+ * multiple Store calls.
+ *
+ * Testing:
+ * Pass an instance of TracingHeaderValidator to registerListener() of ABFS
+ * filesystem/stream class before calling the API in tests.
+ */
+
+public class TracingContext {
+ private final String clientCorrelationID;
+ private final String fileSystemID;
+ private String clientRequestId = EMPTY_STRING;
+ private String primaryRequestID;
+ private String streamID;
+ private int retryCount;
+ private FSOperationType hadoopOpName;
+ private final TracingHeaderFormat format;
+ private Listener listener = null;
+
+ private static final Logger LOG = LoggerFactory.getLogger(AbfsClient.class);
+ public static final int MAX_CLIENT_CORRELATION_ID_LENGTH = 72;
+ public static final String CLIENT_CORRELATION_ID_PATTERN = "[a-zA-Z0-9-]*";
+
+ /**
+ * Initialize TracingContext
+ * @param clientCorrelationID Provided over config by client
+ * @param fileSystemID Unique guid for AzureBlobFileSystem instance
+ * @param hadoopOpName Code indicating the high-level Hadoop operation that
+ * triggered the current Store request
+ * @param tracingHeaderFormat Format of IDs to be printed in header and logs
+ * @param listener Holds instance of TracingHeaderValidator during testing,
+ * null otherwise
+ */
+ public TracingContext(String clientCorrelationID, String fileSystemID,
+ FSOperationType hadoopOpName, TracingHeaderFormat tracingHeaderFormat,
+ Listener listener) {
+ this.fileSystemID = fileSystemID;
+ this.hadoopOpName = hadoopOpName;
+ this.clientCorrelationID = validateClientCorrelationID(clientCorrelationID);
+ streamID = EMPTY_STRING;
+ retryCount = 0;
+ primaryRequestID = EMPTY_STRING;
+ format = tracingHeaderFormat;
+ this.listener = listener;
+ }
+
+ public TracingContext(String clientCorrelationID, String fileSystemID,
+ FSOperationType hadoopOpName, boolean needsPrimaryReqId,
+ TracingHeaderFormat tracingHeaderFormat, Listener listener) {
+ this(clientCorrelationID, fileSystemID, hadoopOpName, tracingHeaderFormat,
+ listener);
+ primaryRequestID = needsPrimaryReqId ? UUID.randomUUID().toString() : "";
+ if (listener != null) {
+ listener.updatePrimaryRequestID(primaryRequestID);
+ }
+ }
+
+ public TracingContext(TracingContext originalTracingContext) {
+ this.fileSystemID = originalTracingContext.fileSystemID;
+ this.streamID = originalTracingContext.streamID;
+ this.clientCorrelationID = originalTracingContext.clientCorrelationID;
+ this.hadoopOpName = originalTracingContext.hadoopOpName;
+ this.retryCount = 0;
+ this.primaryRequestID = originalTracingContext.primaryRequestID;
+ this.format = originalTracingContext.format;
+ if (originalTracingContext.listener != null) {
+ this.listener = originalTracingContext.listener.getClone();
+ }
+ }
+
+ public String validateClientCorrelationID(String clientCorrelationID) {
+ if ((clientCorrelationID.length() > MAX_CLIENT_CORRELATION_ID_LENGTH)
+ || (!clientCorrelationID.matches(CLIENT_CORRELATION_ID_PATTERN))) {
Review comment:
true. Changed the validate method to be static, calling from ABFS constructor only now
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/utils/TracingContext.java
##########
@@ -0,0 +1,170 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.azurebfs.utils;
+
+import java.util.UUID;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import org.apache.hadoop.fs.azurebfs.constants.FSOperationType;
+import org.apache.hadoop.fs.azurebfs.services.AbfsClient;
+
+import static org.apache.hadoop.fs.azurebfs.constants.AbfsHttpConstants.EMPTY_STRING;
+
+/**
+ * The TracingContext class to correlate Store requests using unique
+ * identifiers and resources common to requests (e.g. filesystem, stream)
+ *
+ * Implementing new HDFS method:
+ * Create TracingContext instance in method of outer layer of
+ * ABFS driver (AzureBlobFileSystem/AbfsInputStream/AbfsOutputStream), to be
+ * passed through ABFS layers up to AbfsRestOperation.
+ *
+ * Add new operations to HdfsOperationConstants file.
+ *
+ * PrimaryRequestId can be enabled for individual HDFS API that invoke
+ * multiple Store calls.
+ *
+ * Testing:
+ * Pass an instance of TracingHeaderValidator to registerListener() of ABFS
+ * filesystem/stream class before calling the API in tests.
+ */
+
+public class TracingContext {
+ private final String clientCorrelationID;
+ private final String fileSystemID;
+ private String clientRequestId = EMPTY_STRING;
+ private String primaryRequestID;
+ private String streamID;
+ private int retryCount;
+ private FSOperationType hadoopOpName;
+ private final TracingHeaderFormat format;
+ private Listener listener = null;
+
+ private static final Logger LOG = LoggerFactory.getLogger(AbfsClient.class);
+ public static final int MAX_CLIENT_CORRELATION_ID_LENGTH = 72;
+ public static final String CLIENT_CORRELATION_ID_PATTERN = "[a-zA-Z0-9-]*";
+
+ /**
+ * Initialize TracingContext
+ * @param clientCorrelationID Provided over config by client
+ * @param fileSystemID Unique guid for AzureBlobFileSystem instance
+ * @param hadoopOpName Code indicating the high-level Hadoop operation that
+ * triggered the current Store request
+ * @param tracingHeaderFormat Format of IDs to be printed in header and logs
+ * @param listener Holds instance of TracingHeaderValidator during testing,
+ * null otherwise
+ */
+ public TracingContext(String clientCorrelationID, String fileSystemID,
+ FSOperationType hadoopOpName, TracingHeaderFormat tracingHeaderFormat,
+ Listener listener) {
+ this.fileSystemID = fileSystemID;
+ this.hadoopOpName = hadoopOpName;
+ this.clientCorrelationID = validateClientCorrelationID(clientCorrelationID);
+ streamID = EMPTY_STRING;
+ retryCount = 0;
+ primaryRequestID = EMPTY_STRING;
+ format = tracingHeaderFormat;
+ this.listener = listener;
+ }
+
+ public TracingContext(String clientCorrelationID, String fileSystemID,
+ FSOperationType hadoopOpName, boolean needsPrimaryReqId,
+ TracingHeaderFormat tracingHeaderFormat, Listener listener) {
+ this(clientCorrelationID, fileSystemID, hadoopOpName, tracingHeaderFormat,
+ listener);
+ primaryRequestID = needsPrimaryReqId ? UUID.randomUUID().toString() : "";
+ if (listener != null) {
+ listener.updatePrimaryRequestID(primaryRequestID);
+ }
+ }
+
+ public TracingContext(TracingContext originalTracingContext) {
+ this.fileSystemID = originalTracingContext.fileSystemID;
+ this.streamID = originalTracingContext.streamID;
+ this.clientCorrelationID = originalTracingContext.clientCorrelationID;
+ this.hadoopOpName = originalTracingContext.hadoopOpName;
+ this.retryCount = 0;
+ this.primaryRequestID = originalTracingContext.primaryRequestID;
+ this.format = originalTracingContext.format;
+ if (originalTracingContext.listener != null) {
+ this.listener = originalTracingContext.listener.getClone();
+ }
+ }
+
+ public String validateClientCorrelationID(String clientCorrelationID) {
+ if ((clientCorrelationID.length() > MAX_CLIENT_CORRELATION_ID_LENGTH)
+ || (!clientCorrelationID.matches(CLIENT_CORRELATION_ID_PATTERN))) {
+ LOG.debug(
+ "Invalid config provided; correlation id not included in header.");
+ return EMPTY_STRING;
+ }
+ return clientCorrelationID;
+ }
+
+ public void generateClientRequestId() {
+ clientRequestId = UUID.randomUUID().toString();
+ }
+
+ public void setPrimaryRequestID() {
+ primaryRequestID = UUID.randomUUID().toString();
+ if (listener != null) {
+ listener.updatePrimaryRequestID(primaryRequestID);
+ }
+ }
+
+ public void setStreamID(String stream) {
+ streamID = stream;
+ }
+
+ public void setOperation(FSOperationType operation) {
+ this.hadoopOpName = operation;
+ }
+
+ public void setRetryCount(int retryCount) {
+ this.retryCount = retryCount;
+ }
+
+ public void setListener(Listener listener) {
+ this.listener = listener;
+ }
+
+ public String constructHeader() {
+ String header;
+ switch (format) {
+ case ALL_ID_FORMAT:
+ header =
Review comment:
Added description to member declaration and added note in the fn
Yes the number of separators (:) is kept constant for parsing
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/utils/TracingHeaderFormat.java
##########
@@ -0,0 +1,28 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.azurebfs.utils;
+
+public enum TracingHeaderFormat {
Review comment:
switched enum, but keeping correlationId at the start would be easier to distinguish it as a custom id of variable length
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #2520: HADOOP-17290. ABFS: Add Identifiers to Client Request Header
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on pull request #2520:
URL: https://github.com/apache/hadoop/pull/2520#issuecomment-867930679
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 37s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | markdownlint | 0m 0s | | markdownlint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 37 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 31m 46s | | trunk passed |
| +1 :green_heart: | compile | 0m 48s | | trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 0m 40s | | trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | checkstyle | 0m 31s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 47s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 37s | | trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 0m 34s | | trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 1m 11s | | trunk passed |
| +1 :green_heart: | shadedclient | 19m 17s | | branch has no errors when building and testing our client artifacts. |
| -0 :warning: | patch | 19m 40s | | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 37s | | the patch passed |
| +1 :green_heart: | compile | 0m 40s | | the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javac | 0m 40s | | the patch passed |
| +1 :green_heart: | compile | 0m 31s | | the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | javac | 0m 31s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| -0 :warning: | checkstyle | 0m 19s | [/results-checkstyle-hadoop-tools_hadoop-azure.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2520/20/artifact/out/results-checkstyle-hadoop-tools_hadoop-azure.txt) | hadoop-tools/hadoop-azure: The patch generated 3 new + 17 unchanged - 0 fixed = 20 total (was 17) |
| +1 :green_heart: | mvnsite | 0m 34s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 26s | | the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 0m 24s | | the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 1m 11s | | the patch passed |
| +1 :green_heart: | shadedclient | 19m 7s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 2m 27s | | hadoop-azure in the patch passed. |
| +1 :green_heart: | asflicense | 0m 39s | | The patch does not generate ASF License warnings. |
| | | 85m 13s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2520/20/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/2520 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell markdownlint |
| uname | Linux 65d8e2dfdfcb 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 1567e3e82240a242a6f11970d6d313db575a3787 |
| Default Java | Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2520/20/testReport/ |
| Max. process+thread count | 567 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2520/20/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] sumangala-patki commented on a change in pull request #2520: HADOOP-17290. ABFS: Add Identifiers to Client Request Header
Posted by GitBox <gi...@apache.org>.
sumangala-patki commented on a change in pull request #2520:
URL: https://github.com/apache/hadoop/pull/2520#discussion_r660084530
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/utils/TracingContext.java
##########
@@ -0,0 +1,170 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.azurebfs.utils;
+
+import java.util.UUID;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import org.apache.hadoop.fs.azurebfs.constants.FSOperationType;
+import org.apache.hadoop.fs.azurebfs.services.AbfsClient;
+
+import static org.apache.hadoop.fs.azurebfs.constants.AbfsHttpConstants.EMPTY_STRING;
+
+/**
+ * The TracingContext class to correlate Store requests using unique
+ * identifiers and resources common to requests (e.g. filesystem, stream)
+ *
+ * Implementing new HDFS method:
+ * Create TracingContext instance in method of outer layer of
+ * ABFS driver (AzureBlobFileSystem/AbfsInputStream/AbfsOutputStream), to be
+ * passed through ABFS layers up to AbfsRestOperation.
+ *
+ * Add new operations to HdfsOperationConstants file.
+ *
+ * PrimaryRequestId can be enabled for individual HDFS API that invoke
+ * multiple Store calls.
+ *
+ * Testing:
+ * Pass an instance of TracingHeaderValidator to registerListener() of ABFS
+ * filesystem/stream class before calling the API in tests.
+ */
+
+public class TracingContext {
+ private final String clientCorrelationID;
+ private final String fileSystemID;
+ private String clientRequestId = EMPTY_STRING;
+ private String primaryRequestID;
Review comment:
primaryRequestId is applicable for any method call that triggers more than one http request. For example, methods using continuation logic like listStatus and rename
Have added comments explaining use of tracingContext and its members at the beginning of file
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/utils/TracingContext.java
##########
@@ -0,0 +1,170 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.azurebfs.utils;
+
+import java.util.UUID;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import org.apache.hadoop.fs.azurebfs.constants.FSOperationType;
+import org.apache.hadoop.fs.azurebfs.services.AbfsClient;
+
+import static org.apache.hadoop.fs.azurebfs.constants.AbfsHttpConstants.EMPTY_STRING;
+
+/**
+ * The TracingContext class to correlate Store requests using unique
+ * identifiers and resources common to requests (e.g. filesystem, stream)
+ *
+ * Implementing new HDFS method:
+ * Create TracingContext instance in method of outer layer of
+ * ABFS driver (AzureBlobFileSystem/AbfsInputStream/AbfsOutputStream), to be
+ * passed through ABFS layers up to AbfsRestOperation.
+ *
+ * Add new operations to HdfsOperationConstants file.
+ *
+ * PrimaryRequestId can be enabled for individual HDFS API that invoke
+ * multiple Store calls.
+ *
+ * Testing:
+ * Pass an instance of TracingHeaderValidator to registerListener() of ABFS
+ * filesystem/stream class before calling the API in tests.
+ */
+
+public class TracingContext {
+ private final String clientCorrelationID;
+ private final String fileSystemID;
+ private String clientRequestId = EMPTY_STRING;
+ private String primaryRequestID;
+ private String streamID;
+ private int retryCount;
+ private FSOperationType hadoopOpName;
+ private final TracingHeaderFormat format;
+ private Listener listener = null;
+
+ private static final Logger LOG = LoggerFactory.getLogger(AbfsClient.class);
+ public static final int MAX_CLIENT_CORRELATION_ID_LENGTH = 72;
+ public static final String CLIENT_CORRELATION_ID_PATTERN = "[a-zA-Z0-9-]*";
+
+ /**
+ * Initialize TracingContext
+ * @param clientCorrelationID Provided over config by client
+ * @param fileSystemID Unique guid for AzureBlobFileSystem instance
+ * @param hadoopOpName Code indicating the high-level Hadoop operation that
+ * triggered the current Store request
+ * @param tracingHeaderFormat Format of IDs to be printed in header and logs
+ * @param listener Holds instance of TracingHeaderValidator during testing,
+ * null otherwise
+ */
+ public TracingContext(String clientCorrelationID, String fileSystemID,
+ FSOperationType hadoopOpName, TracingHeaderFormat tracingHeaderFormat,
+ Listener listener) {
+ this.fileSystemID = fileSystemID;
+ this.hadoopOpName = hadoopOpName;
Review comment:
done
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/utils/TracingContext.java
##########
@@ -0,0 +1,170 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.azurebfs.utils;
+
+import java.util.UUID;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import org.apache.hadoop.fs.azurebfs.constants.FSOperationType;
+import org.apache.hadoop.fs.azurebfs.services.AbfsClient;
+
+import static org.apache.hadoop.fs.azurebfs.constants.AbfsHttpConstants.EMPTY_STRING;
+
+/**
+ * The TracingContext class to correlate Store requests using unique
+ * identifiers and resources common to requests (e.g. filesystem, stream)
+ *
+ * Implementing new HDFS method:
+ * Create TracingContext instance in method of outer layer of
+ * ABFS driver (AzureBlobFileSystem/AbfsInputStream/AbfsOutputStream), to be
+ * passed through ABFS layers up to AbfsRestOperation.
+ *
+ * Add new operations to HdfsOperationConstants file.
+ *
+ * PrimaryRequestId can be enabled for individual HDFS API that invoke
+ * multiple Store calls.
+ *
+ * Testing:
+ * Pass an instance of TracingHeaderValidator to registerListener() of ABFS
+ * filesystem/stream class before calling the API in tests.
+ */
+
+public class TracingContext {
+ private final String clientCorrelationID;
+ private final String fileSystemID;
+ private String clientRequestId = EMPTY_STRING;
+ private String primaryRequestID;
+ private String streamID;
+ private int retryCount;
+ private FSOperationType hadoopOpName;
+ private final TracingHeaderFormat format;
+ private Listener listener = null;
+
+ private static final Logger LOG = LoggerFactory.getLogger(AbfsClient.class);
+ public static final int MAX_CLIENT_CORRELATION_ID_LENGTH = 72;
+ public static final String CLIENT_CORRELATION_ID_PATTERN = "[a-zA-Z0-9-]*";
+
+ /**
+ * Initialize TracingContext
+ * @param clientCorrelationID Provided over config by client
+ * @param fileSystemID Unique guid for AzureBlobFileSystem instance
+ * @param hadoopOpName Code indicating the high-level Hadoop operation that
+ * triggered the current Store request
+ * @param tracingHeaderFormat Format of IDs to be printed in header and logs
+ * @param listener Holds instance of TracingHeaderValidator during testing,
+ * null otherwise
+ */
+ public TracingContext(String clientCorrelationID, String fileSystemID,
+ FSOperationType hadoopOpName, TracingHeaderFormat tracingHeaderFormat,
+ Listener listener) {
+ this.fileSystemID = fileSystemID;
+ this.hadoopOpName = hadoopOpName;
+ this.clientCorrelationID = validateClientCorrelationID(clientCorrelationID);
+ streamID = EMPTY_STRING;
+ retryCount = 0;
+ primaryRequestID = EMPTY_STRING;
+ format = tracingHeaderFormat;
+ this.listener = listener;
+ }
+
+ public TracingContext(String clientCorrelationID, String fileSystemID,
+ FSOperationType hadoopOpName, boolean needsPrimaryReqId,
+ TracingHeaderFormat tracingHeaderFormat, Listener listener) {
+ this(clientCorrelationID, fileSystemID, hadoopOpName, tracingHeaderFormat,
+ listener);
+ primaryRequestID = needsPrimaryReqId ? UUID.randomUUID().toString() : "";
+ if (listener != null) {
+ listener.updatePrimaryRequestID(primaryRequestID);
+ }
+ }
+
+ public TracingContext(TracingContext originalTracingContext) {
+ this.fileSystemID = originalTracingContext.fileSystemID;
+ this.streamID = originalTracingContext.streamID;
+ this.clientCorrelationID = originalTracingContext.clientCorrelationID;
+ this.hadoopOpName = originalTracingContext.hadoopOpName;
+ this.retryCount = 0;
+ this.primaryRequestID = originalTracingContext.primaryRequestID;
+ this.format = originalTracingContext.format;
+ if (originalTracingContext.listener != null) {
+ this.listener = originalTracingContext.listener.getClone();
+ }
+ }
+
+ public String validateClientCorrelationID(String clientCorrelationID) {
+ if ((clientCorrelationID.length() > MAX_CLIENT_CORRELATION_ID_LENGTH)
+ || (!clientCorrelationID.matches(CLIENT_CORRELATION_ID_PATTERN))) {
Review comment:
true. Changed the validate method to be static, calling from ABFS constructor only now
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] snvijaya commented on a change in pull request #2520: HADOOP-17290. ABFS: Add Identifiers to Client Request Header
Posted by GitBox <gi...@apache.org>.
snvijaya commented on a change in pull request #2520:
URL: https://github.com/apache/hadoop/pull/2520#discussion_r543089409
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AzureBlobFileSystem.java
##########
@@ -1202,6 +1287,10 @@ public String getCanonicalServiceName() {
return this.statistics;
}
+ public void setListenerOperation(String operation) {
Review comment:
This will only be used by test code ? If so, lower accessibility to private and set VisibleForTesting.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] snvijaya commented on a change in pull request #2520: HADOOP-17290. ABFS: Add Identifiers to Client Request Header
Posted by GitBox <gi...@apache.org>.
snvijaya commented on a change in pull request #2520:
URL: https://github.com/apache/hadoop/pull/2520#discussion_r543098890
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsInputStream.java
##########
@@ -107,6 +122,15 @@ public String getPath() {
return path;
}
+ private String getInputStreamID() {
+ return StringUtils.right(UUID.randomUUID().toString(), STREAM_ID_LEN);
+ }
+
+ public void registerListener(Listener listener1) {
Review comment:
Same as before. Methods used by test need to be private and annotated with VisibleForTesting
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] sumangala-patki commented on pull request #2520: HADOOP-17290. ABFS: Add Identifiers to Client Request Header
Posted by GitBox <gi...@apache.org>.
sumangala-patki commented on pull request #2520:
URL: https://github.com/apache/hadoop/pull/2520#issuecomment-827392911
TEST RESULTS
HNS Account Location: East US 2
NonHNS Account Location: East US 2, Central US
```
HNS-OAuth
[WARNING] Tests run: 100, Failures: 0, Errors: 0, Skipped: 1
[WARNING] Tests run: 530, Failures: 0, Errors: 0, Skipped: 70
[WARNING] Tests run: 265, Failures: 0, Errors: 0, Skipped: 50
HNS-SharedKey
[WARNING] Tests run: 100, Failures: 0, Errors: 0, Skipped: 2
[ERROR] Tests run: 530, Failures: 0, Errors: 0, Skipped: 26
[WARNING] Tests run: 265, Failures: 0, Errors: 0, Skipped: 40
NonHNS-SharedKey
[WARNING] Tests run: 100, Failures: 0, Errors: 0, Skipped: 2
[WARNING] Tests run: 530, Failures: 0, Errors: 0, Skipped: 248
[WARNING] Tests run: 265, Failures: 0, Errors: 0, Skipped: 40
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #2520: HADOOP-17290. ABFS: Add Identifiers to Client Request Header
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on pull request #2520:
URL: https://github.com/apache/hadoop/pull/2520#issuecomment-799192776
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 34s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | markdownlint | 0m 0s | | markdownlint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 34 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 32m 51s | | trunk passed |
| +1 :green_heart: | compile | 0m 38s | | trunk passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | compile | 0m 35s | | trunk passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | checkstyle | 0m 26s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 36s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 31s | | trunk passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 0m 29s | | trunk passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | spotbugs | 1m 1s | | trunk passed |
| +1 :green_heart: | shadedclient | 14m 5s | | branch has no errors when building and testing our client artifacts. |
| -0 :warning: | patch | 14m 23s | | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 29s | | the patch passed |
| +1 :green_heart: | compile | 0m 31s | | the patch passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javac | 0m 31s | | the patch passed |
| +1 :green_heart: | compile | 0m 26s | | the patch passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | javac | 0m 26s | | the patch passed |
| +1 :green_heart: | blanks | 0m 1s | | The patch has no blanks issues. |
| +1 :green_heart: | checkstyle | 0m 17s | | the patch passed |
| +1 :green_heart: | mvnsite | 0m 28s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 23s | | the patch passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 0m 21s | | the patch passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | spotbugs | 0m 59s | | the patch passed |
| +1 :green_heart: | shadedclient | 13m 42s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 2m 8s | | hadoop-azure in the patch passed. |
| +1 :green_heart: | asflicense | 0m 32s | | The patch does not generate ASF License warnings. |
| | | 73m 38s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2520/10/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/2520 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell markdownlint |
| uname | Linux 0022b4d70dd0 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 1e529f4d5b3d70eae74fbeceb7365925f9c17b68 |
| Default Java | Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2520/10/testReport/ |
| Max. process+thread count | 735 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2520/10/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] sumangala-patki commented on a change in pull request #2520: HADOOP-17290. ABFS: Add Identifiers to Client Request Header
Posted by GitBox <gi...@apache.org>.
sumangala-patki commented on a change in pull request #2520:
URL: https://github.com/apache/hadoop/pull/2520#discussion_r548384518
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsInputStream.java
##########
@@ -107,6 +122,15 @@ public String getPath() {
return path;
}
+ private String getInputStreamID() {
+ return StringUtils.right(UUID.randomUUID().toString(), STREAM_ID_LEN);
+ }
+
+ public void registerListener(Listener listener1) {
Review comment:
Cannot make it private since the test classes using it are in a different package. Have moved to the end of file where other VisibleForTesting methods are written (which are also public due to the same reason).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] sumangala-patki commented on a change in pull request #2520: HADOOP-17290. ABFS: Add Identifiers to Client Request Header
Posted by GitBox <gi...@apache.org>.
sumangala-patki commented on a change in pull request #2520:
URL: https://github.com/apache/hadoop/pull/2520#discussion_r658212324
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AzureBlobFileSystem.java
##########
@@ -1071,7 +1155,10 @@ private boolean fileSystemExists() throws IOException {
LOG.debug(
"AzureBlobFileSystem.fileSystemExists uri: {}", uri);
try {
- abfsStore.getFilesystemProperties();
+ TracingContext tracingContext = new TracingContext(clientCorrelationID,
+ fileSystemID, HdfsOperationConstants.GET_FILESTATUS,
Review comment:
This is a private method that does not implement any Hadoop function, and is never used in the Driver. However, we need to pass in a dummy tracing object for syntax. Modified to use test operation type
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] sumangala-patki commented on pull request #2520: HADOOP-17290. ABFS: Add Identifiers to Client Request Header
Posted by GitBox <gi...@apache.org>.
sumangala-patki commented on pull request #2520:
URL: https://github.com/apache/hadoop/pull/2520#issuecomment-739860111
TEST RESULTS
HNS Account Location: East US 2
NonHNS Account Location: East US 2, Central US
```
HNS OAuth
[WARNING] Tests run: 93, Failures: 0, Errors: 0, Skipped: 1
[WARNING] Tests run: 464, Failures: 0, Errors: 0, Skipped: 68
[WARNING] Tests run: 212, Failures: 0, Errors: 0, Skipped: 24
HNS SharedKey
[WARNING] Tests run: 93, Failures: 0, Errors: 0, Skipped: 2
[WARNING] Tests run: 464, Failures: 0, Errors: 0, Skipped: 24
[WARNING] Tests run: 212, Failures: 0, Errors: 0, Skipped: 16
Non-HNS SharedKey
[WARNING] Tests run: 93, Failures: 0, Errors: 0, Skipped: 2
[WARNING] Tests run: 464, Failures: 0, Errors: 0, Skipped: 249
[WARNING] Tests run: 212, Failures: 0, Errors: 0, Skipped: 16
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] sumangala-patki commented on a change in pull request #2520: HADOOP-17290. ABFS: Add Identifiers to Client Request Header
Posted by GitBox <gi...@apache.org>.
sumangala-patki commented on a change in pull request #2520:
URL: https://github.com/apache/hadoop/pull/2520#discussion_r658209651
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AzureBlobFileSystem.java
##########
@@ -111,10 +116,14 @@
private Path workingDir;
private AzureBlobFileSystemStore abfsStore;
private boolean isClosed;
+ private final String fileSystemID = UUID.randomUUID().toString();
Review comment:
yes, done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #2520: HADOOP-17290. ABFS: Add Identifiers to Client Request Header
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on pull request #2520:
URL: https://github.com/apache/hadoop/pull/2520#issuecomment-870058533
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 34s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | markdownlint | 0m 0s | | markdownlint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 37 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 33m 28s | | trunk passed |
| +1 :green_heart: | compile | 0m 40s | | trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 0m 35s | | trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | checkstyle | 0m 26s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 40s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 32s | | trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 0m 31s | | trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 1m 3s | | trunk passed |
| +1 :green_heart: | shadedclient | 16m 23s | | branch has no errors when building and testing our client artifacts. |
| -0 :warning: | patch | 16m 43s | | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 32s | | the patch passed |
| +1 :green_heart: | compile | 0m 33s | | the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javac | 0m 33s | | the patch passed |
| +1 :green_heart: | compile | 0m 28s | | the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | javac | 0m 28s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| -0 :warning: | checkstyle | 0m 18s | [/results-checkstyle-hadoop-tools_hadoop-azure.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2520/21/artifact/out/results-checkstyle-hadoop-tools_hadoop-azure.txt) | hadoop-tools/hadoop-azure: The patch generated 2 new + 17 unchanged - 0 fixed = 19 total (was 17) |
| +1 :green_heart: | mvnsite | 0m 34s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 25s | | the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 0m 22s | | the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 1m 9s | | the patch passed |
| +1 :green_heart: | shadedclient | 16m 14s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 2m 12s | | hadoop-azure in the patch passed. |
| +1 :green_heart: | asflicense | 0m 33s | | The patch does not generate ASF License warnings. |
| | | 79m 47s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2520/21/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/2520 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell markdownlint |
| uname | Linux 31624da3802c 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / ecee1809de89ed5b4fc76df79dd64895537aab99 |
| Default Java | Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2520/21/testReport/ |
| Max. process+thread count | 678 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2520/21/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] sumangala-patki commented on pull request #2520: HADOOP-17290. ABFS: Add Identifiers to Client Request Header
Posted by GitBox <gi...@apache.org>.
sumangala-patki commented on pull request #2520:
URL: https://github.com/apache/hadoop/pull/2520#issuecomment-796472068
TEST RESULTS
HNS Account Location: East US 2
NonHNS Account Location: East US 2, Central US
```
HNS OAuth
[WARNING] Tests run: 96, Failures: 0, Errors: 0, Skipped: 1
[WARNING] Tests run: 504, Failures: 0, Errors: 0, Skipped: 68
[WARNING] Tests run: 261, Failures: 0, Errors: 0, Skipped: 48
HNS SharedKey
[WARNING] Tests run: 96, Failures: 0, Errors: 0, Skipped: 2
[WARNING] Tests run: 500, Failures: 0, Errors: 0, Skipped: 24
[WARNING] Tests run: 261, Failures: 0, Errors: 0, Skipped: 40
Non-HNS SharedKey
[WARNING] Tests run: 96, Failures: 0, Errors: 0, Skipped: 2
[WARNING] Tests run: 504, Failures: 0, Errors: 0, Skipped: 249
[WARNING] Tests run: 261, Failures: 0, Errors: 0, Skipped: 40
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #2520: HADOOP-17290. ABFS: Add Identifiers to Client Request Header
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on pull request #2520:
URL: https://github.com/apache/hadoop/pull/2520#issuecomment-850982108
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 51s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | markdownlint | 0m 1s | | markdownlint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 37 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 33m 14s | | trunk passed |
| +1 :green_heart: | compile | 0m 36s | | trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 0m 34s | | trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | checkstyle | 0m 24s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 38s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 30s | | trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 0m 26s | | trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 0m 59s | | trunk passed |
| +1 :green_heart: | shadedclient | 16m 34s | | branch has no errors when building and testing our client artifacts. |
| -0 :warning: | patch | 16m 51s | | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 28s | | the patch passed |
| +1 :green_heart: | compile | 0m 29s | | the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javac | 0m 29s | | the patch passed |
| +1 :green_heart: | compile | 0m 26s | | the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | javac | 0m 26s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| +1 :green_heart: | checkstyle | 0m 16s | | the patch passed |
| +1 :green_heart: | mvnsite | 0m 30s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 22s | | the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 0m 19s | | the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 1m 1s | | the patch passed |
| +1 :green_heart: | shadedclient | 16m 36s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 1m 58s | | hadoop-azure in the patch passed. |
| +1 :green_heart: | asflicense | 0m 30s | | The patch does not generate ASF License warnings. |
| | | 79m 2s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2520/18/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/2520 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell markdownlint |
| uname | Linux 25dff7969f14 4.15.0-136-generic #140-Ubuntu SMP Thu Jan 28 05:20:47 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / d0b33c151b9331b9f542aa2a94d48b1a25e14d8c |
| Default Java | Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2520/18/testReport/ |
| Max. process+thread count | 574 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2520/18/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #2520: HADOOP-17290. ABFS: Add Identifiers to Client Request Header
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on pull request #2520:
URL: https://github.com/apache/hadoop/pull/2520#issuecomment-850978636
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 48s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | markdownlint | 0m 1s | | markdownlint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 37 new or modified test files. |
|||| _ trunk Compile Tests _ |
| -1 :x: | mvninstall | 41m 51s | [/branch-mvninstall-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2520/17/artifact/out/branch-mvninstall-root.txt) | root in trunk failed. |
| -1 :x: | compile | 0m 29s | [/branch-compile-hadoop-tools_hadoop-azure-jdkUbuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2520/17/artifact/out/branch-compile-hadoop-tools_hadoop-azure-jdkUbuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04.txt) | hadoop-azure in trunk failed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04. |
| -1 :x: | compile | 0m 29s | [/branch-compile-hadoop-tools_hadoop-azure-jdkPrivateBuild-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2520/17/artifact/out/branch-compile-hadoop-tools_hadoop-azure-jdkPrivateBuild-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08.txt) | hadoop-azure in trunk failed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08. |
| -0 :warning: | checkstyle | 0m 26s | [/buildtool-branch-checkstyle-hadoop-tools_hadoop-azure.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2520/17/artifact/out/buildtool-branch-checkstyle-hadoop-tools_hadoop-azure.txt) | The patch fails to run checkstyle in hadoop-azure |
| -1 :x: | mvnsite | 0m 28s | [/branch-mvnsite-hadoop-tools_hadoop-azure.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2520/17/artifact/out/branch-mvnsite-hadoop-tools_hadoop-azure.txt) | hadoop-azure in trunk failed. |
| -1 :x: | javadoc | 0m 29s | [/branch-javadoc-hadoop-tools_hadoop-azure-jdkUbuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2520/17/artifact/out/branch-javadoc-hadoop-tools_hadoop-azure-jdkUbuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04.txt) | hadoop-azure in trunk failed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04. |
| -1 :x: | javadoc | 0m 29s | [/branch-javadoc-hadoop-tools_hadoop-azure-jdkPrivateBuild-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2520/17/artifact/out/branch-javadoc-hadoop-tools_hadoop-azure-jdkPrivateBuild-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08.txt) | hadoop-azure in trunk failed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08. |
| -1 :x: | spotbugs | 0m 29s | [/branch-spotbugs-hadoop-tools_hadoop-azure.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2520/17/artifact/out/branch-spotbugs-hadoop-tools_hadoop-azure.txt) | hadoop-azure in trunk failed. |
| +1 :green_heart: | shadedclient | 2m 59s | | branch has no errors when building and testing our client artifacts. |
| -0 :warning: | patch | 3m 26s | | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. |
|||| _ Patch Compile Tests _ |
| -1 :x: | mvninstall | 0m 22s | [/patch-mvninstall-hadoop-tools_hadoop-azure.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2520/17/artifact/out/patch-mvninstall-hadoop-tools_hadoop-azure.txt) | hadoop-azure in the patch failed. |
| -1 :x: | compile | 0m 22s | [/patch-compile-hadoop-tools_hadoop-azure-jdkUbuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2520/17/artifact/out/patch-compile-hadoop-tools_hadoop-azure-jdkUbuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04.txt) | hadoop-azure in the patch failed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04. |
| -1 :x: | javac | 0m 22s | [/patch-compile-hadoop-tools_hadoop-azure-jdkUbuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2520/17/artifact/out/patch-compile-hadoop-tools_hadoop-azure-jdkUbuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04.txt) | hadoop-azure in the patch failed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04. |
| -1 :x: | compile | 0m 23s | [/patch-compile-hadoop-tools_hadoop-azure-jdkPrivateBuild-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2520/17/artifact/out/patch-compile-hadoop-tools_hadoop-azure-jdkPrivateBuild-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08.txt) | hadoop-azure in the patch failed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08. |
| -1 :x: | javac | 0m 23s | [/patch-compile-hadoop-tools_hadoop-azure-jdkPrivateBuild-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2520/17/artifact/out/patch-compile-hadoop-tools_hadoop-azure-jdkPrivateBuild-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08.txt) | hadoop-azure in the patch failed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08. |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| -0 :warning: | checkstyle | 0m 21s | [/buildtool-patch-checkstyle-hadoop-tools_hadoop-azure.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2520/17/artifact/out/buildtool-patch-checkstyle-hadoop-tools_hadoop-azure.txt) | The patch fails to run checkstyle in hadoop-azure |
| -1 :x: | mvnsite | 0m 22s | [/patch-mvnsite-hadoop-tools_hadoop-azure.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2520/17/artifact/out/patch-mvnsite-hadoop-tools_hadoop-azure.txt) | hadoop-azure in the patch failed. |
| -1 :x: | javadoc | 0m 22s | [/patch-javadoc-hadoop-tools_hadoop-azure-jdkUbuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2520/17/artifact/out/patch-javadoc-hadoop-tools_hadoop-azure-jdkUbuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04.txt) | hadoop-azure in the patch failed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04. |
| -1 :x: | javadoc | 0m 23s | [/patch-javadoc-hadoop-tools_hadoop-azure-jdkPrivateBuild-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2520/17/artifact/out/patch-javadoc-hadoop-tools_hadoop-azure-jdkPrivateBuild-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08.txt) | hadoop-azure in the patch failed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08. |
| -1 :x: | spotbugs | 0m 22s | [/patch-spotbugs-hadoop-tools_hadoop-azure.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2520/17/artifact/out/patch-spotbugs-hadoop-tools_hadoop-azure.txt) | hadoop-azure in the patch failed. |
| +1 :green_heart: | shadedclient | 4m 6s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| -1 :x: | unit | 0m 23s | [/patch-unit-hadoop-tools_hadoop-azure.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2520/17/artifact/out/patch-unit-hadoop-tools_hadoop-azure.txt) | hadoop-azure in the patch failed. |
| +0 :ok: | asflicense | 0m 23s | | ASF License check generated no output? |
| | | 54m 50s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2520/17/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/2520 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell markdownlint |
| uname | Linux 922d72e4eb9d 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / d0b33c151b9331b9f542aa2a94d48b1a25e14d8c |
| Default Java | Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2520/17/testReport/ |
| Max. process+thread count | 94 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2520/17/console |
| versions | git=2.25.1 maven=3.6.3 |
| Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] sumangala-patki commented on a change in pull request #2520: HADOOP-17290. ABFS: Add Identifiers to Client Request Header
Posted by GitBox <gi...@apache.org>.
sumangala-patki commented on a change in pull request #2520:
URL: https://github.com/apache/hadoop/pull/2520#discussion_r658209239
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AbfsConfiguration.java
##########
@@ -264,6 +266,10 @@
DefaultValue = DEFAULT_VALUE_UNKNOWN)
private String clusterType;
+ @StringConfigurationValidatorAnnotation(ConfigurationKey = FS_AZURE_CLIENT_CORRELATIONID,
+ DefaultValue = EMPTY_STRING)
+ private String clientCorrelationID;
Review comment:
renamed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] sumangala-patki commented on a change in pull request #2520: HADOOP-17290. ABFS: Add Identifiers to Client Request Header
Posted by GitBox <gi...@apache.org>.
sumangala-patki commented on a change in pull request #2520:
URL: https://github.com/apache/hadoop/pull/2520#discussion_r658212957
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsRestOperation.java
##########
@@ -221,17 +226,27 @@ private void completeExecute() throws AzureBlobFileSystemException {
LOG.trace("{} REST operation complete", operationType);
}
+ private void updateClientRequestHeader(AbfsHttpOperation httpOperation,
+ TracingContext tracingContext) {
+ tracingContext.generateClientRequestID();
Review comment:
Generating the clientRequestId in the toString() method of tracingContext is not a good option as it will re-generate it every time the method is called. Also, we need tracingContext to store the id for consistency
Have renamed toString to constructHeader. Format is passed over config and already available to tracingContext object
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] sumangala-patki commented on a change in pull request #2520: HADOOP-17290. ABFS: Add Identifiers to Client Request Header
Posted by GitBox <gi...@apache.org>.
sumangala-patki commented on a change in pull request #2520:
URL: https://github.com/apache/hadoop/pull/2520#discussion_r548383155
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AzureBlobFileSystem.java
##########
@@ -1202,6 +1287,10 @@ public String getCanonicalServiceName() {
return this.statistics;
}
+ public void setListenerOperation(String operation) {
Review comment:
done
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #2520: HADOOP-17290. ABFS: Add Identifiers to Client Request Header
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on pull request #2520:
URL: https://github.com/apache/hadoop/pull/2520#issuecomment-826757918
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 38s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | markdownlint | 0m 1s | | markdownlint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 36 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 34m 21s | | trunk passed |
| +1 :green_heart: | compile | 0m 39s | | trunk passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | compile | 0m 36s | | trunk passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | checkstyle | 0m 27s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 39s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 33s | | trunk passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 0m 30s | | trunk passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | spotbugs | 0m 58s | | trunk passed |
| +1 :green_heart: | shadedclient | 13m 59s | | branch has no errors when building and testing our client artifacts. |
| -0 :warning: | patch | 14m 18s | | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 28s | | the patch passed |
| +1 :green_heart: | compile | 0m 29s | | the patch passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javac | 0m 29s | | the patch passed |
| +1 :green_heart: | compile | 0m 26s | | the patch passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | javac | 0m 26s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| +1 :green_heart: | checkstyle | 0m 18s | | the patch passed |
| +1 :green_heart: | mvnsite | 0m 31s | | the patch passed |
| -1 :x: | javadoc | 0m 23s | [/results-javadoc-javadoc-hadoop-tools_hadoop-azure-jdkUbuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2520/15/artifact/out/results-javadoc-javadoc-hadoop-tools_hadoop-azure-jdkUbuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04.txt) | hadoop-tools_hadoop-azure-jdkUbuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 generated 1 new + 15 unchanged - 0 fixed = 16 total (was 15) |
| -1 :x: | javadoc | 0m 21s | [/results-javadoc-javadoc-hadoop-tools_hadoop-azure-jdkPrivateBuild-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2520/15/artifact/out/results-javadoc-javadoc-hadoop-tools_hadoop-azure-jdkPrivateBuild-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08.txt) | hadoop-tools_hadoop-azure-jdkPrivateBuild-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 generated 1 new + 15 unchanged - 0 fixed = 16 total (was 15) |
| +1 :green_heart: | spotbugs | 0m 59s | | the patch passed |
| +1 :green_heart: | shadedclient | 13m 47s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 2m 9s | | hadoop-azure in the patch passed. |
| +1 :green_heart: | asflicense | 0m 33s | | The patch does not generate ASF License warnings. |
| | | 75m 26s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2520/15/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/2520 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell markdownlint |
| uname | Linux 25e58210a1db 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / fd631219582b5bc6db47b69a4aa496682ac335a4 |
| Default Java | Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2520/15/testReport/ |
| Max. process+thread count | 541 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2520/15/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #2520: HADOOP-17290. ABFS: Add Identifiers to Client Request Header
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on pull request #2520:
URL: https://github.com/apache/hadoop/pull/2520#issuecomment-826595085
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 36s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | markdownlint | 0m 0s | | markdownlint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 36 new or modified test files. |
|||| _ trunk Compile Tests _ |
| -1 :x: | mvninstall | 2m 22s | [/branch-mvninstall-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2520/14/artifact/out/branch-mvninstall-root.txt) | root in trunk failed. |
| -1 :x: | compile | 2m 31s | [/branch-compile-hadoop-tools_hadoop-azure-jdkUbuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2520/14/artifact/out/branch-compile-hadoop-tools_hadoop-azure-jdkUbuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04.txt) | hadoop-azure in trunk failed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04. |
| -1 :x: | compile | 0m 15s | [/branch-compile-hadoop-tools_hadoop-azure-jdkPrivateBuild-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2520/14/artifact/out/branch-compile-hadoop-tools_hadoop-azure-jdkPrivateBuild-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08.txt) | hadoop-azure in trunk failed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08. |
| +1 :green_heart: | checkstyle | 0m 20s | | trunk passed |
| -1 :x: | mvnsite | 0m 18s | [/branch-mvnsite-hadoop-tools_hadoop-azure.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2520/14/artifact/out/branch-mvnsite-hadoop-tools_hadoop-azure.txt) | hadoop-azure in trunk failed. |
| +1 :green_heart: | javadoc | 0m 24s | | trunk passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 0m 22s | | trunk passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| -1 :x: | spotbugs | 0m 15s | [/branch-spotbugs-hadoop-tools_hadoop-azure.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2520/14/artifact/out/branch-spotbugs-hadoop-tools_hadoop-azure.txt) | hadoop-azure in trunk failed. |
| +1 :green_heart: | shadedclient | 19m 33s | | branch has no errors when building and testing our client artifacts. |
| -0 :warning: | patch | 19m 51s | | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. |
|||| _ Patch Compile Tests _ |
| -1 :x: | mvninstall | 0m 15s | [/patch-mvninstall-hadoop-tools_hadoop-azure.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2520/14/artifact/out/patch-mvninstall-hadoop-tools_hadoop-azure.txt) | hadoop-azure in the patch failed. |
| -1 :x: | compile | 0m 14s | [/patch-compile-hadoop-tools_hadoop-azure-jdkUbuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2520/14/artifact/out/patch-compile-hadoop-tools_hadoop-azure-jdkUbuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04.txt) | hadoop-azure in the patch failed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04. |
| -1 :x: | javac | 0m 14s | [/patch-compile-hadoop-tools_hadoop-azure-jdkUbuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2520/14/artifact/out/patch-compile-hadoop-tools_hadoop-azure-jdkUbuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04.txt) | hadoop-azure in the patch failed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04. |
| -1 :x: | compile | 0m 15s | [/patch-compile-hadoop-tools_hadoop-azure-jdkPrivateBuild-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2520/14/artifact/out/patch-compile-hadoop-tools_hadoop-azure-jdkPrivateBuild-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08.txt) | hadoop-azure in the patch failed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08. |
| -1 :x: | javac | 0m 15s | [/patch-compile-hadoop-tools_hadoop-azure-jdkPrivateBuild-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2520/14/artifact/out/patch-compile-hadoop-tools_hadoop-azure-jdkPrivateBuild-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08.txt) | hadoop-azure in the patch failed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08. |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| -0 :warning: | checkstyle | 0m 18s | [/results-checkstyle-hadoop-tools_hadoop-azure.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2520/14/artifact/out/results-checkstyle-hadoop-tools_hadoop-azure.txt) | hadoop-tools/hadoop-azure: The patch generated 2 new + 17 unchanged - 0 fixed = 19 total (was 17) |
| -1 :x: | mvnsite | 0m 16s | [/patch-mvnsite-hadoop-tools_hadoop-azure.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2520/14/artifact/out/patch-mvnsite-hadoop-tools_hadoop-azure.txt) | hadoop-azure in the patch failed. |
| -1 :x: | javadoc | 0m 22s | [/results-javadoc-javadoc-hadoop-tools_hadoop-azure-jdkUbuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2520/14/artifact/out/results-javadoc-javadoc-hadoop-tools_hadoop-azure-jdkUbuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04.txt) | hadoop-tools_hadoop-azure-jdkUbuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 generated 1 new + 15 unchanged - 0 fixed = 16 total (was 15) |
| -1 :x: | javadoc | 0m 20s | [/results-javadoc-javadoc-hadoop-tools_hadoop-azure-jdkPrivateBuild-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2520/14/artifact/out/results-javadoc-javadoc-hadoop-tools_hadoop-azure-jdkPrivateBuild-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08.txt) | hadoop-tools_hadoop-azure-jdkPrivateBuild-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 generated 1 new + 15 unchanged - 0 fixed = 16 total (was 15) |
| -1 :x: | spotbugs | 0m 15s | [/patch-spotbugs-hadoop-tools_hadoop-azure.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2520/14/artifact/out/patch-spotbugs-hadoop-tools_hadoop-azure.txt) | hadoop-azure in the patch failed. |
| +1 :green_heart: | shadedclient | 15m 29s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| -1 :x: | unit | 0m 16s | [/patch-unit-hadoop-tools_hadoop-azure.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2520/14/artifact/out/patch-unit-hadoop-tools_hadoop-azure.txt) | hadoop-azure in the patch failed. |
| +1 :green_heart: | asflicense | 0m 34s | | The patch does not generate ASF License warnings. |
| | | 44m 16s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2520/14/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/2520 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell markdownlint |
| uname | Linux 3250296d8043 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / c9217d9492486f401cbcb9320e8d473f98595c1b |
| Default Java | Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2520/14/testReport/ |
| Max. process+thread count | 719 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2520/14/console |
| versions | git=2.25.1 maven=3.6.3 |
| Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #2520: HADOOP-17290. ABFS: Add Identifiers to Client Request Header
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on pull request #2520:
URL: https://github.com/apache/hadoop/pull/2520#issuecomment-872998551
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 58s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 2s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | markdownlint | 0m 0s | | markdownlint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 37 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 30m 40s | | trunk passed |
| +1 :green_heart: | compile | 0m 39s | | trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 0m 36s | | trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | checkstyle | 0m 30s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 42s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 33s | | trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 0m 31s | | trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 1m 1s | | trunk passed |
| +1 :green_heart: | shadedclient | 13m 58s | | branch has no errors when building and testing our client artifacts. |
| -0 :warning: | patch | 14m 17s | | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 30s | | the patch passed |
| +1 :green_heart: | compile | 0m 31s | | the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javac | 0m 31s | | the patch passed |
| +1 :green_heart: | compile | 0m 27s | | the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | javac | 0m 27s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| +1 :green_heart: | checkstyle | 0m 19s | | the patch passed |
| +1 :green_heart: | mvnsite | 0m 30s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 24s | | the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 0m 23s | | the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 1m 3s | | the patch passed |
| +1 :green_heart: | shadedclient | 13m 47s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 2m 3s | | hadoop-azure in the patch passed. |
| +1 :green_heart: | asflicense | 0m 34s | | The patch does not generate ASF License warnings. |
| | | 72m 18s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2520/24/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/2520 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell markdownlint |
| uname | Linux f7e3def8158c 4.15.0-65-generic #74-Ubuntu SMP Tue Sep 17 17:06:04 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 0dc83f4367c6d674a7d8f0bfba0498c26c4551e1 |
| Default Java | Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2520/24/testReport/ |
| Max. process+thread count | 615 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2520/24/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #2520: HADOOP-17290. ABFS: Add Identifiers to Client Request Header
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on pull request #2520:
URL: https://github.com/apache/hadoop/pull/2520#issuecomment-820684101
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 34s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | markdownlint | 0m 0s | | markdownlint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 35 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 33m 47s | | trunk passed |
| +1 :green_heart: | compile | 0m 37s | | trunk passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | compile | 0m 33s | | trunk passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | checkstyle | 0m 28s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 39s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 33s | | trunk passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 0m 30s | | trunk passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | spotbugs | 1m 0s | | trunk passed |
| +1 :green_heart: | shadedclient | 14m 5s | | branch has no errors when building and testing our client artifacts. |
| -0 :warning: | patch | 14m 24s | | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 29s | | the patch passed |
| +1 :green_heart: | compile | 0m 30s | | the patch passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javac | 0m 30s | | the patch passed |
| +1 :green_heart: | compile | 0m 25s | | the patch passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | javac | 0m 25s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| -0 :warning: | checkstyle | 0m 18s | [/results-checkstyle-hadoop-tools_hadoop-azure.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2520/13/artifact/out/results-checkstyle-hadoop-tools_hadoop-azure.txt) | hadoop-tools/hadoop-azure: The patch generated 2 new + 17 unchanged - 0 fixed = 19 total (was 17) |
| +1 :green_heart: | mvnsite | 0m 29s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 21s | | the patch passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 0m 21s | | the patch passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | spotbugs | 1m 1s | | the patch passed |
| +1 :green_heart: | shadedclient | 14m 40s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 1m 59s | | hadoop-azure in the patch passed. |
| +1 :green_heart: | asflicense | 0m 34s | | The patch does not generate ASF License warnings. |
| | | 75m 34s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2520/13/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/2520 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell markdownlint |
| uname | Linux d5ae3a69a668 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 3447977af7444bdd033ba091124bc70b130954e3 |
| Default Java | Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2520/13/testReport/ |
| Max. process+thread count | 545 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2520/13/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus removed a comment on pull request #2520: HADOOP-17290. ABFS: Add Identifiers to Client Request Header
Posted by GitBox <gi...@apache.org>.
hadoop-yetus removed a comment on pull request #2520:
URL: https://github.com/apache/hadoop/pull/2520#issuecomment-872998551
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 58s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 2s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | markdownlint | 0m 0s | | markdownlint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 37 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 30m 40s | | trunk passed |
| +1 :green_heart: | compile | 0m 39s | | trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 0m 36s | | trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | checkstyle | 0m 30s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 42s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 33s | | trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 0m 31s | | trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 1m 1s | | trunk passed |
| +1 :green_heart: | shadedclient | 13m 58s | | branch has no errors when building and testing our client artifacts. |
| -0 :warning: | patch | 14m 17s | | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 30s | | the patch passed |
| +1 :green_heart: | compile | 0m 31s | | the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javac | 0m 31s | | the patch passed |
| +1 :green_heart: | compile | 0m 27s | | the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | javac | 0m 27s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| +1 :green_heart: | checkstyle | 0m 19s | | the patch passed |
| +1 :green_heart: | mvnsite | 0m 30s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 24s | | the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 0m 23s | | the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 1m 3s | | the patch passed |
| +1 :green_heart: | shadedclient | 13m 47s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 2m 3s | | hadoop-azure in the patch passed. |
| +1 :green_heart: | asflicense | 0m 34s | | The patch does not generate ASF License warnings. |
| | | 72m 18s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2520/24/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/2520 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell markdownlint |
| uname | Linux f7e3def8158c 4.15.0-65-generic #74-Ubuntu SMP Tue Sep 17 17:06:04 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 0dc83f4367c6d674a7d8f0bfba0498c26c4551e1 |
| Default Java | Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2520/24/testReport/ |
| Max. process+thread count | 615 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2520/24/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus removed a comment on pull request #2520: HADOOP-17290. ABFS: Add Identifiers to Client Request Header
Posted by GitBox <gi...@apache.org>.
hadoop-yetus removed a comment on pull request #2520:
URL: https://github.com/apache/hadoop/pull/2520#issuecomment-870058533
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] anoopsjohn commented on a change in pull request #2520: HADOOP-17290. ABFS: Add Identifiers to Client Request Header
Posted by GitBox <gi...@apache.org>.
anoopsjohn commented on a change in pull request #2520:
URL: https://github.com/apache/hadoop/pull/2520#discussion_r657346332
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AzureBlobFileSystem.java
##########
@@ -1071,7 +1155,10 @@ private boolean fileSystemExists() throws IOException {
LOG.debug(
"AzureBlobFileSystem.fileSystemExists uri: {}", uri);
try {
- abfsStore.getFilesystemProperties();
+ TracingContext tracingContext = new TracingContext(clientCorrelationID,
+ fileSystemID, HdfsOperationConstants.GET_FILESTATUS,
Review comment:
GET_FILESTATUS op?
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AbfsConfiguration.java
##########
@@ -264,6 +266,10 @@
DefaultValue = DEFAULT_VALUE_UNKNOWN)
private String clusterType;
+ @StringConfigurationValidatorAnnotation(ConfigurationKey = FS_AZURE_CLIENT_CORRELATIONID,
+ DefaultValue = EMPTY_STRING)
+ private String clientCorrelationID;
Review comment:
clientCorrelationId ? To be similar as 'userAgentId' etc? And the getter also
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AzureBlobFileSystem.java
##########
@@ -335,7 +361,10 @@ public boolean rename(final Path src, final Path dst) throws IOException {
}
// Non-HNS account need to check dst status on driver side.
- if (!abfsStore.getIsNamespaceEnabled() && dstFileStatus == null) {
+ TracingContext tracingContext = new TracingContext(clientCorrelationID,
+ fileSystemID, HdfsOperationConstants.RENAME, true, tracingContextFormat,
+ listener);
+ if (!abfsStore.getIsNamespaceEnabled(tracingContext) && dstFileStatus == null) {
Review comment:
Within tryGetFileStatus() there is call to getFileStatus. We should be using this context created here.
tryGetFileStatus() been called by createNonRecursive API also.
Have to handle these.
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AzureBlobFileSystem.java
##########
@@ -1049,8 +1130,11 @@ public boolean exists(Path f) throws IOException {
throws IOException {
LOG.debug("AzureBlobFileSystem.listStatusIterator path : {}", path);
if (abfsStore.getAbfsConfiguration().enableAbfsListIterator()) {
+ TracingContext tracingContext = new TracingContext(clientCorrelationID,
+ fileSystemID, HdfsOperationConstants.LISTSTATUS, true,
+ tracingContextFormat, listener);
AbfsListStatusRemoteIterator abfsLsItr =
- new AbfsListStatusRemoteIterator(getFileStatus(path), abfsStore);
+ new AbfsListStatusRemoteIterator(getFileStatus(path), abfsStore, tracingContext);
Review comment:
Here is a call to getFileStatus(Path) but as part of LISTSTATUS op. So we should the context created above , during getFileStatus also right?
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsInputStream.java
##########
@@ -451,15 +472,15 @@ private int readInternal(final long position, final byte[] b, final int offset,
}
// got nothing from read-ahead, do our own read now
- receivedBytes = readRemote(position, b, offset, length);
+ receivedBytes = readRemote(position, b, offset, length, new TracingContext(tracingContext));
return receivedBytes;
} else {
LOG.debug("read ahead disabled, reading remote");
- return readRemote(position, b, offset, length);
+ return readRemote(position, b, offset, length, new TracingContext(tracingContext));
}
}
- int readRemote(long position, byte[] b, int offset, int length) throws IOException {
+ int readRemote(long position, byte[] b, int offset, int length, TracingContext tracingContext) throws IOException {
Review comment:
Why passing 'tracingContext' when its set as instance member?
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/constants/ConfigurationKeys.java
##########
@@ -109,6 +109,12 @@
* Default value of this config is true. **/
public static final String FS_AZURE_DISABLE_OUTPUTSTREAM_FLUSH = "fs.azure.disable.outputstream.flush";
public static final String FS_AZURE_USER_AGENT_PREFIX_KEY = "fs.azure.user.agent.prefix";
+ /**
+ * The client correlation ID provided over config that will be added to
+ * x-ms-client-request-Id header. Defaults to empty string if the length and
+ * character constraints are not satisfied. **/
+ public static final String FS_AZURE_CLIENT_CORRELATIONID = "fs.azure.client.correlationid";
+ public static final String FS_AZURE_TRACINGCONTEXT_FORMAT = "fs.azure.tracingcontext.format";
Review comment:
This is the tracing header format right? Will that be a better name?
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsHttpOperation.java
##########
@@ -166,6 +166,11 @@ public String getResponseHeader(String httpHeader) {
return connection.getHeaderField(httpHeader);
}
+ @VisibleForTesting
+ public String getRequestHeader(String httpHeader) {
+ return connection.getRequestProperties().get(httpHeader).toString();
Review comment:
Only used for tests? Should be returning List only?
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsRestOperation.java
##########
@@ -221,17 +226,27 @@ private void completeExecute() throws AzureBlobFileSystemException {
LOG.trace("{} REST operation complete", operationType);
}
+ private void updateClientRequestHeader(AbfsHttpOperation httpOperation,
+ TracingContext tracingContext) {
+ tracingContext.generateClientRequestID();
Review comment:
Can avoid this way of explicit call which for sure should get executed before setting of this header.
Can we have a better method name than toString() for generation of required header value? This has to consider the format and generate.
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AzureBlobFileSystem.java
##########
@@ -1049,8 +1130,11 @@ public boolean exists(Path f) throws IOException {
throws IOException {
LOG.debug("AzureBlobFileSystem.listStatusIterator path : {}", path);
if (abfsStore.getAbfsConfiguration().enableAbfsListIterator()) {
+ TracingContext tracingContext = new TracingContext(clientCorrelationID,
+ fileSystemID, HdfsOperationConstants.LISTSTATUS, true,
+ tracingContextFormat, listener);
AbfsListStatusRemoteIterator abfsLsItr =
- new AbfsListStatusRemoteIterator(getFileStatus(path), abfsStore);
+ new AbfsListStatusRemoteIterator(getFileStatus(path), abfsStore, tracingContext);
return RemoteIterators.typeCastingRemoteIterator(abfsLsItr);
} else {
return super.listStatusIterator(path);
Review comment:
Even this call to FileSystem#listStatusIterator() will create ADL gen2 calls right? So should there be way for having a single context(above created) to be used for call from there too?
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AzureBlobFileSystem.java
##########
@@ -1283,6 +1373,11 @@ public String getCanonicalServiceName() {
return this.statistics;
}
+ @VisibleForTesting
+ void setListenerOperation(String operation) {
+ listener.setOperation(operation);
Review comment:
What is this operation been set on Listener?
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/constants/HdfsOperationConstants.java
##########
@@ -0,0 +1,50 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.azurebfs.constants;
+
+public final class HdfsOperationConstants {
Review comment:
Can we avoid the hdfs in this ? Its after all Hadoop common FS API types
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AzureBlobFileSystem.java
##########
@@ -111,10 +116,14 @@
private Path workingDir;
private AzureBlobFileSystemStore abfsStore;
private boolean isClosed;
+ private final String fileSystemID = UUID.randomUUID().toString();
Review comment:
All places ID to Id?
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/constants/HdfsOperationConstants.java
##########
@@ -0,0 +1,50 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.azurebfs.constants;
+
+public final class HdfsOperationConstants {
Review comment:
Can be Enum?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] sumangala-patki commented on a change in pull request #2520: HADOOP-17290. ABFS: Add Identifiers to Client Request Header
Posted by GitBox <gi...@apache.org>.
sumangala-patki commented on a change in pull request #2520:
URL: https://github.com/apache/hadoop/pull/2520#discussion_r658213074
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsInputStream.java
##########
@@ -451,15 +472,15 @@ private int readInternal(final long position, final byte[] b, final int offset,
}
// got nothing from read-ahead, do our own read now
- receivedBytes = readRemote(position, b, offset, length);
+ receivedBytes = readRemote(position, b, offset, length, new TracingContext(tracingContext));
return receivedBytes;
} else {
LOG.debug("read ahead disabled, reading remote");
- return readRemote(position, b, offset, length);
+ return readRemote(position, b, offset, length, new TracingContext(tracingContext));
}
}
- int readRemote(long position, byte[] b, int offset, int length) throws IOException {
+ int readRemote(long position, byte[] b, int offset, int length, TracingContext tracingContext) throws IOException {
Review comment:
The readRemote function is also used by the readahead worker threads (in ReadBufferWorker). The tracingContext passed from the readahead buffers may correspond to independent read requests
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] sumangala-patki commented on pull request #2520: HADOOP-17290. ABFS: Add Identifiers to Client Request Header
Posted by GitBox <gi...@apache.org>.
sumangala-patki commented on pull request #2520:
URL: https://github.com/apache/hadoop/pull/2520#issuecomment-871438512
TEST RESULTS
HNS Account Location: East US 2
NonHNS Account Location: East US 2, Central US
```
HNS OAuth
[WARNING] Tests run: 102, Failures: 0, Errors: 0, Skipped: 1
[ERROR] Failures:
[ERROR] ITestAzureBlobFileSystemFileStatus.testLastModifiedTime:140->Assert.assertTrue:42->Assert.fail:89 lastModifiedTime should be before createEndTime
[ERROR] Errors:
[ERROR] ITestAzureBlobFileSystemLease.testWriteAfterBreakLease:240 » TestTimedOut test...
[ERROR] Tests run: 558, Failures: 1, Errors: 1, Skipped: 98
[ERROR] Errors:
[ERROR] ITestAbfsFileSystemContractSecureDistCp>AbstractContractDistCpTest.testDistCpWithIterator:635 » TestTimedOut
[ERROR] Tests run: 269, Failures: 0, Errors: 1, Skipped: 52
HNS SharedKey
[WARNING] Tests run: 102, Failures: 0, Errors: 0, Skipped: 2
[ERROR] Failures:
[ERROR] ITestAzureBlobFileSystemFileStatus.testLastModifiedTime:140->Assert.assertTrue:42->Assert.fail:89 lastModifiedTime should be before createEndTime
[ERROR] Tests run: 558, Failures: 1, Errors: 0, Skipped: 54
[ERROR] Errors:
[ERROR] ITestAbfsFileSystemContractDistCp>AbstractContractDistCpTest.testDistCpWithIterator:635 » TestTimedOut
[ERROR] ITestAbfsFileSystemContractSecureDistCp>AbstractContractDistCpTest.testDistCpWithIterator:635 » TestTimedOut
[ERROR] Tests run: 269, Failures: 0, Errors: 2, Skipped: 40
NonHNS SharedKey
[WARNING] Tests run: 102, Failures: 0, Errors: 0, Skipped: 2
[ERROR] Failures:
[ERROR] ITestAzureBlobFileSystemFileStatus.testLastModifiedTime:140->Assert.assertTrue:42->Assert.fail:89 lastModifiedTime should be before createEndTime
[ERROR] Tests run: 558, Failures: 1, Errors: 0, Skipped: 276
[ERROR] Errors:
[ERROR] ITestAbfsFileSystemContractDistCp>AbstractContractDistCpTest.testDistCpWithIterator:635 » TestTimedOut
[ERROR] ITestAbfsFileSystemContractSecureDistCp>AbstractContractDistCpTest.testDistCpWithIterator:635 » TestTimedOut
[ERROR] Tests run: 269, Failures: 0, Errors: 2, Skipped: 40
```
Existing JIRAs to track failures: [LMT](https://issues.apache.org/jira/browse/HADOOP-17640), [ContractDistCp](https://issues.apache.org/jira/browse/HADOOP-17628)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #2520: HADOOP-17290. ABFS: Add Identifiers to Client Request Header
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on pull request #2520:
URL: https://github.com/apache/hadoop/pull/2520#issuecomment-870058533
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 34s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | markdownlint | 0m 0s | | markdownlint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 37 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 33m 28s | | trunk passed |
| +1 :green_heart: | compile | 0m 40s | | trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 0m 35s | | trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | checkstyle | 0m 26s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 40s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 32s | | trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 0m 31s | | trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 1m 3s | | trunk passed |
| +1 :green_heart: | shadedclient | 16m 23s | | branch has no errors when building and testing our client artifacts. |
| -0 :warning: | patch | 16m 43s | | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 32s | | the patch passed |
| +1 :green_heart: | compile | 0m 33s | | the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javac | 0m 33s | | the patch passed |
| +1 :green_heart: | compile | 0m 28s | | the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | javac | 0m 28s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| -0 :warning: | checkstyle | 0m 18s | [/results-checkstyle-hadoop-tools_hadoop-azure.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2520/21/artifact/out/results-checkstyle-hadoop-tools_hadoop-azure.txt) | hadoop-tools/hadoop-azure: The patch generated 2 new + 17 unchanged - 0 fixed = 19 total (was 17) |
| +1 :green_heart: | mvnsite | 0m 34s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 25s | | the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 0m 22s | | the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 1m 9s | | the patch passed |
| +1 :green_heart: | shadedclient | 16m 14s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 2m 12s | | hadoop-azure in the patch passed. |
| +1 :green_heart: | asflicense | 0m 33s | | The patch does not generate ASF License warnings. |
| | | 79m 47s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2520/21/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/2520 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell markdownlint |
| uname | Linux 31624da3802c 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / ecee1809de89ed5b4fc76df79dd64895537aab99 |
| Default Java | Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2520/21/testReport/ |
| Max. process+thread count | 678 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2520/21/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #2520: HADOOP-17290. ABFS: Add Identifiers to Client Request Header
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on pull request #2520:
URL: https://github.com/apache/hadoop/pull/2520#issuecomment-820436221
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 1m 19s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | markdownlint | 0m 0s | | markdownlint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 35 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 36m 50s | | trunk passed |
| +1 :green_heart: | compile | 0m 39s | | trunk passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | compile | 0m 33s | | trunk passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | checkstyle | 0m 27s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 40s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 33s | | trunk passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 0m 30s | | trunk passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | spotbugs | 1m 2s | | trunk passed |
| +1 :green_heart: | shadedclient | 14m 51s | | branch has no errors when building and testing our client artifacts. |
| -0 :warning: | patch | 15m 10s | | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 29s | | the patch passed |
| +1 :green_heart: | compile | 0m 30s | | the patch passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javac | 0m 30s | | the patch passed |
| +1 :green_heart: | compile | 0m 27s | | the patch passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | javac | 0m 27s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| +1 :green_heart: | checkstyle | 0m 19s | | the patch passed |
| +1 :green_heart: | mvnsite | 0m 29s | | the patch passed |
| -1 :x: | javadoc | 0m 23s | [/results-javadoc-javadoc-hadoop-tools_hadoop-azure-jdkUbuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2520/12/artifact/out/results-javadoc-javadoc-hadoop-tools_hadoop-azure-jdkUbuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04.txt) | hadoop-tools_hadoop-azure-jdkUbuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 generated 1 new + 15 unchanged - 0 fixed = 16 total (was 15) |
| -1 :x: | javadoc | 0m 20s | [/results-javadoc-javadoc-hadoop-tools_hadoop-azure-jdkPrivateBuild-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2520/12/artifact/out/results-javadoc-javadoc-hadoop-tools_hadoop-azure-jdkPrivateBuild-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08.txt) | hadoop-tools_hadoop-azure-jdkPrivateBuild-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 generated 1 new + 15 unchanged - 0 fixed = 16 total (was 15) |
| +1 :green_heart: | spotbugs | 1m 4s | | the patch passed |
| +1 :green_heart: | shadedclient | 14m 24s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 2m 1s | | hadoop-azure in the patch passed. |
| +1 :green_heart: | asflicense | 0m 32s | | The patch does not generate ASF License warnings. |
| | | 79m 58s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2520/12/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/2520 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell markdownlint |
| uname | Linux f40da7e58c43 4.15.0-65-generic #74-Ubuntu SMP Tue Sep 17 17:06:04 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 302fc06772362b0d108f3aab42cc7b353d7a7f58 |
| Default Java | Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2520/12/testReport/ |
| Max. process+thread count | 539 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2520/12/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #2520: HADOOP-17290. ABFS: Add Identifiers to Client Request Header
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on pull request #2520:
URL: https://github.com/apache/hadoop/pull/2520#issuecomment-866736669
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 59s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | markdownlint | 0m 0s | | markdownlint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 37 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 34m 44s | | trunk passed |
| +1 :green_heart: | compile | 0m 46s | | trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 0m 40s | | trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | checkstyle | 0m 29s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 43s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 33s | | trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 0m 30s | | trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 1m 10s | | trunk passed |
| +1 :green_heart: | shadedclient | 19m 5s | | branch has no errors when building and testing our client artifacts. |
| -0 :warning: | patch | 19m 25s | | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 36s | | the patch passed |
| +1 :green_heart: | compile | 0m 37s | | the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javac | 0m 37s | | the patch passed |
| +1 :green_heart: | compile | 0m 30s | | the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | javac | 0m 30s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| +1 :green_heart: | checkstyle | 0m 18s | | the patch passed |
| +1 :green_heart: | mvnsite | 0m 33s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 24s | | the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 0m 24s | | the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 1m 9s | | the patch passed |
| +1 :green_heart: | shadedclient | 18m 51s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 2m 12s | | hadoop-azure in the patch passed. |
| +1 :green_heart: | asflicense | 0m 36s | | The patch does not generate ASF License warnings. |
| | | 87m 22s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2520/19/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/2520 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell markdownlint |
| uname | Linux faab5f080aa6 4.15.0-65-generic #74-Ubuntu SMP Tue Sep 17 17:06:04 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 6250d04f25d1efed187a0835e70f53415f0e1378 |
| Default Java | Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2520/19/testReport/ |
| Max. process+thread count | 715 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2520/19/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org