You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by GitBox <gi...@apache.org> on 2020/01/30 20:40:36 UTC
[GitHub] [hadoop] lukmajercak opened a new pull request #1825: HDFS-15151
Use TransmitFile for file to socket data transfer
lukmajercak opened a new pull request #1825: HDFS-15151 Use TransmitFile for file to socket data transfer
URL: https://github.com/apache/hadoop/pull/1825
## NOTICE
Please create an issue in ASF JIRA before opening a pull request,
and you need to set the title of the pull request which starts with
the corresponding JIRA issue number. (e.g. HADOOP-XXXXX. Fix a typo in YYY.)
For more details, please see https://cwiki.apache.org/confluence/display/HADOOP/How+To+Contribute
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] goiri commented on a change in pull request #1825:
HDFS-15151 Use TransmitFile for file to socket data transfer
Posted by GitBox <gi...@apache.org>.
goiri commented on a change in pull request #1825: HDFS-15151 Use TransmitFile for file to socket data transfer
URL: https://github.com/apache/hadoop/pull/1825#discussion_r373806641
##########
File path: hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/nativeio/NativeIO.c
##########
@@ -1341,6 +1350,68 @@ Java_org_apache_hadoop_io_nativeio_NativeIO_00024Windows_extendWorkingSetSize(
#endif
}
+jint
+fdval(JNIEnv *env, jobject fdo)
+{
+ return (*env)->GetIntField(env, fdo, fd_fdID);
+}
+
+jlong
+handleval(JNIEnv *env, jobject fdo)
+{
+ return (*env)->GetLongField(env, fdo, handle_fdID);
+}
+
+JNIEXPORT jlong JNICALL
+Java_org_apache_hadoop_io_nativeio_NativeIO_00024Windows_transmitFile(
+ JNIEnv *env, jobject jObject, jobject srcFD, jlong position, jlong count,
+ jobject dstFD)
+{
+#ifdef UNIX
+ THROW(env, "java/io/IOException",
+ "The function transmitFile is not supported on Unix");
+#endif
+
+#ifdef WINDOWS
+ jclass clazz;
+
+ HANDLE src;
+ HANDLE dst;
+ LARGE_INTEGER srcPosition;
+ DWORD numberOfBytesToWrite;
+ BOOL result;
+
+ clazz = (*env)->FindClass(env, "java/io/FileDescriptor");
+ fd_fdID = (*env)->GetFieldID(env, clazz, "fd", "I");
+ handle_fdID = (*env)->GetFieldID(env, clazz, "handle", "J");
+
+ src = (HANDLE)(handleval(env, srcFD));
+ dst = (SOCKET)(fdval(env, dstFD));
+
+ numberOfBytesToWrite = (DWORD)count;
+
+ srcPosition.QuadPart = position;
+ result = SetFilePointerEx(src, srcPosition, &srcPosition, FILE_BEGIN);
+ if (result == 0) {
+ throw_ioe(env, WSAGetLastError());
+ }
+
+ result = TransmitFile(
+ dst,
+ src,
+ numberOfBytesToWrite,
+ 0,
+ NULL,
+ NULL,
+ TF_USE_KERNEL_APC
+ );
+ if (!result) {
+ throw_ioe(env, WSAGetLastError());
+ }
+ return numberOfBytesToWrite;
+#endif
Review comment:
Anyway to rewrite this so we don't get:
NativeIO.c:1413:1: warning: control reaches end of non-void function [-Wreturn-type]
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] lukmajercak commented on a change in pull request #1825:
HDFS-15151 Use TransmitFile for file to socket data transfer
Posted by GitBox <gi...@apache.org>.
lukmajercak commented on a change in pull request #1825: HDFS-15151 Use TransmitFile for file to socket data transfer
URL: https://github.com/apache/hadoop/pull/1825#discussion_r373748571
##########
File path: hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/nativeio/NativeIO.c
##########
@@ -1341,6 +1350,68 @@ Java_org_apache_hadoop_io_nativeio_NativeIO_00024Windows_extendWorkingSetSize(
#endif
}
+jint
+fdval(JNIEnv *env, jobject fdo)
+{
+ return (*env)->GetIntField(env, fdo, fd_fdID);
+}
+
+jlong
+handleval(JNIEnv *env, jobject fdo)
+{
+ return (*env)->GetLongField(env, fdo, handle_fdID);
+}
+
+JNIEXPORT jlong JNICALL
+Java_org_apache_hadoop_io_nativeio_NativeIO_00024Windows_transmitFile(
+ JNIEnv *env, jobject jObject, jobject srcFD, jlong position, jlong count,
+ jobject dstFD)
+{
+#ifdef UNIX
+ THROW(env, "java/io/IOException",
+ "The function transmitFile is not supported on Unix");
+#endif
+
+#ifdef WINDOWS
Review comment:
i was following the way this was done just above for Windows_extendWorkingSetSize
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] lukmajercak commented on a change in pull request #1825:
HDFS-15151 Use TransmitFile for file to socket data transfer
Posted by GitBox <gi...@apache.org>.
lukmajercak commented on a change in pull request #1825: HDFS-15151 Use TransmitFile for file to socket data transfer
URL: https://github.com/apache/hadoop/pull/1825#discussion_r373711295
##########
File path: hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/nativeio/NativeIO.java
##########
@@ -839,6 +875,15 @@ public static boolean access(String path, AccessRight desiredAccess)
}
}
+ /**
+ * Set the configuration to be used by this object.
+ * @param conf configuration to be used
+ */
+ public static void setConf(Configuration conf) {
Review comment:
same as above
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on issue #1825: HDFS-15151 Use
TransmitFile for file to socket data transfer
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on issue #1825: HDFS-15151 Use TransmitFile for file to socket data transfer
URL: https://github.com/apache/hadoop/pull/1825#issuecomment-580997948
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 1m 13s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 1 new or modified test files. |
||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 1m 9s | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 21m 4s | trunk passed |
| +1 :green_heart: | compile | 17m 31s | trunk passed |
| +1 :green_heart: | checkstyle | 2m 57s | trunk passed |
| +1 :green_heart: | mvnsite | 2m 42s | trunk passed |
| +1 :green_heart: | shadedclient | 20m 55s | branch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 2m 47s | trunk passed |
| +0 :ok: | spotbugs | 3m 5s | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 5m 7s | trunk passed |
| -0 :warning: | patch | 3m 25s | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 21s | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 54s | the patch passed |
| +1 :green_heart: | compile | 16m 49s | the patch passed |
| -1 :x: | cc | 16m 49s | root generated 5 new + 23 unchanged - 3 fixed = 28 total (was 26) |
| +1 :green_heart: | golang | 16m 49s | the patch passed |
| -1 :x: | javac | 16m 49s | root generated 4 new + 1873 unchanged - 0 fixed = 1877 total (was 1873) |
| -0 :warning: | checkstyle | 2m 48s | root: The patch generated 4 new + 285 unchanged - 2 fixed = 289 total (was 287) |
| +1 :green_heart: | mvnsite | 2m 36s | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | xml | 0m 1s | The patch has no ill-formed XML file. |
| +1 :green_heart: | shadedclient | 13m 51s | patch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 2m 50s | the patch passed |
| +1 :green_heart: | findbugs | 5m 25s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 10m 9s | hadoop-common in the patch passed. |
| -1 :x: | unit | 107m 54s | hadoop-hdfs in the patch failed. |
| +1 :green_heart: | asflicense | 0m 53s | The patch does not generate ASF License warnings. |
| | | 240m 4s | |
| Reason | Tests |
|-------:|:------|
| Failed junit tests | hadoop.hdfs.server.namenode.ha.TestDelegationTokensWithHA |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | Client=19.03.5 Server=19.03.5 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1825/4/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/1825 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle cc golang xml |
| uname | Linux 0847e8f1d4ff 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19 08:06:28 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | personality/hadoop.sh |
| git revision | trunk / 1e3a0b0 |
| Default Java | 1.8.0_232 |
| cc | https://builds.apache.org/job/hadoop-multibranch/job/PR-1825/4/artifact/out/diff-compile-cc-root.txt |
| javac | https://builds.apache.org/job/hadoop-multibranch/job/PR-1825/4/artifact/out/diff-compile-javac-root.txt |
| checkstyle | https://builds.apache.org/job/hadoop-multibranch/job/PR-1825/4/artifact/out/diff-checkstyle-root.txt |
| unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-1825/4/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt |
| Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1825/4/testReport/ |
| Max. process+thread count | 2937 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: . |
| Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1825/4/console |
| versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on issue #1825: HDFS-15151 Use
TransmitFile for file to socket data transfer
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on issue #1825: HDFS-15151 Use TransmitFile for file to socket data transfer
URL: https://github.com/apache/hadoop/pull/1825#issuecomment-580967422
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 0m 33s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | No case conflicting files found. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 1 new or modified test files. |
||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 1m 15s | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 22m 28s | trunk passed |
| +1 :green_heart: | compile | 18m 58s | trunk passed |
| +1 :green_heart: | checkstyle | 3m 18s | trunk passed |
| +1 :green_heart: | mvnsite | 3m 12s | trunk passed |
| +1 :green_heart: | shadedclient | 23m 4s | branch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 3m 1s | trunk passed |
| +0 :ok: | spotbugs | 3m 36s | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 6m 22s | trunk passed |
| -0 :warning: | patch | 3m 57s | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 24s | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 2m 10s | the patch passed |
| +1 :green_heart: | compile | 19m 35s | the patch passed |
| -1 :x: | cc | 19m 35s | root generated 5 new + 23 unchanged - 3 fixed = 28 total (was 26) |
| +1 :green_heart: | golang | 19m 35s | the patch passed |
| -1 :x: | javac | 19m 35s | root generated 4 new + 1873 unchanged - 0 fixed = 1877 total (was 1873) |
| -0 :warning: | checkstyle | 3m 14s | root: The patch generated 6 new + 285 unchanged - 2 fixed = 291 total (was 287) |
| +1 :green_heart: | mvnsite | 3m 2s | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | xml | 0m 1s | The patch has no ill-formed XML file. |
| +1 :green_heart: | shadedclient | 14m 51s | patch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 3m 9s | the patch passed |
| +1 :green_heart: | findbugs | 6m 44s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 10m 35s | hadoop-common in the patch passed. |
| -1 :x: | unit | 93m 29s | hadoop-hdfs in the patch failed. |
| +1 :green_heart: | asflicense | 0m 52s | The patch does not generate ASF License warnings. |
| | | 238m 52s | |
| Reason | Tests |
|-------:|:------|
| Failed junit tests | hadoop.hdfs.TestDFSInotifyEventInputStreamKerberized |
| | hadoop.hdfs.TestDeadNodeDetection |
| | hadoop.hdfs.server.namenode.ha.TestDelegationTokensWithHA |
| | hadoop.hdfs.server.blockmanagement.TestUnderReplicatedBlocks |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | Client=19.03.4 Server=19.03.4 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1825/3/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/1825 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle cc golang xml |
| uname | Linux 64ff69155737 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19 08:06:28 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | personality/hadoop.sh |
| git revision | trunk / bf8686f |
| Default Java | 1.8.0_242 |
| cc | https://builds.apache.org/job/hadoop-multibranch/job/PR-1825/3/artifact/out/diff-compile-cc-root.txt |
| javac | https://builds.apache.org/job/hadoop-multibranch/job/PR-1825/3/artifact/out/diff-compile-javac-root.txt |
| checkstyle | https://builds.apache.org/job/hadoop-multibranch/job/PR-1825/3/artifact/out/diff-checkstyle-root.txt |
| unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-1825/3/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt |
| Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1825/3/testReport/ |
| Max. process+thread count | 3496 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: . |
| Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1825/3/console |
| versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] goiri commented on a change in pull request #1825:
HDFS-15151 Use TransmitFile for file to socket data transfer
Posted by GitBox <gi...@apache.org>.
goiri commented on a change in pull request #1825: HDFS-15151 Use TransmitFile for file to socket data transfer
URL: https://github.com/apache/hadoop/pull/1825#discussion_r373709113
##########
File path: hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/nativeio/NativeIO.java
##########
@@ -354,6 +347,19 @@ public boolean verifyCanMlock() {
}
}
+ public static void setConf(Configuration conf) {
+ workaroundNonThreadSafePasswdCalls = conf.getBoolean(
+ WORKAROUND_NON_THREADSAFE_CALLS_KEY,
+ WORKAROUND_NON_THREADSAFE_CALLS_DEFAULT);
+
+ cacheTimeout = conf.getLong(
+ CommonConfigurationKeys.HADOOP_SECURITY_UID_NAME_CACHE_TIMEOUT_KEY,
+ CommonConfigurationKeys.HADOOP_SECURITY_UID_NAME_CACHE_TIMEOUT_DEFAULT) *
+ 1000;
+ LOG.debug("Initialized cache for IDs to User/Group mapping with a " +
Review comment:
Use the logger format, even though we need to do the division by 1000. I would even change the message to show milliseconds and save the division.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on issue #1825: HDFS-15151 Use
TransmitFile for file to socket data transfer
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on issue #1825: HDFS-15151 Use TransmitFile for file to socket data transfer
URL: https://github.com/apache/hadoop/pull/1825#issuecomment-580533837
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 27m 10s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 1 new or modified test files. |
||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 1m 19s | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 19m 58s | trunk passed |
| +1 :green_heart: | compile | 18m 25s | trunk passed |
| +1 :green_heart: | checkstyle | 2m 53s | trunk passed |
| +1 :green_heart: | mvnsite | 3m 1s | trunk passed |
| +1 :green_heart: | shadedclient | 20m 0s | branch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 3m 24s | trunk passed |
| +0 :ok: | spotbugs | 3m 42s | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 6m 12s | trunk passed |
| -0 :warning: | patch | 4m 6s | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 27s | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 2m 18s | the patch passed |
| -1 :x: | compile | 1m 15s | root in the patch failed. |
| -1 :x: | cc | 1m 15s | root in the patch failed. |
| -1 :x: | golang | 1m 15s | root in the patch failed. |
| -1 :x: | javac | 1m 15s | root in the patch failed. |
| -0 :warning: | checkstyle | 3m 37s | root: The patch generated 9 new + 285 unchanged - 2 fixed = 294 total (was 287) |
| +1 :green_heart: | mvnsite | 2m 48s | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | xml | 0m 2s | The patch has no ill-formed XML file. |
| +1 :green_heart: | shadedclient | 14m 24s | patch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 2m 50s | the patch passed |
| +1 :green_heart: | findbugs | 5m 43s | the patch passed |
||| _ Other Tests _ |
| -1 :x: | unit | 0m 43s | hadoop-common in the patch failed. |
| -1 :x: | unit | 105m 52s | hadoop-hdfs in the patch failed. |
| +1 :green_heart: | asflicense | 0m 45s | The patch does not generate ASF License warnings. |
| | | 241m 47s | |
| Reason | Tests |
|-------:|:------|
| Failed junit tests | hadoop.fs.TestHdfsNativeCodeLoader |
| | hadoop.hdfs.server.namenode.ha.TestDelegationTokensWithHA |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | Client=19.03.5 Server=19.03.5 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1825/1/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/1825 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle cc golang xml |
| uname | Linux a243d09618ce 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | personality/hadoop.sh |
| git revision | trunk / a7d72c5 |
| Default Java | 1.8.0_242 |
| compile | https://builds.apache.org/job/hadoop-multibranch/job/PR-1825/1/artifact/out/patch-compile-root.txt |
| cc | https://builds.apache.org/job/hadoop-multibranch/job/PR-1825/1/artifact/out/patch-compile-root.txt |
| golang | https://builds.apache.org/job/hadoop-multibranch/job/PR-1825/1/artifact/out/patch-compile-root.txt |
| javac | https://builds.apache.org/job/hadoop-multibranch/job/PR-1825/1/artifact/out/patch-compile-root.txt |
| checkstyle | https://builds.apache.org/job/hadoop-multibranch/job/PR-1825/1/artifact/out/diff-checkstyle-root.txt |
| unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-1825/1/artifact/out/patch-unit-hadoop-common-project_hadoop-common.txt |
| unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-1825/1/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt |
| Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1825/1/testReport/ |
| Max. process+thread count | 3781 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: . |
| Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1825/1/console |
| versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] goiri commented on a change in pull request #1825:
HDFS-15151 Use TransmitFile for file to socket data transfer
Posted by GitBox <gi...@apache.org>.
goiri commented on a change in pull request #1825: HDFS-15151 Use TransmitFile for file to socket data transfer
URL: https://github.com/apache/hadoop/pull/1825#discussion_r373709672
##########
File path: hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/nativeio/NativeIO.java
##########
@@ -839,6 +875,15 @@ public static boolean access(String path, AccessRight desiredAccess)
}
}
+ /**
+ * Set the configuration to be used by this object.
+ * @param conf configuration to be used
+ */
+ public static void setConf(Configuration conf) {
Review comment:
Name this more specifically.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] lukmajercak commented on a change in pull request #1825:
HDFS-15151 Use TransmitFile for file to socket data transfer
Posted by GitBox <gi...@apache.org>.
lukmajercak commented on a change in pull request #1825: HDFS-15151 Use TransmitFile for file to socket data transfer
URL: https://github.com/apache/hadoop/pull/1825#discussion_r373711031
##########
File path: hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/nativeio/NativeIO.java
##########
@@ -354,6 +347,19 @@ public boolean verifyCanMlock() {
}
}
+ public static void setConf(Configuration conf) {
+ workaroundNonThreadSafePasswdCalls = conf.getBoolean(
+ WORKAROUND_NON_THREADSAFE_CALLS_KEY,
+ WORKAROUND_NON_THREADSAFE_CALLS_DEFAULT);
+
+ cacheTimeout = conf.getLong(
+ CommonConfigurationKeys.HADOOP_SECURITY_UID_NAME_CACHE_TIMEOUT_KEY,
+ CommonConfigurationKeys.HADOOP_SECURITY_UID_NAME_CACHE_TIMEOUT_DEFAULT) *
+ 1000;
+ LOG.debug("Initialized cache for IDs to User/Group mapping with a " +
Review comment:
This is old code, not in scope of this change.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] goiri commented on a change in pull request #1825:
HDFS-15151 Use TransmitFile for file to socket data transfer
Posted by GitBox <gi...@apache.org>.
goiri commented on a change in pull request #1825: HDFS-15151 Use TransmitFile for file to socket data transfer
URL: https://github.com/apache/hadoop/pull/1825#discussion_r373806607
##########
File path: hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/net/SocketOutputStream.java
##########
@@ -219,7 +226,9 @@ public void transferToFully(FileChannel fileCh, long position, int count,
waitForWritable();
long wait = System.nanoTime();
- int nTransfered = (int) fileCh.transferTo(position, count, getChannel());
+ int nTransfered = (int) NativeIO.transferTo(fileCh,
+ fileInputStream.getFD(), getChannel(),
+ ((SelChImpl) getChannel()).getFD(), position, count);
Review comment:
I cannot find an easy way out but can you see if we can remove:
SelChImpl is internal proprietary API and may be removed in a future release
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] goiri commented on a change in pull request #1825:
HDFS-15151 Use TransmitFile for file to socket data transfer
Posted by GitBox <gi...@apache.org>.
goiri commented on a change in pull request #1825: HDFS-15151 Use TransmitFile for file to socket data transfer
URL: https://github.com/apache/hadoop/pull/1825#discussion_r373708397
##########
File path: hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/nativeio/NativeIO.c
##########
@@ -1341,6 +1350,68 @@ Java_org_apache_hadoop_io_nativeio_NativeIO_00024Windows_extendWorkingSetSize(
#endif
}
+jint
+fdval(JNIEnv *env, jobject fdo)
+{
+ return (*env)->GetIntField(env, fdo, fd_fdID);
+}
+
+jlong
+handleval(JNIEnv *env, jobject fdo)
+{
+ return (*env)->GetLongField(env, fdo, handle_fdID);
+}
+
+JNIEXPORT jlong JNICALL
+Java_org_apache_hadoop_io_nativeio_NativeIO_00024Windows_transmitFile(
+ JNIEnv *env, jobject jObject, jobject srcFD, jlong position, jlong count,
+ jobject dstFD)
+{
+#ifdef UNIX
+ THROW(env, "java/io/IOException",
+ "The function transmitFile is not supported on Unix");
+#endif
+
+#ifdef WINDOWS
Review comment:
#elseif WINDOWS
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] lukmajercak commented on a change in pull request #1825:
HDFS-15151 Use TransmitFile for file to socket data transfer
Posted by GitBox <gi...@apache.org>.
lukmajercak commented on a change in pull request #1825: HDFS-15151 Use TransmitFile for file to socket data transfer
URL: https://github.com/apache/hadoop/pull/1825#discussion_r373711257
##########
File path: hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/nativeio/NativeIO.java
##########
@@ -819,6 +849,12 @@ public static boolean access(String path, AccessRight desiredAccess)
}
}
}
+
+ public static void setConf(Configuration conf) {
Review comment:
it's not meant to be only for enabling transmitfile, but generally for reloading/resetting the configuration
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on issue #1825: HDFS-15151 Use
TransmitFile for file to socket data transfer
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on issue #1825: HDFS-15151 Use TransmitFile for file to socket data transfer
URL: https://github.com/apache/hadoop/pull/1825#issuecomment-580561908
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 1m 14s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | No case conflicting files found. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 1 new or modified test files. |
||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 1m 8s | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 20m 34s | trunk passed |
| +1 :green_heart: | compile | 19m 0s | trunk passed |
| +1 :green_heart: | checkstyle | 3m 54s | trunk passed |
| +1 :green_heart: | mvnsite | 2m 44s | trunk passed |
| +1 :green_heart: | shadedclient | 21m 52s | branch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 2m 53s | trunk passed |
| +0 :ok: | spotbugs | 3m 5s | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 5m 6s | trunk passed |
| -0 :warning: | patch | 3m 26s | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 21s | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 53s | the patch passed |
| -1 :x: | compile | 0m 55s | root in the patch failed. |
| -1 :x: | cc | 0m 55s | root in the patch failed. |
| -1 :x: | golang | 0m 55s | root in the patch failed. |
| -1 :x: | javac | 0m 55s | root in the patch failed. |
| -0 :warning: | checkstyle | 2m 36s | root: The patch generated 9 new + 285 unchanged - 2 fixed = 294 total (was 287) |
| +1 :green_heart: | mvnsite | 2m 15s | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | xml | 0m 1s | The patch has no ill-formed XML file. |
| +1 :green_heart: | shadedclient | 14m 38s | patch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 2m 31s | the patch passed |
| +1 :green_heart: | findbugs | 5m 0s | the patch passed |
||| _ Other Tests _ |
| -1 :x: | unit | 0m 34s | hadoop-common in the patch failed. |
| -1 :x: | unit | 110m 10s | hadoop-hdfs in the patch failed. |
| +1 :green_heart: | asflicense | 0m 35s | The patch does not generate ASF License warnings. |
| | | 217m 12s | |
| Reason | Tests |
|-------:|:------|
| Failed junit tests | hadoop.fs.TestHdfsNativeCodeLoader |
| | hadoop.hdfs.server.namenode.ha.TestDelegationTokensWithHA |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | Client=19.03.5 Server=19.03.5 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1825/2/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/1825 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle cc golang xml |
| uname | Linux 0f8477e5d918 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19 08:06:28 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | personality/hadoop.sh |
| git revision | trunk / a7d72c5 |
| Default Java | 1.8.0_232 |
| compile | https://builds.apache.org/job/hadoop-multibranch/job/PR-1825/2/artifact/out/patch-compile-root.txt |
| cc | https://builds.apache.org/job/hadoop-multibranch/job/PR-1825/2/artifact/out/patch-compile-root.txt |
| golang | https://builds.apache.org/job/hadoop-multibranch/job/PR-1825/2/artifact/out/patch-compile-root.txt |
| javac | https://builds.apache.org/job/hadoop-multibranch/job/PR-1825/2/artifact/out/patch-compile-root.txt |
| checkstyle | https://builds.apache.org/job/hadoop-multibranch/job/PR-1825/2/artifact/out/diff-checkstyle-root.txt |
| unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-1825/2/artifact/out/patch-unit-hadoop-common-project_hadoop-common.txt |
| unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-1825/2/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt |
| Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1825/2/testReport/ |
| Max. process+thread count | 3059 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: . |
| Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1825/2/console |
| versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] goiri commented on a change in pull request #1825:
HDFS-15151 Use TransmitFile for file to socket data transfer
Posted by GitBox <gi...@apache.org>.
goiri commented on a change in pull request #1825: HDFS-15151 Use TransmitFile for file to socket data transfer
URL: https://github.com/apache/hadoop/pull/1825#discussion_r373709478
##########
File path: hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/nativeio/NativeIO.java
##########
@@ -819,6 +849,12 @@ public static boolean access(String path, AccessRight desiredAccess)
}
}
}
+
+ public static void setConf(Configuration conf) {
Review comment:
Should we name this a little bit more specific and not only setConf? initTransmitFileEnabledConf()?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org