You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "ChenSammi (via GitHub)" <gi...@apache.org> on 2024/12/04 14:15:21 UTC
[PR] HDDS-11626. Document short-circuit read feature [ozone]
ChenSammi opened a new pull request, #7529:
URL: https://github.com/apache/ozone/pull/7529
## What changes were proposed in this pull request?
Document the precondition and configuration of short-circuit local read feature
## What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-11626
## How was this patch tested?
previewed in intellij
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-11626. Document short-circuit read feature [ozone]
Posted by "errose28 (via GitHub)" <gi...@apache.org>.
errose28 commented on code in PR #7529:
URL: https://github.com/apache/ozone/pull/7529#discussion_r1870390885
##########
hadoop-hdds/docs/content/feature/Short-Circuit-Read.md:
##########
@@ -0,0 +1,76 @@
+---
+title: "Short Circuit Local Read in Datanode"
+weight: 2
+menu:
+ main:
+ parent: Features
+summary: Introduction to Ozone Datanode Short Circuit Local Read Feature
+---
+<!---
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+
+Current in Ozone, client reads data over GRPC from Datanode. When the client asks the DataNode to read a file, the DataNode reads that file off of the disk and sends the data to the client over GRPC connection.
Review Comment:
```suggestion
By default, the client reads data over GRPC from the Datanode. When the client asks the DataNode to read a file, the DataNode reads that file off of the disk and sends the data to the client over a GRPC connection.
```
nit. There's a mix of Datanode and DataNode in this doc. We should standardize on Datanode. The new website strictly enforces this.
##########
hadoop-hdds/docs/content/feature/Short-Circuit-Read.md:
##########
@@ -0,0 +1,76 @@
+---
+title: "Short Circuit Local Read in Datanode"
+weight: 2
+menu:
+ main:
+ parent: Features
+summary: Introduction to Ozone Datanode Short Circuit Local Read Feature
+---
+<!---
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+
+Current in Ozone, client reads data over GRPC from Datanode. When the client asks the DataNode to read a file, the DataNode reads that file off of the disk and sends the data to the client over GRPC connection.
+
+This “short-circuit” local read feature will bypass the DataNode, allowing the client to read the file from local disk directly when the client is co-located with the data on the same server.
+
+Short-circuit local read can provide a substantial performance boost to many applications, by removing the overhead of network communication.
+
+## Prerequisite
+
+Short-circuit local reads make use of a UNIX domain socket. This is a special path in the filesystem that allows the client and the DataNodes to communicate.
+
+The Hadoop native library "libhadoop.so" provides the support to use the Unix domain socket. Please refer to Native Libraries ("https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-common/NativeLibraries.html") for details of this library.
+
+Before enabling short-circuit local reads, you must have a proper libhadoop.so, and make sure it's under the directory where Java can find and load it through "System.loadLibrary()" call.
+
+The paths that Java will search for libraries are specified by the "java.library.path" property. The default value of "java.library.path" depends on the OS and Java version. For example, on Linux with OpenJDK 8 it is `/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib`.
+
+Command "ozone checknative" can be used to detect whether libhadoop.so can be loaded successfully by Ozone service.
+
+## Configuration
+
+Short-circuit local reads need to be configured on both the DataNode and the client. By default, it is disabled.
+
+```XML
+<property>
+ <name>ozone.client.read.short-circuit</name>
+ <value>false</value>
+ <description>Disable or enable the short-circuit local read feature.</description>
+</property>
+```
+
+It makes use of a UNIX domain socket, a special path in the filesystem. You will need to set a path to this socket.
+
+```XML
+<property>
+ <name>ozone.domain.socket.path</name>
+ <value>/var/lib/ozone_dn_socket</value>
+ <description>The path used to create domain socket.</description>
+</property>
+```
+
+The DataNode needs to be able to create this path. On the other hand, it should not be possible for any user except the Ozone user(user who launches Ozone service) or root to create this path. For this reason, paths under /var/run or /var/lib are often used， just like the current default value "/var/lib/ozone_dn_socket".
+
+If you configure the "ozone.domain.socket.path" to other value, for example "/dir1/dir2/ozone_dn_socket", please make sure that both dir1 and dir2 are exiting directories, and there is no ozone_dn_socket under dir2. ozone_dn_socket will be created by Ozone Datanode later during Datanode start.
+
+### Security Consideration
+
+To ensure data security and integrity, Ozone will follow the rule as Hadoop ("https://cwiki.apache.org/confluence/display/HADOOP2/SocketPathSecurity") on "ozone.domain.socket.path" path permission check.
+It will fail the "ozone.domain.socket.path" verification and disable the feature if the filesystem permissions of the specified path are inadequate.
+The verification failure message carries detail instruction about how to fix the problem. Following is an example,
+
+"The path component: '/etc/hadoop' in '/etc/hadoop/ozone_dn_socket' has permissions 0777 uid 0 and gid 0. It is not protected because it is world-writable. This might help: 'chmod o-w /etc/hadoop'. For more information: https://wiki.apache.org/hadoop/SocketPathSecurity"
Review Comment:
```suggestion
The verification failure message carries instructions about how to fix the problem. Here is an example:
\```
The path component: '/etc/hadoop' in '/etc/hadoop/ozone_dn_socket' has permissions 0777 uid 0 and gid 0. It is not protected because it is world-writable. This might help: 'chmod o-w /etc/hadoop'. For more information: https://wiki.apache.org/hadoop/SocketPathSecurity
\```
```
##########
hadoop-hdds/docs/content/feature/Short-Circuit-Read.md:
##########
@@ -0,0 +1,76 @@
+---
+title: "Short Circuit Local Read in Datanode"
+weight: 2
+menu:
+ main:
+ parent: Features
+summary: Introduction to Ozone Datanode Short Circuit Local Read Feature
+---
+<!---
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+
+Current in Ozone, client reads data over GRPC from Datanode. When the client asks the DataNode to read a file, the DataNode reads that file off of the disk and sends the data to the client over GRPC connection.
+
+This “short-circuit” local read feature will bypass the DataNode, allowing the client to read the file from local disk directly when the client is co-located with the data on the same server.
+
+Short-circuit local read can provide a substantial performance boost to many applications, by removing the overhead of network communication.
+
+## Prerequisite
+
+Short-circuit local reads make use of a UNIX domain socket. This is a special path in the filesystem that allows the client and the DataNodes to communicate.
+
+The Hadoop native library "libhadoop.so" provides the support to use the Unix domain socket. Please refer to Native Libraries ("https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-common/NativeLibraries.html") for details of this library.
+
+Before enabling short-circuit local reads, you must have a proper libhadoop.so, and make sure it's under the directory where Java can find and load it through "System.loadLibrary()" call.
+
+The paths that Java will search for libraries are specified by the "java.library.path" property. The default value of "java.library.path" depends on the OS and Java version. For example, on Linux with OpenJDK 8 it is `/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib`.
+
+Command "ozone checknative" can be used to detect whether libhadoop.so can be loaded successfully by Ozone service.
+
+## Configuration
+
+Short-circuit local reads need to be configured on both the DataNode and the client. By default, it is disabled.
+
+```XML
+<property>
+ <name>ozone.client.read.short-circuit</name>
+ <value>false</value>
+ <description>Disable or enable the short-circuit local read feature.</description>
+</property>
+```
+
+It makes use of a UNIX domain socket, a special path in the filesystem. You will need to set a path to this socket.
+
+```XML
+<property>
+ <name>ozone.domain.socket.path</name>
+ <value>/var/lib/ozone_dn_socket</value>
+ <description>The path used to create domain socket.</description>
+</property>
+```
+
+The DataNode needs to be able to create this path. On the other hand, it should not be possible for any user except the Ozone user(user who launches Ozone service) or root to create this path. For this reason, paths under /var/run or /var/lib are often used， just like the current default value "/var/lib/ozone_dn_socket".
+
+If you configure the "ozone.domain.socket.path" to other value, for example "/dir1/dir2/ozone_dn_socket", please make sure that both dir1 and dir2 are exiting directories, and there is no ozone_dn_socket under dir2. ozone_dn_socket will be created by Ozone Datanode later during Datanode start.
+
+### Security Consideration
+
+To ensure data security and integrity, Ozone will follow the rule as Hadoop ("https://cwiki.apache.org/confluence/display/HADOOP2/SocketPathSecurity") on "ozone.domain.socket.path" path permission check.
Review Comment:
```suggestion
To ensure data security and integrity, Ozone will follow the same rules as Hadoop to check permission on the `ozone.domain.socket.path` path as documented in [Socket Path Security](https://cwiki.apache.org/confluence/display/HADOOP2/SocketPathSecurity).
```
Is there a better resource for this somewhere on hadoop.apache.org/docs? Linking to confluence instead of official docs is not great, and this link is hardcoded to an old hadoop 2 url that will not get updated with subsequent versions.
##########
hadoop-hdds/docs/content/feature/Short-Circuit-Read.md:
##########
@@ -0,0 +1,76 @@
+---
+title: "Short Circuit Local Read in Datanode"
+weight: 2
+menu:
+ main:
+ parent: Features
+summary: Introduction to Ozone Datanode Short Circuit Local Read Feature
+---
+<!---
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+
+Current in Ozone, client reads data over GRPC from Datanode. When the client asks the DataNode to read a file, the DataNode reads that file off of the disk and sends the data to the client over GRPC connection.
+
+This “short-circuit” local read feature will bypass the DataNode, allowing the client to read the file from local disk directly when the client is co-located with the data on the same server.
+
+Short-circuit local read can provide a substantial performance boost to many applications, by removing the overhead of network communication.
+
+## Prerequisite
+
+Short-circuit local reads make use of a UNIX domain socket. This is a special path in the filesystem that allows the client and the DataNodes to communicate.
+
+The Hadoop native library "libhadoop.so" provides the support to use the Unix domain socket. Please refer to Native Libraries ("https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-common/NativeLibraries.html") for details of this library.
+
+Before enabling short-circuit local reads, you must have a proper libhadoop.so, and make sure it's under the directory where Java can find and load it through "System.loadLibrary()" call.
+
+The paths that Java will search for libraries are specified by the "java.library.path" property. The default value of "java.library.path" depends on the OS and Java version. For example, on Linux with OpenJDK 8 it is `/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib`.
+
+Command "ozone checknative" can be used to detect whether libhadoop.so can be loaded successfully by Ozone service.
+
+## Configuration
+
+Short-circuit local reads need to be configured on both the DataNode and the client. By default, it is disabled.
Review Comment:
It may seem obvious that we need to enable it to use it, but the docs don't actually show that here. Ideally copy/pasting the provided xml into a config would yield the desired result.
##########
hadoop-hdds/docs/content/feature/Short-Circuit-Read.md:
##########
@@ -0,0 +1,76 @@
+---
+title: "Short Circuit Local Read in Datanode"
+weight: 2
+menu:
+ main:
+ parent: Features
+summary: Introduction to Ozone Datanode Short Circuit Local Read Feature
+---
+<!---
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+
+Current in Ozone, client reads data over GRPC from Datanode. When the client asks the DataNode to read a file, the DataNode reads that file off of the disk and sends the data to the client over GRPC connection.
+
+This “short-circuit” local read feature will bypass the DataNode, allowing the client to read the file from local disk directly when the client is co-located with the data on the same server.
+
+Short-circuit local read can provide a substantial performance boost to many applications, by removing the overhead of network communication.
+
+## Prerequisite
+
+Short-circuit local reads make use of a UNIX domain socket. This is a special path in the filesystem that allows the client and the DataNodes to communicate.
+
+The Hadoop native library "libhadoop.so" provides the support to use the Unix domain socket. Please refer to Native Libraries ("https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-common/NativeLibraries.html") for details of this library.
Review Comment:
```suggestion
The Hadoop native library `libhadoop.so` provides support for Unix domain sockets. Please refer to Hadoop's [Native Libraries Guide](https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-common/NativeLibraries.html) for details.
```
Let's use markdown formatting as much as possible to help readability. There's other instances later in this page where we can replace "" with `` and use markdown links instead of plain ones.
##########
hadoop-hdds/docs/content/feature/Short-Circuit-Read.md:
##########
@@ -0,0 +1,76 @@
+---
+title: "Short Circuit Local Read in Datanode"
+weight: 2
+menu:
+ main:
+ parent: Features
+summary: Introduction to Ozone Datanode Short Circuit Local Read Feature
+---
+<!---
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+
+Current in Ozone, client reads data over GRPC from Datanode. When the client asks the DataNode to read a file, the DataNode reads that file off of the disk and sends the data to the client over GRPC connection.
+
+This “short-circuit” local read feature will bypass the DataNode, allowing the client to read the file from local disk directly when the client is co-located with the data on the same server.
+
+Short-circuit local read can provide a substantial performance boost to many applications, by removing the overhead of network communication.
+
+## Prerequisite
+
+Short-circuit local reads make use of a UNIX domain socket. This is a special path in the filesystem that allows the client and the DataNodes to communicate.
+
+The Hadoop native library "libhadoop.so" provides the support to use the Unix domain socket. Please refer to Native Libraries ("https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-common/NativeLibraries.html") for details of this library.
+
+Before enabling short-circuit local reads, you must have a proper libhadoop.so, and make sure it's under the directory where Java can find and load it through "System.loadLibrary()" call.
+
+The paths that Java will search for libraries are specified by the "java.library.path" property. The default value of "java.library.path" depends on the OS and Java version. For example, on Linux with OpenJDK 8 it is `/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib`.
+
+Command "ozone checknative" can be used to detect whether libhadoop.so can be loaded successfully by Ozone service.
+
+## Configuration
+
+Short-circuit local reads need to be configured on both the DataNode and the client. By default, it is disabled.
+
+```XML
+<property>
+ <name>ozone.client.read.short-circuit</name>
+ <value>false</value>
+ <description>Disable or enable the short-circuit local read feature.</description>
+</property>
+```
+
+It makes use of a UNIX domain socket, a special path in the filesystem. You will need to set a path to this socket.
+
+```XML
+<property>
+ <name>ozone.domain.socket.path</name>
+ <value>/var/lib/ozone_dn_socket</value>
+ <description>The path used to create domain socket.</description>
+</property>
+```
+
+The DataNode needs to be able to create this path. On the other hand, it should not be possible for any user except the Ozone user(user who launches Ozone service) or root to create this path. For this reason, paths under /var/run or /var/lib are often used， just like the current default value "/var/lib/ozone_dn_socket".
+
+If you configure the "ozone.domain.socket.path" to other value, for example "/dir1/dir2/ozone_dn_socket", please make sure that both dir1 and dir2 are exiting directories, and there is no ozone_dn_socket under dir2. ozone_dn_socket will be created by Ozone Datanode later during Datanode start.
Review Comment:
```suggestion
If you configure the "ozone.domain.socket.path" to another value, for example `/dir1/dir2/ozone_dn_socket`, please make sure that both `dir1` and `dir2` are existing directories, but the file `ozone_dn_socket` does not exist under `dir2`. `ozone_dn_socket` will be created during Datanode startup.
```
##########
hadoop-hdds/docs/content/feature/Short-Circuit-Read.md:
##########
@@ -0,0 +1,76 @@
+---
+title: "Short Circuit Local Read in Datanode"
+weight: 2
+menu:
+ main:
+ parent: Features
+summary: Introduction to Ozone Datanode Short Circuit Local Read Feature
+---
+<!---
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+
+Current in Ozone, client reads data over GRPC from Datanode. When the client asks the DataNode to read a file, the DataNode reads that file off of the disk and sends the data to the client over GRPC connection.
+
+This “short-circuit” local read feature will bypass the DataNode, allowing the client to read the file from local disk directly when the client is co-located with the data on the same server.
+
+Short-circuit local read can provide a substantial performance boost to many applications, by removing the overhead of network communication.
+
+## Prerequisite
+
+Short-circuit local reads make use of a UNIX domain socket. This is a special path in the filesystem that allows the client and the DataNodes to communicate.
+
+The Hadoop native library "libhadoop.so" provides the support to use the Unix domain socket. Please refer to Native Libraries ("https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-common/NativeLibraries.html") for details of this library.
+
+Before enabling short-circuit local reads, you must have a proper libhadoop.so, and make sure it's under the directory where Java can find and load it through "System.loadLibrary()" call.
+
+The paths that Java will search for libraries are specified by the "java.library.path" property. The default value of "java.library.path" depends on the OS and Java version. For example, on Linux with OpenJDK 8 it is `/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib`.
+
+Command "ozone checknative" can be used to detect whether libhadoop.so can be loaded successfully by Ozone service.
Review Comment:
```suggestion
The `ozone checknative` command can be used to detect whether `libhadoop.so` can be loaded successfully by Ozone.
```
##########
hadoop-hdds/docs/content/feature/Short-Circuit-Read.md:
##########
@@ -0,0 +1,76 @@
+---
+title: "Short Circuit Local Read in Datanode"
+weight: 2
+menu:
+ main:
+ parent: Features
+summary: Introduction to Ozone Datanode Short Circuit Local Read Feature
+---
+<!---
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+
+Current in Ozone, client reads data over GRPC from Datanode. When the client asks the DataNode to read a file, the DataNode reads that file off of the disk and sends the data to the client over GRPC connection.
+
+This “short-circuit” local read feature will bypass the DataNode, allowing the client to read the file from local disk directly when the client is co-located with the data on the same server.
+
+Short-circuit local read can provide a substantial performance boost to many applications, by removing the overhead of network communication.
+
+## Prerequisite
+
+Short-circuit local reads make use of a UNIX domain socket. This is a special path in the filesystem that allows the client and the DataNodes to communicate.
+
+The Hadoop native library "libhadoop.so" provides the support to use the Unix domain socket. Please refer to Native Libraries ("https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-common/NativeLibraries.html") for details of this library.
+
+Before enabling short-circuit local reads, you must have a proper libhadoop.so, and make sure it's under the directory where Java can find and load it through "System.loadLibrary()" call.
Review Comment:
I'm not sure what is meant by a "proper" libhadoop.so. These three mini paragraphs all address the same topic and can be grouped into one paragraph. Additionally we should have instructions for how to make java see the library before talking about loading it.
##########
hadoop-hdds/docs/content/feature/Short-Circuit-Read.md:
##########
@@ -0,0 +1,76 @@
+---
+title: "Short Circuit Local Read in Datanode"
+weight: 2
+menu:
+ main:
+ parent: Features
+summary: Introduction to Ozone Datanode Short Circuit Local Read Feature
+---
+<!---
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+
+Current in Ozone, client reads data over GRPC from Datanode. When the client asks the DataNode to read a file, the DataNode reads that file off of the disk and sends the data to the client over GRPC connection.
+
+This “short-circuit” local read feature will bypass the DataNode, allowing the client to read the file from local disk directly when the client is co-located with the data on the same server.
+
+Short-circuit local read can provide a substantial performance boost to many applications, by removing the overhead of network communication.
+
+## Prerequisite
+
+Short-circuit local reads make use of a UNIX domain socket. This is a special path in the filesystem that allows the client and the DataNodes to communicate.
+
+The Hadoop native library "libhadoop.so" provides the support to use the Unix domain socket. Please refer to Native Libraries ("https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-common/NativeLibraries.html") for details of this library.
+
+Before enabling short-circuit local reads, you must have a proper libhadoop.so, and make sure it's under the directory where Java can find and load it through "System.loadLibrary()" call.
+
+The paths that Java will search for libraries are specified by the "java.library.path" property. The default value of "java.library.path" depends on the OS and Java version. For example, on Linux with OpenJDK 8 it is `/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib`.
+
+Command "ozone checknative" can be used to detect whether libhadoop.so can be loaded successfully by Ozone service.
+
+## Configuration
+
+Short-circuit local reads need to be configured on both the DataNode and the client. By default, it is disabled.
+
+```XML
+<property>
+ <name>ozone.client.read.short-circuit</name>
+ <value>false</value>
+ <description>Disable or enable the short-circuit local read feature.</description>
+</property>
+```
+
+It makes use of a UNIX domain socket, a special path in the filesystem. You will need to set a path to this socket.
+
+```XML
+<property>
+ <name>ozone.domain.socket.path</name>
+ <value>/var/lib/ozone_dn_socket</value>
+ <description>The path used to create domain socket.</description>
+</property>
+```
+
+The DataNode needs to be able to create this path. On the other hand, it should not be possible for any user except the Ozone user(user who launches Ozone service) or root to create this path. For this reason, paths under /var/run or /var/lib are often used， just like the current default value "/var/lib/ozone_dn_socket".
+
+If you configure the "ozone.domain.socket.path" to other value, for example "/dir1/dir2/ozone_dn_socket", please make sure that both dir1 and dir2 are exiting directories, and there is no ozone_dn_socket under dir2. ozone_dn_socket will be created by Ozone Datanode later during Datanode start.
+
+### Security Consideration
+
+To ensure data security and integrity, Ozone will follow the rule as Hadoop ("https://cwiki.apache.org/confluence/display/HADOOP2/SocketPathSecurity") on "ozone.domain.socket.path" path permission check.
Review Comment:
Also, is our implementation using hadoop linraries to do these permission checks or did we re-implement them ourselves? If we are using hadoop then it makes since to link to hadoop docs so they update with the code. If we re-implemented them ourselves we should also "re-implement" our own documentation so they match now and in the future.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-11626. Document short-circuit read feature [ozone]
Posted by "ChenSammi (via GitHub)" <gi...@apache.org>.
ChenSammi commented on code in PR #7529:
URL: https://github.com/apache/ozone/pull/7529#discussion_r1875655570
##########
hadoop-hdds/docs/content/feature/Short-Circuit-Read.md:
##########
@@ -0,0 +1,76 @@
+---
+title: "Short Circuit Local Read in Datanode"
+weight: 2
+menu:
+ main:
+ parent: Features
+summary: Introduction to Ozone Datanode Short Circuit Local Read Feature
+---
+<!---
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+
+Current in Ozone, client reads data over GRPC from Datanode. When the client asks the DataNode to read a file, the DataNode reads that file off of the disk and sends the data to the client over GRPC connection.
+
+This “short-circuit” local read feature will bypass the DataNode, allowing the client to read the file from local disk directly when the client is co-located with the data on the same server.
+
+Short-circuit local read can provide a substantial performance boost to many applications, by removing the overhead of network communication.
+
+## Prerequisite
+
+Short-circuit local reads make use of a UNIX domain socket. This is a special path in the filesystem that allows the client and the DataNodes to communicate.
+
+The Hadoop native library "libhadoop.so" provides the support to use the Unix domain socket. Please refer to Native Libraries ("https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-common/NativeLibraries.html") for details of this library.
+
+Before enabling short-circuit local reads, you must have a proper libhadoop.so, and make sure it's under the directory where Java can find and load it through "System.loadLibrary()" call.
+
+The paths that Java will search for libraries are specified by the "java.library.path" property. The default value of "java.library.path" depends on the OS and Java version. For example, on Linux with OpenJDK 8 it is `/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib`.
+
+Command "ozone checknative" can be used to detect whether libhadoop.so can be loaded successfully by Ozone service.
+
+## Configuration
+
+Short-circuit local reads need to be configured on both the DataNode and the client. By default, it is disabled.
Review Comment:
Let's keep it by default disabled now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-11626. Document short-circuit read feature [ozone]
Posted by "ChenSammi (via GitHub)" <gi...@apache.org>.
ChenSammi commented on code in PR #7529:
URL: https://github.com/apache/ozone/pull/7529#discussion_r1928025544
##########
hadoop-hdds/docs/content/feature/Short-Circuit-Read.md:
##########
@@ -0,0 +1,78 @@
+---
+title: "Short Circuit Local Read in Datanode"
+weight: 2
+menu:
+ main:
+ parent: Features
+summary: Introduction to Ozone Datanode Short Circuit Local Read Feature
+---
+<!---
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+
+By default, client reads data over GRPC from the Datanode. When the client asks the Datanode to read a file, the DataNode reads that file off of the disk and sends the data to the client over a GRPC connection.
+
+This “short-circuit” local read feature will bypass the DataNode, allowing the client to read the file from local disk directly when the client is co-located with the data on the same server.
+
+Short-circuit local read can provide a substantial performance boost to many applications, by removing the overhead of network communication.
+
+## Prerequisite
+
+Short-circuit local reads make use of a UNIX domain socket. This is a special path in the filesystem that allows the client and the DataNodes to communicate.
+
+The Hadoop native library `libhadoop.so` provides support to for Unix domain sockets. Please refer to Hadoop's [Native Libraries Guide](https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-common/NativeLibraries.html) for details.
+
+The Hadoop version used in Ozone is defined by `hadoop.version` in pom.xml. Before enabling short-circuit local reads, find the `libhadoop.so` from the corresponding version Hadoop release package, put it under one of the directories specified by Java `java.library.path` property. The default value of `java.library.path` depends on the OS and Java version. For example, on Linux with OpenJDK 8 it is `/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib`.
Review Comment:
"hadoop.version" value is specified in Ozone pom.xml.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-11626. Document short-circuit read feature [ozone]
Posted by "jojochuang (via GitHub)" <gi...@apache.org>.
jojochuang commented on code in PR #7529:
URL: https://github.com/apache/ozone/pull/7529#discussion_r1960113015
##########
hadoop-hdds/docs/content/feature/Short-Circuit-Read.zh.md:
##########
@@ -0,0 +1,88 @@
+---
+title: "Datanode 本地短路读"
+weight: 2
+menu:
+ main:
+ parent: 特性
+summary: Ozone Datanode 本地短路读功能介绍
+---
+<!---
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+
+当前在 Ozone 中，客户端使用 GRPC 通道从 Datanode 读取数据。当客户端向 Datanode 请求读取一个文件时，Datanode 将文件从本次磁盘读到内存，然后通过 GRPC 通道发回给客户端。
+
+Datanode 本地短路读功能，当客户端和 Datanode 在同一个机器时，允许客户端绕过 Datanode，直接从本地磁盘读取文件内容。通过绕过 Datanode，去掉网络通信带来的开销，Datanode 本地短路读功能将帮助许多 Ozone 应用，提升读性能。
+
+## 前提
+
+Datanode 本地短路读功能基于 Unix domain socket 实现。 Unix domain socket 是一个特殊的文件系统路径，支持客户端和 Datanode 通过它交互传递信息。
+
+Datanode 本地短路读功能需要用到 Hadoop 本地库 `libhadoop.so`。 `libhadoop.so` 提供了调用 Unix domain socket 的功能。该本地库的详细信息，详见 [Native Libraries](https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-common/NativeLibraries.html)。
+
+Ozone 依赖的 Hadoop 版本，由 pom.xml 里的 `hadoop.version` 变量定义. 在启用 Datanode 本地短路读功能前，从对应的 Hadoop 版本发布获取对应的libhadoop.so 文件，将该文件放置在任一 Java 变量 `java.library.path` 定义的目录下。`java.library.path` 的默认值取决于操作系统和 JAVA 版本。例如，在 Linux 上 OpenJDK 8 的默认值是 `/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib`。
+
+在放置好 `libhadoop.so` 后，可使用命令 `ozone checknative` 来查看 `libhadoop.so` 是否能被 Ozone的服务进程正确的搜寻和加载到。
+
+
+## 配置
+
+Datanode 本地短路读功能需要在客户端和 Datanode 端同时配置。 默认情况下，它是关闭的。
+
+```XML
+<property>
+ <name>ozone.client.read.short-circuit</name>
+ <value>false</value>
+ <description>Disable or enable the short-circuit local read feature.</description>
+</property>
+```
+
+Datanode 本地短路读基于 UNIX domain socket。以下变量将配置 domain socket 路径。
+
+```XML
+<property>
+ <name>ozone.domain.socket.path</name>
+ <value></value>
+ <description>This is a path to a UNIX domain socket that will be used for
+ communication between the Datanode and local Ozone clients.
+ If the string "_PORT" is present in this path, it will be replaced by the TCP port of the Datanode.
+ </description>
+</property>
+```
+
+Datanode 需要能创建该路径. 同时，除了启动 Ozone 服务的用户和 root 用户，其他用户不能创建该路径。 由于有这些限制，路径经常使用 `/var/run` 或者 `/var/lib` 下的子目录。
+
+如果将 `ozone.domain.socket.path` 值设置成比如 `/dir1/dir2/ozone_dn_socket`，请确保 `dir1` 和 `dir2` 是已存在的目录，并且 `dir2` 下还没有 `ozone_dn_socket` 文件。 `ozone_dn_socket` 将在 Datanode 启动的时候由 Datanode 创建。
+
+### 参考配置
+可参考如下配置，启用短路读功能。
+
+```XML
+<property>
+ <name>ozone.client.read.short-circuit</name>
+ <value>false</value>
+</property>
+<property>
+ <name>ozone.domain.socket.path</name>
+ <value>/var/run/ozone_dn_socket</value>
+</property>
+```
+
+### 安全考量
+
+为了确保数据的安全和完整性，Ozone 在 `ozone.domain.socket.path` 路径的权限检查上，将遵守和 Hadoop [Socket路径安全](https://wiki.apache.org/hadoop/SocketPathSecurity) 一样的规则。 如果 `ozone.domain.socket.path` 路径权限检查失败，该功能将自动关闭。 检查失败返回的信息包含修复问题的指引，例如
+
+`The path component: '/etc/hadoop' in '/etc/hadoop/ozone_dn_socket' has permissions 0777 uid 0 and gid 0. It is not protected because it is world-writable. This might help: 'chmod o-w /etc/hadoop'. For more information: https://wiki.apache.org/hadoop/SocketPathSecurity`
Review Comment:
The translation page is good. The last paragraph needs to update too.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-11626. Document short-circuit read feature [ozone]
Posted by "errose28 (via GitHub)" <gi...@apache.org>.
errose28 commented on code in PR #7529:
URL: https://github.com/apache/ozone/pull/7529#discussion_r1925723835
##########
hadoop-hdds/docs/content/feature/Short-Circuit-Read.md:
##########
@@ -0,0 +1,78 @@
+---
+title: "Short Circuit Local Read in Datanode"
+weight: 2
+menu:
+ main:
+ parent: Features
+summary: Introduction to Ozone Datanode Short Circuit Local Read Feature
+---
+<!---
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+
+By default, client reads data over GRPC from the Datanode. When the client asks the Datanode to read a file, the DataNode reads that file off of the disk and sends the data to the client over a GRPC connection.
Review Comment:
```suggestion
By default, client reads data over GRPC from the Datanode. When the client asks the Datanode to read a file, the Datanode reads that file off of the disk and sends the data to the client over a GRPC connection.
```
##########
hadoop-hdds/docs/content/design/short-circuit-read.md:
##########
@@ -0,0 +1,29 @@
+---
+title: Short Circuit Local Read in DN
+summary: Support read data from local disk file directly when the client and data are co-located on the same server
+date: 2024-12-04
+jira: HDDS-10685
+status: implemented
+
+---
+<!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+# Abstract
+
+This “short-circuit” local read feature bypassing the Datanode, allows the client to read the file from local disk directly when the client is co-located with the data on the same server.
Review Comment:
```suggestion
The short-circuit local read feature bypasses the Datanode, allowing the client to read the file from local disk directly when the client is co-located with the data on the same server.
```
##########
hadoop-hdds/docs/content/feature/Short-Circuit-Read.md:
##########
@@ -0,0 +1,78 @@
+---
+title: "Short Circuit Local Read in Datanode"
+weight: 2
+menu:
+ main:
+ parent: Features
+summary: Introduction to Ozone Datanode Short Circuit Local Read Feature
+---
+<!---
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+
+By default, client reads data over GRPC from the Datanode. When the client asks the Datanode to read a file, the DataNode reads that file off of the disk and sends the data to the client over a GRPC connection.
+
+This “short-circuit” local read feature will bypass the DataNode, allowing the client to read the file from local disk directly when the client is co-located with the data on the same server.
Review Comment:
```suggestion
Short-circuit local read will bypass the Datanode, allowing the client to read the file from the local disk directly when the client is co-located with the data on the same server.
```
##########
hadoop-hdds/docs/content/feature/Short-Circuit-Read.md:
##########
@@ -0,0 +1,78 @@
+---
+title: "Short Circuit Local Read in Datanode"
+weight: 2
+menu:
+ main:
+ parent: Features
+summary: Introduction to Ozone Datanode Short Circuit Local Read Feature
+---
+<!---
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+
+By default, client reads data over GRPC from the Datanode. When the client asks the Datanode to read a file, the DataNode reads that file off of the disk and sends the data to the client over a GRPC connection.
+
+This “short-circuit” local read feature will bypass the DataNode, allowing the client to read the file from local disk directly when the client is co-located with the data on the same server.
+
+Short-circuit local read can provide a substantial performance boost to many applications, by removing the overhead of network communication.
+
+## Prerequisite
+
+Short-circuit local reads make use of a UNIX domain socket. This is a special path in the filesystem that allows the client and the DataNodes to communicate.
+
+The Hadoop native library `libhadoop.so` provides support to for Unix domain sockets. Please refer to Hadoop's [Native Libraries Guide](https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-common/NativeLibraries.html) for details.
+
+The Hadoop version used in Ozone is defined by `hadoop.version` in pom.xml. Before enabling short-circuit local reads, find the `libhadoop.so` from the corresponding version Hadoop release package, put it under one of the directories specified by Java `java.library.path` property. The default value of `java.library.path` depends on the OS and Java version. For example, on Linux with OpenJDK 8 it is `/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib`.
Review Comment:
Does `hadoop.version` have a default value, or are users required to populate it?
```suggestion
The Hadoop version used in Ozone is defined by `hadoop.version` in pom.xml. Before enabling short-circuit local reads, find the `libhadoop.so` from the release package of the corresponding Hadoop version, put it under one of the directories specified by Java's `java.library.path` property. The default value of `java.library.path` depends on the OS and Java version. For example, on Linux with OpenJDK 8 it is `/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib`.
```
##########
hadoop-hdds/docs/content/feature/Short-Circuit-Read.md:
##########
@@ -0,0 +1,78 @@
+---
+title: "Short Circuit Local Read in Datanode"
+weight: 2
+menu:
+ main:
+ parent: Features
+summary: Introduction to Ozone Datanode Short Circuit Local Read Feature
+---
+<!---
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+
+By default, client reads data over GRPC from the Datanode. When the client asks the Datanode to read a file, the DataNode reads that file off of the disk and sends the data to the client over a GRPC connection.
+
+This “short-circuit” local read feature will bypass the DataNode, allowing the client to read the file from local disk directly when the client is co-located with the data on the same server.
+
+Short-circuit local read can provide a substantial performance boost to many applications, by removing the overhead of network communication.
+
+## Prerequisite
+
+Short-circuit local reads make use of a UNIX domain socket. This is a special path in the filesystem that allows the client and the DataNodes to communicate.
+
+The Hadoop native library `libhadoop.so` provides support to for Unix domain sockets. Please refer to Hadoop's [Native Libraries Guide](https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-common/NativeLibraries.html) for details.
+
+The Hadoop version used in Ozone is defined by `hadoop.version` in pom.xml. Before enabling short-circuit local reads, find the `libhadoop.so` from the corresponding version Hadoop release package, put it under one of the directories specified by Java `java.library.path` property. The default value of `java.library.path` depends on the OS and Java version. For example, on Linux with OpenJDK 8 it is `/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib`.
+
+The `ozone checknative` command can be used to detect whether `libhadoop.so` can be found and loaded successfully by Ozone service.
+
+
+## Configuration
+
+Short-circuit local reads need to be configured on both the DataNode and the client. By default, it is disabled.
+
+```XML
+<property>
+ <name>ozone.client.read.short-circuit</name>
+ <value>false</value>
+ <description>Disable or enable the short-circuit local read feature.</description>
+</property>
+```
+
+It makes use of a UNIX domain socket, a special path in the filesystem. You will need to set a path to this socket.
+
+```XML
+<property>
+ <name>ozone.domain.socket.path</name>
+ <value></value>
+ <description> This is a path to a UNIX domain socket that will be used for
+ communication between the DataNode and local Ozone clients.
+ If the string "_PORT" is present in this path, it will be replaced by the TCP port of the DataNode.
+ </description>
+</property>
+```
+
+The DataNode needs to be able to create this path. On the other hand, it should not be possible for any user except the Ozone user(user who launches Ozone service) or root to create this path. For this reason, paths under `/var/run` or `/var/lib` are often used.
Review Comment:
```suggestion
The Datanode needs to be able to create this path. On the other hand, it should not be possible for any user except the user who launches Ozone service or root to create this path. For this reason, paths under `/var/run` or `/var/lib` are often used.
```
##########
hadoop-hdds/docs/content/feature/Short-Circuit-Read.md:
##########
@@ -0,0 +1,78 @@
+---
+title: "Short Circuit Local Read in Datanode"
+weight: 2
+menu:
+ main:
+ parent: Features
+summary: Introduction to Ozone Datanode Short Circuit Local Read Feature
+---
+<!---
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+
+By default, client reads data over GRPC from the Datanode. When the client asks the Datanode to read a file, the DataNode reads that file off of the disk and sends the data to the client over a GRPC connection.
+
+This “short-circuit” local read feature will bypass the DataNode, allowing the client to read the file from local disk directly when the client is co-located with the data on the same server.
+
+Short-circuit local read can provide a substantial performance boost to many applications, by removing the overhead of network communication.
+
+## Prerequisite
Review Comment:
Let's combine this all under the configuration section and make it a step-by-step list, which will be easier for users to follow.
##########
hadoop-hdds/docs/content/feature/Short-Circuit-Read.md:
##########
@@ -0,0 +1,76 @@
+---
+title: "Short Circuit Local Read in Datanode"
+weight: 2
+menu:
+ main:
+ parent: Features
+summary: Introduction to Ozone Datanode Short Circuit Local Read Feature
+---
+<!---
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+
+Current in Ozone, client reads data over GRPC from Datanode. When the client asks the DataNode to read a file, the DataNode reads that file off of the disk and sends the data to the client over GRPC connection.
+
+This “short-circuit” local read feature will bypass the DataNode, allowing the client to read the file from local disk directly when the client is co-located with the data on the same server.
+
+Short-circuit local read can provide a substantial performance boost to many applications, by removing the overhead of network communication.
+
+## Prerequisite
+
+Short-circuit local reads make use of a UNIX domain socket. This is a special path in the filesystem that allows the client and the DataNodes to communicate.
+
+The Hadoop native library "libhadoop.so" provides the support to use the Unix domain socket. Please refer to Native Libraries ("https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-common/NativeLibraries.html") for details of this library.
+
+Before enabling short-circuit local reads, you must have a proper libhadoop.so, and make sure it's under the directory where Java can find and load it through "System.loadLibrary()" call.
+
+The paths that Java will search for libraries are specified by the "java.library.path" property. The default value of "java.library.path" depends on the OS and Java version. For example, on Linux with OpenJDK 8 it is `/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib`.
+
+Command "ozone checknative" can be used to detect whether libhadoop.so can be loaded successfully by Ozone service.
+
+## Configuration
+
+Short-circuit local reads need to be configured on both the DataNode and the client. By default, it is disabled.
Review Comment:
Shipping with it disabled in Ozone is fine, my comment here is that by copy/pasting the config snippet provided in the doc, the feature should work. This is currently not the case because the doc has it disabled. We should combine both config examples here into one block and provide a default path. The [Hadoop docs](https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-hdfs/ShortCircuitLocalReads.html) do this under the Example Configuration section.
##########
hadoop-hdds/client/src/main/java/org/apache/hadoop/hdds/scm/OzoneClientConfig.java:
##########
@@ -39,6 +39,60 @@ public class OzoneClientConfig {
private static final Logger LOG =
LoggerFactory.getLogger(OzoneClientConfig.class);
+ public static final boolean OZONE_READ_SHORT_CIRCUIT_DEFAULT = false;
Review Comment:
Are these code changes supposed to be here? Let's keep the code and docs PRs separate
##########
hadoop-hdds/docs/content/feature/Short-Circuit-Read.md:
##########
@@ -0,0 +1,78 @@
+---
+title: "Short Circuit Local Read in Datanode"
+weight: 2
+menu:
+ main:
+ parent: Features
+summary: Introduction to Ozone Datanode Short Circuit Local Read Feature
+---
+<!---
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+
+By default, client reads data over GRPC from the Datanode. When the client asks the Datanode to read a file, the DataNode reads that file off of the disk and sends the data to the client over a GRPC connection.
+
+This “short-circuit” local read feature will bypass the DataNode, allowing the client to read the file from local disk directly when the client is co-located with the data on the same server.
Review Comment:
Is short circuit supposed to have a dash? We should standardize this between the title of the page and its content. Also I don't think it should be quoted.
##########
hadoop-hdds/docs/content/feature/Short-Circuit-Read.md:
##########
@@ -0,0 +1,78 @@
+---
+title: "Short Circuit Local Read in Datanode"
+weight: 2
+menu:
+ main:
+ parent: Features
+summary: Introduction to Ozone Datanode Short Circuit Local Read Feature
+---
+<!---
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+
+By default, client reads data over GRPC from the Datanode. When the client asks the Datanode to read a file, the DataNode reads that file off of the disk and sends the data to the client over a GRPC connection.
+
+This “short-circuit” local read feature will bypass the DataNode, allowing the client to read the file from local disk directly when the client is co-located with the data on the same server.
+
+Short-circuit local read can provide a substantial performance boost to many applications, by removing the overhead of network communication.
Review Comment:
```suggestion
Short-circuit local read can provide a substantial performance boost to many applications by removing the overhead of network communication.
```
##########
hadoop-hdds/docs/content/feature/Short-Circuit-Read.md:
##########
@@ -0,0 +1,78 @@
+---
+title: "Short Circuit Local Read in Datanode"
+weight: 2
+menu:
+ main:
+ parent: Features
+summary: Introduction to Ozone Datanode Short Circuit Local Read Feature
+---
+<!---
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+
+By default, client reads data over GRPC from the Datanode. When the client asks the Datanode to read a file, the DataNode reads that file off of the disk and sends the data to the client over a GRPC connection.
Review Comment:
Similar comments throughout the doc. Datanode should not be pascal case.
##########
hadoop-hdds/docs/content/feature/Short-Circuit-Read.md:
##########
@@ -0,0 +1,78 @@
+---
+title: "Short Circuit Local Read in Datanode"
+weight: 2
+menu:
+ main:
+ parent: Features
+summary: Introduction to Ozone Datanode Short Circuit Local Read Feature
+---
+<!---
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+
+By default, client reads data over GRPC from the Datanode. When the client asks the Datanode to read a file, the DataNode reads that file off of the disk and sends the data to the client over a GRPC connection.
+
+This “short-circuit” local read feature will bypass the DataNode, allowing the client to read the file from local disk directly when the client is co-located with the data on the same server.
+
+Short-circuit local read can provide a substantial performance boost to many applications, by removing the overhead of network communication.
+
+## Prerequisite
+
+Short-circuit local reads make use of a UNIX domain socket. This is a special path in the filesystem that allows the client and the DataNodes to communicate.
+
+The Hadoop native library `libhadoop.so` provides support to for Unix domain sockets. Please refer to Hadoop's [Native Libraries Guide](https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-common/NativeLibraries.html) for details.
+
+The Hadoop version used in Ozone is defined by `hadoop.version` in pom.xml. Before enabling short-circuit local reads, find the `libhadoop.so` from the corresponding version Hadoop release package, put it under one of the directories specified by Java `java.library.path` property. The default value of `java.library.path` depends on the OS and Java version. For example, on Linux with OpenJDK 8 it is `/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib`.
+
+The `ozone checknative` command can be used to detect whether `libhadoop.so` can be found and loaded successfully by Ozone service.
+
+
+## Configuration
+
+Short-circuit local reads need to be configured on both the DataNode and the client. By default, it is disabled.
+
+```XML
+<property>
+ <name>ozone.client.read.short-circuit</name>
+ <value>false</value>
+ <description>Disable or enable the short-circuit local read feature.</description>
+</property>
+```
+
+It makes use of a UNIX domain socket, a special path in the filesystem. You will need to set a path to this socket.
+
+```XML
+<property>
+ <name>ozone.domain.socket.path</name>
+ <value></value>
+ <description> This is a path to a UNIX domain socket that will be used for
+ communication between the DataNode and local Ozone clients.
+ If the string "_PORT" is present in this path, it will be replaced by the TCP port of the DataNode.
+ </description>
+</property>
+```
+
+The DataNode needs to be able to create this path. On the other hand, it should not be possible for any user except the Ozone user(user who launches Ozone service) or root to create this path. For this reason, paths under `/var/run` or `/var/lib` are often used.
+
+If you configure the `ozone.domain.socket.path` to some value, for example `/dir1/dir2/ozone_dn_socket`, please make sure that both `dir1` and `dir2` are existing directories, but the file `ozone_dn_socket` does not exist under `dir2`. `ozone_dn_socket` will be created by Ozone Datanode later during its startup.
+
+### Security Consideration
+
+To ensure data security and integrity, Ozone will follow the same rules as Hadoop to check permission on the `ozone.domain.socket.path` path as documented in [Socket Path Security](https://wiki.apache.org/hadoop/SocketPathSecurity).
+It will fail the `ozone.domain.socket.path` verification and disable the feature if the filesystem permissions of the specified path are inadequate.
+The verification failure message carries detail instruction about how to fix the problem. Following is an example,
Review Comment:
These should be soft wrapped. The hard wrapping is not preserved in the rendered markdown and it makes the text harder to edit later. For easier editing set your markdown editor to use soft wraps.
```suggestion
To ensure data security and integrity, Ozone will follow the same rules as Hadoop to check permissions on the `ozone.domain.socket.path` path as documented in [Socket Path Security](https://wiki.apache.org/hadoop/SocketPathSecurity). It will fail the `ozone.domain.socket.path` verification and disable the feature if the filesystem permissions of the specified path are inadequate. The verification failure message carries detailed instructions about how to fix the problem. Following is an example:
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-11626. Document short-circuit read feature [ozone]
Posted by "ChenSammi (via GitHub)" <gi...@apache.org>.
ChenSammi merged PR #7529:
URL: https://github.com/apache/ozone/pull/7529
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-11626. Document short-circuit read feature [ozone]
Posted by "errose28 (via GitHub)" <gi...@apache.org>.
errose28 commented on code in PR #7529:
URL: https://github.com/apache/ozone/pull/7529#discussion_r1929008627
##########
hadoop-hdds/docs/content/feature/Short-Circuit-Read.md:
##########
@@ -0,0 +1,78 @@
+---
+title: "Short Circuit Local Read in Datanode"
+weight: 2
+menu:
+ main:
+ parent: Features
+summary: Introduction to Ozone Datanode Short Circuit Local Read Feature
+---
+<!---
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+
+By default, client reads data over GRPC from the Datanode. When the client asks the Datanode to read a file, the DataNode reads that file off of the disk and sends the data to the client over a GRPC connection.
Review Comment:
I think you misread the comment. It is [here](https://github.com/apache/ozone/pull/7529#discussion_r1870390885) and unedited.
> We should standardize on Datanode. The new website strictly enforces this
[Here](https://github.com/apache/ozone-site/blob/d5d14a76aa16420b1e69bac06b2bcd1a5f5cb698/.markdownlint.yaml#L71) is where the capitalization check is enforced on the new website
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-11626. Document short-circuit read feature [ozone]
Posted by "ChenSammi (via GitHub)" <gi...@apache.org>.
ChenSammi commented on code in PR #7529:
URL: https://github.com/apache/ozone/pull/7529#discussion_r1928022014
##########
hadoop-hdds/docs/content/feature/Short-Circuit-Read.md:
##########
@@ -0,0 +1,78 @@
+---
+title: "Short Circuit Local Read in Datanode"
+weight: 2
+menu:
+ main:
+ parent: Features
+summary: Introduction to Ozone Datanode Short Circuit Local Read Feature
+---
+<!---
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+
+By default, client reads data over GRPC from the Datanode. When the client asks the Datanode to read a file, the DataNode reads that file off of the disk and sends the data to the client over a GRPC connection.
Review Comment:
Ethan, you previously have left comment about prefer "DataNode" over "Datanode". So I changed all "Datanode" to "DataNode".
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-11626. Document short-circuit read feature [ozone]
Posted by "ChenSammi (via GitHub)" <gi...@apache.org>.
ChenSammi commented on PR #7529:
URL: https://github.com/apache/ozone/pull/7529#issuecomment-2527562592
@errose28 , thanks for the review. The patch is improved accordingly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-11626. Document short-circuit read feature [ozone]
Posted by "ChenSammi (via GitHub)" <gi...@apache.org>.
ChenSammi commented on code in PR #7529:
URL: https://github.com/apache/ozone/pull/7529#discussion_r1929944474
##########
hadoop-hdds/docs/content/feature/Short-Circuit-Read.md:
##########
@@ -0,0 +1,78 @@
+---
+title: "Short Circuit Local Read in Datanode"
+weight: 2
+menu:
+ main:
+ parent: Features
+summary: Introduction to Ozone Datanode Short Circuit Local Read Feature
+---
+<!---
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+
+By default, client reads data over GRPC from the Datanode. When the client asks the Datanode to read a file, the DataNode reads that file off of the disk and sends the data to the client over a GRPC connection.
+
+This “short-circuit” local read feature will bypass the DataNode, allowing the client to read the file from local disk directly when the client is co-located with the data on the same server.
+
+Short-circuit local read can provide a substantial performance boost to many applications, by removing the overhead of network communication.
+
+## Prerequisite
+
+Short-circuit local reads make use of a UNIX domain socket. This is a special path in the filesystem that allows the client and the DataNodes to communicate.
+
+The Hadoop native library `libhadoop.so` provides support to for Unix domain sockets. Please refer to Hadoop's [Native Libraries Guide](https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-common/NativeLibraries.html) for details.
+
+The Hadoop version used in Ozone is defined by `hadoop.version` in pom.xml. Before enabling short-circuit local reads, find the `libhadoop.so` from the corresponding version Hadoop release package, put it under one of the directories specified by Java `java.library.path` property. The default value of `java.library.path` depends on the OS and Java version. For example, on Linux with OpenJDK 8 it is `/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib`.
Review Comment:
It's Ozone project's pom.xml, not ozone-site.xml. User don't need to add this key and value. The statement here is to guide the user to find the proper libhadoop.so file from hadoop releases based on the "hadoop.version" used in Ozone project pom.xml.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-11626. Document short-circuit read feature [ozone]
Posted by "errose28 (via GitHub)" <gi...@apache.org>.
errose28 commented on code in PR #7529:
URL: https://github.com/apache/ozone/pull/7529#discussion_r1929012100
##########
hadoop-hdds/docs/content/feature/Short-Circuit-Read.md:
##########
@@ -0,0 +1,78 @@
+---
+title: "Short Circuit Local Read in Datanode"
+weight: 2
+menu:
+ main:
+ parent: Features
+summary: Introduction to Ozone Datanode Short Circuit Local Read Feature
+---
+<!---
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+
+By default, client reads data over GRPC from the Datanode. When the client asks the Datanode to read a file, the DataNode reads that file off of the disk and sends the data to the client over a GRPC connection.
+
+This “short-circuit” local read feature will bypass the DataNode, allowing the client to read the file from local disk directly when the client is co-located with the data on the same server.
+
+Short-circuit local read can provide a substantial performance boost to many applications, by removing the overhead of network communication.
+
+## Prerequisite
+
+Short-circuit local reads make use of a UNIX domain socket. This is a special path in the filesystem that allows the client and the DataNodes to communicate.
+
+The Hadoop native library `libhadoop.so` provides support to for Unix domain sockets. Please refer to Hadoop's [Native Libraries Guide](https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-common/NativeLibraries.html) for details.
+
+The Hadoop version used in Ozone is defined by `hadoop.version` in pom.xml. Before enabling short-circuit local reads, find the `libhadoop.so` from the corresponding version Hadoop release package, put it under one of the directories specified by Java `java.library.path` property. The default value of `java.library.path` depends on the OS and Java version. For example, on Linux with OpenJDK 8 it is `/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib`.
Review Comment:
Yes but Ozone ships with an empty pom by default. So I'm assuming the user is required to add this key and value and there's no default built into the code, but the docs don't specify this. The current phrasing sounds like the user should just check the xml file and the value will already be there.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-11626. Document short-circuit read feature [ozone]
Posted by "jojochuang (via GitHub)" <gi...@apache.org>.
jojochuang commented on PR #7529:
URL: https://github.com/apache/ozone/pull/7529#issuecomment-2549166779
cc @LZD-PratyushBhatt
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-11626. Document short-circuit read feature [ozone]
Posted by "ChenSammi (via GitHub)" <gi...@apache.org>.
ChenSammi commented on code in PR #7529:
URL: https://github.com/apache/ozone/pull/7529#discussion_r1929944474
##########
hadoop-hdds/docs/content/feature/Short-Circuit-Read.md:
##########
@@ -0,0 +1,78 @@
+---
+title: "Short Circuit Local Read in Datanode"
+weight: 2
+menu:
+ main:
+ parent: Features
+summary: Introduction to Ozone Datanode Short Circuit Local Read Feature
+---
+<!---
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+
+By default, client reads data over GRPC from the Datanode. When the client asks the Datanode to read a file, the DataNode reads that file off of the disk and sends the data to the client over a GRPC connection.
+
+This “short-circuit” local read feature will bypass the DataNode, allowing the client to read the file from local disk directly when the client is co-located with the data on the same server.
+
+Short-circuit local read can provide a substantial performance boost to many applications, by removing the overhead of network communication.
+
+## Prerequisite
+
+Short-circuit local reads make use of a UNIX domain socket. This is a special path in the filesystem that allows the client and the DataNodes to communicate.
+
+The Hadoop native library `libhadoop.so` provides support to for Unix domain sockets. Please refer to Hadoop's [Native Libraries Guide](https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-common/NativeLibraries.html) for details.
+
+The Hadoop version used in Ozone is defined by `hadoop.version` in pom.xml. Before enabling short-circuit local reads, find the `libhadoop.so` from the corresponding version Hadoop release package, put it under one of the directories specified by Java `java.library.path` property. The default value of `java.library.path` depends on the OS and Java version. For example, on Linux with OpenJDK 8 it is `/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib`.
Review Comment:
It's Ozone project's pom.xml, not ozone-site.xml.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-11626. Document short-circuit read feature [ozone]
Posted by "ChenSammi (via GitHub)" <gi...@apache.org>.
ChenSammi commented on code in PR #7529:
URL: https://github.com/apache/ozone/pull/7529#discussion_r1929944474
##########
hadoop-hdds/docs/content/feature/Short-Circuit-Read.md:
##########
@@ -0,0 +1,78 @@
+---
+title: "Short Circuit Local Read in Datanode"
+weight: 2
+menu:
+ main:
+ parent: Features
+summary: Introduction to Ozone Datanode Short Circuit Local Read Feature
+---
+<!---
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+
+By default, client reads data over GRPC from the Datanode. When the client asks the Datanode to read a file, the DataNode reads that file off of the disk and sends the data to the client over a GRPC connection.
+
+This “short-circuit” local read feature will bypass the DataNode, allowing the client to read the file from local disk directly when the client is co-located with the data on the same server.
+
+Short-circuit local read can provide a substantial performance boost to many applications, by removing the overhead of network communication.
+
+## Prerequisite
+
+Short-circuit local reads make use of a UNIX domain socket. This is a special path in the filesystem that allows the client and the DataNodes to communicate.
+
+The Hadoop native library `libhadoop.so` provides support to for Unix domain sockets. Please refer to Hadoop's [Native Libraries Guide](https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-common/NativeLibraries.html) for details.
+
+The Hadoop version used in Ozone is defined by `hadoop.version` in pom.xml. Before enabling short-circuit local reads, find the `libhadoop.so` from the corresponding version Hadoop release package, put it under one of the directories specified by Java `java.library.path` property. The default value of `java.library.path` depends on the OS and Java version. For example, on Linux with OpenJDK 8 it is `/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib`.
Review Comment:
It's Ozone project's pom.xml, not ozone-site.xml. User don't need to add this key and value. The statement here is to guide the user to find the proper libhadoop.so file from hadoop releases.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-11626. Document short-circuit read feature [ozone]
Posted by "ChenSammi (via GitHub)" <gi...@apache.org>.
ChenSammi commented on PR #7529:
URL: https://github.com/apache/ozone/pull/7529#issuecomment-2673336222
Thanks @jojochuang , @errose28 , @ashishkumar50 , @LZD-PratyushBhatt for the review.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-11626. Document short-circuit read feature [ozone]
Posted by "ChenSammi (via GitHub)" <gi...@apache.org>.
ChenSammi commented on PR #7529:
URL: https://github.com/apache/ozone/pull/7529#issuecomment-2527358821
> Thanks for adding the docs @ChenSammi. Can you also file a PR to get this into the new Ozone site as well, under the [performance configuration](https://ozone-site-v2.staged.apache.org/docs/administrator-guide/configuration/performance/) section?
Filed [HDDS-11884](https://issues.apache.org/jira/browse/HDDS-11884) to track it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-11626. Document short-circuit read feature [ozone]
Posted by "ChenSammi (via GitHub)" <gi...@apache.org>.
ChenSammi commented on code in PR #7529:
URL: https://github.com/apache/ozone/pull/7529#discussion_r1875663504
##########
hadoop-hdds/docs/content/feature/Short-Circuit-Read.md:
##########
@@ -0,0 +1,76 @@
+---
+title: "Short Circuit Local Read in Datanode"
+weight: 2
+menu:
+ main:
+ parent: Features
+summary: Introduction to Ozone Datanode Short Circuit Local Read Feature
+---
+<!---
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+
+Current in Ozone, client reads data over GRPC from Datanode. When the client asks the DataNode to read a file, the DataNode reads that file off of the disk and sends the data to the client over GRPC connection.
+
+This “short-circuit” local read feature will bypass the DataNode, allowing the client to read the file from local disk directly when the client is co-located with the data on the same server.
+
+Short-circuit local read can provide a substantial performance boost to many applications, by removing the overhead of network communication.
+
+## Prerequisite
+
+Short-circuit local reads make use of a UNIX domain socket. This is a special path in the filesystem that allows the client and the DataNodes to communicate.
+
+The Hadoop native library "libhadoop.so" provides the support to use the Unix domain socket. Please refer to Native Libraries ("https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-common/NativeLibraries.html") for details of this library.
+
+Before enabling short-circuit local reads, you must have a proper libhadoop.so, and make sure it's under the directory where Java can find and load it through "System.loadLibrary()" call.
+
+The paths that Java will search for libraries are specified by the "java.library.path" property. The default value of "java.library.path" depends on the OS and Java version. For example, on Linux with OpenJDK 8 it is `/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib`.
+
+Command "ozone checknative" can be used to detect whether libhadoop.so can be loaded successfully by Ozone service.
+
+## Configuration
+
+Short-circuit local reads need to be configured on both the DataNode and the client. By default, it is disabled.
+
+```XML
+<property>
+ <name>ozone.client.read.short-circuit</name>
+ <value>false</value>
+ <description>Disable or enable the short-circuit local read feature.</description>
+</property>
+```
+
+It makes use of a UNIX domain socket, a special path in the filesystem. You will need to set a path to this socket.
+
+```XML
+<property>
+ <name>ozone.domain.socket.path</name>
+ <value>/var/lib/ozone_dn_socket</value>
+ <description>The path used to create domain socket.</description>
+</property>
+```
+
+The DataNode needs to be able to create this path. On the other hand, it should not be possible for any user except the Ozone user(user who launches Ozone service) or root to create this path. For this reason, paths under /var/run or /var/lib are often used， just like the current default value "/var/lib/ozone_dn_socket".
+
+If you configure the "ozone.domain.socket.path" to other value, for example "/dir1/dir2/ozone_dn_socket", please make sure that both dir1 and dir2 are exiting directories, and there is no ozone_dn_socket under dir2. ozone_dn_socket will be created by Ozone Datanode later during Datanode start.
+
+### Security Consideration
+
+To ensure data security and integrity, Ozone will follow the rule as Hadoop ("https://cwiki.apache.org/confluence/display/HADOOP2/SocketPathSecurity") on "ozone.domain.socket.path" path permission check.
Review Comment:
The same libhadoop.so is used as Hadoop. No re-implementation of native code part.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-11626. Document short-circuit read feature [ozone]
Posted by "LZD-PratyushBhatt (via GitHub)" <gi...@apache.org>.
LZD-PratyushBhatt commented on code in PR #7529:
URL: https://github.com/apache/ozone/pull/7529#discussion_r1881427563
##########
hadoop-hdds/docs/content/feature/Short-Circuit-Read.md:
##########
@@ -0,0 +1,75 @@
+---
+title: "Short Circuit Local Read in Datanode"
+weight: 2
+menu:
+ main:
+ parent: Features
+summary: Introduction to Ozone Datanode Short Circuit Local Read Feature
+---
+<!---
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+
+By default, client reads data over GRPC from the Datanode. When the client asks the Datanode to read a file, the DataNode reads that file off of the disk and sends the data to the client over a GRPC connection.
+
+This “short-circuit” local read feature will bypass the DataNode, allowing the client to read the file from local disk directly when the client is co-located with the data on the same server.
+
+Short-circuit local read can provide a substantial performance boost to many applications, by removing the overhead of network communication.
+
+## Prerequisite
+
+Short-circuit local reads make use of a UNIX domain socket. This is a special path in the filesystem that allows the client and the DataNodes to communicate.
+
+The Hadoop native library `libhadoop.so` provides support to for Unix domain sockets. Please refer to Hadoop's [Native Libraries Guide](https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-common/NativeLibraries.html) for details.
+
+The Hadoop version used in Ozone is defined by `hadoop.version` in pom.xml. Before enabling short-circuit local reads, find the `libhadoop.so` from the corresponding version Hadoop release package, put it under one of the directories specified by Java `java.library.path` property. The default value of `java.library.path` depends on the OS and Java version. For example, on Linux with OpenJDK 8 it is `/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib`.
+
+The `ozone checknative` command can be used to detect whether `libhadoop.so` can be found and loaded successfully by Ozone service.
+
+
+## Configuration
+
+Short-circuit local reads need to be configured on both the DataNode and the client. By default, it is disabled.
+
+```XML
+<property>
+ <name>ozone.client.read.short-circuit</name>
+ <value>false</value>
+ <description>Disable or enable the short-circuit local read feature.</description>
+</property>
+```
+
+It makes use of a UNIX domain socket, a special path in the filesystem. You will need to set a path to this socket.
+
+```XML
+<property>
+ <name>ozone.domain.socket.path</name>
+ <value>/var/lib/ozone_dn_socket</value>
+ <description>The path used to create domain socket.</description>
+</property>
+```
+
+The DataNode needs to be able to create this path. On the other hand, it should not be possible for any user except the Ozone user(user who launches Ozone service) or root to create this path. For this reason, paths under `/var/run` or `/var/lib` are often used， just like the current default value `/var/lib/ozone_dn_socket`.
+
+If you configure the `ozone.domain.socket.path` to another value, for example `/dir1/dir2/ozone_dn_socket`, please make sure that both `dir1` and `dir2` are exiting directories, but the file `ozone_dn_socket` does not exist under `dir2`. `ozone_dn_socket` will be created by Ozone Datanode later during its startup.
Review Comment:
Small nit: Typo in 'exiting', it should be 'existing'.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-11626. Document short-circuit read feature [ozone]
Posted by "ashishkumar50 (via GitHub)" <gi...@apache.org>.
ashishkumar50 commented on code in PR #7529:
URL: https://github.com/apache/ozone/pull/7529#discussion_r1959354714
##########
hadoop-hdds/docs/content/feature/Short-Circuit-Read.md:
##########
@@ -0,0 +1,90 @@
+---
+title: "Short Circuit Local Read in Datanode"
+weight: 2
+menu:
+ main:
+ parent: Features
+summary: Introduction to Ozone Datanode Short Circuit Local Read Feature
+---
+<!---
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+
+By default, client reads data over GRPC from the Datanode. When the client asks the Datanode to read a file, the Datanode reads that file off of the disk and sends the data to the client over a GRPC connection.
+
+This short-circuit local read feature will bypass the Datanode, allowing the client to read the file from local disk directly when the client is co-located with the data on the same server.
+
+Short-circuit local read can provide a substantial performance boost to many applications by removing the overhead of network communication.
+
+## Prerequisite
+
+Short-circuit local reads make use of a UNIX domain socket. This is a special path in the filesystem that allows the client and the DataNodes to communicate.
+
+The Hadoop native library `libhadoop.so` provides support to for Unix domain sockets. Please refer to Hadoop's [Native Libraries Guide](https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-common/NativeLibraries.html) for details.
+
+The Hadoop version used in Ozone is defined by `hadoop.version` in pom.xml. Before enabling short-circuit local reads, find the `libhadoop.so` from the release package of the corresponding Hadoop version, put it under one of the directories specified by Java `java.library.path` property. The default value of `java.library.path` depends on the OS and Java version. For example, on Linux with OpenJDK 8 it is `/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib`.
+
+The `ozone checknative` command can be used to detect whether `libhadoop.so` can be found and loaded successfully by Ozone service.
+
+
+## Configuration
+
+Short-circuit local reads need to be configured on both the Datanode and the client. By default, it is disabled.
+
+```XML
+<property>
+ <name>ozone.client.read.short-circuit</name>
+ <value>false</value>
+ <description>Disable or enable the short-circuit local read feature.</description>
+</property>
+```
+
+It makes use of a UNIX domain socket, a special path in the filesystem. You will need to set a path to this socket.
+
+```XML
+<property>
+ <name>ozone.domain.socket.path</name>
+ <value></value>
+ <description> This is a path to a UNIX domain socket that will be used for
+ communication between the Datanode and local Ozone clients.
+ If the string "_PORT" is present in this path, it will be replaced by the TCP port of the Datanode.
+ </description>
+</property>
+```
+
+The Datanode needs to be able to create this path. On the other hand, it should not be possible for any user except the user who launches Ozone service or root to create this path. For this reason, paths under `/var/run` or `/var/lib` are often used.
+
+If you configure the `ozone.domain.socket.path` to some value, for example `/dir1/dir2/ozone_dn_socket`, please make sure that both `dir1` and `dir2` are existing directories, but the file `ozone_dn_socket` does not exist under `dir2`. `ozone_dn_socket` will be created by Ozone Datanode later during its startup.
+
+### Example Configuration
+To enable short-circuit read, here is an example configuration.
+
+```XML
+<property>
+ <name>ozone.client.read.short-circuit</name>
+ <value>false</value>
Review Comment:
To enable needs to change to `"true"`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-11626. Document short-circuit read feature [ozone]
Posted by "ChenSammi (via GitHub)" <gi...@apache.org>.
ChenSammi commented on code in PR #7529:
URL: https://github.com/apache/ozone/pull/7529#discussion_r1929948043
##########
hadoop-hdds/docs/content/feature/Short-Circuit-Read.md:
##########
@@ -0,0 +1,78 @@
+---
+title: "Short Circuit Local Read in Datanode"
+weight: 2
+menu:
+ main:
+ parent: Features
+summary: Introduction to Ozone Datanode Short Circuit Local Read Feature
+---
+<!---
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+
+By default, client reads data over GRPC from the Datanode. When the client asks the Datanode to read a file, the DataNode reads that file off of the disk and sends the data to the client over a GRPC connection.
Review Comment:
Okay, I changed it oppositely :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org