You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@orc.apache.org by "cxzl25 (via GitHub)" <gi...@apache.org> on 2024/01/22 11:23:06 UTC
[PR] ORC-1458: Reduce HDFS NameNode getFileInfo RPC [orc]
cxzl25 opened a new pull request, #1767:
URL: https://github.com/apache/orc/pull/1767
### What changes were proposed in this pull request?
If the file in HDFS is in a completed state, avoid calling the HDFS getFileInfo RPC.
Provide `orc.file.length.fast` configuration to enable this behavior.
### Why are the changes needed?
Now reading an ORC file in HDFS will generate at least one `open` and `getFileInfo` RPC. This optimization can remove getFileInfo RPC as much as possible, improve ORC reading efficiency, and reduce the number of HDFS RPCs.
### How was this patch tested?
The production environment has been running stably for several months
### Was this patch authored or co-authored using generative AI tooling?
No
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@orc.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] ORC-1458: Reduce HDFS NameNode getFileInfo RPC [orc]
Posted by "cxzl25 (via GitHub)" <gi...@apache.org>.
cxzl25 commented on code in PR #1767:
URL: https://github.com/apache/orc/pull/1767#discussion_r1464385716
##########
java/core/src/java/org/apache/orc/util/OrcInputStreamUtil.java:
##########
@@ -0,0 +1,79 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.orc.util;
+
+import org.apache.hadoop.fs.FSDataInputStream;
+
+import java.io.InputStream;
+import java.lang.reflect.InvocationTargetException;
+import java.lang.reflect.Method;
+
+public class OrcInputStreamUtil {
+
+ private static final String DFS_CLASS = "org.apache.hadoop.hdfs.DFSInputStream";
Review Comment:
`org.apache.hadoop.hdfs.DFSInputStream.shortCircuitForbidden` Method is not public.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@orc.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] ORC-1458: Reduce HDFS NameNode getFileInfo RPC [orc]
Posted by "cxzl25 (via GitHub)" <gi...@apache.org>.
cxzl25 commented on PR #1767:
URL: https://github.com/apache/orc/pull/1767#issuecomment-1907494810
> This is a high-risky way which Apache Hadoop community is also not recommended.
>
> May I ask if you have any reference in ASF communities which adopts this hack
@Hexiaoqiao Can you help review this PR?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@orc.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] ORC-1458: Reduce HDFS NameNode getFileInfo RPC [orc]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on PR #1767:
URL: https://github.com/apache/orc/pull/1767#issuecomment-2573611479
Thank you for closing, @cxzl25 .
For the following, could you file a JIRA Issue too? If we are aware of issues, the community can make alternative solution .
> BTW S3 has a similar problem.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@orc.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] ORC-1458: Reduce HDFS NameNode getFileInfo RPC [orc]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on code in PR #1767:
URL: https://github.com/apache/orc/pull/1767#discussion_r1462155390
##########
java/core/src/java/org/apache/orc/util/OrcInputStreamUtil.java:
##########
@@ -0,0 +1,79 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.orc.util;
+
+import org.apache.hadoop.fs.FSDataInputStream;
+
+import java.io.InputStream;
+import java.lang.reflect.InvocationTargetException;
+import java.lang.reflect.Method;
+
+public class OrcInputStreamUtil {
+
+ private static final String DFS_CLASS = "org.apache.hadoop.hdfs.DFSInputStream";
Review Comment:
Why do we reflect this when Apache ORC can access this directly?
```
$ javap -cp hadoop-client-api-3.3.6.jar org.apache.hadoop.hdfs.DFSInputStream | head -n3
Compiled from "DFSInputStream.java"
public class org.apache.hadoop.hdfs.DFSInputStream extends org.apache.hadoop.fs.FSInputStream implements org.apache.hadoop.fs.ByteBufferReadable,org.apache.hadoop.fs.CanSetDropBehind,org.apache.hadoop.fs.CanSetReadahead,org.apache.hadoop.fs.HasEnhancedByteBufferAccess,org.apache.hadoop.fs.CanUnbuffer,org.apache.hadoop.fs.StreamCapabilities,org.apache.hadoop.fs.ByteBufferPositionedReadable {
public static boolean tcpReadsDisabledForTesting;
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@orc.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] ORC-1458: Reduce HDFS NameNode getFileInfo RPC [orc]
Posted by "cxzl25 (via GitHub)" <gi...@apache.org>.
cxzl25 commented on PR #1767:
URL: https://github.com/apache/orc/pull/1767#issuecomment-2572254383
Happy New Year. @dongjoon-hyun
> Do you still want to introduce this feature in Apache ORC for your use case
Let me close this PR first, this is a bit tricky, BTW S3 has a similar problem.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@orc.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] ORC-1458: Reduce HDFS NameNode getFileInfo RPC [orc]
Posted by "cxzl25 (via GitHub)" <gi...@apache.org>.
cxzl25 commented on PR #1767:
URL: https://github.com/apache/orc/pull/1767#issuecomment-2603605859
> For the following, could you file a JIRA Issue too? If we are aware of issues, the community can make alternative solution
Sorry for the late reply.
Created https://issues.apache.org/jira/browse/ORC-1845
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@orc.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] ORC-1458: Reduce HDFS NameNode getFileInfo RPC [orc]
Posted by "cxzl25 (via GitHub)" <gi...@apache.org>.
cxzl25 commented on code in PR #1767:
URL: https://github.com/apache/orc/pull/1767#discussion_r1464394725
##########
java/core/src/java/org/apache/orc/util/OrcInputStreamUtil.java:
##########
@@ -0,0 +1,79 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.orc.util;
+
+import org.apache.hadoop.fs.FSDataInputStream;
+
+import java.io.InputStream;
+import java.lang.reflect.InvocationTargetException;
+import java.lang.reflect.Method;
+
+public class OrcInputStreamUtil {
+
+ private static final String DFS_CLASS = "org.apache.hadoop.hdfs.DFSInputStream";
Review Comment:
Before [ORC-1430](https://issues.apache.org/jira/browse/ORC-1430), we could not access hdfs-related classes in the core module, so I used reflection.
Now I have done some refactoring, but the method of `shortCircuitForbidden` is not public, so I still need to use reflection.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@orc.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] ORC-1458: Reduce HDFS NameNode getFileInfo RPC [orc]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on PR #1767:
URL: https://github.com/apache/orc/pull/1767#issuecomment-2571718066
Happy New Year. Do you still want to introduce this feature in Apache ORC for your use case, @cxzl25 ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@orc.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] ORC-1458: Reduce HDFS NameNode getFileInfo RPC [orc]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on PR #1767:
URL: https://github.com/apache/orc/pull/1767#issuecomment-1906571492
Gentle ping, @cxzl25 .
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@orc.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] ORC-1458: Reduce HDFS NameNode getFileInfo RPC [orc]
Posted by "Hexiaoqiao (via GitHub)" <gi...@apache.org>.
Hexiaoqiao commented on PR #1767:
URL: https://github.com/apache/orc/pull/1767#issuecomment-1907590652
Thanks involve me here. I have left my concerns [here](https://issues.apache.org/jira/browse/ORC-1458?focusedCommentId=17739786&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17739786), I would like to reassert: I believe this could reduce load to NameNode when using ORC on HDFS but I am worried it involve any potential issues from HDFS view.
Back to this PR, IIUC this is used only for extract footer, right? Technically, for one immutable file, the result from `DFSInputStream#getFileLength()` and `FileStatus#getLen()` will be same, but it is not one recommended usage as @dongjoon-hyun mentioned above.
Thanks again.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@orc.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] ORC-1458: Reduce HDFS NameNode getFileInfo RPC [orc]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on PR #1767:
URL: https://github.com/apache/orc/pull/1767#issuecomment-2603634375
Thank you, @cxzl25 .
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@orc.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] ORC-1458: Reduce HDFS NameNode getFileInfo RPC [orc]
Posted by "cxzl25 (via GitHub)" <gi...@apache.org>.
cxzl25 closed pull request #1767: ORC-1458: Reduce HDFS NameNode getFileInfo RPC
URL: https://github.com/apache/orc/pull/1767
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@orc.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org