You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "lohit vijayarenu (JIRA)" <ji...@apache.org> on 2008/04/11 21:22:05 UTC
[jira] Created: (HADOOP-3239) exists() calls logs
FileNotFoundException in namenode log
exists() calls logs FileNotFoundException in namenode log
---------------------------------------------------------
Key: HADOOP-3239
URL: https://issues.apache.org/jira/browse/HADOOP-3239
Project: Hadoop Core
Issue Type: Bug
Affects Versions: 0.17.0
Reporter: lohit vijayarenu
exists() was modified to invoke getFileStatus() internally. But getFileStatus() throws FileNotFoundException for files which does not exists and this is logged in RPC$Server for each exists() call. One way to get rid of these messages is at the Name Node, catch FileNotFoundException and return null. In this case, RPC would not log it in namenode log. But at the client end we might have to check for null on all calls of getFileStatus(). Other option at client end is to construct FileNotFoundException() when getFileInfo() returns null.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3239) exists() calls logs
FileNotFoundException in namenode log
Posted by "Konstantin Shvachko (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3239?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Konstantin Shvachko updated HADOOP-3239:
----------------------------------------
Resolution: Fixed
Hadoop Flags: [Incompatible change, Reviewed] (was: [Incompatible change])
Status: Resolved (was: Patch Available)
I just committed this. Thank you Lohit.
> exists() calls logs FileNotFoundException in namenode log
> ---------------------------------------------------------
>
> Key: HADOOP-3239
> URL: https://issues.apache.org/jira/browse/HADOOP-3239
> Project: Hadoop Core
> Issue Type: Bug
> Affects Versions: 0.17.0
> Reporter: lohit vijayarenu
> Assignee: lohit vijayarenu
> Fix For: 0.17.0
>
> Attachments: HADOOP-3239-1.patch, HADOOP-3239-2.patch
>
>
> exists() was modified to invoke getFileStatus() internally. But getFileStatus() throws FileNotFoundException for files which does not exists and this is logged in RPC$Server for each exists() call. One way to get rid of these messages is at the Name Node, catch FileNotFoundException and return null. In this case, RPC would not log it in namenode log. But at the client end we might have to check for null on all calls of getFileStatus(). Other option at client end is to construct FileNotFoundException() when getFileInfo() returns null.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3239) exists() calls logs
FileNotFoundException in namenode log
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12588086#action_12588086 ]
Doug Cutting commented on HADOOP-3239:
--------------------------------------
I think the namenode should return null for getFileInfo(), without having to catch an exception. And the client should throw an exception for null in getFileStatus() but not in exists().
> exists() calls logs FileNotFoundException in namenode log
> ---------------------------------------------------------
>
> Key: HADOOP-3239
> URL: https://issues.apache.org/jira/browse/HADOOP-3239
> Project: Hadoop Core
> Issue Type: Bug
> Affects Versions: 0.17.0
> Reporter: lohit vijayarenu
>
> exists() was modified to invoke getFileStatus() internally. But getFileStatus() throws FileNotFoundException for files which does not exists and this is logged in RPC$Server for each exists() call. One way to get rid of these messages is at the Name Node, catch FileNotFoundException and return null. In this case, RPC would not log it in namenode log. But at the client end we might have to check for null on all calls of getFileStatus(). Other option at client end is to construct FileNotFoundException() when getFileInfo() returns null.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3239) exists() calls logs
FileNotFoundException in namenode log
Posted by "Raghu Angadi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12588115#action_12588115 ]
Raghu Angadi commented on HADOOP-3239:
--------------------------------------
Last comment :). Also even if some logging is required for some operations, I think it should be logged at NameNode level and not at the RPC level, since only NameNode knows what it means.
> exists() calls logs FileNotFoundException in namenode log
> ---------------------------------------------------------
>
> Key: HADOOP-3239
> URL: https://issues.apache.org/jira/browse/HADOOP-3239
> Project: Hadoop Core
> Issue Type: Bug
> Affects Versions: 0.17.0
> Reporter: lohit vijayarenu
>
> exists() was modified to invoke getFileStatus() internally. But getFileStatus() throws FileNotFoundException for files which does not exists and this is logged in RPC$Server for each exists() call. One way to get rid of these messages is at the Name Node, catch FileNotFoundException and return null. In this case, RPC would not log it in namenode log. But at the client end we might have to check for null on all calls of getFileStatus(). Other option at client end is to construct FileNotFoundException() when getFileInfo() returns null.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3239) exists() calls logs
FileNotFoundException in namenode log
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12589052#action_12589052 ]
Hudson commented on HADOOP-3239:
--------------------------------
Integrated in Hadoop-trunk #461 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-trunk/461/])
> exists() calls logs FileNotFoundException in namenode log
> ---------------------------------------------------------
>
> Key: HADOOP-3239
> URL: https://issues.apache.org/jira/browse/HADOOP-3239
> Project: Hadoop Core
> Issue Type: Bug
> Affects Versions: 0.17.0
> Reporter: lohit vijayarenu
> Assignee: lohit vijayarenu
> Fix For: 0.17.0
>
> Attachments: HADOOP-3239-1.patch, HADOOP-3239-2.patch
>
>
> exists() was modified to invoke getFileStatus() internally. But getFileStatus() throws FileNotFoundException for files which does not exists and this is logged in RPC$Server for each exists() call. One way to get rid of these messages is at the Name Node, catch FileNotFoundException and return null. In this case, RPC would not log it in namenode log. But at the client end we might have to check for null on all calls of getFileStatus(). Other option at client end is to construct FileNotFoundException() when getFileInfo() returns null.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3239) exists() calls logs
FileNotFoundException in namenode log
Posted by "lohit vijayarenu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3239?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
lohit vijayarenu updated HADOOP-3239:
-------------------------------------
Fix Version/s: 0.17.0
> exists() calls logs FileNotFoundException in namenode log
> ---------------------------------------------------------
>
> Key: HADOOP-3239
> URL: https://issues.apache.org/jira/browse/HADOOP-3239
> Project: Hadoop Core
> Issue Type: Bug
> Affects Versions: 0.17.0
> Reporter: lohit vijayarenu
> Assignee: lohit vijayarenu
> Fix For: 0.17.0
>
> Attachments: HADOOP-3239-1.patch
>
>
> exists() was modified to invoke getFileStatus() internally. But getFileStatus() throws FileNotFoundException for files which does not exists and this is logged in RPC$Server for each exists() call. One way to get rid of these messages is at the Name Node, catch FileNotFoundException and return null. In this case, RPC would not log it in namenode log. But at the client end we might have to check for null on all calls of getFileStatus(). Other option at client end is to construct FileNotFoundException() when getFileInfo() returns null.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3239) exists() calls logs
FileNotFoundException in namenode log
Posted by "Konstantin Shvachko (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12588111#action_12588111 ]
Konstantin Shvachko commented on HADOOP-3239:
---------------------------------------------
> should go in 0.17?
Formally, it is not a regression in terms of performance or functionality.
Informally, it is an annoying message, and we will have to answer questions until people will get used to it.
I'd schedule it for 0.17.
> exists() calls logs FileNotFoundException in namenode log
> ---------------------------------------------------------
>
> Key: HADOOP-3239
> URL: https://issues.apache.org/jira/browse/HADOOP-3239
> Project: Hadoop Core
> Issue Type: Bug
> Affects Versions: 0.17.0
> Reporter: lohit vijayarenu
>
> exists() was modified to invoke getFileStatus() internally. But getFileStatus() throws FileNotFoundException for files which does not exists and this is logged in RPC$Server for each exists() call. One way to get rid of these messages is at the Name Node, catch FileNotFoundException and return null. In this case, RPC would not log it in namenode log. But at the client end we might have to check for null on all calls of getFileStatus(). Other option at client end is to construct FileNotFoundException() when getFileInfo() returns null.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3239) exists() calls logs
FileNotFoundException in namenode log
Posted by "Raghu Angadi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12588114#action_12588114 ]
Raghu Angadi commented on HADOOP-3239:
--------------------------------------
> E.g. the FileNotFoundException being thrown in case of open() needs to be logged while thrown in exists() should not be.
I don't think FNF during open() should be logged either. It is not any more severe than FNF during existss(). In fact, many place we discouraged code that first invoked exists() and threw a FNF or opened the file depending on the result.
> exists() calls logs FileNotFoundException in namenode log
> ---------------------------------------------------------
>
> Key: HADOOP-3239
> URL: https://issues.apache.org/jira/browse/HADOOP-3239
> Project: Hadoop Core
> Issue Type: Bug
> Affects Versions: 0.17.0
> Reporter: lohit vijayarenu
>
> exists() was modified to invoke getFileStatus() internally. But getFileStatus() throws FileNotFoundException for files which does not exists and this is logged in RPC$Server for each exists() call. One way to get rid of these messages is at the Name Node, catch FileNotFoundException and return null. In this case, RPC would not log it in namenode log. But at the client end we might have to check for null on all calls of getFileStatus(). Other option at client end is to construct FileNotFoundException() when getFileInfo() returns null.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3239) exists() calls logs
FileNotFoundException in namenode log
Posted by "Raghu Angadi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12588112#action_12588112 ]
Raghu Angadi commented on HADOOP-3239:
--------------------------------------
Yes, I would prefer returning null from getFileInfo() as well. But I don't remember the reason why it was changed couple of months back.
The main issue in this jira I think is about logging in RPC server at the NameNode. What I meant earlier was that returning null or throwing FileNotFound should be independent of RPC logging.
> exists() calls logs FileNotFoundException in namenode log
> ---------------------------------------------------------
>
> Key: HADOOP-3239
> URL: https://issues.apache.org/jira/browse/HADOOP-3239
> Project: Hadoop Core
> Issue Type: Bug
> Affects Versions: 0.17.0
> Reporter: lohit vijayarenu
>
> exists() was modified to invoke getFileStatus() internally. But getFileStatus() throws FileNotFoundException for files which does not exists and this is logged in RPC$Server for each exists() call. One way to get rid of these messages is at the Name Node, catch FileNotFoundException and return null. In this case, RPC would not log it in namenode log. But at the client end we might have to check for null on all calls of getFileStatus(). Other option at client end is to construct FileNotFoundException() when getFileInfo() returns null.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3239) exists() calls logs
FileNotFoundException in namenode log
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12588689#action_12588689 ]
Doug Cutting commented on HADOOP-3239:
--------------------------------------
+1 This looks good to me.
> exists() calls logs FileNotFoundException in namenode log
> ---------------------------------------------------------
>
> Key: HADOOP-3239
> URL: https://issues.apache.org/jira/browse/HADOOP-3239
> Project: Hadoop Core
> Issue Type: Bug
> Affects Versions: 0.17.0
> Reporter: lohit vijayarenu
> Assignee: lohit vijayarenu
> Fix For: 0.17.0
>
> Attachments: HADOOP-3239-1.patch
>
>
> exists() was modified to invoke getFileStatus() internally. But getFileStatus() throws FileNotFoundException for files which does not exists and this is logged in RPC$Server for each exists() call. One way to get rid of these messages is at the Name Node, catch FileNotFoundException and return null. In this case, RPC would not log it in namenode log. But at the client end we might have to check for null on all calls of getFileStatus(). Other option at client end is to construct FileNotFoundException() when getFileInfo() returns null.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3239) exists() calls logs
FileNotFoundException in namenode log
Posted by "Konstantin Shvachko (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12588109#action_12588109 ]
Konstantin Shvachko commented on HADOOP-3239:
---------------------------------------------
> Is it enough if RPC does not log these?
That would be enough if RPC could distinguish between exceptions that need to be logged and those that don't.
But this is impossible in general. E.g. the FileNotFoundException being thrown in case of open() needs to be logged while thrown in exists() should not be.
> exists() calls logs FileNotFoundException in namenode log
> ---------------------------------------------------------
>
> Key: HADOOP-3239
> URL: https://issues.apache.org/jira/browse/HADOOP-3239
> Project: Hadoop Core
> Issue Type: Bug
> Affects Versions: 0.17.0
> Reporter: lohit vijayarenu
>
> exists() was modified to invoke getFileStatus() internally. But getFileStatus() throws FileNotFoundException for files which does not exists and this is logged in RPC$Server for each exists() call. One way to get rid of these messages is at the Name Node, catch FileNotFoundException and return null. In this case, RPC would not log it in namenode log. But at the client end we might have to check for null on all calls of getFileStatus(). Other option at client end is to construct FileNotFoundException() when getFileInfo() returns null.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Issue Comment Edited: (HADOOP-3239) exists() calls logs
FileNotFoundException in namenode log
Posted by "Raghu Angadi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12588143#action_12588143 ]
rangadi edited comment on HADOOP-3239 at 4/11/08 3:02 PM:
---------------------------------------------------------------
You are right. What was changed recently was to throw FileNotFoundException rather than IOException. The extra logging is because of DFSClient.exists() using getFileInfo().
I still think eventually it would be nice to be able to tell RPC not to log user exceptions or be able to tell which ones to log.
was (Author: rangadi):
You are right. What was changed recently was to throw FileNotFoundException rather than IOException. The extra logging is because of DFSClient.exists() using getFileInfo().
I sting think eventually it would be nice to be able to tell RPC not to log user exceptions or be able to tell which ones to log.
> exists() calls logs FileNotFoundException in namenode log
> ---------------------------------------------------------
>
> Key: HADOOP-3239
> URL: https://issues.apache.org/jira/browse/HADOOP-3239
> Project: Hadoop Core
> Issue Type: Bug
> Affects Versions: 0.17.0
> Reporter: lohit vijayarenu
>
> exists() was modified to invoke getFileStatus() internally. But getFileStatus() throws FileNotFoundException for files which does not exists and this is logged in RPC$Server for each exists() call. One way to get rid of these messages is at the Name Node, catch FileNotFoundException and return null. In this case, RPC would not log it in namenode log. But at the client end we might have to check for null on all calls of getFileStatus(). Other option at client end is to construct FileNotFoundException() when getFileInfo() returns null.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3239) exists() calls logs
FileNotFoundException in namenode log
Posted by "Konstantin Shvachko (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12588090#action_12588090 ]
Konstantin Shvachko commented on HADOOP-3239:
---------------------------------------------
I agree with Doug:
- NameNode.getFileInfo() should return null not throwing the FileNotFoundException.
- DFSClient.getFileInfo() should throw FileNotFoundException if the name-node returned null.
We should also decide whether that should go in 0.17?
> exists() calls logs FileNotFoundException in namenode log
> ---------------------------------------------------------
>
> Key: HADOOP-3239
> URL: https://issues.apache.org/jira/browse/HADOOP-3239
> Project: Hadoop Core
> Issue Type: Bug
> Affects Versions: 0.17.0
> Reporter: lohit vijayarenu
>
> exists() was modified to invoke getFileStatus() internally. But getFileStatus() throws FileNotFoundException for files which does not exists and this is logged in RPC$Server for each exists() call. One way to get rid of these messages is at the Name Node, catch FileNotFoundException and return null. In this case, RPC would not log it in namenode log. But at the client end we might have to check for null on all calls of getFileStatus(). Other option at client end is to construct FileNotFoundException() when getFileInfo() returns null.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3239) exists() calls logs
FileNotFoundException in namenode log
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12588107#action_12588107 ]
Doug Cutting commented on HADOOP-3239:
--------------------------------------
> Is it enough if RPC does not log these?
That sounds like a hack. If some consumers of getFileInfo() don't consider FileNotFound exceptional, then returning null is the right contract for that protocol method. As I've said before, using exception handling for normal control flow is a bad design. The exists() method should be able to return false via normal control flow.
> exists() calls logs FileNotFoundException in namenode log
> ---------------------------------------------------------
>
> Key: HADOOP-3239
> URL: https://issues.apache.org/jira/browse/HADOOP-3239
> Project: Hadoop Core
> Issue Type: Bug
> Affects Versions: 0.17.0
> Reporter: lohit vijayarenu
>
> exists() was modified to invoke getFileStatus() internally. But getFileStatus() throws FileNotFoundException for files which does not exists and this is logged in RPC$Server for each exists() call. One way to get rid of these messages is at the Name Node, catch FileNotFoundException and return null. In this case, RPC would not log it in namenode log. But at the client end we might have to check for null on all calls of getFileStatus(). Other option at client end is to construct FileNotFoundException() when getFileInfo() returns null.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3239) exists() calls logs
FileNotFoundException in namenode log
Posted by "lohit vijayarenu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3239?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
lohit vijayarenu updated HADOOP-3239:
-------------------------------------
Attachment: HADOOP-3239-1.patch
Attached patch does these
- getFileInfo at namenode returns null if file not found
- getFileInfo at DFSClient passes null above as is
- exists uses null to identify if file exists
- getFileStatus checks for null and throws FileNotFoundException
- added a testcase to make sure getFileInfo returns null for non existing files
> exists() calls logs FileNotFoundException in namenode log
> ---------------------------------------------------------
>
> Key: HADOOP-3239
> URL: https://issues.apache.org/jira/browse/HADOOP-3239
> Project: Hadoop Core
> Issue Type: Bug
> Affects Versions: 0.17.0
> Reporter: lohit vijayarenu
> Assignee: lohit vijayarenu
> Attachments: HADOOP-3239-1.patch
>
>
> exists() was modified to invoke getFileStatus() internally. But getFileStatus() throws FileNotFoundException for files which does not exists and this is logged in RPC$Server for each exists() call. One way to get rid of these messages is at the Name Node, catch FileNotFoundException and return null. In this case, RPC would not log it in namenode log. But at the client end we might have to check for null on all calls of getFileStatus(). Other option at client end is to construct FileNotFoundException() when getFileInfo() returns null.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3239) exists() calls logs
FileNotFoundException in namenode log
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12588130#action_12588130 ]
Doug Cutting commented on HADOOP-3239:
--------------------------------------
This issue is about log messages generated by calls to exists() that didn't used to be generated by calls to exists(). It is a regression. It is not about namenode exception logging more generally. (About that, however, I think folks have found it convenient that whenever an exception is received from RPC that it is also logged on the server side.)
> getFileInfo() used to return null.
That's not true. It has always thrown an exception for this case, since that was always before the contract of its corresponding FileSystem methods. Now that it is overloaded to support exists() too, it should no longer throw the exception at this level.
http://svn.apache.org/viewvc/lucene/hadoop/trunk/src/java/org/apache/hadoop/dfs/FSDirectory.java?r1=549977&r2=549976&pathrev=549977
> exists() calls logs FileNotFoundException in namenode log
> ---------------------------------------------------------
>
> Key: HADOOP-3239
> URL: https://issues.apache.org/jira/browse/HADOOP-3239
> Project: Hadoop Core
> Issue Type: Bug
> Affects Versions: 0.17.0
> Reporter: lohit vijayarenu
>
> exists() was modified to invoke getFileStatus() internally. But getFileStatus() throws FileNotFoundException for files which does not exists and this is logged in RPC$Server for each exists() call. One way to get rid of these messages is at the Name Node, catch FileNotFoundException and return null. In this case, RPC would not log it in namenode log. But at the client end we might have to check for null on all calls of getFileStatus(). Other option at client end is to construct FileNotFoundException() when getFileInfo() returns null.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3239) exists() calls logs
FileNotFoundException in namenode log
Posted by "Raghu Angadi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12588143#action_12588143 ]
Raghu Angadi commented on HADOOP-3239:
--------------------------------------
You are right. What was changed recently was to throw FileNotFoundException rather than IOException. The extra logging is because of DFSClient.exists() using getFileInfo().
I sting think eventually it would be nice to be able to tell RPC not to log user exceptions or be able to tell which ones to log.
> exists() calls logs FileNotFoundException in namenode log
> ---------------------------------------------------------
>
> Key: HADOOP-3239
> URL: https://issues.apache.org/jira/browse/HADOOP-3239
> Project: Hadoop Core
> Issue Type: Bug
> Affects Versions: 0.17.0
> Reporter: lohit vijayarenu
>
> exists() was modified to invoke getFileStatus() internally. But getFileStatus() throws FileNotFoundException for files which does not exists and this is logged in RPC$Server for each exists() call. One way to get rid of these messages is at the Name Node, catch FileNotFoundException and return null. In this case, RPC would not log it in namenode log. But at the client end we might have to check for null on all calls of getFileStatus(). Other option at client end is to construct FileNotFoundException() when getFileInfo() returns null.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Assigned: (HADOOP-3239) exists() calls logs
FileNotFoundException in namenode log
Posted by "lohit vijayarenu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3239?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
lohit vijayarenu reassigned HADOOP-3239:
----------------------------------------
Assignee: lohit vijayarenu
> exists() calls logs FileNotFoundException in namenode log
> ---------------------------------------------------------
>
> Key: HADOOP-3239
> URL: https://issues.apache.org/jira/browse/HADOOP-3239
> Project: Hadoop Core
> Issue Type: Bug
> Affects Versions: 0.17.0
> Reporter: lohit vijayarenu
> Assignee: lohit vijayarenu
>
> exists() was modified to invoke getFileStatus() internally. But getFileStatus() throws FileNotFoundException for files which does not exists and this is logged in RPC$Server for each exists() call. One way to get rid of these messages is at the Name Node, catch FileNotFoundException and return null. In this case, RPC would not log it in namenode log. But at the client end we might have to check for null on all calls of getFileStatus(). Other option at client end is to construct FileNotFoundException() when getFileInfo() returns null.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3239) exists() calls logs
FileNotFoundException in namenode log
Posted by "Nigel Daley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3239?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Nigel Daley updated HADOOP-3239:
--------------------------------
Component/s: dfs
Hadoop Flags: [Incompatible change, Reviewed] (was: [Reviewed, Incompatible change])
> exists() calls logs FileNotFoundException in namenode log
> ---------------------------------------------------------
>
> Key: HADOOP-3239
> URL: https://issues.apache.org/jira/browse/HADOOP-3239
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs
> Affects Versions: 0.17.0
> Reporter: lohit vijayarenu
> Assignee: lohit vijayarenu
> Fix For: 0.17.0
>
> Attachments: HADOOP-3239-1.patch, HADOOP-3239-2.patch
>
>
> exists() was modified to invoke getFileStatus() internally. But getFileStatus() throws FileNotFoundException for files which does not exists and this is logged in RPC$Server for each exists() call. One way to get rid of these messages is at the Name Node, catch FileNotFoundException and return null. In this case, RPC would not log it in namenode log. But at the client end we might have to check for null on all calls of getFileStatus(). Other option at client end is to construct FileNotFoundException() when getFileInfo() returns null.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3239) exists() calls logs
FileNotFoundException in namenode log
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12588218#action_12588218 ]
Hadoop QA commented on HADOOP-3239:
-----------------------------------
+1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12379975/HADOOP-3239-1.patch
against trunk revision 645773.
@author +1. The patch does not contain any @author tags.
tests included +1. The patch appears to include 3 new or modified tests.
javadoc +1. The javadoc tool did not generate any warning messages.
javac +1. The applied patch does not generate any new javac compiler warnings.
release audit +1. The applied patch does not generate any new release audit warnings.
findbugs +1. The patch does not introduce any new Findbugs warnings.
core tests +1. The patch passed core unit tests.
contrib tests +1. The patch passed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2217/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2217/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2217/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2217/console
This message is automatically generated.
> exists() calls logs FileNotFoundException in namenode log
> ---------------------------------------------------------
>
> Key: HADOOP-3239
> URL: https://issues.apache.org/jira/browse/HADOOP-3239
> Project: Hadoop Core
> Issue Type: Bug
> Affects Versions: 0.17.0
> Reporter: lohit vijayarenu
> Assignee: lohit vijayarenu
> Attachments: HADOOP-3239-1.patch
>
>
> exists() was modified to invoke getFileStatus() internally. But getFileStatus() throws FileNotFoundException for files which does not exists and this is logged in RPC$Server for each exists() call. One way to get rid of these messages is at the Name Node, catch FileNotFoundException and return null. In this case, RPC would not log it in namenode log. But at the client end we might have to check for null on all calls of getFileStatus(). Other option at client end is to construct FileNotFoundException() when getFileInfo() returns null.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3239) exists() calls logs
FileNotFoundException in namenode log
Posted by "lohit vijayarenu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3239?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
lohit vijayarenu updated HADOOP-3239:
-------------------------------------
Attachment: HADOOP-3239-2.patch
Thanks Doug and Konstantine. Attached patch modifies javadoc as suggested by Konstantine
> exists() calls logs FileNotFoundException in namenode log
> ---------------------------------------------------------
>
> Key: HADOOP-3239
> URL: https://issues.apache.org/jira/browse/HADOOP-3239
> Project: Hadoop Core
> Issue Type: Bug
> Affects Versions: 0.17.0
> Reporter: lohit vijayarenu
> Assignee: lohit vijayarenu
> Fix For: 0.17.0
>
> Attachments: HADOOP-3239-1.patch, HADOOP-3239-2.patch
>
>
> exists() was modified to invoke getFileStatus() internally. But getFileStatus() throws FileNotFoundException for files which does not exists and this is logged in RPC$Server for each exists() call. One way to get rid of these messages is at the Name Node, catch FileNotFoundException and return null. In this case, RPC would not log it in namenode log. But at the client end we might have to check for null on all calls of getFileStatus(). Other option at client end is to construct FileNotFoundException() when getFileInfo() returns null.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3239) exists() calls logs
FileNotFoundException in namenode log
Posted by "Raghu Angadi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12588095#action_12588095 ]
Raghu Angadi commented on HADOOP-3239:
--------------------------------------
Is it enough if RPC does not log these?
> exists() calls logs FileNotFoundException in namenode log
> ---------------------------------------------------------
>
> Key: HADOOP-3239
> URL: https://issues.apache.org/jira/browse/HADOOP-3239
> Project: Hadoop Core
> Issue Type: Bug
> Affects Versions: 0.17.0
> Reporter: lohit vijayarenu
>
> exists() was modified to invoke getFileStatus() internally. But getFileStatus() throws FileNotFoundException for files which does not exists and this is logged in RPC$Server for each exists() call. One way to get rid of these messages is at the Name Node, catch FileNotFoundException and return null. In this case, RPC would not log it in namenode log. But at the client end we might have to check for null on all calls of getFileStatus(). Other option at client end is to construct FileNotFoundException() when getFileInfo() returns null.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3239) exists() calls logs
FileNotFoundException in namenode log
Posted by "lohit vijayarenu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3239?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
lohit vijayarenu updated HADOOP-3239:
-------------------------------------
Release Note: getFileInfo returns null for File not found instead of throwing FileNotFoundException
Hadoop Flags: [Incompatible change]
Status: Patch Available (was: Open)
> exists() calls logs FileNotFoundException in namenode log
> ---------------------------------------------------------
>
> Key: HADOOP-3239
> URL: https://issues.apache.org/jira/browse/HADOOP-3239
> Project: Hadoop Core
> Issue Type: Bug
> Affects Versions: 0.17.0
> Reporter: lohit vijayarenu
> Assignee: lohit vijayarenu
> Attachments: HADOOP-3239-1.patch
>
>
> exists() was modified to invoke getFileStatus() internally. But getFileStatus() throws FileNotFoundException for files which does not exists and this is logged in RPC$Server for each exists() call. One way to get rid of these messages is at the Name Node, catch FileNotFoundException and return null. In this case, RPC would not log it in namenode log. But at the client end we might have to check for null on all calls of getFileStatus(). Other option at client end is to construct FileNotFoundException() when getFileInfo() returns null.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3239) exists() calls logs
FileNotFoundException in namenode log
Posted by "Raghu Angadi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12588779#action_12588779 ]
Raghu Angadi commented on HADOOP-3239:
--------------------------------------
+1.
> exists() calls logs FileNotFoundException in namenode log
> ---------------------------------------------------------
>
> Key: HADOOP-3239
> URL: https://issues.apache.org/jira/browse/HADOOP-3239
> Project: Hadoop Core
> Issue Type: Bug
> Affects Versions: 0.17.0
> Reporter: lohit vijayarenu
> Assignee: lohit vijayarenu
> Fix For: 0.17.0
>
> Attachments: HADOOP-3239-1.patch, HADOOP-3239-2.patch
>
>
> exists() was modified to invoke getFileStatus() internally. But getFileStatus() throws FileNotFoundException for files which does not exists and this is logged in RPC$Server for each exists() call. One way to get rid of these messages is at the Name Node, catch FileNotFoundException and return null. In this case, RPC would not log it in namenode log. But at the client end we might have to check for null on all calls of getFileStatus(). Other option at client end is to construct FileNotFoundException() when getFileInfo() returns null.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3239) exists() calls logs
FileNotFoundException in namenode log
Posted by "Konstantin Shvachko (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12588725#action_12588725 ]
Konstantin Shvachko commented on HADOOP-3239:
---------------------------------------------
+1
In JavaDoc the following
{code}
* @return object containing information regarding the file
* null if file not found
{code}
will look like
{code}
Returns:
object containing information regarding the file null if file not found
{code}
I'd say "or null if..." or use a comma.
> exists() calls logs FileNotFoundException in namenode log
> ---------------------------------------------------------
>
> Key: HADOOP-3239
> URL: https://issues.apache.org/jira/browse/HADOOP-3239
> Project: Hadoop Core
> Issue Type: Bug
> Affects Versions: 0.17.0
> Reporter: lohit vijayarenu
> Assignee: lohit vijayarenu
> Fix For: 0.17.0
>
> Attachments: HADOOP-3239-1.patch
>
>
> exists() was modified to invoke getFileStatus() internally. But getFileStatus() throws FileNotFoundException for files which does not exists and this is logged in RPC$Server for each exists() call. One way to get rid of these messages is at the Name Node, catch FileNotFoundException and return null. In this case, RPC would not log it in namenode log. But at the client end we might have to check for null on all calls of getFileStatus(). Other option at client end is to construct FileNotFoundException() when getFileInfo() returns null.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3239) exists() calls logs
FileNotFoundException in namenode log
Posted by "Raghu Angadi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12588092#action_12588092 ]
Raghu Angadi commented on HADOOP-3239:
--------------------------------------
getFileInfo() used to return null. This was changed couple of months back during fixes to various shell commands and APIs. I am not sure the actual reason behind it, may be consistency. Returning null for non-existent files is good.
> exists() calls logs FileNotFoundException in namenode log
> ---------------------------------------------------------
>
> Key: HADOOP-3239
> URL: https://issues.apache.org/jira/browse/HADOOP-3239
> Project: Hadoop Core
> Issue Type: Bug
> Affects Versions: 0.17.0
> Reporter: lohit vijayarenu
>
> exists() was modified to invoke getFileStatus() internally. But getFileStatus() throws FileNotFoundException for files which does not exists and this is logged in RPC$Server for each exists() call. One way to get rid of these messages is at the Name Node, catch FileNotFoundException and return null. In this case, RPC would not log it in namenode log. But at the client end we might have to check for null on all calls of getFileStatus(). Other option at client end is to construct FileNotFoundException() when getFileInfo() returns null.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Issue Comment Edited: (HADOOP-3239) exists() calls logs
FileNotFoundException in namenode log
Posted by "lohit vijayarenu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12588741#action_12588741 ]
lohit edited comment on HADOOP-3239 at 4/14/08 12:40 PM:
--------------------------------------------------------------------
Thanks Doug and Konstantin. Attached patch modifies javadoc as suggested by Konstantin
was (Author: lohit):
Thanks Doug and Konstantine. Attached patch modifies javadoc as suggested by Konstantine
> exists() calls logs FileNotFoundException in namenode log
> ---------------------------------------------------------
>
> Key: HADOOP-3239
> URL: https://issues.apache.org/jira/browse/HADOOP-3239
> Project: Hadoop Core
> Issue Type: Bug
> Affects Versions: 0.17.0
> Reporter: lohit vijayarenu
> Assignee: lohit vijayarenu
> Fix For: 0.17.0
>
> Attachments: HADOOP-3239-1.patch, HADOOP-3239-2.patch
>
>
> exists() was modified to invoke getFileStatus() internally. But getFileStatus() throws FileNotFoundException for files which does not exists and this is logged in RPC$Server for each exists() call. One way to get rid of these messages is at the Name Node, catch FileNotFoundException and return null. In this case, RPC would not log it in namenode log. But at the client end we might have to check for null on all calls of getFileStatus(). Other option at client end is to construct FileNotFoundException() when getFileInfo() returns null.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3239) exists() calls logs
FileNotFoundException in namenode log
Posted by "Raghu Angadi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12588088#action_12588088 ]
Raghu Angadi commented on HADOOP-3239:
--------------------------------------
I don't think RPC should dictate what exception NameNode should throw. I think it should still throw FileNotFound. There is probably a way not to log selected exceptions at the RPC server. In fact the server should not log the exceptions that it is sending the client actually! may be except RuntimeExceptions.
> exists() calls logs FileNotFoundException in namenode log
> ---------------------------------------------------------
>
> Key: HADOOP-3239
> URL: https://issues.apache.org/jira/browse/HADOOP-3239
> Project: Hadoop Core
> Issue Type: Bug
> Affects Versions: 0.17.0
> Reporter: lohit vijayarenu
>
> exists() was modified to invoke getFileStatus() internally. But getFileStatus() throws FileNotFoundException for files which does not exists and this is logged in RPC$Server for each exists() call. One way to get rid of these messages is at the Name Node, catch FileNotFoundException and return null. In this case, RPC would not log it in namenode log. But at the client end we might have to check for null on all calls of getFileStatus(). Other option at client end is to construct FileNotFoundException() when getFileInfo() returns null.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.