You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "Raghu Angadi (JIRA)" <ji...@apache.org> on 2007/06/02 01:25:15 UTC
[jira] Created: (HADOOP-1453) exists() not necessary before
DFS.open
exists() not necessary before DFS.open
--------------------------------------
Key: HADOOP-1453
URL: https://issues.apache.org/jira/browse/HADOOP-1453
Project: Hadoop
Issue Type: Bug
Components: dfs
Reporter: Raghu Angadi
Fix For: 0.14.0
{code:title=DistributedFileSystem.java:131|borderStyle=solid}
public FSDataInputStream open(Path f, int bufferSize) throws IOException {
if (!exists(f)) {
throw new FileNotFoundException(f.toString());
}
return new DFSClient.DFSDataInputStream(dfs.open(getPath(f)), bufferSize);
}
{code}
{{exists(f)}} adds extra namenode interaction that is not really required. Open is a critical DFS call.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-1453) exists() not necessary before
DFS.open
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1453?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Doug Cutting updated HADOOP-1453:
---------------------------------
Resolution: Fixed
Status: Resolved (was: Patch Available)
I just committed this. Thanks, Raghu!
> exists() not necessary before DFS.open
> --------------------------------------
>
> Key: HADOOP-1453
> URL: https://issues.apache.org/jira/browse/HADOOP-1453
> Project: Hadoop
> Issue Type: Bug
> Components: dfs
> Reporter: Raghu Angadi
> Assignee: Raghu Angadi
> Fix For: 0.14.0
>
> Attachments: HADOOP-1453-01.patch, HADOOP-1453-02.patch, HADOOP-1453-02.patch
>
>
> {code:title=DistributedFileSystem.java:131|borderStyle=solid}
> public FSDataInputStream open(Path f, int bufferSize) throws IOException {
> if (!exists(f)) {
> throw new FileNotFoundException(f.toString());
> }
> return new DFSClient.DFSDataInputStream(dfs.open(getPath(f)), bufferSize);
> }
> {code}
> {{exists(f)}} adds extra namenode interaction that is not really required. Open is a critical DFS call.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-1453) exists() not necessary before
DFS.open
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12506066 ]
dhruba borthakur commented on HADOOP-1453:
------------------------------------------
+1 code looks good.
> exists() not necessary before DFS.open
> --------------------------------------
>
> Key: HADOOP-1453
> URL: https://issues.apache.org/jira/browse/HADOOP-1453
> Project: Hadoop
> Issue Type: Bug
> Components: dfs
> Reporter: Raghu Angadi
> Assignee: Raghu Angadi
> Fix For: 0.14.0
>
> Attachments: HADOOP-1453-01.patch, HADOOP-1453-02.patch, HADOOP-1453-02.patch
>
>
> {code:title=DistributedFileSystem.java:131|borderStyle=solid}
> public FSDataInputStream open(Path f, int bufferSize) throws IOException {
> if (!exists(f)) {
> throw new FileNotFoundException(f.toString());
> }
> return new DFSClient.DFSDataInputStream(dfs.open(getPath(f)), bufferSize);
> }
> {code}
> {{exists(f)}} adds extra namenode interaction that is not really required. Open is a critical DFS call.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Assigned: (HADOOP-1453) exists() not necessary before
DFS.open
Posted by "Raghu Angadi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1453?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Raghu Angadi reassigned HADOOP-1453:
------------------------------------
Assignee: Raghu Angadi
> exists() not necessary before DFS.open
> --------------------------------------
>
> Key: HADOOP-1453
> URL: https://issues.apache.org/jira/browse/HADOOP-1453
> Project: Hadoop
> Issue Type: Bug
> Components: dfs
> Reporter: Raghu Angadi
> Assignee: Raghu Angadi
> Fix For: 0.14.0
>
>
> {code:title=DistributedFileSystem.java:131|borderStyle=solid}
> public FSDataInputStream open(Path f, int bufferSize) throws IOException {
> if (!exists(f)) {
> throw new FileNotFoundException(f.toString());
> }
> return new DFSClient.DFSDataInputStream(dfs.open(getPath(f)), bufferSize);
> }
> {code}
> {{exists(f)}} adds extra namenode interaction that is not really required. Open is a critical DFS call.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-1453) exists() not necessary before
DFS.open
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12505982 ]
dhruba borthakur commented on HADOOP-1453:
------------------------------------------
Both your patches are for DistributedFileSystem whereas your comment says that there are *two* places where a redundant exists() call is made: DistributedFileSystem and ChecksumFileSystem. Can you pl clarify?
> exists() not necessary before DFS.open
> --------------------------------------
>
> Key: HADOOP-1453
> URL: https://issues.apache.org/jira/browse/HADOOP-1453
> Project: Hadoop
> Issue Type: Bug
> Components: dfs
> Reporter: Raghu Angadi
> Assignee: Raghu Angadi
> Fix For: 0.14.0
>
> Attachments: HADOOP-1453-01.patch, HADOOP-1453-02.patch
>
>
> {code:title=DistributedFileSystem.java:131|borderStyle=solid}
> public FSDataInputStream open(Path f, int bufferSize) throws IOException {
> if (!exists(f)) {
> throw new FileNotFoundException(f.toString());
> }
> return new DFSClient.DFSDataInputStream(dfs.open(getPath(f)), bufferSize);
> }
> {code}
> {{exists(f)}} adds extra namenode interaction that is not really required. Open is a critical DFS call.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-1453) exists() not necessary before
DFS.open
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12506239 ]
Hadoop QA commented on HADOOP-1453:
-----------------------------------
-1, build or testing failed
2 attempts failed to build and test the latest attachment http://issues.apache.org/jira/secure/attachment/12360067/HADOOP-1453-02.patch against trunk revision r548523.
Test results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/298/testReport/
Console output: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/298/console
Please note that this message is automatically generated and may represent a problem with the automation system and not the patch.
> exists() not necessary before DFS.open
> --------------------------------------
>
> Key: HADOOP-1453
> URL: https://issues.apache.org/jira/browse/HADOOP-1453
> Project: Hadoop
> Issue Type: Bug
> Components: dfs
> Reporter: Raghu Angadi
> Assignee: Raghu Angadi
> Fix For: 0.14.0
>
> Attachments: HADOOP-1453-01.patch, HADOOP-1453-02.patch, HADOOP-1453-02.patch
>
>
> {code:title=DistributedFileSystem.java:131|borderStyle=solid}
> public FSDataInputStream open(Path f, int bufferSize) throws IOException {
> if (!exists(f)) {
> throw new FileNotFoundException(f.toString());
> }
> return new DFSClient.DFSDataInputStream(dfs.open(getPath(f)), bufferSize);
> }
> {code}
> {{exists(f)}} adds extra namenode interaction that is not really required. Open is a critical DFS call.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-1453) exists() not necessary before
DFS.open
Posted by "Raghu Angadi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1453?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Raghu Angadi updated HADOOP-1453:
---------------------------------
Attachment: HADOOP-1453-02.patch
Thanks Dhruba. Now it is fixed. Please review 02.patch.
> exists() not necessary before DFS.open
> --------------------------------------
>
> Key: HADOOP-1453
> URL: https://issues.apache.org/jira/browse/HADOOP-1453
> Project: Hadoop
> Issue Type: Bug
> Components: dfs
> Reporter: Raghu Angadi
> Assignee: Raghu Angadi
> Fix For: 0.14.0
>
> Attachments: HADOOP-1453-01.patch, HADOOP-1453-02.patch, HADOOP-1453-02.patch
>
>
> {code:title=DistributedFileSystem.java:131|borderStyle=solid}
> public FSDataInputStream open(Path f, int bufferSize) throws IOException {
> if (!exists(f)) {
> throw new FileNotFoundException(f.toString());
> }
> return new DFSClient.DFSDataInputStream(dfs.open(getPath(f)), bufferSize);
> }
> {code}
> {{exists(f)}} adds extra namenode interaction that is not really required. Open is a critical DFS call.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-1453) exists() not necessary before
DFS.open
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12506346 ]
Hadoop QA commented on HADOOP-1453:
-----------------------------------
+1
http://issues.apache.org/jira/secure/attachment/12360067/HADOOP-1453-02.patch applied and successfully tested against trunk revision r548794.
Test results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/308/testReport/
Console output: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/308/console
> exists() not necessary before DFS.open
> --------------------------------------
>
> Key: HADOOP-1453
> URL: https://issues.apache.org/jira/browse/HADOOP-1453
> Project: Hadoop
> Issue Type: Bug
> Components: dfs
> Reporter: Raghu Angadi
> Assignee: Raghu Angadi
> Fix For: 0.14.0
>
> Attachments: HADOOP-1453-01.patch, HADOOP-1453-02.patch, HADOOP-1453-02.patch
>
>
> {code:title=DistributedFileSystem.java:131|borderStyle=solid}
> public FSDataInputStream open(Path f, int bufferSize) throws IOException {
> if (!exists(f)) {
> throw new FileNotFoundException(f.toString());
> }
> return new DFSClient.DFSDataInputStream(dfs.open(getPath(f)), bufferSize);
> }
> {code}
> {{exists(f)}} adds extra namenode interaction that is not really required. Open is a critical DFS call.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-1453) exists() not necessary before
DFS.open
Posted by "Raghu Angadi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1453?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Raghu Angadi updated HADOOP-1453:
---------------------------------
Attachment: HADOOP-1453-01.patch
Attached simple patch:
{code}
Index: src/java/org/apache/hadoop/dfs/DistributedFileSystem.java
===================================================================
--- src/java/org/apache/hadoop/dfs/DistributedFileSystem.java (revision 546652)
+++ src/java/org/apache/hadoop/dfs/DistributedFileSystem.java (working copy)
@@ -129,10 +129,6 @@
}
public FSDataInputStream open(Path f, int bufferSize) throws IOException {
- if (!exists(f)) {
- throw new FileNotFoundException(f.toString());
- }
-
return new DFSClient.DFSDataInputStream(dfs.open(getPath(f)), bufferSize);
}
{code}
> exists() not necessary before DFS.open
> --------------------------------------
>
> Key: HADOOP-1453
> URL: https://issues.apache.org/jira/browse/HADOOP-1453
> Project: Hadoop
> Issue Type: Bug
> Components: dfs
> Reporter: Raghu Angadi
> Assignee: Raghu Angadi
> Fix For: 0.14.0
>
> Attachments: HADOOP-1453-01.patch
>
>
> {code:title=DistributedFileSystem.java:131|borderStyle=solid}
> public FSDataInputStream open(Path f, int bufferSize) throws IOException {
> if (!exists(f)) {
> throw new FileNotFoundException(f.toString());
> }
> return new DFSClient.DFSDataInputStream(dfs.open(getPath(f)), bufferSize);
> }
> {code}
> {{exists(f)}} adds extra namenode interaction that is not really required. Open is a critical DFS call.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-1453) exists() not necessary before
DFS.open
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12506868 ]
Hudson commented on HADOOP-1453:
--------------------------------
Integrated in Hadoop-Nightly #131 (See [http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Nightly/131/])
> exists() not necessary before DFS.open
> --------------------------------------
>
> Key: HADOOP-1453
> URL: https://issues.apache.org/jira/browse/HADOOP-1453
> Project: Hadoop
> Issue Type: Bug
> Components: dfs
> Reporter: Raghu Angadi
> Assignee: Raghu Angadi
> Fix For: 0.14.0
>
> Attachments: HADOOP-1453-01.patch, HADOOP-1453-02.patch, HADOOP-1453-02.patch
>
>
> {code:title=DistributedFileSystem.java:131|borderStyle=solid}
> public FSDataInputStream open(Path f, int bufferSize) throws IOException {
> if (!exists(f)) {
> throw new FileNotFoundException(f.toString());
> }
> return new DFSClient.DFSDataInputStream(dfs.open(getPath(f)), bufferSize);
> }
> {code}
> {{exists(f)}} adds extra namenode interaction that is not really required. Open is a critical DFS call.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-1453) exists() not necessary before
DFS.open
Posted by "Raghu Angadi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1453?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Raghu Angadi updated HADOOP-1453:
---------------------------------
Status: Open (was: Patch Available)
> exists() not necessary before DFS.open
> --------------------------------------
>
> Key: HADOOP-1453
> URL: https://issues.apache.org/jira/browse/HADOOP-1453
> Project: Hadoop
> Issue Type: Bug
> Components: dfs
> Reporter: Raghu Angadi
> Assignee: Raghu Angadi
> Fix For: 0.14.0
>
> Attachments: HADOOP-1453-01.patch, HADOOP-1453-02.patch, HADOOP-1453-02.patch
>
>
> {code:title=DistributedFileSystem.java:131|borderStyle=solid}
> public FSDataInputStream open(Path f, int bufferSize) throws IOException {
> if (!exists(f)) {
> throw new FileNotFoundException(f.toString());
> }
> return new DFSClient.DFSDataInputStream(dfs.open(getPath(f)), bufferSize);
> }
> {code}
> {{exists(f)}} adds extra namenode interaction that is not really required. Open is a critical DFS call.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-1453) exists() not necessary before
DFS.open
Posted by "Raghu Angadi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1453?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Raghu Angadi updated HADOOP-1453:
---------------------------------
Status: Patch Available (was: Open)
> exists() not necessary before DFS.open
> --------------------------------------
>
> Key: HADOOP-1453
> URL: https://issues.apache.org/jira/browse/HADOOP-1453
> Project: Hadoop
> Issue Type: Bug
> Components: dfs
> Reporter: Raghu Angadi
> Assignee: Raghu Angadi
> Fix For: 0.14.0
>
> Attachments: HADOOP-1453-01.patch, HADOOP-1453-02.patch, HADOOP-1453-02.patch
>
>
> {code:title=DistributedFileSystem.java:131|borderStyle=solid}
> public FSDataInputStream open(Path f, int bufferSize) throws IOException {
> if (!exists(f)) {
> throw new FileNotFoundException(f.toString());
> }
> return new DFSClient.DFSDataInputStream(dfs.open(getPath(f)), bufferSize);
> }
> {code}
> {{exists(f)}} adds extra namenode interaction that is not really required. Open is a critical DFS call.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-1453) exists() not necessary before
DFS.open
Posted by "Raghu Angadi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1453?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Raghu Angadi updated HADOOP-1453:
---------------------------------
Attachment: HADOOP-1453-02.patch
Attaching an updated patch. I think it is correct and can be checked in if anyone reviews it.
> exists() not necessary before DFS.open
> --------------------------------------
>
> Key: HADOOP-1453
> URL: https://issues.apache.org/jira/browse/HADOOP-1453
> Project: Hadoop
> Issue Type: Bug
> Components: dfs
> Reporter: Raghu Angadi
> Assignee: Raghu Angadi
> Fix For: 0.14.0
>
> Attachments: HADOOP-1453-01.patch, HADOOP-1453-02.patch
>
>
> {code:title=DistributedFileSystem.java:131|borderStyle=solid}
> public FSDataInputStream open(Path f, int bufferSize) throws IOException {
> if (!exists(f)) {
> throw new FileNotFoundException(f.toString());
> }
> return new DFSClient.DFSDataInputStream(dfs.open(getPath(f)), bufferSize);
> }
> {code}
> {{exists(f)}} adds extra namenode interaction that is not really required. Open is a critical DFS call.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-1453) exists() not necessary before
DFS.open
Posted by "Raghu Angadi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1453?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Raghu Angadi updated HADOOP-1453:
---------------------------------
Status: Patch Available (was: Open)
> exists() not necessary before DFS.open
> --------------------------------------
>
> Key: HADOOP-1453
> URL: https://issues.apache.org/jira/browse/HADOOP-1453
> Project: Hadoop
> Issue Type: Bug
> Components: dfs
> Reporter: Raghu Angadi
> Assignee: Raghu Angadi
> Fix For: 0.14.0
>
> Attachments: HADOOP-1453-01.patch, HADOOP-1453-02.patch, HADOOP-1453-02.patch
>
>
> {code:title=DistributedFileSystem.java:131|borderStyle=solid}
> public FSDataInputStream open(Path f, int bufferSize) throws IOException {
> if (!exists(f)) {
> throw new FileNotFoundException(f.toString());
> }
> return new DFSClient.DFSDataInputStream(dfs.open(getPath(f)), bufferSize);
> }
> {code}
> {{exists(f)}} adds extra namenode interaction that is not really required. Open is a critical DFS call.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-1453) exists() not necessary before
DFS.open
Posted by "Raghu Angadi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12504082 ]
Raghu Angadi commented on HADOOP-1453:
--------------------------------------
This implies there are 3 extra RPCs to NameNode for typical DFS file open(). One in ChecksumFS and one each for file and .file.crc in DFS.
> exists() not necessary before DFS.open
> --------------------------------------
>
> Key: HADOOP-1453
> URL: https://issues.apache.org/jira/browse/HADOOP-1453
> Project: Hadoop
> Issue Type: Bug
> Components: dfs
> Reporter: Raghu Angadi
> Assignee: Raghu Angadi
> Fix For: 0.14.0
>
> Attachments: HADOOP-1453-01.patch
>
>
> {code:title=DistributedFileSystem.java:131|borderStyle=solid}
> public FSDataInputStream open(Path f, int bufferSize) throws IOException {
> if (!exists(f)) {
> throw new FileNotFoundException(f.toString());
> }
> return new DFSClient.DFSDataInputStream(dfs.open(getPath(f)), bufferSize);
> }
> {code}
> {{exists(f)}} adds extra namenode interaction that is not really required. Open is a critical DFS call.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-1453) exists() not necessary before
DFS.open
Posted by "Raghu Angadi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12504078 ]
Raghu Angadi commented on HADOOP-1453:
--------------------------------------
I guess this is not enough. There is an exists() in ChecksumFileSystem.open() as well. Should that be removed as well? :
{code}
public FSDataInputStream open(Path f, int bufferSize) throws IOException {
if (!exists(f)) {
throw new FileNotFoundException(f.toString());
}
return new FSDataInputStream(new FSInputChecker(this, f, bufferSize),
bufferSize);
}
{code}
> exists() not necessary before DFS.open
> --------------------------------------
>
> Key: HADOOP-1453
> URL: https://issues.apache.org/jira/browse/HADOOP-1453
> Project: Hadoop
> Issue Type: Bug
> Components: dfs
> Reporter: Raghu Angadi
> Assignee: Raghu Angadi
> Fix For: 0.14.0
>
> Attachments: HADOOP-1453-01.patch
>
>
> {code:title=DistributedFileSystem.java:131|borderStyle=solid}
> public FSDataInputStream open(Path f, int bufferSize) throws IOException {
> if (!exists(f)) {
> throw new FileNotFoundException(f.toString());
> }
> return new DFSClient.DFSDataInputStream(dfs.open(getPath(f)), bufferSize);
> }
> {code}
> {{exists(f)}} adds extra namenode interaction that is not really required. Open is a critical DFS call.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.