You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "Milind Bhandarkar (JIRA)" <ji...@apache.org> on 2006/09/09 01:20:22 UTC
[jira] Created: (HADOOP-519) HDFS File API should be extended to
include positional read
HDFS File API should be extended to include positional read
-----------------------------------------------------------
Key: HADOOP-519
URL: http://issues.apache.org/jira/browse/HADOOP-519
Project: Hadoop
Issue Type: New Feature
Components: dfs
Affects Versions: 0.6.0
Environment: All
Reporter: Milind Bhandarkar
Assigned To: Milind Bhandarkar
Fix For: 0.7.0
HDFS Input streams should support positional read. Positional read (such as the pread syscall on linux) allows reading for a specified offset without affecting the current file offset. Since the underlying file state is not touched, pread can be used efficiently in multi-threaded programs.
Here is how I plan to implement it.
Provide PositionedReadable interface, with the following methods:
int read(long position, byte[] buffer, int offset, int length);
void readFully(long position, byte[] buffer, int offset, int length);
void readFully(long position, byte[] buffer);
Abstract class FSInputStream would provide default implementation of the above methods using getPos(), seek() and read() methods. The default implementation is inefficient in multi-threaded programs since it locks the object while seeking, reading, and restoring to old state.
DFSClient.DFSInputStream, which extends FSInputStream will provide an efficient non-synchronized implementation for above calls.
In addition, FSDataInputStream, which is a wrapper around FSInputStream, will provide wrapper methods for above read methods as well.
Patch forthcoming early next week.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (HADOOP-519) HDFS File API should be extended to
include positional read
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/HADOOP-519?page=all ]
Doug Cutting updated HADOOP-519:
--------------------------------
Status: Open (was: Patch Available)
You use an array of Object to return multiple values. Wouldn't this be a lot cleaner with a small, private, static inner class?
> HDFS File API should be extended to include positional read
> -----------------------------------------------------------
>
> Key: HADOOP-519
> URL: http://issues.apache.org/jira/browse/HADOOP-519
> Project: Hadoop
> Issue Type: New Feature
> Components: dfs
> Affects Versions: 0.6.0
> Environment: All
> Reporter: Milind Bhandarkar
> Assigned To: Milind Bhandarkar
> Fix For: 0.7.0
>
> Attachments: pread.patch
>
>
> HDFS Input streams should support positional read. Positional read (such as the pread syscall on linux) allows reading for a specified offset without affecting the current file offset. Since the underlying file state is not touched, pread can be used efficiently in multi-threaded programs.
> Here is how I plan to implement it.
> Provide PositionedReadable interface, with the following methods:
> int read(long position, byte[] buffer, int offset, int length);
> void readFully(long position, byte[] buffer, int offset, int length);
> void readFully(long position, byte[] buffer);
> Abstract class FSInputStream would provide default implementation of the above methods using getPos(), seek() and read() methods. The default implementation is inefficient in multi-threaded programs since it locks the object while seeking, reading, and restoring to old state.
> DFSClient.DFSInputStream, which extends FSInputStream will provide an efficient non-synchronized implementation for above calls.
> In addition, FSDataInputStream, which is a wrapper around FSInputStream, will provide wrapper methods for above read methods as well.
> Patch forthcoming early next week.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HADOOP-519) HDFS File API should be extended to
include positional read
Posted by "Milind Bhandarkar (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/HADOOP-519?page=comments#action_12435595 ]
Milind Bhandarkar commented on HADOOP-519:
------------------------------------------
>In DFSClient, you've duplicated a lot of code from blockSeekTo in fetchBlockByteRange.
>Can you perhaps instead add one or two more methods that capture this common code?
Thats what I plan to do in another patch later, when I fix the read implementation and implement pipelining in writes.
> The javadoc for read(long,byte[], int,int) should say "read up to" or "attempt to read",
>since it may not read all of the bytes (that's what readFully is for).
Yes, I will make that modification.
>The javadoc comments on the new FSInputStream methods do not add anything useful to
>what would be inherited from the interface, and what they do add makes them inappropriate
>for inheritance by subclasses. So these should be removed.
You mean, just the commenst should be removed or the methods ? Methods cannot be implemented by making PositionedReadable an abstract class, because FSDataInputStream already extends an abstract class.
> HDFS File API should be extended to include positional read
> -----------------------------------------------------------
>
> Key: HADOOP-519
> URL: http://issues.apache.org/jira/browse/HADOOP-519
> Project: Hadoop
> Issue Type: New Feature
> Components: dfs
> Affects Versions: 0.6.0
> Environment: All
> Reporter: Milind Bhandarkar
> Assigned To: Milind Bhandarkar
> Fix For: 0.7.0
>
> Attachments: pread.patch
>
>
> HDFS Input streams should support positional read. Positional read (such as the pread syscall on linux) allows reading for a specified offset without affecting the current file offset. Since the underlying file state is not touched, pread can be used efficiently in multi-threaded programs.
> Here is how I plan to implement it.
> Provide PositionedReadable interface, with the following methods:
> int read(long position, byte[] buffer, int offset, int length);
> void readFully(long position, byte[] buffer, int offset, int length);
> void readFully(long position, byte[] buffer);
> Abstract class FSInputStream would provide default implementation of the above methods using getPos(), seek() and read() methods. The default implementation is inefficient in multi-threaded programs since it locks the object while seeking, reading, and restoring to old state.
> DFSClient.DFSInputStream, which extends FSInputStream will provide an efficient non-synchronized implementation for above calls.
> In addition, FSDataInputStream, which is a wrapper around FSInputStream, will provide wrapper methods for above read methods as well.
> Patch forthcoming early next week.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (HADOOP-519) HDFS File API should be extended to
include positional read
Posted by "Milind Bhandarkar (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/HADOOP-519?page=all ]
Milind Bhandarkar updated HADOOP-519:
-------------------------------------
Status: Patch Available (was: Open)
Patch submitted.
> HDFS File API should be extended to include positional read
> -----------------------------------------------------------
>
> Key: HADOOP-519
> URL: http://issues.apache.org/jira/browse/HADOOP-519
> Project: Hadoop
> Issue Type: New Feature
> Components: dfs
> Affects Versions: 0.6.0
> Environment: All
> Reporter: Milind Bhandarkar
> Assigned To: Milind Bhandarkar
> Fix For: 0.7.0
>
> Attachments: pread.patch
>
>
> HDFS Input streams should support positional read. Positional read (such as the pread syscall on linux) allows reading for a specified offset without affecting the current file offset. Since the underlying file state is not touched, pread can be used efficiently in multi-threaded programs.
> Here is how I plan to implement it.
> Provide PositionedReadable interface, with the following methods:
> int read(long position, byte[] buffer, int offset, int length);
> void readFully(long position, byte[] buffer, int offset, int length);
> void readFully(long position, byte[] buffer);
> Abstract class FSInputStream would provide default implementation of the above methods using getPos(), seek() and read() methods. The default implementation is inefficient in multi-threaded programs since it locks the object while seeking, reading, and restoring to old state.
> DFSClient.DFSInputStream, which extends FSInputStream will provide an efficient non-synchronized implementation for above calls.
> In addition, FSDataInputStream, which is a wrapper around FSInputStream, will provide wrapper methods for above read methods as well.
> Patch forthcoming early next week.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (HADOOP-519) HDFS File API should be extended to
include positional read
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/HADOOP-519?page=all ]
Doug Cutting updated HADOOP-519:
--------------------------------
Status: Open (was: Patch Available)
Overall this looks like a great addition that is well-implemented. A few nits:
In DFSClient, you've duplicated a lot of code from blockSeekTo in fetchBlockByteRange. Can you perhaps instead add one or two more methods that capture this common code?
The javadoc for read(long,byte[], int,int) should say "read up to" or "attempt to read", since it may not read all of the bytes (that's what readFully is for).
The javadoc comments on the new FSInputStream methods do not add anything useful to what would be inherited from the interface, and what they do add makes them inappropriate for inheritance by subclasses. So these should be removed.
> HDFS File API should be extended to include positional read
> -----------------------------------------------------------
>
> Key: HADOOP-519
> URL: http://issues.apache.org/jira/browse/HADOOP-519
> Project: Hadoop
> Issue Type: New Feature
> Components: dfs
> Affects Versions: 0.6.0
> Environment: All
> Reporter: Milind Bhandarkar
> Assigned To: Milind Bhandarkar
> Fix For: 0.7.0
>
> Attachments: pread.patch
>
>
> HDFS Input streams should support positional read. Positional read (such as the pread syscall on linux) allows reading for a specified offset without affecting the current file offset. Since the underlying file state is not touched, pread can be used efficiently in multi-threaded programs.
> Here is how I plan to implement it.
> Provide PositionedReadable interface, with the following methods:
> int read(long position, byte[] buffer, int offset, int length);
> void readFully(long position, byte[] buffer, int offset, int length);
> void readFully(long position, byte[] buffer);
> Abstract class FSInputStream would provide default implementation of the above methods using getPos(), seek() and read() methods. The default implementation is inefficient in multi-threaded programs since it locks the object while seeking, reading, and restoring to old state.
> DFSClient.DFSInputStream, which extends FSInputStream will provide an efficient non-synchronized implementation for above calls.
> In addition, FSDataInputStream, which is a wrapper around FSInputStream, will provide wrapper methods for above read methods as well.
> Patch forthcoming early next week.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (HADOOP-519) HDFS File API should be extended to
include positional read
Posted by "Milind Bhandarkar (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/HADOOP-519?page=all ]
Milind Bhandarkar updated HADOOP-519:
-------------------------------------
Attachment: pread.patch
This patch implements positional read for FSInputStream, FSDataInputStream, DFSInputStream, and LocalFSFileInputStream.
I will later re-implement "normal" read in terms of pread in a separate patch.
> HDFS File API should be extended to include positional read
> -----------------------------------------------------------
>
> Key: HADOOP-519
> URL: http://issues.apache.org/jira/browse/HADOOP-519
> Project: Hadoop
> Issue Type: New Feature
> Components: dfs
> Affects Versions: 0.6.0
> Environment: All
> Reporter: Milind Bhandarkar
> Assigned To: Milind Bhandarkar
> Fix For: 0.7.0
>
> Attachments: pread.patch
>
>
> HDFS Input streams should support positional read. Positional read (such as the pread syscall on linux) allows reading for a specified offset without affecting the current file offset. Since the underlying file state is not touched, pread can be used efficiently in multi-threaded programs.
> Here is how I plan to implement it.
> Provide PositionedReadable interface, with the following methods:
> int read(long position, byte[] buffer, int offset, int length);
> void readFully(long position, byte[] buffer, int offset, int length);
> void readFully(long position, byte[] buffer);
> Abstract class FSInputStream would provide default implementation of the above methods using getPos(), seek() and read() methods. The default implementation is inefficient in multi-threaded programs since it locks the object while seeking, reading, and restoring to old state.
> DFSClient.DFSInputStream, which extends FSInputStream will provide an efficient non-synchronized implementation for above calls.
> In addition, FSDataInputStream, which is a wrapper around FSInputStream, will provide wrapper methods for above read methods as well.
> Patch forthcoming early next week.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HADOOP-519) HDFS File API should be extended to
include positional read
Posted by "Milind Bhandarkar (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/HADOOP-519?page=comments#action_12439269 ]
Milind Bhandarkar commented on HADOOP-519:
------------------------------------------
Indeed. I will resubmit the patch.
> HDFS File API should be extended to include positional read
> -----------------------------------------------------------
>
> Key: HADOOP-519
> URL: http://issues.apache.org/jira/browse/HADOOP-519
> Project: Hadoop
> Issue Type: New Feature
> Components: dfs
> Affects Versions: 0.6.0
> Environment: All
> Reporter: Milind Bhandarkar
> Assigned To: Milind Bhandarkar
> Fix For: 0.7.0
>
> Attachments: pread.patch
>
>
> HDFS Input streams should support positional read. Positional read (such as the pread syscall on linux) allows reading for a specified offset without affecting the current file offset. Since the underlying file state is not touched, pread can be used efficiently in multi-threaded programs.
> Here is how I plan to implement it.
> Provide PositionedReadable interface, with the following methods:
> int read(long position, byte[] buffer, int offset, int length);
> void readFully(long position, byte[] buffer, int offset, int length);
> void readFully(long position, byte[] buffer);
> Abstract class FSInputStream would provide default implementation of the above methods using getPos(), seek() and read() methods. The default implementation is inefficient in multi-threaded programs since it locks the object while seeking, reading, and restoring to old state.
> DFSClient.DFSInputStream, which extends FSInputStream will provide an efficient non-synchronized implementation for above calls.
> In addition, FSDataInputStream, which is a wrapper around FSInputStream, will provide wrapper methods for above read methods as well.
> Patch forthcoming early next week.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (HADOOP-519) HDFS File API should be extended to
include positional read
Posted by "Milind Bhandarkar (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/HADOOP-519?page=all ]
Milind Bhandarkar updated HADOOP-519:
-------------------------------------
Attachment: (was: pread.patch)
> HDFS File API should be extended to include positional read
> -----------------------------------------------------------
>
> Key: HADOOP-519
> URL: http://issues.apache.org/jira/browse/HADOOP-519
> Project: Hadoop
> Issue Type: New Feature
> Components: dfs
> Affects Versions: 0.6.0
> Environment: All
> Reporter: Milind Bhandarkar
> Assigned To: Milind Bhandarkar
> Fix For: 0.7.0
>
>
> HDFS Input streams should support positional read. Positional read (such as the pread syscall on linux) allows reading for a specified offset without affecting the current file offset. Since the underlying file state is not touched, pread can be used efficiently in multi-threaded programs.
> Here is how I plan to implement it.
> Provide PositionedReadable interface, with the following methods:
> int read(long position, byte[] buffer, int offset, int length);
> void readFully(long position, byte[] buffer, int offset, int length);
> void readFully(long position, byte[] buffer);
> Abstract class FSInputStream would provide default implementation of the above methods using getPos(), seek() and read() methods. The default implementation is inefficient in multi-threaded programs since it locks the object while seeking, reading, and restoring to old state.
> DFSClient.DFSInputStream, which extends FSInputStream will provide an efficient non-synchronized implementation for above calls.
> In addition, FSDataInputStream, which is a wrapper around FSInputStream, will provide wrapper methods for above read methods as well.
> Patch forthcoming early next week.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HADOOP-519) HDFS File API should be extended to
include positional read
Posted by "Milind Bhandarkar (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/HADOOP-519?page=comments#action_12435611 ]
Milind Bhandarkar commented on HADOOP-519:
------------------------------------------
> Why not do this now?
Because the normal read method has some other deficiencies as well. For example, even for reading just a few bytes from the block, the datanode tries to send out the full block. Also, even when the seeks target is within the same block, the existing connection to the datanode is clossed anyway and a new one opened. Since these changes are related to clean-up of duplicate code, I had planned to do them as a different patch.
> HDFS File API should be extended to include positional read
> -----------------------------------------------------------
>
> Key: HADOOP-519
> URL: http://issues.apache.org/jira/browse/HADOOP-519
> Project: Hadoop
> Issue Type: New Feature
> Components: dfs
> Affects Versions: 0.6.0
> Environment: All
> Reporter: Milind Bhandarkar
> Assigned To: Milind Bhandarkar
> Fix For: 0.7.0
>
> Attachments: pread.patch
>
>
> HDFS Input streams should support positional read. Positional read (such as the pread syscall on linux) allows reading for a specified offset without affecting the current file offset. Since the underlying file state is not touched, pread can be used efficiently in multi-threaded programs.
> Here is how I plan to implement it.
> Provide PositionedReadable interface, with the following methods:
> int read(long position, byte[] buffer, int offset, int length);
> void readFully(long position, byte[] buffer, int offset, int length);
> void readFully(long position, byte[] buffer);
> Abstract class FSInputStream would provide default implementation of the above methods using getPos(), seek() and read() methods. The default implementation is inefficient in multi-threaded programs since it locks the object while seeking, reading, and restoring to old state.
> DFSClient.DFSInputStream, which extends FSInputStream will provide an efficient non-synchronized implementation for above calls.
> In addition, FSDataInputStream, which is a wrapper around FSInputStream, will provide wrapper methods for above read methods as well.
> Patch forthcoming early next week.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (HADOOP-519) HDFS File API should be extended to
include positional read
Posted by "Milind Bhandarkar (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/HADOOP-519?page=all ]
Milind Bhandarkar updated HADOOP-519:
-------------------------------------
Attachment: (was: pread.patch)
> HDFS File API should be extended to include positional read
> -----------------------------------------------------------
>
> Key: HADOOP-519
> URL: http://issues.apache.org/jira/browse/HADOOP-519
> Project: Hadoop
> Issue Type: New Feature
> Components: dfs
> Affects Versions: 0.6.0
> Environment: All
> Reporter: Milind Bhandarkar
> Assigned To: Milind Bhandarkar
> Fix For: 0.7.0
>
>
> HDFS Input streams should support positional read. Positional read (such as the pread syscall on linux) allows reading for a specified offset without affecting the current file offset. Since the underlying file state is not touched, pread can be used efficiently in multi-threaded programs.
> Here is how I plan to implement it.
> Provide PositionedReadable interface, with the following methods:
> int read(long position, byte[] buffer, int offset, int length);
> void readFully(long position, byte[] buffer, int offset, int length);
> void readFully(long position, byte[] buffer);
> Abstract class FSInputStream would provide default implementation of the above methods using getPos(), seek() and read() methods. The default implementation is inefficient in multi-threaded programs since it locks the object while seeking, reading, and restoring to old state.
> DFSClient.DFSInputStream, which extends FSInputStream will provide an efficient non-synchronized implementation for above calls.
> In addition, FSDataInputStream, which is a wrapper around FSInputStream, will provide wrapper methods for above read methods as well.
> Patch forthcoming early next week.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (HADOOP-519) HDFS File API should be extended to
include positional read
Posted by "Milind Bhandarkar (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/HADOOP-519?page=all ]
Milind Bhandarkar updated HADOOP-519:
-------------------------------------
Attachment: pread.patch
This is the new patch that fixes the problems in earlier patch mentioned by Doug.
> HDFS File API should be extended to include positional read
> -----------------------------------------------------------
>
> Key: HADOOP-519
> URL: http://issues.apache.org/jira/browse/HADOOP-519
> Project: Hadoop
> Issue Type: New Feature
> Components: dfs
> Affects Versions: 0.6.0
> Environment: All
> Reporter: Milind Bhandarkar
> Assigned To: Milind Bhandarkar
> Fix For: 0.7.0
>
> Attachments: pread.patch
>
>
> HDFS Input streams should support positional read. Positional read (such as the pread syscall on linux) allows reading for a specified offset without affecting the current file offset. Since the underlying file state is not touched, pread can be used efficiently in multi-threaded programs.
> Here is how I plan to implement it.
> Provide PositionedReadable interface, with the following methods:
> int read(long position, byte[] buffer, int offset, int length);
> void readFully(long position, byte[] buffer, int offset, int length);
> void readFully(long position, byte[] buffer);
> Abstract class FSInputStream would provide default implementation of the above methods using getPos(), seek() and read() methods. The default implementation is inefficient in multi-threaded programs since it locks the object while seeking, reading, and restoring to old state.
> DFSClient.DFSInputStream, which extends FSInputStream will provide an efficient non-synchronized implementation for above calls.
> In addition, FSDataInputStream, which is a wrapper around FSInputStream, will provide wrapper methods for above read methods as well.
> Patch forthcoming early next week.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HADOOP-519) HDFS File API should be extended to
include positional read
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/HADOOP-519?page=comments#action_12435614 ]
Doug Cutting commented on HADOOP-519:
-------------------------------------
> Because the normal read method has some other deficiencies as well.
I still don't see the point in having two identical copies of these deficiencies.
Is there a follow-on issue that you have in mind that will redesign the block reading protocol?
> HDFS File API should be extended to include positional read
> -----------------------------------------------------------
>
> Key: HADOOP-519
> URL: http://issues.apache.org/jira/browse/HADOOP-519
> Project: Hadoop
> Issue Type: New Feature
> Components: dfs
> Affects Versions: 0.6.0
> Environment: All
> Reporter: Milind Bhandarkar
> Assigned To: Milind Bhandarkar
> Fix For: 0.7.0
>
> Attachments: pread.patch
>
>
> HDFS Input streams should support positional read. Positional read (such as the pread syscall on linux) allows reading for a specified offset without affecting the current file offset. Since the underlying file state is not touched, pread can be used efficiently in multi-threaded programs.
> Here is how I plan to implement it.
> Provide PositionedReadable interface, with the following methods:
> int read(long position, byte[] buffer, int offset, int length);
> void readFully(long position, byte[] buffer, int offset, int length);
> void readFully(long position, byte[] buffer);
> Abstract class FSInputStream would provide default implementation of the above methods using getPos(), seek() and read() methods. The default implementation is inefficient in multi-threaded programs since it locks the object while seeking, reading, and restoring to old state.
> DFSClient.DFSInputStream, which extends FSInputStream will provide an efficient non-synchronized implementation for above calls.
> In addition, FSDataInputStream, which is a wrapper around FSInputStream, will provide wrapper methods for above read methods as well.
> Patch forthcoming early next week.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (HADOOP-519) HDFS File API should be extended to
include positional read
Posted by "Milind Bhandarkar (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/HADOOP-519?page=all ]
Milind Bhandarkar updated HADOOP-519:
-------------------------------------
Attachment: pread.patch
Object[] returned pair is replaced by a private static class.
> HDFS File API should be extended to include positional read
> -----------------------------------------------------------
>
> Key: HADOOP-519
> URL: http://issues.apache.org/jira/browse/HADOOP-519
> Project: Hadoop
> Issue Type: New Feature
> Components: dfs
> Affects Versions: 0.6.0
> Environment: All
> Reporter: Milind Bhandarkar
> Assigned To: Milind Bhandarkar
> Fix For: 0.7.0
>
> Attachments: pread.patch
>
>
> HDFS Input streams should support positional read. Positional read (such as the pread syscall on linux) allows reading for a specified offset without affecting the current file offset. Since the underlying file state is not touched, pread can be used efficiently in multi-threaded programs.
> Here is how I plan to implement it.
> Provide PositionedReadable interface, with the following methods:
> int read(long position, byte[] buffer, int offset, int length);
> void readFully(long position, byte[] buffer, int offset, int length);
> void readFully(long position, byte[] buffer);
> Abstract class FSInputStream would provide default implementation of the above methods using getPos(), seek() and read() methods. The default implementation is inefficient in multi-threaded programs since it locks the object while seeking, reading, and restoring to old state.
> DFSClient.DFSInputStream, which extends FSInputStream will provide an efficient non-synchronized implementation for above calls.
> In addition, FSDataInputStream, which is a wrapper around FSInputStream, will provide wrapper methods for above read methods as well.
> Patch forthcoming early next week.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (HADOOP-519) HDFS File API should be extended to
include positional read
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/HADOOP-519?page=all ]
Doug Cutting updated HADOOP-519:
--------------------------------
Status: Resolved (was: Patch Available)
Resolution: Fixed
I committed this. Thanks, Milind!
> HDFS File API should be extended to include positional read
> -----------------------------------------------------------
>
> Key: HADOOP-519
> URL: http://issues.apache.org/jira/browse/HADOOP-519
> Project: Hadoop
> Issue Type: New Feature
> Components: dfs
> Affects Versions: 0.6.0
> Environment: All
> Reporter: Milind Bhandarkar
> Assigned To: Milind Bhandarkar
> Fix For: 0.7.0
>
> Attachments: pread.patch
>
>
> HDFS Input streams should support positional read. Positional read (such as the pread syscall on linux) allows reading for a specified offset without affecting the current file offset. Since the underlying file state is not touched, pread can be used efficiently in multi-threaded programs.
> Here is how I plan to implement it.
> Provide PositionedReadable interface, with the following methods:
> int read(long position, byte[] buffer, int offset, int length);
> void readFully(long position, byte[] buffer, int offset, int length);
> void readFully(long position, byte[] buffer);
> Abstract class FSInputStream would provide default implementation of the above methods using getPos(), seek() and read() methods. The default implementation is inefficient in multi-threaded programs since it locks the object while seeking, reading, and restoring to old state.
> DFSClient.DFSInputStream, which extends FSInputStream will provide an efficient non-synchronized implementation for above calls.
> In addition, FSDataInputStream, which is a wrapper around FSInputStream, will provide wrapper methods for above read methods as well.
> Patch forthcoming early next week.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (HADOOP-519) HDFS File API should be extended to
include positional read
Posted by "Milind Bhandarkar (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/HADOOP-519?page=all ]
Milind Bhandarkar updated HADOOP-519:
-------------------------------------
Status: Patch Available (was: Open)
attached.
> HDFS File API should be extended to include positional read
> -----------------------------------------------------------
>
> Key: HADOOP-519
> URL: http://issues.apache.org/jira/browse/HADOOP-519
> Project: Hadoop
> Issue Type: New Feature
> Components: dfs
> Affects Versions: 0.6.0
> Environment: All
> Reporter: Milind Bhandarkar
> Assigned To: Milind Bhandarkar
> Fix For: 0.7.0
>
> Attachments: pread.patch
>
>
> HDFS Input streams should support positional read. Positional read (such as the pread syscall on linux) allows reading for a specified offset without affecting the current file offset. Since the underlying file state is not touched, pread can be used efficiently in multi-threaded programs.
> Here is how I plan to implement it.
> Provide PositionedReadable interface, with the following methods:
> int read(long position, byte[] buffer, int offset, int length);
> void readFully(long position, byte[] buffer, int offset, int length);
> void readFully(long position, byte[] buffer);
> Abstract class FSInputStream would provide default implementation of the above methods using getPos(), seek() and read() methods. The default implementation is inefficient in multi-threaded programs since it locks the object while seeking, reading, and restoring to old state.
> DFSClient.DFSInputStream, which extends FSInputStream will provide an efficient non-synchronized implementation for above calls.
> In addition, FSDataInputStream, which is a wrapper around FSInputStream, will provide wrapper methods for above read methods as well.
> Patch forthcoming early next week.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HADOOP-519) HDFS File API should be extended to
include positional read
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/HADOOP-519?page=comments#action_12435606 ]
Doug Cutting commented on HADOOP-519:
-------------------------------------
> Thats what I plan to do in another patch later [ ... ]
Why not do this now?
> You mean, just the commenst should be removed or the methods ?
Just the javadoc comments.
> HDFS File API should be extended to include positional read
> -----------------------------------------------------------
>
> Key: HADOOP-519
> URL: http://issues.apache.org/jira/browse/HADOOP-519
> Project: Hadoop
> Issue Type: New Feature
> Components: dfs
> Affects Versions: 0.6.0
> Environment: All
> Reporter: Milind Bhandarkar
> Assigned To: Milind Bhandarkar
> Fix For: 0.7.0
>
> Attachments: pread.patch
>
>
> HDFS Input streams should support positional read. Positional read (such as the pread syscall on linux) allows reading for a specified offset without affecting the current file offset. Since the underlying file state is not touched, pread can be used efficiently in multi-threaded programs.
> Here is how I plan to implement it.
> Provide PositionedReadable interface, with the following methods:
> int read(long position, byte[] buffer, int offset, int length);
> void readFully(long position, byte[] buffer, int offset, int length);
> void readFully(long position, byte[] buffer);
> Abstract class FSInputStream would provide default implementation of the above methods using getPos(), seek() and read() methods. The default implementation is inefficient in multi-threaded programs since it locks the object while seeking, reading, and restoring to old state.
> DFSClient.DFSInputStream, which extends FSInputStream will provide an efficient non-synchronized implementation for above calls.
> In addition, FSDataInputStream, which is a wrapper around FSInputStream, will provide wrapper methods for above read methods as well.
> Patch forthcoming early next week.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HADOOP-519) HDFS File API should be extended to
include positional read
Posted by "Milind Bhandarkar (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/HADOOP-519?page=comments#action_12433929 ]
Milind Bhandarkar commented on HADOOP-519:
------------------------------------------
>>So, to be clear, you will modify both FSInputStream and FSDataInputStream to implement the new PositionReadable interface, right?
Yes.
>>Only the primitive FSInputStream.read(long,byte[],int,int) method needs to be synchronized. The others can be unsynchronized and implemented >>only in base classes, inherited by optimized subclasses.
That's right.
>>An optimized implementation of read(long,byte[],int,int) can be provided in both DFSInputStream and LocalFSInputStream (the latter using nio's >>FileChannel.read(ByteBuffer,long)). It might be simpler if the PositionReadble API were instead read(ByteBuffer, long), so that the client can >>manage ByteBuffer allocation.
I was going to make LocalFSInputStream to use the default synchronized implementation. I will study the solution you have suggested and will try to see if there are any tangible benefits.
>>Finally, we should change implementations of read(byte[],int,int) and seek(long) to be synchronized. This won't hurt, since they're not currently >>thread safe, and it will make the positioned-read methods thread-safe.
Yes.
> HDFS File API should be extended to include positional read
> -----------------------------------------------------------
>
> Key: HADOOP-519
> URL: http://issues.apache.org/jira/browse/HADOOP-519
> Project: Hadoop
> Issue Type: New Feature
> Components: dfs
> Affects Versions: 0.6.0
> Environment: All
> Reporter: Milind Bhandarkar
> Assigned To: Milind Bhandarkar
> Fix For: 0.7.0
>
>
> HDFS Input streams should support positional read. Positional read (such as the pread syscall on linux) allows reading for a specified offset without affecting the current file offset. Since the underlying file state is not touched, pread can be used efficiently in multi-threaded programs.
> Here is how I plan to implement it.
> Provide PositionedReadable interface, with the following methods:
> int read(long position, byte[] buffer, int offset, int length);
> void readFully(long position, byte[] buffer, int offset, int length);
> void readFully(long position, byte[] buffer);
> Abstract class FSInputStream would provide default implementation of the above methods using getPos(), seek() and read() methods. The default implementation is inefficient in multi-threaded programs since it locks the object while seeking, reading, and restoring to old state.
> DFSClient.DFSInputStream, which extends FSInputStream will provide an efficient non-synchronized implementation for above calls.
> In addition, FSDataInputStream, which is a wrapper around FSInputStream, will provide wrapper methods for above read methods as well.
> Patch forthcoming early next week.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HADOOP-519) HDFS File API should be extended to
include positional read
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/HADOOP-519?page=comments#action_12433910 ]
Doug Cutting commented on HADOOP-519:
-------------------------------------
So, to be clear, you will modify both FSInputStream and FSDataInputStream to implement the new PositionReadable interface, right? (FSDataInputStream should also implement Seekable too: it already has the methods, but does not yet reference the interface.)
Only the primitive FSInputStream.read(long,byte[],int,int) method needs to be synchronized. The others can be unsynchronized and implemented only in base classes, inherited by optimized subclasses.
An optimized implementation of read(long,byte[],int,int) can be provided in both DFSInputStream and LocalFSInputStream (the latter using nio's FileChannel.read(ByteBuffer,long)). It might be simpler if the PositionReadble API were instead read(ByteBuffer, long), so that the client can manage ByteBuffer allocation.
Finally, we should change implementations of read(byte[],int,int) and seek(long) to be synchronized. This won't hurt, since they're not currently thread safe, and it will make the positioned-read methods thread-safe.
> HDFS File API should be extended to include positional read
> -----------------------------------------------------------
>
> Key: HADOOP-519
> URL: http://issues.apache.org/jira/browse/HADOOP-519
> Project: Hadoop
> Issue Type: New Feature
> Components: dfs
> Affects Versions: 0.6.0
> Environment: All
> Reporter: Milind Bhandarkar
> Assigned To: Milind Bhandarkar
> Fix For: 0.7.0
>
>
> HDFS Input streams should support positional read. Positional read (such as the pread syscall on linux) allows reading for a specified offset without affecting the current file offset. Since the underlying file state is not touched, pread can be used efficiently in multi-threaded programs.
> Here is how I plan to implement it.
> Provide PositionedReadable interface, with the following methods:
> int read(long position, byte[] buffer, int offset, int length);
> void readFully(long position, byte[] buffer, int offset, int length);
> void readFully(long position, byte[] buffer);
> Abstract class FSInputStream would provide default implementation of the above methods using getPos(), seek() and read() methods. The default implementation is inefficient in multi-threaded programs since it locks the object while seeking, reading, and restoring to old state.
> DFSClient.DFSInputStream, which extends FSInputStream will provide an efficient non-synchronized implementation for above calls.
> In addition, FSDataInputStream, which is a wrapper around FSInputStream, will provide wrapper methods for above read methods as well.
> Patch forthcoming early next week.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (HADOOP-519) HDFS File API should be extended to
include positional read
Posted by "Milind Bhandarkar (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/HADOOP-519?page=all ]
Milind Bhandarkar updated HADOOP-519:
-------------------------------------
Status: Patch Available (was: Open)
Patch submitted.
> HDFS File API should be extended to include positional read
> -----------------------------------------------------------
>
> Key: HADOOP-519
> URL: http://issues.apache.org/jira/browse/HADOOP-519
> Project: Hadoop
> Issue Type: New Feature
> Components: dfs
> Affects Versions: 0.6.0
> Environment: All
> Reporter: Milind Bhandarkar
> Assigned To: Milind Bhandarkar
> Fix For: 0.7.0
>
> Attachments: pread.patch
>
>
> HDFS Input streams should support positional read. Positional read (such as the pread syscall on linux) allows reading for a specified offset without affecting the current file offset. Since the underlying file state is not touched, pread can be used efficiently in multi-threaded programs.
> Here is how I plan to implement it.
> Provide PositionedReadable interface, with the following methods:
> int read(long position, byte[] buffer, int offset, int length);
> void readFully(long position, byte[] buffer, int offset, int length);
> void readFully(long position, byte[] buffer);
> Abstract class FSInputStream would provide default implementation of the above methods using getPos(), seek() and read() methods. The default implementation is inefficient in multi-threaded programs since it locks the object while seeking, reading, and restoring to old state.
> DFSClient.DFSInputStream, which extends FSInputStream will provide an efficient non-synchronized implementation for above calls.
> In addition, FSDataInputStream, which is a wrapper around FSInputStream, will provide wrapper methods for above read methods as well.
> Patch forthcoming early next week.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira