You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-dev@hadoop.apache.org by Todd Lipcon <to...@cloudera.com> on 2012/06/07 23:51:26 UTC
Merging some improvements to branch-2
Hi all,
I plan to merge the following JIRAs to branch-2 this week unless there
are any objections:
"Direct read" optimization:
6e51b33 HADOOP-8135. Add ByteBufferReadable interface to
FSDataInputStream. Contributed by Henry Robinson.
4418682 HADOOP-8244. Improve comments on ByteBufferReadable.read.
Contributed by Henry Robinson.
c66f982 HDFS-2834. Add a ByteBuffer-based read API to DFSInputStream.
Contributed by Henry Robinson.
467acd1 HDFS-3110. Use directRead API to reduce the number of buffer
copies in libhdfs. Contributed by Henry Robinson.
These improvements give a substantial savings in CPU to applications
using libhdfs, and have some potential for usage in apps like HBase
and MR as well.
MiniDFSClusterManager:
bfa5c0a HDFS-3167. CLI-based driver for MiniDFSCluster. Contributed by
Henry Robinson.
21dfa6a HDFS-3235. MiniDFSClusterManager doesn't correctly support
-format option. Contributed by Henry Robinson.
These test-only improvements make it easier to construct system tests
against realistic pseudo-distributed clusters.
All of the above patches have been baking in trunk for quite some time.
Thanks
-Todd
--
Todd Lipcon
Software Engineer, Cloudera
Re: Merging some improvements to branch-2
Posted by Todd Lipcon <to...@cloudera.com>.
Thanks all, I merged these as well as HDFS-3243 and HDFS-3514, some
small test fixes I forgot to add to my original list. Hopefully I got
all the CHANGES.txt, etc, right :)
-Todd
On Fri, Jun 8, 2012 at 11:53 AM, Todd Lipcon <to...@cloudera.com> wrote:
> Thanks. I'll merge these this this afternoon.
>
> -Todd
>
> On Thu, Jun 7, 2012 at 4:25 PM, Eli Collins <el...@cloudera.com> wrote:
>> +1
>>
>> Will be great to have the direct read optimization (thanks Henry) in 2.x.
>>
>> Thanks,
>> Eli
>>
>> On Thu, Jun 7, 2012 at 2:51 PM, Todd Lipcon <to...@cloudera.com> wrote:
>>> Hi all,
>>>
>>> I plan to merge the following JIRAs to branch-2 this week unless there
>>> are any objections:
>>>
>>> "Direct read" optimization:
>>>
>>> 6e51b33 HADOOP-8135. Add ByteBufferReadable interface to
>>> FSDataInputStream. Contributed by Henry Robinson.
>>> 4418682 HADOOP-8244. Improve comments on ByteBufferReadable.read.
>>> Contributed by Henry Robinson.
>>> c66f982 HDFS-2834. Add a ByteBuffer-based read API to DFSInputStream.
>>> Contributed by Henry Robinson.
>>> 467acd1 HDFS-3110. Use directRead API to reduce the number of buffer
>>> copies in libhdfs. Contributed by Henry Robinson.
>>>
>>> These improvements give a substantial savings in CPU to applications
>>> using libhdfs, and have some potential for usage in apps like HBase
>>> and MR as well.
>>>
>>> MiniDFSClusterManager:
>>> bfa5c0a HDFS-3167. CLI-based driver for MiniDFSCluster. Contributed by
>>> Henry Robinson.
>>> 21dfa6a HDFS-3235. MiniDFSClusterManager doesn't correctly support
>>> -format option. Contributed by Henry Robinson.
>>>
>>> These test-only improvements make it easier to construct system tests
>>> against realistic pseudo-distributed clusters.
>>>
>>> All of the above patches have been baking in trunk for quite some time.
>>>
>>> Thanks
>>> -Todd
>>> --
>>> Todd Lipcon
>>> Software Engineer, Cloudera
>
>
>
> --
> Todd Lipcon
> Software Engineer, Cloudera
--
Todd Lipcon
Software Engineer, Cloudera
Re: Merging some improvements to branch-2
Posted by Todd Lipcon <to...@cloudera.com>.
Thanks. I'll merge these this this afternoon.
-Todd
On Thu, Jun 7, 2012 at 4:25 PM, Eli Collins <el...@cloudera.com> wrote:
> +1
>
> Will be great to have the direct read optimization (thanks Henry) in 2.x.
>
> Thanks,
> Eli
>
> On Thu, Jun 7, 2012 at 2:51 PM, Todd Lipcon <to...@cloudera.com> wrote:
>> Hi all,
>>
>> I plan to merge the following JIRAs to branch-2 this week unless there
>> are any objections:
>>
>> "Direct read" optimization:
>>
>> 6e51b33 HADOOP-8135. Add ByteBufferReadable interface to
>> FSDataInputStream. Contributed by Henry Robinson.
>> 4418682 HADOOP-8244. Improve comments on ByteBufferReadable.read.
>> Contributed by Henry Robinson.
>> c66f982 HDFS-2834. Add a ByteBuffer-based read API to DFSInputStream.
>> Contributed by Henry Robinson.
>> 467acd1 HDFS-3110. Use directRead API to reduce the number of buffer
>> copies in libhdfs. Contributed by Henry Robinson.
>>
>> These improvements give a substantial savings in CPU to applications
>> using libhdfs, and have some potential for usage in apps like HBase
>> and MR as well.
>>
>> MiniDFSClusterManager:
>> bfa5c0a HDFS-3167. CLI-based driver for MiniDFSCluster. Contributed by
>> Henry Robinson.
>> 21dfa6a HDFS-3235. MiniDFSClusterManager doesn't correctly support
>> -format option. Contributed by Henry Robinson.
>>
>> These test-only improvements make it easier to construct system tests
>> against realistic pseudo-distributed clusters.
>>
>> All of the above patches have been baking in trunk for quite some time.
>>
>> Thanks
>> -Todd
>> --
>> Todd Lipcon
>> Software Engineer, Cloudera
--
Todd Lipcon
Software Engineer, Cloudera
Re: Merging some improvements to branch-2
Posted by Eli Collins <el...@cloudera.com>.
+1
Will be great to have the direct read optimization (thanks Henry) in 2.x.
Thanks,
Eli
On Thu, Jun 7, 2012 at 2:51 PM, Todd Lipcon <to...@cloudera.com> wrote:
> Hi all,
>
> I plan to merge the following JIRAs to branch-2 this week unless there
> are any objections:
>
> "Direct read" optimization:
>
> 6e51b33 HADOOP-8135. Add ByteBufferReadable interface to
> FSDataInputStream. Contributed by Henry Robinson.
> 4418682 HADOOP-8244. Improve comments on ByteBufferReadable.read.
> Contributed by Henry Robinson.
> c66f982 HDFS-2834. Add a ByteBuffer-based read API to DFSInputStream.
> Contributed by Henry Robinson.
> 467acd1 HDFS-3110. Use directRead API to reduce the number of buffer
> copies in libhdfs. Contributed by Henry Robinson.
>
> These improvements give a substantial savings in CPU to applications
> using libhdfs, and have some potential for usage in apps like HBase
> and MR as well.
>
> MiniDFSClusterManager:
> bfa5c0a HDFS-3167. CLI-based driver for MiniDFSCluster. Contributed by
> Henry Robinson.
> 21dfa6a HDFS-3235. MiniDFSClusterManager doesn't correctly support
> -format option. Contributed by Henry Robinson.
>
> These test-only improvements make it easier to construct system tests
> against realistic pseudo-distributed clusters.
>
> All of the above patches have been baking in trunk for quite some time.
>
> Thanks
> -Todd
> --
> Todd Lipcon
> Software Engineer, Cloudera
Re: Merging some improvements to branch-2
Posted by "Aaron T. Myers" <at...@cloudera.com>.
+1
I've reviewed several of these patches and think they should be fine to go
into branch-2.
--
Aaron T. Myers
Software Engineer, Cloudera
On Thu, Jun 7, 2012 at 2:51 PM, Todd Lipcon <to...@cloudera.com> wrote:
> Hi all,
>
> I plan to merge the following JIRAs to branch-2 this week unless there
> are any objections:
>
> "Direct read" optimization:
>
> 6e51b33 HADOOP-8135. Add ByteBufferReadable interface to
> FSDataInputStream. Contributed by Henry Robinson.
> 4418682 HADOOP-8244. Improve comments on ByteBufferReadable.read.
> Contributed by Henry Robinson.
> c66f982 HDFS-2834. Add a ByteBuffer-based read API to DFSInputStream.
> Contributed by Henry Robinson.
> 467acd1 HDFS-3110. Use directRead API to reduce the number of buffer
> copies in libhdfs. Contributed by Henry Robinson.
>
> These improvements give a substantial savings in CPU to applications
> using libhdfs, and have some potential for usage in apps like HBase
> and MR as well.
>
> MiniDFSClusterManager:
> bfa5c0a HDFS-3167. CLI-based driver for MiniDFSCluster. Contributed by
> Henry Robinson.
> 21dfa6a HDFS-3235. MiniDFSClusterManager doesn't correctly support
> -format option. Contributed by Henry Robinson.
>
> These test-only improvements make it easier to construct system tests
> against realistic pseudo-distributed clusters.
>
> All of the above patches have been baking in trunk for quite some time.
>
> Thanks
> -Todd
> --
> Todd Lipcon
> Software Engineer, Cloudera
>