You are viewing a plain text version of this content. The canonical link for it is here.
Posted to general@hadoop.apache.org by "Xiaoyi Lu@cse.osu.edu" <lu...@cse.ohio-state.edu> on 2013/11/09 14:44:47 UTC
RDMA (InfiniBand and RoCE) Support for Hadoop 1.2.1
The Hadoop-RDMA team is pleased to announce the release of Hadoop-RDMA 0.9.8
(technology preview) package with the following features and bug-fixes:
* Hadoop-RDMA 0.9.8 Features
- Based on Apache Hadoop 1.2.1
- High performance design with native InfiniBand and RoCE support
at the verbs-level for HDFS, MapReduce, and RPC components
- Compliant with Apache Hadoop 1.2.1 APIs and applications
- Easily configurable for native InfiniBand, RoCE and the traditional
sockets-based support (Ethernet and InfiniBand with IPoIB)
- HDFS over native InfiniBand and RoCE
- RDMA-based write
- RDMA-based replication
- Tuned packet size
- MapReduce over native InfiniBand and RoCE
- RDMA-based shuffle
- Pre-fetching and caching of map outputs
- In-memory merge
- Overlapping of merge and reduce
- RPC over native InfiniBand and RoCE
- JVM-bypassed buffer management
- On-demand InfiniBand connection setup
- RDMA or send/recv based adaptive communication
- Tested with
- Mellanox InfiniBand adapters (DDR, QDR, and FDR)
- RoCE
- Various multi-core platforms
- Different file systems with disks and SSDs
* Bug Fixes (since 0.9.1)
- MapReduce over InfiniBand
- Reducing the waiting time for the pre-fetch cache read
- Enabling Reduce tasks to randomize the shuffle pattern
- HDFS over InfiniBand
- Fix an issue in processing multiple blocks concurrently in a
single task
For downloading Hadoop-RDMA 0.9.8 package and the associated user guide,
please visit the following URL:
http://hadoop-rdma.cse.ohio-state.edu
Sample performance numbers for benchmarks using Hadoop-RDMA 0.9.8
version can be viewed by visiting the `Performance' tab of the above website.
All questions, feedbacks and bug reports are welcome. Please post it
to the hadoop-rdma-discuss mailing list (hadoop-rdma-discuss at
cse.ohio-state.edu).
Thanks,
The Hadoop-RDMA Team
Re: RDMA (InfiniBand and RoCE) Support for Hadoop 1.2.1
Posted by "Xiaoyi Lu@cse.osu.edu" <lu...@cse.ohio-state.edu>.
Hey, Milind,
Thanks for your interests.
- We are working on moving the design to Hadoop 2.x. It will be
available in the near future.
- This is an independent project.
Xiaoyi
On Mon, Nov 11, 2013 at 2:43 PM, Milind Bhandarkar
<mb...@gopivotal.com> wrote:
> Hi Xiaoyi,
>
> This is great !
>
> For clusters that have moved to Hadoop 2.x versions, do you have any work
> in progress to rebase Hadoop-RDMA to 2.x ?
>
> Also, I am wondering if the development that you are doing will be
> available in Mellanox UDA ? Or is it an independent project ?
>
> - milind
>
>
> ---
> Milind Bhandarkar
> Chief Scientist
> Pivotal
> +1-650-523-3858 (W)
> +1-408-666-8483 (C)
>
>
> On Sat, Nov 9, 2013 at 5:44 AM, Xiaoyi Lu@cse.osu.edu <
> luxi@cse.ohio-state.edu> wrote:
>
>> The Hadoop-RDMA team is pleased to announce the release of Hadoop-RDMA
>> 0.9.8
>> (technology preview) package with the following features and bug-fixes:
>>
>> * Hadoop-RDMA 0.9.8 Features
>>
>> - Based on Apache Hadoop 1.2.1
>> - High performance design with native InfiniBand and RoCE support
>> at the verbs-level for HDFS, MapReduce, and RPC components
>> - Compliant with Apache Hadoop 1.2.1 APIs and applications
>> - Easily configurable for native InfiniBand, RoCE and the traditional
>> sockets-based support (Ethernet and InfiniBand with IPoIB)
>> - HDFS over native InfiniBand and RoCE
>> - RDMA-based write
>> - RDMA-based replication
>> - Tuned packet size
>> - MapReduce over native InfiniBand and RoCE
>> - RDMA-based shuffle
>> - Pre-fetching and caching of map outputs
>> - In-memory merge
>> - Overlapping of merge and reduce
>> - RPC over native InfiniBand and RoCE
>> - JVM-bypassed buffer management
>> - On-demand InfiniBand connection setup
>> - RDMA or send/recv based adaptive communication
>> - Tested with
>> - Mellanox InfiniBand adapters (DDR, QDR, and FDR)
>> - RoCE
>> - Various multi-core platforms
>> - Different file systems with disks and SSDs
>>
>> * Bug Fixes (since 0.9.1)
>>
>> - MapReduce over InfiniBand
>> - Reducing the waiting time for the pre-fetch cache read
>> - Enabling Reduce tasks to randomize the shuffle pattern
>> - HDFS over InfiniBand
>> - Fix an issue in processing multiple blocks concurrently in a
>> single task
>>
>> For downloading Hadoop-RDMA 0.9.8 package and the associated user guide,
>> please visit the following URL:
>>
>> http://hadoop-rdma.cse.ohio-state.edu
>>
>> Sample performance numbers for benchmarks using Hadoop-RDMA 0.9.8
>> version can be viewed by visiting the `Performance' tab of the above
>> website.
>>
>> All questions, feedbacks and bug reports are welcome. Please post it
>> to the hadoop-rdma-discuss mailing list (hadoop-rdma-discuss at
>> cse.ohio-state.edu).
>>
>> Thanks,
>>
>> The Hadoop-RDMA Team
>>
Re: RDMA (InfiniBand and RoCE) Support for Hadoop 1.2.1
Posted by Milind Bhandarkar <mb...@gopivotal.com>.
Hi Xiaoyi,
This is great !
For clusters that have moved to Hadoop 2.x versions, do you have any work
in progress to rebase Hadoop-RDMA to 2.x ?
Also, I am wondering if the development that you are doing will be
available in Mellanox UDA ? Or is it an independent project ?
- milind
---
Milind Bhandarkar
Chief Scientist
Pivotal
+1-650-523-3858 (W)
+1-408-666-8483 (C)
On Sat, Nov 9, 2013 at 5:44 AM, Xiaoyi Lu@cse.osu.edu <
luxi@cse.ohio-state.edu> wrote:
> The Hadoop-RDMA team is pleased to announce the release of Hadoop-RDMA
> 0.9.8
> (technology preview) package with the following features and bug-fixes:
>
> * Hadoop-RDMA 0.9.8 Features
>
> - Based on Apache Hadoop 1.2.1
> - High performance design with native InfiniBand and RoCE support
> at the verbs-level for HDFS, MapReduce, and RPC components
> - Compliant with Apache Hadoop 1.2.1 APIs and applications
> - Easily configurable for native InfiniBand, RoCE and the traditional
> sockets-based support (Ethernet and InfiniBand with IPoIB)
> - HDFS over native InfiniBand and RoCE
> - RDMA-based write
> - RDMA-based replication
> - Tuned packet size
> - MapReduce over native InfiniBand and RoCE
> - RDMA-based shuffle
> - Pre-fetching and caching of map outputs
> - In-memory merge
> - Overlapping of merge and reduce
> - RPC over native InfiniBand and RoCE
> - JVM-bypassed buffer management
> - On-demand InfiniBand connection setup
> - RDMA or send/recv based adaptive communication
> - Tested with
> - Mellanox InfiniBand adapters (DDR, QDR, and FDR)
> - RoCE
> - Various multi-core platforms
> - Different file systems with disks and SSDs
>
> * Bug Fixes (since 0.9.1)
>
> - MapReduce over InfiniBand
> - Reducing the waiting time for the pre-fetch cache read
> - Enabling Reduce tasks to randomize the shuffle pattern
> - HDFS over InfiniBand
> - Fix an issue in processing multiple blocks concurrently in a
> single task
>
> For downloading Hadoop-RDMA 0.9.8 package and the associated user guide,
> please visit the following URL:
>
> http://hadoop-rdma.cse.ohio-state.edu
>
> Sample performance numbers for benchmarks using Hadoop-RDMA 0.9.8
> version can be viewed by visiting the `Performance' tab of the above
> website.
>
> All questions, feedbacks and bug reports are welcome. Please post it
> to the hadoop-rdma-discuss mailing list (hadoop-rdma-discuss at
> cse.ohio-state.edu).
>
> Thanks,
>
> The Hadoop-RDMA Team
>