You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-dev@hadoop.apache.org by Yasin Celik <ya...@gmail.com> on 2013/10/09 01:49:28 UTC
Fwd: SDP support for Hadoop RPC
Hello,
I looked around but I think it seems there is no Sockets Direct
Protocol(SDP) support in Hadoop RPC.
I basically want to take advantage of Infiniband when transferring data.
Is there any known support for this? Has anybody tried this before?
Best
---------
Yasin Celik
Re: SDP support for Hadoop RPC
Posted by Milind Bhandarkar <mb...@gopivotal.com>.
Have you checked out Mellanox's UDA ?
There have been a lot of efforts from RoCE providers to get Hadoop
optimized implementations with modern networking hardware. Even RedHat has
announced HPN (High Performance Networking) interfaces to allow modern
networks. It might take a few years to get there for Open Source Hadoop
implementations, but believe me, every Hadoop vendor based off of open
source Hadoop is incorporating new network architectures in their
proprietary distro. We were there first. That's all.
- milind
---
Milind Bhandarkar
Chief Scientist
Pivotal
+1-650-523-3858 (W)
+1-408-666-8483 (C)
On Tue, Oct 8, 2013 at 6:13 PM, Yasin Celik <ya...@gmail.com> wrote:
> Hello
>
> How do you get hadoop worked with infiniband? Do you use SDP?
> I only need to use RPC of hadoop for now. Any help on how to get it worked?
>
> Thanks
> Yasin
>
>
> On Tue, Oct 8, 2013 at 8:57 PM, Milind Bhandarkar <
> mbhandarkar@gopivotal.com
> > wrote:
>
> > Yes, we have. It works very well, but it is considered too niche by folks
> > who insist on buying the least capable hardware for their test clusters,
> > and therefore, recommend such underpowered clusters to customers as well.
> >
> > - milind
> >
> > > On Oct 8, 2013, at 16:49, Yasin Celik <ya...@gmail.com> wrote:
> > >
> > > Hello,
> > >
> > > I looked around but I think it seems there is no Sockets Direct
> > > Protocol(SDP) support in Hadoop RPC.
> > > I basically want to take advantage of Infiniband when transferring
> data.
> > > Is there any known support for this? Has anybody tried this before?
> > >
> > > Best
> > >
> > > ---------
> > > Yasin Celik
> >
>
>
>
> --
> ---------
> Yasin Celik
>
Re: SDP support for Hadoop RPC
Posted by Yasin Celik <ya...@gmail.com>.
Hello
How do you get hadoop worked with infiniband? Do you use SDP?
I only need to use RPC of hadoop for now. Any help on how to get it worked?
Thanks
Yasin
On Tue, Oct 8, 2013 at 8:57 PM, Milind Bhandarkar <mbhandarkar@gopivotal.com
> wrote:
> Yes, we have. It works very well, but it is considered too niche by folks
> who insist on buying the least capable hardware for their test clusters,
> and therefore, recommend such underpowered clusters to customers as well.
>
> - milind
>
> > On Oct 8, 2013, at 16:49, Yasin Celik <ya...@gmail.com> wrote:
> >
> > Hello,
> >
> > I looked around but I think it seems there is no Sockets Direct
> > Protocol(SDP) support in Hadoop RPC.
> > I basically want to take advantage of Infiniband when transferring data.
> > Is there any known support for this? Has anybody tried this before?
> >
> > Best
> >
> > ---------
> > Yasin Celik
>
--
---------
Yasin Celik
Re: SDP support for Hadoop RPC
Posted by Milind Bhandarkar <mb...@gopivotal.com>.
I cannot believe i made the same mistake twice, of reply-to to mailing list. I need to get off the outlook client.
once again, sorry folks. this was supposed to go to steve's personal email.
- milind
Sent from my iPhone
> On Oct 9, 2013, at 8:47, Suresh Srinivas <su...@hortonworks.com> wrote:
>
> Milind, please stop this. The topic here is not what Steve's employers
> wants to sell or recommend. Please stick to the technical issue. This is
> the second time this week where a thread unnecessarily goes beyond
> technical issues. If you have an axe to grind, please keep it off this
> forum. It is getting annoying.
>
> I am also annoyed by the frequent references to private communications made
> in this public forum.
>
>> Let's meet in the same chinese restaurant, and I will give you the
> numbers on performance and cost
> Instead of conspiracy theories and employer bashing, adding details such as
> this is what is useful to this forum!
>
>
> On Wed, Oct 9, 2013 at 2:09 AM, Milind Bhandarkar <mbhandarkar@gopivotal.com
>> wrote:
>
>> No, Steve, I meant exactly what I wrote. One day, when you are here, Let's
>> meet in the same chinese restaurant, and I will give you the numbers on
>> performance and cost, and let us do the division of these two numbers.
>>
>> Let's talk about how the nn latency becomes a bottleneck for rest of the
>> cluster's throughput, and why the networking world's advances cannot be
>> pushed under the rug.
>>
>> Let's talk about why your employers are cozying up with DSSD and engenio
>> while you and others in open source are insisting on 1GbE and DAS SATA
>> disks being the most suitable for Hadoop.
>>
>> And most of all, lets chat about why business aspects of Hadoop are acting
>> against the open source from the same orgs' folks.
>>
>> Tomorrow and day after we are conducting big data benchmarking workshop in
>> San Jose, where your partners and other open-core hadoop company's partners
>> will demonstrate how advanced hardware (cpus, networks, storage) is more
>> cost effective that what you are recommending.
>>
>> I had recognized this phenomenon very early, and wrote a blog post
>> comparing open source hadoop development to charlie chaplin, who missed the
>> color and talky movie technology, by sticking to mute black and white
>> technology. I know your employer has moved beyond cheap hardware, based on
>> what I hear from customers where we compete. I an wondering why you still
>> keep on insisting new technologies are not worth it.
>>
>> - milind
>>
>> Sent from my iPhone
>>
>>> On Oct 9, 2013, at 1:45, Steve Loughran <st...@hortonworks.com> wrote:
>>>
>>> On 9 October 2013 01:57, Milind Bhandarkar <mbhandarkar@gopivotal.com
>>> wrote:
>>>
>>>> Yes, we have. It works very well, but it is considered too niche by
>> folks
>>>> who insist on buying the least capable hardware for their test clusters,
>>>> and therefore, recommend such underpowered clusters to customers as
>> well.
>>>
>>>
>>> surely you meant to say "take advantage of the cost model of JBOD
>> storage
>>> and ethernet to allow data to be stored and accessed at significantly
>> lower
>>> price points than for legacy storage architectures and pricing models -so
>>> enabling their customers to store and process data they would have
>>> previously had to discard" (0)
>>>
>>> IB should be most interesting at the app level -for apps > classic MR.
>>> That's giraph, streaming work, Tez. I'd like to see some numbers there.
>> As
>>> the oracle
>>>
>>> For storage, IB would make locality less of an issue (1,2), and instead
>>> make the level of storage: SSD vs HDD more significant in terms of
>>> performance (2). There is ongoing work there in a set of JIRAs about
>>> multi-tier storage.
>>>
>>> I don't know the current state of Hadoop on IB, or even if
>> allocateDirect()
>>> of NIO has been picked up. For IPC there should be some latency
>>> improvements, while for the Datanodes its the bulk data you want to push
>>> around faster. If you want to work on either of those problems you'd be
>>> very welcome.
>>>
>>>
>>>
>>> (0) I also have a VMWare test cluster for some HA work and VM capacity
>> from
>>> Rackspace for a broader pool of deployment options.
>>> (1) Hadoop 2.1 supports Unix Domain Sockets for a direct-yet-secure
>>> connection from a local app (HBase, ...) and the Datanode. This bypasses
>>> the network stack entirely
>>> (2)
>> http://nowlab.cse.ohio-state.edu/publications/conf-papers/2010/sur-masvdc10.pdf
>>> (3) http://www.cs.berkeley.edu/%7Eganesha/disk-irrelevant_hotos2011.pdf
>>>
>>> --
>>> CONFIDENTIALITY NOTICE
>>> NOTICE: This message is intended for the use of the individual or entity
>> to
>>> which it is addressed and may contain information that is confidential,
>>> privileged and exempt from disclosure under applicable law. If the reader
>>> of this message is not the intended recipient, you are hereby notified
>> that
>>> any printing, copying, dissemination, distribution, disclosure or
>>> forwarding of this communication is strictly prohibited. If you have
>>> received this communication in error, please contact the sender
>> immediately
>>> and delete it from your system. Thank You.
>
>
>
> --
> http://hortonworks.com/download/
>
> --
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity to
> which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
Re: SDP support for Hadoop RPC
Posted by Suresh Srinivas <su...@hortonworks.com>.
Milind, please stop this. The topic here is not what Steve's employers
wants to sell or recommend. Please stick to the technical issue. This is
the second time this week where a thread unnecessarily goes beyond
technical issues. If you have an axe to grind, please keep it off this
forum. It is getting annoying.
I am also annoyed by the frequent references to private communications made
in this public forum.
> Let's meet in the same chinese restaurant, and I will give you the
numbers on performance and cost
Instead of conspiracy theories and employer bashing, adding details such as
this is what is useful to this forum!
On Wed, Oct 9, 2013 at 2:09 AM, Milind Bhandarkar <mbhandarkar@gopivotal.com
> wrote:
> No, Steve, I meant exactly what I wrote. One day, when you are here, Let's
> meet in the same chinese restaurant, and I will give you the numbers on
> performance and cost, and let us do the division of these two numbers.
>
> Let's talk about how the nn latency becomes a bottleneck for rest of the
> cluster's throughput, and why the networking world's advances cannot be
> pushed under the rug.
>
> Let's talk about why your employers are cozying up with DSSD and engenio
> while you and others in open source are insisting on 1GbE and DAS SATA
> disks being the most suitable for Hadoop.
>
> And most of all, lets chat about why business aspects of Hadoop are acting
> against the open source from the same orgs' folks.
>
> Tomorrow and day after we are conducting big data benchmarking workshop in
> San Jose, where your partners and other open-core hadoop company's partners
> will demonstrate how advanced hardware (cpus, networks, storage) is more
> cost effective that what you are recommending.
>
> I had recognized this phenomenon very early, and wrote a blog post
> comparing open source hadoop development to charlie chaplin, who missed the
> color and talky movie technology, by sticking to mute black and white
> technology. I know your employer has moved beyond cheap hardware, based on
> what I hear from customers where we compete. I an wondering why you still
> keep on insisting new technologies are not worth it.
>
> - milind
>
> Sent from my iPhone
>
> > On Oct 9, 2013, at 1:45, Steve Loughran <st...@hortonworks.com> wrote:
> >
> > On 9 October 2013 01:57, Milind Bhandarkar <mbhandarkar@gopivotal.com
> >wrote:
> >
> >> Yes, we have. It works very well, but it is considered too niche by
> folks
> >> who insist on buying the least capable hardware for their test clusters,
> >> and therefore, recommend such underpowered clusters to customers as
> well.
> >
> >
> > surely you meant to say "take advantage of the cost model of JBOD
> storage
> > and ethernet to allow data to be stored and accessed at significantly
> lower
> > price points than for legacy storage architectures and pricing models -so
> > enabling their customers to store and process data they would have
> > previously had to discard" (0)
> >
> > IB should be most interesting at the app level -for apps > classic MR.
> > That's giraph, streaming work, Tez. I'd like to see some numbers there.
> As
> > the oracle
> >
> > For storage, IB would make locality less of an issue (1,2), and instead
> > make the level of storage: SSD vs HDD more significant in terms of
> > performance (2). There is ongoing work there in a set of JIRAs about
> > multi-tier storage.
> >
> > I don't know the current state of Hadoop on IB, or even if
> allocateDirect()
> > of NIO has been picked up. For IPC there should be some latency
> > improvements, while for the Datanodes its the bulk data you want to push
> > around faster. If you want to work on either of those problems you'd be
> > very welcome.
> >
> >
> >
> > (0) I also have a VMWare test cluster for some HA work and VM capacity
> from
> > Rackspace for a broader pool of deployment options.
> > (1) Hadoop 2.1 supports Unix Domain Sockets for a direct-yet-secure
> > connection from a local app (HBase, ...) and the Datanode. This bypasses
> > the network stack entirely
> > (2)
> >
> http://nowlab.cse.ohio-state.edu/publications/conf-papers/2010/sur-masvdc10.pdf
> > (3) http://www.cs.berkeley.edu/%7Eganesha/disk-irrelevant_hotos2011.pdf
> >
> > --
> > CONFIDENTIALITY NOTICE
> > NOTICE: This message is intended for the use of the individual or entity
> to
> > which it is addressed and may contain information that is confidential,
> > privileged and exempt from disclosure under applicable law. If the reader
> > of this message is not the intended recipient, you are hereby notified
> that
> > any printing, copying, dissemination, distribution, disclosure or
> > forwarding of this communication is strictly prohibited. If you have
> > received this communication in error, please contact the sender
> immediately
> > and delete it from your system. Thank You.
>
--
http://hortonworks.com/download/
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: SDP support for Hadoop RPC
Posted by Milind Bhandarkar <mb...@gopivotal.com>.
No, Steve, I meant exactly what I wrote. One day, when you are here, Let's meet in the same chinese restaurant, and I will give you the numbers on performance and cost, and let us do the division of these two numbers.
Let's talk about how the nn latency becomes a bottleneck for rest of the cluster's throughput, and why the networking world's advances cannot be pushed under the rug.
Let's talk about why your employers are cozying up with DSSD and engenio while you and others in open source are insisting on 1GbE and DAS SATA disks being the most suitable for Hadoop.
And most of all, lets chat about why business aspects of Hadoop are acting against the open source from the same orgs' folks.
Tomorrow and day after we are conducting big data benchmarking workshop in San Jose, where your partners and other open-core hadoop company's partners will demonstrate how advanced hardware (cpus, networks, storage) is more cost effective that what you are recommending.
I had recognized this phenomenon very early, and wrote a blog post comparing open source hadoop development to charlie chaplin, who missed the color and talky movie technology, by sticking to mute black and white technology. I know your employer has moved beyond cheap hardware, based on what I hear from customers where we compete. I an wondering why you still keep on insisting new technologies are not worth it.
- milind
Sent from my iPhone
> On Oct 9, 2013, at 1:45, Steve Loughran <st...@hortonworks.com> wrote:
>
> On 9 October 2013 01:57, Milind Bhandarkar <mb...@gopivotal.com>wrote:
>
>> Yes, we have. It works very well, but it is considered too niche by folks
>> who insist on buying the least capable hardware for their test clusters,
>> and therefore, recommend such underpowered clusters to customers as well.
>
>
> surely you meant to say "take advantage of the cost model of JBOD storage
> and ethernet to allow data to be stored and accessed at significantly lower
> price points than for legacy storage architectures and pricing models -so
> enabling their customers to store and process data they would have
> previously had to discard" (0)
>
> IB should be most interesting at the app level -for apps > classic MR.
> That's giraph, streaming work, Tez. I'd like to see some numbers there. As
> the oracle
>
> For storage, IB would make locality less of an issue (1,2), and instead
> make the level of storage: SSD vs HDD more significant in terms of
> performance (2). There is ongoing work there in a set of JIRAs about
> multi-tier storage.
>
> I don't know the current state of Hadoop on IB, or even if allocateDirect()
> of NIO has been picked up. For IPC there should be some latency
> improvements, while for the Datanodes its the bulk data you want to push
> around faster. If you want to work on either of those problems you'd be
> very welcome.
>
>
>
> (0) I also have a VMWare test cluster for some HA work and VM capacity from
> Rackspace for a broader pool of deployment options.
> (1) Hadoop 2.1 supports Unix Domain Sockets for a direct-yet-secure
> connection from a local app (HBase, ...) and the Datanode. This bypasses
> the network stack entirely
> (2)
> http://nowlab.cse.ohio-state.edu/publications/conf-papers/2010/sur-masvdc10.pdf
> (3) http://www.cs.berkeley.edu/%7Eganesha/disk-irrelevant_hotos2011.pdf
>
> --
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity to
> which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
Re: SDP support for Hadoop RPC
Posted by Steve Loughran <st...@hortonworks.com>.
oops, didn't finish a sentence:
On 9 October 2013 09:45, Steve Loughran <st...@hortonworks.com> wrote:
>
>
>
> IB should be most interesting at the app level -for apps > classic MR.
> That's giraph, streaming work, Tez. I'd like to see some numbers there. As
> the oracle
>
As the oracle exadata appliance has IB, there may be some numbers -but I
don't know if/how Hadoop takes advantage IB there.
I have seen numbers related to 10GbE and they weren't that compelling for
very large clusters, where you have multiple racks and enough nodes that
server failures are handled easily. 2x1GbE or 4x1GbE bonded still has a
price advantage there -though you've just massively complicated your wiring
and stand-up costs.
For smaller clusters where the servers are going towards
many-disks-per-box, 10GbE is compelling from a recovery perspective,
especially as the cost of the backbone switches will be less.
Steve
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: SDP support for Hadoop RPC
Posted by Steve Loughran <st...@hortonworks.com>.
On 9 October 2013 01:57, Milind Bhandarkar <mb...@gopivotal.com>wrote:
> Yes, we have. It works very well, but it is considered too niche by folks
> who insist on buying the least capable hardware for their test clusters,
> and therefore, recommend such underpowered clusters to customers as well.
>
surely you meant to say "take advantage of the cost model of JBOD storage
and ethernet to allow data to be stored and accessed at significantly lower
price points than for legacy storage architectures and pricing models -so
enabling their customers to store and process data they would have
previously had to discard" (0)
IB should be most interesting at the app level -for apps > classic MR.
That's giraph, streaming work, Tez. I'd like to see some numbers there. As
the oracle
For storage, IB would make locality less of an issue (1,2), and instead
make the level of storage: SSD vs HDD more significant in terms of
performance (2). There is ongoing work there in a set of JIRAs about
multi-tier storage.
I don't know the current state of Hadoop on IB, or even if allocateDirect()
of NIO has been picked up. For IPC there should be some latency
improvements, while for the Datanodes its the bulk data you want to push
around faster. If you want to work on either of those problems you'd be
very welcome.
(0) I also have a VMWare test cluster for some HA work and VM capacity from
Rackspace for a broader pool of deployment options.
(1) Hadoop 2.1 supports Unix Domain Sockets for a direct-yet-secure
connection from a local app (HBase, ...) and the Datanode. This bypasses
the network stack entirely
(2)
http://nowlab.cse.ohio-state.edu/publications/conf-papers/2010/sur-masvdc10.pdf
(3) http://www.cs.berkeley.edu/%7Eganesha/disk-irrelevant_hotos2011.pdf
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: SDP support for Hadoop RPC
Posted by Milind Bhandarkar <mb...@gopivotal.com>.
Yes, we have. It works very well, but it is considered too niche by folks who insist on buying the least capable hardware for their test clusters, and therefore, recommend such underpowered clusters to customers as well.
- milind
> On Oct 8, 2013, at 16:49, Yasin Celik <ya...@gmail.com> wrote:
>
> Hello,
>
> I looked around but I think it seems there is no Sockets Direct
> Protocol(SDP) support in Hadoop RPC.
> I basically want to take advantage of Infiniband when transferring data.
> Is there any known support for this? Has anybody tried this before?
>
> Best
>
> ---------
> Yasin Celik
Re: SDP support for Hadoop RPC
Posted by Steve Loughran <st...@hortonworks.com>.
On 10 October 2013 09:00, Luke Lu <ll...@apache.org> wrote:
> Hadoop works with Java 7, which has builtin SDP support. You'll need to
> pass SDP related properties (-Dcom.sun.sdp.conf=your_sdp.conf) to hadoop
> JVMs via env var HADOOP_OPTS.
>
> One caveat of SDP is the 64KB message size (performance cliff beyond 64KB),
> which _should_ work with default hadoop config, where dfs packet size is
> 64KB.
>
> __Luke
>
Fancy starting a hadoop wiki page on this? Maybe a base Hadoop Networking
with sections underneath. Numbers would be nice, but "things you need to do
at both ends to get it working" would come first.
What happens if if you enable sdp on the server but forget to do it on the
client?
-steve
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: SDP support for Hadoop RPC
Posted by Luke Lu <ll...@vicaya.com>.
0.20.2 is too old. The current stable release of hadoop is 1.2.1.
Anyway, for 0.20.2, you probably need -Djava.net.preferIPv4Stack=true as
well, which is already set for Hadoop 1+. BTW, you'll need to set all these
in mapred.child.java.opts as well.
On Thu, Oct 10, 2013 at 5:12 AM, Yasin Celik <ya...@gmail.com> wrote:
> Hi Luke,
>
> I actually tested hadoop RPC using SDP. But it did not work for me.
> I used hadoop-core 0.20.2
> Following is the error I received:
>
> java.io.IOException: Address family not supported by protocol
> at sun.net.sdp.SdpSupport.
> convert0(Native Method)
> at sun.net.sdp.SdpSupport.convertSocket(SdpSupport.java:70)
> at sun.net.sdp.SdpProvider.convertTcpToSdpIfMatch(SdpProvider.java:300)
> at sun.net.sdp.SdpProvider.implBeforeTcpBind(SdpProvider.java:323)
> at sun.net.NetHooks.beforeTcpBind(NetHooks.java:86)
> at
> sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:198)
> at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
> at org.apache.hadoop.ipc.Server.bind(Server.java:188)
> at org.apache.hadoop.ipc.Server$Listener.<init>(Server.java:253)
> at org.apache.hadoop.ipc.Server.<init>(Server.java:1026)
> at org.apache.hadoop.ipc.RPC$Server.<init>(RPC.java:488)
> at org.apache.hadoop.ipc.RPC.getServer(RPC.java:450)
>
> My sdp.conf file is like:
> bind * *
> connect 10.xx.xx.0/24 <http://10.80.70.0/24> 50030-50050
>
> I also tried this for one specific node:
>
> bind 10.80.70.13 50031
> connect 10.xx.xx.0/24 <http://10.80.70.0/24> 50030-50050
>
>
> Thank you very much.
>
>
> On Thu, Oct 10, 2013 at 4:00 AM, Luke Lu <ll...@apache.org> wrote:
>
> > Hadoop works with Java 7, which has builtin SDP support. You'll need to
> > pass SDP related properties (-Dcom.sun.sdp.conf=your_sdp.conf) to hadoop
> > JVMs via env var HADOOP_OPTS.
> >
> > One caveat of SDP is the 64KB message size (performance cliff beyond
> 64KB),
> > which _should_ work with default hadoop config, where dfs packet size is
> > 64KB.
> >
> > __Luke
> >
> >
> > On Tue, Oct 8, 2013 at 4:49 PM, Yasin Celik <ya...@gmail.com>
> > wrote:
> >
> > > Hello,
> > >
> > > I looked around but I think it seems there is no Sockets Direct
> > > Protocol(SDP) support in Hadoop RPC.
> > > I basically want to take advantage of Infiniband when transferring
> data.
> > > Is there any known support for this? Has anybody tried this before?
> > >
> > > Best
> > >
> > > ---------
> > > Yasin Celik
> > >
> >
>
>
>
> --
> ---------
> Yasin Celik
>
Re: SDP support for Hadoop RPC
Posted by Yasin Celik <ya...@gmail.com>.
Hi Luke,
I actually tested hadoop RPC using SDP. But it did not work for me.
I used hadoop-core 0.20.2
Following is the error I received:
java.io.IOException: Address family not supported by protocol
at sun.net.sdp.SdpSupport.
convert0(Native Method)
at sun.net.sdp.SdpSupport.convertSocket(SdpSupport.java:70)
at sun.net.sdp.SdpProvider.convertTcpToSdpIfMatch(SdpProvider.java:300)
at sun.net.sdp.SdpProvider.implBeforeTcpBind(SdpProvider.java:323)
at sun.net.NetHooks.beforeTcpBind(NetHooks.java:86)
at
sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:198)
at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
at org.apache.hadoop.ipc.Server.bind(Server.java:188)
at org.apache.hadoop.ipc.Server$Listener.<init>(Server.java:253)
at org.apache.hadoop.ipc.Server.<init>(Server.java:1026)
at org.apache.hadoop.ipc.RPC$Server.<init>(RPC.java:488)
at org.apache.hadoop.ipc.RPC.getServer(RPC.java:450)
My sdp.conf file is like:
bind * *
connect 10.xx.xx.0/24 <http://10.80.70.0/24> 50030-50050
I also tried this for one specific node:
bind 10.80.70.13 50031
connect 10.xx.xx.0/24 <http://10.80.70.0/24> 50030-50050
Thank you very much.
On Thu, Oct 10, 2013 at 4:00 AM, Luke Lu <ll...@apache.org> wrote:
> Hadoop works with Java 7, which has builtin SDP support. You'll need to
> pass SDP related properties (-Dcom.sun.sdp.conf=your_sdp.conf) to hadoop
> JVMs via env var HADOOP_OPTS.
>
> One caveat of SDP is the 64KB message size (performance cliff beyond 64KB),
> which _should_ work with default hadoop config, where dfs packet size is
> 64KB.
>
> __Luke
>
>
> On Tue, Oct 8, 2013 at 4:49 PM, Yasin Celik <ya...@gmail.com>
> wrote:
>
> > Hello,
> >
> > I looked around but I think it seems there is no Sockets Direct
> > Protocol(SDP) support in Hadoop RPC.
> > I basically want to take advantage of Infiniband when transferring data.
> > Is there any known support for this? Has anybody tried this before?
> >
> > Best
> >
> > ---------
> > Yasin Celik
> >
>
--
---------
Yasin Celik
Re: SDP support for Hadoop RPC
Posted by Luke Lu <ll...@apache.org>.
Hadoop works with Java 7, which has builtin SDP support. You'll need to
pass SDP related properties (-Dcom.sun.sdp.conf=your_sdp.conf) to hadoop
JVMs via env var HADOOP_OPTS.
One caveat of SDP is the 64KB message size (performance cliff beyond 64KB),
which _should_ work with default hadoop config, where dfs packet size is
64KB.
__Luke
On Tue, Oct 8, 2013 at 4:49 PM, Yasin Celik <ya...@gmail.com> wrote:
> Hello,
>
> I looked around but I think it seems there is no Sockets Direct
> Protocol(SDP) support in Hadoop RPC.
> I basically want to take advantage of Infiniband when transferring data.
> Is there any known support for this? Has anybody tried this before?
>
> Best
>
> ---------
> Yasin Celik
>