You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-user@hadoop.apache.org by Aji Janis <aj...@gmail.com> on 2013/09/09 19:19:12 UTC
Hadoop on IPv6
We are moving to a IPv6 network. I know Hadoop installation says to disable
IPv6. Does anyone have any suggestions for work arround? Currently, using
Cloudera version 0.20.2-cdh3u5. Thanks in advance.
Re: Hadoop on IPv6
Posted by Aji Janis <aj...@gmail.com>.
I really appreciate the suggestions. Thank you.
On Mon, Sep 9, 2013 at 4:48 PM, Adam Muise <am...@hortonworks.com> wrote:
> I suggest you go to the mighty Google. This is not a Hadoop configuration
> question, it's a web server / proxy configuration.
>
> Starters:
>
> http://www.progob.nl/robmaaseu/index.php/an-easy-way-to-get-ipv6-ready-on-the-outside-ipv6-to-ipv4-proxy/
>
> and proxying with nginx
> http://nginx.org/en/docs/http/ngx_http_proxy_module.html
>
> Eventually Knox will help out by adding a REST/Web set of gateways for
> Hadoop services:
> http://knox.incubator.apache.org
> For now, your web based gateway services on the cluster can be proxied and
> you can still land on your gateway box with ipv6 capabilities on the
> outside of the secure cluster. That gateway box would have direct ipv4
> connections to the cluster services.
>
> Cheers,
>
>
> On Mon, Sep 9, 2013 at 4:35 PM, Aji Janis <aj...@gmail.com> wrote:
>
>> Adam,
>> Thank you for the suggestion. Could you point me to any documentation
>> that can help with the configuration changes I need to make? Thanks.
>>
>> Aji
>>
>>
>> On Mon, Sep 9, 2013 at 1:30 PM, Adam Muise <am...@hortonworks.com>wrote:
>>
>>> Keep the Hadoop cluster and it's network backbone on IPv4. It should be
>>> isolated from the rest of your network for security and performance reasons
>>> already anyway.
>>>
>>> Enable IPv6 on your gateway nodes and let clients jump in for access via
>>> the ipv6 network and use the cluster via ipv4 on the gateway node. You
>>> could even setup a web proxy for some of the required sites using nginx or
>>> your favorite proxy server. Leave the ipv6 conversion of the Hadoop cluster
>>> until it is fully supported and well-tested.
>>>
>>>
>>> On Mon, Sep 9, 2013 at 1:19 PM, Aji Janis <aj...@gmail.com> wrote:
>>>
>>>> We are moving to a IPv6 network. I know Hadoop installation says to
>>>> disable IPv6. Does anyone have any suggestions for work arround? Currently,
>>>> using Cloudera version 0.20.2-cdh3u5. Thanks in advance.
>>>>
>>>
>>>
>>>
>>> --
>>> *
>>> *
>>> *
>>> *
>>> *Adam Muise*
>>> Solution Engineer
>>> *Hortonworks*
>>> amuise@hortonworks.com
>>> 416-417-4037
>>>
>>> Hortonworks - Develops, Distributes and Supports Enterprise Apache
>>> Hadoop. <http://hortonworks.com/>
>>>
>>> Hortonworks Virtual Sandbox <http://hortonworks.com/sandbox>
>>>
>>> Hadoop: Disruptive Possibilities by Jeff Needham<http://hortonworks.com/resources/?did=72&cat=1>
>>>
>>> CONFIDENTIALITY NOTICE
>>> NOTICE: This message is intended for the use of the individual or entity
>>> to which it is addressed and may contain information that is confidential,
>>> privileged and exempt from disclosure under applicable law. If the reader
>>> of this message is not the intended recipient, you are hereby notified that
>>> any printing, copying, dissemination, distribution, disclosure or
>>> forwarding of this communication is strictly prohibited. If you have
>>> received this communication in error, please contact the sender immediately
>>> and delete it from your system. Thank You.
>>
>>
>>
>
>
> --
> *
> *
> *
> *
> *Adam Muise*
> Solution Engineer
> *Hortonworks*
> amuise@hortonworks.com
> 416-417-4037
>
> Hortonworks - Develops, Distributes and Supports Enterprise Apache Hadoop.<http://hortonworks.com/>
>
> Hortonworks Virtual Sandbox <http://hortonworks.com/sandbox>
>
> Hadoop: Disruptive Possibilities by Jeff Needham<http://hortonworks.com/resources/?did=72&cat=1>
>
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity
> to which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
>
Re: Hadoop on IPv6
Posted by Aji Janis <aj...@gmail.com>.
I really appreciate the suggestions. Thank you.
On Mon, Sep 9, 2013 at 4:48 PM, Adam Muise <am...@hortonworks.com> wrote:
> I suggest you go to the mighty Google. This is not a Hadoop configuration
> question, it's a web server / proxy configuration.
>
> Starters:
>
> http://www.progob.nl/robmaaseu/index.php/an-easy-way-to-get-ipv6-ready-on-the-outside-ipv6-to-ipv4-proxy/
>
> and proxying with nginx
> http://nginx.org/en/docs/http/ngx_http_proxy_module.html
>
> Eventually Knox will help out by adding a REST/Web set of gateways for
> Hadoop services:
> http://knox.incubator.apache.org
> For now, your web based gateway services on the cluster can be proxied and
> you can still land on your gateway box with ipv6 capabilities on the
> outside of the secure cluster. That gateway box would have direct ipv4
> connections to the cluster services.
>
> Cheers,
>
>
> On Mon, Sep 9, 2013 at 4:35 PM, Aji Janis <aj...@gmail.com> wrote:
>
>> Adam,
>> Thank you for the suggestion. Could you point me to any documentation
>> that can help with the configuration changes I need to make? Thanks.
>>
>> Aji
>>
>>
>> On Mon, Sep 9, 2013 at 1:30 PM, Adam Muise <am...@hortonworks.com>wrote:
>>
>>> Keep the Hadoop cluster and it's network backbone on IPv4. It should be
>>> isolated from the rest of your network for security and performance reasons
>>> already anyway.
>>>
>>> Enable IPv6 on your gateway nodes and let clients jump in for access via
>>> the ipv6 network and use the cluster via ipv4 on the gateway node. You
>>> could even setup a web proxy for some of the required sites using nginx or
>>> your favorite proxy server. Leave the ipv6 conversion of the Hadoop cluster
>>> until it is fully supported and well-tested.
>>>
>>>
>>> On Mon, Sep 9, 2013 at 1:19 PM, Aji Janis <aj...@gmail.com> wrote:
>>>
>>>> We are moving to a IPv6 network. I know Hadoop installation says to
>>>> disable IPv6. Does anyone have any suggestions for work arround? Currently,
>>>> using Cloudera version 0.20.2-cdh3u5. Thanks in advance.
>>>>
>>>
>>>
>>>
>>> --
>>> *
>>> *
>>> *
>>> *
>>> *Adam Muise*
>>> Solution Engineer
>>> *Hortonworks*
>>> amuise@hortonworks.com
>>> 416-417-4037
>>>
>>> Hortonworks - Develops, Distributes and Supports Enterprise Apache
>>> Hadoop. <http://hortonworks.com/>
>>>
>>> Hortonworks Virtual Sandbox <http://hortonworks.com/sandbox>
>>>
>>> Hadoop: Disruptive Possibilities by Jeff Needham<http://hortonworks.com/resources/?did=72&cat=1>
>>>
>>> CONFIDENTIALITY NOTICE
>>> NOTICE: This message is intended for the use of the individual or entity
>>> to which it is addressed and may contain information that is confidential,
>>> privileged and exempt from disclosure under applicable law. If the reader
>>> of this message is not the intended recipient, you are hereby notified that
>>> any printing, copying, dissemination, distribution, disclosure or
>>> forwarding of this communication is strictly prohibited. If you have
>>> received this communication in error, please contact the sender immediately
>>> and delete it from your system. Thank You.
>>
>>
>>
>
>
> --
> *
> *
> *
> *
> *Adam Muise*
> Solution Engineer
> *Hortonworks*
> amuise@hortonworks.com
> 416-417-4037
>
> Hortonworks - Develops, Distributes and Supports Enterprise Apache Hadoop.<http://hortonworks.com/>
>
> Hortonworks Virtual Sandbox <http://hortonworks.com/sandbox>
>
> Hadoop: Disruptive Possibilities by Jeff Needham<http://hortonworks.com/resources/?did=72&cat=1>
>
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity
> to which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
>
Re: Hadoop on IPv6
Posted by Aji Janis <aj...@gmail.com>.
I really appreciate the suggestions. Thank you.
On Mon, Sep 9, 2013 at 4:48 PM, Adam Muise <am...@hortonworks.com> wrote:
> I suggest you go to the mighty Google. This is not a Hadoop configuration
> question, it's a web server / proxy configuration.
>
> Starters:
>
> http://www.progob.nl/robmaaseu/index.php/an-easy-way-to-get-ipv6-ready-on-the-outside-ipv6-to-ipv4-proxy/
>
> and proxying with nginx
> http://nginx.org/en/docs/http/ngx_http_proxy_module.html
>
> Eventually Knox will help out by adding a REST/Web set of gateways for
> Hadoop services:
> http://knox.incubator.apache.org
> For now, your web based gateway services on the cluster can be proxied and
> you can still land on your gateway box with ipv6 capabilities on the
> outside of the secure cluster. That gateway box would have direct ipv4
> connections to the cluster services.
>
> Cheers,
>
>
> On Mon, Sep 9, 2013 at 4:35 PM, Aji Janis <aj...@gmail.com> wrote:
>
>> Adam,
>> Thank you for the suggestion. Could you point me to any documentation
>> that can help with the configuration changes I need to make? Thanks.
>>
>> Aji
>>
>>
>> On Mon, Sep 9, 2013 at 1:30 PM, Adam Muise <am...@hortonworks.com>wrote:
>>
>>> Keep the Hadoop cluster and it's network backbone on IPv4. It should be
>>> isolated from the rest of your network for security and performance reasons
>>> already anyway.
>>>
>>> Enable IPv6 on your gateway nodes and let clients jump in for access via
>>> the ipv6 network and use the cluster via ipv4 on the gateway node. You
>>> could even setup a web proxy for some of the required sites using nginx or
>>> your favorite proxy server. Leave the ipv6 conversion of the Hadoop cluster
>>> until it is fully supported and well-tested.
>>>
>>>
>>> On Mon, Sep 9, 2013 at 1:19 PM, Aji Janis <aj...@gmail.com> wrote:
>>>
>>>> We are moving to a IPv6 network. I know Hadoop installation says to
>>>> disable IPv6. Does anyone have any suggestions for work arround? Currently,
>>>> using Cloudera version 0.20.2-cdh3u5. Thanks in advance.
>>>>
>>>
>>>
>>>
>>> --
>>> *
>>> *
>>> *
>>> *
>>> *Adam Muise*
>>> Solution Engineer
>>> *Hortonworks*
>>> amuise@hortonworks.com
>>> 416-417-4037
>>>
>>> Hortonworks - Develops, Distributes and Supports Enterprise Apache
>>> Hadoop. <http://hortonworks.com/>
>>>
>>> Hortonworks Virtual Sandbox <http://hortonworks.com/sandbox>
>>>
>>> Hadoop: Disruptive Possibilities by Jeff Needham<http://hortonworks.com/resources/?did=72&cat=1>
>>>
>>> CONFIDENTIALITY NOTICE
>>> NOTICE: This message is intended for the use of the individual or entity
>>> to which it is addressed and may contain information that is confidential,
>>> privileged and exempt from disclosure under applicable law. If the reader
>>> of this message is not the intended recipient, you are hereby notified that
>>> any printing, copying, dissemination, distribution, disclosure or
>>> forwarding of this communication is strictly prohibited. If you have
>>> received this communication in error, please contact the sender immediately
>>> and delete it from your system. Thank You.
>>
>>
>>
>
>
> --
> *
> *
> *
> *
> *Adam Muise*
> Solution Engineer
> *Hortonworks*
> amuise@hortonworks.com
> 416-417-4037
>
> Hortonworks - Develops, Distributes and Supports Enterprise Apache Hadoop.<http://hortonworks.com/>
>
> Hortonworks Virtual Sandbox <http://hortonworks.com/sandbox>
>
> Hadoop: Disruptive Possibilities by Jeff Needham<http://hortonworks.com/resources/?did=72&cat=1>
>
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity
> to which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
>
Re: Hadoop on IPv6
Posted by Aji Janis <aj...@gmail.com>.
I really appreciate the suggestions. Thank you.
On Mon, Sep 9, 2013 at 4:48 PM, Adam Muise <am...@hortonworks.com> wrote:
> I suggest you go to the mighty Google. This is not a Hadoop configuration
> question, it's a web server / proxy configuration.
>
> Starters:
>
> http://www.progob.nl/robmaaseu/index.php/an-easy-way-to-get-ipv6-ready-on-the-outside-ipv6-to-ipv4-proxy/
>
> and proxying with nginx
> http://nginx.org/en/docs/http/ngx_http_proxy_module.html
>
> Eventually Knox will help out by adding a REST/Web set of gateways for
> Hadoop services:
> http://knox.incubator.apache.org
> For now, your web based gateway services on the cluster can be proxied and
> you can still land on your gateway box with ipv6 capabilities on the
> outside of the secure cluster. That gateway box would have direct ipv4
> connections to the cluster services.
>
> Cheers,
>
>
> On Mon, Sep 9, 2013 at 4:35 PM, Aji Janis <aj...@gmail.com> wrote:
>
>> Adam,
>> Thank you for the suggestion. Could you point me to any documentation
>> that can help with the configuration changes I need to make? Thanks.
>>
>> Aji
>>
>>
>> On Mon, Sep 9, 2013 at 1:30 PM, Adam Muise <am...@hortonworks.com>wrote:
>>
>>> Keep the Hadoop cluster and it's network backbone on IPv4. It should be
>>> isolated from the rest of your network for security and performance reasons
>>> already anyway.
>>>
>>> Enable IPv6 on your gateway nodes and let clients jump in for access via
>>> the ipv6 network and use the cluster via ipv4 on the gateway node. You
>>> could even setup a web proxy for some of the required sites using nginx or
>>> your favorite proxy server. Leave the ipv6 conversion of the Hadoop cluster
>>> until it is fully supported and well-tested.
>>>
>>>
>>> On Mon, Sep 9, 2013 at 1:19 PM, Aji Janis <aj...@gmail.com> wrote:
>>>
>>>> We are moving to a IPv6 network. I know Hadoop installation says to
>>>> disable IPv6. Does anyone have any suggestions for work arround? Currently,
>>>> using Cloudera version 0.20.2-cdh3u5. Thanks in advance.
>>>>
>>>
>>>
>>>
>>> --
>>> *
>>> *
>>> *
>>> *
>>> *Adam Muise*
>>> Solution Engineer
>>> *Hortonworks*
>>> amuise@hortonworks.com
>>> 416-417-4037
>>>
>>> Hortonworks - Develops, Distributes and Supports Enterprise Apache
>>> Hadoop. <http://hortonworks.com/>
>>>
>>> Hortonworks Virtual Sandbox <http://hortonworks.com/sandbox>
>>>
>>> Hadoop: Disruptive Possibilities by Jeff Needham<http://hortonworks.com/resources/?did=72&cat=1>
>>>
>>> CONFIDENTIALITY NOTICE
>>> NOTICE: This message is intended for the use of the individual or entity
>>> to which it is addressed and may contain information that is confidential,
>>> privileged and exempt from disclosure under applicable law. If the reader
>>> of this message is not the intended recipient, you are hereby notified that
>>> any printing, copying, dissemination, distribution, disclosure or
>>> forwarding of this communication is strictly prohibited. If you have
>>> received this communication in error, please contact the sender immediately
>>> and delete it from your system. Thank You.
>>
>>
>>
>
>
> --
> *
> *
> *
> *
> *Adam Muise*
> Solution Engineer
> *Hortonworks*
> amuise@hortonworks.com
> 416-417-4037
>
> Hortonworks - Develops, Distributes and Supports Enterprise Apache Hadoop.<http://hortonworks.com/>
>
> Hortonworks Virtual Sandbox <http://hortonworks.com/sandbox>
>
> Hadoop: Disruptive Possibilities by Jeff Needham<http://hortonworks.com/resources/?did=72&cat=1>
>
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity
> to which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
>
Re: Hadoop on IPv6
Posted by Adam Muise <am...@hortonworks.com>.
I suggest you go to the mighty Google. This is not a Hadoop configuration
question, it's a web server / proxy configuration.
Starters:
http://www.progob.nl/robmaaseu/index.php/an-easy-way-to-get-ipv6-ready-on-the-outside-ipv6-to-ipv4-proxy/
and proxying with nginx
http://nginx.org/en/docs/http/ngx_http_proxy_module.html
Eventually Knox will help out by adding a REST/Web set of gateways for
Hadoop services:
http://knox.incubator.apache.org
For now, your web based gateway services on the cluster can be proxied and
you can still land on your gateway box with ipv6 capabilities on the
outside of the secure cluster. That gateway box would have direct ipv4
connections to the cluster services.
Cheers,
On Mon, Sep 9, 2013 at 4:35 PM, Aji Janis <aj...@gmail.com> wrote:
> Adam,
> Thank you for the suggestion. Could you point me to any documentation that
> can help with the configuration changes I need to make? Thanks.
>
> Aji
>
>
> On Mon, Sep 9, 2013 at 1:30 PM, Adam Muise <am...@hortonworks.com> wrote:
>
>> Keep the Hadoop cluster and it's network backbone on IPv4. It should be
>> isolated from the rest of your network for security and performance reasons
>> already anyway.
>>
>> Enable IPv6 on your gateway nodes and let clients jump in for access via
>> the ipv6 network and use the cluster via ipv4 on the gateway node. You
>> could even setup a web proxy for some of the required sites using nginx or
>> your favorite proxy server. Leave the ipv6 conversion of the Hadoop cluster
>> until it is fully supported and well-tested.
>>
>>
>> On Mon, Sep 9, 2013 at 1:19 PM, Aji Janis <aj...@gmail.com> wrote:
>>
>>> We are moving to a IPv6 network. I know Hadoop installation says to
>>> disable IPv6. Does anyone have any suggestions for work arround? Currently,
>>> using Cloudera version 0.20.2-cdh3u5. Thanks in advance.
>>>
>>
>>
>>
>> --
>> *
>> *
>> *
>> *
>> *Adam Muise*
>> Solution Engineer
>> *Hortonworks*
>> amuise@hortonworks.com
>> 416-417-4037
>>
>> Hortonworks - Develops, Distributes and Supports Enterprise Apache Hadoop.<http://hortonworks.com/>
>>
>> Hortonworks Virtual Sandbox <http://hortonworks.com/sandbox>
>>
>> Hadoop: Disruptive Possibilities by Jeff Needham<http://hortonworks.com/resources/?did=72&cat=1>
>>
>> CONFIDENTIALITY NOTICE
>> NOTICE: This message is intended for the use of the individual or entity
>> to which it is addressed and may contain information that is confidential,
>> privileged and exempt from disclosure under applicable law. If the reader
>> of this message is not the intended recipient, you are hereby notified that
>> any printing, copying, dissemination, distribution, disclosure or
>> forwarding of this communication is strictly prohibited. If you have
>> received this communication in error, please contact the sender immediately
>> and delete it from your system. Thank You.
>
>
>
--
*
*
*
*
*Adam Muise*
Solution Engineer
*Hortonworks*
amuise@hortonworks.com
416-417-4037
Hortonworks - Develops, Distributes and Supports Enterprise Apache
Hadoop.<http://hortonworks.com/>
Hortonworks Virtual Sandbox <http://hortonworks.com/sandbox>
Hadoop: Disruptive Possibilities by Jeff
Needham<http://hortonworks.com/resources/?did=72&cat=1>
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: Hadoop on IPv6
Posted by Aji Janis <aj...@gmail.com>.
Thank you for the clarification Adam.
On Tue, Sep 10, 2013 at 12:34 PM, Adam Muise <am...@hortonworks.com> wrote:
> Harsh is giving you a best practice for JVMs using IPv4 in general. As
> what I am suggesting is IPv4-only connections to the Hadoop daemons and
> clients on the cluster and gateway, you would not have issues heeding his
> advice.
>
> Obviously you would not set "-Djava.net.preferIPv4Stack=true" to any java
> client connecting to the proxy server as the proxy would expect IPv6 and
> you stated that's what you wanted your network to run on. The assumption is
> that any application or client outside of the cluster and gateway node only
> uses IPv6.
>
> I suggest you identify how your clients access the Hadoop cluster (based
> on Hive JDBC/ODBC, JobTracker, Datanodes, NameNode), whether they use a
> gateway node or simply log into cluster boxes, and what applications rely
> on connections to this cluster. That is your list of hosts and ports that
> you will have to find a network solution for or hide behind a gateway node.
> Make your decisions around this.
>
> Remember that my proxy suggestion only worked for RESTful web based
> traffic and that's not the entire stack that typically gets used in Hadoop.
> ODBC/JDBC does NOT fall under this category for example. If you are
> upgrading your network already, ask the network specialists or consultants
> about NAT64 and equivalent options.
>
>
> On Tue, Sep 10, 2013 at 12:12 PM, Aji Janis <aj...@gmail.com> wrote:
>
>> Harsh,
>>
>> We have the config params that you mentioned, but to be clear are you
>> saying that the suggestions Adam made won't work because of these
>> configurations ?
>>
>> Thanks,
>> Aji
>>
>>
>> On Mon, Sep 9, 2013 at 4:55 PM, Harsh J <ha...@cloudera.com> wrote:
>>
>>> For Hadoop JVMs, generally speaking, need to have the following in
>>> your hadoop-env.sh for HADOOP_OPTS:
>>>
>>> export HADOOP_OPTS="-Djava.net.preferIPv4Stack=true $HADOOP_OPTS"
>>>
>>> We also ship with this pre-added in the default configs of more recent
>>> releases.
>>>
>>> You may also need these on other JVM based daemons' opts, such as
>>> HBASE_OPTS/etc. depending on what all you run.
>>>
>>> On Tue, Sep 10, 2013 at 2:05 AM, Aji Janis <aj...@gmail.com> wrote:
>>> > Adam,
>>> > Thank you for the suggestion. Could you point me to any documentation
>>> that
>>> > can help with the configuration changes I need to make? Thanks.
>>> >
>>> > Aji
>>> >
>>> >
>>> > On Mon, Sep 9, 2013 at 1:30 PM, Adam Muise <am...@hortonworks.com>
>>> wrote:
>>> >>
>>> >> Keep the Hadoop cluster and it's network backbone on IPv4. It should
>>> be
>>> >> isolated from the rest of your network for security and performance
>>> reasons
>>> >> already anyway.
>>> >>
>>> >> Enable IPv6 on your gateway nodes and let clients jump in for access
>>> via
>>> >> the ipv6 network and use the cluster via ipv4 on the gateway node.
>>> You could
>>> >> even setup a web proxy for some of the required sites using nginx or
>>> your
>>> >> favorite proxy server. Leave the ipv6 conversion of the Hadoop
>>> cluster until
>>> >> it is fully supported and well-tested.
>>> >>
>>> >>
>>> >> On Mon, Sep 9, 2013 at 1:19 PM, Aji Janis <aj...@gmail.com> wrote:
>>> >>>
>>> >>> We are moving to a IPv6 network. I know Hadoop installation says to
>>> >>> disable IPv6. Does anyone have any suggestions for work arround?
>>> Currently,
>>> >>> using Cloudera version 0.20.2-cdh3u5. Thanks in advance.
>>> >>
>>> >>
>>> >>
>>> >>
>>> >> --
>>> >>
>>> >>
>>> >> Adam Muise
>>> >> Solution Engineer
>>> >> Hortonworks
>>> >> amuise@hortonworks.com
>>> >> 416-417-4037
>>> >>
>>> >> Hortonworks - Develops, Distributes and Supports Enterprise Apache
>>> Hadoop.
>>> >>
>>> >> Hortonworks Virtual Sandbox
>>> >>
>>> >> Hadoop: Disruptive Possibilities by Jeff Needham
>>> >>
>>> >> CONFIDENTIALITY NOTICE
>>> >> NOTICE: This message is intended for the use of the individual or
>>> entity
>>> >> to which it is addressed and may contain information that is
>>> confidential,
>>> >> privileged and exempt from disclosure under applicable law. If the
>>> reader of
>>> >> this message is not the intended recipient, you are hereby notified
>>> that any
>>> >> printing, copying, dissemination, distribution, disclosure or
>>> forwarding of
>>> >> this communication is strictly prohibited. If you have received this
>>> >> communication in error, please contact the sender immediately and
>>> delete it
>>> >> from your system. Thank You.
>>> >
>>> >
>>>
>>>
>>>
>>> --
>>> Harsh J
>>>
>>
>>
>
>
> --
> *
> *
> *
> *
> *Adam Muise*
> Solution Engineer
> *Hortonworks*
> amuise@hortonworks.com
> 416-417-4037
>
> Hortonworks - Develops, Distributes and Supports Enterprise Apache Hadoop.<http://hortonworks.com/>
>
> Hortonworks Virtual Sandbox <http://hortonworks.com/sandbox>
>
> Hadoop: Disruptive Possibilities by Jeff Needham<http://hortonworks.com/resources/?did=72&cat=1>
>
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity
> to which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
>
Re: Hadoop on IPv6
Posted by Aji Janis <aj...@gmail.com>.
Thank you for the clarification Adam.
On Tue, Sep 10, 2013 at 12:34 PM, Adam Muise <am...@hortonworks.com> wrote:
> Harsh is giving you a best practice for JVMs using IPv4 in general. As
> what I am suggesting is IPv4-only connections to the Hadoop daemons and
> clients on the cluster and gateway, you would not have issues heeding his
> advice.
>
> Obviously you would not set "-Djava.net.preferIPv4Stack=true" to any java
> client connecting to the proxy server as the proxy would expect IPv6 and
> you stated that's what you wanted your network to run on. The assumption is
> that any application or client outside of the cluster and gateway node only
> uses IPv6.
>
> I suggest you identify how your clients access the Hadoop cluster (based
> on Hive JDBC/ODBC, JobTracker, Datanodes, NameNode), whether they use a
> gateway node or simply log into cluster boxes, and what applications rely
> on connections to this cluster. That is your list of hosts and ports that
> you will have to find a network solution for or hide behind a gateway node.
> Make your decisions around this.
>
> Remember that my proxy suggestion only worked for RESTful web based
> traffic and that's not the entire stack that typically gets used in Hadoop.
> ODBC/JDBC does NOT fall under this category for example. If you are
> upgrading your network already, ask the network specialists or consultants
> about NAT64 and equivalent options.
>
>
> On Tue, Sep 10, 2013 at 12:12 PM, Aji Janis <aj...@gmail.com> wrote:
>
>> Harsh,
>>
>> We have the config params that you mentioned, but to be clear are you
>> saying that the suggestions Adam made won't work because of these
>> configurations ?
>>
>> Thanks,
>> Aji
>>
>>
>> On Mon, Sep 9, 2013 at 4:55 PM, Harsh J <ha...@cloudera.com> wrote:
>>
>>> For Hadoop JVMs, generally speaking, need to have the following in
>>> your hadoop-env.sh for HADOOP_OPTS:
>>>
>>> export HADOOP_OPTS="-Djava.net.preferIPv4Stack=true $HADOOP_OPTS"
>>>
>>> We also ship with this pre-added in the default configs of more recent
>>> releases.
>>>
>>> You may also need these on other JVM based daemons' opts, such as
>>> HBASE_OPTS/etc. depending on what all you run.
>>>
>>> On Tue, Sep 10, 2013 at 2:05 AM, Aji Janis <aj...@gmail.com> wrote:
>>> > Adam,
>>> > Thank you for the suggestion. Could you point me to any documentation
>>> that
>>> > can help with the configuration changes I need to make? Thanks.
>>> >
>>> > Aji
>>> >
>>> >
>>> > On Mon, Sep 9, 2013 at 1:30 PM, Adam Muise <am...@hortonworks.com>
>>> wrote:
>>> >>
>>> >> Keep the Hadoop cluster and it's network backbone on IPv4. It should
>>> be
>>> >> isolated from the rest of your network for security and performance
>>> reasons
>>> >> already anyway.
>>> >>
>>> >> Enable IPv6 on your gateway nodes and let clients jump in for access
>>> via
>>> >> the ipv6 network and use the cluster via ipv4 on the gateway node.
>>> You could
>>> >> even setup a web proxy for some of the required sites using nginx or
>>> your
>>> >> favorite proxy server. Leave the ipv6 conversion of the Hadoop
>>> cluster until
>>> >> it is fully supported and well-tested.
>>> >>
>>> >>
>>> >> On Mon, Sep 9, 2013 at 1:19 PM, Aji Janis <aj...@gmail.com> wrote:
>>> >>>
>>> >>> We are moving to a IPv6 network. I know Hadoop installation says to
>>> >>> disable IPv6. Does anyone have any suggestions for work arround?
>>> Currently,
>>> >>> using Cloudera version 0.20.2-cdh3u5. Thanks in advance.
>>> >>
>>> >>
>>> >>
>>> >>
>>> >> --
>>> >>
>>> >>
>>> >> Adam Muise
>>> >> Solution Engineer
>>> >> Hortonworks
>>> >> amuise@hortonworks.com
>>> >> 416-417-4037
>>> >>
>>> >> Hortonworks - Develops, Distributes and Supports Enterprise Apache
>>> Hadoop.
>>> >>
>>> >> Hortonworks Virtual Sandbox
>>> >>
>>> >> Hadoop: Disruptive Possibilities by Jeff Needham
>>> >>
>>> >> CONFIDENTIALITY NOTICE
>>> >> NOTICE: This message is intended for the use of the individual or
>>> entity
>>> >> to which it is addressed and may contain information that is
>>> confidential,
>>> >> privileged and exempt from disclosure under applicable law. If the
>>> reader of
>>> >> this message is not the intended recipient, you are hereby notified
>>> that any
>>> >> printing, copying, dissemination, distribution, disclosure or
>>> forwarding of
>>> >> this communication is strictly prohibited. If you have received this
>>> >> communication in error, please contact the sender immediately and
>>> delete it
>>> >> from your system. Thank You.
>>> >
>>> >
>>>
>>>
>>>
>>> --
>>> Harsh J
>>>
>>
>>
>
>
> --
> *
> *
> *
> *
> *Adam Muise*
> Solution Engineer
> *Hortonworks*
> amuise@hortonworks.com
> 416-417-4037
>
> Hortonworks - Develops, Distributes and Supports Enterprise Apache Hadoop.<http://hortonworks.com/>
>
> Hortonworks Virtual Sandbox <http://hortonworks.com/sandbox>
>
> Hadoop: Disruptive Possibilities by Jeff Needham<http://hortonworks.com/resources/?did=72&cat=1>
>
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity
> to which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
>
Re: Hadoop on IPv6
Posted by Aji Janis <aj...@gmail.com>.
Thank you for the clarification Adam.
On Tue, Sep 10, 2013 at 12:34 PM, Adam Muise <am...@hortonworks.com> wrote:
> Harsh is giving you a best practice for JVMs using IPv4 in general. As
> what I am suggesting is IPv4-only connections to the Hadoop daemons and
> clients on the cluster and gateway, you would not have issues heeding his
> advice.
>
> Obviously you would not set "-Djava.net.preferIPv4Stack=true" to any java
> client connecting to the proxy server as the proxy would expect IPv6 and
> you stated that's what you wanted your network to run on. The assumption is
> that any application or client outside of the cluster and gateway node only
> uses IPv6.
>
> I suggest you identify how your clients access the Hadoop cluster (based
> on Hive JDBC/ODBC, JobTracker, Datanodes, NameNode), whether they use a
> gateway node or simply log into cluster boxes, and what applications rely
> on connections to this cluster. That is your list of hosts and ports that
> you will have to find a network solution for or hide behind a gateway node.
> Make your decisions around this.
>
> Remember that my proxy suggestion only worked for RESTful web based
> traffic and that's not the entire stack that typically gets used in Hadoop.
> ODBC/JDBC does NOT fall under this category for example. If you are
> upgrading your network already, ask the network specialists or consultants
> about NAT64 and equivalent options.
>
>
> On Tue, Sep 10, 2013 at 12:12 PM, Aji Janis <aj...@gmail.com> wrote:
>
>> Harsh,
>>
>> We have the config params that you mentioned, but to be clear are you
>> saying that the suggestions Adam made won't work because of these
>> configurations ?
>>
>> Thanks,
>> Aji
>>
>>
>> On Mon, Sep 9, 2013 at 4:55 PM, Harsh J <ha...@cloudera.com> wrote:
>>
>>> For Hadoop JVMs, generally speaking, need to have the following in
>>> your hadoop-env.sh for HADOOP_OPTS:
>>>
>>> export HADOOP_OPTS="-Djava.net.preferIPv4Stack=true $HADOOP_OPTS"
>>>
>>> We also ship with this pre-added in the default configs of more recent
>>> releases.
>>>
>>> You may also need these on other JVM based daemons' opts, such as
>>> HBASE_OPTS/etc. depending on what all you run.
>>>
>>> On Tue, Sep 10, 2013 at 2:05 AM, Aji Janis <aj...@gmail.com> wrote:
>>> > Adam,
>>> > Thank you for the suggestion. Could you point me to any documentation
>>> that
>>> > can help with the configuration changes I need to make? Thanks.
>>> >
>>> > Aji
>>> >
>>> >
>>> > On Mon, Sep 9, 2013 at 1:30 PM, Adam Muise <am...@hortonworks.com>
>>> wrote:
>>> >>
>>> >> Keep the Hadoop cluster and it's network backbone on IPv4. It should
>>> be
>>> >> isolated from the rest of your network for security and performance
>>> reasons
>>> >> already anyway.
>>> >>
>>> >> Enable IPv6 on your gateway nodes and let clients jump in for access
>>> via
>>> >> the ipv6 network and use the cluster via ipv4 on the gateway node.
>>> You could
>>> >> even setup a web proxy for some of the required sites using nginx or
>>> your
>>> >> favorite proxy server. Leave the ipv6 conversion of the Hadoop
>>> cluster until
>>> >> it is fully supported and well-tested.
>>> >>
>>> >>
>>> >> On Mon, Sep 9, 2013 at 1:19 PM, Aji Janis <aj...@gmail.com> wrote:
>>> >>>
>>> >>> We are moving to a IPv6 network. I know Hadoop installation says to
>>> >>> disable IPv6. Does anyone have any suggestions for work arround?
>>> Currently,
>>> >>> using Cloudera version 0.20.2-cdh3u5. Thanks in advance.
>>> >>
>>> >>
>>> >>
>>> >>
>>> >> --
>>> >>
>>> >>
>>> >> Adam Muise
>>> >> Solution Engineer
>>> >> Hortonworks
>>> >> amuise@hortonworks.com
>>> >> 416-417-4037
>>> >>
>>> >> Hortonworks - Develops, Distributes and Supports Enterprise Apache
>>> Hadoop.
>>> >>
>>> >> Hortonworks Virtual Sandbox
>>> >>
>>> >> Hadoop: Disruptive Possibilities by Jeff Needham
>>> >>
>>> >> CONFIDENTIALITY NOTICE
>>> >> NOTICE: This message is intended for the use of the individual or
>>> entity
>>> >> to which it is addressed and may contain information that is
>>> confidential,
>>> >> privileged and exempt from disclosure under applicable law. If the
>>> reader of
>>> >> this message is not the intended recipient, you are hereby notified
>>> that any
>>> >> printing, copying, dissemination, distribution, disclosure or
>>> forwarding of
>>> >> this communication is strictly prohibited. If you have received this
>>> >> communication in error, please contact the sender immediately and
>>> delete it
>>> >> from your system. Thank You.
>>> >
>>> >
>>>
>>>
>>>
>>> --
>>> Harsh J
>>>
>>
>>
>
>
> --
> *
> *
> *
> *
> *Adam Muise*
> Solution Engineer
> *Hortonworks*
> amuise@hortonworks.com
> 416-417-4037
>
> Hortonworks - Develops, Distributes and Supports Enterprise Apache Hadoop.<http://hortonworks.com/>
>
> Hortonworks Virtual Sandbox <http://hortonworks.com/sandbox>
>
> Hadoop: Disruptive Possibilities by Jeff Needham<http://hortonworks.com/resources/?did=72&cat=1>
>
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity
> to which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
>
Re: Hadoop on IPv6
Posted by Aji Janis <aj...@gmail.com>.
Thank you for the clarification Adam.
On Tue, Sep 10, 2013 at 12:34 PM, Adam Muise <am...@hortonworks.com> wrote:
> Harsh is giving you a best practice for JVMs using IPv4 in general. As
> what I am suggesting is IPv4-only connections to the Hadoop daemons and
> clients on the cluster and gateway, you would not have issues heeding his
> advice.
>
> Obviously you would not set "-Djava.net.preferIPv4Stack=true" to any java
> client connecting to the proxy server as the proxy would expect IPv6 and
> you stated that's what you wanted your network to run on. The assumption is
> that any application or client outside of the cluster and gateway node only
> uses IPv6.
>
> I suggest you identify how your clients access the Hadoop cluster (based
> on Hive JDBC/ODBC, JobTracker, Datanodes, NameNode), whether they use a
> gateway node or simply log into cluster boxes, and what applications rely
> on connections to this cluster. That is your list of hosts and ports that
> you will have to find a network solution for or hide behind a gateway node.
> Make your decisions around this.
>
> Remember that my proxy suggestion only worked for RESTful web based
> traffic and that's not the entire stack that typically gets used in Hadoop.
> ODBC/JDBC does NOT fall under this category for example. If you are
> upgrading your network already, ask the network specialists or consultants
> about NAT64 and equivalent options.
>
>
> On Tue, Sep 10, 2013 at 12:12 PM, Aji Janis <aj...@gmail.com> wrote:
>
>> Harsh,
>>
>> We have the config params that you mentioned, but to be clear are you
>> saying that the suggestions Adam made won't work because of these
>> configurations ?
>>
>> Thanks,
>> Aji
>>
>>
>> On Mon, Sep 9, 2013 at 4:55 PM, Harsh J <ha...@cloudera.com> wrote:
>>
>>> For Hadoop JVMs, generally speaking, need to have the following in
>>> your hadoop-env.sh for HADOOP_OPTS:
>>>
>>> export HADOOP_OPTS="-Djava.net.preferIPv4Stack=true $HADOOP_OPTS"
>>>
>>> We also ship with this pre-added in the default configs of more recent
>>> releases.
>>>
>>> You may also need these on other JVM based daemons' opts, such as
>>> HBASE_OPTS/etc. depending on what all you run.
>>>
>>> On Tue, Sep 10, 2013 at 2:05 AM, Aji Janis <aj...@gmail.com> wrote:
>>> > Adam,
>>> > Thank you for the suggestion. Could you point me to any documentation
>>> that
>>> > can help with the configuration changes I need to make? Thanks.
>>> >
>>> > Aji
>>> >
>>> >
>>> > On Mon, Sep 9, 2013 at 1:30 PM, Adam Muise <am...@hortonworks.com>
>>> wrote:
>>> >>
>>> >> Keep the Hadoop cluster and it's network backbone on IPv4. It should
>>> be
>>> >> isolated from the rest of your network for security and performance
>>> reasons
>>> >> already anyway.
>>> >>
>>> >> Enable IPv6 on your gateway nodes and let clients jump in for access
>>> via
>>> >> the ipv6 network and use the cluster via ipv4 on the gateway node.
>>> You could
>>> >> even setup a web proxy for some of the required sites using nginx or
>>> your
>>> >> favorite proxy server. Leave the ipv6 conversion of the Hadoop
>>> cluster until
>>> >> it is fully supported and well-tested.
>>> >>
>>> >>
>>> >> On Mon, Sep 9, 2013 at 1:19 PM, Aji Janis <aj...@gmail.com> wrote:
>>> >>>
>>> >>> We are moving to a IPv6 network. I know Hadoop installation says to
>>> >>> disable IPv6. Does anyone have any suggestions for work arround?
>>> Currently,
>>> >>> using Cloudera version 0.20.2-cdh3u5. Thanks in advance.
>>> >>
>>> >>
>>> >>
>>> >>
>>> >> --
>>> >>
>>> >>
>>> >> Adam Muise
>>> >> Solution Engineer
>>> >> Hortonworks
>>> >> amuise@hortonworks.com
>>> >> 416-417-4037
>>> >>
>>> >> Hortonworks - Develops, Distributes and Supports Enterprise Apache
>>> Hadoop.
>>> >>
>>> >> Hortonworks Virtual Sandbox
>>> >>
>>> >> Hadoop: Disruptive Possibilities by Jeff Needham
>>> >>
>>> >> CONFIDENTIALITY NOTICE
>>> >> NOTICE: This message is intended for the use of the individual or
>>> entity
>>> >> to which it is addressed and may contain information that is
>>> confidential,
>>> >> privileged and exempt from disclosure under applicable law. If the
>>> reader of
>>> >> this message is not the intended recipient, you are hereby notified
>>> that any
>>> >> printing, copying, dissemination, distribution, disclosure or
>>> forwarding of
>>> >> this communication is strictly prohibited. If you have received this
>>> >> communication in error, please contact the sender immediately and
>>> delete it
>>> >> from your system. Thank You.
>>> >
>>> >
>>>
>>>
>>>
>>> --
>>> Harsh J
>>>
>>
>>
>
>
> --
> *
> *
> *
> *
> *Adam Muise*
> Solution Engineer
> *Hortonworks*
> amuise@hortonworks.com
> 416-417-4037
>
> Hortonworks - Develops, Distributes and Supports Enterprise Apache Hadoop.<http://hortonworks.com/>
>
> Hortonworks Virtual Sandbox <http://hortonworks.com/sandbox>
>
> Hadoop: Disruptive Possibilities by Jeff Needham<http://hortonworks.com/resources/?did=72&cat=1>
>
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity
> to which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
>
Re: Hadoop on IPv6
Posted by Adam Muise <am...@hortonworks.com>.
Harsh is giving you a best practice for JVMs using IPv4 in general. As what
I am suggesting is IPv4-only connections to the Hadoop daemons and clients
on the cluster and gateway, you would not have issues heeding his advice.
Obviously you would not set "-Djava.net.preferIPv4Stack=true" to any java
client connecting to the proxy server as the proxy would expect IPv6 and
you stated that's what you wanted your network to run on. The assumption is
that any application or client outside of the cluster and gateway node only
uses IPv6.
I suggest you identify how your clients access the Hadoop cluster (based on
Hive JDBC/ODBC, JobTracker, Datanodes, NameNode), whether they use a
gateway node or simply log into cluster boxes, and what applications rely
on connections to this cluster. That is your list of hosts and ports that
you will have to find a network solution for or hide behind a gateway node.
Make your decisions around this.
Remember that my proxy suggestion only worked for RESTful web based traffic
and that's not the entire stack that typically gets used in Hadoop.
ODBC/JDBC does NOT fall under this category for example. If you are
upgrading your network already, ask the network specialists or consultants
about NAT64 and equivalent options.
On Tue, Sep 10, 2013 at 12:12 PM, Aji Janis <aj...@gmail.com> wrote:
> Harsh,
>
> We have the config params that you mentioned, but to be clear are you
> saying that the suggestions Adam made won't work because of these
> configurations ?
>
> Thanks,
> Aji
>
>
> On Mon, Sep 9, 2013 at 4:55 PM, Harsh J <ha...@cloudera.com> wrote:
>
>> For Hadoop JVMs, generally speaking, need to have the following in
>> your hadoop-env.sh for HADOOP_OPTS:
>>
>> export HADOOP_OPTS="-Djava.net.preferIPv4Stack=true $HADOOP_OPTS"
>>
>> We also ship with this pre-added in the default configs of more recent
>> releases.
>>
>> You may also need these on other JVM based daemons' opts, such as
>> HBASE_OPTS/etc. depending on what all you run.
>>
>> On Tue, Sep 10, 2013 at 2:05 AM, Aji Janis <aj...@gmail.com> wrote:
>> > Adam,
>> > Thank you for the suggestion. Could you point me to any documentation
>> that
>> > can help with the configuration changes I need to make? Thanks.
>> >
>> > Aji
>> >
>> >
>> > On Mon, Sep 9, 2013 at 1:30 PM, Adam Muise <am...@hortonworks.com>
>> wrote:
>> >>
>> >> Keep the Hadoop cluster and it's network backbone on IPv4. It should be
>> >> isolated from the rest of your network for security and performance
>> reasons
>> >> already anyway.
>> >>
>> >> Enable IPv6 on your gateway nodes and let clients jump in for access
>> via
>> >> the ipv6 network and use the cluster via ipv4 on the gateway node. You
>> could
>> >> even setup a web proxy for some of the required sites using nginx or
>> your
>> >> favorite proxy server. Leave the ipv6 conversion of the Hadoop cluster
>> until
>> >> it is fully supported and well-tested.
>> >>
>> >>
>> >> On Mon, Sep 9, 2013 at 1:19 PM, Aji Janis <aj...@gmail.com> wrote:
>> >>>
>> >>> We are moving to a IPv6 network. I know Hadoop installation says to
>> >>> disable IPv6. Does anyone have any suggestions for work arround?
>> Currently,
>> >>> using Cloudera version 0.20.2-cdh3u5. Thanks in advance.
>> >>
>> >>
>> >>
>> >>
>> >> --
>> >>
>> >>
>> >> Adam Muise
>> >> Solution Engineer
>> >> Hortonworks
>> >> amuise@hortonworks.com
>> >> 416-417-4037
>> >>
>> >> Hortonworks - Develops, Distributes and Supports Enterprise Apache
>> Hadoop.
>> >>
>> >> Hortonworks Virtual Sandbox
>> >>
>> >> Hadoop: Disruptive Possibilities by Jeff Needham
>> >>
>> >> CONFIDENTIALITY NOTICE
>> >> NOTICE: This message is intended for the use of the individual or
>> entity
>> >> to which it is addressed and may contain information that is
>> confidential,
>> >> privileged and exempt from disclosure under applicable law. If the
>> reader of
>> >> this message is not the intended recipient, you are hereby notified
>> that any
>> >> printing, copying, dissemination, distribution, disclosure or
>> forwarding of
>> >> this communication is strictly prohibited. If you have received this
>> >> communication in error, please contact the sender immediately and
>> delete it
>> >> from your system. Thank You.
>> >
>> >
>>
>>
>>
>> --
>> Harsh J
>>
>
>
--
*
*
*
*
*Adam Muise*
Solution Engineer
*Hortonworks*
amuise@hortonworks.com
416-417-4037
Hortonworks - Develops, Distributes and Supports Enterprise Apache
Hadoop.<http://hortonworks.com/>
Hortonworks Virtual Sandbox <http://hortonworks.com/sandbox>
Hadoop: Disruptive Possibilities by Jeff
Needham<http://hortonworks.com/resources/?did=72&cat=1>
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: Hadoop on IPv6
Posted by Adam Muise <am...@hortonworks.com>.
Harsh is giving you a best practice for JVMs using IPv4 in general. As what
I am suggesting is IPv4-only connections to the Hadoop daemons and clients
on the cluster and gateway, you would not have issues heeding his advice.
Obviously you would not set "-Djava.net.preferIPv4Stack=true" to any java
client connecting to the proxy server as the proxy would expect IPv6 and
you stated that's what you wanted your network to run on. The assumption is
that any application or client outside of the cluster and gateway node only
uses IPv6.
I suggest you identify how your clients access the Hadoop cluster (based on
Hive JDBC/ODBC, JobTracker, Datanodes, NameNode), whether they use a
gateway node or simply log into cluster boxes, and what applications rely
on connections to this cluster. That is your list of hosts and ports that
you will have to find a network solution for or hide behind a gateway node.
Make your decisions around this.
Remember that my proxy suggestion only worked for RESTful web based traffic
and that's not the entire stack that typically gets used in Hadoop.
ODBC/JDBC does NOT fall under this category for example. If you are
upgrading your network already, ask the network specialists or consultants
about NAT64 and equivalent options.
On Tue, Sep 10, 2013 at 12:12 PM, Aji Janis <aj...@gmail.com> wrote:
> Harsh,
>
> We have the config params that you mentioned, but to be clear are you
> saying that the suggestions Adam made won't work because of these
> configurations ?
>
> Thanks,
> Aji
>
>
> On Mon, Sep 9, 2013 at 4:55 PM, Harsh J <ha...@cloudera.com> wrote:
>
>> For Hadoop JVMs, generally speaking, need to have the following in
>> your hadoop-env.sh for HADOOP_OPTS:
>>
>> export HADOOP_OPTS="-Djava.net.preferIPv4Stack=true $HADOOP_OPTS"
>>
>> We also ship with this pre-added in the default configs of more recent
>> releases.
>>
>> You may also need these on other JVM based daemons' opts, such as
>> HBASE_OPTS/etc. depending on what all you run.
>>
>> On Tue, Sep 10, 2013 at 2:05 AM, Aji Janis <aj...@gmail.com> wrote:
>> > Adam,
>> > Thank you for the suggestion. Could you point me to any documentation
>> that
>> > can help with the configuration changes I need to make? Thanks.
>> >
>> > Aji
>> >
>> >
>> > On Mon, Sep 9, 2013 at 1:30 PM, Adam Muise <am...@hortonworks.com>
>> wrote:
>> >>
>> >> Keep the Hadoop cluster and it's network backbone on IPv4. It should be
>> >> isolated from the rest of your network for security and performance
>> reasons
>> >> already anyway.
>> >>
>> >> Enable IPv6 on your gateway nodes and let clients jump in for access
>> via
>> >> the ipv6 network and use the cluster via ipv4 on the gateway node. You
>> could
>> >> even setup a web proxy for some of the required sites using nginx or
>> your
>> >> favorite proxy server. Leave the ipv6 conversion of the Hadoop cluster
>> until
>> >> it is fully supported and well-tested.
>> >>
>> >>
>> >> On Mon, Sep 9, 2013 at 1:19 PM, Aji Janis <aj...@gmail.com> wrote:
>> >>>
>> >>> We are moving to a IPv6 network. I know Hadoop installation says to
>> >>> disable IPv6. Does anyone have any suggestions for work arround?
>> Currently,
>> >>> using Cloudera version 0.20.2-cdh3u5. Thanks in advance.
>> >>
>> >>
>> >>
>> >>
>> >> --
>> >>
>> >>
>> >> Adam Muise
>> >> Solution Engineer
>> >> Hortonworks
>> >> amuise@hortonworks.com
>> >> 416-417-4037
>> >>
>> >> Hortonworks - Develops, Distributes and Supports Enterprise Apache
>> Hadoop.
>> >>
>> >> Hortonworks Virtual Sandbox
>> >>
>> >> Hadoop: Disruptive Possibilities by Jeff Needham
>> >>
>> >> CONFIDENTIALITY NOTICE
>> >> NOTICE: This message is intended for the use of the individual or
>> entity
>> >> to which it is addressed and may contain information that is
>> confidential,
>> >> privileged and exempt from disclosure under applicable law. If the
>> reader of
>> >> this message is not the intended recipient, you are hereby notified
>> that any
>> >> printing, copying, dissemination, distribution, disclosure or
>> forwarding of
>> >> this communication is strictly prohibited. If you have received this
>> >> communication in error, please contact the sender immediately and
>> delete it
>> >> from your system. Thank You.
>> >
>> >
>>
>>
>>
>> --
>> Harsh J
>>
>
>
--
*
*
*
*
*Adam Muise*
Solution Engineer
*Hortonworks*
amuise@hortonworks.com
416-417-4037
Hortonworks - Develops, Distributes and Supports Enterprise Apache
Hadoop.<http://hortonworks.com/>
Hortonworks Virtual Sandbox <http://hortonworks.com/sandbox>
Hadoop: Disruptive Possibilities by Jeff
Needham<http://hortonworks.com/resources/?did=72&cat=1>
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: Hadoop on IPv6
Posted by Adam Muise <am...@hortonworks.com>.
Harsh is giving you a best practice for JVMs using IPv4 in general. As what
I am suggesting is IPv4-only connections to the Hadoop daemons and clients
on the cluster and gateway, you would not have issues heeding his advice.
Obviously you would not set "-Djava.net.preferIPv4Stack=true" to any java
client connecting to the proxy server as the proxy would expect IPv6 and
you stated that's what you wanted your network to run on. The assumption is
that any application or client outside of the cluster and gateway node only
uses IPv6.
I suggest you identify how your clients access the Hadoop cluster (based on
Hive JDBC/ODBC, JobTracker, Datanodes, NameNode), whether they use a
gateway node or simply log into cluster boxes, and what applications rely
on connections to this cluster. That is your list of hosts and ports that
you will have to find a network solution for or hide behind a gateway node.
Make your decisions around this.
Remember that my proxy suggestion only worked for RESTful web based traffic
and that's not the entire stack that typically gets used in Hadoop.
ODBC/JDBC does NOT fall under this category for example. If you are
upgrading your network already, ask the network specialists or consultants
about NAT64 and equivalent options.
On Tue, Sep 10, 2013 at 12:12 PM, Aji Janis <aj...@gmail.com> wrote:
> Harsh,
>
> We have the config params that you mentioned, but to be clear are you
> saying that the suggestions Adam made won't work because of these
> configurations ?
>
> Thanks,
> Aji
>
>
> On Mon, Sep 9, 2013 at 4:55 PM, Harsh J <ha...@cloudera.com> wrote:
>
>> For Hadoop JVMs, generally speaking, need to have the following in
>> your hadoop-env.sh for HADOOP_OPTS:
>>
>> export HADOOP_OPTS="-Djava.net.preferIPv4Stack=true $HADOOP_OPTS"
>>
>> We also ship with this pre-added in the default configs of more recent
>> releases.
>>
>> You may also need these on other JVM based daemons' opts, such as
>> HBASE_OPTS/etc. depending on what all you run.
>>
>> On Tue, Sep 10, 2013 at 2:05 AM, Aji Janis <aj...@gmail.com> wrote:
>> > Adam,
>> > Thank you for the suggestion. Could you point me to any documentation
>> that
>> > can help with the configuration changes I need to make? Thanks.
>> >
>> > Aji
>> >
>> >
>> > On Mon, Sep 9, 2013 at 1:30 PM, Adam Muise <am...@hortonworks.com>
>> wrote:
>> >>
>> >> Keep the Hadoop cluster and it's network backbone on IPv4. It should be
>> >> isolated from the rest of your network for security and performance
>> reasons
>> >> already anyway.
>> >>
>> >> Enable IPv6 on your gateway nodes and let clients jump in for access
>> via
>> >> the ipv6 network and use the cluster via ipv4 on the gateway node. You
>> could
>> >> even setup a web proxy for some of the required sites using nginx or
>> your
>> >> favorite proxy server. Leave the ipv6 conversion of the Hadoop cluster
>> until
>> >> it is fully supported and well-tested.
>> >>
>> >>
>> >> On Mon, Sep 9, 2013 at 1:19 PM, Aji Janis <aj...@gmail.com> wrote:
>> >>>
>> >>> We are moving to a IPv6 network. I know Hadoop installation says to
>> >>> disable IPv6. Does anyone have any suggestions for work arround?
>> Currently,
>> >>> using Cloudera version 0.20.2-cdh3u5. Thanks in advance.
>> >>
>> >>
>> >>
>> >>
>> >> --
>> >>
>> >>
>> >> Adam Muise
>> >> Solution Engineer
>> >> Hortonworks
>> >> amuise@hortonworks.com
>> >> 416-417-4037
>> >>
>> >> Hortonworks - Develops, Distributes and Supports Enterprise Apache
>> Hadoop.
>> >>
>> >> Hortonworks Virtual Sandbox
>> >>
>> >> Hadoop: Disruptive Possibilities by Jeff Needham
>> >>
>> >> CONFIDENTIALITY NOTICE
>> >> NOTICE: This message is intended for the use of the individual or
>> entity
>> >> to which it is addressed and may contain information that is
>> confidential,
>> >> privileged and exempt from disclosure under applicable law. If the
>> reader of
>> >> this message is not the intended recipient, you are hereby notified
>> that any
>> >> printing, copying, dissemination, distribution, disclosure or
>> forwarding of
>> >> this communication is strictly prohibited. If you have received this
>> >> communication in error, please contact the sender immediately and
>> delete it
>> >> from your system. Thank You.
>> >
>> >
>>
>>
>>
>> --
>> Harsh J
>>
>
>
--
*
*
*
*
*Adam Muise*
Solution Engineer
*Hortonworks*
amuise@hortonworks.com
416-417-4037
Hortonworks - Develops, Distributes and Supports Enterprise Apache
Hadoop.<http://hortonworks.com/>
Hortonworks Virtual Sandbox <http://hortonworks.com/sandbox>
Hadoop: Disruptive Possibilities by Jeff
Needham<http://hortonworks.com/resources/?did=72&cat=1>
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: Hadoop on IPv6
Posted by Adam Muise <am...@hortonworks.com>.
Harsh is giving you a best practice for JVMs using IPv4 in general. As what
I am suggesting is IPv4-only connections to the Hadoop daemons and clients
on the cluster and gateway, you would not have issues heeding his advice.
Obviously you would not set "-Djava.net.preferIPv4Stack=true" to any java
client connecting to the proxy server as the proxy would expect IPv6 and
you stated that's what you wanted your network to run on. The assumption is
that any application or client outside of the cluster and gateway node only
uses IPv6.
I suggest you identify how your clients access the Hadoop cluster (based on
Hive JDBC/ODBC, JobTracker, Datanodes, NameNode), whether they use a
gateway node or simply log into cluster boxes, and what applications rely
on connections to this cluster. That is your list of hosts and ports that
you will have to find a network solution for or hide behind a gateway node.
Make your decisions around this.
Remember that my proxy suggestion only worked for RESTful web based traffic
and that's not the entire stack that typically gets used in Hadoop.
ODBC/JDBC does NOT fall under this category for example. If you are
upgrading your network already, ask the network specialists or consultants
about NAT64 and equivalent options.
On Tue, Sep 10, 2013 at 12:12 PM, Aji Janis <aj...@gmail.com> wrote:
> Harsh,
>
> We have the config params that you mentioned, but to be clear are you
> saying that the suggestions Adam made won't work because of these
> configurations ?
>
> Thanks,
> Aji
>
>
> On Mon, Sep 9, 2013 at 4:55 PM, Harsh J <ha...@cloudera.com> wrote:
>
>> For Hadoop JVMs, generally speaking, need to have the following in
>> your hadoop-env.sh for HADOOP_OPTS:
>>
>> export HADOOP_OPTS="-Djava.net.preferIPv4Stack=true $HADOOP_OPTS"
>>
>> We also ship with this pre-added in the default configs of more recent
>> releases.
>>
>> You may also need these on other JVM based daemons' opts, such as
>> HBASE_OPTS/etc. depending on what all you run.
>>
>> On Tue, Sep 10, 2013 at 2:05 AM, Aji Janis <aj...@gmail.com> wrote:
>> > Adam,
>> > Thank you for the suggestion. Could you point me to any documentation
>> that
>> > can help with the configuration changes I need to make? Thanks.
>> >
>> > Aji
>> >
>> >
>> > On Mon, Sep 9, 2013 at 1:30 PM, Adam Muise <am...@hortonworks.com>
>> wrote:
>> >>
>> >> Keep the Hadoop cluster and it's network backbone on IPv4. It should be
>> >> isolated from the rest of your network for security and performance
>> reasons
>> >> already anyway.
>> >>
>> >> Enable IPv6 on your gateway nodes and let clients jump in for access
>> via
>> >> the ipv6 network and use the cluster via ipv4 on the gateway node. You
>> could
>> >> even setup a web proxy for some of the required sites using nginx or
>> your
>> >> favorite proxy server. Leave the ipv6 conversion of the Hadoop cluster
>> until
>> >> it is fully supported and well-tested.
>> >>
>> >>
>> >> On Mon, Sep 9, 2013 at 1:19 PM, Aji Janis <aj...@gmail.com> wrote:
>> >>>
>> >>> We are moving to a IPv6 network. I know Hadoop installation says to
>> >>> disable IPv6. Does anyone have any suggestions for work arround?
>> Currently,
>> >>> using Cloudera version 0.20.2-cdh3u5. Thanks in advance.
>> >>
>> >>
>> >>
>> >>
>> >> --
>> >>
>> >>
>> >> Adam Muise
>> >> Solution Engineer
>> >> Hortonworks
>> >> amuise@hortonworks.com
>> >> 416-417-4037
>> >>
>> >> Hortonworks - Develops, Distributes and Supports Enterprise Apache
>> Hadoop.
>> >>
>> >> Hortonworks Virtual Sandbox
>> >>
>> >> Hadoop: Disruptive Possibilities by Jeff Needham
>> >>
>> >> CONFIDENTIALITY NOTICE
>> >> NOTICE: This message is intended for the use of the individual or
>> entity
>> >> to which it is addressed and may contain information that is
>> confidential,
>> >> privileged and exempt from disclosure under applicable law. If the
>> reader of
>> >> this message is not the intended recipient, you are hereby notified
>> that any
>> >> printing, copying, dissemination, distribution, disclosure or
>> forwarding of
>> >> this communication is strictly prohibited. If you have received this
>> >> communication in error, please contact the sender immediately and
>> delete it
>> >> from your system. Thank You.
>> >
>> >
>>
>>
>>
>> --
>> Harsh J
>>
>
>
--
*
*
*
*
*Adam Muise*
Solution Engineer
*Hortonworks*
amuise@hortonworks.com
416-417-4037
Hortonworks - Develops, Distributes and Supports Enterprise Apache
Hadoop.<http://hortonworks.com/>
Hortonworks Virtual Sandbox <http://hortonworks.com/sandbox>
Hadoop: Disruptive Possibilities by Jeff
Needham<http://hortonworks.com/resources/?did=72&cat=1>
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: Hadoop on IPv6
Posted by Aji Janis <aj...@gmail.com>.
Harsh,
We have the config params that you mentioned, but to be clear are you
saying that the suggestions Adam made won't work because of these
configurations ?
Thanks,
Aji
On Mon, Sep 9, 2013 at 4:55 PM, Harsh J <ha...@cloudera.com> wrote:
> For Hadoop JVMs, generally speaking, need to have the following in
> your hadoop-env.sh for HADOOP_OPTS:
>
> export HADOOP_OPTS="-Djava.net.preferIPv4Stack=true $HADOOP_OPTS"
>
> We also ship with this pre-added in the default configs of more recent
> releases.
>
> You may also need these on other JVM based daemons' opts, such as
> HBASE_OPTS/etc. depending on what all you run.
>
> On Tue, Sep 10, 2013 at 2:05 AM, Aji Janis <aj...@gmail.com> wrote:
> > Adam,
> > Thank you for the suggestion. Could you point me to any documentation
> that
> > can help with the configuration changes I need to make? Thanks.
> >
> > Aji
> >
> >
> > On Mon, Sep 9, 2013 at 1:30 PM, Adam Muise <am...@hortonworks.com>
> wrote:
> >>
> >> Keep the Hadoop cluster and it's network backbone on IPv4. It should be
> >> isolated from the rest of your network for security and performance
> reasons
> >> already anyway.
> >>
> >> Enable IPv6 on your gateway nodes and let clients jump in for access via
> >> the ipv6 network and use the cluster via ipv4 on the gateway node. You
> could
> >> even setup a web proxy for some of the required sites using nginx or
> your
> >> favorite proxy server. Leave the ipv6 conversion of the Hadoop cluster
> until
> >> it is fully supported and well-tested.
> >>
> >>
> >> On Mon, Sep 9, 2013 at 1:19 PM, Aji Janis <aj...@gmail.com> wrote:
> >>>
> >>> We are moving to a IPv6 network. I know Hadoop installation says to
> >>> disable IPv6. Does anyone have any suggestions for work arround?
> Currently,
> >>> using Cloudera version 0.20.2-cdh3u5. Thanks in advance.
> >>
> >>
> >>
> >>
> >> --
> >>
> >>
> >> Adam Muise
> >> Solution Engineer
> >> Hortonworks
> >> amuise@hortonworks.com
> >> 416-417-4037
> >>
> >> Hortonworks - Develops, Distributes and Supports Enterprise Apache
> Hadoop.
> >>
> >> Hortonworks Virtual Sandbox
> >>
> >> Hadoop: Disruptive Possibilities by Jeff Needham
> >>
> >> CONFIDENTIALITY NOTICE
> >> NOTICE: This message is intended for the use of the individual or entity
> >> to which it is addressed and may contain information that is
> confidential,
> >> privileged and exempt from disclosure under applicable law. If the
> reader of
> >> this message is not the intended recipient, you are hereby notified
> that any
> >> printing, copying, dissemination, distribution, disclosure or
> forwarding of
> >> this communication is strictly prohibited. If you have received this
> >> communication in error, please contact the sender immediately and
> delete it
> >> from your system. Thank You.
> >
> >
>
>
>
> --
> Harsh J
>
Re: Hadoop on IPv6
Posted by Aji Janis <aj...@gmail.com>.
Harsh,
We have the config params that you mentioned, but to be clear are you
saying that the suggestions Adam made won't work because of these
configurations ?
Thanks,
Aji
On Mon, Sep 9, 2013 at 4:55 PM, Harsh J <ha...@cloudera.com> wrote:
> For Hadoop JVMs, generally speaking, need to have the following in
> your hadoop-env.sh for HADOOP_OPTS:
>
> export HADOOP_OPTS="-Djava.net.preferIPv4Stack=true $HADOOP_OPTS"
>
> We also ship with this pre-added in the default configs of more recent
> releases.
>
> You may also need these on other JVM based daemons' opts, such as
> HBASE_OPTS/etc. depending on what all you run.
>
> On Tue, Sep 10, 2013 at 2:05 AM, Aji Janis <aj...@gmail.com> wrote:
> > Adam,
> > Thank you for the suggestion. Could you point me to any documentation
> that
> > can help with the configuration changes I need to make? Thanks.
> >
> > Aji
> >
> >
> > On Mon, Sep 9, 2013 at 1:30 PM, Adam Muise <am...@hortonworks.com>
> wrote:
> >>
> >> Keep the Hadoop cluster and it's network backbone on IPv4. It should be
> >> isolated from the rest of your network for security and performance
> reasons
> >> already anyway.
> >>
> >> Enable IPv6 on your gateway nodes and let clients jump in for access via
> >> the ipv6 network and use the cluster via ipv4 on the gateway node. You
> could
> >> even setup a web proxy for some of the required sites using nginx or
> your
> >> favorite proxy server. Leave the ipv6 conversion of the Hadoop cluster
> until
> >> it is fully supported and well-tested.
> >>
> >>
> >> On Mon, Sep 9, 2013 at 1:19 PM, Aji Janis <aj...@gmail.com> wrote:
> >>>
> >>> We are moving to a IPv6 network. I know Hadoop installation says to
> >>> disable IPv6. Does anyone have any suggestions for work arround?
> Currently,
> >>> using Cloudera version 0.20.2-cdh3u5. Thanks in advance.
> >>
> >>
> >>
> >>
> >> --
> >>
> >>
> >> Adam Muise
> >> Solution Engineer
> >> Hortonworks
> >> amuise@hortonworks.com
> >> 416-417-4037
> >>
> >> Hortonworks - Develops, Distributes and Supports Enterprise Apache
> Hadoop.
> >>
> >> Hortonworks Virtual Sandbox
> >>
> >> Hadoop: Disruptive Possibilities by Jeff Needham
> >>
> >> CONFIDENTIALITY NOTICE
> >> NOTICE: This message is intended for the use of the individual or entity
> >> to which it is addressed and may contain information that is
> confidential,
> >> privileged and exempt from disclosure under applicable law. If the
> reader of
> >> this message is not the intended recipient, you are hereby notified
> that any
> >> printing, copying, dissemination, distribution, disclosure or
> forwarding of
> >> this communication is strictly prohibited. If you have received this
> >> communication in error, please contact the sender immediately and
> delete it
> >> from your system. Thank You.
> >
> >
>
>
>
> --
> Harsh J
>
Re: Hadoop on IPv6
Posted by Aji Janis <aj...@gmail.com>.
Harsh,
We have the config params that you mentioned, but to be clear are you
saying that the suggestions Adam made won't work because of these
configurations ?
Thanks,
Aji
On Mon, Sep 9, 2013 at 4:55 PM, Harsh J <ha...@cloudera.com> wrote:
> For Hadoop JVMs, generally speaking, need to have the following in
> your hadoop-env.sh for HADOOP_OPTS:
>
> export HADOOP_OPTS="-Djava.net.preferIPv4Stack=true $HADOOP_OPTS"
>
> We also ship with this pre-added in the default configs of more recent
> releases.
>
> You may also need these on other JVM based daemons' opts, such as
> HBASE_OPTS/etc. depending on what all you run.
>
> On Tue, Sep 10, 2013 at 2:05 AM, Aji Janis <aj...@gmail.com> wrote:
> > Adam,
> > Thank you for the suggestion. Could you point me to any documentation
> that
> > can help with the configuration changes I need to make? Thanks.
> >
> > Aji
> >
> >
> > On Mon, Sep 9, 2013 at 1:30 PM, Adam Muise <am...@hortonworks.com>
> wrote:
> >>
> >> Keep the Hadoop cluster and it's network backbone on IPv4. It should be
> >> isolated from the rest of your network for security and performance
> reasons
> >> already anyway.
> >>
> >> Enable IPv6 on your gateway nodes and let clients jump in for access via
> >> the ipv6 network and use the cluster via ipv4 on the gateway node. You
> could
> >> even setup a web proxy for some of the required sites using nginx or
> your
> >> favorite proxy server. Leave the ipv6 conversion of the Hadoop cluster
> until
> >> it is fully supported and well-tested.
> >>
> >>
> >> On Mon, Sep 9, 2013 at 1:19 PM, Aji Janis <aj...@gmail.com> wrote:
> >>>
> >>> We are moving to a IPv6 network. I know Hadoop installation says to
> >>> disable IPv6. Does anyone have any suggestions for work arround?
> Currently,
> >>> using Cloudera version 0.20.2-cdh3u5. Thanks in advance.
> >>
> >>
> >>
> >>
> >> --
> >>
> >>
> >> Adam Muise
> >> Solution Engineer
> >> Hortonworks
> >> amuise@hortonworks.com
> >> 416-417-4037
> >>
> >> Hortonworks - Develops, Distributes and Supports Enterprise Apache
> Hadoop.
> >>
> >> Hortonworks Virtual Sandbox
> >>
> >> Hadoop: Disruptive Possibilities by Jeff Needham
> >>
> >> CONFIDENTIALITY NOTICE
> >> NOTICE: This message is intended for the use of the individual or entity
> >> to which it is addressed and may contain information that is
> confidential,
> >> privileged and exempt from disclosure under applicable law. If the
> reader of
> >> this message is not the intended recipient, you are hereby notified
> that any
> >> printing, copying, dissemination, distribution, disclosure or
> forwarding of
> >> this communication is strictly prohibited. If you have received this
> >> communication in error, please contact the sender immediately and
> delete it
> >> from your system. Thank You.
> >
> >
>
>
>
> --
> Harsh J
>
Re: Hadoop on IPv6
Posted by Aji Janis <aj...@gmail.com>.
Harsh,
We have the config params that you mentioned, but to be clear are you
saying that the suggestions Adam made won't work because of these
configurations ?
Thanks,
Aji
On Mon, Sep 9, 2013 at 4:55 PM, Harsh J <ha...@cloudera.com> wrote:
> For Hadoop JVMs, generally speaking, need to have the following in
> your hadoop-env.sh for HADOOP_OPTS:
>
> export HADOOP_OPTS="-Djava.net.preferIPv4Stack=true $HADOOP_OPTS"
>
> We also ship with this pre-added in the default configs of more recent
> releases.
>
> You may also need these on other JVM based daemons' opts, such as
> HBASE_OPTS/etc. depending on what all you run.
>
> On Tue, Sep 10, 2013 at 2:05 AM, Aji Janis <aj...@gmail.com> wrote:
> > Adam,
> > Thank you for the suggestion. Could you point me to any documentation
> that
> > can help with the configuration changes I need to make? Thanks.
> >
> > Aji
> >
> >
> > On Mon, Sep 9, 2013 at 1:30 PM, Adam Muise <am...@hortonworks.com>
> wrote:
> >>
> >> Keep the Hadoop cluster and it's network backbone on IPv4. It should be
> >> isolated from the rest of your network for security and performance
> reasons
> >> already anyway.
> >>
> >> Enable IPv6 on your gateway nodes and let clients jump in for access via
> >> the ipv6 network and use the cluster via ipv4 on the gateway node. You
> could
> >> even setup a web proxy for some of the required sites using nginx or
> your
> >> favorite proxy server. Leave the ipv6 conversion of the Hadoop cluster
> until
> >> it is fully supported and well-tested.
> >>
> >>
> >> On Mon, Sep 9, 2013 at 1:19 PM, Aji Janis <aj...@gmail.com> wrote:
> >>>
> >>> We are moving to a IPv6 network. I know Hadoop installation says to
> >>> disable IPv6. Does anyone have any suggestions for work arround?
> Currently,
> >>> using Cloudera version 0.20.2-cdh3u5. Thanks in advance.
> >>
> >>
> >>
> >>
> >> --
> >>
> >>
> >> Adam Muise
> >> Solution Engineer
> >> Hortonworks
> >> amuise@hortonworks.com
> >> 416-417-4037
> >>
> >> Hortonworks - Develops, Distributes and Supports Enterprise Apache
> Hadoop.
> >>
> >> Hortonworks Virtual Sandbox
> >>
> >> Hadoop: Disruptive Possibilities by Jeff Needham
> >>
> >> CONFIDENTIALITY NOTICE
> >> NOTICE: This message is intended for the use of the individual or entity
> >> to which it is addressed and may contain information that is
> confidential,
> >> privileged and exempt from disclosure under applicable law. If the
> reader of
> >> this message is not the intended recipient, you are hereby notified
> that any
> >> printing, copying, dissemination, distribution, disclosure or
> forwarding of
> >> this communication is strictly prohibited. If you have received this
> >> communication in error, please contact the sender immediately and
> delete it
> >> from your system. Thank You.
> >
> >
>
>
>
> --
> Harsh J
>
Re: Hadoop on IPv6
Posted by Harsh J <ha...@cloudera.com>.
For Hadoop JVMs, generally speaking, need to have the following in
your hadoop-env.sh for HADOOP_OPTS:
export HADOOP_OPTS="-Djava.net.preferIPv4Stack=true $HADOOP_OPTS"
We also ship with this pre-added in the default configs of more recent releases.
You may also need these on other JVM based daemons' opts, such as
HBASE_OPTS/etc. depending on what all you run.
On Tue, Sep 10, 2013 at 2:05 AM, Aji Janis <aj...@gmail.com> wrote:
> Adam,
> Thank you for the suggestion. Could you point me to any documentation that
> can help with the configuration changes I need to make? Thanks.
>
> Aji
>
>
> On Mon, Sep 9, 2013 at 1:30 PM, Adam Muise <am...@hortonworks.com> wrote:
>>
>> Keep the Hadoop cluster and it's network backbone on IPv4. It should be
>> isolated from the rest of your network for security and performance reasons
>> already anyway.
>>
>> Enable IPv6 on your gateway nodes and let clients jump in for access via
>> the ipv6 network and use the cluster via ipv4 on the gateway node. You could
>> even setup a web proxy for some of the required sites using nginx or your
>> favorite proxy server. Leave the ipv6 conversion of the Hadoop cluster until
>> it is fully supported and well-tested.
>>
>>
>> On Mon, Sep 9, 2013 at 1:19 PM, Aji Janis <aj...@gmail.com> wrote:
>>>
>>> We are moving to a IPv6 network. I know Hadoop installation says to
>>> disable IPv6. Does anyone have any suggestions for work arround? Currently,
>>> using Cloudera version 0.20.2-cdh3u5. Thanks in advance.
>>
>>
>>
>>
>> --
>>
>>
>> Adam Muise
>> Solution Engineer
>> Hortonworks
>> amuise@hortonworks.com
>> 416-417-4037
>>
>> Hortonworks - Develops, Distributes and Supports Enterprise Apache Hadoop.
>>
>> Hortonworks Virtual Sandbox
>>
>> Hadoop: Disruptive Possibilities by Jeff Needham
>>
>> CONFIDENTIALITY NOTICE
>> NOTICE: This message is intended for the use of the individual or entity
>> to which it is addressed and may contain information that is confidential,
>> privileged and exempt from disclosure under applicable law. If the reader of
>> this message is not the intended recipient, you are hereby notified that any
>> printing, copying, dissemination, distribution, disclosure or forwarding of
>> this communication is strictly prohibited. If you have received this
>> communication in error, please contact the sender immediately and delete it
>> from your system. Thank You.
>
>
--
Harsh J
Re: Hadoop on IPv6
Posted by Harsh J <ha...@cloudera.com>.
For Hadoop JVMs, generally speaking, need to have the following in
your hadoop-env.sh for HADOOP_OPTS:
export HADOOP_OPTS="-Djava.net.preferIPv4Stack=true $HADOOP_OPTS"
We also ship with this pre-added in the default configs of more recent releases.
You may also need these on other JVM based daemons' opts, such as
HBASE_OPTS/etc. depending on what all you run.
On Tue, Sep 10, 2013 at 2:05 AM, Aji Janis <aj...@gmail.com> wrote:
> Adam,
> Thank you for the suggestion. Could you point me to any documentation that
> can help with the configuration changes I need to make? Thanks.
>
> Aji
>
>
> On Mon, Sep 9, 2013 at 1:30 PM, Adam Muise <am...@hortonworks.com> wrote:
>>
>> Keep the Hadoop cluster and it's network backbone on IPv4. It should be
>> isolated from the rest of your network for security and performance reasons
>> already anyway.
>>
>> Enable IPv6 on your gateway nodes and let clients jump in for access via
>> the ipv6 network and use the cluster via ipv4 on the gateway node. You could
>> even setup a web proxy for some of the required sites using nginx or your
>> favorite proxy server. Leave the ipv6 conversion of the Hadoop cluster until
>> it is fully supported and well-tested.
>>
>>
>> On Mon, Sep 9, 2013 at 1:19 PM, Aji Janis <aj...@gmail.com> wrote:
>>>
>>> We are moving to a IPv6 network. I know Hadoop installation says to
>>> disable IPv6. Does anyone have any suggestions for work arround? Currently,
>>> using Cloudera version 0.20.2-cdh3u5. Thanks in advance.
>>
>>
>>
>>
>> --
>>
>>
>> Adam Muise
>> Solution Engineer
>> Hortonworks
>> amuise@hortonworks.com
>> 416-417-4037
>>
>> Hortonworks - Develops, Distributes and Supports Enterprise Apache Hadoop.
>>
>> Hortonworks Virtual Sandbox
>>
>> Hadoop: Disruptive Possibilities by Jeff Needham
>>
>> CONFIDENTIALITY NOTICE
>> NOTICE: This message is intended for the use of the individual or entity
>> to which it is addressed and may contain information that is confidential,
>> privileged and exempt from disclosure under applicable law. If the reader of
>> this message is not the intended recipient, you are hereby notified that any
>> printing, copying, dissemination, distribution, disclosure or forwarding of
>> this communication is strictly prohibited. If you have received this
>> communication in error, please contact the sender immediately and delete it
>> from your system. Thank You.
>
>
--
Harsh J
Re: Hadoop on IPv6
Posted by Adam Muise <am...@hortonworks.com>.
I suggest you go to the mighty Google. This is not a Hadoop configuration
question, it's a web server / proxy configuration.
Starters:
http://www.progob.nl/robmaaseu/index.php/an-easy-way-to-get-ipv6-ready-on-the-outside-ipv6-to-ipv4-proxy/
and proxying with nginx
http://nginx.org/en/docs/http/ngx_http_proxy_module.html
Eventually Knox will help out by adding a REST/Web set of gateways for
Hadoop services:
http://knox.incubator.apache.org
For now, your web based gateway services on the cluster can be proxied and
you can still land on your gateway box with ipv6 capabilities on the
outside of the secure cluster. That gateway box would have direct ipv4
connections to the cluster services.
Cheers,
On Mon, Sep 9, 2013 at 4:35 PM, Aji Janis <aj...@gmail.com> wrote:
> Adam,
> Thank you for the suggestion. Could you point me to any documentation that
> can help with the configuration changes I need to make? Thanks.
>
> Aji
>
>
> On Mon, Sep 9, 2013 at 1:30 PM, Adam Muise <am...@hortonworks.com> wrote:
>
>> Keep the Hadoop cluster and it's network backbone on IPv4. It should be
>> isolated from the rest of your network for security and performance reasons
>> already anyway.
>>
>> Enable IPv6 on your gateway nodes and let clients jump in for access via
>> the ipv6 network and use the cluster via ipv4 on the gateway node. You
>> could even setup a web proxy for some of the required sites using nginx or
>> your favorite proxy server. Leave the ipv6 conversion of the Hadoop cluster
>> until it is fully supported and well-tested.
>>
>>
>> On Mon, Sep 9, 2013 at 1:19 PM, Aji Janis <aj...@gmail.com> wrote:
>>
>>> We are moving to a IPv6 network. I know Hadoop installation says to
>>> disable IPv6. Does anyone have any suggestions for work arround? Currently,
>>> using Cloudera version 0.20.2-cdh3u5. Thanks in advance.
>>>
>>
>>
>>
>> --
>> *
>> *
>> *
>> *
>> *Adam Muise*
>> Solution Engineer
>> *Hortonworks*
>> amuise@hortonworks.com
>> 416-417-4037
>>
>> Hortonworks - Develops, Distributes and Supports Enterprise Apache Hadoop.<http://hortonworks.com/>
>>
>> Hortonworks Virtual Sandbox <http://hortonworks.com/sandbox>
>>
>> Hadoop: Disruptive Possibilities by Jeff Needham<http://hortonworks.com/resources/?did=72&cat=1>
>>
>> CONFIDENTIALITY NOTICE
>> NOTICE: This message is intended for the use of the individual or entity
>> to which it is addressed and may contain information that is confidential,
>> privileged and exempt from disclosure under applicable law. If the reader
>> of this message is not the intended recipient, you are hereby notified that
>> any printing, copying, dissemination, distribution, disclosure or
>> forwarding of this communication is strictly prohibited. If you have
>> received this communication in error, please contact the sender immediately
>> and delete it from your system. Thank You.
>
>
>
--
*
*
*
*
*Adam Muise*
Solution Engineer
*Hortonworks*
amuise@hortonworks.com
416-417-4037
Hortonworks - Develops, Distributes and Supports Enterprise Apache
Hadoop.<http://hortonworks.com/>
Hortonworks Virtual Sandbox <http://hortonworks.com/sandbox>
Hadoop: Disruptive Possibilities by Jeff
Needham<http://hortonworks.com/resources/?did=72&cat=1>
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: Hadoop on IPv6
Posted by Harsh J <ha...@cloudera.com>.
For Hadoop JVMs, generally speaking, need to have the following in
your hadoop-env.sh for HADOOP_OPTS:
export HADOOP_OPTS="-Djava.net.preferIPv4Stack=true $HADOOP_OPTS"
We also ship with this pre-added in the default configs of more recent releases.
You may also need these on other JVM based daemons' opts, such as
HBASE_OPTS/etc. depending on what all you run.
On Tue, Sep 10, 2013 at 2:05 AM, Aji Janis <aj...@gmail.com> wrote:
> Adam,
> Thank you for the suggestion. Could you point me to any documentation that
> can help with the configuration changes I need to make? Thanks.
>
> Aji
>
>
> On Mon, Sep 9, 2013 at 1:30 PM, Adam Muise <am...@hortonworks.com> wrote:
>>
>> Keep the Hadoop cluster and it's network backbone on IPv4. It should be
>> isolated from the rest of your network for security and performance reasons
>> already anyway.
>>
>> Enable IPv6 on your gateway nodes and let clients jump in for access via
>> the ipv6 network and use the cluster via ipv4 on the gateway node. You could
>> even setup a web proxy for some of the required sites using nginx or your
>> favorite proxy server. Leave the ipv6 conversion of the Hadoop cluster until
>> it is fully supported and well-tested.
>>
>>
>> On Mon, Sep 9, 2013 at 1:19 PM, Aji Janis <aj...@gmail.com> wrote:
>>>
>>> We are moving to a IPv6 network. I know Hadoop installation says to
>>> disable IPv6. Does anyone have any suggestions for work arround? Currently,
>>> using Cloudera version 0.20.2-cdh3u5. Thanks in advance.
>>
>>
>>
>>
>> --
>>
>>
>> Adam Muise
>> Solution Engineer
>> Hortonworks
>> amuise@hortonworks.com
>> 416-417-4037
>>
>> Hortonworks - Develops, Distributes and Supports Enterprise Apache Hadoop.
>>
>> Hortonworks Virtual Sandbox
>>
>> Hadoop: Disruptive Possibilities by Jeff Needham
>>
>> CONFIDENTIALITY NOTICE
>> NOTICE: This message is intended for the use of the individual or entity
>> to which it is addressed and may contain information that is confidential,
>> privileged and exempt from disclosure under applicable law. If the reader of
>> this message is not the intended recipient, you are hereby notified that any
>> printing, copying, dissemination, distribution, disclosure or forwarding of
>> this communication is strictly prohibited. If you have received this
>> communication in error, please contact the sender immediately and delete it
>> from your system. Thank You.
>
>
--
Harsh J
Re: Hadoop on IPv6
Posted by Adam Muise <am...@hortonworks.com>.
I suggest you go to the mighty Google. This is not a Hadoop configuration
question, it's a web server / proxy configuration.
Starters:
http://www.progob.nl/robmaaseu/index.php/an-easy-way-to-get-ipv6-ready-on-the-outside-ipv6-to-ipv4-proxy/
and proxying with nginx
http://nginx.org/en/docs/http/ngx_http_proxy_module.html
Eventually Knox will help out by adding a REST/Web set of gateways for
Hadoop services:
http://knox.incubator.apache.org
For now, your web based gateway services on the cluster can be proxied and
you can still land on your gateway box with ipv6 capabilities on the
outside of the secure cluster. That gateway box would have direct ipv4
connections to the cluster services.
Cheers,
On Mon, Sep 9, 2013 at 4:35 PM, Aji Janis <aj...@gmail.com> wrote:
> Adam,
> Thank you for the suggestion. Could you point me to any documentation that
> can help with the configuration changes I need to make? Thanks.
>
> Aji
>
>
> On Mon, Sep 9, 2013 at 1:30 PM, Adam Muise <am...@hortonworks.com> wrote:
>
>> Keep the Hadoop cluster and it's network backbone on IPv4. It should be
>> isolated from the rest of your network for security and performance reasons
>> already anyway.
>>
>> Enable IPv6 on your gateway nodes and let clients jump in for access via
>> the ipv6 network and use the cluster via ipv4 on the gateway node. You
>> could even setup a web proxy for some of the required sites using nginx or
>> your favorite proxy server. Leave the ipv6 conversion of the Hadoop cluster
>> until it is fully supported and well-tested.
>>
>>
>> On Mon, Sep 9, 2013 at 1:19 PM, Aji Janis <aj...@gmail.com> wrote:
>>
>>> We are moving to a IPv6 network. I know Hadoop installation says to
>>> disable IPv6. Does anyone have any suggestions for work arround? Currently,
>>> using Cloudera version 0.20.2-cdh3u5. Thanks in advance.
>>>
>>
>>
>>
>> --
>> *
>> *
>> *
>> *
>> *Adam Muise*
>> Solution Engineer
>> *Hortonworks*
>> amuise@hortonworks.com
>> 416-417-4037
>>
>> Hortonworks - Develops, Distributes and Supports Enterprise Apache Hadoop.<http://hortonworks.com/>
>>
>> Hortonworks Virtual Sandbox <http://hortonworks.com/sandbox>
>>
>> Hadoop: Disruptive Possibilities by Jeff Needham<http://hortonworks.com/resources/?did=72&cat=1>
>>
>> CONFIDENTIALITY NOTICE
>> NOTICE: This message is intended for the use of the individual or entity
>> to which it is addressed and may contain information that is confidential,
>> privileged and exempt from disclosure under applicable law. If the reader
>> of this message is not the intended recipient, you are hereby notified that
>> any printing, copying, dissemination, distribution, disclosure or
>> forwarding of this communication is strictly prohibited. If you have
>> received this communication in error, please contact the sender immediately
>> and delete it from your system. Thank You.
>
>
>
--
*
*
*
*
*Adam Muise*
Solution Engineer
*Hortonworks*
amuise@hortonworks.com
416-417-4037
Hortonworks - Develops, Distributes and Supports Enterprise Apache
Hadoop.<http://hortonworks.com/>
Hortonworks Virtual Sandbox <http://hortonworks.com/sandbox>
Hadoop: Disruptive Possibilities by Jeff
Needham<http://hortonworks.com/resources/?did=72&cat=1>
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: Hadoop on IPv6
Posted by Harsh J <ha...@cloudera.com>.
For Hadoop JVMs, generally speaking, need to have the following in
your hadoop-env.sh for HADOOP_OPTS:
export HADOOP_OPTS="-Djava.net.preferIPv4Stack=true $HADOOP_OPTS"
We also ship with this pre-added in the default configs of more recent releases.
You may also need these on other JVM based daemons' opts, such as
HBASE_OPTS/etc. depending on what all you run.
On Tue, Sep 10, 2013 at 2:05 AM, Aji Janis <aj...@gmail.com> wrote:
> Adam,
> Thank you for the suggestion. Could you point me to any documentation that
> can help with the configuration changes I need to make? Thanks.
>
> Aji
>
>
> On Mon, Sep 9, 2013 at 1:30 PM, Adam Muise <am...@hortonworks.com> wrote:
>>
>> Keep the Hadoop cluster and it's network backbone on IPv4. It should be
>> isolated from the rest of your network for security and performance reasons
>> already anyway.
>>
>> Enable IPv6 on your gateway nodes and let clients jump in for access via
>> the ipv6 network and use the cluster via ipv4 on the gateway node. You could
>> even setup a web proxy for some of the required sites using nginx or your
>> favorite proxy server. Leave the ipv6 conversion of the Hadoop cluster until
>> it is fully supported and well-tested.
>>
>>
>> On Mon, Sep 9, 2013 at 1:19 PM, Aji Janis <aj...@gmail.com> wrote:
>>>
>>> We are moving to a IPv6 network. I know Hadoop installation says to
>>> disable IPv6. Does anyone have any suggestions for work arround? Currently,
>>> using Cloudera version 0.20.2-cdh3u5. Thanks in advance.
>>
>>
>>
>>
>> --
>>
>>
>> Adam Muise
>> Solution Engineer
>> Hortonworks
>> amuise@hortonworks.com
>> 416-417-4037
>>
>> Hortonworks - Develops, Distributes and Supports Enterprise Apache Hadoop.
>>
>> Hortonworks Virtual Sandbox
>>
>> Hadoop: Disruptive Possibilities by Jeff Needham
>>
>> CONFIDENTIALITY NOTICE
>> NOTICE: This message is intended for the use of the individual or entity
>> to which it is addressed and may contain information that is confidential,
>> privileged and exempt from disclosure under applicable law. If the reader of
>> this message is not the intended recipient, you are hereby notified that any
>> printing, copying, dissemination, distribution, disclosure or forwarding of
>> this communication is strictly prohibited. If you have received this
>> communication in error, please contact the sender immediately and delete it
>> from your system. Thank You.
>
>
--
Harsh J
Re: Hadoop on IPv6
Posted by Adam Muise <am...@hortonworks.com>.
I suggest you go to the mighty Google. This is not a Hadoop configuration
question, it's a web server / proxy configuration.
Starters:
http://www.progob.nl/robmaaseu/index.php/an-easy-way-to-get-ipv6-ready-on-the-outside-ipv6-to-ipv4-proxy/
and proxying with nginx
http://nginx.org/en/docs/http/ngx_http_proxy_module.html
Eventually Knox will help out by adding a REST/Web set of gateways for
Hadoop services:
http://knox.incubator.apache.org
For now, your web based gateway services on the cluster can be proxied and
you can still land on your gateway box with ipv6 capabilities on the
outside of the secure cluster. That gateway box would have direct ipv4
connections to the cluster services.
Cheers,
On Mon, Sep 9, 2013 at 4:35 PM, Aji Janis <aj...@gmail.com> wrote:
> Adam,
> Thank you for the suggestion. Could you point me to any documentation that
> can help with the configuration changes I need to make? Thanks.
>
> Aji
>
>
> On Mon, Sep 9, 2013 at 1:30 PM, Adam Muise <am...@hortonworks.com> wrote:
>
>> Keep the Hadoop cluster and it's network backbone on IPv4. It should be
>> isolated from the rest of your network for security and performance reasons
>> already anyway.
>>
>> Enable IPv6 on your gateway nodes and let clients jump in for access via
>> the ipv6 network and use the cluster via ipv4 on the gateway node. You
>> could even setup a web proxy for some of the required sites using nginx or
>> your favorite proxy server. Leave the ipv6 conversion of the Hadoop cluster
>> until it is fully supported and well-tested.
>>
>>
>> On Mon, Sep 9, 2013 at 1:19 PM, Aji Janis <aj...@gmail.com> wrote:
>>
>>> We are moving to a IPv6 network. I know Hadoop installation says to
>>> disable IPv6. Does anyone have any suggestions for work arround? Currently,
>>> using Cloudera version 0.20.2-cdh3u5. Thanks in advance.
>>>
>>
>>
>>
>> --
>> *
>> *
>> *
>> *
>> *Adam Muise*
>> Solution Engineer
>> *Hortonworks*
>> amuise@hortonworks.com
>> 416-417-4037
>>
>> Hortonworks - Develops, Distributes and Supports Enterprise Apache Hadoop.<http://hortonworks.com/>
>>
>> Hortonworks Virtual Sandbox <http://hortonworks.com/sandbox>
>>
>> Hadoop: Disruptive Possibilities by Jeff Needham<http://hortonworks.com/resources/?did=72&cat=1>
>>
>> CONFIDENTIALITY NOTICE
>> NOTICE: This message is intended for the use of the individual or entity
>> to which it is addressed and may contain information that is confidential,
>> privileged and exempt from disclosure under applicable law. If the reader
>> of this message is not the intended recipient, you are hereby notified that
>> any printing, copying, dissemination, distribution, disclosure or
>> forwarding of this communication is strictly prohibited. If you have
>> received this communication in error, please contact the sender immediately
>> and delete it from your system. Thank You.
>
>
>
--
*
*
*
*
*Adam Muise*
Solution Engineer
*Hortonworks*
amuise@hortonworks.com
416-417-4037
Hortonworks - Develops, Distributes and Supports Enterprise Apache
Hadoop.<http://hortonworks.com/>
Hortonworks Virtual Sandbox <http://hortonworks.com/sandbox>
Hadoop: Disruptive Possibilities by Jeff
Needham<http://hortonworks.com/resources/?did=72&cat=1>
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: Hadoop on IPv6
Posted by Aji Janis <aj...@gmail.com>.
Adam,
Thank you for the suggestion. Could you point me to any documentation that
can help with the configuration changes I need to make? Thanks.
Aji
On Mon, Sep 9, 2013 at 1:30 PM, Adam Muise <am...@hortonworks.com> wrote:
> Keep the Hadoop cluster and it's network backbone on IPv4. It should be
> isolated from the rest of your network for security and performance reasons
> already anyway.
>
> Enable IPv6 on your gateway nodes and let clients jump in for access via
> the ipv6 network and use the cluster via ipv4 on the gateway node. You
> could even setup a web proxy for some of the required sites using nginx or
> your favorite proxy server. Leave the ipv6 conversion of the Hadoop cluster
> until it is fully supported and well-tested.
>
>
> On Mon, Sep 9, 2013 at 1:19 PM, Aji Janis <aj...@gmail.com> wrote:
>
>> We are moving to a IPv6 network. I know Hadoop installation says to
>> disable IPv6. Does anyone have any suggestions for work arround? Currently,
>> using Cloudera version 0.20.2-cdh3u5. Thanks in advance.
>>
>
>
>
> --
> *
> *
> *
> *
> *Adam Muise*
> Solution Engineer
> *Hortonworks*
> amuise@hortonworks.com
> 416-417-4037
>
> Hortonworks - Develops, Distributes and Supports Enterprise Apache Hadoop.<http://hortonworks.com/>
>
> Hortonworks Virtual Sandbox <http://hortonworks.com/sandbox>
>
> Hadoop: Disruptive Possibilities by Jeff Needham<http://hortonworks.com/resources/?did=72&cat=1>
>
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity
> to which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
Re: Hadoop on IPv6
Posted by Aji Janis <aj...@gmail.com>.
Adam,
Thank you for the suggestion. Could you point me to any documentation that
can help with the configuration changes I need to make? Thanks.
Aji
On Mon, Sep 9, 2013 at 1:30 PM, Adam Muise <am...@hortonworks.com> wrote:
> Keep the Hadoop cluster and it's network backbone on IPv4. It should be
> isolated from the rest of your network for security and performance reasons
> already anyway.
>
> Enable IPv6 on your gateway nodes and let clients jump in for access via
> the ipv6 network and use the cluster via ipv4 on the gateway node. You
> could even setup a web proxy for some of the required sites using nginx or
> your favorite proxy server. Leave the ipv6 conversion of the Hadoop cluster
> until it is fully supported and well-tested.
>
>
> On Mon, Sep 9, 2013 at 1:19 PM, Aji Janis <aj...@gmail.com> wrote:
>
>> We are moving to a IPv6 network. I know Hadoop installation says to
>> disable IPv6. Does anyone have any suggestions for work arround? Currently,
>> using Cloudera version 0.20.2-cdh3u5. Thanks in advance.
>>
>
>
>
> --
> *
> *
> *
> *
> *Adam Muise*
> Solution Engineer
> *Hortonworks*
> amuise@hortonworks.com
> 416-417-4037
>
> Hortonworks - Develops, Distributes and Supports Enterprise Apache Hadoop.<http://hortonworks.com/>
>
> Hortonworks Virtual Sandbox <http://hortonworks.com/sandbox>
>
> Hadoop: Disruptive Possibilities by Jeff Needham<http://hortonworks.com/resources/?did=72&cat=1>
>
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity
> to which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
Re: Hadoop on IPv6
Posted by Aji Janis <aj...@gmail.com>.
Adam,
Thank you for the suggestion. Could you point me to any documentation that
can help with the configuration changes I need to make? Thanks.
Aji
On Mon, Sep 9, 2013 at 1:30 PM, Adam Muise <am...@hortonworks.com> wrote:
> Keep the Hadoop cluster and it's network backbone on IPv4. It should be
> isolated from the rest of your network for security and performance reasons
> already anyway.
>
> Enable IPv6 on your gateway nodes and let clients jump in for access via
> the ipv6 network and use the cluster via ipv4 on the gateway node. You
> could even setup a web proxy for some of the required sites using nginx or
> your favorite proxy server. Leave the ipv6 conversion of the Hadoop cluster
> until it is fully supported and well-tested.
>
>
> On Mon, Sep 9, 2013 at 1:19 PM, Aji Janis <aj...@gmail.com> wrote:
>
>> We are moving to a IPv6 network. I know Hadoop installation says to
>> disable IPv6. Does anyone have any suggestions for work arround? Currently,
>> using Cloudera version 0.20.2-cdh3u5. Thanks in advance.
>>
>
>
>
> --
> *
> *
> *
> *
> *Adam Muise*
> Solution Engineer
> *Hortonworks*
> amuise@hortonworks.com
> 416-417-4037
>
> Hortonworks - Develops, Distributes and Supports Enterprise Apache Hadoop.<http://hortonworks.com/>
>
> Hortonworks Virtual Sandbox <http://hortonworks.com/sandbox>
>
> Hadoop: Disruptive Possibilities by Jeff Needham<http://hortonworks.com/resources/?did=72&cat=1>
>
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity
> to which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
Re: Hadoop on IPv6
Posted by Aji Janis <aj...@gmail.com>.
Adam,
Thank you for the suggestion. Could you point me to any documentation that
can help with the configuration changes I need to make? Thanks.
Aji
On Mon, Sep 9, 2013 at 1:30 PM, Adam Muise <am...@hortonworks.com> wrote:
> Keep the Hadoop cluster and it's network backbone on IPv4. It should be
> isolated from the rest of your network for security and performance reasons
> already anyway.
>
> Enable IPv6 on your gateway nodes and let clients jump in for access via
> the ipv6 network and use the cluster via ipv4 on the gateway node. You
> could even setup a web proxy for some of the required sites using nginx or
> your favorite proxy server. Leave the ipv6 conversion of the Hadoop cluster
> until it is fully supported and well-tested.
>
>
> On Mon, Sep 9, 2013 at 1:19 PM, Aji Janis <aj...@gmail.com> wrote:
>
>> We are moving to a IPv6 network. I know Hadoop installation says to
>> disable IPv6. Does anyone have any suggestions for work arround? Currently,
>> using Cloudera version 0.20.2-cdh3u5. Thanks in advance.
>>
>
>
>
> --
> *
> *
> *
> *
> *Adam Muise*
> Solution Engineer
> *Hortonworks*
> amuise@hortonworks.com
> 416-417-4037
>
> Hortonworks - Develops, Distributes and Supports Enterprise Apache Hadoop.<http://hortonworks.com/>
>
> Hortonworks Virtual Sandbox <http://hortonworks.com/sandbox>
>
> Hadoop: Disruptive Possibilities by Jeff Needham<http://hortonworks.com/resources/?did=72&cat=1>
>
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity
> to which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
Re: Hadoop on IPv6
Posted by Adam Muise <am...@hortonworks.com>.
Keep the Hadoop cluster and it's network backbone on IPv4. It should be
isolated from the rest of your network for security and performance reasons
already anyway.
Enable IPv6 on your gateway nodes and let clients jump in for access via
the ipv6 network and use the cluster via ipv4 on the gateway node. You
could even setup a web proxy for some of the required sites using nginx or
your favorite proxy server. Leave the ipv6 conversion of the Hadoop cluster
until it is fully supported and well-tested.
On Mon, Sep 9, 2013 at 1:19 PM, Aji Janis <aj...@gmail.com> wrote:
> We are moving to a IPv6 network. I know Hadoop installation says to
> disable IPv6. Does anyone have any suggestions for work arround? Currently,
> using Cloudera version 0.20.2-cdh3u5. Thanks in advance.
>
--
*
*
*
*
*Adam Muise*
Solution Engineer
*Hortonworks*
amuise@hortonworks.com
416-417-4037
Hortonworks - Develops, Distributes and Supports Enterprise Apache
Hadoop.<http://hortonworks.com/>
Hortonworks Virtual Sandbox <http://hortonworks.com/sandbox>
Hadoop: Disruptive Possibilities by Jeff
Needham<http://hortonworks.com/resources/?did=72&cat=1>
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: Hadoop on IPv6
Posted by Adam Muise <am...@hortonworks.com>.
Keep the Hadoop cluster and it's network backbone on IPv4. It should be
isolated from the rest of your network for security and performance reasons
already anyway.
Enable IPv6 on your gateway nodes and let clients jump in for access via
the ipv6 network and use the cluster via ipv4 on the gateway node. You
could even setup a web proxy for some of the required sites using nginx or
your favorite proxy server. Leave the ipv6 conversion of the Hadoop cluster
until it is fully supported and well-tested.
On Mon, Sep 9, 2013 at 1:19 PM, Aji Janis <aj...@gmail.com> wrote:
> We are moving to a IPv6 network. I know Hadoop installation says to
> disable IPv6. Does anyone have any suggestions for work arround? Currently,
> using Cloudera version 0.20.2-cdh3u5. Thanks in advance.
>
--
*
*
*
*
*Adam Muise*
Solution Engineer
*Hortonworks*
amuise@hortonworks.com
416-417-4037
Hortonworks - Develops, Distributes and Supports Enterprise Apache
Hadoop.<http://hortonworks.com/>
Hortonworks Virtual Sandbox <http://hortonworks.com/sandbox>
Hadoop: Disruptive Possibilities by Jeff
Needham<http://hortonworks.com/resources/?did=72&cat=1>
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: Hadoop on IPv6
Posted by Adam Muise <am...@hortonworks.com>.
Keep the Hadoop cluster and it's network backbone on IPv4. It should be
isolated from the rest of your network for security and performance reasons
already anyway.
Enable IPv6 on your gateway nodes and let clients jump in for access via
the ipv6 network and use the cluster via ipv4 on the gateway node. You
could even setup a web proxy for some of the required sites using nginx or
your favorite proxy server. Leave the ipv6 conversion of the Hadoop cluster
until it is fully supported and well-tested.
On Mon, Sep 9, 2013 at 1:19 PM, Aji Janis <aj...@gmail.com> wrote:
> We are moving to a IPv6 network. I know Hadoop installation says to
> disable IPv6. Does anyone have any suggestions for work arround? Currently,
> using Cloudera version 0.20.2-cdh3u5. Thanks in advance.
>
--
*
*
*
*
*Adam Muise*
Solution Engineer
*Hortonworks*
amuise@hortonworks.com
416-417-4037
Hortonworks - Develops, Distributes and Supports Enterprise Apache
Hadoop.<http://hortonworks.com/>
Hortonworks Virtual Sandbox <http://hortonworks.com/sandbox>
Hadoop: Disruptive Possibilities by Jeff
Needham<http://hortonworks.com/resources/?did=72&cat=1>
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: Hadoop on IPv6
Posted by Adam Muise <am...@hortonworks.com>.
Keep the Hadoop cluster and it's network backbone on IPv4. It should be
isolated from the rest of your network for security and performance reasons
already anyway.
Enable IPv6 on your gateway nodes and let clients jump in for access via
the ipv6 network and use the cluster via ipv4 on the gateway node. You
could even setup a web proxy for some of the required sites using nginx or
your favorite proxy server. Leave the ipv6 conversion of the Hadoop cluster
until it is fully supported and well-tested.
On Mon, Sep 9, 2013 at 1:19 PM, Aji Janis <aj...@gmail.com> wrote:
> We are moving to a IPv6 network. I know Hadoop installation says to
> disable IPv6. Does anyone have any suggestions for work arround? Currently,
> using Cloudera version 0.20.2-cdh3u5. Thanks in advance.
>
--
*
*
*
*
*Adam Muise*
Solution Engineer
*Hortonworks*
amuise@hortonworks.com
416-417-4037
Hortonworks - Develops, Distributes and Supports Enterprise Apache
Hadoop.<http://hortonworks.com/>
Hortonworks Virtual Sandbox <http://hortonworks.com/sandbox>
Hadoop: Disruptive Possibilities by Jeff
Needham<http://hortonworks.com/resources/?did=72&cat=1>
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.