You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-dev@hadoop.apache.org by Haohui Mai <hm...@hortonworks.com> on 2013/11/26 21:50:36 UTC
Deprecate hftp / hsftp
Hi,
Recently I've been focusing on fixing hftp / hsftp / webhdfs / swebhdfs in
various set ups. Now we have reached the state that all the above file
systems can work in both secure and insecure clusters, and transfer data
through both http and https.
Taking a step back, these file systems are very similar, and I'm wondering
whether it is a good time to deprecate hftp and hsftp in Hadoop right now.
The main reason is that hftp / hsftp only provides a strict subset of
functionalities that webhdfs / swebhdfs offer. Notably, webhdfs / swebhdfs
support writes and HA which hftp / hsftp do not support. It's more natural
to move forward with webhdfs / swebhdfs, keeping both hftp / hsftp around
seems introducing more work.
Another reason is that webhdfs has been supported since Hadoop 1, thus
getting rid of hftp / hsftp does not seem removing any features, even the
users are trying to migrate from Hadoop 1 to Hadoop 2.
Your ideas are appreciated.
Thanks,
Haohui
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: Deprecate hftp / hsftp
Posted by Eli Collins <el...@cloudera.com>.
Hey Haohui,
Seems reasonable to mark hftp and hsftp as deprecated in 2.x and remove
them from trunk (for 3.x).
Thanks,
Eli
On Tuesday, November 26, 2013, Haohui Mai wrote:
> Hi,
>
> Recently I've been focusing on fixing hftp / hsftp / webhdfs / swebhdfs in
> various set ups. Now we have reached the state that all the above file
> systems can work in both secure and insecure clusters, and transfer data
> through both http and https.
>
> Taking a step back, these file systems are very similar, and I'm wondering
> whether it is a good time to deprecate hftp and hsftp in Hadoop right now.
>
> The main reason is that hftp / hsftp only provides a strict subset of
> functionalities that webhdfs / swebhdfs offer. Notably, webhdfs / swebhdfs
> support writes and HA which hftp / hsftp do not support. It's more natural
> to move forward with webhdfs / swebhdfs, keeping both hftp / hsftp around
> seems introducing more work.
>
> Another reason is that webhdfs has been supported since Hadoop 1, thus
> getting rid of hftp / hsftp does not seem removing any features, even the
> users are trying to migrate from Hadoop 1 to Hadoop 2.
>
>
> Your ideas are appreciated.
>
> Thanks,
> Haohui
>
> --
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity to
> which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
>
Re: Deprecate hftp / hsftp
Posted by Haohui Mai <hm...@hortonworks.com>.
Hi Adam,
I agree that having some admission controls in webhdfs would be great, and
you can sort of work around it by keeping both hftp and webhdfs around.
However, I think this is irrelevant to whether we should deprecate hftp /
hsftp.
~Haohui
On Tue, Nov 26, 2013 at 2:06 PM, Adam Faris <af...@linkedin.com> wrote:
> The problem with replacing hftp with WebHDFS is caused by both the
> datanode web interface and the WebHDFS servlet shareing the same jetty
> container. If I want to prevent downloads via webhdfs to corportate
> workstations, I have to firewall the jetty port. But firewalling the jetty
> port on the datanode then breaks the HDFS file viewer. Alternatively I
> could disable WebHDFS but then I no longer am able to transfer data between
> different version hadoop clusters.
>
> I think what's missing from webhdfs is a deny/allow list similar to what
> httpd has. This would allow the operations teams to configure WebHDFS so
> that file transfers only happen between defined networks, and still allow
> the other jetty servlets to be available as we won't have to firewall the
> jetty port.
>
> I should probably go search through open tickets to see if deny/allow list
> has been requested and if not, open a jira. :)
>
>
>
> On Nov 26, 2013, at 1:40 PM, Suresh Srinivas <su...@hortonworks.com>
> wrote:
>
> > Thanks Haohui for all your hard work in this area. I am +1 on this
> proposal.
> >
> >
> > On Tue, Nov 26, 2013 at 12:50 PM, Haohui Mai <hm...@hortonworks.com>
> wrote:
> >
> >> Hi,
> >>
> >> Recently I've been focusing on fixing hftp / hsftp / webhdfs / swebhdfs
> in
> >> various set ups. Now we have reached the state that all the above file
> >> systems can work in both secure and insecure clusters, and transfer data
> >> through both http and https.
> >>
> >> Taking a step back, these file systems are very similar, and I'm
> wondering
> >> whether it is a good time to deprecate hftp and hsftp in Hadoop right
> now.
> >>
> >> The main reason is that hftp / hsftp only provides a strict subset of
> >> functionalities that webhdfs / swebhdfs offer. Notably, webhdfs /
> swebhdfs
> >> support writes and HA which hftp / hsftp do not support. It's more
> natural
> >> to move forward with webhdfs / swebhdfs, keeping both hftp / hsftp
> around
> >> seems introducing more work.
> >>
> >> Another reason is that webhdfs has been supported since Hadoop 1, thus
> >> getting rid of hftp / hsftp does not seem removing any features, even
> the
> >> users are trying to migrate from Hadoop 1 to Hadoop 2.
> >>
> >>
> >> Your ideas are appreciated.
> >>
> >> Thanks,
> >> Haohui
> >>
> >> --
> >> CONFIDENTIALITY NOTICE
> >> NOTICE: This message is intended for the use of the individual or
> entity to
> >> which it is addressed and may contain information that is confidential,
> >> privileged and exempt from disclosure under applicable law. If the
> reader
> >> of this message is not the intended recipient, you are hereby notified
> that
> >> any printing, copying, dissemination, distribution, disclosure or
> >> forwarding of this communication is strictly prohibited. If you have
> >> received this communication in error, please contact the sender
> immediately
> >> and delete it from your system. Thank You.
> >>
> >
> >
> >
> > --
> > http://hortonworks.com/download/
> >
> > --
> > CONFIDENTIALITY NOTICE
> > NOTICE: This message is intended for the use of the individual or entity
> to
> > which it is addressed and may contain information that is confidential,
> > privileged and exempt from disclosure under applicable law. If the reader
> > of this message is not the intended recipient, you are hereby notified
> that
> > any printing, copying, dissemination, distribution, disclosure or
> > forwarding of this communication is strictly prohibited. If you have
> > received this communication in error, please contact the sender
> immediately
> > and delete it from your system. Thank You.
>
>
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: Deprecate hftp / hsftp
Posted by Adam Faris <af...@linkedin.com>.
The problem with replacing hftp with WebHDFS is caused by both the datanode web interface and the WebHDFS servlet shareing the same jetty container. If I want to prevent downloads via webhdfs to corportate workstations, I have to firewall the jetty port. But firewalling the jetty port on the datanode then breaks the HDFS file viewer. Alternatively I could disable WebHDFS but then I no longer am able to transfer data between different version hadoop clusters.
I think what's missing from webhdfs is a deny/allow list similar to what httpd has. This would allow the operations teams to configure WebHDFS so that file transfers only happen between defined networks, and still allow the other jetty servlets to be available as we won't have to firewall the jetty port.
I should probably go search through open tickets to see if deny/allow list has been requested and if not, open a jira. :)
On Nov 26, 2013, at 1:40 PM, Suresh Srinivas <su...@hortonworks.com> wrote:
> Thanks Haohui for all your hard work in this area. I am +1 on this proposal.
>
>
> On Tue, Nov 26, 2013 at 12:50 PM, Haohui Mai <hm...@hortonworks.com> wrote:
>
>> Hi,
>>
>> Recently I've been focusing on fixing hftp / hsftp / webhdfs / swebhdfs in
>> various set ups. Now we have reached the state that all the above file
>> systems can work in both secure and insecure clusters, and transfer data
>> through both http and https.
>>
>> Taking a step back, these file systems are very similar, and I'm wondering
>> whether it is a good time to deprecate hftp and hsftp in Hadoop right now.
>>
>> The main reason is that hftp / hsftp only provides a strict subset of
>> functionalities that webhdfs / swebhdfs offer. Notably, webhdfs / swebhdfs
>> support writes and HA which hftp / hsftp do not support. It's more natural
>> to move forward with webhdfs / swebhdfs, keeping both hftp / hsftp around
>> seems introducing more work.
>>
>> Another reason is that webhdfs has been supported since Hadoop 1, thus
>> getting rid of hftp / hsftp does not seem removing any features, even the
>> users are trying to migrate from Hadoop 1 to Hadoop 2.
>>
>>
>> Your ideas are appreciated.
>>
>> Thanks,
>> Haohui
>>
>> --
>> CONFIDENTIALITY NOTICE
>> NOTICE: This message is intended for the use of the individual or entity to
>> which it is addressed and may contain information that is confidential,
>> privileged and exempt from disclosure under applicable law. If the reader
>> of this message is not the intended recipient, you are hereby notified that
>> any printing, copying, dissemination, distribution, disclosure or
>> forwarding of this communication is strictly prohibited. If you have
>> received this communication in error, please contact the sender immediately
>> and delete it from your system. Thank You.
>>
>
>
>
> --
> http://hortonworks.com/download/
>
> --
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity to
> which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
Re: Deprecate hftp / hsftp
Posted by Haohui Mai <hm...@hortonworks.com>.
Filed HDFS-5570 for removing hftp / hsftp from trunk.
On Tue, Nov 26, 2013 at 6:01 PM, Jing Zhao <ji...@hortonworks.com> wrote:
> +1
>
> So I've reviewed most of Haohui's patch for developing swebhdfs,
> fixing https setup for hsftp, and supporting HA in webhdfs. For both
> webhdfs and swebhdfs we have run system tests on both secured and
> insecure cluster, including running distcp to transfer data between
> 1.x and 2.x. I believe currently webhdfs/swebhdfs is in a very good
> shape.
>
> Also, recently the new web UI in HDFS requires enabling WebHDFS by
> default (HDFS-5532). Thus it is a good time to deprecate HFTP.
>
> On Tue, Nov 26, 2013 at 5:33 PM, Tsz Wo (Nicholas), Sze
> <s2...@yahoo.com> wrote:
> > +1
> >
> > WebHDFS has been around for two years and has become mature. It is a
> rewrite of HFTP and is intended to replace HFTP as stated in my blog [1].
> Now is indeed a good time to deprecate HFTP.
> >
> > Haohui, thanks a lot for adding swebhdfs and fixing the secure setups!
> >
> > Tsz-Wo
> > [1] http://hortonworks.com/blog/webhdfs---http-rest-access-to-hdfs/
> >
> >
> >
> >
> > On Tuesday, November 26, 2013 2:30 PM, Haohui Mai <hm...@hortonworks.com>
> wrote:
> >
> > I meant HDP 1.3, which is based on Hadoop 1.2.
> >>
> >>~Haohui
> >>
> >>
> >>On Tue, Nov 26, 2013 at 2:19 PM, Haohui Mai <hm...@hortonworks.com>
> wrote:
> >>
> >>> Hi Lohit,
> >>>
> >>> Some of our customers are running Hadoop 1.3, and they run distcp to
> >>> transfer data via webhdfs in production.
> >>>
> >>> We use distcp to transfer data between Hadoop 1.x and Hadoop 2.x. As
> far
> >>> as I know webhdfs is the most reliable way to transfer data between
> secure
> >>> and insecure clusters across different Hadoop versions.
> >>>
> >>> ~Haohui
> >>>
> >>>
> >>> On Tue, Nov 26, 2013 at 1:51 PM, lohit <lo...@gmail.com>
> wrote:
> >>>
> >>>> Curious to know usage of webhdfs for Hadoop 1.0. Are people using it
> in
> >>>> production?
> >>>> Reason I ask is because we use lot of hftp to transfer data between
> Hadoop
> >>>> 1.0 and Hadoop 2.0 as of today.
> >>>> Removing hftp would mean that anyone using Hadoop 1.0 and Hadoop 3.0
> would
> >>>> have to move to webhdfs, right?
> >>>>
> >>>>
> >>>>
> >>>> 2013/11/26 Suresh Srinivas <su...@hortonworks.com>
> >>>>
> >>>> > Thanks Haohui for all your hard work in this area. I am +1 on this
> >>>> > proposal.
> >>>> >
> >>>> >
> >>>> > On Tue, Nov 26, 2013 at 12:50 PM, Haohui Mai <hm...@hortonworks.com>
> >>>> wrote:
> >>>> >
> >>>> > > Hi,
> >>>> > >
> >>>> > > Recently I've been focusing on fixing hftp / hsftp / webhdfs /
> >>>> swebhdfs
> >>>> > in
> >>>> > > various set ups. Now we have reached the state that all the above
> file
> >>>> > > systems can work in both secure and insecure clusters, and
> transfer
> >>>> data
> >>>> > > through both http and https.
> >>>> > >
> >>>> > > Taking a step back, these file systems are very similar, and I'm
> >>>> > wondering
> >>>> > > whether it is a good time to deprecate hftp and hsftp in Hadoop
> right
> >>>> > now.
> >>>> > >
> >>>> > > The main reason is that hftp / hsftp only provides a strict
> subset of
> >>>> > > functionalities that webhdfs / swebhdfs offer. Notably, webhdfs /
> >>>> > swebhdfs
> >>>> > > support writes and HA which hftp / hsftp do not support. It's more
> >>>> > natural
> >>>> > > to move forward with webhdfs / swebhdfs, keeping both hftp / hsftp
> >>>> around
> >>>> > > seems introducing more work.
> >>>> > >
> >>>> > > Another reason is that webhdfs has been supported since Hadoop 1,
> thus
> >>>> > > getting rid of hftp / hsftp does not seem removing any features,
> even
> >>>> the
> >>>> > > users are trying to migrate from Hadoop 1 to Hadoop 2.
> >>>> > >
> >>>> > >
> >>>> > > Your ideas are appreciated.
> >>>> > >
> >>>> > > Thanks,
> >>>> > > Haohui
> >>>> > >
> >>>> > > --
> >>>> > > CONFIDENTIALITY NOTICE
> >>>> > > NOTICE: This message is intended for the use of the individual or
> >>>> entity
> >>>> > to
> >>>> > > which it is addressed and may contain information that is
> >>>> confidential,
> >>>> > > privileged and exempt from disclosure under applicable law. If the
> >>>> reader
> >>>> > > of this message is not the intended recipient, you are hereby
> notified
> >>>> > that
> >>>> > > any printing, copying, dissemination, distribution, disclosure or
> >>>> > > forwarding of this communication is strictly prohibited. If you
> have
> >>>> > > received this communication in error, please contact the sender
> >>>> > immediately
> >>>> > > and delete it from your system. Thank You.
> >>>> > >
> >>>> >
> >>>> >
> >>>> >
> >>>> > --
> >>>> > http://hortonworks.com/download/
> >>>> >
> >>>> > --
> >>>> > CONFIDENTIALITY NOTICE
> >>>> > NOTICE: This message is intended for the use of the individual or
> >>>> entity to
> >>>> > which it is addressed and may contain information that is
> confidential,
> >>>> > privileged and exempt from disclosure under applicable law. If the
> >>>> reader
> >>>> > of this message is not the intended recipient, you are hereby
> notified
> >>>> that
> >>>> > any printing, copying, dissemination, distribution, disclosure or
> >>>> > forwarding of this communication is strictly prohibited. If you have
> >>>> > received this communication in error, please contact the sender
> >>>> immediately
> >>>> > and delete it from your system. Thank You.
> >>>> >
> >>>>
> >>>>
> >>>>
> >>>> --
> >>>> Have a Nice Day!
> >>>> Lohit
> >>>>
> >>>
> >>>
> >>
> >>--
> >>CONFIDENTIALITY NOTICE
> >>NOTICE: This message is intended for the use of the individual or entity
> to
> >>which it is addressed and may contain information that is confidential,
> >>privileged and exempt from disclosure under applicable law. If the reader
> >>of this message is not the intended recipient, you are hereby notified
> that
> >>any printing, copying, dissemination, distribution, disclosure or
> >>forwarding of this communication is strictly prohibited. If you have
> >>received this communication in error, please contact the sender
> immediately
> >>and delete it from your system. Thank You.
> >>
> >>
> >>
>
> --
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity to
> which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
>
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: Deprecate hftp / hsftp
Posted by Jing Zhao <ji...@hortonworks.com>.
+1
So I've reviewed most of Haohui's patch for developing swebhdfs,
fixing https setup for hsftp, and supporting HA in webhdfs. For both
webhdfs and swebhdfs we have run system tests on both secured and
insecure cluster, including running distcp to transfer data between
1.x and 2.x. I believe currently webhdfs/swebhdfs is in a very good
shape.
Also, recently the new web UI in HDFS requires enabling WebHDFS by
default (HDFS-5532). Thus it is a good time to deprecate HFTP.
On Tue, Nov 26, 2013 at 5:33 PM, Tsz Wo (Nicholas), Sze
<s2...@yahoo.com> wrote:
> +1
>
> WebHDFS has been around for two years and has become mature. It is a rewrite of HFTP and is intended to replace HFTP as stated in my blog [1]. Now is indeed a good time to deprecate HFTP.
>
> Haohui, thanks a lot for adding swebhdfs and fixing the secure setups!
>
> Tsz-Wo
> [1] http://hortonworks.com/blog/webhdfs-–-http-rest-access-to-hdfs/
>
>
>
>
> On Tuesday, November 26, 2013 2:30 PM, Haohui Mai <hm...@hortonworks.com> wrote:
>
> I meant HDP 1.3, which is based on Hadoop 1.2.
>>
>>~Haohui
>>
>>
>>On Tue, Nov 26, 2013 at 2:19 PM, Haohui Mai <hm...@hortonworks.com> wrote:
>>
>>> Hi Lohit,
>>>
>>> Some of our customers are running Hadoop 1.3, and they run distcp to
>>> transfer data via webhdfs in production.
>>>
>>> We use distcp to transfer data between Hadoop 1.x and Hadoop 2.x. As far
>>> as I know webhdfs is the most reliable way to transfer data between secure
>>> and insecure clusters across different Hadoop versions.
>>>
>>> ~Haohui
>>>
>>>
>>> On Tue, Nov 26, 2013 at 1:51 PM, lohit <lo...@gmail.com> wrote:
>>>
>>>> Curious to know usage of webhdfs for Hadoop 1.0. Are people using it in
>>>> production?
>>>> Reason I ask is because we use lot of hftp to transfer data between Hadoop
>>>> 1.0 and Hadoop 2.0 as of today.
>>>> Removing hftp would mean that anyone using Hadoop 1.0 and Hadoop 3.0 would
>>>> have to move to webhdfs, right?
>>>>
>>>>
>>>>
>>>> 2013/11/26 Suresh Srinivas <su...@hortonworks.com>
>>>>
>>>> > Thanks Haohui for all your hard work in this area. I am +1 on this
>>>> > proposal.
>>>> >
>>>> >
>>>> > On Tue, Nov 26, 2013 at 12:50 PM, Haohui Mai <hm...@hortonworks.com>
>>>> wrote:
>>>> >
>>>> > > Hi,
>>>> > >
>>>> > > Recently I've been focusing on fixing hftp / hsftp / webhdfs /
>>>> swebhdfs
>>>> > in
>>>> > > various set ups. Now we have reached the state that all the above file
>>>> > > systems can work in both secure and insecure clusters, and transfer
>>>> data
>>>> > > through both http and https.
>>>> > >
>>>> > > Taking a step back, these file systems are very similar, and I'm
>>>> > wondering
>>>> > > whether it is a good time to deprecate hftp and hsftp in Hadoop right
>>>> > now.
>>>> > >
>>>> > > The main reason is that hftp / hsftp only provides a strict subset of
>>>> > > functionalities that webhdfs / swebhdfs offer. Notably, webhdfs /
>>>> > swebhdfs
>>>> > > support writes and HA which hftp / hsftp do not support. It's more
>>>> > natural
>>>> > > to move forward with webhdfs / swebhdfs, keeping both hftp / hsftp
>>>> around
>>>> > > seems introducing more work.
>>>> > >
>>>> > > Another reason is that webhdfs has been supported since Hadoop 1, thus
>>>> > > getting rid of hftp / hsftp does not seem removing any features, even
>>>> the
>>>> > > users are trying to migrate from Hadoop 1 to Hadoop 2.
>>>> > >
>>>> > >
>>>> > > Your ideas are appreciated.
>>>> > >
>>>> > > Thanks,
>>>> > > Haohui
>>>> > >
>>>> > > --
>>>> > > CONFIDENTIALITY NOTICE
>>>> > > NOTICE: This message is intended for the use of the individual or
>>>> entity
>>>> > to
>>>> > > which it is addressed and may contain information that is
>>>> confidential,
>>>> > > privileged and exempt from disclosure under applicable law. If the
>>>> reader
>>>> > > of this message is not the intended recipient, you are hereby notified
>>>> > that
>>>> > > any printing, copying, dissemination, distribution, disclosure or
>>>> > > forwarding of this communication is strictly prohibited. If you have
>>>> > > received this communication in error, please contact the sender
>>>> > immediately
>>>> > > and delete it from your system. Thank You.
>>>> > >
>>>> >
>>>> >
>>>> >
>>>> > --
>>>> > http://hortonworks.com/download/
>>>> >
>>>> > --
>>>> > CONFIDENTIALITY NOTICE
>>>> > NOTICE: This message is intended for the use of the individual or
>>>> entity to
>>>> > which it is addressed and may contain information that is confidential,
>>>> > privileged and exempt from disclosure under applicable law. If the
>>>> reader
>>>> > of this message is not the intended recipient, you are hereby notified
>>>> that
>>>> > any printing, copying, dissemination, distribution, disclosure or
>>>> > forwarding of this communication is strictly prohibited. If you have
>>>> > received this communication in error, please contact the sender
>>>> immediately
>>>> > and delete it from your system. Thank You.
>>>> >
>>>>
>>>>
>>>>
>>>> --
>>>> Have a Nice Day!
>>>> Lohit
>>>>
>>>
>>>
>>
>>--
>>CONFIDENTIALITY NOTICE
>>NOTICE: This message is intended for the use of the individual or entity to
>>which it is addressed and may contain information that is confidential,
>>privileged and exempt from disclosure under applicable law. If the reader
>>of this message is not the intended recipient, you are hereby notified that
>>any printing, copying, dissemination, distribution, disclosure or
>>forwarding of this communication is strictly prohibited. If you have
>>received this communication in error, please contact the sender immediately
>>and delete it from your system. Thank You.
>>
>>
>>
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: Deprecate hftp / hsftp
Posted by "Tsz Wo (Nicholas), Sze" <s2...@yahoo.com>.
+1
WebHDFS has been around for two years and has become mature. It is a rewrite of HFTP and is intended to replace HFTP as stated in my blog [1]. Now is indeed a good time to deprecate HFTP.
Haohui, thanks a lot for adding swebhdfs and fixing the secure setups!
Tsz-Wo
[1] http://hortonworks.com/blog/webhdfs-–-http-rest-access-to-hdfs/
On Tuesday, November 26, 2013 2:30 PM, Haohui Mai <hm...@hortonworks.com> wrote:
I meant HDP 1.3, which is based on Hadoop 1.2.
>
>~Haohui
>
>
>On Tue, Nov 26, 2013 at 2:19 PM, Haohui Mai <hm...@hortonworks.com> wrote:
>
>> Hi Lohit,
>>
>> Some of our customers are running Hadoop 1.3, and they run distcp to
>> transfer data via webhdfs in production.
>>
>> We use distcp to transfer data between Hadoop 1.x and Hadoop 2.x. As far
>> as I know webhdfs is the most reliable way to transfer data between secure
>> and insecure clusters across different Hadoop versions.
>>
>> ~Haohui
>>
>>
>> On Tue, Nov 26, 2013 at 1:51 PM, lohit <lo...@gmail.com> wrote:
>>
>>> Curious to know usage of webhdfs for Hadoop 1.0. Are people using it in
>>> production?
>>> Reason I ask is because we use lot of hftp to transfer data between Hadoop
>>> 1.0 and Hadoop 2.0 as of today.
>>> Removing hftp would mean that anyone using Hadoop 1.0 and Hadoop 3.0 would
>>> have to move to webhdfs, right?
>>>
>>>
>>>
>>> 2013/11/26 Suresh Srinivas <su...@hortonworks.com>
>>>
>>> > Thanks Haohui for all your hard work in this area. I am +1 on this
>>> > proposal.
>>> >
>>> >
>>> > On Tue, Nov 26, 2013 at 12:50 PM, Haohui Mai <hm...@hortonworks.com>
>>> wrote:
>>> >
>>> > > Hi,
>>> > >
>>> > > Recently I've been focusing on fixing hftp / hsftp / webhdfs /
>>> swebhdfs
>>> > in
>>> > > various set ups. Now we have reached the state that all the above file
>>> > > systems can work in both secure and insecure clusters, and transfer
>>> data
>>> > > through both http and https.
>>> > >
>>> > > Taking a step back, these file systems are very similar, and I'm
>>> > wondering
>>> > > whether it is a good time to deprecate hftp and hsftp in Hadoop right
>>> > now.
>>> > >
>>> > > The main reason is that hftp / hsftp only provides a strict subset of
>>> > > functionalities that webhdfs / swebhdfs offer. Notably, webhdfs /
>>> > swebhdfs
>>> > > support writes and HA which hftp / hsftp do not support. It's more
>>> > natural
>>> > > to move forward with webhdfs / swebhdfs, keeping both hftp / hsftp
>>> around
>>> > > seems introducing more work.
>>> > >
>>> > > Another reason is that webhdfs has been supported since Hadoop 1, thus
>>> > > getting rid of hftp / hsftp does not seem removing any features, even
>>> the
>>> > > users are trying to migrate from Hadoop 1 to Hadoop 2.
>>> > >
>>> > >
>>> > > Your ideas are appreciated.
>>> > >
>>> > > Thanks,
>>> > > Haohui
>>> > >
>>> > > --
>>> > > CONFIDENTIALITY NOTICE
>>> > > NOTICE: This message is intended for the use of the individual or
>>> entity
>>> > to
>>> > > which it is addressed and may contain information that is
>>> confidential,
>>> > > privileged and exempt from disclosure under applicable law. If the
>>> reader
>>> > > of this message is not the intended recipient, you are hereby notified
>>> > that
>>> > > any printing, copying, dissemination, distribution, disclosure or
>>> > > forwarding of this communication is strictly prohibited. If you have
>>> > > received this communication in error, please contact the sender
>>> > immediately
>>> > > and delete it from your system. Thank You.
>>> > >
>>> >
>>> >
>>> >
>>> > --
>>> > http://hortonworks.com/download/
>>> >
>>> > --
>>> > CONFIDENTIALITY NOTICE
>>> > NOTICE: This message is intended for the use of the individual or
>>> entity to
>>> > which it is addressed and may contain information that is confidential,
>>> > privileged and exempt from disclosure under applicable law. If the
>>> reader
>>> > of this message is not the intended recipient, you are hereby notified
>>> that
>>> > any printing, copying, dissemination, distribution, disclosure or
>>> > forwarding of this communication is strictly prohibited. If you have
>>> > received this communication in error, please contact the sender
>>> immediately
>>> > and delete it from your system. Thank You.
>>> >
>>>
>>>
>>>
>>> --
>>> Have a Nice Day!
>>> Lohit
>>>
>>
>>
>
>--
>CONFIDENTIALITY NOTICE
>NOTICE: This message is intended for the use of the individual or entity to
>which it is addressed and may contain information that is confidential,
>privileged and exempt from disclosure under applicable law. If the reader
>of this message is not the intended recipient, you are hereby notified that
>any printing, copying, dissemination, distribution, disclosure or
>forwarding of this communication is strictly prohibited. If you have
>received this communication in error, please contact the sender immediately
>and delete it from your system. Thank You.
>
>
>
Re: Deprecate hftp / hsftp
Posted by Haohui Mai <hm...@hortonworks.com>.
I meant HDP 1.3, which is based on Hadoop 1.2.
~Haohui
On Tue, Nov 26, 2013 at 2:19 PM, Haohui Mai <hm...@hortonworks.com> wrote:
> Hi Lohit,
>
> Some of our customers are running Hadoop 1.3, and they run distcp to
> transfer data via webhdfs in production.
>
> We use distcp to transfer data between Hadoop 1.x and Hadoop 2.x. As far
> as I know webhdfs is the most reliable way to transfer data between secure
> and insecure clusters across different Hadoop versions.
>
> ~Haohui
>
>
> On Tue, Nov 26, 2013 at 1:51 PM, lohit <lo...@gmail.com> wrote:
>
>> Curious to know usage of webhdfs for Hadoop 1.0. Are people using it in
>> production?
>> Reason I ask is because we use lot of hftp to transfer data between Hadoop
>> 1.0 and Hadoop 2.0 as of today.
>> Removing hftp would mean that anyone using Hadoop 1.0 and Hadoop 3.0 would
>> have to move to webhdfs, right?
>>
>>
>>
>> 2013/11/26 Suresh Srinivas <su...@hortonworks.com>
>>
>> > Thanks Haohui for all your hard work in this area. I am +1 on this
>> > proposal.
>> >
>> >
>> > On Tue, Nov 26, 2013 at 12:50 PM, Haohui Mai <hm...@hortonworks.com>
>> wrote:
>> >
>> > > Hi,
>> > >
>> > > Recently I've been focusing on fixing hftp / hsftp / webhdfs /
>> swebhdfs
>> > in
>> > > various set ups. Now we have reached the state that all the above file
>> > > systems can work in both secure and insecure clusters, and transfer
>> data
>> > > through both http and https.
>> > >
>> > > Taking a step back, these file systems are very similar, and I'm
>> > wondering
>> > > whether it is a good time to deprecate hftp and hsftp in Hadoop right
>> > now.
>> > >
>> > > The main reason is that hftp / hsftp only provides a strict subset of
>> > > functionalities that webhdfs / swebhdfs offer. Notably, webhdfs /
>> > swebhdfs
>> > > support writes and HA which hftp / hsftp do not support. It's more
>> > natural
>> > > to move forward with webhdfs / swebhdfs, keeping both hftp / hsftp
>> around
>> > > seems introducing more work.
>> > >
>> > > Another reason is that webhdfs has been supported since Hadoop 1, thus
>> > > getting rid of hftp / hsftp does not seem removing any features, even
>> the
>> > > users are trying to migrate from Hadoop 1 to Hadoop 2.
>> > >
>> > >
>> > > Your ideas are appreciated.
>> > >
>> > > Thanks,
>> > > Haohui
>> > >
>> > > --
>> > > CONFIDENTIALITY NOTICE
>> > > NOTICE: This message is intended for the use of the individual or
>> entity
>> > to
>> > > which it is addressed and may contain information that is
>> confidential,
>> > > privileged and exempt from disclosure under applicable law. If the
>> reader
>> > > of this message is not the intended recipient, you are hereby notified
>> > that
>> > > any printing, copying, dissemination, distribution, disclosure or
>> > > forwarding of this communication is strictly prohibited. If you have
>> > > received this communication in error, please contact the sender
>> > immediately
>> > > and delete it from your system. Thank You.
>> > >
>> >
>> >
>> >
>> > --
>> > http://hortonworks.com/download/
>> >
>> > --
>> > CONFIDENTIALITY NOTICE
>> > NOTICE: This message is intended for the use of the individual or
>> entity to
>> > which it is addressed and may contain information that is confidential,
>> > privileged and exempt from disclosure under applicable law. If the
>> reader
>> > of this message is not the intended recipient, you are hereby notified
>> that
>> > any printing, copying, dissemination, distribution, disclosure or
>> > forwarding of this communication is strictly prohibited. If you have
>> > received this communication in error, please contact the sender
>> immediately
>> > and delete it from your system. Thank You.
>> >
>>
>>
>>
>> --
>> Have a Nice Day!
>> Lohit
>>
>
>
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: Deprecate hftp / hsftp
Posted by Haohui Mai <hm...@hortonworks.com>.
Hi Lohit,
Some of our customers are running Hadoop 1.3, and they run distcp to
transfer data via webhdfs in production.
We use distcp to transfer data between Hadoop 1.x and Hadoop 2.x. As far as
I know webhdfs is the most reliable way to transfer data between secure and
insecure clusters across different Hadoop versions.
~Haohui
On Tue, Nov 26, 2013 at 1:51 PM, lohit <lo...@gmail.com> wrote:
> Curious to know usage of webhdfs for Hadoop 1.0. Are people using it in
> production?
> Reason I ask is because we use lot of hftp to transfer data between Hadoop
> 1.0 and Hadoop 2.0 as of today.
> Removing hftp would mean that anyone using Hadoop 1.0 and Hadoop 3.0 would
> have to move to webhdfs, right?
>
>
>
> 2013/11/26 Suresh Srinivas <su...@hortonworks.com>
>
> > Thanks Haohui for all your hard work in this area. I am +1 on this
> > proposal.
> >
> >
> > On Tue, Nov 26, 2013 at 12:50 PM, Haohui Mai <hm...@hortonworks.com>
> wrote:
> >
> > > Hi,
> > >
> > > Recently I've been focusing on fixing hftp / hsftp / webhdfs / swebhdfs
> > in
> > > various set ups. Now we have reached the state that all the above file
> > > systems can work in both secure and insecure clusters, and transfer
> data
> > > through both http and https.
> > >
> > > Taking a step back, these file systems are very similar, and I'm
> > wondering
> > > whether it is a good time to deprecate hftp and hsftp in Hadoop right
> > now.
> > >
> > > The main reason is that hftp / hsftp only provides a strict subset of
> > > functionalities that webhdfs / swebhdfs offer. Notably, webhdfs /
> > swebhdfs
> > > support writes and HA which hftp / hsftp do not support. It's more
> > natural
> > > to move forward with webhdfs / swebhdfs, keeping both hftp / hsftp
> around
> > > seems introducing more work.
> > >
> > > Another reason is that webhdfs has been supported since Hadoop 1, thus
> > > getting rid of hftp / hsftp does not seem removing any features, even
> the
> > > users are trying to migrate from Hadoop 1 to Hadoop 2.
> > >
> > >
> > > Your ideas are appreciated.
> > >
> > > Thanks,
> > > Haohui
> > >
> > > --
> > > CONFIDENTIALITY NOTICE
> > > NOTICE: This message is intended for the use of the individual or
> entity
> > to
> > > which it is addressed and may contain information that is confidential,
> > > privileged and exempt from disclosure under applicable law. If the
> reader
> > > of this message is not the intended recipient, you are hereby notified
> > that
> > > any printing, copying, dissemination, distribution, disclosure or
> > > forwarding of this communication is strictly prohibited. If you have
> > > received this communication in error, please contact the sender
> > immediately
> > > and delete it from your system. Thank You.
> > >
> >
> >
> >
> > --
> > http://hortonworks.com/download/
> >
> > --
> > CONFIDENTIALITY NOTICE
> > NOTICE: This message is intended for the use of the individual or entity
> to
> > which it is addressed and may contain information that is confidential,
> > privileged and exempt from disclosure under applicable law. If the reader
> > of this message is not the intended recipient, you are hereby notified
> that
> > any printing, copying, dissemination, distribution, disclosure or
> > forwarding of this communication is strictly prohibited. If you have
> > received this communication in error, please contact the sender
> immediately
> > and delete it from your system. Thank You.
> >
>
>
>
> --
> Have a Nice Day!
> Lohit
>
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: Deprecate hftp / hsftp
Posted by lohit <lo...@gmail.com>.
Curious to know usage of webhdfs for Hadoop 1.0. Are people using it in
production?
Reason I ask is because we use lot of hftp to transfer data between Hadoop
1.0 and Hadoop 2.0 as of today.
Removing hftp would mean that anyone using Hadoop 1.0 and Hadoop 3.0 would
have to move to webhdfs, right?
2013/11/26 Suresh Srinivas <su...@hortonworks.com>
> Thanks Haohui for all your hard work in this area. I am +1 on this
> proposal.
>
>
> On Tue, Nov 26, 2013 at 12:50 PM, Haohui Mai <hm...@hortonworks.com> wrote:
>
> > Hi,
> >
> > Recently I've been focusing on fixing hftp / hsftp / webhdfs / swebhdfs
> in
> > various set ups. Now we have reached the state that all the above file
> > systems can work in both secure and insecure clusters, and transfer data
> > through both http and https.
> >
> > Taking a step back, these file systems are very similar, and I'm
> wondering
> > whether it is a good time to deprecate hftp and hsftp in Hadoop right
> now.
> >
> > The main reason is that hftp / hsftp only provides a strict subset of
> > functionalities that webhdfs / swebhdfs offer. Notably, webhdfs /
> swebhdfs
> > support writes and HA which hftp / hsftp do not support. It's more
> natural
> > to move forward with webhdfs / swebhdfs, keeping both hftp / hsftp around
> > seems introducing more work.
> >
> > Another reason is that webhdfs has been supported since Hadoop 1, thus
> > getting rid of hftp / hsftp does not seem removing any features, even the
> > users are trying to migrate from Hadoop 1 to Hadoop 2.
> >
> >
> > Your ideas are appreciated.
> >
> > Thanks,
> > Haohui
> >
> > --
> > CONFIDENTIALITY NOTICE
> > NOTICE: This message is intended for the use of the individual or entity
> to
> > which it is addressed and may contain information that is confidential,
> > privileged and exempt from disclosure under applicable law. If the reader
> > of this message is not the intended recipient, you are hereby notified
> that
> > any printing, copying, dissemination, distribution, disclosure or
> > forwarding of this communication is strictly prohibited. If you have
> > received this communication in error, please contact the sender
> immediately
> > and delete it from your system. Thank You.
> >
>
>
>
> --
> http://hortonworks.com/download/
>
> --
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity to
> which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
>
--
Have a Nice Day!
Lohit
Re: Deprecate hftp / hsftp
Posted by Suresh Srinivas <su...@hortonworks.com>.
Thanks Haohui for all your hard work in this area. I am +1 on this proposal.
On Tue, Nov 26, 2013 at 12:50 PM, Haohui Mai <hm...@hortonworks.com> wrote:
> Hi,
>
> Recently I've been focusing on fixing hftp / hsftp / webhdfs / swebhdfs in
> various set ups. Now we have reached the state that all the above file
> systems can work in both secure and insecure clusters, and transfer data
> through both http and https.
>
> Taking a step back, these file systems are very similar, and I'm wondering
> whether it is a good time to deprecate hftp and hsftp in Hadoop right now.
>
> The main reason is that hftp / hsftp only provides a strict subset of
> functionalities that webhdfs / swebhdfs offer. Notably, webhdfs / swebhdfs
> support writes and HA which hftp / hsftp do not support. It's more natural
> to move forward with webhdfs / swebhdfs, keeping both hftp / hsftp around
> seems introducing more work.
>
> Another reason is that webhdfs has been supported since Hadoop 1, thus
> getting rid of hftp / hsftp does not seem removing any features, even the
> users are trying to migrate from Hadoop 1 to Hadoop 2.
>
>
> Your ideas are appreciated.
>
> Thanks,
> Haohui
>
> --
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity to
> which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
>
--
http://hortonworks.com/download/
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.