You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by 小学园PHP <xx...@qq.com> on 2013/03/20 04:24:46 UTC
Put a file to HDFS with WebHDFS ?
When i use the WebHDFS, i wanna put a file to Hadoop.my curl command:
curl -i -X PUT -T /home/levi/3 "http://localhost:50070/webhdfs/v1/levi/3?op=CREATE"
But i got a return:
HTTP/1.1 100 Continue
HTTP/1.1 307 TEMPORARY_REDIRECT
Content-Type: application/octet-stream
Location: http://58.53.211.46:50075/webhdfs/v1/levi/3?op=CREATE&overwrite=false
Content-Length: 0
Server: Jetty(6.1.26)
OK, it failed.
Do somebody know what has happened?
Re: Put a file to HDFS with WebHDFS ?
Posted by Adam Faris <af...@linkedin.com>.
The WebHDFS wiki has bad information regarding curl. According to RFC 2616 Section 10.3.8, a web client shall not automatically follow a 307 redirect unless the response is from a GET or HEAD request. As jetty returns a 307 response for the PUT request against the WebHDFS api, curl does the proper thing and does not automatically follow the redirect. You need to add "-L" to override the default behavior in curl, forcing it to follow the Location header in the HTTP response. Here's an example from a secure grid.
$ curl --negotiate -u : -i -X PUT -T /etc/motd -L 'http://mynamenode.example.com:50070/webhdfs/v1/tmp/motd?op=CREATE'
HTTP/1.1 100 Continue
HTTP/1.1 401 Unauthorized
Content-Type: text/html; charset=utf-8
WWW-Authenticate: Negotiate
Content-Length: 0
Server: Jetty(6.1.26)
HTTP/1.1 100 Continue
HTTP/1.1 307 TEMPORARY_REDIRECT
Content-Type: application/octet-stream
Expires: Thu, 01-Jan-1970 00:00:00 GMT
Set-Cookie: hadoop.auth="xxxxx";Path=/
Location: http://datanode1.example.com:70/webhdfs/v1/tmp/motd?op=CREATE&delegation=xxxxx&overwrite=false
Content-Length: 0
Server: Jetty(6.1.26)
HTTP/1.1 100 Continue
HTTP/1.1 201 Created
Content-Type: application/octet-stream
Location: webhdfs://mynamenode.example.com:50070/tmp/motd
Content-Length: 0
Server: Jetty(6.1.26)
-- Good luck, Adam
On Mar 19, 2013, at 11:19 PM, ken barclay <ke...@att.net> wrote:
> It didn't fail. If you hit the URL it returned, it will do the create.
>
> --- On Wed, 3/20/13, 小学园PHP <xx...@qq.com> wrote:
>
> From: 小学园PHP <xx...@qq.com>
> Subject: Put a file to HDFS with WebHDFS ?
> To: "user" <us...@hadoop.apache.org>
> Date: Wednesday, March 20, 2013, 3:24 AM
>
> When i use the WebHDFS, i wanna put a file to Hadoop.
> my curl command:
> curl -i -X PUT -T /home/levi/3 "http://localhost:50070/webhdfs/v1/levi/3?op=CREATE"
>
> But i got a return:
> HTTP/1.1 100 Continue
>
> HTTP/1.1 307 TEMPORARY_REDIRECT
> Content-Type: application/octet-stream
> Location: http://58.53.211.46:50075/webhdfs/v1/levi/3?op=CREATE&overwrite=false
> Content-Length: 0
> Server: Jetty(6.1.26)
>
> OK, it failed.
> Do somebody know what has happened?
Re: Put a file to HDFS with WebHDFS ?
Posted by Adam Faris <af...@linkedin.com>.
The WebHDFS wiki has bad information regarding curl. According to RFC 2616 Section 10.3.8, a web client shall not automatically follow a 307 redirect unless the response is from a GET or HEAD request. As jetty returns a 307 response for the PUT request against the WebHDFS api, curl does the proper thing and does not automatically follow the redirect. You need to add "-L" to override the default behavior in curl, forcing it to follow the Location header in the HTTP response. Here's an example from a secure grid.
$ curl --negotiate -u : -i -X PUT -T /etc/motd -L 'http://mynamenode.example.com:50070/webhdfs/v1/tmp/motd?op=CREATE'
HTTP/1.1 100 Continue
HTTP/1.1 401 Unauthorized
Content-Type: text/html; charset=utf-8
WWW-Authenticate: Negotiate
Content-Length: 0
Server: Jetty(6.1.26)
HTTP/1.1 100 Continue
HTTP/1.1 307 TEMPORARY_REDIRECT
Content-Type: application/octet-stream
Expires: Thu, 01-Jan-1970 00:00:00 GMT
Set-Cookie: hadoop.auth="xxxxx";Path=/
Location: http://datanode1.example.com:70/webhdfs/v1/tmp/motd?op=CREATE&delegation=xxxxx&overwrite=false
Content-Length: 0
Server: Jetty(6.1.26)
HTTP/1.1 100 Continue
HTTP/1.1 201 Created
Content-Type: application/octet-stream
Location: webhdfs://mynamenode.example.com:50070/tmp/motd
Content-Length: 0
Server: Jetty(6.1.26)
-- Good luck, Adam
On Mar 19, 2013, at 11:19 PM, ken barclay <ke...@att.net> wrote:
> It didn't fail. If you hit the URL it returned, it will do the create.
>
> --- On Wed, 3/20/13, 小学园PHP <xx...@qq.com> wrote:
>
> From: 小学园PHP <xx...@qq.com>
> Subject: Put a file to HDFS with WebHDFS ?
> To: "user" <us...@hadoop.apache.org>
> Date: Wednesday, March 20, 2013, 3:24 AM
>
> When i use the WebHDFS, i wanna put a file to Hadoop.
> my curl command:
> curl -i -X PUT -T /home/levi/3 "http://localhost:50070/webhdfs/v1/levi/3?op=CREATE"
>
> But i got a return:
> HTTP/1.1 100 Continue
>
> HTTP/1.1 307 TEMPORARY_REDIRECT
> Content-Type: application/octet-stream
> Location: http://58.53.211.46:50075/webhdfs/v1/levi/3?op=CREATE&overwrite=false
> Content-Length: 0
> Server: Jetty(6.1.26)
>
> OK, it failed.
> Do somebody know what has happened?
Re: Put a file to HDFS with WebHDFS ?
Posted by Adam Faris <af...@linkedin.com>.
The WebHDFS wiki has bad information regarding curl. According to RFC 2616 Section 10.3.8, a web client shall not automatically follow a 307 redirect unless the response is from a GET or HEAD request. As jetty returns a 307 response for the PUT request against the WebHDFS api, curl does the proper thing and does not automatically follow the redirect. You need to add "-L" to override the default behavior in curl, forcing it to follow the Location header in the HTTP response. Here's an example from a secure grid.
$ curl --negotiate -u : -i -X PUT -T /etc/motd -L 'http://mynamenode.example.com:50070/webhdfs/v1/tmp/motd?op=CREATE'
HTTP/1.1 100 Continue
HTTP/1.1 401 Unauthorized
Content-Type: text/html; charset=utf-8
WWW-Authenticate: Negotiate
Content-Length: 0
Server: Jetty(6.1.26)
HTTP/1.1 100 Continue
HTTP/1.1 307 TEMPORARY_REDIRECT
Content-Type: application/octet-stream
Expires: Thu, 01-Jan-1970 00:00:00 GMT
Set-Cookie: hadoop.auth="xxxxx";Path=/
Location: http://datanode1.example.com:70/webhdfs/v1/tmp/motd?op=CREATE&delegation=xxxxx&overwrite=false
Content-Length: 0
Server: Jetty(6.1.26)
HTTP/1.1 100 Continue
HTTP/1.1 201 Created
Content-Type: application/octet-stream
Location: webhdfs://mynamenode.example.com:50070/tmp/motd
Content-Length: 0
Server: Jetty(6.1.26)
-- Good luck, Adam
On Mar 19, 2013, at 11:19 PM, ken barclay <ke...@att.net> wrote:
> It didn't fail. If you hit the URL it returned, it will do the create.
>
> --- On Wed, 3/20/13, 小学园PHP <xx...@qq.com> wrote:
>
> From: 小学园PHP <xx...@qq.com>
> Subject: Put a file to HDFS with WebHDFS ?
> To: "user" <us...@hadoop.apache.org>
> Date: Wednesday, March 20, 2013, 3:24 AM
>
> When i use the WebHDFS, i wanna put a file to Hadoop.
> my curl command:
> curl -i -X PUT -T /home/levi/3 "http://localhost:50070/webhdfs/v1/levi/3?op=CREATE"
>
> But i got a return:
> HTTP/1.1 100 Continue
>
> HTTP/1.1 307 TEMPORARY_REDIRECT
> Content-Type: application/octet-stream
> Location: http://58.53.211.46:50075/webhdfs/v1/levi/3?op=CREATE&overwrite=false
> Content-Length: 0
> Server: Jetty(6.1.26)
>
> OK, it failed.
> Do somebody know what has happened?
Re: Put a file to HDFS with WebHDFS ?
Posted by Adam Faris <af...@linkedin.com>.
The WebHDFS wiki has bad information regarding curl. According to RFC 2616 Section 10.3.8, a web client shall not automatically follow a 307 redirect unless the response is from a GET or HEAD request. As jetty returns a 307 response for the PUT request against the WebHDFS api, curl does the proper thing and does not automatically follow the redirect. You need to add "-L" to override the default behavior in curl, forcing it to follow the Location header in the HTTP response. Here's an example from a secure grid.
$ curl --negotiate -u : -i -X PUT -T /etc/motd -L 'http://mynamenode.example.com:50070/webhdfs/v1/tmp/motd?op=CREATE'
HTTP/1.1 100 Continue
HTTP/1.1 401 Unauthorized
Content-Type: text/html; charset=utf-8
WWW-Authenticate: Negotiate
Content-Length: 0
Server: Jetty(6.1.26)
HTTP/1.1 100 Continue
HTTP/1.1 307 TEMPORARY_REDIRECT
Content-Type: application/octet-stream
Expires: Thu, 01-Jan-1970 00:00:00 GMT
Set-Cookie: hadoop.auth="xxxxx";Path=/
Location: http://datanode1.example.com:70/webhdfs/v1/tmp/motd?op=CREATE&delegation=xxxxx&overwrite=false
Content-Length: 0
Server: Jetty(6.1.26)
HTTP/1.1 100 Continue
HTTP/1.1 201 Created
Content-Type: application/octet-stream
Location: webhdfs://mynamenode.example.com:50070/tmp/motd
Content-Length: 0
Server: Jetty(6.1.26)
-- Good luck, Adam
On Mar 19, 2013, at 11:19 PM, ken barclay <ke...@att.net> wrote:
> It didn't fail. If you hit the URL it returned, it will do the create.
>
> --- On Wed, 3/20/13, 小学园PHP <xx...@qq.com> wrote:
>
> From: 小学园PHP <xx...@qq.com>
> Subject: Put a file to HDFS with WebHDFS ?
> To: "user" <us...@hadoop.apache.org>
> Date: Wednesday, March 20, 2013, 3:24 AM
>
> When i use the WebHDFS, i wanna put a file to Hadoop.
> my curl command:
> curl -i -X PUT -T /home/levi/3 "http://localhost:50070/webhdfs/v1/levi/3?op=CREATE"
>
> But i got a return:
> HTTP/1.1 100 Continue
>
> HTTP/1.1 307 TEMPORARY_REDIRECT
> Content-Type: application/octet-stream
> Location: http://58.53.211.46:50075/webhdfs/v1/levi/3?op=CREATE&overwrite=false
> Content-Length: 0
> Server: Jetty(6.1.26)
>
> OK, it failed.
> Do somebody know what has happened?
Re: Put a file to HDFS with WebHDFS ?
Posted by ken barclay <ke...@att.net>.
It didn't fail. If you hit the URL it returned, it will do the create.
--- On Wed, 3/20/13, 小学园PHP <xx...@qq.com> wrote:
From: 小学园PHP <xx...@qq.com>
Subject: Put a file to HDFS with WebHDFS ?
To: "user" <us...@hadoop.apache.org>
Date: Wednesday, March 20, 2013, 3:24 AM
When i use the WebHDFS, i wanna put a file to Hadoop.my curl command:curl -i -X PUT -T /home/levi/3 "http://localhost:50070/webhdfs/v1/levi/3?op=CREATE"
But i got a return:HTTP/1.1 100 Continue
HTTP/1.1 307 TEMPORARY_REDIRECTContent-Type: application/octet-streamLocation: http://58.53.211.46:50075/webhdfs/v1/levi/3?op=CREATE&overwrite=falseContent-Length: 0Server: Jetty(6.1.26)
OK, it failed.Do somebody know what has happened?
Re: Put a file to HDFS with WebHDFS ?
Posted by ken barclay <ke...@att.net>.
It didn't fail. If you hit the URL it returned, it will do the create.
--- On Wed, 3/20/13, 小学园PHP <xx...@qq.com> wrote:
From: 小学园PHP <xx...@qq.com>
Subject: Put a file to HDFS with WebHDFS ?
To: "user" <us...@hadoop.apache.org>
Date: Wednesday, March 20, 2013, 3:24 AM
When i use the WebHDFS, i wanna put a file to Hadoop.my curl command:curl -i -X PUT -T /home/levi/3 "http://localhost:50070/webhdfs/v1/levi/3?op=CREATE"
But i got a return:HTTP/1.1 100 Continue
HTTP/1.1 307 TEMPORARY_REDIRECTContent-Type: application/octet-streamLocation: http://58.53.211.46:50075/webhdfs/v1/levi/3?op=CREATE&overwrite=falseContent-Length: 0Server: Jetty(6.1.26)
OK, it failed.Do somebody know what has happened?
Re: Put a file to HDFS with WebHDFS ?
Posted by ken barclay <ke...@att.net>.
It didn't fail. If you hit the URL it returned, it will do the create.
--- On Wed, 3/20/13, 小学园PHP <xx...@qq.com> wrote:
From: 小学园PHP <xx...@qq.com>
Subject: Put a file to HDFS with WebHDFS ?
To: "user" <us...@hadoop.apache.org>
Date: Wednesday, March 20, 2013, 3:24 AM
When i use the WebHDFS, i wanna put a file to Hadoop.my curl command:curl -i -X PUT -T /home/levi/3 "http://localhost:50070/webhdfs/v1/levi/3?op=CREATE"
But i got a return:HTTP/1.1 100 Continue
HTTP/1.1 307 TEMPORARY_REDIRECTContent-Type: application/octet-streamLocation: http://58.53.211.46:50075/webhdfs/v1/levi/3?op=CREATE&overwrite=falseContent-Length: 0Server: Jetty(6.1.26)
OK, it failed.Do somebody know what has happened?
Re: Put a file to HDFS with WebHDFS ?
Posted by ken barclay <ke...@att.net>.
It didn't fail. If you hit the URL it returned, it will do the create.
--- On Wed, 3/20/13, 小学园PHP <xx...@qq.com> wrote:
From: 小学园PHP <xx...@qq.com>
Subject: Put a file to HDFS with WebHDFS ?
To: "user" <us...@hadoop.apache.org>
Date: Wednesday, March 20, 2013, 3:24 AM
When i use the WebHDFS, i wanna put a file to Hadoop.my curl command:curl -i -X PUT -T /home/levi/3 "http://localhost:50070/webhdfs/v1/levi/3?op=CREATE"
But i got a return:HTTP/1.1 100 Continue
HTTP/1.1 307 TEMPORARY_REDIRECTContent-Type: application/octet-streamLocation: http://58.53.211.46:50075/webhdfs/v1/levi/3?op=CREATE&overwrite=falseContent-Length: 0Server: Jetty(6.1.26)
OK, it failed.Do somebody know what has happened?