You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by 超级塞亚人 <sh...@gmail.com> on 2013/04/19 06:35:12 UTC
Uploading file to HDFS
I have a problem. Our cluster has 32 nodes. Each disk is 1TB. I wanna
upload 2TB file to HDFS.How can I put the file to the namenode and upload
to HDFS?
Re: Uploading file to HDFS
Posted by Olivier Renault <or...@hortonworks.com>.
You should be able to do it using WebHDFS.
WebHDFS is a REST API so there is no need to have hadoop installed on the
client where the file is located. YOu can find an example on how to copy a
file at the following URL.
http://hadoop.apache.org/docs/r1.0.4/webhdfs.html#CREATE
Olivier
On 19 April 2013 11:01, Wellington Chevreuil <wellington.chevreuil@gmail.com
> wrote:
> Can't you use flume for that?
>
>
> 2013/4/19 David Parks <da...@yahoo.com>
>
>> I just realized another trick you might trying. The Hadoop dfs client can
>> read input from STDIN, you could use netcat to pipe the stuff across to
>> HDFS without hitting the hard drive, I haven’t tried it, but here’s what I
>> would think might work:****
>>
>> ** **
>>
>> On the Hadoop box, open a listening port and feed that to the HDFS
>> command:****
>>
>> nc -l 2342 | hdfs dfs -copyFromLocal - /tmp/x.txt****
>>
>> ** **
>>
>> On the remote server:****
>>
>> cat my_big_2tb_file > nc 10.1.1.1 2342****
>>
>> ** **
>>
>> I haven’t tried it yet, but in theory this would work. I just happened to
>> test out the hdfs dfs command reading from stdin. You might have to correct
>> the above syntax, I just wrote it off the top of my head.****
>>
>> ** **
>>
>> Dave****
>>
>> ** **
>>
>> ** **
>>
>> *From:* 超级塞亚人 [mailto:sheldom@gmail.com]
>> *Sent:* Friday, April 19, 2013 11:35 AM
>> *To:* user@hadoop.apache.org
>> *Subject:* Uploading file to HDFS****
>>
>> ** **
>>
>> I have a problem. Our cluster has 32 nodes. Each disk is 1TB. I wanna
>> upload 2TB file to HDFS.How can I put the file to the namenode and upload
>> to HDFS? ****
>>
>
>
--
Olivier Renault
Solution Engineer - Big Data - Hortonworks, Inc.
+44 7500 933 036
orenault@hortonworks.com
www.hortonworks.com
<http://hortonworks.com/products/hortonworks-sandbox/>
Re: Uploading file to HDFS
Posted by Olivier Renault <or...@hortonworks.com>.
You should be able to do it using WebHDFS.
WebHDFS is a REST API so there is no need to have hadoop installed on the
client where the file is located. YOu can find an example on how to copy a
file at the following URL.
http://hadoop.apache.org/docs/r1.0.4/webhdfs.html#CREATE
Olivier
On 19 April 2013 11:01, Wellington Chevreuil <wellington.chevreuil@gmail.com
> wrote:
> Can't you use flume for that?
>
>
> 2013/4/19 David Parks <da...@yahoo.com>
>
>> I just realized another trick you might trying. The Hadoop dfs client can
>> read input from STDIN, you could use netcat to pipe the stuff across to
>> HDFS without hitting the hard drive, I haven’t tried it, but here’s what I
>> would think might work:****
>>
>> ** **
>>
>> On the Hadoop box, open a listening port and feed that to the HDFS
>> command:****
>>
>> nc -l 2342 | hdfs dfs -copyFromLocal - /tmp/x.txt****
>>
>> ** **
>>
>> On the remote server:****
>>
>> cat my_big_2tb_file > nc 10.1.1.1 2342****
>>
>> ** **
>>
>> I haven’t tried it yet, but in theory this would work. I just happened to
>> test out the hdfs dfs command reading from stdin. You might have to correct
>> the above syntax, I just wrote it off the top of my head.****
>>
>> ** **
>>
>> Dave****
>>
>> ** **
>>
>> ** **
>>
>> *From:* 超级塞亚人 [mailto:sheldom@gmail.com]
>> *Sent:* Friday, April 19, 2013 11:35 AM
>> *To:* user@hadoop.apache.org
>> *Subject:* Uploading file to HDFS****
>>
>> ** **
>>
>> I have a problem. Our cluster has 32 nodes. Each disk is 1TB. I wanna
>> upload 2TB file to HDFS.How can I put the file to the namenode and upload
>> to HDFS? ****
>>
>
>
--
Olivier Renault
Solution Engineer - Big Data - Hortonworks, Inc.
+44 7500 933 036
orenault@hortonworks.com
www.hortonworks.com
<http://hortonworks.com/products/hortonworks-sandbox/>
Re: Uploading file to HDFS
Posted by Olivier Renault <or...@hortonworks.com>.
You should be able to do it using WebHDFS.
WebHDFS is a REST API so there is no need to have hadoop installed on the
client where the file is located. YOu can find an example on how to copy a
file at the following URL.
http://hadoop.apache.org/docs/r1.0.4/webhdfs.html#CREATE
Olivier
On 19 April 2013 11:01, Wellington Chevreuil <wellington.chevreuil@gmail.com
> wrote:
> Can't you use flume for that?
>
>
> 2013/4/19 David Parks <da...@yahoo.com>
>
>> I just realized another trick you might trying. The Hadoop dfs client can
>> read input from STDIN, you could use netcat to pipe the stuff across to
>> HDFS without hitting the hard drive, I haven��t tried it, but here��s what I
>> would think might work:****
>>
>> ** **
>>
>> On the Hadoop box, open a listening port and feed that to the HDFS
>> command:****
>>
>> nc -l 2342 | hdfs dfs -copyFromLocal - /tmp/x.txt****
>>
>> ** **
>>
>> On the remote server:****
>>
>> cat my_big_2tb_file > nc 10.1.1.1 2342****
>>
>> ** **
>>
>> I haven��t tried it yet, but in theory this would work. I just happened to
>> test out the hdfs dfs command reading from stdin. You might have to correct
>> the above syntax, I just wrote it off the top of my head.****
>>
>> ** **
>>
>> Dave****
>>
>> ** **
>>
>> ** **
>>
>> *From:* ���������� [mailto:sheldom@gmail.com]
>> *Sent:* Friday, April 19, 2013 11:35 AM
>> *To:* user@hadoop.apache.org
>> *Subject:* Uploading file to HDFS****
>>
>> ** **
>>
>> I have a problem. Our cluster has 32 nodes. Each disk is 1TB. I wanna
>> upload 2TB file to HDFS.How can I put the file to the namenode and upload
>> to HDFS? ****
>>
>
>
--
Olivier Renault
Solution Engineer - Big Data - Hortonworks, Inc.
+44 7500 933 036
orenault@hortonworks.com
www.hortonworks.com
<http://hortonworks.com/products/hortonworks-sandbox/>
Re: Uploading file to HDFS
Posted by Olivier Renault <or...@hortonworks.com>.
You should be able to do it using WebHDFS.
WebHDFS is a REST API so there is no need to have hadoop installed on the
client where the file is located. YOu can find an example on how to copy a
file at the following URL.
http://hadoop.apache.org/docs/r1.0.4/webhdfs.html#CREATE
Olivier
On 19 April 2013 11:01, Wellington Chevreuil <wellington.chevreuil@gmail.com
> wrote:
> Can't you use flume for that?
>
>
> 2013/4/19 David Parks <da...@yahoo.com>
>
>> I just realized another trick you might trying. The Hadoop dfs client can
>> read input from STDIN, you could use netcat to pipe the stuff across to
>> HDFS without hitting the hard drive, I haven��t tried it, but here��s what I
>> would think might work:****
>>
>> ** **
>>
>> On the Hadoop box, open a listening port and feed that to the HDFS
>> command:****
>>
>> nc -l 2342 | hdfs dfs -copyFromLocal - /tmp/x.txt****
>>
>> ** **
>>
>> On the remote server:****
>>
>> cat my_big_2tb_file > nc 10.1.1.1 2342****
>>
>> ** **
>>
>> I haven��t tried it yet, but in theory this would work. I just happened to
>> test out the hdfs dfs command reading from stdin. You might have to correct
>> the above syntax, I just wrote it off the top of my head.****
>>
>> ** **
>>
>> Dave****
>>
>> ** **
>>
>> ** **
>>
>> *From:* ���������� [mailto:sheldom@gmail.com]
>> *Sent:* Friday, April 19, 2013 11:35 AM
>> *To:* user@hadoop.apache.org
>> *Subject:* Uploading file to HDFS****
>>
>> ** **
>>
>> I have a problem. Our cluster has 32 nodes. Each disk is 1TB. I wanna
>> upload 2TB file to HDFS.How can I put the file to the namenode and upload
>> to HDFS? ****
>>
>
>
--
Olivier Renault
Solution Engineer - Big Data - Hortonworks, Inc.
+44 7500 933 036
orenault@hortonworks.com
www.hortonworks.com
<http://hortonworks.com/products/hortonworks-sandbox/>
Re: Uploading file to HDFS
Posted by Wellington Chevreuil <we...@gmail.com>.
Can't you use flume for that?
2013/4/19 David Parks <da...@yahoo.com>
> I just realized another trick you might trying. The Hadoop dfs client can
> read input from STDIN, you could use netcat to pipe the stuff across to
> HDFS without hitting the hard drive, I haven’t tried it, but here’s what I
> would think might work:****
>
> ** **
>
> On the Hadoop box, open a listening port and feed that to the HDFS command:
> ****
>
> nc -l 2342 | hdfs dfs -copyFromLocal - /tmp/x.txt****
>
> ** **
>
> On the remote server:****
>
> cat my_big_2tb_file > nc 10.1.1.1 2342****
>
> ** **
>
> I haven’t tried it yet, but in theory this would work. I just happened to
> test out the hdfs dfs command reading from stdin. You might have to correct
> the above syntax, I just wrote it off the top of my head.****
>
> ** **
>
> Dave****
>
> ** **
>
> ** **
>
> *From:* 超级塞亚人 [mailto:sheldom@gmail.com]
> *Sent:* Friday, April 19, 2013 11:35 AM
> *To:* user@hadoop.apache.org
> *Subject:* Uploading file to HDFS****
>
> ** **
>
> I have a problem. Our cluster has 32 nodes. Each disk is 1TB. I wanna
> upload 2TB file to HDFS.How can I put the file to the namenode and upload
> to HDFS? ****
>
Re: Uploading file to HDFS
Posted by Wellington Chevreuil <we...@gmail.com>.
Can't you use flume for that?
2013/4/19 David Parks <da...@yahoo.com>
> I just realized another trick you might trying. The Hadoop dfs client can
> read input from STDIN, you could use netcat to pipe the stuff across to
> HDFS without hitting the hard drive, I haven’t tried it, but here’s what I
> would think might work:****
>
> ** **
>
> On the Hadoop box, open a listening port and feed that to the HDFS command:
> ****
>
> nc -l 2342 | hdfs dfs -copyFromLocal - /tmp/x.txt****
>
> ** **
>
> On the remote server:****
>
> cat my_big_2tb_file > nc 10.1.1.1 2342****
>
> ** **
>
> I haven’t tried it yet, but in theory this would work. I just happened to
> test out the hdfs dfs command reading from stdin. You might have to correct
> the above syntax, I just wrote it off the top of my head.****
>
> ** **
>
> Dave****
>
> ** **
>
> ** **
>
> *From:* 超级塞亚人 [mailto:sheldom@gmail.com]
> *Sent:* Friday, April 19, 2013 11:35 AM
> *To:* user@hadoop.apache.org
> *Subject:* Uploading file to HDFS****
>
> ** **
>
> I have a problem. Our cluster has 32 nodes. Each disk is 1TB. I wanna
> upload 2TB file to HDFS.How can I put the file to the namenode and upload
> to HDFS? ****
>
Re: Uploading file to HDFS
Posted by Wellington Chevreuil <we...@gmail.com>.
Can't you use flume for that?
2013/4/19 David Parks <da...@yahoo.com>
> I just realized another trick you might trying. The Hadoop dfs client can
> read input from STDIN, you could use netcat to pipe the stuff across to
> HDFS without hitting the hard drive, I haven’t tried it, but here’s what I
> would think might work:****
>
> ** **
>
> On the Hadoop box, open a listening port and feed that to the HDFS command:
> ****
>
> nc -l 2342 | hdfs dfs -copyFromLocal - /tmp/x.txt****
>
> ** **
>
> On the remote server:****
>
> cat my_big_2tb_file > nc 10.1.1.1 2342****
>
> ** **
>
> I haven’t tried it yet, but in theory this would work. I just happened to
> test out the hdfs dfs command reading from stdin. You might have to correct
> the above syntax, I just wrote it off the top of my head.****
>
> ** **
>
> Dave****
>
> ** **
>
> ** **
>
> *From:* 超级塞亚人 [mailto:sheldom@gmail.com]
> *Sent:* Friday, April 19, 2013 11:35 AM
> *To:* user@hadoop.apache.org
> *Subject:* Uploading file to HDFS****
>
> ** **
>
> I have a problem. Our cluster has 32 nodes. Each disk is 1TB. I wanna
> upload 2TB file to HDFS.How can I put the file to the namenode and upload
> to HDFS? ****
>
Re: Uploading file to HDFS
Posted by Wellington Chevreuil <we...@gmail.com>.
Can't you use flume for that?
2013/4/19 David Parks <da...@yahoo.com>
> I just realized another trick you might trying. The Hadoop dfs client can
> read input from STDIN, you could use netcat to pipe the stuff across to
> HDFS without hitting the hard drive, I haven’t tried it, but here’s what I
> would think might work:****
>
> ** **
>
> On the Hadoop box, open a listening port and feed that to the HDFS command:
> ****
>
> nc -l 2342 | hdfs dfs -copyFromLocal - /tmp/x.txt****
>
> ** **
>
> On the remote server:****
>
> cat my_big_2tb_file > nc 10.1.1.1 2342****
>
> ** **
>
> I haven’t tried it yet, but in theory this would work. I just happened to
> test out the hdfs dfs command reading from stdin. You might have to correct
> the above syntax, I just wrote it off the top of my head.****
>
> ** **
>
> Dave****
>
> ** **
>
> ** **
>
> *From:* 超级塞亚人 [mailto:sheldom@gmail.com]
> *Sent:* Friday, April 19, 2013 11:35 AM
> *To:* user@hadoop.apache.org
> *Subject:* Uploading file to HDFS****
>
> ** **
>
> I have a problem. Our cluster has 32 nodes. Each disk is 1TB. I wanna
> upload 2TB file to HDFS.How can I put the file to the namenode and upload
> to HDFS? ****
>
RE: Uploading file to HDFS
Posted by David Parks <da...@yahoo.com>.
I just realized another trick you might trying. The Hadoop dfs client can
read input from STDIN, you could use netcat to pipe the stuff across to HDFS
without hitting the hard drive, I haven’t tried it, but here’s what I
would think might work:
On the Hadoop box, open a listening port and feed that to the HDFS command:
nc -l 2342 | hdfs dfs -copyFromLocal - /tmp/x.txt
On the remote server:
cat my_big_2tb_file > nc 10.1.1.1 2342
I haven’t tried it yet, but in theory this would work. I just happened to
test out the hdfs dfs command reading from stdin. You might have to correct
the above syntax, I just wrote it off the top of my head.
Dave
From: 超级塞亚人 [mailto:sheldom@gmail.com]
Sent: Friday, April 19, 2013 11:35 AM
To: user@hadoop.apache.org
Subject: Uploading file to HDFS
I have a problem. Our cluster has 32 nodes. Each disk is 1TB. I wanna upload
2TB file to HDFS.How can I put the file to the namenode and upload to HDFS?
RE: Uploading file to HDFS
Posted by David Parks <da...@yahoo.com>.
I just realized another trick you might trying. The Hadoop dfs client can
read input from STDIN, you could use netcat to pipe the stuff across to HDFS
without hitting the hard drive, I haven’t tried it, but here’s what I
would think might work:
On the Hadoop box, open a listening port and feed that to the HDFS command:
nc -l 2342 | hdfs dfs -copyFromLocal - /tmp/x.txt
On the remote server:
cat my_big_2tb_file > nc 10.1.1.1 2342
I haven’t tried it yet, but in theory this would work. I just happened to
test out the hdfs dfs command reading from stdin. You might have to correct
the above syntax, I just wrote it off the top of my head.
Dave
From: 超级塞亚人 [mailto:sheldom@gmail.com]
Sent: Friday, April 19, 2013 11:35 AM
To: user@hadoop.apache.org
Subject: Uploading file to HDFS
I have a problem. Our cluster has 32 nodes. Each disk is 1TB. I wanna upload
2TB file to HDFS.How can I put the file to the namenode and upload to HDFS?
RE: Uploading file to HDFS
Posted by David Parks <da...@yahoo.com>.
I think the problem here is that he doesn't have Hadoop installed on this
other location so there's no Hadoop DFS client to do the put directly into
HDFS on, he would normally copy the file to one of the nodes in the cluster
where the client files are installed. I've had the same problem recently.
I've tried setting up dfs-hdfs-proxy, though I must say that it's been
crashing when I try to put modest to large files through it (but I've got a
thread going with the developer on that issue). That, or one of the other
remote mount options might work well.
https://www.google.com/url?sa=t&rct=j&q=&esrc=s&source=web&cd=1&cad=rja&ved=
0CDQQFjAA&url=http%3A%2F%2Fwiki.apache.org%2Fhadoop%2FMountableHDFS&ei=T-pwU
Y74A8jPrQfYooHoBw&usg=AFQjCNEQbxmrMGKAETj3FPEw3Lr1PBHz-w&sig2=4JpEzZ_8IAyJ-N
PofSRmMg&bvm=bv.45373924,d.bmk
You could also install Hadoop on the box that has the 2TB file (I realize
that you might not control it or want to do that depending on the
configuration).
A remote NFS mount that you can access from one of the Hadoop boxes... ?
Split up the file into smaller pieces?
There are some ideas. I'd love to hear your final solution as I've also been
having fits getting into HDFS from outside the Hadoop environment. I wish
it natively supported NFS mounts or some light weight/easy to install remote
DFS tools.
Dave
-----Original Message-----
From: Harsh J [mailto:harsh@cloudera.com]
Sent: Friday, April 19, 2013 1:40 PM
To: <us...@hadoop.apache.org>
Subject: Re: Uploading file to HDFS
Can you not simply do a fs -put from the location where the 2 TB file
currently resides? HDFS should be able to consume it just fine, as the
client chunks them into fixed size blocks.
On Fri, Apr 19, 2013 at 10:05 AM, 超级塞亚人 <sh...@gmail.com> wrote:
> I have a problem. Our cluster has 32 nodes. Each disk is 1TB. I wanna
> upload 2TB file to HDFS.How can I put the file to the namenode and upload
to HDFS?
--
Harsh J
RE: Uploading file to HDFS
Posted by David Parks <da...@yahoo.com>.
I think the problem here is that he doesn't have Hadoop installed on this
other location so there's no Hadoop DFS client to do the put directly into
HDFS on, he would normally copy the file to one of the nodes in the cluster
where the client files are installed. I've had the same problem recently.
I've tried setting up dfs-hdfs-proxy, though I must say that it's been
crashing when I try to put modest to large files through it (but I've got a
thread going with the developer on that issue). That, or one of the other
remote mount options might work well.
https://www.google.com/url?sa=t&rct=j&q=&esrc=s&source=web&cd=1&cad=rja&ved=
0CDQQFjAA&url=http%3A%2F%2Fwiki.apache.org%2Fhadoop%2FMountableHDFS&ei=T-pwU
Y74A8jPrQfYooHoBw&usg=AFQjCNEQbxmrMGKAETj3FPEw3Lr1PBHz-w&sig2=4JpEzZ_8IAyJ-N
PofSRmMg&bvm=bv.45373924,d.bmk
You could also install Hadoop on the box that has the 2TB file (I realize
that you might not control it or want to do that depending on the
configuration).
A remote NFS mount that you can access from one of the Hadoop boxes... ?
Split up the file into smaller pieces?
There are some ideas. I'd love to hear your final solution as I've also been
having fits getting into HDFS from outside the Hadoop environment. I wish
it natively supported NFS mounts or some light weight/easy to install remote
DFS tools.
Dave
-----Original Message-----
From: Harsh J [mailto:harsh@cloudera.com]
Sent: Friday, April 19, 2013 1:40 PM
To: <us...@hadoop.apache.org>
Subject: Re: Uploading file to HDFS
Can you not simply do a fs -put from the location where the 2 TB file
currently resides? HDFS should be able to consume it just fine, as the
client chunks them into fixed size blocks.
On Fri, Apr 19, 2013 at 10:05 AM, 超级塞亚人 <sh...@gmail.com> wrote:
> I have a problem. Our cluster has 32 nodes. Each disk is 1TB. I wanna
> upload 2TB file to HDFS.How can I put the file to the namenode and upload
to HDFS?
--
Harsh J
RE: Uploading file to HDFS
Posted by David Parks <da...@yahoo.com>.
I think the problem here is that he doesn't have Hadoop installed on this
other location so there's no Hadoop DFS client to do the put directly into
HDFS on, he would normally copy the file to one of the nodes in the cluster
where the client files are installed. I've had the same problem recently.
I've tried setting up dfs-hdfs-proxy, though I must say that it's been
crashing when I try to put modest to large files through it (but I've got a
thread going with the developer on that issue). That, or one of the other
remote mount options might work well.
https://www.google.com/url?sa=t&rct=j&q=&esrc=s&source=web&cd=1&cad=rja&ved=
0CDQQFjAA&url=http%3A%2F%2Fwiki.apache.org%2Fhadoop%2FMountableHDFS&ei=T-pwU
Y74A8jPrQfYooHoBw&usg=AFQjCNEQbxmrMGKAETj3FPEw3Lr1PBHz-w&sig2=4JpEzZ_8IAyJ-N
PofSRmMg&bvm=bv.45373924,d.bmk
You could also install Hadoop on the box that has the 2TB file (I realize
that you might not control it or want to do that depending on the
configuration).
A remote NFS mount that you can access from one of the Hadoop boxes... ?
Split up the file into smaller pieces?
There are some ideas. I'd love to hear your final solution as I've also been
having fits getting into HDFS from outside the Hadoop environment. I wish
it natively supported NFS mounts or some light weight/easy to install remote
DFS tools.
Dave
-----Original Message-----
From: Harsh J [mailto:harsh@cloudera.com]
Sent: Friday, April 19, 2013 1:40 PM
To: <us...@hadoop.apache.org>
Subject: Re: Uploading file to HDFS
Can you not simply do a fs -put from the location where the 2 TB file
currently resides? HDFS should be able to consume it just fine, as the
client chunks them into fixed size blocks.
On Fri, Apr 19, 2013 at 10:05 AM, 超级塞亚人 <sh...@gmail.com> wrote:
> I have a problem. Our cluster has 32 nodes. Each disk is 1TB. I wanna
> upload 2TB file to HDFS.How can I put the file to the namenode and upload
to HDFS?
--
Harsh J
RE: Uploading file to HDFS
Posted by David Parks <da...@yahoo.com>.
I think the problem here is that he doesn't have Hadoop installed on this
other location so there's no Hadoop DFS client to do the put directly into
HDFS on, he would normally copy the file to one of the nodes in the cluster
where the client files are installed. I've had the same problem recently.
I've tried setting up dfs-hdfs-proxy, though I must say that it's been
crashing when I try to put modest to large files through it (but I've got a
thread going with the developer on that issue). That, or one of the other
remote mount options might work well.
https://www.google.com/url?sa=t&rct=j&q=&esrc=s&source=web&cd=1&cad=rja&ved=
0CDQQFjAA&url=http%3A%2F%2Fwiki.apache.org%2Fhadoop%2FMountableHDFS&ei=T-pwU
Y74A8jPrQfYooHoBw&usg=AFQjCNEQbxmrMGKAETj3FPEw3Lr1PBHz-w&sig2=4JpEzZ_8IAyJ-N
PofSRmMg&bvm=bv.45373924,d.bmk
You could also install Hadoop on the box that has the 2TB file (I realize
that you might not control it or want to do that depending on the
configuration).
A remote NFS mount that you can access from one of the Hadoop boxes... ?
Split up the file into smaller pieces?
There are some ideas. I'd love to hear your final solution as I've also been
having fits getting into HDFS from outside the Hadoop environment. I wish
it natively supported NFS mounts or some light weight/easy to install remote
DFS tools.
Dave
-----Original Message-----
From: Harsh J [mailto:harsh@cloudera.com]
Sent: Friday, April 19, 2013 1:40 PM
To: <us...@hadoop.apache.org>
Subject: Re: Uploading file to HDFS
Can you not simply do a fs -put from the location where the 2 TB file
currently resides? HDFS should be able to consume it just fine, as the
client chunks them into fixed size blocks.
On Fri, Apr 19, 2013 at 10:05 AM, 超级塞亚人 <sh...@gmail.com> wrote:
> I have a problem. Our cluster has 32 nodes. Each disk is 1TB. I wanna
> upload 2TB file to HDFS.How can I put the file to the namenode and upload
to HDFS?
--
Harsh J
Re: Uploading file to HDFS
Posted by Harsh J <ha...@cloudera.com>.
Can you not simply do a fs -put from the location where the 2 TB file
currently resides? HDFS should be able to consume it just fine, as the
client chunks them into fixed size blocks.
On Fri, Apr 19, 2013 at 10:05 AM, 超级塞亚人 <sh...@gmail.com> wrote:
> I have a problem. Our cluster has 32 nodes. Each disk is 1TB. I wanna upload
> 2TB file to HDFS.How can I put the file to the namenode and upload to HDFS?
--
Harsh J
RE: Uploading file to HDFS
Posted by David Parks <da...@yahoo.com>.
I just realized another trick you might trying. The Hadoop dfs client can
read input from STDIN, you could use netcat to pipe the stuff across to HDFS
without hitting the hard drive, I haven’t tried it, but here’s what I
would think might work:
On the Hadoop box, open a listening port and feed that to the HDFS command:
nc -l 2342 | hdfs dfs -copyFromLocal - /tmp/x.txt
On the remote server:
cat my_big_2tb_file > nc 10.1.1.1 2342
I haven’t tried it yet, but in theory this would work. I just happened to
test out the hdfs dfs command reading from stdin. You might have to correct
the above syntax, I just wrote it off the top of my head.
Dave
From: 超级塞亚人 [mailto:sheldom@gmail.com]
Sent: Friday, April 19, 2013 11:35 AM
To: user@hadoop.apache.org
Subject: Uploading file to HDFS
I have a problem. Our cluster has 32 nodes. Each disk is 1TB. I wanna upload
2TB file to HDFS.How can I put the file to the namenode and upload to HDFS?
Re: Uploading file to HDFS
Posted by Harsh J <ha...@cloudera.com>.
Can you not simply do a fs -put from the location where the 2 TB file
currently resides? HDFS should be able to consume it just fine, as the
client chunks them into fixed size blocks.
On Fri, Apr 19, 2013 at 10:05 AM, 超级塞亚人 <sh...@gmail.com> wrote:
> I have a problem. Our cluster has 32 nodes. Each disk is 1TB. I wanna upload
> 2TB file to HDFS.How can I put the file to the namenode and upload to HDFS?
--
Harsh J
Re: Uploading file to HDFS
Posted by Harsh J <ha...@cloudera.com>.
Can you not simply do a fs -put from the location where the 2 TB file
currently resides? HDFS should be able to consume it just fine, as the
client chunks them into fixed size blocks.
On Fri, Apr 19, 2013 at 10:05 AM, 超级塞亚人 <sh...@gmail.com> wrote:
> I have a problem. Our cluster has 32 nodes. Each disk is 1TB. I wanna upload
> 2TB file to HDFS.How can I put the file to the namenode and upload to HDFS?
--
Harsh J
Re: Uploading file to HDFS
Posted by Harsh J <ha...@cloudera.com>.
Can you not simply do a fs -put from the location where the 2 TB file
currently resides? HDFS should be able to consume it just fine, as the
client chunks them into fixed size blocks.
On Fri, Apr 19, 2013 at 10:05 AM, 超级塞亚人 <sh...@gmail.com> wrote:
> I have a problem. Our cluster has 32 nodes. Each disk is 1TB. I wanna upload
> 2TB file to HDFS.How can I put the file to the namenode and upload to HDFS?
--
Harsh J
Re: Uploading file to HDFS
Posted by shashwat shriparv <dw...@gmail.com>.
On Tue, Apr 23, 2013 at 9:23 PM, Mohammad Tariq <do...@gmail.com> wrote:
> What should I do on namenode and datanode? Thank you very much
As Tariq has ask, can you provide datanode logs snapshots??
*Thanks & Regards *
∞
Shashwat Shriparv
Re: Uploading file to HDFS
Posted by shashwat shriparv <dw...@gmail.com>.
On Tue, Apr 23, 2013 at 9:23 PM, Mohammad Tariq <do...@gmail.com> wrote:
> What should I do on namenode and datanode? Thank you very much
As Tariq has ask, can you provide datanode logs snapshots??
*Thanks & Regards *
∞
Shashwat Shriparv
Re: Uploading file to HDFS
Posted by shashwat shriparv <dw...@gmail.com>.
On Tue, Apr 23, 2013 at 9:23 PM, Mohammad Tariq <do...@gmail.com> wrote:
> What should I do on namenode and datanode? Thank you very much
As Tariq has ask, can you provide datanode logs snapshots??
*Thanks & Regards *
∞
Shashwat Shriparv
Re: Uploading file to HDFS
Posted by shashwat shriparv <dw...@gmail.com>.
On Tue, Apr 23, 2013 at 9:23 PM, Mohammad Tariq <do...@gmail.com> wrote:
> What should I do on namenode and datanode? Thank you very much
As Tariq has ask, can you provide datanode logs snapshots??
*Thanks & Regards *
∞
Shashwat Shriparv
Re: Uploading file to HDFS
Posted by Mohammad Tariq <do...@gmail.com>.
Hi there,
Could you plz show me your config files and DN error logs?
Warm Regards,
Tariq
https://mtariq.jux.com/
cloudfront.blogspot.com
On Tue, Apr 23, 2013 at 4:35 PM, 超级塞亚人 <sh...@gmail.com> wrote:
> Asking for help! I'm facing the problem that no datanode to stop. Namenode
> has been started but datanode can't be started. What should I do on
> namenode and datanode? Thank you very much
>
>
> 2013/4/19 超级塞亚人 <sh...@gmail.com>
>
>> I have a problem. Our cluster has 32 nodes. Each disk is 1TB. I wanna
>> upload 2TB file to HDFS.How can I put the file to the namenode and upload
>> to HDFS?
>
>
>
Re: Uploading file to HDFS
Posted by Mohammad Tariq <do...@gmail.com>.
Hi there,
Could you plz show me your config files and DN error logs?
Warm Regards,
Tariq
https://mtariq.jux.com/
cloudfront.blogspot.com
On Tue, Apr 23, 2013 at 4:35 PM, 超级塞亚人 <sh...@gmail.com> wrote:
> Asking for help! I'm facing the problem that no datanode to stop. Namenode
> has been started but datanode can't be started. What should I do on
> namenode and datanode? Thank you very much
>
>
> 2013/4/19 超级塞亚人 <sh...@gmail.com>
>
>> I have a problem. Our cluster has 32 nodes. Each disk is 1TB. I wanna
>> upload 2TB file to HDFS.How can I put the file to the namenode and upload
>> to HDFS?
>
>
>
Re: Uploading file to HDFS
Posted by Mohammad Tariq <do...@gmail.com>.
Hi there,
Could you plz show me your config files and DN error logs?
Warm Regards,
Tariq
https://mtariq.jux.com/
cloudfront.blogspot.com
On Tue, Apr 23, 2013 at 4:35 PM, 超级塞亚人 <sh...@gmail.com> wrote:
> Asking for help! I'm facing the problem that no datanode to stop. Namenode
> has been started but datanode can't be started. What should I do on
> namenode and datanode? Thank you very much
>
>
> 2013/4/19 超级塞亚人 <sh...@gmail.com>
>
>> I have a problem. Our cluster has 32 nodes. Each disk is 1TB. I wanna
>> upload 2TB file to HDFS.How can I put the file to the namenode and upload
>> to HDFS?
>
>
>
Re: Uploading file to HDFS
Posted by Mohammad Tariq <do...@gmail.com>.
Hi there,
Could you plz show me your config files and DN error logs?
Warm Regards,
Tariq
https://mtariq.jux.com/
cloudfront.blogspot.com
On Tue, Apr 23, 2013 at 4:35 PM, 超级塞亚人 <sh...@gmail.com> wrote:
> Asking for help! I'm facing the problem that no datanode to stop. Namenode
> has been started but datanode can't be started. What should I do on
> namenode and datanode? Thank you very much
>
>
> 2013/4/19 超级塞亚人 <sh...@gmail.com>
>
>> I have a problem. Our cluster has 32 nodes. Each disk is 1TB. I wanna
>> upload 2TB file to HDFS.How can I put the file to the namenode and upload
>> to HDFS?
>
>
>
Re: Uploading file to HDFS
Posted by 超级塞亚人 <sh...@gmail.com>.
Asking for help! I'm facing the problem that no datanode to stop. Namenode
has been started but datanode can't be started. What should I do on
namenode and datanode? Thank you very much
2013/4/19 超级塞亚人 <sh...@gmail.com>
> I have a problem. Our cluster has 32 nodes. Each disk is 1TB. I wanna
> upload 2TB file to HDFS.How can I put the file to the namenode and upload
> to HDFS?
RE: Uploading file to HDFS
Posted by David Parks <da...@yahoo.com>.
I just realized another trick you might trying. The Hadoop dfs client can
read input from STDIN, you could use netcat to pipe the stuff across to HDFS
without hitting the hard drive, I haven’t tried it, but here’s what I
would think might work:
On the Hadoop box, open a listening port and feed that to the HDFS command:
nc -l 2342 | hdfs dfs -copyFromLocal - /tmp/x.txt
On the remote server:
cat my_big_2tb_file > nc 10.1.1.1 2342
I haven’t tried it yet, but in theory this would work. I just happened to
test out the hdfs dfs command reading from stdin. You might have to correct
the above syntax, I just wrote it off the top of my head.
Dave
From: 超级塞亚人 [mailto:sheldom@gmail.com]
Sent: Friday, April 19, 2013 11:35 AM
To: user@hadoop.apache.org
Subject: Uploading file to HDFS
I have a problem. Our cluster has 32 nodes. Each disk is 1TB. I wanna upload
2TB file to HDFS.How can I put the file to the namenode and upload to HDFS?
Re: Uploading file to HDFS
Posted by 超级塞亚人 <sh...@gmail.com>.
Asking for help! I'm facing the problem that no datanode to stop. Namenode
has been started but datanode can't be started. What should I do on
namenode and datanode? Thank you very much
2013/4/19 超级塞亚人 <sh...@gmail.com>
> I have a problem. Our cluster has 32 nodes. Each disk is 1TB. I wanna
> upload 2TB file to HDFS.How can I put the file to the namenode and upload
> to HDFS?
Re: Uploading file to HDFS
Posted by 超级塞亚人 <sh...@gmail.com>.
Asking for help! I'm facing the problem that no datanode to stop. Namenode
has been started but datanode can't be started. What should I do on
namenode and datanode? Thank you very much
2013/4/19 超级塞亚人 <sh...@gmail.com>
> I have a problem. Our cluster has 32 nodes. Each disk is 1TB. I wanna
> upload 2TB file to HDFS.How can I put the file to the namenode and upload
> to HDFS?
Re: Uploading file to HDFS
Posted by 超级塞亚人 <sh...@gmail.com>.
Asking for help! I'm facing the problem that no datanode to stop. Namenode
has been started but datanode can't be started. What should I do on
namenode and datanode? Thank you very much
2013/4/19 超级塞亚人 <sh...@gmail.com>
> I have a problem. Our cluster has 32 nodes. Each disk is 1TB. I wanna
> upload 2TB file to HDFS.How can I put the file to the namenode and upload
> to HDFS?