You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by Joe Greenawalt <jo...@gmail.com> on 2011/06/03 22:55:25 UTC
remotely downloading file
Hi,
We're interested in using HDFS to store several large file sets to be
available for download from our customers in the following paradigm:
Customer <- | APPSERVER-CLUSTER {app1,app2,app3} | <- | HDFS |
I had assumed that pulling the file from HDFS to the APPSERVER-CLUSTER could
be done program-ably remotely after browsing the documentation. But after
reading the API, it seems that writing Java code to interface with HDFS
needs to happen locally? Is that correct?
If it is correct, what is the best/recommended way to
deliver downloadables to the APPSERVERS (and vice versa) which are hosted in
the same network but on different machines?
Thanks,
Joe
Re: remotely downloading file
Posted by Joe Greenawalt <jo...@gmail.com>.
Thanks, and I've seen this example. I think after I can connect, i'm ok,
but i'm not sure how to do it remotely. I'm writing a groovy script just to
test connection and i'll paste it below so you can see what i'm trying to
do.
@GrabResolver(name='org.apache.mahout.hadoop', root='
> http://mymavenrepo/nexus/content/repositories/thirdparty')
> @Grab(group='org.apache.mahout.hadoop', module='hadoop-core',
> version='0.20.203.0')
> import org.apache.hadoop.hdfs.DFSClient
> import org.apache.hadoop.hdfs.DistributedFileSystem
> import org.apache.hadoop.conf.Configuration
> org.apache.hadoop.hdfs.protocol.HdfsFileStatus
> def DIR_HADOOP = "1.1.1.1";
> def PORT_HADOOP = "9000";
> def config = new Configuration()
> config.set("fs.default.name", DIR_HADOOP + ":" + PORT_HADOOP) //got this
> from from some site
> def dfs = new DistributedFileSystem()
> def dfsClient = dfs.getClient()
> def fileInfo = dfsClient.getFileInfo("/DEV")
> println fileInfo.isDir()
On Tue, Jun 7, 2011 at 2:11 PM, Habermaas, William <
William.Habermaas@fatwire.com> wrote:
> Joe,
>
> Take a look at http://wiki.apache.org/hadoop/HadoopDfsReadWriteExample
>
> It should give you an idea of how to read and write HDFS files. This page
> is somewhat old and the package names have changed a bit between versions
> but I hope it will get you on the right track. If you don't want to write
> code there are HDFS copy utilities that you can use instead from shell
> scripts.
>
> Bill
>
> -----Original Message-----
> From: Joe Greenawalt [mailto:joe.greenawalt@gmail.com]
> Sent: Tuesday, June 07, 2011 1:38 PM
> To: common-user@hadoop.apache.org
> Subject: Re: remotely downloading file
>
> Bill,
> thanks for the reply, is there a resource that you have available that i
> can
> look at the correct way to connect remotely?
> I seem to be seeing conflicting ways on doing that.
>
> I'm looking at:
>
> http://hadoop.apache.org/hdfs/docs/current/api/org/apache/hadoop/hdfs/DFSClient.html
>
> http://hadoop.apache.org/hdfs/docs/current/api/org/apache/hadoop/hdfs/DistributedFileSystem.html
>
> But the examples i'm seeing are using the Configuration but i don't see
> that
> being used in those classes.
>
> Thanks again,
>
> Joe
>
> On Fri, Jun 3, 2011 at 5:05 PM, Habermaas, William <
> William.Habermaas@fatwire.com> wrote:
>
> > You can access HDFS for reading and writing from other machines. The API
> > works through the HDFS client which can be anywhere on the network and
> not
> > just on the namenode. You just have to have the Hadoop core jar with your
> > application wherever it is going to run.
> >
> > Bill
> >
> > -----Original Message-----
> > From: Joe Greenawalt [mailto:joe.greenawalt@gmail.com]
> > Sent: Friday, June 03, 2011 4:55 PM
> > To: common-user@hadoop.apache.org
> > Subject: remotely downloading file
> >
> > Hi,
> > We're interested in using HDFS to store several large file sets to be
> > available for download from our customers in the following paradigm:
> >
> > Customer <- | APPSERVER-CLUSTER {app1,app2,app3} | <- | HDFS |
> >
> > I had assumed that pulling the file from HDFS to the APPSERVER-CLUSTER
> > could
> > be done program-ably remotely after browsing the documentation. But
> after
> > reading the API, it seems that writing Java code to interface with HDFS
> > needs to happen locally? Is that correct?
> >
> > If it is correct, what is the best/recommended way to
> > deliver downloadables to the APPSERVERS (and vice versa) which are hosted
> > in
> > the same network but on different machines?
> >
> > Thanks,
> > Joe
> >
>
RE: remotely downloading file
Posted by "Habermaas, William" <Wi...@fatwire.com>.
Joe,
Take a look at http://wiki.apache.org/hadoop/HadoopDfsReadWriteExample
It should give you an idea of how to read and write HDFS files. This page is somewhat old and the package names have changed a bit between versions but I hope it will get you on the right track. If you don't want to write code there are HDFS copy utilities that you can use instead from shell scripts.
Bill
-----Original Message-----
From: Joe Greenawalt [mailto:joe.greenawalt@gmail.com]
Sent: Tuesday, June 07, 2011 1:38 PM
To: common-user@hadoop.apache.org
Subject: Re: remotely downloading file
Bill,
thanks for the reply, is there a resource that you have available that i can
look at the correct way to connect remotely?
I seem to be seeing conflicting ways on doing that.
I'm looking at:
http://hadoop.apache.org/hdfs/docs/current/api/org/apache/hadoop/hdfs/DFSClient.html
http://hadoop.apache.org/hdfs/docs/current/api/org/apache/hadoop/hdfs/DistributedFileSystem.html
But the examples i'm seeing are using the Configuration but i don't see that
being used in those classes.
Thanks again,
Joe
On Fri, Jun 3, 2011 at 5:05 PM, Habermaas, William <
William.Habermaas@fatwire.com> wrote:
> You can access HDFS for reading and writing from other machines. The API
> works through the HDFS client which can be anywhere on the network and not
> just on the namenode. You just have to have the Hadoop core jar with your
> application wherever it is going to run.
>
> Bill
>
> -----Original Message-----
> From: Joe Greenawalt [mailto:joe.greenawalt@gmail.com]
> Sent: Friday, June 03, 2011 4:55 PM
> To: common-user@hadoop.apache.org
> Subject: remotely downloading file
>
> Hi,
> We're interested in using HDFS to store several large file sets to be
> available for download from our customers in the following paradigm:
>
> Customer <- | APPSERVER-CLUSTER {app1,app2,app3} | <- | HDFS |
>
> I had assumed that pulling the file from HDFS to the APPSERVER-CLUSTER
> could
> be done program-ably remotely after browsing the documentation. But after
> reading the API, it seems that writing Java code to interface with HDFS
> needs to happen locally? Is that correct?
>
> If it is correct, what is the best/recommended way to
> deliver downloadables to the APPSERVERS (and vice versa) which are hosted
> in
> the same network but on different machines?
>
> Thanks,
> Joe
>
Re: remotely downloading file
Posted by Joe Greenawalt <jo...@gmail.com>.
Bill,
thanks for the reply, is there a resource that you have available that i can
look at the correct way to connect remotely?
I seem to be seeing conflicting ways on doing that.
I'm looking at:
http://hadoop.apache.org/hdfs/docs/current/api/org/apache/hadoop/hdfs/DFSClient.html
http://hadoop.apache.org/hdfs/docs/current/api/org/apache/hadoop/hdfs/DistributedFileSystem.html
But the examples i'm seeing are using the Configuration but i don't see that
being used in those classes.
Thanks again,
Joe
On Fri, Jun 3, 2011 at 5:05 PM, Habermaas, William <
William.Habermaas@fatwire.com> wrote:
> You can access HDFS for reading and writing from other machines. The API
> works through the HDFS client which can be anywhere on the network and not
> just on the namenode. You just have to have the Hadoop core jar with your
> application wherever it is going to run.
>
> Bill
>
> -----Original Message-----
> From: Joe Greenawalt [mailto:joe.greenawalt@gmail.com]
> Sent: Friday, June 03, 2011 4:55 PM
> To: common-user@hadoop.apache.org
> Subject: remotely downloading file
>
> Hi,
> We're interested in using HDFS to store several large file sets to be
> available for download from our customers in the following paradigm:
>
> Customer <- | APPSERVER-CLUSTER {app1,app2,app3} | <- | HDFS |
>
> I had assumed that pulling the file from HDFS to the APPSERVER-CLUSTER
> could
> be done program-ably remotely after browsing the documentation. But after
> reading the API, it seems that writing Java code to interface with HDFS
> needs to happen locally? Is that correct?
>
> If it is correct, what is the best/recommended way to
> deliver downloadables to the APPSERVERS (and vice versa) which are hosted
> in
> the same network but on different machines?
>
> Thanks,
> Joe
>
RE: remotely downloading file
Posted by "Habermaas, William" <Wi...@fatwire.com>.
You can access HDFS for reading and writing from other machines. The API works through the HDFS client which can be anywhere on the network and not just on the namenode. You just have to have the Hadoop core jar with your application wherever it is going to run.
Bill
-----Original Message-----
From: Joe Greenawalt [mailto:joe.greenawalt@gmail.com]
Sent: Friday, June 03, 2011 4:55 PM
To: common-user@hadoop.apache.org
Subject: remotely downloading file
Hi,
We're interested in using HDFS to store several large file sets to be
available for download from our customers in the following paradigm:
Customer <- | APPSERVER-CLUSTER {app1,app2,app3} | <- | HDFS |
I had assumed that pulling the file from HDFS to the APPSERVER-CLUSTER could
be done program-ably remotely after browsing the documentation. But after
reading the API, it seems that writing Java code to interface with HDFS
needs to happen locally? Is that correct?
If it is correct, what is the best/recommended way to
deliver downloadables to the APPSERVERS (and vice versa) which are hosted in
the same network but on different machines?
Thanks,
Joe