Re: How to -moveToLocal to local machine instead of remote machine?

[Gabriele Kahlout <ga...@mysimpatico.com>]

It looks like karmasphere can handle it:

INFO
com.karmasphere.studio.hadoop.netbeans.filesystem.CommonsVfsBrowserTransferHandler
importData: Past filenode: FileNode for file:///Users/simpatico/Documents,
state=Updated; type=folder
INFO com.karmasphere.studio.hadoop.netbeans.filesystem.FileOperation$Type$2
run: Moving Files: 1 to FileNode for file:///Users/simpatico/Documents,
state=Updated; type=folder



On Sun, Jul 31, 2011 at 8:41 PM, Gabriele Kahlout
<ga...@mysimpatico.com>wrote:

> I have sort of the same problem I feel.
> I've a large segment I cannot index because there's not enough room, nor
> -copyToLocal, local being the server.
> How can I instead -copyToLocal where local is my local terminal machine,
> instead of the server?
>
>
> On Tue, Jun 7, 2011 at 9:30 PM, Joe Greenawalt <jo...@gmail.com>wrote:
>
>> Thanks, and I've seen this example.  I think after I can connect, i'm ok,
>> but i'm not sure how to do it remotely.  I'm writing a groovy script just
>> to
>> test connection and i'll paste it below so you can see what i'm trying to
>> do.
>>
>> @GrabResolver(name='org.apache.mahout.hadoop', root='
>> > http://mymavenrepo/nexus/content/repositories/thirdparty')
>> > @Grab(group='org.apache.mahout.hadoop', module='hadoop-core',
>> > version='0.20.203.0')
>> > import org.apache.hadoop.hdfs.DFSClient
>> > import org.apache.hadoop.hdfs.DistributedFileSystem
>> > import org.apache.hadoop.conf.Configuration
>> > org.apache.hadoop.hdfs.protocol.HdfsFileStatus
>> > def DIR_HADOOP = "1.1.1.1";
>> > def PORT_HADOOP = "9000";
>> > def config = new Configuration()
>> > config.set("fs.default.name", DIR_HADOOP + ":" + PORT_HADOOP)  //got
>> this
>> > from from some site
>> > def dfs = new DistributedFileSystem()
>> > def dfsClient = dfs.getClient()
>> > def fileInfo = dfsClient.getFileInfo("/DEV")
>> > println fileInfo.isDir()
>>
>>
>> On Tue, Jun 7, 2011 at 2:11 PM, Habermaas, William <
>> William.Habermaas@fatwire.com> wrote:
>>
>> > Joe,
>> >
>> > Take a look at http://wiki.apache.org/hadoop/HadoopDfsReadWriteExample
>> >
>> > It should give you an idea of how to read and write HDFS files. This
>> page
>> > is somewhat old and the package names have changed a bit between
>> versions
>> > but I hope it will get you on the right track. If you don't want to
>> write
>> > code there are HDFS copy utilities that you can use instead from shell
>> > scripts.
>> >
>> > Bill
>> >
>> > -----Original Message-----
>> > From: Joe Greenawalt [mailto:joe.greenawalt@gmail.com]
>> > Sent: Tuesday, June 07, 2011 1:38 PM
>> > To: common-user@hadoop.apache.org
>> > Subject: Re: remotely downloading file
>> >
>> > Bill,
>> > thanks for the reply, is there a resource that you have available that i
>> > can
>> > look at the correct way to connect remotely?
>> > I seem to be seeing conflicting ways on doing that.
>> >
>> > I'm looking at:
>> >
>> >
>> http://hadoop.apache.org/hdfs/docs/current/api/org/apache/hadoop/hdfs/DFSClient.html
>> >
>> >
>> http://hadoop.apache.org/hdfs/docs/current/api/org/apache/hadoop/hdfs/DistributedFileSystem.html
>> >
>> > But the examples i'm seeing are using the Configuration but i don't see
>> > that
>> > being used in those classes.
>> >
>> > Thanks again,
>> >
>> > Joe
>> >
>> > On Fri, Jun 3, 2011 at 5:05 PM, Habermaas, William <
>> > William.Habermaas@fatwire.com> wrote:
>> >
>> > > You can access HDFS for reading and writing from other machines. The
>> API
>> > > works through the HDFS client which can be anywhere on the network and
>> > not
>> > > just on the namenode. You just have to have the Hadoop core jar with
>> your
>> > > application wherever it is going to run.
>> > >
>> > > Bill
>> > >
>> > > -----Original Message-----
>> > > From: Joe Greenawalt [mailto:joe.greenawalt@gmail.com]
>> > > Sent: Friday, June 03, 2011 4:55 PM
>> > > To: common-user@hadoop.apache.org
>> > > Subject: remotely downloading file
>> > >
>> > > Hi,
>> > > We're interested in using HDFS to store several large file sets to be
>> > > available for download from our customers  in the following paradigm:
>> > >
>> > > Customer  <-  | APPSERVER-CLUSTER {app1,app2,app3} |  <-  | HDFS |
>> > >
>> > > I had assumed that pulling the file from HDFS to the APPSERVER-CLUSTER
>> > > could
>> > > be done program-ably remotely after browsing the documentation.  But
>> > after
>> > > reading the API, it seems that writing Java code to interface with
>> HDFS
>> > > needs to happen locally?  Is that correct?
>> > >
>> > > If it is correct, what is the best/recommended way to
>> > > deliver downloadables to the APPSERVERS (and vice versa) which are
>> hosted
>> > > in
>> > > the same network but on different machines?
>> > >
>> > > Thanks,
>> > > Joe
>> > >
>> >
>>
>
>
>
> --
> Regards,
> K. Gabriele
>
> --- unchanged since 20/9/10 ---
> P.S. If the subject contains "[LON]" or the addressee acknowledges the
> receipt within 48 hours then I don't resend the email.
> subject(this) ∈ L(LON*) ∨ ∃x. (x ∈ MyInbox ∧ Acknowledges(x, this) ∧
> time(x) < Now + 48h) ⇒ ¬resend(I, this).
>
> If an email is sent by a sender that is not a trusted contact or the email
> does not contain a valid code then the email is not received. A valid code
> starts with a hyphen and ends with "X".
> ∀x. x ∈ MyInbox ⇒ from(x) ∈ MySafeSenderList ∨ (∃y. y ∈ subject(x) ∧ y ∈
> L(-[a-z]+[0-9]X)).
>
>


-- 
Regards,
K. Gabriele

--- unchanged since 20/9/10 ---
P.S. If the subject contains "[LON]" or the addressee acknowledges the
receipt within 48 hours then I don't resend the email.
subject(this) ∈ L(LON*) ∨ ∃x. (x ∈ MyInbox ∧ Acknowledges(x, this) ∧ time(x)
< Now + 48h) ⇒ ¬resend(I, this).

If an email is sent by a sender that is not a trusted contact or the email
does not contain a valid code then the email is not received. A valid code
starts with a hyphen and ends with "X".
∀x. x ∈ MyInbox ⇒ from(x) ∈ MySafeSenderList ∨ (∃y. y ∈ subject(x) ∧ y ∈
L(-[a-z]+[0-9]X)).