You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hadoop.apache.org by "Chandra Mohan, Ananda Vel Murugan" <An...@honeywell.com> on 2013/09/23 15:28:17 UTC
Distributed cache in command line
Hi,
Is it possible to access distributed cache in command line? I have written a custom InputFormat implementation which I want to add to distributed cache. Using libjars is not an option for me as I am not running Hadoop job in command line. I am running it using RHadoop package in R which internally uses Hadoop streaming. Please help. Thanks.
Regards,
Anand.C
RE: Distributed cache in command line
Posted by "Chandra Mohan, Ananda Vel Murugan" <An...@honeywell.com>.
Hi,
Thanks for the response. I can create symlinks for the files. But I don't know how to add jar to distributed cache. I found one way is by using libjars argument while running hadoop job. Is it possible to add a jar file directly to distributed cache? Is there any specific folder in HDFS which acts as distributed cache?
Regards,
Anand.C
From: Omkar Joshi [mailto:ojoshi@hortonworks.com]
Sent: Monday, September 23, 2013 11:57 PM
To: user@hadoop.apache.org
Subject: Re: Distributed cache in command line
Hi,
I have no idea about RHadoop but in general in YARN we do create symlinks for the files in distributed cache in the current working directory of every container. You may be able to use that somehow.
Thanks,
Omkar Joshi
Hortonworks Inc.<http://www.hortonworks.com>
On Mon, Sep 23, 2013 at 6:28 AM, Chandra Mohan, Ananda Vel Murugan <An...@honeywell.com>> wrote:
Hi,
Is it possible to access distributed cache in command line? I have written a custom InputFormat implementation which I want to add to distributed cache. Using libjars is not an option for me as I am not running Hadoop job in command line. I am running it using RHadoop package in R which internally uses Hadoop streaming. Please help. Thanks.
Regards,
Anand.C
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
RE: Distributed cache in command line
Posted by "Chandra Mohan, Ananda Vel Murugan" <An...@honeywell.com>.
Hi,
Thanks for the response. I can create symlinks for the files. But I don't know how to add jar to distributed cache. I found one way is by using libjars argument while running hadoop job. Is it possible to add a jar file directly to distributed cache? Is there any specific folder in HDFS which acts as distributed cache?
Regards,
Anand.C
From: Omkar Joshi [mailto:ojoshi@hortonworks.com]
Sent: Monday, September 23, 2013 11:57 PM
To: user@hadoop.apache.org
Subject: Re: Distributed cache in command line
Hi,
I have no idea about RHadoop but in general in YARN we do create symlinks for the files in distributed cache in the current working directory of every container. You may be able to use that somehow.
Thanks,
Omkar Joshi
Hortonworks Inc.<http://www.hortonworks.com>
On Mon, Sep 23, 2013 at 6:28 AM, Chandra Mohan, Ananda Vel Murugan <An...@honeywell.com>> wrote:
Hi,
Is it possible to access distributed cache in command line? I have written a custom InputFormat implementation which I want to add to distributed cache. Using libjars is not an option for me as I am not running Hadoop job in command line. I am running it using RHadoop package in R which internally uses Hadoop streaming. Please help. Thanks.
Regards,
Anand.C
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
RE: Distributed cache in command line
Posted by "Chandra Mohan, Ananda Vel Murugan" <An...@honeywell.com>.
Hi,
Thanks for the response. I can create symlinks for the files. But I don't know how to add jar to distributed cache. I found one way is by using libjars argument while running hadoop job. Is it possible to add a jar file directly to distributed cache? Is there any specific folder in HDFS which acts as distributed cache?
Regards,
Anand.C
From: Omkar Joshi [mailto:ojoshi@hortonworks.com]
Sent: Monday, September 23, 2013 11:57 PM
To: user@hadoop.apache.org
Subject: Re: Distributed cache in command line
Hi,
I have no idea about RHadoop but in general in YARN we do create symlinks for the files in distributed cache in the current working directory of every container. You may be able to use that somehow.
Thanks,
Omkar Joshi
Hortonworks Inc.<http://www.hortonworks.com>
On Mon, Sep 23, 2013 at 6:28 AM, Chandra Mohan, Ananda Vel Murugan <An...@honeywell.com>> wrote:
Hi,
Is it possible to access distributed cache in command line? I have written a custom InputFormat implementation which I want to add to distributed cache. Using libjars is not an option for me as I am not running Hadoop job in command line. I am running it using RHadoop package in R which internally uses Hadoop streaming. Please help. Thanks.
Regards,
Anand.C
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
RE: Distributed cache in command line
Posted by "Chandra Mohan, Ananda Vel Murugan" <An...@honeywell.com>.
Hi,
Thanks for the response. I can create symlinks for the files. But I don't know how to add jar to distributed cache. I found one way is by using libjars argument while running hadoop job. Is it possible to add a jar file directly to distributed cache? Is there any specific folder in HDFS which acts as distributed cache?
Regards,
Anand.C
From: Omkar Joshi [mailto:ojoshi@hortonworks.com]
Sent: Monday, September 23, 2013 11:57 PM
To: user@hadoop.apache.org
Subject: Re: Distributed cache in command line
Hi,
I have no idea about RHadoop but in general in YARN we do create symlinks for the files in distributed cache in the current working directory of every container. You may be able to use that somehow.
Thanks,
Omkar Joshi
Hortonworks Inc.<http://www.hortonworks.com>
On Mon, Sep 23, 2013 at 6:28 AM, Chandra Mohan, Ananda Vel Murugan <An...@honeywell.com>> wrote:
Hi,
Is it possible to access distributed cache in command line? I have written a custom InputFormat implementation which I want to add to distributed cache. Using libjars is not an option for me as I am not running Hadoop job in command line. I am running it using RHadoop package in R which internally uses Hadoop streaming. Please help. Thanks.
Regards,
Anand.C
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
Re: Distributed cache in command line
Posted by Omkar Joshi <oj...@hortonworks.com>.
Hi,
I have no idea about RHadoop but in general in YARN we do create symlinks
for the files in distributed cache in the current working directory of
every container. You may be able to use that somehow.
Thanks,
Omkar Joshi
*Hortonworks Inc.* <http://www.hortonworks.com>
On Mon, Sep 23, 2013 at 6:28 AM, Chandra Mohan, Ananda Vel Murugan <
Ananda.Murugan@honeywell.com> wrote:
> Hi, ****
>
> ** **
>
> Is it possible to access distributed cache in command line? I have written
> a custom InputFormat implementation which I want to add to distributed
> cache. Using *libjars *is not an option for me as I am not running Hadoop
> job in command line. I am running it using RHadoop package in R which
> internally uses Hadoop streaming. Please help. Thanks. ****
>
> ** **
>
> Regards,****
>
> Anand.C****
>
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: Distributed cache in command line
Posted by Omkar Joshi <oj...@hortonworks.com>.
Hi,
I have no idea about RHadoop but in general in YARN we do create symlinks
for the files in distributed cache in the current working directory of
every container. You may be able to use that somehow.
Thanks,
Omkar Joshi
*Hortonworks Inc.* <http://www.hortonworks.com>
On Mon, Sep 23, 2013 at 6:28 AM, Chandra Mohan, Ananda Vel Murugan <
Ananda.Murugan@honeywell.com> wrote:
> Hi, ****
>
> ** **
>
> Is it possible to access distributed cache in command line? I have written
> a custom InputFormat implementation which I want to add to distributed
> cache. Using *libjars *is not an option for me as I am not running Hadoop
> job in command line. I am running it using RHadoop package in R which
> internally uses Hadoop streaming. Please help. Thanks. ****
>
> ** **
>
> Regards,****
>
> Anand.C****
>
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: Distributed cache in command line
Posted by Omkar Joshi <oj...@hortonworks.com>.
Hi,
I have no idea about RHadoop but in general in YARN we do create symlinks
for the files in distributed cache in the current working directory of
every container. You may be able to use that somehow.
Thanks,
Omkar Joshi
*Hortonworks Inc.* <http://www.hortonworks.com>
On Mon, Sep 23, 2013 at 6:28 AM, Chandra Mohan, Ananda Vel Murugan <
Ananda.Murugan@honeywell.com> wrote:
> Hi, ****
>
> ** **
>
> Is it possible to access distributed cache in command line? I have written
> a custom InputFormat implementation which I want to add to distributed
> cache. Using *libjars *is not an option for me as I am not running Hadoop
> job in command line. I am running it using RHadoop package in R which
> internally uses Hadoop streaming. Please help. Thanks. ****
>
> ** **
>
> Regards,****
>
> Anand.C****
>
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: Distributed cache in command line
Posted by Omkar Joshi <oj...@hortonworks.com>.
Hi,
I have no idea about RHadoop but in general in YARN we do create symlinks
for the files in distributed cache in the current working directory of
every container. You may be able to use that somehow.
Thanks,
Omkar Joshi
*Hortonworks Inc.* <http://www.hortonworks.com>
On Mon, Sep 23, 2013 at 6:28 AM, Chandra Mohan, Ananda Vel Murugan <
Ananda.Murugan@honeywell.com> wrote:
> Hi, ****
>
> ** **
>
> Is it possible to access distributed cache in command line? I have written
> a custom InputFormat implementation which I want to add to distributed
> cache. Using *libjars *is not an option for me as I am not running Hadoop
> job in command line. I am running it using RHadoop package in R which
> internally uses Hadoop streaming. Please help. Thanks. ****
>
> ** **
>
> Regards,****
>
> Anand.C****
>
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.