You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@pig.apache.org by Jim Green <op...@gmail.com> on 2015/04/23 20:57:33 UTC
Any way to set hadoop.tmp.dir in pig except putting it in core-site.xml?
Hi Team,
Pig will put “unbar-xxx” directory in /tmp/hadoop-<user> by default, and
this is controlled by hadoop.tmp.dir.
If many pig jobs are running in parallel, it may fill /tmp or it may cause
IO storm to /tmp.
The goal is to set hadoop.tmp.dir to other directories to give a better
life to /tmp.
I know we can set hadoop.tmp.dir in core-site.xml and it works fine.
However I could not find a way to set it at job level.
I tried “set hadoop.tmp.dir /xxx” in pig scripts and also put it in:
export PIG_OPTS="-Dhadoop.tmp.dir=/xxx"
However none of them works.
*So is there anyway to set hadoop.tmp.dir instead of putting it in
core-site.xml?*
--
Thanks,
www.openkb.info
(Open KnowledgeBase for Hadoop/Database/OS/Network/Tool)
Re: Any way to set hadoop.tmp.dir in pig except putting it in core-site.xml?
Posted by Jim Green <op...@gmail.com>.
Hi Billy,
Yes, I tried both and none of them work.
Thanks.
On Wed, Apr 29, 2015 at 6:21 PM, Billy Watson <wi...@gmail.com>
wrote:
> Have you tried putting it in HADOOP_OPTS instead of PIG_OPTS?
>
> William Watson
> Software Engineer
> (904) 705-7056 PCS
>
> On Wed, Apr 29, 2015 at 6:18 PM, Jim Green <op...@gmail.com> wrote:
>
> > Pig 0.13.
> >
> > Here is one typo in my previous email: “unbar” => “unjar”.
> >
> >
> > On Wed, Apr 29, 2015 at 12:09 PM, Billy Watson <williamrwatson@gmail.com
> >
> > wrote:
> >
> > > What version of pig are you using?
> > >
> > > William Watson
> > > Software Engineer
> > > (904) 705-7056 PCS
> > >
> > > On Thu, Apr 23, 2015 at 2:57 PM, Jim Green <op...@gmail.com>
> wrote:
> > >
> > > > Hi Team,
> > > >
> > > > Pig will put “unbar-xxx” directory in /tmp/hadoop-<user> by default,
> > and
> > > > this is controlled by hadoop.tmp.dir.
> > > > If many pig jobs are running in parallel, it may fill /tmp or it may
> > > cause
> > > > IO storm to /tmp.
> > > > The goal is to set hadoop.tmp.dir to other directories to give a
> better
> > > > life to /tmp.
> > > >
> > > > I know we can set hadoop.tmp.dir in core-site.xml and it works fine.
> > > > However I could not find a way to set it at job level.
> > > >
> > > > I tried “set hadoop.tmp.dir /xxx” in pig scripts and also put it in:
> > > > export PIG_OPTS="-Dhadoop.tmp.dir=/xxx"
> > > > However none of them works.
> > > >
> > > > *So is there anyway to set hadoop.tmp.dir instead of putting it in
> > > > core-site.xml?*
> > > > --
> > > > Thanks,
> > > > www.openkb.info
> > > > (Open KnowledgeBase for Hadoop/Database/OS/Network/Tool)
> > > >
> > >
> >
> >
> >
> > --
> > Thanks,
> > www.openkb.info
> > (Open KnowledgeBase for Hadoop/Database/OS/Network/Tool)
> >
>
--
Thanks,
www.openkb.info
(Open KnowledgeBase for Hadoop/Database/OS/Network/Tool)
Re: Any way to set hadoop.tmp.dir in pig except putting it in core-site.xml?
Posted by Billy Watson <wi...@gmail.com>.
Have you tried putting it in HADOOP_OPTS instead of PIG_OPTS?
William Watson
Software Engineer
(904) 705-7056 PCS
On Wed, Apr 29, 2015 at 6:18 PM, Jim Green <op...@gmail.com> wrote:
> Pig 0.13.
>
> Here is one typo in my previous email: “unbar” => “unjar”.
>
>
> On Wed, Apr 29, 2015 at 12:09 PM, Billy Watson <wi...@gmail.com>
> wrote:
>
> > What version of pig are you using?
> >
> > William Watson
> > Software Engineer
> > (904) 705-7056 PCS
> >
> > On Thu, Apr 23, 2015 at 2:57 PM, Jim Green <op...@gmail.com> wrote:
> >
> > > Hi Team,
> > >
> > > Pig will put “unbar-xxx” directory in /tmp/hadoop-<user> by default,
> and
> > > this is controlled by hadoop.tmp.dir.
> > > If many pig jobs are running in parallel, it may fill /tmp or it may
> > cause
> > > IO storm to /tmp.
> > > The goal is to set hadoop.tmp.dir to other directories to give a better
> > > life to /tmp.
> > >
> > > I know we can set hadoop.tmp.dir in core-site.xml and it works fine.
> > > However I could not find a way to set it at job level.
> > >
> > > I tried “set hadoop.tmp.dir /xxx” in pig scripts and also put it in:
> > > export PIG_OPTS="-Dhadoop.tmp.dir=/xxx"
> > > However none of them works.
> > >
> > > *So is there anyway to set hadoop.tmp.dir instead of putting it in
> > > core-site.xml?*
> > > --
> > > Thanks,
> > > www.openkb.info
> > > (Open KnowledgeBase for Hadoop/Database/OS/Network/Tool)
> > >
> >
>
>
>
> --
> Thanks,
> www.openkb.info
> (Open KnowledgeBase for Hadoop/Database/OS/Network/Tool)
>
Re: Any way to set hadoop.tmp.dir in pig except putting it in core-site.xml?
Posted by Jim Green <op...@gmail.com>.
Pig 0.13.
Here is one typo in my previous email: “unbar” => “unjar”.
On Wed, Apr 29, 2015 at 12:09 PM, Billy Watson <wi...@gmail.com>
wrote:
> What version of pig are you using?
>
> William Watson
> Software Engineer
> (904) 705-7056 PCS
>
> On Thu, Apr 23, 2015 at 2:57 PM, Jim Green <op...@gmail.com> wrote:
>
> > Hi Team,
> >
> > Pig will put “unbar-xxx” directory in /tmp/hadoop-<user> by default, and
> > this is controlled by hadoop.tmp.dir.
> > If many pig jobs are running in parallel, it may fill /tmp or it may
> cause
> > IO storm to /tmp.
> > The goal is to set hadoop.tmp.dir to other directories to give a better
> > life to /tmp.
> >
> > I know we can set hadoop.tmp.dir in core-site.xml and it works fine.
> > However I could not find a way to set it at job level.
> >
> > I tried “set hadoop.tmp.dir /xxx” in pig scripts and also put it in:
> > export PIG_OPTS="-Dhadoop.tmp.dir=/xxx"
> > However none of them works.
> >
> > *So is there anyway to set hadoop.tmp.dir instead of putting it in
> > core-site.xml?*
> > --
> > Thanks,
> > www.openkb.info
> > (Open KnowledgeBase for Hadoop/Database/OS/Network/Tool)
> >
>
--
Thanks,
www.openkb.info
(Open KnowledgeBase for Hadoop/Database/OS/Network/Tool)
Re: Any way to set hadoop.tmp.dir in pig except putting it in core-site.xml?
Posted by Billy Watson <wi...@gmail.com>.
What version of pig are you using?
William Watson
Software Engineer
(904) 705-7056 PCS
On Thu, Apr 23, 2015 at 2:57 PM, Jim Green <op...@gmail.com> wrote:
> Hi Team,
>
> Pig will put “unbar-xxx” directory in /tmp/hadoop-<user> by default, and
> this is controlled by hadoop.tmp.dir.
> If many pig jobs are running in parallel, it may fill /tmp or it may cause
> IO storm to /tmp.
> The goal is to set hadoop.tmp.dir to other directories to give a better
> life to /tmp.
>
> I know we can set hadoop.tmp.dir in core-site.xml and it works fine.
> However I could not find a way to set it at job level.
>
> I tried “set hadoop.tmp.dir /xxx” in pig scripts and also put it in:
> export PIG_OPTS="-Dhadoop.tmp.dir=/xxx"
> However none of them works.
>
> *So is there anyway to set hadoop.tmp.dir instead of putting it in
> core-site.xml?*
> --
> Thanks,
> www.openkb.info
> (Open KnowledgeBase for Hadoop/Database/OS/Network/Tool)
>