You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@nutch.apache.org by Bai Shen <ba...@gmail.com> on 2012/10/11 13:20:23 UTC
Referencing files in job from plugin
I need to reference a file from my plugin. However, when I try to call it
using File(blah.txt), it looks for the file at the location where I run
nutch from, not in the job file.
What is the proper way to refer to the files in the job file?
Thanks.
Re: Referencing files in job from plugin
Posted by Bai Shen <ba...@gmail.com>.
Okay, thanks.
On Thu, Oct 11, 2012 at 10:02 AM, Ferdy Galema <fe...@kalooga.com>wrote:
> I'm slightly confused how it works in code that is executed in the job
> submitter context (so outside of mappers/reducers code). You might want to
> check Hadoop documentation on that.
>
> On Thu, Oct 11, 2012 at 3:41 PM, Bai Shen <ba...@gmail.com> wrote:
>
> > I'll give that a try. I know when I was doing new File("dir/blah.txt")
> it
> > was looking in the directory on the client machine that I was running the
> > job from.
> >
> > On Thu, Oct 11, 2012 at 9:30 AM, Ferdy Galema <ferdy.galema@kalooga.com
> > >wrote:
> >
> > > new File("./blah.txt") should do it. It's a relative path.
> > >
> > > On Thu, Oct 11, 2012 at 2:52 PM, Bai Shen <ba...@gmail.com>
> > wrote:
> > >
> > > > I'm trying the first method now. However, some of my code requires a
> > > file
> > > > object that is a directory. So I can't use the first method with it.
> > > >
> > > > If I put something in the /classes, what path do I reference it with?
> > > Do I
> > > > just do new File(/classes/blah.txt")?
> > > >
> > > > Thanks.
> > > >
> > > > On Thu, Oct 11, 2012 at 7:40 AM, Ferdy Galema <
> > ferdy.galema@kalooga.com
> > > > >wrote:
> > > >
> > > > > There are some options. The best way imho is to use
> > > > getResourceAsStream().
> > > > > Put the file in the same package as the calling java code. Then do
> > > > > something like
> > > > > getClass().getClassLoader().getResourceAsStream("myfile");
> > > > >
> > > > > If you really want to directly access the file then you can put it
> in
> > > > > /classes in the job file. This directory is expanded into the
> current
> > > > > working directory of a running task.
> > > > >
> > > > > Last but not least you are able to use shared filesystem, for
> example
> > > the
> > > > > HDFS or the mapreduce DistributedCache. This is useful if the files
> > are
> > > > > big, or change a lot.
> > > > >
> > > > > On Thu, Oct 11, 2012 at 1:20 PM, Bai Shen <baishen.lists@gmail.com
> >
> > > > wrote:
> > > > >
> > > > > > I need to reference a file from my plugin. However, when I try
> to
> > > call
> > > > > it
> > > > > > using File(blah.txt), it looks for the file at the location
> where I
> > > run
> > > > > > nutch from, not in the job file.
> > > > > >
> > > > > > What is the proper way to refer to the files in the job file?
> > > > > >
> > > > > > Thanks.
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: Referencing files in job from plugin
Posted by Ferdy Galema <fe...@kalooga.com>.
I'm slightly confused how it works in code that is executed in the job
submitter context (so outside of mappers/reducers code). You might want to
check Hadoop documentation on that.
On Thu, Oct 11, 2012 at 3:41 PM, Bai Shen <ba...@gmail.com> wrote:
> I'll give that a try. I know when I was doing new File("dir/blah.txt") it
> was looking in the directory on the client machine that I was running the
> job from.
>
> On Thu, Oct 11, 2012 at 9:30 AM, Ferdy Galema <ferdy.galema@kalooga.com
> >wrote:
>
> > new File("./blah.txt") should do it. It's a relative path.
> >
> > On Thu, Oct 11, 2012 at 2:52 PM, Bai Shen <ba...@gmail.com>
> wrote:
> >
> > > I'm trying the first method now. However, some of my code requires a
> > file
> > > object that is a directory. So I can't use the first method with it.
> > >
> > > If I put something in the /classes, what path do I reference it with?
> > Do I
> > > just do new File(/classes/blah.txt")?
> > >
> > > Thanks.
> > >
> > > On Thu, Oct 11, 2012 at 7:40 AM, Ferdy Galema <
> ferdy.galema@kalooga.com
> > > >wrote:
> > >
> > > > There are some options. The best way imho is to use
> > > getResourceAsStream().
> > > > Put the file in the same package as the calling java code. Then do
> > > > something like
> > > > getClass().getClassLoader().getResourceAsStream("myfile");
> > > >
> > > > If you really want to directly access the file then you can put it in
> > > > /classes in the job file. This directory is expanded into the current
> > > > working directory of a running task.
> > > >
> > > > Last but not least you are able to use shared filesystem, for example
> > the
> > > > HDFS or the mapreduce DistributedCache. This is useful if the files
> are
> > > > big, or change a lot.
> > > >
> > > > On Thu, Oct 11, 2012 at 1:20 PM, Bai Shen <ba...@gmail.com>
> > > wrote:
> > > >
> > > > > I need to reference a file from my plugin. However, when I try to
> > call
> > > > it
> > > > > using File(blah.txt), it looks for the file at the location where I
> > run
> > > > > nutch from, not in the job file.
> > > > >
> > > > > What is the proper way to refer to the files in the job file?
> > > > >
> > > > > Thanks.
> > > > >
> > > >
> > >
> >
>
Re: Referencing files in job from plugin
Posted by Bai Shen <ba...@gmail.com>.
I'll give that a try. I know when I was doing new File("dir/blah.txt") it
was looking in the directory on the client machine that I was running the
job from.
On Thu, Oct 11, 2012 at 9:30 AM, Ferdy Galema <fe...@kalooga.com>wrote:
> new File("./blah.txt") should do it. It's a relative path.
>
> On Thu, Oct 11, 2012 at 2:52 PM, Bai Shen <ba...@gmail.com> wrote:
>
> > I'm trying the first method now. However, some of my code requires a
> file
> > object that is a directory. So I can't use the first method with it.
> >
> > If I put something in the /classes, what path do I reference it with?
> Do I
> > just do new File(/classes/blah.txt")?
> >
> > Thanks.
> >
> > On Thu, Oct 11, 2012 at 7:40 AM, Ferdy Galema <ferdy.galema@kalooga.com
> > >wrote:
> >
> > > There are some options. The best way imho is to use
> > getResourceAsStream().
> > > Put the file in the same package as the calling java code. Then do
> > > something like
> > > getClass().getClassLoader().getResourceAsStream("myfile");
> > >
> > > If you really want to directly access the file then you can put it in
> > > /classes in the job file. This directory is expanded into the current
> > > working directory of a running task.
> > >
> > > Last but not least you are able to use shared filesystem, for example
> the
> > > HDFS or the mapreduce DistributedCache. This is useful if the files are
> > > big, or change a lot.
> > >
> > > On Thu, Oct 11, 2012 at 1:20 PM, Bai Shen <ba...@gmail.com>
> > wrote:
> > >
> > > > I need to reference a file from my plugin. However, when I try to
> call
> > > it
> > > > using File(blah.txt), it looks for the file at the location where I
> run
> > > > nutch from, not in the job file.
> > > >
> > > > What is the proper way to refer to the files in the job file?
> > > >
> > > > Thanks.
> > > >
> > >
> >
>
Re: Referencing files in job from plugin
Posted by Ferdy Galema <fe...@kalooga.com>.
new File("./blah.txt") should do it. It's a relative path.
On Thu, Oct 11, 2012 at 2:52 PM, Bai Shen <ba...@gmail.com> wrote:
> I'm trying the first method now. However, some of my code requires a file
> object that is a directory. So I can't use the first method with it.
>
> If I put something in the /classes, what path do I reference it with? Do I
> just do new File(/classes/blah.txt")?
>
> Thanks.
>
> On Thu, Oct 11, 2012 at 7:40 AM, Ferdy Galema <ferdy.galema@kalooga.com
> >wrote:
>
> > There are some options. The best way imho is to use
> getResourceAsStream().
> > Put the file in the same package as the calling java code. Then do
> > something like
> > getClass().getClassLoader().getResourceAsStream("myfile");
> >
> > If you really want to directly access the file then you can put it in
> > /classes in the job file. This directory is expanded into the current
> > working directory of a running task.
> >
> > Last but not least you are able to use shared filesystem, for example the
> > HDFS or the mapreduce DistributedCache. This is useful if the files are
> > big, or change a lot.
> >
> > On Thu, Oct 11, 2012 at 1:20 PM, Bai Shen <ba...@gmail.com>
> wrote:
> >
> > > I need to reference a file from my plugin. However, when I try to call
> > it
> > > using File(blah.txt), it looks for the file at the location where I run
> > > nutch from, not in the job file.
> > >
> > > What is the proper way to refer to the files in the job file?
> > >
> > > Thanks.
> > >
> >
>
Re: Referencing files in job from plugin
Posted by Bai Shen <ba...@gmail.com>.
I'm trying the first method now. However, some of my code requires a file
object that is a directory. So I can't use the first method with it.
If I put something in the /classes, what path do I reference it with? Do I
just do new File(/classes/blah.txt")?
Thanks.
On Thu, Oct 11, 2012 at 7:40 AM, Ferdy Galema <fe...@kalooga.com>wrote:
> There are some options. The best way imho is to use getResourceAsStream().
> Put the file in the same package as the calling java code. Then do
> something like
> getClass().getClassLoader().getResourceAsStream("myfile");
>
> If you really want to directly access the file then you can put it in
> /classes in the job file. This directory is expanded into the current
> working directory of a running task.
>
> Last but not least you are able to use shared filesystem, for example the
> HDFS or the mapreduce DistributedCache. This is useful if the files are
> big, or change a lot.
>
> On Thu, Oct 11, 2012 at 1:20 PM, Bai Shen <ba...@gmail.com> wrote:
>
> > I need to reference a file from my plugin. However, when I try to call
> it
> > using File(blah.txt), it looks for the file at the location where I run
> > nutch from, not in the job file.
> >
> > What is the proper way to refer to the files in the job file?
> >
> > Thanks.
> >
>
Re: Referencing files in job from plugin
Posted by Ferdy Galema <fe...@kalooga.com>.
When you have a Configuration object you are good to go. Try any of the
static DistributedCache.get*() methods, such as getCacheFiles or
getFileStatus.
On Thu, Oct 11, 2012 at 5:11 PM, Bai Shen <ba...@gmail.com> wrote:
> Okay, I think I can use the DistributedCache, but I'm not sure on one
> thing. How do I get to the cache from the plugin? I need the hadoop
> context in order to make a call to the cache.
>
> On Thu, Oct 11, 2012 at 7:40 AM, Ferdy Galema <ferdy.galema@kalooga.com
> >wrote:
>
> > There are some options. The best way imho is to use
> getResourceAsStream().
> > Put the file in the same package as the calling java code. Then do
> > something like
> > getClass().getClassLoader().getResourceAsStream("myfile");
> >
> > If you really want to directly access the file then you can put it in
> > /classes in the job file. This directory is expanded into the current
> > working directory of a running task.
> >
> > Last but not least you are able to use shared filesystem, for example the
> > HDFS or the mapreduce DistributedCache. This is useful if the files are
> > big, or change a lot.
> >
> > On Thu, Oct 11, 2012 at 1:20 PM, Bai Shen <ba...@gmail.com>
> wrote:
> >
> > > I need to reference a file from my plugin. However, when I try to call
> > it
> > > using File(blah.txt), it looks for the file at the location where I run
> > > nutch from, not in the job file.
> > >
> > > What is the proper way to refer to the files in the job file?
> > >
> > > Thanks.
> > >
> >
>
Re: Referencing files in job from plugin
Posted by Bai Shen <ba...@gmail.com>.
Okay, I think I can use the DistributedCache, but I'm not sure on one
thing. How do I get to the cache from the plugin? I need the hadoop
context in order to make a call to the cache.
On Thu, Oct 11, 2012 at 7:40 AM, Ferdy Galema <fe...@kalooga.com>wrote:
> There are some options. The best way imho is to use getResourceAsStream().
> Put the file in the same package as the calling java code. Then do
> something like
> getClass().getClassLoader().getResourceAsStream("myfile");
>
> If you really want to directly access the file then you can put it in
> /classes in the job file. This directory is expanded into the current
> working directory of a running task.
>
> Last but not least you are able to use shared filesystem, for example the
> HDFS or the mapreduce DistributedCache. This is useful if the files are
> big, or change a lot.
>
> On Thu, Oct 11, 2012 at 1:20 PM, Bai Shen <ba...@gmail.com> wrote:
>
> > I need to reference a file from my plugin. However, when I try to call
> it
> > using File(blah.txt), it looks for the file at the location where I run
> > nutch from, not in the job file.
> >
> > What is the proper way to refer to the files in the job file?
> >
> > Thanks.
> >
>
Re: Referencing files in job from plugin
Posted by Ferdy Galema <fe...@kalooga.com>.
There are some options. The best way imho is to use getResourceAsStream().
Put the file in the same package as the calling java code. Then do
something like
getClass().getClassLoader().getResourceAsStream("myfile");
If you really want to directly access the file then you can put it in
/classes in the job file. This directory is expanded into the current
working directory of a running task.
Last but not least you are able to use shared filesystem, for example the
HDFS or the mapreduce DistributedCache. This is useful if the files are
big, or change a lot.
On Thu, Oct 11, 2012 at 1:20 PM, Bai Shen <ba...@gmail.com> wrote:
> I need to reference a file from my plugin. However, when I try to call it
> using File(blah.txt), it looks for the file at the location where I run
> nutch from, not in the job file.
>
> What is the proper way to refer to the files in the job file?
>
> Thanks.
>