You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-user@hadoop.apache.org by Cristi Cioriia <cr...@1and1.ro> on 2010/09/07 15:02:47 UTC
How to use distributed cache api
Hi guys,
I want to distribute an xml file to the tasks of a MR job and then be
able to retrieve it using code like this:
DatabaseDescriptor.class.getClassLoader().getResource("storage-conf.xml");
where DatabaseDescriptor is a class.
In order to do that I added the file to the distributed cache in the job
driver like this:
Path filePath = new Path("/user/ccioriia/storage-conf.xml");
DistributedCache.addFileToClassPath(filePath, conf);
Then, for providing the mapper with the dataI need, I have a record
reader class that calls the line :
DatabaseDescriptor.class.getClassLoader().getResource("storage-conf.xml");
which returns null. The file is added in the cache and is copied to the
task tracker, because I checked it with the following code in the
mapper:
Path[] localFiles;
try
{
localFiles = DistributedCache.getLocalCacheFiles(job);
}
catch (IOException e)
{
throw new RuntimeException(e);
}
cassConfig = localFiles[0].getParent().toString();
LOG.info(cassConfig);//prints: /tmp/hadoop-hadoop/mapred/local/taskTracker/archive/cccwork/user/ccioriia/storage-conf.xml
LOG.info(localFiles[0].getName();//prints: storage-config.xml
So I guess what I would like to have is something like
DistributedCache.addCacheFile(new
URI("/user/ccioriia/storage-conf.xml#storage-conf.xml"), conf);
which would also add it to the classpath so that the call
DatabaseDescriptor.class.getClassLoader().getResource("storage-conf.xml");
returns the URL to the file instead of null.
Can you provide me an advice about how I could do this?
Thanks,
--
Cristian-Andrei Cioriia
Programator Java
1&1 Internet Development S.R.L. - Bucharest/Romania - Web Applications
Hosting Romania
18 Mircea Eliade St
Sect 1, Bucharest
RO Bucharest, 012015
Număr de telefon: +40 751 812 984
cristian-andrei.cioriia@1and1.ro
Re:Re: How to use distributed cache api
Posted by Cristi Cioriia <cr...@1and1.ro>.
Hi Arun,
Your suggestion worked perfectly. My code looks like this now:
DistributedCache.addCacheFile(new URI(uri), conf);
DistributedCache.createSymlink(conf);
Thanks,
Cristi
> On Sep 7, 2010, at 6:02 AM, Cristi Cioriia wrote:
> > So I guess what I would like to have is something like
> > DistributedCache.addCacheFile(new
> > URI("/user/ccioriia/storage-conf.xml#storage-conf.xml"), conf);
> > which would also add it to the classpath so that the call
> > DatabaseDescriptor.class.getClassLoader().getResource("storage-
> > conf.xml");
> > returns the URL to the file instead of null.
> > Can you provide me an advice about how I could do this?
>
>
> Yep, also call DC.createSymlink(Configuration) in your launching
> program.
>
> Arun
--
Cristian-Andrei Cioriia
Programator Java
1&1 Internet Development S.R.L. - Bucharest/Romania - Web Applications
Hosting Romania
18 Mircea Eliade St
Sect 1, Bucharest
RO Bucharest, 012015
Număr de telefon: +40 751 812 984
cristian-andrei.cioriia@1and1.ro
Re: How to use distributed cache api
Posted by Arun C Murthy <ac...@yahoo-inc.com>.
On Sep 7, 2010, at 6:02 AM, Cristi Cioriia wrote:
> So I guess what I would like to have is something like
> DistributedCache.addCacheFile(new
> URI("/user/ccioriia/storage-conf.xml#storage-conf.xml"), conf);
> which would also add it to the classpath so that the call
> DatabaseDescriptor.class.getClassLoader().getResource("storage-
> conf.xml");
> returns the URL to the file instead of null.
> Can you provide me an advice about how I could do this?
Yep, also call DC.createSymlink(Configuration) in your launching
program.
Arun
Re: How to use distributed cache api
Posted by Ted Yu <yu...@gmail.com>.
Consider using the following to retrieve file:
Path[] cacheFiles=DistributedCache.getFileClassPaths(conf);
BufferedReader joinReader=new BufferedReader(new
FileReader(cacheFiles[0].toString()));
On Tue, Sep 7, 2010 at 6:02 AM, Cristi Cioriia <
cristian-andrei.cioriia@1and1.ro> wrote:
> Hi guys,
>
> I want to distribute an xml file to the tasks of a MR job and then be
> able to retrieve it using code like this:
>
> DatabaseDescriptor.class.getClassLoader().getResource("storage-conf.xml");
>
> where DatabaseDescriptor is a class.
>
> In order to do that I added the file to the distributed cache in the job
> driver like this:
>
> Path filePath = new Path("/user/ccioriia/storage-conf.xml");
> DistributedCache.addFileToClassPath(filePath, conf);
>
> Then, for providing the mapper with the dataI need, I have a record
> reader class that calls the line :
> DatabaseDescriptor.class.getClassLoader().getResource("storage-conf.xml");
>
> which returns null. The file is added in the cache and is copied to the
> task tracker, because I checked it with the following code in the
> mapper:
>
> Path[] localFiles;
>
> try
> {
> localFiles = DistributedCache.getLocalCacheFiles(job);
> }
> catch (IOException e)
> {
> throw new RuntimeException(e);
> }
> cassConfig = localFiles[0].getParent().toString();
>
> LOG.info(cassConfig);//prints:
> /tmp/hadoop-hadoop/mapred/local/taskTracker/archive/cccwork/user/ccioriia/storage-conf.xml
> LOG.info(localFiles[0].getName();//prints: storage-config.xml
>
> So I guess what I would like to have is something like
> DistributedCache.addCacheFile(new
> URI("/user/ccioriia/storage-conf.xml#storage-conf.xml"), conf);
> which would also add it to the classpath so that the call
> DatabaseDescriptor.class.getClassLoader().getResource("storage-conf.xml");
> returns the URL to the file instead of null.
> Can you provide me an advice about how I could do this?
>
> Thanks,
> --
> Cristian-Andrei Cioriia
> Programator Java
>
> 1&1 Internet Development S.R.L. - Bucharest/Romania - Web Applications
> Hosting Romania
> 18 Mircea Eliade St
> Sect 1, Bucharest
> RO Bucharest, 012015
> Număr de telefon: +40 751 812 984
> cristian-andrei.cioriia@1and1.ro
>
>
>
>