You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-user@hadoop.apache.org by Mark Olimpiati <ma...@gmail.com> on 2012/10/11 01:45:09 UTC
Reading Sequence File from Hadoop Distributed Cache ..
Hi,
I'm storing sequence files in the distributed cache which seems to be
stored somewher under each node's /tmp .../local/archive/ ... path.
In mapper code, I tried using SequenceFile.Reader with all possible
configurations (locally, distribtued) however, it can't find it. Are
sequence files supported in the distribued cache??
Thank you,
Mark
Re: Reading Sequence File from Hadoop Distributed Cache ..
Posted by Mark Olimpiati <ma...@gmail.com>.
I'll try that thanks for the suggestion Steve!
Mark
On Fri, Oct 12, 2012 at 11:27 AM, Steve Loughran <st...@hortonworks.com>wrote:
>
>
> On 11 October 2012 20:53, Mark Olimpiati <ma...@gmail.com> wrote:
>
>> Thanks for the reply Harsh, but as I said I tried locally too by using
>> the following:
>>
>> FileSystem localFs = cachedFiles[0].getFileSystem(new
>> Configuration());
>>
>>
>> Isn't the above supposed to give me the local file system ?? If yes, I
>> tried that but didn't work, if not, can you show one example?
>>
>> Thank you,
>> Mark
>
>
> gives you the default FS for that process, which can be different.
>
> what about FileSystem.get("file://tmp",new Configuration())
>
Re: Reading Sequence File from Hadoop Distributed Cache ..
Posted by Mark Olimpiati <ma...@gmail.com>.
I'll try that thanks for the suggestion Steve!
Mark
On Fri, Oct 12, 2012 at 11:27 AM, Steve Loughran <st...@hortonworks.com>wrote:
>
>
> On 11 October 2012 20:53, Mark Olimpiati <ma...@gmail.com> wrote:
>
>> Thanks for the reply Harsh, but as I said I tried locally too by using
>> the following:
>>
>> FileSystem localFs = cachedFiles[0].getFileSystem(new
>> Configuration());
>>
>>
>> Isn't the above supposed to give me the local file system ?? If yes, I
>> tried that but didn't work, if not, can you show one example?
>>
>> Thank you,
>> Mark
>
>
> gives you the default FS for that process, which can be different.
>
> what about FileSystem.get("file://tmp",new Configuration())
>
Re: Reading Sequence File from Hadoop Distributed Cache ..
Posted by Mark Olimpiati <ma...@gmail.com>.
I'll try that thanks for the suggestion Steve!
Mark
On Fri, Oct 12, 2012 at 11:27 AM, Steve Loughran <st...@hortonworks.com>wrote:
>
>
> On 11 October 2012 20:53, Mark Olimpiati <ma...@gmail.com> wrote:
>
>> Thanks for the reply Harsh, but as I said I tried locally too by using
>> the following:
>>
>> FileSystem localFs = cachedFiles[0].getFileSystem(new
>> Configuration());
>>
>>
>> Isn't the above supposed to give me the local file system ?? If yes, I
>> tried that but didn't work, if not, can you show one example?
>>
>> Thank you,
>> Mark
>
>
> gives you the default FS for that process, which can be different.
>
> what about FileSystem.get("file://tmp",new Configuration())
>
Re: Reading Sequence File from Hadoop Distributed Cache ..
Posted by Mark Olimpiati <ma...@gmail.com>.
I'll try that thanks for the suggestion Steve!
Mark
On Fri, Oct 12, 2012 at 11:27 AM, Steve Loughran <st...@hortonworks.com>wrote:
>
>
> On 11 October 2012 20:53, Mark Olimpiati <ma...@gmail.com> wrote:
>
>> Thanks for the reply Harsh, but as I said I tried locally too by using
>> the following:
>>
>> FileSystem localFs = cachedFiles[0].getFileSystem(new
>> Configuration());
>>
>>
>> Isn't the above supposed to give me the local file system ?? If yes, I
>> tried that but didn't work, if not, can you show one example?
>>
>> Thank you,
>> Mark
>
>
> gives you the default FS for that process, which can be different.
>
> what about FileSystem.get("file://tmp",new Configuration())
>
Re: Reading Sequence File from Hadoop Distributed Cache ..
Posted by Steve Loughran <st...@hortonworks.com>.
On 11 October 2012 20:53, Mark Olimpiati <ma...@gmail.com> wrote:
> Thanks for the reply Harsh, but as I said I tried locally too by using the
> following:
>
> FileSystem localFs = cachedFiles[0].getFileSystem(new
> Configuration());
>
>
> Isn't the above supposed to give me the local file system ?? If yes, I
> tried that but didn't work, if not, can you show one example?
>
> Thank you,
> Mark
gives you the default FS for that process, which can be different.
what about FileSystem.get("file://tmp",new Configuration())
Re: Reading Sequence File from Hadoop Distributed Cache ..
Posted by Steve Loughran <st...@hortonworks.com>.
On 11 October 2012 20:53, Mark Olimpiati <ma...@gmail.com> wrote:
> Thanks for the reply Harsh, but as I said I tried locally too by using the
> following:
>
> FileSystem localFs = cachedFiles[0].getFileSystem(new
> Configuration());
>
>
> Isn't the above supposed to give me the local file system ?? If yes, I
> tried that but didn't work, if not, can you show one example?
>
> Thank you,
> Mark
gives you the default FS for that process, which can be different.
what about FileSystem.get("file://tmp",new Configuration())
Re: Reading Sequence File from Hadoop Distributed Cache ..
Posted by Steve Loughran <st...@hortonworks.com>.
On 11 October 2012 20:53, Mark Olimpiati <ma...@gmail.com> wrote:
> Thanks for the reply Harsh, but as I said I tried locally too by using the
> following:
>
> FileSystem localFs = cachedFiles[0].getFileSystem(new
> Configuration());
>
>
> Isn't the above supposed to give me the local file system ?? If yes, I
> tried that but didn't work, if not, can you show one example?
>
> Thank you,
> Mark
gives you the default FS for that process, which can be different.
what about FileSystem.get("file://tmp",new Configuration())
Re: Reading Sequence File from Hadoop Distributed Cache ..
Posted by Steve Loughran <st...@hortonworks.com>.
On 11 October 2012 20:53, Mark Olimpiati <ma...@gmail.com> wrote:
> Thanks for the reply Harsh, but as I said I tried locally too by using the
> following:
>
> FileSystem localFs = cachedFiles[0].getFileSystem(new
> Configuration());
>
>
> Isn't the above supposed to give me the local file system ?? If yes, I
> tried that but didn't work, if not, can you show one example?
>
> Thank you,
> Mark
gives you the default FS for that process, which can be different.
what about FileSystem.get("file://tmp",new Configuration())
Re: Reading Sequence File from Hadoop Distributed Cache ..
Posted by Mark Olimpiati <ma...@gmail.com>.
Thanks for the reply Harsh, but as I said I tried locally too by using the
following:
FileSystem localFs = cachedFiles[0].getFileSystem(new
Configuration());
Isn't the above supposed to give me the local file system ?? If yes, I
tried that but didn't work, if not, can you show one example?
Thank you,
Mark
On Wed, Oct 10, 2012 at 7:33 PM, Harsh J <ha...@cloudera.com> wrote:
> Hi Mark,
>
> DistributedCache files, when accessed from a Task, exist on the local
> file system. You should make sure the SequenceFile.Reader tries to
> read it with a LocalFS than a HDFS instance.
>
> On Thu, Oct 11, 2012 at 5:15 AM, Mark Olimpiati <ma...@gmail.com>
> wrote:
> > Hi,
> >
> > I'm storing sequence files in the distributed cache which seems to be
> > stored somewher under each node's /tmp .../local/archive/ ... path.
> >
> > In mapper code, I tried using SequenceFile.Reader with all possible
> > configurations (locally, distribtued) however, it can't find it. Are
> > sequence files supported in the distribued cache??
> >
> > Thank you,
> > Mark
>
>
>
> --
> Harsh J
>
Re: Reading Sequence File from Hadoop Distributed Cache ..
Posted by Mark Olimpiati <ma...@gmail.com>.
Thanks for the reply Harsh, but as I said I tried locally too by using the
following:
FileSystem localFs = cachedFiles[0].getFileSystem(new
Configuration());
Isn't the above supposed to give me the local file system ?? If yes, I
tried that but didn't work, if not, can you show one example?
Thank you,
Mark
On Wed, Oct 10, 2012 at 7:33 PM, Harsh J <ha...@cloudera.com> wrote:
> Hi Mark,
>
> DistributedCache files, when accessed from a Task, exist on the local
> file system. You should make sure the SequenceFile.Reader tries to
> read it with a LocalFS than a HDFS instance.
>
> On Thu, Oct 11, 2012 at 5:15 AM, Mark Olimpiati <ma...@gmail.com>
> wrote:
> > Hi,
> >
> > I'm storing sequence files in the distributed cache which seems to be
> > stored somewher under each node's /tmp .../local/archive/ ... path.
> >
> > In mapper code, I tried using SequenceFile.Reader with all possible
> > configurations (locally, distribtued) however, it can't find it. Are
> > sequence files supported in the distribued cache??
> >
> > Thank you,
> > Mark
>
>
>
> --
> Harsh J
>
Re: Reading Sequence File from Hadoop Distributed Cache ..
Posted by Mark Olimpiati <ma...@gmail.com>.
Thanks for the reply Harsh, but as I said I tried locally too by using the
following:
FileSystem localFs = cachedFiles[0].getFileSystem(new
Configuration());
Isn't the above supposed to give me the local file system ?? If yes, I
tried that but didn't work, if not, can you show one example?
Thank you,
Mark
On Wed, Oct 10, 2012 at 7:33 PM, Harsh J <ha...@cloudera.com> wrote:
> Hi Mark,
>
> DistributedCache files, when accessed from a Task, exist on the local
> file system. You should make sure the SequenceFile.Reader tries to
> read it with a LocalFS than a HDFS instance.
>
> On Thu, Oct 11, 2012 at 5:15 AM, Mark Olimpiati <ma...@gmail.com>
> wrote:
> > Hi,
> >
> > I'm storing sequence files in the distributed cache which seems to be
> > stored somewher under each node's /tmp .../local/archive/ ... path.
> >
> > In mapper code, I tried using SequenceFile.Reader with all possible
> > configurations (locally, distribtued) however, it can't find it. Are
> > sequence files supported in the distribued cache??
> >
> > Thank you,
> > Mark
>
>
>
> --
> Harsh J
>
Re: Reading Sequence File from Hadoop Distributed Cache ..
Posted by Mark Olimpiati <ma...@gmail.com>.
Thanks for the reply Harsh, but as I said I tried locally too by using the
following:
FileSystem localFs = cachedFiles[0].getFileSystem(new
Configuration());
Isn't the above supposed to give me the local file system ?? If yes, I
tried that but didn't work, if not, can you show one example?
Thank you,
Mark
On Wed, Oct 10, 2012 at 7:33 PM, Harsh J <ha...@cloudera.com> wrote:
> Hi Mark,
>
> DistributedCache files, when accessed from a Task, exist on the local
> file system. You should make sure the SequenceFile.Reader tries to
> read it with a LocalFS than a HDFS instance.
>
> On Thu, Oct 11, 2012 at 5:15 AM, Mark Olimpiati <ma...@gmail.com>
> wrote:
> > Hi,
> >
> > I'm storing sequence files in the distributed cache which seems to be
> > stored somewher under each node's /tmp .../local/archive/ ... path.
> >
> > In mapper code, I tried using SequenceFile.Reader with all possible
> > configurations (locally, distribtued) however, it can't find it. Are
> > sequence files supported in the distribued cache??
> >
> > Thank you,
> > Mark
>
>
>
> --
> Harsh J
>
Re: Reading Sequence File from Hadoop Distributed Cache ..
Posted by Harsh J <ha...@cloudera.com>.
Hi Mark,
DistributedCache files, when accessed from a Task, exist on the local
file system. You should make sure the SequenceFile.Reader tries to
read it with a LocalFS than a HDFS instance.
On Thu, Oct 11, 2012 at 5:15 AM, Mark Olimpiati <ma...@gmail.com> wrote:
> Hi,
>
> I'm storing sequence files in the distributed cache which seems to be
> stored somewher under each node's /tmp .../local/archive/ ... path.
>
> In mapper code, I tried using SequenceFile.Reader with all possible
> configurations (locally, distribtued) however, it can't find it. Are
> sequence files supported in the distribued cache??
>
> Thank you,
> Mark
--
Harsh J
Re: Reading Sequence File from Hadoop Distributed Cache ..
Posted by Harsh J <ha...@cloudera.com>.
Hi Mark,
DistributedCache files, when accessed from a Task, exist on the local
file system. You should make sure the SequenceFile.Reader tries to
read it with a LocalFS than a HDFS instance.
On Thu, Oct 11, 2012 at 5:15 AM, Mark Olimpiati <ma...@gmail.com> wrote:
> Hi,
>
> I'm storing sequence files in the distributed cache which seems to be
> stored somewher under each node's /tmp .../local/archive/ ... path.
>
> In mapper code, I tried using SequenceFile.Reader with all possible
> configurations (locally, distribtued) however, it can't find it. Are
> sequence files supported in the distribued cache??
>
> Thank you,
> Mark
--
Harsh J
Re: Reading Sequence File from Hadoop Distributed Cache ..
Posted by Harsh J <ha...@cloudera.com>.
Hi Mark,
DistributedCache files, when accessed from a Task, exist on the local
file system. You should make sure the SequenceFile.Reader tries to
read it with a LocalFS than a HDFS instance.
On Thu, Oct 11, 2012 at 5:15 AM, Mark Olimpiati <ma...@gmail.com> wrote:
> Hi,
>
> I'm storing sequence files in the distributed cache which seems to be
> stored somewher under each node's /tmp .../local/archive/ ... path.
>
> In mapper code, I tried using SequenceFile.Reader with all possible
> configurations (locally, distribtued) however, it can't find it. Are
> sequence files supported in the distribued cache??
>
> Thank you,
> Mark
--
Harsh J
Re: Reading Sequence File from Hadoop Distributed Cache ..
Posted by Harsh J <ha...@cloudera.com>.
Hi Mark,
DistributedCache files, when accessed from a Task, exist on the local
file system. You should make sure the SequenceFile.Reader tries to
read it with a LocalFS than a HDFS instance.
On Thu, Oct 11, 2012 at 5:15 AM, Mark Olimpiati <ma...@gmail.com> wrote:
> Hi,
>
> I'm storing sequence files in the distributed cache which seems to be
> stored somewher under each node's /tmp .../local/archive/ ... path.
>
> In mapper code, I tried using SequenceFile.Reader with all possible
> configurations (locally, distribtued) however, it can't find it. Are
> sequence files supported in the distribued cache??
>
> Thank you,
> Mark
--
Harsh J