You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-user@hadoop.apache.org by Hemanth Yamijala <yh...@thoughtworks.com> on 2013/01/25 02:26:26 UTC
Filesystem closed exception
Hi,
We are noticing a problem where we get a filesystem closed exception when a
map task is done and is finishing execution. By map task, I literally mean
the MapTask class of the map reduce code. Debugging this we found that the
mapper is getting a handle to the filesystem object and itself calling a
close on it. Because filesystem objects are cached, I believe the behaviour
is as expected in terms of the exception.
I just wanted to confirm that:
- if we do have a requirement to use a filesystem object in a mapper or
reducer, we should either not close it ourselves
- or (seems better to me) ask for a new version of the filesystem instance
by setting the fs.hdfs.impl.disable.cache property to true in job
configuration.
Also, does anyone know if this behaviour was any different in Hadoop 0.20 ?
For some context, this behaviour is actually seen in Oozie, which runs a
launcher mapper for a simple java action. Hence, the java action could very
well interact with a file system. I know this is probably better addressed
in Oozie context, but wanted to get the map reduce view of things.
Thanks,
Hemanth
Re: Filesystem closed exception
Posted by Hemanth Yamijala <yh...@thoughtworks.com>.
FS Caching is enabled on the cluster (i.e. the default is not changed).
Our code isn't actually mapper code, but a standalone java program being
run as part of Oozie. It just seemed confusing and not a very clear
strategy to leave unclosed resources. Hence my suggestion to get an
uncached FS handle for this use case alone. Note, I am not suggesting to
disable FS caching in general.
Thanks
Hemanth
On Thu, Jan 31, 2013 at 12:19 AM, Alejandro Abdelnur <tu...@cloudera.com>wrote:
> Hemanth,
>
> Is FS caching enabled or not in your cluster?
>
> A simple solution would be to modify your mapper code not to close the FS.
> It will go away when the task ends anyway.
>
> Thx
>
>
> On Thu, Jan 24, 2013 at 5:26 PM, Hemanth Yamijala <
> yhemanth@thoughtworks.com> wrote:
>
>> Hi,
>>
>> We are noticing a problem where we get a filesystem closed exception when
>> a map task is done and is finishing execution. By map task, I literally
>> mean the MapTask class of the map reduce code. Debugging this we found that
>> the mapper is getting a handle to the filesystem object and itself calling
>> a close on it. Because filesystem objects are cached, I believe the
>> behaviour is as expected in terms of the exception.
>>
>> I just wanted to confirm that:
>>
>> - if we do have a requirement to use a filesystem object in a mapper or
>> reducer, we should either not close it ourselves
>> - or (seems better to me) ask for a new version of the filesystem
>> instance by setting the fs.hdfs.impl.disable.cache property to true in job
>> configuration.
>>
>> Also, does anyone know if this behaviour was any different in Hadoop 0.20
>> ?
>>
>> For some context, this behaviour is actually seen in Oozie, which runs a
>> launcher mapper for a simple java action. Hence, the java action could very
>> well interact with a file system. I know this is probably better addressed
>> in Oozie context, but wanted to get the map reduce view of things.
>>
>>
>> Thanks,
>> Hemanth
>>
>
>
>
> --
> Alejandro
>
Re: Filesystem closed exception
Posted by Hemanth Yamijala <yh...@thoughtworks.com>.
FS Caching is enabled on the cluster (i.e. the default is not changed).
Our code isn't actually mapper code, but a standalone java program being
run as part of Oozie. It just seemed confusing and not a very clear
strategy to leave unclosed resources. Hence my suggestion to get an
uncached FS handle for this use case alone. Note, I am not suggesting to
disable FS caching in general.
Thanks
Hemanth
On Thu, Jan 31, 2013 at 12:19 AM, Alejandro Abdelnur <tu...@cloudera.com>wrote:
> Hemanth,
>
> Is FS caching enabled or not in your cluster?
>
> A simple solution would be to modify your mapper code not to close the FS.
> It will go away when the task ends anyway.
>
> Thx
>
>
> On Thu, Jan 24, 2013 at 5:26 PM, Hemanth Yamijala <
> yhemanth@thoughtworks.com> wrote:
>
>> Hi,
>>
>> We are noticing a problem where we get a filesystem closed exception when
>> a map task is done and is finishing execution. By map task, I literally
>> mean the MapTask class of the map reduce code. Debugging this we found that
>> the mapper is getting a handle to the filesystem object and itself calling
>> a close on it. Because filesystem objects are cached, I believe the
>> behaviour is as expected in terms of the exception.
>>
>> I just wanted to confirm that:
>>
>> - if we do have a requirement to use a filesystem object in a mapper or
>> reducer, we should either not close it ourselves
>> - or (seems better to me) ask for a new version of the filesystem
>> instance by setting the fs.hdfs.impl.disable.cache property to true in job
>> configuration.
>>
>> Also, does anyone know if this behaviour was any different in Hadoop 0.20
>> ?
>>
>> For some context, this behaviour is actually seen in Oozie, which runs a
>> launcher mapper for a simple java action. Hence, the java action could very
>> well interact with a file system. I know this is probably better addressed
>> in Oozie context, but wanted to get the map reduce view of things.
>>
>>
>> Thanks,
>> Hemanth
>>
>
>
>
> --
> Alejandro
>
Re: Filesystem closed exception
Posted by Hemanth Yamijala <yh...@thoughtworks.com>.
FS Caching is enabled on the cluster (i.e. the default is not changed).
Our code isn't actually mapper code, but a standalone java program being
run as part of Oozie. It just seemed confusing and not a very clear
strategy to leave unclosed resources. Hence my suggestion to get an
uncached FS handle for this use case alone. Note, I am not suggesting to
disable FS caching in general.
Thanks
Hemanth
On Thu, Jan 31, 2013 at 12:19 AM, Alejandro Abdelnur <tu...@cloudera.com>wrote:
> Hemanth,
>
> Is FS caching enabled or not in your cluster?
>
> A simple solution would be to modify your mapper code not to close the FS.
> It will go away when the task ends anyway.
>
> Thx
>
>
> On Thu, Jan 24, 2013 at 5:26 PM, Hemanth Yamijala <
> yhemanth@thoughtworks.com> wrote:
>
>> Hi,
>>
>> We are noticing a problem where we get a filesystem closed exception when
>> a map task is done and is finishing execution. By map task, I literally
>> mean the MapTask class of the map reduce code. Debugging this we found that
>> the mapper is getting a handle to the filesystem object and itself calling
>> a close on it. Because filesystem objects are cached, I believe the
>> behaviour is as expected in terms of the exception.
>>
>> I just wanted to confirm that:
>>
>> - if we do have a requirement to use a filesystem object in a mapper or
>> reducer, we should either not close it ourselves
>> - or (seems better to me) ask for a new version of the filesystem
>> instance by setting the fs.hdfs.impl.disable.cache property to true in job
>> configuration.
>>
>> Also, does anyone know if this behaviour was any different in Hadoop 0.20
>> ?
>>
>> For some context, this behaviour is actually seen in Oozie, which runs a
>> launcher mapper for a simple java action. Hence, the java action could very
>> well interact with a file system. I know this is probably better addressed
>> in Oozie context, but wanted to get the map reduce view of things.
>>
>>
>> Thanks,
>> Hemanth
>>
>
>
>
> --
> Alejandro
>
Re: Filesystem closed exception
Posted by Hemanth Yamijala <yh...@thoughtworks.com>.
FS Caching is enabled on the cluster (i.e. the default is not changed).
Our code isn't actually mapper code, but a standalone java program being
run as part of Oozie. It just seemed confusing and not a very clear
strategy to leave unclosed resources. Hence my suggestion to get an
uncached FS handle for this use case alone. Note, I am not suggesting to
disable FS caching in general.
Thanks
Hemanth
On Thu, Jan 31, 2013 at 12:19 AM, Alejandro Abdelnur <tu...@cloudera.com>wrote:
> Hemanth,
>
> Is FS caching enabled or not in your cluster?
>
> A simple solution would be to modify your mapper code not to close the FS.
> It will go away when the task ends anyway.
>
> Thx
>
>
> On Thu, Jan 24, 2013 at 5:26 PM, Hemanth Yamijala <
> yhemanth@thoughtworks.com> wrote:
>
>> Hi,
>>
>> We are noticing a problem where we get a filesystem closed exception when
>> a map task is done and is finishing execution. By map task, I literally
>> mean the MapTask class of the map reduce code. Debugging this we found that
>> the mapper is getting a handle to the filesystem object and itself calling
>> a close on it. Because filesystem objects are cached, I believe the
>> behaviour is as expected in terms of the exception.
>>
>> I just wanted to confirm that:
>>
>> - if we do have a requirement to use a filesystem object in a mapper or
>> reducer, we should either not close it ourselves
>> - or (seems better to me) ask for a new version of the filesystem
>> instance by setting the fs.hdfs.impl.disable.cache property to true in job
>> configuration.
>>
>> Also, does anyone know if this behaviour was any different in Hadoop 0.20
>> ?
>>
>> For some context, this behaviour is actually seen in Oozie, which runs a
>> launcher mapper for a simple java action. Hence, the java action could very
>> well interact with a file system. I know this is probably better addressed
>> in Oozie context, but wanted to get the map reduce view of things.
>>
>>
>> Thanks,
>> Hemanth
>>
>
>
>
> --
> Alejandro
>
Re: Filesystem closed exception
Posted by Alejandro Abdelnur <tu...@cloudera.com>.
Hemanth,
Is FS caching enabled or not in your cluster?
A simple solution would be to modify your mapper code not to close the FS.
It will go away when the task ends anyway.
Thx
On Thu, Jan 24, 2013 at 5:26 PM, Hemanth Yamijala <yhemanth@thoughtworks.com
> wrote:
> Hi,
>
> We are noticing a problem where we get a filesystem closed exception when
> a map task is done and is finishing execution. By map task, I literally
> mean the MapTask class of the map reduce code. Debugging this we found that
> the mapper is getting a handle to the filesystem object and itself calling
> a close on it. Because filesystem objects are cached, I believe the
> behaviour is as expected in terms of the exception.
>
> I just wanted to confirm that:
>
> - if we do have a requirement to use a filesystem object in a mapper or
> reducer, we should either not close it ourselves
> - or (seems better to me) ask for a new version of the filesystem instance
> by setting the fs.hdfs.impl.disable.cache property to true in job
> configuration.
>
> Also, does anyone know if this behaviour was any different in Hadoop 0.20 ?
>
> For some context, this behaviour is actually seen in Oozie, which runs a
> launcher mapper for a simple java action. Hence, the java action could very
> well interact with a file system. I know this is probably better addressed
> in Oozie context, but wanted to get the map reduce view of things.
>
>
> Thanks,
> Hemanth
>
--
Alejandro
Re: Filesystem closed exception
Posted by Alejandro Abdelnur <tu...@cloudera.com>.
Hemanth,
Is FS caching enabled or not in your cluster?
A simple solution would be to modify your mapper code not to close the FS.
It will go away when the task ends anyway.
Thx
On Thu, Jan 24, 2013 at 5:26 PM, Hemanth Yamijala <yhemanth@thoughtworks.com
> wrote:
> Hi,
>
> We are noticing a problem where we get a filesystem closed exception when
> a map task is done and is finishing execution. By map task, I literally
> mean the MapTask class of the map reduce code. Debugging this we found that
> the mapper is getting a handle to the filesystem object and itself calling
> a close on it. Because filesystem objects are cached, I believe the
> behaviour is as expected in terms of the exception.
>
> I just wanted to confirm that:
>
> - if we do have a requirement to use a filesystem object in a mapper or
> reducer, we should either not close it ourselves
> - or (seems better to me) ask for a new version of the filesystem instance
> by setting the fs.hdfs.impl.disable.cache property to true in job
> configuration.
>
> Also, does anyone know if this behaviour was any different in Hadoop 0.20 ?
>
> For some context, this behaviour is actually seen in Oozie, which runs a
> launcher mapper for a simple java action. Hence, the java action could very
> well interact with a file system. I know this is probably better addressed
> in Oozie context, but wanted to get the map reduce view of things.
>
>
> Thanks,
> Hemanth
>
--
Alejandro
Re: Filesystem closed exception
Posted by Alejandro Abdelnur <tu...@cloudera.com>.
Hemanth,
Is FS caching enabled or not in your cluster?
A simple solution would be to modify your mapper code not to close the FS.
It will go away when the task ends anyway.
Thx
On Thu, Jan 24, 2013 at 5:26 PM, Hemanth Yamijala <yhemanth@thoughtworks.com
> wrote:
> Hi,
>
> We are noticing a problem where we get a filesystem closed exception when
> a map task is done and is finishing execution. By map task, I literally
> mean the MapTask class of the map reduce code. Debugging this we found that
> the mapper is getting a handle to the filesystem object and itself calling
> a close on it. Because filesystem objects are cached, I believe the
> behaviour is as expected in terms of the exception.
>
> I just wanted to confirm that:
>
> - if we do have a requirement to use a filesystem object in a mapper or
> reducer, we should either not close it ourselves
> - or (seems better to me) ask for a new version of the filesystem instance
> by setting the fs.hdfs.impl.disable.cache property to true in job
> configuration.
>
> Also, does anyone know if this behaviour was any different in Hadoop 0.20 ?
>
> For some context, this behaviour is actually seen in Oozie, which runs a
> launcher mapper for a simple java action. Hence, the java action could very
> well interact with a file system. I know this is probably better addressed
> in Oozie context, but wanted to get the map reduce view of things.
>
>
> Thanks,
> Hemanth
>
--
Alejandro
Re: Filesystem closed exception
Posted by Hemanth Yamijala <yh...@thoughtworks.com>.
Thanks, Harsh. Particularly for pointing out HADOOP-7973.
On Fri, Jan 25, 2013 at 11:51 AM, Harsh J <ha...@cloudera.com> wrote:
> It is pretty much the same in 0.20.x as well, IIRC. Your two points
> are also correct (for a fix to this). Also see:
> https://issues.apache.org/jira/browse/HADOOP-7973.
>
> On Fri, Jan 25, 2013 at 6:56 AM, Hemanth Yamijala
> <yh...@thoughtworks.com> wrote:
> > Hi,
> >
> > We are noticing a problem where we get a filesystem closed exception
> when a
> > map task is done and is finishing execution. By map task, I literally
> mean
> > the MapTask class of the map reduce code. Debugging this we found that
> the
> > mapper is getting a handle to the filesystem object and itself calling a
> > close on it. Because filesystem objects are cached, I believe the
> behaviour
> > is as expected in terms of the exception.
> >
> > I just wanted to confirm that:
> >
> > - if we do have a requirement to use a filesystem object in a mapper or
> > reducer, we should either not close it ourselves
> > - or (seems better to me) ask for a new version of the filesystem
> instance
> > by setting the fs.hdfs.impl.disable.cache property to true in job
> > configuration.
> >
> > Also, does anyone know if this behaviour was any different in Hadoop
> 0.20 ?
> >
> > For some context, this behaviour is actually seen in Oozie, which runs a
> > launcher mapper for a simple java action. Hence, the java action could
> very
> > well interact with a file system. I know this is probably better
> addressed
> > in Oozie context, but wanted to get the map reduce view of things.
> >
> >
> > Thanks,
> > Hemanth
>
>
>
> --
> Harsh J
>
Re: Filesystem closed exception
Posted by Hemanth Yamijala <yh...@thoughtworks.com>.
Thanks, Harsh. Particularly for pointing out HADOOP-7973.
On Fri, Jan 25, 2013 at 11:51 AM, Harsh J <ha...@cloudera.com> wrote:
> It is pretty much the same in 0.20.x as well, IIRC. Your two points
> are also correct (for a fix to this). Also see:
> https://issues.apache.org/jira/browse/HADOOP-7973.
>
> On Fri, Jan 25, 2013 at 6:56 AM, Hemanth Yamijala
> <yh...@thoughtworks.com> wrote:
> > Hi,
> >
> > We are noticing a problem where we get a filesystem closed exception
> when a
> > map task is done and is finishing execution. By map task, I literally
> mean
> > the MapTask class of the map reduce code. Debugging this we found that
> the
> > mapper is getting a handle to the filesystem object and itself calling a
> > close on it. Because filesystem objects are cached, I believe the
> behaviour
> > is as expected in terms of the exception.
> >
> > I just wanted to confirm that:
> >
> > - if we do have a requirement to use a filesystem object in a mapper or
> > reducer, we should either not close it ourselves
> > - or (seems better to me) ask for a new version of the filesystem
> instance
> > by setting the fs.hdfs.impl.disable.cache property to true in job
> > configuration.
> >
> > Also, does anyone know if this behaviour was any different in Hadoop
> 0.20 ?
> >
> > For some context, this behaviour is actually seen in Oozie, which runs a
> > launcher mapper for a simple java action. Hence, the java action could
> very
> > well interact with a file system. I know this is probably better
> addressed
> > in Oozie context, but wanted to get the map reduce view of things.
> >
> >
> > Thanks,
> > Hemanth
>
>
>
> --
> Harsh J
>
Re: Filesystem closed exception
Posted by Hemanth Yamijala <yh...@thoughtworks.com>.
Thanks, Harsh. Particularly for pointing out HADOOP-7973.
On Fri, Jan 25, 2013 at 11:51 AM, Harsh J <ha...@cloudera.com> wrote:
> It is pretty much the same in 0.20.x as well, IIRC. Your two points
> are also correct (for a fix to this). Also see:
> https://issues.apache.org/jira/browse/HADOOP-7973.
>
> On Fri, Jan 25, 2013 at 6:56 AM, Hemanth Yamijala
> <yh...@thoughtworks.com> wrote:
> > Hi,
> >
> > We are noticing a problem where we get a filesystem closed exception
> when a
> > map task is done and is finishing execution. By map task, I literally
> mean
> > the MapTask class of the map reduce code. Debugging this we found that
> the
> > mapper is getting a handle to the filesystem object and itself calling a
> > close on it. Because filesystem objects are cached, I believe the
> behaviour
> > is as expected in terms of the exception.
> >
> > I just wanted to confirm that:
> >
> > - if we do have a requirement to use a filesystem object in a mapper or
> > reducer, we should either not close it ourselves
> > - or (seems better to me) ask for a new version of the filesystem
> instance
> > by setting the fs.hdfs.impl.disable.cache property to true in job
> > configuration.
> >
> > Also, does anyone know if this behaviour was any different in Hadoop
> 0.20 ?
> >
> > For some context, this behaviour is actually seen in Oozie, which runs a
> > launcher mapper for a simple java action. Hence, the java action could
> very
> > well interact with a file system. I know this is probably better
> addressed
> > in Oozie context, but wanted to get the map reduce view of things.
> >
> >
> > Thanks,
> > Hemanth
>
>
>
> --
> Harsh J
>
Re: Filesystem closed exception
Posted by Hemanth Yamijala <yh...@thoughtworks.com>.
Thanks, Harsh. Particularly for pointing out HADOOP-7973.
On Fri, Jan 25, 2013 at 11:51 AM, Harsh J <ha...@cloudera.com> wrote:
> It is pretty much the same in 0.20.x as well, IIRC. Your two points
> are also correct (for a fix to this). Also see:
> https://issues.apache.org/jira/browse/HADOOP-7973.
>
> On Fri, Jan 25, 2013 at 6:56 AM, Hemanth Yamijala
> <yh...@thoughtworks.com> wrote:
> > Hi,
> >
> > We are noticing a problem where we get a filesystem closed exception
> when a
> > map task is done and is finishing execution. By map task, I literally
> mean
> > the MapTask class of the map reduce code. Debugging this we found that
> the
> > mapper is getting a handle to the filesystem object and itself calling a
> > close on it. Because filesystem objects are cached, I believe the
> behaviour
> > is as expected in terms of the exception.
> >
> > I just wanted to confirm that:
> >
> > - if we do have a requirement to use a filesystem object in a mapper or
> > reducer, we should either not close it ourselves
> > - or (seems better to me) ask for a new version of the filesystem
> instance
> > by setting the fs.hdfs.impl.disable.cache property to true in job
> > configuration.
> >
> > Also, does anyone know if this behaviour was any different in Hadoop
> 0.20 ?
> >
> > For some context, this behaviour is actually seen in Oozie, which runs a
> > launcher mapper for a simple java action. Hence, the java action could
> very
> > well interact with a file system. I know this is probably better
> addressed
> > in Oozie context, but wanted to get the map reduce view of things.
> >
> >
> > Thanks,
> > Hemanth
>
>
>
> --
> Harsh J
>
Re: Filesystem closed exception
Posted by Harsh J <ha...@cloudera.com>.
It is pretty much the same in 0.20.x as well, IIRC. Your two points
are also correct (for a fix to this). Also see:
https://issues.apache.org/jira/browse/HADOOP-7973.
On Fri, Jan 25, 2013 at 6:56 AM, Hemanth Yamijala
<yh...@thoughtworks.com> wrote:
> Hi,
>
> We are noticing a problem where we get a filesystem closed exception when a
> map task is done and is finishing execution. By map task, I literally mean
> the MapTask class of the map reduce code. Debugging this we found that the
> mapper is getting a handle to the filesystem object and itself calling a
> close on it. Because filesystem objects are cached, I believe the behaviour
> is as expected in terms of the exception.
>
> I just wanted to confirm that:
>
> - if we do have a requirement to use a filesystem object in a mapper or
> reducer, we should either not close it ourselves
> - or (seems better to me) ask for a new version of the filesystem instance
> by setting the fs.hdfs.impl.disable.cache property to true in job
> configuration.
>
> Also, does anyone know if this behaviour was any different in Hadoop 0.20 ?
>
> For some context, this behaviour is actually seen in Oozie, which runs a
> launcher mapper for a simple java action. Hence, the java action could very
> well interact with a file system. I know this is probably better addressed
> in Oozie context, but wanted to get the map reduce view of things.
>
>
> Thanks,
> Hemanth
--
Harsh J
Re: Filesystem closed exception
Posted by Harsh J <ha...@cloudera.com>.
It is pretty much the same in 0.20.x as well, IIRC. Your two points
are also correct (for a fix to this). Also see:
https://issues.apache.org/jira/browse/HADOOP-7973.
On Fri, Jan 25, 2013 at 6:56 AM, Hemanth Yamijala
<yh...@thoughtworks.com> wrote:
> Hi,
>
> We are noticing a problem where we get a filesystem closed exception when a
> map task is done and is finishing execution. By map task, I literally mean
> the MapTask class of the map reduce code. Debugging this we found that the
> mapper is getting a handle to the filesystem object and itself calling a
> close on it. Because filesystem objects are cached, I believe the behaviour
> is as expected in terms of the exception.
>
> I just wanted to confirm that:
>
> - if we do have a requirement to use a filesystem object in a mapper or
> reducer, we should either not close it ourselves
> - or (seems better to me) ask for a new version of the filesystem instance
> by setting the fs.hdfs.impl.disable.cache property to true in job
> configuration.
>
> Also, does anyone know if this behaviour was any different in Hadoop 0.20 ?
>
> For some context, this behaviour is actually seen in Oozie, which runs a
> launcher mapper for a simple java action. Hence, the java action could very
> well interact with a file system. I know this is probably better addressed
> in Oozie context, but wanted to get the map reduce view of things.
>
>
> Thanks,
> Hemanth
--
Harsh J
Re: Filesystem closed exception
Posted by Harsh J <ha...@cloudera.com>.
It is pretty much the same in 0.20.x as well, IIRC. Your two points
are also correct (for a fix to this). Also see:
https://issues.apache.org/jira/browse/HADOOP-7973.
On Fri, Jan 25, 2013 at 6:56 AM, Hemanth Yamijala
<yh...@thoughtworks.com> wrote:
> Hi,
>
> We are noticing a problem where we get a filesystem closed exception when a
> map task is done and is finishing execution. By map task, I literally mean
> the MapTask class of the map reduce code. Debugging this we found that the
> mapper is getting a handle to the filesystem object and itself calling a
> close on it. Because filesystem objects are cached, I believe the behaviour
> is as expected in terms of the exception.
>
> I just wanted to confirm that:
>
> - if we do have a requirement to use a filesystem object in a mapper or
> reducer, we should either not close it ourselves
> - or (seems better to me) ask for a new version of the filesystem instance
> by setting the fs.hdfs.impl.disable.cache property to true in job
> configuration.
>
> Also, does anyone know if this behaviour was any different in Hadoop 0.20 ?
>
> For some context, this behaviour is actually seen in Oozie, which runs a
> launcher mapper for a simple java action. Hence, the java action could very
> well interact with a file system. I know this is probably better addressed
> in Oozie context, but wanted to get the map reduce view of things.
>
>
> Thanks,
> Hemanth
--
Harsh J
Re: Filesystem closed exception
Posted by Harsh J <ha...@cloudera.com>.
It is pretty much the same in 0.20.x as well, IIRC. Your two points
are also correct (for a fix to this). Also see:
https://issues.apache.org/jira/browse/HADOOP-7973.
On Fri, Jan 25, 2013 at 6:56 AM, Hemanth Yamijala
<yh...@thoughtworks.com> wrote:
> Hi,
>
> We are noticing a problem where we get a filesystem closed exception when a
> map task is done and is finishing execution. By map task, I literally mean
> the MapTask class of the map reduce code. Debugging this we found that the
> mapper is getting a handle to the filesystem object and itself calling a
> close on it. Because filesystem objects are cached, I believe the behaviour
> is as expected in terms of the exception.
>
> I just wanted to confirm that:
>
> - if we do have a requirement to use a filesystem object in a mapper or
> reducer, we should either not close it ourselves
> - or (seems better to me) ask for a new version of the filesystem instance
> by setting the fs.hdfs.impl.disable.cache property to true in job
> configuration.
>
> Also, does anyone know if this behaviour was any different in Hadoop 0.20 ?
>
> For some context, this behaviour is actually seen in Oozie, which runs a
> launcher mapper for a simple java action. Hence, the java action could very
> well interact with a file system. I know this is probably better addressed
> in Oozie context, but wanted to get the map reduce view of things.
>
>
> Thanks,
> Hemanth
--
Harsh J
Re: Filesystem closed exception
Posted by Alejandro Abdelnur <tu...@cloudera.com>.
Hemanth,
Is FS caching enabled or not in your cluster?
A simple solution would be to modify your mapper code not to close the FS.
It will go away when the task ends anyway.
Thx
On Thu, Jan 24, 2013 at 5:26 PM, Hemanth Yamijala <yhemanth@thoughtworks.com
> wrote:
> Hi,
>
> We are noticing a problem where we get a filesystem closed exception when
> a map task is done and is finishing execution. By map task, I literally
> mean the MapTask class of the map reduce code. Debugging this we found that
> the mapper is getting a handle to the filesystem object and itself calling
> a close on it. Because filesystem objects are cached, I believe the
> behaviour is as expected in terms of the exception.
>
> I just wanted to confirm that:
>
> - if we do have a requirement to use a filesystem object in a mapper or
> reducer, we should either not close it ourselves
> - or (seems better to me) ask for a new version of the filesystem instance
> by setting the fs.hdfs.impl.disable.cache property to true in job
> configuration.
>
> Also, does anyone know if this behaviour was any different in Hadoop 0.20 ?
>
> For some context, this behaviour is actually seen in Oozie, which runs a
> launcher mapper for a simple java action. Hence, the java action could very
> well interact with a file system. I know this is probably better addressed
> in Oozie context, but wanted to get the map reduce view of things.
>
>
> Thanks,
> Hemanth
>
--
Alejandro