You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@nutch.apache.org by "Stefan Groschupf (JIRA)" <ji...@apache.org> on 2006/02/15 12:11:08 UTC

[jira] Created: (NUTCH-211) FetchedSegments leave readers open

FetchedSegments leave readers open 
-----------------------------------

         Key: NUTCH-211
         URL: http://issues.apache.org/jira/browse/NUTCH-211
     Project: Nutch
        Type: Bug
    Versions: 0.8-dev    
    Reporter: Stefan Groschupf
    Priority: Critical
     Fix For: 0.8-dev


I have a case here where the NutchBean is instantiated more than once, however I do cache the nutch bean, but in some situations the bean needs to re created. The problem is the  FetchedSegments leaves open all reads it uses. So a nio Exception is thrown as soon I try to create the NutchBean again. 
I would suggest to add a close method to  FetchedSegments and all involved objects to be able cleanly shutting down the NutchBean.
Any comments? Would a patch be welcome?

Caused by: java.nio.channels.ClosedChannelException
at sun.nio.ch.FileChannelImpl.ensureOpen(FileChannelImpl.java:89)
at sun.nio.ch.FileChannelImpl.position(FileChannelImpl.java:272)
at org.apache.nutch.fs.LocalFileSystem$LocalNFSFileInputStream.seek(LocalFileSystem.java:83)
at org.apache.nutch.fs.NFSDataInputStream$Checker.seek(NFSDataInputStream.java:66)
at org.apache.nutch.fs.NFSDataInputStream$PositionCache.seek(NFSDataInputStream.java:162)
at org.apache.nutch.fs.NFSDataInputStream$Buffer.seek(NFSDataInputStream.java:191)
at org.apache.nutch.fs.NFSDataInputStream.seek(NFSDataInputStream.java:241)
at org.apache.nutch.io.SequenceFile$Reader.seek(SequenceFile.java:403)
at org.apache.nutch.io.MapFile$Reader.seek(MapFile.java:329)
at org.apache.nutch.io.MapFile$Reader.get(MapFile.java:374)
at org.apache.nutch.mapred.MapFileOutputFormat.getEntry(MapFileOutputFormat.java:76)
at org.apache.nutch.searcher.FetchedSegments$Segment.getEntry(FetchedSegments.java:93)
at org.apache.nutch.searcher.FetchedSegments$Segment.getParseText(FetchedSegments.java:84)
at org.apache.nutch.searcher.FetchedSegments.getSummary(FetchedSegments.java:147)
at org.apache.nutch.searcher.NutchBean.getSummary(NutchBean.java:321)


-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
   http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
   http://www.atlassian.com/software/jira


[jira] Updated: (NUTCH-211) FetchedSegments leave readers open

Posted by "Stefan Groschupf (JIRA)" <ji...@apache.org>.
     [ http://issues.apache.org/jira/browse/NUTCH-211?page=all ]

Stefan Groschupf updated NUTCH-211:
-----------------------------------

    Attachment: closeFetchSegments.patch

NutchBean, FetchedSegments,FetchedSegments.Segment IndexSearcher and HitContent now extends / implements the hadoop Closeable interface.
A NutchBean should now be able to cleanly shutdown without leaving open file handles or cocket clients.

> FetchedSegments leave readers open
> ----------------------------------
>
>          Key: NUTCH-211
>          URL: http://issues.apache.org/jira/browse/NUTCH-211
>      Project: Nutch
>         Type: Bug
>     Versions: 0.8-dev
>     Reporter: Stefan Groschupf
>     Assignee: Stefan Groschupf
>     Priority: Critical
>      Fix For: 0.8-dev
>  Attachments: closeFetchSegments.patch
>
> I have a case here where the NutchBean is instantiated more than once, however I do cache the nutch bean, but in some situations the bean needs to re created. The problem is the  FetchedSegments leaves open all reads it uses. So a nio Exception is thrown as soon I try to create the NutchBean again. 
> I would suggest to add a close method to  FetchedSegments and all involved objects to be able cleanly shutting down the NutchBean.
> Any comments? Would a patch be welcome?
> Caused by: java.nio.channels.ClosedChannelException
> at sun.nio.ch.FileChannelImpl.ensureOpen(FileChannelImpl.java:89)
> at sun.nio.ch.FileChannelImpl.position(FileChannelImpl.java:272)
> at org.apache.nutch.fs.LocalFileSystem$LocalNFSFileInputStream.seek(LocalFileSystem.java:83)
> at org.apache.nutch.fs.NFSDataInputStream$Checker.seek(NFSDataInputStream.java:66)
> at org.apache.nutch.fs.NFSDataInputStream$PositionCache.seek(NFSDataInputStream.java:162)
> at org.apache.nutch.fs.NFSDataInputStream$Buffer.seek(NFSDataInputStream.java:191)
> at org.apache.nutch.fs.NFSDataInputStream.seek(NFSDataInputStream.java:241)
> at org.apache.nutch.io.SequenceFile$Reader.seek(SequenceFile.java:403)
> at org.apache.nutch.io.MapFile$Reader.seek(MapFile.java:329)
> at org.apache.nutch.io.MapFile$Reader.get(MapFile.java:374)
> at org.apache.nutch.mapred.MapFileOutputFormat.getEntry(MapFileOutputFormat.java:76)
> at org.apache.nutch.searcher.FetchedSegments$Segment.getEntry(FetchedSegments.java:93)
> at org.apache.nutch.searcher.FetchedSegments$Segment.getParseText(FetchedSegments.java:84)
> at org.apache.nutch.searcher.FetchedSegments.getSummary(FetchedSegments.java:147)
> at org.apache.nutch.searcher.NutchBean.getSummary(NutchBean.java:321)

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
   http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
   http://www.atlassian.com/software/jira


Re: Global locking

Posted by Gal Nitzan <gn...@usa.net>.
well, at the moment it solve the problem I mentioned yesterday where all
tasktrackers will access the same site with hadoop. it seems that the
use of job.setBoolean("mapred.speculative.execution", false); didn't
help and I'm not sure why.

However, though it is one more software it removes the need for special
treatment for fetcher, i.e. special fetch lists built by the generator.
So now fetcher/tasktracker suppose to access politely to hosts but still
its list contains various hosts. Sometimes I noticed that generator
created a fetchlist where (only 2 hosts in the seed) were put in the
same fetchlist which made only one tasktracker work instead of two.

I'm sorry if It sound a little confusing :) or unreasonable... :)

Gal



On Thu, 2006-02-16 at 13:47 -0800, Doug Cutting wrote:
> Gal Nitzan wrote:
> > I have implemented a down and dirty Global Locking:
> >  [ ... ]
> > 
> > I changed FetcherThread constructor to create an instance of
> > SyncManager.
> > 
> > And in also in the run method I try to get a lock on the host. If not
> > successful I add the url into a ListArray<key,datum> for a later
> > processing...
> > 
> > I also changed generator to put each url into a separate array so all
> > fetchlists are even.
> 
> What problem does this fix?
> 
> Doug
> 



Re: Global locking

Posted by Doug Cutting <cu...@apache.org>.
Gal Nitzan wrote:
> I have implemented a down and dirty Global Locking:
>  [ ... ]
> 
> I changed FetcherThread constructor to create an instance of
> SyncManager.
> 
> And in also in the run method I try to get a lock on the host. If not
> successful I add the url into a ListArray<key,datum> for a later
> processing...
> 
> I also changed generator to put each url into a separate array so all
> fetchlists are even.

What problem does this fix?

Doug

Global locking

Posted by Gal Nitzan <gn...@usa.net>.
I have implemented a down and dirty Global Locking:

I am currently testing it but I would like to get other people idea on
this:

I used RMI for this purpose:

A RMI server which implements two methods {
boolean lock(String urlString);
void unlock(String urlString);
}

the server holds a map<key,val> where key is an Integer(host hash) the
val is a very simplistic class:

public class LockObj {
  private int hash;
  private long start;
  private long timeout;
  private int max_locks;
  private int locks = 0;
  private Object sync_obj = new Object();

  public LockObj(int hash, long timeout, int max_locks) {
    this.hash = hash;
    this.timeout = timeout;
    start = new Date().getTime();
    this.max_locks = max_locks;
  }

  public synchronized boolean lock() {
    boolean ret = false;

    if (locks+1 < max_locks) {
      synchronized(sync_obj) {
        locks++;
      }
      ret = true;
    }
    return ret;
  }

  public synchronized void unlock() {
    if (locks > 0) {
      synchronized(sync_obj) {
        locks--;
      }
    }
  }

  public int locks() {
    return locks;
  }

  // convert the host part of a url to hash
  // if url exception. use the string input for hash
  public static int make_hash(String urlString) {
    URL url = null;
    try {
      url = new URL(urlString);
    } catch (MalformedURLException e) {
    }

    return (url==null ? urlString : url.getHost()).hashCode();
  }

  // check if this object timeout has reached.
  // later implement a listener event
  public boolean timeout_reached() {
    long current = new Date().getTime();

    return (current - start) > timeout;
  }

  // free all
  public void unlock_all() {
    synchronized(sync_obj) {
      while (locks != 0)
        locks--;
    }
  }

  public int hash() {
    return hash;
  }
}

not the prettiest thing but just finished the first barrier... it
worked!!!


I changed FetcherThread constructor to create an instance of
SyncManager.

And in also in the run method I try to get a lock on the host. If not
successful I add the url into a ListArray<key,datum> for a later
processing...

I also changed generator to put each url into a separate array so all
fetchlists are even.

Would appreciate your comments and any way to improve.

The RMI is a little cumbersome but hay... for now it works for 5 task
trackers without a problem (so it seems) :)


Gal




On Wed, 2006-02-15 at 14:55 -0800, Doug Cutting wrote:
> Andrzej Bialecki wrote:
> > (FYI: if you wonder how it was working before, the trick was to generate 
> > just 1 split for the fetch job, which then lead to just one task being 
> > created for any input fetchlist.
> 
> I don't think that's right.  The generator uses setNumReduceTasks() to 
> the desired number of fetch tasks, to control how many host-disjoint 
> fetchlists are generated.  Then the fetcher does not permit input files 
> to be split, so that fetch tasks remain host-disjoint.  So lots of 
> splits can be generated, by default one per mapred.map.tasks, permitting 
> lots of parallel fetching.
> 
> This should still work.  If it does not, I'd be interested to hear more 
> details.
> 
> Doug
> 



Re: All tasktrackers access same site at the same time (hadoop) please help

Posted by Doug Cutting <cu...@apache.org>.
Andrzej Bialecki wrote:
> (FYI: if you wonder how it was working before, the trick was to generate 
> just 1 split for the fetch job, which then lead to just one task being 
> created for any input fetchlist.

I don't think that's right.  The generator uses setNumReduceTasks() to 
the desired number of fetch tasks, to control how many host-disjoint 
fetchlists are generated.  Then the fetcher does not permit input files 
to be split, so that fetch tasks remain host-disjoint.  So lots of 
splits can be generated, by default one per mapred.map.tasks, permitting 
lots of parallel fetching.

This should still work.  If it does not, I'd be interested to hear more 
details.

Doug

Re: All tasktrackers access same site at the same time (hadoop) please help

Posted by Andrzej Bialecki <ab...@getopt.org>.
Gal Nitzan wrote:
> Hi,
>
> Just installed 0-8 with hadoop from trunk.
>
> I noticed all tasktrackers are participating in the fetch.
>
> I have only one site in the injected seed file
>
> I have 5 tasktrackers all except one access the same site.
>
> I am using nu0.8 dev with hadoop.
>
> Please, any idea?
>   

Hadoop doesn't have any mechanism for coordinating simultaneous access 
to resources across the cluster (global locking). I described the 
problem on the hadoop-dev list, no comments yet...

(FYI: if you wonder how it was working before, the trick was to generate 
just 1 split for the fetch job, which then lead to just one task being 
created for any input fetchlist. This was a hack that apparently stopped 
working after Hadoop was moved to its own codebase; the proper solution 
is to implement a global lock manager).

-- 
Best regards,
Andrzej Bialecki     <><
 ___. ___ ___ ___ _ _   __________________________________
[__ || __|__/|__||\/|  Information Retrieval, Semantic Web
___|||__||  \|  ||  |  Embedded Unix, System Integration
http://www.sigram.com  Contact: info at sigram dot com



Re: All tasktrackers access same site at the same time (hadoop) please help

Posted by Gal Nitzan <gn...@usa.net>.
Thanks for the prompt reply.

I have updated Fetcher.java and hadoop.jar from trunk but I still get
the aforementioned behavior.



On Wed, 2006-02-15 at 15:02 -0800, Doug Cutting wrote:
> Gal Nitzan wrote:
> > I noticed all tasktrackers are participating in the fetch.
> > 
> > I have only one site in the injected seed file
> > 
> > I have 5 tasktrackers all except one access the same site.
> 
> I just fixed a bug related to this.  Please try updating.
> 
> The problem was that MapReduce recently started supporting speculative 
> execution, where, if some tasks appear to be executing slowly and there 
> are idle nodes, then these tasks automatically are run in parallel on 
> another node, and the results of the first that finishes are used.  But 
> this is not appropriate for fetching.  So I just added a mechanism to 
> Hadoop to disable it and then disabled it in the Fetcher.
> 
> Note also that the slaves file is now located in the conf/ directory, as 
> is a new file named hadoop-env.sh.  This contains all relevant 
> environment variables, so that we no longer have to rely on ssh's 
> environment passing feature.
> 
> Doug
> 



Re: All tasktrackers access same site at the same time (hadoop) please help

Posted by Doug Cutting <cu...@apache.org>.
Gal Nitzan wrote:
> I noticed all tasktrackers are participating in the fetch.
> 
> I have only one site in the injected seed file
> 
> I have 5 tasktrackers all except one access the same site.

I just fixed a bug related to this.  Please try updating.

The problem was that MapReduce recently started supporting speculative 
execution, where, if some tasks appear to be executing slowly and there 
are idle nodes, then these tasks automatically are run in parallel on 
another node, and the results of the first that finishes are used.  But 
this is not appropriate for fetching.  So I just added a mechanism to 
Hadoop to disable it and then disabled it in the Fetcher.

Note also that the slaves file is now located in the conf/ directory, as 
is a new file named hadoop-env.sh.  This contains all relevant 
environment variables, so that we no longer have to rely on ssh's 
environment passing feature.

Doug

All tasktrackers access same site at the same time (hadoop) please help

Posted by Gal Nitzan <gn...@usa.net>.
Hi,

Just installed 0-8 with hadoop from trunk.

I noticed all tasktrackers are participating in the fetch.

I have only one site in the injected seed file

I have 5 tasktrackers all except one access the same site.

I am using nu0.8 dev with hadoop.

Please, any idea?

Thanks.





[jira] Commented: (NUTCH-211) FetchedSegments leave readers open

Posted by "Stefan Groschupf (JIRA)" <ji...@apache.org>.
    [ http://issues.apache.org/jira/browse/NUTCH-211?page=comments#action_12366533 ] 

Stefan Groschupf commented on NUTCH-211:
----------------------------------------

I'm already in process of creating the patch, however I spend some more time to find the real problem source. It is just a guess, but could it be that the problem source is that the NutchBean.getDetail method is concurently invoked by different threads, what is throwing a java.nio.channels.ClosedByInterruptException? Shouldn't be the access to a nio channel syncronized? Means shouldn't be the LocalFileSystem$LocalNFSFileInputStream.seek method syncronized, or is that nonsense? May this by now more a hadoop releated question, but it is somehow part of this problem. Any hints?


Caused by: java.nio.channels.ClosedByInterruptException
        at java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:184)
        at sun.nio.ch.FileChannelImpl.position(FileChannelImpl.java:289)
        at org.apache.nutch.fs.LocalFileSystem$LocalNFSFileInputStream.seek(LocalFileSystem.java:83)
        at org.apache.nutch.fs.NFSDataInputStream$Checker.seek(NFSDataInputStream.java:66)
        at org.apache.nutch.fs.NFSDataInputStream$PositionCache.seek(NFSDataInputStream.java:162)
        at org.apache.nutch.fs.NFSDataInputStream$Buffer.seek(NFSDataInputStream.java:191)
        at org.apache.nutch.fs.NFSDataInputStream.seek(NFSDataInputStream.java:241)
        at org.apache.nutch.io.SequenceFile$Reader.seek(SequenceFile.java:403)
        at org.apache.nutch.io.MapFile$Reader.seek(MapFile.java:329)
        at org.apache.nutch.io.MapFile$Reader.get(MapFile.java:374)
        at org.apache.nutch.mapred.MapFileOutputFormat.getEntry(MapFileOutputFormat.java:76)
        at org.apache.nutch.searcher.FetchedSegments$Segment.getEntry(FetchedSegments.java:93)
        at org.apache.nutch.searcher.FetchedSegments$Segment.getParseText(FetchedSegments.java:84)
        at org.apache.nutch.searcher.FetchedSegments.getSummary(FetchedSegments.java:147)
        at org.apache.nutch.searcher.NutchBean.getSummary(NutchBean.java:321)
        at de.ingrid.iplug.se.NutchSearcher.getDetail(NutchSearcher.java:219)

> FetchedSegments leave readers open
> ----------------------------------
>
>          Key: NUTCH-211
>          URL: http://issues.apache.org/jira/browse/NUTCH-211
>      Project: Nutch
>         Type: Bug
>     Versions: 0.8-dev
>     Reporter: Stefan Groschupf
>     Assignee: Stefan Groschupf
>     Priority: Critical
>      Fix For: 0.8-dev

>
> I have a case here where the NutchBean is instantiated more than once, however I do cache the nutch bean, but in some situations the bean needs to re created. The problem is the  FetchedSegments leaves open all reads it uses. So a nio Exception is thrown as soon I try to create the NutchBean again. 
> I would suggest to add a close method to  FetchedSegments and all involved objects to be able cleanly shutting down the NutchBean.
> Any comments? Would a patch be welcome?
> Caused by: java.nio.channels.ClosedChannelException
> at sun.nio.ch.FileChannelImpl.ensureOpen(FileChannelImpl.java:89)
> at sun.nio.ch.FileChannelImpl.position(FileChannelImpl.java:272)
> at org.apache.nutch.fs.LocalFileSystem$LocalNFSFileInputStream.seek(LocalFileSystem.java:83)
> at org.apache.nutch.fs.NFSDataInputStream$Checker.seek(NFSDataInputStream.java:66)
> at org.apache.nutch.fs.NFSDataInputStream$PositionCache.seek(NFSDataInputStream.java:162)
> at org.apache.nutch.fs.NFSDataInputStream$Buffer.seek(NFSDataInputStream.java:191)
> at org.apache.nutch.fs.NFSDataInputStream.seek(NFSDataInputStream.java:241)
> at org.apache.nutch.io.SequenceFile$Reader.seek(SequenceFile.java:403)
> at org.apache.nutch.io.MapFile$Reader.seek(MapFile.java:329)
> at org.apache.nutch.io.MapFile$Reader.get(MapFile.java:374)
> at org.apache.nutch.mapred.MapFileOutputFormat.getEntry(MapFileOutputFormat.java:76)
> at org.apache.nutch.searcher.FetchedSegments$Segment.getEntry(FetchedSegments.java:93)
> at org.apache.nutch.searcher.FetchedSegments$Segment.getParseText(FetchedSegments.java:84)
> at org.apache.nutch.searcher.FetchedSegments.getSummary(FetchedSegments.java:147)
> at org.apache.nutch.searcher.NutchBean.getSummary(NutchBean.java:321)

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
   http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
   http://www.atlassian.com/software/jira


[jira] Assigned: (NUTCH-211) FetchedSegments leave readers open

Posted by "Stefan Groschupf (JIRA)" <ji...@apache.org>.
     [ http://issues.apache.org/jira/browse/NUTCH-211?page=all ]

Stefan Groschupf reassigned NUTCH-211:
--------------------------------------

    Assign To: Stefan Groschupf

> FetchedSegments leave readers open
> ----------------------------------
>
>          Key: NUTCH-211
>          URL: http://issues.apache.org/jira/browse/NUTCH-211
>      Project: Nutch
>         Type: Bug
>     Versions: 0.8-dev
>     Reporter: Stefan Groschupf
>     Assignee: Stefan Groschupf
>     Priority: Critical
>      Fix For: 0.8-dev

>
> I have a case here where the NutchBean is instantiated more than once, however I do cache the nutch bean, but in some situations the bean needs to re created. The problem is the  FetchedSegments leaves open all reads it uses. So a nio Exception is thrown as soon I try to create the NutchBean again. 
> I would suggest to add a close method to  FetchedSegments and all involved objects to be able cleanly shutting down the NutchBean.
> Any comments? Would a patch be welcome?
> Caused by: java.nio.channels.ClosedChannelException
> at sun.nio.ch.FileChannelImpl.ensureOpen(FileChannelImpl.java:89)
> at sun.nio.ch.FileChannelImpl.position(FileChannelImpl.java:272)
> at org.apache.nutch.fs.LocalFileSystem$LocalNFSFileInputStream.seek(LocalFileSystem.java:83)
> at org.apache.nutch.fs.NFSDataInputStream$Checker.seek(NFSDataInputStream.java:66)
> at org.apache.nutch.fs.NFSDataInputStream$PositionCache.seek(NFSDataInputStream.java:162)
> at org.apache.nutch.fs.NFSDataInputStream$Buffer.seek(NFSDataInputStream.java:191)
> at org.apache.nutch.fs.NFSDataInputStream.seek(NFSDataInputStream.java:241)
> at org.apache.nutch.io.SequenceFile$Reader.seek(SequenceFile.java:403)
> at org.apache.nutch.io.MapFile$Reader.seek(MapFile.java:329)
> at org.apache.nutch.io.MapFile$Reader.get(MapFile.java:374)
> at org.apache.nutch.mapred.MapFileOutputFormat.getEntry(MapFileOutputFormat.java:76)
> at org.apache.nutch.searcher.FetchedSegments$Segment.getEntry(FetchedSegments.java:93)
> at org.apache.nutch.searcher.FetchedSegments$Segment.getParseText(FetchedSegments.java:84)
> at org.apache.nutch.searcher.FetchedSegments.getSummary(FetchedSegments.java:147)
> at org.apache.nutch.searcher.NutchBean.getSummary(NutchBean.java:321)

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
   http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
   http://www.atlassian.com/software/jira


[jira] Updated: (NUTCH-211) FetchedSegments leave readers open

Posted by "Stefan Groschupf (JIRA)" <ji...@apache.org>.
     [ http://issues.apache.org/jira/browse/NUTCH-211?page=all ]

Stefan Groschupf updated NUTCH-211:
-----------------------------------

    Attachment: closeable160206.patch

Now also closing linkdb reader and file system, thanks to Raghavendra.

> FetchedSegments leave readers open
> ----------------------------------
>
>          Key: NUTCH-211
>          URL: http://issues.apache.org/jira/browse/NUTCH-211
>      Project: Nutch
>         Type: Bug
>     Versions: 0.8-dev
>     Reporter: Stefan Groschupf
>     Assignee: Stefan Groschupf
>     Priority: Critical
>      Fix For: 0.8-dev
>  Attachments: closeFetchSegments.patch, closeable160206.patch
>
> I have a case here where the NutchBean is instantiated more than once, however I do cache the nutch bean, but in some situations the bean needs to re created. The problem is the  FetchedSegments leaves open all reads it uses. So a nio Exception is thrown as soon I try to create the NutchBean again. 
> I would suggest to add a close method to  FetchedSegments and all involved objects to be able cleanly shutting down the NutchBean.
> Any comments? Would a patch be welcome?
> Caused by: java.nio.channels.ClosedChannelException
> at sun.nio.ch.FileChannelImpl.ensureOpen(FileChannelImpl.java:89)
> at sun.nio.ch.FileChannelImpl.position(FileChannelImpl.java:272)
> at org.apache.nutch.fs.LocalFileSystem$LocalNFSFileInputStream.seek(LocalFileSystem.java:83)
> at org.apache.nutch.fs.NFSDataInputStream$Checker.seek(NFSDataInputStream.java:66)
> at org.apache.nutch.fs.NFSDataInputStream$PositionCache.seek(NFSDataInputStream.java:162)
> at org.apache.nutch.fs.NFSDataInputStream$Buffer.seek(NFSDataInputStream.java:191)
> at org.apache.nutch.fs.NFSDataInputStream.seek(NFSDataInputStream.java:241)
> at org.apache.nutch.io.SequenceFile$Reader.seek(SequenceFile.java:403)
> at org.apache.nutch.io.MapFile$Reader.seek(MapFile.java:329)
> at org.apache.nutch.io.MapFile$Reader.get(MapFile.java:374)
> at org.apache.nutch.mapred.MapFileOutputFormat.getEntry(MapFileOutputFormat.java:76)
> at org.apache.nutch.searcher.FetchedSegments$Segment.getEntry(FetchedSegments.java:93)
> at org.apache.nutch.searcher.FetchedSegments$Segment.getParseText(FetchedSegments.java:84)
> at org.apache.nutch.searcher.FetchedSegments.getSummary(FetchedSegments.java:147)
> at org.apache.nutch.searcher.NutchBean.getSummary(NutchBean.java:321)

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
   http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
   http://www.atlassian.com/software/jira


[jira] Commented: (NUTCH-211) FetchedSegments leave readers open

Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
    [ http://issues.apache.org/jira/browse/NUTCH-211?page=comments#action_12366505 ] 

Doug Cutting commented on NUTCH-211:
------------------------------------

The interfaces that FetchedSegments implements should have a close method.  Moreover, these interfaces should extend a Closeable interface.  JDK 1.5 has added such an interface, and, in the meantime, I can  add one to Hadoop (org.apache.hadoop.io.Closeable) that Nutch can use until we upgrade to Java 1.5.

So, once I've added Closeable to Hadoop, please submit a patch that makes HitContent and HitSummarizer extend Closeable, and FetchedSegments implement it.

Does that sound reasonable?

> FetchedSegments leave readers open
> ----------------------------------
>
>          Key: NUTCH-211
>          URL: http://issues.apache.org/jira/browse/NUTCH-211
>      Project: Nutch
>         Type: Bug
>     Versions: 0.8-dev
>     Reporter: Stefan Groschupf
>     Priority: Critical
>      Fix For: 0.8-dev

>
> I have a case here where the NutchBean is instantiated more than once, however I do cache the nutch bean, but in some situations the bean needs to re created. The problem is the  FetchedSegments leaves open all reads it uses. So a nio Exception is thrown as soon I try to create the NutchBean again. 
> I would suggest to add a close method to  FetchedSegments and all involved objects to be able cleanly shutting down the NutchBean.
> Any comments? Would a patch be welcome?
> Caused by: java.nio.channels.ClosedChannelException
> at sun.nio.ch.FileChannelImpl.ensureOpen(FileChannelImpl.java:89)
> at sun.nio.ch.FileChannelImpl.position(FileChannelImpl.java:272)
> at org.apache.nutch.fs.LocalFileSystem$LocalNFSFileInputStream.seek(LocalFileSystem.java:83)
> at org.apache.nutch.fs.NFSDataInputStream$Checker.seek(NFSDataInputStream.java:66)
> at org.apache.nutch.fs.NFSDataInputStream$PositionCache.seek(NFSDataInputStream.java:162)
> at org.apache.nutch.fs.NFSDataInputStream$Buffer.seek(NFSDataInputStream.java:191)
> at org.apache.nutch.fs.NFSDataInputStream.seek(NFSDataInputStream.java:241)
> at org.apache.nutch.io.SequenceFile$Reader.seek(SequenceFile.java:403)
> at org.apache.nutch.io.MapFile$Reader.seek(MapFile.java:329)
> at org.apache.nutch.io.MapFile$Reader.get(MapFile.java:374)
> at org.apache.nutch.mapred.MapFileOutputFormat.getEntry(MapFileOutputFormat.java:76)
> at org.apache.nutch.searcher.FetchedSegments$Segment.getEntry(FetchedSegments.java:93)
> at org.apache.nutch.searcher.FetchedSegments$Segment.getParseText(FetchedSegments.java:84)
> at org.apache.nutch.searcher.FetchedSegments.getSummary(FetchedSegments.java:147)
> at org.apache.nutch.searcher.NutchBean.getSummary(NutchBean.java:321)

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
   http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
   http://www.atlassian.com/software/jira


[jira] Commented: (NUTCH-211) FetchedSegments leave readers open

Posted by "Stefan Groschupf (JIRA)" <ji...@apache.org>.
    [ http://issues.apache.org/jira/browse/NUTCH-211?page=comments#action_12366507 ] 

Stefan Groschupf commented on NUTCH-211:
----------------------------------------

Sounds good, as soon there is a Closeable interface in hadoop I will create a patch.
Thanks!

> FetchedSegments leave readers open
> ----------------------------------
>
>          Key: NUTCH-211
>          URL: http://issues.apache.org/jira/browse/NUTCH-211
>      Project: Nutch
>         Type: Bug
>     Versions: 0.8-dev
>     Reporter: Stefan Groschupf
>     Priority: Critical
>      Fix For: 0.8-dev

>
> I have a case here where the NutchBean is instantiated more than once, however I do cache the nutch bean, but in some situations the bean needs to re created. The problem is the  FetchedSegments leaves open all reads it uses. So a nio Exception is thrown as soon I try to create the NutchBean again. 
> I would suggest to add a close method to  FetchedSegments and all involved objects to be able cleanly shutting down the NutchBean.
> Any comments? Would a patch be welcome?
> Caused by: java.nio.channels.ClosedChannelException
> at sun.nio.ch.FileChannelImpl.ensureOpen(FileChannelImpl.java:89)
> at sun.nio.ch.FileChannelImpl.position(FileChannelImpl.java:272)
> at org.apache.nutch.fs.LocalFileSystem$LocalNFSFileInputStream.seek(LocalFileSystem.java:83)
> at org.apache.nutch.fs.NFSDataInputStream$Checker.seek(NFSDataInputStream.java:66)
> at org.apache.nutch.fs.NFSDataInputStream$PositionCache.seek(NFSDataInputStream.java:162)
> at org.apache.nutch.fs.NFSDataInputStream$Buffer.seek(NFSDataInputStream.java:191)
> at org.apache.nutch.fs.NFSDataInputStream.seek(NFSDataInputStream.java:241)
> at org.apache.nutch.io.SequenceFile$Reader.seek(SequenceFile.java:403)
> at org.apache.nutch.io.MapFile$Reader.seek(MapFile.java:329)
> at org.apache.nutch.io.MapFile$Reader.get(MapFile.java:374)
> at org.apache.nutch.mapred.MapFileOutputFormat.getEntry(MapFileOutputFormat.java:76)
> at org.apache.nutch.searcher.FetchedSegments$Segment.getEntry(FetchedSegments.java:93)
> at org.apache.nutch.searcher.FetchedSegments$Segment.getParseText(FetchedSegments.java:84)
> at org.apache.nutch.searcher.FetchedSegments.getSummary(FetchedSegments.java:147)
> at org.apache.nutch.searcher.NutchBean.getSummary(NutchBean.java:321)

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
   http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
   http://www.atlassian.com/software/jira


[jira] Commented: (NUTCH-211) FetchedSegments leave readers open

Posted by "raghavendra prabhu (JIRA)" <ji...@apache.org>.
    [ http://issues.apache.org/jira/browse/NUTCH-211?page=comments#action_12366635 ] 

raghavendra prabhu commented on NUTCH-211:
------------------------------------------

A same problem like this exist in nutch-0.7.2 and i had implemented a similar interface for closing the readers . This should free up the nutchbean.
Are you also closing the linkdb readers in the nutchbean.(nutch-0.8) ?

> FetchedSegments leave readers open
> ----------------------------------
>
>          Key: NUTCH-211
>          URL: http://issues.apache.org/jira/browse/NUTCH-211
>      Project: Nutch
>         Type: Bug
>     Versions: 0.8-dev
>     Reporter: Stefan Groschupf
>     Assignee: Stefan Groschupf
>     Priority: Critical
>      Fix For: 0.8-dev
>  Attachments: closeFetchSegments.patch
>
> I have a case here where the NutchBean is instantiated more than once, however I do cache the nutch bean, but in some situations the bean needs to re created. The problem is the  FetchedSegments leaves open all reads it uses. So a nio Exception is thrown as soon I try to create the NutchBean again. 
> I would suggest to add a close method to  FetchedSegments and all involved objects to be able cleanly shutting down the NutchBean.
> Any comments? Would a patch be welcome?
> Caused by: java.nio.channels.ClosedChannelException
> at sun.nio.ch.FileChannelImpl.ensureOpen(FileChannelImpl.java:89)
> at sun.nio.ch.FileChannelImpl.position(FileChannelImpl.java:272)
> at org.apache.nutch.fs.LocalFileSystem$LocalNFSFileInputStream.seek(LocalFileSystem.java:83)
> at org.apache.nutch.fs.NFSDataInputStream$Checker.seek(NFSDataInputStream.java:66)
> at org.apache.nutch.fs.NFSDataInputStream$PositionCache.seek(NFSDataInputStream.java:162)
> at org.apache.nutch.fs.NFSDataInputStream$Buffer.seek(NFSDataInputStream.java:191)
> at org.apache.nutch.fs.NFSDataInputStream.seek(NFSDataInputStream.java:241)
> at org.apache.nutch.io.SequenceFile$Reader.seek(SequenceFile.java:403)
> at org.apache.nutch.io.MapFile$Reader.seek(MapFile.java:329)
> at org.apache.nutch.io.MapFile$Reader.get(MapFile.java:374)
> at org.apache.nutch.mapred.MapFileOutputFormat.getEntry(MapFileOutputFormat.java:76)
> at org.apache.nutch.searcher.FetchedSegments$Segment.getEntry(FetchedSegments.java:93)
> at org.apache.nutch.searcher.FetchedSegments$Segment.getParseText(FetchedSegments.java:84)
> at org.apache.nutch.searcher.FetchedSegments.getSummary(FetchedSegments.java:147)
> at org.apache.nutch.searcher.NutchBean.getSummary(NutchBean.java:321)

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
   http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
   http://www.atlassian.com/software/jira


[jira] Commented: (NUTCH-211) FetchedSegments leave readers open

Posted by "Stefan Groschupf (JIRA)" <ji...@apache.org>.
    [ http://issues.apache.org/jira/browse/NUTCH-211?page=comments#action_12366645 ] 

Stefan Groschupf commented on NUTCH-211:
----------------------------------------

Raghavendra, I'm not sure if I also close the linkDB reader, may be I missed that. I will check this later today and may come with a improved version is I missed it. Thanks for catching this.

> FetchedSegments leave readers open
> ----------------------------------
>
>          Key: NUTCH-211
>          URL: http://issues.apache.org/jira/browse/NUTCH-211
>      Project: Nutch
>         Type: Bug
>     Versions: 0.8-dev
>     Reporter: Stefan Groschupf
>     Assignee: Stefan Groschupf
>     Priority: Critical
>      Fix For: 0.8-dev
>  Attachments: closeFetchSegments.patch
>
> I have a case here where the NutchBean is instantiated more than once, however I do cache the nutch bean, but in some situations the bean needs to re created. The problem is the  FetchedSegments leaves open all reads it uses. So a nio Exception is thrown as soon I try to create the NutchBean again. 
> I would suggest to add a close method to  FetchedSegments and all involved objects to be able cleanly shutting down the NutchBean.
> Any comments? Would a patch be welcome?
> Caused by: java.nio.channels.ClosedChannelException
> at sun.nio.ch.FileChannelImpl.ensureOpen(FileChannelImpl.java:89)
> at sun.nio.ch.FileChannelImpl.position(FileChannelImpl.java:272)
> at org.apache.nutch.fs.LocalFileSystem$LocalNFSFileInputStream.seek(LocalFileSystem.java:83)
> at org.apache.nutch.fs.NFSDataInputStream$Checker.seek(NFSDataInputStream.java:66)
> at org.apache.nutch.fs.NFSDataInputStream$PositionCache.seek(NFSDataInputStream.java:162)
> at org.apache.nutch.fs.NFSDataInputStream$Buffer.seek(NFSDataInputStream.java:191)
> at org.apache.nutch.fs.NFSDataInputStream.seek(NFSDataInputStream.java:241)
> at org.apache.nutch.io.SequenceFile$Reader.seek(SequenceFile.java:403)
> at org.apache.nutch.io.MapFile$Reader.seek(MapFile.java:329)
> at org.apache.nutch.io.MapFile$Reader.get(MapFile.java:374)
> at org.apache.nutch.mapred.MapFileOutputFormat.getEntry(MapFileOutputFormat.java:76)
> at org.apache.nutch.searcher.FetchedSegments$Segment.getEntry(FetchedSegments.java:93)
> at org.apache.nutch.searcher.FetchedSegments$Segment.getParseText(FetchedSegments.java:84)
> at org.apache.nutch.searcher.FetchedSegments.getSummary(FetchedSegments.java:147)
> at org.apache.nutch.searcher.NutchBean.getSummary(NutchBean.java:321)

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
   http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
   http://www.atlassian.com/software/jira


[jira] Resolved: (NUTCH-211) FetchedSegments leave readers open

Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
     [ http://issues.apache.org/jira/browse/NUTCH-211?page=all ]
     
Doug Cutting resolved NUTCH-211:
--------------------------------

    Resolution: Fixed

I committed this, with a bunch of  whitespace fixes.

> FetchedSegments leave readers open
> ----------------------------------
>
>          Key: NUTCH-211
>          URL: http://issues.apache.org/jira/browse/NUTCH-211
>      Project: Nutch
>         Type: Bug
>     Versions: 0.8-dev
>     Reporter: Stefan Groschupf
>     Assignee: Stefan Groschupf
>     Priority: Critical
>      Fix For: 0.8-dev
>  Attachments: closeFetchSegments.patch, closeable160206.patch
>
> I have a case here where the NutchBean is instantiated more than once, however I do cache the nutch bean, but in some situations the bean needs to re created. The problem is the  FetchedSegments leaves open all reads it uses. So a nio Exception is thrown as soon I try to create the NutchBean again. 
> I would suggest to add a close method to  FetchedSegments and all involved objects to be able cleanly shutting down the NutchBean.
> Any comments? Would a patch be welcome?
> Caused by: java.nio.channels.ClosedChannelException
> at sun.nio.ch.FileChannelImpl.ensureOpen(FileChannelImpl.java:89)
> at sun.nio.ch.FileChannelImpl.position(FileChannelImpl.java:272)
> at org.apache.nutch.fs.LocalFileSystem$LocalNFSFileInputStream.seek(LocalFileSystem.java:83)
> at org.apache.nutch.fs.NFSDataInputStream$Checker.seek(NFSDataInputStream.java:66)
> at org.apache.nutch.fs.NFSDataInputStream$PositionCache.seek(NFSDataInputStream.java:162)
> at org.apache.nutch.fs.NFSDataInputStream$Buffer.seek(NFSDataInputStream.java:191)
> at org.apache.nutch.fs.NFSDataInputStream.seek(NFSDataInputStream.java:241)
> at org.apache.nutch.io.SequenceFile$Reader.seek(SequenceFile.java:403)
> at org.apache.nutch.io.MapFile$Reader.seek(MapFile.java:329)
> at org.apache.nutch.io.MapFile$Reader.get(MapFile.java:374)
> at org.apache.nutch.mapred.MapFileOutputFormat.getEntry(MapFileOutputFormat.java:76)
> at org.apache.nutch.searcher.FetchedSegments$Segment.getEntry(FetchedSegments.java:93)
> at org.apache.nutch.searcher.FetchedSegments$Segment.getParseText(FetchedSegments.java:84)
> at org.apache.nutch.searcher.FetchedSegments.getSummary(FetchedSegments.java:147)
> at org.apache.nutch.searcher.NutchBean.getSummary(NutchBean.java:321)

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
   http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
   http://www.atlassian.com/software/jira