[jira] [Updated] (JENA-91) extremely large buffer is being created in ObjectFileStorage

["Simon Helsen (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/JENA-91?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Simon Helsen updated JENA-91:
-----------------------------

    Description: 
I tried to debug the OME and check why a bytebuffer is causing my native memory to explode in almost no time. It all seems to happen in this bit of code in com.hp.hpl.jena.tdb.base.objectfile.ObjectFileStorage (lines 243 onwards)

  // No - it's in the underlying file storage.
        lengthBuffer.clear() ;
        int x = file.read(lengthBuffer, loc) ;
        if ( x != 4 )
            throw new FileException("ObjectFile.read("+loc+")["+filesize+"]["+file.size()+"]: Failed to read the length : got "+x+" bytes") ;
        int len = lengthBuffer.getInt(0) ;
        ByteBuffer bb = ByteBuffer.allocate(len) ;

My debugger shows that x==4. It also shows the lengthBuffer has the following content: [111, 110, 61, 95]. This amounts to the value of len=1869495647, which is rather a lot :-) Obviously, the next statement (ByteBuffer.allocate) causes the OME.


  was:
I tried to debug the OME and check why a bytebuffer is causing my native memory to explode in almost no time. It all seems to happen in this bit of code in com.hp.hpl.jena.tdb.base.objectfile.ObjectFileStorage (lines 243 onwards)

  // No - it's in the underlying file storage.
        lengthBuffer.clear() ;
        int x = file.read(lengthBuffer, loc) ;
        if ( x != 4 )
            throw new FileException("ObjectFile.read("+loc+")["+filesize+"]["+file.size()+"]: Failed to read the length : got "+x+" bytes") ;
        int len = lengthBuffer.getInt(0) ;
        ByteBuffer bb = ByteBuffer.allocate(len) ;

My debugger shows that x==4. It also shows the lengthBuffer has the following content: [111, 110, 61, 95], however the value of len=1869495647, which is rather a lot :-) Obviously, the next statement (ByteBuffer.allocate) causes the OME

So, looking into it, lengthBuffer.getInt(0)==1869495647, but lenghtBuffer.get(0)==111, which seems more correct. Then I noticed that the bytebuffer also says bigEndian==true. I am running all of this on Windows 7. which is to the best of my knowledge little endian

I think this can only work correctly if the byteBuffer for lengths (at least) have their order explicitly set to that of the platform. So, somewhere in the code, you'd need to make sure to set

lengthBuffer.order(ByteOrder.nativeOrder())

I think this only affects this particular one because it manages lengths, but I don't know the code well enough to know if there are other places that may be affected

        Summary: extremely large buffer is being created in ObjectFileStorage  (was: byte area for lengths is read in platform endian, but expects big endian)

> extremely large buffer is being created in ObjectFileStorage
> ------------------------------------------------------------
>
>                 Key: JENA-91
>                 URL: https://issues.apache.org/jira/browse/JENA-91
>             Project: Jena
>          Issue Type: Bug
>          Components: TDB
>         Environment: Windows (and I presume any little endian system)
>            Reporter: Simon Helsen
>            Priority: Critical
>
> I tried to debug the OME and check why a bytebuffer is causing my native memory to explode in almost no time. It all seems to happen in this bit of code in com.hp.hpl.jena.tdb.base.objectfile.ObjectFileStorage (lines 243 onwards)
>   // No - it's in the underlying file storage.
>         lengthBuffer.clear() ;
>         int x = file.read(lengthBuffer, loc) ;
>         if ( x != 4 )
>             throw new FileException("ObjectFile.read("+loc+")["+filesize+"]["+file.size()+"]: Failed to read the length : got "+x+" bytes") ;
>         int len = lengthBuffer.getInt(0) ;
>         ByteBuffer bb = ByteBuffer.allocate(len) ;
> My debugger shows that x==4. It also shows the lengthBuffer has the following content: [111, 110, 61, 95]. This amounts to the value of len=1869495647, which is rather a lot :-) Obviously, the next statement (ByteBuffer.allocate) causes the OME.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira