You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@jena.apache.org by Simon Helsen <sh...@ca.ibm.com> on 2011/08/08 21:42:03 UTC
Current ByteBuffers only work for bigEndian systems...
Andy, others,
I tried to debug the OME and check why the bytebuffer is causing my native
memory to explode in almost no time. It all seems to happen in this bit of
code in com.hp.hpl.jena.tdb.base.objectfile.ObjectFileStorage (lines 243
onwards)
// No - it's in the underlying file storage.
lengthBuffer.clear() ;
int x = file.read(lengthBuffer, loc) ;
if ( x != 4 )
throw new FileException("ObjectFile.read("+loc+")["+filesize+
"]["+file.size()+"]: Failed to read the length : got "+x+" bytes") ;
int len = lengthBuffer.getInt(0) ;
ByteBuffer bb = ByteBuffer.allocate(len) ;
My debugger shows that x==4. It also shows the lengthBuffer has the
following content: [111, 110, 61, 95], however the value of len=1869495647
, which is rather a lot :-) Obviously, the next statement
(ByteBuffer.allocate) causes the OME
So, looking into it, lengthBuffer.getInt(0)==1869495647, but
lenghtBuffer.get(0)==111, which seems more correct. Then I noticed that
the bytebuffer also says bigEndian==true. I am running all of this on
Windows 7. which is to the best of my knowledge little endian
I think this can only work correctly if these byteBuffers have their order
explicilty set to that of the platform. So, somewhere in the code, you'd
need to make sure to set
lengthBuffer.order(ByteOrder.nativeOrder())
And this probably for all usages of ByteBuffer. That will probably solve
the problem on little endian systems such as Windows
Any thoughts or should I create a Jira Issue?
Simon
Re: Current ByteBuffers only work for bigEndian systems...
Posted by Simon Helsen <sh...@ca.ibm.com>.
this analysis I made below is not correct - I'll move the discussion to
the JIRA issue for those interested
Simon
From:
Simon Helsen/Toronto/IBM@IBMCA
To:
jena-dev@incubator.apache.org
Cc:
jena-dev@incubator.apache.org
Date:
08/08/2011 03:53 PM
Subject:
Re: Current ByteBuffers only work for bigEndian systems...
I created https://issues.apache.org/jira/browse/JENA-91, and I adjusted my
comment a bit because it is presumably only an issue if the bytes are used
a length markers and read with getInt or other type-getters which use the
platform endianness.
However, I suggest that the code is checked for other occurrences of the
same problem
thanks,
Simon
From:
Simon Helsen/Toronto/IBM@IBMCA
To:
jena-dev@incubator.apache.org
Date:
08/08/2011 03:42 PM
Subject:
Current ByteBuffers only work for bigEndian systems...
Andy, others,
I tried to debug the OME and check why the bytebuffer is causing my native
memory to explode in almost no time. It all seems to happen in this bit of
code in com.hp.hpl.jena.tdb.base.objectfile.ObjectFileStorage (lines 243
onwards)
// No - it's in the underlying file storage.
lengthBuffer.clear() ;
int x = file.read(lengthBuffer, loc) ;
if ( x != 4 )
throw new FileException("ObjectFile.read("+loc+")["+filesize+
"]["+file.size()+"]: Failed to read the length : got "+x+" bytes") ;
int len = lengthBuffer.getInt(0) ;
ByteBuffer bb = ByteBuffer.allocate(len) ;
My debugger shows that x==4. It also shows the lengthBuffer has the
following content: [111, 110, 61, 95], however the value of len=1869495647
, which is rather a lot :-) Obviously, the next statement
(ByteBuffer.allocate) causes the OME
So, looking into it, lengthBuffer.getInt(0)==1869495647, but
lenghtBuffer.get(0)==111, which seems more correct. Then I noticed that
the bytebuffer also says bigEndian==true. I am running all of this on
Windows 7. which is to the best of my knowledge little endian
I think this can only work correctly if these byteBuffers have their order
explicilty set to that of the platform. So, somewhere in the code, you'd
need to make sure to set
lengthBuffer.order(ByteOrder.nativeOrder())
And this probably for all usages of ByteBuffer. That will probably solve
the problem on little endian systems such as Windows
Any thoughts or should I create a Jira Issue?
Simon
Re: Current ByteBuffers only work for bigEndian systems...
Posted by Simon Helsen <sh...@ca.ibm.com>.
I created https://issues.apache.org/jira/browse/JENA-91, and I adjusted my
comment a bit because it is presumably only an issue if the bytes are used
a length markers and read with getInt or other type-getters which use the
platform endianness.
However, I suggest that the code is checked for other occurrences of the
same problem
thanks,
Simon
From:
Simon Helsen/Toronto/IBM@IBMCA
To:
jena-dev@incubator.apache.org
Date:
08/08/2011 03:42 PM
Subject:
Current ByteBuffers only work for bigEndian systems...
Andy, others,
I tried to debug the OME and check why the bytebuffer is causing my native
memory to explode in almost no time. It all seems to happen in this bit of
code in com.hp.hpl.jena.tdb.base.objectfile.ObjectFileStorage (lines 243
onwards)
// No - it's in the underlying file storage.
lengthBuffer.clear() ;
int x = file.read(lengthBuffer, loc) ;
if ( x != 4 )
throw new FileException("ObjectFile.read("+loc+")["+filesize+
"]["+file.size()+"]: Failed to read the length : got "+x+" bytes") ;
int len = lengthBuffer.getInt(0) ;
ByteBuffer bb = ByteBuffer.allocate(len) ;
My debugger shows that x==4. It also shows the lengthBuffer has the
following content: [111, 110, 61, 95], however the value of len=1869495647
, which is rather a lot :-) Obviously, the next statement
(ByteBuffer.allocate) causes the OME
So, looking into it, lengthBuffer.getInt(0)==1869495647, but
lenghtBuffer.get(0)==111, which seems more correct. Then I noticed that
the bytebuffer also says bigEndian==true. I am running all of this on
Windows 7. which is to the best of my knowledge little endian
I think this can only work correctly if these byteBuffers have their order
explicilty set to that of the platform. So, somewhere in the code, you'd
need to make sure to set
lengthBuffer.order(ByteOrder.nativeOrder())
And this probably for all usages of ByteBuffer. That will probably solve
the problem on little endian systems such as Windows
Any thoughts or should I create a Jira Issue?
Simon