You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@jena.apache.org by Simon Helsen <sh...@ca.ibm.com> on 2012/05/17 00:50:10 UTC
API changes
Andy,
I know you asked a while back about what API changes I had noticed. There
are a number of things that have changed since 2.6.3, most of them are
smaller and manageable (albeit annoying). Here a few examples:
1) DatasetGraphTDB subset = TDBFactory.createDatasetGraph();
now has to be
DatasetGraph subset = TDBFactory.createDatasetGraph();
2) Another example is:
queryContext.setDataset(new DatasetImpl(subset));
now has to be
queryContext.setDataset(DatasetImpl.wrap(subset));
3) TupleIndex newIndex = SetupTDB.makeTupleIndex(tmpLocation,
srcGraph.getConfig(), primary, name, name, indexRecordLen);
became
TupleIndex newIndex = SetupTDB.makeTupleIndex(tmpLocation, primary, name,
name, indexRecordLen);
4) More obscure is the following change:
SetupTDB.globalConfig.setProperty(Names.pBlockReadCacheSize,
blockReadCacheSize.toString());
That does not work anymore. It is unclear where I can set this property
now. I could use help with this one
In general, API changes are unavoidable, but they should be documented
per release in a readme.
Simon
Re: API changes
Posted by Andy Seaborne <an...@apache.org>.
On 18/05/12 03:23, Paul Gearon wrote:
> There's a standard practice on Linux/Unix systems of opening a file
> and deleting it while keeping it open. This file is fully visible
> through it's handle, but the inode has been released on disk and no
> other process can see it any more. It's been a while, but I don't
> think this works on Windows.
Indeed - windows does not treat open files as if a hard link existed
like Linux does. Linux keeps a reference to files which is incremented
for open files and all hard links (in the in-memory inode). When it
goes to zero, the file pace can be reclaimed.
In Windows, open files can't be deleted.
Truncate is move the file pointer to zero without releasing the disk
inode or OS filehandle.
> What we found was that Linux and Solaris behaved themselves quite well
> when we tried to release a mapping. Run the GC, and the mapping has
> probably been released. If not, then run it again the mapping has
> *definitely* been released.
>
> This seemed to work in general, until we had a problem with Windows
> where it wasn't released on the second GC. So we called the GC a third
> time. About a month later we had an occasion where that failed too ...
> so we increased it more. Then 6 months later it failed again (it had
> worked thousands of times in between). Finally we took the approach of
> counting down from about 10 and re-attempting the truncation after a
> GC on each iteration. This appeared to fix the problem, and we never
> saw it go over 4 attempts in the log (though we don't log the number
> of attempts any more).
>
> At some point, someone (I think David) discovered the "clean" method,
> and this gets called on objects in Windows. You'll see it gets pulled
> out via reflection.
>
> We recently pulled most of this code out and put it into a single file.
>
> http://mulgara.org/svn/mulgara/trunk/src/jar/util/java/org/mulgara/util/io/MappingUtil.java
>
> Hopefully it's clear enough, but ask questions if there's anything too obtuse.
Thanks for that.
TDB itself does not delete files (the tests do ... but the test
framework wokrs differently on Windows and Linux for portability) -
Simon is doing the delete in his code.
Andy
Re: API changes
Posted by Moritz Hoffmann <an...@gmail.com>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256
On 05/18/2012 08:43 AM, Paolo Castagna wrote:
> Hi Paul, first of all, thanks for your comments and suggestions.
>
> Paul Gearon wrote:
>> We recently pulled most of this code out and put it into a single
>> file.
>>
>> http://mulgara.org/svn/mulgara/trunk/src/jar/util/java/org/mulgara/util/io/MappingUtil.java
>>
>>
>>
Hopefully it's clear enough, but ask questions if there's anything too
obtuse.
>
> The clean(...) method uses sun.misc.Cleaner and MappingUtil.java
> imports it, so I assume this would work on with Sun/Oracle JVMs, am
> I correct? Some of the Jena users (and some of those having
> problems with memory mapped files on Windows) do not use Oracle
> JVM, so I guess that approach would not work for them.
I just had a quick glimpse at IBM JRE 1.6, and it has sun.misc.Cleaner
and the interface sun.nio.ch.DirectBuffer also exists. Still, it's not
guaranteed to be portable, but reflection could solve the problem.
Best,
Moritz
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.12 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
iF4EAREIAAYFAk+2ABoACgkQ264ap80Va8dbGgEAq5tvpX8wubawdABBgVfMC4Lg
2q1bSG9DKBWSXs1dRPAA+QEufI+GpvNJqYzv0wxbvUrODbnrYt7rCZb/T/UyGH1Q
=Y2Xq
-----END PGP SIGNATURE-----
Re: API changes
Posted by Simon Helsen <sh...@ca.ibm.com>.
Thanks everyone for the discussion. Will take a look at the Mulgara code.
Simon
From:
Paul Gearon <ge...@ieee.org>
To:
dev@jena.apache.org
Date:
05/18/2012 11:56 AM
Subject:
Re: API changes
On Fri, May 18, 2012 at 2:43 AM, Paolo Castagna
<ca...@googlemail.com> wrote:
> Hi Paul,
> first of all, thanks for your comments and suggestions.
>
> Paul Gearon wrote:
>> We recently pulled most of this code out and put it into a single file.
>>
>>
http://mulgara.org/svn/mulgara/trunk/src/jar/util/java/org/mulgara/util/io/MappingUtil.java
>>
>> Hopefully it's clear enough, but ask questions if there's anything too
obtuse.
>
> The clean(...) method uses sun.misc.Cleaner and MappingUtil.java imports
it, so I assume this would work on with Sun/Oracle JVMs, am I correct?
> Some of the Jena users (and some of those having problems with memory
mapped files on Windows) do not use Oracle JVM, so I guess that approach
would not work for them.
>
> The last comment on http://bugs.sun.com/view_bug.do?bug_id=4724038
suggests a similar solution: ((sun.nio.ch.DirectBuffer)
buf).cleaner().clean()
> It would be interesting to know if there are equivalent
.cleaner().clean() methods in other JVMs, if so, someone might use
reflection and route the .clean() method to the various internal objects.
>
> Or, are sun.misc.Cleaner and/or sun.nio.ch.DirectBuffer available in
other JVMs? (I don't know)
The clean() method is advertised as the appropriate mechanism, but as
it's not part of the Sun/Oracle spec there is no need for any other
JVM to include it. I have no idea if anyone does.
The technique of re-running a GC works empirically, though it has a
performance issue, and can't actually be *guaranteed*. In practice, we
found that the probability of an object surviving more than a few
consecutive CGs diminished rapidly, to the point where nothing seems
to get past about 6. Bringing a cleaner into it avoids the need for
consecutive GCs (meaning that the object *can* be guaranteed to be
removed, and avoiding the multiple GC overhead), but since we can't be
guaranteed that cleaners will be available then we keep the option
available.
> Thanks again,
> Paolo
>
> PS:
> I am a Linux users (from many, many years... my last Windows was Windows
98) so, sorry, but I cannot help testing this.
> I also tend to use Oracle's JVMs on 64-bits Linux OS and JVMs... which
do not have the problem. ;-)
I moved to Linux in 98, and last worked on Windows in '04. These days
I'm mostly on Mac, and my only access to Windows is through
"Parallels". However, we ran into these issues nearly 10 years ago,
and after many months of debugging and testing (since the bug was
rare, and only on Windows) we found that this approach fixed
everything for us.
Regards,
Paul
Re: API changes
Posted by Paul Gearon <ge...@ieee.org>.
On Fri, May 18, 2012 at 2:43 AM, Paolo Castagna
<ca...@googlemail.com> wrote:
> Hi Paul,
> first of all, thanks for your comments and suggestions.
>
> Paul Gearon wrote:
>> We recently pulled most of this code out and put it into a single file.
>>
>> http://mulgara.org/svn/mulgara/trunk/src/jar/util/java/org/mulgara/util/io/MappingUtil.java
>>
>> Hopefully it's clear enough, but ask questions if there's anything too obtuse.
>
> The clean(...) method uses sun.misc.Cleaner and MappingUtil.java imports it, so I assume this would work on with Sun/Oracle JVMs, am I correct?
> Some of the Jena users (and some of those having problems with memory mapped files on Windows) do not use Oracle JVM, so I guess that approach would not work for them.
>
> The last comment on http://bugs.sun.com/view_bug.do?bug_id=4724038 suggests a similar solution: ((sun.nio.ch.DirectBuffer) buf).cleaner().clean()
> It would be interesting to know if there are equivalent .cleaner().clean() methods in other JVMs, if so, someone might use reflection and route the .clean() method to the various internal objects.
>
> Or, are sun.misc.Cleaner and/or sun.nio.ch.DirectBuffer available in other JVMs? (I don't know)
The clean() method is advertised as the appropriate mechanism, but as
it's not part of the Sun/Oracle spec there is no need for any other
JVM to include it. I have no idea if anyone does.
The technique of re-running a GC works empirically, though it has a
performance issue, and can't actually be *guaranteed*. In practice, we
found that the probability of an object surviving more than a few
consecutive CGs diminished rapidly, to the point where nothing seems
to get past about 6. Bringing a cleaner into it avoids the need for
consecutive GCs (meaning that the object *can* be guaranteed to be
removed, and avoiding the multiple GC overhead), but since we can't be
guaranteed that cleaners will be available then we keep the option
available.
> Thanks again,
> Paolo
>
> PS:
> I am a Linux users (from many, many years... my last Windows was Windows 98) so, sorry, but I cannot help testing this.
> I also tend to use Oracle's JVMs on 64-bits Linux OS and JVMs... which do not have the problem. ;-)
I moved to Linux in 98, and last worked on Windows in '04. These days
I'm mostly on Mac, and my only access to Windows is through
"Parallels". However, we ran into these issues nearly 10 years ago,
and after many months of debugging and testing (since the bug was
rare, and only on Windows) we found that this approach fixed
everything for us.
Regards,
Paul
Re: API changes
Posted by Paolo Castagna <ca...@googlemail.com>.
Hi Paul,
first of all, thanks for your comments and suggestions.
Paul Gearon wrote:
> We recently pulled most of this code out and put it into a single file.
>
> http://mulgara.org/svn/mulgara/trunk/src/jar/util/java/org/mulgara/util/io/MappingUtil.java
>
> Hopefully it's clear enough, but ask questions if there's anything too obtuse.
The clean(...) method uses sun.misc.Cleaner and MappingUtil.java imports it, so I assume this would work on with Sun/Oracle JVMs, am I correct?
Some of the Jena users (and some of those having problems with memory mapped files on Windows) do not use Oracle JVM, so I guess that approach would not work for them.
The last comment on http://bugs.sun.com/view_bug.do?bug_id=4724038 suggests a similar solution: ((sun.nio.ch.DirectBuffer) buf).cleaner().clean()
It would be interesting to know if there are equivalent .cleaner().clean() methods in other JVMs, if so, someone might use reflection and route the .clean() method to the various internal objects.
Or, are sun.misc.Cleaner and/or sun.nio.ch.DirectBuffer available in other JVMs? (I don't know)
Thanks again,
Paolo
PS:
I am a Linux users (from many, many years... my last Windows was Windows 98) so, sorry, but I cannot help testing this.
I also tend to use Oracle's JVMs on 64-bits Linux OS and JVMs... which do not have the problem. ;-)
Re: API changes
Posted by Paul Gearon <ge...@ieee.org>.
On Thu, May 17, 2012 at 5:30 PM, Andy Seaborne <an...@apache.org> wrote:
> Paul,
>
> Do you have a pointer to the Mulgara code? It would be great to fix this.
>
> It's not truncation nor access after closing though: it's deletion of the
> files that's the issue. close(), no use whatsoever then file delete does
> not delete the file. The files are held by the JVM until exit.
>
> http://bugs.sun.com/view_bug.do?bug_id=4724038
The reason that close() won't help is because it's perfectly legal to
map a file and then close the file handle used to create the mapping.
It's the mapping that has to be released.
We don't often delete files that have been mapped, so I can't claim
unequivocally that it will work, but it ought to. I believe we've
deleted files, just not often. Since the problem with truncation and
with deletion is similar (due to open objects), then I'd expect the
solution to work again.
> Works just fine on Linux, does not work on MS Windows. maybe it's because
> open files count to the reference count in Linux but not on MS windows.
There's a standard practice on Linux/Unix systems of opening a file
and deleting it while keeping it open. This file is fully visible
through it's handle, but the inode has been released on disk and no
other process can see it any more. It's been a while, but I don't
think this works on Windows.
What we found was that Linux and Solaris behaved themselves quite well
when we tried to release a mapping. Run the GC, and the mapping has
probably been released. If not, then run it again the mapping has
*definitely* been released.
This seemed to work in general, until we had a problem with Windows
where it wasn't released on the second GC. So we called the GC a third
time. About a month later we had an occasion where that failed too ...
so we increased it more. Then 6 months later it failed again (it had
worked thousands of times in between). Finally we took the approach of
counting down from about 10 and re-attempting the truncation after a
GC on each iteration. This appeared to fix the problem, and we never
saw it go over 4 attempts in the log (though we don't log the number
of attempts any more).
At some point, someone (I think David) discovered the "clean" method,
and this gets called on objects in Windows. You'll see it gets pulled
out via reflection.
We recently pulled most of this code out and put it into a single file.
http://mulgara.org/svn/mulgara/trunk/src/jar/util/java/org/mulgara/util/io/MappingUtil.java
Hopefully it's clear enough, but ask questions if there's anything too obtuse.
Paul
Re: API changes
Posted by Andy Seaborne <an...@apache.org>.
Paul,
Do you have a pointer to the Mulgara code? It would be great to fix this.
It's not truncation nor access after closing though: it's deletion of
the files that's the issue. close(), no use whatsoever then file delete
does not delete the file. The files are held by the JVM until exit.
http://bugs.sun.com/view_bug.do?bug_id=4724038
Works just fine on Linux, does not work on MS Windows. maybe it's
because open files count to the reference count in Linux but not on MS
windows.
The 25-MAR-2011 comment may work ... on Sun JVMs.
Andy
On 17/05/12 21:34, Paul Gearon wrote:
> On an side note here (and hence, may be ignored the the majority)...
>
> On Thu, May 17, 2012 at 2:59 PM, Simon Helsen<sh...@ca.ibm.com> wrote:
>> We run only on 64 bit, but always in direct mode
>> because mapped mode does not allow us to remove the index (as a reindex
>> operation) because of the infamous JVM bug which holds a lock on the
>> involved files, at least on Windows.
>
> It's not a bug. It's necessary in order to maintain memory integrity
> in Java. Java has no documented way to force an object to be garbage
> collected, and if the file is closed before the mapping object is
> collected then a get() or put() will give a GPF/segfault (depending on
> Windows/Linux). get/put doesn't check if the file is still there,
> since the whole point of mapping is to avoid overhead.
>
> These problems CAN be avoided... even on Windows. Mulgara has it and
> uses it in order to truncate mapped files. Andy, does Jena care about
> this?
>
>> Moreover, it is unstable under system
>> crashes since the actual flush may not have taken place after a close.
>> That too is probably more a problem on Windows.
>
> Well, a system crash can lead to unstable files no matter the access
> type. However, if you're ever relying on data in a mapped file being
> in a stable state, then you MUST call flush on it before closing.
> Otherwise the OS will only write the data out when it has time. There
> are some conditions when this will be too long. In fact, you shouldn't
> be calling flush before closing, but rather, call flush before writing
> any data structures that expect the modified portions of that file to
> be in a consistent state.
>
> Regards,
> Paul Gearon
Re: API changes
Posted by Paul Gearon <ge...@ieee.org>.
On an side note here (and hence, may be ignored the the majority)...
On Thu, May 17, 2012 at 2:59 PM, Simon Helsen <sh...@ca.ibm.com> wrote:
> We run only on 64 bit, but always in direct mode
> because mapped mode does not allow us to remove the index (as a reindex
> operation) because of the infamous JVM bug which holds a lock on the
> involved files, at least on Windows.
It's not a bug. It's necessary in order to maintain memory integrity
in Java. Java has no documented way to force an object to be garbage
collected, and if the file is closed before the mapping object is
collected then a get() or put() will give a GPF/segfault (depending on
Windows/Linux). get/put doesn't check if the file is still there,
since the whole point of mapping is to avoid overhead.
These problems CAN be avoided... even on Windows. Mulgara has it and
uses it in order to truncate mapped files. Andy, does Jena care about
this?
> Moreover, it is unstable under system
> crashes since the actual flush may not have taken place after a close.
> That too is probably more a problem on Windows.
Well, a system crash can lead to unstable files no matter the access
type. However, if you're ever relying on data in a mapped file being
in a stable state, then you MUST call flush on it before closing.
Otherwise the OS will only write the data out when it has time. There
are some conditions when this will be too long. In fact, you shouldn't
be calling flush before closing, but rather, call flush before writing
any data structures that expect the modified portions of that file to
be in a consistent state.
Regards,
Paul Gearon
Re: API changes
Posted by Andy Seaborne <an...@apache.org>.
On 17/05/12 20:47, Simon Helsen wrote:
> yes, doing a gc has its issues in general, especially on a server, but
> we'd only do it at the very beginning (after startup) or offline. As I
> mentioned before, it doesn't really work reliably. It only increases the
> chance the lock is released.
>
> But for that reason, we are stuck to direct. It also worth noting that
> direct is faster on Windows, at least, it was last time we tested. On
> linux, mapped seems somewhat faster
>
> Simon
A thing to try is to use Google's guava cache code.
Even better, would be a scan resistant algorithm, like LRU-2 (LRU-K), 2Q
or ARC. This is a patented area ... it's IBM who patented ARC.
For the PostgreSQL story:
http://www.varlena.com/GeneralBits/96.php
(which means the postgresql algorithm is an interesting choice)
Andy
Re: API changes
Posted by Simon Helsen <sh...@ca.ibm.com>.
yes, doing a gc has its issues in general, especially on a server, but
we'd only do it at the very beginning (after startup) or offline. As I
mentioned before, it doesn't really work reliably. It only increases the
chance the lock is released.
But for that reason, we are stuck to direct. It also worth noting that
direct is faster on Windows, at least, it was last time we tested. On
linux, mapped seems somewhat faster
Simon
From:
Andy Seaborne <an...@apache.org>
To:
Simon Helsen/Toronto/IBM@IBMCA
Cc:
dev@jena.apache.org, Andy Seaborne <an...@gmail.com>
Date:
05/17/2012 03:43 PM
Subject:
Re: API changes
On 17/05/12 20:21, Simon Helsen wrote:
> Andy,
>
> the workaround you mention, what is that? I know I read somewhere that
> forcing a full GC before trying to delete it may solve it, but we
> experimented with that and it only reduced the likelihood of it
> occurring. The lock was not always released when we tried. Is that the
> workaround you know of?
The bug report at "Sun" has lots of suggestions.
I've not tried any of them - assigning null and forcing a full gc looked
like a possibility. But forcing a GC in some hidden piece of code has
its own issues for a general solution.
Andy
>
> Simon
>
>
> From: Andy Seaborne <an...@apache.org>
> To: Simon Helsen/Toronto/IBM@IBMCA
> Cc: dev@jena.apache.org, Andy Seaborne
<an...@gmail.com>
> Date: 05/17/2012 03:09 PM
> Subject: Re: API changes
>
>
> ------------------------------------------------------------------------
>
>
>
> On 17/05/12 19:59, Simon Helsen wrote:
> > Andy,
> >
> > let me pick out this comment of yours:
> >
> > "You may wish to submit an enhancement if you want to manipulate
cache
> > sizes in detail. It's been on the "it would be nice" list but it
isn't
> > something many people need as they run on 64bit in production. I
wonder
> > if parameters ought to be a related to location.
> > "
> >
> > why do you say this?
>
> "Direct mode" if you like. For most people, direct = 32 bit, mapped =
> 64 bit. It's a unique feature of your deployment.
>
> > We run only on 64 bit, but always in direct mode
> > because mapped mode does not allow us to remove the index (as a
reindex
> > operation) because of the infamous JVM bug which holds a lock on the
> > involved files, at least on Windows. Moreover, it is unstable under
> > system crashes since the actual flush may not have taken place after
a
> > close.
>
> We have transactions now.
>
> (But you still can't delete datasets on Windows without using one of the
> (partial) workarounds for that bug.)
>
> > That too is probably more a problem on Windows. The latter may be
> > outdated now with transactions and journaling, although it has to be
> > seen if we could still corrupt the index this way. Either way, I
would
> > never recommend using mapped I/O for production purposes. We tried
and
> > miserably failed. And yes, not all production systems are linux
> >
> > Anyhow, I'll look into the enhancement
>
> Great.
>
> Andy
>
> >
> > Simon
> >
> >
> >
> > From: Andy Seaborne <an...@apache.org>
> > To: dev@jena.apache.org
> > Date: 05/17/2012 04:28 AM
> > Subject: Re: API changes
> >
> >
> >
------------------------------------------------------------------------
> >
> >
> >
> > On 16/05/12 23:50, Simon Helsen wrote:
> > > Andy,
> > >
> > > I know you asked a while back about what API changes I had noticed.
> There
> > > are a number of things that have changed since 2.6.3,
> >
> > October 2010
> >
> > > most of them are
> > > smaller and manageable (albeit annoying). Here a few examples:
> > >
> > > 1) DatasetGraphTDB subset = TDBFactory.createDatasetGraph();
> > >
> > > now has to be
> > >
> > > DatasetGraph subset = TDBFactory.createDatasetGraph();
> >
> > DatasetGraphTDB is an internal class (and it's not DatasetGraphTDB
> now :-)
> >
> > >
> > > 2) Another example is:
> > >
> > > queryContext.setDataset(new DatasetImpl(subset));
> > >
> > > now has to be
> > >
> > > queryContext.setDataset(DatasetImpl.wrap(subset));
> >
> > I don't understand this one - what's queryContext?
> > The only setDataset is on ExecutionContext in
> >
> > com.hp.hpl.jena.sparql.engine
> >
> > which is inside the engine and not an API call.
> >
> > There is QueryExecutionFactory for creating a QueryExecution for
dataset.
> >
> > The call you want is
> > DatasetFactory.create(DatasetGraph)
> >
> > DatasetImpl is, well, an Impl class.
> >
> > > 3) TupleIndex newIndex = SetupTDB.makeTupleIndex(tmpLocation,
> > > srcGraph.getConfig(), primary, name, name, indexRecordLen);
> > >
> > > became
> > >
> > > TupleIndex newIndex = SetupTDB.makeTupleIndex(tmpLocation, primary,
> name,
> > > name, indexRecordLen);
> >
> > Doing anything with internal classes is at your own risk!
> >
> > >
> > > 4) More obscure is the following change:
> > >
> > > SetupTDB.globalConfig.setProperty(Names.pBlockReadCacheSize,
> > > blockReadCacheSize.toString());
> >
> > SetupTDB is a wrapper to old code.
> >
> > SystemParams has the constants as finals.
> >
> > See SetupTDB for setting constant from properties.
> >
> > You may wish to submit an enhancement if you want to manipulate cache
> > sizes in detail. It's been on the "it would be nice" list but it
isn't
> > something many people need as they run on 64bit in production. I
wonder
> > if parameters ought to be a related to location.
> >
> > >
> > > That does not work anymore. It is unclear where I can set this
property
> > > now. I could use help with this one
> > >
> > >
> > > In general, API changes are unavoidable, but they should be
documented
> > > per release in a readme.
> > >
> > > Simon
> > >
> > >
> >
> > Andy
> >
> >
> >
>
>
>
Re: API changes
Posted by Andy Seaborne <an...@apache.org>.
On 17/05/12 20:21, Simon Helsen wrote:
> Andy,
>
> the workaround you mention, what is that? I know I read somewhere that
> forcing a full GC before trying to delete it may solve it, but we
> experimented with that and it only reduced the likelihood of it
> occurring. The lock was not always released when we tried. Is that the
> workaround you know of?
The bug report at "Sun" has lots of suggestions.
I've not tried any of them - assigning null and forcing a full gc looked
like a possibility. But forcing a GC in some hidden piece of code has
its own issues for a general solution.
Andy
>
> Simon
>
>
> From: Andy Seaborne <an...@apache.org>
> To: Simon Helsen/Toronto/IBM@IBMCA
> Cc: dev@jena.apache.org, Andy Seaborne <an...@gmail.com>
> Date: 05/17/2012 03:09 PM
> Subject: Re: API changes
>
>
> ------------------------------------------------------------------------
>
>
>
> On 17/05/12 19:59, Simon Helsen wrote:
> > Andy,
> >
> > let me pick out this comment of yours:
> >
> > "You may wish to submit an enhancement if you want to manipulate cache
> > sizes in detail. It's been on the "it would be nice" list but it isn't
> > something many people need as they run on 64bit in production. I wonder
> > if parameters ought to be a related to location.
> > "
> >
> > why do you say this?
>
> "Direct mode" if you like. For most people, direct = 32 bit, mapped =
> 64 bit. It's a unique feature of your deployment.
>
> > We run only on 64 bit, but always in direct mode
> > because mapped mode does not allow us to remove the index (as a reindex
> > operation) because of the infamous JVM bug which holds a lock on the
> > involved files, at least on Windows. Moreover, it is unstable under
> > system crashes since the actual flush may not have taken place after a
> > close.
>
> We have transactions now.
>
> (But you still can't delete datasets on Windows without using one of the
> (partial) workarounds for that bug.)
>
> > That too is probably more a problem on Windows. The latter may be
> > outdated now with transactions and journaling, although it has to be
> > seen if we could still corrupt the index this way. Either way, I would
> > never recommend using mapped I/O for production purposes. We tried and
> > miserably failed. And yes, not all production systems are linux
> >
> > Anyhow, I'll look into the enhancement
>
> Great.
>
> Andy
>
> >
> > Simon
> >
> >
> >
> > From: Andy Seaborne <an...@apache.org>
> > To: dev@jena.apache.org
> > Date: 05/17/2012 04:28 AM
> > Subject: Re: API changes
> >
> >
> > ------------------------------------------------------------------------
> >
> >
> >
> > On 16/05/12 23:50, Simon Helsen wrote:
> > > Andy,
> > >
> > > I know you asked a while back about what API changes I had noticed.
> There
> > > are a number of things that have changed since 2.6.3,
> >
> > October 2010
> >
> > > most of them are
> > > smaller and manageable (albeit annoying). Here a few examples:
> > >
> > > 1) DatasetGraphTDB subset = TDBFactory.createDatasetGraph();
> > >
> > > now has to be
> > >
> > > DatasetGraph subset = TDBFactory.createDatasetGraph();
> >
> > DatasetGraphTDB is an internal class (and it's not DatasetGraphTDB
> now :-)
> >
> > >
> > > 2) Another example is:
> > >
> > > queryContext.setDataset(new DatasetImpl(subset));
> > >
> > > now has to be
> > >
> > > queryContext.setDataset(DatasetImpl.wrap(subset));
> >
> > I don't understand this one - what's queryContext?
> > The only setDataset is on ExecutionContext in
> >
> > com.hp.hpl.jena.sparql.engine
> >
> > which is inside the engine and not an API call.
> >
> > There is QueryExecutionFactory for creating a QueryExecution for dataset.
> >
> > The call you want is
> > DatasetFactory.create(DatasetGraph)
> >
> > DatasetImpl is, well, an Impl class.
> >
> > > 3) TupleIndex newIndex = SetupTDB.makeTupleIndex(tmpLocation,
> > > srcGraph.getConfig(), primary, name, name, indexRecordLen);
> > >
> > > became
> > >
> > > TupleIndex newIndex = SetupTDB.makeTupleIndex(tmpLocation, primary,
> name,
> > > name, indexRecordLen);
> >
> > Doing anything with internal classes is at your own risk!
> >
> > >
> > > 4) More obscure is the following change:
> > >
> > > SetupTDB.globalConfig.setProperty(Names.pBlockReadCacheSize,
> > > blockReadCacheSize.toString());
> >
> > SetupTDB is a wrapper to old code.
> >
> > SystemParams has the constants as finals.
> >
> > See SetupTDB for setting constant from properties.
> >
> > You may wish to submit an enhancement if you want to manipulate cache
> > sizes in detail. It's been on the "it would be nice" list but it isn't
> > something many people need as they run on 64bit in production. I wonder
> > if parameters ought to be a related to location.
> >
> > >
> > > That does not work anymore. It is unclear where I can set this property
> > > now. I could use help with this one
> > >
> > >
> > > In general, API changes are unavoidable, but they should be documented
> > > per release in a readme.
> > >
> > > Simon
> > >
> > >
> >
> > Andy
> >
> >
> >
>
>
>
Re: API changes
Posted by Simon Helsen <sh...@ca.ibm.com>.
Andy,
the workaround you mention, what is that? I know I read somewhere that
forcing a full GC before trying to delete it may solve it, but we
experimented with that and it only reduced the likelihood of it occurring.
The lock was not always released when we tried. Is that the workaround you
know of?
Simon
From:
Andy Seaborne <an...@apache.org>
To:
Simon Helsen/Toronto/IBM@IBMCA
Cc:
dev@jena.apache.org, Andy Seaborne <an...@gmail.com>
Date:
05/17/2012 03:09 PM
Subject:
Re: API changes
On 17/05/12 19:59, Simon Helsen wrote:
> Andy,
>
> let me pick out this comment of yours:
>
> "You may wish to submit an enhancement if you want to manipulate cache
> sizes in detail. It's been on the "it would be nice" list but it isn't
> something many people need as they run on 64bit in production. I wonder
> if parameters ought to be a related to location.
> "
>
> why do you say this?
"Direct mode" if you like. For most people, direct = 32 bit, mapped =
64 bit. It's a unique feature of your deployment.
> We run only on 64 bit, but always in direct mode
> because mapped mode does not allow us to remove the index (as a reindex
> operation) because of the infamous JVM bug which holds a lock on the
> involved files, at least on Windows. Moreover, it is unstable under
> system crashes since the actual flush may not have taken place after a
> close.
We have transactions now.
(But you still can't delete datasets on Windows without using one of the
(partial) workarounds for that bug.)
> That too is probably more a problem on Windows. The latter may be
> outdated now with transactions and journaling, although it has to be
> seen if we could still corrupt the index this way. Either way, I would
> never recommend using mapped I/O for production purposes. We tried and
> miserably failed. And yes, not all production systems are linux
>
> Anyhow, I'll look into the enhancement
Great.
Andy
>
> Simon
>
>
>
> From: Andy Seaborne <an...@apache.org>
> To: dev@jena.apache.org
> Date: 05/17/2012 04:28 AM
> Subject: Re: API changes
>
>
> ------------------------------------------------------------------------
>
>
>
> On 16/05/12 23:50, Simon Helsen wrote:
> > Andy,
> >
> > I know you asked a while back about what API changes I had noticed.
There
> > are a number of things that have changed since 2.6.3,
>
> October 2010
>
> > most of them are
> > smaller and manageable (albeit annoying). Here a few examples:
> >
> > 1) DatasetGraphTDB subset = TDBFactory.createDatasetGraph();
> >
> > now has to be
> >
> > DatasetGraph subset = TDBFactory.createDatasetGraph();
>
> DatasetGraphTDB is an internal class (and it's not DatasetGraphTDB now
:-)
>
> >
> > 2) Another example is:
> >
> > queryContext.setDataset(new DatasetImpl(subset));
> >
> > now has to be
> >
> > queryContext.setDataset(DatasetImpl.wrap(subset));
>
> I don't understand this one - what's queryContext?
> The only setDataset is on ExecutionContext in
>
> com.hp.hpl.jena.sparql.engine
>
> which is inside the engine and not an API call.
>
> There is QueryExecutionFactory for creating a QueryExecution for
dataset.
>
> The call you want is
> DatasetFactory.create(DatasetGraph)
>
> DatasetImpl is, well, an Impl class.
>
> > 3) TupleIndex newIndex = SetupTDB.makeTupleIndex(tmpLocation,
> > srcGraph.getConfig(), primary, name, name, indexRecordLen);
> >
> > became
> >
> > TupleIndex newIndex = SetupTDB.makeTupleIndex(tmpLocation, primary,
name,
> > name, indexRecordLen);
>
> Doing anything with internal classes is at your own risk!
>
> >
> > 4) More obscure is the following change:
> >
> > SetupTDB.globalConfig.setProperty(Names.pBlockReadCacheSize,
> > blockReadCacheSize.toString());
>
> SetupTDB is a wrapper to old code.
>
> SystemParams has the constants as finals.
>
> See SetupTDB for setting constant from properties.
>
> You may wish to submit an enhancement if you want to manipulate cache
> sizes in detail. It's been on the "it would be nice" list but it isn't
> something many people need as they run on 64bit in production. I wonder
> if parameters ought to be a related to location.
>
> >
> > That does not work anymore. It is unclear where I can set this
property
> > now. I could use help with this one
> >
> >
> > In general, API changes are unavoidable, but they should be
documented
> > per release in a readme.
> >
> > Simon
> >
> >
>
> Andy
>
>
>
Re: API changes
Posted by Andy Seaborne <an...@apache.org>.
On 17/05/12 19:59, Simon Helsen wrote:
> Andy,
>
> let me pick out this comment of yours:
>
> "You may wish to submit an enhancement if you want to manipulate cache
> sizes in detail. It's been on the "it would be nice" list but it isn't
> something many people need as they run on 64bit in production. I wonder
> if parameters ought to be a related to location.
> "
>
> why do you say this?
"Direct mode" if you like. For most people, direct = 32 bit, mapped =
64 bit. It's a unique feature of your deployment.
> We run only on 64 bit, but always in direct mode
> because mapped mode does not allow us to remove the index (as a reindex
> operation) because of the infamous JVM bug which holds a lock on the
> involved files, at least on Windows. Moreover, it is unstable under
> system crashes since the actual flush may not have taken place after a
> close.
We have transactions now.
(But you still can't delete datasets on Windows without using one of the
(partial) workarounds for that bug.)
> That too is probably more a problem on Windows. The latter may be
> outdated now with transactions and journaling, although it has to be
> seen if we could still corrupt the index this way. Either way, I would
> never recommend using mapped I/O for production purposes. We tried and
> miserably failed. And yes, not all production systems are linux
>
> Anyhow, I'll look into the enhancement
Great.
Andy
>
> Simon
>
>
>
> From: Andy Seaborne <an...@apache.org>
> To: dev@jena.apache.org
> Date: 05/17/2012 04:28 AM
> Subject: Re: API changes
>
>
> ------------------------------------------------------------------------
>
>
>
> On 16/05/12 23:50, Simon Helsen wrote:
> > Andy,
> >
> > I know you asked a while back about what API changes I had noticed. There
> > are a number of things that have changed since 2.6.3,
>
> October 2010
>
> > most of them are
> > smaller and manageable (albeit annoying). Here a few examples:
> >
> > 1) DatasetGraphTDB subset = TDBFactory.createDatasetGraph();
> >
> > now has to be
> >
> > DatasetGraph subset = TDBFactory.createDatasetGraph();
>
> DatasetGraphTDB is an internal class (and it's not DatasetGraphTDB now :-)
>
> >
> > 2) Another example is:
> >
> > queryContext.setDataset(new DatasetImpl(subset));
> >
> > now has to be
> >
> > queryContext.setDataset(DatasetImpl.wrap(subset));
>
> I don't understand this one - what's queryContext?
> The only setDataset is on ExecutionContext in
>
> com.hp.hpl.jena.sparql.engine
>
> which is inside the engine and not an API call.
>
> There is QueryExecutionFactory for creating a QueryExecution for dataset.
>
> The call you want is
> DatasetFactory.create(DatasetGraph)
>
> DatasetImpl is, well, an Impl class.
>
> > 3) TupleIndex newIndex = SetupTDB.makeTupleIndex(tmpLocation,
> > srcGraph.getConfig(), primary, name, name, indexRecordLen);
> >
> > became
> >
> > TupleIndex newIndex = SetupTDB.makeTupleIndex(tmpLocation, primary, name,
> > name, indexRecordLen);
>
> Doing anything with internal classes is at your own risk!
>
> >
> > 4) More obscure is the following change:
> >
> > SetupTDB.globalConfig.setProperty(Names.pBlockReadCacheSize,
> > blockReadCacheSize.toString());
>
> SetupTDB is a wrapper to old code.
>
> SystemParams has the constants as finals.
>
> See SetupTDB for setting constant from properties.
>
> You may wish to submit an enhancement if you want to manipulate cache
> sizes in detail. It's been on the "it would be nice" list but it isn't
> something many people need as they run on 64bit in production. I wonder
> if parameters ought to be a related to location.
>
> >
> > That does not work anymore. It is unclear where I can set this property
> > now. I could use help with this one
> >
> >
> > In general, API changes are unavoidable, but they should be documented
> > per release in a readme.
> >
> > Simon
> >
> >
>
> Andy
>
>
>
Re: API changes
Posted by Simon Helsen <sh...@ca.ibm.com>.
Andy,
let me pick out this comment of yours:
"You may wish to submit an enhancement if you want to manipulate cache
sizes in detail. It's been on the "it would be nice" list but it isn't
something many people need as they run on 64bit in production. I wonder
if parameters ought to be a related to location.
"
why do you say this? We run only on 64 bit, but always in direct mode
because mapped mode does not allow us to remove the index (as a reindex
operation) because of the infamous JVM bug which holds a lock on the
involved files, at least on Windows. Moreover, it is unstable under system
crashes since the actual flush may not have taken place after a close.
That too is probably more a problem on Windows. The latter may be outdated
now with transactions and journaling, although it has to be seen if we
could still corrupt the index this way. Either way, I would never
recommend using mapped I/O for production purposes. We tried and miserably
failed. And yes, not all production systems are linux
Anyhow, I'll look into the enhancement
Simon
From:
Andy Seaborne <an...@apache.org>
To:
dev@jena.apache.org
Date:
05/17/2012 04:28 AM
Subject:
Re: API changes
On 16/05/12 23:50, Simon Helsen wrote:
> Andy,
>
> I know you asked a while back about what API changes I had noticed.
There
> are a number of things that have changed since 2.6.3,
October 2010
> most of them are
> smaller and manageable (albeit annoying). Here a few examples:
>
> 1) DatasetGraphTDB subset = TDBFactory.createDatasetGraph();
>
> now has to be
>
> DatasetGraph subset = TDBFactory.createDatasetGraph();
DatasetGraphTDB is an internal class (and it's not DatasetGraphTDB now :-)
>
> 2) Another example is:
>
> queryContext.setDataset(new DatasetImpl(subset));
>
> now has to be
>
> queryContext.setDataset(DatasetImpl.wrap(subset));
I don't understand this one - what's queryContext?
The only setDataset is on ExecutionContext in
com.hp.hpl.jena.sparql.engine
which is inside the engine and not an API call.
There is QueryExecutionFactory for creating a QueryExecution for dataset.
The call you want is
DatasetFactory.create(DatasetGraph)
DatasetImpl is, well, an Impl class.
> 3) TupleIndex newIndex = SetupTDB.makeTupleIndex(tmpLocation,
> srcGraph.getConfig(), primary, name, name, indexRecordLen);
>
> became
>
> TupleIndex newIndex = SetupTDB.makeTupleIndex(tmpLocation, primary,
name,
> name, indexRecordLen);
Doing anything with internal classes is at your own risk!
>
> 4) More obscure is the following change:
>
> SetupTDB.globalConfig.setProperty(Names.pBlockReadCacheSize,
> blockReadCacheSize.toString());
SetupTDB is a wrapper to old code.
SystemParams has the constants as finals.
See SetupTDB for setting constant from properties.
You may wish to submit an enhancement if you want to manipulate cache
sizes in detail. It's been on the "it would be nice" list but it isn't
something many people need as they run on 64bit in production. I wonder
if parameters ought to be a related to location.
>
> That does not work anymore. It is unclear where I can set this property
> now. I could use help with this one
>
>
> In general, API changes are unavoidable, but they should be documented
> per release in a readme.
>
> Simon
>
>
Andy
Re: API changes
Posted by Paolo Castagna <ca...@googlemail.com>.
Andy Seaborne wrote:
> You may wish to submit an enhancement if you want to manipulate cache
> sizes in detail. It's been on the "it would be nice" list but it isn't
> something many people need as they run on 64bit in production. I wonder
> if parameters ought to be a related to location.
Marked as 'new feature' in JIRA: JENA-133
https://issues.apache.org/jira/browse/JENA-133
Paolo
Re: API changes
Posted by Andy Seaborne <an...@apache.org>.
On 16/05/12 23:50, Simon Helsen wrote:
> Andy,
>
> I know you asked a while back about what API changes I had noticed. There
> are a number of things that have changed since 2.6.3,
October 2010
> most of them are
> smaller and manageable (albeit annoying). Here a few examples:
>
> 1) DatasetGraphTDB subset = TDBFactory.createDatasetGraph();
>
> now has to be
>
> DatasetGraph subset = TDBFactory.createDatasetGraph();
DatasetGraphTDB is an internal class (and it's not DatasetGraphTDB now :-)
>
> 2) Another example is:
>
> queryContext.setDataset(new DatasetImpl(subset));
>
> now has to be
>
> queryContext.setDataset(DatasetImpl.wrap(subset));
I don't understand this one - what's queryContext?
The only setDataset is on ExecutionContext in
com.hp.hpl.jena.sparql.engine
which is inside the engine and not an API call.
There is QueryExecutionFactory for creating a QueryExecution for dataset.
The call you want is
DatasetFactory.create(DatasetGraph)
DatasetImpl is, well, an Impl class.
> 3) TupleIndex newIndex = SetupTDB.makeTupleIndex(tmpLocation,
> srcGraph.getConfig(), primary, name, name, indexRecordLen);
>
> became
>
> TupleIndex newIndex = SetupTDB.makeTupleIndex(tmpLocation, primary, name,
> name, indexRecordLen);
Doing anything with internal classes is at your own risk!
>
> 4) More obscure is the following change:
>
> SetupTDB.globalConfig.setProperty(Names.pBlockReadCacheSize,
> blockReadCacheSize.toString());
SetupTDB is a wrapper to old code.
SystemParams has the constants as finals.
See SetupTDB for setting constant from properties.
You may wish to submit an enhancement if you want to manipulate cache
sizes in detail. It's been on the "it would be nice" list but it isn't
something many people need as they run on 64bit in production. I wonder
if parameters ought to be a related to location.
>
> That does not work anymore. It is unclear where I can set this property
> now. I could use help with this one
>
>
> In general, API changes are unavoidable, but they should be documented
> per release in a readme.
>
> Simon
>
>
Andy