You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by "Kristofer Karlsson (JIRA)" <ji...@apache.org> on 2013/02/05 14:09:12 UTC
[jira] [Updated] (LUCENE-4755) Unmap MMapIndexInput's in a delayed
way, and avoid WeakReference usage.
[ https://issues.apache.org/jira/browse/LUCENE-4755?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Kristofer Karlsson updated LUCENE-4755:
---------------------------------------
Description:
(Most of this is shamelessly borrowed from Uwe Schindler)
It would be nice to move away from using WeakReference to clean up clones.
Instead, the clones could depend on the master by using a shared boolean closed-flag.
In order to ensure visibility of this value, or at least make it less likely to crash, we could delay the unmapping operation.
Rough suggestion of changes:
{code:java}
public class ByteBufferUnmapper {
/**
* <code>true</code>, if this platform supports unmapping mmapped files.
*/
public static final boolean UNMAP_SUPPORTED;
private static final ScheduledExecutorService SCHEDULED_EXECUTOR_SERVICE = Executors.newSingleThreadScheduledExecutor(new NamedThreadFactory("unmapper-"));
static {
boolean v = false;
try {
Class.forName("sun.misc.Cleaner");
Class.forName("java.nio.DirectByteBuffer")
.getMethod("cleaner");
v = true;
} catch (Exception e) {
// Do nothing
} finally {
UNMAP_SUPPORTED = v;
}
}
public static void unmap(final ByteBuffer buffer) throws IOException {
try {
AccessController.doPrivileged(new PrivilegedExceptionAction<Object>() {
public Object run() throws Exception {
Method getCleanerMethod = buffer.getClass().getMethod("cleaner");
getCleanerMethod.setAccessible(true);
final Object cleaner = getCleanerMethod.invoke(buffer);
if (cleaner != null) {
cleaner.getClass().getMethod("clean").invoke(cleaner);
}
return null;
}
});
} catch (PrivilegedActionException e) {
final IOException ioe = new IOException("unable to unmap the mapped buffer");
ioe.initCause(e.getCause());
throw ioe;
}
}
public static void unmapLater(final ByteBuffer buffer, long delay, TimeUnit unit) {
SCHEDULED_EXECUTOR_SERVICE.schedule(new Runnable() {
public void run() {
try {
unmap(buffer);
} catch (IOException e) {
e.printStackTrace();
}
}
}, delay, unit);
}
}
{code}
{code:java}
// MMapDirectory
final void cleanMapping(final ByteBuffer buffer) throws IOException {
if (useUnmapHack) {
ByteBufferUnmapper.unmapLater(buffer, 10, TimeUnit.SECONDS);
}
}
{code}
{code:java}
// MMapIndexInput
@Override
public short readShort() throws IOException {
if (closed[0]) {
throw new AlreadyClosedException("MMapIndexInput already closed: " + this);
}
{code}
{code:java}
@Override
public void close() throws IOException {
try {
if (isClone || buffers == null) return;
closed[0] = true;
// make local copy, then un-set early
final ByteBuffer[] bufs = buffers;
for (final ByteBuffer b : bufs) {
cleanMapping(b);
}
} finally {
unsetBuffers();
}
}
{code}
was:
(Most of this is shamelessly borrowed from Uwe Schindler)
It would be nice to move away from using WeakReference to clean up clones.
Instead, the clones could depend on the master by using a shared boolean closed-flag.
In order to ensure visibility of this value, or at least make it less likely to crash, we could delay the unmapping operation.
Rough suggestion of changes:
<code>
public class ByteBufferUnmapper {
/**
* <code>true</code>, if this platform supports unmapping mmapped files.
*/
public static final boolean UNMAP_SUPPORTED;
private static final ScheduledExecutorService SCHEDULED_EXECUTOR_SERVICE = Executors.newSingleThreadScheduledExecutor(new NamedThreadFactory("unmapper-"));
static {
boolean v = false;
try {
Class.forName("sun.misc.Cleaner");
Class.forName("java.nio.DirectByteBuffer")
.getMethod("cleaner");
v = true;
} catch (Exception e) {
// Do nothing
} finally {
UNMAP_SUPPORTED = v;
}
}
public static void unmap(final ByteBuffer buffer) throws IOException {
try {
AccessController.doPrivileged(new PrivilegedExceptionAction<Object>() {
public Object run() throws Exception {
Method getCleanerMethod = buffer.getClass().getMethod("cleaner");
getCleanerMethod.setAccessible(true);
final Object cleaner = getCleanerMethod.invoke(buffer);
if (cleaner != null) {
cleaner.getClass().getMethod("clean").invoke(cleaner);
}
return null;
}
});
} catch (PrivilegedActionException e) {
final IOException ioe = new IOException("unable to unmap the mapped buffer");
ioe.initCause(e.getCause());
throw ioe;
}
}
public static void unmapLater(final ByteBuffer buffer, long delay, TimeUnit unit) {
SCHEDULED_EXECUTOR_SERVICE.schedule(new Runnable() {
public void run() {
try {
unmap(buffer);
} catch (IOException e) {
e.printStackTrace();
}
}
}, delay, unit);
}
}
</code>
<code>
// MMapDirectory
final void cleanMapping(final ByteBuffer buffer) throws IOException {
if (useUnmapHack) {
ByteBufferUnmapper.unmapLater(buffer, 10, TimeUnit.SECONDS);
}
}
</code>
<code>
// MMapIndexInput
@Override
public short readShort() throws IOException {
if (closed[0]) {
throw new AlreadyClosedException("MMapIndexInput already closed: " + this);
}
</code>
<code>
@Override
public void close() throws IOException {
try {
if (isClone || buffers == null) return;
closed[0] = true;
// make local copy, then un-set early
final ByteBuffer[] bufs = buffers;
for (final ByteBuffer b : bufs) {
cleanMapping(b);
}
} finally {
unsetBuffers();
}
}
</code>
> Unmap MMapIndexInput's in a delayed way, and avoid WeakReference usage.
> -----------------------------------------------------------------------
>
> Key: LUCENE-4755
> URL: https://issues.apache.org/jira/browse/LUCENE-4755
> Project: Lucene - Core
> Issue Type: Improvement
> Components: core/store
> Reporter: Kristofer Karlsson
>
> (Most of this is shamelessly borrowed from Uwe Schindler)
> It would be nice to move away from using WeakReference to clean up clones.
> Instead, the clones could depend on the master by using a shared boolean closed-flag.
> In order to ensure visibility of this value, or at least make it less likely to crash, we could delay the unmapping operation.
> Rough suggestion of changes:
> {code:java}
> public class ByteBufferUnmapper {
> /**
> * <code>true</code>, if this platform supports unmapping mmapped files.
> */
> public static final boolean UNMAP_SUPPORTED;
> private static final ScheduledExecutorService SCHEDULED_EXECUTOR_SERVICE = Executors.newSingleThreadScheduledExecutor(new NamedThreadFactory("unmapper-"));
> static {
> boolean v = false;
> try {
> Class.forName("sun.misc.Cleaner");
> Class.forName("java.nio.DirectByteBuffer")
> .getMethod("cleaner");
> v = true;
> } catch (Exception e) {
> // Do nothing
> } finally {
> UNMAP_SUPPORTED = v;
> }
> }
> public static void unmap(final ByteBuffer buffer) throws IOException {
> try {
> AccessController.doPrivileged(new PrivilegedExceptionAction<Object>() {
> public Object run() throws Exception {
> Method getCleanerMethod = buffer.getClass().getMethod("cleaner");
> getCleanerMethod.setAccessible(true);
> final Object cleaner = getCleanerMethod.invoke(buffer);
> if (cleaner != null) {
> cleaner.getClass().getMethod("clean").invoke(cleaner);
> }
> return null;
> }
> });
> } catch (PrivilegedActionException e) {
> final IOException ioe = new IOException("unable to unmap the mapped buffer");
> ioe.initCause(e.getCause());
> throw ioe;
> }
> }
> public static void unmapLater(final ByteBuffer buffer, long delay, TimeUnit unit) {
> SCHEDULED_EXECUTOR_SERVICE.schedule(new Runnable() {
> public void run() {
> try {
> unmap(buffer);
> } catch (IOException e) {
> e.printStackTrace();
> }
> }
> }, delay, unit);
> }
> }
> {code}
> {code:java}
> // MMapDirectory
> final void cleanMapping(final ByteBuffer buffer) throws IOException {
> if (useUnmapHack) {
> ByteBufferUnmapper.unmapLater(buffer, 10, TimeUnit.SECONDS);
> }
> }
> {code}
> {code:java}
> // MMapIndexInput
> @Override
> public short readShort() throws IOException {
> if (closed[0]) {
> throw new AlreadyClosedException("MMapIndexInput already closed: " + this);
> }
> {code}
> {code:java}
> @Override
> public void close() throws IOException {
> try {
> if (isClone || buffers == null) return;
> closed[0] = true;
>
> // make local copy, then un-set early
> final ByteBuffer[] bufs = buffers;
>
> for (final ByteBuffer b : bufs) {
> cleanMapping(b);
> }
> } finally {
> unsetBuffers();
> }
> }
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org