[jira] [Comment Edited] (ARTEMIS-2131) Error compacting journal

["Francesco Nigro (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/ARTEMIS-2131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16653386#comment-16653386 ] 

Francesco Nigro edited comment on ARTEMIS-2131 at 10/17/18 11:25 AM:
---------------------------------------------------------------------

I have a strong feeling these issues are somehow related: [https://github.com/apache/activemq-artemis/pull/2250]

If you can try that branch we can check if it fixes the issue.

The original problem seems that the buffer used while compacting is considered a duplicate or a slice:
{code:java}
sequentialFile.writeDirect(writingChannel.toByteBuffer(), true);{code}
Due to this reason it cannot be released correctly while the mentioned PR is making use of a call that won't try to pool or release it:
{code:java}
 final ByteBuffer byteBuffer = writingChannelNioBuffer;
 final int readerIndex = writingChannel.readerIndex();
 byteBuffer.clear().position(readerIndex).limit(readerIndex + writingChannel.readableBytes());
 sequentialFile.blockingWriteDirect(byteBuffer, false, false);{code}

With blockingWriteDirect defined as:

{code:java}
   /**
    * Write directly to the file without using any intermediate buffer and wait completion.<br>
    * If {@code releaseBuffer} is {@code true} the provided {@code bytes} should be released
    * through {@link SequentialFileFactory#releaseBuffer(ByteBuffer)}, if supported.
    *
    * @param bytes         the ByteBuffer must be compatible with the SequentialFile implementation (AIO or
    *                      NIO). If {@code releaseBuffer} is {@code true} use a buffer from
    *                      {@link SequentialFileFactory#newBuffer(int)}, {@link SequentialFileFactory#allocateDirectBuffer(int)}
    *                      otherwise.
    * @param sync          if {@code true} will durable flush the written data on the file, {@code false} otherwise
    * @param releaseBuffer if {@code true} will release the buffer, {@code false} otherwise
    */
   void blockingWriteDirect(ByteBuffer bytes, boolean sync, boolean releaseBuffer) throws Exception;
{code}



was (Author: nigro.fra@gmail.com):
I have a strong feeling these issues are somehow related: [https://github.com/apache/activemq-artemis/pull/2250]

If you can try that branch we can check if it fixes the issue.

The original problem seems that the buffer used while compacting is considered a duplicate or a slice:
{code:java}
sequentialFile.writeDirect(writingChannel.toByteBuffer(), true);{code}
Due to this reason it cannot be released correctly while the mentioned PR is making use of a call that won't try to pool or release it:
{code:java}
 final ByteBuffer byteBuffer = writingChannelNioBuffer;
 final int readerIndex = writingChannel.readerIndex();
 byteBuffer.clear().position(readerIndex).limit(readerIndex + writingChannel.readableBytes());
 sequentialFile.blockingWriteDirect(byteBuffer, false, false);{code}

> Error compacting journal
> ------------------------
>
>                 Key: ARTEMIS-2131
>                 URL: https://issues.apache.org/jira/browse/ARTEMIS-2131
>             Project: ActiveMQ Artemis
>          Issue Type: Bug
>    Affects Versions: 2.6.3
>         Environment: OpenJDK 11
>            Reporter: Borna
>            Priority: Major
>
> Sometimes (especially when under higher load), Artemis throws the following exception:
> {{[10:03:01.377] ERROR [Thread-9 (ActiveMQ-IO-server-org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$6@2484c868)] o.a.a.a.journal - AMQ144003: Error compacting java.lang.reflect.InvocationTargetException: null at java.base/jdk.internal.reflect.GeneratedMethodAccessor991.invoke(Unknown Source) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:566) at io.netty.util.internal.CleanerJava9.freeDirectBuffer(CleanerJava9.java:88) at io.netty.util.internal.PlatformDependent.freeDirectBuffer(PlatformDependent.java:392) at org.apache.activemq.artemis.core.io.mapped.MappedSequentialFileFactory.releaseBuffer(MappedSequentialFileFactory.java:134) at org.apache.activemq.artemis.core.io.mapped.MappedSequentialFile.writeDirect(MappedSequentialFile.java:272) at org.apache.activemq.artemis.core.io.mapped.TimedSequentialFile.writeDirect(TimedSequentialFile.java:164) at org.apache.activemq.artemis.core.journal.impl.AbstractJournalUpdateTask.flush(AbstractJournalUpdateTask.java:217) at org.apache.activemq.artemis.core.journal.impl.JournalImpl.compact(JournalImpl.java:1613) at org.apache.activemq.artemis.core.journal.impl.JournalImpl$14.run(JournalImpl.java:2159) at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:42) at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:31) at org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:66) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118) Caused by: java.lang.IllegalArgumentException: duplicate or slice at jdk.unsupported/sun.misc.Unsafe.invokeCleaner(Unsafe.java:1238) ... 17 common frames omitted}}
>  
> As a consequence, it looks like old journal files are not reclaimed and consumption of disk space grows indefinitely.
> Tested using OpenJDK 11.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)