You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kafka.apache.org by elizabeth <el...@gmail.com> on 2013/08/13 07:07:43 UTC
Unmap before resizing
Hi, I just filed the JIRA as instructed.
https://issues.apache.org/jira/browse/KAFKA-1008
Thanks!
=====
Hi, can you file a JIRA and attach the patch there? This does the Apache
copyright stuff...
Instructions here:http://kafka.apache.org/contributing.html
This seems like a good thing to have. Is there a more portable way to do
this? Obviously this would not work on a non-sun JVM. I think the ideal
would be to add a tryUnmap() method that attempts the case but swallows any
exception.
Also I would like to fully understand the Sun's objection to allowing user
code to unmap buffers. In that ticket on the previous thread on
Windows+mmap there was a mention that forcing the unmap was somehow a
security hole. I don't understand this. Unix obviously supports unmap and
there is no security issue with that as normal process-level memory
isolation protects processes from reading each others memory or files. I
suspect maybe this would only be true in cases where you are trying to
isolate different code in the same JVM which is not a valid use case. But I
want to fully understand the issue before we add something someone claims
is unsafe.
-Jay
On Thu, Jul 25, 2013 at 12:07 AM, lizziew <gi...@git.apache.org> wrote:
> GitHub user lizziew opened a pull request:
>
> https://github.com/apache/kafka/pull/6
>
> Unmap before resizing
>
> While I was studying how MappedByteBuffer works, I saw a sharing
> runtime exception on Windows. I applied what I learned to generate a patch
> which uses an internal open JDK API to solve this problem.
>
> ---
>
> Caused by: java.io.IOException: The requested operation cannot be
> performed
> on a
> file with a user-mapped section open
> at java.io.RandomAccessFile.setLength(Native Method)
> at kafka.log.OffsetIndex.liftedTree2$1(OffsetIndex.scala:263)
> at kafka.log.OffsetIndex.resize(OffsetIndex.scala:262)
> at kafka.log.OffsetIndex.trimToValidSize(OffsetIndex.scala:247)
> at kafka.log.Log.rollToOffset(Log.scala:518)
> at kafka.log.Log.roll(Log.scala:502)
> at kafka.log.Log.maybeRoll(Log.scala:484)
> at kafka.log.Log.append(Log.scala:297)
>
> You can merge this pull request into a Git repository by running:
>
> $ git pull https://github.com/lizziew/kafka 0.8
>
> Alternatively you can review and apply these changes as the patch at:
>
> https://github.com/apache/kafka/pull/6.patch
>
> ----
>
> ----
>
>
Re: Unmap before resizing
Posted by Jay Kreps <ja...@gmail.com>.
Awesome, thanks for the contribution! I posted a few follow-up items on the
ticket.
-Jay
On Mon, Aug 12, 2013 at 10:07 PM, elizabeth <el...@gmail.com>wrote:
> Hi, I just filed the JIRA as instructed.
>
> https://issues.apache.org/jira/browse/KAFKA-1008
>
> Thanks!
>
> =====
>
> Hi, can you file a JIRA and attach the patch there? This does the Apache
> copyright stuff...
>
> Instructions here:http://kafka.apache.org/contributing.html
>
> This seems like a good thing to have. Is there a more portable way to do
> this? Obviously this would not work on a non-sun JVM. I think the ideal
> would be to add a tryUnmap() method that attempts the case but swallows any
> exception.
>
> Also I would like to fully understand the Sun's objection to allowing user
> code to unmap buffers. In that ticket on the previous thread on
> Windows+mmap there was a mention that forcing the unmap was somehow a
> security hole. I don't understand this. Unix obviously supports unmap and
> there is no security issue with that as normal process-level memory
> isolation protects processes from reading each others memory or files. I
> suspect maybe this would only be true in cases where you are trying to
> isolate different code in the same JVM which is not a valid use case. But I
> want to fully understand the issue before we add something someone claims
> is unsafe.
>
> -Jay
>
>
> On Thu, Jul 25, 2013 at 12:07 AM, lizziew <gi...@git.apache.org> wrote:
>
> > GitHub user lizziew opened a pull request:
> >
> > https://github.com/apache/kafka/pull/6
> >
> > Unmap before resizing
> >
> > While I was studying how MappedByteBuffer works, I saw a sharing
> > runtime exception on Windows. I applied what I learned to generate a
> patch
> > which uses an internal open JDK API to solve this problem.
> >
> > ---
> >
> > Caused by: java.io.IOException: The requested operation cannot be
> > performed
> > on a
> > file with a user-mapped section open
> > at java.io.RandomAccessFile.setLength(Native Method)
> > at kafka.log.OffsetIndex.liftedTree2$1(OffsetIndex.scala:263)
> > at kafka.log.OffsetIndex.resize(OffsetIndex.scala:262)
> > at
> kafka.log.OffsetIndex.trimToValidSize(OffsetIndex.scala:247)
> > at kafka.log.Log.rollToOffset(Log.scala:518)
> > at kafka.log.Log.roll(Log.scala:502)
> > at kafka.log.Log.maybeRoll(Log.scala:484)
> > at kafka.log.Log.append(Log.scala:297)
> >
> > You can merge this pull request into a Git repository by running:
> >
> > $ git pull https://github.com/lizziew/kafka 0.8
> >
> > Alternatively you can review and apply these changes as the patch at:
> >
> > https://github.com/apache/kafka/pull/6.patch
> >
> > ----
> >
> > ----
> >
> >
>