You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@ignite.apache.org by Kamil Misuth <ki...@ethome.sk> on 2017/04/09 23:53:41 UTC
Ignite on FreeBSD 11 and OpenJDK
Greetings,
OpenJDK (7 and 8) HotSpot JVM SIGSEGVs on FreeBSD 11 as soon as node
joins a topology and starts to communicate via DirectNioClientWorker.
The root cause is DirectByteBufferStreamImpl (both versions) which uses
GridUnsafe.getXXX/putXXX(Object object, offset, value) methods to
manipulate DirectByteBuffer, whereas it should really be using
GridUnsafe.getXXX/putXXX(address, value), since DirectByteBuffer is
allocated on C heap (off java heap).
Notice that at least one instance of the same problem is known to exist
in another project using Unsafe
https://issues.apache.org/jira/browse/CASSANDRA-8325 .
The OpenJDK source of Unsafe is more or less clear on this
http://hg.openjdk.java.net/jdk8u/jdk8u60/jdk/file/935758609767/src/share/classes/sun/misc/Unsafe.java#l391
I have prepared a simple fix here
https://github.com/apache/ignite/compare/1.9.0-rc2...kimec:freebsd-support .
However, I am not sure if the solution is right in regard to overall
ignite performance.
I've tried to compile ignite-core with tests and after applying my
changes was able to pass all the basic stuff until the performance test
stage at which point my machine run out of RAM and swap space (some 10
GB)... Not sure if this is how the tests are supposed to be. After
compiling with -DskipTests I was able to create FreeBSD 11 - CentOS 7
two node cluster and everything seemed OK (the two nodes shared an IGFS
instance backed by replicated caches).
Please note that OpenJDK on different systems as well as Oracle JDK (via
Linux compatility layer) on FreeBSD seem to be more forgiving and does
not SIGSEGV.
I've based my branch on 1.9.0-rc2 since tag 1.9.0 has already POM with
version 2.0.
Kamil
Re: Ignite on FreeBSD 11 and OpenJDK
Posted by Kamil Misuth <ki...@ethome.sk>.
Hi Denis,
that is great. Thank you for the link!
I hope Ignite 2.0 or some later version will eventually work on FreeBSD
with OpenJDK.
Kamil
On 2017-04-19 00:53, Denis Magda wrote:
> Hi Kamil,
>
> The documentation for the new off-heap page memory is being prepared
> and will become public right after 2.0 release. You can track the
> progress here:
> https://issues.apache.org/jira/browse/IGNITE-4961
>
> \u2014
> Denis
>
>> On Apr 18, 2017, at 3:47 PM, Kamil Misuth <ki...@ethome.sk> wrote:
>>
>> Replying, to my own question.
>>
>> Indeed, it seems to be the case that for Ignite 2.0, everything will
>> be off heap. Denis Magda just (23 hours ago) updated the relevant wiki
>> page.
>>
>> https://cwiki.apache.org/confluence/pages/diffpagesbyversion.action?pageId=67638199&selectedPageVersions=10&selectedPageVersions=8
>>
>> Sounds cool to me :).
>>
>> Kamil
>>
>> On 2017-04-19 00:43, Kamil Misuth wrote:
>>> Sure, Andrey.
>>> In mean time, a fact-checking question:
>>> I haven't really gone through the new memory policy functionality in
>>> depth, but it does seem that for Ignite 2.0 your aim is to move data
>>> off heap implicitly for all caches (and possibly backup the off heap
>>> region by memory mapped file(s)). Is this what you meant by "many
>>> memory related fixes"?
>>> Thanks.
>>> Kamil
>>> On 2017-04-13 13:08, Andrey Gura wrote:
>>>> Kamil,
>>>> thanks a lot for your help. As I told before I need some time for
>>>> problem analyse. After it I'll create JIRA ticket and share link to
>>>> it
>>>> in this thread.
>>>> Thank you again!
>>>> On Wed, Apr 12, 2017 at 11:56 PM, Kamil Misuth <ki...@ethome.sk>
>>>> wrote:
>>>>> Hi Andrey,
>>>>> I've built ignite-3477-master (commit hash 5839f481b7) today.
>>>>> Apart from the fact that some other Configuration APIs changed,
>>>>> CacheMemoryMode disapeared. I guess this is related to
>>>>> http://apache-ignite-developers.2346864.n4.nabble.com/IGNITE-4758-introducing-cache-memory-policies-td14957.html
>>>>> Anyway, I've got the following exception on the first try, since
>>>>> ignite-indexing 2.0.0 is compiled against H2 1.4.191 and Spring
>>>>> Boot
>>>>> 1.5.2.RELEASE managed H2 version is 1.4.193 and apparently
>>>>> org.h2.result.Row
>>>>> changed from abstract class to an interface between the two
>>>>> versions.
>>>>> Caused by: java.lang.IncompatibleClassChangeError: class
>>>>> org.apache.ignite.internal.processors.query.h2.opt.GridH2Row has
>>>>> interface
>>>>> org.h2.result.Row as super class
>>>>> After fixing the dependency issue, I've tried to create two node
>>>>> cluster and
>>>>> JVM SIGSEGVed in DirectNioClientWorker just as before.
>>>>> Attaching crash logs from both nodes (running on the same machine
>>>>> FreeBSD
>>>>> 11) to this e-mail.
>>>>> Kamil
>>>>> On 2017-04-12 00:49, Kamil Misuth wrote:
>>>>>> Sure thing.
>>>>>> I will check out ignite-3477-master as soon as I have some time
>>>>>> tomorrow.
>>>>>> Kamil
>>>>>> On 04/11/2017 05:48 PM, Andrey Gura wrote:
>>>>>>> Thanks for provided information. I need additional time for
>>>>>>> problem
>>>>>>> investigation.
>>>>>>> You can also try code from ignite-3477-master branch. This branch
>>>>>>> contains many memory related fixes but it isn't stable yet.
>>>>>>> On Mon, Apr 10, 2017 at 11:37 PM, kimec.ethome.sk
>>>>>>> <ki...@ethome.sk>
>>>>>>> wrote:
>>>>>>>> Hi Andrey,
>>>>>>>> sorry, I've got ahead of my self.
>>>>>>>> I am on FreeBSD 11.0-RELEASE-p1 amd64
>>>>>>>> With OpenJDK Runtime Environment 1.8.0_121-b13 Oracle
>>>>>>>> Corporation
>>>>>>>> OpenJDK
>>>>>>>> 64-Bit Server VM 25.121-b13
>>>>>>>> hw.model: Intel(R) Core(TM) i7-4702MQ CPU @ 2.20GHz
>>>>>>>> hw.machine_arch: amd64
>>>>>>>> hw.ncpu: 8
>>>>>>>> hw.physmem: 8251813888
>>>>>>>> Core dump is 1 GB, so I guess that is no go. I am attaching
>>>>>>>> crash log to
>>>>>>>> this e-mail.
>>>>>>>> I have uploaded the project I've used during my testing here
>>>>>>>> https://github.com/kimec/ignite-spring-boot .
>>>>>>>> The sample works perfectly well with stock ignite-core on Linux
>>>>>>>> OpenJDK
>>>>>>>> 8
>>>>>>>> xs64 CentOS 7 .
>>>>>>>> Kamil
>>>>>>>> On 2017-04-10 12:24, Andrey Gura wrote:
>>>>>>>>> Hi,
>>>>>>>>> could you please share core dump file? If not, it would be
>>>>>>>>> helpful to
>>>>>>>>> know what is CPU architecture on this server.
>>>>>>>>> On Mon, Apr 10, 2017 at 2:53 AM, Kamil Misuth <ki...@ethome.sk>
>>>>>>>>> wrote:
>>>>>>>>>> Greetings,
>>>>>>>>>> OpenJDK (7 and 8) HotSpot JVM SIGSEGVs on FreeBSD 11 as soon
>>>>>>>>>> as node
>>>>>>>>>> joins a
>>>>>>>>>> topology and starts to communicate via DirectNioClientWorker.
>>>>>>>>>> The root cause is DirectByteBufferStreamImpl (both versions)
>>>>>>>>>> which
>>>>>>>>>> uses
>>>>>>>>>> GridUnsafe.getXXX/putXXX(Object object, offset, value) methods
>>>>>>>>>> to
>>>>>>>>>> manipulate
>>>>>>>>>> DirectByteBuffer, whereas it should really be using
>>>>>>>>>> GridUnsafe.getXXX/putXXX(address, value), since
>>>>>>>>>> DirectByteBuffer is
>>>>>>>>>> allocated on C heap (off java heap).
>>>>>>>>>> Notice that at least one instance of the same problem is known
>>>>>>>>>> to
>>>>>>>>>> exist
>>>>>>>>>> in
>>>>>>>>>> another project using Unsafe
>>>>>>>>>> https://issues.apache.org/jira/browse/CASSANDRA-8325 .
>>>>>>>>>> The OpenJDK source of Unsafe is more or less clear on this
>>>>>>>>>> http://hg.openjdk.java.net/jdk8u/jdk8u60/jdk/file/935758609767/src/share/classes/sun/misc/Unsafe.java#l391
>>>>>>>>>> I have prepared a simple fix here
>>>>>>>>>> https://github.com/apache/ignite/compare/1.9.0-rc2...kimec:freebsd-support
>>>>>>>>>> .
>>>>>>>>>> However, I am not sure if the solution is right in regard to
>>>>>>>>>> overall
>>>>>>>>>> ignite
>>>>>>>>>> performance.
>>>>>>>>>> I've tried to compile ignite-core with tests and after
>>>>>>>>>> applying my
>>>>>>>>>> changes
>>>>>>>>>> was able to pass all the basic stuff until the performance
>>>>>>>>>> test stage
>>>>>>>>>> at
>>>>>>>>>> which point my machine run out of RAM and swap space (some 10
>>>>>>>>>> GB)...
>>>>>>>>>> Not
>>>>>>>>>> sure if this is how the tests are supposed to be. After
>>>>>>>>>> compiling with
>>>>>>>>>> -DskipTests I was able to create FreeBSD 11 - CentOS 7 two
>>>>>>>>>> node
>>>>>>>>>> cluster
>>>>>>>>>> and
>>>>>>>>>> everything seemed OK (the two nodes shared an IGFS instance
>>>>>>>>>> backed by
>>>>>>>>>> replicated caches).
>>>>>>>>>> Please note that OpenJDK on different systems as well as
>>>>>>>>>> Oracle JDK
>>>>>>>>>> (via
>>>>>>>>>> Linux compatility layer) on FreeBSD seem to be more forgiving
>>>>>>>>>> and does
>>>>>>>>>> not
>>>>>>>>>> SIGSEGV.
>>>>>>>>>> I've based my branch on 1.9.0-rc2 since tag 1.9.0 has already
>>>>>>>>>> POM with
>>>>>>>>>> version 2.0.
>>>>>>>>>> Kamil
Re: Ignite on FreeBSD 11 and OpenJDK
Posted by Denis Magda <dm...@apache.org>.
Hi Kamil,
The documentation for the new off-heap page memory is being prepared and will become public right after 2.0 release. You can track the progress here:
https://issues.apache.org/jira/browse/IGNITE-4961
—
Denis
> On Apr 18, 2017, at 3:47 PM, Kamil Misuth <ki...@ethome.sk> wrote:
>
> Replying, to my own question.
>
> Indeed, it seems to be the case that for Ignite 2.0, everything will be off heap. Denis Magda just (23 hours ago) updated the relevant wiki page.
>
> https://cwiki.apache.org/confluence/pages/diffpagesbyversion.action?pageId=67638199&selectedPageVersions=10&selectedPageVersions=8
>
> Sounds cool to me :).
>
> Kamil
>
> On 2017-04-19 00:43, Kamil Misuth wrote:
>> Sure, Andrey.
>> In mean time, a fact-checking question:
>> I haven't really gone through the new memory policy functionality in
>> depth, but it does seem that for Ignite 2.0 your aim is to move data
>> off heap implicitly for all caches (and possibly backup the off heap
>> region by memory mapped file(s)). Is this what you meant by "many
>> memory related fixes"?
>> Thanks.
>> Kamil
>> On 2017-04-13 13:08, Andrey Gura wrote:
>>> Kamil,
>>> thanks a lot for your help. As I told before I need some time for
>>> problem analyse. After it I'll create JIRA ticket and share link to it
>>> in this thread.
>>> Thank you again!
>>> On Wed, Apr 12, 2017 at 11:56 PM, Kamil Misuth <ki...@ethome.sk> wrote:
>>>> Hi Andrey,
>>>> I've built ignite-3477-master (commit hash 5839f481b7) today.
>>>> Apart from the fact that some other Configuration APIs changed,
>>>> CacheMemoryMode disapeared. I guess this is related to
>>>> http://apache-ignite-developers.2346864.n4.nabble.com/IGNITE-4758-introducing-cache-memory-policies-td14957.html
>>>> Anyway, I've got the following exception on the first try, since
>>>> ignite-indexing 2.0.0 is compiled against H2 1.4.191 and Spring Boot
>>>> 1.5.2.RELEASE managed H2 version is 1.4.193 and apparently org.h2.result.Row
>>>> changed from abstract class to an interface between the two versions.
>>>> Caused by: java.lang.IncompatibleClassChangeError: class
>>>> org.apache.ignite.internal.processors.query.h2.opt.GridH2Row has interface
>>>> org.h2.result.Row as super class
>>>> After fixing the dependency issue, I've tried to create two node cluster and
>>>> JVM SIGSEGVed in DirectNioClientWorker just as before.
>>>> Attaching crash logs from both nodes (running on the same machine FreeBSD
>>>> 11) to this e-mail.
>>>> Kamil
>>>> On 2017-04-12 00:49, Kamil Misuth wrote:
>>>>> Sure thing.
>>>>> I will check out ignite-3477-master as soon as I have some time tomorrow.
>>>>> Kamil
>>>>> On 04/11/2017 05:48 PM, Andrey Gura wrote:
>>>>>> Thanks for provided information. I need additional time for problem
>>>>>> investigation.
>>>>>> You can also try code from ignite-3477-master branch. This branch
>>>>>> contains many memory related fixes but it isn't stable yet.
>>>>>> On Mon, Apr 10, 2017 at 11:37 PM, kimec.ethome.sk <ki...@ethome.sk>
>>>>>> wrote:
>>>>>>> Hi Andrey,
>>>>>>> sorry, I've got ahead of my self.
>>>>>>> I am on FreeBSD 11.0-RELEASE-p1 amd64
>>>>>>> With OpenJDK Runtime Environment 1.8.0_121-b13 Oracle Corporation
>>>>>>> OpenJDK
>>>>>>> 64-Bit Server VM 25.121-b13
>>>>>>> hw.model: Intel(R) Core(TM) i7-4702MQ CPU @ 2.20GHz
>>>>>>> hw.machine_arch: amd64
>>>>>>> hw.ncpu: 8
>>>>>>> hw.physmem: 8251813888
>>>>>>> Core dump is 1 GB, so I guess that is no go. I am attaching crash log to
>>>>>>> this e-mail.
>>>>>>> I have uploaded the project I've used during my testing here
>>>>>>> https://github.com/kimec/ignite-spring-boot .
>>>>>>> The sample works perfectly well with stock ignite-core on Linux OpenJDK
>>>>>>> 8
>>>>>>> xs64 CentOS 7 .
>>>>>>> Kamil
>>>>>>> On 2017-04-10 12:24, Andrey Gura wrote:
>>>>>>>> Hi,
>>>>>>>> could you please share core dump file? If not, it would be helpful to
>>>>>>>> know what is CPU architecture on this server.
>>>>>>>> On Mon, Apr 10, 2017 at 2:53 AM, Kamil Misuth <ki...@ethome.sk> wrote:
>>>>>>>>> Greetings,
>>>>>>>>> OpenJDK (7 and 8) HotSpot JVM SIGSEGVs on FreeBSD 11 as soon as node
>>>>>>>>> joins a
>>>>>>>>> topology and starts to communicate via DirectNioClientWorker.
>>>>>>>>> The root cause is DirectByteBufferStreamImpl (both versions) which
>>>>>>>>> uses
>>>>>>>>> GridUnsafe.getXXX/putXXX(Object object, offset, value) methods to
>>>>>>>>> manipulate
>>>>>>>>> DirectByteBuffer, whereas it should really be using
>>>>>>>>> GridUnsafe.getXXX/putXXX(address, value), since DirectByteBuffer is
>>>>>>>>> allocated on C heap (off java heap).
>>>>>>>>> Notice that at least one instance of the same problem is known to
>>>>>>>>> exist
>>>>>>>>> in
>>>>>>>>> another project using Unsafe
>>>>>>>>> https://issues.apache.org/jira/browse/CASSANDRA-8325 .
>>>>>>>>> The OpenJDK source of Unsafe is more or less clear on this
>>>>>>>>> http://hg.openjdk.java.net/jdk8u/jdk8u60/jdk/file/935758609767/src/share/classes/sun/misc/Unsafe.java#l391
>>>>>>>>> I have prepared a simple fix here
>>>>>>>>> https://github.com/apache/ignite/compare/1.9.0-rc2...kimec:freebsd-support .
>>>>>>>>> However, I am not sure if the solution is right in regard to overall
>>>>>>>>> ignite
>>>>>>>>> performance.
>>>>>>>>> I've tried to compile ignite-core with tests and after applying my
>>>>>>>>> changes
>>>>>>>>> was able to pass all the basic stuff until the performance test stage
>>>>>>>>> at
>>>>>>>>> which point my machine run out of RAM and swap space (some 10 GB)...
>>>>>>>>> Not
>>>>>>>>> sure if this is how the tests are supposed to be. After compiling with
>>>>>>>>> -DskipTests I was able to create FreeBSD 11 - CentOS 7 two node
>>>>>>>>> cluster
>>>>>>>>> and
>>>>>>>>> everything seemed OK (the two nodes shared an IGFS instance backed by
>>>>>>>>> replicated caches).
>>>>>>>>> Please note that OpenJDK on different systems as well as Oracle JDK
>>>>>>>>> (via
>>>>>>>>> Linux compatility layer) on FreeBSD seem to be more forgiving and does
>>>>>>>>> not
>>>>>>>>> SIGSEGV.
>>>>>>>>> I've based my branch on 1.9.0-rc2 since tag 1.9.0 has already POM with
>>>>>>>>> version 2.0.
>>>>>>>>> Kamil
Re: Ignite on FreeBSD 11 and OpenJDK
Posted by Kamil Misuth <ki...@ethome.sk>.
Replying, to my own question.
Indeed, it seems to be the case that for Ignite 2.0, everything will be
off heap. Denis Magda just (23 hours ago) updated the relevant wiki
page.
https://cwiki.apache.org/confluence/pages/diffpagesbyversion.action?pageId=67638199&selectedPageVersions=10&selectedPageVersions=8
Sounds cool to me :).
Kamil
On 2017-04-19 00:43, Kamil Misuth wrote:
> Sure, Andrey.
>
> In mean time, a fact-checking question:
>
> I haven't really gone through the new memory policy functionality in
> depth, but it does seem that for Ignite 2.0 your aim is to move data
> off heap implicitly for all caches (and possibly backup the off heap
> region by memory mapped file(s)). Is this what you meant by "many
> memory related fixes"?
>
> Thanks.
>
> Kamil
>
> On 2017-04-13 13:08, Andrey Gura wrote:
>> Kamil,
>>
>> thanks a lot for your help. As I told before I need some time for
>> problem analyse. After it I'll create JIRA ticket and share link to it
>> in this thread.
>>
>> Thank you again!
>>
>> On Wed, Apr 12, 2017 at 11:56 PM, Kamil Misuth <ki...@ethome.sk>
>> wrote:
>>> Hi Andrey,
>>>
>>> I've built ignite-3477-master (commit hash 5839f481b7) today.
>>> Apart from the fact that some other Configuration APIs changed,
>>> CacheMemoryMode disapeared. I guess this is related to
>>> http://apache-ignite-developers.2346864.n4.nabble.com/IGNITE-4758-introducing-cache-memory-policies-td14957.html
>>>
>>> Anyway, I've got the following exception on the first try, since
>>> ignite-indexing 2.0.0 is compiled against H2 1.4.191 and Spring Boot
>>> 1.5.2.RELEASE managed H2 version is 1.4.193 and apparently
>>> org.h2.result.Row
>>> changed from abstract class to an interface between the two versions.
>>>
>>> Caused by: java.lang.IncompatibleClassChangeError: class
>>> org.apache.ignite.internal.processors.query.h2.opt.GridH2Row has
>>> interface
>>> org.h2.result.Row as super class
>>>
>>> After fixing the dependency issue, I've tried to create two node
>>> cluster and
>>> JVM SIGSEGVed in DirectNioClientWorker just as before.
>>>
>>> Attaching crash logs from both nodes (running on the same machine
>>> FreeBSD
>>> 11) to this e-mail.
>>>
>>> Kamil
>>>
>>>
>>> On 2017-04-12 00:49, Kamil Misuth wrote:
>>>>
>>>> Sure thing.
>>>>
>>>> I will check out ignite-3477-master as soon as I have some time
>>>> tomorrow.
>>>>
>>>> Kamil
>>>>
>>>>
>>>> On 04/11/2017 05:48 PM, Andrey Gura wrote:
>>>>>
>>>>> Thanks for provided information. I need additional time for problem
>>>>> investigation.
>>>>>
>>>>> You can also try code from ignite-3477-master branch. This branch
>>>>> contains many memory related fixes but it isn't stable yet.
>>>>>
>>>>> On Mon, Apr 10, 2017 at 11:37 PM, kimec.ethome.sk <ki...@ethome.sk>
>>>>> wrote:
>>>>>>
>>>>>> Hi Andrey,
>>>>>>
>>>>>> sorry, I've got ahead of my self.
>>>>>>
>>>>>> I am on FreeBSD 11.0-RELEASE-p1 amd64
>>>>>> With OpenJDK Runtime Environment 1.8.0_121-b13 Oracle Corporation
>>>>>> OpenJDK
>>>>>> 64-Bit Server VM 25.121-b13
>>>>>> hw.model: Intel(R) Core(TM) i7-4702MQ CPU @ 2.20GHz
>>>>>> hw.machine_arch: amd64
>>>>>> hw.ncpu: 8
>>>>>> hw.physmem: 8251813888
>>>>>>
>>>>>> Core dump is 1 GB, so I guess that is no go. I am attaching crash
>>>>>> log to
>>>>>> this e-mail.
>>>>>> I have uploaded the project I've used during my testing here
>>>>>> https://github.com/kimec/ignite-spring-boot .
>>>>>> The sample works perfectly well with stock ignite-core on Linux
>>>>>> OpenJDK
>>>>>> 8
>>>>>> xs64 CentOS 7 .
>>>>>>
>>>>>> Kamil
>>>>>>
>>>>>>
>>>>>>
>>>>>> On 2017-04-10 12:24, Andrey Gura wrote:
>>>>>>>
>>>>>>> Hi,
>>>>>>>
>>>>>>> could you please share core dump file? If not, it would be
>>>>>>> helpful to
>>>>>>> know what is CPU architecture on this server.
>>>>>>>
>>>>>>> On Mon, Apr 10, 2017 at 2:53 AM, Kamil Misuth <ki...@ethome.sk>
>>>>>>> wrote:
>>>>>>>>
>>>>>>>> Greetings,
>>>>>>>>
>>>>>>>> OpenJDK (7 and 8) HotSpot JVM SIGSEGVs on FreeBSD 11 as soon as
>>>>>>>> node
>>>>>>>> joins a
>>>>>>>> topology and starts to communicate via DirectNioClientWorker.
>>>>>>>> The root cause is DirectByteBufferStreamImpl (both versions)
>>>>>>>> which
>>>>>>>> uses
>>>>>>>> GridUnsafe.getXXX/putXXX(Object object, offset, value) methods
>>>>>>>> to
>>>>>>>> manipulate
>>>>>>>> DirectByteBuffer, whereas it should really be using
>>>>>>>> GridUnsafe.getXXX/putXXX(address, value), since DirectByteBuffer
>>>>>>>> is
>>>>>>>> allocated on C heap (off java heap).
>>>>>>>> Notice that at least one instance of the same problem is known
>>>>>>>> to
>>>>>>>> exist
>>>>>>>> in
>>>>>>>> another project using Unsafe
>>>>>>>> https://issues.apache.org/jira/browse/CASSANDRA-8325 .
>>>>>>>> The OpenJDK source of Unsafe is more or less clear on this
>>>>>>>>
>>>>>>>>
>>>>>>>> http://hg.openjdk.java.net/jdk8u/jdk8u60/jdk/file/935758609767/src/share/classes/sun/misc/Unsafe.java#l391
>>>>>>>> I have prepared a simple fix here
>>>>>>>>
>>>>>>>>
>>>>>>>> https://github.com/apache/ignite/compare/1.9.0-rc2...kimec:freebsd-support
>>>>>>>> .
>>>>>>>> However, I am not sure if the solution is right in regard to
>>>>>>>> overall
>>>>>>>> ignite
>>>>>>>> performance.
>>>>>>>> I've tried to compile ignite-core with tests and after applying
>>>>>>>> my
>>>>>>>> changes
>>>>>>>> was able to pass all the basic stuff until the performance test
>>>>>>>> stage
>>>>>>>> at
>>>>>>>> which point my machine run out of RAM and swap space (some 10
>>>>>>>> GB)...
>>>>>>>> Not
>>>>>>>> sure if this is how the tests are supposed to be. After
>>>>>>>> compiling with
>>>>>>>> -DskipTests I was able to create FreeBSD 11 - CentOS 7 two node
>>>>>>>> cluster
>>>>>>>> and
>>>>>>>> everything seemed OK (the two nodes shared an IGFS instance
>>>>>>>> backed by
>>>>>>>> replicated caches).
>>>>>>>> Please note that OpenJDK on different systems as well as Oracle
>>>>>>>> JDK
>>>>>>>> (via
>>>>>>>> Linux compatility layer) on FreeBSD seem to be more forgiving
>>>>>>>> and does
>>>>>>>> not
>>>>>>>> SIGSEGV.
>>>>>>>> I've based my branch on 1.9.0-rc2 since tag 1.9.0 has already
>>>>>>>> POM with
>>>>>>>> version 2.0.
>>>>>>>>
>>>>>>>> Kamil
Re: Ignite on FreeBSD 11 and OpenJDK
Posted by Kamil Misuth <ki...@ethome.sk>.
Sure, Andrey.
In mean time, a fact-checking question:
I haven't really gone through the new memory policy functionality in
depth, but it does seem that for Ignite 2.0 your aim is to move data off
heap implicitly for all caches (and possibly backup the off heap region
by memory mapped file(s)). Is this what you meant by "many memory
related fixes"?
Thanks.
Kamil
On 2017-04-13 13:08, Andrey Gura wrote:
> Kamil,
>
> thanks a lot for your help. As I told before I need some time for
> problem analyse. After it I'll create JIRA ticket and share link to it
> in this thread.
>
> Thank you again!
>
> On Wed, Apr 12, 2017 at 11:56 PM, Kamil Misuth <ki...@ethome.sk> wrote:
>> Hi Andrey,
>>
>> I've built ignite-3477-master (commit hash 5839f481b7) today.
>> Apart from the fact that some other Configuration APIs changed,
>> CacheMemoryMode disapeared. I guess this is related to
>> http://apache-ignite-developers.2346864.n4.nabble.com/IGNITE-4758-introducing-cache-memory-policies-td14957.html
>>
>> Anyway, I've got the following exception on the first try, since
>> ignite-indexing 2.0.0 is compiled against H2 1.4.191 and Spring Boot
>> 1.5.2.RELEASE managed H2 version is 1.4.193 and apparently
>> org.h2.result.Row
>> changed from abstract class to an interface between the two versions.
>>
>> Caused by: java.lang.IncompatibleClassChangeError: class
>> org.apache.ignite.internal.processors.query.h2.opt.GridH2Row has
>> interface
>> org.h2.result.Row as super class
>>
>> After fixing the dependency issue, I've tried to create two node
>> cluster and
>> JVM SIGSEGVed in DirectNioClientWorker just as before.
>>
>> Attaching crash logs from both nodes (running on the same machine
>> FreeBSD
>> 11) to this e-mail.
>>
>> Kamil
>>
>>
>> On 2017-04-12 00:49, Kamil Misuth wrote:
>>>
>>> Sure thing.
>>>
>>> I will check out ignite-3477-master as soon as I have some time
>>> tomorrow.
>>>
>>> Kamil
>>>
>>>
>>> On 04/11/2017 05:48 PM, Andrey Gura wrote:
>>>>
>>>> Thanks for provided information. I need additional time for problem
>>>> investigation.
>>>>
>>>> You can also try code from ignite-3477-master branch. This branch
>>>> contains many memory related fixes but it isn't stable yet.
>>>>
>>>> On Mon, Apr 10, 2017 at 11:37 PM, kimec.ethome.sk <ki...@ethome.sk>
>>>> wrote:
>>>>>
>>>>> Hi Andrey,
>>>>>
>>>>> sorry, I've got ahead of my self.
>>>>>
>>>>> I am on FreeBSD 11.0-RELEASE-p1 amd64
>>>>> With OpenJDK Runtime Environment 1.8.0_121-b13 Oracle Corporation
>>>>> OpenJDK
>>>>> 64-Bit Server VM 25.121-b13
>>>>> hw.model: Intel(R) Core(TM) i7-4702MQ CPU @ 2.20GHz
>>>>> hw.machine_arch: amd64
>>>>> hw.ncpu: 8
>>>>> hw.physmem: 8251813888
>>>>>
>>>>> Core dump is 1 GB, so I guess that is no go. I am attaching crash
>>>>> log to
>>>>> this e-mail.
>>>>> I have uploaded the project I've used during my testing here
>>>>> https://github.com/kimec/ignite-spring-boot .
>>>>> The sample works perfectly well with stock ignite-core on Linux
>>>>> OpenJDK
>>>>> 8
>>>>> xs64 CentOS 7 .
>>>>>
>>>>> Kamil
>>>>>
>>>>>
>>>>>
>>>>> On 2017-04-10 12:24, Andrey Gura wrote:
>>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> could you please share core dump file? If not, it would be helpful
>>>>>> to
>>>>>> know what is CPU architecture on this server.
>>>>>>
>>>>>> On Mon, Apr 10, 2017 at 2:53 AM, Kamil Misuth <ki...@ethome.sk>
>>>>>> wrote:
>>>>>>>
>>>>>>> Greetings,
>>>>>>>
>>>>>>> OpenJDK (7 and 8) HotSpot JVM SIGSEGVs on FreeBSD 11 as soon as
>>>>>>> node
>>>>>>> joins a
>>>>>>> topology and starts to communicate via DirectNioClientWorker.
>>>>>>> The root cause is DirectByteBufferStreamImpl (both versions)
>>>>>>> which
>>>>>>> uses
>>>>>>> GridUnsafe.getXXX/putXXX(Object object, offset, value) methods to
>>>>>>> manipulate
>>>>>>> DirectByteBuffer, whereas it should really be using
>>>>>>> GridUnsafe.getXXX/putXXX(address, value), since DirectByteBuffer
>>>>>>> is
>>>>>>> allocated on C heap (off java heap).
>>>>>>> Notice that at least one instance of the same problem is known to
>>>>>>> exist
>>>>>>> in
>>>>>>> another project using Unsafe
>>>>>>> https://issues.apache.org/jira/browse/CASSANDRA-8325 .
>>>>>>> The OpenJDK source of Unsafe is more or less clear on this
>>>>>>>
>>>>>>>
>>>>>>> http://hg.openjdk.java.net/jdk8u/jdk8u60/jdk/file/935758609767/src/share/classes/sun/misc/Unsafe.java#l391
>>>>>>> I have prepared a simple fix here
>>>>>>>
>>>>>>>
>>>>>>> https://github.com/apache/ignite/compare/1.9.0-rc2...kimec:freebsd-support
>>>>>>> .
>>>>>>> However, I am not sure if the solution is right in regard to
>>>>>>> overall
>>>>>>> ignite
>>>>>>> performance.
>>>>>>> I've tried to compile ignite-core with tests and after applying
>>>>>>> my
>>>>>>> changes
>>>>>>> was able to pass all the basic stuff until the performance test
>>>>>>> stage
>>>>>>> at
>>>>>>> which point my machine run out of RAM and swap space (some 10
>>>>>>> GB)...
>>>>>>> Not
>>>>>>> sure if this is how the tests are supposed to be. After compiling
>>>>>>> with
>>>>>>> -DskipTests I was able to create FreeBSD 11 - CentOS 7 two node
>>>>>>> cluster
>>>>>>> and
>>>>>>> everything seemed OK (the two nodes shared an IGFS instance
>>>>>>> backed by
>>>>>>> replicated caches).
>>>>>>> Please note that OpenJDK on different systems as well as Oracle
>>>>>>> JDK
>>>>>>> (via
>>>>>>> Linux compatility layer) on FreeBSD seem to be more forgiving and
>>>>>>> does
>>>>>>> not
>>>>>>> SIGSEGV.
>>>>>>> I've based my branch on 1.9.0-rc2 since tag 1.9.0 has already POM
>>>>>>> with
>>>>>>> version 2.0.
>>>>>>>
>>>>>>> Kamil
Re: Ignite on FreeBSD 11 and OpenJDK
Posted by Andrey Gura <ag...@apache.org>.
Kamil,
thanks a lot for your help. As I told before I need some time for
problem analyse. After it I'll create JIRA ticket and share link to it
in this thread.
Thank you again!
On Wed, Apr 12, 2017 at 11:56 PM, Kamil Misuth <ki...@ethome.sk> wrote:
> Hi Andrey,
>
> I've built ignite-3477-master (commit hash 5839f481b7) today.
> Apart from the fact that some other Configuration APIs changed,
> CacheMemoryMode disapeared. I guess this is related to
> http://apache-ignite-developers.2346864.n4.nabble.com/IGNITE-4758-introducing-cache-memory-policies-td14957.html
>
> Anyway, I've got the following exception on the first try, since
> ignite-indexing 2.0.0 is compiled against H2 1.4.191 and Spring Boot
> 1.5.2.RELEASE managed H2 version is 1.4.193 and apparently org.h2.result.Row
> changed from abstract class to an interface between the two versions.
>
> Caused by: java.lang.IncompatibleClassChangeError: class
> org.apache.ignite.internal.processors.query.h2.opt.GridH2Row has interface
> org.h2.result.Row as super class
>
> After fixing the dependency issue, I've tried to create two node cluster and
> JVM SIGSEGVed in DirectNioClientWorker just as before.
>
> Attaching crash logs from both nodes (running on the same machine FreeBSD
> 11) to this e-mail.
>
> Kamil
>
>
> On 2017-04-12 00:49, Kamil Misuth wrote:
>>
>> Sure thing.
>>
>> I will check out ignite-3477-master as soon as I have some time tomorrow.
>>
>> Kamil
>>
>>
>> On 04/11/2017 05:48 PM, Andrey Gura wrote:
>>>
>>> Thanks for provided information. I need additional time for problem
>>> investigation.
>>>
>>> You can also try code from ignite-3477-master branch. This branch
>>> contains many memory related fixes but it isn't stable yet.
>>>
>>> On Mon, Apr 10, 2017 at 11:37 PM, kimec.ethome.sk <ki...@ethome.sk>
>>> wrote:
>>>>
>>>> Hi Andrey,
>>>>
>>>> sorry, I've got ahead of my self.
>>>>
>>>> I am on FreeBSD 11.0-RELEASE-p1 amd64
>>>> With OpenJDK Runtime Environment 1.8.0_121-b13 Oracle Corporation
>>>> OpenJDK
>>>> 64-Bit Server VM 25.121-b13
>>>> hw.model: Intel(R) Core(TM) i7-4702MQ CPU @ 2.20GHz
>>>> hw.machine_arch: amd64
>>>> hw.ncpu: 8
>>>> hw.physmem: 8251813888
>>>>
>>>> Core dump is 1 GB, so I guess that is no go. I am attaching crash log to
>>>> this e-mail.
>>>> I have uploaded the project I've used during my testing here
>>>> https://github.com/kimec/ignite-spring-boot .
>>>> The sample works perfectly well with stock ignite-core on Linux OpenJDK
>>>> 8
>>>> xs64 CentOS 7 .
>>>>
>>>> Kamil
>>>>
>>>>
>>>>
>>>> On 2017-04-10 12:24, Andrey Gura wrote:
>>>>>
>>>>> Hi,
>>>>>
>>>>> could you please share core dump file? If not, it would be helpful to
>>>>> know what is CPU architecture on this server.
>>>>>
>>>>> On Mon, Apr 10, 2017 at 2:53 AM, Kamil Misuth <ki...@ethome.sk> wrote:
>>>>>>
>>>>>> Greetings,
>>>>>>
>>>>>> OpenJDK (7 and 8) HotSpot JVM SIGSEGVs on FreeBSD 11 as soon as node
>>>>>> joins a
>>>>>> topology and starts to communicate via DirectNioClientWorker.
>>>>>> The root cause is DirectByteBufferStreamImpl (both versions) which
>>>>>> uses
>>>>>> GridUnsafe.getXXX/putXXX(Object object, offset, value) methods to
>>>>>> manipulate
>>>>>> DirectByteBuffer, whereas it should really be using
>>>>>> GridUnsafe.getXXX/putXXX(address, value), since DirectByteBuffer is
>>>>>> allocated on C heap (off java heap).
>>>>>> Notice that at least one instance of the same problem is known to
>>>>>> exist
>>>>>> in
>>>>>> another project using Unsafe
>>>>>> https://issues.apache.org/jira/browse/CASSANDRA-8325 .
>>>>>> The OpenJDK source of Unsafe is more or less clear on this
>>>>>>
>>>>>>
>>>>>> http://hg.openjdk.java.net/jdk8u/jdk8u60/jdk/file/935758609767/src/share/classes/sun/misc/Unsafe.java#l391
>>>>>> I have prepared a simple fix here
>>>>>>
>>>>>>
>>>>>> https://github.com/apache/ignite/compare/1.9.0-rc2...kimec:freebsd-support .
>>>>>> However, I am not sure if the solution is right in regard to overall
>>>>>> ignite
>>>>>> performance.
>>>>>> I've tried to compile ignite-core with tests and after applying my
>>>>>> changes
>>>>>> was able to pass all the basic stuff until the performance test stage
>>>>>> at
>>>>>> which point my machine run out of RAM and swap space (some 10 GB)...
>>>>>> Not
>>>>>> sure if this is how the tests are supposed to be. After compiling with
>>>>>> -DskipTests I was able to create FreeBSD 11 - CentOS 7 two node
>>>>>> cluster
>>>>>> and
>>>>>> everything seemed OK (the two nodes shared an IGFS instance backed by
>>>>>> replicated caches).
>>>>>> Please note that OpenJDK on different systems as well as Oracle JDK
>>>>>> (via
>>>>>> Linux compatility layer) on FreeBSD seem to be more forgiving and does
>>>>>> not
>>>>>> SIGSEGV.
>>>>>> I've based my branch on 1.9.0-rc2 since tag 1.9.0 has already POM with
>>>>>> version 2.0.
>>>>>>
>>>>>> Kamil
Re: Ignite on FreeBSD 11 and OpenJDK
Posted by Kamil Misuth <ki...@ethome.sk>.
Hi Andrey,
I've built ignite-3477-master (commit hash 5839f481b7) today.
Apart from the fact that some other Configuration APIs changed,
CacheMemoryMode disapeared. I guess this is related to
http://apache-ignite-developers.2346864.n4.nabble.com/IGNITE-4758-introducing-cache-memory-policies-td14957.html
Anyway, I've got the following exception on the first try, since
ignite-indexing 2.0.0 is compiled against H2 1.4.191 and Spring Boot
1.5.2.RELEASE managed H2 version is 1.4.193 and apparently
org.h2.result.Row changed from abstract class to an interface between
the two versions.
Caused by: java.lang.IncompatibleClassChangeError: class
org.apache.ignite.internal.processors.query.h2.opt.GridH2Row has
interface org.h2.result.Row as super class
After fixing the dependency issue, I've tried to create two node cluster
and JVM SIGSEGVed in DirectNioClientWorker just as before.
Attaching crash logs from both nodes (running on the same machine
FreeBSD 11) to this e-mail.
Kamil
On 2017-04-12 00:49, Kamil Misuth wrote:
> Sure thing.
>
> I will check out ignite-3477-master as soon as I have some time
> tomorrow.
>
> Kamil
>
>
> On 04/11/2017 05:48 PM, Andrey Gura wrote:
>> Thanks for provided information. I need additional time for problem
>> investigation.
>>
>> You can also try code from ignite-3477-master branch. This branch
>> contains many memory related fixes but it isn't stable yet.
>>
>> On Mon, Apr 10, 2017 at 11:37 PM, kimec.ethome.sk <ki...@ethome.sk>
>> wrote:
>>> Hi Andrey,
>>>
>>> sorry, I've got ahead of my self.
>>>
>>> I am on FreeBSD 11.0-RELEASE-p1 amd64
>>> With OpenJDK Runtime Environment 1.8.0_121-b13 Oracle Corporation
>>> OpenJDK
>>> 64-Bit Server VM 25.121-b13
>>> hw.model: Intel(R) Core(TM) i7-4702MQ CPU @ 2.20GHz
>>> hw.machine_arch: amd64
>>> hw.ncpu: 8
>>> hw.physmem: 8251813888
>>>
>>> Core dump is 1 GB, so I guess that is no go. I am attaching crash log
>>> to
>>> this e-mail.
>>> I have uploaded the project I've used during my testing here
>>> https://github.com/kimec/ignite-spring-boot .
>>> The sample works perfectly well with stock ignite-core on Linux
>>> OpenJDK 8
>>> xs64 CentOS 7 .
>>>
>>> Kamil
>>>
>>>
>>>
>>> On 2017-04-10 12:24, Andrey Gura wrote:
>>>> Hi,
>>>>
>>>> could you please share core dump file? If not, it would be helpful
>>>> to
>>>> know what is CPU architecture on this server.
>>>>
>>>> On Mon, Apr 10, 2017 at 2:53 AM, Kamil Misuth <ki...@ethome.sk>
>>>> wrote:
>>>>> Greetings,
>>>>>
>>>>> OpenJDK (7 and 8) HotSpot JVM SIGSEGVs on FreeBSD 11 as soon as
>>>>> node
>>>>> joins a
>>>>> topology and starts to communicate via DirectNioClientWorker.
>>>>> The root cause is DirectByteBufferStreamImpl (both versions) which
>>>>> uses
>>>>> GridUnsafe.getXXX/putXXX(Object object, offset, value) methods to
>>>>> manipulate
>>>>> DirectByteBuffer, whereas it should really be using
>>>>> GridUnsafe.getXXX/putXXX(address, value), since DirectByteBuffer is
>>>>> allocated on C heap (off java heap).
>>>>> Notice that at least one instance of the same problem is known to
>>>>> exist
>>>>> in
>>>>> another project using Unsafe
>>>>> https://issues.apache.org/jira/browse/CASSANDRA-8325 .
>>>>> The OpenJDK source of Unsafe is more or less clear on this
>>>>>
>>>>> http://hg.openjdk.java.net/jdk8u/jdk8u60/jdk/file/935758609767/src/share/classes/sun/misc/Unsafe.java#l391
>>>>> I have prepared a simple fix here
>>>>>
>>>>> https://github.com/apache/ignite/compare/1.9.0-rc2...kimec:freebsd-support
>>>>> .
>>>>> However, I am not sure if the solution is right in regard to
>>>>> overall
>>>>> ignite
>>>>> performance.
>>>>> I've tried to compile ignite-core with tests and after applying my
>>>>> changes
>>>>> was able to pass all the basic stuff until the performance test
>>>>> stage at
>>>>> which point my machine run out of RAM and swap space (some 10
>>>>> GB)... Not
>>>>> sure if this is how the tests are supposed to be. After compiling
>>>>> with
>>>>> -DskipTests I was able to create FreeBSD 11 - CentOS 7 two node
>>>>> cluster
>>>>> and
>>>>> everything seemed OK (the two nodes shared an IGFS instance backed
>>>>> by
>>>>> replicated caches).
>>>>> Please note that OpenJDK on different systems as well as Oracle JDK
>>>>> (via
>>>>> Linux compatility layer) on FreeBSD seem to be more forgiving and
>>>>> does
>>>>> not
>>>>> SIGSEGV.
>>>>> I've based my branch on 1.9.0-rc2 since tag 1.9.0 has already POM
>>>>> with
>>>>> version 2.0.
>>>>>
>>>>> Kamil
>>>>>
Re: Ignite on FreeBSD 11 and OpenJDK
Posted by Kamil Misuth <ki...@ethome.sk>.
Sure thing.
I will check out ignite-3477-master as soon as I have some time tomorrow.
Kamil
On 04/11/2017 05:48 PM, Andrey Gura wrote:
> Thanks for provided information. I need additional time for problem
> investigation.
>
> You can also try code from ignite-3477-master branch. This branch
> contains many memory related fixes but it isn't stable yet.
>
> On Mon, Apr 10, 2017 at 11:37 PM, kimec.ethome.sk <ki...@ethome.sk> wrote:
>> Hi Andrey,
>>
>> sorry, I've got ahead of my self.
>>
>> I am on FreeBSD 11.0-RELEASE-p1 amd64
>> With OpenJDK Runtime Environment 1.8.0_121-b13 Oracle Corporation OpenJDK
>> 64-Bit Server VM 25.121-b13
>> hw.model: Intel(R) Core(TM) i7-4702MQ CPU @ 2.20GHz
>> hw.machine_arch: amd64
>> hw.ncpu: 8
>> hw.physmem: 8251813888
>>
>> Core dump is 1 GB, so I guess that is no go. I am attaching crash log to
>> this e-mail.
>> I have uploaded the project I've used during my testing here
>> https://github.com/kimec/ignite-spring-boot .
>> The sample works perfectly well with stock ignite-core on Linux OpenJDK 8
>> xs64 CentOS 7 .
>>
>> Kamil
>>
>>
>>
>> On 2017-04-10 12:24, Andrey Gura wrote:
>>> Hi,
>>>
>>> could you please share core dump file? If not, it would be helpful to
>>> know what is CPU architecture on this server.
>>>
>>> On Mon, Apr 10, 2017 at 2:53 AM, Kamil Misuth <ki...@ethome.sk> wrote:
>>>> Greetings,
>>>>
>>>> OpenJDK (7 and 8) HotSpot JVM SIGSEGVs on FreeBSD 11 as soon as node
>>>> joins a
>>>> topology and starts to communicate via DirectNioClientWorker.
>>>> The root cause is DirectByteBufferStreamImpl (both versions) which uses
>>>> GridUnsafe.getXXX/putXXX(Object object, offset, value) methods to
>>>> manipulate
>>>> DirectByteBuffer, whereas it should really be using
>>>> GridUnsafe.getXXX/putXXX(address, value), since DirectByteBuffer is
>>>> allocated on C heap (off java heap).
>>>> Notice that at least one instance of the same problem is known to exist
>>>> in
>>>> another project using Unsafe
>>>> https://issues.apache.org/jira/browse/CASSANDRA-8325 .
>>>> The OpenJDK source of Unsafe is more or less clear on this
>>>>
>>>> http://hg.openjdk.java.net/jdk8u/jdk8u60/jdk/file/935758609767/src/share/classes/sun/misc/Unsafe.java#l391
>>>> I have prepared a simple fix here
>>>>
>>>> https://github.com/apache/ignite/compare/1.9.0-rc2...kimec:freebsd-support .
>>>> However, I am not sure if the solution is right in regard to overall
>>>> ignite
>>>> performance.
>>>> I've tried to compile ignite-core with tests and after applying my
>>>> changes
>>>> was able to pass all the basic stuff until the performance test stage at
>>>> which point my machine run out of RAM and swap space (some 10 GB)... Not
>>>> sure if this is how the tests are supposed to be. After compiling with
>>>> -DskipTests I was able to create FreeBSD 11 - CentOS 7 two node cluster
>>>> and
>>>> everything seemed OK (the two nodes shared an IGFS instance backed by
>>>> replicated caches).
>>>> Please note that OpenJDK on different systems as well as Oracle JDK (via
>>>> Linux compatility layer) on FreeBSD seem to be more forgiving and does
>>>> not
>>>> SIGSEGV.
>>>> I've based my branch on 1.9.0-rc2 since tag 1.9.0 has already POM with
>>>> version 2.0.
>>>>
>>>> Kamil
>>>>
Re: Ignite on FreeBSD 11 and OpenJDK
Posted by Andrey Gura <ag...@apache.org>.
Thanks for provided information. I need additional time for problem
investigation.
You can also try code from ignite-3477-master branch. This branch
contains many memory related fixes but it isn't stable yet.
On Mon, Apr 10, 2017 at 11:37 PM, kimec.ethome.sk <ki...@ethome.sk> wrote:
> Hi Andrey,
>
> sorry, I've got ahead of my self.
>
> I am on FreeBSD 11.0-RELEASE-p1 amd64
> With OpenJDK Runtime Environment 1.8.0_121-b13 Oracle Corporation OpenJDK
> 64-Bit Server VM 25.121-b13
> hw.model: Intel(R) Core(TM) i7-4702MQ CPU @ 2.20GHz
> hw.machine_arch: amd64
> hw.ncpu: 8
> hw.physmem: 8251813888
>
> Core dump is 1 GB, so I guess that is no go. I am attaching crash log to
> this e-mail.
> I have uploaded the project I've used during my testing here
> https://github.com/kimec/ignite-spring-boot .
> The sample works perfectly well with stock ignite-core on Linux OpenJDK 8
> xs64 CentOS 7 .
>
> Kamil
>
>
>
> On 2017-04-10 12:24, Andrey Gura wrote:
>>
>> Hi,
>>
>> could you please share core dump file? If not, it would be helpful to
>> know what is CPU architecture on this server.
>>
>> On Mon, Apr 10, 2017 at 2:53 AM, Kamil Misuth <ki...@ethome.sk> wrote:
>>>
>>> Greetings,
>>>
>>> OpenJDK (7 and 8) HotSpot JVM SIGSEGVs on FreeBSD 11 as soon as node
>>> joins a
>>> topology and starts to communicate via DirectNioClientWorker.
>>> The root cause is DirectByteBufferStreamImpl (both versions) which uses
>>> GridUnsafe.getXXX/putXXX(Object object, offset, value) methods to
>>> manipulate
>>> DirectByteBuffer, whereas it should really be using
>>> GridUnsafe.getXXX/putXXX(address, value), since DirectByteBuffer is
>>> allocated on C heap (off java heap).
>>> Notice that at least one instance of the same problem is known to exist
>>> in
>>> another project using Unsafe
>>> https://issues.apache.org/jira/browse/CASSANDRA-8325 .
>>> The OpenJDK source of Unsafe is more or less clear on this
>>>
>>> http://hg.openjdk.java.net/jdk8u/jdk8u60/jdk/file/935758609767/src/share/classes/sun/misc/Unsafe.java#l391
>>> I have prepared a simple fix here
>>>
>>> https://github.com/apache/ignite/compare/1.9.0-rc2...kimec:freebsd-support .
>>> However, I am not sure if the solution is right in regard to overall
>>> ignite
>>> performance.
>>> I've tried to compile ignite-core with tests and after applying my
>>> changes
>>> was able to pass all the basic stuff until the performance test stage at
>>> which point my machine run out of RAM and swap space (some 10 GB)... Not
>>> sure if this is how the tests are supposed to be. After compiling with
>>> -DskipTests I was able to create FreeBSD 11 - CentOS 7 two node cluster
>>> and
>>> everything seemed OK (the two nodes shared an IGFS instance backed by
>>> replicated caches).
>>> Please note that OpenJDK on different systems as well as Oracle JDK (via
>>> Linux compatility layer) on FreeBSD seem to be more forgiving and does
>>> not
>>> SIGSEGV.
>>> I've based my branch on 1.9.0-rc2 since tag 1.9.0 has already POM with
>>> version 2.0.
>>>
>>> Kamil
>>>
>
Re: Ignite on FreeBSD 11 and OpenJDK
Posted by "kimec.ethome.sk" <ki...@ethome.sk>.
Hi Andrey,
sorry, I've got ahead of my self.
I am on FreeBSD 11.0-RELEASE-p1 amd64
With OpenJDK Runtime Environment 1.8.0_121-b13 Oracle Corporation
OpenJDK 64-Bit Server VM 25.121-b13
hw.model: Intel(R) Core(TM) i7-4702MQ CPU @ 2.20GHz
hw.machine_arch: amd64
hw.ncpu: 8
hw.physmem: 8251813888
Core dump is 1 GB, so I guess that is no go. I am attaching crash log to
this e-mail.
I have uploaded the project I've used during my testing here
https://github.com/kimec/ignite-spring-boot .
The sample works perfectly well with stock ignite-core on Linux OpenJDK
8 xs64 CentOS 7 .
Kamil
On 2017-04-10 12:24, Andrey Gura wrote:
> Hi,
>
> could you please share core dump file? If not, it would be helpful to
> know what is CPU architecture on this server.
>
> On Mon, Apr 10, 2017 at 2:53 AM, Kamil Misuth <ki...@ethome.sk> wrote:
>> Greetings,
>>
>> OpenJDK (7 and 8) HotSpot JVM SIGSEGVs on FreeBSD 11 as soon as node
>> joins a
>> topology and starts to communicate via DirectNioClientWorker.
>> The root cause is DirectByteBufferStreamImpl (both versions) which
>> uses
>> GridUnsafe.getXXX/putXXX(Object object, offset, value) methods to
>> manipulate
>> DirectByteBuffer, whereas it should really be using
>> GridUnsafe.getXXX/putXXX(address, value), since DirectByteBuffer is
>> allocated on C heap (off java heap).
>> Notice that at least one instance of the same problem is known to
>> exist in
>> another project using Unsafe
>> https://issues.apache.org/jira/browse/CASSANDRA-8325 .
>> The OpenJDK source of Unsafe is more or less clear on this
>> http://hg.openjdk.java.net/jdk8u/jdk8u60/jdk/file/935758609767/src/share/classes/sun/misc/Unsafe.java#l391
>> I have prepared a simple fix here
>> https://github.com/apache/ignite/compare/1.9.0-rc2...kimec:freebsd-support
>> .
>> However, I am not sure if the solution is right in regard to overall
>> ignite
>> performance.
>> I've tried to compile ignite-core with tests and after applying my
>> changes
>> was able to pass all the basic stuff until the performance test stage
>> at
>> which point my machine run out of RAM and swap space (some 10 GB)...
>> Not
>> sure if this is how the tests are supposed to be. After compiling with
>> -DskipTests I was able to create FreeBSD 11 - CentOS 7 two node
>> cluster and
>> everything seemed OK (the two nodes shared an IGFS instance backed by
>> replicated caches).
>> Please note that OpenJDK on different systems as well as Oracle JDK
>> (via
>> Linux compatility layer) on FreeBSD seem to be more forgiving and does
>> not
>> SIGSEGV.
>> I've based my branch on 1.9.0-rc2 since tag 1.9.0 has already POM with
>> version 2.0.
>>
>> Kamil
>>
>>
Re: Ignite on FreeBSD 11 and OpenJDK
Posted by Andrey Gura <ag...@apache.org>.
Hi,
could you please share core dump file? If not, it would be helpful to
know what is CPU architecture on this server.
On Mon, Apr 10, 2017 at 2:53 AM, Kamil Misuth <ki...@ethome.sk> wrote:
> Greetings,
>
> OpenJDK (7 and 8) HotSpot JVM SIGSEGVs on FreeBSD 11 as soon as node joins a
> topology and starts to communicate via DirectNioClientWorker.
> The root cause is DirectByteBufferStreamImpl (both versions) which uses
> GridUnsafe.getXXX/putXXX(Object object, offset, value) methods to manipulate
> DirectByteBuffer, whereas it should really be using
> GridUnsafe.getXXX/putXXX(address, value), since DirectByteBuffer is
> allocated on C heap (off java heap).
> Notice that at least one instance of the same problem is known to exist in
> another project using Unsafe
> https://issues.apache.org/jira/browse/CASSANDRA-8325 .
> The OpenJDK source of Unsafe is more or less clear on this
> http://hg.openjdk.java.net/jdk8u/jdk8u60/jdk/file/935758609767/src/share/classes/sun/misc/Unsafe.java#l391
> I have prepared a simple fix here
> https://github.com/apache/ignite/compare/1.9.0-rc2...kimec:freebsd-support .
> However, I am not sure if the solution is right in regard to overall ignite
> performance.
> I've tried to compile ignite-core with tests and after applying my changes
> was able to pass all the basic stuff until the performance test stage at
> which point my machine run out of RAM and swap space (some 10 GB)... Not
> sure if this is how the tests are supposed to be. After compiling with
> -DskipTests I was able to create FreeBSD 11 - CentOS 7 two node cluster and
> everything seemed OK (the two nodes shared an IGFS instance backed by
> replicated caches).
> Please note that OpenJDK on different systems as well as Oracle JDK (via
> Linux compatility layer) on FreeBSD seem to be more forgiving and does not
> SIGSEGV.
> I've based my branch on 1.9.0-rc2 since tag 1.9.0 has already POM with
> version 2.0.
>
> Kamil
>
>