Re: PageMemory approach for Ignite 2.0

[Denis Magda <dm...@apache.org>]

Here we need to define what’s meant under *fast enough*. Java is unmanageable in terms of memory and it’s unlikely that any custom memory manageable solution like the PageMemory will outperform it ever. Simply because the Java heap will still be an intermediate layer between an application and the PageMemory passing objects back and forth. Also the PageMemory manages data, as I understand, with JNI-based Unsafe that also brings performance hit, etc.

So, personally I share Vladimir’s point of view and would not discontinue the on-heap implementation.

What I can’t is why the PageMemory so slower than the current *off*-heap based implementation? The latter has comparable performance with the on-heap impl and it’s not twice times slower for sure. Alex G., could you elaborate on this?

—
Denis

> On Dec 31, 2016, at 8:29 AM, Dmitriy Setrakyan <ds...@apache.org> wrote:
> 
> On Sat, Dec 31, 2016 at 7:07 AM, Vladimir Ozerov <vo...@gridgain.com>
> wrote:
> 
> So my main concern is *what about current heap mode*? It must stay alive.
>> Page-memory approach should be abstracted out and implemented in addition
>> to current heap-approach, not instead of it. Have high-end machine and
>> suffer from GC? Pick offheap mode. Have a commodity machine? Old good heap
>> mode is your choice.
>> 
> 
> Vova, disagree. I don't see a reason to maintain on-heap implementation, if
> we can make the off-heap one work fast enough. Remember, this is the 1st
> draft only. Once we optimize it, it will get a lot faster.

Re: PageMemory approach for Ignite 2.0

[Sergi Vladykin <se...@gmail.com>]

Agree with Dmitriy. We should avoid having multiple implementations of the
same thing if possible. Lets put our efforts on fixing issues with
PageMemory.

Sergi

2017-01-03 20:11 GMT+03:00 Dmitriy Setrakyan <ds...@apache.org>:

> Vova,
>
> I would qualify the need for PageMemory as strategic for Apache Ignite.
> With addition of SQL Grid component, Ignite can now also satisfy in-memory
> database use cases, which are very space consuming and require a new memory
> management approach. Basic distributed hash map is not going to work for
> such use cases.
>
> Once PageMemory becomes stable and fast, I don't believe we can afford to
> maintain two types of memory management in the project. It will be just too
> time consuming. On top of that, the old approach will simply not be needed
> anymore.
>
> I am OK with maintaining 2 implementations, assuming we can have a good
> abstraction for the memory management.However, even in that case, we should
> all weigh whether it makes sense to spend time on migrating the current
> on-heap implementation to use these new memory APIs.
>
> D.
>
> On Sun, Jan 1, 2017 at 10:47 PM, Vladimir Ozerov <vo...@gridgain.com>
> wrote:
>
> > Dima,
> >
> > Performance is a serious concern, but not the main one. My point is that
> > standard users working on commodity hardware and requiring only in-memory
> > mode simply do not need page memory. They need distributed HashMap. We
> > already have it. It is fast, it is stable, it have been tested rigorously
> > for years. It does what users need.
> >
> > PageMemory approach targets high-end deployments which is hardly
> represents
> > majority of our users. Less than 10% I think. Or may be <5%, or even <1%.
> > This is who may benefit from page memory. Others will benefit nothing
> > except of additional layer of indirection, drop in performance, risks of
> > instability. And problems with capacity planning, because it is much
> harder
> > to plan two memory regions properly than a single one.
> >
> > I talked to Alexey Goncharuk some time ago, and he told it is not a big
> > deal to abstract out PageMemory. Alex, please confirm. I encourage
> everyone
> > to stop thinking of dropping "old" before you have built "new" and
> > confirmed that it is better.
> >
> > Let's ensure that new approach is well-abstracted, add it to 2.0, let it
> > maturate for 1-2 years, and then think of dropping current approach in
> 3.0.
> > This sounds much better to me.
> >
> >
> > On Sun, Jan 1, 2017 at 10:42 AM, Denis Magda <dm...@apache.org> wrote:
> >
> > > Sorry, just recalled that Unsafe is not JNI based. However, my previous
> > > point of view still remains the same.
> > >
> > > > On Dec 31, 2016, at 11:39 PM, Denis Magda <dm...@apache.org> wrote:
> > > >
> > > > JNI-based Unsafe that also brings performance hit
> > >
> > > —
> > > Denis
> > >
> > >
> >
>

Re: PageMemory approach for Ignite 2.0

[Dmitriy Setrakyan <ds...@apache.org>]

Vova,

I would qualify the need for PageMemory as strategic for Apache Ignite.
With addition of SQL Grid component, Ignite can now also satisfy in-memory
database use cases, which are very space consuming and require a new memory
management approach. Basic distributed hash map is not going to work for
such use cases.

Once PageMemory becomes stable and fast, I don't believe we can afford to
maintain two types of memory management in the project. It will be just too
time consuming. On top of that, the old approach will simply not be needed
anymore.

I am OK with maintaining 2 implementations, assuming we can have a good
abstraction for the memory management.However, even in that case, we should
all weigh whether it makes sense to spend time on migrating the current
on-heap implementation to use these new memory APIs.

D.

On Sun, Jan 1, 2017 at 10:47 PM, Vladimir Ozerov <vo...@gridgain.com>
wrote:

> Dima,
>
> Performance is a serious concern, but not the main one. My point is that
> standard users working on commodity hardware and requiring only in-memory
> mode simply do not need page memory. They need distributed HashMap. We
> already have it. It is fast, it is stable, it have been tested rigorously
> for years. It does what users need.
>
> PageMemory approach targets high-end deployments which is hardly represents
> majority of our users. Less than 10% I think. Or may be <5%, or even <1%.
> This is who may benefit from page memory. Others will benefit nothing
> except of additional layer of indirection, drop in performance, risks of
> instability. And problems with capacity planning, because it is much harder
> to plan two memory regions properly than a single one.
>
> I talked to Alexey Goncharuk some time ago, and he told it is not a big
> deal to abstract out PageMemory. Alex, please confirm. I encourage everyone
> to stop thinking of dropping "old" before you have built "new" and
> confirmed that it is better.
>
> Let's ensure that new approach is well-abstracted, add it to 2.0, let it
> maturate for 1-2 years, and then think of dropping current approach in 3.0.
> This sounds much better to me.
>
>
> On Sun, Jan 1, 2017 at 10:42 AM, Denis Magda <dm...@apache.org> wrote:
>
> > Sorry, just recalled that Unsafe is not JNI based. However, my previous
> > point of view still remains the same.
> >
> > > On Dec 31, 2016, at 11:39 PM, Denis Magda <dm...@apache.org> wrote:
> > >
> > > JNI-based Unsafe that also brings performance hit
> >
> > —
> > Denis
> >
> >
>

Re: PageMemory approach for Ignite 2.0

[Vladimir Ozerov <vo...@gridgain.com>]

Dima,

Performance is a serious concern, but not the main one. My point is that
standard users working on commodity hardware and requiring only in-memory
mode simply do not need page memory. They need distributed HashMap. We
already have it. It is fast, it is stable, it have been tested rigorously
for years. It does what users need.

PageMemory approach targets high-end deployments which is hardly represents
majority of our users. Less than 10% I think. Or may be <5%, or even <1%.
This is who may benefit from page memory. Others will benefit nothing
except of additional layer of indirection, drop in performance, risks of
instability. And problems with capacity planning, because it is much harder
to plan two memory regions properly than a single one.

I talked to Alexey Goncharuk some time ago, and he told it is not a big
deal to abstract out PageMemory. Alex, please confirm. I encourage everyone
to stop thinking of dropping "old" before you have built "new" and
confirmed that it is better.

Let's ensure that new approach is well-abstracted, add it to 2.0, let it
maturate for 1-2 years, and then think of dropping current approach in 3.0.
This sounds much better to me.


On Sun, Jan 1, 2017 at 10:42 AM, Denis Magda <dm...@apache.org> wrote:

> Sorry, just recalled that Unsafe is not JNI based. However, my previous
> point of view still remains the same.
>
> > On Dec 31, 2016, at 11:39 PM, Denis Magda <dm...@apache.org> wrote:
> >
> > JNI-based Unsafe that also brings performance hit
>
> —
> Denis
>
>

Re: PageMemory approach for Ignite 2.0

[Denis Magda <dm...@apache.org>]

Sorry, just recalled that Unsafe is not JNI based. However, my previous point of view still remains the same.

> On Dec 31, 2016, at 11:39 PM, Denis Magda <dm...@apache.org> wrote:
> 
> JNI-based Unsafe that also brings performance hit

—
Denis