IEP-57: API to implement different storage engines, modularization

[Maksim Stepachev <ma...@gmail.com>]

Igniters,

I want to be able to add another storage for Ignite like column-oriented
storage. This requires some changes and defining the abstraction layer.
I don't have a definite plan yet for how this could be done?
Who wants to take part in this activity?
I need the approval of committers for it.

https://cwiki.apache.org/confluence/display/IGNITE/IEP-57%3A+API+to+implement+different+storage+engines%2C+modularization

Re: IEP-57: API to implement different storage engines, modularization

[Pavel Kovalenko <jo...@gmail.com>]

Hi Maksim,

I think your proposal is very good and useful not only for introducing new
types of storage engine but also it's a good trigger to rework the existing
storage engine to decouple its components of each other.

I have a key question that we should answer. Where we should place our
abstraction point?
If we talk about another kind of B-tree we can relatively easily re-use our
page memory and persisting mechanics, so the problem, in this case, is to
implement the right data layout and operations over page memory.
Other things (persisting, recovery) will be applied almost automatically.

If we talk about something completely different like LSM we need to solve
different problems to achieve acceptable performance
a) Split entries by columns and store them continuously to each other as
independent chunks. We need also to handle them in batch mode to achieve
vectorization power.
b) LSM is an append-only engine while our page memory is implemented for
random-access. We need to support different data layout in memory and on
disk. This also requires a different persisting mechanism rather than we
have.
c) Even if we do all these things we will face with new problems related to
merging and compacting our data chunks [1]

This will require a lot of time to bring it to production-ready form with
acceptable performance, while resources are limited and there are a lot of
OLAP databases that solve these problems more efficiently.

I think we should focus on OLTP cases and row-oriented data layout and do
things that match with our page memory and persisting mechanics.

[1]
https://www.scylladb.com/2018/01/31/compaction-series-leveled-compaction/




пт, 18 сент. 2020 г. в 13:56, Ilya Kasnacheev <il...@gmail.com>:

> Hello!
>
> My opinion is that we should have a second storage option first, add an
> abstraction layer later.
>
> It's not possible to make a good abstraction layer out of 1 implementation
> anymore than it's possible to definitely draw a straight line through one
> point.
>
> WRT columnar storage itself, it is an interesting idea.
>
> Regards,
> --
> Ilya Kasnacheev
>
>
> пт, 18 сент. 2020 г. в 13:48, Maksim Stepachev <maksim.stepachev@gmail.com
> >:
>
> > Igniters,
> >
> > I want to be able to add another storage for Ignite like column-oriented
> > storage. This requires some changes and defining the abstraction layer.
> > I don't have a definite plan yet for how this could be done?
> > Who wants to take part in this activity?
> > I need the approval of committers for it.
> >
> >
> >
> https://cwiki.apache.org/confluence/display/IGNITE/IEP-57%3A+API+to+implement+different+storage+engines%2C+modularization
> >
>

Re: IEP-57: API to implement different storage engines, modularization

[Ilya Kasnacheev <il...@gmail.com>]

Hello!

My opinion is that we should have a second storage option first, add an
abstraction layer later.

It's not possible to make a good abstraction layer out of 1 implementation
anymore than it's possible to definitely draw a straight line through one
point.

WRT columnar storage itself, it is an interesting idea.

Regards,
-- 
Ilya Kasnacheev


пт, 18 сент. 2020 г. в 13:48, Maksim Stepachev <ma...@gmail.com>:

> Igniters,
>
> I want to be able to add another storage for Ignite like column-oriented
> storage. This requires some changes and defining the abstraction layer.
> I don't have a definite plan yet for how this could be done?
> Who wants to take part in this activity?
> I need the approval of committers for it.
>
>
> https://cwiki.apache.org/confluence/display/IGNITE/IEP-57%3A+API+to+implement+different+storage+engines%2C+modularization
>