You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@ignite.apache.org by Denis Magda <dm...@apache.org> on 2019/10/01 00:20:21 UTC
Re: New SQL execution engine
Ivan, we need more of these discussions, totally agree with you ;)
I've updated the Motivation paragraph outlining some high-level users we
see by working with our users. Hope it helps. Let's carry on and let me
send a note to Apache Calcite community.
-
Denis
On Mon, Sep 30, 2019 at 1:56 AM Ivan Pavlukhin <vo...@gmail.com> wrote:
> Folks,
>
> Thanks everyone for a hot discussion! Not every open source community
> has such open and boiling discussions. It means that people here
> really do care. And I am proud of it!
>
> As I understood, nobody is strictly against the proposed initiative.
> And I am glad that we can move forward (with some steps back along the
> way).
>
> пт, 27 сент. 2019 г. в 19:29, Nikolay Izhikov <ni...@apache.org>:
> >
> > Hello, Denis.
> >
> > Thanks for the clarifications.
> >
> > Sounds good for me.
> > All I try to say in this thread:
> > Guys, please, let's take a step back and write down requirements(what we
> want to get with SQL engine).
> > Which features and use-cases are primary for us.
> >
> > I'm sure you have done it, already during your research.
> >
> > Please, share it with the community.
> >
> > I'm pretty sure we would back to this document again and again during
> migration.
> > So good written design is worth it.
> >
> > В Пт, 27/09/2019 в 09:10 -0700, Denis Magda пишет:
> > > Ignite mates, let me try to move the discussion in a constructive way.
> It
> > > looks like we set a wrong context from the very beginning.
> > >
> > > Before proposing this idea to the community, some of us were
> > > discussing/researching the topic in different groups (the one need to
> think
> > > it through first before even suggesting to consider changes of this
> > > magnitude). The day has come to share this idea with the whole
> community
> > > and outline the next actions. But (!) nobody is 100% sure that that's
> the
> > > right decision. Thus, this will be an *experiment*, some of our
> community
> > > members will be developing a *prototype* and only based on the
> prototype
> > > outcomes we shall make a final decision. Igor, Roman, Ivan, Andrey,
> hope
> > > that nothing has changed and we're on the same page here.
> > >
> > > Many technical and architectural reasons that justify this project have
> > > been shared but let me throw in my perspective. There is nothing wrong
> with
> > > H2, that was the right choice for that time. Thanks to H2 and Ignite
> SQL
> > > APIs, our project is used across hundreds of deployments who are
> > > accelerating relational databases or use Ignite as a system of records.
> > > However, these days many more companies are migrating to *distributed*
> > > databases that speak SQL. For instance, if a couple of years ago 1 out
> of
> > > 10 use cases needed support for multi-joins queries or queries with
> > > subselects or efficient memory usage then today there are 5 out of 10
> use
> > > cases of this kind; in the foreseeable future, it will be a 10 out of
> 10.
> > > So, the evolution is in progress -- the relational world goes
> distributed,
> > > it became exhaustive for both Ignite SQL maintainers and experts who
> help
> > > to tune it for production usage to keep pace with the evolution mostly
> due
> > > to the H2-dependency. Thus, Ignite SQL has to evolve and has to be
> ready to
> > > face the future reality.
> > >
> > > Luckily, we don't need to rush and don't have the right to rush because
> > > hundreds existing users have already trusted their production
> environments
> > > to Ignite SQL and we need to roll out changes with such a big impact
> > > carefully. So, I'm excited that Roman, Igor, Ivan, Andrey stepped in
> and
> > > agreed to be the first contributors who will be *experimenting* with
> the
> > > new SQL engine. Let's support them; let's connect them with Apache
> Calcite
> > > community and see how this story evolves. Folks, please keep the
> community
> > > aware of the progress, let us know when help is needed, some of us
> will be
> > > ready to support with development once you create a solid foundation
> for
> > > the prototype.
> > >
> > > -
> > > Denis
> > >
> > >
> > > On Fri, Sep 27, 2019 at 1:45 AM Igor Seliverstov <
> gvvinblade@apache.org>
> > > wrote:
> > >
> > > > Hi Igniters!
> > > >
> > > > As you might know currently we have many open issues relating to
> current
> > > > H2 based engine and its execution flow.
> > > >
> > > > Some of them are critical (like impossibility to execute particular
> > > > queries), some of them are majors (like impossibility to execute
> particular
> > > > queries without pre-preparation your data to have a collocation) and
> many
> > > > minors.
> > > >
> > > > Most of the issues cannot be solved without whole engine redesign.
> > > >
> > > > So, here the proposal:
> > > >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=130028084
> > > >
> > > > I'll appreciate if you share your thoughts on top of that.
> > > >
> > > > Regards,
> > > > Igor
> > > >
>
>
>
> --
> Best regards,
> Ivan Pavlukhin
>
RE: New SQL execution engine
Posted by "Hostettler, Steve" <St...@wolterskluwer.com>.
Hi Roman,
Thanks a lot for the answer (and the pull request). As I said initially, I was under the impression that the reason was the lack of affinity.
I understand the reason and the current design and I think we all agreed that this is not optimal and that it should be reworked in the new design. Especially the sort of silent behavior. That being said, more than a warning : having joins in // inter partitions would be very helpful but I understand that it is not straightforward.
As always you guys are very reactive and helpful. Keep up the great work. Appreciate it.
-----Original Message-----
From: Roman Kondakov <ko...@mail.ru.INVALID>
Sent: Monday, November 18, 2019 11:04 AM
To: dev@ignite.apache.org
Subject: Re: New SQL execution engine
Hi, Steve
This behavior is actually not a bug, but this is not obvious. I'll try to explain.
When query parallelism = N is turned on, it means that each cache is divided into N parts from the SQL point of view. Every SQL query is executed independently over each particular part, and then results are merged together during the reducer step.
This is absolutely identical to the distributed query execution, where instead of a single node with query parallelism = N, we have N nodes with query parallelism = 1. SQL query is executed over each partition of data on all nodes and then results are merged on reducer.
As we can see, query parallelism is equivalent to the distributed query execution. When we do joins over distributed tables, we need to think about the collocation of data [1]. If data is not collocated, we get a wrong result. This happens silently, which is not good, IMO.
I reworked your example a bit in order to impose collocation on the joining key and now join returns correct result [2].
Current approach in configuration and query execution looks very uncomfortable and should be completely redesigned in the new engine.
[1] https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fapacheignite-sql.readme.io%2Fdocs%2Fdistributed-joins&data=02%7C01%7CSteve.Hostettler%40wolterskluwer.com%7C68a93ad417fc4e70ed1808d76c0e9f53%7C8ac76c91e7f141ffa89c3553b2da2c17%7C0%7C0%7C637096682368420072&sdata=82bDWI1PHUOzNz95A5F%2Flyiqlrb9aQ2vadxhE%2FK47LM%3D&reserved=0
[2] https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fhostettler%2FigniteParallelQueries%2Fpull%2F1&data=02%7C01%7CSteve.Hostettler%40wolterskluwer.com%7C68a93ad417fc4e70ed1808d76c0e9f53%7C8ac76c91e7f141ffa89c3553b2da2c17%7C0%7C0%7C637096682368420072&sdata=QCvNEKqGGyZYOXQbF0sG0DUCzYJCnKoWleFTMtngcsc%3D&reserved=0
--
Kind Regards
Roman Kondakov
On 16.11.2019 12:50, steve.hostettler@gmail.com wrote:
> Actually I am now wondering whether this is not just a bug and that I
> should record it as such. As the behavior is different with and
> without the parallelism and there is no warning during execution or in the api.
>
> Any thought?
>
>
>
> --
> Sent from:
> https://nam04.safelinks.protection.outlook.com/?url=http%3A%2F%2Fapach
> e-ignite-developers.2346864.n4.nabble.com%2F&data=02%7C01%7CSteve.
> Hostettler%40wolterskluwer.com%7C68a93ad417fc4e70ed1808d76c0e9f53%7C8a
> c76c91e7f141ffa89c3553b2da2c17%7C0%7C0%7C637096682368420072&sdata=
> LzUii%2BuNqHhS1YbFLNwpe7cn6XRRpKrrSO6wS5zNlSU%3D&reserved=0
Re: New SQL execution engine
Posted by Roman Kondakov <ko...@mail.ru.INVALID>.
Hi, Steve
This behavior is actually not a bug, but this is not obvious. I'll try
to explain.
When query parallelism = N is turned on, it means that each cache is
divided into N parts from the SQL point of view. Every SQL query is
executed independently over each particular part, and then results are
merged together during the reducer step.
This is absolutely identical to the distributed query execution, where
instead of a single node with query parallelism = N, we have N nodes
with query parallelism = 1. SQL query is executed over each partition of
data on all nodes and then results are merged on reducer.
As we can see, query parallelism is equivalent to the distributed query
execution. When we do joins over distributed tables, we need to think
about the collocation of data [1]. If data is not collocated, we get a
wrong result. This happens silently, which is not good, IMO.
I reworked your example a bit in order to impose collocation on the
joining key and now join returns correct result [2].
Current approach in configuration and query execution looks very
uncomfortable and should be completely redesigned in the new engine.
[1] https://apacheignite-sql.readme.io/docs/distributed-joins
[2] https://github.com/hostettler/igniteParallelQueries/pull/1
--
Kind Regards
Roman Kondakov
On 16.11.2019 12:50, steve.hostettler@gmail.com wrote:
> Actually I am now wondering whether this is not just a bug and that I should
> record it as such. As the behavior is different with and without the
> parallelism and there is no warning during execution or in the api.
>
> Any thought?
>
>
>
> --
> Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/
RE: New SQL execution engine
Posted by "Hostettler, Steve" <St...@wolterskluwer.com>.
Ivan,
Thanks that is good news. I use ignite as a platform and not directly to exec in-house application so these types of things are making the generic code less generic 😊.
Thanks a lot for the great work.
-----Original Message-----
From: Ivan Pavlukhin <vo...@gmail.com>
Sent: Monday, November 18, 2019 10:13 AM
To: dev <de...@ignite.apache.org>
Subject: Re: New SQL execution engine
Steve,
Yep, unfortunately query parallelism in current flavor is counter-intuitive. But it was designed so =( As Roman wrote
> And of course this feature should also be available in the new engine, though it's architecture may be changed.
The architecture of parallel execution will be definitely reconsidered. And currently we are targeted to do it so in one node cluster query will return the same results regardless parallelism.
сб, 16 нояб. 2019 г. в 12:48, steve.hostettler@gmail.com
<st...@gmail.com>:
>
> Actually I am now wondering whether this is not just a bug and that I
> should record it as such. As the behavior is different with and
> without the parallelism and there is no warning during execution or in the api.
>
> Any thought?
>
>
>
> --
> Sent from:
> https://nam04.safelinks.protection.outlook.com/?url=http%3A%2F%2Fapach
> e-ignite-developers.2346864.n4.nabble.com%2F&data=02%7C01%7CSteve.
> Hostettler%40wolterskluwer.com%7Cac6000fb14834d1abfa108d76c079273%7C8a
> c76c91e7f141ffa89c3553b2da2c17%7C0%7C0%7C637096652092270800&sdata=
> PcitGXmdx5DittW1RMAOEeneiLfKVrydUHL8uCKGi3g%3D&reserved=0
--
Best regards,
Ivan Pavlukhin
Re: New SQL execution engine
Posted by Ivan Pavlukhin <vo...@gmail.com>.
Steve,
Yep, unfortunately query parallelism in current flavor is
counter-intuitive. But it was designed so =( As Roman wrote
> And of course this feature should also be available in the new engine, though it's architecture may be changed.
The architecture of parallel execution will be definitely
reconsidered. And currently we are targeted to do it so in one node
cluster query will return the same results regardless parallelism.
сб, 16 нояб. 2019 г. в 12:48, steve.hostettler@gmail.com
<st...@gmail.com>:
>
> Actually I am now wondering whether this is not just a bug and that I should
> record it as such. As the behavior is different with and without the
> parallelism and there is no warning during execution or in the api.
>
> Any thought?
>
>
>
> --
> Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/
--
Best regards,
Ivan Pavlukhin
RE: New SQL execution engine
Posted by "steve.hostettler@gmail.com" <st...@gmail.com>.
Actually I am now wondering whether this is not just a bug and that I should
record it as such. As the behavior is different with and without the
parallelism and there is no warning during execution or in the api.
Any thought?
--
Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/
RE: New SQL execution engine
Posted by "Hostettler, Steve" <St...@wolterskluwer.com>.
Hi Roman,
Actually it does not work as I expect it. Please see https://github.com/hostettler/igniteParallelQueries
Do mvn clean install and then java -jar target/ignite-parallel-query-1.0.0-SNAPSHOT-jar-with-dependencies.jar
This demonstrates that with or without the flag the query does not return the same result. I understand that it probably because I did not set an affinity but it is very counter-intuitive.
Am I missing something?
-----Original Message-----
From: Roman Kondakov <ko...@mail.ru.INVALID>
Sent: Friday, November 15, 2019 11:46 AM
To: dev@ignite.apache.org
Subject: Re: New SQL execution engine
Hi Steve,
it is possible to execute queries in parallel even in the current engine, see docs here [1]. And of course this feature should also be available in the new engine, though it's architecture may be changed.
[1]
https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fapacheignite.readme.io%2Fv2.0%2Fdocs%2Fsql-performance-and-debugging%23query-parallelism&data=02%7C01%7CSteve.Hostettler%40wolterskluwer.com%7C2b752425baeb422af60408d769b9159d%7C8ac76c91e7f141ffa89c3553b2da2c17%7C0%7C0%7C637094115967087030&sdata=eN7b2RCJegg8J9KQVK6TIFhcS6NG7j5pWKFxX9GWyYk%3D&reserved=0
--
Kind Regards
Roman Kondakov
On 15.11.2019 12:53, steve.hostettler@gmail.com wrote:
> Dear all,
>
> would it be possible to also have then // execution of sql queries on single
> node with that approach?
> My understanding is that, for the moment, the SQL queries a re
> single-threaded for a given node if there is no affinity.
>
> Best Regards
>
>
>
> --
> Sent from: https://nam04.safelinks.protection.outlook.com/?url=http%3A%2F%2Fapache-ignite-developers.2346864.n4.nabble.com%2F&data=02%7C01%7CSteve.Hostettler%40wolterskluwer.com%7C2b752425baeb422af60408d769b9159d%7C8ac76c91e7f141ffa89c3553b2da2c17%7C0%7C0%7C637094115967087030&sdata=jXtLMt2dWYqM4KcRFkw4lby6K0o8glKnrLFgxZ96LbQ%3D&reserved=0
Re: New SQL execution engine
Posted by Roman Kondakov <ko...@mail.ru.INVALID>.
Hi Steve,
it is possible to execute queries in parallel even in the current
engine, see docs here [1]. And of course this feature should also be
available in the new engine, though it's architecture may be changed.
[1]
https://apacheignite.readme.io/v2.0/docs/sql-performance-and-debugging#query-parallelism
--
Kind Regards
Roman Kondakov
On 15.11.2019 12:53, steve.hostettler@gmail.com wrote:
> Dear all,
>
> would it be possible to also have then // execution of sql queries on single
> node with that approach?
> My understanding is that, for the moment, the SQL queries a re
> single-threaded for a given node if there is no affinity.
>
> Best Regards
>
>
>
> --
> Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/
Re: New SQL execution engine
Posted by "steve.hostettler@gmail.com" <st...@gmail.com>.
Dear all,
would it be possible to also have then // execution of sql queries on single
node with that approach?
My understanding is that, for the moment, the SQL queries a re
single-threaded for a given node if there is no affinity.
Best Regards
--
Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/
Re: New SQL execution engine
Posted by Ivan Pavlukhin <vo...@gmail.com>.
Nikolay,
Guys updated IEP [1]. Could you please check it? Are there any missing
parts needed at that stage?
[1] https://cwiki.apache.org/confluence/display/IGNITE/IEP-37%3A+New+query+execution+engine
вт, 1 окт. 2019 г. в 12:19, Ivan Pavlukhin <vo...@gmail.com>:
>
> Folks,
>
> I marked IEP-33 as obsolete. Also now the IEP-37 we currently are
> working with has a pretty URL
> https://cwiki.apache.org/confluence/display/IGNITE/IEP-37%3A+New+query+execution+engine
>
> вт, 1 окт. 2019 г. в 11:17, Seliverstov Igor <gv...@gmail.com>:
> >
> > Nikolay,
> >
> > The document you edited is wrong (and outdated).
> >
> > Since the author meant another idea, I decided not to change IEP-35 and
> > create a new one - IEP-37 (https://cwiki.apache.org/confluence/x/NBLABw).
> > It's already have a number of key requirements.
> >
> > Regards,
> > Igor
> >
> > вт, 1 окт. 2019 г., 6:14 Nikolay Izhikov <ni...@apache.org>:
> >
> > > Hello, Igniters.
> > >
> > > I extends IEP [1] with the tickets caused by H2 limitations.
> > >
> > > Please, let's write down requirements for engine in the IEP.
> > >
> > >
> > > https://cwiki.apache.org/confluence/display/IGNITE/IEP-33%3A+New+SQL+executor+engine+infrastructure
> > >
> > > В Пн, 30/09/2019 в 17:20 -0700, Denis Magda пишет:
> > > > Ivan, we need more of these discussions, totally agree with you ;)
> > > >
> > > > I've updated the Motivation paragraph outlining some high-level users we
> > > > see by working with our users. Hope it helps. Let's carry on and let me
> > > > send a note to Apache Calcite community.
> > > >
> > > > -
> > > > Denis
> > > >
> > > >
> > > > On Mon, Sep 30, 2019 at 1:56 AM Ivan Pavlukhin <vo...@gmail.com>
> > > wrote:
> > > >
> > > > > Folks,
> > > > >
> > > > > Thanks everyone for a hot discussion! Not every open source community
> > > > > has such open and boiling discussions. It means that people here
> > > > > really do care. And I am proud of it!
> > > > >
> > > > > As I understood, nobody is strictly against the proposed initiative.
> > > > > And I am glad that we can move forward (with some steps back along the
> > > > > way).
> > > > >
> > > > > пт, 27 сент. 2019 г. в 19:29, Nikolay Izhikov <ni...@apache.org>:
> > > > > >
> > > > > > Hello, Denis.
> > > > > >
> > > > > > Thanks for the clarifications.
> > > > > >
> > > > > > Sounds good for me.
> > > > > > All I try to say in this thread:
> > > > > > Guys, please, let's take a step back and write down
> > > requirements(what we
> > > > >
> > > > > want to get with SQL engine).
> > > > > > Which features and use-cases are primary for us.
> > > > > >
> > > > > > I'm sure you have done it, already during your research.
> > > > > >
> > > > > > Please, share it with the community.
> > > > > >
> > > > > > I'm pretty sure we would back to this document again and again during
> > > > >
> > > > > migration.
> > > > > > So good written design is worth it.
> > > > > >
> > > > > > В Пт, 27/09/2019 в 09:10 -0700, Denis Magda пишет:
> > > > > > > Ignite mates, let me try to move the discussion in a constructive
> > > way.
> > > > >
> > > > > It
> > > > > > > looks like we set a wrong context from the very beginning.
> > > > > > >
> > > > > > > Before proposing this idea to the community, some of us were
> > > > > > > discussing/researching the topic in different groups (the one need
> > > to
> > > > >
> > > > > think
> > > > > > > it through first before even suggesting to consider changes of this
> > > > > > > magnitude). The day has come to share this idea with the whole
> > > > >
> > > > > community
> > > > > > > and outline the next actions. But (!) nobody is 100% sure that
> > > that's
> > > > >
> > > > > the
> > > > > > > right decision. Thus, this will be an *experiment*, some of our
> > > > >
> > > > > community
> > > > > > > members will be developing a *prototype* and only based on the
> > > > >
> > > > > prototype
> > > > > > > outcomes we shall make a final decision. Igor, Roman, Ivan, Andrey,
> > > > >
> > > > > hope
> > > > > > > that nothing has changed and we're on the same page here.
> > > > > > >
> > > > > > > Many technical and architectural reasons that justify this project
> > > have
> > > > > > > been shared but let me throw in my perspective. There is nothing
> > > wrong
> > > > >
> > > > > with
> > > > > > > H2, that was the right choice for that time. Thanks to H2 and
> > > Ignite
> > > > >
> > > > > SQL
> > > > > > > APIs, our project is used across hundreds of deployments who are
> > > > > > > accelerating relational databases or use Ignite as a system of
> > > records.
> > > > > > > However, these days many more companies are migrating to
> > > *distributed*
> > > > > > > databases that speak SQL. For instance, if a couple of years ago 1
> > > out
> > > > >
> > > > > of
> > > > > > > 10 use cases needed support for multi-joins queries or queries with
> > > > > > > subselects or efficient memory usage then today there are 5 out of
> > > 10
> > > > >
> > > > > use
> > > > > > > cases of this kind; in the foreseeable future, it will be a 10 out
> > > of
> > > > >
> > > > > 10.
> > > > > > > So, the evolution is in progress -- the relational world goes
> > > > >
> > > > > distributed,
> > > > > > > it became exhaustive for both Ignite SQL maintainers and experts
> > > who
> > > > >
> > > > > help
> > > > > > > to tune it for production usage to keep pace with the evolution
> > > mostly
> > > > >
> > > > > due
> > > > > > > to the H2-dependency. Thus, Ignite SQL has to evolve and has to be
> > > > >
> > > > > ready to
> > > > > > > face the future reality.
> > > > > > >
> > > > > > > Luckily, we don't need to rush and don't have the right to rush
> > > because
> > > > > > > hundreds existing users have already trusted their production
> > > > >
> > > > > environments
> > > > > > > to Ignite SQL and we need to roll out changes with such a big
> > > impact
> > > > > > > carefully. So, I'm excited that Roman, Igor, Ivan, Andrey stepped
> > > in
> > > > >
> > > > > and
> > > > > > > agreed to be the first contributors who will be *experimenting*
> > > with
> > > > >
> > > > > the
> > > > > > > new SQL engine. Let's support them; let's connect them with Apache
> > > > >
> > > > > Calcite
> > > > > > > community and see how this story evolves. Folks, please keep the
> > > > >
> > > > > community
> > > > > > > aware of the progress, let us know when help is needed, some of us
> > > > >
> > > > > will be
> > > > > > > ready to support with development once you create a solid
> > > foundation
> > > > >
> > > > > for
> > > > > > > the prototype.
> > > > > > >
> > > > > > > -
> > > > > > > Denis
> > > > > > >
> > > > > > >
> > > > > > > On Fri, Sep 27, 2019 at 1:45 AM Igor Seliverstov <
> > > > >
> > > > > gvvinblade@apache.org>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Hi Igniters!
> > > > > > > >
> > > > > > > > As you might know currently we have many open issues relating to
> > > > >
> > > > > current
> > > > > > > > H2 based engine and its execution flow.
> > > > > > > >
> > > > > > > > Some of them are critical (like impossibility to execute
> > > particular
> > > > > > > > queries), some of them are majors (like impossibility to execute
> > > > >
> > > > > particular
> > > > > > > > queries without pre-preparation your data to have a collocation)
> > > and
> > > > >
> > > > > many
> > > > > > > > minors.
> > > > > > > >
> > > > > > > > Most of the issues cannot be solved without whole engine
> > > redesign.
> > > > > > > >
> > > > > > > > So, here the proposal:
> > > > > > > >
> > > > >
> > > > >
> > > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=130028084
> > > > > > > >
> > > > > > > > I'll appreciate if you share your thoughts on top of that.
> > > > > > > >
> > > > > > > > Regards,
> > > > > > > > Igor
> > > > > > > >
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Best regards,
> > > > > Ivan Pavlukhin
> > > > >
> > >
>
>
>
> --
> Best regards,
> Ivan Pavlukhin
--
Best regards,
Ivan Pavlukhin
Re: New SQL execution engine
Posted by Ivan Pavlukhin <vo...@gmail.com>.
Folks,
I marked IEP-33 as obsolete. Also now the IEP-37 we currently are
working with has a pretty URL
https://cwiki.apache.org/confluence/display/IGNITE/IEP-37%3A+New+query+execution+engine
вт, 1 окт. 2019 г. в 11:17, Seliverstov Igor <gv...@gmail.com>:
>
> Nikolay,
>
> The document you edited is wrong (and outdated).
>
> Since the author meant another idea, I decided not to change IEP-35 and
> create a new one - IEP-37 (https://cwiki.apache.org/confluence/x/NBLABw).
> It's already have a number of key requirements.
>
> Regards,
> Igor
>
> вт, 1 окт. 2019 г., 6:14 Nikolay Izhikov <ni...@apache.org>:
>
> > Hello, Igniters.
> >
> > I extends IEP [1] with the tickets caused by H2 limitations.
> >
> > Please, let's write down requirements for engine in the IEP.
> >
> >
> > https://cwiki.apache.org/confluence/display/IGNITE/IEP-33%3A+New+SQL+executor+engine+infrastructure
> >
> > В Пн, 30/09/2019 в 17:20 -0700, Denis Magda пишет:
> > > Ivan, we need more of these discussions, totally agree with you ;)
> > >
> > > I've updated the Motivation paragraph outlining some high-level users we
> > > see by working with our users. Hope it helps. Let's carry on and let me
> > > send a note to Apache Calcite community.
> > >
> > > -
> > > Denis
> > >
> > >
> > > On Mon, Sep 30, 2019 at 1:56 AM Ivan Pavlukhin <vo...@gmail.com>
> > wrote:
> > >
> > > > Folks,
> > > >
> > > > Thanks everyone for a hot discussion! Not every open source community
> > > > has such open and boiling discussions. It means that people here
> > > > really do care. And I am proud of it!
> > > >
> > > > As I understood, nobody is strictly against the proposed initiative.
> > > > And I am glad that we can move forward (with some steps back along the
> > > > way).
> > > >
> > > > пт, 27 сент. 2019 г. в 19:29, Nikolay Izhikov <ni...@apache.org>:
> > > > >
> > > > > Hello, Denis.
> > > > >
> > > > > Thanks for the clarifications.
> > > > >
> > > > > Sounds good for me.
> > > > > All I try to say in this thread:
> > > > > Guys, please, let's take a step back and write down
> > requirements(what we
> > > >
> > > > want to get with SQL engine).
> > > > > Which features and use-cases are primary for us.
> > > > >
> > > > > I'm sure you have done it, already during your research.
> > > > >
> > > > > Please, share it with the community.
> > > > >
> > > > > I'm pretty sure we would back to this document again and again during
> > > >
> > > > migration.
> > > > > So good written design is worth it.
> > > > >
> > > > > В Пт, 27/09/2019 в 09:10 -0700, Denis Magda пишет:
> > > > > > Ignite mates, let me try to move the discussion in a constructive
> > way.
> > > >
> > > > It
> > > > > > looks like we set a wrong context from the very beginning.
> > > > > >
> > > > > > Before proposing this idea to the community, some of us were
> > > > > > discussing/researching the topic in different groups (the one need
> > to
> > > >
> > > > think
> > > > > > it through first before even suggesting to consider changes of this
> > > > > > magnitude). The day has come to share this idea with the whole
> > > >
> > > > community
> > > > > > and outline the next actions. But (!) nobody is 100% sure that
> > that's
> > > >
> > > > the
> > > > > > right decision. Thus, this will be an *experiment*, some of our
> > > >
> > > > community
> > > > > > members will be developing a *prototype* and only based on the
> > > >
> > > > prototype
> > > > > > outcomes we shall make a final decision. Igor, Roman, Ivan, Andrey,
> > > >
> > > > hope
> > > > > > that nothing has changed and we're on the same page here.
> > > > > >
> > > > > > Many technical and architectural reasons that justify this project
> > have
> > > > > > been shared but let me throw in my perspective. There is nothing
> > wrong
> > > >
> > > > with
> > > > > > H2, that was the right choice for that time. Thanks to H2 and
> > Ignite
> > > >
> > > > SQL
> > > > > > APIs, our project is used across hundreds of deployments who are
> > > > > > accelerating relational databases or use Ignite as a system of
> > records.
> > > > > > However, these days many more companies are migrating to
> > *distributed*
> > > > > > databases that speak SQL. For instance, if a couple of years ago 1
> > out
> > > >
> > > > of
> > > > > > 10 use cases needed support for multi-joins queries or queries with
> > > > > > subselects or efficient memory usage then today there are 5 out of
> > 10
> > > >
> > > > use
> > > > > > cases of this kind; in the foreseeable future, it will be a 10 out
> > of
> > > >
> > > > 10.
> > > > > > So, the evolution is in progress -- the relational world goes
> > > >
> > > > distributed,
> > > > > > it became exhaustive for both Ignite SQL maintainers and experts
> > who
> > > >
> > > > help
> > > > > > to tune it for production usage to keep pace with the evolution
> > mostly
> > > >
> > > > due
> > > > > > to the H2-dependency. Thus, Ignite SQL has to evolve and has to be
> > > >
> > > > ready to
> > > > > > face the future reality.
> > > > > >
> > > > > > Luckily, we don't need to rush and don't have the right to rush
> > because
> > > > > > hundreds existing users have already trusted their production
> > > >
> > > > environments
> > > > > > to Ignite SQL and we need to roll out changes with such a big
> > impact
> > > > > > carefully. So, I'm excited that Roman, Igor, Ivan, Andrey stepped
> > in
> > > >
> > > > and
> > > > > > agreed to be the first contributors who will be *experimenting*
> > with
> > > >
> > > > the
> > > > > > new SQL engine. Let's support them; let's connect them with Apache
> > > >
> > > > Calcite
> > > > > > community and see how this story evolves. Folks, please keep the
> > > >
> > > > community
> > > > > > aware of the progress, let us know when help is needed, some of us
> > > >
> > > > will be
> > > > > > ready to support with development once you create a solid
> > foundation
> > > >
> > > > for
> > > > > > the prototype.
> > > > > >
> > > > > > -
> > > > > > Denis
> > > > > >
> > > > > >
> > > > > > On Fri, Sep 27, 2019 at 1:45 AM Igor Seliverstov <
> > > >
> > > > gvvinblade@apache.org>
> > > > > > wrote:
> > > > > >
> > > > > > > Hi Igniters!
> > > > > > >
> > > > > > > As you might know currently we have many open issues relating to
> > > >
> > > > current
> > > > > > > H2 based engine and its execution flow.
> > > > > > >
> > > > > > > Some of them are critical (like impossibility to execute
> > particular
> > > > > > > queries), some of them are majors (like impossibility to execute
> > > >
> > > > particular
> > > > > > > queries without pre-preparation your data to have a collocation)
> > and
> > > >
> > > > many
> > > > > > > minors.
> > > > > > >
> > > > > > > Most of the issues cannot be solved without whole engine
> > redesign.
> > > > > > >
> > > > > > > So, here the proposal:
> > > > > > >
> > > >
> > > >
> > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=130028084
> > > > > > >
> > > > > > > I'll appreciate if you share your thoughts on top of that.
> > > > > > >
> > > > > > > Regards,
> > > > > > > Igor
> > > > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > Best regards,
> > > > Ivan Pavlukhin
> > > >
> >
--
Best regards,
Ivan Pavlukhin
Re: New SQL execution engine
Posted by Seliverstov Igor <gv...@gmail.com>.
Nikolay,
The document you edited is wrong (and outdated).
Since the author meant another idea, I decided not to change IEP-35 and
create a new one - IEP-37 (https://cwiki.apache.org/confluence/x/NBLABw).
It's already have a number of key requirements.
Regards,
Igor
вт, 1 окт. 2019 г., 6:14 Nikolay Izhikov <ni...@apache.org>:
> Hello, Igniters.
>
> I extends IEP [1] with the tickets caused by H2 limitations.
>
> Please, let's write down requirements for engine in the IEP.
>
>
> https://cwiki.apache.org/confluence/display/IGNITE/IEP-33%3A+New+SQL+executor+engine+infrastructure
>
> В Пн, 30/09/2019 в 17:20 -0700, Denis Magda пишет:
> > Ivan, we need more of these discussions, totally agree with you ;)
> >
> > I've updated the Motivation paragraph outlining some high-level users we
> > see by working with our users. Hope it helps. Let's carry on and let me
> > send a note to Apache Calcite community.
> >
> > -
> > Denis
> >
> >
> > On Mon, Sep 30, 2019 at 1:56 AM Ivan Pavlukhin <vo...@gmail.com>
> wrote:
> >
> > > Folks,
> > >
> > > Thanks everyone for a hot discussion! Not every open source community
> > > has such open and boiling discussions. It means that people here
> > > really do care. And I am proud of it!
> > >
> > > As I understood, nobody is strictly against the proposed initiative.
> > > And I am glad that we can move forward (with some steps back along the
> > > way).
> > >
> > > пт, 27 сент. 2019 г. в 19:29, Nikolay Izhikov <ni...@apache.org>:
> > > >
> > > > Hello, Denis.
> > > >
> > > > Thanks for the clarifications.
> > > >
> > > > Sounds good for me.
> > > > All I try to say in this thread:
> > > > Guys, please, let's take a step back and write down
> requirements(what we
> > >
> > > want to get with SQL engine).
> > > > Which features and use-cases are primary for us.
> > > >
> > > > I'm sure you have done it, already during your research.
> > > >
> > > > Please, share it with the community.
> > > >
> > > > I'm pretty sure we would back to this document again and again during
> > >
> > > migration.
> > > > So good written design is worth it.
> > > >
> > > > В Пт, 27/09/2019 в 09:10 -0700, Denis Magda пишет:
> > > > > Ignite mates, let me try to move the discussion in a constructive
> way.
> > >
> > > It
> > > > > looks like we set a wrong context from the very beginning.
> > > > >
> > > > > Before proposing this idea to the community, some of us were
> > > > > discussing/researching the topic in different groups (the one need
> to
> > >
> > > think
> > > > > it through first before even suggesting to consider changes of this
> > > > > magnitude). The day has come to share this idea with the whole
> > >
> > > community
> > > > > and outline the next actions. But (!) nobody is 100% sure that
> that's
> > >
> > > the
> > > > > right decision. Thus, this will be an *experiment*, some of our
> > >
> > > community
> > > > > members will be developing a *prototype* and only based on the
> > >
> > > prototype
> > > > > outcomes we shall make a final decision. Igor, Roman, Ivan, Andrey,
> > >
> > > hope
> > > > > that nothing has changed and we're on the same page here.
> > > > >
> > > > > Many technical and architectural reasons that justify this project
> have
> > > > > been shared but let me throw in my perspective. There is nothing
> wrong
> > >
> > > with
> > > > > H2, that was the right choice for that time. Thanks to H2 and
> Ignite
> > >
> > > SQL
> > > > > APIs, our project is used across hundreds of deployments who are
> > > > > accelerating relational databases or use Ignite as a system of
> records.
> > > > > However, these days many more companies are migrating to
> *distributed*
> > > > > databases that speak SQL. For instance, if a couple of years ago 1
> out
> > >
> > > of
> > > > > 10 use cases needed support for multi-joins queries or queries with
> > > > > subselects or efficient memory usage then today there are 5 out of
> 10
> > >
> > > use
> > > > > cases of this kind; in the foreseeable future, it will be a 10 out
> of
> > >
> > > 10.
> > > > > So, the evolution is in progress -- the relational world goes
> > >
> > > distributed,
> > > > > it became exhaustive for both Ignite SQL maintainers and experts
> who
> > >
> > > help
> > > > > to tune it for production usage to keep pace with the evolution
> mostly
> > >
> > > due
> > > > > to the H2-dependency. Thus, Ignite SQL has to evolve and has to be
> > >
> > > ready to
> > > > > face the future reality.
> > > > >
> > > > > Luckily, we don't need to rush and don't have the right to rush
> because
> > > > > hundreds existing users have already trusted their production
> > >
> > > environments
> > > > > to Ignite SQL and we need to roll out changes with such a big
> impact
> > > > > carefully. So, I'm excited that Roman, Igor, Ivan, Andrey stepped
> in
> > >
> > > and
> > > > > agreed to be the first contributors who will be *experimenting*
> with
> > >
> > > the
> > > > > new SQL engine. Let's support them; let's connect them with Apache
> > >
> > > Calcite
> > > > > community and see how this story evolves. Folks, please keep the
> > >
> > > community
> > > > > aware of the progress, let us know when help is needed, some of us
> > >
> > > will be
> > > > > ready to support with development once you create a solid
> foundation
> > >
> > > for
> > > > > the prototype.
> > > > >
> > > > > -
> > > > > Denis
> > > > >
> > > > >
> > > > > On Fri, Sep 27, 2019 at 1:45 AM Igor Seliverstov <
> > >
> > > gvvinblade@apache.org>
> > > > > wrote:
> > > > >
> > > > > > Hi Igniters!
> > > > > >
> > > > > > As you might know currently we have many open issues relating to
> > >
> > > current
> > > > > > H2 based engine and its execution flow.
> > > > > >
> > > > > > Some of them are critical (like impossibility to execute
> particular
> > > > > > queries), some of them are majors (like impossibility to execute
> > >
> > > particular
> > > > > > queries without pre-preparation your data to have a collocation)
> and
> > >
> > > many
> > > > > > minors.
> > > > > >
> > > > > > Most of the issues cannot be solved without whole engine
> redesign.
> > > > > >
> > > > > > So, here the proposal:
> > > > > >
> > >
> > >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=130028084
> > > > > >
> > > > > > I'll appreciate if you share your thoughts on top of that.
> > > > > >
> > > > > > Regards,
> > > > > > Igor
> > > > > >
> > >
> > >
> > >
> > > --
> > > Best regards,
> > > Ivan Pavlukhin
> > >
>
Re: New SQL execution engine
Posted by Nikolay Izhikov <ni...@apache.org>.
Hello, Igniters.
I extends IEP [1] with the tickets caused by H2 limitations.
Please, let's write down requirements for engine in the IEP.
https://cwiki.apache.org/confluence/display/IGNITE/IEP-33%3A+New+SQL+executor+engine+infrastructure
В Пн, 30/09/2019 в 17:20 -0700, Denis Magda пишет:
> Ivan, we need more of these discussions, totally agree with you ;)
>
> I've updated the Motivation paragraph outlining some high-level users we
> see by working with our users. Hope it helps. Let's carry on and let me
> send a note to Apache Calcite community.
>
> -
> Denis
>
>
> On Mon, Sep 30, 2019 at 1:56 AM Ivan Pavlukhin <vo...@gmail.com> wrote:
>
> > Folks,
> >
> > Thanks everyone for a hot discussion! Not every open source community
> > has such open and boiling discussions. It means that people here
> > really do care. And I am proud of it!
> >
> > As I understood, nobody is strictly against the proposed initiative.
> > And I am glad that we can move forward (with some steps back along the
> > way).
> >
> > пт, 27 сент. 2019 г. в 19:29, Nikolay Izhikov <ni...@apache.org>:
> > >
> > > Hello, Denis.
> > >
> > > Thanks for the clarifications.
> > >
> > > Sounds good for me.
> > > All I try to say in this thread:
> > > Guys, please, let's take a step back and write down requirements(what we
> >
> > want to get with SQL engine).
> > > Which features and use-cases are primary for us.
> > >
> > > I'm sure you have done it, already during your research.
> > >
> > > Please, share it with the community.
> > >
> > > I'm pretty sure we would back to this document again and again during
> >
> > migration.
> > > So good written design is worth it.
> > >
> > > В Пт, 27/09/2019 в 09:10 -0700, Denis Magda пишет:
> > > > Ignite mates, let me try to move the discussion in a constructive way.
> >
> > It
> > > > looks like we set a wrong context from the very beginning.
> > > >
> > > > Before proposing this idea to the community, some of us were
> > > > discussing/researching the topic in different groups (the one need to
> >
> > think
> > > > it through first before even suggesting to consider changes of this
> > > > magnitude). The day has come to share this idea with the whole
> >
> > community
> > > > and outline the next actions. But (!) nobody is 100% sure that that's
> >
> > the
> > > > right decision. Thus, this will be an *experiment*, some of our
> >
> > community
> > > > members will be developing a *prototype* and only based on the
> >
> > prototype
> > > > outcomes we shall make a final decision. Igor, Roman, Ivan, Andrey,
> >
> > hope
> > > > that nothing has changed and we're on the same page here.
> > > >
> > > > Many technical and architectural reasons that justify this project have
> > > > been shared but let me throw in my perspective. There is nothing wrong
> >
> > with
> > > > H2, that was the right choice for that time. Thanks to H2 and Ignite
> >
> > SQL
> > > > APIs, our project is used across hundreds of deployments who are
> > > > accelerating relational databases or use Ignite as a system of records.
> > > > However, these days many more companies are migrating to *distributed*
> > > > databases that speak SQL. For instance, if a couple of years ago 1 out
> >
> > of
> > > > 10 use cases needed support for multi-joins queries or queries with
> > > > subselects or efficient memory usage then today there are 5 out of 10
> >
> > use
> > > > cases of this kind; in the foreseeable future, it will be a 10 out of
> >
> > 10.
> > > > So, the evolution is in progress -- the relational world goes
> >
> > distributed,
> > > > it became exhaustive for both Ignite SQL maintainers and experts who
> >
> > help
> > > > to tune it for production usage to keep pace with the evolution mostly
> >
> > due
> > > > to the H2-dependency. Thus, Ignite SQL has to evolve and has to be
> >
> > ready to
> > > > face the future reality.
> > > >
> > > > Luckily, we don't need to rush and don't have the right to rush because
> > > > hundreds existing users have already trusted their production
> >
> > environments
> > > > to Ignite SQL and we need to roll out changes with such a big impact
> > > > carefully. So, I'm excited that Roman, Igor, Ivan, Andrey stepped in
> >
> > and
> > > > agreed to be the first contributors who will be *experimenting* with
> >
> > the
> > > > new SQL engine. Let's support them; let's connect them with Apache
> >
> > Calcite
> > > > community and see how this story evolves. Folks, please keep the
> >
> > community
> > > > aware of the progress, let us know when help is needed, some of us
> >
> > will be
> > > > ready to support with development once you create a solid foundation
> >
> > for
> > > > the prototype.
> > > >
> > > > -
> > > > Denis
> > > >
> > > >
> > > > On Fri, Sep 27, 2019 at 1:45 AM Igor Seliverstov <
> >
> > gvvinblade@apache.org>
> > > > wrote:
> > > >
> > > > > Hi Igniters!
> > > > >
> > > > > As you might know currently we have many open issues relating to
> >
> > current
> > > > > H2 based engine and its execution flow.
> > > > >
> > > > > Some of them are critical (like impossibility to execute particular
> > > > > queries), some of them are majors (like impossibility to execute
> >
> > particular
> > > > > queries without pre-preparation your data to have a collocation) and
> >
> > many
> > > > > minors.
> > > > >
> > > > > Most of the issues cannot be solved without whole engine redesign.
> > > > >
> > > > > So, here the proposal:
> > > > >
> >
> > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=130028084
> > > > >
> > > > > I'll appreciate if you share your thoughts on top of that.
> > > > >
> > > > > Regards,
> > > > > Igor
> > > > >
> >
> >
> >
> > --
> > Best regards,
> > Ivan Pavlukhin
> >