You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@calcite.apache.org by Atri Sharma <at...@gmail.com> on 2021/05/13 10:27:51 UTC
Using Calcite as a Distributed Optimizer
Hi All,
Are there examples of using Calcite to compile and optimize queries to
be run on a set of nodes, and then merge partial results back?
Atri
--
Regards,
Atri
l'apprenant
Re: Using Calcite as a Distributed Optimizer
Posted by Fan Liya <li...@gmail.com>.
I think you can refer to Flink
Best,
Liya Fan
On Fri, May 14, 2021 at 12:32 PM Atri Sharma <at...@gmail.com> wrote:
> Is there an example I can refer to?
>
> On Fri, 14 May 2021, 04:59 Haisheng Yuan, <hy...@apache.org> wrote:
>
> > Yes, definitely. Many distributed big data systems use Apache Calcite to
> > optimize queries and generate distributed plans.
> >
> > On 2021/05/13 23:16:10, Atri Sharma <at...@gmail.com> wrote:
> > > Thank you.
> > >
> > > So my use case is such that I wish to use Calcite as a two phase
> > optimizer
> > > -- Get a SQL query, compile it and optimize it and convert it to a SQL
> > > fragment.
> > >
> > > Then run the query on worker nodes, get results on master and merge
> > results.
> > >
> > > This question spans both Calcite and Avatica, but I wanted to
> understand
> > if
> > > achieving the above is possible with Calcite today.
> > >
> > > Atri
> > >
> > > On Fri, 14 May 2021, 01:20 Julian Hyde, <jh...@gmail.com>
> wrote:
> > >
> > > > Calcite has no user@ list. So, ask away here!
> > > >
> > > > > On May 13, 2021, at 10:52 AM, Atri Sharma <at...@gmail.com>
> > wrote:
> > > > >
> > > > > Sorry, didn't realize I had sent to the dev list. I will send to
> the
> > user
> > > > > list
> > > > >
> > > > > On Thu, 13 May 2021, 15:57 Atri Sharma, <at...@gmail.com>
> wrote:
> > > > >
> > > > >> Hi All,
> > > > >>
> > > > >> Are there examples of using Calcite to compile and optimize
> queries
> > to
> > > > >> be run on a set of nodes, and then merge partial results back?
> > > > >>
> > > > >> Atri
> > > > >>
> > > > >> --
> > > > >> Regards,
> > > > >>
> > > > >> Atri
> > > > >> l'apprenant
> > > > >>
> > > >
> > > >
> > >
> >
>
Re: Using Calcite as a Distributed Optimizer
Posted by Roman Kondakov <ko...@mail.ru.INVALID>.
Hi Atri,
you can refer to this list [1]. I also aware that Apache Ignite and
Hazelcast made some efforts to employ Calcite as a distributed query
planner.
[1] https://calcite.apache.org/docs/powered_by.html
--
Roman Kondakov
On 14.05.2021 14:31, Atri Sharma wrote:
> Is there an example I can refer to?
>
> On Fri, 14 May 2021, 04:59 Haisheng Yuan, <hy...@apache.org> wrote:
>
>> Yes, definitely. Many distributed big data systems use Apache Calcite to
>> optimize queries and generate distributed plans.
>>
>> On 2021/05/13 23:16:10, Atri Sharma <at...@gmail.com> wrote:
>>> Thank you.
>>>
>>> So my use case is such that I wish to use Calcite as a two phase
>> optimizer
>>> -- Get a SQL query, compile it and optimize it and convert it to a SQL
>>> fragment.
>>>
>>> Then run the query on worker nodes, get results on master and merge
>> results.
>>>
>>> This question spans both Calcite and Avatica, but I wanted to understand
>> if
>>> achieving the above is possible with Calcite today.
>>>
>>> Atri
>>>
>>> On Fri, 14 May 2021, 01:20 Julian Hyde, <jh...@gmail.com> wrote:
>>>
>>>> Calcite has no user@ list. So, ask away here!
>>>>
>>>>> On May 13, 2021, at 10:52 AM, Atri Sharma <at...@gmail.com>
>> wrote:
>>>>>
>>>>> Sorry, didn't realize I had sent to the dev list. I will send to the
>> user
>>>>> list
>>>>>
>>>>> On Thu, 13 May 2021, 15:57 Atri Sharma, <at...@gmail.com> wrote:
>>>>>
>>>>>> Hi All,
>>>>>>
>>>>>> Are there examples of using Calcite to compile and optimize queries
>> to
>>>>>> be run on a set of nodes, and then merge partial results back?
>>>>>>
>>>>>> Atri
>>>>>>
>>>>>> --
>>>>>> Regards,
>>>>>>
>>>>>> Atri
>>>>>> l'apprenant
>>>>>>
>>>>
>>>>
>>>
>>
>
Re: Using Calcite as a Distributed Optimizer
Posted by Atri Sharma <at...@gmail.com>.
Is there an example I can refer to?
On Fri, 14 May 2021, 04:59 Haisheng Yuan, <hy...@apache.org> wrote:
> Yes, definitely. Many distributed big data systems use Apache Calcite to
> optimize queries and generate distributed plans.
>
> On 2021/05/13 23:16:10, Atri Sharma <at...@gmail.com> wrote:
> > Thank you.
> >
> > So my use case is such that I wish to use Calcite as a two phase
> optimizer
> > -- Get a SQL query, compile it and optimize it and convert it to a SQL
> > fragment.
> >
> > Then run the query on worker nodes, get results on master and merge
> results.
> >
> > This question spans both Calcite and Avatica, but I wanted to understand
> if
> > achieving the above is possible with Calcite today.
> >
> > Atri
> >
> > On Fri, 14 May 2021, 01:20 Julian Hyde, <jh...@gmail.com> wrote:
> >
> > > Calcite has no user@ list. So, ask away here!
> > >
> > > > On May 13, 2021, at 10:52 AM, Atri Sharma <at...@gmail.com>
> wrote:
> > > >
> > > > Sorry, didn't realize I had sent to the dev list. I will send to the
> user
> > > > list
> > > >
> > > > On Thu, 13 May 2021, 15:57 Atri Sharma, <at...@gmail.com> wrote:
> > > >
> > > >> Hi All,
> > > >>
> > > >> Are there examples of using Calcite to compile and optimize queries
> to
> > > >> be run on a set of nodes, and then merge partial results back?
> > > >>
> > > >> Atri
> > > >>
> > > >> --
> > > >> Regards,
> > > >>
> > > >> Atri
> > > >> l'apprenant
> > > >>
> > >
> > >
> >
>
Re: Using Calcite as a Distributed Optimizer
Posted by Haisheng Yuan <hy...@apache.org>.
Yes, definitely. Many distributed big data systems use Apache Calcite to optimize queries and generate distributed plans.
On 2021/05/13 23:16:10, Atri Sharma <at...@gmail.com> wrote:
> Thank you.
>
> So my use case is such that I wish to use Calcite as a two phase optimizer
> -- Get a SQL query, compile it and optimize it and convert it to a SQL
> fragment.
>
> Then run the query on worker nodes, get results on master and merge results.
>
> This question spans both Calcite and Avatica, but I wanted to understand if
> achieving the above is possible with Calcite today.
>
> Atri
>
> On Fri, 14 May 2021, 01:20 Julian Hyde, <jh...@gmail.com> wrote:
>
> > Calcite has no user@ list. So, ask away here!
> >
> > > On May 13, 2021, at 10:52 AM, Atri Sharma <at...@gmail.com> wrote:
> > >
> > > Sorry, didn't realize I had sent to the dev list. I will send to the user
> > > list
> > >
> > > On Thu, 13 May 2021, 15:57 Atri Sharma, <at...@gmail.com> wrote:
> > >
> > >> Hi All,
> > >>
> > >> Are there examples of using Calcite to compile and optimize queries to
> > >> be run on a set of nodes, and then merge partial results back?
> > >>
> > >> Atri
> > >>
> > >> --
> > >> Regards,
> > >>
> > >> Atri
> > >> l'apprenant
> > >>
> >
> >
>
Re: Using Calcite as a Distributed Optimizer
Posted by Atri Sharma <at...@gmail.com>.
Thank you.
So my use case is such that I wish to use Calcite as a two phase optimizer
-- Get a SQL query, compile it and optimize it and convert it to a SQL
fragment.
Then run the query on worker nodes, get results on master and merge results.
This question spans both Calcite and Avatica, but I wanted to understand if
achieving the above is possible with Calcite today.
Atri
On Fri, 14 May 2021, 01:20 Julian Hyde, <jh...@gmail.com> wrote:
> Calcite has no user@ list. So, ask away here!
>
> > On May 13, 2021, at 10:52 AM, Atri Sharma <at...@gmail.com> wrote:
> >
> > Sorry, didn't realize I had sent to the dev list. I will send to the user
> > list
> >
> > On Thu, 13 May 2021, 15:57 Atri Sharma, <at...@gmail.com> wrote:
> >
> >> Hi All,
> >>
> >> Are there examples of using Calcite to compile and optimize queries to
> >> be run on a set of nodes, and then merge partial results back?
> >>
> >> Atri
> >>
> >> --
> >> Regards,
> >>
> >> Atri
> >> l'apprenant
> >>
>
>
Re: Using Calcite as a Distributed Optimizer
Posted by Julian Hyde <jh...@gmail.com>.
Calcite has no user@ list. So, ask away here!
> On May 13, 2021, at 10:52 AM, Atri Sharma <at...@gmail.com> wrote:
>
> Sorry, didn't realize I had sent to the dev list. I will send to the user
> list
>
> On Thu, 13 May 2021, 15:57 Atri Sharma, <at...@gmail.com> wrote:
>
>> Hi All,
>>
>> Are there examples of using Calcite to compile and optimize queries to
>> be run on a set of nodes, and then merge partial results back?
>>
>> Atri
>>
>> --
>> Regards,
>>
>> Atri
>> l'apprenant
>>
Re: Using Calcite as a Distributed Optimizer
Posted by Atri Sharma <at...@gmail.com>.
Sorry, didn't realize I had sent to the dev list. I will send to the user
list
On Thu, 13 May 2021, 15:57 Atri Sharma, <at...@gmail.com> wrote:
> Hi All,
>
> Are there examples of using Calcite to compile and optimize queries to
> be run on a set of nodes, and then merge partial results back?
>
> Atri
>
> --
> Regards,
>
> Atri
> l'apprenant
>