You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@calcite.apache.org by Chunwei Lei <ch...@gmail.com> on 2020/07/01 02:57:25 UTC
Re: Recommendation for matchLimit for HepProgram
Hi, Maytas.
In my system, we set the matchLimit to 1200, which can cover most cases.
Best,
Chunwei
On Tue, Jun 30, 2020 at 4:30 PM Maytas Monsereenusorn <ma...@apache.org>
wrote:
> Hi,
>
> I am encountering a problem in Calcite that causes my query to result in an
> StackOverflow exception. A simple query that can reproduces this issue is
> something liken SELECT ARRAY ['Hello', NULL]
>
> I believe this is the cause (and also the fix)
> https://issues.apache.org/jira/browse/CALCITE-3845
> However, we are still currently using Calcite 1.21.0 which does not have
> this fix (fix above is in 1.23.0). I found that a workaround that can work
> without upgrading Calcite is to set currentProgram.matchLimit. I was able
> to confirm that the workaround (setting a limit on matchLimit) fixes this
> issue without needing to upgrade. I am wondering if there is any
> recommendation on a sensible value to use for the matchLimit
> for HepProgramBuilder?
>
> The default value is Integer.MAX_VALUE which I don't think will ever be
> possible due to stack / memory limitations.
>
> - Maytas
>
Re: Recommendation for matchLimit for HepProgram
Posted by Maytas Monsereenusorn <ma...@gmail.com>.
Hi Chunwei,
Thank you for your suggestion and insight!
-Maytas
On Wed, Jul 1, 2020 at 8:16 PM Chunwei Lei <ch...@gmail.com> wrote:
> Hi, Maytas. Our system is MaxCompute[1], which is a big data computing and
> storage platform. We use
> Calcite for cost-based query optimization.
>
> To reduce the search space, before entering VolcanoPlanner we use
> HepPlanner to do some
> optimizations, such as column pruning, predicate pushdown, and so on.
>
>
>
> [1]https://www.alibabacloud.com/product/maxcompute
>
>
>
> Best,
> Chunwei
>
>
> On Wed, Jul 1, 2020 at 1:29 PM Maytas Monsereenusorn <ma...@gmail.com>
> wrote:
>
> > Hi Chunwei,
> >
> > Thank you for your reply. Can you share a little bit more about your
> > particular system?
> > For context, our system is Apache Druid.
> >
> > Best Regards,
> > Maytas
> >
> > On Tue, Jun 30, 2020 at 7:57 PM Chunwei Lei <ch...@gmail.com>
> > wrote:
> >
> > > Hi, Maytas.
> > >
> > > In my system, we set the matchLimit to 1200, which can cover most
> cases.
> > >
> > > Best,
> > > Chunwei
> > >
> > >
> > > On Tue, Jun 30, 2020 at 4:30 PM Maytas Monsereenusorn <
> > maytasm@apache.org>
> > > wrote:
> > >
> > > > Hi,
> > > >
> > > > I am encountering a problem in Calcite that causes my query to result
> > in
> > > an
> > > > StackOverflow exception. A simple query that can reproduces this
> issue
> > is
> > > > something liken SELECT ARRAY ['Hello', NULL]
> > > >
> > > > I believe this is the cause (and also the fix)
> > > > https://issues.apache.org/jira/browse/CALCITE-3845
> > > > However, we are still currently using Calcite 1.21.0 which does not
> > have
> > > > this fix (fix above is in 1.23.0). I found that a workaround that can
> > > work
> > > > without upgrading Calcite is to set currentProgram.matchLimit. I was
> > able
> > > > to confirm that the workaround (setting a limit on matchLimit) fixes
> > this
> > > > issue without needing to upgrade. I am wondering if there is any
> > > > recommendation on a sensible value to use for the matchLimit
> > > > for HepProgramBuilder?
> > > >
> > > > The default value is Integer.MAX_VALUE which I don't think will ever
> be
> > > > possible due to stack / memory limitations.
> > > >
> > > > - Maytas
> > > >
> > >
> >
>
Re: Recommendation for matchLimit for HepProgram
Posted by Chunwei Lei <ch...@gmail.com>.
Hi, Maytas. Our system is MaxCompute[1], which is a big data computing and
storage platform. We use
Calcite for cost-based query optimization.
To reduce the search space, before entering VolcanoPlanner we use
HepPlanner to do some
optimizations, such as column pruning, predicate pushdown, and so on.
[1]https://www.alibabacloud.com/product/maxcompute
Best,
Chunwei
On Wed, Jul 1, 2020 at 1:29 PM Maytas Monsereenusorn <ma...@gmail.com>
wrote:
> Hi Chunwei,
>
> Thank you for your reply. Can you share a little bit more about your
> particular system?
> For context, our system is Apache Druid.
>
> Best Regards,
> Maytas
>
> On Tue, Jun 30, 2020 at 7:57 PM Chunwei Lei <ch...@gmail.com>
> wrote:
>
> > Hi, Maytas.
> >
> > In my system, we set the matchLimit to 1200, which can cover most cases.
> >
> > Best,
> > Chunwei
> >
> >
> > On Tue, Jun 30, 2020 at 4:30 PM Maytas Monsereenusorn <
> maytasm@apache.org>
> > wrote:
> >
> > > Hi,
> > >
> > > I am encountering a problem in Calcite that causes my query to result
> in
> > an
> > > StackOverflow exception. A simple query that can reproduces this issue
> is
> > > something liken SELECT ARRAY ['Hello', NULL]
> > >
> > > I believe this is the cause (and also the fix)
> > > https://issues.apache.org/jira/browse/CALCITE-3845
> > > However, we are still currently using Calcite 1.21.0 which does not
> have
> > > this fix (fix above is in 1.23.0). I found that a workaround that can
> > work
> > > without upgrading Calcite is to set currentProgram.matchLimit. I was
> able
> > > to confirm that the workaround (setting a limit on matchLimit) fixes
> this
> > > issue without needing to upgrade. I am wondering if there is any
> > > recommendation on a sensible value to use for the matchLimit
> > > for HepProgramBuilder?
> > >
> > > The default value is Integer.MAX_VALUE which I don't think will ever be
> > > possible due to stack / memory limitations.
> > >
> > > - Maytas
> > >
> >
>
Re: Recommendation for matchLimit for HepProgram
Posted by Maytas Monsereenusorn <ma...@gmail.com>.
Hi Chunwei,
Thank you for your reply. Can you share a little bit more about your
particular system?
For context, our system is Apache Druid.
Best Regards,
Maytas
On Tue, Jun 30, 2020 at 7:57 PM Chunwei Lei <ch...@gmail.com> wrote:
> Hi, Maytas.
>
> In my system, we set the matchLimit to 1200, which can cover most cases.
>
> Best,
> Chunwei
>
>
> On Tue, Jun 30, 2020 at 4:30 PM Maytas Monsereenusorn <ma...@apache.org>
> wrote:
>
> > Hi,
> >
> > I am encountering a problem in Calcite that causes my query to result in
> an
> > StackOverflow exception. A simple query that can reproduces this issue is
> > something liken SELECT ARRAY ['Hello', NULL]
> >
> > I believe this is the cause (and also the fix)
> > https://issues.apache.org/jira/browse/CALCITE-3845
> > However, we are still currently using Calcite 1.21.0 which does not have
> > this fix (fix above is in 1.23.0). I found that a workaround that can
> work
> > without upgrading Calcite is to set currentProgram.matchLimit. I was able
> > to confirm that the workaround (setting a limit on matchLimit) fixes this
> > issue without needing to upgrade. I am wondering if there is any
> > recommendation on a sensible value to use for the matchLimit
> > for HepProgramBuilder?
> >
> > The default value is Integer.MAX_VALUE which I don't think will ever be
> > possible due to stack / memory limitations.
> >
> > - Maytas
> >
>