You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@calcite.apache.org by Jeremy Dyer <jd...@gmail.com> on 2022/01/05 14:30:22 UTC
StackOverflowError in HepPlanner
Hello,
I am experiencing a java.lang.StackOverFlowError in the HepPlanner [1]. The
error is occurring in a unit test that had been working for some time until
I recently upgraded to Calcite 1.29.0. It is very possible I am doing
something wrong in my planner configuration but I find it odd that it
worked before the upgrade? The query is this [2]. Does anyone have ideas
how I might resolve this issue?
- Jeremy Dyer
[1]
https://github.com/apache/calcite/blob/cbfe0609edcc4a843d71497f159e3687a834119e/core/src/main/java/org/apache/calcite/plan/hep/HepPlanner.java#L391
[2]
SELECT
(CASE WHEN a = 3 THEN 1 END) AS "S1",
(CASE WHEN a > 0 THEN a ELSE 1 END) AS "S2",
(CASE WHEN a = 4 THEN 3 ELSE a + 1 END) AS "S3",
(CASE WHEN a = 3 THEN 1 WHEN a > 0 THEN 2 ELSE a END) AS "S4",
CASE
WHEN (a >= 1 AND a < 2) OR (a > 2) THEN CAST('in-between' AS VARCHAR) ELSE
CAST('out-of-range' AS VARCHAR)
END AS "S5",
CASE
WHEN (a < 2) OR (3 < a AND a < 4) THEN 42 ELSE 47
END AS "S6",
CASE WHEN (1 < a AND a <= 4) THEN 1 ELSE 0 END AS "S7"
FROM df
Re: StackOverflowError in HepPlanner
Posted by Stamatis Zampetakis <za...@gmail.com>.
Hi Jeremy,
Please log a JIRA case and attach the stacktrace. Apart from the query it
would help if you can provide the reproducer as a unit test in Calcite
(e.g., in RelOptRulesTest [1]).
Certain combinations of rules can lead to infinite rule matching.
Best,
Stamatis
[1]
https://github.com/apache/calcite/blob/ea4a5f36085553d427a339270a6e0b9b844f2364/core/src/test/java/org/apache/calcite/test/RelOptRulesTest.java
On Wed, Jan 5, 2022 at 4:57 PM Justin Swett <js...@google.com.invalid>
wrote:
> Does it only happen with the full SQL statement or can you repro with
> just one of those columns? S5, S6 and S7 seem like the fields that
> might be problematic.
>
> On Wed, Jan 5, 2022 at 6:31 AM Jeremy Dyer <jd...@gmail.com> wrote:
> >
> > Hello,
> >
> > I am experiencing a java.lang.StackOverFlowError in the HepPlanner [1].
> The
> > error is occurring in a unit test that had been working for some time
> until
> > I recently upgraded to Calcite 1.29.0. It is very possible I am doing
> > something wrong in my planner configuration but I find it odd that it
> > worked before the upgrade? The query is this [2]. Does anyone have ideas
> > how I might resolve this issue?
> >
> > - Jeremy Dyer
> >
> > [1]
> >
> https://github.com/apache/calcite/blob/cbfe0609edcc4a843d71497f159e3687a834119e/core/src/main/java/org/apache/calcite/plan/hep/HepPlanner.java#L391
> >
> > [2]
> > SELECT
> > (CASE WHEN a = 3 THEN 1 END) AS "S1",
> > (CASE WHEN a > 0 THEN a ELSE 1 END) AS "S2",
> > (CASE WHEN a = 4 THEN 3 ELSE a + 1 END) AS "S3",
> > (CASE WHEN a = 3 THEN 1 WHEN a > 0 THEN 2 ELSE a END) AS "S4",
> > CASE
> > WHEN (a >= 1 AND a < 2) OR (a > 2) THEN CAST('in-between' AS VARCHAR)
> ELSE
> > CAST('out-of-range' AS VARCHAR)
> > END AS "S5",
> > CASE
> > WHEN (a < 2) OR (3 < a AND a < 4) THEN 42 ELSE 47
> > END AS "S6",
> > CASE WHEN (1 < a AND a <= 4) THEN 1 ELSE 0 END AS "S7"
> > FROM df
>
Re: StackOverflowError in HepPlanner
Posted by Jacques Nadeau <ja...@apache.org>.
Stackoverflow typically means one of two things: infinite planning or very
deep trees being created that go beyond stack height. It's important to
remember that this is a relatively arbitrary magic threshold so small
non-functional changes can potentially influence whether you hit this
condition or not. It doesn't necessarily guarantee that there is something
functionally wrong, just that the call depth is very deep. I've seen it
when using sub-optimal valid patterns with very large queries. That being
said, given the relative simplicity of the query, I wouldn't expect a
valid deep tree.
I've definitely experienced both infinite planning and very deep planning
trees when mixing rules that can reorder inner expressions of rex trees.
These rules can create reordering projects above transformed nodes and then
fire again and again. I suggest you start removing rules in your case to
get a minimal reproduction. In most cases, two or three interacting rules
can cause this behavior with more complex Rex trees.
On Wed, Jan 5, 2022 at 8:38 AM Alessandro Solimando <
alessandro.solimando@gmail.com> wrote:
> Hi Jeremy,
> if you manage to repro as a unit test in Calcite (as Stamatis suggested)
> you can "git bisect" on that specific test to find the right commit.
>
> Best regards,
> Alessandro
>
> On Wed, 5 Jan 2022 at 17:12, Jeremy Dyer <jd...@gmail.com> wrote:
>
> > I just tried removing the "S7" CASE WHEN portion and now it works. Why is
> > that however? Also just for version understanding my previous version was
> > Calcite 1.26.0 and this begins to occur at 1.27.0 so I would assume that
> > something in [1] diff would be the reason I just don't know where to
> look?
> >
> > [1]
> >
> https://github.com/apache/calcite/compare/calcite-1.26.0...calcite-1.27.0
> >
> > On Wed, Jan 5, 2022 at 10:57 AM Justin Swett <js...@google.com.invalid>
> > wrote:
> >
> > > Does it only happen with the full SQL statement or can you repro with
> > > just one of those columns? S5, S6 and S7 seem like the fields that
> > > might be problematic.
> > >
> > > On Wed, Jan 5, 2022 at 6:31 AM Jeremy Dyer <jd...@gmail.com> wrote:
> > > >
> > > > Hello,
> > > >
> > > > I am experiencing a java.lang.StackOverFlowError in the HepPlanner
> [1].
> > > The
> > > > error is occurring in a unit test that had been working for some time
> > > until
> > > > I recently upgraded to Calcite 1.29.0. It is very possible I am doing
> > > > something wrong in my planner configuration but I find it odd that it
> > > > worked before the upgrade? The query is this [2]. Does anyone have
> > ideas
> > > > how I might resolve this issue?
> > > >
> > > > - Jeremy Dyer
> > > >
> > > > [1]
> > > >
> > >
> >
> https://github.com/apache/calcite/blob/cbfe0609edcc4a843d71497f159e3687a834119e/core/src/main/java/org/apache/calcite/plan/hep/HepPlanner.java#L391
> > > >
> > > > [2]
> > > > SELECT
> > > > (CASE WHEN a = 3 THEN 1 END) AS "S1",
> > > > (CASE WHEN a > 0 THEN a ELSE 1 END) AS "S2",
> > > > (CASE WHEN a = 4 THEN 3 ELSE a + 1 END) AS "S3",
> > > > (CASE WHEN a = 3 THEN 1 WHEN a > 0 THEN 2 ELSE a END) AS "S4",
> > > > CASE
> > > > WHEN (a >= 1 AND a < 2) OR (a > 2) THEN CAST('in-between' AS VARCHAR)
> > > ELSE
> > > > CAST('out-of-range' AS VARCHAR)
> > > > END AS "S5",
> > > > CASE
> > > > WHEN (a < 2) OR (3 < a AND a < 4) THEN 42 ELSE 47
> > > > END AS "S6",
> > > > CASE WHEN (1 < a AND a <= 4) THEN 1 ELSE 0 END AS "S7"
> > > > FROM df
> > >
> >
>
Re: StackOverflowError in HepPlanner
Posted by Alessandro Solimando <al...@gmail.com>.
Hi Jeremy,
if you manage to repro as a unit test in Calcite (as Stamatis suggested)
you can "git bisect" on that specific test to find the right commit.
Best regards,
Alessandro
On Wed, 5 Jan 2022 at 17:12, Jeremy Dyer <jd...@gmail.com> wrote:
> I just tried removing the "S7" CASE WHEN portion and now it works. Why is
> that however? Also just for version understanding my previous version was
> Calcite 1.26.0 and this begins to occur at 1.27.0 so I would assume that
> something in [1] diff would be the reason I just don't know where to look?
>
> [1]
> https://github.com/apache/calcite/compare/calcite-1.26.0...calcite-1.27.0
>
> On Wed, Jan 5, 2022 at 10:57 AM Justin Swett <js...@google.com.invalid>
> wrote:
>
> > Does it only happen with the full SQL statement or can you repro with
> > just one of those columns? S5, S6 and S7 seem like the fields that
> > might be problematic.
> >
> > On Wed, Jan 5, 2022 at 6:31 AM Jeremy Dyer <jd...@gmail.com> wrote:
> > >
> > > Hello,
> > >
> > > I am experiencing a java.lang.StackOverFlowError in the HepPlanner [1].
> > The
> > > error is occurring in a unit test that had been working for some time
> > until
> > > I recently upgraded to Calcite 1.29.0. It is very possible I am doing
> > > something wrong in my planner configuration but I find it odd that it
> > > worked before the upgrade? The query is this [2]. Does anyone have
> ideas
> > > how I might resolve this issue?
> > >
> > > - Jeremy Dyer
> > >
> > > [1]
> > >
> >
> https://github.com/apache/calcite/blob/cbfe0609edcc4a843d71497f159e3687a834119e/core/src/main/java/org/apache/calcite/plan/hep/HepPlanner.java#L391
> > >
> > > [2]
> > > SELECT
> > > (CASE WHEN a = 3 THEN 1 END) AS "S1",
> > > (CASE WHEN a > 0 THEN a ELSE 1 END) AS "S2",
> > > (CASE WHEN a = 4 THEN 3 ELSE a + 1 END) AS "S3",
> > > (CASE WHEN a = 3 THEN 1 WHEN a > 0 THEN 2 ELSE a END) AS "S4",
> > > CASE
> > > WHEN (a >= 1 AND a < 2) OR (a > 2) THEN CAST('in-between' AS VARCHAR)
> > ELSE
> > > CAST('out-of-range' AS VARCHAR)
> > > END AS "S5",
> > > CASE
> > > WHEN (a < 2) OR (3 < a AND a < 4) THEN 42 ELSE 47
> > > END AS "S6",
> > > CASE WHEN (1 < a AND a <= 4) THEN 1 ELSE 0 END AS "S7"
> > > FROM df
> >
>
Re: StackOverflowError in HepPlanner
Posted by Jeremy Dyer <jd...@gmail.com>.
I just tried removing the "S7" CASE WHEN portion and now it works. Why is
that however? Also just for version understanding my previous version was
Calcite 1.26.0 and this begins to occur at 1.27.0 so I would assume that
something in [1] diff would be the reason I just don't know where to look?
[1]
https://github.com/apache/calcite/compare/calcite-1.26.0...calcite-1.27.0
On Wed, Jan 5, 2022 at 10:57 AM Justin Swett <js...@google.com.invalid>
wrote:
> Does it only happen with the full SQL statement or can you repro with
> just one of those columns? S5, S6 and S7 seem like the fields that
> might be problematic.
>
> On Wed, Jan 5, 2022 at 6:31 AM Jeremy Dyer <jd...@gmail.com> wrote:
> >
> > Hello,
> >
> > I am experiencing a java.lang.StackOverFlowError in the HepPlanner [1].
> The
> > error is occurring in a unit test that had been working for some time
> until
> > I recently upgraded to Calcite 1.29.0. It is very possible I am doing
> > something wrong in my planner configuration but I find it odd that it
> > worked before the upgrade? The query is this [2]. Does anyone have ideas
> > how I might resolve this issue?
> >
> > - Jeremy Dyer
> >
> > [1]
> >
> https://github.com/apache/calcite/blob/cbfe0609edcc4a843d71497f159e3687a834119e/core/src/main/java/org/apache/calcite/plan/hep/HepPlanner.java#L391
> >
> > [2]
> > SELECT
> > (CASE WHEN a = 3 THEN 1 END) AS "S1",
> > (CASE WHEN a > 0 THEN a ELSE 1 END) AS "S2",
> > (CASE WHEN a = 4 THEN 3 ELSE a + 1 END) AS "S3",
> > (CASE WHEN a = 3 THEN 1 WHEN a > 0 THEN 2 ELSE a END) AS "S4",
> > CASE
> > WHEN (a >= 1 AND a < 2) OR (a > 2) THEN CAST('in-between' AS VARCHAR)
> ELSE
> > CAST('out-of-range' AS VARCHAR)
> > END AS "S5",
> > CASE
> > WHEN (a < 2) OR (3 < a AND a < 4) THEN 42 ELSE 47
> > END AS "S6",
> > CASE WHEN (1 < a AND a <= 4) THEN 1 ELSE 0 END AS "S7"
> > FROM df
>
Re: StackOverflowError in HepPlanner
Posted by Justin Swett <js...@google.com.INVALID>.
Does it only happen with the full SQL statement or can you repro with
just one of those columns? S5, S6 and S7 seem like the fields that
might be problematic.
On Wed, Jan 5, 2022 at 6:31 AM Jeremy Dyer <jd...@gmail.com> wrote:
>
> Hello,
>
> I am experiencing a java.lang.StackOverFlowError in the HepPlanner [1]. The
> error is occurring in a unit test that had been working for some time until
> I recently upgraded to Calcite 1.29.0. It is very possible I am doing
> something wrong in my planner configuration but I find it odd that it
> worked before the upgrade? The query is this [2]. Does anyone have ideas
> how I might resolve this issue?
>
> - Jeremy Dyer
>
> [1]
> https://github.com/apache/calcite/blob/cbfe0609edcc4a843d71497f159e3687a834119e/core/src/main/java/org/apache/calcite/plan/hep/HepPlanner.java#L391
>
> [2]
> SELECT
> (CASE WHEN a = 3 THEN 1 END) AS "S1",
> (CASE WHEN a > 0 THEN a ELSE 1 END) AS "S2",
> (CASE WHEN a = 4 THEN 3 ELSE a + 1 END) AS "S3",
> (CASE WHEN a = 3 THEN 1 WHEN a > 0 THEN 2 ELSE a END) AS "S4",
> CASE
> WHEN (a >= 1 AND a < 2) OR (a > 2) THEN CAST('in-between' AS VARCHAR) ELSE
> CAST('out-of-range' AS VARCHAR)
> END AS "S5",
> CASE
> WHEN (a < 2) OR (3 < a AND a < 4) THEN 42 ELSE 47
> END AS "S6",
> CASE WHEN (1 < a AND a <= 4) THEN 1 ELSE 0 END AS "S7"
> FROM df