You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@impala.apache.org by "Steve Carlin (Code Review)" <ge...@cloudera.org> on 2026/08/17 12:16:02 UTC

[Impala-ASF-CR] IMPALA-15266: Fix flaky test due to slow Calcite first query.

Steve Carlin has uploaded this change for review. ( http://gerrit.cloudera.org:8080/24699


Change subject: IMPALA-15266: Fix flaky test due to slow Calcite first query.
......................................................................

IMPALA-15266: Fix flaky test due to slow Calcite first query.

The test test_invalidation_metrics in the
test_automatic_invalidation.py file is flaky for the Calcite planner.
On the first query, some static variables are being initialized.

Some of these static initializations are not done at startup time.
One of special note here is the RelMetadataQuery initializations.
Calcite generates java code on the first call to the RelMetadataQuery
functions. The JaninoRelMetadataProvider.revise() method is now called
for these RelMetadataQuery queries to compile the code before the
first query is run.

Also of note: The test_invalidation_metrics calls a "select 1" before
calling any queries. There is still some initialization of static
variables done within Calcite, but the quick "select 1" query gets
compiled way below the 1s threshold and subsequent queries clear the
1s barrier with ease as well.

Change-Id: I90b167055cb8d8b0045344085083160f1d98f5a4
---
M java/calcite-planner/src/main/java/org/apache/impala/calcite/service/CalciteCompilerFactory.java
M java/calcite-planner/src/main/java/org/apache/impala/calcite/service/CalciteQueryParser.java
M java/calcite-planner/src/main/java/org/apache/impala/calcite/service/CalciteRelNodeConverter.java
M tests/custom_cluster/test_automatic_invalidation.py
4 files changed, 44 insertions(+), 10 deletions(-)



  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/99/24699/2
-- 
To view, visit http://gerrit.cloudera.org:8080/24699
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: I90b167055cb8d8b0045344085083160f1d98f5a4
Gerrit-Change-Number: 24699
Gerrit-PatchSet: 2
Gerrit-Owner: Steve Carlin <sc...@cloudera.com>

[Impala-ASF-CR] IMPALA-15266: Fix flaky test due to slow Calcite first query.

Posted by "Steve Carlin (Code Review)" <ge...@cloudera.org>.
Steve Carlin has posted comments on this change. ( http://gerrit.cloudera.org:8080/24699 )

Change subject: IMPALA-15266: Fix flaky test due to slow Calcite first query.
......................................................................


Patch Set 5:

(3 comments)

http://gerrit.cloudera.org:8080/#/c/24699/5/java/calcite-planner/src/main/java/org/apache/impala/calcite/service/CalciteCompilerFactory.java
File java/calcite-planner/src/main/java/org/apache/impala/calcite/service/CalciteCompilerFactory.java:

http://gerrit.cloudera.org:8080/#/c/24699/5/java/calcite-planner/src/main/java/org/apache/impala/calcite/service/CalciteCompilerFactory.java@55
PS5, Line 55:     ImpalaRelMetadataProvider.DEFAULT.revise(
> Small thing about where this runs: Frontend's static init does CALCITE_COMP
I think it's fine.

When the project first started, we had this initial thought; that Calcite would not necessarily be on every server. This is  one of the reasons it is in its own jar file.

At some point, we made the decision to include it. 

But my thought here is that if we are loading the Calcite jar file, then it's fair game to initialize all variables as well.  Indeed, we are already initializing the ones in the Impala Calcite jar file.

The Janino compilation, as far as I can tell, is generating code for the metadata queries.  This is code that theoretically should be there anyway. 

My guess as to why it is compiled code instead of actual code is that they were worried about code bloat?  In earlier versions, I think they generated code for each RelNode for each MetadataQuery, and maybe some systems had a ton of different RelNodes?  Just a guess here.  ButI don't think it's a huge hit memory-wise with our limited MetadataQueries (and even our limited RelNodes, but I don't think that matters anymore).

So I'm not too worried about this being done for all servers, even if they don't use Calcite.


http://gerrit.cloudera.org:8080/#/c/24699/5/java/calcite-planner/src/main/java/org/apache/impala/calcite/service/CalciteCompilerFactory.java@76
PS5, Line 76:     RelMetadataQuery.THREAD_PROVIDERS.set(
> Do we still need this one? THREAD_PROVIDERS is a ThreadLocal, so it covers 
Removed the code


http://gerrit.cloudera.org:8080/#/c/24699/5/tests/custom_cluster/test_automatic_invalidation.py
File tests/custom_cluster/test_automatic_invalidation.py:

http://gerrit.cloudera.org:8080/#/c/24699/5/tests/custom_cluster/test_automatic_invalidation.py@256
PS5, Line 256:     self.execute_query("select 1")
> Out of curiosity: with the static init happening at startup now, do you kno
Filed IMPALA-15283 as a catch-all to find these.



-- 
To view, visit http://gerrit.cloudera.org:8080/24699
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I90b167055cb8d8b0045344085083160f1d98f5a4
Gerrit-Change-Number: 24699
Gerrit-PatchSet: 5
Gerrit-Owner: Steve Carlin <sc...@cloudera.com>
Gerrit-Reviewer: Aleksandr Efimov <dz...@gmail.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Arnab Karmakar <ar...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Steve Carlin <sc...@cloudera.com>
Gerrit-Comment-Date: Sun, 23 Aug 2026 15:11:10 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-15266: Fix flaky test due to slow Calcite first query.

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24699 )

Change subject: IMPALA-15266: Fix flaky test due to slow Calcite first query.
......................................................................


Patch Set 2:

Build Failed 

https://jenkins.impala.io/job/gerrit-code-review-checks/23635/ : Initial code review checks failed. See linked job for details on the failure.


-- 
To view, visit http://gerrit.cloudera.org:8080/24699
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I90b167055cb8d8b0045344085083160f1d98f5a4
Gerrit-Change-Number: 24699
Gerrit-PatchSet: 2
Gerrit-Owner: Steve Carlin <sc...@cloudera.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Arnab Karmakar <ar...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Comment-Date: Mon, 17 Aug 2026 12:37:50 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-15266: Fix flaky test due to slow Calcite first query.

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24699 )

Change subject: IMPALA-15266: Fix flaky test due to slow Calcite first query.
......................................................................


Patch Set 3:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/23637/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/24699
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I90b167055cb8d8b0045344085083160f1d98f5a4
Gerrit-Change-Number: 24699
Gerrit-PatchSet: 3
Gerrit-Owner: Steve Carlin <sc...@cloudera.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Arnab Karmakar <ar...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Comment-Date: Mon, 17 Aug 2026 12:48:25 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-15266: Fix flaky test due to slow Calcite first query.

Posted by "Steve Carlin (Code Review)" <ge...@cloudera.org>.
Hello Aman Sinha, Arnab Karmakar, Aleksandr Efimov, Joe McDonnell, Michael Smith, Impala Public Jenkins, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/24699

to look at the new patch set (#7).

Change subject: IMPALA-15266: Fix flaky test due to slow Calcite first query.
......................................................................

IMPALA-15266: Fix flaky test due to slow Calcite first query.

The test test_invalidation_metrics in the
test_automatic_invalidation.py file is flaky for the Calcite planner.
On the first query, some static variables are being initialized.

Some of these static initializations are not done at startup time.
One of special note here is the RelMetadataQuery initializations.
Calcite generates java code on the first call to the RelMetadataQuery
functions. The JaninoRelMetadataProvider.revise() method is now called
for these RelMetadataQuery queries to compile the code before the
first query is run.

Also of note: The test_invalidation_metrics calls a "select 1" before
calling any queries. There is still some initialization of static
variables done within Calcite, but the quick "select 1" query gets
compiled way below the 1s threshold and subsequent queries clear the
1s barrier with ease as well.

Change-Id: I90b167055cb8d8b0045344085083160f1d98f5a4
---
M java/calcite-planner/src/main/java/org/apache/impala/calcite/service/CalciteCompilerFactory.java
M java/calcite-planner/src/main/java/org/apache/impala/calcite/service/CalciteQueryParser.java
M java/calcite-planner/src/main/java/org/apache/impala/calcite/service/CalciteRelNodeConverter.java
M tests/custom_cluster/test_automatic_invalidation.py
4 files changed, 52 insertions(+), 13 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/99/24699/7
-- 
To view, visit http://gerrit.cloudera.org:8080/24699
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I90b167055cb8d8b0045344085083160f1d98f5a4
Gerrit-Change-Number: 24699
Gerrit-PatchSet: 7
Gerrit-Owner: Steve Carlin <sc...@cloudera.com>
Gerrit-Reviewer: Aleksandr Efimov <dz...@gmail.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Arnab Karmakar <ar...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>

[Impala-ASF-CR] IMPALA-15266: Fix flaky test due to slow Calcite first query.

Posted by "Steve Carlin (Code Review)" <ge...@cloudera.org>.
Steve Carlin has posted comments on this change. ( http://gerrit.cloudera.org:8080/24699 )

Change subject: IMPALA-15266: Fix flaky test due to slow Calcite first query.
......................................................................


Patch Set 5:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/24699/5/java/calcite-planner/src/main/java/org/apache/impala/calcite/service/CalciteCompilerFactory.java
File java/calcite-planner/src/main/java/org/apache/impala/calcite/service/CalciteCompilerFactory.java:

http://gerrit.cloudera.org:8080/#/c/24699/5/java/calcite-planner/src/main/java/org/apache/impala/calcite/service/CalciteCompilerFactory.java@55
PS5, Line 55:     ImpalaRelMetadataProvider.DEFAULT.revise(
> Memory-wise I'm convinced.
Timewise it was maybe half a second.  Maybe not even.

It's not a lot.  But it was enough to cause problems with that one test.  So I don't think it's worth putting it in a background thread.



-- 
To view, visit http://gerrit.cloudera.org:8080/24699
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I90b167055cb8d8b0045344085083160f1d98f5a4
Gerrit-Change-Number: 24699
Gerrit-PatchSet: 5
Gerrit-Owner: Steve Carlin <sc...@cloudera.com>
Gerrit-Reviewer: Aleksandr Efimov <dz...@gmail.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Arnab Karmakar <ar...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Steve Carlin <sc...@cloudera.com>
Gerrit-Comment-Date: Sun, 23 Aug 2026 18:30:18 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-15266: Fix flaky test due to slow Calcite first query.

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24699 )

Change subject: IMPALA-15266: Fix flaky test due to slow Calcite first query.
......................................................................


Patch Set 6:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/23725/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/24699
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I90b167055cb8d8b0045344085083160f1d98f5a4
Gerrit-Change-Number: 24699
Gerrit-PatchSet: 6
Gerrit-Owner: Steve Carlin <sc...@cloudera.com>
Gerrit-Reviewer: Aleksandr Efimov <dz...@gmail.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Arnab Karmakar <ar...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Steve Carlin <sc...@cloudera.com>
Gerrit-Comment-Date: Sun, 23 Aug 2026 15:22:23 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-15266: Fix flaky test due to slow Calcite first query.

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24699 )

Change subject: IMPALA-15266: Fix flaky test due to slow Calcite first query.
......................................................................


Patch Set 7:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/23726/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/24699
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I90b167055cb8d8b0045344085083160f1d98f5a4
Gerrit-Change-Number: 24699
Gerrit-PatchSet: 7
Gerrit-Owner: Steve Carlin <sc...@cloudera.com>
Gerrit-Reviewer: Aleksandr Efimov <dz...@gmail.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Arnab Karmakar <ar...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Steve Carlin <sc...@cloudera.com>
Gerrit-Comment-Date: Sun, 23 Aug 2026 15:32:13 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-15266: Fix flaky test due to slow Calcite first query.

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24699 )

Change subject: IMPALA-15266: Fix flaky test due to slow Calcite first query.
......................................................................


Patch Set 4:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/23638/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/24699
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I90b167055cb8d8b0045344085083160f1d98f5a4
Gerrit-Change-Number: 24699
Gerrit-PatchSet: 4
Gerrit-Owner: Steve Carlin <sc...@cloudera.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Arnab Karmakar <ar...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Comment-Date: Mon, 17 Aug 2026 13:32:12 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-15266: Fix flaky test due to slow Calcite first query.

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24699 )

Change subject: IMPALA-15266: Fix flaky test due to slow Calcite first query.
......................................................................


Patch Set 3:

(5 comments)

gerrit-auto-critic failed. You can reproduce it locally using command:

  python3 bin/jenkins/critique-gerrit-review.py --dryrun

To run it, you might need a virtual env with Python3's venv installed.

http://gerrit.cloudera.org:8080/#/c/24699/3/java/calcite-planner/src/main/java/org/apache/impala/calcite/service/CalciteCompilerFactory.java
File java/calcite-planner/src/main/java/org/apache/impala/calcite/service/CalciteCompilerFactory.java:

http://gerrit.cloudera.org:8080/#/c/24699/3/java/calcite-planner/src/main/java/org/apache/impala/calcite/service/CalciteCompilerFactory.java@40
PS3, Line 40: import org.apache.calcite.sql2rel.SqlToRelConverter;
unused import: org.apache.calcite.sql2rel.SqlToRelConverter


http://gerrit.cloudera.org:8080/#/c/24699/3/java/calcite-planner/src/main/java/org/apache/impala/calcite/service/CalciteCompilerFactory.java@57
PS3, Line 57:     ImpalaRelMetadataProvider.DEFAULT.revise(BuiltInMetadata.ExplainVisibility.Handler.class);
line too long (94 > 90)


http://gerrit.cloudera.org:8080/#/c/24699/3/java/calcite-planner/src/main/java/org/apache/impala/calcite/service/CalciteCompilerFactory.java@61
PS3, Line 61:     ImpalaRelMetadataProvider.DEFAULT.revise(BuiltInMetadata.DistinctRowCount.Handler.class);
line too long (93 > 90)


http://gerrit.cloudera.org:8080/#/c/24699/3/java/calcite-planner/src/main/java/org/apache/impala/calcite/service/CalciteCompilerFactory.java@64
PS3, Line 64:     ImpalaRelMetadataProvider.DEFAULT.revise(BuiltInMetadata.CumulativeCost.Handler.class);
line too long (91 > 90)


http://gerrit.cloudera.org:8080/#/c/24699/3/java/calcite-planner/src/main/java/org/apache/impala/calcite/service/CalciteCompilerFactory.java@65
PS3, Line 65:     ImpalaRelMetadataProvider.DEFAULT.revise(BuiltInMetadata.NonCumulativeCost.Handler.class);
line too long (94 > 90)



-- 
To view, visit http://gerrit.cloudera.org:8080/24699
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I90b167055cb8d8b0045344085083160f1d98f5a4
Gerrit-Change-Number: 24699
Gerrit-PatchSet: 3
Gerrit-Owner: Steve Carlin <sc...@cloudera.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Arnab Karmakar <ar...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Comment-Date: Mon, 17 Aug 2026 12:28:55 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-15266: Fix flaky test due to slow Calcite first query.

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24699 )

Change subject: IMPALA-15266: Fix flaky test due to slow Calcite first query.
......................................................................


Patch Set 5:

Build Failed 

https://jenkins.impala.io/job/gerrit-code-review-checks/23640/ : Initial code review checks failed. See linked job for details on the failure.


-- 
To view, visit http://gerrit.cloudera.org:8080/24699
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I90b167055cb8d8b0045344085083160f1d98f5a4
Gerrit-Change-Number: 24699
Gerrit-PatchSet: 5
Gerrit-Owner: Steve Carlin <sc...@cloudera.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Arnab Karmakar <ar...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Comment-Date: Mon, 17 Aug 2026 14:33:51 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-15266: Fix flaky test due to slow Calcite first query.

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24699 )

Change subject: IMPALA-15266: Fix flaky test due to slow Calcite first query.
......................................................................


Patch Set 6:

(3 comments)

gerrit-auto-critic failed. You can reproduce it locally using command:

  python3 bin/jenkins/critique-gerrit-review.py --dryrun

To run it, you might need a virtual env with Python3's venv installed.

http://gerrit.cloudera.org:8080/#/c/24699/6/java/calcite-planner/src/main/java/org/apache/impala/calcite/service/CalciteCompilerFactory.java
File java/calcite-planner/src/main/java/org/apache/impala/calcite/service/CalciteCompilerFactory.java:

http://gerrit.cloudera.org:8080/#/c/24699/6/java/calcite-planner/src/main/java/org/apache/impala/calcite/service/CalciteCompilerFactory.java@21
PS6, Line 21: import org.apache.calcite.rel.metadata.DefaultRelMetadataProvider;
unused import: org.apache.calcite.rel.metadata.DefaultRelMetadataProvider


http://gerrit.cloudera.org:8080/#/c/24699/6/java/calcite-planner/src/main/java/org/apache/impala/calcite/service/CalciteCompilerFactory.java@22
PS6, Line 22: import org.apache.calcite.rel.metadata.JaninoRelMetadataProvider;
unused import: org.apache.calcite.rel.metadata.JaninoRelMetadataProvider


http://gerrit.cloudera.org:8080/#/c/24699/6/java/calcite-planner/src/main/java/org/apache/impala/calcite/service/CalciteCompilerFactory.java@23
PS6, Line 23: import org.apache.calcite.rel.metadata.RelMetadataQuery;
unused import: org.apache.calcite.rel.metadata.RelMetadataQuery



-- 
To view, visit http://gerrit.cloudera.org:8080/24699
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I90b167055cb8d8b0045344085083160f1d98f5a4
Gerrit-Change-Number: 24699
Gerrit-PatchSet: 6
Gerrit-Owner: Steve Carlin <sc...@cloudera.com>
Gerrit-Reviewer: Aleksandr Efimov <dz...@gmail.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Arnab Karmakar <ar...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Comment-Date: Sun, 23 Aug 2026 15:02:05 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-15266: Fix flaky test due to slow Calcite first query.

Posted by "Aleksandr Efimov (Code Review)" <ge...@cloudera.org>.
Aleksandr Efimov has posted comments on this change. ( http://gerrit.cloudera.org:8080/24699 )

Change subject: IMPALA-15266: Fix flaky test due to slow Calcite first query.
......................................................................


Patch Set 5:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/24699/5/java/calcite-planner/src/main/java/org/apache/impala/calcite/service/CalciteCompilerFactory.java
File java/calcite-planner/src/main/java/org/apache/impala/calcite/service/CalciteCompilerFactory.java:

http://gerrit.cloudera.org:8080/#/c/24699/5/java/calcite-planner/src/main/java/org/apache/impala/calcite/service/CalciteCompilerFactory.java@55
PS5, Line 55:     ImpalaRelMetadataProvider.DEFAULT.revise(
> Timewise it was maybe half a second.  Maybe not even.
Thanks



-- 
To view, visit http://gerrit.cloudera.org:8080/24699
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I90b167055cb8d8b0045344085083160f1d98f5a4
Gerrit-Change-Number: 24699
Gerrit-PatchSet: 5
Gerrit-Owner: Steve Carlin <sc...@cloudera.com>
Gerrit-Reviewer: Aleksandr Efimov <dz...@gmail.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Arnab Karmakar <ar...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Steve Carlin <sc...@cloudera.com>
Gerrit-Comment-Date: Sun, 23 Aug 2026 18:43:44 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-15266: Fix flaky test due to slow Calcite first query.

Posted by "Aleksandr Efimov (Code Review)" <ge...@cloudera.org>.
Aleksandr Efimov has posted comments on this change. ( http://gerrit.cloudera.org:8080/24699 )

Change subject: IMPALA-15266: Fix flaky test due to slow Calcite first query.
......................................................................


Patch Set 5:

(3 comments)

http://gerrit.cloudera.org:8080/#/c/24699/5/java/calcite-planner/src/main/java/org/apache/impala/calcite/service/CalciteCompilerFactory.java
File java/calcite-planner/src/main/java/org/apache/impala/calcite/service/CalciteCompilerFactory.java:

http://gerrit.cloudera.org:8080/#/c/24699/5/java/calcite-planner/src/main/java/org/apache/impala/calcite/service/CalciteCompilerFactory.java@55
PS5, Line 55:     ImpalaRelMetadataProvider.DEFAULT.revise(
> I think it's fine.
Memory-wise I'm convinced.

What I still can't tell is the startup cost: after the move every impalad pays that compilation while starting, Calcite or not. Do you have a number from a restart? If it turns out to be a noticeable slice of that second, a background thread would give you both.


http://gerrit.cloudera.org:8080/#/c/24699/5/java/calcite-planner/src/main/java/org/apache/impala/calcite/service/CalciteCompilerFactory.java@76
PS5, Line 76:     RelMetadataQuery.THREAD_PROVIDERS.set(
> Removed the code
Checked on PS7 - every metadata query in the module comes from the cluster, and the provider is set in CalciteRelNodeConverter, so nothing is left on the thread-local. Thanks!


http://gerrit.cloudera.org:8080/#/c/24699/5/tests/custom_cluster/test_automatic_invalidation.py
File tests/custom_cluster/test_automatic_invalidation.py:

http://gerrit.cloudera.org:8080/#/c/24699/5/tests/custom_cluster/test_automatic_invalidation.py@256
PS5, Line 256:     self.execute_query("select 1")
> Filed IMPALA-15283 as a catch-all to find these.
Thanks for filing IMPALA-15283.



-- 
To view, visit http://gerrit.cloudera.org:8080/24699
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I90b167055cb8d8b0045344085083160f1d98f5a4
Gerrit-Change-Number: 24699
Gerrit-PatchSet: 5
Gerrit-Owner: Steve Carlin <sc...@cloudera.com>
Gerrit-Reviewer: Aleksandr Efimov <dz...@gmail.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Arnab Karmakar <ar...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Steve Carlin <sc...@cloudera.com>
Gerrit-Comment-Date: Sun, 23 Aug 2026 17:09:10 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-15266: Fix flaky test due to slow Calcite first query.

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24699 )

Change subject: IMPALA-15266: Fix flaky test due to slow Calcite first query.
......................................................................


Patch Set 2:

(5 comments)

gerrit-auto-critic failed. You can reproduce it locally using command:

  python3 bin/jenkins/critique-gerrit-review.py --dryrun

To run it, you might need a virtual env with Python3's venv installed.

http://gerrit.cloudera.org:8080/#/c/24699/2/java/calcite-planner/src/main/java/org/apache/impala/calcite/service/CalciteCompilerFactory.java
File java/calcite-planner/src/main/java/org/apache/impala/calcite/service/CalciteCompilerFactory.java:

http://gerrit.cloudera.org:8080/#/c/24699/2/java/calcite-planner/src/main/java/org/apache/impala/calcite/service/CalciteCompilerFactory.java@40
PS2, Line 40: import org.apache.calcite.sql2rel.SqlToRelConverter;
unused import: org.apache.calcite.sql2rel.SqlToRelConverter


http://gerrit.cloudera.org:8080/#/c/24699/2/java/calcite-planner/src/main/java/org/apache/impala/calcite/service/CalciteCompilerFactory.java@57
PS2, Line 57:     ImpalaRelMetadataProvider.DEFAULT.revise(BuiltInMetadata.ExplainVisibility.Handler.class);
line too long (94 > 90)


http://gerrit.cloudera.org:8080/#/c/24699/2/java/calcite-planner/src/main/java/org/apache/impala/calcite/service/CalciteCompilerFactory.java@61
PS2, Line 61:     ImpalaRelMetadataProvider.DEFAULT.revise(BuiltInMetadata.DistinctRowCount.Handler.class);
line too long (93 > 90)


http://gerrit.cloudera.org:8080/#/c/24699/2/java/calcite-planner/src/main/java/org/apache/impala/calcite/service/CalciteCompilerFactory.java@64
PS2, Line 64:     ImpalaRelMetadataProvider.DEFAULT.revise(BuiltInMetadata.CumulativeCost.Handler.class);
line too long (91 > 90)


http://gerrit.cloudera.org:8080/#/c/24699/2/java/calcite-planner/src/main/java/org/apache/impala/calcite/service/CalciteCompilerFactory.java@65
PS2, Line 65:     ImpalaRelMetadataProvider.DEFAULT.revise(BuiltInMetadata.NonCumulativeCost.Handler.class);
line too long (94 > 90)



-- 
To view, visit http://gerrit.cloudera.org:8080/24699
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I90b167055cb8d8b0045344085083160f1d98f5a4
Gerrit-Change-Number: 24699
Gerrit-PatchSet: 2
Gerrit-Owner: Steve Carlin <sc...@cloudera.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Arnab Karmakar <ar...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Comment-Date: Mon, 17 Aug 2026 12:18:15 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-15266: Fix flaky test due to slow Calcite first query.

Posted by "Steve Carlin (Code Review)" <ge...@cloudera.org>.
Hello Aman Sinha, Arnab Karmakar, Joe McDonnell, Michael Smith, Impala Public Jenkins, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/24699

to look at the new patch set (#4).

Change subject: IMPALA-15266: Fix flaky test due to slow Calcite first query.
......................................................................

IMPALA-15266: Fix flaky test due to slow Calcite first query.

The test test_invalidation_metrics in the
test_automatic_invalidation.py file is flaky for the Calcite planner.
On the first query, some static variables are being initialized.

Some of these static initializations are not done at startup time.
One of special note here is the RelMetadataQuery initializations.
Calcite generates java code on the first call to the RelMetadataQuery
functions. The JaninoRelMetadataProvider.revise() method is now called
for these RelMetadataQuery queries to compile the code before the
first query is run.

Also of note: The test_invalidation_metrics calls a "select 1" before
calling any queries. There is still some initialization of static
variables done within Calcite, but the quick "select 1" query gets
compiled way below the 1s threshold and subsequent queries clear the
1s barrier with ease as well.

Change-Id: I90b167055cb8d8b0045344085083160f1d98f5a4
---
M java/calcite-planner/src/main/java/org/apache/impala/calcite/service/CalciteCompilerFactory.java
M java/calcite-planner/src/main/java/org/apache/impala/calcite/service/CalciteQueryParser.java
M java/calcite-planner/src/main/java/org/apache/impala/calcite/service/CalciteRelNodeConverter.java
M tests/custom_cluster/test_automatic_invalidation.py
4 files changed, 57 insertions(+), 10 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/99/24699/4
-- 
To view, visit http://gerrit.cloudera.org:8080/24699
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I90b167055cb8d8b0045344085083160f1d98f5a4
Gerrit-Change-Number: 24699
Gerrit-PatchSet: 4
Gerrit-Owner: Steve Carlin <sc...@cloudera.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Arnab Karmakar <ar...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>

[Impala-ASF-CR] IMPALA-15266: Fix flaky test due to slow Calcite first query.

Posted by "Steve Carlin (Code Review)" <ge...@cloudera.org>.
Hello Aman Sinha, Arnab Karmakar, Joe McDonnell, Michael Smith, Impala Public Jenkins, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/24699

to look at the new patch set (#3).

Change subject: IMPALA-15266: Fix flaky test due to slow Calcite first query.
......................................................................

IMPALA-15266: Fix flaky test due to slow Calcite first query.

The test test_invalidation_metrics in the
test_automatic_invalidation.py file is flaky for the Calcite planner.
On the first query, some static variables are being initialized.

Some of these static initializations are not done at startup time.
One of special note here is the RelMetadataQuery initializations.
Calcite generates java code on the first call to the RelMetadataQuery
functions. The JaninoRelMetadataProvider.revise() method is now called
for these RelMetadataQuery queries to compile the code before the
first query is run.

Also of note: The test_invalidation_metrics calls a "select 1" before
calling any queries. There is still some initialization of static
variables done within Calcite, but the quick "select 1" query gets
compiled way below the 1s threshold and subsequent queries clear the
1s barrier with ease as well.

Change-Id: I90b167055cb8d8b0045344085083160f1d98f5a4
---
M java/calcite-planner/src/main/java/org/apache/impala/calcite/service/CalciteCompilerFactory.java
M java/calcite-planner/src/main/java/org/apache/impala/calcite/service/CalciteQueryParser.java
M java/calcite-planner/src/main/java/org/apache/impala/calcite/service/CalciteRelNodeConverter.java
M tests/custom_cluster/test_automatic_invalidation.py
4 files changed, 47 insertions(+), 10 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/99/24699/3
-- 
To view, visit http://gerrit.cloudera.org:8080/24699
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I90b167055cb8d8b0045344085083160f1d98f5a4
Gerrit-Change-Number: 24699
Gerrit-PatchSet: 3
Gerrit-Owner: Steve Carlin <sc...@cloudera.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Arnab Karmakar <ar...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>

[Impala-ASF-CR] IMPALA-15266: Fix flaky test due to slow Calcite first query.

Posted by "Aleksandr Efimov (Code Review)" <ge...@cloudera.org>.
Aleksandr Efimov has posted comments on this change. ( http://gerrit.cloudera.org:8080/24699 )

Change subject: IMPALA-15266: Fix flaky test due to slow Calcite first query.
......................................................................


Patch Set 5:

(3 comments)

A couple of questions about the init path, nothing blocking. On the red check on PS5: the pipeline failed while fetching a child job's console (wget exit 8) - the jobs I can still see (ubuntu-20.04-build-only, the ARM build, auto-critic) are green, and the clang-tidy/rat/python-check builds have rotated away, so it may just need a re-trigger.

http://gerrit.cloudera.org:8080/#/c/24699/5/java/calcite-planner/src/main/java/org/apache/impala/calcite/service/CalciteCompilerFactory.java
File java/calcite-planner/src/main/java/org/apache/impala/calcite/service/CalciteCompilerFactory.java:

http://gerrit.cloudera.org:8080/#/c/24699/5/java/calcite-planner/src/main/java/org/apache/impala/calcite/service/CalciteCompilerFactory.java@55
PS5, Line 55:     ImpalaRelMetadataProvider.DEFAULT.revise(
Small thing about where this runs: Frontend's static init does CALCITE_COMPILER_FACTORY = loadCalciteCompilerFactory(), so this block executes on every impalad start, including clusters that never switch the Calcite planner on. Do you have a feel for what the Janino compilation adds there? If it is more than noise, doing it on a background thread, or only when the planner is enabled, would leave the startup path as it is for everyone else.


http://gerrit.cloudera.org:8080/#/c/24699/5/java/calcite-planner/src/main/java/org/apache/impala/calcite/service/CalciteCompilerFactory.java@76
PS5, Line 76:     RelMetadataQuery.THREAD_PROVIDERS.set(
Do we still need this one? THREAD_PROVIDERS is a ThreadLocal, so it covers only the thread that loads the class, and each query thread gets its own value from RelOptCluster.create() and then from cluster_.setMetadataProvider(ImpalaRelMetadataProvider.DEFAULT) in the CalciteRelNodeConverter constructor - which is also what makes dropping the per-query set below safe. The revise() calls above are the part that warms Janino up, and this one uses DefaultRelMetadataProvider rather than the Impala provider, so it looks droppable to me.


http://gerrit.cloudera.org:8080/#/c/24699/5/tests/custom_cluster/test_automatic_invalidation.py
File tests/custom_cluster/test_automatic_invalidation.py:

http://gerrit.cloudera.org:8080/#/c/24699/5/tests/custom_cluster/test_automatic_invalidation.py@256
PS5, Line 256:     self.execute_query("select 1")
Out of curiosity: with the static init happening at startup now, do you know what is left initializing on the first query? You mention in the commit message that something still is, and if that is what keeps the first query near the 1s mark, the next test with a similar budget will need the same warm-up line - might be worth a JIRA if the profile shows an obvious remaining chunk.



-- 
To view, visit http://gerrit.cloudera.org:8080/24699
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I90b167055cb8d8b0045344085083160f1d98f5a4
Gerrit-Change-Number: 24699
Gerrit-PatchSet: 5
Gerrit-Owner: Steve Carlin <sc...@cloudera.com>
Gerrit-Reviewer: Aleksandr Efimov <dz...@gmail.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Arnab Karmakar <ar...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Comment-Date: Sun, 23 Aug 2026 13:59:26 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-15266: Fix flaky test due to slow Calcite first query.

Posted by "Steve Carlin (Code Review)" <ge...@cloudera.org>.
Hello Aman Sinha, Arnab Karmakar, Aleksandr Efimov, Joe McDonnell, Michael Smith, Impala Public Jenkins, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/24699

to look at the new patch set (#6).

Change subject: IMPALA-15266: Fix flaky test due to slow Calcite first query.
......................................................................

IMPALA-15266: Fix flaky test due to slow Calcite first query.

The test test_invalidation_metrics in the
test_automatic_invalidation.py file is flaky for the Calcite planner.
On the first query, some static variables are being initialized.

Some of these static initializations are not done at startup time.
One of special note here is the RelMetadataQuery initializations.
Calcite generates java code on the first call to the RelMetadataQuery
functions. The JaninoRelMetadataProvider.revise() method is now called
for these RelMetadataQuery queries to compile the code before the
first query is run.

Also of note: The test_invalidation_metrics calls a "select 1" before
calling any queries. There is still some initialization of static
variables done within Calcite, but the quick "select 1" query gets
compiled way below the 1s threshold and subsequent queries clear the
1s barrier with ease as well.

Change-Id: I90b167055cb8d8b0045344085083160f1d98f5a4
---
M java/calcite-planner/src/main/java/org/apache/impala/calcite/service/CalciteCompilerFactory.java
M java/calcite-planner/src/main/java/org/apache/impala/calcite/service/CalciteQueryParser.java
M java/calcite-planner/src/main/java/org/apache/impala/calcite/service/CalciteRelNodeConverter.java
M tests/custom_cluster/test_automatic_invalidation.py
4 files changed, 52 insertions(+), 10 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/99/24699/6
-- 
To view, visit http://gerrit.cloudera.org:8080/24699
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I90b167055cb8d8b0045344085083160f1d98f5a4
Gerrit-Change-Number: 24699
Gerrit-PatchSet: 6
Gerrit-Owner: Steve Carlin <sc...@cloudera.com>
Gerrit-Reviewer: Aleksandr Efimov <dz...@gmail.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Arnab Karmakar <ar...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>

[Impala-ASF-CR] IMPALA-15266: Fix flaky test due to slow Calcite first query.

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24699 )

Change subject: IMPALA-15266: Fix flaky test due to slow Calcite first query.
......................................................................


Patch Set 4:

(1 comment)

gerrit-auto-critic failed. You can reproduce it locally using command:

  python3 bin/jenkins/critique-gerrit-review.py --dryrun

To run it, you might need a virtual env with Python3's venv installed.

http://gerrit.cloudera.org:8080/#/c/24699/4/java/calcite-planner/src/main/java/org/apache/impala/calcite/service/CalciteCompilerFactory.java
File java/calcite-planner/src/main/java/org/apache/impala/calcite/service/CalciteCompilerFactory.java:

http://gerrit.cloudera.org:8080/#/c/24699/4/java/calcite-planner/src/main/java/org/apache/impala/calcite/service/CalciteCompilerFactory.java@40
PS4, Line 40: import org.apache.calcite.sql2rel.SqlToRelConverter;
unused import: org.apache.calcite.sql2rel.SqlToRelConverter



-- 
To view, visit http://gerrit.cloudera.org:8080/24699
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I90b167055cb8d8b0045344085083160f1d98f5a4
Gerrit-Change-Number: 24699
Gerrit-PatchSet: 4
Gerrit-Owner: Steve Carlin <sc...@cloudera.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Arnab Karmakar <ar...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Comment-Date: Mon, 17 Aug 2026 13:11:26 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-15266: Fix flaky test due to slow Calcite first query.

Posted by "Steve Carlin (Code Review)" <ge...@cloudera.org>.
Hello Aman Sinha, Arnab Karmakar, Joe McDonnell, Michael Smith, Impala Public Jenkins, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/24699

to look at the new patch set (#5).

Change subject: IMPALA-15266: Fix flaky test due to slow Calcite first query.
......................................................................

IMPALA-15266: Fix flaky test due to slow Calcite first query.

The test test_invalidation_metrics in the
test_automatic_invalidation.py file is flaky for the Calcite planner.
On the first query, some static variables are being initialized.

Some of these static initializations are not done at startup time.
One of special note here is the RelMetadataQuery initializations.
Calcite generates java code on the first call to the RelMetadataQuery
functions. The JaninoRelMetadataProvider.revise() method is now called
for these RelMetadataQuery queries to compile the code before the
first query is run.

Also of note: The test_invalidation_metrics calls a "select 1" before
calling any queries. There is still some initialization of static
variables done within Calcite, but the quick "select 1" query gets
compiled way below the 1s threshold and subsequent queries clear the
1s barrier with ease as well.

Change-Id: I90b167055cb8d8b0045344085083160f1d98f5a4
---
M java/calcite-planner/src/main/java/org/apache/impala/calcite/service/CalciteCompilerFactory.java
M java/calcite-planner/src/main/java/org/apache/impala/calcite/service/CalciteQueryParser.java
M java/calcite-planner/src/main/java/org/apache/impala/calcite/service/CalciteRelNodeConverter.java
M tests/custom_cluster/test_automatic_invalidation.py
4 files changed, 55 insertions(+), 10 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/99/24699/5
-- 
To view, visit http://gerrit.cloudera.org:8080/24699
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I90b167055cb8d8b0045344085083160f1d98f5a4
Gerrit-Change-Number: 24699
Gerrit-PatchSet: 5
Gerrit-Owner: Steve Carlin <sc...@cloudera.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Arnab Karmakar <ar...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>