You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@grails.apache.org by "jdaugherty (via GitHub)" <gi...@apache.org> on 2026/10/13 09:07:00 UTC
[PR] Add the sql-tail command to find query-heavy pages [grails-core]
jdaugherty opened a new pull request, #16631:
URL: https://github.com/apache/grails-core/pull/16631
A page that is slow because it runs hundreds of queries is easy to miss in a log: `logSql` prints every statement, but not which page ran it or how many a request added up to. This adds the `sql-tail` command, which shows the pages of a running application ranked by the statements they run per request, and for each page the statement that repeats most within one request, which is what an N+1 looks like.
```
grails run-app --sql-tail # or ./gradlew bootRun -Pgrails.gorm.sqlTail.enabled
grails sql-tail # in a second terminal
```
```
sql-tail build/sql-tail/capture.log 312 statements in 14 requests (refreshes as statements run; Ctrl-C to stop)
PAGE REQUESTS AVG/REQ MAX TOTAL MOST REPEATED IN ONE REQUEST
book/show 6 48.0 61 288 44× select ... from author a1_0 where a1_0.id=?
book/index 5 4.2 5 21 5× select ... from book b1_0
```
`--statements` ranks the statements, `--page=book/show` shows one page's, `--stream` prints each statement as it runs with a header per request, and `--all --once` reports a whole run, including an integration test run.
### Design
- **No cost unless asked.** Capture is on only when `grails.gorm.sqlTail.enabled` is set at startup. An application started without it carries no statement inspector, no thread and no check per statement. `run-app --sql-tail` and `bootRun -Pgrails.gorm.sqlTail.enabled` set it for one run; it can also be kept in config for development.
- **Attribution per page.** `SqlTailStatementInspector`, installed by the Hibernate 5 and Hibernate 7 connection source factories, records each statement on the request thread. `SqlTailRequestAttribution` reads the controller, action and namespace Grails resolved for the request and gives every statement of one request the same id, so interceptors, the action and GSP rendering count for the same page. Work on other threads is shown under the thread name. An inspector the application configured itself through `hibernate.session_factory.statement_inspector` keeps running; the capture records the statement it returns.
- **Where it lives.** The writer, record format and settings are in `grails-datamapping-core` (Spring Web compile-only and guarded). The command is a script command in a new `grails-datamapping-core-cli` companion, so the `grails` shell runs it without booting the application; the reader, collapser, summary, renderer and runner sit beside it. The capture file is `sql-tail/capture.log` under the build directory, resolved as `BuildSettings` resolves it, tab separated and truncated per run; the reader resets when the application restarts.
- **Gradle.** The Grails Gradle plugin hands `-Pgrails.gorm.sqlTail.*` properties to forked application and test JVMs through a `GrailsPropertyArgumentProvider` that the startup progress forwarding now shares. No Gradle task is added for the command itself; script commands have no Gradle entry point today, and that is left for a separate decision.
### Documentation
- New reference page `sql-tail`, and the `--sql-tail` flag on the `run-app` page.
- New guide section *Finding Query-Heavy Pages with sql-tail* under *The DataSource*, including the shell's limitation with a build directory moved in `build.gradle`.
- What's New entry, and configuration metadata for `grails.gorm.sqlTail.enabled` and `grails.gorm.sqlTail.file`.
### Verification
- `grails-datamapping-core`: full `test` and the new `testCli` phase (settings, record, attribution, capture; collapser, summary, reader, renderer, runner, and the script compiled and run the way the shell does).
- `grails-data-hibernate7-core` and `grails-data-hibernate5-core`: full suites, including the new inspector specs (no inspector without the flag, capture with data source names, every data source to one file, a configured inspector by class name and as an instance, an unloadable one reported).
- `grails-gradle`: `GrailsGradlePluginToolchainSpec` with the new fork-settings features.
- `codeStyle` and `validateDependencyVersions` on the touched modules.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@grails.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add the sql-tail command to find query-heavy pages [grails-core]
Posted by "jamesfredley (via GitHub)" <gi...@apache.org>.
jamesfredley commented on code in PR #16631:
URL: https://github.com/apache/grails-core/pull/16631#discussion_r4341353089
##########
grails-datamapping-core/src/main/groovy/org/grails/datastore/gorm/jdbc/sqltail/SqlTailCapture.groovy:
##########
@@ -0,0 +1,155 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * https://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.grails.datastore.gorm.jdbc.sqltail
+
+import java.nio.charset.StandardCharsets
+import java.nio.file.Files
+import java.nio.file.StandardOpenOption
+
+import groovy.transform.CompileStatic
+import groovy.util.logging.Slf4j
+
+/**
+ * Writes the statements an application runs to the capture file the {@code sql-tail} command
+ * tails. One capture is open per file, shared by every data source of the application; the file is
+ * truncated when the capture opens, so it holds the statements of the current run only. Each
+ * statement is flushed as it is written so that the command sees it at once.
+ *
+ * <p>A capture is only opened when {@link SqlTailSettings#SETTING_ENABLED} is set; an application
+ * started without it never reaches this class.</p>
+ *
+ * @since 8.1
+ */
+@CompileStatic
+@Slf4j
+class SqlTailCapture implements Closeable {
+
+ private static final Map<String, SqlTailCapture> OPEN = new HashMap<>()
+
+ /** The file the statements are written to. */
+ final File file
+
+ private final SqlTailRequestAttribution attribution = new SqlTailRequestAttribution()
+ private final String key
+ private Writer writer
+
+ private SqlTailCapture(File file, String key) {
+ this.file = file
+ this.key = key
+ File parent = file.absoluteFile.parentFile
+ try {
+ if (parent != null && !parent.isDirectory() && !parent.mkdirs() && !parent.isDirectory()) {
+ throw new IOException("Cannot create the sql-tail capture directory ${parent}")
+ }
+ this.writer = Files.newBufferedWriter(file.toPath(), StandardCharsets.UTF_8,
+ StandardOpenOption.CREATE, StandardOpenOption.TRUNCATE_EXISTING, StandardOpenOption.WRITE)
+ }
+ catch (IOException e) {
+ throw new UncheckedIOException("Cannot open the sql-tail capture file ${file}: ${e.message}", e)
+ }
+ }
+
+ /**
+ * Opens the capture for a file, or returns the one already open for it.
+ *
+ * @param file the capture file
+ * @return the capture
+ * @throws UncheckedIOException when the file cannot be created
+ */
+ static synchronized SqlTailCapture open(File file) {
+ String key = file.canonicalPath
+ SqlTailCapture capture = OPEN.get(key)
Review Comment:
Nothing in the datastore or application shutdown path calls `close()`. This map keeps every writer for the life of the classloader. Closing a session factory leaves the file open. Opening another datastore on the same path reuses the old capture instead of truncating it. A different path leaks another handle.
Own the capture from the session factory. Reference-count it when several datasources share a file, and close and remove it when the last owner closes. Please test final-owner cleanup and reopening the same path.
##########
grails-datamapping-core/src/main/groovy/org/grails/datastore/gorm/jdbc/sqltail/SqlTailCapture.groovy:
##########
@@ -0,0 +1,155 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * https://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.grails.datastore.gorm.jdbc.sqltail
+
+import java.nio.charset.StandardCharsets
+import java.nio.file.Files
+import java.nio.file.StandardOpenOption
+
+import groovy.transform.CompileStatic
+import groovy.util.logging.Slf4j
+
+/**
+ * Writes the statements an application runs to the capture file the {@code sql-tail} command
+ * tails. One capture is open per file, shared by every data source of the application; the file is
+ * truncated when the capture opens, so it holds the statements of the current run only. Each
+ * statement is flushed as it is written so that the command sees it at once.
+ *
+ * <p>A capture is only opened when {@link SqlTailSettings#SETTING_ENABLED} is set; an application
+ * started without it never reaches this class.</p>
+ *
+ * @since 8.1
+ */
+@CompileStatic
+@Slf4j
+class SqlTailCapture implements Closeable {
+
+ private static final Map<String, SqlTailCapture> OPEN = new HashMap<>()
+
+ /** The file the statements are written to. */
+ final File file
+
+ private final SqlTailRequestAttribution attribution = new SqlTailRequestAttribution()
+ private final String key
+ private Writer writer
+
+ private SqlTailCapture(File file, String key) {
+ this.file = file
+ this.key = key
+ File parent = file.absoluteFile.parentFile
+ try {
+ if (parent != null && !parent.isDirectory() && !parent.mkdirs() && !parent.isDirectory()) {
+ throw new IOException("Cannot create the sql-tail capture directory ${parent}")
+ }
+ this.writer = Files.newBufferedWriter(file.toPath(), StandardCharsets.UTF_8,
+ StandardOpenOption.CREATE, StandardOpenOption.TRUNCATE_EXISTING, StandardOpenOption.WRITE)
+ }
+ catch (IOException e) {
+ throw new UncheckedIOException("Cannot open the sql-tail capture file ${file}: ${e.message}", e)
+ }
+ }
+
+ /**
+ * Opens the capture for a file, or returns the one already open for it.
+ *
+ * @param file the capture file
+ * @return the capture
+ * @throws UncheckedIOException when the file cannot be created
+ */
+ static synchronized SqlTailCapture open(File file) {
+ String key = file.canonicalPath
+ SqlTailCapture capture = OPEN.get(key)
+ if (capture == null) {
+ capture = new SqlTailCapture(file, key)
+ OPEN.put(key, capture)
+ log.info('Capturing SQL statements to {} for sql-tail', file)
+ }
+ capture
+ }
+
+ /**
+ * Opens the capture the settings name.
+ *
+ * @param settings the sql-tail settings
+ * @return the capture
+ */
+ static SqlTailCapture open(SqlTailSettings settings) {
+ open(settings.resolveFile())
+ }
+
+ /**
+ * Records a statement that is about to run on the current thread.
+ *
+ * @param dataSource the connection source name the statement runs against
+ * @param sql the statement
+ */
+ void record(String dataSource, String sql) {
+ SqlTailRecord record = new SqlTailRecord(timestamp: System.currentTimeMillis(), dataSource: dataSource, sql: sql)
Review Comment:
This stores the SQL Hibernate is about to run, unchanged. `toLine()` only collapses whitespace. A statement such as `select * from account where password='example-secret'` or a comment that contains a key is written to the capture file and can be printed by the CLI.
Redact literals and comments on the recorded copy before it is written. Leave the SQL returned to Hibernate unchanged. If the statement cannot be parsed, fail closed on the recorded copy rather than writing the raw text. Please assert that an embedded secret never appears in the capture or the CLI output, and that the database still receives the original statement.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@grails.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add the sql-tail command to find query-heavy pages [grails-core]
Posted by "borinquenkid (via GitHub)" <gi...@apache.org>.
borinquenkid commented on PR #16631:
URL: https://github.com/apache/grails-core/pull/16631#issuecomment-6294424723
Pushed two small commits on top of a3f5590b08, @jdaugherty — shout if you'd rather have them elsewhere.
**97e21ab3e9 (build-logic): report a module's `cli` source set to JaCoCo.** The aggregate fed JaCoCo only each module's `main` classes and sources, so the eight command classes under `grails-datamapping-core/src/cli` (~610 lines: collapser, reader, renderer, runner, summary, options, mode, type) were absent from the report and from Codecov — the 85% patch figure covered only the `main` tier. `testCli`'s exec data was already collected; it just had no classes to match. `GrailsJacocoPlugin` now adds the `cli` source set whenever a module has one, with a case in `GrailsJacocoPluginSpec`. Regenerating the aggregate locally (`testCli jacocoAggregateReport`) now shows the tier: 299 lines covered, 9 missed (`SqlTailRunner` 7, `SqlStatementCollapser` 2). This gap is pre-existing — grails-core's `src/cli` has the same blind spot — so happy to split it into its own PR if you prefer.
**13d66f218e (test): the mkdirs branch of `SqlTailCapture.open`** (`:57`), the one line your a3f5590b08 left: the capture creating its parent directory rather than finding it.
Everything else Codecov listed as missing in `grails-datamapping-core` is covered by your commit; the remaining `GrailsGradlePlugin` / `GrailsPropertyArgumentProvider` lines run in TestKit's daemon, where JaCoCo can't see them.
Locally on this head: sqltail specs 33 passed (1 skipped: the `/dev/full` case, Linux only), `codenarcTest` clean; `GrailsJacocoPluginSpec` 7/7.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@grails.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add the sql-tail command to find query-heavy pages [grails-core]
Posted by "testlens-app[bot] (via GitHub)" <gi...@apache.org>.
testlens-app[bot] commented on PR #16631:
URL: https://github.com/apache/grails-core/pull/16631#issuecomment-6297435828
## ✅ All tests passed ✅
🏷️ Commit: 153dc6da675d9041f6f78cdc114addfe561d834c
▶️ Tests: 6938 executed
⚪️ Checks: 89/89 completed
---
_Learn more about TestLens at [testlens.app/docs](https://testlens.app/docs/features/pr-comment/)._
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@grails.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add the query-tail command to find query-heavy pages [grails-core]
Posted by "testlens-app[bot] (via GitHub)" <gi...@apache.org>.
testlens-app[bot] commented on PR #16631:
URL: https://github.com/apache/grails-core/pull/16631#issuecomment-6388381590
## ✅ All tests passed ✅
🏷️ Commit: b0e9a8c9a450daa269fbf3df8388a8e885ba4e82
▶️ Tests: 138171 executed
⚪️ Checks: 97/97 completed
---
_Learn more about TestLens at [testlens.app/docs](https://testlens.app/docs/features/pr-comment/)._
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@grails.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add the query-tail command to find query-heavy pages [grails-core]
Posted by "testlens-app[bot] (via GitHub)" <gi...@apache.org>.
testlens-app[bot] commented on PR #16631:
URL: https://github.com/apache/grails-core/pull/16631#issuecomment-6397261655
## 🚨 TestLens detected 1 failed test 🚨
### Test Summary
> [!TIP]
> Inspect all details with the [TestLens CLI](https://testlens.app/docs/features/cli): `testlens pr 16631`
#### [CI / Functional Tests \(Java 25, indy\=false, shard 1\)](https://github.com/apache/grails-core/actions/runs/40189574456/job/121157519322?pr=16631) > :grails-test-examples-app1:integrationTest
| Test | Runs | Flakiness |
|---|---|--:|
| RedirectWithAndWithoutParamsFunctionalSpec > Params are not added to the url after a redirect even if they are passed to the redirect | ❌ | [1% 🟡](https://github.com/apache/grails-core/issues/16030) |
🏷️ Commit: 73c20a13c7207c7a86ace3ca24b04d40be40569a
▶️ Tests: 21188 executed
🟡 Checks: 39/92 completed
### Test Failures
<details><summary><strong>RedirectWithAndWithoutParamsFunctionalSpec > Params are not added to the url after a redirect even if they are passed to the redirect</strong> (:grails-test-examples-app1:integrationTest in <a href="https://github.com/apache/grails-core/actions/runs/40189574456/job/121157519322?pr=16631">CI / Functional Tests (Java 25, indy=false, shard 1)</a>)</summary>
```
Condition not satisfied:
pageSource.contains('"id":')
| |
| false
<html><head></head><body><form action="save" method="post">
<input type="text" name="name">
<input type="submit">
</form>
</body></html>
at functionaltests.RedirectWithAndWithoutParamsFunctionalSpec.$tt__$spock_feature_1_0(RedirectWithAndWithoutParamsFunctionalSpec.groovy:41)
at functionaltests.RedirectWithAndWithoutParamsFunctionalSpec.Params are not added to the url after a redirect even if they are passed to the redirect_closure1(RedirectWithAndWithoutParamsFunctionalSpec.groovy)
at grails.gorm.transactions.GrailsTransactionTemplate$1.doInTransaction(GrailsTransactionTemplate.groovy:108)
at org.springframework.transaction.support.TransactionTemplate.execute(TransactionTemplate.java:137)
at grails.gorm.transactions.GrailsTransactionTemplate.doExecuteAndRollback(GrailsTransactionTemplate.groovy:105)
at grails.gorm.transactions.GrailsTransactionTemplate.executeAndRollback_closure1(GrailsTransactionTemplate.groovy:100)
at grails.gorm.transactions.GrailsTransactionTemplate.inConnectionScope(GrailsTransactionTemplate.groovy:96)
at grails.gorm.transactions.GrailsTransactionTemplate.executeAndRollback(GrailsTransactionTemplate.groovy:100)
at functionaltests.RedirectWithAndWithoutParamsFunctionalSpec.Params are not added to the url after a redirect even if they are passed to the redirect(RedirectWithAndWithoutParamsFunctionalSpec.groovy)
```
</details>
### Rerun Controls
> [!NOTE]
> Checks are currently running using the configuration below.
Select tests to mute in this pull request:
🔲 RedirectWithAndWithoutParamsFunctionalSpec > Params are not added to the url after a redirect even if they are passed to the redirect <!-- uniqueId=[engine:spock]/[spec:functionaltests.RedirectWithAndWithoutParamsFunctionalSpec]/[feature:$spock_feature_1_0] -->
Reuse successful test results:
🔲 ♻️ Only rerun the tests that failed or were muted before
Click the checkbox to trigger a rerun:
🔲 **Rerun jobs**
---
_Learn more about TestLens at [testlens.app/docs](https://testlens.app/docs/features/pr-comment/)._
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@grails.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add the query-tail command to find query-heavy pages [grails-core]
Posted by "jamesfredley (via GitHub)" <gi...@apache.org>.
jamesfredley commented on code in PR #16631:
URL: https://github.com/apache/grails-core/pull/16631#discussion_r4385378854
##########
grails-datamapping-core/src/main/groovy/org/apache/grails/datamapping/core/querytail/CypherSyntax.groovy:
##########
@@ -0,0 +1,110 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * https://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.grails.datamapping.core.querytail
+
+import groovy.transform.CompileStatic
+import groovy.transform.PackageScope
+
+/**
+ * Cypher, the query language of Neo4j: single and double quotes delimit a string, in which a
+ * backslash escapes the character after it and a quote written twice is two strings; backticks
+ * delimit a name, in which a backtick is written twice; {@code //} starts a line comment, and
+ * {@code --} and {@code #} do not, as {@code -->} draws a relationship; block comments do not nest;
+ * and a dollar starts a parameter. Neo4j 5 ends a line comment at a carriage return, which Neo4j 3.5
+ * does not, so a statement with a carriage return that no line feed follows is read both ways. Neo4j 5
+ * decodes a {@code \u} escape before it parses, where 3.5 reads it only inside a string, so a
+ * backslash outside a string or a comment, which may spell a quote, withholds the statement.
+ */
+@CompileStatic
+@PackageScope
+final class CypherSyntax extends StandardSqlSyntax {
+
+ private static final char BACKSLASH = '\\'
+ private static final char QUOTE = '\''
+ private static final char DOUBLE_QUOTE = '"'
+ private static final char BACKTICK = '`'
+
+ private final boolean carriageReturnEndsLine
+
+ CypherSyntax(boolean carriageReturnEndsLine) {
+ super('Cypher')
+ this.carriageReturnEndsLine = carriageReturnEndsLine
+ }
+
+ @Override
+ List<QueryTailSyntax> readings(String sql) {
+ if (!hasLoneCarriageReturn(sql)) {
+ return List.<QueryTailSyntax>of(this)
+ }
+ List.<QueryTailSyntax>of(new CypherSyntax(false), new CypherSyntax(true))
+ }
+
+ @Override
+ int endOfComment(String sql, int index) {
+ if (sql.startsWith('//', index)) {
+ return endOfLine(sql, index)
+ }
+ if (sql.startsWith('/*', index)) {
+ return endOfBlockComment(sql, index)
+ }
+ NONE
+ }
+
+ @Override
+ int endOfValue(String sql, int index) {
+ char c = sql.charAt(index)
+ if (c == BACKSLASH) {
+ return UNTERMINATED
+ }
+ if (c == QUOTE || c == DOUBLE_QUOTE) {
Review Comment:
Neo4j 5 reads `RETURN 'a\u0027 + 'secret' + \u0027b'` as three strings. This redactor returns `RETURN ?secret?`, so `secret` reaches the file. The backslash check does not see an escape that is inside a string.
Account for that Unicode quote, or withhold the statement. Please assert `secret` is absent from the capture.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@grails.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add the query-tail command to find query-heavy pages [grails-core]
Posted by "jdaugherty (via GitHub)" <gi...@apache.org>.
jdaugherty commented on PR #16631:
URL: https://github.com/apache/grails-core/pull/16631#issuecomment-6385237757
@jamesfredley re review 5638523900: both items are fixed on head b3a97c2543. A numeric key is kept only where the path before it is a field the mapping knows as an array; in a map property, under an unknown key or at the top of a document it is recorded as `?`, asserted in `MongoCommandRedactorSpec` and through the capture API in `QueryTailCommandListenerSpec` against a real mongod (the numeric key of an `attrs` map reaches neither the insert nor the find in the file). The run header is now written while the starter still holds the `STARTING` lock, so a joiner cannot append before it. Details are in the replies on your two threads, left open for you.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@grails.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add the sql-tail command to find query-heavy pages [grails-core]
Posted by "testlens-app[bot] (via GitHub)" <gi...@apache.org>.
testlens-app[bot] commented on PR #16631:
URL: https://github.com/apache/grails-core/pull/16631#issuecomment-6373776883
## 🚨 All tests passed but jobs failed 🚨
### Failed Jobs without Test Failures
❌ [CI / Mongodb Functional Tests \(Java 21, MongoDB 8.0, indy\=false\)](https://github.com/apache/grails-core/actions/runs/40011004465/job/120611839274?pr=16631)
🏷️ Commit: 9f418b24179ae4cf5c1e104103f35e30c5eae5d0
▶️ Tests: 42414 executed
🟡 Checks: 54/88 completed
### Rerun Controls
Click the checkbox to trigger a rerun:
🔲 **Rerun jobs**
---
_Learn more about TestLens at [testlens.app/docs](https://testlens.app/docs/features/pr-comment/)._
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@grails.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add the query-tail command to find query-heavy pages [grails-core]
Posted by "jdaugherty (via GitHub)" <gi...@apache.org>.
jdaugherty commented on code in PR #16631:
URL: https://github.com/apache/grails-core/pull/16631#discussion_r4393424575
##########
grails-data-mongodb/core/src/main/groovy/org/apache/grails/data/mongodb/core/querytail/MongoCommandRedactor.groovy:
##########
@@ -0,0 +1,511 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * https://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.grails.data.mongodb.core.querytail
+
+import groovy.transform.CompileStatic
+import org.bson.BsonDocument
+import org.bson.BsonValue
+
+import org.apache.grails.datamapping.core.querytail.QueryTailRedactor
+
+/**
+ * Takes the values out of a MongoDB command before it is captured. A command is a document, so
+ * there is nothing to parse: every value becomes {@link QueryTailRedactor#PLACEHOLDER}, and the
+ * keys, which are the options of the command, its operators and the field names of the collection,
+ * are kept, except where a key may be data. The keys of a document stored in a map property, or in
+ * a property or a collection the {@linkplain MongoMappedFields mapping} does not know, are recorded
+ * as the placeholder too, and so is every segment of a dotted path past such a key. So is a key at
+ * the top of stored data, the documents an insert writes, a replacement document, the fields an
+ * update writes and the operand of a literal, unless the mapping knows it, as the name of a dynamic
+ * attribute may be data; the keys at the top of a condition, a pipeline, a sort or a projection
+ * are the fields the application asks for, and are kept. An update that is a pipeline writes the
+ * documents its stages build: the keys at the top of what a {@code $set}, {@code $addFields} or
+ * {@code $project} stage holds, of the new root of a {@code $replaceWith} or {@code $replaceRoot}, and of a
+ * document {@code $mergeObjects} merges into it are fields written, kept only when the mapping knows
+ * them, as those of a classic update are. A numeric key
+ * is kept only where it indexes an array the mapping knows; in a map, in an element of an array,
+ * which is a map or a document unless the field is a list of lists, or at the top of a document,
+ * it is data. A {@code $} key is kept as an operator only where operators go, in a condition, a
+ * pipeline and the operators of an update; in stored data, the documents an insert writes, a
+ * replacement document and the values an update operator writes, it is a field name, which MongoDB
+ * allows to start with {@code $}, and is recorded as the placeholder unless the mapping knows it.
+ * So is a {@code $} key in the operand of a comparison in a condition, such as {@code $eq} or
+ * {@code $in}, or an implicit document or array equality, which holds the value the field is compared
+ * with, and in a DBRef, whose {@code $ref}, {@code $id} and {@code $db} a condition reads as fields
+ * compared with their whole value, in any order; in an aggregation expression, which
+ * {@code $expr}, the {@code let} of a command and the stages that compute fields hold, the operands
+ * of a comparison are expressions, and their operators are kept, except in the operand of a
+ * {@code $literal} or {@code $const}, which is stored data. A comment, and the constants of an
+ * update, may be any document, and none of their keys are kept. The entries the driver adds for its
+ * session, the cluster time and the transaction, which say nothing about the command, are left out.
+ *
+ * <p>A command is recorded as its name, the database and collection it runs against, and the
+ * document of its remaining entries: {@code find shop.book {"filter": {"title": ?}, "limit": ?}}.</p>
+ *
+ * @since 9.0
+ */
+@CompileStatic
+final class MongoCommandRedactor {
+
+ private static final String PLACEHOLDER = QueryTailRedactor.PLACEHOLDER
+
+ private static final String DATABASE = '$db'
+
+ private static final String COLLECTION = 'collection'
+
+ /** The entries the driver adds, which describe its session and transaction rather than the command. */
+ private static final Set<String> LEFT_OUT = ['lsid', '$clusterTime', 'txnNumber', 'autocommit',
+ 'startTransaction', '$readPreference', 'apiVersion', 'apiStrict', 'apiDeprecationErrors'] as Set<String>
+
+ /**
+ * The options whose document, or documents, are keyed by the fields of the collection, wherever they appear,
+ * including those of the operations of a client-level {@code bulkWrite}.
+ */
+ private static final Set<String> FIELD_KEYED = ['filter', 'query', 'q', 'u', 'update', 'document', 'documents',
+ 'updateMods', 'pipeline', 'sort', 'projection', 'fields', 'key', 'hint', 'min', 'max', 'arrayFilters',
+ 'partialFilterExpression', 'wildcardProjection', 'validator', 'weights'] as Set<String>
+
+ /** Documents that match fields, whose field values may be implicit equality operands. */
+ private static final Set<String> CONDITION_KEYED = ['filter', 'query', 'q', 'arrayFilters',
+ 'partialFilterExpression', 'validator', 'restrictSearchWithMatch'] as Set<String>
+
+ /** The options whose documents are stored data: what an insert writes. */
+ private static final Set<String> DATA_KEYED = ['documents', 'document'] as Set<String>
+
+ /** The options and the operator that may hold any document, none of whose keys say what the command does. */
+ private static final Set<String> ANY_VALUE = ['comment', '$comment', 'c'] as Set<String>
+
+ /** The option whose values are aggregation expressions. */
+ private static final String LET = 'let'
+
+ /** The options that hold an update: update operators, a replacement document or a pipeline. */
+ private static final Set<String> UPDATE_KEYED = ['u', 'update', 'updateMods'] as Set<String>
+
+ /** The update operators whose fields take a condition rather than the value to write. */
+ private static final Set<String> CONDITION_OPERATORS = ['$pull', '$currentDate'] as Set<String>
+
+ /** The update operators that add to an array, whose value may hold modifiers. */
+ private static final Set<String> PUSH_OPERATORS = ['$push', '$addToSet'] as Set<String>
+
+ private static final String EACH = '$each'
+
+ /** The modifiers of a {@code $push} or an {@code $addToSet}. */
+ private static final Set<String> PUSH_MODIFIERS = [EACH, '$slice', '$sort', '$position'] as Set<String>
+
+ /**
+ * The operators of a pipeline whose operand is literal data rather than an expression, including
+ * {@code $const}, which MongoDB accepts for {@code $literal}.
+ */
+ private static final Set<String> LITERAL_OPERATORS = ['$literal', '$const', '$documents'] as Set<String>
+
+ /**
+ * The fields of a DBRef, which a condition reads as fields compared with their whole value, never as
+ * operators, in any order and wherever a condition has fields.
+ */
+ private static final Set<String> DBREF_FIELDS = ['$ref', '$id', '$db'] as Set<String>
+
+ /**
+ * The query operators whose operand holds values rather than a condition: the comparisons, whose operand
+ * is the value a field is compared with, a JSON schema, whose {@code enum} lists values, and a
+ * GeoJSON geometry, which can carry additional data alongside its type and coordinates.
+ */
+ private static final Set<String> VALUE_OPERATORS = ['$eq', '$ne', '$gt', '$gte', '$lt', '$lte', '$in', '$nin',
+ '$all', '$jsonSchema', '$geometry'] as Set<String>
+
+ /** The operator of a condition, and the stages of a pipeline, whose operand is an aggregation expression. */
+ private static final Set<String> EXPRESSION_OPERATORS = ['$expr', '$addFields', '$set', '$project', '$group',
+ '$replaceRoot', '$replaceWith', '$redact', '$bucket', '$bucketAuto', '$sortByCount', '$setWindowFields',
+ '$fill'] as Set<String>
+
+ /** The stages of an update pipeline whose operand is the document they write, or the new root that replaces it. */
+ private static final Set<String> WRITING_STAGES = ['$set', '$addFields', '$project', '$replaceWith'] as Set<String>
+
+ /** The stage of an update pipeline whose options name the new root that replaces the document. */
+ private static final String REPLACE_ROOT = '$replaceRoot'
+
+ /** The option of {@code $replaceRoot} that holds the new root. */
+ private static final String NEW_ROOT = 'newRoot'
+
+ /** The operator whose operands are documents merged into one, which in an update pipeline is written. */
+ private static final String MERGE_OBJECTS = '$mergeObjects'
+
+ /** The kinds of document whose keys at the top are stored data: what an insert, a replacement or an update writes. */
+ private static final Set<Kind> STORED = EnumSet.of(Kind.DATA, Kind.UPDATE, Kind.UPDATE_FIELDS, Kind.WRITTEN)
+
+ private MongoCommandRedactor() {
+ }
+
+ /**
+ * Redacts a command.
+ *
+ * @param database the database the command runs against
+ * @param commandName the name of the command, which is its first key
+ * @param command the command as the driver sends it
+ * @param fields the fields the mapping of the datastore knows, or null before the datastore is built
+ * @return the command with its values replaced
+ */
+ static String redact(String database, String commandName, BsonDocument command, MongoMappedFields fields) {
+ String collection = null
+ String collectionKey = null
+ BsonValue first = command.get(commandName)
+ if (first != null && first.isString()) {
Review Comment:
Fixed in fe3a01a69b. A collection is now taken only from a command that runs against one. That means an allowlist of the commands whose first value is their collection: `find`, `insert`, `update`, `delete`, `aggregate`, `count`, `distinct`, `findAndModify`, `mapReduce`, the collection and index commands, `killCursors`, and the search-index and plan-cache commands. The `collection` option is used only for `getMore`. The first value of any other command is recorded as `?` like every other value, so your command is recorded as `dropUser admin {"dropUser": ?}`. A command that is not on the list, including one MongoDB adds later, is redacted rather than named.
Tests:
- `MongoCommandRedactorSpec`: "only a command that runs against a collection names one..." covers `dropUser`, `usersInfo`, `grantRolesToUser`, `renameCollection` (whose full namespace is redacted too), `enableSharding` and an unknown command with a `collection` option, each asserting the secret is absent. The `findandmodify`, `distinct`, `listIndexes`, `create` and `drop` rows still name their collection.
- `QueryTailCommandListenerSpec`: creates `alice@example.com` on the test MongoDB and drops it with `dropUser`. It asserts the user is gone, that `dropUser` and `usersInfo` are recorded with `?`, and that neither the account name nor its password is in the file.
##########
grails-data-mongodb/core/src/main/groovy/org/apache/grails/data/mongodb/core/querytail/MongoCommandRedactor.groovy:
##########
@@ -0,0 +1,511 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * https://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.grails.data.mongodb.core.querytail
+
+import groovy.transform.CompileStatic
+import org.bson.BsonDocument
+import org.bson.BsonValue
+
+import org.apache.grails.datamapping.core.querytail.QueryTailRedactor
+
+/**
+ * Takes the values out of a MongoDB command before it is captured. A command is a document, so
+ * there is nothing to parse: every value becomes {@link QueryTailRedactor#PLACEHOLDER}, and the
+ * keys, which are the options of the command, its operators and the field names of the collection,
+ * are kept, except where a key may be data. The keys of a document stored in a map property, or in
+ * a property or a collection the {@linkplain MongoMappedFields mapping} does not know, are recorded
+ * as the placeholder too, and so is every segment of a dotted path past such a key. So is a key at
+ * the top of stored data, the documents an insert writes, a replacement document, the fields an
+ * update writes and the operand of a literal, unless the mapping knows it, as the name of a dynamic
+ * attribute may be data; the keys at the top of a condition, a pipeline, a sort or a projection
+ * are the fields the application asks for, and are kept. An update that is a pipeline writes the
+ * documents its stages build: the keys at the top of what a {@code $set}, {@code $addFields} or
+ * {@code $project} stage holds, of the new root of a {@code $replaceWith} or {@code $replaceRoot}, and of a
+ * document {@code $mergeObjects} merges into it are fields written, kept only when the mapping knows
+ * them, as those of a classic update are. A numeric key
+ * is kept only where it indexes an array the mapping knows; in a map, in an element of an array,
+ * which is a map or a document unless the field is a list of lists, or at the top of a document,
+ * it is data. A {@code $} key is kept as an operator only where operators go, in a condition, a
+ * pipeline and the operators of an update; in stored data, the documents an insert writes, a
+ * replacement document and the values an update operator writes, it is a field name, which MongoDB
+ * allows to start with {@code $}, and is recorded as the placeholder unless the mapping knows it.
+ * So is a {@code $} key in the operand of a comparison in a condition, such as {@code $eq} or
+ * {@code $in}, or an implicit document or array equality, which holds the value the field is compared
+ * with, and in a DBRef, whose {@code $ref}, {@code $id} and {@code $db} a condition reads as fields
+ * compared with their whole value, in any order; in an aggregation expression, which
+ * {@code $expr}, the {@code let} of a command and the stages that compute fields hold, the operands
+ * of a comparison are expressions, and their operators are kept, except in the operand of a
+ * {@code $literal} or {@code $const}, which is stored data. A comment, and the constants of an
+ * update, may be any document, and none of their keys are kept. The entries the driver adds for its
+ * session, the cluster time and the transaction, which say nothing about the command, are left out.
+ *
+ * <p>A command is recorded as its name, the database and collection it runs against, and the
+ * document of its remaining entries: {@code find shop.book {"filter": {"title": ?}, "limit": ?}}.</p>
+ *
+ * @since 9.0
+ */
+@CompileStatic
+final class MongoCommandRedactor {
+
+ private static final String PLACEHOLDER = QueryTailRedactor.PLACEHOLDER
+
+ private static final String DATABASE = '$db'
+
+ private static final String COLLECTION = 'collection'
+
+ /** The entries the driver adds, which describe its session and transaction rather than the command. */
+ private static final Set<String> LEFT_OUT = ['lsid', '$clusterTime', 'txnNumber', 'autocommit',
+ 'startTransaction', '$readPreference', 'apiVersion', 'apiStrict', 'apiDeprecationErrors'] as Set<String>
+
+ /**
+ * The options whose document, or documents, are keyed by the fields of the collection, wherever they appear,
+ * including those of the operations of a client-level {@code bulkWrite}.
+ */
+ private static final Set<String> FIELD_KEYED = ['filter', 'query', 'q', 'u', 'update', 'document', 'documents',
+ 'updateMods', 'pipeline', 'sort', 'projection', 'fields', 'key', 'hint', 'min', 'max', 'arrayFilters',
+ 'partialFilterExpression', 'wildcardProjection', 'validator', 'weights'] as Set<String>
+
+ /** Documents that match fields, whose field values may be implicit equality operands. */
+ private static final Set<String> CONDITION_KEYED = ['filter', 'query', 'q', 'arrayFilters',
+ 'partialFilterExpression', 'validator', 'restrictSearchWithMatch'] as Set<String>
+
+ /** The options whose documents are stored data: what an insert writes. */
+ private static final Set<String> DATA_KEYED = ['documents', 'document'] as Set<String>
+
+ /** The options and the operator that may hold any document, none of whose keys say what the command does. */
+ private static final Set<String> ANY_VALUE = ['comment', '$comment', 'c'] as Set<String>
+
+ /** The option whose values are aggregation expressions. */
+ private static final String LET = 'let'
+
+ /** The options that hold an update: update operators, a replacement document or a pipeline. */
+ private static final Set<String> UPDATE_KEYED = ['u', 'update', 'updateMods'] as Set<String>
+
+ /** The update operators whose fields take a condition rather than the value to write. */
+ private static final Set<String> CONDITION_OPERATORS = ['$pull', '$currentDate'] as Set<String>
+
+ /** The update operators that add to an array, whose value may hold modifiers. */
+ private static final Set<String> PUSH_OPERATORS = ['$push', '$addToSet'] as Set<String>
+
+ private static final String EACH = '$each'
+
+ /** The modifiers of a {@code $push} or an {@code $addToSet}. */
+ private static final Set<String> PUSH_MODIFIERS = [EACH, '$slice', '$sort', '$position'] as Set<String>
+
+ /**
+ * The operators of a pipeline whose operand is literal data rather than an expression, including
+ * {@code $const}, which MongoDB accepts for {@code $literal}.
+ */
+ private static final Set<String> LITERAL_OPERATORS = ['$literal', '$const', '$documents'] as Set<String>
+
+ /**
+ * The fields of a DBRef, which a condition reads as fields compared with their whole value, never as
+ * operators, in any order and wherever a condition has fields.
+ */
+ private static final Set<String> DBREF_FIELDS = ['$ref', '$id', '$db'] as Set<String>
+
+ /**
+ * The query operators whose operand holds values rather than a condition: the comparisons, whose operand
+ * is the value a field is compared with, a JSON schema, whose {@code enum} lists values, and a
+ * GeoJSON geometry, which can carry additional data alongside its type and coordinates.
+ */
+ private static final Set<String> VALUE_OPERATORS = ['$eq', '$ne', '$gt', '$gte', '$lt', '$lte', '$in', '$nin',
+ '$all', '$jsonSchema', '$geometry'] as Set<String>
+
+ /** The operator of a condition, and the stages of a pipeline, whose operand is an aggregation expression. */
+ private static final Set<String> EXPRESSION_OPERATORS = ['$expr', '$addFields', '$set', '$project', '$group',
+ '$replaceRoot', '$replaceWith', '$redact', '$bucket', '$bucketAuto', '$sortByCount', '$setWindowFields',
+ '$fill'] as Set<String>
+
+ /** The stages of an update pipeline whose operand is the document they write, or the new root that replaces it. */
+ private static final Set<String> WRITING_STAGES = ['$set', '$addFields', '$project', '$replaceWith'] as Set<String>
+
+ /** The stage of an update pipeline whose options name the new root that replaces the document. */
+ private static final String REPLACE_ROOT = '$replaceRoot'
+
+ /** The option of {@code $replaceRoot} that holds the new root. */
+ private static final String NEW_ROOT = 'newRoot'
+
+ /** The operator whose operands are documents merged into one, which in an update pipeline is written. */
+ private static final String MERGE_OBJECTS = '$mergeObjects'
+
+ /** The kinds of document whose keys at the top are stored data: what an insert, a replacement or an update writes. */
+ private static final Set<Kind> STORED = EnumSet.of(Kind.DATA, Kind.UPDATE, Kind.UPDATE_FIELDS, Kind.WRITTEN)
+
+ private MongoCommandRedactor() {
+ }
+
+ /**
+ * Redacts a command.
+ *
+ * @param database the database the command runs against
+ * @param commandName the name of the command, which is its first key
+ * @param command the command as the driver sends it
+ * @param fields the fields the mapping of the datastore knows, or null before the datastore is built
+ * @return the command with its values replaced
+ */
+ static String redact(String database, String commandName, BsonDocument command, MongoMappedFields fields) {
+ String collection = null
+ String collectionKey = null
+ BsonValue first = command.get(commandName)
+ if (first != null && first.isString()) {
+ collection = first.asString().value
+ collectionKey = commandName
+ }
+ else if (command.get(COLLECTION)?.isString()) {
+ collection = command.getString(COLLECTION).value
+ collectionKey = COLLECTION
+ }
+ StringBuilder out = new StringBuilder(commandName)
+ if (database != null || collection != null) {
+ out.append(' ')
+ if (database != null) {
+ out.append(database)
+ }
+ if (collection != null) {
+ out.append(database != null ? '.' : '').append(collection)
+ }
+ }
+ Scope options = Scope.options(collection, fields)
+ StringBuilder body = new StringBuilder()
+ for (Map.Entry<String, BsonValue> entry in command.entrySet()) {
+ String key = entry.key
+ if (key == DATABASE || key == collectionKey || LEFT_OUT.contains(key)) {
+ continue
+ }
+ body.append(body.length() == 0 ? '{' : ', ')
+ appendEntry(body, key, entry.value, options)
+ }
+ if (body.length() > 0) {
+ out.append(' ').append(body).append('}')
+ }
+ out.toString()
+ }
+
+ private static void appendEntry(StringBuilder out, String key, BsonValue value, Scope scope) {
+ Scope child = scope.of(key, value)
+ out.append(child.keyText).append(': ')
+ appendValue(out, value, child)
+ }
+
+ private static void appendValue(StringBuilder out, BsonValue value, Scope scope) {
+ if (value.isDocument()) {
+ out.append('{')
+ boolean first = true
+ for (Map.Entry<String, BsonValue> entry in value.asDocument().entrySet()) {
+ if (!first) {
+ out.append(', ')
+ }
+ first = false
+ appendEntry(out, entry.key, entry.value, scope)
+ }
+ out.append('}')
+ }
+ else if (value.isArray()) {
+ out.append('[')
+ boolean first = true
+ Scope element = scope.element()
+ for (BsonValue item in value.asArray()) {
+ if (!first) {
+ out.append(', ')
+ }
+ first = false
+ appendValue(out, item, element)
+ }
+ out.append(']')
+ }
+ else {
+ out.append(PLACEHOLDER)
+ }
+ }
+
+ private static String quote(String key) {
+ StringBuilder out = new StringBuilder(key.length() + 2).append('"')
+ for (int i = 0; i < key.length(); i++) {
+ char c = key.charAt(i)
+ if (c == '"' || c == '\\') {
+ out.append('\\').append(c)
+ }
+ else if (c < 0x20) {
+ out.append(String.format('\\u%04x', (int) c))
+ }
+ else {
+ out.append(c)
+ }
+ }
+ out.append('"').toString()
+ }
+
+ /**
+ * Where a document sits in the command, which decides what its keys are: the options of the
+ * command, kept; the fields and operators of a condition, a pipeline, a sort or a projection, where
+ * a {@code $} key is an operator; an aggregation expression, whose {@code $} keys are operators
+ * too; an update, whose {@code $} keys are update operators; or stored data, the documents an
+ * insert writes, the values an update writes and the values a condition compares with, where a
+ * {@code $} key is a field name like any other and is kept only when the mapping knows it.
+ */
+ @CompileStatic
+ private static final class Scope {
+
+ final String collection
+ final MongoMappedFields fields
+ final Kind kind
+ /**
+ * The update operator whose fields a document of kind {@link Kind#UPDATE_FIELDS} holds, or the stage
+ * whose document one of kind {@link Kind#WRITTEN} is.
+ */
+ final String operator
+ /** The field path of the document; null in a document keyed by fields the mapping does not know. */
+ final List<String> path
+ /** How many indices into the field at the path have been taken: 0 at the field, 1 in one of its elements. */
+ final int depth
+ final String keyText
+
+ private Scope(String collection, MongoMappedFields fields, Kind kind, String operator, List<String> path,
+ int depth, String keyText) {
+ this.collection = collection
+ this.fields = fields
+ this.kind = kind
+ this.operator = operator
+ this.path = path
+ this.depth = depth
+ this.keyText = keyText
+ }
+
+ static Scope options(String collection, MongoMappedFields fields) {
+ new Scope(collection, fields, Kind.OPTIONS, null, Collections.<String>emptyList(), 0, null)
+ }
+
+ /**
+ * @return the scope of an element of an array in this scope: one index deeper into the field
+ * at the path, so that a numeric key inside the element is data unless the elements are arrays;
+ * the elements of an update that is an array are the stages of a pipeline, and an element of an
+ * expression at the top of a document, as the operands of an operator {@code $expr} holds are, is a
+ * value the expression builds, whose keys are kept only when the mapping knows them, as those of a
+ * document a stage of an update pipeline writes are
+ */
+ Scope element() {
+ if (kind == Kind.UPDATE) {
+ return child(Kind.UPDATE_PIPELINE, null, path, depth, keyText)
+ }
+ if (kind == Kind.PUSH) {
+ return child(Kind.DATA, null, path, depth + 2, keyText)
+ }
+ if ((kind == Kind.QUERY || kind == Kind.CONDITION || kind == Kind.EXPRESSION || kind == Kind.DATA) && path != null && !path.isEmpty()) {
+ return child(kind, null, path, depth + 1, keyText)
+ }
+ if (kind == Kind.EXPRESSION && path != null) {
+ return child(Kind.WRITTEN, null, path, depth, keyText)
+ }
+ this
+ }
+
+ /**
+ * @return the scope of the value of a key in a document of this scope, with the key as it is recorded
+ */
+ Scope of(String key, BsonValue value) {
+ if (kind == Kind.OPTIONS) {
+ if (ANY_VALUE.contains(key)) {
+ return child(Kind.DATA, null, null, 0, quote(key))
+ }
+ Kind valueKind = DATA_KEYED.contains(key) ? Kind.DATA :
+ UPDATE_KEYED.contains(key) ? Kind.UPDATE :
+ CONDITION_KEYED.contains(key) ? Kind.CONDITION :
+ FIELD_KEYED.contains(key) ? Kind.QUERY :
+ key == LET ? Kind.EXPRESSION : Kind.OPTIONS
+ return child(valueKind, null, Collections.<String>emptyList(), 0, quote(key))
Review Comment:
Fixed in fe3a01a69b. A map-reduce `scope` is now treated like a comment: none of its keys are kept, at any depth, inside documents and arrays, including keys that match a mapped field or start with `$`. Your example is recorded as `"scope": {?: {?: ?}}`.
Tests:
- `MongoCommandRedactorSpec`: "map-reduce scope keys are data even when they match mapped fields or operators" asserts `"scope": {?: {?: ?}, ?: {?: ?}, ?: [{?: {?: ?}}]}`, both with the mapping bound and before it binds.
- `QueryTailCommandListenerSpec`: runs a real `mapReduce` whose map function reads the scope. It checks the result, so the database still received the scope as written, and asserts that `example-secret`, `9876543210` and the queried title are absent from the file.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@grails.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add the query-tail command to find query-heavy pages [grails-core]
Posted by "jamesfredley (via GitHub)" <gi...@apache.org>.
jamesfredley commented on code in PR #16631:
URL: https://github.com/apache/grails-core/pull/16631#discussion_r4390104171
##########
grails-datamapping-core/src/main/groovy/org/apache/grails/datamapping/core/querytail/StandardSqlSyntax.groovy:
##########
@@ -0,0 +1,310 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * https://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.grails.datamapping.core.querytail
+
+import groovy.transform.CompileStatic
+import groovy.transform.PackageScope
+
+/**
+ * The rules of standard SQL, which every database shares: a single-quoted string in which the
+ * quote is written twice, a double-quoted name in which the quote is written twice, a number, a
+ * {@code --} comment to the end of the line, and a {@code /* ... *}{@code /} comment that does not
+ * nest. The syntax of a database extends it and overrides the hooks its database differs on.
+ */
+@CompileStatic
+@PackageScope
+abstract class StandardSqlSyntax implements QueryTailSyntax {
+
+ private static final char SPACE = ' '
+ private static final char ZERO = '0'
+ private static final char NINE = '9'
+ private static final char LOWER_A = 'a'
+ private static final char LOWER_Z = 'z'
+ private static final char UPPER_A = 'A'
+ private static final char UPPER_Z = 'Z'
+ private static final char UNDERSCORE = '_'
+ private static final char ASCII_LAST = 0x7F as char
+
+ private final String name
+
+ protected StandardSqlSyntax(String name) {
+ this.name = name
+ }
+
+ @Override
+ int endOfComment(String sql, int index) {
+ if (sql.startsWith('/*', index)) {
+ return endOfBlockComment(sql, index)
+ }
+ if (sql.startsWith('--', index)) {
+ return endOfLine(sql, index)
+ }
+ NONE
+ }
+
+ @Override
+ int endOfValue(String sql, int index) {
+ char c = sql.charAt(index)
+ if (c == '\'') {
+ return endOfString(sql, index)
+ }
+ if (c == '"' && doubleQuotedStrings()) {
+ return endOfQuoted(sql, index, '"' as char, backslashEscapes(), true)
+ }
+ if (startsNumber(sql, index) && standalone(sql, index)) {
+ return endOfNumber(sql, index)
+ }
+ NONE
+ }
+
+ @Override
+ int endOfQuotedName(String sql, int index) {
Review Comment:
`ALTER USER app_user IDENTIFIED BY "ExampleSecret123"` is written unchanged. Oracle accepts a password in double quotes or bare. Neither is a string literal this scanner redacts.
Withhold a credential statement, or mask the password in both forms. Please assert that password is absent from the capture.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@grails.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add the query-tail command to find query-heavy pages [grails-core]
Posted by "jdaugherty (via GitHub)" <gi...@apache.org>.
jdaugherty commented on code in PR #16631:
URL: https://github.com/apache/grails-core/pull/16631#discussion_r4372245786
##########
grails-data-hibernate7/core/src/main/groovy/org/apache/grails/data/hibernate7/core/querytail/QueryTailStatementInspector.groovy:
##########
@@ -0,0 +1,319 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * https://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.grails.data.hibernate7.core.querytail
+
+import groovy.transform.CompileStatic
+import groovy.transform.PackageScope
+import org.hibernate.SessionFactory
+import org.hibernate.SessionFactoryObserver
+import org.hibernate.cfg.AvailableSettings
+import org.hibernate.cfg.Configuration
+import org.hibernate.dialect.AbstractTransactSQLDialect
+import org.hibernate.dialect.CockroachDialect
+import org.hibernate.dialect.DB2Dialect
+import org.hibernate.dialect.Dialect
+import org.hibernate.dialect.H2Dialect
+import org.hibernate.dialect.HANADialect
+import org.hibernate.dialect.HSQLDialect
+import org.hibernate.dialect.MySQLDialect
+import org.hibernate.dialect.OracleDialect
+import org.hibernate.dialect.PostgreSQLDialect
+import org.hibernate.dialect.SpannerDialect
+import org.hibernate.dialect.SpannerPostgreSQLDialect
+import org.hibernate.engine.spi.SessionFactoryImplementor
+import org.hibernate.resource.jdbc.spi.StatementInspector
+import org.springframework.beans.BeanUtils
+import org.springframework.util.ClassUtils
+
+import org.apache.grails.data.core.querytail.QueryTailCapture
+import org.apache.grails.data.core.querytail.QueryTailSettings
+import org.apache.grails.data.core.querytail.QueryTailSyntax
+import org.grails.datastore.mapping.core.exceptions.ConfigurationException
+
+/**
+ * The statement inspector GORM installs on a session factory when query-tail capture is enabled. It
+ * records every statement Hibernate is about to run to the {@link QueryTailCapture} of the
+ * application, without its values, and returns the statement as it found it: the database runs
+ * it unchanged. An inspector the application configured itself through
+ * {@code hibernate.session_factory.statement_inspector} keeps running: it sees the statement
+ * first, and the statement it returns is the one run and recorded.
+ *
+ * <p>The session factory owns the capture. The inspector observes it: it opens the capture when
+ * the session factory is created, takes from its dialect how the database quotes values, and
+ * closes the capture when the session factory closes, so a session factory that fails to start
+ * leaves no capture open either. An observer the application set on the configuration keeps
+ * being called.</p>
+ *
+ * <p>Nothing of this is installed unless {@link QueryTailSettings#SETTING_ENABLED} is set, so an
+ * application that does not capture pays nothing per statement.</p>
+ *
+ * @since 8.1
+ */
+@CompileStatic
+class QueryTailStatementInspector implements StatementInspector, SessionFactoryObserver {
+
+ @Serial
+ private static final long serialVersionUID = 1L
+
+ /** The community dialects, which are not a dependency, that the inspector tells apart by name. */
+ private static final String COMMUNITY_INFORMIX = 'org.hibernate.community.dialect.InformixDialect'
+ private static final String COMMUNITY_SQL_ANYWHERE = 'org.hibernate.community.dialect.SybaseAnywhereDialect'
+
+ /** The connection source name statements are recorded under. */
+ final String dataSourceName
+
+ /** The file statements are recorded to. */
+ final File file
+
+ /** The inspector the application configured, or null. */
+ final StatementInspector delegate
+
+ /** The session factory observer the application configured, or null. */
+ final SessionFactoryObserver observer
+
+ private transient volatile QueryTailCapture capture
+ private transient boolean closed
+
+ /**
+ * @param dataSourceName the connection source the inspected session factory belongs to
+ * @param file the capture file statements are recorded to
+ * @param delegate the inspector the application configured, or null
+ * @param observer the session factory observer the application configured, or null
+ */
+ QueryTailStatementInspector(String dataSourceName, File file, StatementInspector delegate,
+ SessionFactoryObserver observer) {
+ this.dataSourceName = dataSourceName
+ this.file = file
+ this.delegate = delegate
+ this.observer = observer
+ }
+
+ /**
+ * Installs the inspector on the configuration of a session factory when capture is enabled, as
+ * its statement inspector and as the observer of the session factory it builds.
+ *
+ * @param dataSourceName the connection source the session factory is built for
+ * @param settings the query-tail settings of the connection source
+ * @param configuration the configuration the session factory is built from
+ * @return the installed inspector, or null when capture is not enabled
+ * @throws ConfigurationException when the inspector the application configured cannot be created
+ */
+ static QueryTailStatementInspector install(String dataSourceName, QueryTailSettings settings,
+ Configuration configuration) {
+ if (settings == null || !settings.enabled) {
+ return null
+ }
+ Properties properties = configuration.properties
+ StatementInspector configured = resolve(properties.get(AvailableSettings.STATEMENT_INSPECTOR))
Review Comment:
Follow-up in ea3558f3c6. The capture now chooses the inspector to wrap when the session factory is built, not when the connection source is configured. That covers every way a `configClass` can set an inspector:
- **`setStatementInspector`:** called in the constructor, or in an overridden `buildSessionFactory` before it calls `super` (Hibernate 7).
- **Registry builder:** a setting applied to the builder that an overridden `createStandardServiceRegistryBuilder` returns (Hibernate 5 and 7). Before this change, GORM applied the configuration's properties, which held the capture, over the builder's settings. Once capture was on, the application's inspector stopped running.
The installer now only registers the capture on GORM's configuration. Once the configuration has merged the registry settings, it hands them to the capture. The capture then wraps the inspector the session factory would otherwise run:
- the one set on the configuration, on Hibernate 7;
- else the one the merged registry settings name.
A new data-driven feature on both versions checks, for each way of configuring an inspector, that the inspector that runs with capture on is the one that runs with it off. On b8c346b48c, the registry-builder row fails on both versions, and so does the overridden-`buildSessionFactory` row on Hibernate 7.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@grails.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add the query-tail command to find query-heavy pages [grails-core]
Posted by "jamesfredley (via GitHub)" <gi...@apache.org>.
jamesfredley commented on code in PR #16631:
URL: https://github.com/apache/grails-core/pull/16631#discussion_r4372863783
##########
grails-datamapping-core/src/main/groovy/org/apache/grails/data/core/querytail/QueryTailRequestAttribution.groovy:
##########
@@ -0,0 +1,427 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * https://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.grails.data.core.querytail
+
+import java.lang.reflect.Method
+import java.nio.charset.StandardCharsets
+import java.util.concurrent.atomic.AtomicLong
+import java.util.regex.Pattern
+
+import groovy.transform.CompileStatic
+
+import org.springframework.util.ClassUtils
+import org.springframework.web.context.request.RequestAttributes
+import org.springframework.web.context.request.RequestContextHolder
+import org.springframework.web.context.request.ServletRequestAttributes
+
+/**
+ * Attributes a captured statement to the web request on the current thread. Grails keeps the
+ * request, and the controller and action it resolved, on the thread that serves it, so a statement
+ * run by an interceptor, the action or the view all land on the same page. The first statement of
+ * a request gives it an id that every later statement of that request repeats, which is how the
+ * {@code query-tail} command counts statements per request. Work on any other thread is attributed to
+ * the thread by name.
+ *
+ * <p>The request is recorded as its method and path, with the values in the path replaced by
+ * {@code ?}: the values of the parameters the URL mapping took from it and those of its query, found
+ * whether or not the client percent-encoded them, and any segment that is a number, a UUID or a long
+ * hexadecimal token, so that an id, a token or a key carried in a path never reaches the capture
+ * file. A segment a value is glued into, as a mapping such as {@code /reset/${token}_confirm}
+ * produces, is replaced whole, whether the value shows as sent or decoded, and so is every path
+ * parameter after a semicolon, such as a {@code jsessionid} the container rewrote the URL with.
+ * Until the URL mapping has run, as for a statement of a
+ * filter that runs before the controller is chosen, or for a request that never reaches one, no
+ * segment's role is known and every segment is replaced; only the controller the framework set on
+ * the request says that it has run, never a parameter, which a client can send.</p>
+ *
+ * <p>Spring Web is optional: without it on the classpath every statement is attributed to its
+ * thread.</p>
+ *
+ * @since 8.1
+ */
+@CompileStatic
+class QueryTailRequestAttribution {
+
+ /** The request attribute holding the id this attribution gave a request. */
+ public static final String REQUEST_ID_ATTRIBUTE = 'org.apache.grails.data.core.querytail.REQUEST_ID'
+
+ /** The request attribute Grails sets the controller name in. */
+ public static final String CONTROLLER_NAME_ATTRIBUTE = 'org.grails.CONTROLLER_NAME_ATTRIBUTE'
+
+ /** The request attribute Grails sets the controller namespace in. */
+ public static final String CONTROLLER_NAMESPACE_ATTRIBUTE = 'org.grails.CONTROLLER_NAMESPACE_ATTRIBUTE'
+
+ /** The request attribute Grails sets the action name in. */
+ public static final String ACTION_NAME_ATTRIBUTE = 'org.grails.ACTION_NAME_ATTRIBUTE'
+
+ private static final boolean SPRING_WEB_PRESENT = ClassUtils.isPresent(
+ 'org.springframework.web.context.request.RequestContextHolder', QueryTailRequestAttribution.classLoader)
+
+ private final AtomicLong requestCounter = new AtomicLong()
+ private final String requestIdPrefix
+
+ /**
+ * An attribution whose request ids are {@code r1}, {@code r2} and so on.
+ */
+ QueryTailRequestAttribution() {
+ this('')
+ }
+
+ /**
+ * @param requestIdPrefix what every request id starts with, which tells the requests of one JVM
+ * from those of another writing the same capture
+ */
+ QueryTailRequestAttribution(String requestIdPrefix) {
+ this.requestIdPrefix = requestIdPrefix ?: ''
+ }
+
+ /**
+ * Fills in the request id, page and request of a record from the current thread.
+ *
+ * @param record the record of a statement that just ran on this thread
+ * @return the same record
+ */
+ QueryTailRecord attribute(QueryTailRecord record) {
+ if (SPRING_WEB_PRESENT && WebRequestLookup.attribute(record, requestCounter, requestIdPrefix)) {
+ return record
+ }
+ record.requestId = null
+ record.request = null
+ record.page = QueryTailRecord.THREAD_PAGE_PREFIX + Thread.currentThread().name
+ record
+ }
+
+ /**
+ * Keeps every reference to Spring Web, in a signature as well as in a body, in a class that is only
+ * loaded once Spring Web is known to be present. The outer class can then be loaded, and reflected
+ * on, without Spring Web on the classpath.
+ */
+ @CompileStatic
+ private static class WebRequestLookup {
+
+ private static final String REQUEST_ID_PREFIX = 'r'
+
+ private static final String PLACEHOLDER = '?'
+
+ /** The parameters a URL mapping sets that name code rather than carry a value. */
+ private static final Set<String> MAPPING_PARAMETERS = ['controller', 'action', 'namespace', 'plugin', 'format'] as Set<String>
+
+ /** A segment that is a number, a UUID or a long hexadecimal token is a value whatever the mapping says. */
+ private static final Pattern VALUE_SHAPED = Pattern.compile(
+ '(?i)[0-9]+|[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}|[0-9a-f]{32,}')
+
+ private static final boolean SERVLET_PRESENT = ClassUtils.isPresent(
+ 'jakarta.servlet.http.HttpServletRequest', WebRequestLookup.classLoader)
+
+ /** The {@code getParams()} of a Grails web request, which holds the parameters its URL mapping took, by class. */
+ private static final ClassValue<Method> PARAMETERS = new ClassValue<Method>() {
+ @Override
+ protected Method computeValue(Class<?> type) {
+ try {
+ return type.getMethod('getParams')
+ }
+ catch (NoSuchMethodException ignored) {
+ return null
+ }
+ }
+ }
+
+ /**
+ * Fills in a record from the request on the current thread.
+ *
+ * @param record the record to fill in
+ * @param requestCounter the counter the next request id is taken from
+ * @param requestIdPrefix what every request id starts with
+ * @return whether there is a request on the current thread
+ */
+ static boolean attribute(QueryTailRecord record, AtomicLong requestCounter, String requestIdPrefix) {
+ RequestAttributes attributes = RequestContextHolder.getRequestAttributes()
+ if (attributes == null) {
+ return false
+ }
+ record.requestId = requestId(attributes, requestCounter, requestIdPrefix)
+ record.page = page(attributes)
+ record.request = SERVLET_PRESENT ? ServletRequestLookup.describe(attributes) : null
+ true
+ }
+
+ /**
+ * Builds the request line: the method and the path, with the values in the path, the parameters
+ * the URL mapping took from it among them, replaced.
+ *
+ * @param method the HTTP method
+ * @param path the request path
+ * @param attributes the request, whose mapped parameters are read when it is a Grails web request
+ * @param parameters the parameters of the servlet request, by name, each a {@code String[]}
+ */
+ static String describe(String method, String path, RequestAttributes attributes, Map<?, ?> parameters) {
+ // only the framework sets the controller attribute; a client can send a controller parameter
+ if (attributes.getAttribute(CONTROLLER_NAME_ATTRIBUTE, RequestAttributes.SCOPE_REQUEST) == null) {
+ return "${method} ${withholdPath(path)}".toString()
+ }
+ Set<String> values = new HashSet<>()
+ addValues(values, mappedParameters(attributes))
+ addValues(values, parameters)
+ "${method} ${redactPath(path, values)}".toString()
+ }
+
+ /**
+ * Replaces every segment of a path: before the URL mapping has run, nothing tells a literal
+ * segment from a token, so only the shape of the path is recorded.
+ */
+ private static String withholdPath(String path) {
+ String[] segments = path.split('/', -1)
+ StringBuilder out = new StringBuilder(path.length())
+ for (int i = 0; i < segments.length; i++) {
+ if (i > 0) {
+ out.append('/')
+ }
+ out.append(segments[i].isEmpty() ? segments[i] : PLACEHOLDER)
+ }
+ out.toString()
+ }
+
+ private static String requestId(RequestAttributes attributes, AtomicLong requestCounter, String requestIdPrefix) {
+ synchronized (attributes) {
+ Object existing = attributes.getAttribute(REQUEST_ID_ATTRIBUTE, RequestAttributes.SCOPE_REQUEST)
+ if (existing != null) {
+ return existing.toString()
+ }
+ String id = requestIdPrefix + REQUEST_ID_PREFIX + requestCounter.incrementAndGet()
+ attributes.setAttribute(REQUEST_ID_ATTRIBUTE, id, RequestAttributes.SCOPE_REQUEST)
+ return id
+ }
+ }
+
+ private static String page(RequestAttributes attributes) {
+ Object controller = attributes.getAttribute(CONTROLLER_NAME_ATTRIBUTE, RequestAttributes.SCOPE_REQUEST)
+ if (controller == null) {
+ return null
+ }
+ Object namespace = attributes.getAttribute(CONTROLLER_NAMESPACE_ATTRIBUTE, RequestAttributes.SCOPE_REQUEST)
+ Object action = attributes.getAttribute(ACTION_NAME_ATTRIBUTE, RequestAttributes.SCOPE_REQUEST)
+ StringBuilder page = new StringBuilder()
+ if (namespace != null) {
+ page.append(namespace).append(':')
+ }
+ page.append(controller)
+ if (action != null) {
+ page.append('/').append(action)
+ }
+ page.toString()
+ }
+
+ /**
+ * @return the parameters of a Grails web request, which include those its URL mapping took from
+ * the path, or null for a request that is not one
+ */
+ private static Map<?, ?> mappedParameters(RequestAttributes attributes) {
+ Method parameters = PARAMETERS.get(attributes.getClass())
+ if (parameters == null) {
+ return null
+ }
+ try {
+ Object value = parameters.invoke(attributes)
+ return value instanceof Map ? (Map<?, ?>) value : null
+ }
+ catch (ReflectiveOperationException | RuntimeException ignored) {
+ return null
+ }
+ }
+
+ private static void addValues(Set<String> values, Map<?, ?> parameters) {
+ if (parameters == null) {
+ return
+ }
+ for (Map.Entry<?, ?> entry in parameters.entrySet()) {
+ if (MAPPING_PARAMETERS.contains(String.valueOf(entry.key))) {
+ continue
+ }
+ Object value = entry.value
+ if (value instanceof CharSequence) {
+ values.add(value.toString())
+ }
+ else if (value instanceof Object[]) {
+ for (Object element in (Object[]) value) {
+ if (element instanceof CharSequence) {
+ values.add(element.toString())
+ }
+ }
+ }
+ }
+ }
+
+ /**
+ * Replaces the values in a path: a segment that is one of the values, with or without an
+ * extension, a run of segments that spells a value holding slashes, any segment shaped like a
+ * value, and any segment a value is glued into. The segments are matched as the client sent
+ * them and percent-decoded, since the parameters are decoded while the request URI is not; the
+ * path is written as sent. The path parameters of a segment, after a semicolon, are not part of
+ * what is matched, and each of them is written as the placeholder: neither a session id the
+ * container rewrote the URL with nor any other is a parameter of the request.
+ */
+ private static String redactPath(String path, Set<String> values) {
+ String[] segments = path.split('/', -1)
+ String[] pathParameters = new String[segments.length]
+ String[] decoded = new String[segments.length]
+ boolean[] redacted = new boolean[segments.length]
+ for (int i = 0; i < segments.length; i++) {
+ int semicolon = segments[i].indexOf(';')
+ pathParameters[i] = semicolon < 0 ? '' : withheldPathParameters(segments[i].substring(semicolon))
+ segments[i] = semicolon < 0 ? segments[i] : segments[i].substring(0, semicolon)
+ decoded[i] = decode(segments[i])
+ redacted[i] = isValue(segments[i], values) || isValue(decoded[i], values)
+ }
+ for (String value in values) {
Review Comment:
The documented `/images/$name**.jpg` mapping leaves `/images/customer-secret/private-secret.jpg` unchanged. The `.jpg` suffix stops the whole-value match, and a per-segment check cannot match a value that contains a slash.
Mask a multi-segment value that has a literal prefix or suffix. Please assert that path does not reach the file.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@grails.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add the sql-tail command to find query-heavy pages [grails-core]
Posted by "jdaugherty (via GitHub)" <gi...@apache.org>.
jdaugherty commented on PR #16631:
URL: https://github.com/apache/grails-core/pull/16631#issuecomment-6372063084
Two follow-ups on the syntax split, in ab9131b53a:
- Every file this branch adds is now Groovy under `@CompileStatic`. The redactor, the syntax classes and the two statement inspectors were Java.
- The sql-tail code moved to a package root each jar owns, nested under its project, so no package of one jar nests inside another's and modules can be introduced later: `org.apache.grails.data.core.sqltail` in grails-datamapping-core, `org.apache.grails.data.core.cli.sqltail` in its CLI artifact, and `org.apache.grails.data.hibernate5.core.sqltail` / `org.apache.grails.data.hibernate7.core.sqltail` in the Hibernate cores, following the `org.apache.grails.data.hibernate*.core` packages already there.
The configuration keys, the command and the capture format are unchanged, and the redactor and inspector specs pass as before.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@grails.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add the query-tail command to find query-heavy pages [grails-core]
Posted by "jdaugherty (via GitHub)" <gi...@apache.org>.
jdaugherty commented on PR #16631:
URL: https://github.com/apache/grails-core/pull/16631#issuecomment-6374258809
Two changes to the branch itself, now a single commit, b429e3794c:
- **Renamed to `query-tail`.** With Cypher and MongoDB commands captured alongside SQL, `sql-tail` no longer said what it does. The command is `grails query-tail`, the flag `run-app --query-tail`, the settings `grails.gorm.queryTail.enabled` and `.file`, the capture file `build/query-tail/capture.log`, the packages `org.apache.grails.data.*.querytail`, and the classes `QueryTail*`; the CLI's `SqlStatementType` and `SqlStatementCollapser` became `StatementType` and `StatementCollapser` for the same reason. The guide section, the reference page, the What's New entry and the configuration metadata follow. The PR title and description are updated.
- **Squashed.** The fourteen commits are one, so the earlier line-comment anchors point at history that is no longer on the branch; the threads themselves are all resolved.
Also in this commit, from a review pass over the Neo4j/MongoDB work: a client-level MongoDB `bulkWrite` now has the keys of its `document` and `updateMods` entries resolved against the mapping like `insert` and `update` do, so a map property's keys cannot leak through it; a capture path that cannot be resolved raises the documented `UncheckedIOException` rather than a sneaky `IOException` through the Java Neo4j factory; the run-app flag, the command description and the configuration metadata describe all three stores; and the connection-source and factory specs of the MongoDB and Hibernate modules gained the query-tail settings and listener lifecycle cases.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@grails.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add the query-tail command to find query-heavy pages [grails-core]
Posted by "jamesfredley (via GitHub)" <gi...@apache.org>.
jamesfredley commented on code in PR #16631:
URL: https://github.com/apache/grails-core/pull/16631#discussion_r4367356838
##########
grails-datamapping-core/src/main/groovy/org/apache/grails/data/core/querytail/QueryTailCapture.groovy:
##########
@@ -0,0 +1,356 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * https://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.grails.data.core.querytail
+
+import java.nio.ByteBuffer
+import java.nio.channels.FileChannel
+import java.nio.channels.FileLock
+import java.nio.channels.OverlappingFileLockException
+import java.nio.charset.StandardCharsets
+import java.nio.file.Files
+import java.nio.file.StandardOpenOption
+import java.util.concurrent.atomic.AtomicBoolean
+
+import groovy.transform.CompileStatic
+import groovy.util.logging.Slf4j
+
+/**
+ * Writes the statements an application runs to the capture file the {@code query-tail} command
+ * tails. Each session factory, Neo4j driver or MongoDB client that captures opens its own capture
+ * and closes it when it closes; the captures of one file share it, so every data source of an
+ * application writes to the one file. The file is truncated when the first capture opens it, so it
+ * holds the statements of the current run only, and its first line names that run. It is closed
+ * when the last capture closes, and opened afresh by the next one. Each statement is written whole
+ * and at once, so that the command sees it as soon as it is recorded.
+ *
+ * <p>JVMs that run at the same time with capture enabled, such as the parallel forks of a test task,
+ * share the file: a lock file beside it tells the first to start, which truncates the file and names
+ * the run, from the others, which append to that run without truncating it and number their
+ * requests apart, so that the command reads one run from all of them.</p>
+ *
+ * <p>A statement is recorded without its values: {@link QueryTailRedactor} replaces its literals and
+ * drops its comments before it is written, in the {@link #syntax} of the database it runs on. A
+ * statement that is not text, such as a MongoDB command, has its values taken out by whoever records
+ * it, through {@link #recordRedacted}.</p>
+ *
+ * <p>A capture is only opened when {@link QueryTailSettings#SETTING_ENABLED} is set; an application
+ * started without it never reaches this class.</p>
+ *
+ * @since 8.1
+ */
+@CompileStatic
+@Slf4j
+class QueryTailCapture implements Closeable {
+
+ private static final Map<String, CaptureFile> OPEN = new HashMap<>()
+
+ /** The file the statements are written to. */
+ final File file
+
+ /** The syntax of the database the statements of this capture run on, which their values are found by. */
+ volatile QueryTailSyntax syntax = QueryTailSyntax.UNKNOWN
+
+ private final CaptureFile captureFile
+ private final AtomicBoolean closed = new AtomicBoolean()
+
+ private QueryTailCapture(File file, CaptureFile captureFile) {
+ this.file = file
+ this.captureFile = captureFile
+ }
+
+ /**
+ * Opens a capture of a file. The first capture of a file truncates it; a capture opened while
+ * another of the same file is open shares it. Every capture opened must be closed.
+ *
+ * @param file the capture file
+ * @return the capture
+ * @throws UncheckedIOException when the file cannot be created
+ */
+ static QueryTailCapture open(File file) {
+ String key
+ try {
+ key = file.canonicalPath
+ }
+ catch (IOException e) {
+ throw new UncheckedIOException("Cannot resolve the query-tail capture file ${file}: ${e.message}", e)
+ }
+ synchronized (OPEN) {
+ CaptureFile captureFile = OPEN.get(key)
+ if (captureFile == null) {
+ captureFile = new CaptureFile(file, key)
+ OPEN.put(key, captureFile)
+ if (captureFile.joined) {
+ log.info('Capturing statements to {} for query-tail, in the run of another JVM writing it', file)
+ }
+ else {
+ log.info('Capturing statements to {} for query-tail', file)
+ }
+ }
+ captureFile.captures++
+ return new QueryTailCapture(file, captureFile)
+ }
+ }
+
+ /**
+ * Opens the capture the settings name.
+ *
+ * @param settings the query-tail settings
+ * @return the capture
+ */
+ static QueryTailCapture open(QueryTailSettings settings) {
+ open(settings.resolveFile())
+ }
+
+ /**
+ * Records a statement that is about to run on the current thread, without its values.
+ *
+ * @param dataSource the connection source name the statement runs against
+ * @param sql the statement
+ */
+ void record(String dataSource, String sql) {
+ write(dataSource, QueryTailRedactor.redact(sql, syntax))
+ }
+
+ /**
+ * Records a statement that is about to run on the current thread and that holds no value any
+ * more: a MongoDB command, which is a document rather than text, is redacted field by field by
+ * the listener that records it. Nothing of the statement is changed here.
+ *
+ * @param dataSource the connection source name the statement runs against
+ * @param statement the statement, with its values already taken out
+ */
+ void recordRedacted(String dataSource, String statement) {
+ write(dataSource, statement)
+ }
+
+ private void write(String dataSource, String statement) {
+ if (closed.get()) {
+ return
+ }
+ QueryTailRecord record = new QueryTailRecord(timestamp: System.currentTimeMillis(), dataSource: dataSource,
+ sql: statement)
+ captureFile.write(captureFile.attribution.attribute(record))
+ }
+
+ /**
+ * @return whether this capture is still writing to its file
+ */
+ boolean isOpen() {
+ !closed.get() && captureFile.open
+ }
+
+ /**
+ * Closes this capture. The file is closed once every capture of it is.
+ */
+ @Override
+ void close() {
+ if (!closed.compareAndSet(false, true)) {
+ return
+ }
+ boolean last
+ synchronized (OPEN) {
+ last = --captureFile.captures == 0
+ if (last && OPEN.get(captureFile.key).is(captureFile)) {
+ OPEN.remove(captureFile.key)
+ }
+ }
+ if (last) {
+ captureFile.close()
+ }
+ }
+
+ /**
+ * A capture file and the captures writing to it. The file is written by one channel, a line at a
+ * time, and the requests of every data source are numbered by one attribution, so that two
+ * requests never share an id.
+ *
+ * <p>A lock file beside the capture coordinates the JVMs writing it. Every writer holds a shared
+ * lock on the {@link #WRITING} byte of the lock file while it writes. A starting writer first
+ * takes the {@link #STARTING} byte exclusively, so that starters take turns, then tries the
+ * {@link #WRITING} byte exclusively: when it gets it, no other JVM is writing, and it truncates the
+ * file and writes the header of a new run; when it does not, it joins the run being written. Either
+ * way it then holds the {@link #WRITING} byte shared until it closes. A lock file that cannot be
+ * created, as beside a device, leaves the writer uncoordinated, as the only one.</p>
+ */
+ @CompileStatic
+ private static class CaptureFile {
+
+ /** The byte of the lock file a starting writer holds, exclusively, while it decides whether to truncate. */
+ private static final long STARTING = 0L
+
+ /** The byte of the lock file every writer holds, shared, while it writes. */
+ private static final long WRITING = 1L
+
+ final File file
+ final String key
+ final QueryTailRequestAttribution attribution
+
+ /** Whether this JVM joined the run of another JVM writing the file, rather than starting one. */
+ final boolean joined
+
+ /** How many captures are open on the file; guarded by {@link QueryTailCapture#OPEN}. */
+ int captures
+
+ private FileChannel channel
+ private FileChannel lockChannel
+ private FileLock writing
+
+ CaptureFile(File file, String key) {
+ this.file = file
+ this.key = key
+ File parent = file.absoluteFile.parentFile
+ boolean first
+ try {
+ if (parent != null && !parent.isDirectory() && !parent.mkdirs() && !parent.isDirectory()) {
+ throw new IOException("Cannot create the query-tail capture directory ${parent}")
+ }
+ lockChannel = openLockFile(file)
+ FileLock starting = lockChannel == null ? null : lockChannel.lock(STARTING, 1L, false)
+ try {
+ FileLock alone = lockChannel == null ? null : lock(lockChannel, false, true)
+ first = lockChannel == null || alone != null
+ alone?.release()
+ if (first) {
+ Files.newByteChannel(file.toPath(), StandardOpenOption.CREATE, StandardOpenOption.WRITE,
+ StandardOpenOption.TRUNCATE_EXISTING).close()
+ }
+ channel = FileChannel.open(file.toPath(), StandardOpenOption.CREATE, StandardOpenOption.WRITE,
+ StandardOpenOption.APPEND)
+ writing = lockChannel == null ? null : lock(lockChannel, true, false)
+ }
+ finally {
Review Comment:
The first JVM releases `STARTING` before it writes the run header. A second JVM can append a statement first, so the header is not the first line and a later restart cannot identify the run.
Hold the startup lock until the header is written.
##########
grails-data-mongodb/core/src/main/groovy/org/apache/grails/data/mongo/core/querytail/MongoCommandRedactor.groovy:
##########
@@ -0,0 +1,237 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * https://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.grails.data.mongo.core.querytail
+
+import groovy.transform.CompileStatic
+import org.bson.BsonDocument
+import org.bson.BsonValue
+
+import org.apache.grails.data.core.querytail.QueryTailRedactor
+
+/**
+ * Takes the values out of a MongoDB command before it is captured. A command is a document, so
+ * there is nothing to parse: every value becomes {@link QueryTailRedactor#PLACEHOLDER}, and the
+ * keys, which are the options of the command, its operators and the field names of the collection,
+ * are kept, except where a key may be data. The keys of a document stored in a map property, or in
+ * a property or a collection the {@linkplain MongoMappedFields mapping} does not know, are recorded
+ * as the placeholder too, and so is every segment of a dotted path past such a key. The entries the
+ * driver adds for its session, the cluster time and the transaction, which say nothing about the
+ * command, are left out.
+ *
+ * <p>A command is recorded as its name, the database and collection it runs against, and the
+ * document of its remaining entries: {@code find shop.book {"filter": {"title": ?}, "limit": ?}}.</p>
+ *
+ * @since 8.1
+ */
+@CompileStatic
+final class MongoCommandRedactor {
+
+ private static final String PLACEHOLDER = QueryTailRedactor.PLACEHOLDER
+
+ private static final String DATABASE = '$db'
+
+ private static final String COLLECTION = 'collection'
+
+ /** The entries the driver adds, which describe its session and transaction rather than the command. */
+ private static final Set<String> LEFT_OUT = ['lsid', '$clusterTime', 'txnNumber', 'autocommit',
+ 'startTransaction', '$readPreference', 'apiVersion', 'apiStrict', 'apiDeprecationErrors'] as Set<String>
+
+ /**
+ * The options whose document, or documents, are keyed by the fields of the collection, wherever they appear,
+ * including those of the operations of a client-level {@code bulkWrite}.
+ */
+ private static final Set<String> FIELD_KEYED = ['filter', 'query', 'q', 'u', 'update', 'document', 'documents',
+ 'updateMods', 'pipeline', 'sort', 'projection', 'fields', 'key', 'hint', 'min', 'max', 'arrayFilters',
+ 'partialFilterExpression', 'wildcardProjection'] as Set<String>
+
+ private MongoCommandRedactor() {
+ }
+
+ /**
+ * Redacts a command.
+ *
+ * @param database the database the command runs against
+ * @param commandName the name of the command, which is its first key
+ * @param command the command as the driver sends it
+ * @param fields the fields the mapping of the datastore knows, or null before the datastore is built
+ * @return the command with its values replaced
+ */
+ static String redact(String database, String commandName, BsonDocument command, MongoMappedFields fields) {
+ String collection = null
+ String collectionKey = null
+ BsonValue first = command.get(commandName)
+ if (first != null && first.isString()) {
+ collection = first.asString().value
+ collectionKey = commandName
+ }
+ else if (command.get(COLLECTION)?.isString()) {
+ collection = command.getString(COLLECTION).value
+ collectionKey = COLLECTION
+ }
+ StringBuilder out = new StringBuilder(commandName)
+ if (database != null || collection != null) {
+ out.append(' ')
+ if (database != null) {
+ out.append(database)
+ }
+ if (collection != null) {
+ out.append(database != null ? '.' : '').append(collection)
+ }
+ }
+ Scope options = Scope.options(collection, fields)
+ StringBuilder body = new StringBuilder()
+ for (Map.Entry<String, BsonValue> entry in command.entrySet()) {
+ String key = entry.key
+ if (key == DATABASE || key == collectionKey || LEFT_OUT.contains(key)) {
+ continue
+ }
+ body.append(body.length() == 0 ? '{' : ', ')
+ appendEntry(body, key, entry.value, options)
+ }
+ if (body.length() > 0) {
+ out.append(' ').append(body).append('}')
+ }
+ out.toString()
+ }
+
+ private static void appendEntry(StringBuilder out, String key, BsonValue value, Scope scope) {
+ Scope child = scope.of(key)
+ out.append(child.keyText).append(': ')
+ appendValue(out, value, child)
+ }
+
+ private static void appendValue(StringBuilder out, BsonValue value, Scope scope) {
+ if (value.isDocument()) {
+ out.append('{')
+ boolean first = true
+ for (Map.Entry<String, BsonValue> entry in value.asDocument().entrySet()) {
+ if (!first) {
+ out.append(', ')
+ }
+ first = false
+ appendEntry(out, entry.key, entry.value, scope)
+ }
+ out.append('}')
+ }
+ else if (value.isArray()) {
+ out.append('[')
+ boolean first = true
+ for (BsonValue element in value.asArray()) {
+ if (!first) {
+ out.append(', ')
+ }
+ first = false
+ appendValue(out, element, scope)
+ }
+ out.append(']')
+ }
+ else {
+ out.append(PLACEHOLDER)
+ }
+ }
+
+ private static String quote(String key) {
+ StringBuilder out = new StringBuilder(key.length() + 2).append('"')
+ for (int i = 0; i < key.length(); i++) {
+ char c = key.charAt(i)
+ if (c == '"' || c == '\\') {
+ out.append('\\').append(c)
+ }
+ else if (c < 0x20) {
+ out.append(String.format('\\u%04x', (int) c))
+ }
+ else {
+ out.append(c)
+ }
+ }
+ out.append('"').toString()
+ }
+
+ /**
+ * Where a document sits in the command: among the options of the command, whose keys are kept
+ * and name what their documents hold, or in a document keyed by fields, where a key is kept when
+ * it is an operator or a field the mapping knows at its path.
+ */
+ @CompileStatic
+ private static final class Scope {
+
+ final String collection
+ final MongoMappedFields fields
+ /** The field path of the document; null in a document keyed by fields the mapping does not know. */
+ final List<String> path
+ final boolean optionKeyed
+ final String keyText
+
+ private Scope(String collection, MongoMappedFields fields, List<String> path, boolean optionKeyed,
+ String keyText) {
+ this.collection = collection
+ this.fields = fields
+ this.path = path
+ this.optionKeyed = optionKeyed
+ this.keyText = keyText
+ }
+
+ static Scope options(String collection, MongoMappedFields fields) {
+ new Scope(collection, fields, Collections.<String>emptyList(), true, null)
+ }
+
+ /**
+ * @return the scope of the value of a key in a document of this scope, with the key as it is recorded
+ */
+ Scope of(String key) {
+ if (optionKeyed) {
+ boolean fieldKeyed = FIELD_KEYED.contains(key)
+ return new Scope(collection, fields, Collections.<String>emptyList(), !fieldKeyed, quote(key))
+ }
+ if (key.startsWith('$')) {
+ return new Scope(collection, fields, path, false, quote(key))
+ }
+ List<String> current = path
+ String[] segments = key.split('\\.', -1)
+ StringBuilder recorded = new StringBuilder()
+ boolean anyKept = false
+ for (int i = 0; i < segments.length; i++) {
+ String segment = segments[i]
+ boolean kept
+ if (current == null) {
+ kept = false
+ }
+ else if (MongoMappedFields.isIndex(segment)) {
Review Comment:
A numeric key in a map is kept as if it were an array index. `attrs: {"1234567890": "value"}` writes `1234567890` into the capture. The same happens for a dotted path.
Mask map keys. Keep a numeric segment only when the parent is a known indexed collection. Please assert that key does not reach the file.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@grails.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add the query-tail command to find query-heavy pages [grails-core]
Posted by "jdaugherty (via GitHub)" <gi...@apache.org>.
jdaugherty commented on PR #16631:
URL: https://github.com/apache/grails-core/pull/16631#issuecomment-6398454890
@jamesfredley, review 5643345657 is addressed in f69625477f, still a single commit on 9.0.x.
- **`$` key inside `$eq`**: in a condition, the operand of `$eq`, `$ne`, `$gt`, `$gte`, `$lt`, `$lte`, `$in`, `$nin` and `$all` is now data, so `{"attrs":{"$eq":{"$example-secret":"value"}}}` records `{"attrs": {"$eq": {?: ?}}}`. In aggregation expressions (`$expr`, the `let` of a command, and stages such as `$project` and `$group`), the operators of a comparison's operands are still kept. The listener spec runs your filter and an `$expr` aggregation against a real mongod and asserts that the key does not reach the file.
- **Same class, also fixed**: keys of `comment`/`$comment` documents, update constants (`c`), `$jsonSchema`, the `validator` and text-index `weights` documents, and a `$literal` in a command's `let`.
The thread is resolved with the details.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@grails.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add the query-tail command to find query-heavy pages [grails-core]
Posted by "jamesfredley (via GitHub)" <gi...@apache.org>.
jamesfredley commented on code in PR #16631:
URL: https://github.com/apache/grails-core/pull/16631#discussion_r4371526281
##########
grails-datamapping-core/src/cli/groovy/org/apache/grails/data/core/cli/querytail/QueryTailFileReader.groovy:
##########
@@ -0,0 +1,259 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * https://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.grails.data.core.cli.querytail
+
+import java.nio.charset.StandardCharsets
+
+import groovy.transform.CompileStatic
+
+import org.apache.grails.data.core.querytail.QueryTailRecord
+
+/**
+ * Follows a capture file the way {@code tail -f} does: each poll returns the records written
+ * since the last one, up to a {@link #CHUNK chunk} of the file at a time, so that a capture of any
+ * size is read in pieces; a reader that is {@link #isBehind() behind} has more to read without
+ * waiting. A line that has no terminator yet is left for the next poll, so a record the
+ * application is still writing is never read in half, and a line longer than a chunk is dropped
+ * up to its end, as a line that is not a record is. An application that starts again
+ * truncates the file and names its new run in the first line; when that line names another run,
+ * or the file shrinks, reading starts over from the beginning. The run tells a restart apart from
+ * a file that grew past where the reader was before it was next polled. A file truncated between
+ * being measured and being read, so that the read comes up short, is a restart too: it is read again
+ * from the beginning in the same poll. A poll that finds the file as long as it left it reads nothing,
+ * not even its first line, so an idle command costs nothing; a restart that has written nothing yet
+ * is noticed with its first statement.
+ *
+ * @since 8.1
+ */
+@CompileStatic
+class QueryTailFileReader implements Closeable {
+
+ private static final byte NEWLINE = (byte) 10
+
+ /** How much of the file the first line is looked for in; a header is far shorter. */
+ private static final int HEADER_LIMIT = 256
+
+ /** How much of the file one poll reads at most, in bytes. */
+ public static final int CHUNK = 1024 * 1024
+
+ final File file
+ final int chunk
+ private long position
+ private long measured = -1L
+ private String run
+ private boolean restarted
+ private boolean behind
+ private boolean skipping
+
+ /**
+ * @param file the capture file
+ * @param fromStart whether to read what the file already holds, or only what is written from now on
+ */
+ QueryTailFileReader(File file, boolean fromStart) {
+ this(file, fromStart, CHUNK)
+ }
+
+ /**
+ * @param file the capture file
+ * @param fromStart whether to read what the file already holds, or only what is written from now on
+ * @param chunk how much of the file one poll reads at most, in bytes
+ */
+ QueryTailFileReader(File file, boolean fromStart, int chunk) {
+ this.file = file
+ this.chunk = chunk
+ this.position = fromStart ? 0 : file.length()
+ this.run = file.exists() ? readRun() : null
+ // opened in the middle of a line being written: the rest of that line is not a record
+ this.skipping = position > 0 && !endsLine(readFrom(position - 1, position))
+ }
+
+ /**
+ * @return the records written since the last poll, or since the reader was opened, up to a chunk of the file
+ */
+ List<QueryTailRecord> poll() {
+ restarted = false
+ behind = false
+ // a read that comes up short is retried once, from the start; the next poll takes it from there
+ for (int attempt = 0; attempt < 2; attempt++) {
+ long length = file.length()
+ if (length == position && length == measured) {
Review Comment:
If the application restarts between polls and the new capture is the same number of bytes, this returns before it reads the new run header. `poll()` then reports no records and no restart, so the summary stays on the old run.
Check the header, or another change marker, before treating an unchanged length as idle.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@grails.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add the query-tail command to find query-heavy pages [grails-core]
Posted by "jdaugherty (via GitHub)" <gi...@apache.org>.
jdaugherty commented on PR #16631:
URL: https://github.com/apache/grails-core/pull/16631#issuecomment-6417931886
@jamesfredley, I pushed 6b3f7c9935 over 28b2772301, still a single commit on 9.0.x. It fixes four more problems:
- **Inspector names (H5 and H7):** Hibernate can select a statement inspector by a name registered with its `StrategySelector`. With capture on, such a name failed the start, because the capture loaded every string as a class name. The capture now resolves `hibernate.session_factory.statement_inspector` through the session factory's strategy selector, as Hibernate does. A value Hibernate rejects, such as an empty or space-padded name, now fails the same way with capture on as without it.
- **Observer set during the build (H5 and H7):** the capture took the configuration's observer slot when the connection source was created. A `hibernate.configClass` that set its own observer in `buildSessionFactory()` replaced it, and the capture never closed. The capture now chains the application's observer when the session factory is built. On H5 that happens before the registry-closing observer wraps it, and building the same configuration again no longer chains the capture to itself.
- **Symbolic links across JVMs:** the lock file was named after the path each JVM opened. Two JVMs naming the capture through a link and through its target took different locks, and the second truncated the first's run. The lock is now keyed by the canonical path. Links are followed by hand first, so a link to a file that doesn't exist yet resolves too.
- **CLI memory:** the summary kept a statement-count map for every request in every mode. `--statements` now keeps only each request's key. Page mode folds each request's totals into its page as statements arrive, and keeps per-request counts only for the last 1000 active requests of each page. The command reference describes this.
Tests:
- **H5/H7 `QueryTailStatementInspectorSpec`:** a named inspector with and without capture, names Hibernate rejects, a configured class that isn't an inspector, and an observer set in the constructor or during the build. The observer cases check that the capture closes and that reopening the file starts a new run.
- **H5/H7 `HibernateConnectionSourceFactorySpec`:** builds one configuration twice with capture off and on. Without the self-chaining guard, the capture-on case overflows the stack.
- **H5 `HibernateMappingContextConfigurationSpec`:** checks the service registry is still destroyed for both observer timings, with capture off and on.
- **`QueryTailCaptureSpec`:** two JVMs write through the file and through a link, in both directions. A third feature opens a link before its file exists, in one JVM.
- **First failed write:** this feature now runs in a child JVM with a zero file-size limit instead of a link to `/dev/full`. It runs on macOS and on CI without root.
- **`QueryTailRunnerSpec`:** summarizes 350,000 records, each its own request, in a child JVM limited to 96 MB, in both modes, plus a one-request control. The previous summary needed 192 MB for that capture.
Locally, the full `:grails-data-hibernate5-core` and `:grails-data-hibernate7-core` suites pass, and so does `:grails-datamapping-core:check`. The MongoDB and Neo4j query-tail specs pass too. `QueryTailCaptureSpec` also passes in a Linux container, as root and as a non-root user.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@grails.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add the query-tail command to find query-heavy pages [grails-core]
Posted by "jdaugherty (via GitHub)" <gi...@apache.org>.
jdaugherty commented on PR #16631:
URL: https://github.com/apache/grails-core/pull/16631#issuecomment-6418642193
@jamesfredley, I pushed dcc4f0646e over 6b3f7c9935, still a single commit on 9.0.x. It fixes two file-handling problems:
- **Records longer than a read chunk:** the command reads the capture at most 1 MiB per poll and dropped any line without a line end in a full chunk. That included valid records, such as a redacted MongoDB `$in` over 350,000 ids, so large commands vanished from the counts. A line that runs past a chunk is now carried from poll to poll until it ends. Up to 16 chunks (16 MiB by default) are kept whole. A longer line still counts, with its request and page, and its statement is cut at the limit and marked `... (cut)`. A restart drops a line that was still being read.
- **Capture through a symbolic link into missing directories:** the capture created the directory of the link, not of the file it points to. After a clean build removed `build/query-tail`, opening through the link failed the start. The capture now creates the directories of the resolved file, where the lock file goes too.
Tests:
- **`QueryTailFileReaderSpec`:** a line longer than a chunk read across polls, a line past the limit cut with its metadata kept, a long line read once its writer finishes it, and a restart while a long line is pending. These replace the feature that expected long lines to be dropped.
- **`QueryTailRunnerSpec`:** writes a 350,000-id `$in` through the capture API, followed by a small command, and checks that `--all --once --statements` counts both.
- **`QueryTailCaptureSpec`:** opens a capture by its own name and through a link, with the target's directories missing.
- Against the previous reader and capture, the new reader and directory cases fail; with the fix they pass.
`:grails-datamapping-core:check` passes locally. `QueryTailCaptureSpec` also passes in a Linux container, as root and as a non-root user.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@grails.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add the sql-tail command to find query-heavy pages [grails-core]
Posted by "jdaugherty (via GitHub)" <gi...@apache.org>.
jdaugherty commented on code in PR #16631:
URL: https://github.com/apache/grails-core/pull/16631#discussion_r4362041015
##########
grails-datamapping-core/src/main/groovy/org/grails/datastore/gorm/jdbc/sqltail/SqlTailRedactor.java:
##########
@@ -0,0 +1,452 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * https://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.grails.datastore.gorm.jdbc.sqltail;
+
+import java.util.ArrayList;
+import java.util.List;
+
+/**
+ * Takes the values out of a statement before it is captured. Every string, number and
+ * dollar-quoted literal becomes {@link #PLACEHOLDER} and every comment is dropped, so that a value
+ * written into the SQL text, a password in a native query or a key in a comment, never reaches the
+ * capture file. Only the recorded copy is redacted: the statement the database runs is not changed.
+ *
+ * <p>Databases disagree on parts of the lexical syntax: whether a backslash escapes a quote, whether
+ * comments nest, whether double quotes delimit an identifier or a string, whether a quote written
+ * twice escapes it or ends one string and starts another, whether a quote written three times
+ * opens a string that quotes can appear in, whether {@code #} starts a comment, and whether
+ * {@code #} and {@code @} are part of a name or operators. A statement is redacted under every
+ * reading its {@link Syntax} allows, and is recorded only when all of them give the same text. When
+ * they do not, or when a literal or a comment never ends, the statement is withheld:
+ * {@link #WITHHELD} is recorded in its place, never the statement.</p>
+ *
+ * @since 8.1
+ */
+public final class SqlTailRedactor {
+
+ /** What a literal is replaced with. */
+ public static final String PLACEHOLDER = "?";
+
+ /** What is recorded in place of a statement whose values cannot be told apart from its SQL. */
+ public static final String WITHHELD = "(statement withheld: its values could not be redacted)";
+
+ private static final boolean[] NO = {false};
+ private static final boolean[] YES = {true};
+ private static final boolean[] EITHER = {false, true};
+
+ /**
+ * The lexical syntax of the database a statement runs on, as far as finding its values goes.
+ */
+ public enum Syntax {
+
+ /**
+ * Not known, or a database that can read double-quoted text as a string although it quotes
+ * identifiers with double quotes, such as SQLite and Informix: a statement is recorded only when
+ * the readings of every syntax below agree.
+ */
+ UNKNOWN,
+
+ /**
+ * Standard SQL, where double quotes delimit identifiers: H2, PostgreSQL, Oracle, DB2, HSQLDB and HANA.
+ * They differ on {@code #} and {@code @}, which are operators in PostgreSQL and part of a name in Oracle
+ * and DB2, so a statement that has one next to a number is withheld.
+ */
+ STANDARD,
+
+ /** MySQL and MariaDB: double quotes delimit strings, and {@code #} and {@code -- } start comments. */
+ MYSQL,
+
+ /**
+ * Microsoft SQL Server and Sybase: square brackets delimit identifiers, comments nest, a dollar before
+ * digits is an amount, and double quotes delimit a string when {@code QUOTED_IDENTIFIER} is off, as it is
+ * by default on Sybase.
+ */
+ TRANSACT_SQL,
+
+ /**
+ * GoogleSQL, the dialect of Cloud Spanner: backticks delimit identifiers, single and double quotes
+ * delimit strings, a string that opens with its quote written three times ends only at the quote
+ * written three times again and holds any other quote, a backslash always escapes the character
+ * after it, a quote written twice is two strings, and {@code #} starts a comment.
+ */
+ GOOGLE_SQL
+ }
+
+ private SqlTailRedactor() {
+ }
+
+ /**
+ * Redacts a statement.
+ *
+ * @param sql the statement
+ * @param syntax the syntax of the database it runs on
+ * @return the statement with its literals replaced and its comments dropped, or {@link #WITHHELD}
+ */
+ public static String redact(String sql, Syntax syntax) {
+ if (sql == null) {
+ return null;
+ }
+ String redacted = null;
+ for (Reading reading : readings(sql, syntax == null ? Syntax.UNKNOWN : syntax)) {
+ String candidate = reading.redact(sql);
+ if (candidate == null || (redacted != null && !redacted.equals(candidate))) {
+ return WITHHELD;
+ }
+ redacted = candidate;
+ }
+ return redacted;
+ }
+
+ /**
+ * The readings a statement has to agree under. A reading that depends on a character the
+ * statement does not contain is the same as the one that does not, so it is left out.
+ */
+ private static List<Reading> readings(String sql, Syntax syntax) {
+ boolean[] backslash = sql.indexOf('\\') >= 0 ? EITHER : NO;
+ int comment = sql.indexOf("/*");
+ boolean[] nesting = comment >= 0 && sql.indexOf("/*", comment + 2) >= 0 ? EITHER : NO;
+ boolean[] symbols = sql.indexOf('#') >= 0 || sql.indexOf('@') >= 0 ? EITHER : YES;
+ List<Reading> readings = new ArrayList<>();
+ if (syntax == Syntax.STANDARD || syntax == Syntax.UNKNOWN) {
+ for (boolean backslashEscapes : backslash) {
+ for (boolean nestedComments : nesting) {
+ for (boolean symbolsInIdentifiers : symbols) {
+ readings.add(new Reading(Syntax.STANDARD, backslashEscapes, nestedComments,
+ symbolsInIdentifiers));
+ }
+ }
+ }
+ }
+ if (syntax == Syntax.MYSQL || syntax == Syntax.UNKNOWN) {
+ for (boolean backslashEscapes : backslash) {
+ readings.add(new Reading(Syntax.MYSQL, backslashEscapes, false, true));
+ }
+ }
+ if (syntax == Syntax.TRANSACT_SQL || syntax == Syntax.UNKNOWN) {
+ readings.add(new Reading(Syntax.TRANSACT_SQL, false, true, true));
+ }
+ if (syntax == Syntax.GOOGLE_SQL || syntax == Syntax.UNKNOWN) {
+ readings.add(new Reading(Syntax.GOOGLE_SQL, true, false, true));
+ }
+ return readings;
+ }
+
+ /**
+ * One way of reading a statement. MySQL and GoogleSQL delimit identifiers with backticks and start a
+ * comment with {@code #}; only MySQL needs whitespace after {@code --}; only Transact-SQL delimits
+ * identifiers with square brackets. Outside standard SQL, double quotes can delimit a string: in
+ * MySQL unless {@code ANSI_QUOTES} is set, in Transact-SQL when {@code QUOTED_IDENTIFIER} is off, and
+ * in GoogleSQL always. Either way the quoted run ends at the same place, so it is replaced as a
+ * value. A GoogleSQL string is read on its own terms: a quote written twice ends it, a backslash
+ * escapes, and a quote written three times opens a string that ends only at the next three. In
+ * Transact-SQL a dollar before digits prefixes an amount, which is a value; elsewhere it prefixes a
+ * parameter or a name.
+ *
+ * @param syntax the syntax read in, never {@link Syntax#UNKNOWN}
+ * @param backslashEscapes whether a backslash escapes the character after it in a string
+ * @param nestedComments whether a block comment can contain another
+ * @param symbolsInIdentifiers whether {@code #} and {@code @} are part of a name, as in Oracle, DB2 and
+ * Transact-SQL, rather than operators, as in PostgreSQL, where a number after one is a value
+ */
+ private record Reading(Syntax syntax, boolean backslashEscapes, boolean nestedComments,
+ boolean symbolsInIdentifiers) {
+
+ /**
+ * @return the redacted statement, or null when a literal, a quoted identifier or a comment never ends
+ */
+ String redact(String sql) {
+ boolean mysql = syntax == Syntax.MYSQL;
+ boolean googleSql = syntax == Syntax.GOOGLE_SQL;
+ boolean doubleQuotedValue = syntax != Syntax.STANDARD;
+ int length = sql.length();
+ StringBuilder out = new StringBuilder(length);
+ int i = 0;
+ while (i < length) {
+ char c = sql.charAt(i);
+ char next = i + 1 < length ? sql.charAt(i + 1) : 0;
+ char previous = i > 0 ? sql.charAt(i - 1) : 0;
+ int dollarTag = c == '$' && !isIdentifierPart(previous) ? dollarTagLength(sql, i) : 0;
+ int end;
+ if (googleSql && (c == '\'' || c == '"')) {
+ end = endOfGoogleSqlString(sql, i);
+ out.append(PLACEHOLDER);
+ }
+ else if (c == '\'') {
+ end = endOfQuotedLiteral(sql, i);
+ out.append(PLACEHOLDER);
+ }
+ else if (c == '"') {
+ end = endOfQuoted(sql, i, '"', mysql && backslashEscapes, true);
+ if (end >= 0) {
+ out.append(doubleQuotedValue ? PLACEHOLDER : sql.substring(i, end));
+ }
+ }
+ else if (c == '`' && (mysql || googleSql)) {
+ end = endOfQuoted(sql, i, '`', googleSql, mysql);
+ if (end >= 0) {
+ out.append(sql, i, end);
+ }
+ }
+ else if (c == '[' && syntax == Syntax.TRANSACT_SQL) {
+ end = endOfQuoted(sql, i, ']', false, true);
+ if (end >= 0) {
+ out.append(sql, i, end);
+ }
+ }
+ else if (c == '/' && next == '*') {
+ end = endOfBlockComment(sql, i);
+ out.append(' ');
+ }
+ else if ((c == '-' && next == '-' && (!mysql || i + 2 >= length || Character.isWhitespace(sql.charAt(i + 2)))) ||
Review Comment:
Fixed in 0dc78c283a. `--` now starts a MySQL comment before any character at or below U+0020 or at U+007F, which is what the server's whitespace-or-control test accepts for ASCII. A character outside ASCII after `--` depends on the connection's character set: a no-break space is whitespace in latin1 and a name character in utf8mb4 (checked against MySQL 8.0 and 9.x and MariaDB), so a statement with one is read both ways and withheld.
While here I ran a lexical probe against every database the inspector maps (H2, HSQLDB, Derby and SQLite in memory; PostgreSQL, CockroachDB, MySQL, MariaDB, SQL Server, Oracle, Db2, Firebird, Informix and the Spanner emulator in containers) and replaced the one `STANDARD` reading with a `SqlTailSyntax` per database, as they did not share its rules: H2 reads `//` as a line comment, which was kept before; Oracle accepts the quote itself as the `q'` delimiter, so `q''a''` is a string; CockroachDB, Oracle, Informix and MySQL end a line comment only at a line feed; MySQL runs the content of `/*!...*/` depending on the server version, so such a statement is withheld.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@grails.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add the sql-tail command to find query-heavy pages [grails-core]
Posted by "testlens-app[bot] (via GitHub)" <gi...@apache.org>.
testlens-app[bot] commented on PR #16631:
URL: https://github.com/apache/grails-core/pull/16631#issuecomment-6347958597
## 🚨 All tests passed but jobs failed 🚨
### Failed Jobs without Test Failures
❌ [CI / Functional Tests \(Java 21, indy\=false\)](https://github.com/apache/grails-core/actions/runs/39818831780/job/119965591434?pr=16631)
🏷️ Commit: a562379a664d93f686e1bc660c733a4cfe4538fa
▶️ Tests: 48101 executed
🟡 Checks: 78/88 completed
### Rerun Controls
Click the checkbox to trigger a rerun:
🔲 **Rerun jobs**
---
_Learn more about TestLens at [testlens.app/docs](https://testlens.app/docs/features/pr-comment/)._
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@grails.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add the query-tail command to find query-heavy pages [grails-core]
Posted by "jdaugherty (via GitHub)" <gi...@apache.org>.
jdaugherty commented on PR #16631:
URL: https://github.com/apache/grails-core/pull/16631#issuecomment-6374599189
Nine fixes from a review pass over the branch, folded into the single commit, now 83e2720c65:
- **JVMs running at the same time share one capture.** The file is written a line at a time through an append channel, and a lock file beside it tells the first JVM to start, which truncates the file and names the run, from the ones that join it: they append to that run without truncating, with their requests numbered `<pid>-r1`, `<pid>-r2`. The parallel forks of a test task therefore land in one run the command reads whole. A lock file that cannot be created, as beside a device, leaves the writer uncoordinated.
- **Request paths lose their values.** The request line is recorded with every path segment that holds a value replaced by `?`: the values the URL mapping took from the path, those of the query string, and any segment that is a number, a UUID or a long hexadecimal token. `/password/reset/<token>` records as `GET /password/reset/?`.
- **A capture whose writes fail is let go**, so the next capture opened on that file starts afresh instead of joining a dead one.
- **The reader aligns to a line when it attaches** to a live file, so a statement being written as the command starts is never read in half.
- **An idle poll reads nothing**, not even the first line, when the file is as long as the reader left it.
- **`--once` stops when told** while it is still catching up on a large capture.
- **A bare `--file`** no longer names a file called `true`.
- **The average column** is written with a point whatever the default locale.
- **`--page=<name>`** no longer matches the `thread:<name>` pages.
Each has a spec: the shared-file protocol through the lock file, the failed capture through a named pipe whose reader goes away, the attaching reader through a half-written line, the idle poll through a file that refuses to be opened, and the rest through their tables.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@grails.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add the query-tail command to find query-heavy pages [grails-core]
Posted by "testlens-app[bot] (via GitHub)" <gi...@apache.org>.
testlens-app[bot] commented on PR #16631:
URL: https://github.com/apache/grails-core/pull/16631#issuecomment-6472316343
## 🚨 TestLens detected 1 failed test 🚨
### Test Summary
> [!TIP]
> Inspect all details with the [TestLens CLI](https://testlens.app/docs/features/cli): `testlens pr 16631`
#### [CI / Build Grails Forge \(Java 21, indy\=false\)](https://github.com/apache/grails-core/actions/runs/40701249120/job/122862832757?pr=16631) > :test-core:test [grails-forge]
| Test | Runs | Flakiness |
|---|---|--:|
| AsciidoctorBuildSpec > an app with asciidoctor builds its guide, without the configuration cache the plugin does not support | ❌ | [2% 🟡](https://github.com/apache/grails-core/issues/16030) |
🏷️ Commit: 77bd7c097f6c71fa51eaf0caa31a3f1e90521b18
▶️ Tests: 43458 executed
🟡 Checks: 62/93 completed
### Test Failures
<details><summary><strong>AsciidoctorBuildSpec > an app with asciidoctor builds its guide, without the configuration cache the plugin does not support</strong> (:test-core:test [grails-forge] in <a href="https://github.com/apache/grails-core/actions/runs/40701249120/job/122862832757?pr=16631">CI / Build Grails Forge (Java 21, indy=false)</a>)</summary>
```
org.gradle.testkit.runner.UnexpectedBuildFailure: Unexpected build execution failure in /tmp/testasciidoctor223555917924703965 with arguments [--init-script, /home/runner/work/grails-core/grails-core/grails-forge/gradle/classic-framework.init.gradle, asciidoctor]
Output:
Calculating task graph as no cached configuration is available for tasks: asciidoctor
FAILURE: Build failed with an exception.
* What went wrong:
A problem occurred configuring root project 'testasciidoctor223555917924703965'.
> Could not resolve all artifacts for configuration 'classpath'.
> Could not download asciidoctor-gradle-jvm-4.0.5.jar (org.asciidoctor:asciidoctor-gradle-jvm:4.0.5): Repository maven2 is disabled due to earlier error below:
> Could not download org.asciidoctor.jvm.convert.gradle.plugin-4.0.5.jar (org.asciidoctor.jvm.convert:org.asciidoctor.jvm.convert.gradle.plugin:4.0.5)
> Could not get resource 'https://repo.grails.org/grails/restricted/org/asciidoctor/jvm/convert/org.asciidoctor.jvm.convert.gradle.plugin/4.0.5/org.asciidoctor.jvm.convert.gradle.plugin-4.0.5.jar'.
> Could not HEAD 'https://repo.grails.org/grails/restricted/org/asciidoctor/jvm/convert/org.asciidoctor.jvm.convert.gradle.plugin/4.0.5/org.asciidoctor.jvm.convert.gradle.plugin-4.0.5.jar'.
> Received status code 429 from server: Too Many Requests
> There are 2 more failures with identical causes.
* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights from a Build Scan (powered by Develocity).
> Get more help at https://help.gradle.org.
BUILD FAILED in 45s
Configuration cache entry stored.
at org.gradle.testkit.runner.internal.DefaultGradleRunner.lambda$build$0(DefaultGradleRunner.java:275)
at org.gradle.testkit.runner.internal.DefaultGradleRunner.run(DefaultGradleRunner.java:374)
at org.gradle.testkit.runner.internal.DefaultGradleRunner.build(DefaultGradleRunner.java:273)
at org.grails.forge.utils.CommandSpec.executeGradle(CommandSpec.groovy:81)
at org.grails.forge.utils.CommandSpec.executeGradle(CommandSpec.groovy:76)
at org.grails.forge.features.asciidoctor.AsciidoctorBuildSpec.an app with asciidoctor builds its guide, without the configuration cache the plugin does not support(AsciidoctorBuildSpec.groovy:33)
```
</details>
### Rerun Controls
> [!NOTE]
> Checks are currently running using the configuration below.
Select tests to mute in this pull request:
🔲 AsciidoctorBuildSpec > an app with asciidoctor builds its guide, without the configuration cache the plugin does not support <!-- uniqueId=[engine:spock]/[spec:org.grails.forge.features.asciidoctor.AsciidoctorBuildSpec]/[feature:$spock_feature_1_0] -->
Reuse successful test results:
🔲 ♻️ Only rerun the tests that failed or were muted before
Click the checkbox to trigger a rerun:
🔲 **Rerun jobs**
---
_Learn more about TestLens at [testlens.app/docs](https://testlens.app/docs/features/pr-comment/)._
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@grails.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add the query-tail command to find query-heavy pages [grails-core]
Posted by "testlens-app[bot] (via GitHub)" <gi...@apache.org>.
testlens-app[bot] commented on PR #16631:
URL: https://github.com/apache/grails-core/pull/16631#issuecomment-6431795297
## 🚨 All tests passed but jobs failed 🚨
### Failed Jobs without Test Failures
❌ [End to End / End to End Tests \(end-to-end build only\)](https://github.com/apache/grails-core/actions/runs/40432892622/job/121955116737?pr=16631)
🏷️ Commit: 01ef1fbbabbeec0ef1785873fbd958c22c7494c3
▶️ Tests: 63104 executed
🟡 Checks: 41/92 completed
### Rerun Controls
Click the checkbox to trigger a rerun:
🔲 **Rerun jobs**
---
_Learn more about TestLens at [testlens.app/docs](https://testlens.app/docs/features/pr-comment/)._
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@grails.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add the query-tail command to find query-heavy pages [grails-core]
Posted by "jdaugherty (via GitHub)" <gi...@apache.org>.
jdaugherty commented on PR #16631:
URL: https://github.com/apache/grails-core/pull/16631#issuecomment-6414947946
@jamesfredley, review 5651210439 is addressed in da16a91754, still a single commit on 9.0.x.
- **Failed first write:** the start lock is released only while it is still valid, so `open()` returns a stopped capture instead of throwing `ClosedChannelException`. A capture file that stops while it starts is no longer registered as open, so the next capture of the same path tries the file again instead of joining the stopped one. Details are on the thread.
`:grails-datamapping-core:check` passes locally. The `/dev/full` features run only on Linux, so I also ran `QueryTailCaptureSpec` in a Linux container, where it passes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@grails.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add the query-tail command to find query-heavy pages [grails-core]
Posted by "jamesfredley (via GitHub)" <gi...@apache.org>.
jamesfredley commented on code in PR #16631:
URL: https://github.com/apache/grails-core/pull/16631#discussion_r4396159328
##########
grails-data-hibernate7/core/src/main/groovy/org/apache/grails/data/hibernate7/core/querytail/QueryTailConnectionProvider.groovy:
##########
@@ -0,0 +1,127 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * https://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.grails.data.hibernate7.core.querytail
+
+import java.lang.reflect.InvocationHandler
+import java.lang.reflect.InvocationTargetException
+import java.lang.reflect.Method
+import java.lang.reflect.Proxy
+import java.sql.Connection
+
+import groovy.transform.CompileStatic
+import org.hibernate.boot.registry.StandardServiceInitiator
+import org.hibernate.boot.registry.StandardServiceRegistryBuilder
+import org.hibernate.engine.jdbc.connections.internal.ConnectionProviderInitiator
+import org.hibernate.engine.jdbc.connections.internal.MultiTenantConnectionProviderInitiator
+import org.hibernate.service.Service
+import org.hibernate.service.spi.Configurable
+import org.hibernate.service.spi.ServiceBinding
+import org.hibernate.service.spi.ServiceRegistryAwareService
+import org.hibernate.service.spi.ServiceRegistryImplementor
+import org.hibernate.service.spi.Startable
+import org.hibernate.service.spi.Stoppable
+
+import org.apache.grails.datamapping.core.querytail.QueryTailJdbcConnection
+import org.apache.grails.datamapping.core.querytail.QueryTailRecorder
+
+/**
+ * Decorates the provider Hibernate selects, preserving its configuration and service lifecycle.
+ * Connections are captured at execution and returned to their provider as the original instances.
+ *
+ * @since 9.0
+ */
+@CompileStatic
+final class QueryTailConnectionProvider<S extends Service> implements StandardServiceInitiator<S> {
+
+ private final StandardServiceInitiator<S> delegate
+ private final QueryTailRecorder recorder
+
+ private QueryTailConnectionProvider(StandardServiceInitiator<S> delegate, QueryTailRecorder recorder) {
+ this.delegate = delegate
+ this.recorder = recorder
+ }
+
+ static void install(StandardServiceRegistryBuilder builder, QueryTailRecorder recorder) {
Review Comment:
install registers wrappers around Hibernate's stock connection-provider initiators. A hibernate.configClass that registered its own initiator is no longer called, so capture can take connections from a different source.
Wrap the initiator the registry already selected. The Hibernate 5 installer does the same.
##########
grails-data-mongodb/core/src/main/groovy/org/apache/grails/data/mongodb/core/querytail/MongoCommandRedactor.groovy:
##########
@@ -0,0 +1,529 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * https://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.grails.data.mongodb.core.querytail
+
+import groovy.transform.CompileStatic
+import org.bson.BsonDocument
+import org.bson.BsonValue
+
+import org.apache.grails.datamapping.core.querytail.QueryTailRedactor
+
+/**
+ * Takes the values out of a MongoDB command before it is captured. A command is a document, so
+ * there is nothing to parse: every value becomes {@link QueryTailRedactor#PLACEHOLDER}, and the
+ * keys, which are the options of the command, its operators and the field names of the collection,
+ * are kept, except where a key may be data. The keys of a document stored in a map property, or in
+ * a property or a collection the {@linkplain MongoMappedFields mapping} does not know, are recorded
+ * as the placeholder too, and so is every segment of a dotted path past such a key. So is a key at
+ * the top of stored data, the documents an insert writes, a replacement document, the fields an
+ * update writes and the operand of a literal, unless the mapping knows it, as the name of a dynamic
+ * attribute may be data; the keys at the top of a condition, a pipeline, a sort or a projection
+ * are the fields the application asks for, and are kept. An update that is a pipeline writes the
+ * documents its stages build: the keys at the top of what a {@code $set}, {@code $addFields} or
+ * {@code $project} stage holds, of the new root of a {@code $replaceWith} or {@code $replaceRoot}, and of a
+ * document {@code $mergeObjects} merges into it are fields written, kept only when the mapping knows
+ * them, as those of a classic update are. A numeric key
+ * is kept only where it indexes an array the mapping knows; in a map, in an element of an array,
+ * which is a map or a document unless the field is a list of lists, or at the top of a document,
+ * it is data. A {@code $} key is kept as an operator only where operators go, in a condition, a
+ * pipeline and the operators of an update; in stored data, the documents an insert writes, a
+ * replacement document and the values an update operator writes, it is a field name, which MongoDB
+ * allows to start with {@code $}, and is recorded as the placeholder unless the mapping knows it.
+ * So is a {@code $} key in the operand of a comparison in a condition, such as {@code $eq} or
+ * {@code $in}, or an implicit document or array equality, which holds the value the field is compared
+ * with, and in a DBRef, whose {@code $ref}, {@code $id} and {@code $db} a condition reads as fields
+ * compared with their whole value, in any order; in an aggregation expression, which
+ * {@code $expr}, the {@code let} of a command and the stages that compute fields hold, the operands
+ * of a comparison are expressions, and their operators are kept, except in the operand of a
+ * {@code $literal} or {@code $const}, which is stored data. A comment, and the constants of an
+ * update, and the scope of a map-reduce command, may be any document, and none of their keys are
+ * kept. The entries the driver adds for its
+ * session, the cluster time and the transaction, which say nothing about the command, are left out.
+ *
+ * <p>A command is recorded as its name, the database and collection it runs against, and the
+ * document of its remaining entries: {@code find shop.book {"filter": {"title": ?}, "limit": ?}}.
+ * The collection is taken only from a command that runs against one; the first value of any other
+ * command is an argument, such as a user name, and is recorded as a value:
+ * {@code dropUser admin {"dropUser": ?}}.</p>
+ *
+ * @since 9.0
+ */
+@CompileStatic
+final class MongoCommandRedactor {
+
+ private static final String PLACEHOLDER = QueryTailRedactor.PLACEHOLDER
+
+ private static final String DATABASE = '$db'
+
+ private static final String COLLECTION = 'collection'
+
+ /**
+ * The commands whose first value is the collection they run against. The first value of any other
+ * command, such as the user {@code dropUser} drops, is an argument, and is recorded as a value.
+ */
+ private static final Set<String> COLLECTION_COMMANDS = ['find', 'insert', 'update', 'delete', 'aggregate',
+ 'count', 'distinct', 'findAndModify', 'findandmodify', 'mapReduce', 'mapreduce', 'killCursors',
+ 'create', 'drop', 'collMod', 'createIndexes', 'dropIndexes', 'deleteIndexes', 'listIndexes', 'reIndex',
+ 'compact', 'validate', 'collStats', 'convertToCapped', 'cloneCollectionAsCapped', 'analyze',
+ 'checkMetadataConsistency', 'createSearchIndexes', 'dropSearchIndex', 'updateSearchIndex',
+ 'planCacheClear', 'planCacheClearFilters', 'planCacheListFilters', 'planCacheSetFilter'] as Set<String>
+
+ /** The command that names its collection in its {@code collection} option, as it continues a cursor. */
+ private static final String GET_MORE = 'getMore'
+
+ /** The entries the driver adds, which describe its session and transaction rather than the command. */
+ private static final Set<String> LEFT_OUT = ['lsid', '$clusterTime', 'txnNumber', 'autocommit',
+ 'startTransaction', '$readPreference', 'apiVersion', 'apiStrict', 'apiDeprecationErrors'] as Set<String>
+
+ /**
+ * The options whose document, or documents, are keyed by the fields of the collection, wherever they appear,
+ * including those of the operations of a client-level {@code bulkWrite}.
+ */
+ private static final Set<String> FIELD_KEYED = ['filter', 'query', 'q', 'u', 'update', 'document', 'documents',
+ 'updateMods', 'pipeline', 'sort', 'projection', 'fields', 'key', 'hint', 'min', 'max', 'arrayFilters',
+ 'partialFilterExpression', 'wildcardProjection', 'validator', 'weights'] as Set<String>
+
+ /** Documents that match fields, whose field values may be implicit equality operands. */
+ private static final Set<String> CONDITION_KEYED = ['filter', 'query', 'q', 'arrayFilters',
+ 'partialFilterExpression', 'validator', 'restrictSearchWithMatch'] as Set<String>
+
+ /** The options whose documents are stored data: what an insert writes. */
+ private static final Set<String> DATA_KEYED = ['documents', 'document'] as Set<String>
+
+ /** The options and the operator that may hold any document, none of whose keys say what the command does. */
+ private static final Set<String> ANY_VALUE = ['comment', '$comment', 'c', 'scope'] as Set<String>
+
+ /** The option whose values are aggregation expressions. */
+ private static final String LET = 'let'
+
+ /** The options that hold an update: update operators, a replacement document or a pipeline. */
+ private static final Set<String> UPDATE_KEYED = ['u', 'update', 'updateMods'] as Set<String>
+
+ /** The update operators whose fields take a condition rather than the value to write. */
+ private static final Set<String> CONDITION_OPERATORS = ['$pull', '$currentDate'] as Set<String>
+
+ /** The update operators that add to an array, whose value may hold modifiers. */
+ private static final Set<String> PUSH_OPERATORS = ['$push', '$addToSet'] as Set<String>
+
+ private static final String EACH = '$each'
+
+ /** The modifiers of a {@code $push} or an {@code $addToSet}. */
+ private static final Set<String> PUSH_MODIFIERS = [EACH, '$slice', '$sort', '$position'] as Set<String>
+
+ /**
+ * The operators of a pipeline whose operand is literal data rather than an expression, including
+ * {@code $const}, which MongoDB accepts for {@code $literal}.
+ */
+ private static final Set<String> LITERAL_OPERATORS = ['$literal', '$const', '$documents'] as Set<String>
+
+ /**
+ * The fields of a DBRef, which a condition reads as fields compared with their whole value, never as
+ * operators, in any order and wherever a condition has fields.
+ */
+ private static final Set<String> DBREF_FIELDS = ['$ref', '$id', '$db'] as Set<String>
+
+ /**
+ * The query operators whose operand holds values rather than a condition: the comparisons, whose operand
+ * is the value a field is compared with, a JSON schema, whose {@code enum} lists values, and a
+ * GeoJSON geometry, which can carry additional data alongside its type and coordinates.
+ */
+ private static final Set<String> VALUE_OPERATORS = ['$eq', '$ne', '$gt', '$gte', '$lt', '$lte', '$in', '$nin',
+ '$all', '$jsonSchema', '$geometry'] as Set<String>
+
+ /** The operator of a condition, and the stages of a pipeline, whose operand is an aggregation expression. */
+ private static final Set<String> EXPRESSION_OPERATORS = ['$expr', '$addFields', '$set', '$project', '$group',
+ '$replaceRoot', '$replaceWith', '$redact', '$bucket', '$bucketAuto', '$sortByCount', '$setWindowFields',
+ '$fill'] as Set<String>
+
+ /** The stages of an update pipeline whose operand is the document they write, or the new root that replaces it. */
+ private static final Set<String> WRITING_STAGES = ['$set', '$addFields', '$project', '$replaceWith'] as Set<String>
+
+ /** The stage of an update pipeline whose options name the new root that replaces the document. */
+ private static final String REPLACE_ROOT = '$replaceRoot'
+
+ /** The option of {@code $replaceRoot} that holds the new root. */
+ private static final String NEW_ROOT = 'newRoot'
+
+ /** The operator whose operands are documents merged into one, which in an update pipeline is written. */
+ private static final String MERGE_OBJECTS = '$mergeObjects'
+
+ /** The kinds of document whose keys at the top are stored data: what an insert, a replacement or an update writes. */
+ private static final Set<Kind> STORED = EnumSet.of(Kind.DATA, Kind.UPDATE, Kind.UPDATE_FIELDS, Kind.WRITTEN)
+
+ private MongoCommandRedactor() {
+ }
+
+ /**
+ * Redacts a command.
+ *
+ * @param database the database the command runs against
+ * @param commandName the name of the command, which is its first key
+ * @param command the command as the driver sends it
+ * @param fields the fields the mapping of the datastore knows, or null before the datastore is built
+ * @return the command with its values replaced
+ */
+ static String redact(String database, String commandName, BsonDocument command, MongoMappedFields fields) {
+ String collection = null
+ String collectionKey = null
+ BsonValue first = command.get(commandName)
+ if (first != null && first.isString() && COLLECTION_COMMANDS.contains(commandName)) {
+ collection = first.asString().value
+ collectionKey = commandName
+ }
+ else if (commandName == GET_MORE && command.get(COLLECTION)?.isString()) {
+ collection = command.getString(COLLECTION).value
+ collectionKey = COLLECTION
+ }
+ StringBuilder out = new StringBuilder(commandName)
+ if (database != null || collection != null) {
+ out.append(' ')
+ if (database != null) {
+ out.append(database)
+ }
+ if (collection != null) {
+ out.append(database != null ? '.' : '').append(collection)
+ }
+ }
+ Scope options = Scope.options(collection, fields)
+ StringBuilder body = new StringBuilder()
+ for (Map.Entry<String, BsonValue> entry in command.entrySet()) {
+ String key = entry.key
+ if (key == DATABASE || key == collectionKey || LEFT_OUT.contains(key)) {
+ continue
+ }
+ body.append(body.length() == 0 ? '{' : ', ')
+ appendEntry(body, key, entry.value, options)
+ }
+ if (body.length() > 0) {
+ out.append(' ').append(body).append('}')
+ }
+ out.toString()
+ }
+
+ private static void appendEntry(StringBuilder out, String key, BsonValue value, Scope scope) {
+ Scope child = scope.of(key, value)
+ out.append(child.keyText).append(': ')
+ appendValue(out, value, child)
+ }
+
+ private static void appendValue(StringBuilder out, BsonValue value, Scope scope) {
+ if (value.isDocument()) {
+ out.append('{')
+ boolean first = true
+ for (Map.Entry<String, BsonValue> entry in value.asDocument().entrySet()) {
+ if (!first) {
+ out.append(', ')
+ }
+ first = false
+ appendEntry(out, entry.key, entry.value, scope)
+ }
+ out.append('}')
+ }
+ else if (value.isArray()) {
+ out.append('[')
+ boolean first = true
+ Scope element = scope.element()
+ for (BsonValue item in value.asArray()) {
+ if (!first) {
+ out.append(', ')
+ }
+ first = false
+ appendValue(out, item, element)
+ }
+ out.append(']')
+ }
+ else {
+ out.append(PLACEHOLDER)
+ }
+ }
+
+ private static String quote(String key) {
+ StringBuilder out = new StringBuilder(key.length() + 2).append('"')
+ for (int i = 0; i < key.length(); i++) {
+ char c = key.charAt(i)
+ if (c == '"' || c == '\\') {
+ out.append('\\').append(c)
+ }
+ else if (c < 0x20) {
+ out.append(String.format('\\u%04x', (int) c))
+ }
+ else {
+ out.append(c)
+ }
+ }
+ out.append('"').toString()
+ }
+
+ /**
+ * Where a document sits in the command, which decides what its keys are: the options of the
+ * command, kept; the fields and operators of a condition, a pipeline, a sort or a projection, where
+ * a {@code $} key is an operator; an aggregation expression, whose {@code $} keys are operators
+ * too; an update, whose {@code $} keys are update operators; or stored data, the documents an
+ * insert writes, the values an update writes and the values a condition compares with, where a
+ * {@code $} key is a field name like any other and is kept only when the mapping knows it.
+ */
+ @CompileStatic
+ private static final class Scope {
+
+ final String collection
+ final MongoMappedFields fields
+ final Kind kind
+ /**
+ * The update operator whose fields a document of kind {@link Kind#UPDATE_FIELDS} holds, or the stage
+ * whose document one of kind {@link Kind#WRITTEN} is.
+ */
+ final String operator
+ /** The field path of the document; null in a document keyed by fields the mapping does not know. */
+ final List<String> path
+ /** How many indices into the field at the path have been taken: 0 at the field, 1 in one of its elements. */
+ final int depth
+ final String keyText
+
+ private Scope(String collection, MongoMappedFields fields, Kind kind, String operator, List<String> path,
+ int depth, String keyText) {
+ this.collection = collection
+ this.fields = fields
+ this.kind = kind
+ this.operator = operator
+ this.path = path
+ this.depth = depth
+ this.keyText = keyText
+ }
+
+ static Scope options(String collection, MongoMappedFields fields) {
+ new Scope(collection, fields, Kind.OPTIONS, null, Collections.<String>emptyList(), 0, null)
+ }
+
+ /**
+ * @return the scope of an element of an array in this scope: one index deeper into the field
+ * at the path, so that a numeric key inside the element is data unless the elements are arrays;
+ * the elements of an update that is an array are the stages of a pipeline, and an element of an
+ * expression at the top of a document, as the operands of an operator {@code $expr} holds are, is a
+ * value the expression builds, whose keys are kept only when the mapping knows them, as those of a
+ * document a stage of an update pipeline writes are
+ */
+ Scope element() {
+ if (kind == Kind.UPDATE) {
+ return child(Kind.UPDATE_PIPELINE, null, path, depth, keyText)
+ }
+ if (kind == Kind.PUSH) {
+ return child(Kind.DATA, null, path, depth + 2, keyText)
+ }
+ if ((kind == Kind.QUERY || kind == Kind.CONDITION || kind == Kind.EXPRESSION || kind == Kind.DATA) && path != null && !path.isEmpty()) {
+ return child(kind, null, path, depth + 1, keyText)
+ }
+ if (kind == Kind.EXPRESSION && path != null) {
+ return child(Kind.WRITTEN, null, path, depth, keyText)
+ }
+ this
+ }
+
+ /**
+ * @return the scope of the value of a key in a document of this scope, with the key as it is recorded
+ */
+ Scope of(String key, BsonValue value) {
+ if (kind == Kind.OPTIONS) {
+ if (ANY_VALUE.contains(key)) {
+ return child(Kind.DATA, null, null, 0, quote(key))
+ }
+ Kind valueKind = DATA_KEYED.contains(key) ? Kind.DATA :
+ UPDATE_KEYED.contains(key) ? Kind.UPDATE :
+ CONDITION_KEYED.contains(key) ? Kind.CONDITION :
+ FIELD_KEYED.contains(key) ? Kind.QUERY :
+ key == LET ? Kind.EXPRESSION : Kind.OPTIONS
+ return child(valueKind, null, Collections.<String>emptyList(), 0, quote(key))
+ }
+ if ((kind == Kind.QUERY || kind == Kind.CONDITION) && key.startsWith('$')) {
+ if (ANY_VALUE.contains(key)) {
+ return child(Kind.DATA, null, null, 0, quote(key))
+ }
+ if (EXPRESSION_OPERATORS.contains(key)) {
+ return child(Kind.EXPRESSION, null, path, depth, quote(key))
+ }
+ if (key == '$match') {
+ return child(Kind.CONDITION, null, path, depth, quote(key))
+ }
+ if (kind == Kind.CONDITION && DBREF_FIELDS.contains(key)) {
+ return field(key, Kind.DATA)
Review Comment:
find with filter location.$near containing metadata.$example-secret records $example-secret. $near is not treated as a value operator, so that key stays. $nearSphere does the same. MongoDB accepts the extra fields and ignores them.
Treat the geometry document as data. Please assert the secret key is absent.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@grails.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add the query-tail command to find query-heavy pages [grails-core]
Posted by "testlens-app[bot] (via GitHub)" <gi...@apache.org>.
testlens-app[bot] commented on PR #16631:
URL: https://github.com/apache/grails-core/pull/16631#issuecomment-6427243116
## ✅ All tests passed ✅
🏷️ Commit: 52808b36418857b1a881da165c512d4b47b7dd95
▶️ Tests: 139419 executed
⚪️ Checks: 97/97 completed
---
_Learn more about TestLens at [testlens.app/docs](https://testlens.app/docs/features/pr-comment/)._
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@grails.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add the query-tail command to find query-heavy pages [grails-core]
Posted by "testlens-app[bot] (via GitHub)" <gi...@apache.org>.
testlens-app[bot] commented on PR #16631:
URL: https://github.com/apache/grails-core/pull/16631#issuecomment-6389778117
## 🚨 TestLens detected 1 failed test 🚨
### Test Summary
> [!TIP]
> Inspect all details with the [TestLens CLI](https://testlens.app/docs/features/cli): `testlens pr 16631`
#### [CI / Build Grails-Core \(Windows JDK 25 shard 2\)](https://github.com/apache/grails-core/actions/runs/40131866286/job/120985098136?pr=16631) > :grails-startup-progress:test
| Test | Runs | Flakiness |
|---|---|--:|
| StartupProgressSpec > tells a watching page why the start failed while <span>#</span>stage, then releases the port > tells a watching page why the start failed while creating beans, then releases the port | ❌ | 0% 🟢 |
🏷️ Commit: a50ddf396beb7ac25d53380fe55c30a8388301d6
▶️ Tests: 92426 executed
🟡 Checks: 49/92 completed
### Test Failures
<details><summary><strong>StartupProgressSpec > tells a watching page why the start failed while <span>#</span>stage, then releases the port > tells a watching page why the start failed while creating beans, then releases the port</strong> (:grails-startup-progress:test in <a href="https://github.com/apache/grails-core/actions/runs/40131866286/job/120985098136?pr=16631">CI / Build Grails-Core (Windows JDK 25 shard 2)</a>)</summary>
```
java.lang.AssertionError: No FAILED status within 30 seconds
at org.apache.grails.startup.StartupProgressSpec.awaitStatus(StartupProgressSpec.groovy:1032)
at org.apache.grails.startup.StartupProgressSpec.tells a watching page why the start failed while #stage, then releases the port(StartupProgressSpec.groovy:319)
```
</details>
### Rerun Controls
> [!NOTE]
> Checks are currently running using the configuration below.
Select tests to mute in this pull request:
🔲 StartupProgressSpec > tells a watching page why the start failed while <span>#</span>stage, then releases the port <!-- uniqueId=[engine:spock]/[spec:org.apache.grails.startup.StartupProgressSpec]/[feature:$spock_feature_0_2] -->
Reuse successful test results:
🔲 ♻️ Only rerun the tests that failed or were muted before
Click the checkbox to trigger a rerun:
🔲 **Rerun jobs**
---
_Learn more about TestLens at [testlens.app/docs](https://testlens.app/docs/features/pr-comment/)._
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@grails.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add the query-tail command to find query-heavy pages [grails-core]
Posted by "jamesfredley (via GitHub)" <gi...@apache.org>.
jamesfredley commented on code in PR #16631:
URL: https://github.com/apache/grails-core/pull/16631#discussion_r4369042371
##########
grails-datamapping-core/src/main/groovy/org/apache/grails/data/core/querytail/QueryTailRequestAttribution.groovy:
##########
@@ -0,0 +1,425 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * https://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.grails.data.core.querytail
+
+import java.lang.reflect.Method
+import java.nio.charset.StandardCharsets
+import java.util.concurrent.atomic.AtomicLong
+import java.util.regex.Pattern
+
+import groovy.transform.CompileStatic
+
+import org.springframework.util.ClassUtils
+import org.springframework.web.context.request.RequestAttributes
+import org.springframework.web.context.request.RequestContextHolder
+import org.springframework.web.context.request.ServletRequestAttributes
+
+/**
+ * Attributes a captured statement to the web request on the current thread. Grails keeps the
+ * request, and the controller and action it resolved, on the thread that serves it, so a statement
+ * run by an interceptor, the action or the view all land on the same page. The first statement of
+ * a request gives it an id that every later statement of that request repeats, which is how the
+ * {@code query-tail} command counts statements per request. Work on any other thread is attributed to
+ * the thread by name.
+ *
+ * <p>The request is recorded as its method and path, with the values in the path replaced by
+ * {@code ?}: the values of the parameters the URL mapping took from it and those of its query, found
+ * whether or not the client percent-encoded them, and any segment that is a number, a UUID or a long
+ * hexadecimal token, so that an id, a token or a key carried in a path never reaches the capture
+ * file. A value glued to static text in a segment, as a mapping such as {@code /reset/${token}_confirm}
+ * produces, is replaced within the segment. Until the URL mapping has run, as for a statement of a
+ * filter that runs before the controller is chosen, or for a request that never reaches one, no
+ * segment's role is known and every segment is replaced; only the controller the framework set on
+ * the request says that it has run, never a parameter, which a client can send.</p>
+ *
+ * <p>Spring Web is optional: without it on the classpath every statement is attributed to its
+ * thread.</p>
+ *
+ * @since 8.1
+ */
+@CompileStatic
+class QueryTailRequestAttribution {
+
+ /** The request attribute holding the id this attribution gave a request. */
+ public static final String REQUEST_ID_ATTRIBUTE = 'org.apache.grails.data.core.querytail.REQUEST_ID'
+
+ /** The request attribute Grails sets the controller name in. */
+ public static final String CONTROLLER_NAME_ATTRIBUTE = 'org.grails.CONTROLLER_NAME_ATTRIBUTE'
+
+ /** The request attribute Grails sets the controller namespace in. */
+ public static final String CONTROLLER_NAMESPACE_ATTRIBUTE = 'org.grails.CONTROLLER_NAMESPACE_ATTRIBUTE'
+
+ /** The request attribute Grails sets the action name in. */
+ public static final String ACTION_NAME_ATTRIBUTE = 'org.grails.ACTION_NAME_ATTRIBUTE'
+
+ private static final boolean SPRING_WEB_PRESENT = ClassUtils.isPresent(
+ 'org.springframework.web.context.request.RequestContextHolder', QueryTailRequestAttribution.classLoader)
+
+ private final AtomicLong requestCounter = new AtomicLong()
+ private final String requestIdPrefix
+
+ /**
+ * An attribution whose request ids are {@code r1}, {@code r2} and so on.
+ */
+ QueryTailRequestAttribution() {
+ this('')
+ }
+
+ /**
+ * @param requestIdPrefix what every request id starts with, which tells the requests of one JVM
+ * from those of another writing the same capture
+ */
+ QueryTailRequestAttribution(String requestIdPrefix) {
+ this.requestIdPrefix = requestIdPrefix ?: ''
+ }
+
+ /**
+ * Fills in the request id, page and request of a record from the current thread.
+ *
+ * @param record the record of a statement that just ran on this thread
+ * @return the same record
+ */
+ QueryTailRecord attribute(QueryTailRecord record) {
+ if (SPRING_WEB_PRESENT && WebRequestLookup.attribute(record, requestCounter, requestIdPrefix)) {
+ return record
+ }
+ record.requestId = null
+ record.request = null
+ record.page = QueryTailRecord.THREAD_PAGE_PREFIX + Thread.currentThread().name
+ record
+ }
+
+ /**
+ * Keeps every reference to Spring Web, in a signature as well as in a body, in a class that is only
+ * loaded once Spring Web is known to be present. The outer class can then be loaded, and reflected
+ * on, without Spring Web on the classpath.
+ */
+ @CompileStatic
+ private static class WebRequestLookup {
+
+ private static final String REQUEST_ID_PREFIX = 'r'
+
+ private static final String PLACEHOLDER = '?'
+
+ /** The parameters a URL mapping sets that name code rather than carry a value. */
+ private static final Set<String> MAPPING_PARAMETERS = ['controller', 'action', 'namespace', 'plugin', 'format'] as Set<String>
+
+ /** A segment that is a number, a UUID or a long hexadecimal token is a value whatever the mapping says. */
+ private static final Pattern VALUE_SHAPED = Pattern.compile(
+ '(?i)[0-9]+|[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}|[0-9a-f]{32,}')
+
+ private static final boolean SERVLET_PRESENT = ClassUtils.isPresent(
+ 'jakarta.servlet.http.HttpServletRequest', WebRequestLookup.classLoader)
+
+ /** The {@code getParams()} of a Grails web request, which holds the parameters its URL mapping took, by class. */
+ private static final ClassValue<Method> PARAMETERS = new ClassValue<Method>() {
+ @Override
+ protected Method computeValue(Class<?> type) {
+ try {
+ return type.getMethod('getParams')
+ }
+ catch (NoSuchMethodException ignored) {
+ return null
+ }
+ }
+ }
+
+ /**
+ * Fills in a record from the request on the current thread.
+ *
+ * @param record the record to fill in
+ * @param requestCounter the counter the next request id is taken from
+ * @param requestIdPrefix what every request id starts with
+ * @return whether there is a request on the current thread
+ */
+ static boolean attribute(QueryTailRecord record, AtomicLong requestCounter, String requestIdPrefix) {
+ RequestAttributes attributes = RequestContextHolder.getRequestAttributes()
+ if (attributes == null) {
+ return false
+ }
+ record.requestId = requestId(attributes, requestCounter, requestIdPrefix)
+ record.page = page(attributes)
+ record.request = SERVLET_PRESENT ? ServletRequestLookup.describe(attributes) : null
+ true
+ }
+
+ /**
+ * Builds the request line: the method and the path, with the values in the path, the parameters
+ * the URL mapping took from it among them, replaced.
+ *
+ * @param method the HTTP method
+ * @param path the request path
+ * @param attributes the request, whose mapped parameters are read when it is a Grails web request
+ * @param parameters the parameters of the servlet request, by name, each a {@code String[]}
+ */
+ static String describe(String method, String path, RequestAttributes attributes, Map<?, ?> parameters) {
+ // only the framework sets the controller attribute; a client can send a controller parameter
+ if (attributes.getAttribute(CONTROLLER_NAME_ATTRIBUTE, RequestAttributes.SCOPE_REQUEST) == null) {
+ return "${method} ${withholdPath(path)}".toString()
+ }
+ Set<String> values = new HashSet<>()
+ addValues(values, mappedParameters(attributes))
+ addValues(values, parameters)
+ "${method} ${redactPath(path, values)}".toString()
+ }
+
+ /**
+ * Replaces every segment of a path: before the URL mapping has run, nothing tells a literal
+ * segment from a token, so only the shape of the path is recorded.
+ */
+ private static String withholdPath(String path) {
+ String[] segments = path.split('/', -1)
+ StringBuilder out = new StringBuilder(path.length())
+ for (int i = 0; i < segments.length; i++) {
+ if (i > 0) {
+ out.append('/')
+ }
+ out.append(segments[i].isEmpty() ? segments[i] : PLACEHOLDER)
+ }
+ out.toString()
+ }
+
+ private static String requestId(RequestAttributes attributes, AtomicLong requestCounter, String requestIdPrefix) {
+ synchronized (attributes) {
+ Object existing = attributes.getAttribute(REQUEST_ID_ATTRIBUTE, RequestAttributes.SCOPE_REQUEST)
+ if (existing != null) {
+ return existing.toString()
+ }
+ String id = requestIdPrefix + REQUEST_ID_PREFIX + requestCounter.incrementAndGet()
+ attributes.setAttribute(REQUEST_ID_ATTRIBUTE, id, RequestAttributes.SCOPE_REQUEST)
+ return id
+ }
+ }
+
+ private static String page(RequestAttributes attributes) {
+ Object controller = attributes.getAttribute(CONTROLLER_NAME_ATTRIBUTE, RequestAttributes.SCOPE_REQUEST)
+ if (controller == null) {
+ return null
+ }
+ Object namespace = attributes.getAttribute(CONTROLLER_NAMESPACE_ATTRIBUTE, RequestAttributes.SCOPE_REQUEST)
+ Object action = attributes.getAttribute(ACTION_NAME_ATTRIBUTE, RequestAttributes.SCOPE_REQUEST)
+ StringBuilder page = new StringBuilder()
+ if (namespace != null) {
+ page.append(namespace).append(':')
+ }
+ page.append(controller)
+ if (action != null) {
+ page.append('/').append(action)
+ }
+ page.toString()
+ }
+
+ /**
+ * @return the parameters of a Grails web request, which include those its URL mapping took from
+ * the path, or null for a request that is not one
+ */
+ private static Map<?, ?> mappedParameters(RequestAttributes attributes) {
+ Method parameters = PARAMETERS.get(attributes.getClass())
+ if (parameters == null) {
+ return null
+ }
+ try {
+ Object value = parameters.invoke(attributes)
+ return value instanceof Map ? (Map<?, ?>) value : null
+ }
+ catch (ReflectiveOperationException | RuntimeException ignored) {
+ return null
+ }
+ }
+
+ private static void addValues(Set<String> values, Map<?, ?> parameters) {
+ if (parameters == null) {
+ return
+ }
+ for (Map.Entry<?, ?> entry in parameters.entrySet()) {
+ if (MAPPING_PARAMETERS.contains(String.valueOf(entry.key))) {
+ continue
+ }
+ Object value = entry.value
+ if (value instanceof CharSequence) {
+ values.add(value.toString())
+ }
+ else if (value instanceof Object[]) {
+ for (Object element in (Object[]) value) {
+ if (element instanceof CharSequence) {
+ values.add(element.toString())
+ }
+ }
+ }
+ }
+ }
+
+ /**
+ * Replaces the values in a path: a segment that is one of the values, with or without an
+ * extension, a run of segments that spells a value holding slashes, any segment shaped like a
+ * value, and a value glued to static text within a segment. The segments are matched as the
+ * client sent them and percent-decoded, since the parameters are decoded while the request URI
+ * is not; the path is written as sent, except that a segment a value was cut out of is written
+ * in the form it was found in.
+ */
+ private static String redactPath(String path, Set<String> values) {
+ String[] segments = path.split('/', -1)
+ String[] decoded = new String[segments.length]
+ boolean[] redacted = new boolean[segments.length]
+ for (int i = 0; i < segments.length; i++) {
+ decoded[i] = decode(segments[i])
+ redacted[i] = isValue(segments[i], values) || isValue(decoded[i], values)
+ }
+ for (String value in values) {
+ if (value.contains('/')) {
+ String[] run = value.split('/', -1)
+ markRun(segments, redacted, run)
+ markDecodedRun(decoded, redacted, run)
+ }
+ }
+ StringBuilder out = new StringBuilder(path.length())
+ for (int i = 0; i < segments.length; i++) {
+ if (i > 0) {
+ out.append('/')
+ }
+ out.append(redacted[i] ? PLACEHOLDER : withoutValues(segments[i], decoded[i], values))
+ }
+ out.toString()
+ }
+
+ /**
+ * Cuts the values out of a segment that is not one itself, as a mapping that glues static text
+ * to a variable produces: a value of three or more characters wherever it appears, a shorter one
+ * only where no letter or digit continues it on either side, so that a one-letter query value
+ * does not pick holes in every word of the path.
+ *
+ * @return the segment with the values replaced, in the form, as sent or decoded, they were found in
+ */
+ private static String withoutValues(String segment, String decoded, Set<String> values) {
+ String cut = cutValues(segment, values)
+ if (cut != segment) {
Review Comment:
Changing the raw segment returns before the decoded match. `/reset/%6B9fQ2x_confirm?q=confirm` with token `k9fQ2x` becomes `GET /reset/%6B9fQ2x_?`. The encoded token is still in the file.
If either the raw or the decoded form changed, replace the whole segment. Please cover that request through `attribute()`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@grails.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add the query-tail command to find query-heavy pages [grails-core]
Posted by "testlens-app[bot] (via GitHub)" <gi...@apache.org>.
testlens-app[bot] commented on PR #16631:
URL: https://github.com/apache/grails-core/pull/16631#issuecomment-6391915529
## 🚨 All tests passed but jobs failed 🚨
### Failed Jobs without Test Failures
❌ [CI / Hibernate7 Functional Tests \(Java 25, indy\=false\)](https://github.com/apache/grails-core/actions/runs/40145921895/job/121027135873?pr=16631)
🏷️ Commit: 49a8b1d769d2ad603a7219f9eccbd40a4fdb07ff
▶️ Tests: 127661 executed
🟡 Checks: 84/93 completed
### Rerun Controls
Click the checkbox to trigger a rerun:
🔲 **Rerun jobs**
---
_Learn more about TestLens at [testlens.app/docs](https://testlens.app/docs/features/pr-comment/)._
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@grails.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add the query-tail command to find query-heavy pages [grails-core]
Posted by "jamesfredley (via GitHub)" <gi...@apache.org>.
jamesfredley commented on code in PR #16631:
URL: https://github.com/apache/grails-core/pull/16631#discussion_r4392889924
##########
grails-data-mongodb/core/src/main/groovy/org/apache/grails/data/mongodb/core/querytail/MongoCommandRedactor.groovy:
##########
@@ -0,0 +1,511 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * https://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.grails.data.mongodb.core.querytail
+
+import groovy.transform.CompileStatic
+import org.bson.BsonDocument
+import org.bson.BsonValue
+
+import org.apache.grails.datamapping.core.querytail.QueryTailRedactor
+
+/**
+ * Takes the values out of a MongoDB command before it is captured. A command is a document, so
+ * there is nothing to parse: every value becomes {@link QueryTailRedactor#PLACEHOLDER}, and the
+ * keys, which are the options of the command, its operators and the field names of the collection,
+ * are kept, except where a key may be data. The keys of a document stored in a map property, or in
+ * a property or a collection the {@linkplain MongoMappedFields mapping} does not know, are recorded
+ * as the placeholder too, and so is every segment of a dotted path past such a key. So is a key at
+ * the top of stored data, the documents an insert writes, a replacement document, the fields an
+ * update writes and the operand of a literal, unless the mapping knows it, as the name of a dynamic
+ * attribute may be data; the keys at the top of a condition, a pipeline, a sort or a projection
+ * are the fields the application asks for, and are kept. An update that is a pipeline writes the
+ * documents its stages build: the keys at the top of what a {@code $set}, {@code $addFields} or
+ * {@code $project} stage holds, of the new root of a {@code $replaceWith} or {@code $replaceRoot}, and of a
+ * document {@code $mergeObjects} merges into it are fields written, kept only when the mapping knows
+ * them, as those of a classic update are. A numeric key
+ * is kept only where it indexes an array the mapping knows; in a map, in an element of an array,
+ * which is a map or a document unless the field is a list of lists, or at the top of a document,
+ * it is data. A {@code $} key is kept as an operator only where operators go, in a condition, a
+ * pipeline and the operators of an update; in stored data, the documents an insert writes, a
+ * replacement document and the values an update operator writes, it is a field name, which MongoDB
+ * allows to start with {@code $}, and is recorded as the placeholder unless the mapping knows it.
+ * So is a {@code $} key in the operand of a comparison in a condition, such as {@code $eq} or
+ * {@code $in}, or an implicit document or array equality, which holds the value the field is compared
+ * with, and in a DBRef, whose {@code $ref}, {@code $id} and {@code $db} a condition reads as fields
+ * compared with their whole value, in any order; in an aggregation expression, which
+ * {@code $expr}, the {@code let} of a command and the stages that compute fields hold, the operands
+ * of a comparison are expressions, and their operators are kept, except in the operand of a
+ * {@code $literal} or {@code $const}, which is stored data. A comment, and the constants of an
+ * update, may be any document, and none of their keys are kept. The entries the driver adds for its
+ * session, the cluster time and the transaction, which say nothing about the command, are left out.
+ *
+ * <p>A command is recorded as its name, the database and collection it runs against, and the
+ * document of its remaining entries: {@code find shop.book {"filter": {"title": ?}, "limit": ?}}.</p>
+ *
+ * @since 9.0
+ */
+@CompileStatic
+final class MongoCommandRedactor {
+
+ private static final String PLACEHOLDER = QueryTailRedactor.PLACEHOLDER
+
+ private static final String DATABASE = '$db'
+
+ private static final String COLLECTION = 'collection'
+
+ /** The entries the driver adds, which describe its session and transaction rather than the command. */
+ private static final Set<String> LEFT_OUT = ['lsid', '$clusterTime', 'txnNumber', 'autocommit',
+ 'startTransaction', '$readPreference', 'apiVersion', 'apiStrict', 'apiDeprecationErrors'] as Set<String>
+
+ /**
+ * The options whose document, or documents, are keyed by the fields of the collection, wherever they appear,
+ * including those of the operations of a client-level {@code bulkWrite}.
+ */
+ private static final Set<String> FIELD_KEYED = ['filter', 'query', 'q', 'u', 'update', 'document', 'documents',
+ 'updateMods', 'pipeline', 'sort', 'projection', 'fields', 'key', 'hint', 'min', 'max', 'arrayFilters',
+ 'partialFilterExpression', 'wildcardProjection', 'validator', 'weights'] as Set<String>
+
+ /** Documents that match fields, whose field values may be implicit equality operands. */
+ private static final Set<String> CONDITION_KEYED = ['filter', 'query', 'q', 'arrayFilters',
+ 'partialFilterExpression', 'validator', 'restrictSearchWithMatch'] as Set<String>
+
+ /** The options whose documents are stored data: what an insert writes. */
+ private static final Set<String> DATA_KEYED = ['documents', 'document'] as Set<String>
+
+ /** The options and the operator that may hold any document, none of whose keys say what the command does. */
+ private static final Set<String> ANY_VALUE = ['comment', '$comment', 'c'] as Set<String>
+
+ /** The option whose values are aggregation expressions. */
+ private static final String LET = 'let'
+
+ /** The options that hold an update: update operators, a replacement document or a pipeline. */
+ private static final Set<String> UPDATE_KEYED = ['u', 'update', 'updateMods'] as Set<String>
+
+ /** The update operators whose fields take a condition rather than the value to write. */
+ private static final Set<String> CONDITION_OPERATORS = ['$pull', '$currentDate'] as Set<String>
+
+ /** The update operators that add to an array, whose value may hold modifiers. */
+ private static final Set<String> PUSH_OPERATORS = ['$push', '$addToSet'] as Set<String>
+
+ private static final String EACH = '$each'
+
+ /** The modifiers of a {@code $push} or an {@code $addToSet}. */
+ private static final Set<String> PUSH_MODIFIERS = [EACH, '$slice', '$sort', '$position'] as Set<String>
+
+ /**
+ * The operators of a pipeline whose operand is literal data rather than an expression, including
+ * {@code $const}, which MongoDB accepts for {@code $literal}.
+ */
+ private static final Set<String> LITERAL_OPERATORS = ['$literal', '$const', '$documents'] as Set<String>
+
+ /**
+ * The fields of a DBRef, which a condition reads as fields compared with their whole value, never as
+ * operators, in any order and wherever a condition has fields.
+ */
+ private static final Set<String> DBREF_FIELDS = ['$ref', '$id', '$db'] as Set<String>
+
+ /**
+ * The query operators whose operand holds values rather than a condition: the comparisons, whose operand
+ * is the value a field is compared with, a JSON schema, whose {@code enum} lists values, and a
+ * GeoJSON geometry, which can carry additional data alongside its type and coordinates.
+ */
+ private static final Set<String> VALUE_OPERATORS = ['$eq', '$ne', '$gt', '$gte', '$lt', '$lte', '$in', '$nin',
+ '$all', '$jsonSchema', '$geometry'] as Set<String>
+
+ /** The operator of a condition, and the stages of a pipeline, whose operand is an aggregation expression. */
+ private static final Set<String> EXPRESSION_OPERATORS = ['$expr', '$addFields', '$set', '$project', '$group',
+ '$replaceRoot', '$replaceWith', '$redact', '$bucket', '$bucketAuto', '$sortByCount', '$setWindowFields',
+ '$fill'] as Set<String>
+
+ /** The stages of an update pipeline whose operand is the document they write, or the new root that replaces it. */
+ private static final Set<String> WRITING_STAGES = ['$set', '$addFields', '$project', '$replaceWith'] as Set<String>
+
+ /** The stage of an update pipeline whose options name the new root that replaces the document. */
+ private static final String REPLACE_ROOT = '$replaceRoot'
+
+ /** The option of {@code $replaceRoot} that holds the new root. */
+ private static final String NEW_ROOT = 'newRoot'
+
+ /** The operator whose operands are documents merged into one, which in an update pipeline is written. */
+ private static final String MERGE_OBJECTS = '$mergeObjects'
+
+ /** The kinds of document whose keys at the top are stored data: what an insert, a replacement or an update writes. */
+ private static final Set<Kind> STORED = EnumSet.of(Kind.DATA, Kind.UPDATE, Kind.UPDATE_FIELDS, Kind.WRITTEN)
+
+ private MongoCommandRedactor() {
+ }
+
+ /**
+ * Redacts a command.
+ *
+ * @param database the database the command runs against
+ * @param commandName the name of the command, which is its first key
+ * @param command the command as the driver sends it
+ * @param fields the fields the mapping of the datastore knows, or null before the datastore is built
+ * @return the command with its values replaced
+ */
+ static String redact(String database, String commandName, BsonDocument command, MongoMappedFields fields) {
+ String collection = null
+ String collectionKey = null
+ BsonValue first = command.get(commandName)
+ if (first != null && first.isString()) {
+ collection = first.asString().value
+ collectionKey = commandName
+ }
+ else if (command.get(COLLECTION)?.isString()) {
+ collection = command.getString(COLLECTION).value
+ collectionKey = COLLECTION
+ }
+ StringBuilder out = new StringBuilder(commandName)
+ if (database != null || collection != null) {
+ out.append(' ')
+ if (database != null) {
+ out.append(database)
+ }
+ if (collection != null) {
+ out.append(database != null ? '.' : '').append(collection)
+ }
+ }
+ Scope options = Scope.options(collection, fields)
+ StringBuilder body = new StringBuilder()
+ for (Map.Entry<String, BsonValue> entry in command.entrySet()) {
+ String key = entry.key
+ if (key == DATABASE || key == collectionKey || LEFT_OUT.contains(key)) {
+ continue
+ }
+ body.append(body.length() == 0 ? '{' : ', ')
+ appendEntry(body, key, entry.value, options)
+ }
+ if (body.length() > 0) {
+ out.append(' ').append(body).append('}')
+ }
+ out.toString()
+ }
+
+ private static void appendEntry(StringBuilder out, String key, BsonValue value, Scope scope) {
+ Scope child = scope.of(key, value)
+ out.append(child.keyText).append(': ')
+ appendValue(out, value, child)
+ }
+
+ private static void appendValue(StringBuilder out, BsonValue value, Scope scope) {
+ if (value.isDocument()) {
+ out.append('{')
+ boolean first = true
+ for (Map.Entry<String, BsonValue> entry in value.asDocument().entrySet()) {
+ if (!first) {
+ out.append(', ')
+ }
+ first = false
+ appendEntry(out, entry.key, entry.value, scope)
+ }
+ out.append('}')
+ }
+ else if (value.isArray()) {
+ out.append('[')
+ boolean first = true
+ Scope element = scope.element()
+ for (BsonValue item in value.asArray()) {
+ if (!first) {
+ out.append(', ')
+ }
+ first = false
+ appendValue(out, item, element)
+ }
+ out.append(']')
+ }
+ else {
+ out.append(PLACEHOLDER)
+ }
+ }
+
+ private static String quote(String key) {
+ StringBuilder out = new StringBuilder(key.length() + 2).append('"')
+ for (int i = 0; i < key.length(); i++) {
+ char c = key.charAt(i)
+ if (c == '"' || c == '\\') {
+ out.append('\\').append(c)
+ }
+ else if (c < 0x20) {
+ out.append(String.format('\\u%04x', (int) c))
+ }
+ else {
+ out.append(c)
+ }
+ }
+ out.append('"').toString()
+ }
+
+ /**
+ * Where a document sits in the command, which decides what its keys are: the options of the
+ * command, kept; the fields and operators of a condition, a pipeline, a sort or a projection, where
+ * a {@code $} key is an operator; an aggregation expression, whose {@code $} keys are operators
+ * too; an update, whose {@code $} keys are update operators; or stored data, the documents an
+ * insert writes, the values an update writes and the values a condition compares with, where a
+ * {@code $} key is a field name like any other and is kept only when the mapping knows it.
+ */
+ @CompileStatic
+ private static final class Scope {
+
+ final String collection
+ final MongoMappedFields fields
+ final Kind kind
+ /**
+ * The update operator whose fields a document of kind {@link Kind#UPDATE_FIELDS} holds, or the stage
+ * whose document one of kind {@link Kind#WRITTEN} is.
+ */
+ final String operator
+ /** The field path of the document; null in a document keyed by fields the mapping does not know. */
+ final List<String> path
+ /** How many indices into the field at the path have been taken: 0 at the field, 1 in one of its elements. */
+ final int depth
+ final String keyText
+
+ private Scope(String collection, MongoMappedFields fields, Kind kind, String operator, List<String> path,
+ int depth, String keyText) {
+ this.collection = collection
+ this.fields = fields
+ this.kind = kind
+ this.operator = operator
+ this.path = path
+ this.depth = depth
+ this.keyText = keyText
+ }
+
+ static Scope options(String collection, MongoMappedFields fields) {
+ new Scope(collection, fields, Kind.OPTIONS, null, Collections.<String>emptyList(), 0, null)
+ }
+
+ /**
+ * @return the scope of an element of an array in this scope: one index deeper into the field
+ * at the path, so that a numeric key inside the element is data unless the elements are arrays;
+ * the elements of an update that is an array are the stages of a pipeline, and an element of an
+ * expression at the top of a document, as the operands of an operator {@code $expr} holds are, is a
+ * value the expression builds, whose keys are kept only when the mapping knows them, as those of a
+ * document a stage of an update pipeline writes are
+ */
+ Scope element() {
+ if (kind == Kind.UPDATE) {
+ return child(Kind.UPDATE_PIPELINE, null, path, depth, keyText)
+ }
+ if (kind == Kind.PUSH) {
+ return child(Kind.DATA, null, path, depth + 2, keyText)
+ }
+ if ((kind == Kind.QUERY || kind == Kind.CONDITION || kind == Kind.EXPRESSION || kind == Kind.DATA) && path != null && !path.isEmpty()) {
+ return child(kind, null, path, depth + 1, keyText)
+ }
+ if (kind == Kind.EXPRESSION && path != null) {
+ return child(Kind.WRITTEN, null, path, depth, keyText)
+ }
+ this
+ }
+
+ /**
+ * @return the scope of the value of a key in a document of this scope, with the key as it is recorded
+ */
+ Scope of(String key, BsonValue value) {
+ if (kind == Kind.OPTIONS) {
+ if (ANY_VALUE.contains(key)) {
+ return child(Kind.DATA, null, null, 0, quote(key))
+ }
+ Kind valueKind = DATA_KEYED.contains(key) ? Kind.DATA :
+ UPDATE_KEYED.contains(key) ? Kind.UPDATE :
+ CONDITION_KEYED.contains(key) ? Kind.CONDITION :
+ FIELD_KEYED.contains(key) ? Kind.QUERY :
+ key == LET ? Kind.EXPRESSION : Kind.OPTIONS
+ return child(valueKind, null, Collections.<String>emptyList(), 0, quote(key))
Review Comment:
A map-reduce `scope` such as `{payload: {'alice@example.com': 1}}` stays as `Kind.OPTIONS`, so the email is written unchanged.
Scope documents are application data. Redact their keys and values.
##########
grails-data-mongodb/core/src/main/groovy/org/apache/grails/data/mongodb/core/querytail/MongoCommandRedactor.groovy:
##########
@@ -0,0 +1,511 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * https://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.grails.data.mongodb.core.querytail
+
+import groovy.transform.CompileStatic
+import org.bson.BsonDocument
+import org.bson.BsonValue
+
+import org.apache.grails.datamapping.core.querytail.QueryTailRedactor
+
+/**
+ * Takes the values out of a MongoDB command before it is captured. A command is a document, so
+ * there is nothing to parse: every value becomes {@link QueryTailRedactor#PLACEHOLDER}, and the
+ * keys, which are the options of the command, its operators and the field names of the collection,
+ * are kept, except where a key may be data. The keys of a document stored in a map property, or in
+ * a property or a collection the {@linkplain MongoMappedFields mapping} does not know, are recorded
+ * as the placeholder too, and so is every segment of a dotted path past such a key. So is a key at
+ * the top of stored data, the documents an insert writes, a replacement document, the fields an
+ * update writes and the operand of a literal, unless the mapping knows it, as the name of a dynamic
+ * attribute may be data; the keys at the top of a condition, a pipeline, a sort or a projection
+ * are the fields the application asks for, and are kept. An update that is a pipeline writes the
+ * documents its stages build: the keys at the top of what a {@code $set}, {@code $addFields} or
+ * {@code $project} stage holds, of the new root of a {@code $replaceWith} or {@code $replaceRoot}, and of a
+ * document {@code $mergeObjects} merges into it are fields written, kept only when the mapping knows
+ * them, as those of a classic update are. A numeric key
+ * is kept only where it indexes an array the mapping knows; in a map, in an element of an array,
+ * which is a map or a document unless the field is a list of lists, or at the top of a document,
+ * it is data. A {@code $} key is kept as an operator only where operators go, in a condition, a
+ * pipeline and the operators of an update; in stored data, the documents an insert writes, a
+ * replacement document and the values an update operator writes, it is a field name, which MongoDB
+ * allows to start with {@code $}, and is recorded as the placeholder unless the mapping knows it.
+ * So is a {@code $} key in the operand of a comparison in a condition, such as {@code $eq} or
+ * {@code $in}, or an implicit document or array equality, which holds the value the field is compared
+ * with, and in a DBRef, whose {@code $ref}, {@code $id} and {@code $db} a condition reads as fields
+ * compared with their whole value, in any order; in an aggregation expression, which
+ * {@code $expr}, the {@code let} of a command and the stages that compute fields hold, the operands
+ * of a comparison are expressions, and their operators are kept, except in the operand of a
+ * {@code $literal} or {@code $const}, which is stored data. A comment, and the constants of an
+ * update, may be any document, and none of their keys are kept. The entries the driver adds for its
+ * session, the cluster time and the transaction, which say nothing about the command, are left out.
+ *
+ * <p>A command is recorded as its name, the database and collection it runs against, and the
+ * document of its remaining entries: {@code find shop.book {"filter": {"title": ?}, "limit": ?}}.</p>
+ *
+ * @since 9.0
+ */
+@CompileStatic
+final class MongoCommandRedactor {
+
+ private static final String PLACEHOLDER = QueryTailRedactor.PLACEHOLDER
+
+ private static final String DATABASE = '$db'
+
+ private static final String COLLECTION = 'collection'
+
+ /** The entries the driver adds, which describe its session and transaction rather than the command. */
+ private static final Set<String> LEFT_OUT = ['lsid', '$clusterTime', 'txnNumber', 'autocommit',
+ 'startTransaction', '$readPreference', 'apiVersion', 'apiStrict', 'apiDeprecationErrors'] as Set<String>
+
+ /**
+ * The options whose document, or documents, are keyed by the fields of the collection, wherever they appear,
+ * including those of the operations of a client-level {@code bulkWrite}.
+ */
+ private static final Set<String> FIELD_KEYED = ['filter', 'query', 'q', 'u', 'update', 'document', 'documents',
+ 'updateMods', 'pipeline', 'sort', 'projection', 'fields', 'key', 'hint', 'min', 'max', 'arrayFilters',
+ 'partialFilterExpression', 'wildcardProjection', 'validator', 'weights'] as Set<String>
+
+ /** Documents that match fields, whose field values may be implicit equality operands. */
+ private static final Set<String> CONDITION_KEYED = ['filter', 'query', 'q', 'arrayFilters',
+ 'partialFilterExpression', 'validator', 'restrictSearchWithMatch'] as Set<String>
+
+ /** The options whose documents are stored data: what an insert writes. */
+ private static final Set<String> DATA_KEYED = ['documents', 'document'] as Set<String>
+
+ /** The options and the operator that may hold any document, none of whose keys say what the command does. */
+ private static final Set<String> ANY_VALUE = ['comment', '$comment', 'c'] as Set<String>
+
+ /** The option whose values are aggregation expressions. */
+ private static final String LET = 'let'
+
+ /** The options that hold an update: update operators, a replacement document or a pipeline. */
+ private static final Set<String> UPDATE_KEYED = ['u', 'update', 'updateMods'] as Set<String>
+
+ /** The update operators whose fields take a condition rather than the value to write. */
+ private static final Set<String> CONDITION_OPERATORS = ['$pull', '$currentDate'] as Set<String>
+
+ /** The update operators that add to an array, whose value may hold modifiers. */
+ private static final Set<String> PUSH_OPERATORS = ['$push', '$addToSet'] as Set<String>
+
+ private static final String EACH = '$each'
+
+ /** The modifiers of a {@code $push} or an {@code $addToSet}. */
+ private static final Set<String> PUSH_MODIFIERS = [EACH, '$slice', '$sort', '$position'] as Set<String>
+
+ /**
+ * The operators of a pipeline whose operand is literal data rather than an expression, including
+ * {@code $const}, which MongoDB accepts for {@code $literal}.
+ */
+ private static final Set<String> LITERAL_OPERATORS = ['$literal', '$const', '$documents'] as Set<String>
+
+ /**
+ * The fields of a DBRef, which a condition reads as fields compared with their whole value, never as
+ * operators, in any order and wherever a condition has fields.
+ */
+ private static final Set<String> DBREF_FIELDS = ['$ref', '$id', '$db'] as Set<String>
+
+ /**
+ * The query operators whose operand holds values rather than a condition: the comparisons, whose operand
+ * is the value a field is compared with, a JSON schema, whose {@code enum} lists values, and a
+ * GeoJSON geometry, which can carry additional data alongside its type and coordinates.
+ */
+ private static final Set<String> VALUE_OPERATORS = ['$eq', '$ne', '$gt', '$gte', '$lt', '$lte', '$in', '$nin',
+ '$all', '$jsonSchema', '$geometry'] as Set<String>
+
+ /** The operator of a condition, and the stages of a pipeline, whose operand is an aggregation expression. */
+ private static final Set<String> EXPRESSION_OPERATORS = ['$expr', '$addFields', '$set', '$project', '$group',
+ '$replaceRoot', '$replaceWith', '$redact', '$bucket', '$bucketAuto', '$sortByCount', '$setWindowFields',
+ '$fill'] as Set<String>
+
+ /** The stages of an update pipeline whose operand is the document they write, or the new root that replaces it. */
+ private static final Set<String> WRITING_STAGES = ['$set', '$addFields', '$project', '$replaceWith'] as Set<String>
+
+ /** The stage of an update pipeline whose options name the new root that replaces the document. */
+ private static final String REPLACE_ROOT = '$replaceRoot'
+
+ /** The option of {@code $replaceRoot} that holds the new root. */
+ private static final String NEW_ROOT = 'newRoot'
+
+ /** The operator whose operands are documents merged into one, which in an update pipeline is written. */
+ private static final String MERGE_OBJECTS = '$mergeObjects'
+
+ /** The kinds of document whose keys at the top are stored data: what an insert, a replacement or an update writes. */
+ private static final Set<Kind> STORED = EnumSet.of(Kind.DATA, Kind.UPDATE, Kind.UPDATE_FIELDS, Kind.WRITTEN)
+
+ private MongoCommandRedactor() {
+ }
+
+ /**
+ * Redacts a command.
+ *
+ * @param database the database the command runs against
+ * @param commandName the name of the command, which is its first key
+ * @param command the command as the driver sends it
+ * @param fields the fields the mapping of the datastore knows, or null before the datastore is built
+ * @return the command with its values replaced
+ */
+ static String redact(String database, String commandName, BsonDocument command, MongoMappedFields fields) {
+ String collection = null
+ String collectionKey = null
+ BsonValue first = command.get(commandName)
+ if (first != null && first.isString()) {
Review Comment:
`{"dropUser": "alice@example.com"}` is recorded as `dropUser admin.alice@example.com`. The first string argument is treated as a collection name, and the driver does not redact `dropUser`, so the account name reaches the capture.
Take a namespace only from commands that target a collection. Redact the other string arguments.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@grails.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add the sql-tail command to find query-heavy pages [grails-core]
Posted by "testlens-app[bot] (via GitHub)" <gi...@apache.org>.
testlens-app[bot] commented on PR #16631:
URL: https://github.com/apache/grails-core/pull/16631#issuecomment-6156485813
## ✅ All tests passed ✅
🏷️ Commit: a22e6fe7e53e6212d6d29befa29ce7bc8bbe379a
▶️ Tests: 102498 executed
⚪️ Checks: 89/89 completed
---
_Learn more about TestLens at [testlens.app/docs](https://testlens.app/docs/features/pr-comment/)._
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@grails.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add the query-tail command to find query-heavy pages [grails-core]
Posted by "jdaugherty (via GitHub)" <gi...@apache.org>.
jdaugherty commented on code in PR #16631:
URL: https://github.com/apache/grails-core/pull/16631#discussion_r4397776714
##########
grails-data-mongodb/core/src/main/groovy/org/apache/grails/data/mongodb/core/querytail/MongoCommandRedactor.groovy:
##########
@@ -0,0 +1,574 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * https://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.grails.data.mongodb.core.querytail
+
+import groovy.transform.CompileStatic
+import org.bson.BsonDocument
+import org.bson.BsonValue
+
+import org.apache.grails.datamapping.core.querytail.QueryTailRedactor
+
+/**
+ * Takes the values out of a MongoDB command before it is captured. A command is a document, so
+ * there is nothing to parse: every value becomes {@link QueryTailRedactor#PLACEHOLDER}, and the
+ * keys, which are the options of the command, its operators and the field names of the collection,
+ * are kept, except where a key may be data. The keys of a document stored in a map property, or in
+ * a property or a collection the {@linkplain MongoMappedFields mapping} does not know, are recorded
+ * as the placeholder too, and so is every segment of a dotted path past such a key. So is a key at
+ * the top of stored data, the documents an insert writes, a replacement document, the fields an
+ * update writes and the operand of a literal, unless the mapping knows it, as the name of a dynamic
+ * attribute may be data; the keys at the top of a condition, a pipeline, a sort or a projection
+ * are the fields the application asks for, and are kept. An update that is a pipeline writes the
+ * documents its stages build: the keys at the top of what a {@code $set}, {@code $addFields} or
+ * {@code $project} stage holds, of the new root of a {@code $replaceWith} or {@code $replaceRoot}, and of a
+ * document {@code $mergeObjects} merges into it are fields written, kept only when the mapping knows
+ * them, as those of a classic update are. A numeric key
+ * is kept only where it indexes an array the mapping knows; in a map, in an element of an array,
+ * which is a map or a document unless the field is a list of lists, or at the top of a document,
+ * it is data. A {@code $} key is kept as an operator only where operators go, in a condition, a
+ * pipeline and the operators of an update, and in a condition only when it is one of MongoDB's query
+ * operators: any other, such as a key beside the geometry of {@code $near}, is recorded as the
+ * placeholder, and so is everything under it; in stored data, the documents an insert writes, a
+ * replacement document and the values an update operator writes, it is a field name, which MongoDB
+ * allows to start with {@code $}, and is recorded as the placeholder unless the mapping knows it.
+ * So is a {@code $} key in the operand of a comparison in a condition, such as {@code $eq} or
+ * {@code $in}, or an implicit document or array equality, which holds the value the field is compared
+ * with, and in a DBRef, whose {@code $ref}, {@code $id} and {@code $db} a condition reads as fields
+ * compared with their whole value, in any order; in an aggregation expression, which
+ * {@code $expr}, the {@code let} of a command and the stages that compute fields hold, the operands
+ * of a comparison are expressions, and their operators are kept, except in the operand of a
+ * {@code $literal} or {@code $const}, which is stored data. A comment, and the constants of an
+ * update, and the scope of a map-reduce command, may be any document, and none of their keys are
+ * kept. The entries the driver adds for its
+ * session, the cluster time and the transaction, which say nothing about the command, are left out.
+ *
+ * <p>An array is recorded with each of its different elements once: an element recorded as one before
+ * it in the array is left out, so that the documents of a batch and the values of an {@code $in} are
+ * recorded the same whatever their number, one or many, and a command is counted as the one statement
+ * it is: {@code {"$in": [?]}}. An empty array is recorded as {@code []}.</p>
+ *
+ * <p>A command is recorded as its name, the database and collection it runs against, and the
+ * document of its remaining entries: {@code find shop.book {"filter": {"title": ?}, "limit": ?}}.
+ * The collection is taken only from a command that runs against one; the first value of any other
+ * command is an argument, such as a user name, and is recorded as a value:
+ * {@code dropUser admin {"dropUser": ?}}.</p>
+ *
+ * @since 9.0
+ */
+@CompileStatic
+final class MongoCommandRedactor {
+
+ private static final String PLACEHOLDER = QueryTailRedactor.PLACEHOLDER
+
+ private static final String DATABASE = '$db'
+
+ private static final String COLLECTION = 'collection'
+
+ /**
+ * The commands whose first value is the collection they run against. The first value of any other
+ * command, such as the user {@code dropUser} drops, is an argument, and is recorded as a value.
+ */
+ private static final Set<String> COLLECTION_COMMANDS = ['find', 'insert', 'update', 'delete', 'aggregate',
+ 'count', 'distinct', 'findAndModify', 'findandmodify', 'mapReduce', 'mapreduce', 'killCursors',
+ 'create', 'drop', 'collMod', 'createIndexes', 'dropIndexes', 'deleteIndexes', 'listIndexes', 'reIndex',
+ 'compact', 'validate', 'collStats', 'convertToCapped', 'cloneCollectionAsCapped', 'analyze',
+ 'checkMetadataConsistency', 'createSearchIndexes', 'dropSearchIndex', 'updateSearchIndex',
+ 'planCacheClear', 'planCacheClearFilters', 'planCacheListFilters', 'planCacheSetFilter'] as Set<String>
+
+ /** The command that names its collection in its {@code collection} option, as it continues a cursor. */
+ private static final String GET_MORE = 'getMore'
+
+ /** The entries the driver adds, which describe its session and transaction rather than the command. */
+ private static final Set<String> LEFT_OUT = ['lsid', '$clusterTime', 'txnNumber', 'autocommit',
+ 'startTransaction', '$readPreference', 'apiVersion', 'apiStrict', 'apiDeprecationErrors'] as Set<String>
+
+ /**
+ * The options whose document, or documents, are keyed by the fields of the collection, wherever they appear,
+ * including those of the operations of a client-level {@code bulkWrite}.
+ */
+ private static final Set<String> FIELD_KEYED = ['filter', 'query', 'q', 'u', 'update', 'document', 'documents',
+ 'updateMods', 'pipeline', 'sort', 'projection', 'fields', 'key', 'hint', 'min', 'max', 'arrayFilters',
+ 'partialFilterExpression', 'wildcardProjection', 'validator', 'weights'] as Set<String>
+
+ /** Documents that match fields, whose field values may be implicit equality operands. */
+ private static final Set<String> CONDITION_KEYED = ['filter', 'query', 'q', 'arrayFilters',
+ 'partialFilterExpression', 'validator', 'restrictSearchWithMatch'] as Set<String>
+
+ /** The options whose documents are stored data: what an insert writes. */
+ private static final Set<String> DATA_KEYED = ['documents', 'document'] as Set<String>
+
+ /** The options and the operator that may hold any document, none of whose keys say what the command does. */
+ private static final Set<String> ANY_VALUE = ['comment', '$comment', 'c', 'scope'] as Set<String>
+
+ /** The option whose values are aggregation expressions. */
+ private static final String LET = 'let'
+
+ /** The options that hold an update: update operators, a replacement document or a pipeline. */
+ private static final Set<String> UPDATE_KEYED = ['u', 'update', 'updateMods'] as Set<String>
+
+ /** The update operators whose fields take a condition rather than the value to write. */
+ private static final Set<String> CONDITION_OPERATORS = ['$pull', '$currentDate'] as Set<String>
+
+ /** The update operators that add to an array, whose value may hold modifiers. */
+ private static final Set<String> PUSH_OPERATORS = ['$push', '$addToSet'] as Set<String>
+
+ private static final String EACH = '$each'
+
+ /** The modifiers of a {@code $push} or an {@code $addToSet}. */
+ private static final Set<String> PUSH_MODIFIERS = [EACH, '$slice', '$sort', '$position'] as Set<String>
+
+ /**
+ * The operators of a pipeline whose operand is literal data rather than an expression, including
+ * {@code $const}, which MongoDB accepts for {@code $literal}.
+ */
+ private static final Set<String> LITERAL_OPERATORS = ['$literal', '$const', '$documents'] as Set<String>
+
+ /**
+ * The fields of a DBRef, which a condition reads as fields compared with their whole value, never as
+ * operators, in any order and wherever a condition has fields.
+ */
+ private static final Set<String> DBREF_FIELDS = ['$ref', '$id', '$db'] as Set<String>
+
+ /**
+ * The query operators whose operand holds values rather than a condition: the comparisons, whose operand
+ * is the value a field is compared with, a JSON schema, whose {@code enum} lists values, and a
+ * GeoJSON geometry, which can carry additional data alongside its type and coordinates.
+ */
+ private static final Set<String> VALUE_OPERATORS = ['$eq', '$ne', '$gt', '$gte', '$lt', '$lte', '$in', '$nin',
+ '$all', '$jsonSchema', '$geometry'] as Set<String>
+
+ /**
+ * The operators of a condition, the only {@code $} keys a condition holds but the fields of a DBRef: the query
+ * operators of MongoDB, and the options of {@code $regex}, {@code $text} and the geospatial ones.
+ */
+ private static final Set<String> QUERY_OPERATORS = ['$eq', '$ne', '$gt', '$gte', '$lt', '$lte', '$in', '$nin',
+ '$and', '$or', '$not', '$nor', '$exists', '$type', '$expr', '$jsonSchema', '$mod', '$regex', '$options',
+ '$text', '$search', '$language', '$caseSensitive', '$diacriticSensitive', '$where', '$geoIntersects',
+ '$geoWithin', '$near', '$nearSphere', '$geometry', '$maxDistance', '$minDistance', '$box', '$center',
+ '$centerSphere', '$polygon', '$uniqueDocs', '$all', '$elemMatch', '$size', '$bitsAllClear', '$bitsAllSet',
+ '$bitsAnyClear', '$bitsAnySet', '$comment', '$rand', '$sampleRate'] as Set<String>
+
+ /** The operator of a condition, and the stages of a pipeline, whose operand is an aggregation expression. */
+ private static final Set<String> EXPRESSION_OPERATORS = ['$expr', '$addFields', '$set', '$project', '$group',
+ '$replaceRoot', '$replaceWith', '$redact', '$bucket', '$bucketAuto', '$sortByCount', '$setWindowFields',
+ '$fill'] as Set<String>
+
+ /** The stages of an update pipeline whose operand is the document they write, or the new root that replaces it. */
+ private static final Set<String> WRITING_STAGES = ['$set', '$addFields', '$project', '$replaceWith'] as Set<String>
+
+ /** The stage of an update pipeline whose options name the new root that replaces the document. */
+ private static final String REPLACE_ROOT = '$replaceRoot'
+
+ /** The option of {@code $replaceRoot} that holds the new root. */
+ private static final String NEW_ROOT = 'newRoot'
+
+ /** The operator whose operands are documents merged into one, which in an update pipeline is written. */
+ private static final String MERGE_OBJECTS = '$mergeObjects'
+
+ /** The kinds of document whose keys at the top are stored data: what an insert, a replacement or an update writes. */
+ private static final Set<Kind> STORED = EnumSet.of(Kind.DATA, Kind.UPDATE, Kind.UPDATE_FIELDS, Kind.WRITTEN)
+
+ private MongoCommandRedactor() {
+ }
+
+ /**
+ * Redacts a command.
+ *
+ * @param database the database the command runs against
+ * @param commandName the name of the command, which is its first key
+ * @param command the command as the driver sends it
+ * @param fields the fields the mapping of the datastore knows, or null before the datastore is built
+ * @return the command with its values replaced
+ */
+ static String redact(String database, String commandName, BsonDocument command, MongoMappedFields fields) {
+ String collection = null
+ String collectionKey = null
+ BsonValue first = command.get(commandName)
+ if (first != null && first.isString() && COLLECTION_COMMANDS.contains(commandName)) {
+ collection = first.asString().value
+ collectionKey = commandName
+ }
+ else if (commandName == GET_MORE && command.get(COLLECTION)?.isString()) {
+ collection = command.getString(COLLECTION).value
+ collectionKey = COLLECTION
+ }
+ StringBuilder out = new StringBuilder(commandName)
+ if (database != null || collection != null) {
+ out.append(' ')
+ if (database != null) {
+ out.append(database)
+ }
+ if (collection != null) {
+ out.append(database != null ? '.' : '').append(collection)
+ }
+ }
+ Scope options = Scope.options(collection, fields)
+ StringBuilder body = new StringBuilder()
+ for (Map.Entry<String, BsonValue> entry in command.entrySet()) {
+ String key = entry.key
+ if (key == DATABASE || key == collectionKey || LEFT_OUT.contains(key)) {
+ continue
+ }
+ body.append(body.length() == 0 ? '{' : ', ')
+ appendEntry(body, key, entry.value, options)
+ }
+ if (body.length() > 0) {
+ out.append(' ').append(body).append('}')
+ }
+ out.toString()
+ }
+
+ private static void appendEntry(StringBuilder out, String key, BsonValue value, Scope scope) {
+ Scope child = scope.of(key, value)
+ out.append(child.keyText).append(': ')
+ appendValue(out, value, child)
+ }
+
+ private static void appendValue(StringBuilder out, BsonValue value, Scope scope) {
+ if (value.isDocument()) {
+ out.append('{')
+ boolean first = true
+ for (Map.Entry<String, BsonValue> entry in value.asDocument().entrySet()) {
+ if (!first) {
+ out.append(', ')
+ }
+ first = false
+ appendEntry(out, entry.key, entry.value, scope)
+ }
+ out.append('}')
+ }
+ else if (value.isArray()) {
+ out.append('[')
+ Scope element = scope.element()
+ // a value is recorded as the placeholder whatever it is, so only a document or an array is rendered
+ // to be told apart from the elements before it, and a list of values costs no more than a look at each
+ boolean placeholder = false
+ Set<String> written = null
+ boolean first = true
+ for (BsonValue item in value.asArray()) {
+ String recorded
+ if (item.isDocument() || item.isArray()) {
+ StringBuilder text = new StringBuilder()
+ appendValue(text, item, element)
+ recorded = text.toString()
+ if (written == null) {
+ written = new HashSet<>()
+ }
+ if (!written.add(recorded)) {
+ continue
+ }
+ }
+ else if (placeholder) {
+ continue
+ }
+ else {
+ placeholder = true
+ recorded = PLACEHOLDER
+ }
+ if (!first) {
+ out.append(', ')
+ }
+ first = false
+ out.append(recorded)
+ }
+ out.append(']')
+ }
+ else {
+ out.append(PLACEHOLDER)
+ }
+ }
+
+ private static String quote(String key) {
+ StringBuilder out = new StringBuilder(key.length() + 2).append('"')
+ for (int i = 0; i < key.length(); i++) {
+ char c = key.charAt(i)
+ if (c == '"' || c == '\\') {
+ out.append('\\').append(c)
+ }
+ else if (c < 0x20) {
+ out.append(String.format('\\u%04x', (int) c))
+ }
+ else {
+ out.append(c)
+ }
+ }
+ out.append('"').toString()
+ }
+
+ /**
+ * Where a document sits in the command, which decides what its keys are: the options of the
+ * command, kept; the fields and operators of a condition, a pipeline, a sort or a projection, where
+ * a {@code $} key is an operator; an aggregation expression, whose {@code $} keys are operators
+ * too; an update, whose {@code $} keys are update operators; or stored data, the documents an
+ * insert writes, the values an update writes and the values a condition compares with, where a
+ * {@code $} key is a field name like any other and is kept only when the mapping knows it.
+ */
+ @CompileStatic
+ private static final class Scope {
+
+ final String collection
+ final MongoMappedFields fields
+ final Kind kind
+ /**
+ * The update operator whose fields a document of kind {@link Kind#UPDATE_FIELDS} holds, or the stage
+ * whose document one of kind {@link Kind#WRITTEN} is.
+ */
+ final String operator
+ /** The field path of the document; null in a document keyed by fields the mapping does not know. */
+ final List<String> path
+ /** How many indices into the field at the path have been taken: 0 at the field, 1 in one of its elements. */
+ final int depth
+ final String keyText
+
+ private Scope(String collection, MongoMappedFields fields, Kind kind, String operator, List<String> path,
+ int depth, String keyText) {
+ this.collection = collection
+ this.fields = fields
+ this.kind = kind
+ this.operator = operator
+ this.path = path
+ this.depth = depth
+ this.keyText = keyText
+ }
+
+ static Scope options(String collection, MongoMappedFields fields) {
+ new Scope(collection, fields, Kind.OPTIONS, null, Collections.<String>emptyList(), 0, null)
+ }
+
+ /**
+ * @return the scope of an element of an array in this scope: one index deeper into the field
+ * at the path, so that a numeric key inside the element is data unless the elements are arrays;
+ * the elements of an update that is an array are the stages of a pipeline, and an element of an
+ * expression at the top of a document, as the operands of an operator {@code $expr} holds are, is a
+ * value the expression builds, whose keys are kept only when the mapping knows them, as those of a
+ * document a stage of an update pipeline writes are
+ */
+ Scope element() {
+ if (kind == Kind.UPDATE) {
+ return child(Kind.UPDATE_PIPELINE, null, path, depth, keyText)
+ }
+ if (kind == Kind.PUSH) {
+ return child(Kind.DATA, null, path, depth + 2, keyText)
+ }
+ if ((kind == Kind.QUERY || kind == Kind.CONDITION || kind == Kind.EXPRESSION || kind == Kind.DATA) && path != null && !path.isEmpty()) {
+ return child(kind, null, path, depth + 1, keyText)
+ }
+ if (kind == Kind.EXPRESSION && path != null) {
+ return child(Kind.WRITTEN, null, path, depth, keyText)
+ }
+ this
+ }
+
+ /**
+ * @return the scope of the value of a key in a document of this scope, with the key as it is recorded
+ */
+ Scope of(String key, BsonValue value) {
+ if (kind == Kind.OPTIONS) {
+ if (ANY_VALUE.contains(key)) {
+ return child(Kind.DATA, null, null, 0, quote(key))
+ }
+ Kind valueKind = DATA_KEYED.contains(key) ? Kind.DATA :
+ UPDATE_KEYED.contains(key) ? Kind.UPDATE :
+ CONDITION_KEYED.contains(key) ? Kind.CONDITION :
+ FIELD_KEYED.contains(key) ? Kind.QUERY :
+ key == LET ? Kind.EXPRESSION : Kind.OPTIONS
+ return child(valueKind, null, Collections.<String>emptyList(), 0, quote(key))
+ }
+ if ((kind == Kind.QUERY || kind == Kind.CONDITION) && key.startsWith('$')) {
+ if (ANY_VALUE.contains(key)) {
+ return child(Kind.DATA, null, null, 0, quote(key))
+ }
+ if (kind == Kind.CONDITION && DBREF_FIELDS.contains(key)) {
+ return field(key, Kind.DATA)
+ }
+ if (kind == Kind.CONDITION && !QUERY_OPERATORS.contains(key)) {
Review Comment:
Fixed in 45b61d2aa2. `$within` is now one of the query operators a condition keeps. It is the older name of `$geoWithin`, and GORM writes it for `WithinBox`, `WithinPolygon` and `WithinCircle`. `$box`, `$polygon` and `$center` were already in the list, so `location.$within.$box` is now recorded as `{"location": {"$within": {"$box": [[?]]}}}`. A `$` key beside the shape that is not an operator is still recorded as `?`, along with everything under it.
The redactor spec has rows for all three shapes, plus one with an unknown key beside `$box`. `QueryTailCommandListenerSpec` runs `findByLocationWithinBox`, `findByLocationWithinPolygon` and `findByLocationWithinCircle` against MongoDB: each finds its document, and the capture keeps `$within` and the shape operator.
##########
grails-data-mongodb/core/src/main/groovy/org/apache/grails/data/mongodb/core/querytail/MongoCommandRedactor.groovy:
##########
@@ -0,0 +1,574 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * https://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.grails.data.mongodb.core.querytail
+
+import groovy.transform.CompileStatic
+import org.bson.BsonDocument
+import org.bson.BsonValue
+
+import org.apache.grails.datamapping.core.querytail.QueryTailRedactor
+
+/**
+ * Takes the values out of a MongoDB command before it is captured. A command is a document, so
+ * there is nothing to parse: every value becomes {@link QueryTailRedactor#PLACEHOLDER}, and the
+ * keys, which are the options of the command, its operators and the field names of the collection,
+ * are kept, except where a key may be data. The keys of a document stored in a map property, or in
+ * a property or a collection the {@linkplain MongoMappedFields mapping} does not know, are recorded
+ * as the placeholder too, and so is every segment of a dotted path past such a key. So is a key at
+ * the top of stored data, the documents an insert writes, a replacement document, the fields an
+ * update writes and the operand of a literal, unless the mapping knows it, as the name of a dynamic
+ * attribute may be data; the keys at the top of a condition, a pipeline, a sort or a projection
+ * are the fields the application asks for, and are kept. An update that is a pipeline writes the
+ * documents its stages build: the keys at the top of what a {@code $set}, {@code $addFields} or
+ * {@code $project} stage holds, of the new root of a {@code $replaceWith} or {@code $replaceRoot}, and of a
+ * document {@code $mergeObjects} merges into it are fields written, kept only when the mapping knows
+ * them, as those of a classic update are. A numeric key
+ * is kept only where it indexes an array the mapping knows; in a map, in an element of an array,
+ * which is a map or a document unless the field is a list of lists, or at the top of a document,
+ * it is data. A {@code $} key is kept as an operator only where operators go, in a condition, a
+ * pipeline and the operators of an update, and in a condition only when it is one of MongoDB's query
+ * operators: any other, such as a key beside the geometry of {@code $near}, is recorded as the
+ * placeholder, and so is everything under it; in stored data, the documents an insert writes, a
+ * replacement document and the values an update operator writes, it is a field name, which MongoDB
+ * allows to start with {@code $}, and is recorded as the placeholder unless the mapping knows it.
+ * So is a {@code $} key in the operand of a comparison in a condition, such as {@code $eq} or
+ * {@code $in}, or an implicit document or array equality, which holds the value the field is compared
+ * with, and in a DBRef, whose {@code $ref}, {@code $id} and {@code $db} a condition reads as fields
+ * compared with their whole value, in any order; in an aggregation expression, which
+ * {@code $expr}, the {@code let} of a command and the stages that compute fields hold, the operands
+ * of a comparison are expressions, and their operators are kept, except in the operand of a
+ * {@code $literal} or {@code $const}, which is stored data. A comment, and the constants of an
+ * update, and the scope of a map-reduce command, may be any document, and none of their keys are
+ * kept. The entries the driver adds for its
+ * session, the cluster time and the transaction, which say nothing about the command, are left out.
+ *
+ * <p>An array is recorded with each of its different elements once: an element recorded as one before
+ * it in the array is left out, so that the documents of a batch and the values of an {@code $in} are
+ * recorded the same whatever their number, one or many, and a command is counted as the one statement
+ * it is: {@code {"$in": [?]}}. An empty array is recorded as {@code []}.</p>
+ *
+ * <p>A command is recorded as its name, the database and collection it runs against, and the
+ * document of its remaining entries: {@code find shop.book {"filter": {"title": ?}, "limit": ?}}.
+ * The collection is taken only from a command that runs against one; the first value of any other
+ * command is an argument, such as a user name, and is recorded as a value:
+ * {@code dropUser admin {"dropUser": ?}}.</p>
+ *
+ * @since 9.0
+ */
+@CompileStatic
+final class MongoCommandRedactor {
+
+ private static final String PLACEHOLDER = QueryTailRedactor.PLACEHOLDER
+
+ private static final String DATABASE = '$db'
+
+ private static final String COLLECTION = 'collection'
+
+ /**
+ * The commands whose first value is the collection they run against. The first value of any other
+ * command, such as the user {@code dropUser} drops, is an argument, and is recorded as a value.
+ */
+ private static final Set<String> COLLECTION_COMMANDS = ['find', 'insert', 'update', 'delete', 'aggregate',
+ 'count', 'distinct', 'findAndModify', 'findandmodify', 'mapReduce', 'mapreduce', 'killCursors',
+ 'create', 'drop', 'collMod', 'createIndexes', 'dropIndexes', 'deleteIndexes', 'listIndexes', 'reIndex',
+ 'compact', 'validate', 'collStats', 'convertToCapped', 'cloneCollectionAsCapped', 'analyze',
+ 'checkMetadataConsistency', 'createSearchIndexes', 'dropSearchIndex', 'updateSearchIndex',
+ 'planCacheClear', 'planCacheClearFilters', 'planCacheListFilters', 'planCacheSetFilter'] as Set<String>
+
+ /** The command that names its collection in its {@code collection} option, as it continues a cursor. */
+ private static final String GET_MORE = 'getMore'
+
+ /** The entries the driver adds, which describe its session and transaction rather than the command. */
+ private static final Set<String> LEFT_OUT = ['lsid', '$clusterTime', 'txnNumber', 'autocommit',
+ 'startTransaction', '$readPreference', 'apiVersion', 'apiStrict', 'apiDeprecationErrors'] as Set<String>
+
+ /**
+ * The options whose document, or documents, are keyed by the fields of the collection, wherever they appear,
+ * including those of the operations of a client-level {@code bulkWrite}.
+ */
+ private static final Set<String> FIELD_KEYED = ['filter', 'query', 'q', 'u', 'update', 'document', 'documents',
+ 'updateMods', 'pipeline', 'sort', 'projection', 'fields', 'key', 'hint', 'min', 'max', 'arrayFilters',
+ 'partialFilterExpression', 'wildcardProjection', 'validator', 'weights'] as Set<String>
+
+ /** Documents that match fields, whose field values may be implicit equality operands. */
+ private static final Set<String> CONDITION_KEYED = ['filter', 'query', 'q', 'arrayFilters',
+ 'partialFilterExpression', 'validator', 'restrictSearchWithMatch'] as Set<String>
+
+ /** The options whose documents are stored data: what an insert writes. */
+ private static final Set<String> DATA_KEYED = ['documents', 'document'] as Set<String>
+
+ /** The options and the operator that may hold any document, none of whose keys say what the command does. */
+ private static final Set<String> ANY_VALUE = ['comment', '$comment', 'c', 'scope'] as Set<String>
+
+ /** The option whose values are aggregation expressions. */
+ private static final String LET = 'let'
+
+ /** The options that hold an update: update operators, a replacement document or a pipeline. */
+ private static final Set<String> UPDATE_KEYED = ['u', 'update', 'updateMods'] as Set<String>
+
+ /** The update operators whose fields take a condition rather than the value to write. */
+ private static final Set<String> CONDITION_OPERATORS = ['$pull', '$currentDate'] as Set<String>
+
+ /** The update operators that add to an array, whose value may hold modifiers. */
+ private static final Set<String> PUSH_OPERATORS = ['$push', '$addToSet'] as Set<String>
+
+ private static final String EACH = '$each'
+
+ /** The modifiers of a {@code $push} or an {@code $addToSet}. */
+ private static final Set<String> PUSH_MODIFIERS = [EACH, '$slice', '$sort', '$position'] as Set<String>
+
+ /**
+ * The operators of a pipeline whose operand is literal data rather than an expression, including
+ * {@code $const}, which MongoDB accepts for {@code $literal}.
+ */
+ private static final Set<String> LITERAL_OPERATORS = ['$literal', '$const', '$documents'] as Set<String>
+
+ /**
+ * The fields of a DBRef, which a condition reads as fields compared with their whole value, never as
+ * operators, in any order and wherever a condition has fields.
+ */
+ private static final Set<String> DBREF_FIELDS = ['$ref', '$id', '$db'] as Set<String>
+
+ /**
+ * The query operators whose operand holds values rather than a condition: the comparisons, whose operand
+ * is the value a field is compared with, a JSON schema, whose {@code enum} lists values, and a
+ * GeoJSON geometry, which can carry additional data alongside its type and coordinates.
+ */
+ private static final Set<String> VALUE_OPERATORS = ['$eq', '$ne', '$gt', '$gte', '$lt', '$lte', '$in', '$nin',
+ '$all', '$jsonSchema', '$geometry'] as Set<String>
+
+ /**
+ * The operators of a condition, the only {@code $} keys a condition holds but the fields of a DBRef: the query
+ * operators of MongoDB, and the options of {@code $regex}, {@code $text} and the geospatial ones.
+ */
+ private static final Set<String> QUERY_OPERATORS = ['$eq', '$ne', '$gt', '$gte', '$lt', '$lte', '$in', '$nin',
+ '$and', '$or', '$not', '$nor', '$exists', '$type', '$expr', '$jsonSchema', '$mod', '$regex', '$options',
+ '$text', '$search', '$language', '$caseSensitive', '$diacriticSensitive', '$where', '$geoIntersects',
+ '$geoWithin', '$near', '$nearSphere', '$geometry', '$maxDistance', '$minDistance', '$box', '$center',
+ '$centerSphere', '$polygon', '$uniqueDocs', '$all', '$elemMatch', '$size', '$bitsAllClear', '$bitsAllSet',
+ '$bitsAnyClear', '$bitsAnySet', '$comment', '$rand', '$sampleRate'] as Set<String>
+
+ /** The operator of a condition, and the stages of a pipeline, whose operand is an aggregation expression. */
+ private static final Set<String> EXPRESSION_OPERATORS = ['$expr', '$addFields', '$set', '$project', '$group',
+ '$replaceRoot', '$replaceWith', '$redact', '$bucket', '$bucketAuto', '$sortByCount', '$setWindowFields',
+ '$fill'] as Set<String>
+
+ /** The stages of an update pipeline whose operand is the document they write, or the new root that replaces it. */
+ private static final Set<String> WRITING_STAGES = ['$set', '$addFields', '$project', '$replaceWith'] as Set<String>
+
+ /** The stage of an update pipeline whose options name the new root that replaces the document. */
+ private static final String REPLACE_ROOT = '$replaceRoot'
+
+ /** The option of {@code $replaceRoot} that holds the new root. */
+ private static final String NEW_ROOT = 'newRoot'
+
+ /** The operator whose operands are documents merged into one, which in an update pipeline is written. */
+ private static final String MERGE_OBJECTS = '$mergeObjects'
+
+ /** The kinds of document whose keys at the top are stored data: what an insert, a replacement or an update writes. */
+ private static final Set<Kind> STORED = EnumSet.of(Kind.DATA, Kind.UPDATE, Kind.UPDATE_FIELDS, Kind.WRITTEN)
+
+ private MongoCommandRedactor() {
+ }
+
+ /**
+ * Redacts a command.
+ *
+ * @param database the database the command runs against
+ * @param commandName the name of the command, which is its first key
+ * @param command the command as the driver sends it
+ * @param fields the fields the mapping of the datastore knows, or null before the datastore is built
+ * @return the command with its values replaced
+ */
+ static String redact(String database, String commandName, BsonDocument command, MongoMappedFields fields) {
+ String collection = null
+ String collectionKey = null
+ BsonValue first = command.get(commandName)
+ if (first != null && first.isString() && COLLECTION_COMMANDS.contains(commandName)) {
+ collection = first.asString().value
+ collectionKey = commandName
+ }
+ else if (commandName == GET_MORE && command.get(COLLECTION)?.isString()) {
+ collection = command.getString(COLLECTION).value
+ collectionKey = COLLECTION
+ }
+ StringBuilder out = new StringBuilder(commandName)
+ if (database != null || collection != null) {
+ out.append(' ')
+ if (database != null) {
+ out.append(database)
+ }
+ if (collection != null) {
+ out.append(database != null ? '.' : '').append(collection)
+ }
+ }
+ Scope options = Scope.options(collection, fields)
+ StringBuilder body = new StringBuilder()
+ for (Map.Entry<String, BsonValue> entry in command.entrySet()) {
+ String key = entry.key
+ if (key == DATABASE || key == collectionKey || LEFT_OUT.contains(key)) {
+ continue
+ }
+ body.append(body.length() == 0 ? '{' : ', ')
+ appendEntry(body, key, entry.value, options)
+ }
+ if (body.length() > 0) {
+ out.append(' ').append(body).append('}')
+ }
+ out.toString()
+ }
+
+ private static void appendEntry(StringBuilder out, String key, BsonValue value, Scope scope) {
+ Scope child = scope.of(key, value)
+ out.append(child.keyText).append(': ')
+ appendValue(out, value, child)
+ }
+
+ private static void appendValue(StringBuilder out, BsonValue value, Scope scope) {
+ if (value.isDocument()) {
+ out.append('{')
+ boolean first = true
+ for (Map.Entry<String, BsonValue> entry in value.asDocument().entrySet()) {
+ if (!first) {
+ out.append(', ')
+ }
+ first = false
+ appendEntry(out, entry.key, entry.value, scope)
+ }
+ out.append('}')
+ }
+ else if (value.isArray()) {
+ out.append('[')
+ Scope element = scope.element()
+ // a value is recorded as the placeholder whatever it is, so only a document or an array is rendered
+ // to be told apart from the elements before it, and a list of values costs no more than a look at each
+ boolean placeholder = false
+ Set<String> written = null
+ boolean first = true
+ for (BsonValue item in value.asArray()) {
+ String recorded
+ if (item.isDocument() || item.isArray()) {
+ StringBuilder text = new StringBuilder()
+ appendValue(text, item, element)
+ recorded = text.toString()
+ if (written == null) {
+ written = new HashSet<>()
+ }
+ if (!written.add(recorded)) {
+ continue
+ }
+ }
+ else if (placeholder) {
+ continue
+ }
+ else {
+ placeholder = true
+ recorded = PLACEHOLDER
+ }
+ if (!first) {
+ out.append(', ')
+ }
+ first = false
+ out.append(recorded)
+ }
+ out.append(']')
+ }
+ else {
+ out.append(PLACEHOLDER)
+ }
+ }
+
+ private static String quote(String key) {
+ StringBuilder out = new StringBuilder(key.length() + 2).append('"')
+ for (int i = 0; i < key.length(); i++) {
+ char c = key.charAt(i)
+ if (c == '"' || c == '\\') {
+ out.append('\\').append(c)
+ }
+ else if (c < 0x20) {
+ out.append(String.format('\\u%04x', (int) c))
+ }
+ else {
+ out.append(c)
+ }
+ }
+ out.append('"').toString()
+ }
+
+ /**
+ * Where a document sits in the command, which decides what its keys are: the options of the
+ * command, kept; the fields and operators of a condition, a pipeline, a sort or a projection, where
+ * a {@code $} key is an operator; an aggregation expression, whose {@code $} keys are operators
+ * too; an update, whose {@code $} keys are update operators; or stored data, the documents an
+ * insert writes, the values an update writes and the values a condition compares with, where a
+ * {@code $} key is a field name like any other and is kept only when the mapping knows it.
+ */
+ @CompileStatic
+ private static final class Scope {
+
+ final String collection
+ final MongoMappedFields fields
+ final Kind kind
+ /**
+ * The update operator whose fields a document of kind {@link Kind#UPDATE_FIELDS} holds, or the stage
+ * whose document one of kind {@link Kind#WRITTEN} is.
+ */
+ final String operator
+ /** The field path of the document; null in a document keyed by fields the mapping does not know. */
+ final List<String> path
+ /** How many indices into the field at the path have been taken: 0 at the field, 1 in one of its elements. */
+ final int depth
+ final String keyText
+
+ private Scope(String collection, MongoMappedFields fields, Kind kind, String operator, List<String> path,
+ int depth, String keyText) {
+ this.collection = collection
+ this.fields = fields
+ this.kind = kind
+ this.operator = operator
+ this.path = path
+ this.depth = depth
+ this.keyText = keyText
+ }
+
+ static Scope options(String collection, MongoMappedFields fields) {
+ new Scope(collection, fields, Kind.OPTIONS, null, Collections.<String>emptyList(), 0, null)
+ }
+
+ /**
+ * @return the scope of an element of an array in this scope: one index deeper into the field
+ * at the path, so that a numeric key inside the element is data unless the elements are arrays;
+ * the elements of an update that is an array are the stages of a pipeline, and an element of an
+ * expression at the top of a document, as the operands of an operator {@code $expr} holds are, is a
+ * value the expression builds, whose keys are kept only when the mapping knows them, as those of a
+ * document a stage of an update pipeline writes are
+ */
+ Scope element() {
+ if (kind == Kind.UPDATE) {
+ return child(Kind.UPDATE_PIPELINE, null, path, depth, keyText)
+ }
+ if (kind == Kind.PUSH) {
+ return child(Kind.DATA, null, path, depth + 2, keyText)
+ }
+ if ((kind == Kind.QUERY || kind == Kind.CONDITION || kind == Kind.EXPRESSION || kind == Kind.DATA) && path != null && !path.isEmpty()) {
+ return child(kind, null, path, depth + 1, keyText)
+ }
+ if (kind == Kind.EXPRESSION && path != null) {
+ return child(Kind.WRITTEN, null, path, depth, keyText)
+ }
+ this
+ }
+
+ /**
+ * @return the scope of the value of a key in a document of this scope, with the key as it is recorded
+ */
+ Scope of(String key, BsonValue value) {
+ if (kind == Kind.OPTIONS) {
+ if (ANY_VALUE.contains(key)) {
+ return child(Kind.DATA, null, null, 0, quote(key))
+ }
+ Kind valueKind = DATA_KEYED.contains(key) ? Kind.DATA :
+ UPDATE_KEYED.contains(key) ? Kind.UPDATE :
+ CONDITION_KEYED.contains(key) ? Kind.CONDITION :
+ FIELD_KEYED.contains(key) ? Kind.QUERY :
+ key == LET ? Kind.EXPRESSION : Kind.OPTIONS
Review Comment:
Fixed in 45b61d2aa2, for `applyOps` and for every other option of this kind. Until now, a document under an option the redactor did not recognise was read as more options, so its keys were kept. Now it reads a document under an option only when it knows the option holds one: fields, a condition, a pipeline, an update, stored data, or more options. Options in that last group are the write and read concerns, `collation`, `cursor`, the `updates`/`deletes` statements, `indexes`, the `ops`/`nsInfo` of a client `bulkWrite`, `applyOps` and its `preCondition`, the map-reduce `out`, the command `explain` runs, and the `timeseries`/`clusteredIndex`/`changeStreamPreAndPostImages` options of `create`. Any other option's document is data, and none of its keys are recorded. So the `o` and `o2` of an `applyOps` operation and the `res` of a precondition are now `{?: {?: ?}}`, while `op` and `ns` are still kept: `applyOps admin {"applyOps": [{"op": ?, "ns": ?, "o": {?: ?, ?: {?: ?}}}]}`. This also
covers `comment`, the `c` constants of an update and the map-reduce `scope`, which used to be listed one by one. It also covers any option a later server adds.
Spec rows cover the `applyOps` payload, a precondition's `res`, an unknown option at the top of a command and inside a write statement, a comment, and `explain` and `writeConcern`, which keep their keys.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@grails.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add the query-tail command to find query-heavy pages [grails-core]
Posted by "jdaugherty (via GitHub)" <gi...@apache.org>.
jdaugherty commented on PR #16631:
URL: https://github.com/apache/grails-core/pull/16631#issuecomment-6460570620
Pushed two more commits on top of 72d9720740: cae80795d1 fixes when captures open and close and what is recorded, and ecd64813fa lets the simple datastore capture connection sources added at runtime. The head is now ecd64813fa.
**When a capture opens and closes**
- **A datastore that fails to build (MongoDB, simple datastore):** a datastore now opens the captures of its connections only once it is fully built, together with those of its child datastores. One that throws partway through its constructor no longer leaves the file open and locked, which made the next datastore in the JVM add to the stale run instead of starting a new one.
- **Schema drop (Hibernate 5 and 7):** the capture now closes when the session factory starts to close, so the statements `create-drop` runs to drop the schema are not recorded. The statements that create it never were.
- **How the observer is installed (Hibernate 5 and 7):** the capture's observer, and the Hibernate 5 observer that destroys the service registry, are added to the session factory's builder through a `SessionFactoryBuilderFactory` that the configuration's class-loader service returns. The configuration's own `sessionFactoryObserver` is no longer swapped out during a build and put back afterwards.
- **Connection providers added as services (Hibernate 5 and 7):** a `ConnectionProvider` that a configuration class registers with `addService` replaces the one Hibernate would choose, so the capture never saw it. It is now wrapped right after the service registry is built.
- **Two applications in one JVM:** when a second application in the same JVM (with its own copy of the capture classes, such as a second web application in one container) finds the file locked by the first, it now stops with a warning telling it to set its own `grails.gorm.queryTail.file`. Before, it could keep writing without holding any lock. File locks belong to the process, so neither application could keep its own lock, or the other's, once the other closed the file.
- **Neo4j around an application's driver:** only `grails.gorm.queryTail` is read, through the new `QueryTailSettings.of(PropertyResolver)`. Before, the whole `grails.gorm` tree was bound, so a `grails.gorm` setting meant for another datastore, such as a tenant resolver class it can't load, could fail the Neo4j datastore.
- **Runtime connection sources (simple datastore):** a connection source added at runtime, as `addTenantForSchema` adds one, now uses the application's query-tail settings unless its own configuration sets them. Before, it never captured.
**What is recorded and shown**
- **Subqueries in a select list:** a select whose column list contains a subquery, as a formula property adds, is shown in full. Selects of the same table that read different tables in their subqueries no longer count as one statement.
- **MongoDB arrays:** an array is recorded as its different elements, each once, without a trailing `...`. One element and many give the same statement (`{"$in": [?]}`), and an empty array is `[]`. Plain values (anything other than a document or an array) are no longer rendered one by one, so a large `$in` costs a single pass.
- **A `--file` that can't be read:** the command now reports a capture file that is a directory or can't be read, and exits with 1. Before, it showed an empty summary or kept waiting forever.
**Tests:** I checked that each new spec fails without its fix. The one for a directory found while the command is following the file hangs without the fix, so it has a 30-second timeout. Specs that imitated another JVM writing the file by holding its lock from the same JVM now start a real second JVM, because a lock held within the same JVM now means a second application.
The guide is updated to match.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@grails.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add the sql-tail command to find query-heavy pages [grails-core]
Posted by "jdaugherty (via GitHub)" <gi...@apache.org>.
jdaugherty commented on code in PR #16631:
URL: https://github.com/apache/grails-core/pull/16631#discussion_r4352616291
##########
grails-datamapping-core/src/main/groovy/org/grails/datastore/gorm/jdbc/sqltail/SqlTailRedactor.java:
##########
@@ -0,0 +1,360 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * https://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.grails.datastore.gorm.jdbc.sqltail;
+
+import java.util.ArrayList;
+import java.util.List;
+
+/**
+ * Takes the values out of a statement before it is captured. Every string, number and
+ * dollar-quoted literal becomes {@link #PLACEHOLDER} and every comment is dropped, so that a value
+ * written into the SQL text, a password in a native query or a key in a comment, never reaches the
+ * capture file. Only the recorded copy is redacted: the statement the database runs is not changed.
+ *
+ * <p>Databases disagree on parts of the lexical syntax: whether a backslash escapes a quote, whether
+ * comments nest, whether double quotes delimit an identifier or a string, and whether {@code #}
+ * starts a comment. A statement is redacted under every reading its {@link Syntax} allows, and is
+ * recorded only when all of them give the same text. When they do not, or when a literal or a
+ * comment never ends, the statement is withheld: {@link #WITHHELD} is recorded in its place, never
+ * the statement.</p>
+ *
+ * @since 8.1
+ */
+public final class SqlTailRedactor {
+
+ /** What a literal is replaced with. */
+ public static final String PLACEHOLDER = "?";
+
+ /** What is recorded in place of a statement whose values cannot be told apart from its SQL. */
+ public static final String WITHHELD = "(statement withheld: its values could not be redacted)";
+
+ private static final boolean[] NO = {false};
+ private static final boolean[] EITHER = {false, true};
+
+ /**
+ * The lexical syntax of the database a statement runs on, as far as finding its values goes.
+ */
+ public enum Syntax {
+
+ /**
+ * Not known, or a database that can read double-quoted text as a string although it quotes
+ * identifiers with double quotes, such as SQLite and Informix: a statement is recorded only when
+ * the readings of every syntax below agree.
+ */
+ UNKNOWN,
+
+ /** Standard SQL, where double quotes delimit identifiers: H2, PostgreSQL, Oracle, DB2, HSQLDB and HANA. */
+ STANDARD,
+
+ /** MySQL and MariaDB: double quotes delimit strings, and {@code #} and {@code -- } start comments. */
+ MYSQL,
+
+ /**
+ * Microsoft SQL Server and Sybase: square brackets delimit identifiers, comments nest, and double
+ * quotes delimit a string when {@code QUOTED_IDENTIFIER} is off, as it is by default on Sybase.
+ */
+ TRANSACT_SQL
+ }
+
+ private SqlTailRedactor() {
+ }
+
+ /**
+ * Redacts a statement.
+ *
+ * @param sql the statement
+ * @param syntax the syntax of the database it runs on
+ * @return the statement with its literals replaced and its comments dropped, or {@link #WITHHELD}
+ */
+ public static String redact(String sql, Syntax syntax) {
+ if (sql == null) {
+ return null;
+ }
+ String redacted = null;
+ for (Reading reading : readings(sql, syntax == null ? Syntax.UNKNOWN : syntax)) {
+ String candidate = reading.redact(sql);
+ if (candidate == null || (redacted != null && !redacted.equals(candidate))) {
+ return WITHHELD;
+ }
+ redacted = candidate;
+ }
+ return redacted;
+ }
+
+ /**
+ * The readings a statement has to agree under. A reading that depends on a character the
+ * statement does not contain is the same as the one that does not, so it is left out.
+ */
+ private static List<Reading> readings(String sql, Syntax syntax) {
+ boolean[] backslash = sql.indexOf('\\') >= 0 ? EITHER : NO;
+ int comment = sql.indexOf("/*");
+ boolean[] nesting = comment >= 0 && sql.indexOf("/*", comment + 2) >= 0 ? EITHER : NO;
+ List<Reading> readings = new ArrayList<>();
+ if (syntax == Syntax.STANDARD || syntax == Syntax.UNKNOWN) {
+ for (boolean backslashEscapes : backslash) {
+ for (boolean nestedComments : nesting) {
+ readings.add(new Reading(Syntax.STANDARD, backslashEscapes, nestedComments));
+ }
+ }
+ }
+ if (syntax == Syntax.MYSQL || syntax == Syntax.UNKNOWN) {
+ for (boolean backslashEscapes : backslash) {
+ readings.add(new Reading(Syntax.MYSQL, backslashEscapes, false));
+ }
+ }
+ if (syntax == Syntax.TRANSACT_SQL || syntax == Syntax.UNKNOWN) {
+ readings.add(new Reading(Syntax.TRANSACT_SQL, false, true));
+ }
+ return readings;
+ }
+
+ /**
+ * One way of reading a statement. Only MySQL delimits identifiers with backticks, starts a comment
+ * with {@code #}, and needs whitespace after {@code --}; only Transact-SQL delimits identifiers with
+ * square brackets. Outside standard SQL, double quotes can delimit a string: in MySQL unless
+ * {@code ANSI_QUOTES} is set, and in Transact-SQL when {@code QUOTED_IDENTIFIER} is off. Either way the
+ * quoted run ends at the same place, so it is replaced as a value.
+ *
+ * @param syntax the syntax read in, never {@link Syntax#UNKNOWN}
+ * @param backslashEscapes whether a backslash escapes the character after it in a string
+ * @param nestedComments whether a block comment can contain another
+ */
+ private record Reading(Syntax syntax, boolean backslashEscapes, boolean nestedComments) {
+
+ /**
+ * @return the redacted statement, or null when a literal, a quoted identifier or a comment never ends
+ */
+ String redact(String sql) {
+ boolean mysql = syntax == Syntax.MYSQL;
+ boolean doubleQuotedValue = syntax != Syntax.STANDARD;
+ int length = sql.length();
+ StringBuilder out = new StringBuilder(length);
+ int i = 0;
+ while (i < length) {
+ char c = sql.charAt(i);
+ char next = i + 1 < length ? sql.charAt(i + 1) : 0;
+ char previous = i > 0 ? sql.charAt(i - 1) : 0;
+ int dollarTag = c == '$' && !isIdentifierPart(previous) ? dollarTagLength(sql, i) : 0;
+ int end;
+ if (c == '\'') {
+ end = endOfQuotedLiteral(sql, i);
+ out.append(PLACEHOLDER);
+ }
+ else if (c == '"') {
Review Comment:
Fixed in afaaabb58a. GoogleSQL is a `Syntax` of its own now: a string that opens with its quote written three times holds any other quote and ends only at the next three, a backslash always escapes the character after it, a quote written twice is two strings, backticks delimit identifiers and `#` starts a comment. Hibernate 7's `SpannerDialect` is classified as it, and `select """a"example-secret"b""" from t` records as `select ? from t`.
The backtick heuristic is gone with it: only `MySQLDialect` and its subclasses read as MySQL, so a backtick-quoting dialect Grails does not know, such as Google's Spanner dialect for Hibernate 5 (Hibernate 5 ships none), is read in every syntax, GoogleSQL among them. A triple-quoted string is withheld there, and so is a quote doubled inside a string, which GoogleSQL reads as two strings, so `'O''Brien'` is withheld before the dialect is known. Covered in `SqlTailRedactorSpec` (a GoogleSQL table, the withheld and unknown rows, and a GoogleSQL no-survival list), `SqlTailCaptureSpec`, and both inspector specs; the guide paragraph on double quotes says the same.
##########
grails-datamapping-core/src/cli/groovy/org/grails/datastore/gorm/jdbc/sqltail/cli/SqlTailFileReader.groovy:
##########
@@ -0,0 +1,152 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * https://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.grails.datastore.gorm.jdbc.sqltail.cli
+
+import java.nio.charset.StandardCharsets
+
+import groovy.transform.CompileStatic
+
+import org.grails.datastore.gorm.jdbc.sqltail.SqlTailRecord
+
+/**
+ * Follows a capture file the way {@code tail -f} does: each poll returns the records written
+ * since the last one. A line that has no terminator yet is left for the next poll, so a record
+ * the application is still writing is never read in half. An application that starts again
+ * truncates the file and names its new run in the first line; when that line names another run,
+ * or the file shrinks, reading starts over from the beginning. The run tells a restart apart from
+ * a file that grew past where the reader was before it was next polled.
+ *
+ * @since 8.1
+ */
+@CompileStatic
+class SqlTailFileReader implements Closeable {
+
+ private static final byte NEWLINE = (byte) 10
+
+ /** How much of the file the first line is looked for in; a header is far shorter. */
+ private static final int HEADER_LIMIT = 256
+
+ final File file
+ private long position
+ private String run
+ private boolean restarted
+
+ /**
+ * @param file the capture file
+ * @param fromStart whether to read what the file already holds, or only what is written from now on
+ */
+ SqlTailFileReader(File file, boolean fromStart) {
+ this.file = file
+ this.position = fromStart ? 0 : file.length()
+ this.run = file.exists() ? readRun() : null
+ }
+
+ /**
+ * @return the records written since the last poll, or since the reader was opened
+ */
+ List<SqlTailRecord> poll() {
+ restarted = false
+ long length = file.length()
+ String current = length > 0 ? readRun() : null
+ if (length < position || (current != null && run != null && current != run)) {
+ position = 0
+ restarted = true
+ }
+ if (current != null || restarted) {
+ // a file truncated before its header was written names its run on a later poll
+ run = current
+ }
+ if (length <= position) {
+ return []
+ }
+ byte[] bytes = readFrom(position, length)
+ int consumed = lastLineEnd(bytes)
+ if (consumed == 0) {
+ return []
+ }
+ position += consumed
+ String text = new String(bytes, 0, consumed, StandardCharsets.UTF_8)
+ List<SqlTailRecord> records = []
+ for (String line : text.split('\n')) {
+ SqlTailRecord record = SqlTailRecord.parse(line)
+ if (record != null) {
+ records.add(record)
+ }
+ }
+ records
+ }
+
+ /**
+ * @return whether the last poll found a new run in the file, which an application restart writes
+ */
+ boolean isRestarted() {
+ restarted
+ }
+
+ /**
+ * @return how far the file has been read
+ */
+ long getPosition() {
+ position
+ }
+
+ /**
+ * @return the run that wrote the file, as its first line names it; null before the line is complete
+ */
+ String getRun() {
+ run
+ }
+
+ @Override
+ void close() {
+ // the file is opened and closed per poll
+ }
+
+ private static int lastLineEnd(byte[] bytes) {
+ for (int i = bytes.length - 1; i >= 0; i--) {
+ if (bytes[i] == NEWLINE) {
+ return i + 1
+ }
+ }
+ 0
+ }
+
+ private String readRun() {
+ byte[] head = readFrom(0, Math.min(file.length(), (long) HEADER_LIMIT))
+ for (int i = 0; i < head.length; i++) {
+ if (head[i] == NEWLINE) {
+ return SqlTailRecord.parseHeader(new String(head, 0, i, StandardCharsets.UTF_8))
+ }
+ }
+ null
+ }
+
+ private byte[] readFrom(long from, long to) {
+ RandomAccessFile input = new RandomAccessFile(file, 'r')
+ try {
+ input.seek(from)
+ byte[] bytes = new byte[(int) Math.max(0L, Math.min(to, input.length()) - from)]
Review Comment:
Fixed in afaaabb58a. The reader no longer reads a length it measured earlier with `readFully`: it reads what the file holds, and a read that comes up short, or finds the file gone between measure and open, is a restart. The poll starts over from the beginning once, in the same poll, and the next poll takes it from there, so a file that keeps shrinking cannot spin it. `SqlTailFileReaderSpec` covers the truncated and the removed case with a `File` whose `length()` truncates or deletes the file the moment it has been measured; before the fix that poll ended in `EOFException`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@grails.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add the query-tail command to find query-heavy pages [grails-core]
Posted by "testlens-app[bot] (via GitHub)" <gi...@apache.org>.
testlens-app[bot] commented on PR #16631:
URL: https://github.com/apache/grails-core/pull/16631#issuecomment-6385030372
## ✅ All tests passed ✅
🏷️ Commit: 5e5feb5e4d3cb870b7e758a394433eac0b293d3d
▶️ Tests: 138155 executed
⚪️ Checks: 97/97 completed
---
_Learn more about TestLens at [testlens.app/docs](https://testlens.app/docs/features/pr-comment/)._
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@grails.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add the query-tail command to find query-heavy pages [grails-core]
Posted by "jamesfredley (via GitHub)" <gi...@apache.org>.
jamesfredley commented on code in PR #16631:
URL: https://github.com/apache/grails-core/pull/16631#discussion_r4365978367
##########
grails-datamapping-core/src/main/groovy/org/apache/grails/data/core/querytail/H2SqlSyntax.groovy:
##########
@@ -0,0 +1,102 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * https://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.grails.data.core.querytail
+
+import groovy.transform.CompileStatic
+import groovy.transform.PackageScope
+
+/**
+ * H2: block comments nest, {@code //} starts a line comment, {@code $$} quotes a string without a
+ * tag, backticks delimit a name in every compatibility mode, and square brackets delimit a name in
+ * the SQL Server mode, where they are never a subscript, which is what a bracket after a name, a
+ * closing parenthesis or a closing bracket is in the other modes. A bracket after whitespace, or
+ * after a comment, is a name in the SQL Server mode but opens an array literal after the
+ * {@code ARRAY} keyword, which may stand any distance before it, so a statement with one is read
+ * both ways: a name, which is recorded as written, agrees with itself, while an array literal,
+ * whose elements are values, does not and withholds the statement.
+ */
+@CompileStatic
+@PackageScope
+final class H2SqlSyntax extends StandardSqlSyntax {
+
+ private final boolean bracketAfterWhitespaceIsName
+
+ H2SqlSyntax() {
+ this(true)
+ }
+
+ private H2SqlSyntax(boolean bracketAfterWhitespaceIsName) {
+ super('H2')
+ this.bracketAfterWhitespaceIsName = bracketAfterWhitespaceIsName
+ }
+
+ @Override
+ List<QueryTailSyntax> readings(String sql) {
+ if (!hasBracketAfterWhitespace(sql)) {
+ return List.<QueryTailSyntax>of(this)
+ }
+ List.<QueryTailSyntax>of(new H2SqlSyntax(true), new H2SqlSyntax(false))
+ }
+
+ @Override
+ int endOfComment(String sql, int index) {
+ sql.startsWith('//', index) ? endOfLine(sql, index) : super.endOfComment(sql, index)
+ }
+
+ @Override
+ int endOfValue(String sql, int index) {
+ int end = endOfDollarQuoted(sql, index, false)
+ end != NONE ? end : super.endOfValue(sql, index)
+ }
+
+ @Override
+ int endOfQuotedName(String sql, int index) {
+ char c = sql.charAt(index)
+ if (c == '`') {
+ return endOfQuoted(sql, index, '`' as char, false, true)
+ }
+ if (c == '[' && !subscript(sql, index) && (bracketAfterWhitespaceIsName || !afterWhitespace(sql, index))) {
+ return endOfQuoted(sql, index, ']' as char, false, false)
+ }
+ super.endOfQuotedName(sql, index)
+ }
+
+ @Override
+ protected boolean nestedComments() {
+ true
+ }
+
+ private boolean subscript(String sql, int index) {
+ char previous = previous(sql, index)
+ isNamePart(previous) || previous == ')' || previous == ']'
+ }
+
+ private static boolean afterWhitespace(String sql, int index) {
+ index > 0 && Character.isWhitespace(sql.charAt(index - 1))
+ }
+
+ private static boolean hasBracketAfterWhitespace(String sql) {
Review Comment:
This checks only the character before `[`. `select ARRAY/**/['example-secret']` has `/` there, so the span is copied as an identifier and the secret reaches the file. The new regression has whitespace after `*/`, so it misses this.
Treat a comment immediately before `[` as ambiguous, or withhold the statement. Please cover `ARRAY/**/['example-secret']` and `ARRAY /* c */['example-secret']` through the capture API.
##########
build-logic/plugins/src/main/groovy/org/apache/grails/buildsrc/CompilePlugin.groovy:
##########
@@ -181,7 +181,9 @@ class CompilePlugin implements Plugin<Project> {
it.options.fork = true
// always set an isolated build to ensure grails.factories aren't accidentally merged since every project
// in this mono repo should be an isolated projected
- it.options.forkOptions.jvmArgs = ['-Xms128M', '-Xmx2G', '-Dgrails.isolated.build=true']
+ it.options.forkOptions.jvmArgs = ['-Xms128M', '-Xmx2G', '-Dgrails.isolated.build=true',
Review Comment:
This commit is marked to be dropped, and it must be. Every Groovy compiler fork now heap-dumps to a fixed `/tmp/querytail-heap` path and runs `jcmd` on OOM. The workflow uploads those dumps on failure. That is shared build behavior, not a switch.
Remove this commit before merge.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@grails.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add the query-tail command to find query-heavy pages [grails-core]
Posted by "jdaugherty (via GitHub)" <gi...@apache.org>.
jdaugherty commented on code in PR #16631:
URL: https://github.com/apache/grails-core/pull/16631#discussion_r4366432130
##########
grails-datamapping-core/src/main/groovy/org/apache/grails/data/core/querytail/H2SqlSyntax.groovy:
##########
@@ -0,0 +1,102 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * https://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.grails.data.core.querytail
+
+import groovy.transform.CompileStatic
+import groovy.transform.PackageScope
+
+/**
+ * H2: block comments nest, {@code //} starts a line comment, {@code $$} quotes a string without a
+ * tag, backticks delimit a name in every compatibility mode, and square brackets delimit a name in
+ * the SQL Server mode, where they are never a subscript, which is what a bracket after a name, a
+ * closing parenthesis or a closing bracket is in the other modes. A bracket after whitespace, or
+ * after a comment, is a name in the SQL Server mode but opens an array literal after the
+ * {@code ARRAY} keyword, which may stand any distance before it, so a statement with one is read
+ * both ways: a name, which is recorded as written, agrees with itself, while an array literal,
+ * whose elements are values, does not and withholds the statement.
+ */
+@CompileStatic
+@PackageScope
+final class H2SqlSyntax extends StandardSqlSyntax {
+
+ private final boolean bracketAfterWhitespaceIsName
+
+ H2SqlSyntax() {
+ this(true)
+ }
+
+ private H2SqlSyntax(boolean bracketAfterWhitespaceIsName) {
+ super('H2')
+ this.bracketAfterWhitespaceIsName = bracketAfterWhitespaceIsName
+ }
+
+ @Override
+ List<QueryTailSyntax> readings(String sql) {
+ if (!hasBracketAfterWhitespace(sql)) {
+ return List.<QueryTailSyntax>of(this)
+ }
+ List.<QueryTailSyntax>of(new H2SqlSyntax(true), new H2SqlSyntax(false))
+ }
+
+ @Override
+ int endOfComment(String sql, int index) {
+ sql.startsWith('//', index) ? endOfLine(sql, index) : super.endOfComment(sql, index)
+ }
+
+ @Override
+ int endOfValue(String sql, int index) {
+ int end = endOfDollarQuoted(sql, index, false)
+ end != NONE ? end : super.endOfValue(sql, index)
+ }
+
+ @Override
+ int endOfQuotedName(String sql, int index) {
+ char c = sql.charAt(index)
+ if (c == '`') {
+ return endOfQuoted(sql, index, '`' as char, false, true)
+ }
+ if (c == '[' && !subscript(sql, index) && (bracketAfterWhitespaceIsName || !afterWhitespace(sql, index))) {
+ return endOfQuoted(sql, index, ']' as char, false, false)
+ }
+ super.endOfQuotedName(sql, index)
+ }
+
+ @Override
+ protected boolean nestedComments() {
+ true
+ }
+
+ private boolean subscript(String sql, int index) {
+ char previous = previous(sql, index)
+ isNamePart(previous) || previous == ')' || previous == ']'
+ }
+
+ private static boolean afterWhitespace(String sql, int index) {
+ index > 0 && Character.isWhitespace(sql.charAt(index - 1))
+ }
+
+ private static boolean hasBracketAfterWhitespace(String sql) {
Review Comment:
Fixed in db833afb85 (folded into the feature commit). `H2SqlSyntax` now treats the two characters that close a block comment like whitespace before a `[`: `afterGap` replaces `afterWhitespace` in both the readings decision and `endOfQuotedName`, so `ARRAY/**/[...]` and `ARRAY /* c */[...]` get the name and the array-literal reading and are withheld; a line comment ends with a line break, which was already whitespace. Covered in `QueryTailRedactorSpec` (withheld table and the never-survives list, plus `ARRAY -- c\n[...]`) and through the capture API in `QueryTailCaptureSpec`, where the three `ARRAY` forms are withheld and `[order] from [user]` is still recorded; the guide paragraph names the comment case.
##########
grails-datamapping-core/src/main/groovy/org/apache/grails/data/core/querytail/H2SqlSyntax.groovy:
##########
@@ -0,0 +1,102 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * https://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.grails.data.core.querytail
+
+import groovy.transform.CompileStatic
+import groovy.transform.PackageScope
+
+/**
+ * H2: block comments nest, {@code //} starts a line comment, {@code $$} quotes a string without a
+ * tag, backticks delimit a name in every compatibility mode, and square brackets delimit a name in
+ * the SQL Server mode, where they are never a subscript, which is what a bracket after a name, a
+ * closing parenthesis or a closing bracket is in the other modes. A bracket after whitespace, or
+ * after a comment, is a name in the SQL Server mode but opens an array literal after the
+ * {@code ARRAY} keyword, which may stand any distance before it, so a statement with one is read
+ * both ways: a name, which is recorded as written, agrees with itself, while an array literal,
+ * whose elements are values, does not and withholds the statement.
+ */
+@CompileStatic
+@PackageScope
+final class H2SqlSyntax extends StandardSqlSyntax {
+
+ private final boolean bracketAfterWhitespaceIsName
+
+ H2SqlSyntax() {
+ this(true)
+ }
+
+ private H2SqlSyntax(boolean bracketAfterWhitespaceIsName) {
+ super('H2')
+ this.bracketAfterWhitespaceIsName = bracketAfterWhitespaceIsName
+ }
+
+ @Override
+ List<QueryTailSyntax> readings(String sql) {
+ if (!hasBracketAfterWhitespace(sql)) {
+ return List.<QueryTailSyntax>of(this)
+ }
+ List.<QueryTailSyntax>of(new H2SqlSyntax(true), new H2SqlSyntax(false))
+ }
+
+ @Override
+ int endOfComment(String sql, int index) {
+ sql.startsWith('//', index) ? endOfLine(sql, index) : super.endOfComment(sql, index)
+ }
+
+ @Override
+ int endOfValue(String sql, int index) {
+ int end = endOfDollarQuoted(sql, index, false)
+ end != NONE ? end : super.endOfValue(sql, index)
+ }
+
+ @Override
+ int endOfQuotedName(String sql, int index) {
+ char c = sql.charAt(index)
+ if (c == '`') {
+ return endOfQuoted(sql, index, '`' as char, false, true)
+ }
+ if (c == '[' && !subscript(sql, index) && (bracketAfterWhitespaceIsName || !afterWhitespace(sql, index))) {
+ return endOfQuoted(sql, index, ']' as char, false, false)
+ }
+ super.endOfQuotedName(sql, index)
+ }
+
+ @Override
+ protected boolean nestedComments() {
+ true
+ }
+
+ private boolean subscript(String sql, int index) {
+ char previous = previous(sql, index)
+ isNamePart(previous) || previous == ')' || previous == ']'
+ }
+
+ private static boolean afterWhitespace(String sql, int index) {
+ index > 0 && Character.isWhitespace(sql.charAt(index - 1))
+ }
+
+ private static boolean hasBracketAfterWhitespace(String sql) {
Review Comment:
Fixed in db833afb85 (folded into the feature commit). `H2SqlSyntax` now treats the two characters that close a block comment like whitespace before a `[`: `afterGap` replaces `afterWhitespace` in both the readings decision and `endOfQuotedName`, so `ARRAY/**/[...]` and `ARRAY /* c */[...]` get the name and the array-literal reading and are withheld; a line comment ends with a line break, which was already whitespace. Covered in `QueryTailRedactorSpec` (withheld table and the never-survives list, plus `ARRAY -- c\n[...]`) and through the capture API in `QueryTailCaptureSpec`, where the three `ARRAY` forms are withheld and `[order] from [user]` is still recorded; the guide paragraph names the comment case.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@grails.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add the query-tail command to find query-heavy pages [grails-core]
Posted by "testlens-app[bot] (via GitHub)" <gi...@apache.org>.
testlens-app[bot] commented on PR #16631:
URL: https://github.com/apache/grails-core/pull/16631#issuecomment-6399218090
## ✅ All tests passed ✅
🏷️ Commit: f69625477f456b50da07ae6eb874b9162fd3b119
▶️ Tests: 138081 executed
⚪️ Checks: 97/97 completed
---
_Learn more about TestLens at [testlens.app/docs](https://testlens.app/docs/features/pr-comment/)._
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@grails.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add the query-tail command to find query-heavy pages [grails-core]
Posted by "jdaugherty (via GitHub)" <gi...@apache.org>.
jdaugherty commented on PR #16631:
URL: https://github.com/apache/grails-core/pull/16631#issuecomment-6402662376
@jamesfredley one follow-up to the statement-inspector item, in ea3558f3c6 (still a single commit). An application inspector could also be lost when a `configClass` set it on the service registry builder, by overriding `createStandardServiceRegistryBuilder`. GORM applies the configuration's properties, which held the capture, over the builder's settings, so the capture replaced the inspector. This affected Hibernate 5 and 7.
The capture now picks the inspector to wrap when the session factory is built, from the merged registry settings and, on Hibernate 7, the inspector set on the configuration. A new test runs on both versions for each way of setting an inspector, and checks that the inspector that runs with capture on is the one that runs with it off. Details are on the thread.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@grails.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add the query-tail command to find query-heavy pages [grails-core]
Posted by "testlens-app[bot] (via GitHub)" <gi...@apache.org>.
testlens-app[bot] commented on PR #16631:
URL: https://github.com/apache/grails-core/pull/16631#issuecomment-6441634086
## 🚨 All tests passed but jobs failed 🚨
### Failed Jobs without Test Failures
❌ [Coverage / Coverage - grails-gradle \(ubuntu-24.04\)](https://github.com/apache/grails-core/actions/runs/40499634101/job/122177454051?pr=16631)
🏷️ Commit: d8f76c42e32afa49838b817989bd8c7180b27604
▶️ Tests: 4166 executed
🟡 Checks: 17/88 completed
### Rerun Controls
Click the checkbox to trigger a rerun:
🔲 **Rerun jobs**
---
_Learn more about TestLens at [testlens.app/docs](https://testlens.app/docs/features/pr-comment/)._
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@grails.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add the sql-tail command to find query-heavy pages [grails-core]
Posted by "jdaugherty (via GitHub)" <gi...@apache.org>.
jdaugherty commented on code in PR #16631:
URL: https://github.com/apache/grails-core/pull/16631#discussion_r4349802341
##########
grails-datamapping-core/src/main/groovy/org/grails/datastore/gorm/jdbc/sqltail/SqlTailCapture.groovy:
##########
@@ -0,0 +1,155 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * https://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.grails.datastore.gorm.jdbc.sqltail
+
+import java.nio.charset.StandardCharsets
+import java.nio.file.Files
+import java.nio.file.StandardOpenOption
+
+import groovy.transform.CompileStatic
+import groovy.util.logging.Slf4j
+
+/**
+ * Writes the statements an application runs to the capture file the {@code sql-tail} command
+ * tails. One capture is open per file, shared by every data source of the application; the file is
+ * truncated when the capture opens, so it holds the statements of the current run only. Each
+ * statement is flushed as it is written so that the command sees it at once.
+ *
+ * <p>A capture is only opened when {@link SqlTailSettings#SETTING_ENABLED} is set; an application
+ * started without it never reaches this class.</p>
+ *
+ * @since 8.1
+ */
+@CompileStatic
+@Slf4j
+class SqlTailCapture implements Closeable {
+
+ private static final Map<String, SqlTailCapture> OPEN = new HashMap<>()
+
+ /** The file the statements are written to. */
+ final File file
+
+ private final SqlTailRequestAttribution attribution = new SqlTailRequestAttribution()
+ private final String key
+ private Writer writer
+
+ private SqlTailCapture(File file, String key) {
+ this.file = file
+ this.key = key
+ File parent = file.absoluteFile.parentFile
+ try {
+ if (parent != null && !parent.isDirectory() && !parent.mkdirs() && !parent.isDirectory()) {
+ throw new IOException("Cannot create the sql-tail capture directory ${parent}")
+ }
+ this.writer = Files.newBufferedWriter(file.toPath(), StandardCharsets.UTF_8,
+ StandardOpenOption.CREATE, StandardOpenOption.TRUNCATE_EXISTING, StandardOpenOption.WRITE)
+ }
+ catch (IOException e) {
+ throw new UncheckedIOException("Cannot open the sql-tail capture file ${file}: ${e.message}", e)
+ }
+ }
+
+ /**
+ * Opens the capture for a file, or returns the one already open for it.
+ *
+ * @param file the capture file
+ * @return the capture
+ * @throws UncheckedIOException when the file cannot be created
+ */
+ static synchronized SqlTailCapture open(File file) {
+ String key = file.canonicalPath
+ SqlTailCapture capture = OPEN.get(key)
+ if (capture == null) {
+ capture = new SqlTailCapture(file, key)
+ OPEN.put(key, capture)
+ log.info('Capturing SQL statements to {} for sql-tail', file)
+ }
+ capture
+ }
+
+ /**
+ * Opens the capture the settings name.
+ *
+ * @param settings the sql-tail settings
+ * @return the capture
+ */
+ static SqlTailCapture open(SqlTailSettings settings) {
+ open(settings.resolveFile())
+ }
+
+ /**
+ * Records a statement that is about to run on the current thread.
+ *
+ * @param dataSource the connection source name the statement runs against
+ * @param sql the statement
+ */
+ void record(String dataSource, String sql) {
+ SqlTailRecord record = new SqlTailRecord(timestamp: System.currentTimeMillis(), dataSource: dataSource, sql: sql)
Review Comment:
Fixed in 43640050e9. `SqlTailCapture.record()` now redacts the recorded copy before it is written: string, number and dollar-quoted literals become `?` and comments are dropped. The statement returned to Hibernate is untouched.
The statement is read in the syntax of the session factory's dialect (from `Dialect.openQuote()`). Where databases disagree on where a value ends (a backslash before a quote, nested comments, double quotes as strings on MySQL, or any statement run before the dialect is known), every reading has to produce the same redacted text. If they differ, or a literal or comment never ends, the line is recorded as `(statement withheld: its values could not be redacted)` instead of the statement.
Tests:
- `SqlTailRedactorSpec`: the cases per syntax, the fail-closed cases, and a check that an embedded value survives under no syntax.
- `SqlTailStatementInspectorSpec` (H5 and H7): with `hibernate.use_sql_comments` on, a native query with `'example-secret'` and a key in a comment, and an HQL query with the same literal (H7 inlines it into both the SQL and its generated comment). The database returns the original value, a delegate inspector sees the original statement, and the capture file holds neither the value nor the key.
- `SqlTailRunnerSpec`: the same statement, captured and then shown in the pages, statements and stream views, never shows the value or the key.
##########
grails-datamapping-core/src/main/groovy/org/grails/datastore/gorm/jdbc/sqltail/SqlTailCapture.groovy:
##########
@@ -0,0 +1,155 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * https://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.grails.datastore.gorm.jdbc.sqltail
+
+import java.nio.charset.StandardCharsets
+import java.nio.file.Files
+import java.nio.file.StandardOpenOption
+
+import groovy.transform.CompileStatic
+import groovy.util.logging.Slf4j
+
+/**
+ * Writes the statements an application runs to the capture file the {@code sql-tail} command
+ * tails. One capture is open per file, shared by every data source of the application; the file is
+ * truncated when the capture opens, so it holds the statements of the current run only. Each
+ * statement is flushed as it is written so that the command sees it at once.
+ *
+ * <p>A capture is only opened when {@link SqlTailSettings#SETTING_ENABLED} is set; an application
+ * started without it never reaches this class.</p>
+ *
+ * @since 8.1
+ */
+@CompileStatic
+@Slf4j
+class SqlTailCapture implements Closeable {
+
+ private static final Map<String, SqlTailCapture> OPEN = new HashMap<>()
+
+ /** The file the statements are written to. */
+ final File file
+
+ private final SqlTailRequestAttribution attribution = new SqlTailRequestAttribution()
+ private final String key
+ private Writer writer
+
+ private SqlTailCapture(File file, String key) {
+ this.file = file
+ this.key = key
+ File parent = file.absoluteFile.parentFile
+ try {
+ if (parent != null && !parent.isDirectory() && !parent.mkdirs() && !parent.isDirectory()) {
+ throw new IOException("Cannot create the sql-tail capture directory ${parent}")
+ }
+ this.writer = Files.newBufferedWriter(file.toPath(), StandardCharsets.UTF_8,
+ StandardOpenOption.CREATE, StandardOpenOption.TRUNCATE_EXISTING, StandardOpenOption.WRITE)
+ }
+ catch (IOException e) {
+ throw new UncheckedIOException("Cannot open the sql-tail capture file ${file}: ${e.message}", e)
+ }
+ }
+
+ /**
+ * Opens the capture for a file, or returns the one already open for it.
+ *
+ * @param file the capture file
+ * @return the capture
+ * @throws UncheckedIOException when the file cannot be created
+ */
+ static synchronized SqlTailCapture open(File file) {
+ String key = file.canonicalPath
+ SqlTailCapture capture = OPEN.get(key)
Review Comment:
Fixed in 43640050e9. The session factory now owns its capture: `SqlTailStatementInspector` is also installed as the configuration's `SessionFactoryObserver`, chaining any observer already set. It opens its capture when the factory is created (or on the first statement, if that comes earlier) and closes it in `sessionFactoryClosed`, which both Hibernate versions also call when construction fails. A failure before the factory exists opens nothing.
`SqlTailCapture.open()` now returns a handle per owner over a reference-counted file. Closing a handle is idempotent; when the last one closes, the file is closed and removed from the map, so the next open truncates it and starts a new run.
Hibernate 5's `HibernateMappingContextConfiguration` replaced any observer set before the build; it now chains it before destroying the service registry.
Tests: `SqlTailCaptureSpec` covers last-owner cleanup and reopening the same path. `SqlTailStatementInspectorSpec` (H5 and H7) closes a datastore with two data sources and checks both captures close and the next start truncates the file, keeps a second datastore on the same file writing after the first closes, and checks that an observer the application set is still called. `HibernateMappingContextConfigurationSpec` (H5) checks the chained observer and that the registry is still destroyed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@grails.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add the sql-tail command to find query-heavy pages [grails-core]
Posted by "jamesfredley (via GitHub)" <gi...@apache.org>.
jamesfredley commented on code in PR #16631:
URL: https://github.com/apache/grails-core/pull/16631#discussion_r4352333988
##########
grails-datamapping-core/src/main/groovy/org/grails/datastore/gorm/jdbc/sqltail/SqlTailRedactor.java:
##########
@@ -0,0 +1,360 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * https://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.grails.datastore.gorm.jdbc.sqltail;
+
+import java.util.ArrayList;
+import java.util.List;
+
+/**
+ * Takes the values out of a statement before it is captured. Every string, number and
+ * dollar-quoted literal becomes {@link #PLACEHOLDER} and every comment is dropped, so that a value
+ * written into the SQL text, a password in a native query or a key in a comment, never reaches the
+ * capture file. Only the recorded copy is redacted: the statement the database runs is not changed.
+ *
+ * <p>Databases disagree on parts of the lexical syntax: whether a backslash escapes a quote, whether
+ * comments nest, whether double quotes delimit an identifier or a string, and whether {@code #}
+ * starts a comment. A statement is redacted under every reading its {@link Syntax} allows, and is
+ * recorded only when all of them give the same text. When they do not, or when a literal or a
+ * comment never ends, the statement is withheld: {@link #WITHHELD} is recorded in its place, never
+ * the statement.</p>
+ *
+ * @since 8.1
+ */
+public final class SqlTailRedactor {
+
+ /** What a literal is replaced with. */
+ public static final String PLACEHOLDER = "?";
+
+ /** What is recorded in place of a statement whose values cannot be told apart from its SQL. */
+ public static final String WITHHELD = "(statement withheld: its values could not be redacted)";
+
+ private static final boolean[] NO = {false};
+ private static final boolean[] EITHER = {false, true};
+
+ /**
+ * The lexical syntax of the database a statement runs on, as far as finding its values goes.
+ */
+ public enum Syntax {
+
+ /**
+ * Not known, or a database that can read double-quoted text as a string although it quotes
+ * identifiers with double quotes, such as SQLite and Informix: a statement is recorded only when
+ * the readings of every syntax below agree.
+ */
+ UNKNOWN,
+
+ /** Standard SQL, where double quotes delimit identifiers: H2, PostgreSQL, Oracle, DB2, HSQLDB and HANA. */
+ STANDARD,
+
+ /** MySQL and MariaDB: double quotes delimit strings, and {@code #} and {@code -- } start comments. */
+ MYSQL,
+
+ /**
+ * Microsoft SQL Server and Sybase: square brackets delimit identifiers, comments nest, and double
+ * quotes delimit a string when {@code QUOTED_IDENTIFIER} is off, as it is by default on Sybase.
+ */
+ TRANSACT_SQL
+ }
+
+ private SqlTailRedactor() {
+ }
+
+ /**
+ * Redacts a statement.
+ *
+ * @param sql the statement
+ * @param syntax the syntax of the database it runs on
+ * @return the statement with its literals replaced and its comments dropped, or {@link #WITHHELD}
+ */
+ public static String redact(String sql, Syntax syntax) {
+ if (sql == null) {
+ return null;
+ }
+ String redacted = null;
+ for (Reading reading : readings(sql, syntax == null ? Syntax.UNKNOWN : syntax)) {
+ String candidate = reading.redact(sql);
+ if (candidate == null || (redacted != null && !redacted.equals(candidate))) {
+ return WITHHELD;
+ }
+ redacted = candidate;
+ }
+ return redacted;
+ }
+
+ /**
+ * The readings a statement has to agree under. A reading that depends on a character the
+ * statement does not contain is the same as the one that does not, so it is left out.
+ */
+ private static List<Reading> readings(String sql, Syntax syntax) {
+ boolean[] backslash = sql.indexOf('\\') >= 0 ? EITHER : NO;
+ int comment = sql.indexOf("/*");
+ boolean[] nesting = comment >= 0 && sql.indexOf("/*", comment + 2) >= 0 ? EITHER : NO;
+ List<Reading> readings = new ArrayList<>();
+ if (syntax == Syntax.STANDARD || syntax == Syntax.UNKNOWN) {
+ for (boolean backslashEscapes : backslash) {
+ for (boolean nestedComments : nesting) {
+ readings.add(new Reading(Syntax.STANDARD, backslashEscapes, nestedComments));
+ }
+ }
+ }
+ if (syntax == Syntax.MYSQL || syntax == Syntax.UNKNOWN) {
+ for (boolean backslashEscapes : backslash) {
+ readings.add(new Reading(Syntax.MYSQL, backslashEscapes, false));
+ }
+ }
+ if (syntax == Syntax.TRANSACT_SQL || syntax == Syntax.UNKNOWN) {
+ readings.add(new Reading(Syntax.TRANSACT_SQL, false, true));
+ }
+ return readings;
+ }
+
+ /**
+ * One way of reading a statement. Only MySQL delimits identifiers with backticks, starts a comment
+ * with {@code #}, and needs whitespace after {@code --}; only Transact-SQL delimits identifiers with
+ * square brackets. Outside standard SQL, double quotes can delimit a string: in MySQL unless
+ * {@code ANSI_QUOTES} is set, and in Transact-SQL when {@code QUOTED_IDENTIFIER} is off. Either way the
+ * quoted run ends at the same place, so it is replaced as a value.
+ *
+ * @param syntax the syntax read in, never {@link Syntax#UNKNOWN}
+ * @param backslashEscapes whether a backslash escapes the character after it in a string
+ * @param nestedComments whether a block comment can contain another
+ */
+ private record Reading(Syntax syntax, boolean backslashEscapes, boolean nestedComments) {
+
+ /**
+ * @return the redacted statement, or null when a literal, a quoted identifier or a comment never ends
+ */
+ String redact(String sql) {
+ boolean mysql = syntax == Syntax.MYSQL;
+ boolean doubleQuotedValue = syntax != Syntax.STANDARD;
+ int length = sql.length();
+ StringBuilder out = new StringBuilder(length);
+ int i = 0;
+ while (i < length) {
+ char c = sql.charAt(i);
+ char next = i + 1 < length ? sql.charAt(i + 1) : 0;
+ char previous = i > 0 ? sql.charAt(i - 1) : 0;
+ int dollarTag = c == '$' && !isIdentifierPart(previous) ? dollarTagLength(sql, i) : 0;
+ int end;
+ if (c == '\'') {
+ end = endOfQuotedLiteral(sql, i);
+ out.append(PLACEHOLDER);
+ }
+ else if (c == '"') {
Review Comment:
Spanner is classified as MySQL, and GoogleSQL triple quotes are not a literal. `select """a"example-secret"b"""` becomes `select ?example-secret?`, so part of the value is written.
Handle the triple-quoted form, or withhold a statement that contains it.
##########
grails-datamapping-core/src/cli/groovy/org/grails/datastore/gorm/jdbc/sqltail/cli/SqlTailFileReader.groovy:
##########
@@ -0,0 +1,152 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * https://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.grails.datastore.gorm.jdbc.sqltail.cli
+
+import java.nio.charset.StandardCharsets
+
+import groovy.transform.CompileStatic
+
+import org.grails.datastore.gorm.jdbc.sqltail.SqlTailRecord
+
+/**
+ * Follows a capture file the way {@code tail -f} does: each poll returns the records written
+ * since the last one. A line that has no terminator yet is left for the next poll, so a record
+ * the application is still writing is never read in half. An application that starts again
+ * truncates the file and names its new run in the first line; when that line names another run,
+ * or the file shrinks, reading starts over from the beginning. The run tells a restart apart from
+ * a file that grew past where the reader was before it was next polled.
+ *
+ * @since 8.1
+ */
+@CompileStatic
+class SqlTailFileReader implements Closeable {
+
+ private static final byte NEWLINE = (byte) 10
+
+ /** How much of the file the first line is looked for in; a header is far shorter. */
+ private static final int HEADER_LIMIT = 256
+
+ final File file
+ private long position
+ private String run
+ private boolean restarted
+
+ /**
+ * @param file the capture file
+ * @param fromStart whether to read what the file already holds, or only what is written from now on
+ */
+ SqlTailFileReader(File file, boolean fromStart) {
+ this.file = file
+ this.position = fromStart ? 0 : file.length()
+ this.run = file.exists() ? readRun() : null
+ }
+
+ /**
+ * @return the records written since the last poll, or since the reader was opened
+ */
+ List<SqlTailRecord> poll() {
+ restarted = false
+ long length = file.length()
+ String current = length > 0 ? readRun() : null
+ if (length < position || (current != null && run != null && current != run)) {
+ position = 0
+ restarted = true
+ }
+ if (current != null || restarted) {
+ // a file truncated before its header was written names its run on a later poll
+ run = current
+ }
+ if (length <= position) {
+ return []
+ }
+ byte[] bytes = readFrom(position, length)
+ int consumed = lastLineEnd(bytes)
+ if (consumed == 0) {
+ return []
+ }
+ position += consumed
+ String text = new String(bytes, 0, consumed, StandardCharsets.UTF_8)
+ List<SqlTailRecord> records = []
+ for (String line : text.split('\n')) {
+ SqlTailRecord record = SqlTailRecord.parse(line)
+ if (record != null) {
+ records.add(record)
+ }
+ }
+ records
+ }
+
+ /**
+ * @return whether the last poll found a new run in the file, which an application restart writes
+ */
+ boolean isRestarted() {
+ restarted
+ }
+
+ /**
+ * @return how far the file has been read
+ */
+ long getPosition() {
+ position
+ }
+
+ /**
+ * @return the run that wrote the file, as its first line names it; null before the line is complete
+ */
+ String getRun() {
+ run
+ }
+
+ @Override
+ void close() {
+ // the file is opened and closed per poll
+ }
+
+ private static int lastLineEnd(byte[] bytes) {
+ for (int i = bytes.length - 1; i >= 0; i--) {
+ if (bytes[i] == NEWLINE) {
+ return i + 1
+ }
+ }
+ 0
+ }
+
+ private String readRun() {
+ byte[] head = readFrom(0, Math.min(file.length(), (long) HEADER_LIMIT))
+ for (int i = 0; i < head.length; i++) {
+ if (head[i] == NEWLINE) {
+ return SqlTailRecord.parseHeader(new String(head, 0, i, StandardCharsets.UTF_8))
+ }
+ }
+ null
+ }
+
+ private byte[] readFrom(long from, long to) {
+ RandomAccessFile input = new RandomAccessFile(file, 'r')
+ try {
+ input.seek(from)
+ byte[] bytes = new byte[(int) Math.max(0L, Math.min(to, input.length()) - from)]
Review Comment:
If the application truncates the capture between `length()` and `readFully`, the read throws `EOFException` and sql-tail exits. A restart should be a retry, not a command failure.
Treat a short read caused by truncation as a restart and poll again.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@grails.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add the query-tail command to find query-heavy pages [grails-core]
Posted by "jamesfredley (via GitHub)" <gi...@apache.org>.
jamesfredley commented on code in PR #16631:
URL: https://github.com/apache/grails-core/pull/16631#discussion_r4369531231
##########
grails-datamapping-core/src/main/groovy/org/apache/grails/data/core/querytail/QueryTailRequestAttribution.groovy:
##########
@@ -0,0 +1,410 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * https://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.grails.data.core.querytail
+
+import java.lang.reflect.Method
+import java.nio.charset.StandardCharsets
+import java.util.concurrent.atomic.AtomicLong
+import java.util.regex.Pattern
+
+import groovy.transform.CompileStatic
+
+import org.springframework.util.ClassUtils
+import org.springframework.web.context.request.RequestAttributes
+import org.springframework.web.context.request.RequestContextHolder
+import org.springframework.web.context.request.ServletRequestAttributes
+
+/**
+ * Attributes a captured statement to the web request on the current thread. Grails keeps the
+ * request, and the controller and action it resolved, on the thread that serves it, so a statement
+ * run by an interceptor, the action or the view all land on the same page. The first statement of
+ * a request gives it an id that every later statement of that request repeats, which is how the
+ * {@code query-tail} command counts statements per request. Work on any other thread is attributed to
+ * the thread by name.
+ *
+ * <p>The request is recorded as its method and path, with the values in the path replaced by
+ * {@code ?}: the values of the parameters the URL mapping took from it and those of its query, found
+ * whether or not the client percent-encoded them, and any segment that is a number, a UUID or a long
+ * hexadecimal token, so that an id, a token or a key carried in a path never reaches the capture
+ * file. A segment a value is glued into, as a mapping such as {@code /reset/${token}_confirm}
+ * produces, is replaced whole, whether the value shows as sent or decoded. Until the URL mapping has run, as for a statement of a
+ * filter that runs before the controller is chosen, or for a request that never reaches one, no
+ * segment's role is known and every segment is replaced; only the controller the framework set on
+ * the request says that it has run, never a parameter, which a client can send.</p>
+ *
+ * <p>Spring Web is optional: without it on the classpath every statement is attributed to its
+ * thread.</p>
+ *
+ * @since 8.1
+ */
+@CompileStatic
+class QueryTailRequestAttribution {
+
+ /** The request attribute holding the id this attribution gave a request. */
+ public static final String REQUEST_ID_ATTRIBUTE = 'org.apache.grails.data.core.querytail.REQUEST_ID'
+
+ /** The request attribute Grails sets the controller name in. */
+ public static final String CONTROLLER_NAME_ATTRIBUTE = 'org.grails.CONTROLLER_NAME_ATTRIBUTE'
+
+ /** The request attribute Grails sets the controller namespace in. */
+ public static final String CONTROLLER_NAMESPACE_ATTRIBUTE = 'org.grails.CONTROLLER_NAMESPACE_ATTRIBUTE'
+
+ /** The request attribute Grails sets the action name in. */
+ public static final String ACTION_NAME_ATTRIBUTE = 'org.grails.ACTION_NAME_ATTRIBUTE'
+
+ private static final boolean SPRING_WEB_PRESENT = ClassUtils.isPresent(
+ 'org.springframework.web.context.request.RequestContextHolder', QueryTailRequestAttribution.classLoader)
+
+ private final AtomicLong requestCounter = new AtomicLong()
+ private final String requestIdPrefix
+
+ /**
+ * An attribution whose request ids are {@code r1}, {@code r2} and so on.
+ */
+ QueryTailRequestAttribution() {
+ this('')
+ }
+
+ /**
+ * @param requestIdPrefix what every request id starts with, which tells the requests of one JVM
+ * from those of another writing the same capture
+ */
+ QueryTailRequestAttribution(String requestIdPrefix) {
+ this.requestIdPrefix = requestIdPrefix ?: ''
+ }
+
+ /**
+ * Fills in the request id, page and request of a record from the current thread.
+ *
+ * @param record the record of a statement that just ran on this thread
+ * @return the same record
+ */
+ QueryTailRecord attribute(QueryTailRecord record) {
+ if (SPRING_WEB_PRESENT && WebRequestLookup.attribute(record, requestCounter, requestIdPrefix)) {
+ return record
+ }
+ record.requestId = null
+ record.request = null
+ record.page = QueryTailRecord.THREAD_PAGE_PREFIX + Thread.currentThread().name
+ record
+ }
+
+ /**
+ * Keeps every reference to Spring Web, in a signature as well as in a body, in a class that is only
+ * loaded once Spring Web is known to be present. The outer class can then be loaded, and reflected
+ * on, without Spring Web on the classpath.
+ */
+ @CompileStatic
+ private static class WebRequestLookup {
+
+ private static final String REQUEST_ID_PREFIX = 'r'
+
+ private static final String PLACEHOLDER = '?'
+
+ /** The parameters a URL mapping sets that name code rather than carry a value. */
+ private static final Set<String> MAPPING_PARAMETERS = ['controller', 'action', 'namespace', 'plugin', 'format'] as Set<String>
+
+ /** A segment that is a number, a UUID or a long hexadecimal token is a value whatever the mapping says. */
+ private static final Pattern VALUE_SHAPED = Pattern.compile(
+ '(?i)[0-9]+|[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}|[0-9a-f]{32,}')
+
+ private static final boolean SERVLET_PRESENT = ClassUtils.isPresent(
+ 'jakarta.servlet.http.HttpServletRequest', WebRequestLookup.classLoader)
+
+ /** The {@code getParams()} of a Grails web request, which holds the parameters its URL mapping took, by class. */
+ private static final ClassValue<Method> PARAMETERS = new ClassValue<Method>() {
+ @Override
+ protected Method computeValue(Class<?> type) {
+ try {
+ return type.getMethod('getParams')
+ }
+ catch (NoSuchMethodException ignored) {
+ return null
+ }
+ }
+ }
+
+ /**
+ * Fills in a record from the request on the current thread.
+ *
+ * @param record the record to fill in
+ * @param requestCounter the counter the next request id is taken from
+ * @param requestIdPrefix what every request id starts with
+ * @return whether there is a request on the current thread
+ */
+ static boolean attribute(QueryTailRecord record, AtomicLong requestCounter, String requestIdPrefix) {
+ RequestAttributes attributes = RequestContextHolder.getRequestAttributes()
+ if (attributes == null) {
+ return false
+ }
+ record.requestId = requestId(attributes, requestCounter, requestIdPrefix)
+ record.page = page(attributes)
+ record.request = SERVLET_PRESENT ? ServletRequestLookup.describe(attributes) : null
+ true
+ }
+
+ /**
+ * Builds the request line: the method and the path, with the values in the path, the parameters
+ * the URL mapping took from it among them, replaced.
+ *
+ * @param method the HTTP method
+ * @param path the request path
+ * @param attributes the request, whose mapped parameters are read when it is a Grails web request
+ * @param parameters the parameters of the servlet request, by name, each a {@code String[]}
+ */
+ static String describe(String method, String path, RequestAttributes attributes, Map<?, ?> parameters) {
+ // only the framework sets the controller attribute; a client can send a controller parameter
+ if (attributes.getAttribute(CONTROLLER_NAME_ATTRIBUTE, RequestAttributes.SCOPE_REQUEST) == null) {
+ return "${method} ${withholdPath(path)}".toString()
+ }
+ Set<String> values = new HashSet<>()
+ addValues(values, mappedParameters(attributes))
+ addValues(values, parameters)
+ "${method} ${redactPath(path, values)}".toString()
+ }
+
+ /**
+ * Replaces every segment of a path: before the URL mapping has run, nothing tells a literal
+ * segment from a token, so only the shape of the path is recorded.
+ */
+ private static String withholdPath(String path) {
+ String[] segments = path.split('/', -1)
+ StringBuilder out = new StringBuilder(path.length())
+ for (int i = 0; i < segments.length; i++) {
+ if (i > 0) {
+ out.append('/')
+ }
+ out.append(segments[i].isEmpty() ? segments[i] : PLACEHOLDER)
+ }
+ out.toString()
+ }
+
+ private static String requestId(RequestAttributes attributes, AtomicLong requestCounter, String requestIdPrefix) {
+ synchronized (attributes) {
+ Object existing = attributes.getAttribute(REQUEST_ID_ATTRIBUTE, RequestAttributes.SCOPE_REQUEST)
+ if (existing != null) {
+ return existing.toString()
+ }
+ String id = requestIdPrefix + REQUEST_ID_PREFIX + requestCounter.incrementAndGet()
+ attributes.setAttribute(REQUEST_ID_ATTRIBUTE, id, RequestAttributes.SCOPE_REQUEST)
+ return id
+ }
+ }
+
+ private static String page(RequestAttributes attributes) {
+ Object controller = attributes.getAttribute(CONTROLLER_NAME_ATTRIBUTE, RequestAttributes.SCOPE_REQUEST)
+ if (controller == null) {
+ return null
+ }
+ Object namespace = attributes.getAttribute(CONTROLLER_NAMESPACE_ATTRIBUTE, RequestAttributes.SCOPE_REQUEST)
+ Object action = attributes.getAttribute(ACTION_NAME_ATTRIBUTE, RequestAttributes.SCOPE_REQUEST)
+ StringBuilder page = new StringBuilder()
+ if (namespace != null) {
+ page.append(namespace).append(':')
+ }
+ page.append(controller)
+ if (action != null) {
+ page.append('/').append(action)
+ }
+ page.toString()
+ }
+
+ /**
+ * @return the parameters of a Grails web request, which include those its URL mapping took from
+ * the path, or null for a request that is not one
+ */
+ private static Map<?, ?> mappedParameters(RequestAttributes attributes) {
+ Method parameters = PARAMETERS.get(attributes.getClass())
+ if (parameters == null) {
+ return null
+ }
+ try {
+ Object value = parameters.invoke(attributes)
+ return value instanceof Map ? (Map<?, ?>) value : null
+ }
+ catch (ReflectiveOperationException | RuntimeException ignored) {
+ return null
+ }
+ }
+
+ private static void addValues(Set<String> values, Map<?, ?> parameters) {
+ if (parameters == null) {
+ return
+ }
+ for (Map.Entry<?, ?> entry in parameters.entrySet()) {
+ if (MAPPING_PARAMETERS.contains(String.valueOf(entry.key))) {
+ continue
+ }
+ Object value = entry.value
+ if (value instanceof CharSequence) {
+ values.add(value.toString())
+ }
+ else if (value instanceof Object[]) {
+ for (Object element in (Object[]) value) {
+ if (element instanceof CharSequence) {
+ values.add(element.toString())
+ }
+ }
+ }
+ }
+ }
+
+ /**
+ * Replaces the values in a path: a segment that is one of the values, with or without an
+ * extension, a run of segments that spells a value holding slashes, any segment shaped like a
+ * value, and any segment a value is glued into. The segments are matched as the client sent
+ * them and percent-decoded, since the parameters are decoded while the request URI is not; the
+ * path is written as sent.
+ */
+ private static String redactPath(String path, Set<String> values) {
+ String[] segments = path.split('/', -1)
+ String[] decoded = new String[segments.length]
+ boolean[] redacted = new boolean[segments.length]
+ for (int i = 0; i < segments.length; i++) {
+ decoded[i] = decode(segments[i])
+ redacted[i] = isValue(segments[i], values) || isValue(decoded[i], values)
+ }
+ for (String value in values) {
+ if (value.contains('/')) {
+ String[] run = value.split('/', -1)
+ markRun(segments, redacted, run)
+ markDecodedRun(decoded, redacted, run)
+ }
+ }
+ StringBuilder out = new StringBuilder(path.length())
+ for (int i = 0; i < segments.length; i++) {
+ if (i > 0) {
+ out.append('/')
+ }
+ boolean replaced = redacted[i] || holdsValue(segments[i], values) || holdsValue(decoded[i], values)
+ out.append(replaced ? PLACEHOLDER : segments[i])
+ }
+ out.toString()
+ }
+
+ /**
+ * Whether a value is glued into a segment that is not one itself, as a mapping that glues static
+ * text to a variable produces. A value of three or more characters counts wherever it appears, a
+ * shorter one only where no letter or digit continues it on either side, so that a one-letter
+ * query value does not take out every word of the path.
+ */
+ private static boolean holdsValue(String text, Set<String> values) {
+ for (String value in values) {
+ if (value.isEmpty()) {
+ continue
+ }
+ for (int at = text.indexOf(value); at >= 0; at = text.indexOf(value, at + 1)) {
+ int end = at + value.length()
+ if (value.length() >= 3 ||
+ ((at == 0 || !Character.isLetterOrDigit(text.charAt(at - 1))) &&
+ (end == text.length() || !Character.isLetterOrDigit(text.charAt(end))))) {
+ return true
+ }
+ }
+ }
+ false
+ }
+
+ /**
+ * @return the segment with its percent-encoding decoded, or as it is when it has none or is not
+ * valid percent-encoding; a plus sign is kept, since it is literal in a path
+ */
+ private static String decode(String segment) {
+ if (segment.indexOf('%') < 0) {
+ return segment
+ }
+ try {
+ return URLDecoder.decode(segment.replace('+', '%2B'), StandardCharsets.UTF_8)
+ }
+ catch (IllegalArgumentException ignored) {
+ return segment
+ }
+ }
+
+ private static boolean isValue(String segment, Set<String> values) {
+ if (segment.isEmpty()) {
+ return false
+ }
+ int dot = segment.lastIndexOf('.')
+ String bare = dot > 0 ? segment.substring(0, dot) : segment
+ values.contains(segment) || values.contains(bare) || VALUE_SHAPED.matcher(bare).matches()
+ }
+
+ /**
+ * Marks the segments whose decoded forms spell the run; a decoded segment may itself hold
+ * slashes, which the client encoded, so the decoded forms are split again before matching
+ * and a match marks the segments the matched parts came from.
+ */
+ private static void markDecodedRun(String[] decoded, boolean[] redacted, String[] run) {
+ List<String> parts = []
+ List<Integer> owners = []
+ for (int i = 0; i < decoded.length; i++) {
+ for (String part in decoded[i].split('/', -1)) {
+ parts.add(part)
+ owners.add(i)
+ }
+ }
+ boolean[] partRedacted = new boolean[parts.size()]
+ markRun(parts as String[], partRedacted, run)
+ for (int i = 0; i < partRedacted.length; i++) {
+ if (partRedacted[i]) {
+ redacted[owners.get(i)] = true
+ }
+ }
+ }
+
+ private static void markRun(String[] segments, boolean[] redacted, String[] run) {
+ if (run.length == 0 || run.length > segments.length) {
+ return
+ }
+ for (int start = 0; start + run.length <= segments.length; start++) {
+ boolean matches = true
+ for (int i = 0; i < run.length && matches; i++) {
+ matches = segments[start + i] == run[i]
+ }
+ if (matches) {
+ for (int i = 0; i < run.length; i++) {
+ redacted[start + i] = true
+ }
+ }
+ }
+ }
+ }
+
+ /**
+ * Keeps every reference to the servlet API in a class that is only loaded once the servlet API is
+ * known to be present, as {@link WebRequestLookup} does for Spring Web.
+ */
+ @CompileStatic
+ private static class ServletRequestLookup {
+
+ /**
+ * @return the request line of the servlet request on the current thread, or null when the request
+ * is not a servlet request
+ */
+ static String describe(RequestAttributes attributes) {
+ if (!(attributes instanceof ServletRequestAttributes)) {
+ return null
+ }
+ def request = ((ServletRequestAttributes) attributes).request
Review Comment:
`/book/index;jsessionid=abc123secretXYZ` is recorded unchanged. The session id is not a parameter and does not match the numeric, UUID, or hex patterns.
Mask a semicolon path parameter, especially `jsessionid`, before the request is written. Please assert that value does not reach the file.
##########
grails-data-mongodb/core/src/main/groovy/org/apache/grails/data/mongo/core/querytail/MongoCommandRedactor.groovy:
##########
@@ -0,0 +1,265 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * https://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.grails.data.mongo.core.querytail
+
+import groovy.transform.CompileStatic
+import org.bson.BsonDocument
+import org.bson.BsonValue
+
+import org.apache.grails.data.core.querytail.QueryTailRedactor
+
+/**
+ * Takes the values out of a MongoDB command before it is captured. A command is a document, so
+ * there is nothing to parse: every value becomes {@link QueryTailRedactor#PLACEHOLDER}, and the
+ * keys, which are the options of the command, its operators and the field names of the collection,
+ * are kept, except where a key may be data. The keys of a document stored in a map property, or in
+ * a property or a collection the {@linkplain MongoMappedFields mapping} does not know, are recorded
+ * as the placeholder too, and so is every segment of a dotted path past such a key. A numeric key
+ * is kept only where it indexes an array the mapping knows; in a map, in an element of an array,
+ * which is a map or a document unless the field is a list of lists, or at the top of a document,
+ * it is data. The entries the driver adds for its session, the cluster time and the transaction,
+ * which say nothing about the command, are left out.
+ *
+ * <p>A command is recorded as its name, the database and collection it runs against, and the
+ * document of its remaining entries: {@code find shop.book {"filter": {"title": ?}, "limit": ?}}.</p>
+ *
+ * @since 8.1
+ */
+@CompileStatic
+final class MongoCommandRedactor {
+
+ private static final String PLACEHOLDER = QueryTailRedactor.PLACEHOLDER
+
+ private static final String DATABASE = '$db'
+
+ private static final String COLLECTION = 'collection'
+
+ /** The entries the driver adds, which describe its session and transaction rather than the command. */
+ private static final Set<String> LEFT_OUT = ['lsid', '$clusterTime', 'txnNumber', 'autocommit',
+ 'startTransaction', '$readPreference', 'apiVersion', 'apiStrict', 'apiDeprecationErrors'] as Set<String>
+
+ /**
+ * The options whose document, or documents, are keyed by the fields of the collection, wherever they appear,
+ * including those of the operations of a client-level {@code bulkWrite}.
+ */
+ private static final Set<String> FIELD_KEYED = ['filter', 'query', 'q', 'u', 'update', 'document', 'documents',
+ 'updateMods', 'pipeline', 'sort', 'projection', 'fields', 'key', 'hint', 'min', 'max', 'arrayFilters',
+ 'partialFilterExpression', 'wildcardProjection'] as Set<String>
+
+ private MongoCommandRedactor() {
+ }
+
+ /**
+ * Redacts a command.
+ *
+ * @param database the database the command runs against
+ * @param commandName the name of the command, which is its first key
+ * @param command the command as the driver sends it
+ * @param fields the fields the mapping of the datastore knows, or null before the datastore is built
+ * @return the command with its values replaced
+ */
+ static String redact(String database, String commandName, BsonDocument command, MongoMappedFields fields) {
+ String collection = null
+ String collectionKey = null
+ BsonValue first = command.get(commandName)
+ if (first != null && first.isString()) {
+ collection = first.asString().value
+ collectionKey = commandName
+ }
+ else if (command.get(COLLECTION)?.isString()) {
+ collection = command.getString(COLLECTION).value
+ collectionKey = COLLECTION
+ }
+ StringBuilder out = new StringBuilder(commandName)
+ if (database != null || collection != null) {
+ out.append(' ')
+ if (database != null) {
+ out.append(database)
+ }
+ if (collection != null) {
+ out.append(database != null ? '.' : '').append(collection)
+ }
+ }
+ Scope options = Scope.options(collection, fields)
+ StringBuilder body = new StringBuilder()
+ for (Map.Entry<String, BsonValue> entry in command.entrySet()) {
+ String key = entry.key
+ if (key == DATABASE || key == collectionKey || LEFT_OUT.contains(key)) {
+ continue
+ }
+ body.append(body.length() == 0 ? '{' : ', ')
+ appendEntry(body, key, entry.value, options)
+ }
+ if (body.length() > 0) {
+ out.append(' ').append(body).append('}')
+ }
+ out.toString()
+ }
+
+ private static void appendEntry(StringBuilder out, String key, BsonValue value, Scope scope) {
+ Scope child = scope.of(key)
+ out.append(child.keyText).append(': ')
+ appendValue(out, value, child)
+ }
+
+ private static void appendValue(StringBuilder out, BsonValue value, Scope scope) {
+ if (value.isDocument()) {
+ out.append('{')
+ boolean first = true
+ for (Map.Entry<String, BsonValue> entry in value.asDocument().entrySet()) {
+ if (!first) {
+ out.append(', ')
+ }
+ first = false
+ appendEntry(out, entry.key, entry.value, scope)
+ }
+ out.append('}')
+ }
+ else if (value.isArray()) {
+ out.append('[')
+ boolean first = true
+ Scope element = scope.element()
+ for (BsonValue item in value.asArray()) {
+ if (!first) {
+ out.append(', ')
+ }
+ first = false
+ appendValue(out, item, element)
+ }
+ out.append(']')
+ }
+ else {
+ out.append(PLACEHOLDER)
+ }
+ }
+
+ private static String quote(String key) {
+ StringBuilder out = new StringBuilder(key.length() + 2).append('"')
+ for (int i = 0; i < key.length(); i++) {
+ char c = key.charAt(i)
+ if (c == '"' || c == '\\') {
+ out.append('\\').append(c)
+ }
+ else if (c < 0x20) {
+ out.append(String.format('\\u%04x', (int) c))
+ }
+ else {
+ out.append(c)
+ }
+ }
+ out.append('"').toString()
+ }
+
+ /**
+ * Where a document sits in the command: among the options of the command, whose keys are kept
+ * and name what their documents hold, or in a document keyed by fields, where a key is kept when
+ * it is an operator or a field the mapping knows at its path.
+ */
+ @CompileStatic
+ private static final class Scope {
+
+ final String collection
+ final MongoMappedFields fields
+ /** The field path of the document; null in a document keyed by fields the mapping does not know. */
+ final List<String> path
+ final boolean optionKeyed
+ final String keyText
+ /** How many indices into the field at the path have been taken: 0 at the field, 1 in one of its elements. */
+ final int depth
+
+ private Scope(String collection, MongoMappedFields fields, List<String> path, boolean optionKeyed,
+ String keyText, int depth) {
+ this.collection = collection
+ this.fields = fields
+ this.path = path
+ this.optionKeyed = optionKeyed
+ this.keyText = keyText
+ this.depth = depth
+ }
+
+ static Scope options(String collection, MongoMappedFields fields) {
+ new Scope(collection, fields, Collections.<String>emptyList(), true, null, 0)
+ }
+
+ /**
+ * @return the scope of an element of an array in this scope: one index deeper into the field
+ * at the path, so that a numeric key inside the element is data unless the elements are arrays
+ */
+ Scope element() {
+ optionKeyed || path == null || path.isEmpty() ? this : new Scope(collection, fields, path, false, keyText, depth + 1)
+ }
+
+ private boolean indexed(List<String> at, int atDepth) {
+ at != null && !at.isEmpty() && fields != null && fields.isIndexed(collection, at, atDepth)
+ }
+
+ /**
+ * @return the scope of the value of a key in a document of this scope, with the key as it is recorded
+ */
+ Scope of(String key) {
+ if (optionKeyed) {
+ boolean fieldKeyed = FIELD_KEYED.contains(key)
+ return new Scope(collection, fields, Collections.<String>emptyList(), !fieldKeyed, quote(key), 0)
+ }
+ if (key.startsWith('$')) {
Review Comment:
MongoDB 5+ allows a stored field name to start with `$`. Inserting `attrs: ['$example-secret': 'v']` keeps `$example-secret` because every dollar-prefixed key is treated as an operator, including inside a map.
Keep operator names only in an operator context. Mask a dollar-prefixed key in stored data.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@grails.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add the query-tail command to find query-heavy pages [grails-core]
Posted by "jdaugherty (via GitHub)" <gi...@apache.org>.
jdaugherty commented on code in PR #16631:
URL: https://github.com/apache/grails-core/pull/16631#discussion_r4365605734
##########
grails-datamapping-core/src/main/groovy/org/apache/grails/data/core/querytail/H2SqlSyntax.groovy:
##########
@@ -0,0 +1,65 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * https://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.grails.datastore.gorm.jdbc.sqltail;
+
+/**
+ * H2: block comments nest, {@code //} starts a line comment, {@code $$} quotes a string without a
+ * tag, backticks delimit a name in every compatibility mode, and square brackets delimit a name in
+ * the SQL Server mode, where they are never a subscript, which is what a bracket after a name, a
+ * closing parenthesis or a closing bracket is in the other modes.
+ */
+final class H2SqlSyntax extends StandardSqlSyntax {
+
+ H2SqlSyntax() {
+ super("H2");
+ }
+
+ @Override
+ public int endOfComment(String sql, int index) {
+ return sql.startsWith("//", index) ? endOfLine(sql, index) : super.endOfComment(sql, index);
+ }
+
+ @Override
+ public int endOfValue(String sql, int index) {
+ int end = endOfDollarQuoted(sql, index, false);
+ return end != NONE ? end : super.endOfValue(sql, index);
+ }
+
+ @Override
+ public int endOfQuotedName(String sql, int index) {
+ char c = sql.charAt(index);
+ if (c == '`') {
+ return endOfQuoted(sql, index, '`', false, true);
+ }
+ if (c == '[' && !subscript(sql, index)) {
+ return endOfQuoted(sql, index, ']', false, false);
+ }
+ return super.endOfQuotedName(sql, index);
+ }
+
+ @Override
+ protected boolean nestedComments() {
+ return true;
+ }
+
+ private boolean subscript(String sql, int index) {
Review Comment:
Fixed in 67ba541f11. H2 now reads a bracket that follows whitespace both ways, as a name in the SQL Server mode and as the punctuation an array literal opens with after `ARRAY`, however far before it the keyword stands. A name agrees with itself and is recorded; an array literal, whose elements are values, does not and the statement is withheld. `QueryTailRedactorSpec` asserts `select ARRAY ['example-secret'] from t`, the same with a block comment and with a line comment between `ARRAY` and `[`, and `select [it's] from t` are all withheld, and the never-survives list for H2 holds the literal in those positions, including `ARRAY ['a', 'secret'][1]`.
##########
grails-datamapping-core/src/main/groovy/org/apache/grails/data/core/querytail/H2SqlSyntax.groovy:
##########
@@ -0,0 +1,70 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * https://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.grails.data.core.querytail
+
+import groovy.transform.CompileStatic
+import groovy.transform.PackageScope
+
+/**
+ * H2: block comments nest, {@code //} starts a line comment, {@code $$} quotes a string without a
+ * tag, backticks delimit a name in every compatibility mode, and square brackets delimit a name in
+ * the SQL Server mode, where they are never a subscript, which is what a bracket after a name, a
+ * closing parenthesis or a closing bracket is in the other modes.
+ */
+@CompileStatic
+@PackageScope
+final class H2SqlSyntax extends StandardSqlSyntax {
+
+ H2SqlSyntax() {
+ super('H2')
+ }
+
+ @Override
+ int endOfComment(String sql, int index) {
+ sql.startsWith('//', index) ? endOfLine(sql, index) : super.endOfComment(sql, index)
+ }
+
+ @Override
+ int endOfValue(String sql, int index) {
+ int end = endOfDollarQuoted(sql, index, false)
+ end != NONE ? end : super.endOfValue(sql, index)
+ }
+
+ @Override
+ int endOfQuotedName(String sql, int index) {
+ char c = sql.charAt(index)
+ if (c == '`') {
+ return endOfQuoted(sql, index, '`' as char, false, true)
+ }
+ if (c == '[' && !subscript(sql, index)) {
Review Comment:
Fixed in 67ba541f11. H2 now reads a bracket that follows whitespace both ways, as a name in the SQL Server mode and as the punctuation an array literal opens with after `ARRAY`, however far before it the keyword stands. A name agrees with itself and is recorded; an array literal, whose elements are values, does not and the statement is withheld. `QueryTailRedactorSpec` asserts `select ARRAY ['example-secret'] from t`, the same with a block comment and with a line comment between `ARRAY` and `[`, and `select [it's] from t` are all withheld, and the never-survives list for H2 holds the literal in those positions, including `ARRAY ['a', 'secret'][1]`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@grails.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add the query-tail command to find query-heavy pages [grails-core]
Posted by "jdaugherty (via GitHub)" <gi...@apache.org>.
jdaugherty commented on PR #16631:
URL: https://github.com/apache/grails-core/pull/16631#issuecomment-6378483128
Three commits on the branch for the moment, 67ba541f11, 5ed78b9dee and 44b44e9fb2:
- **67ba541f11, the feature**, now with the H2 bracket fix from the last review: a bracket after whitespace or a comment is read both as a name and as the punctuation an array literal opens with, so `ARRAY ['x']` is withheld and `[name]` still recorded. Both threads are answered and resolved.
- **5ed78b9dee, a memory leak in the compiler, kept separate because it is not part of the feature.** `AstUtils.isDomainClass` was `@Memoized`, an unbounded JVM-wide cache keyed by class name that pinned every `ClassNode` it was asked about, and with it the node's whole compile unit and class loader. Measured in a Gradle compiler fork after the grails-datamapping-core test compile: 194,420 `ClassNode`s and 154 `ModuleNode`s still held, 222 MB after a full collection, as the next compile started; with the fix, one `ModuleNode` and 76 MB. The verdict now lives as node metadata, which also ends the same-name collision #16653 had worked around. The memo dates from GORM itself (2015) and is on every branch, so it is backported as its own pull request against 7.0.x, #16747, to merge up; this commit goes once that reaches 9.0.x.
- **44b44e9fb2 is a temporary diagnostic and will be dropped.** The MongoDB core test compile still runs out of its 2 GB on CI after the `where` closure went, including in the end-to-end lane where its main compile came from the cache, so that fork was fresh. Here the same compile takes 7 seconds and fits in 600 MB. The commit makes a fork that exhausts its heap print a class histogram into the build log and upload its heap dump from the end-to-end lane, so the next failure says what fills it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@grails.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add the query-tail command to find query-heavy pages [grails-core]
Posted by "jdaugherty (via GitHub)" <gi...@apache.org>.
jdaugherty commented on PR #16631:
URL: https://github.com/apache/grails-core/pull/16631#issuecomment-6381719926
Rebased onto 9.0.x at 5175f69a00. The branch is back to the single feature commit, 54a6574fc5 (same patch as before, unchanged content): the `isDomainClass` verdict fix and the removal of the MongoDB modules' 768 MB compiler cap now reach 9.0.x through #16747's merge-up, so their copies here are dropped. Verified on the new base: datastore-core compile, the query-tail specs (376), the datamapping-core cli tests, style check, and the MongoDB/Neo4j/Hibernate 7 core test compiles.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@grails.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add the query-tail command to find query-heavy pages [grails-core]
Posted by "jdaugherty (via GitHub)" <gi...@apache.org>.
jdaugherty commented on code in PR #16631:
URL: https://github.com/apache/grails-core/pull/16631#discussion_r4376728360
##########
grails-datamapping-core/src/main/groovy/org/apache/grails/data/core/querytail/QueryTailCapture.groovy:
##########
@@ -0,0 +1,368 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * https://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.grails.data.core.querytail
+
+import java.nio.ByteBuffer
+import java.nio.channels.FileChannel
+import java.nio.channels.FileLock
+import java.nio.channels.OverlappingFileLockException
+import java.nio.charset.StandardCharsets
+import java.nio.file.Files
+import java.nio.file.StandardOpenOption
+import java.util.concurrent.atomic.AtomicBoolean
+import java.util.concurrent.atomic.AtomicLong
+
+import groovy.transform.CompileStatic
+import groovy.util.logging.Slf4j
+
+/**
+ * Writes the statements an application runs to the capture file the {@code query-tail} command
+ * tails. Each session factory, Neo4j driver or MongoDB client that captures opens its own capture
+ * and closes it when it closes; the captures of one file share it, so every data source of an
+ * application writes to the one file. The file is truncated when the first capture opens it, so it
+ * holds the statements of the current run only, and its first line names that run. It is closed
+ * when the last capture closes, and opened afresh by the next one. Each statement is written whole
+ * and at once, so that the command sees it as soon as it is recorded.
+ *
+ * <p>JVMs that run at the same time with capture enabled, such as the parallel forks of a test task,
+ * share the file: a lock file beside it tells the first to start, which truncates the file and names
+ * the run, from the others, which append to that run without truncating it and number their
+ * requests apart, so that the command reads one run from all of them.</p>
+ *
+ * <p>A statement is recorded without its values: {@link QueryTailRedactor} replaces its literals and
+ * drops its comments before it is written, in the {@link #syntax} of the database it runs on. A
+ * statement that is not text, such as a MongoDB command, has its values taken out by whoever records
+ * it, through {@link #recordRedacted}.</p>
+ *
+ * <p>A capture is only opened when {@link QueryTailSettings#SETTING_ENABLED} is set; an application
+ * started without it never reaches this class.</p>
+ *
+ * @since 8.1
+ */
+@CompileStatic
+@Slf4j
+class QueryTailCapture implements Closeable {
+
+ private static final Map<String, CaptureFile> OPEN = new HashMap<>()
+
+ /** The file the statements are written to. */
+ final File file
+
+ /** The syntax of the database the statements of this capture run on, which their values are found by. */
+ volatile QueryTailSyntax syntax = QueryTailSyntax.UNKNOWN
+
+ private final CaptureFile captureFile
+ private final AtomicBoolean closed = new AtomicBoolean()
+
+ private QueryTailCapture(File file, CaptureFile captureFile) {
+ this.file = file
+ this.captureFile = captureFile
+ }
+
+ /**
+ * Opens a capture of a file. The first capture of a file truncates it; a capture opened while
+ * another of the same file is open shares it. Every capture opened must be closed.
+ *
+ * @param file the capture file
+ * @return the capture
+ * @throws UncheckedIOException when the file cannot be created
+ */
+ static QueryTailCapture open(File file) {
+ String key
+ try {
+ key = file.canonicalPath
+ }
+ catch (IOException e) {
+ throw new UncheckedIOException("Cannot resolve the query-tail capture file ${file}: ${e.message}", e)
+ }
+ synchronized (OPEN) {
+ CaptureFile captureFile = OPEN.get(key)
+ if (captureFile == null) {
+ captureFile = new CaptureFile(file, key)
+ OPEN.put(key, captureFile)
+ if (captureFile.joined) {
+ log.info('Capturing statements to {} for query-tail, in the run of another JVM writing it', file)
+ }
+ else {
+ log.info('Capturing statements to {} for query-tail', file)
+ }
+ }
+ captureFile.captures++
+ return new QueryTailCapture(file, captureFile)
+ }
+ }
+
+ /**
+ * Opens the capture the settings name.
+ *
+ * @param settings the query-tail settings
+ * @return the capture
+ */
+ static QueryTailCapture open(QueryTailSettings settings) {
+ open(settings.resolveFile())
+ }
+
+ /**
+ * Records a statement that is about to run on the current thread, without its values.
+ *
+ * @param dataSource the connection source name the statement runs against
+ * @param sql the statement
+ */
+ void record(String dataSource, String sql) {
+ write(dataSource, QueryTailRedactor.redact(sql, syntax))
+ }
+
+ /**
+ * Records a statement that is about to run on the current thread and that holds no value any
+ * more: a MongoDB command, which is a document rather than text, is redacted field by field by
+ * the listener that records it. Nothing of the statement is changed here.
+ *
+ * @param dataSource the connection source name the statement runs against
+ * @param statement the statement, with its values already taken out
+ */
+ void recordRedacted(String dataSource, String statement) {
+ write(dataSource, statement)
+ }
+
+ private void write(String dataSource, String statement) {
+ if (closed.get()) {
+ return
+ }
+ QueryTailRecord record = new QueryTailRecord(timestamp: System.currentTimeMillis(), dataSource: dataSource,
+ sql: statement)
+ captureFile.write(captureFile.attribution.attribute(record))
+ }
+
+ /**
+ * @return whether this capture is still writing to its file
+ */
+ boolean isOpen() {
+ !closed.get() && captureFile.open
+ }
+
+ /**
+ * Closes this capture. The file is closed once every capture of it is.
+ */
+ @Override
+ void close() {
+ if (!closed.compareAndSet(false, true)) {
+ return
+ }
+ synchronized (OPEN) {
+ if (--captureFile.captures == 0) {
+ // A replacement must not join this writer's run before its coordination lock is released.
+ captureFile.close()
+ if (OPEN.get(captureFile.key).is(captureFile)) {
+ OPEN.remove(captureFile.key)
+ }
+ }
+ }
+ }
+
+ /**
+ * A capture file and the captures writing to it. The file is written by one channel, a line at a
+ * time, and the requests of every data source are numbered by one attribution, so that two
+ * requests never share an id. A JVM that joins the run of another numbers the requests of every run
+ * it joins by one counter, so that it repeats none of its ids when it closes its capture and joins
+ * the same run again.
+ *
+ * <p>A lock file beside the capture coordinates the JVMs writing it. Every writer holds a shared
+ * lock on the {@link #WRITING} byte of the lock file while it writes. A starting writer first
+ * takes the {@link #STARTING} byte exclusively, so that starters take turns, then tries the
+ * {@link #WRITING} byte exclusively: when it gets it, no other JVM is writing, and it truncates the
+ * file and writes the header of a new run before it lets the next starter in, so that the header
+ * is the first line of the run whatever a joiner writes after it; when it does not, it joins the
+ * run being written. Either way it takes the {@link #WRITING} byte shared before it lets go of
+ * {@link #STARTING} and holds it until it closes. A lock file that cannot be created, as beside a
+ * device, leaves the writer uncoordinated, as the only one.</p>
+ */
+ @CompileStatic
+ private static class CaptureFile {
+
+ /** The byte of the lock file a starting writer holds, exclusively, while it decides whether to truncate. */
+ private static final long STARTING = 0L
+
+ /** The byte of the lock file every writer holds, shared, while it writes. */
+ private static final long WRITING = 1L
+
+ /**
+ * Numbers the requests of every run this JVM joins, so that a JVM that closes its capture and
+ * joins the same run again never repeats an id it gave a request of that run before.
+ */
+ private static final AtomicLong JOINED_REQUESTS = new AtomicLong()
+
+ final File file
+ final String key
+ final QueryTailRequestAttribution attribution
+
+ /** Whether this JVM joined the run of another JVM writing the file, rather than starting one. */
+ final boolean joined
+
+ /** How many captures are open on the file; guarded by {@link QueryTailCapture#OPEN}. */
+ int captures
+
+ private FileChannel channel
+ private FileChannel lockChannel
+ private FileLock writing
+
+ CaptureFile(File file, String key) {
+ this.file = file
+ this.key = key
+ File parent = file.absoluteFile.parentFile
+ boolean first
+ try {
+ if (parent != null && !parent.isDirectory() && !parent.mkdirs() && !parent.isDirectory()) {
+ throw new IOException("Cannot create the query-tail capture directory ${parent}")
+ }
+ lockChannel = openLockFile(file)
+ FileLock starting = lockChannel == null ? null : lockChannel.lock(STARTING, 1L, false)
+ try {
+ FileLock alone = lockChannel == null ? null : lock(lockChannel, false, true)
+ first = lockChannel == null || alone != null
+ alone?.release()
+ if (first) {
+ Files.newByteChannel(file.toPath(), StandardOpenOption.CREATE, StandardOpenOption.WRITE,
+ StandardOpenOption.TRUNCATE_EXISTING).close()
+ }
+ channel = FileChannel.open(file.toPath(), StandardOpenOption.CREATE, StandardOpenOption.WRITE,
+ StandardOpenOption.APPEND)
+ writing = lockChannel == null ? null : lock(lockChannel, true, false)
+ if (first) {
+ writeLine(QueryTailRecord.header(UUID.randomUUID().toString()))
+ }
+ }
+ finally {
Review Comment:
Fixed in da16a91754. A failed header write has already closed the lock file, which released the start lock with it, so the `finally` now releases that lock only while it is still valid. `open()` therefore returns a stopped capture instead of throwing.
Returning a stopped capture exposed one more case. When the first write fails, the write path tries to let the file go before `open()` has registered it, so it would have been registered afterwards as a dead file, and the next capture of the same path would have joined it. `open()` now registers a capture file only if it is still open after it starts, the same way a file whose later write fails is let go.
The new feature in `QueryTailCaptureSpec`, "a capture whose first write fails is opened stopped, even with a lock file beside it, and the next one tries the file afresh", links the capture file to `/dev/full` in a temp directory, so the lock file can always be created:
- It checks that `open()` returns a stopped capture.
- It deletes the lock file, opens the path again while the stopped capture is still open, and checks that the new open created the lock file itself.
I ran the spec in a Linux container against the unfixed class first. The new feature failed with the `UncheckedIOException` you described, and so did the existing `/dev/full` feature, since the container runs as root and can create `/dev/full.lock`. With only the lock-release half of the fix, the second open joins the stopped file. With the whole fix, the spec passes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@grails.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add the query-tail command to find query-heavy pages [grails-core]
Posted by "jdaugherty (via GitHub)" <gi...@apache.org>.
jdaugherty commented on PR #16631:
URL: https://github.com/apache/grails-core/pull/16631#issuecomment-6397137136
@jamesfredley re review 5641362789: both items are fixed on head 73c20a13c7. Path parameters after a semicolon are split off before matching and each is recorded as `;?`, so `/book/index;jsessionid=abc123secretXYZ` records `GET /book/index;?`, asserted through the capture API. The MongoDB redactor now tells conditions, update operators and stored data apart: a `$` key stays an operator only in a condition, a pipeline or at the top of an update, and in stored data (inserted documents, replacement documents, values an update operator writes, `$push`/`$addToSet` elements, `$literal`) it is a field name recorded as `?` unless mapped; proven against a real mongod with an insert and a `$set` of `attrs: {"$example-secret": ...}`. Details are in the replies on your two threads, which I resolved since both are fixed; reopen either to discuss further.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@grails.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add the query-tail command to find query-heavy pages [grails-core]
Posted by "jamesfredley (via GitHub)" <gi...@apache.org>.
jamesfredley commented on code in PR #16631:
URL: https://github.com/apache/grails-core/pull/16631#discussion_r4370961857
##########
grails-data-mongodb/core/src/main/groovy/org/apache/grails/data/mongo/core/querytail/MongoCommandRedactor.groovy:
##########
@@ -0,0 +1,363 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * https://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.grails.data.mongo.core.querytail
+
+import groovy.transform.CompileStatic
+import org.bson.BsonDocument
+import org.bson.BsonValue
+
+import org.apache.grails.data.core.querytail.QueryTailRedactor
+
+/**
+ * Takes the values out of a MongoDB command before it is captured. A command is a document, so
+ * there is nothing to parse: every value becomes {@link QueryTailRedactor#PLACEHOLDER}, and the
+ * keys, which are the options of the command, its operators and the field names of the collection,
+ * are kept, except where a key may be data. The keys of a document stored in a map property, or in
+ * a property or a collection the {@linkplain MongoMappedFields mapping} does not know, are recorded
+ * as the placeholder too, and so is every segment of a dotted path past such a key. A numeric key
+ * is kept only where it indexes an array the mapping knows; in a map, in an element of an array,
+ * which is a map or a document unless the field is a list of lists, or at the top of a document,
+ * it is data. A {@code $} key is kept as an operator only where operators go, in a condition, a
+ * pipeline and the operators of an update; in stored data, the documents an insert writes, a
+ * replacement document and the values an update operator writes, it is a field name, which MongoDB
+ * allows to start with {@code $}, and is recorded as the placeholder unless the mapping knows it.
+ * The entries the driver adds for its session, the cluster time and the transaction, which say
+ * nothing about the command, are left out.
+ *
+ * <p>A command is recorded as its name, the database and collection it runs against, and the
+ * document of its remaining entries: {@code find shop.book {"filter": {"title": ?}, "limit": ?}}.</p>
+ *
+ * @since 8.1
+ */
+@CompileStatic
+final class MongoCommandRedactor {
+
+ private static final String PLACEHOLDER = QueryTailRedactor.PLACEHOLDER
+
+ private static final String DATABASE = '$db'
+
+ private static final String COLLECTION = 'collection'
+
+ /** The entries the driver adds, which describe its session and transaction rather than the command. */
+ private static final Set<String> LEFT_OUT = ['lsid', '$clusterTime', 'txnNumber', 'autocommit',
+ 'startTransaction', '$readPreference', 'apiVersion', 'apiStrict', 'apiDeprecationErrors'] as Set<String>
+
+ /**
+ * The options whose document, or documents, are keyed by the fields of the collection, wherever they appear,
+ * including those of the operations of a client-level {@code bulkWrite}.
+ */
+ private static final Set<String> FIELD_KEYED = ['filter', 'query', 'q', 'u', 'update', 'document', 'documents',
+ 'updateMods', 'pipeline', 'sort', 'projection', 'fields', 'key', 'hint', 'min', 'max', 'arrayFilters',
+ 'partialFilterExpression', 'wildcardProjection'] as Set<String>
+
+ /** The options whose documents are stored data: what an insert writes. */
+ private static final Set<String> DATA_KEYED = ['documents', 'document'] as Set<String>
+
+ /** The options that hold an update: update operators, a replacement document or a pipeline. */
+ private static final Set<String> UPDATE_KEYED = ['u', 'update', 'updateMods'] as Set<String>
+
+ /** The update operators whose fields take a condition rather than the value to write. */
+ private static final Set<String> CONDITION_OPERATORS = ['$pull', '$currentDate'] as Set<String>
+
+ /** The update operators that add to an array, whose value may hold modifiers. */
+ private static final Set<String> PUSH_OPERATORS = ['$push', '$addToSet'] as Set<String>
+
+ private static final String EACH = '$each'
+
+ /** The modifiers of a {@code $push} or an {@code $addToSet}. */
+ private static final Set<String> PUSH_MODIFIERS = [EACH, '$slice', '$sort', '$position'] as Set<String>
+
+ /** The operators of a pipeline whose operand is literal data rather than an expression. */
+ private static final Set<String> LITERAL_OPERATORS = ['$literal', '$documents'] as Set<String>
+
+ private MongoCommandRedactor() {
+ }
+
+ /**
+ * Redacts a command.
+ *
+ * @param database the database the command runs against
+ * @param commandName the name of the command, which is its first key
+ * @param command the command as the driver sends it
+ * @param fields the fields the mapping of the datastore knows, or null before the datastore is built
+ * @return the command with its values replaced
+ */
+ static String redact(String database, String commandName, BsonDocument command, MongoMappedFields fields) {
+ String collection = null
+ String collectionKey = null
+ BsonValue first = command.get(commandName)
+ if (first != null && first.isString()) {
+ collection = first.asString().value
+ collectionKey = commandName
+ }
+ else if (command.get(COLLECTION)?.isString()) {
+ collection = command.getString(COLLECTION).value
+ collectionKey = COLLECTION
+ }
+ StringBuilder out = new StringBuilder(commandName)
+ if (database != null || collection != null) {
+ out.append(' ')
+ if (database != null) {
+ out.append(database)
+ }
+ if (collection != null) {
+ out.append(database != null ? '.' : '').append(collection)
+ }
+ }
+ Scope options = Scope.options(collection, fields)
+ StringBuilder body = new StringBuilder()
+ for (Map.Entry<String, BsonValue> entry in command.entrySet()) {
+ String key = entry.key
+ if (key == DATABASE || key == collectionKey || LEFT_OUT.contains(key)) {
+ continue
+ }
+ body.append(body.length() == 0 ? '{' : ', ')
+ appendEntry(body, key, entry.value, options)
+ }
+ if (body.length() > 0) {
+ out.append(' ').append(body).append('}')
+ }
+ out.toString()
+ }
+
+ private static void appendEntry(StringBuilder out, String key, BsonValue value, Scope scope) {
+ Scope child = scope.of(key)
+ out.append(child.keyText).append(': ')
+ appendValue(out, value, child)
+ }
+
+ private static void appendValue(StringBuilder out, BsonValue value, Scope scope) {
+ if (value.isDocument()) {
+ out.append('{')
+ boolean first = true
+ for (Map.Entry<String, BsonValue> entry in value.asDocument().entrySet()) {
+ if (!first) {
+ out.append(', ')
+ }
+ first = false
+ appendEntry(out, entry.key, entry.value, scope)
+ }
+ out.append('}')
+ }
+ else if (value.isArray()) {
+ out.append('[')
+ boolean first = true
+ Scope element = scope.element()
+ for (BsonValue item in value.asArray()) {
+ if (!first) {
+ out.append(', ')
+ }
+ first = false
+ appendValue(out, item, element)
+ }
+ out.append(']')
+ }
+ else {
+ out.append(PLACEHOLDER)
+ }
+ }
+
+ private static String quote(String key) {
+ StringBuilder out = new StringBuilder(key.length() + 2).append('"')
+ for (int i = 0; i < key.length(); i++) {
+ char c = key.charAt(i)
+ if (c == '"' || c == '\\') {
+ out.append('\\').append(c)
+ }
+ else if (c < 0x20) {
+ out.append(String.format('\\u%04x', (int) c))
+ }
+ else {
+ out.append(c)
+ }
+ }
+ out.append('"').toString()
+ }
+
+ /**
+ * Where a document sits in the command, which decides what its keys are: the options of the
+ * command, kept; the fields and operators of a condition, a pipeline, a sort or a projection, where
+ * a {@code $} key is an operator; an update, whose {@code $} keys are update operators; or stored
+ * data, the documents an insert writes and the values an update writes, where a {@code $} key is a
+ * field name like any other and is kept only when the mapping knows it.
+ */
+ @CompileStatic
+ private static final class Scope {
+
+ final String collection
+ final MongoMappedFields fields
+ final Kind kind
+ /** The update operator whose fields a document of kind {@link Kind#UPDATE_FIELDS} holds. */
+ final String operator
+ /** The field path of the document; null in a document keyed by fields the mapping does not know. */
+ final List<String> path
+ /** How many indices into the field at the path have been taken: 0 at the field, 1 in one of its elements. */
+ final int depth
+ final String keyText
+
+ private Scope(String collection, MongoMappedFields fields, Kind kind, String operator, List<String> path,
+ int depth, String keyText) {
+ this.collection = collection
+ this.fields = fields
+ this.kind = kind
+ this.operator = operator
+ this.path = path
+ this.depth = depth
+ this.keyText = keyText
+ }
+
+ static Scope options(String collection, MongoMappedFields fields) {
+ new Scope(collection, fields, Kind.OPTIONS, null, Collections.<String>emptyList(), 0, null)
+ }
+
+ /**
+ * @return the scope of an element of an array in this scope: one index deeper into the field
+ * at the path, so that a numeric key inside the element is data unless the elements are arrays;
+ * the elements of an update that is an array are the stages of a pipeline
+ */
+ Scope element() {
+ if (kind == Kind.UPDATE) {
+ return child(Kind.QUERY, null, path, depth, keyText)
+ }
+ if (kind == Kind.PUSH) {
+ return child(Kind.DATA, null, path, depth + 2, keyText)
+ }
+ if ((kind == Kind.QUERY || kind == Kind.DATA) && path != null && !path.isEmpty()) {
+ return child(kind, null, path, depth + 1, keyText)
+ }
+ this
+ }
+
+ /**
+ * @return the scope of the value of a key in a document of this scope, with the key as it is recorded
+ */
+ Scope of(String key) {
+ if (kind == Kind.OPTIONS) {
+ Kind valueKind = DATA_KEYED.contains(key) ? Kind.DATA :
+ UPDATE_KEYED.contains(key) ? Kind.UPDATE :
+ FIELD_KEYED.contains(key) ? Kind.QUERY : Kind.OPTIONS
+ return child(valueKind, null, Collections.<String>emptyList(), 0, quote(key))
+ }
+ if (kind == Kind.QUERY && key.startsWith('$')) {
Review Comment:
`{"attrs":{"$eq":{"$example-secret":"value"}}}` still records `$example-secret`. The `$eq` operand stays in query scope, so a dollar-prefixed data key is treated as an operator. MongoDB 7 accepts that filter.
Treat a literal comparison operand as data. Please assert that key does not reach the file.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@grails.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add the query-tail command to find query-heavy pages [grails-core]
Posted by "jdaugherty (via GitHub)" <gi...@apache.org>.
jdaugherty commented on PR #16631:
URL: https://github.com/apache/grails-core/pull/16631#issuecomment-6421052656
@jamesfredley I believe this is ready for your review again.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@grails.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add the sql-tail command to find query-heavy pages [grails-core]
Posted by "jdaugherty (via GitHub)" <gi...@apache.org>.
jdaugherty commented on PR #16631:
URL: https://github.com/apache/grails-core/pull/16631#issuecomment-6343657017
@jamesfredley the two line comments are answered on their threads. The three items from the review body are fixed in 43640050e9 as well:
- **A runtime-added connection dropped the application's sql-tail settings.** `HibernateConnectionSourceFactory` (H7) and `AbstractHibernateConnectionSourceFactory` (H5) now read them with `SqlTailSettings.from(configuration, fallback)`, where the fallback is the sql-tail settings of the connection source it is built after. A connection added through `addConnectionSource` captures as the application does unless its own configuration says otherwise. Covered by `a data source added at runtime records as the application does` in both inspector specs, and by `SqlTailSettingsSpec`.
- **Truncation was the only restart signal.** The first line of the capture now names the run (`# sql-tail capture <uuid>`), and `SqlTailFileReader` starts over when that line names a different run, as well as when the file shrinks. A file caught between the truncation and its header being written is picked up without a second restart. `SqlTailFileReaderSpec` covers a new run that grew past the old offset before the next poll; `SqlTailRunnerSpec` covers it end to end in the stream.
- **Request totals versus page labels, and interleaved stream counts.** `SqlTailSummary` now keeps request statistics per page, so a page's MAX and most repeated statement count only what a request ran on that page; a forward, or a filter query before the controller is chosen, counts the request on each page with its own statements. `SqlTailRenderer` keeps a running count per request instead of dropping it when another request interleaves, heads a returning request as `(continued)`, keeps the 1000 most recent requests, and forgets them on a restart. Covered in `SqlTailSummarySpec`, `SqlTailRendererSpec` and `SqlTailRunnerSpec`.
The guide section, the `sql-tail` reference page and What's New now describe the redaction and the capture lifecycle.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@grails.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add the query-tail command to find query-heavy pages [grails-core]
Posted by "testlens-app[bot] (via GitHub)" <gi...@apache.org>.
testlens-app[bot] commented on PR #16631:
URL: https://github.com/apache/grails-core/pull/16631#issuecomment-6462174304
## ✅ All tests passed ✅
🏷️ Commit: 8976ddec9801de57cc106eb045fea86bc79840fd
▶️ Tests: 141370 executed
⚪️ Checks: 97/97 completed
---
_Learn more about TestLens at [testlens.app/docs](https://testlens.app/docs/features/pr-comment/)._
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@grails.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add the sql-tail command to find query-heavy pages [grails-core]
Posted by "jdaugherty (via GitHub)" <gi...@apache.org>.
jdaugherty commented on PR #16631:
URL: https://github.com/apache/grails-core/pull/16631#issuecomment-6297590715
Once the tests pass, I intend to merge this since it's a stand alone feature. If anyone objects, we can iterate on the change.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@grails.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add the query-tail command to find query-heavy pages [grails-core]
Posted by "jdaugherty (via GitHub)" <gi...@apache.org>.
jdaugherty commented on code in PR #16631:
URL: https://github.com/apache/grails-core/pull/16631#discussion_r4388383705
##########
grails-datamapping-core/src/main/groovy/org/apache/grails/datamapping/core/querytail/CypherSyntax.groovy:
##########
@@ -0,0 +1,110 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * https://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.grails.datamapping.core.querytail
+
+import groovy.transform.CompileStatic
+import groovy.transform.PackageScope
+
+/**
+ * Cypher, the query language of Neo4j: single and double quotes delimit a string, in which a
+ * backslash escapes the character after it and a quote written twice is two strings; backticks
+ * delimit a name, in which a backtick is written twice; {@code //} starts a line comment, and
+ * {@code --} and {@code #} do not, as {@code -->} draws a relationship; block comments do not nest;
+ * and a dollar starts a parameter. Neo4j 5 ends a line comment at a carriage return, which Neo4j 3.5
+ * does not, so a statement with a carriage return that no line feed follows is read both ways. Neo4j 5
+ * decodes a {@code \u} escape before it parses, where 3.5 reads it only inside a string, so a
+ * backslash outside a string or a comment, which may spell a quote, withholds the statement.
+ */
+@CompileStatic
+@PackageScope
+final class CypherSyntax extends StandardSqlSyntax {
+
+ private static final char BACKSLASH = '\\'
+ private static final char QUOTE = '\''
+ private static final char DOUBLE_QUOTE = '"'
+ private static final char BACKTICK = '`'
+
+ private final boolean carriageReturnEndsLine
+
+ CypherSyntax(boolean carriageReturnEndsLine) {
+ super('Cypher')
+ this.carriageReturnEndsLine = carriageReturnEndsLine
+ }
+
+ @Override
+ List<QueryTailSyntax> readings(String sql) {
+ if (!hasLoneCarriageReturn(sql)) {
+ return List.<QueryTailSyntax>of(this)
+ }
+ List.<QueryTailSyntax>of(new CypherSyntax(false), new CypherSyntax(true))
+ }
+
+ @Override
+ int endOfComment(String sql, int index) {
+ if (sql.startsWith('//', index)) {
+ return endOfLine(sql, index)
+ }
+ if (sql.startsWith('/*', index)) {
+ return endOfBlockComment(sql, index)
+ }
+ NONE
+ }
+
+ @Override
+ int endOfValue(String sql, int index) {
+ char c = sql.charAt(index)
+ if (c == BACKSLASH) {
+ return UNTERMINATED
+ }
+ if (c == QUOTE || c == DOUBLE_QUOTE) {
Review Comment:
Fixed in d8f76c42e3. Neo4j 5 decodes a `\u` escape that no backslash escapes before it lexes. Its Cypher 5 parser (`CypherCharStream`) and its Cypher 25 parser (`UnicodeEscapeReplacementReader`) both do this. So an escape in a string or a comment can end it somewhere other than where 3.5 and the redactor end it. Such an escape now withholds the statement. An escaped backslash before a `u` (`'C:\\users'`) is not an escape and is still redacted to `?`.
Your statement is now asserted withheld, with `secret` absent from the file. That is checked at three levels:
- `QueryTailRedactorSpec`: withheld rows, plus the never-survives list.
- `QueryTailCaptureSpec`: the capture file.
- `QueryTailDriverSpec`: through the real driver on the embedded server.
The new rows also cover `\uu0022`, an escape after an escaped backslash, and escapes in `//` and `/* */` comments.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@grails.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add the query-tail command to find query-heavy pages [grails-core]
Posted by "jamesfredley (via GitHub)" <gi...@apache.org>.
jamesfredley commented on code in PR #16631:
URL: https://github.com/apache/grails-core/pull/16631#discussion_r4396560865
##########
grails-data-hibernate7/core/src/main/groovy/org/apache/grails/data/hibernate7/core/querytail/QueryTailConnectionProvider.groovy:
##########
@@ -0,0 +1,127 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * https://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.grails.data.hibernate7.core.querytail
+
+import java.lang.reflect.InvocationHandler
+import java.lang.reflect.InvocationTargetException
+import java.lang.reflect.Method
+import java.lang.reflect.Proxy
+import java.sql.Connection
+
+import groovy.transform.CompileStatic
+import org.hibernate.boot.registry.StandardServiceInitiator
+import org.hibernate.boot.registry.StandardServiceRegistryBuilder
+import org.hibernate.engine.jdbc.connections.internal.ConnectionProviderInitiator
+import org.hibernate.engine.jdbc.connections.internal.MultiTenantConnectionProviderInitiator
+import org.hibernate.service.Service
+import org.hibernate.service.spi.Configurable
+import org.hibernate.service.spi.ServiceBinding
+import org.hibernate.service.spi.ServiceRegistryAwareService
+import org.hibernate.service.spi.ServiceRegistryImplementor
+import org.hibernate.service.spi.Startable
+import org.hibernate.service.spi.Stoppable
+
+import org.apache.grails.datamapping.core.querytail.QueryTailJdbcConnection
+import org.apache.grails.datamapping.core.querytail.QueryTailRecorder
+
+/**
+ * Decorates the provider Hibernate selects, preserving its configuration and service lifecycle.
+ * Connections are captured at execution and returned to their provider as the original instances.
+ *
+ * @since 9.0
+ */
+@CompileStatic
+final class QueryTailConnectionProvider<S extends Service> implements StandardServiceInitiator<S> {
+
+ private final StandardServiceInitiator<S> delegate
+ private final QueryTailRecorder recorder
+
+ private QueryTailConnectionProvider(StandardServiceInitiator<S> delegate, QueryTailRecorder recorder) {
+ this.delegate = delegate
+ this.recorder = recorder
+ }
+
+ static void install(StandardServiceRegistryBuilder builder, QueryTailRecorder recorder) {
Review Comment:
install adds initiators for Hibernate's default connection provider and the multi-tenant provider. Those replace a caller-supplied initiator for the same role, so that provider is never used. The Hibernate 5 copy does the same. Keep an initiator that is already registered, and cover a custom initiator rather than only hibernate.connection.provider_class.
##########
grails-data-mongodb/core/src/main/groovy/org/apache/grails/data/mongodb/core/querytail/MongoCommandRedactor.groovy:
##########
@@ -0,0 +1,548 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * https://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.grails.data.mongodb.core.querytail
+
+import groovy.transform.CompileStatic
+import org.bson.BsonDocument
+import org.bson.BsonValue
+
+import org.apache.grails.datamapping.core.querytail.QueryTailRedactor
+
+/**
+ * Takes the values out of a MongoDB command before it is captured. A command is a document, so
+ * there is nothing to parse: every value becomes {@link QueryTailRedactor#PLACEHOLDER}, and the
+ * keys, which are the options of the command, its operators and the field names of the collection,
+ * are kept, except where a key may be data. The keys of a document stored in a map property, or in
+ * a property or a collection the {@linkplain MongoMappedFields mapping} does not know, are recorded
+ * as the placeholder too, and so is every segment of a dotted path past such a key. So is a key at
+ * the top of stored data, the documents an insert writes, a replacement document, the fields an
+ * update writes and the operand of a literal, unless the mapping knows it, as the name of a dynamic
+ * attribute may be data; the keys at the top of a condition, a pipeline, a sort or a projection
+ * are the fields the application asks for, and are kept. An update that is a pipeline writes the
+ * documents its stages build: the keys at the top of what a {@code $set}, {@code $addFields} or
+ * {@code $project} stage holds, of the new root of a {@code $replaceWith} or {@code $replaceRoot}, and of a
+ * document {@code $mergeObjects} merges into it are fields written, kept only when the mapping knows
+ * them, as those of a classic update are. A numeric key
+ * is kept only where it indexes an array the mapping knows; in a map, in an element of an array,
+ * which is a map or a document unless the field is a list of lists, or at the top of a document,
+ * it is data. A {@code $} key is kept as an operator only where operators go, in a condition, a
+ * pipeline and the operators of an update; in stored data, the documents an insert writes, a
+ * replacement document and the values an update operator writes, it is a field name, which MongoDB
+ * allows to start with {@code $}, and is recorded as the placeholder unless the mapping knows it.
+ * So is a {@code $} key in the operand of a comparison in a condition, such as {@code $eq} or
+ * {@code $in}, or an implicit document or array equality, which holds the value the field is compared
+ * with, and in a DBRef, whose {@code $ref}, {@code $id} and {@code $db} a condition reads as fields
+ * compared with their whole value, in any order; in an aggregation expression, which
+ * {@code $expr}, the {@code let} of a command and the stages that compute fields hold, the operands
+ * of a comparison are expressions, and their operators are kept, except in the operand of a
+ * {@code $literal} or {@code $const}, which is stored data. A comment, and the constants of an
+ * update, and the scope of a map-reduce command, may be any document, and none of their keys are
+ * kept. The entries the driver adds for its
+ * session, the cluster time and the transaction, which say nothing about the command, are left out.
+ *
+ * <p>An array is recorded with each of its elements once: an element recorded as one before it in
+ * the array is left out, and the array then ends with {@code ...}, so that the documents of a batch
+ * and the values of an {@code $in} are recorded the same whatever their number, and a command is
+ * counted as the one statement it is: {@code {"$in": [?, ...]}}.</p>
+ *
+ * <p>A command is recorded as its name, the database and collection it runs against, and the
+ * document of its remaining entries: {@code find shop.book {"filter": {"title": ?}, "limit": ?}}.
+ * The collection is taken only from a command that runs against one; the first value of any other
+ * command is an argument, such as a user name, and is recorded as a value:
+ * {@code dropUser admin {"dropUser": ?}}.</p>
+ *
+ * @since 9.0
+ */
+@CompileStatic
+final class MongoCommandRedactor {
+
+ private static final String PLACEHOLDER = QueryTailRedactor.PLACEHOLDER
+
+ /** Ends an array that held an element recorded as one before it, which is left out. */
+ private static final String REPEATED = '...'
+
+ private static final String DATABASE = '$db'
+
+ private static final String COLLECTION = 'collection'
+
+ /**
+ * The commands whose first value is the collection they run against. The first value of any other
+ * command, such as the user {@code dropUser} drops, is an argument, and is recorded as a value.
+ */
+ private static final Set<String> COLLECTION_COMMANDS = ['find', 'insert', 'update', 'delete', 'aggregate',
+ 'count', 'distinct', 'findAndModify', 'findandmodify', 'mapReduce', 'mapreduce', 'killCursors',
+ 'create', 'drop', 'collMod', 'createIndexes', 'dropIndexes', 'deleteIndexes', 'listIndexes', 'reIndex',
+ 'compact', 'validate', 'collStats', 'convertToCapped', 'cloneCollectionAsCapped', 'analyze',
+ 'checkMetadataConsistency', 'createSearchIndexes', 'dropSearchIndex', 'updateSearchIndex',
+ 'planCacheClear', 'planCacheClearFilters', 'planCacheListFilters', 'planCacheSetFilter'] as Set<String>
+
+ /** The command that names its collection in its {@code collection} option, as it continues a cursor. */
+ private static final String GET_MORE = 'getMore'
+
+ /** The entries the driver adds, which describe its session and transaction rather than the command. */
+ private static final Set<String> LEFT_OUT = ['lsid', '$clusterTime', 'txnNumber', 'autocommit',
+ 'startTransaction', '$readPreference', 'apiVersion', 'apiStrict', 'apiDeprecationErrors'] as Set<String>
+
+ /**
+ * The options whose document, or documents, are keyed by the fields of the collection, wherever they appear,
+ * including those of the operations of a client-level {@code bulkWrite}.
+ */
+ private static final Set<String> FIELD_KEYED = ['filter', 'query', 'q', 'u', 'update', 'document', 'documents',
+ 'updateMods', 'pipeline', 'sort', 'projection', 'fields', 'key', 'hint', 'min', 'max', 'arrayFilters',
+ 'partialFilterExpression', 'wildcardProjection', 'validator', 'weights'] as Set<String>
+
+ /** Documents that match fields, whose field values may be implicit equality operands. */
+ private static final Set<String> CONDITION_KEYED = ['filter', 'query', 'q', 'arrayFilters',
+ 'partialFilterExpression', 'validator', 'restrictSearchWithMatch'] as Set<String>
+
+ /** The options whose documents are stored data: what an insert writes. */
+ private static final Set<String> DATA_KEYED = ['documents', 'document'] as Set<String>
+
+ /** The options and the operator that may hold any document, none of whose keys say what the command does. */
+ private static final Set<String> ANY_VALUE = ['comment', '$comment', 'c', 'scope'] as Set<String>
+
+ /** The option whose values are aggregation expressions. */
+ private static final String LET = 'let'
+
+ /** The options that hold an update: update operators, a replacement document or a pipeline. */
+ private static final Set<String> UPDATE_KEYED = ['u', 'update', 'updateMods'] as Set<String>
+
+ /** The update operators whose fields take a condition rather than the value to write. */
+ private static final Set<String> CONDITION_OPERATORS = ['$pull', '$currentDate'] as Set<String>
+
+ /** The update operators that add to an array, whose value may hold modifiers. */
+ private static final Set<String> PUSH_OPERATORS = ['$push', '$addToSet'] as Set<String>
+
+ private static final String EACH = '$each'
+
+ /** The modifiers of a {@code $push} or an {@code $addToSet}. */
+ private static final Set<String> PUSH_MODIFIERS = [EACH, '$slice', '$sort', '$position'] as Set<String>
+
+ /**
+ * The operators of a pipeline whose operand is literal data rather than an expression, including
+ * {@code $const}, which MongoDB accepts for {@code $literal}.
+ */
+ private static final Set<String> LITERAL_OPERATORS = ['$literal', '$const', '$documents'] as Set<String>
+
+ /**
+ * The fields of a DBRef, which a condition reads as fields compared with their whole value, never as
+ * operators, in any order and wherever a condition has fields.
+ */
+ private static final Set<String> DBREF_FIELDS = ['$ref', '$id', '$db'] as Set<String>
+
+ /**
+ * The query operators whose operand holds values rather than a condition: the comparisons, whose operand
+ * is the value a field is compared with, a JSON schema, whose {@code enum} lists values, and a
+ * GeoJSON geometry, which can carry additional data alongside its type and coordinates.
+ */
+ private static final Set<String> VALUE_OPERATORS = ['$eq', '$ne', '$gt', '$gte', '$lt', '$lte', '$in', '$nin',
+ '$all', '$jsonSchema', '$geometry'] as Set<String>
+
+ /** The operator of a condition, and the stages of a pipeline, whose operand is an aggregation expression. */
+ private static final Set<String> EXPRESSION_OPERATORS = ['$expr', '$addFields', '$set', '$project', '$group',
+ '$replaceRoot', '$replaceWith', '$redact', '$bucket', '$bucketAuto', '$sortByCount', '$setWindowFields',
+ '$fill'] as Set<String>
+
+ /** The stages of an update pipeline whose operand is the document they write, or the new root that replaces it. */
+ private static final Set<String> WRITING_STAGES = ['$set', '$addFields', '$project', '$replaceWith'] as Set<String>
+
+ /** The stage of an update pipeline whose options name the new root that replaces the document. */
+ private static final String REPLACE_ROOT = '$replaceRoot'
+
+ /** The option of {@code $replaceRoot} that holds the new root. */
+ private static final String NEW_ROOT = 'newRoot'
+
+ /** The operator whose operands are documents merged into one, which in an update pipeline is written. */
+ private static final String MERGE_OBJECTS = '$mergeObjects'
+
+ /** The kinds of document whose keys at the top are stored data: what an insert, a replacement or an update writes. */
+ private static final Set<Kind> STORED = EnumSet.of(Kind.DATA, Kind.UPDATE, Kind.UPDATE_FIELDS, Kind.WRITTEN)
+
+ private MongoCommandRedactor() {
+ }
+
+ /**
+ * Redacts a command.
+ *
+ * @param database the database the command runs against
+ * @param commandName the name of the command, which is its first key
+ * @param command the command as the driver sends it
+ * @param fields the fields the mapping of the datastore knows, or null before the datastore is built
+ * @return the command with its values replaced
+ */
+ static String redact(String database, String commandName, BsonDocument command, MongoMappedFields fields) {
+ String collection = null
+ String collectionKey = null
+ BsonValue first = command.get(commandName)
+ if (first != null && first.isString() && COLLECTION_COMMANDS.contains(commandName)) {
+ collection = first.asString().value
+ collectionKey = commandName
+ }
+ else if (commandName == GET_MORE && command.get(COLLECTION)?.isString()) {
+ collection = command.getString(COLLECTION).value
+ collectionKey = COLLECTION
+ }
+ StringBuilder out = new StringBuilder(commandName)
+ if (database != null || collection != null) {
+ out.append(' ')
+ if (database != null) {
+ out.append(database)
+ }
+ if (collection != null) {
+ out.append(database != null ? '.' : '').append(collection)
+ }
+ }
+ Scope options = Scope.options(collection, fields)
+ StringBuilder body = new StringBuilder()
+ for (Map.Entry<String, BsonValue> entry in command.entrySet()) {
+ String key = entry.key
+ if (key == DATABASE || key == collectionKey || LEFT_OUT.contains(key)) {
+ continue
+ }
+ body.append(body.length() == 0 ? '{' : ', ')
+ appendEntry(body, key, entry.value, options)
+ }
+ if (body.length() > 0) {
+ out.append(' ').append(body).append('}')
+ }
+ out.toString()
+ }
+
+ private static void appendEntry(StringBuilder out, String key, BsonValue value, Scope scope) {
+ Scope child = scope.of(key, value)
+ out.append(child.keyText).append(': ')
+ appendValue(out, value, child)
+ }
+
+ private static void appendValue(StringBuilder out, BsonValue value, Scope scope) {
+ if (value.isDocument()) {
+ out.append('{')
+ boolean first = true
+ for (Map.Entry<String, BsonValue> entry in value.asDocument().entrySet()) {
+ if (!first) {
+ out.append(', ')
+ }
+ first = false
+ appendEntry(out, entry.key, entry.value, scope)
+ }
+ out.append('}')
+ }
+ else if (value.isArray()) {
+ out.append('[')
+ Scope element = scope.element()
+ Set<String> written = new HashSet<>()
+ boolean repeated = false
+ for (BsonValue item in value.asArray()) {
+ StringBuilder text = new StringBuilder()
+ appendValue(text, item, element)
+ String recorded = text.toString()
+ if (written.add(recorded)) {
+ if (written.size() > 1) {
+ out.append(', ')
+ }
+ out.append(recorded)
+ }
+ else {
+ repeated = true
+ }
+ }
+ if (repeated) {
+ out.append(', ').append(REPEATED)
+ }
+ out.append(']')
+ }
+ else {
+ out.append(PLACEHOLDER)
+ }
+ }
+
+ private static String quote(String key) {
+ StringBuilder out = new StringBuilder(key.length() + 2).append('"')
+ for (int i = 0; i < key.length(); i++) {
+ char c = key.charAt(i)
+ if (c == '"' || c == '\\') {
+ out.append('\\').append(c)
+ }
+ else if (c < 0x20) {
+ out.append(String.format('\\u%04x', (int) c))
+ }
+ else {
+ out.append(c)
+ }
+ }
+ out.append('"').toString()
+ }
+
+ /**
+ * Where a document sits in the command, which decides what its keys are: the options of the
+ * command, kept; the fields and operators of a condition, a pipeline, a sort or a projection, where
+ * a {@code $} key is an operator; an aggregation expression, whose {@code $} keys are operators
+ * too; an update, whose {@code $} keys are update operators; or stored data, the documents an
+ * insert writes, the values an update writes and the values a condition compares with, where a
+ * {@code $} key is a field name like any other and is kept only when the mapping knows it.
+ */
+ @CompileStatic
+ private static final class Scope {
+
+ final String collection
+ final MongoMappedFields fields
+ final Kind kind
+ /**
+ * The update operator whose fields a document of kind {@link Kind#UPDATE_FIELDS} holds, or the stage
+ * whose document one of kind {@link Kind#WRITTEN} is.
+ */
+ final String operator
+ /** The field path of the document; null in a document keyed by fields the mapping does not know. */
+ final List<String> path
+ /** How many indices into the field at the path have been taken: 0 at the field, 1 in one of its elements. */
+ final int depth
+ final String keyText
+
+ private Scope(String collection, MongoMappedFields fields, Kind kind, String operator, List<String> path,
+ int depth, String keyText) {
+ this.collection = collection
+ this.fields = fields
+ this.kind = kind
+ this.operator = operator
+ this.path = path
+ this.depth = depth
+ this.keyText = keyText
+ }
+
+ static Scope options(String collection, MongoMappedFields fields) {
+ new Scope(collection, fields, Kind.OPTIONS, null, Collections.<String>emptyList(), 0, null)
+ }
+
+ /**
+ * @return the scope of an element of an array in this scope: one index deeper into the field
+ * at the path, so that a numeric key inside the element is data unless the elements are arrays;
+ * the elements of an update that is an array are the stages of a pipeline, and an element of an
+ * expression at the top of a document, as the operands of an operator {@code $expr} holds are, is a
+ * value the expression builds, whose keys are kept only when the mapping knows them, as those of a
+ * document a stage of an update pipeline writes are
+ */
+ Scope element() {
+ if (kind == Kind.UPDATE) {
+ return child(Kind.UPDATE_PIPELINE, null, path, depth, keyText)
+ }
+ if (kind == Kind.PUSH) {
+ return child(Kind.DATA, null, path, depth + 2, keyText)
+ }
+ if ((kind == Kind.QUERY || kind == Kind.CONDITION || kind == Kind.EXPRESSION || kind == Kind.DATA) && path != null && !path.isEmpty()) {
+ return child(kind, null, path, depth + 1, keyText)
+ }
+ if (kind == Kind.EXPRESSION && path != null) {
+ return child(Kind.WRITTEN, null, path, depth, keyText)
+ }
+ this
+ }
+
+ /**
+ * @return the scope of the value of a key in a document of this scope, with the key as it is recorded
+ */
+ Scope of(String key, BsonValue value) {
+ if (kind == Kind.OPTIONS) {
+ if (ANY_VALUE.contains(key)) {
+ return child(Kind.DATA, null, null, 0, quote(key))
+ }
+ Kind valueKind = DATA_KEYED.contains(key) ? Kind.DATA :
+ UPDATE_KEYED.contains(key) ? Kind.UPDATE :
+ CONDITION_KEYED.contains(key) ? Kind.CONDITION :
+ FIELD_KEYED.contains(key) ? Kind.QUERY :
+ key == LET ? Kind.EXPRESSION : Kind.OPTIONS
+ return child(valueKind, null, Collections.<String>emptyList(), 0, quote(key))
+ }
+ if ((kind == Kind.QUERY || kind == Kind.CONDITION) && key.startsWith('$')) {
Review Comment:
A key such as $example-secret inside $near is quoted as an operator and kept in the capture. The listener records the command before the server rejects it. Redact a dollar-prefixed key here unless it is an operator, and cover that key directly rather than only a secret under $geometry.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@grails.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add the query-tail command to find query-heavy pages [grails-core]
Posted by "jdaugherty (via GitHub)" <gi...@apache.org>.
jdaugherty commented on code in PR #16631:
URL: https://github.com/apache/grails-core/pull/16631#discussion_r4396907451
##########
grails-data-hibernate7/core/src/main/groovy/org/apache/grails/data/hibernate7/core/querytail/QueryTailConnectionProvider.groovy:
##########
@@ -0,0 +1,127 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * https://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.grails.data.hibernate7.core.querytail
+
+import java.lang.reflect.InvocationHandler
+import java.lang.reflect.InvocationTargetException
+import java.lang.reflect.Method
+import java.lang.reflect.Proxy
+import java.sql.Connection
+
+import groovy.transform.CompileStatic
+import org.hibernate.boot.registry.StandardServiceInitiator
+import org.hibernate.boot.registry.StandardServiceRegistryBuilder
+import org.hibernate.engine.jdbc.connections.internal.ConnectionProviderInitiator
+import org.hibernate.engine.jdbc.connections.internal.MultiTenantConnectionProviderInitiator
+import org.hibernate.service.Service
+import org.hibernate.service.spi.Configurable
+import org.hibernate.service.spi.ServiceBinding
+import org.hibernate.service.spi.ServiceRegistryAwareService
+import org.hibernate.service.spi.ServiceRegistryImplementor
+import org.hibernate.service.spi.Startable
+import org.hibernate.service.spi.Stoppable
+
+import org.apache.grails.datamapping.core.querytail.QueryTailJdbcConnection
+import org.apache.grails.datamapping.core.querytail.QueryTailRecorder
+
+/**
+ * Decorates the provider Hibernate selects, preserving its configuration and service lifecycle.
+ * Connections are captured at execution and returned to their provider as the original instances.
+ *
+ * @since 9.0
+ */
+@CompileStatic
+final class QueryTailConnectionProvider<S extends Service> implements StandardServiceInitiator<S> {
+
+ private final StandardServiceInitiator<S> delegate
+ private final QueryTailRecorder recorder
+
+ private QueryTailConnectionProvider(StandardServiceInitiator<S> delegate, QueryTailRecorder recorder) {
+ this.delegate = delegate
+ this.recorder = recorder
+ }
+
+ static void install(StandardServiceRegistryBuilder builder, QueryTailRecorder recorder) {
Review Comment:
Fixed in 8976ddec98, for Hibernate 5 and 7. The capture no longer adds initiators to the service registry the session factory is built with. It decorates the connection providers in the session factory's own service registry instead, through a `SessionFactoryServiceContributor`, which is where sessions take their connections from. Whatever provider the main registry has is kept and wrapped: Hibernate's default, one from a custom initiator, or one registered with `addService`. That registry still configures, starts and stops it.
`QueryTailSessionFactoryObserverSpec` now covers a configuration class that adds its own initiator: its provider hands out the connections, and their statements are recorded. The spec also checks that a configured provider is stopped exactly once.
##########
grails-data-mongodb/core/src/main/groovy/org/apache/grails/data/mongodb/core/querytail/MongoCommandRedactor.groovy:
##########
@@ -0,0 +1,548 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * https://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.grails.data.mongodb.core.querytail
+
+import groovy.transform.CompileStatic
+import org.bson.BsonDocument
+import org.bson.BsonValue
+
+import org.apache.grails.datamapping.core.querytail.QueryTailRedactor
+
+/**
+ * Takes the values out of a MongoDB command before it is captured. A command is a document, so
+ * there is nothing to parse: every value becomes {@link QueryTailRedactor#PLACEHOLDER}, and the
+ * keys, which are the options of the command, its operators and the field names of the collection,
+ * are kept, except where a key may be data. The keys of a document stored in a map property, or in
+ * a property or a collection the {@linkplain MongoMappedFields mapping} does not know, are recorded
+ * as the placeholder too, and so is every segment of a dotted path past such a key. So is a key at
+ * the top of stored data, the documents an insert writes, a replacement document, the fields an
+ * update writes and the operand of a literal, unless the mapping knows it, as the name of a dynamic
+ * attribute may be data; the keys at the top of a condition, a pipeline, a sort or a projection
+ * are the fields the application asks for, and are kept. An update that is a pipeline writes the
+ * documents its stages build: the keys at the top of what a {@code $set}, {@code $addFields} or
+ * {@code $project} stage holds, of the new root of a {@code $replaceWith} or {@code $replaceRoot}, and of a
+ * document {@code $mergeObjects} merges into it are fields written, kept only when the mapping knows
+ * them, as those of a classic update are. A numeric key
+ * is kept only where it indexes an array the mapping knows; in a map, in an element of an array,
+ * which is a map or a document unless the field is a list of lists, or at the top of a document,
+ * it is data. A {@code $} key is kept as an operator only where operators go, in a condition, a
+ * pipeline and the operators of an update; in stored data, the documents an insert writes, a
+ * replacement document and the values an update operator writes, it is a field name, which MongoDB
+ * allows to start with {@code $}, and is recorded as the placeholder unless the mapping knows it.
+ * So is a {@code $} key in the operand of a comparison in a condition, such as {@code $eq} or
+ * {@code $in}, or an implicit document or array equality, which holds the value the field is compared
+ * with, and in a DBRef, whose {@code $ref}, {@code $id} and {@code $db} a condition reads as fields
+ * compared with their whole value, in any order; in an aggregation expression, which
+ * {@code $expr}, the {@code let} of a command and the stages that compute fields hold, the operands
+ * of a comparison are expressions, and their operators are kept, except in the operand of a
+ * {@code $literal} or {@code $const}, which is stored data. A comment, and the constants of an
+ * update, and the scope of a map-reduce command, may be any document, and none of their keys are
+ * kept. The entries the driver adds for its
+ * session, the cluster time and the transaction, which say nothing about the command, are left out.
+ *
+ * <p>An array is recorded with each of its elements once: an element recorded as one before it in
+ * the array is left out, and the array then ends with {@code ...}, so that the documents of a batch
+ * and the values of an {@code $in} are recorded the same whatever their number, and a command is
+ * counted as the one statement it is: {@code {"$in": [?, ...]}}.</p>
+ *
+ * <p>A command is recorded as its name, the database and collection it runs against, and the
+ * document of its remaining entries: {@code find shop.book {"filter": {"title": ?}, "limit": ?}}.
+ * The collection is taken only from a command that runs against one; the first value of any other
+ * command is an argument, such as a user name, and is recorded as a value:
+ * {@code dropUser admin {"dropUser": ?}}.</p>
+ *
+ * @since 9.0
+ */
+@CompileStatic
+final class MongoCommandRedactor {
+
+ private static final String PLACEHOLDER = QueryTailRedactor.PLACEHOLDER
+
+ /** Ends an array that held an element recorded as one before it, which is left out. */
+ private static final String REPEATED = '...'
+
+ private static final String DATABASE = '$db'
+
+ private static final String COLLECTION = 'collection'
+
+ /**
+ * The commands whose first value is the collection they run against. The first value of any other
+ * command, such as the user {@code dropUser} drops, is an argument, and is recorded as a value.
+ */
+ private static final Set<String> COLLECTION_COMMANDS = ['find', 'insert', 'update', 'delete', 'aggregate',
+ 'count', 'distinct', 'findAndModify', 'findandmodify', 'mapReduce', 'mapreduce', 'killCursors',
+ 'create', 'drop', 'collMod', 'createIndexes', 'dropIndexes', 'deleteIndexes', 'listIndexes', 'reIndex',
+ 'compact', 'validate', 'collStats', 'convertToCapped', 'cloneCollectionAsCapped', 'analyze',
+ 'checkMetadataConsistency', 'createSearchIndexes', 'dropSearchIndex', 'updateSearchIndex',
+ 'planCacheClear', 'planCacheClearFilters', 'planCacheListFilters', 'planCacheSetFilter'] as Set<String>
+
+ /** The command that names its collection in its {@code collection} option, as it continues a cursor. */
+ private static final String GET_MORE = 'getMore'
+
+ /** The entries the driver adds, which describe its session and transaction rather than the command. */
+ private static final Set<String> LEFT_OUT = ['lsid', '$clusterTime', 'txnNumber', 'autocommit',
+ 'startTransaction', '$readPreference', 'apiVersion', 'apiStrict', 'apiDeprecationErrors'] as Set<String>
+
+ /**
+ * The options whose document, or documents, are keyed by the fields of the collection, wherever they appear,
+ * including those of the operations of a client-level {@code bulkWrite}.
+ */
+ private static final Set<String> FIELD_KEYED = ['filter', 'query', 'q', 'u', 'update', 'document', 'documents',
+ 'updateMods', 'pipeline', 'sort', 'projection', 'fields', 'key', 'hint', 'min', 'max', 'arrayFilters',
+ 'partialFilterExpression', 'wildcardProjection', 'validator', 'weights'] as Set<String>
+
+ /** Documents that match fields, whose field values may be implicit equality operands. */
+ private static final Set<String> CONDITION_KEYED = ['filter', 'query', 'q', 'arrayFilters',
+ 'partialFilterExpression', 'validator', 'restrictSearchWithMatch'] as Set<String>
+
+ /** The options whose documents are stored data: what an insert writes. */
+ private static final Set<String> DATA_KEYED = ['documents', 'document'] as Set<String>
+
+ /** The options and the operator that may hold any document, none of whose keys say what the command does. */
+ private static final Set<String> ANY_VALUE = ['comment', '$comment', 'c', 'scope'] as Set<String>
+
+ /** The option whose values are aggregation expressions. */
+ private static final String LET = 'let'
+
+ /** The options that hold an update: update operators, a replacement document or a pipeline. */
+ private static final Set<String> UPDATE_KEYED = ['u', 'update', 'updateMods'] as Set<String>
+
+ /** The update operators whose fields take a condition rather than the value to write. */
+ private static final Set<String> CONDITION_OPERATORS = ['$pull', '$currentDate'] as Set<String>
+
+ /** The update operators that add to an array, whose value may hold modifiers. */
+ private static final Set<String> PUSH_OPERATORS = ['$push', '$addToSet'] as Set<String>
+
+ private static final String EACH = '$each'
+
+ /** The modifiers of a {@code $push} or an {@code $addToSet}. */
+ private static final Set<String> PUSH_MODIFIERS = [EACH, '$slice', '$sort', '$position'] as Set<String>
+
+ /**
+ * The operators of a pipeline whose operand is literal data rather than an expression, including
+ * {@code $const}, which MongoDB accepts for {@code $literal}.
+ */
+ private static final Set<String> LITERAL_OPERATORS = ['$literal', '$const', '$documents'] as Set<String>
+
+ /**
+ * The fields of a DBRef, which a condition reads as fields compared with their whole value, never as
+ * operators, in any order and wherever a condition has fields.
+ */
+ private static final Set<String> DBREF_FIELDS = ['$ref', '$id', '$db'] as Set<String>
+
+ /**
+ * The query operators whose operand holds values rather than a condition: the comparisons, whose operand
+ * is the value a field is compared with, a JSON schema, whose {@code enum} lists values, and a
+ * GeoJSON geometry, which can carry additional data alongside its type and coordinates.
+ */
+ private static final Set<String> VALUE_OPERATORS = ['$eq', '$ne', '$gt', '$gte', '$lt', '$lte', '$in', '$nin',
+ '$all', '$jsonSchema', '$geometry'] as Set<String>
+
+ /** The operator of a condition, and the stages of a pipeline, whose operand is an aggregation expression. */
+ private static final Set<String> EXPRESSION_OPERATORS = ['$expr', '$addFields', '$set', '$project', '$group',
+ '$replaceRoot', '$replaceWith', '$redact', '$bucket', '$bucketAuto', '$sortByCount', '$setWindowFields',
+ '$fill'] as Set<String>
+
+ /** The stages of an update pipeline whose operand is the document they write, or the new root that replaces it. */
+ private static final Set<String> WRITING_STAGES = ['$set', '$addFields', '$project', '$replaceWith'] as Set<String>
+
+ /** The stage of an update pipeline whose options name the new root that replaces the document. */
+ private static final String REPLACE_ROOT = '$replaceRoot'
+
+ /** The option of {@code $replaceRoot} that holds the new root. */
+ private static final String NEW_ROOT = 'newRoot'
+
+ /** The operator whose operands are documents merged into one, which in an update pipeline is written. */
+ private static final String MERGE_OBJECTS = '$mergeObjects'
+
+ /** The kinds of document whose keys at the top are stored data: what an insert, a replacement or an update writes. */
+ private static final Set<Kind> STORED = EnumSet.of(Kind.DATA, Kind.UPDATE, Kind.UPDATE_FIELDS, Kind.WRITTEN)
+
+ private MongoCommandRedactor() {
+ }
+
+ /**
+ * Redacts a command.
+ *
+ * @param database the database the command runs against
+ * @param commandName the name of the command, which is its first key
+ * @param command the command as the driver sends it
+ * @param fields the fields the mapping of the datastore knows, or null before the datastore is built
+ * @return the command with its values replaced
+ */
+ static String redact(String database, String commandName, BsonDocument command, MongoMappedFields fields) {
+ String collection = null
+ String collectionKey = null
+ BsonValue first = command.get(commandName)
+ if (first != null && first.isString() && COLLECTION_COMMANDS.contains(commandName)) {
+ collection = first.asString().value
+ collectionKey = commandName
+ }
+ else if (commandName == GET_MORE && command.get(COLLECTION)?.isString()) {
+ collection = command.getString(COLLECTION).value
+ collectionKey = COLLECTION
+ }
+ StringBuilder out = new StringBuilder(commandName)
+ if (database != null || collection != null) {
+ out.append(' ')
+ if (database != null) {
+ out.append(database)
+ }
+ if (collection != null) {
+ out.append(database != null ? '.' : '').append(collection)
+ }
+ }
+ Scope options = Scope.options(collection, fields)
+ StringBuilder body = new StringBuilder()
+ for (Map.Entry<String, BsonValue> entry in command.entrySet()) {
+ String key = entry.key
+ if (key == DATABASE || key == collectionKey || LEFT_OUT.contains(key)) {
+ continue
+ }
+ body.append(body.length() == 0 ? '{' : ', ')
+ appendEntry(body, key, entry.value, options)
+ }
+ if (body.length() > 0) {
+ out.append(' ').append(body).append('}')
+ }
+ out.toString()
+ }
+
+ private static void appendEntry(StringBuilder out, String key, BsonValue value, Scope scope) {
+ Scope child = scope.of(key, value)
+ out.append(child.keyText).append(': ')
+ appendValue(out, value, child)
+ }
+
+ private static void appendValue(StringBuilder out, BsonValue value, Scope scope) {
+ if (value.isDocument()) {
+ out.append('{')
+ boolean first = true
+ for (Map.Entry<String, BsonValue> entry in value.asDocument().entrySet()) {
+ if (!first) {
+ out.append(', ')
+ }
+ first = false
+ appendEntry(out, entry.key, entry.value, scope)
+ }
+ out.append('}')
+ }
+ else if (value.isArray()) {
+ out.append('[')
+ Scope element = scope.element()
+ Set<String> written = new HashSet<>()
+ boolean repeated = false
+ for (BsonValue item in value.asArray()) {
+ StringBuilder text = new StringBuilder()
+ appendValue(text, item, element)
+ String recorded = text.toString()
+ if (written.add(recorded)) {
+ if (written.size() > 1) {
+ out.append(', ')
+ }
+ out.append(recorded)
+ }
+ else {
+ repeated = true
+ }
+ }
+ if (repeated) {
+ out.append(', ').append(REPEATED)
+ }
+ out.append(']')
+ }
+ else {
+ out.append(PLACEHOLDER)
+ }
+ }
+
+ private static String quote(String key) {
+ StringBuilder out = new StringBuilder(key.length() + 2).append('"')
+ for (int i = 0; i < key.length(); i++) {
+ char c = key.charAt(i)
+ if (c == '"' || c == '\\') {
+ out.append('\\').append(c)
+ }
+ else if (c < 0x20) {
+ out.append(String.format('\\u%04x', (int) c))
+ }
+ else {
+ out.append(c)
+ }
+ }
+ out.append('"').toString()
+ }
+
+ /**
+ * Where a document sits in the command, which decides what its keys are: the options of the
+ * command, kept; the fields and operators of a condition, a pipeline, a sort or a projection, where
+ * a {@code $} key is an operator; an aggregation expression, whose {@code $} keys are operators
+ * too; an update, whose {@code $} keys are update operators; or stored data, the documents an
+ * insert writes, the values an update writes and the values a condition compares with, where a
+ * {@code $} key is a field name like any other and is kept only when the mapping knows it.
+ */
+ @CompileStatic
+ private static final class Scope {
+
+ final String collection
+ final MongoMappedFields fields
+ final Kind kind
+ /**
+ * The update operator whose fields a document of kind {@link Kind#UPDATE_FIELDS} holds, or the stage
+ * whose document one of kind {@link Kind#WRITTEN} is.
+ */
+ final String operator
+ /** The field path of the document; null in a document keyed by fields the mapping does not know. */
+ final List<String> path
+ /** How many indices into the field at the path have been taken: 0 at the field, 1 in one of its elements. */
+ final int depth
+ final String keyText
+
+ private Scope(String collection, MongoMappedFields fields, Kind kind, String operator, List<String> path,
+ int depth, String keyText) {
+ this.collection = collection
+ this.fields = fields
+ this.kind = kind
+ this.operator = operator
+ this.path = path
+ this.depth = depth
+ this.keyText = keyText
+ }
+
+ static Scope options(String collection, MongoMappedFields fields) {
+ new Scope(collection, fields, Kind.OPTIONS, null, Collections.<String>emptyList(), 0, null)
+ }
+
+ /**
+ * @return the scope of an element of an array in this scope: one index deeper into the field
+ * at the path, so that a numeric key inside the element is data unless the elements are arrays;
+ * the elements of an update that is an array are the stages of a pipeline, and an element of an
+ * expression at the top of a document, as the operands of an operator {@code $expr} holds are, is a
+ * value the expression builds, whose keys are kept only when the mapping knows them, as those of a
+ * document a stage of an update pipeline writes are
+ */
+ Scope element() {
+ if (kind == Kind.UPDATE) {
+ return child(Kind.UPDATE_PIPELINE, null, path, depth, keyText)
+ }
+ if (kind == Kind.PUSH) {
+ return child(Kind.DATA, null, path, depth + 2, keyText)
+ }
+ if ((kind == Kind.QUERY || kind == Kind.CONDITION || kind == Kind.EXPRESSION || kind == Kind.DATA) && path != null && !path.isEmpty()) {
+ return child(kind, null, path, depth + 1, keyText)
+ }
+ if (kind == Kind.EXPRESSION && path != null) {
+ return child(Kind.WRITTEN, null, path, depth, keyText)
+ }
+ this
+ }
+
+ /**
+ * @return the scope of the value of a key in a document of this scope, with the key as it is recorded
+ */
+ Scope of(String key, BsonValue value) {
+ if (kind == Kind.OPTIONS) {
+ if (ANY_VALUE.contains(key)) {
+ return child(Kind.DATA, null, null, 0, quote(key))
+ }
+ Kind valueKind = DATA_KEYED.contains(key) ? Kind.DATA :
+ UPDATE_KEYED.contains(key) ? Kind.UPDATE :
+ CONDITION_KEYED.contains(key) ? Kind.CONDITION :
+ FIELD_KEYED.contains(key) ? Kind.QUERY :
+ key == LET ? Kind.EXPRESSION : Kind.OPTIONS
+ return child(valueKind, null, Collections.<String>emptyList(), 0, quote(key))
+ }
+ if ((kind == Kind.QUERY || kind == Kind.CONDITION) && key.startsWith('$')) {
Review Comment:
Fixed in 8976ddec98. In a condition, a `$` key is now kept only if it is one of MongoDB's query operators (or a DBRef field). Any other is recorded as `?`, along with everything under it.
`MongoCommandRedactorSpec` covers the key directly inside `$near` and `$nearSphere`, next to field operators, at the top of a condition, inside `$elemMatch` and in a `$match` stage. It also checks that the query operators themselves, including the geospatial, `$text` and `$regex` options, are still kept. The guide says the same.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@grails.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add the query-tail command to find query-heavy pages [grails-core]
Posted by "jdaugherty (via GitHub)" <gi...@apache.org>.
jdaugherty commented on PR #16631:
URL: https://github.com/apache/grails-core/pull/16631#issuecomment-6452077227
@jamesfredley, review 5672694161 is addressed. The head is now fe3a01a69b, still a single commit on 9.0.x. It folds in b8a79742d1, which I pushed earlier today.
- **Commands not aimed at a collection (MongoDB):** a collection is taken only from a command that runs against one, so `{"dropUser": "alice@example.com"}` is recorded as `dropUser admin {"dropUser": ?}`. Details are on the thread.
- **Map-reduce scope (MongoDB):** none of the keys in a `scope` are recorded, at any depth. Details are on the thread.
The same push changes how Hibernate 5 and 7 statements are counted:
- **Counted at execution:** SQL is now recorded when JDBC runs it, not when Hibernate prepares it. Each execution of a reused statement counts, and so does each entry an `executeBatch` submits, so 50 inserts count as 50 with or without JDBC batching. A prepared statement that never runs, or a batch cleared before it runs, counts nothing. Failed executions and statements later rolled back still count. The capture wraps the connections Hibernate's connection provider hands out and gives the provider back its own connection on release. It never reads bound values. The application's statement inspector still runs first, and the statement it returns is the one recorded.
- **Tests:** `QueryTailStatementInspectorSpec`, for Hibernate 5 and 7, counts every inserted, updated and deleted row across JDBC batch sizes, including a session-level override. It counts executions and submitted batches in direct JDBC work, but not preparation or cleared and abandoned entries. It also checks that a configured connection provider keeps its injections, its lifecycle and the identity of its connections.
The guide is updated to match.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@grails.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add the query-tail command to find query-heavy pages [grails-core]
Posted by "testlens-app[bot] (via GitHub)" <gi...@apache.org>.
testlens-app[bot] commented on PR #16631:
URL: https://github.com/apache/grails-core/pull/16631#issuecomment-6464826509
## ✅ All tests passed ✅
🏷️ Commit: f07667bca628cbaf89efc658e11f25ceeaee6672
▶️ Tests: 141559 executed
⚪️ Checks: 97/97 completed
---
_Learn more about TestLens at [testlens.app/docs](https://testlens.app/docs/features/pr-comment/)._
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@grails.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add the query-tail command to find query-heavy pages [grails-core]
Posted by "jdaugherty (via GitHub)" <gi...@apache.org>.
jdaugherty commented on code in PR #16631:
URL: https://github.com/apache/grails-core/pull/16631#discussion_r4383525360
##########
grails-datamapping-core/src/main/groovy/org/apache/grails/data/core/querytail/QueryTailRequestAttribution.groovy:
##########
@@ -0,0 +1,729 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * https://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.grails.data.core.querytail
+
+import java.lang.reflect.Method
+import java.nio.charset.StandardCharsets
+import java.util.concurrent.atomic.AtomicLong
+import java.util.regex.Pattern
+
+import groovy.transform.CompileStatic
+import groovy.transform.PackageScope
+
+import jakarta.servlet.http.HttpServletRequest
+
+import org.springframework.util.ClassUtils
+import org.springframework.web.context.request.RequestAttributes
+import org.springframework.web.context.request.RequestContextHolder
+import org.springframework.web.context.request.ServletRequestAttributes
+
+/**
+ * Attributes a captured statement to the web request on the current thread. Grails keeps the
+ * request, and the controller and action it resolved, on the thread that serves it, so a statement
+ * run by an interceptor, the action or the view all land on the same page. The first statement of
+ * a request gives it an id that every later statement of that request repeats, which is how the
+ * {@code query-tail} command counts statements per request. Each capture file keeps its own request
+ * ids, so a request recorded to another file cannot consume or reuse one of its ids. Work on any other
+ * thread is attributed to the thread by name.
+ *
+ * <p>The request is recorded as its method and path, with the values in the path replaced by
+ * {@code ?}. Once Grails has chosen the controller, the path is read against the URL mapping it
+ * matched: a segment the mapping spells out is kept, as is a variable segment that names the
+ * controller, action or namespace, and every other variable segment is replaced. A request a Spring
+ * MVC handler serves is recorded as the pattern its handler mapping matched, whose variables are
+ * replaced. When the path cannot be read against the mapping, the values of the parameters the URL
+ * mapping took from it and those of its query are replaced, found
+ * whether or not the client percent-encoded them, and any segment that is a number, a UUID or a long
+ * hexadecimal token, so that an id, a token or a key carried in a path never reaches the capture
+ * file. A segment a value is glued into, as a mapping such as {@code /reset/${token}_confirm}
+ * produces, is replaced whole, whether the value shows as sent or decoded. That also holds for a value
+ * spanning several segments, with literal text before its first segment or after its last, and for every path
+ * parameter after a semicolon, such as a {@code jsessionid} the container rewrote the URL with.
+ * Until the URL mapping has run, as for a statement of a
+ * filter that runs before the controller is chosen, or for a request that never reaches one, no
+ * segment's role is known and every segment is replaced; only the controller, or the pattern, the
+ * framework set on the request says that it has run, never a parameter, which a client can send.
+ * A request is described once and its line reused for its later statements, until its path or the
+ * handler it reached changes, as a forward changes them.</p>
+ *
+ * <p>Work a request starts on another thread that carries the request with it, as Grails promises
+ * and {@code @Async} methods do, counts for the request. A statement that runs once the request
+ * has completed is attributed to its thread.</p>
+ *
+ * <p>Spring Web is optional: without it on the classpath every statement is attributed to its
+ * thread.</p>
+ *
+ * @since 9.0
+ */
+@CompileStatic
+class QueryTailRequestAttribution {
+
+ /** The prefix of the request attribute holding the id each attribution gave a request. */
+ public static final String REQUEST_ID_ATTRIBUTE = 'org.apache.grails.data.core.querytail.REQUEST_ID'
+
+ /** The request attribute Grails sets the controller name in. */
+ public static final String CONTROLLER_NAME_ATTRIBUTE = 'org.grails.CONTROLLER_NAME_ATTRIBUTE'
+
+ /** The request attribute Grails sets the controller namespace in. */
+ public static final String CONTROLLER_NAMESPACE_ATTRIBUTE = 'org.grails.CONTROLLER_NAMESPACE_ATTRIBUTE'
+
+ /** The request attribute Grails sets the action name in. */
+ public static final String ACTION_NAME_ATTRIBUTE = 'org.grails.ACTION_NAME_ATTRIBUTE'
+
+ private static final boolean SPRING_WEB_PRESENT = ClassUtils.isPresent(
+ 'org.springframework.web.context.request.RequestContextHolder', QueryTailRequestAttribution.classLoader)
+
+ private final AtomicLong requestCounter
+ private final String requestIdPrefix
+ private final String requestIdAttribute = REQUEST_ID_ATTRIBUTE + '.' + UUID.randomUUID()
+
+ /**
+ * An attribution whose request ids are {@code r1}, {@code r2} and so on.
+ */
+ QueryTailRequestAttribution() {
+ this('')
+ }
+
+ /**
+ * @param requestIdPrefix what every request id starts with, which tells the requests of one JVM
+ * from those of another writing the same capture
+ */
+ QueryTailRequestAttribution(String requestIdPrefix) {
+ this(requestIdPrefix, new AtomicLong())
+ }
+
+ /**
+ * @param requestIdPrefix what every request id starts with
+ * @param requestCounter the counter the requests are numbered by, which attributions that give
+ * their ids the same prefix share, so that none of them repeats an id another gave
+ */
+ @PackageScope
+ QueryTailRequestAttribution(String requestIdPrefix, AtomicLong requestCounter) {
+ this.requestIdPrefix = requestIdPrefix ?: ''
+ this.requestCounter = requestCounter
+ }
+
+ /**
+ * Fills in the request id, page and request of a record from the current thread.
+ *
+ * @param record the record of a statement that just ran on this thread
+ * @return the same record
+ */
+ QueryTailRecord attribute(QueryTailRecord record) {
+ if (SPRING_WEB_PRESENT) {
+ try {
+ if (WebRequestLookup.attribute(record, requestCounter, requestIdPrefix, requestIdAttribute)) {
+ return record
+ }
+ }
+ catch (IllegalStateException ignored) {
+ // the request has completed, as a task it started can outlive it, and is no longer to be read
+ }
+ }
+ record.requestId = null
+ record.request = null
+ record.page = QueryTailRecord.THREAD_PAGE_PREFIX + Thread.currentThread().name
+ record
+ }
+
+ /**
+ * Keeps every reference to Spring Web, in a signature as well as in a body, in a class that is only
+ * loaded once Spring Web is known to be present. The outer class can then be loaded, and reflected
+ * on, without Spring Web on the classpath.
+ */
+ @CompileStatic
+ private static class WebRequestLookup {
+
+ private static final String REQUEST_ID_PREFIX = 'r'
+
+ private static final String PLACEHOLDER = '?'
+
+ /** The parameters a URL mapping sets that name code rather than carry a value. */
+ private static final Set<String> MAPPING_PARAMETERS = ['controller', 'action', 'namespace', 'plugin', 'format'] as Set<String>
+
+ /** A segment that is a number, a UUID or a long hexadecimal token is a value whatever the mapping says. */
+ private static final Pattern VALUE_SHAPED = Pattern.compile(
+ '(?i)[0-9]+|[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}|[0-9a-f]{32,}')
+
+ private static final boolean SERVLET_PRESENT = ClassUtils.isPresent(
+ 'jakarta.servlet.http.HttpServletRequest', WebRequestLookup.classLoader)
+
+ /** The request attribute Grails keeps the URL mapping it matched a request with in. */
+ private static final String MATCHED_MAPPING_ATTRIBUTE = 'org.grails.url.match.info'
+
+ /** The request attribute a Spring MVC handler mapping keeps the pattern it matched a request with in. */
+ private static final String MATCHED_PATTERN_ATTRIBUTE = 'org.springframework.web.servlet.HandlerMapping.bestMatchingPattern'
+
+ /** The request attribute the line a request was described with is kept in, with what it was described from. */
+ private static final String REQUEST_LINE_ATTRIBUTE = 'org.apache.grails.data.core.querytail.REQUEST_LINE'
+
+ /** The {@code getParams()} of a Grails web request, which holds the parameters its URL mapping took, by class. */
+ private static final ClassValue<Method> PARAMETERS = publicMethod('getParams')
+
+ /** The {@code getUrlData()} of the URL mapping Grails matched a request with, by class. */
+ private static final ClassValue<Method> URL_DATA = publicMethod('getUrlData')
+
+ /** The {@code getTokens()} of the data of a URL mapping, by class. */
+ private static final ClassValue<Method> TOKENS = publicMethod('getTokens')
+
+ private static ClassValue<Method> publicMethod(String name) {
+ new ClassValue<Method>() {
+ @Override
+ protected Method computeValue(Class<?> type) {
+ try {
+ return type.getMethod(name)
+ }
+ catch (NoSuchMethodException ignored) {
+ return null
+ }
+ }
+ }
+ }
+
+ /**
+ * Fills in a record from the request on the current thread.
+ *
+ * @param record the record to fill in
+ * @param requestCounter the counter the next request id is taken from
+ * @param requestIdPrefix what every request id starts with
+ * @param requestIdAttribute the request attribute this capture keeps its ids in
+ * @return whether there is a request on the current thread
+ */
+ static boolean attribute(QueryTailRecord record, AtomicLong requestCounter, String requestIdPrefix, String requestIdAttribute) {
+ RequestAttributes attributes = RequestContextHolder.getRequestAttributes()
+ if (attributes == null) {
+ return false
+ }
+ record.requestId = requestId(attributes, requestCounter, requestIdPrefix, requestIdAttribute)
+ record.page = page(attributes)
+ record.request = SERVLET_PRESENT ? ServletRequestLookup.describe(attributes) : null
+ true
+ }
+
+ /**
+ * Gives the request line of a request: the one described for an earlier statement of the request,
+ * when the request has not moved on since, else one described now.
+ *
+ * @param method the HTTP method
+ * @param path the request path, as the client sent it, with the context path
+ * @param contextPath the context path of the application, which the path starts with
+ * @param queryString the query string of the request, or null
+ * @param attributes the request
+ */
+ static String describe(String method, String path, String contextPath, String queryString, RequestAttributes attributes) {
+ Object controller = attributes.getAttribute(CONTROLLER_NAME_ATTRIBUTE, RequestAttributes.SCOPE_REQUEST)
+ Object action = attributes.getAttribute(ACTION_NAME_ATTRIBUTE, RequestAttributes.SCOPE_REQUEST)
+ Object namespace = attributes.getAttribute(CONTROLLER_NAMESPACE_ATTRIBUTE, RequestAttributes.SCOPE_REQUEST)
+ Object mapping = attributes.getAttribute(MATCHED_MAPPING_ATTRIBUTE, RequestAttributes.SCOPE_REQUEST)
+ Object pattern = attributes.getAttribute(MATCHED_PATTERN_ATTRIBUTE, RequestAttributes.SCOPE_REQUEST)
+ RequestLine described = new RequestLine(method, path, queryString, controller, action, namespace, mapping, pattern)
+ Object earlier = attributes.getAttribute(REQUEST_LINE_ATTRIBUTE, RequestAttributes.SCOPE_REQUEST)
+ if (earlier instanceof RequestLine && ((RequestLine) earlier).sameAs(described)) {
+ return ((RequestLine) earlier).line
+ }
+ described.line = "${method} ${describePath(path, contextPath, queryString, attributes, described)}".toString()
+ attributes.setAttribute(REQUEST_LINE_ATTRIBUTE, described, RequestAttributes.SCOPE_REQUEST)
+ described.line
+ }
+
+ /**
+ * @return the path of a request with its values replaced: read against the mapping that matched it,
+ * or against the pattern a Spring MVC handler mapping matched, or by the values its parameters hold
+ */
+ private static String describePath(String path, String contextPath, String queryString, RequestAttributes attributes,
+ RequestLine request) {
+ // only the framework sets the controller attribute; a client can send a controller parameter
+ if (request.controller != null) {
+ Set<String> names = codeNames(request)
+ String mapped = readAgainstMapping(path, contextPath, request.mapping, names)
+ if (mapped != null) {
+ return mapped
+ }
+ Set<String> values = new HashSet<>()
+ addValues(values, mappedParameters(attributes))
+ addQueryValues(values, queryString)
+ return redactPath(path, values, names)
+ }
+ if (request.pattern != null) {
+ return contextPath + patternPath(request.pattern.toString())
+ }
+ withholdPath(path)
+ }
+
+ /** @return the names of the controller, action and namespace a request reached, which a path may spell out */
+ private static Set<String> codeNames(RequestLine request) {
+ Set<String> names = new HashSet<>()
+ for (Object name in [request.controller, request.action, request.namespace]) {
+ if (name != null && !name.toString().isEmpty()) {
+ names.add(name.toString())
+ }
+ }
+ names
+ }
+
+ /**
+ * Reads a path against the tokens of the URL mapping that matched it: a token the mapping spells
+ * out is kept where the path spells it out too, a variable segment that names the controller,
+ * action or namespace is kept, and every other variable is replaced, as is every segment a greedy
+ * token takes. The path parameters of a segment are replaced each.
+ *
+ * @return the path read so, or null when the mapping does not say how, or the path does not
+ * follow its tokens
+ */
+ private static String readAgainstMapping(String path, String contextPath, Object mapping, Set<String> names) {
+ String[] tokens = mappingTokens(mapping)
+ if (tokens == null) {
+ return null
+ }
+ String prefix = contextPath && path.startsWith(contextPath) ? contextPath : ''
+ String within = path.substring(prefix.length())
+ if (!within.startsWith('/')) {
+ return null
+ }
+ List<String> segments = new ArrayList<>(Arrays.asList(within.substring(1).split('/', -1)))
+ boolean trailingSlash = !segments.isEmpty() && segments.last().isEmpty()
+ if (trailingSlash) {
+ segments.removeLast()
+ }
+ List<String> spelled = tokens.findAll { String token -> !token.isEmpty() }
+ int greedy = spelled.findIndexOf { String token -> token.contains('**') }
+ List<String> out = []
+ int at = 0
+ for (int i = 0; i < spelled.size(); i++) {
+ String token = spelled[i]
+ boolean optional = token.endsWith('?')
+ String bare = optional ? token.substring(0, token.length() - 1) : token
+ if (i == greedy) {
+ int taken = segments.size() - (spelled.size() - 1)
+ if (taken < 0) {
+ return null
+ }
+ for (int k = 0; k < taken; k++) {
+ out.add(PLACEHOLDER + parametersOf(segments[at++]))
+ }
+ continue
+ }
+ if (at >= segments.size()) {
+ if (optional) {
+ continue
+ }
+ return null
+ }
+ String segment = segments[at]
+ int semicolon = segment.indexOf(';')
+ String bareSegment = semicolon < 0 ? segment : segment.substring(0, semicolon)
+ String decoded = decode(bareSegment)
+ boolean last = at == segments.size() - 1
+ if (!bare.contains('(') && !bare.contains('*')) {
+ if (decoded != bare && !(last && withoutExtension(decoded) == bare)) {
+ return null
+ }
+ out.add(bareSegment + parametersOf(segment))
+ }
+ else if (bare == '(*)' && (names.contains(decoded) || (last && names.contains(withoutExtension(decoded))))) {
+ out.add(bareSegment + parametersOf(segment))
+ }
+ else {
+ out.add(PLACEHOLDER + parametersOf(segment))
+ }
+ at++
+ }
+ if (at != segments.size()) {
+ return null
+ }
+ prefix + '/' + out.join('/') + (trailingSlash && !out.isEmpty() ? '/' : '')
+ }
+
+ private static String withoutExtension(String segment) {
+ int dot = segment.lastIndexOf('.')
+ dot > 0 ? segment.substring(0, dot) : segment
+ }
+
+ private static String parametersOf(String segment) {
+ int semicolon = segment.indexOf(';')
+ semicolon < 0 ? '' : withheldPathParameters(segment.substring(semicolon))
+ }
+
+ /**
+ * @return the tokens of the URL mapping Grails matched a request with, which spell out its
+ * literal segments and stand in for its variables, or null when there is none to read
+ */
+ private static String[] mappingTokens(Object mapping) {
+ if (mapping == null) {
+ return null
+ }
+ Object data = invoke(URL_DATA, mapping)
+ Object tokens = data == null ? null : invoke(TOKENS, data)
+ tokens instanceof String[] ? (String[]) tokens : null
+ }
+
+ private static Object invoke(ClassValue<Method> methods, Object target) {
+ Method method = methods.get(target.getClass())
+ if (method == null) {
+ return null
+ }
+ try {
+ return method.invoke(target)
+ }
+ catch (ReflectiveOperationException | RuntimeException ignored) {
+ return null
+ }
+ }
+
+ /**
+ * @return the pattern a Spring MVC handler mapping matched, with every segment that holds a
+ * variable or a wildcard replaced
+ */
+ private static String patternPath(String pattern) {
+ String[] segments = pattern.split('/', -1)
+ for (int i = 0; i < segments.length; i++) {
+ String segment = segments[i]
+ if (segment.contains('{') || segment.contains('*') || segment.contains('?')) {
+ segments[i] = PLACEHOLDER
+ }
+ }
+ segments.join('/')
+ }
+
+ /**
+ * Adds the values of a query string, decoded as the servlet container decodes them. The query
+ * string is read rather than the parameters of the request, as reading those makes the container
+ * parse a form posted in the body and use it up before the application reads it.
+ */
+ private static void addQueryValues(Set<String> values, String queryString) {
+ if (!queryString) {
+ return
+ }
+ for (String pair : queryString.split('&')) {
+ int equals = pair.indexOf('=')
+ if (equals >= 0 && !MAPPING_PARAMETERS.contains(decodeQuery(pair.substring(0, equals)))) {
+ values.add(decodeQuery(pair.substring(equals + 1)))
+ }
+ }
+ }
+
+ private static String decodeQuery(String text) {
+ try {
+ return URLDecoder.decode(text, StandardCharsets.UTF_8)
+ }
+ catch (IllegalArgumentException ignored) {
+ return text
+ }
+ }
+
+ /**
+ * Replaces every segment of a path: before the URL mapping has run, nothing tells a literal
+ * segment from a token, so only the shape of the path is recorded.
+ */
+ private static String withholdPath(String path) {
+ String[] segments = path.split('/', -1)
+ StringBuilder out = new StringBuilder(path.length())
+ for (int i = 0; i < segments.length; i++) {
+ if (i > 0) {
+ out.append('/')
+ }
+ out.append(segments[i].isEmpty() ? segments[i] : PLACEHOLDER)
+ }
+ out.toString()
+ }
+
+ private static String requestId(RequestAttributes attributes, AtomicLong requestCounter, String requestIdPrefix, String requestIdAttribute) {
+ // the request itself, which a task the request starts sees through attributes of its own
+ synchronized (SERVLET_PRESENT ? ServletRequestLookup.mutex(attributes) : attributes) {
+ Object existing = attributes.getAttribute(requestIdAttribute, RequestAttributes.SCOPE_REQUEST)
+ if (existing != null) {
+ return existing.toString()
+ }
+ String id = requestIdPrefix + REQUEST_ID_PREFIX + requestCounter.incrementAndGet()
+ attributes.setAttribute(requestIdAttribute, id, RequestAttributes.SCOPE_REQUEST)
+ return id
+ }
+ }
+
+ private static String page(RequestAttributes attributes) {
Review Comment:
Fixed in 6b133f1c64. While the action attribute is unset, the page and the request line take the default action of the controller the URL mapping matched (`getControllerClass().getDefaultAction()` on `org.grails.url.match.info`). That lookup is checked against the controller name the request carries, so a mapping left from before a forward cannot lend another controller's default.
`QueryTailRequestAttributionSpec`:
- "every statement of a request to a controller's default action is on one page, its before interceptor's too" records a statement before the action is set, as a `before` interceptor sees the request, and one after, then asserts one request id and one page key, `queryTailShelf/browse`, for a controller whose `defaultAction` is not `index`.
- "a mapping that matched another controller than the request names gives it no default action" covers the forward guard.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@grails.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add the query-tail command to find query-heavy pages [grails-core]
Posted by "testlens-app[bot] (via GitHub)" <gi...@apache.org>.
testlens-app[bot] commented on PR #16631:
URL: https://github.com/apache/grails-core/pull/16631#issuecomment-6459978275
## ✅ All tests passed ✅
🏷️ Commit: 72d972074060d20aa3d09798c47a1a9ad5764794
▶️ Tests: 141183 executed
⚪️ Checks: 97/97 completed
---
_Learn more about TestLens at [testlens.app/docs](https://testlens.app/docs/features/pr-comment/)._
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@grails.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add the sql-tail command to find query-heavy pages [grails-core]
Posted by "jdaugherty (via GitHub)" <gi...@apache.org>.
jdaugherty commented on PR #16631:
URL: https://github.com/apache/grails-core/pull/16631#issuecomment-6142233874
As part of our forge discussion, we need to have a serious discussion about commands that require an app, ones that do not, and then just regular project scripts. It seems there's a gap in what's runnable via gradle - the commands that do not require an app start can't be kicked off currently. This PR is not going to try to tackle that and added sql tail only as a grails command as a result.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@grails.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add the sql-tail command to find query-heavy pages [grails-core]
Posted by "jamesfredley (via GitHub)" <gi...@apache.org>.
jamesfredley commented on code in PR #16631:
URL: https://github.com/apache/grails-core/pull/16631#discussion_r4350340665
##########
grails-datamapping-core/src/main/groovy/org/grails/datastore/gorm/jdbc/sqltail/SqlTailRedactor.java:
##########
@@ -0,0 +1,363 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * https://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.grails.datastore.gorm.jdbc.sqltail;
+
+import java.util.ArrayList;
+import java.util.List;
+
+/**
+ * Takes the values out of a statement before it is captured. Every string, number and
+ * dollar-quoted literal becomes {@link #PLACEHOLDER} and every comment is dropped, so that a value
+ * written into the SQL text, a password in a native query or a key in a comment, never reaches the
+ * capture file. Only the recorded copy is redacted: the statement the database runs is not changed.
+ *
+ * <p>Databases disagree on parts of the lexical syntax: whether a backslash escapes a quote, whether
+ * comments nest, whether double quotes delimit an identifier or a string, and whether {@code #}
+ * starts a comment. A statement is redacted under every reading its {@link Syntax} allows, and is
+ * recorded only when all of them give the same text. When they do not, or when a literal or a
+ * comment never ends, the statement is withheld: {@link #WITHHELD} is recorded in its place, never
+ * the statement.</p>
+ *
+ * @since 8.1
+ */
+public final class SqlTailRedactor {
+
+ /** What a literal is replaced with. */
+ public static final String PLACEHOLDER = "?";
+
+ /** What is recorded in place of a statement whose values cannot be told apart from its SQL. */
+ public static final String WITHHELD = "(statement withheld: its values could not be redacted)";
+
+ private static final boolean[] NO = {false};
+ private static final boolean[] EITHER = {false, true};
+
+ /**
+ * The lexical syntax of the database a statement runs on, as far as finding its values goes.
+ */
+ public enum Syntax {
+
+ /** Not known: a statement is recorded only when the readings of every syntax below agree. */
+ UNKNOWN,
+
+ /** Standard SQL, where double quotes delimit identifiers: H2, PostgreSQL, Oracle, DB2 and most others. */
+ STANDARD,
+
+ /** MySQL and MariaDB: double quotes delimit strings, and {@code #} and {@code -- } start comments. */
+ MYSQL,
+
+ /** Microsoft SQL Server and Sybase: square brackets delimit identifiers, and comments nest. */
+ TRANSACT_SQL;
+
+ /**
+ * @param openQuote the character a dialect opens a quoted identifier with
+ * @return the syntax of the databases that quote identifiers with it
+ */
+ public static Syntax forIdentifierQuote(char openQuote) {
+ return switch (openQuote) {
+ case '"' -> STANDARD;
+ case '`' -> MYSQL;
+ case '[' -> TRANSACT_SQL;
+ default -> UNKNOWN;
+ };
+ }
+ }
+
+ private SqlTailRedactor() {
+ }
+
+ /**
+ * Redacts a statement.
+ *
+ * @param sql the statement
+ * @param syntax the syntax of the database it runs on
+ * @return the statement with its literals replaced and its comments dropped, or {@link #WITHHELD}
+ */
+ public static String redact(String sql, Syntax syntax) {
+ if (sql == null) {
+ return null;
+ }
+ String redacted = null;
+ for (Reading reading : readings(sql, syntax == null ? Syntax.UNKNOWN : syntax)) {
+ String candidate = reading.redact(sql);
+ if (candidate == null || (redacted != null && !redacted.equals(candidate))) {
+ return WITHHELD;
+ }
+ redacted = candidate;
+ }
+ return redacted;
+ }
+
+ /**
+ * The readings a statement has to agree under. A reading that depends on a character the
+ * statement does not contain is the same as the one that does not, so it is left out.
+ */
+ private static List<Reading> readings(String sql, Syntax syntax) {
+ boolean[] backslash = sql.indexOf('\\') >= 0 ? EITHER : NO;
+ int comment = sql.indexOf("/*");
+ boolean[] nesting = comment >= 0 && sql.indexOf("/*", comment + 2) >= 0 ? EITHER : NO;
+ List<Reading> readings = new ArrayList<>();
+ if (syntax == Syntax.STANDARD || syntax == Syntax.UNKNOWN) {
+ for (boolean backslashEscapes : backslash) {
+ for (boolean nestedComments : nesting) {
+ readings.add(new Reading(Syntax.STANDARD, backslashEscapes, nestedComments));
+ }
+ }
+ }
+ if (syntax == Syntax.MYSQL || syntax == Syntax.UNKNOWN) {
+ for (boolean backslashEscapes : backslash) {
+ readings.add(new Reading(Syntax.MYSQL, backslashEscapes, false));
+ }
+ }
+ if (syntax == Syntax.TRANSACT_SQL || syntax == Syntax.UNKNOWN) {
+ readings.add(new Reading(Syntax.TRANSACT_SQL, false, true));
+ }
+ return readings;
+ }
+
+ /**
+ * One way of reading a statement. Only MySQL delimits strings with double quotes and identifiers
+ * with backticks, starts a comment with {@code #}, and needs whitespace after {@code --}; only
+ * Transact-SQL delimits identifiers with square brackets.
+ *
+ * @param syntax the syntax read in, never {@link Syntax#UNKNOWN}
+ * @param backslashEscapes whether a backslash escapes the character after it in a string
+ * @param nestedComments whether a block comment can contain another
+ */
+ private record Reading(Syntax syntax, boolean backslashEscapes, boolean nestedComments) {
+
+ /**
+ * @return the redacted statement, or null when a literal, a quoted identifier or a comment never ends
+ */
+ String redact(String sql) {
+ boolean mysql = syntax == Syntax.MYSQL;
+ int length = sql.length();
+ StringBuilder out = new StringBuilder(length);
+ int i = 0;
+ while (i < length) {
+ char c = sql.charAt(i);
+ char next = i + 1 < length ? sql.charAt(i + 1) : 0;
+ char previous = i > 0 ? sql.charAt(i - 1) : 0;
+ int dollarTag = c == '$' && !isIdentifierPart(previous) ? dollarTagLength(sql, i) : 0;
+ int end;
+ if (c == '\'') {
+ end = endOfQuotedLiteral(sql, i);
+ out.append(PLACEHOLDER);
+ }
+ else if (c == '"') {
Review Comment:
Double-quoted text is kept for every syntax except MySQL. SQL Server still accepts double-quoted string literals when `QUOTED_IDENTIFIER` is off. This batch keeps the secret:
```sql
SET QUOTED_IDENTIFIER OFF;
SELECT "example-secret";
```
For Transact-SQL, treat a double-quoted span as ambiguous and withhold the statement, or redact it. Please cover that batch through the capture API.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@grails.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add the sql-tail command to find query-heavy pages [grails-core]
Posted by "testlens-app[bot] (via GitHub)" <gi...@apache.org>.
testlens-app[bot] commented on PR #16631:
URL: https://github.com/apache/grails-core/pull/16631#issuecomment-6354084305
## 🚨 TestLens detected 1 failed test 🚨
### Test Summary
> [!TIP]
> Inspect all details with the [TestLens CLI](https://testlens.app/docs/features/cli): `testlens pr 16631`
#### [Groovy Snapshot Canary Build / Build Grails \(shard 2\)](https://github.com/apache/grails-core/actions/runs/39840275001/job/120118323285?pr=16631) > :grails-data-mongodb-spring-boot:test
| Test | Runs | Flakiness |
|---|---|--:|
| MongoDbGormAutoConfigurationSpec | ❌ | 0% 🟢 |
🏷️ Commit: afaaabb58acb2a5021b98827b98586762cfc4c0a
▶️ Tests: 113631 executed
🟡 Checks: 92/95 completed
### Test Failures
<details><summary><strong>MongoDbGormAutoConfigurationSpec</strong> (:grails-data-mongodb-spring-boot:test in <a href="https://github.com/apache/grails-core/actions/runs/39840275001/job/120118323285?pr=16631">Groovy Snapshot Canary Build / Build Grails (shard 2)</a>)</summary>
```
org.testcontainers.containers.ContainerLaunchException: Container startup failed for image mongo:8.0
at org.testcontainers.containers.GenericContainer.doStart(GenericContainer.java:346)
at org.testcontainers.containers.GenericContainer.start(GenericContainer.java:317)
at org.apache.grails.testing.mongo.MongoContainerHolder.startMongoContainer(MongoContainerHolder.groovy:42)
at org.apache.grails.testing.mongo.MongoContainerHolder.getContainer(MongoContainerHolder.groovy:53)
at org.apache.grails.testing.mongo.StartMongoGrailsUnitExtension.visitSpec_closure1(StartMongoGrailsUnitExtension.groovy:78)
at org.spockframework.runtime.extension.MethodInvocation.proceed(MethodInvocation.java:139)
at org.spockframework.runtime.model.MethodInfo.invoke(MethodInfo.java:157)
at java.base/java.util.ArrayList.forEach(ArrayList.java:1596)
Caused by: org.rnorth.ducttape.RetryCountExceededException: Retry limit hit with exception
at org.rnorth.ducttape.unreliables.Unreliables.retryUntilSuccess(Unreliables.java:88)
at org.testcontainers.containers.GenericContainer.doStart(GenericContainer.java:331)
... 7 more
Caused by: org.testcontainers.containers.ContainerLaunchException: Could not create/start container
at org.testcontainers.containers.GenericContainer.tryStart(GenericContainer.java:551)
at org.testcontainers.containers.GenericContainer.lambda$doStart$0(GenericContainer.java:341)
at org.rnorth.ducttape.unreliables.Unreliables.retryUntilSuccess(Unreliables.java:81)
... 8 more
Caused by: java.lang.IllegalStateException: Wait strategy failed. Container exited with code 1
at org.testcontainers.containers.GenericContainer.tryStart(GenericContainer.java:520)
... 10 more
Caused by: org.testcontainers.containers.ContainerLaunchException: Timed out waiting for log output matching '(?i).*waiting for connections.*'
at org.testcontainers.containers.wait.strategy.LogMessageWaitStrategy.waitUntilReady(LogMessageWaitStrategy.java:47)
at org.testcontainers.containers.wait.strategy.AbstractWaitStrategy.waitUntilReady(AbstractWaitStrategy.java:52)
at org.testcontainers.containers.GenericContainer.waitUntilContainerStarted(GenericContainer.java:904)
at org.testcontainers.containers.GenericContainer.tryStart(GenericContainer.java:487)
... 10 more
```
</details>
### Rerun Controls
> [!NOTE]
> Checks are currently running using the configuration below.
Select tests to mute in this pull request:
🔲 MongoDbGormAutoConfigurationSpec <!-- uniqueId=[engine:spock]/[spec:org.grails.datastore.gorm.mongodb.boot.autoconfigure.MongoDbGormAutoConfigurationSpec] -->
Reuse successful test results:
🔲 ♻️ Only rerun the tests that failed or were muted before
Click the checkbox to trigger a rerun:
🔲 **Rerun jobs**
---
_Learn more about TestLens at [testlens.app/docs](https://testlens.app/docs/features/pr-comment/)._
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@grails.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add the query-tail command to find query-heavy pages [grails-core]
Posted by "jdaugherty (via GitHub)" <gi...@apache.org>.
jdaugherty commented on PR #16631:
URL: https://github.com/apache/grails-core/pull/16631#issuecomment-6383993227
@jamesfredley re review 5637980816: both items are fixed on head 5e5feb5e4d. The request path is now matched percent-decoded as well as as-sent, so `/search/secret%40example.com` with `q=secret@example.com` is recorded as `GET /search/?`, asserted through the capture API in `QueryTailCaptureSpec` (neither the encoded nor the decoded form reaches the file) and in the attribution spec, including values whose slashes the client encoded. `--stream --page=<page>` now applies the page match before a streamed record is rendered, with the same rule as the summary; `QueryTailRunnerSpec` covers `controller/action` and a bare controller name. Details in the two thread replies, which I resolved.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@grails.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add the query-tail command to find query-heavy pages [grails-core]
Posted by "jdaugherty (via GitHub)" <gi...@apache.org>.
jdaugherty commented on code in PR #16631:
URL: https://github.com/apache/grails-core/pull/16631#discussion_r4372207120
##########
grails-data-hibernate7/core/src/main/groovy/org/apache/grails/data/hibernate7/core/querytail/QueryTailStatementInspector.groovy:
##########
@@ -0,0 +1,319 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * https://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.grails.data.hibernate7.core.querytail
+
+import groovy.transform.CompileStatic
+import groovy.transform.PackageScope
+import org.hibernate.SessionFactory
+import org.hibernate.SessionFactoryObserver
+import org.hibernate.cfg.AvailableSettings
+import org.hibernate.cfg.Configuration
+import org.hibernate.dialect.AbstractTransactSQLDialect
+import org.hibernate.dialect.CockroachDialect
+import org.hibernate.dialect.DB2Dialect
+import org.hibernate.dialect.Dialect
+import org.hibernate.dialect.H2Dialect
+import org.hibernate.dialect.HANADialect
+import org.hibernate.dialect.HSQLDialect
+import org.hibernate.dialect.MySQLDialect
+import org.hibernate.dialect.OracleDialect
+import org.hibernate.dialect.PostgreSQLDialect
+import org.hibernate.dialect.SpannerDialect
+import org.hibernate.dialect.SpannerPostgreSQLDialect
+import org.hibernate.engine.spi.SessionFactoryImplementor
+import org.hibernate.resource.jdbc.spi.StatementInspector
+import org.springframework.beans.BeanUtils
+import org.springframework.util.ClassUtils
+
+import org.apache.grails.data.core.querytail.QueryTailCapture
+import org.apache.grails.data.core.querytail.QueryTailSettings
+import org.apache.grails.data.core.querytail.QueryTailSyntax
+import org.grails.datastore.mapping.core.exceptions.ConfigurationException
+
+/**
+ * The statement inspector GORM installs on a session factory when query-tail capture is enabled. It
+ * records every statement Hibernate is about to run to the {@link QueryTailCapture} of the
+ * application, without its values, and returns the statement as it found it: the database runs
+ * it unchanged. An inspector the application configured itself through
+ * {@code hibernate.session_factory.statement_inspector} keeps running: it sees the statement
+ * first, and the statement it returns is the one run and recorded.
+ *
+ * <p>The session factory owns the capture. The inspector observes it: it opens the capture when
+ * the session factory is created, takes from its dialect how the database quotes values, and
+ * closes the capture when the session factory closes, so a session factory that fails to start
+ * leaves no capture open either. An observer the application set on the configuration keeps
+ * being called.</p>
+ *
+ * <p>Nothing of this is installed unless {@link QueryTailSettings#SETTING_ENABLED} is set, so an
+ * application that does not capture pays nothing per statement.</p>
+ *
+ * @since 8.1
+ */
+@CompileStatic
+class QueryTailStatementInspector implements StatementInspector, SessionFactoryObserver {
+
+ @Serial
+ private static final long serialVersionUID = 1L
+
+ /** The community dialects, which are not a dependency, that the inspector tells apart by name. */
+ private static final String COMMUNITY_INFORMIX = 'org.hibernate.community.dialect.InformixDialect'
+ private static final String COMMUNITY_SQL_ANYWHERE = 'org.hibernate.community.dialect.SybaseAnywhereDialect'
+
+ /** The connection source name statements are recorded under. */
+ final String dataSourceName
+
+ /** The file statements are recorded to. */
+ final File file
+
+ /** The inspector the application configured, or null. */
+ final StatementInspector delegate
+
+ /** The session factory observer the application configured, or null. */
+ final SessionFactoryObserver observer
+
+ private transient volatile QueryTailCapture capture
+ private transient boolean closed
+
+ /**
+ * @param dataSourceName the connection source the inspected session factory belongs to
+ * @param file the capture file statements are recorded to
+ * @param delegate the inspector the application configured, or null
+ * @param observer the session factory observer the application configured, or null
+ */
+ QueryTailStatementInspector(String dataSourceName, File file, StatementInspector delegate,
+ SessionFactoryObserver observer) {
+ this.dataSourceName = dataSourceName
+ this.file = file
+ this.delegate = delegate
+ this.observer = observer
+ }
+
+ /**
+ * Installs the inspector on the configuration of a session factory when capture is enabled, as
+ * its statement inspector and as the observer of the session factory it builds.
+ *
+ * @param dataSourceName the connection source the session factory is built for
+ * @param settings the query-tail settings of the connection source
+ * @param configuration the configuration the session factory is built from
+ * @return the installed inspector, or null when capture is not enabled
+ * @throws ConfigurationException when the inspector the application configured cannot be created
+ */
+ static QueryTailStatementInspector install(String dataSourceName, QueryTailSettings settings,
+ Configuration configuration) {
+ if (settings == null || !settings.enabled) {
+ return null
+ }
+ Properties properties = configuration.properties
+ StatementInspector configured = resolve(properties.get(AvailableSettings.STATEMENT_INSPECTOR))
Review Comment:
Fixed in b8c346b48c. On Hibernate 7, `Configuration.buildSessionFactory` applies an inspector set with `setStatementInspector` over the one the property names, so a `configClass` that set one replaced the capture.
`install` now wraps the inspector already set on the configuration, and falls back to the one the property names when none is set. It installs the wrapper with `setStatementInspector` and as the property. When both are set, the wrapper runs the one set on the configuration, which is what Hibernate runs without the capture.
Hibernate 5.6's `Configuration` has no `setStatementInspector`, and no field for one either. Its `buildSessionFactory` applies only the interceptor, the observer, the entity-not-found delegate, the tuplizer factory and the tenant resolver. On Hibernate 5, a `configClass` can therefore set an inspector only through the configuration's properties, which `install` reads after the `configClass` is constructed. I added a Hibernate 5 test for that case.
Tests:
- **Hibernate 7:** a `configClass` that calls `setStatementInspector` in its constructor. Its inspector runs, the query returns its row, and the statement the inspector returns is recorded.
- **Hibernate 7:** the same `configClass` while the property names another inspector. Only the one set on the configuration runs.
- **Hibernate 5:** a `configClass` that puts an inspector in its properties. The inspector keeps running and its statement is recorded.
Both Hibernate 7 tests fail on the previous installer.
##########
grails-datamapping-core/src/cli/groovy/org/apache/grails/data/core/cli/querytail/StatementCollapser.groovy:
##########
@@ -0,0 +1,131 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * https://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.grails.data.core.cli.querytail
+
+import java.util.regex.Pattern
+
+import groovy.transform.CompileStatic
+
+/**
+ * Shortens a statement to the part that identifies it: the table a select reads and its where
+ * clause, the table an update changes and its where clause, the table an insert writes to. The
+ * column lists Hibernate generates are long and say nothing about which code ran the statement,
+ * and dropping them lets the same query count as one statement wherever it is run from. A Cypher
+ * statement is shown whole: its pattern names what it reads or writes, and GORM generates nothing
+ * long in it. So is a MongoDB command, which is recorded as its name, its collection and the
+ * document that shapes it.
+ *
+ * @since 8.1
+ */
+@CompileStatic
+class StatementCollapser {
+
+ private static final Pattern COMMENT = Pattern.compile('/\\*.*?\\*/', Pattern.DOTALL)
+ private static final Pattern WHITESPACE = Pattern.compile('\\s+')
+ private static final String ELLIPSIS = '...'
+
+ /**
+ * @param sql a statement
+ * @return the shortened statement, or the statement with its comments and extra whitespace
+ * removed when it is not a SQL select, update or insert
+ */
+ static String collapse(String sql) {
+ String text = normalize(sql)
+ if (text.isEmpty() || StatementType.isCypher(text)) {
+ return text
+ }
+ switch (StatementType.of(text)) {
+ case StatementType.SELECT:
+ return collapseSelect(text)
+ case StatementType.UPDATE:
+ return collapseUpdate(text)
+ case StatementType.INSERT:
+ return collapseInsert(text)
+ default:
+ return text
+ }
+ }
+
+ /**
+ * @param sql a statement
+ * @return the statement without comments, its whitespace collapsed to single spaces
+ */
+ static String normalize(String sql) {
+ if (sql == null) {
+ return ''
+ }
+ WHITESPACE.matcher(COMMENT.matcher(sql).replaceAll(' ')).replaceAll(' ').trim()
+ }
+
+ private static String collapseSelect(String text) {
+ int from = topLevelIndexOf(text, ' from ')
+ from < 0 ? text : "select ${ELLIPSIS}${text.substring(from)}".toString()
+ }
+
+ private static String collapseUpdate(String text) {
+ int set = topLevelIndexOf(text, ' set ')
+ if (set < 0) {
+ return text
+ }
+ int where = topLevelIndexOf(text, ' where ')
+ String head = text.substring(0, set)
+ where > set ? "${head} ${ELLIPSIS}${text.substring(where)}".toString() : "${head} ${ELLIPSIS}".toString()
+ }
+
+ private static String collapseInsert(String text) {
+ int columns = topLevelIndexOf(text, '(')
+ int values = topLevelIndexOf(text, ' values ')
+ int cut = columns < 0 ? values : (values < 0 ? columns : Math.min(columns, values))
+ cut < 0 ? text : "${text.substring(0, cut).trim()} ${ELLIPSIS}".toString()
+ }
+
+ /**
+ * Finds a keyword outside parentheses and string literals, so that a subquery or a value
+ * never stands in for the outer statement's clause.
+ */
+ static int topLevelIndexOf(String text, String needle) {
+ String haystack = text.toLowerCase(Locale.ROOT)
+ if (haystack.length() != text.length()) {
+ haystack = text
+ }
+ int depth = 0
+ boolean quoted = false
+ for (int i = 0; i < haystack.length(); i++) {
+ char c = haystack.charAt(i)
+ if (c == ('\'' as char)) {
Review Comment:
Fixed in b8c346b48c. The clause scan now skips quoted names the same way it skips string literals. Names can be quoted with double quotes, backticks or square brackets, and a closing quote written twice stands for itself (`""`, ` `` `, `]]`, and `''` in a literal).
Your two updates now collapse to `update "order set lines" ... where id=?` and `update "order set headers" ... where id=?`.
- **Parentheses:** a parenthesis inside a quoted name no longer counts, so `insert into "order (lines)" (id,title) values (?,?)` keeps its table.
- **Comments and whitespace:** these are removed only outside quoted names, so a name that holds two spaces or `/* */` is kept as written.
- **Unclosed quotes:** a statement with a name left open is shown whole.
New rows cover each quote style, doubled quotes, and clause words and parentheses inside names in `select`, `update` and `insert`. Every row that used to collapse wrongly fails on the previous scanner.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@grails.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add the query-tail command to find query-heavy pages [grails-core]
Posted by "testlens-app[bot] (via GitHub)" <gi...@apache.org>.
testlens-app[bot] commented on PR #16631:
URL: https://github.com/apache/grails-core/pull/16631#issuecomment-6378932449
## 🚨 All tests passed but jobs failed 🚨
### Failed Jobs without Test Failures
❌ [CI / Hibernate5 Functional Tests \(Java 21, indy\=true\)](https://github.com/apache/grails-core/actions/runs/40052121561/job/120747100400?pr=16631)
🏷️ Commit: 44b44e9fb272cdd916f83e1922f852decaf06e24
▶️ Tests: 109158 executed
🟡 Checks: 60/92 completed
### Rerun Controls
Click the checkbox to trigger a rerun:
🔲 **Rerun jobs**
---
_Learn more about TestLens at [testlens.app/docs](https://testlens.app/docs/features/pr-comment/)._
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@grails.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add the query-tail command to find query-heavy pages [grails-core]
Posted by "testlens-app[bot] (via GitHub)" <gi...@apache.org>.
testlens-app[bot] commented on PR #16631:
URL: https://github.com/apache/grails-core/pull/16631#issuecomment-6379368364
## 🚨 All tests passed but jobs failed 🚨
### Failed Jobs without Test Failures
❌ [Coverage / Coverage - grails-gradle \(ubuntu-24.04\)](https://github.com/apache/grails-core/actions/runs/40058676471/job/120767231343?pr=16631)
🏷️ Commit: bded3443bf0c44bf94c8cd53c800d16126c20145
▶️ Tests: 374 executed
🟡 Checks: 16/91 completed
### Rerun Controls
Click the checkbox to trigger a rerun:
🔲 **Rerun jobs**
---
_Learn more about TestLens at [testlens.app/docs](https://testlens.app/docs/features/pr-comment/)._
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@grails.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add the query-tail command to find query-heavy pages [grails-core]
Posted by "jdaugherty (via GitHub)" <gi...@apache.org>.
jdaugherty commented on code in PR #16631:
URL: https://github.com/apache/grails-core/pull/16631#discussion_r4367062421
##########
grails-datamapping-core/src/main/groovy/org/apache/grails/data/core/querytail/QueryTailRequestAttribution.groovy:
##########
@@ -0,0 +1,315 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * https://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.grails.data.core.querytail
+
+import java.lang.reflect.Method
+import java.util.concurrent.atomic.AtomicLong
+import java.util.regex.Pattern
+
+import groovy.transform.CompileStatic
+
+import org.springframework.util.ClassUtils
+import org.springframework.web.context.request.RequestAttributes
+import org.springframework.web.context.request.RequestContextHolder
+import org.springframework.web.context.request.ServletRequestAttributes
+
+/**
+ * Attributes a captured statement to the web request on the current thread. Grails keeps the
+ * request, and the controller and action it resolved, on the thread that serves it, so a statement
+ * run by an interceptor, the action or the view all land on the same page. The first statement of
+ * a request gives it an id that every later statement of that request repeats, which is how the
+ * {@code query-tail} command counts statements per request. Work on any other thread is attributed to
+ * the thread by name.
+ *
+ * <p>The request is recorded as its method and path, with the values in the path replaced by
+ * {@code ?}: the values of the parameters the URL mapping took from it and those of its query, and
+ * any segment that is a number, a UUID or a long hexadecimal token, so that an id, a token or a key
+ * carried in a path never reaches the capture file.</p>
+ *
+ * <p>Spring Web is optional: without it on the classpath every statement is attributed to its
+ * thread.</p>
+ *
+ * @since 8.1
+ */
+@CompileStatic
+class QueryTailRequestAttribution {
+
+ /** The request attribute holding the id this attribution gave a request. */
+ public static final String REQUEST_ID_ATTRIBUTE = 'org.apache.grails.data.core.querytail.REQUEST_ID'
+
+ /** The request attribute Grails sets the controller name in. */
+ public static final String CONTROLLER_NAME_ATTRIBUTE = 'org.grails.CONTROLLER_NAME_ATTRIBUTE'
+
+ /** The request attribute Grails sets the controller namespace in. */
+ public static final String CONTROLLER_NAMESPACE_ATTRIBUTE = 'org.grails.CONTROLLER_NAMESPACE_ATTRIBUTE'
+
+ /** The request attribute Grails sets the action name in. */
+ public static final String ACTION_NAME_ATTRIBUTE = 'org.grails.ACTION_NAME_ATTRIBUTE'
+
+ private static final boolean SPRING_WEB_PRESENT = ClassUtils.isPresent(
+ 'org.springframework.web.context.request.RequestContextHolder', QueryTailRequestAttribution.classLoader)
+
+ private final AtomicLong requestCounter = new AtomicLong()
+ private final String requestIdPrefix
+
+ /**
+ * An attribution whose request ids are {@code r1}, {@code r2} and so on.
+ */
+ QueryTailRequestAttribution() {
+ this('')
+ }
+
+ /**
+ * @param requestIdPrefix what every request id starts with, which tells the requests of one JVM
+ * from those of another writing the same capture
+ */
+ QueryTailRequestAttribution(String requestIdPrefix) {
+ this.requestIdPrefix = requestIdPrefix ?: ''
+ }
+
+ /**
+ * Fills in the request id, page and request of a record from the current thread.
+ *
+ * @param record the record of a statement that just ran on this thread
+ * @return the same record
+ */
+ QueryTailRecord attribute(QueryTailRecord record) {
+ if (SPRING_WEB_PRESENT && WebRequestLookup.attribute(record, requestCounter, requestIdPrefix)) {
+ return record
+ }
+ record.requestId = null
+ record.request = null
+ record.page = QueryTailRecord.THREAD_PAGE_PREFIX + Thread.currentThread().name
+ record
+ }
+
+ /**
+ * Keeps every reference to Spring Web, in a signature as well as in a body, in a class that is only
+ * loaded once Spring Web is known to be present. The outer class can then be loaded, and reflected
+ * on, without Spring Web on the classpath.
+ */
+ @CompileStatic
+ private static class WebRequestLookup {
+
+ private static final String REQUEST_ID_PREFIX = 'r'
+
+ private static final String PLACEHOLDER = '?'
+
+ /** The parameters a URL mapping sets that name code rather than carry a value. */
+ private static final Set<String> MAPPING_PARAMETERS = ['controller', 'action', 'namespace', 'plugin', 'format'] as Set<String>
+
+ /** A segment that is a number, a UUID or a long hexadecimal token is a value whatever the mapping says. */
+ private static final Pattern VALUE_SHAPED = Pattern.compile(
+ '(?i)[0-9]+|[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}|[0-9a-f]{32,}')
+
+ private static final boolean SERVLET_PRESENT = ClassUtils.isPresent(
+ 'jakarta.servlet.http.HttpServletRequest', WebRequestLookup.classLoader)
+
+ /** The {@code getParams()} of a Grails web request, which holds the parameters its URL mapping took, by class. */
+ private static final ClassValue<Method> PARAMETERS = new ClassValue<Method>() {
+ @Override
+ protected Method computeValue(Class<?> type) {
+ try {
+ return type.getMethod('getParams')
+ }
+ catch (NoSuchMethodException ignored) {
+ return null
+ }
+ }
+ }
+
+ /**
+ * Fills in a record from the request on the current thread.
+ *
+ * @param record the record to fill in
+ * @param requestCounter the counter the next request id is taken from
+ * @param requestIdPrefix what every request id starts with
+ * @return whether there is a request on the current thread
+ */
+ static boolean attribute(QueryTailRecord record, AtomicLong requestCounter, String requestIdPrefix) {
+ RequestAttributes attributes = RequestContextHolder.getRequestAttributes()
+ if (attributes == null) {
+ return false
+ }
+ record.requestId = requestId(attributes, requestCounter, requestIdPrefix)
+ record.page = page(attributes)
+ record.request = SERVLET_PRESENT ? ServletRequestLookup.describe(attributes) : null
+ true
+ }
+
+ /**
+ * Builds the request line: the method and the path, with the values in the path, the parameters
+ * the URL mapping took from it among them, replaced.
+ *
+ * @param method the HTTP method
+ * @param path the request path
+ * @param attributes the request, whose mapped parameters are read when it is a Grails web request
+ * @param parameters the parameters of the servlet request, by name, each a {@code String[]}
+ */
+ static String describe(String method, String path, RequestAttributes attributes, Map<?, ?> parameters) {
+ Set<String> values = new HashSet<>()
+ addValues(values, mappedParameters(attributes))
+ addValues(values, parameters)
+ "${method} ${redactPath(path, values)}".toString()
+ }
+
+ private static String requestId(RequestAttributes attributes, AtomicLong requestCounter, String requestIdPrefix) {
+ synchronized (attributes) {
+ Object existing = attributes.getAttribute(REQUEST_ID_ATTRIBUTE, RequestAttributes.SCOPE_REQUEST)
+ if (existing != null) {
+ return existing.toString()
+ }
+ String id = requestIdPrefix + REQUEST_ID_PREFIX + requestCounter.incrementAndGet()
+ attributes.setAttribute(REQUEST_ID_ATTRIBUTE, id, RequestAttributes.SCOPE_REQUEST)
+ return id
+ }
+ }
+
+ private static String page(RequestAttributes attributes) {
+ Object controller = attributes.getAttribute(CONTROLLER_NAME_ATTRIBUTE, RequestAttributes.SCOPE_REQUEST)
+ if (controller == null) {
+ return null
+ }
+ Object namespace = attributes.getAttribute(CONTROLLER_NAMESPACE_ATTRIBUTE, RequestAttributes.SCOPE_REQUEST)
+ Object action = attributes.getAttribute(ACTION_NAME_ATTRIBUTE, RequestAttributes.SCOPE_REQUEST)
+ StringBuilder page = new StringBuilder()
+ if (namespace != null) {
+ page.append(namespace).append(':')
+ }
+ page.append(controller)
+ if (action != null) {
+ page.append('/').append(action)
+ }
+ page.toString()
+ }
+
+ /**
+ * @return the parameters of a Grails web request, which include those its URL mapping took from
+ * the path, or null for a request that is not one
+ */
+ private static Map<?, ?> mappedParameters(RequestAttributes attributes) {
+ Method parameters = PARAMETERS.get(attributes.getClass())
+ if (parameters == null) {
+ return null
+ }
+ try {
+ Object value = parameters.invoke(attributes)
+ return value instanceof Map ? (Map<?, ?>) value : null
+ }
+ catch (ReflectiveOperationException | RuntimeException ignored) {
+ return null
+ }
+ }
+
+ private static void addValues(Set<String> values, Map<?, ?> parameters) {
+ if (parameters == null) {
+ return
+ }
+ for (Map.Entry<?, ?> entry in parameters.entrySet()) {
+ if (MAPPING_PARAMETERS.contains(String.valueOf(entry.key))) {
+ continue
+ }
+ Object value = entry.value
+ if (value instanceof CharSequence) {
+ values.add(value.toString())
+ }
+ else if (value instanceof Object[]) {
+ for (Object element in (Object[]) value) {
+ if (element instanceof CharSequence) {
+ values.add(element.toString())
+ }
+ }
+ }
+ }
+ }
+
+ /**
+ * Replaces the values in a path: a segment that is one of the values, with or without an
+ * extension, a run of segments that spells a value holding slashes, and any segment shaped
+ * like a value.
+ */
+ private static String redactPath(String path, Set<String> values) {
+ String[] segments = path.split('/', -1)
Review Comment:
Fixed in 5e5feb5e4d (folded into the feature commit). `redactPath` now percent-decodes every segment (UTF-8; a `+` stays literal; a segment that is not valid percent-encoding is matched as sent) and matches both the raw and the decoded form against the mapped and query parameters, so `/search/secret%40example.com` with `q=secret@example.com` is written as `GET /search/?`. A value holding slashes is also found when the client encoded its slashes into one segment (`/files/reports%2F2026%2Fq3.pdf` → `GET /files/?`) or only some of them: the decoded segments are re-split before the run is matched. The path itself is still written as sent, with the matched segments replaced. Covered in `QueryTailRequestAttributionSpec` (three table rows plus the `%40` and the malformed-encoding cases) and through the capture API in `QueryTailCaptureSpec`, which asserts that neither `secret%40example.com` nor `secret@example.com` reaches the file; the guide says the values are found whether or not the
client encoded them.
##########
grails-datamapping-core/src/cli/groovy/org/apache/grails/data/core/cli/querytail/QueryTailRunner.groovy:
##########
@@ -0,0 +1,181 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * https://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.grails.data.core.cli.querytail
+
+import groovy.transform.CompileStatic
+
+import org.apache.grails.data.core.querytail.QueryTailRecord
+
+/**
+ * Runs the {@code query-tail} command: waits for the capture file, follows it, and prints the
+ * summary or the stream until stopped. The command script hands it the console and stops it when
+ * the shell cancels the command.
+ *
+ * @since 8.1
+ */
+@CompileStatic
+class QueryTailRunner {
+
+ /** How to get an application to write the capture file. */
+ public static final String START_HINT =
+ "Start the application with 'grails run-app --query-tail' or './gradlew bootRun -Pgrails.gorm.queryTail.enabled' " +
+ 'to capture the statements it runs.'
+
+ final QueryTailOptions options
+ final QueryTailRenderer renderer
+
+ /** How long to wait between looks at the file, in milliseconds. */
+ long pollMillis = 250
+
+ /** How long a summary stays on screen before it is printed again, in milliseconds. */
+ long refreshMillis = 1000
+
+ /** How much of the capture one look reads at most, in bytes; what is left is read without waiting. */
+ int chunk = QueryTailFileReader.CHUNK
+
+ private volatile boolean stopped
+
+ QueryTailRunner(QueryTailOptions options, PrintStream out, boolean ansi) {
+ this.options = options
+ this.renderer = new QueryTailRenderer(out, ansi)
+ this.renderer.width = options.width
+ }
+
+ /**
+ * Ends the run; the summary on screen stays.
+ */
+ void stop() {
+ stopped = true
+ }
+
+ /**
+ * @return whether the run was stopped
+ */
+ boolean isStopped() {
+ stopped
+ }
+
+ /**
+ * Runs until stopped, or until the summary has been printed once with {@link QueryTailOptions#once}.
+ *
+ * @return 0 when it ran, 1 when there was nothing to read
+ */
+ int run() {
+ File file = options.file
+ boolean waited = !file.exists()
+ if (waited && !awaitFile(file)) {
+ return stopped ? 0 : 1
+ }
+ // a capture that appeared while waiting is a run that has just started: read it from its first statement
+ QueryTailFileReader reader = new QueryTailFileReader(file, waited || options.all || options.once, chunk)
+ try {
+ follow(reader)
+ }
+ finally {
+ reader.close()
+ }
+ 0
+ }
+
+ private boolean awaitFile(File file) {
+ if (options.once) {
+ renderer.out.println("No query-tail capture at ${file}.")
+ renderer.out.println(START_HINT)
+ return false
+ }
+ renderer.out.println("Waiting for ${file}. ${START_HINT}")
+ while (!file.exists()) {
+ if (stopped || !pause(pollMillis)) {
+ return false
+ }
+ }
+ true
+ }
+
+ private void follow(QueryTailFileReader reader) {
+ QueryTailSummary summary = new QueryTailSummary()
+ QueryTailRecord previous = null
+ long lastRender = 0
+ boolean changed = true
+ while (true) {
+ List<QueryTailRecord> records = reader.poll()
+ if (reader.restarted) {
+ summary = new QueryTailSummary()
+ renderer.reset()
+ previous = null
+ changed = true
+ }
+ for (QueryTailRecord record : records) {
+ if (options.mode == QueryTailMode.STREAM) {
Review Comment:
Fixed in 5e5feb5e4d. `follow` applies `QueryTailSummary.matchesPage(pageOf(record), options.page)` before rendering a streamed record, with the same rule the summary uses (`controller/action`, a controller name, or a thread page), and only rendered records become the `previous` for the request headers. `QueryTailRunnerSpec` streams four requests over three pages with `--page=book/show` and gets the two `book/show` requests, headers and footers included, and with `--page=book` all three book requests and nothing from `author`; the reference page says `--page` filters the stream when `--stream` is given.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@grails.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add the query-tail command to find query-heavy pages [grails-core]
Posted by "jamesfredley (via GitHub)" <gi...@apache.org>.
jamesfredley commented on code in PR #16631:
URL: https://github.com/apache/grails-core/pull/16631#discussion_r4366898664
##########
grails-datamapping-core/src/cli/groovy/org/apache/grails/data/core/cli/querytail/QueryTailRunner.groovy:
##########
@@ -0,0 +1,181 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * https://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.grails.data.core.cli.querytail
+
+import groovy.transform.CompileStatic
+
+import org.apache.grails.data.core.querytail.QueryTailRecord
+
+/**
+ * Runs the {@code query-tail} command: waits for the capture file, follows it, and prints the
+ * summary or the stream until stopped. The command script hands it the console and stops it when
+ * the shell cancels the command.
+ *
+ * @since 8.1
+ */
+@CompileStatic
+class QueryTailRunner {
+
+ /** How to get an application to write the capture file. */
+ public static final String START_HINT =
+ "Start the application with 'grails run-app --query-tail' or './gradlew bootRun -Pgrails.gorm.queryTail.enabled' " +
+ 'to capture the statements it runs.'
+
+ final QueryTailOptions options
+ final QueryTailRenderer renderer
+
+ /** How long to wait between looks at the file, in milliseconds. */
+ long pollMillis = 250
+
+ /** How long a summary stays on screen before it is printed again, in milliseconds. */
+ long refreshMillis = 1000
+
+ /** How much of the capture one look reads at most, in bytes; what is left is read without waiting. */
+ int chunk = QueryTailFileReader.CHUNK
+
+ private volatile boolean stopped
+
+ QueryTailRunner(QueryTailOptions options, PrintStream out, boolean ansi) {
+ this.options = options
+ this.renderer = new QueryTailRenderer(out, ansi)
+ this.renderer.width = options.width
+ }
+
+ /**
+ * Ends the run; the summary on screen stays.
+ */
+ void stop() {
+ stopped = true
+ }
+
+ /**
+ * @return whether the run was stopped
+ */
+ boolean isStopped() {
+ stopped
+ }
+
+ /**
+ * Runs until stopped, or until the summary has been printed once with {@link QueryTailOptions#once}.
+ *
+ * @return 0 when it ran, 1 when there was nothing to read
+ */
+ int run() {
+ File file = options.file
+ boolean waited = !file.exists()
+ if (waited && !awaitFile(file)) {
+ return stopped ? 0 : 1
+ }
+ // a capture that appeared while waiting is a run that has just started: read it from its first statement
+ QueryTailFileReader reader = new QueryTailFileReader(file, waited || options.all || options.once, chunk)
+ try {
+ follow(reader)
+ }
+ finally {
+ reader.close()
+ }
+ 0
+ }
+
+ private boolean awaitFile(File file) {
+ if (options.once) {
+ renderer.out.println("No query-tail capture at ${file}.")
+ renderer.out.println(START_HINT)
+ return false
+ }
+ renderer.out.println("Waiting for ${file}. ${START_HINT}")
+ while (!file.exists()) {
+ if (stopped || !pause(pollMillis)) {
+ return false
+ }
+ }
+ true
+ }
+
+ private void follow(QueryTailFileReader reader) {
+ QueryTailSummary summary = new QueryTailSummary()
+ QueryTailRecord previous = null
+ long lastRender = 0
+ boolean changed = true
+ while (true) {
+ List<QueryTailRecord> records = reader.poll()
+ if (reader.restarted) {
+ summary = new QueryTailSummary()
+ renderer.reset()
+ previous = null
+ changed = true
+ }
+ for (QueryTailRecord record : records) {
+ if (options.mode == QueryTailMode.STREAM) {
Review Comment:
`query-tail --stream --page=book/show` prints every page. The page option is kept, and the summary path filters on it, but this branch renders every record.
Apply the page match before rendering a streamed record.
##########
grails-datamapping-core/src/main/groovy/org/apache/grails/data/core/querytail/QueryTailRequestAttribution.groovy:
##########
@@ -0,0 +1,315 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * https://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.grails.data.core.querytail
+
+import java.lang.reflect.Method
+import java.util.concurrent.atomic.AtomicLong
+import java.util.regex.Pattern
+
+import groovy.transform.CompileStatic
+
+import org.springframework.util.ClassUtils
+import org.springframework.web.context.request.RequestAttributes
+import org.springframework.web.context.request.RequestContextHolder
+import org.springframework.web.context.request.ServletRequestAttributes
+
+/**
+ * Attributes a captured statement to the web request on the current thread. Grails keeps the
+ * request, and the controller and action it resolved, on the thread that serves it, so a statement
+ * run by an interceptor, the action or the view all land on the same page. The first statement of
+ * a request gives it an id that every later statement of that request repeats, which is how the
+ * {@code query-tail} command counts statements per request. Work on any other thread is attributed to
+ * the thread by name.
+ *
+ * <p>The request is recorded as its method and path, with the values in the path replaced by
+ * {@code ?}: the values of the parameters the URL mapping took from it and those of its query, and
+ * any segment that is a number, a UUID or a long hexadecimal token, so that an id, a token or a key
+ * carried in a path never reaches the capture file.</p>
+ *
+ * <p>Spring Web is optional: without it on the classpath every statement is attributed to its
+ * thread.</p>
+ *
+ * @since 8.1
+ */
+@CompileStatic
+class QueryTailRequestAttribution {
+
+ /** The request attribute holding the id this attribution gave a request. */
+ public static final String REQUEST_ID_ATTRIBUTE = 'org.apache.grails.data.core.querytail.REQUEST_ID'
+
+ /** The request attribute Grails sets the controller name in. */
+ public static final String CONTROLLER_NAME_ATTRIBUTE = 'org.grails.CONTROLLER_NAME_ATTRIBUTE'
+
+ /** The request attribute Grails sets the controller namespace in. */
+ public static final String CONTROLLER_NAMESPACE_ATTRIBUTE = 'org.grails.CONTROLLER_NAMESPACE_ATTRIBUTE'
+
+ /** The request attribute Grails sets the action name in. */
+ public static final String ACTION_NAME_ATTRIBUTE = 'org.grails.ACTION_NAME_ATTRIBUTE'
+
+ private static final boolean SPRING_WEB_PRESENT = ClassUtils.isPresent(
+ 'org.springframework.web.context.request.RequestContextHolder', QueryTailRequestAttribution.classLoader)
+
+ private final AtomicLong requestCounter = new AtomicLong()
+ private final String requestIdPrefix
+
+ /**
+ * An attribution whose request ids are {@code r1}, {@code r2} and so on.
+ */
+ QueryTailRequestAttribution() {
+ this('')
+ }
+
+ /**
+ * @param requestIdPrefix what every request id starts with, which tells the requests of one JVM
+ * from those of another writing the same capture
+ */
+ QueryTailRequestAttribution(String requestIdPrefix) {
+ this.requestIdPrefix = requestIdPrefix ?: ''
+ }
+
+ /**
+ * Fills in the request id, page and request of a record from the current thread.
+ *
+ * @param record the record of a statement that just ran on this thread
+ * @return the same record
+ */
+ QueryTailRecord attribute(QueryTailRecord record) {
+ if (SPRING_WEB_PRESENT && WebRequestLookup.attribute(record, requestCounter, requestIdPrefix)) {
+ return record
+ }
+ record.requestId = null
+ record.request = null
+ record.page = QueryTailRecord.THREAD_PAGE_PREFIX + Thread.currentThread().name
+ record
+ }
+
+ /**
+ * Keeps every reference to Spring Web, in a signature as well as in a body, in a class that is only
+ * loaded once Spring Web is known to be present. The outer class can then be loaded, and reflected
+ * on, without Spring Web on the classpath.
+ */
+ @CompileStatic
+ private static class WebRequestLookup {
+
+ private static final String REQUEST_ID_PREFIX = 'r'
+
+ private static final String PLACEHOLDER = '?'
+
+ /** The parameters a URL mapping sets that name code rather than carry a value. */
+ private static final Set<String> MAPPING_PARAMETERS = ['controller', 'action', 'namespace', 'plugin', 'format'] as Set<String>
+
+ /** A segment that is a number, a UUID or a long hexadecimal token is a value whatever the mapping says. */
+ private static final Pattern VALUE_SHAPED = Pattern.compile(
+ '(?i)[0-9]+|[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}|[0-9a-f]{32,}')
+
+ private static final boolean SERVLET_PRESENT = ClassUtils.isPresent(
+ 'jakarta.servlet.http.HttpServletRequest', WebRequestLookup.classLoader)
+
+ /** The {@code getParams()} of a Grails web request, which holds the parameters its URL mapping took, by class. */
+ private static final ClassValue<Method> PARAMETERS = new ClassValue<Method>() {
+ @Override
+ protected Method computeValue(Class<?> type) {
+ try {
+ return type.getMethod('getParams')
+ }
+ catch (NoSuchMethodException ignored) {
+ return null
+ }
+ }
+ }
+
+ /**
+ * Fills in a record from the request on the current thread.
+ *
+ * @param record the record to fill in
+ * @param requestCounter the counter the next request id is taken from
+ * @param requestIdPrefix what every request id starts with
+ * @return whether there is a request on the current thread
+ */
+ static boolean attribute(QueryTailRecord record, AtomicLong requestCounter, String requestIdPrefix) {
+ RequestAttributes attributes = RequestContextHolder.getRequestAttributes()
+ if (attributes == null) {
+ return false
+ }
+ record.requestId = requestId(attributes, requestCounter, requestIdPrefix)
+ record.page = page(attributes)
+ record.request = SERVLET_PRESENT ? ServletRequestLookup.describe(attributes) : null
+ true
+ }
+
+ /**
+ * Builds the request line: the method and the path, with the values in the path, the parameters
+ * the URL mapping took from it among them, replaced.
+ *
+ * @param method the HTTP method
+ * @param path the request path
+ * @param attributes the request, whose mapped parameters are read when it is a Grails web request
+ * @param parameters the parameters of the servlet request, by name, each a {@code String[]}
+ */
+ static String describe(String method, String path, RequestAttributes attributes, Map<?, ?> parameters) {
+ Set<String> values = new HashSet<>()
+ addValues(values, mappedParameters(attributes))
+ addValues(values, parameters)
+ "${method} ${redactPath(path, values)}".toString()
+ }
+
+ private static String requestId(RequestAttributes attributes, AtomicLong requestCounter, String requestIdPrefix) {
+ synchronized (attributes) {
+ Object existing = attributes.getAttribute(REQUEST_ID_ATTRIBUTE, RequestAttributes.SCOPE_REQUEST)
+ if (existing != null) {
+ return existing.toString()
+ }
+ String id = requestIdPrefix + REQUEST_ID_PREFIX + requestCounter.incrementAndGet()
+ attributes.setAttribute(REQUEST_ID_ATTRIBUTE, id, RequestAttributes.SCOPE_REQUEST)
+ return id
+ }
+ }
+
+ private static String page(RequestAttributes attributes) {
+ Object controller = attributes.getAttribute(CONTROLLER_NAME_ATTRIBUTE, RequestAttributes.SCOPE_REQUEST)
+ if (controller == null) {
+ return null
+ }
+ Object namespace = attributes.getAttribute(CONTROLLER_NAMESPACE_ATTRIBUTE, RequestAttributes.SCOPE_REQUEST)
+ Object action = attributes.getAttribute(ACTION_NAME_ATTRIBUTE, RequestAttributes.SCOPE_REQUEST)
+ StringBuilder page = new StringBuilder()
+ if (namespace != null) {
+ page.append(namespace).append(':')
+ }
+ page.append(controller)
+ if (action != null) {
+ page.append('/').append(action)
+ }
+ page.toString()
+ }
+
+ /**
+ * @return the parameters of a Grails web request, which include those its URL mapping took from
+ * the path, or null for a request that is not one
+ */
+ private static Map<?, ?> mappedParameters(RequestAttributes attributes) {
+ Method parameters = PARAMETERS.get(attributes.getClass())
+ if (parameters == null) {
+ return null
+ }
+ try {
+ Object value = parameters.invoke(attributes)
+ return value instanceof Map ? (Map<?, ?>) value : null
+ }
+ catch (ReflectiveOperationException | RuntimeException ignored) {
+ return null
+ }
+ }
+
+ private static void addValues(Set<String> values, Map<?, ?> parameters) {
+ if (parameters == null) {
+ return
+ }
+ for (Map.Entry<?, ?> entry in parameters.entrySet()) {
+ if (MAPPING_PARAMETERS.contains(String.valueOf(entry.key))) {
+ continue
+ }
+ Object value = entry.value
+ if (value instanceof CharSequence) {
+ values.add(value.toString())
+ }
+ else if (value instanceof Object[]) {
+ for (Object element in (Object[]) value) {
+ if (element instanceof CharSequence) {
+ values.add(element.toString())
+ }
+ }
+ }
+ }
+ }
+
+ /**
+ * Replaces the values in a path: a segment that is one of the values, with or without an
+ * extension, a run of segments that spells a value holding slashes, and any segment shaped
+ * like a value.
+ */
+ private static String redactPath(String path, Set<String> values) {
+ String[] segments = path.split('/', -1)
Review Comment:
The path is compared while it is still percent-encoded. `/search/secret%40example.com` with `q=secret@example.com` is written unchanged, because the parameter is decoded and the path is not.
Decode each path segment before matching it to a parameter value. Please assert that encoded form does not reach the file.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@grails.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add the query-tail command to find query-heavy pages [grails-core]
Posted by "jdaugherty (via GitHub)" <gi...@apache.org>.
jdaugherty commented on code in PR #16631:
URL: https://github.com/apache/grails-core/pull/16631#discussion_r4401461320
##########
grails-data-mongodb/core/src/main/groovy/org/apache/grails/data/mongodb/core/querytail/MongoCommandRedactor.groovy:
##########
@@ -0,0 +1,596 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * https://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.grails.data.mongodb.core.querytail
+
+import groovy.transform.CompileStatic
+import org.bson.BsonDocument
+import org.bson.BsonValue
+
+import org.apache.grails.datamapping.core.querytail.QueryTailRedactor
+
+/**
+ * Takes the values out of a MongoDB command before it is captured. A command is a document, so
+ * there is nothing to parse: every value becomes {@link QueryTailRedactor#PLACEHOLDER}, and the
+ * keys, which are the options of the command, its operators and the field names of the collection,
+ * are kept, except where a key may be data. The keys of a document stored in a map property, or in
+ * a property or a collection the {@linkplain MongoMappedFields mapping} does not know, are recorded
+ * as the placeholder too, and so is every segment of a dotted path past such a key. So is a key at
+ * the top of stored data, the documents an insert writes, a replacement document, the fields an
+ * update writes and the operand of a literal, unless the mapping knows it, as the name of a dynamic
+ * attribute may be data; the keys at the top of a condition, a pipeline, a sort or a projection
+ * are the fields the application asks for, and are kept. An update that is a pipeline writes the
+ * documents its stages build: the keys at the top of what a {@code $set}, {@code $addFields} or
+ * {@code $project} stage holds, of the new root of a {@code $replaceWith} or {@code $replaceRoot}, and of a
+ * document {@code $mergeObjects} merges into it are fields written, kept only when the mapping knows
+ * them, as those of a classic update are. A numeric key
+ * is kept only where it indexes an array the mapping knows; in a map, in an element of an array,
+ * which is a map or a document unless the field is a list of lists, or at the top of a document,
+ * it is data. A {@code $} key is kept as an operator only where operators go, in a condition, a
+ * pipeline and the operators of an update, and in a condition only when it is one of MongoDB's query
+ * operators: any other, such as a key beside the geometry of {@code $near}, is recorded as the
+ * placeholder, and so is everything under it; in stored data, the documents an insert writes, a
+ * replacement document and the values an update operator writes, it is a field name, which MongoDB
+ * allows to start with {@code $}, and is recorded as the placeholder unless the mapping knows it.
+ * So is a {@code $} key in the operand of a comparison in a condition, such as {@code $eq} or
+ * {@code $in}, or an implicit document or array equality, which holds the value the field is compared
+ * with, and in a DBRef, whose {@code $ref}, {@code $id} and {@code $db} a condition reads as fields
+ * compared with their whole value, in any order; in an aggregation expression, which
+ * {@code $expr}, the {@code let} of a command and the stages that compute fields hold, the operands
+ * of a comparison are expressions, and their operators are kept, except in the operand of a
+ * {@code $literal} or {@code $const}, which is stored data. The keys of the command itself are its
+ * options, and are kept, but a document under an option is read only where the option is known to
+ * hold one: the fields, a condition, a pipeline, an update or stored data, or options in turn, as
+ * the concerns, the collation and the cursor of a command, the statements of a write, the indexes it
+ * builds and the operations of {@code applyOps} do. Any other option, as a comment, the constants of
+ * an update, the scope of a map-reduce command and the document an operation of {@code applyOps}
+ * applies are, may hold any document, and none of its keys are kept; nor are those of a
+ * {@code $comment}. The entries the driver adds for its
+ * session, the cluster time and the transaction, which say nothing about the command, are left out.
+ *
+ * <p>An array is recorded with each of its different elements once: an element recorded as one before
+ * it in the array is left out, so that the documents of a batch and the values of an {@code $in} are
+ * recorded the same whatever their number, one or many, and a command is counted as the one statement
+ * it is: {@code {"$in": [?]}}. An empty array is recorded as {@code []}.</p>
+ *
+ * <p>A command is recorded as its name, the database and collection it runs against, and the
+ * document of its remaining entries: {@code find shop.book {"filter": {"title": ?}, "limit": ?}}.
+ * The collection is taken only from a command that runs against one; the first value of any other
+ * command is an argument, such as a user name, and is recorded as a value:
+ * {@code dropUser admin {"dropUser": ?}}.</p>
+ *
+ * @since 9.0
+ */
+@CompileStatic
+final class MongoCommandRedactor {
+
+ private static final String PLACEHOLDER = QueryTailRedactor.PLACEHOLDER
+
+ private static final String DATABASE = '$db'
+
+ private static final String COLLECTION = 'collection'
+
+ /**
+ * The commands whose first value is the collection they run against. The first value of any other
+ * command, such as the user {@code dropUser} drops, is an argument, and is recorded as a value.
+ */
+ private static final Set<String> COLLECTION_COMMANDS = ['find', 'insert', 'update', 'delete', 'aggregate',
+ 'count', 'distinct', 'findAndModify', 'findandmodify', 'mapReduce', 'mapreduce', 'killCursors',
+ 'create', 'drop', 'collMod', 'createIndexes', 'dropIndexes', 'deleteIndexes', 'listIndexes', 'reIndex',
+ 'compact', 'validate', 'collStats', 'convertToCapped', 'cloneCollectionAsCapped', 'analyze',
+ 'checkMetadataConsistency', 'createSearchIndexes', 'dropSearchIndex', 'updateSearchIndex',
+ 'planCacheClear', 'planCacheClearFilters', 'planCacheListFilters', 'planCacheSetFilter'] as Set<String>
+
+ /** The command that names its collection in its {@code collection} option, as it continues a cursor. */
+ private static final String GET_MORE = 'getMore'
+
+ /** The entries the driver adds, which describe its session and transaction rather than the command. */
+ private static final Set<String> LEFT_OUT = ['lsid', '$clusterTime', 'txnNumber', 'autocommit',
+ 'startTransaction', '$readPreference', 'apiVersion', 'apiStrict', 'apiDeprecationErrors'] as Set<String>
+
+ /**
+ * The options whose document, or documents, are keyed by the fields of the collection, wherever they appear,
+ * including those of the operations of a client-level {@code bulkWrite}.
+ */
+ private static final Set<String> FIELD_KEYED = ['filter', 'query', 'q', 'u', 'update', 'document', 'documents',
+ 'updateMods', 'pipeline', 'sort', 'projection', 'fields', 'key', 'hint', 'min', 'max', 'arrayFilters',
+ 'partialFilterExpression', 'wildcardProjection', 'validator', 'weights'] as Set<String>
+
+ /** Documents that match fields, whose field values may be implicit equality operands. */
+ private static final Set<String> CONDITION_KEYED = ['filter', 'query', 'q', 'arrayFilters',
+ 'partialFilterExpression', 'validator', 'restrictSearchWithMatch'] as Set<String>
+
+ /** The options whose documents are stored data: what an insert writes. */
+ private static final Set<String> DATA_KEYED = ['documents', 'document'] as Set<String>
+
+ /**
+ * The options whose documents, or the documents of whose arrays, hold options in turn: the concerns and the
+ * collation of a command, the cursor it opens, the statements of a write, the indexes it builds, the operations
+ * of a client-level {@code bulkWrite} and of {@code applyOps} and their preconditions, where a map-reduce
+ * command writes, the command {@code explain} runs, the options of a collection {@code create} builds, and the
+ * sessions {@code endSessions} ends. A
+ * document under any other option may hold any document, as a comment, the constants of an update, the scope
+ * of a map-reduce command and the document an operation of {@code applyOps} applies do, and none of its keys
+ * are kept.
+ */
+ private static final Set<String> OPTION_KEYED = ['writeConcern', 'readConcern', 'collation', 'cursor', 'updates',
+ 'deletes', 'indexes', 'ops', 'nsInfo', 'applyOps', 'preCondition', 'out', 'explain', 'timeseries',
+ 'clusteredIndex', 'changeStreamPreAndPostImages', 'endSessions'] as Set<String>
+
+ /** The operator of a condition that may hold any document, none of whose keys say what the command does. */
+ private static final String COMMENT = '$comment'
+
+ /** The option whose values are aggregation expressions. */
+ private static final String LET = 'let'
+
+ /** The options that hold an update: update operators, a replacement document or a pipeline. */
+ private static final Set<String> UPDATE_KEYED = ['u', 'update', 'updateMods'] as Set<String>
+
+ /** The update operators whose fields take a condition rather than the value to write. */
+ private static final Set<String> CONDITION_OPERATORS = ['$pull', '$currentDate'] as Set<String>
+
+ /** The update operators that add to an array, whose value may hold modifiers. */
+ private static final Set<String> PUSH_OPERATORS = ['$push', '$addToSet'] as Set<String>
+
+ private static final String EACH = '$each'
+
+ /** The modifiers of a {@code $push} or an {@code $addToSet}. */
+ private static final Set<String> PUSH_MODIFIERS = [EACH, '$slice', '$sort', '$position'] as Set<String>
+
+ /**
+ * The operators of a pipeline whose operand is literal data rather than an expression, including
+ * {@code $const}, which MongoDB accepts for {@code $literal}.
+ */
+ private static final Set<String> LITERAL_OPERATORS = ['$literal', '$const', '$documents'] as Set<String>
+
+ /**
+ * The fields of a DBRef, which a condition reads as fields compared with their whole value, never as
+ * operators, in any order and wherever a condition has fields.
+ */
+ private static final Set<String> DBREF_FIELDS = ['$ref', '$id', '$db'] as Set<String>
+
+ /**
+ * The query operators whose operand holds values rather than a condition: the comparisons, whose operand
+ * is the value a field is compared with, a JSON schema, whose {@code enum} lists values, and a
+ * GeoJSON geometry, which can carry additional data alongside its type and coordinates.
+ */
+ private static final Set<String> VALUE_OPERATORS = ['$eq', '$ne', '$gt', '$gte', '$lt', '$lte', '$in', '$nin',
+ '$all', '$jsonSchema', '$geometry'] as Set<String>
+
+ /**
+ * The operators of a condition, the only {@code $} keys a condition holds but the fields of a DBRef: the query
+ * operators of MongoDB, including {@code $within}, the older name of {@code $geoWithin}, which GORM writes for a
+ * box, a polygon or a circle, and the options of {@code $regex}, {@code $text} and the geospatial ones.
+ */
+ private static final Set<String> QUERY_OPERATORS = ['$eq', '$ne', '$gt', '$gte', '$lt', '$lte', '$in', '$nin',
+ '$and', '$or', '$not', '$nor', '$exists', '$type', '$expr', '$jsonSchema', '$mod', '$regex', '$options',
+ '$text', '$search', '$language', '$caseSensitive', '$diacriticSensitive', '$where', '$geoIntersects',
+ '$geoWithin', '$within', '$near', '$nearSphere', '$geometry', '$maxDistance', '$minDistance', '$box', '$center',
+ '$centerSphere', '$polygon', '$uniqueDocs', '$all', '$elemMatch', '$size', '$bitsAllClear', '$bitsAllSet',
+ '$bitsAnyClear', '$bitsAnySet', '$comment', '$rand', '$sampleRate'] as Set<String>
+
+ /** The operator of a condition, and the stages of a pipeline, whose operand is an aggregation expression. */
+ private static final Set<String> EXPRESSION_OPERATORS = ['$expr', '$addFields', '$set', '$project', '$group',
+ '$replaceRoot', '$replaceWith', '$redact', '$bucket', '$bucketAuto', '$sortByCount', '$setWindowFields',
+ '$fill'] as Set<String>
+
+ /** The stages of an update pipeline whose operand is the document they write, or the new root that replaces it. */
+ private static final Set<String> WRITING_STAGES = ['$set', '$addFields', '$project', '$replaceWith'] as Set<String>
+
+ /** The stage of an update pipeline whose options name the new root that replaces the document. */
+ private static final String REPLACE_ROOT = '$replaceRoot'
+
+ /** The option of {@code $replaceRoot} that holds the new root. */
+ private static final String NEW_ROOT = 'newRoot'
+
+ /** The operator whose operands are documents merged into one, which in an update pipeline is written. */
+ private static final String MERGE_OBJECTS = '$mergeObjects'
+
+ /** The kinds of document whose keys at the top are stored data: what an insert, a replacement or an update writes. */
+ private static final Set<Kind> STORED = EnumSet.of(Kind.DATA, Kind.UPDATE, Kind.UPDATE_FIELDS, Kind.WRITTEN)
+
+ private MongoCommandRedactor() {
+ }
+
+ /**
+ * Redacts a command.
+ *
+ * @param database the database the command runs against
+ * @param commandName the name of the command, which is its first key
+ * @param command the command as the driver sends it
+ * @param fields the fields the mapping of the datastore knows, or null before the datastore is built
+ * @return the command with its values replaced
+ */
+ static String redact(String database, String commandName, BsonDocument command, MongoMappedFields fields) {
+ String collection = null
+ String collectionKey = null
+ BsonValue first = command.get(commandName)
+ if (first != null && first.isString() && COLLECTION_COMMANDS.contains(commandName)) {
+ collection = first.asString().value
+ collectionKey = commandName
+ }
+ else if (commandName == GET_MORE && command.get(COLLECTION)?.isString()) {
+ collection = command.getString(COLLECTION).value
+ collectionKey = COLLECTION
+ }
+ StringBuilder out = new StringBuilder(commandName)
+ if (database != null || collection != null) {
+ out.append(' ')
+ if (database != null) {
+ out.append(database)
+ }
+ if (collection != null) {
+ out.append(database != null ? '.' : '').append(collection)
+ }
+ }
+ Scope options = Scope.options(collection, fields)
+ StringBuilder body = new StringBuilder()
+ for (Map.Entry<String, BsonValue> entry in command.entrySet()) {
+ String key = entry.key
+ if (key == DATABASE || key == collectionKey || LEFT_OUT.contains(key)) {
+ continue
+ }
+ body.append(body.length() == 0 ? '{' : ', ')
+ appendEntry(body, key, entry.value, options)
+ }
+ if (body.length() > 0) {
+ out.append(' ').append(body).append('}')
+ }
+ out.toString()
+ }
+
+ private static void appendEntry(StringBuilder out, String key, BsonValue value, Scope scope) {
+ Scope child = scope.of(key, value)
+ out.append(child.keyText).append(': ')
+ appendValue(out, value, child)
+ }
+
+ private static void appendValue(StringBuilder out, BsonValue value, Scope scope) {
+ if (value.isDocument()) {
+ out.append('{')
+ boolean first = true
+ for (Map.Entry<String, BsonValue> entry in value.asDocument().entrySet()) {
+ if (!first) {
+ out.append(', ')
+ }
+ first = false
+ appendEntry(out, entry.key, entry.value, scope)
+ }
+ out.append('}')
+ }
+ else if (value.isArray()) {
+ out.append('[')
+ Scope element = scope.element()
+ // a value is recorded as the placeholder whatever it is, so only a document or an array is rendered
+ // to be told apart from the elements before it, and a list of values costs no more than a look at each
+ boolean placeholder = false
+ Set<String> written = null
+ boolean first = true
+ for (BsonValue item in value.asArray()) {
+ String recorded
+ if (item.isDocument() || item.isArray()) {
+ StringBuilder text = new StringBuilder()
+ appendValue(text, item, element)
+ recorded = text.toString()
+ if (written == null) {
+ written = new HashSet<>()
+ }
+ if (!written.add(recorded)) {
+ continue
+ }
+ }
+ else if (placeholder) {
+ continue
+ }
+ else {
+ placeholder = true
+ recorded = PLACEHOLDER
+ }
+ if (!first) {
+ out.append(', ')
+ }
+ first = false
+ out.append(recorded)
+ }
+ out.append(']')
+ }
+ else {
+ out.append(PLACEHOLDER)
+ }
+ }
+
+ private static String quote(String key) {
+ StringBuilder out = new StringBuilder(key.length() + 2).append('"')
+ for (int i = 0; i < key.length(); i++) {
+ char c = key.charAt(i)
+ if (c == '"' || c == '\\') {
+ out.append('\\').append(c)
+ }
+ else if (c < 0x20) {
+ out.append(String.format('\\u%04x', (int) c))
+ }
+ else {
+ out.append(c)
+ }
+ }
+ out.append('"').toString()
+ }
+
+ /**
+ * Where a document sits in the command, which decides what its keys are: the options of the
+ * command, kept; the fields and operators of a condition, a pipeline, a sort or a projection, where
+ * a {@code $} key is an operator; an aggregation expression, whose {@code $} keys are operators
+ * too; an update, whose {@code $} keys are update operators; or stored data, the documents an
+ * insert writes, the values an update writes and the values a condition compares with, where a
+ * {@code $} key is a field name like any other and is kept only when the mapping knows it.
+ */
+ @CompileStatic
+ private static final class Scope {
+
+ final String collection
+ final MongoMappedFields fields
+ final Kind kind
+ /**
+ * The update operator whose fields a document of kind {@link Kind#UPDATE_FIELDS} holds, or the stage
+ * whose document one of kind {@link Kind#WRITTEN} is.
+ */
+ final String operator
+ /** The field path of the document; null in a document keyed by fields the mapping does not know. */
+ final List<String> path
+ /** How many indices into the field at the path have been taken: 0 at the field, 1 in one of its elements. */
+ final int depth
+ final String keyText
+
+ private Scope(String collection, MongoMappedFields fields, Kind kind, String operator, List<String> path,
+ int depth, String keyText) {
+ this.collection = collection
+ this.fields = fields
+ this.kind = kind
+ this.operator = operator
+ this.path = path
+ this.depth = depth
+ this.keyText = keyText
+ }
+
+ static Scope options(String collection, MongoMappedFields fields) {
+ new Scope(collection, fields, Kind.OPTIONS, null, Collections.<String>emptyList(), 0, null)
+ }
+
+ /**
+ * @return the scope of an element of an array in this scope: one index deeper into the field
+ * at the path, so that a numeric key inside the element is data unless the elements are arrays;
+ * the elements of an update that is an array are the stages of a pipeline, and an element of an
+ * expression at the top of a document, as the operands of an operator {@code $expr} holds are, is a
+ * value the expression builds, whose keys are kept only when the mapping knows them, as those of a
+ * document a stage of an update pipeline writes are
+ */
+ Scope element() {
+ if (kind == Kind.UPDATE) {
+ return child(Kind.UPDATE_PIPELINE, null, path, depth, keyText)
+ }
+ if (kind == Kind.PUSH) {
+ return child(Kind.DATA, null, path, depth + 2, keyText)
+ }
+ if ((kind == Kind.QUERY || kind == Kind.CONDITION || kind == Kind.EXPRESSION || kind == Kind.DATA) && path != null && !path.isEmpty()) {
+ return child(kind, null, path, depth + 1, keyText)
+ }
+ if (kind == Kind.EXPRESSION && path != null) {
+ return child(Kind.WRITTEN, null, path, depth, keyText)
+ }
+ this
+ }
+
+ /**
+ * @return the scope of the value of a key in a document of this scope, with the key as it is recorded
+ */
+ Scope of(String key, BsonValue value) {
+ if (kind == Kind.OPTIONS) {
+ Kind valueKind = DATA_KEYED.contains(key) ? Kind.DATA :
+ UPDATE_KEYED.contains(key) ? Kind.UPDATE :
+ CONDITION_KEYED.contains(key) ? Kind.CONDITION :
+ FIELD_KEYED.contains(key) ? Kind.QUERY :
+ key == LET ? Kind.EXPRESSION :
+ OPTION_KEYED.contains(key) ? Kind.OPTIONS : null
+ if (valueKind == null) {
+ // an option not known to hold options, fields or a pipeline, whose document may be any data
+ return child(Kind.DATA, null, null, 0, quote(key))
+ }
+ return child(valueKind, null, Collections.<String>emptyList(), 0, quote(key))
+ }
+ if ((kind == Kind.QUERY || kind == Kind.CONDITION) && key.startsWith('$')) {
+ if (key == COMMENT) {
+ return child(Kind.DATA, null, null, 0, quote(key))
+ }
+ if (kind == Kind.CONDITION && DBREF_FIELDS.contains(key)) {
+ return field(key, Kind.DATA)
+ }
+ if (kind == Kind.CONDITION && !QUERY_OPERATORS.contains(key)) {
+ // no operator of a condition, as a key beside the geometry of $near is not, so data the server rejects
+ return child(Kind.DATA, null, null, 0, PLACEHOLDER)
+ }
+ if (EXPRESSION_OPERATORS.contains(key)) {
+ return child(Kind.EXPRESSION, null, path, depth, quote(key))
+ }
+ if (key == '$match') {
+ return child(Kind.CONDITION, null, path, depth, quote(key))
+ }
+ // an operator on an array, such as $elemMatch, describes its elements
+ int operandDepth = indexed(path, depth) ? depth + 1 : depth
+ Kind operandKind = LITERAL_OPERATORS.contains(key) || VALUE_OPERATORS.contains(key) ? Kind.DATA : kind
Review Comment:
Fixed in 77bd7c097f. The operand of `$elemMatch` is now a condition wherever it appears, in a projection (`projection` or `findAndModify` `fields`) as under a filter. Your example now records `"projection": {"attributes": {"$elemMatch": {?: {?: ?, ?: ?}}}}`, and an unknown `$` key directly inside a projection `$elemMatch` is recorded as `?` like one in a filter. `QueryTailCommandListenerSpec` runs the projection against MongoDB to confirm the matching element comes back and `example-secret` never reaches the capture. The redactor spec shows a filter and a projection `$elemMatch` record the same way. The guide says so too.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@grails.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add the query-tail command to find query-heavy pages [grails-core]
Posted by "jamesfredley (via GitHub)" <gi...@apache.org>.
jamesfredley commented on code in PR #16631:
URL: https://github.com/apache/grails-core/pull/16631#discussion_r4399321387
##########
grails-data-mongodb/core/src/main/groovy/org/apache/grails/data/mongodb/core/querytail/MongoCommandRedactor.groovy:
##########
@@ -0,0 +1,596 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * https://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.grails.data.mongodb.core.querytail
+
+import groovy.transform.CompileStatic
+import org.bson.BsonDocument
+import org.bson.BsonValue
+
+import org.apache.grails.datamapping.core.querytail.QueryTailRedactor
+
+/**
+ * Takes the values out of a MongoDB command before it is captured. A command is a document, so
+ * there is nothing to parse: every value becomes {@link QueryTailRedactor#PLACEHOLDER}, and the
+ * keys, which are the options of the command, its operators and the field names of the collection,
+ * are kept, except where a key may be data. The keys of a document stored in a map property, or in
+ * a property or a collection the {@linkplain MongoMappedFields mapping} does not know, are recorded
+ * as the placeholder too, and so is every segment of a dotted path past such a key. So is a key at
+ * the top of stored data, the documents an insert writes, a replacement document, the fields an
+ * update writes and the operand of a literal, unless the mapping knows it, as the name of a dynamic
+ * attribute may be data; the keys at the top of a condition, a pipeline, a sort or a projection
+ * are the fields the application asks for, and are kept. An update that is a pipeline writes the
+ * documents its stages build: the keys at the top of what a {@code $set}, {@code $addFields} or
+ * {@code $project} stage holds, of the new root of a {@code $replaceWith} or {@code $replaceRoot}, and of a
+ * document {@code $mergeObjects} merges into it are fields written, kept only when the mapping knows
+ * them, as those of a classic update are. A numeric key
+ * is kept only where it indexes an array the mapping knows; in a map, in an element of an array,
+ * which is a map or a document unless the field is a list of lists, or at the top of a document,
+ * it is data. A {@code $} key is kept as an operator only where operators go, in a condition, a
+ * pipeline and the operators of an update, and in a condition only when it is one of MongoDB's query
+ * operators: any other, such as a key beside the geometry of {@code $near}, is recorded as the
+ * placeholder, and so is everything under it; in stored data, the documents an insert writes, a
+ * replacement document and the values an update operator writes, it is a field name, which MongoDB
+ * allows to start with {@code $}, and is recorded as the placeholder unless the mapping knows it.
+ * So is a {@code $} key in the operand of a comparison in a condition, such as {@code $eq} or
+ * {@code $in}, or an implicit document or array equality, which holds the value the field is compared
+ * with, and in a DBRef, whose {@code $ref}, {@code $id} and {@code $db} a condition reads as fields
+ * compared with their whole value, in any order; in an aggregation expression, which
+ * {@code $expr}, the {@code let} of a command and the stages that compute fields hold, the operands
+ * of a comparison are expressions, and their operators are kept, except in the operand of a
+ * {@code $literal} or {@code $const}, which is stored data. The keys of the command itself are its
+ * options, and are kept, but a document under an option is read only where the option is known to
+ * hold one: the fields, a condition, a pipeline, an update or stored data, or options in turn, as
+ * the concerns, the collation and the cursor of a command, the statements of a write, the indexes it
+ * builds and the operations of {@code applyOps} do. Any other option, as a comment, the constants of
+ * an update, the scope of a map-reduce command and the document an operation of {@code applyOps}
+ * applies are, may hold any document, and none of its keys are kept; nor are those of a
+ * {@code $comment}. The entries the driver adds for its
+ * session, the cluster time and the transaction, which say nothing about the command, are left out.
+ *
+ * <p>An array is recorded with each of its different elements once: an element recorded as one before
+ * it in the array is left out, so that the documents of a batch and the values of an {@code $in} are
+ * recorded the same whatever their number, one or many, and a command is counted as the one statement
+ * it is: {@code {"$in": [?]}}. An empty array is recorded as {@code []}.</p>
+ *
+ * <p>A command is recorded as its name, the database and collection it runs against, and the
+ * document of its remaining entries: {@code find shop.book {"filter": {"title": ?}, "limit": ?}}.
+ * The collection is taken only from a command that runs against one; the first value of any other
+ * command is an argument, such as a user name, and is recorded as a value:
+ * {@code dropUser admin {"dropUser": ?}}.</p>
+ *
+ * @since 9.0
+ */
+@CompileStatic
+final class MongoCommandRedactor {
+
+ private static final String PLACEHOLDER = QueryTailRedactor.PLACEHOLDER
+
+ private static final String DATABASE = '$db'
+
+ private static final String COLLECTION = 'collection'
+
+ /**
+ * The commands whose first value is the collection they run against. The first value of any other
+ * command, such as the user {@code dropUser} drops, is an argument, and is recorded as a value.
+ */
+ private static final Set<String> COLLECTION_COMMANDS = ['find', 'insert', 'update', 'delete', 'aggregate',
+ 'count', 'distinct', 'findAndModify', 'findandmodify', 'mapReduce', 'mapreduce', 'killCursors',
+ 'create', 'drop', 'collMod', 'createIndexes', 'dropIndexes', 'deleteIndexes', 'listIndexes', 'reIndex',
+ 'compact', 'validate', 'collStats', 'convertToCapped', 'cloneCollectionAsCapped', 'analyze',
+ 'checkMetadataConsistency', 'createSearchIndexes', 'dropSearchIndex', 'updateSearchIndex',
+ 'planCacheClear', 'planCacheClearFilters', 'planCacheListFilters', 'planCacheSetFilter'] as Set<String>
+
+ /** The command that names its collection in its {@code collection} option, as it continues a cursor. */
+ private static final String GET_MORE = 'getMore'
+
+ /** The entries the driver adds, which describe its session and transaction rather than the command. */
+ private static final Set<String> LEFT_OUT = ['lsid', '$clusterTime', 'txnNumber', 'autocommit',
+ 'startTransaction', '$readPreference', 'apiVersion', 'apiStrict', 'apiDeprecationErrors'] as Set<String>
+
+ /**
+ * The options whose document, or documents, are keyed by the fields of the collection, wherever they appear,
+ * including those of the operations of a client-level {@code bulkWrite}.
+ */
+ private static final Set<String> FIELD_KEYED = ['filter', 'query', 'q', 'u', 'update', 'document', 'documents',
+ 'updateMods', 'pipeline', 'sort', 'projection', 'fields', 'key', 'hint', 'min', 'max', 'arrayFilters',
+ 'partialFilterExpression', 'wildcardProjection', 'validator', 'weights'] as Set<String>
+
+ /** Documents that match fields, whose field values may be implicit equality operands. */
+ private static final Set<String> CONDITION_KEYED = ['filter', 'query', 'q', 'arrayFilters',
+ 'partialFilterExpression', 'validator', 'restrictSearchWithMatch'] as Set<String>
+
+ /** The options whose documents are stored data: what an insert writes. */
+ private static final Set<String> DATA_KEYED = ['documents', 'document'] as Set<String>
+
+ /**
+ * The options whose documents, or the documents of whose arrays, hold options in turn: the concerns and the
+ * collation of a command, the cursor it opens, the statements of a write, the indexes it builds, the operations
+ * of a client-level {@code bulkWrite} and of {@code applyOps} and their preconditions, where a map-reduce
+ * command writes, the command {@code explain} runs, the options of a collection {@code create} builds, and the
+ * sessions {@code endSessions} ends. A
+ * document under any other option may hold any document, as a comment, the constants of an update, the scope
+ * of a map-reduce command and the document an operation of {@code applyOps} applies do, and none of its keys
+ * are kept.
+ */
+ private static final Set<String> OPTION_KEYED = ['writeConcern', 'readConcern', 'collation', 'cursor', 'updates',
+ 'deletes', 'indexes', 'ops', 'nsInfo', 'applyOps', 'preCondition', 'out', 'explain', 'timeseries',
+ 'clusteredIndex', 'changeStreamPreAndPostImages', 'endSessions'] as Set<String>
+
+ /** The operator of a condition that may hold any document, none of whose keys say what the command does. */
+ private static final String COMMENT = '$comment'
+
+ /** The option whose values are aggregation expressions. */
+ private static final String LET = 'let'
+
+ /** The options that hold an update: update operators, a replacement document or a pipeline. */
+ private static final Set<String> UPDATE_KEYED = ['u', 'update', 'updateMods'] as Set<String>
+
+ /** The update operators whose fields take a condition rather than the value to write. */
+ private static final Set<String> CONDITION_OPERATORS = ['$pull', '$currentDate'] as Set<String>
+
+ /** The update operators that add to an array, whose value may hold modifiers. */
+ private static final Set<String> PUSH_OPERATORS = ['$push', '$addToSet'] as Set<String>
+
+ private static final String EACH = '$each'
+
+ /** The modifiers of a {@code $push} or an {@code $addToSet}. */
+ private static final Set<String> PUSH_MODIFIERS = [EACH, '$slice', '$sort', '$position'] as Set<String>
+
+ /**
+ * The operators of a pipeline whose operand is literal data rather than an expression, including
+ * {@code $const}, which MongoDB accepts for {@code $literal}.
+ */
+ private static final Set<String> LITERAL_OPERATORS = ['$literal', '$const', '$documents'] as Set<String>
+
+ /**
+ * The fields of a DBRef, which a condition reads as fields compared with their whole value, never as
+ * operators, in any order and wherever a condition has fields.
+ */
+ private static final Set<String> DBREF_FIELDS = ['$ref', '$id', '$db'] as Set<String>
+
+ /**
+ * The query operators whose operand holds values rather than a condition: the comparisons, whose operand
+ * is the value a field is compared with, a JSON schema, whose {@code enum} lists values, and a
+ * GeoJSON geometry, which can carry additional data alongside its type and coordinates.
+ */
+ private static final Set<String> VALUE_OPERATORS = ['$eq', '$ne', '$gt', '$gte', '$lt', '$lte', '$in', '$nin',
+ '$all', '$jsonSchema', '$geometry'] as Set<String>
+
+ /**
+ * The operators of a condition, the only {@code $} keys a condition holds but the fields of a DBRef: the query
+ * operators of MongoDB, including {@code $within}, the older name of {@code $geoWithin}, which GORM writes for a
+ * box, a polygon or a circle, and the options of {@code $regex}, {@code $text} and the geospatial ones.
+ */
+ private static final Set<String> QUERY_OPERATORS = ['$eq', '$ne', '$gt', '$gte', '$lt', '$lte', '$in', '$nin',
+ '$and', '$or', '$not', '$nor', '$exists', '$type', '$expr', '$jsonSchema', '$mod', '$regex', '$options',
+ '$text', '$search', '$language', '$caseSensitive', '$diacriticSensitive', '$where', '$geoIntersects',
+ '$geoWithin', '$within', '$near', '$nearSphere', '$geometry', '$maxDistance', '$minDistance', '$box', '$center',
+ '$centerSphere', '$polygon', '$uniqueDocs', '$all', '$elemMatch', '$size', '$bitsAllClear', '$bitsAllSet',
+ '$bitsAnyClear', '$bitsAnySet', '$comment', '$rand', '$sampleRate'] as Set<String>
+
+ /** The operator of a condition, and the stages of a pipeline, whose operand is an aggregation expression. */
+ private static final Set<String> EXPRESSION_OPERATORS = ['$expr', '$addFields', '$set', '$project', '$group',
+ '$replaceRoot', '$replaceWith', '$redact', '$bucket', '$bucketAuto', '$sortByCount', '$setWindowFields',
+ '$fill'] as Set<String>
+
+ /** The stages of an update pipeline whose operand is the document they write, or the new root that replaces it. */
+ private static final Set<String> WRITING_STAGES = ['$set', '$addFields', '$project', '$replaceWith'] as Set<String>
+
+ /** The stage of an update pipeline whose options name the new root that replaces the document. */
+ private static final String REPLACE_ROOT = '$replaceRoot'
+
+ /** The option of {@code $replaceRoot} that holds the new root. */
+ private static final String NEW_ROOT = 'newRoot'
+
+ /** The operator whose operands are documents merged into one, which in an update pipeline is written. */
+ private static final String MERGE_OBJECTS = '$mergeObjects'
+
+ /** The kinds of document whose keys at the top are stored data: what an insert, a replacement or an update writes. */
+ private static final Set<Kind> STORED = EnumSet.of(Kind.DATA, Kind.UPDATE, Kind.UPDATE_FIELDS, Kind.WRITTEN)
+
+ private MongoCommandRedactor() {
+ }
+
+ /**
+ * Redacts a command.
+ *
+ * @param database the database the command runs against
+ * @param commandName the name of the command, which is its first key
+ * @param command the command as the driver sends it
+ * @param fields the fields the mapping of the datastore knows, or null before the datastore is built
+ * @return the command with its values replaced
+ */
+ static String redact(String database, String commandName, BsonDocument command, MongoMappedFields fields) {
+ String collection = null
+ String collectionKey = null
+ BsonValue first = command.get(commandName)
+ if (first != null && first.isString() && COLLECTION_COMMANDS.contains(commandName)) {
+ collection = first.asString().value
+ collectionKey = commandName
+ }
+ else if (commandName == GET_MORE && command.get(COLLECTION)?.isString()) {
+ collection = command.getString(COLLECTION).value
+ collectionKey = COLLECTION
+ }
+ StringBuilder out = new StringBuilder(commandName)
+ if (database != null || collection != null) {
+ out.append(' ')
+ if (database != null) {
+ out.append(database)
+ }
+ if (collection != null) {
+ out.append(database != null ? '.' : '').append(collection)
+ }
+ }
+ Scope options = Scope.options(collection, fields)
+ StringBuilder body = new StringBuilder()
+ for (Map.Entry<String, BsonValue> entry in command.entrySet()) {
+ String key = entry.key
+ if (key == DATABASE || key == collectionKey || LEFT_OUT.contains(key)) {
+ continue
+ }
+ body.append(body.length() == 0 ? '{' : ', ')
+ appendEntry(body, key, entry.value, options)
+ }
+ if (body.length() > 0) {
+ out.append(' ').append(body).append('}')
+ }
+ out.toString()
+ }
+
+ private static void appendEntry(StringBuilder out, String key, BsonValue value, Scope scope) {
+ Scope child = scope.of(key, value)
+ out.append(child.keyText).append(': ')
+ appendValue(out, value, child)
+ }
+
+ private static void appendValue(StringBuilder out, BsonValue value, Scope scope) {
+ if (value.isDocument()) {
+ out.append('{')
+ boolean first = true
+ for (Map.Entry<String, BsonValue> entry in value.asDocument().entrySet()) {
+ if (!first) {
+ out.append(', ')
+ }
+ first = false
+ appendEntry(out, entry.key, entry.value, scope)
+ }
+ out.append('}')
+ }
+ else if (value.isArray()) {
+ out.append('[')
+ Scope element = scope.element()
+ // a value is recorded as the placeholder whatever it is, so only a document or an array is rendered
+ // to be told apart from the elements before it, and a list of values costs no more than a look at each
+ boolean placeholder = false
+ Set<String> written = null
+ boolean first = true
+ for (BsonValue item in value.asArray()) {
+ String recorded
+ if (item.isDocument() || item.isArray()) {
+ StringBuilder text = new StringBuilder()
+ appendValue(text, item, element)
+ recorded = text.toString()
+ if (written == null) {
+ written = new HashSet<>()
+ }
+ if (!written.add(recorded)) {
+ continue
+ }
+ }
+ else if (placeholder) {
+ continue
+ }
+ else {
+ placeholder = true
+ recorded = PLACEHOLDER
+ }
+ if (!first) {
+ out.append(', ')
+ }
+ first = false
+ out.append(recorded)
+ }
+ out.append(']')
+ }
+ else {
+ out.append(PLACEHOLDER)
+ }
+ }
+
+ private static String quote(String key) {
+ StringBuilder out = new StringBuilder(key.length() + 2).append('"')
+ for (int i = 0; i < key.length(); i++) {
+ char c = key.charAt(i)
+ if (c == '"' || c == '\\') {
+ out.append('\\').append(c)
+ }
+ else if (c < 0x20) {
+ out.append(String.format('\\u%04x', (int) c))
+ }
+ else {
+ out.append(c)
+ }
+ }
+ out.append('"').toString()
+ }
+
+ /**
+ * Where a document sits in the command, which decides what its keys are: the options of the
+ * command, kept; the fields and operators of a condition, a pipeline, a sort or a projection, where
+ * a {@code $} key is an operator; an aggregation expression, whose {@code $} keys are operators
+ * too; an update, whose {@code $} keys are update operators; or stored data, the documents an
+ * insert writes, the values an update writes and the values a condition compares with, where a
+ * {@code $} key is a field name like any other and is kept only when the mapping knows it.
+ */
+ @CompileStatic
+ private static final class Scope {
+
+ final String collection
+ final MongoMappedFields fields
+ final Kind kind
+ /**
+ * The update operator whose fields a document of kind {@link Kind#UPDATE_FIELDS} holds, or the stage
+ * whose document one of kind {@link Kind#WRITTEN} is.
+ */
+ final String operator
+ /** The field path of the document; null in a document keyed by fields the mapping does not know. */
+ final List<String> path
+ /** How many indices into the field at the path have been taken: 0 at the field, 1 in one of its elements. */
+ final int depth
+ final String keyText
+
+ private Scope(String collection, MongoMappedFields fields, Kind kind, String operator, List<String> path,
+ int depth, String keyText) {
+ this.collection = collection
+ this.fields = fields
+ this.kind = kind
+ this.operator = operator
+ this.path = path
+ this.depth = depth
+ this.keyText = keyText
+ }
+
+ static Scope options(String collection, MongoMappedFields fields) {
+ new Scope(collection, fields, Kind.OPTIONS, null, Collections.<String>emptyList(), 0, null)
+ }
+
+ /**
+ * @return the scope of an element of an array in this scope: one index deeper into the field
+ * at the path, so that a numeric key inside the element is data unless the elements are arrays;
+ * the elements of an update that is an array are the stages of a pipeline, and an element of an
+ * expression at the top of a document, as the operands of an operator {@code $expr} holds are, is a
+ * value the expression builds, whose keys are kept only when the mapping knows them, as those of a
+ * document a stage of an update pipeline writes are
+ */
+ Scope element() {
+ if (kind == Kind.UPDATE) {
+ return child(Kind.UPDATE_PIPELINE, null, path, depth, keyText)
+ }
+ if (kind == Kind.PUSH) {
+ return child(Kind.DATA, null, path, depth + 2, keyText)
+ }
+ if ((kind == Kind.QUERY || kind == Kind.CONDITION || kind == Kind.EXPRESSION || kind == Kind.DATA) && path != null && !path.isEmpty()) {
+ return child(kind, null, path, depth + 1, keyText)
+ }
+ if (kind == Kind.EXPRESSION && path != null) {
+ return child(Kind.WRITTEN, null, path, depth, keyText)
+ }
+ this
+ }
+
+ /**
+ * @return the scope of the value of a key in a document of this scope, with the key as it is recorded
+ */
+ Scope of(String key, BsonValue value) {
+ if (kind == Kind.OPTIONS) {
+ Kind valueKind = DATA_KEYED.contains(key) ? Kind.DATA :
+ UPDATE_KEYED.contains(key) ? Kind.UPDATE :
+ CONDITION_KEYED.contains(key) ? Kind.CONDITION :
+ FIELD_KEYED.contains(key) ? Kind.QUERY :
+ key == LET ? Kind.EXPRESSION :
+ OPTION_KEYED.contains(key) ? Kind.OPTIONS : null
+ if (valueKind == null) {
+ // an option not known to hold options, fields or a pipeline, whose document may be any data
+ return child(Kind.DATA, null, null, 0, quote(key))
+ }
+ return child(valueKind, null, Collections.<String>emptyList(), 0, quote(key))
+ }
+ if ((kind == Kind.QUERY || kind == Kind.CONDITION) && key.startsWith('$')) {
+ if (key == COMMENT) {
+ return child(Kind.DATA, null, null, 0, quote(key))
+ }
+ if (kind == Kind.CONDITION && DBREF_FIELDS.contains(key)) {
+ return field(key, Kind.DATA)
+ }
+ if (kind == Kind.CONDITION && !QUERY_OPERATORS.contains(key)) {
+ // no operator of a condition, as a key beside the geometry of $near is not, so data the server rejects
+ return child(Kind.DATA, null, null, 0, PLACEHOLDER)
+ }
+ if (EXPRESSION_OPERATORS.contains(key)) {
+ return child(Kind.EXPRESSION, null, path, depth, quote(key))
+ }
+ if (key == '$match') {
+ return child(Kind.CONDITION, null, path, depth, quote(key))
+ }
+ // an operator on an array, such as $elemMatch, describes its elements
+ int operandDepth = indexed(path, depth) ? depth + 1 : depth
+ Kind operandKind = LITERAL_OPERATORS.contains(key) || VALUE_OPERATORS.contains(key) ? Kind.DATA : kind
Review Comment:
A projection $elemMatch stays in query scope, so a dollar-prefixed key in the equality document is kept. find with projection attributes.$elemMatch.nested of {a: 1, $example-secret: v} records $example-secret. The same $elemMatch under filter does not. Treat that operand as a condition.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@grails.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add the query-tail command to find query-heavy pages [grails-core]
Posted by "testlens-app[bot] (via GitHub)" <gi...@apache.org>.
testlens-app[bot] commented on PR #16631:
URL: https://github.com/apache/grails-core/pull/16631#issuecomment-6430070468
## 🚨 All tests passed but jobs failed 🚨
### Failed Jobs without Test Failures
❌ [End to End / End to End Tests \(end-to-end build only\)](https://github.com/apache/grails-core/actions/runs/40418570925/job/121907154553?pr=16631)
🏷️ Commit: 6b133f1c64656f4cec0356a10a5461605b2611b6
▶️ Tests: 77100 executed
🟡 Checks: 47/92 completed
### Rerun Controls
Click the checkbox to trigger a rerun:
🔲 **Rerun jobs**
---
_Learn more about TestLens at [testlens.app/docs](https://testlens.app/docs/features/pr-comment/)._
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@grails.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add the query-tail command to find query-heavy pages [grails-core]
Posted by "testlens-app[bot] (via GitHub)" <gi...@apache.org>.
testlens-app[bot] commented on PR #16631:
URL: https://github.com/apache/grails-core/pull/16631#issuecomment-6415907740
## 🚨 TestLens detected 1 failed test 🚨
### Test Summary
> [!TIP]
> Inspect all details with the [TestLens CLI](https://testlens.app/docs/features/cli): `testlens pr 16631`
#### [CI / Build Grails-Core \(Windows JDK 25 shard 2\)](https://github.com/apache/grails-core/actions/runs/40315621192/job/121555318946?pr=16631) > :grails-startup-progress:test
| Test | Runs | Flakiness |
|---|---|--:|
| StartupProgressSpec > tells a watching page why the start failed while <span>#</span>stage, then releases the port > tells a watching page why the start failed while creating beans, then releases the port | ❌ | 0% 🟢 |
🏷️ Commit: da16a91754467f873fb2b4f4f8ebeff43221def6
▶️ Tests: 30937 executed
🟡 Checks: 40/89 completed
### Test Failures
<details><summary><strong>StartupProgressSpec > tells a watching page why the start failed while <span>#</span>stage, then releases the port > tells a watching page why the start failed while creating beans, then releases the port</strong> (:grails-startup-progress:test in <a href="https://github.com/apache/grails-core/actions/runs/40315621192/job/121555318946?pr=16631">CI / Build Grails-Core (Windows JDK 25 shard 2)</a>)</summary>
```
java.lang.AssertionError: No FAILED status within 30 seconds
at org.apache.grails.startup.StartupProgressSpec.awaitStatus(StartupProgressSpec.groovy:1032)
at org.apache.grails.startup.StartupProgressSpec.tells a watching page why the start failed while #stage, then releases the port(StartupProgressSpec.groovy:319)
```
</details>
### Rerun Controls
> [!NOTE]
> Checks are currently running using the configuration below.
Select tests to mute in this pull request:
🔲 StartupProgressSpec > tells a watching page why the start failed while <span>#</span>stage, then releases the port <!-- uniqueId=[engine:spock]/[spec:org.apache.grails.startup.StartupProgressSpec]/[feature:$spock_feature_0_2] -->
Reuse successful test results:
🔲 ♻️ Only rerun the tests that failed or were muted before
Click the checkbox to trigger a rerun:
🔲 **Rerun jobs**
---
_Learn more about TestLens at [testlens.app/docs](https://testlens.app/docs/features/pr-comment/)._
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@grails.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add the query-tail command to find query-heavy pages [grails-core]
Posted by "jamesfredley (via GitHub)" <gi...@apache.org>.
jamesfredley commented on code in PR #16631:
URL: https://github.com/apache/grails-core/pull/16631#discussion_r4367882558
##########
grails-data-mongodb/core/src/main/groovy/org/apache/grails/data/mongo/core/querytail/MongoCommandRedactor.groovy:
##########
@@ -0,0 +1,241 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * https://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.grails.data.mongo.core.querytail
+
+import groovy.transform.CompileStatic
+import org.bson.BsonDocument
+import org.bson.BsonValue
+
+import org.apache.grails.data.core.querytail.QueryTailRedactor
+
+/**
+ * Takes the values out of a MongoDB command before it is captured. A command is a document, so
+ * there is nothing to parse: every value becomes {@link QueryTailRedactor#PLACEHOLDER}, and the
+ * keys, which are the options of the command, its operators and the field names of the collection,
+ * are kept, except where a key may be data. The keys of a document stored in a map property, or in
+ * a property or a collection the {@linkplain MongoMappedFields mapping} does not know, are recorded
+ * as the placeholder too, and so is every segment of a dotted path past such a key. A numeric key
+ * is kept only where it indexes an array the mapping knows; in a map, or at the top of a document,
+ * it is data. The entries the driver adds for its session, the cluster time and the transaction,
+ * which say nothing about the command, are left out.
+ *
+ * <p>A command is recorded as its name, the database and collection it runs against, and the
+ * document of its remaining entries: {@code find shop.book {"filter": {"title": ?}, "limit": ?}}.</p>
+ *
+ * @since 8.1
+ */
+@CompileStatic
+final class MongoCommandRedactor {
+
+ private static final String PLACEHOLDER = QueryTailRedactor.PLACEHOLDER
+
+ private static final String DATABASE = '$db'
+
+ private static final String COLLECTION = 'collection'
+
+ /** The entries the driver adds, which describe its session and transaction rather than the command. */
+ private static final Set<String> LEFT_OUT = ['lsid', '$clusterTime', 'txnNumber', 'autocommit',
+ 'startTransaction', '$readPreference', 'apiVersion', 'apiStrict', 'apiDeprecationErrors'] as Set<String>
+
+ /**
+ * The options whose document, or documents, are keyed by the fields of the collection, wherever they appear,
+ * including those of the operations of a client-level {@code bulkWrite}.
+ */
+ private static final Set<String> FIELD_KEYED = ['filter', 'query', 'q', 'u', 'update', 'document', 'documents',
+ 'updateMods', 'pipeline', 'sort', 'projection', 'fields', 'key', 'hint', 'min', 'max', 'arrayFilters',
+ 'partialFilterExpression', 'wildcardProjection'] as Set<String>
+
+ private MongoCommandRedactor() {
+ }
+
+ /**
+ * Redacts a command.
+ *
+ * @param database the database the command runs against
+ * @param commandName the name of the command, which is its first key
+ * @param command the command as the driver sends it
+ * @param fields the fields the mapping of the datastore knows, or null before the datastore is built
+ * @return the command with its values replaced
+ */
+ static String redact(String database, String commandName, BsonDocument command, MongoMappedFields fields) {
+ String collection = null
+ String collectionKey = null
+ BsonValue first = command.get(commandName)
+ if (first != null && first.isString()) {
+ collection = first.asString().value
+ collectionKey = commandName
+ }
+ else if (command.get(COLLECTION)?.isString()) {
+ collection = command.getString(COLLECTION).value
+ collectionKey = COLLECTION
+ }
+ StringBuilder out = new StringBuilder(commandName)
+ if (database != null || collection != null) {
+ out.append(' ')
+ if (database != null) {
+ out.append(database)
+ }
+ if (collection != null) {
+ out.append(database != null ? '.' : '').append(collection)
+ }
+ }
+ Scope options = Scope.options(collection, fields)
+ StringBuilder body = new StringBuilder()
+ for (Map.Entry<String, BsonValue> entry in command.entrySet()) {
+ String key = entry.key
+ if (key == DATABASE || key == collectionKey || LEFT_OUT.contains(key)) {
+ continue
+ }
+ body.append(body.length() == 0 ? '{' : ', ')
+ appendEntry(body, key, entry.value, options)
+ }
+ if (body.length() > 0) {
+ out.append(' ').append(body).append('}')
+ }
+ out.toString()
+ }
+
+ private static void appendEntry(StringBuilder out, String key, BsonValue value, Scope scope) {
+ Scope child = scope.of(key)
+ out.append(child.keyText).append(': ')
+ appendValue(out, value, child)
+ }
+
+ private static void appendValue(StringBuilder out, BsonValue value, Scope scope) {
+ if (value.isDocument()) {
+ out.append('{')
+ boolean first = true
+ for (Map.Entry<String, BsonValue> entry in value.asDocument().entrySet()) {
+ if (!first) {
+ out.append(', ')
+ }
+ first = false
+ appendEntry(out, entry.key, entry.value, scope)
+ }
+ out.append('}')
+ }
+ else if (value.isArray()) {
+ out.append('[')
+ boolean first = true
+ for (BsonValue element in value.asArray()) {
+ if (!first) {
+ out.append(', ')
+ }
+ first = false
+ appendValue(out, element, scope)
Review Comment:
A `List<Map>` such as `attributes: [{"1234567890": "value"}]` still records `1234567890`. Each array element keeps the parent scope, so the numeric key is treated as an index of the collection rather than a map key. `attributes.0.1234567890` does the same.
Advance the scope into the element. Mask a numeric key unless the parent is a known indexed collection. Please cover that list through the capture API.
##########
grails-datamapping-core/src/main/groovy/org/apache/grails/data/core/querytail/QueryTailRequestAttribution.groovy:
##########
@@ -0,0 +1,361 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * https://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.grails.data.core.querytail
+
+import java.lang.reflect.Method
+import java.nio.charset.StandardCharsets
+import java.util.concurrent.atomic.AtomicLong
+import java.util.regex.Pattern
+
+import groovy.transform.CompileStatic
+
+import org.springframework.util.ClassUtils
+import org.springframework.web.context.request.RequestAttributes
+import org.springframework.web.context.request.RequestContextHolder
+import org.springframework.web.context.request.ServletRequestAttributes
+
+/**
+ * Attributes a captured statement to the web request on the current thread. Grails keeps the
+ * request, and the controller and action it resolved, on the thread that serves it, so a statement
+ * run by an interceptor, the action or the view all land on the same page. The first statement of
+ * a request gives it an id that every later statement of that request repeats, which is how the
+ * {@code query-tail} command counts statements per request. Work on any other thread is attributed to
+ * the thread by name.
+ *
+ * <p>The request is recorded as its method and path, with the values in the path replaced by
+ * {@code ?}: the values of the parameters the URL mapping took from it and those of its query, found
+ * whether or not the client percent-encoded them, and any segment that is a number, a UUID or a long
+ * hexadecimal token, so that an id, a token or a key carried in a path never reaches the capture
+ * file.</p>
+ *
+ * <p>Spring Web is optional: without it on the classpath every statement is attributed to its
+ * thread.</p>
+ *
+ * @since 8.1
+ */
+@CompileStatic
+class QueryTailRequestAttribution {
+
+ /** The request attribute holding the id this attribution gave a request. */
+ public static final String REQUEST_ID_ATTRIBUTE = 'org.apache.grails.data.core.querytail.REQUEST_ID'
+
+ /** The request attribute Grails sets the controller name in. */
+ public static final String CONTROLLER_NAME_ATTRIBUTE = 'org.grails.CONTROLLER_NAME_ATTRIBUTE'
+
+ /** The request attribute Grails sets the controller namespace in. */
+ public static final String CONTROLLER_NAMESPACE_ATTRIBUTE = 'org.grails.CONTROLLER_NAMESPACE_ATTRIBUTE'
+
+ /** The request attribute Grails sets the action name in. */
+ public static final String ACTION_NAME_ATTRIBUTE = 'org.grails.ACTION_NAME_ATTRIBUTE'
+
+ private static final boolean SPRING_WEB_PRESENT = ClassUtils.isPresent(
+ 'org.springframework.web.context.request.RequestContextHolder', QueryTailRequestAttribution.classLoader)
+
+ private final AtomicLong requestCounter = new AtomicLong()
+ private final String requestIdPrefix
+
+ /**
+ * An attribution whose request ids are {@code r1}, {@code r2} and so on.
+ */
+ QueryTailRequestAttribution() {
+ this('')
+ }
+
+ /**
+ * @param requestIdPrefix what every request id starts with, which tells the requests of one JVM
+ * from those of another writing the same capture
+ */
+ QueryTailRequestAttribution(String requestIdPrefix) {
+ this.requestIdPrefix = requestIdPrefix ?: ''
+ }
+
+ /**
+ * Fills in the request id, page and request of a record from the current thread.
+ *
+ * @param record the record of a statement that just ran on this thread
+ * @return the same record
+ */
+ QueryTailRecord attribute(QueryTailRecord record) {
+ if (SPRING_WEB_PRESENT && WebRequestLookup.attribute(record, requestCounter, requestIdPrefix)) {
+ return record
+ }
+ record.requestId = null
+ record.request = null
+ record.page = QueryTailRecord.THREAD_PAGE_PREFIX + Thread.currentThread().name
+ record
+ }
+
+ /**
+ * Keeps every reference to Spring Web, in a signature as well as in a body, in a class that is only
+ * loaded once Spring Web is known to be present. The outer class can then be loaded, and reflected
+ * on, without Spring Web on the classpath.
+ */
+ @CompileStatic
+ private static class WebRequestLookup {
+
+ private static final String REQUEST_ID_PREFIX = 'r'
+
+ private static final String PLACEHOLDER = '?'
+
+ /** The parameters a URL mapping sets that name code rather than carry a value. */
+ private static final Set<String> MAPPING_PARAMETERS = ['controller', 'action', 'namespace', 'plugin', 'format'] as Set<String>
+
+ /** A segment that is a number, a UUID or a long hexadecimal token is a value whatever the mapping says. */
+ private static final Pattern VALUE_SHAPED = Pattern.compile(
+ '(?i)[0-9]+|[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}|[0-9a-f]{32,}')
+
+ private static final boolean SERVLET_PRESENT = ClassUtils.isPresent(
+ 'jakarta.servlet.http.HttpServletRequest', WebRequestLookup.classLoader)
+
+ /** The {@code getParams()} of a Grails web request, which holds the parameters its URL mapping took, by class. */
+ private static final ClassValue<Method> PARAMETERS = new ClassValue<Method>() {
+ @Override
+ protected Method computeValue(Class<?> type) {
+ try {
+ return type.getMethod('getParams')
+ }
+ catch (NoSuchMethodException ignored) {
+ return null
+ }
+ }
+ }
+
+ /**
+ * Fills in a record from the request on the current thread.
+ *
+ * @param record the record to fill in
+ * @param requestCounter the counter the next request id is taken from
+ * @param requestIdPrefix what every request id starts with
+ * @return whether there is a request on the current thread
+ */
+ static boolean attribute(QueryTailRecord record, AtomicLong requestCounter, String requestIdPrefix) {
+ RequestAttributes attributes = RequestContextHolder.getRequestAttributes()
+ if (attributes == null) {
+ return false
+ }
+ record.requestId = requestId(attributes, requestCounter, requestIdPrefix)
+ record.page = page(attributes)
+ record.request = SERVLET_PRESENT ? ServletRequestLookup.describe(attributes) : null
+ true
+ }
+
+ /**
+ * Builds the request line: the method and the path, with the values in the path, the parameters
+ * the URL mapping took from it among them, replaced.
+ *
+ * @param method the HTTP method
+ * @param path the request path
+ * @param attributes the request, whose mapped parameters are read when it is a Grails web request
+ * @param parameters the parameters of the servlet request, by name, each a {@code String[]}
+ */
+ static String describe(String method, String path, RequestAttributes attributes, Map<?, ?> parameters) {
+ Set<String> values = new HashSet<>()
+ addValues(values, mappedParameters(attributes))
+ addValues(values, parameters)
+ "${method} ${redactPath(path, values)}".toString()
+ }
+
+ private static String requestId(RequestAttributes attributes, AtomicLong requestCounter, String requestIdPrefix) {
+ synchronized (attributes) {
+ Object existing = attributes.getAttribute(REQUEST_ID_ATTRIBUTE, RequestAttributes.SCOPE_REQUEST)
+ if (existing != null) {
+ return existing.toString()
+ }
+ String id = requestIdPrefix + REQUEST_ID_PREFIX + requestCounter.incrementAndGet()
+ attributes.setAttribute(REQUEST_ID_ATTRIBUTE, id, RequestAttributes.SCOPE_REQUEST)
+ return id
+ }
+ }
+
+ private static String page(RequestAttributes attributes) {
+ Object controller = attributes.getAttribute(CONTROLLER_NAME_ATTRIBUTE, RequestAttributes.SCOPE_REQUEST)
+ if (controller == null) {
+ return null
+ }
+ Object namespace = attributes.getAttribute(CONTROLLER_NAMESPACE_ATTRIBUTE, RequestAttributes.SCOPE_REQUEST)
+ Object action = attributes.getAttribute(ACTION_NAME_ATTRIBUTE, RequestAttributes.SCOPE_REQUEST)
+ StringBuilder page = new StringBuilder()
+ if (namespace != null) {
+ page.append(namespace).append(':')
+ }
+ page.append(controller)
+ if (action != null) {
+ page.append('/').append(action)
+ }
+ page.toString()
+ }
+
+ /**
+ * @return the parameters of a Grails web request, which include those its URL mapping took from
+ * the path, or null for a request that is not one
+ */
+ private static Map<?, ?> mappedParameters(RequestAttributes attributes) {
+ Method parameters = PARAMETERS.get(attributes.getClass())
+ if (parameters == null) {
+ return null
+ }
+ try {
+ Object value = parameters.invoke(attributes)
+ return value instanceof Map ? (Map<?, ?>) value : null
+ }
+ catch (ReflectiveOperationException | RuntimeException ignored) {
+ return null
+ }
+ }
+
+ private static void addValues(Set<String> values, Map<?, ?> parameters) {
+ if (parameters == null) {
+ return
+ }
+ for (Map.Entry<?, ?> entry in parameters.entrySet()) {
+ if (MAPPING_PARAMETERS.contains(String.valueOf(entry.key))) {
+ continue
+ }
+ Object value = entry.value
+ if (value instanceof CharSequence) {
+ values.add(value.toString())
+ }
+ else if (value instanceof Object[]) {
+ for (Object element in (Object[]) value) {
+ if (element instanceof CharSequence) {
+ values.add(element.toString())
+ }
+ }
+ }
+ }
+ }
+
+ /**
+ * Replaces the values in a path: a segment that is one of the values, with or without an
+ * extension, a run of segments that spells a value holding slashes, and any segment shaped
+ * like a value. The segments are matched as the client sent them and percent-decoded, since
+ * the parameters are decoded while the request URI is not; the path is written as sent.
+ */
+ private static String redactPath(String path, Set<String> values) {
+ String[] segments = path.split('/', -1)
+ String[] decoded = new String[segments.length]
+ boolean[] redacted = new boolean[segments.length]
+ for (int i = 0; i < segments.length; i++) {
Review Comment:
A query that runs before URL mapping has no parameters yet. `/password/reset/k9fQ2x` matches neither a parameter nor the numeric, UUID, or hex patterns, so the token is written. Mapping the request later cannot remove it.
Withhold an unmatched segment until its role is known, or do not record the path until mapping has run.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@grails.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add the query-tail command to find query-heavy pages [grails-core]
Posted by "jdaugherty (via GitHub)" <gi...@apache.org>.
jdaugherty commented on PR #16631:
URL: https://github.com/apache/grails-core/pull/16631#issuecomment-6391293406
@jamesfredley re review 5640715445: fixed on head 49a8b1d769. A segment a mapped value is glued into is now withheld whole whenever either its sent or its percent-decoded form holds the value, so `/reset/%6B9fQ2x_confirm?q=confirm` with `token=k9fQ2x` records `GET /reset/?`; covered through `attribute()` in the attribution spec, with and without the `q` parameter. Details are in the reply on your thread, which I resolved since it is fixed; reopen it if you want to discuss further.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@grails.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add the query-tail command to find query-heavy pages [grails-core]
Posted by "jdaugherty (via GitHub)" <gi...@apache.org>.
jdaugherty commented on code in PR #16631:
URL: https://github.com/apache/grails-core/pull/16631#discussion_r4367908555
##########
grails-data-mongodb/core/src/main/groovy/org/apache/grails/data/mongo/core/querytail/MongoCommandRedactor.groovy:
##########
@@ -0,0 +1,241 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * https://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.grails.data.mongo.core.querytail
+
+import groovy.transform.CompileStatic
+import org.bson.BsonDocument
+import org.bson.BsonValue
+
+import org.apache.grails.data.core.querytail.QueryTailRedactor
+
+/**
+ * Takes the values out of a MongoDB command before it is captured. A command is a document, so
+ * there is nothing to parse: every value becomes {@link QueryTailRedactor#PLACEHOLDER}, and the
+ * keys, which are the options of the command, its operators and the field names of the collection,
+ * are kept, except where a key may be data. The keys of a document stored in a map property, or in
+ * a property or a collection the {@linkplain MongoMappedFields mapping} does not know, are recorded
+ * as the placeholder too, and so is every segment of a dotted path past such a key. A numeric key
+ * is kept only where it indexes an array the mapping knows; in a map, or at the top of a document,
+ * it is data. The entries the driver adds for its session, the cluster time and the transaction,
+ * which say nothing about the command, are left out.
+ *
+ * <p>A command is recorded as its name, the database and collection it runs against, and the
+ * document of its remaining entries: {@code find shop.book {"filter": {"title": ?}, "limit": ?}}.</p>
+ *
+ * @since 8.1
+ */
+@CompileStatic
+final class MongoCommandRedactor {
+
+ private static final String PLACEHOLDER = QueryTailRedactor.PLACEHOLDER
+
+ private static final String DATABASE = '$db'
+
+ private static final String COLLECTION = 'collection'
+
+ /** The entries the driver adds, which describe its session and transaction rather than the command. */
+ private static final Set<String> LEFT_OUT = ['lsid', '$clusterTime', 'txnNumber', 'autocommit',
+ 'startTransaction', '$readPreference', 'apiVersion', 'apiStrict', 'apiDeprecationErrors'] as Set<String>
+
+ /**
+ * The options whose document, or documents, are keyed by the fields of the collection, wherever they appear,
+ * including those of the operations of a client-level {@code bulkWrite}.
+ */
+ private static final Set<String> FIELD_KEYED = ['filter', 'query', 'q', 'u', 'update', 'document', 'documents',
+ 'updateMods', 'pipeline', 'sort', 'projection', 'fields', 'key', 'hint', 'min', 'max', 'arrayFilters',
+ 'partialFilterExpression', 'wildcardProjection'] as Set<String>
+
+ private MongoCommandRedactor() {
+ }
+
+ /**
+ * Redacts a command.
+ *
+ * @param database the database the command runs against
+ * @param commandName the name of the command, which is its first key
+ * @param command the command as the driver sends it
+ * @param fields the fields the mapping of the datastore knows, or null before the datastore is built
+ * @return the command with its values replaced
+ */
+ static String redact(String database, String commandName, BsonDocument command, MongoMappedFields fields) {
+ String collection = null
+ String collectionKey = null
+ BsonValue first = command.get(commandName)
+ if (first != null && first.isString()) {
+ collection = first.asString().value
+ collectionKey = commandName
+ }
+ else if (command.get(COLLECTION)?.isString()) {
+ collection = command.getString(COLLECTION).value
+ collectionKey = COLLECTION
+ }
+ StringBuilder out = new StringBuilder(commandName)
+ if (database != null || collection != null) {
+ out.append(' ')
+ if (database != null) {
+ out.append(database)
+ }
+ if (collection != null) {
+ out.append(database != null ? '.' : '').append(collection)
+ }
+ }
+ Scope options = Scope.options(collection, fields)
+ StringBuilder body = new StringBuilder()
+ for (Map.Entry<String, BsonValue> entry in command.entrySet()) {
+ String key = entry.key
+ if (key == DATABASE || key == collectionKey || LEFT_OUT.contains(key)) {
+ continue
+ }
+ body.append(body.length() == 0 ? '{' : ', ')
+ appendEntry(body, key, entry.value, options)
+ }
+ if (body.length() > 0) {
+ out.append(' ').append(body).append('}')
+ }
+ out.toString()
+ }
+
+ private static void appendEntry(StringBuilder out, String key, BsonValue value, Scope scope) {
+ Scope child = scope.of(key)
+ out.append(child.keyText).append(': ')
+ appendValue(out, value, child)
+ }
+
+ private static void appendValue(StringBuilder out, BsonValue value, Scope scope) {
+ if (value.isDocument()) {
+ out.append('{')
+ boolean first = true
+ for (Map.Entry<String, BsonValue> entry in value.asDocument().entrySet()) {
+ if (!first) {
+ out.append(', ')
+ }
+ first = false
+ appendEntry(out, entry.key, entry.value, scope)
+ }
+ out.append('}')
+ }
+ else if (value.isArray()) {
+ out.append('[')
+ boolean first = true
+ for (BsonValue element in value.asArray()) {
+ if (!first) {
+ out.append(', ')
+ }
+ first = false
+ appendValue(out, element, scope)
Review Comment:
Fixed in b0e9a8c9a4 (folded into the feature commit). The scope now advances into an array element: `Scope` carries a `depth`, the number of indices taken into the field at its path, an array element gets `depth + 1`, and so does the document of an operator on an array such as `$elemMatch`. A numeric key is kept only when `MongoMappedFields.isIndexed(collection, path, depth)` says the field is an array at that depth: at depth 0 an array- or `Collection`-typed property, at depth 1 only when its element type (the array component type, or the `Basic` property's component type) is itself an array or collection, deeper never. So `attributes: [{"9876543210": "v"}]`, `attributes.0.9876543210`, `attributes: {$elemMatch: {"9876543210": 1}}` and a numeric key in an element of an embedded list all record `?`, while `stops.0.city` and `tags.0` keep their index. Covered in `MongoCommandRedactorSpec` (insert, dotted, `$elemMatch` and nested-array forms, plus the never-survives case with a numer
ic secret in a list of maps) and through the capture API in `QueryTailCommandListenerSpec` against a real mongod, which saves a book whose `List<Map>` property holds a numeric key, queries `attributes.0.<key>`, and asserts the key reaches neither the insert nor the find in the file. The guide says a numeric key inside an element of such a list is data unless the property is a list of lists.
##########
grails-datamapping-core/src/main/groovy/org/apache/grails/data/core/querytail/QueryTailRequestAttribution.groovy:
##########
@@ -0,0 +1,361 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * https://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.grails.data.core.querytail
+
+import java.lang.reflect.Method
+import java.nio.charset.StandardCharsets
+import java.util.concurrent.atomic.AtomicLong
+import java.util.regex.Pattern
+
+import groovy.transform.CompileStatic
+
+import org.springframework.util.ClassUtils
+import org.springframework.web.context.request.RequestAttributes
+import org.springframework.web.context.request.RequestContextHolder
+import org.springframework.web.context.request.ServletRequestAttributes
+
+/**
+ * Attributes a captured statement to the web request on the current thread. Grails keeps the
+ * request, and the controller and action it resolved, on the thread that serves it, so a statement
+ * run by an interceptor, the action or the view all land on the same page. The first statement of
+ * a request gives it an id that every later statement of that request repeats, which is how the
+ * {@code query-tail} command counts statements per request. Work on any other thread is attributed to
+ * the thread by name.
+ *
+ * <p>The request is recorded as its method and path, with the values in the path replaced by
+ * {@code ?}: the values of the parameters the URL mapping took from it and those of its query, found
+ * whether or not the client percent-encoded them, and any segment that is a number, a UUID or a long
+ * hexadecimal token, so that an id, a token or a key carried in a path never reaches the capture
+ * file.</p>
+ *
+ * <p>Spring Web is optional: without it on the classpath every statement is attributed to its
+ * thread.</p>
+ *
+ * @since 8.1
+ */
+@CompileStatic
+class QueryTailRequestAttribution {
+
+ /** The request attribute holding the id this attribution gave a request. */
+ public static final String REQUEST_ID_ATTRIBUTE = 'org.apache.grails.data.core.querytail.REQUEST_ID'
+
+ /** The request attribute Grails sets the controller name in. */
+ public static final String CONTROLLER_NAME_ATTRIBUTE = 'org.grails.CONTROLLER_NAME_ATTRIBUTE'
+
+ /** The request attribute Grails sets the controller namespace in. */
+ public static final String CONTROLLER_NAMESPACE_ATTRIBUTE = 'org.grails.CONTROLLER_NAMESPACE_ATTRIBUTE'
+
+ /** The request attribute Grails sets the action name in. */
+ public static final String ACTION_NAME_ATTRIBUTE = 'org.grails.ACTION_NAME_ATTRIBUTE'
+
+ private static final boolean SPRING_WEB_PRESENT = ClassUtils.isPresent(
+ 'org.springframework.web.context.request.RequestContextHolder', QueryTailRequestAttribution.classLoader)
+
+ private final AtomicLong requestCounter = new AtomicLong()
+ private final String requestIdPrefix
+
+ /**
+ * An attribution whose request ids are {@code r1}, {@code r2} and so on.
+ */
+ QueryTailRequestAttribution() {
+ this('')
+ }
+
+ /**
+ * @param requestIdPrefix what every request id starts with, which tells the requests of one JVM
+ * from those of another writing the same capture
+ */
+ QueryTailRequestAttribution(String requestIdPrefix) {
+ this.requestIdPrefix = requestIdPrefix ?: ''
+ }
+
+ /**
+ * Fills in the request id, page and request of a record from the current thread.
+ *
+ * @param record the record of a statement that just ran on this thread
+ * @return the same record
+ */
+ QueryTailRecord attribute(QueryTailRecord record) {
+ if (SPRING_WEB_PRESENT && WebRequestLookup.attribute(record, requestCounter, requestIdPrefix)) {
+ return record
+ }
+ record.requestId = null
+ record.request = null
+ record.page = QueryTailRecord.THREAD_PAGE_PREFIX + Thread.currentThread().name
+ record
+ }
+
+ /**
+ * Keeps every reference to Spring Web, in a signature as well as in a body, in a class that is only
+ * loaded once Spring Web is known to be present. The outer class can then be loaded, and reflected
+ * on, without Spring Web on the classpath.
+ */
+ @CompileStatic
+ private static class WebRequestLookup {
+
+ private static final String REQUEST_ID_PREFIX = 'r'
+
+ private static final String PLACEHOLDER = '?'
+
+ /** The parameters a URL mapping sets that name code rather than carry a value. */
+ private static final Set<String> MAPPING_PARAMETERS = ['controller', 'action', 'namespace', 'plugin', 'format'] as Set<String>
+
+ /** A segment that is a number, a UUID or a long hexadecimal token is a value whatever the mapping says. */
+ private static final Pattern VALUE_SHAPED = Pattern.compile(
+ '(?i)[0-9]+|[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}|[0-9a-f]{32,}')
+
+ private static final boolean SERVLET_PRESENT = ClassUtils.isPresent(
+ 'jakarta.servlet.http.HttpServletRequest', WebRequestLookup.classLoader)
+
+ /** The {@code getParams()} of a Grails web request, which holds the parameters its URL mapping took, by class. */
+ private static final ClassValue<Method> PARAMETERS = new ClassValue<Method>() {
+ @Override
+ protected Method computeValue(Class<?> type) {
+ try {
+ return type.getMethod('getParams')
+ }
+ catch (NoSuchMethodException ignored) {
+ return null
+ }
+ }
+ }
+
+ /**
+ * Fills in a record from the request on the current thread.
+ *
+ * @param record the record to fill in
+ * @param requestCounter the counter the next request id is taken from
+ * @param requestIdPrefix what every request id starts with
+ * @return whether there is a request on the current thread
+ */
+ static boolean attribute(QueryTailRecord record, AtomicLong requestCounter, String requestIdPrefix) {
+ RequestAttributes attributes = RequestContextHolder.getRequestAttributes()
+ if (attributes == null) {
+ return false
+ }
+ record.requestId = requestId(attributes, requestCounter, requestIdPrefix)
+ record.page = page(attributes)
+ record.request = SERVLET_PRESENT ? ServletRequestLookup.describe(attributes) : null
+ true
+ }
+
+ /**
+ * Builds the request line: the method and the path, with the values in the path, the parameters
+ * the URL mapping took from it among them, replaced.
+ *
+ * @param method the HTTP method
+ * @param path the request path
+ * @param attributes the request, whose mapped parameters are read when it is a Grails web request
+ * @param parameters the parameters of the servlet request, by name, each a {@code String[]}
+ */
+ static String describe(String method, String path, RequestAttributes attributes, Map<?, ?> parameters) {
+ Set<String> values = new HashSet<>()
+ addValues(values, mappedParameters(attributes))
+ addValues(values, parameters)
+ "${method} ${redactPath(path, values)}".toString()
+ }
+
+ private static String requestId(RequestAttributes attributes, AtomicLong requestCounter, String requestIdPrefix) {
+ synchronized (attributes) {
+ Object existing = attributes.getAttribute(REQUEST_ID_ATTRIBUTE, RequestAttributes.SCOPE_REQUEST)
+ if (existing != null) {
+ return existing.toString()
+ }
+ String id = requestIdPrefix + REQUEST_ID_PREFIX + requestCounter.incrementAndGet()
+ attributes.setAttribute(REQUEST_ID_ATTRIBUTE, id, RequestAttributes.SCOPE_REQUEST)
+ return id
+ }
+ }
+
+ private static String page(RequestAttributes attributes) {
+ Object controller = attributes.getAttribute(CONTROLLER_NAME_ATTRIBUTE, RequestAttributes.SCOPE_REQUEST)
+ if (controller == null) {
+ return null
+ }
+ Object namespace = attributes.getAttribute(CONTROLLER_NAMESPACE_ATTRIBUTE, RequestAttributes.SCOPE_REQUEST)
+ Object action = attributes.getAttribute(ACTION_NAME_ATTRIBUTE, RequestAttributes.SCOPE_REQUEST)
+ StringBuilder page = new StringBuilder()
+ if (namespace != null) {
+ page.append(namespace).append(':')
+ }
+ page.append(controller)
+ if (action != null) {
+ page.append('/').append(action)
+ }
+ page.toString()
+ }
+
+ /**
+ * @return the parameters of a Grails web request, which include those its URL mapping took from
+ * the path, or null for a request that is not one
+ */
+ private static Map<?, ?> mappedParameters(RequestAttributes attributes) {
+ Method parameters = PARAMETERS.get(attributes.getClass())
+ if (parameters == null) {
+ return null
+ }
+ try {
+ Object value = parameters.invoke(attributes)
+ return value instanceof Map ? (Map<?, ?>) value : null
+ }
+ catch (ReflectiveOperationException | RuntimeException ignored) {
+ return null
+ }
+ }
+
+ private static void addValues(Set<String> values, Map<?, ?> parameters) {
+ if (parameters == null) {
+ return
+ }
+ for (Map.Entry<?, ?> entry in parameters.entrySet()) {
+ if (MAPPING_PARAMETERS.contains(String.valueOf(entry.key))) {
+ continue
+ }
+ Object value = entry.value
+ if (value instanceof CharSequence) {
+ values.add(value.toString())
+ }
+ else if (value instanceof Object[]) {
+ for (Object element in (Object[]) value) {
+ if (element instanceof CharSequence) {
+ values.add(element.toString())
+ }
+ }
+ }
+ }
+ }
+
+ /**
+ * Replaces the values in a path: a segment that is one of the values, with or without an
+ * extension, a run of segments that spells a value holding slashes, and any segment shaped
+ * like a value. The segments are matched as the client sent them and percent-decoded, since
+ * the parameters are decoded while the request URI is not; the path is written as sent.
+ */
+ private static String redactPath(String path, Set<String> values) {
+ String[] segments = path.split('/', -1)
+ String[] decoded = new String[segments.length]
+ boolean[] redacted = new boolean[segments.length]
+ for (int i = 0; i < segments.length; i++) {
Review Comment:
Fixed in b0e9a8c9a4. Until the URL mapping has run, no segment's role is known, so the request line records only the shape of the path: `describe` treats the mapping as run when the controller attribute is set or the Grails request's parameters carry `controller`, and otherwise replaces every segment, so a statement run before the controller is chosen, or in a request that never reaches one, is written as `GET /?/?/?`; once the mapping has run the later statements of the same request get the mapped line with the literal segments kept. `QueryTailRequestAttributionSpec` records a statement on `/password/reset/k9fQ2x` before and after the mapping (`GET /?/?/?`, then `GET /password/reset/?`), and the rows that relied on the numeric, UUID and hex patterns now run with a mapped request, plus one unmapped row. The guide and the class javadoc say so, and the capture specs that assert a path now bind a controller.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@grails.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add the query-tail command to find query-heavy pages [grails-core]
Posted by "jdaugherty (via GitHub)" <gi...@apache.org>.
jdaugherty commented on PR #16631:
URL: https://github.com/apache/grails-core/pull/16631#issuecomment-6400277602
@jamesfredley, review 5644139826 is addressed in 6945601254, still a single commit on 9.0.x.
- **Same-length restart**: `QueryTailFileReader` now reads the run header on every poll and treats a poll as idle only when the run and the length are both unchanged. A restart that rewrites the capture to the same size starts the command over, in the stream and in the summaries. Reader and runner tests cover it, and they fail against the previous reader.
The thread is resolved with the details.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@grails.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add the query-tail command to find query-heavy pages [grails-core]
Posted by "jdaugherty (via GitHub)" <gi...@apache.org>.
jdaugherty commented on PR #16631:
URL: https://github.com/apache/grails-core/pull/16631#issuecomment-6402514432
@jamesfredley both items from your last review are fixed in b8c346b48c. It is still a single commit.
- **Programmatic statement inspector:** on Hibernate 7, the installer now wraps an inspector a `configClass` sets with `setStatementInspector`, and installs the wrapper the same way. When the property also names an inspector, the wrapper runs the one set on the configuration, as Hibernate does. Hibernate 5.6's `Configuration` has no such setter, so on Hibernate 5 a `configClass` can set an inspector only through the configuration's properties, which the installer already wraps. A Hibernate 5 test now covers that case.
- **Quoted names in the collapser:** clause matching now skips names quoted with double quotes, backticks or brackets, including doubled closing quotes. `update "order set lines" ...` and `update "order set headers" ...` now stay two statements. Comment and whitespace removal also leave quoted names as written.
The new tests fail on the previous code. The Hibernate 5 and 7 query-tail specs, the CLI query-tail specs and CodeNarc pass locally.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@grails.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add the query-tail command to find query-heavy pages [grails-core]
Posted by "testlens-app[bot] (via GitHub)" <gi...@apache.org>.
testlens-app[bot] commented on PR #16631:
URL: https://github.com/apache/grails-core/pull/16631#issuecomment-6403829019
## ✅ All tests passed ✅
🏷️ Commit: ea3558f3c622462817f3cc4897ca48e3937d7710
▶️ Tests: 135595 executed
⚪️ Checks: 97/97 completed
---
_Learn more about TestLens at [testlens.app/docs](https://testlens.app/docs/features/pr-comment/)._
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@grails.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add the sql-tail command to find query-heavy pages [grails-core]
Posted by "codecov[bot] (via GitHub)" <gi...@apache.org>.
codecov[bot] commented on PR #16631:
URL: https://github.com/apache/grails-core/pull/16631#issuecomment-6150845908
## [Codecov](https://app.codecov.io/gh/apache/grails-core/pull/16631?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:x: Patch coverage is `85.00000%` with `18 lines` in your changes missing coverage. Please review.
:white_check_mark: Project coverage is 60.9871%. Comparing base ([`bf9c215`](https://app.codecov.io/gh/apache/grails-core/commit/bf9c215ace37e396fe329b693e8d57232596e390?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`a22e6fe`](https://app.codecov.io/gh/apache/grails-core/commit/a22e6fe7e53e6212d6d29befa29ce7bc8bbe379a?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
:warning: Report is 18 commits behind head on 9.0.x.
| [Files with missing lines](https://app.codecov.io/gh/apache/grails-core/pull/16631?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [.../plugin/core/GrailsPropertyArgumentProvider.groovy](https://app.codecov.io/gh/apache/grails-core/pull/16631?src=pr&el=tree&filepath=grails-gradle%2Fplugins%2Fsrc%2Fmain%2Fgroovy%2Forg%2Fgrails%2Fgradle%2Fplugin%2Fcore%2FGrailsPropertyArgumentProvider.groovy&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-Z3JhaWxzLWdyYWRsZS9wbHVnaW5zL3NyYy9tYWluL2dyb292eS9vcmcvZ3JhaWxzL2dyYWRsZS9wbHVnaW4vY29yZS9HcmFpbHNQcm9wZXJ0eUFyZ3VtZW50UHJvdmlkZXIuZ3Jvb3Z5) | 0.0000% | [6 Missing :warning: ](https://app.codecov.io/gh/apache/grails-core/pull/16631?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...gorm/jdbc/sqltail/SqlTailRequestAttribution.groovy](https://app.codecov.io/gh/apache/grails-core/pull/16631?src=pr&el=tree&filepath=grails-datamapping-core%2Fsrc%2Fmain%2Fgroovy%2Forg%2Fgrails%2Fdatastore%2Fgorm%2Fjdbc%2Fsqltail%2FSqlTailRequestAttribution.groovy&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-Z3JhaWxzLWRhdGFtYXBwaW5nLWNvcmUvc3JjL21haW4vZ3Jvb3Z5L29yZy9ncmFpbHMvZGF0YXN0b3JlL2dvcm0vamRiYy9zcWx0YWlsL1NxbFRhaWxSZXF1ZXN0QXR0cmlidXRpb24uZ3Jvb3Z5) | 86.8421% | [1 Missing and 4 partials :warning: ](https://app.codecov.io/gh/apache/grails-core/pull/16631?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [.../datastore/gorm/jdbc/sqltail/SqlTailCapture.groovy](https://app.codecov.io/gh/apache/grails-core/pull/16631?src=pr&el=tree&filepath=grails-datamapping-core%2Fsrc%2Fmain%2Fgroovy%2Forg%2Fgrails%2Fdatastore%2Fgorm%2Fjdbc%2Fsqltail%2FSqlTailCapture.groovy&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-Z3JhaWxzLWRhdGFtYXBwaW5nLWNvcmUvc3JjL21haW4vZ3Jvb3Z5L29yZy9ncmFpbHMvZGF0YXN0b3JlL2dvcm0vamRiYy9zcWx0YWlsL1NxbFRhaWxDYXB0dXJlLmdyb292eQ==) | 92.5000% | [2 Missing and 1 partial :warning: ](https://app.codecov.io/gh/apache/grails-core/pull/16631?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...rails/gradle/plugin/core/GrailsGradlePlugin.groovy](https://app.codecov.io/gh/apache/grails-core/pull/16631?src=pr&el=tree&filepath=grails-gradle%2Fplugins%2Fsrc%2Fmain%2Fgroovy%2Forg%2Fgrails%2Fgradle%2Fplugin%2Fcore%2FGrailsGradlePlugin.groovy&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-Z3JhaWxzLWdyYWRsZS9wbHVnaW5zL3NyYy9tYWluL2dyb292eS9vcmcvZ3JhaWxzL2dyYWRsZS9wbHVnaW4vY29yZS9HcmFpbHNHcmFkbGVQbHVnaW4uZ3Jvb3Z5) | 0.0000% | [2 Missing :warning: ](https://app.codecov.io/gh/apache/grails-core/pull/16631?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...s/datastore/gorm/jdbc/sqltail/SqlTailRecord.groovy](https://app.codecov.io/gh/apache/grails-core/pull/16631?src=pr&el=tree&filepath=grails-datamapping-core%2Fsrc%2Fmain%2Fgroovy%2Forg%2Fgrails%2Fdatastore%2Fgorm%2Fjdbc%2Fsqltail%2FSqlTailRecord.groovy&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-Z3JhaWxzLWRhdGFtYXBwaW5nLWNvcmUvc3JjL21haW4vZ3Jvb3Z5L29yZy9ncmFpbHMvZGF0YXN0b3JlL2dvcm0vamRiYy9zcWx0YWlsL1NxbFRhaWxSZWNvcmQuZ3Jvb3Z5) | 94.7368% | [0 Missing and 1 partial :warning: ](https://app.codecov.io/gh/apache/grails-core/pull/16631?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...e/plugin/core/GrailsStartupProgressProvider.groovy](https://app.codecov.io/gh/apache/grails-core/pull/16631?src=pr&el=tree&filepath=grails-gradle%2Fplugins%2Fsrc%2Fmain%2Fgroovy%2Forg%2Fgrails%2Fgradle%2Fplugin%2Fcore%2FGrailsStartupProgressProvider.groovy&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-Z3JhaWxzLWdyYWRsZS9wbHVnaW5zL3NyYy9tYWluL2dyb292eS9vcmcvZ3JhaWxzL2dyYWRsZS9wbHVnaW4vY29yZS9HcmFpbHNTdGFydHVwUHJvZ3Jlc3NQcm92aWRlci5ncm9vdnk=) | 0.0000% | [1 Missing :warning: ](https://app.codecov.io/gh/apache/grails-core/pull/16631?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
<details><summary>Additional details and impacted files</summary>
[](https://app.codecov.io/gh/apache/grails-core/pull/16631?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
```diff
@@ Coverage Diff @@
## 9.0.x #16631 +/- ##
==================================================
+ Coverage 60.8588% 60.9871% +0.1283%
- Complexity 26377 26518 +141
==================================================
Files 2226 2234 +8
Lines 110970 111199 +229
Branches 20095 20124 +29
==================================================
+ Hits 67535 67817 +282
+ Misses 34527 34448 -79
- Partials 8908 8934 +26
```
| [Files with missing lines](https://app.codecov.io/gh/apache/grails-core/pull/16631?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [...ions/AbstractHibernateConnectionSourceFactory.java](https://app.codecov.io/gh/apache/grails-core/pull/16631?src=pr&el=tree&filepath=grails-data-hibernate5%2Fcore%2Fsrc%2Fmain%2Fgroovy%2Forg%2Fgrails%2Form%2Fhibernate%2Fconnections%2FAbstractHibernateConnectionSourceFactory.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-Z3JhaWxzLWRhdGEtaGliZXJuYXRlNS9jb3JlL3NyYy9tYWluL2dyb292eS9vcmcvZ3JhaWxzL29ybS9oaWJlcm5hdGUvY29ubmVjdGlvbnMvQWJzdHJhY3RIaWJlcm5hdGVDb25uZWN0aW9uU291cmNlRmFjdG9yeS5qYXZh) | `71.7391% <100.0000%> (+1.2846%)` | :arrow_up: |
| [...datastore/gorm/jdbc/sqltail/SqlTailSettings.groovy](https://app.codecov.io/gh/apache/grails-core/pull/16631?src=pr&el=tree&filepath=grails-datamapping-core%2Fsrc%2Fmain%2Fgroovy%2Forg%2Fgrails%2Fdatastore%2Fgorm%2Fjdbc%2Fsqltail%2FSqlTailSettings.groovy&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-Z3JhaWxzLWRhdGFtYXBwaW5nLWNvcmUvc3JjL21haW4vZ3Jvb3Z5L29yZy9ncmFpbHMvZGF0YXN0b3JlL2dvcm0vamRiYy9zcWx0YWlsL1NxbFRhaWxTZXR0aW5ncy5ncm9vdnk=) | `100.0000% <100.0000%> (ø)` | |
| [...s/datastore/gorm/jdbc/sqltail/SqlTailRecord.groovy](https://app.codecov.io/gh/apache/grails-core/pull/16631?src=pr&el=tree&filepath=grails-datamapping-core%2Fsrc%2Fmain%2Fgroovy%2Forg%2Fgrails%2Fdatastore%2Fgorm%2Fjdbc%2Fsqltail%2FSqlTailRecord.groovy&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-Z3JhaWxzLWRhdGFtYXBwaW5nLWNvcmUvc3JjL21haW4vZ3Jvb3Z5L29yZy9ncmFpbHMvZGF0YXN0b3JlL2dvcm0vamRiYy9zcWx0YWlsL1NxbFRhaWxSZWNvcmQuZ3Jvb3Z5) | `94.7368% <94.7368%> (ø)` | |
| [...e/plugin/core/GrailsStartupProgressProvider.groovy](https://app.codecov.io/gh/apache/grails-core/pull/16631?src=pr&el=tree&filepath=grails-gradle%2Fplugins%2Fsrc%2Fmain%2Fgroovy%2Forg%2Fgrails%2Fgradle%2Fplugin%2Fcore%2FGrailsStartupProgressProvider.groovy&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-Z3JhaWxzLWdyYWRsZS9wbHVnaW5zL3NyYy9tYWluL2dyb292eS9vcmcvZ3JhaWxzL2dyYWRsZS9wbHVnaW4vY29yZS9HcmFpbHNTdGFydHVwUHJvZ3Jlc3NQcm92aWRlci5ncm9vdnk=) | `0.0000% <0.0000%> (ø)` | |
| [...rails/gradle/plugin/core/GrailsGradlePlugin.groovy](https://app.codecov.io/gh/apache/grails-core/pull/16631?src=pr&el=tree&filepath=grails-gradle%2Fplugins%2Fsrc%2Fmain%2Fgroovy%2Forg%2Fgrails%2Fgradle%2Fplugin%2Fcore%2FGrailsGradlePlugin.groovy&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-Z3JhaWxzLWdyYWRsZS9wbHVnaW5zL3NyYy9tYWluL2dyb292eS9vcmcvZ3JhaWxzL2dyYWRsZS9wbHVnaW4vY29yZS9HcmFpbHNHcmFkbGVQbHVnaW4uZ3Jvb3Z5) | `29.5191% <0.0000%> (-0.0982%)` | :arrow_down: |
| [.../datastore/gorm/jdbc/sqltail/SqlTailCapture.groovy](https://app.codecov.io/gh/apache/grails-core/pull/16631?src=pr&el=tree&filepath=grails-datamapping-core%2Fsrc%2Fmain%2Fgroovy%2Forg%2Fgrails%2Fdatastore%2Fgorm%2Fjdbc%2Fsqltail%2FSqlTailCapture.groovy&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-Z3JhaWxzLWRhdGFtYXBwaW5nLWNvcmUvc3JjL21haW4vZ3Jvb3Z5L29yZy9ncmFpbHMvZGF0YXN0b3JlL2dvcm0vamRiYy9zcWx0YWlsL1NxbFRhaWxDYXB0dXJlLmdyb292eQ==) | `92.5000% <92.5000%> (ø)` | |
| [...gorm/jdbc/sqltail/SqlTailRequestAttribution.groovy](https://app.codecov.io/gh/apache/grails-core/pull/16631?src=pr&el=tree&filepath=grails-datamapping-core%2Fsrc%2Fmain%2Fgroovy%2Forg%2Fgrails%2Fdatastore%2Fgorm%2Fjdbc%2Fsqltail%2FSqlTailRequestAttribution.groovy&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-Z3JhaWxzLWRhdGFtYXBwaW5nLWNvcmUvc3JjL21haW4vZ3Jvb3Z5L29yZy9ncmFpbHMvZGF0YXN0b3JlL2dvcm0vamRiYy9zcWx0YWlsL1NxbFRhaWxSZXF1ZXN0QXR0cmlidXRpb24uZ3Jvb3Z5) | `86.8421% <86.8421%> (ø)` | |
| [.../plugin/core/GrailsPropertyArgumentProvider.groovy](https://app.codecov.io/gh/apache/grails-core/pull/16631?src=pr&el=tree&filepath=grails-gradle%2Fplugins%2Fsrc%2Fmain%2Fgroovy%2Forg%2Fgrails%2Fgradle%2Fplugin%2Fcore%2FGrailsPropertyArgumentProvider.groovy&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-Z3JhaWxzLWdyYWRsZS9wbHVnaW5zL3NyYy9tYWluL2dyb292eS9vcmcvZ3JhaWxzL2dyYWRsZS9wbHVnaW4vY29yZS9HcmFpbHNQcm9wZXJ0eUFyZ3VtZW50UHJvdmlkZXIuZ3Jvb3Z5) | `0.0000% <0.0000%> (ø)` | |
... and [31 files with indirect coverage changes](https://app.codecov.io/gh/apache/grails-core/pull/16631/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
</details>
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
- :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@grails.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add the sql-tail command to find query-heavy pages [grails-core]
Posted by "testlens-app[bot] (via GitHub)" <gi...@apache.org>.
testlens-app[bot] commented on PR #16631:
URL: https://github.com/apache/grails-core/pull/16631#issuecomment-6298473124
## 🚨 All tests passed but jobs failed 🚨
### Failed Jobs without Test Failures
❌ [Coverage / Coverage - grails-core \(ubuntu-24.04\)](https://github.com/apache/grails-core/actions/runs/39470034333/job/118791365142?pr=16631)
🏷️ Commit: 554856cf118d7359e9a06bf7d3df5c495bdf7de5
▶️ Tests: 37771 executed
🟡 Checks: 51/84 completed
### Rerun Controls
Click the checkbox to trigger a rerun:
🔲 **Rerun jobs**
---
_Learn more about TestLens at [testlens.app/docs](https://testlens.app/docs/features/pr-comment/)._
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@grails.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add the query-tail command to find query-heavy pages [grails-core]
Posted by "jamesfredley (via GitHub)" <gi...@apache.org>.
jamesfredley commented on code in PR #16631:
URL: https://github.com/apache/grails-core/pull/16631#discussion_r4381376158
##########
grails-data-hibernate7/core/src/main/groovy/org/apache/grails/data/hibernate7/core/querytail/QueryTailStatementInspector.groovy:
##########
@@ -0,0 +1,354 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * https://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.grails.data.hibernate7.core.querytail
+
+import groovy.transform.CompileStatic
+import groovy.transform.PackageScope
+import org.hibernate.SessionFactory
+import org.hibernate.SessionFactoryObserver
+import org.hibernate.boot.registry.StandardServiceRegistryBuilder
+import org.hibernate.boot.registry.selector.spi.StrategySelectionException
+import org.hibernate.boot.registry.selector.spi.StrategySelector
+import org.hibernate.cfg.AvailableSettings
+import org.hibernate.cfg.Configuration
+import org.hibernate.dialect.AbstractTransactSQLDialect
+import org.hibernate.dialect.CockroachDialect
+import org.hibernate.dialect.DB2Dialect
+import org.hibernate.dialect.Dialect
+import org.hibernate.dialect.H2Dialect
+import org.hibernate.dialect.HANADialect
+import org.hibernate.dialect.HSQLDialect
+import org.hibernate.dialect.MySQLDialect
+import org.hibernate.dialect.OracleDialect
+import org.hibernate.dialect.PostgreSQLDialect
+import org.hibernate.dialect.SpannerDialect
+import org.hibernate.dialect.SpannerPostgreSQLDialect
+import org.hibernate.engine.spi.SessionFactoryImplementor
+import org.hibernate.resource.jdbc.spi.StatementInspector
+
+import org.apache.grails.data.core.querytail.QueryTailCapture
+import org.apache.grails.data.core.querytail.QueryTailSettings
+import org.apache.grails.data.core.querytail.QueryTailSyntax
+import org.grails.datastore.mapping.core.exceptions.ConfigurationException
+import org.grails.orm.hibernate.cfg.HibernateMappingContextConfiguration
+
+/**
+ * The statement inspector GORM installs on a session factory when query-tail capture is enabled. It
+ * records every statement Hibernate is about to run to the {@link QueryTailCapture} of the
+ * application, without its values, and returns the statement as it found it: the database runs
+ * it unchanged. The inspector the session factory would run without the capture keeps running,
+ * wherever the application configured it: with {@code hibernate.session_factory.statement_inspector},
+ * on the configuration with {@link Configuration#setStatementInspector}, or in the settings of the
+ * service registry, as a {@code hibernate.configClass} may. It sees the statement first, and the
+ * statement it returns is the one run and recorded.
+ *
+ * <p>The session factory owns the capture. The inspector observes it: it opens the capture when
+ * the session factory is created, takes from its dialect how the database quotes values, and
+ * closes the capture when the session factory closes, so a session factory that fails to start
+ * leaves no capture open either. An observer the application set on the configuration keeps
+ * being called, even one it sets as the configuration builds the session factory.</p>
+ *
+ * <p>Nothing of this is installed unless {@link QueryTailSettings#SETTING_ENABLED} is set, so an
+ * application that does not capture pays nothing per statement.</p>
+ *
+ * @since 8.1
+ */
+@CompileStatic
+class QueryTailStatementInspector implements StatementInspector, SessionFactoryObserver {
+
+ @Serial
+ private static final long serialVersionUID = 1L
+
+ /** The community dialects, which are not a dependency, that the inspector tells apart by name. */
+ private static final String COMMUNITY_INFORMIX = 'org.hibernate.community.dialect.InformixDialect'
+ private static final String COMMUNITY_SQL_ANYWHERE = 'org.hibernate.community.dialect.SybaseAnywhereDialect'
+
+ /** The connection source name statements are recorded under. */
+ final String dataSourceName
+
+ /** The file statements are recorded to. */
+ final File file
+
+ private SessionFactoryObserver observer
+ private StatementInspector delegate
+ private boolean wrapped
+ private transient volatile QueryTailCapture capture
+ private transient boolean closed
+
+ /**
+ * @param dataSourceName the connection source the inspected session factory belongs to
+ * @param file the capture file statements are recorded to
+ */
+ QueryTailStatementInspector(String dataSourceName, File file) {
+ this.dataSourceName = dataSourceName
+ this.file = file
+ }
+
+ /**
+ * Installs the inspector on the configuration of a session factory when capture is enabled, as
+ * the observer and the statement inspector the configuration puts in place of those of the
+ * session factory when it builds the session factory.
+ *
+ * @param dataSourceName the connection source the session factory is built for
+ * @param settings the query-tail settings of the connection source
+ * @param configuration the configuration the session factory is built from
+ * @return the installed inspector, or null when capture is not enabled
+ */
+ static QueryTailStatementInspector install(String dataSourceName, QueryTailSettings settings,
+ HibernateMappingContextConfiguration configuration) {
+ if (settings == null || !settings.enabled) {
+ return null
+ }
+ QueryTailStatementInspector inspector = new QueryTailStatementInspector(dataSourceName, settings.resolveFile())
+ configuration.queryTailStatementInspector = inspector
+ inspector
+ }
+
+ /**
+ * Takes the place of the statement inspector the session factory would run, and runs it: the
+ * one set on the configuration, which Hibernate prefers, else the one the settings of the
+ * service registry name, where the properties of the configuration are applied over the
+ * settings the registry builder came with. The configuration calls this as it builds the
+ * session factory, once every setting is in place.
+ *
+ * @param registry the builder of the service registry the session factory is built with
+ * @param configuration the configuration the session factory is built from
+ * @throws ConfigurationException when the inspector the application configured cannot be created
+ */
+ void wrap(StandardServiceRegistryBuilder registry, Configuration configuration) {
+ if (!wrapped) {
+ delegate = configuration.statementInspector ?: resolve(registry.bootstrapServiceRegistry.getService(StrategySelector),
+ registry.settings.get(AvailableSettings.STATEMENT_INSPECTOR))
+ wrapped = true
+ }
+ registry.applySetting(AvailableSettings.STATEMENT_INSPECTOR, this)
+ configuration.statementInspector = this
+ }
+
+ /**
+ * Takes the place of the session factory observer the configuration holds, and tells that one,
+ * set by the application, of every event first. The configuration calls this as it builds the
+ * session factory, after the application had every chance to set its own, even as the build
+ * began.
+ *
+ * @param configuration the configuration the session factory is built from
+ */
+ void observe(Configuration configuration) {
+ SessionFactoryObserver configured = configuration.sessionFactoryObserver
+ // a configuration built again holds this inspector already, in front of the observer it had
+ if (configured !== this) {
+ observer = configured
+ }
+ configuration.sessionFactoryObserver = this
+ }
+
+ /**
+ * @return the inspector the session factory would run without the capture, or null when it
+ * would run none; known once the session factory is built
+ */
+ StatementInspector getDelegate() {
+ delegate
+ }
+
+ /**
+ * @return the session factory observer the application configured, or null when it configured
+ * none; known once the session factory is built
+ */
+ SessionFactoryObserver getObserver() {
+ observer
+ }
+
+ /**
+ * Resolves the inspector an application configured as Hibernate does: an instance, a class, a
+ * class name or a name registered with the strategy selector of the service registry.
+ *
+ * @param strategies the strategy selector of the service registry the session factory is built with
+ * @param configured the configured value, or null
+ * @return the inspector, or null when nothing was configured
+ * @throws ConfigurationException when the configured inspector cannot be loaded or created
+ */
+ @PackageScope
+ static StatementInspector resolve(StrategySelector strategies, Object configured) {
+ if (configured == null) {
+ return null
+ }
+ String name = configured instanceof Class ? ((Class) configured).name : configured.toString()
+ try {
+ strategies.resolveStrategy(StatementInspector, configured)
+ }
+ catch (StrategySelectionException e) {
+ throw new ConfigurationException("Cannot load the statement inspector [${name}] configured with " +
+ AvailableSettings.STATEMENT_INSPECTOR, e)
+ }
+ catch (ClassCastException e) {
+ // the selector creates whatever class is named, and only the cast of what it creates tells it is no inspector
+ throw new ConfigurationException("The statement inspector [${name}] configured with " +
+ "${AvailableSettings.STATEMENT_INSPECTOR} does not implement ${StatementInspector.name}", e)
+ }
+ }
+
+ @Override
+ String inspect(String sql) {
+ String executed = sql
+ if (delegate != null) {
+ String inspected = delegate.inspect(sql)
+ if (inspected != null) {
+ executed = inspected
+ }
+ }
+ QueryTailCapture current = openCapture()
+ if (current != null) {
+ current.record(dataSourceName, executed)
+ }
+ executed
+ }
+
+ /**
+ * Opens the capture, and records the statements of the session factory in the syntax of its dialect.
+ */
+ @Override
+ void sessionFactoryCreated(SessionFactory factory) {
+ if (observer != null) {
+ observer.sessionFactoryCreated(factory)
+ }
+ QueryTailCapture current = openCapture()
+ if (current != null && factory instanceof SessionFactoryImplementor) {
+ current.syntax = syntaxOf(factory.jdbcServices.dialect)
+ }
+ }
+
+ /**
+ * The syntax the statements of a dialect are redacted in: that of its database, where the
+ * redactor knows the database's lexical rules. Cloud Spanner's PostgreSQL dialect is not known,
+ * as the parser of a Spanner PostgreSQL database has not been verified, and neither is SQL
+ * Anywhere's, a Transact-SQL dialect whose database reads {@code //} as a comment; the Informix
+ * dialect among the community dialects is known. Any other dialect, such as SQLite's, Firebird's
+ * or a dialect that is not Hibernate's, is redacted as for a database that is not known: a
+ * statement is then recorded only when every known syntax reads it the same way.
+ *
+ * @param dialect the dialect of a session factory
+ * @return the syntax its statements are redacted in
+ */
+ static QueryTailSyntax syntaxOf(Dialect dialect) {
+ if (dialect instanceof SpannerPostgreSQLDialect || extendsNamed(dialect, COMMUNITY_SQL_ANYWHERE)) {
+ return QueryTailSyntax.UNKNOWN
+ }
+ if (dialect instanceof AbstractTransactSQLDialect) {
+ return QueryTailSyntax.TRANSACT_SQL
+ }
+ if (dialect instanceof SpannerDialect) {
+ return QueryTailSyntax.GOOGLE_SQL
+ }
+ if (dialect instanceof MySQLDialect) {
+ return QueryTailSyntax.MYSQL
+ }
+ if (dialect instanceof CockroachDialect) {
+ return QueryTailSyntax.COCKROACH
+ }
+ if (dialect instanceof PostgreSQLDialect) {
+ return QueryTailSyntax.POSTGRESQL
+ }
+ if (dialect instanceof H2Dialect) {
+ return QueryTailSyntax.H2
+ }
+ if (dialect instanceof OracleDialect) {
+ return QueryTailSyntax.ORACLE
+ }
+ if (dialect instanceof DB2Dialect) {
+ return QueryTailSyntax.DB2
+ }
+ if (dialect instanceof HSQLDialect) {
+ return QueryTailSyntax.HSQLDB
+ }
+ if (dialect instanceof HANADialect) {
+ return QueryTailSyntax.HANA
+ }
+ if (extendsNamed(dialect, COMMUNITY_INFORMIX)) {
+ return QueryTailSyntax.INFORMIX
+ }
+ QueryTailSyntax.UNKNOWN
+ }
+
+ /** @return whether a dialect is, or extends, the named class, which may not be on the class path */
+ private static boolean extendsNamed(Dialect dialect, String className) {
+ for (Class<?> type = dialect.getClass(); type != null; type = type.superclass) {
+ if (type.name == className) {
+ return true
+ }
+ }
+ false
+ }
+
+ @Override
+ void sessionFactoryClosing(SessionFactory factory) {
+ if (observer != null) {
+ observer.sessionFactoryClosing(factory)
+ }
+ }
+
+ /**
+ * Closes the capture.
+ */
+ @Override
+ void sessionFactoryClosed(SessionFactory factory) {
+ try {
+ if (observer != null) {
+ observer.sessionFactoryClosed(factory)
+ }
+ }
+ finally {
+ close()
+ }
+ }
+
+ /**
+ * Stops recording and closes the capture; a capture that was never opened stays unopened.
+ */
+ synchronized void close() {
Review Comment:
One configuration that builds two session factories reuses this inspector and its one capture. Closing either factory closes that capture. The other factory keeps running queries and nothing is written. Building again after close reuses the closed capture.
Give each factory its own capture, or install a new inspector for each build. Hibernate 5 has the same gap. Please build two factories, close one, and assert the other still records.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@grails.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add the query-tail command to find query-heavy pages [grails-core]
Posted by "jdaugherty (via GitHub)" <gi...@apache.org>.
jdaugherty commented on PR #16631:
URL: https://github.com/apache/grails-core/pull/16631#issuecomment-6416875465
@jamesfredley, I pushed 28b2772301 over da16a91754, still a single commit on 9.0.x. It closes two more gaps in the MongoDB redaction:
- **`$const`:** MongoDB accepts `$const` in place of `$literal`, but its operand was read as an expression, so a `$`-prefixed key inside it was kept as an operator. Its operand is now literal data, as with `$literal`.
- **DBRefs:** a DBRef was recognized only when `$ref` came first. MongoDB reads `$ref`, `$id` and `$db` as fields in any order and compares their values whole. That applies to the value of a field, at the top of a condition or of an `$or`/`$and` element, and in an `$elemMatch`. The redactor now does the same, so no key inside a reference's id is recorded.
`MongoCommandRedactorSpec` has exact-output rows and a leak check for each of those positions. `QueryTailCommandListenerSpec` runs the `$const` projection and each DBRef shape against MongoDB. It checks that the server still matches the stored reference and that the capture holds none of its keys. The full `:grails-data-mongodb-core` suite and its code style checks pass locally. The guide's MongoDB paragraph describes both cases.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@grails.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add the query-tail command to find query-heavy pages [grails-core]
Posted by "jdaugherty (via GitHub)" <gi...@apache.org>.
jdaugherty commented on code in PR #16631:
URL: https://github.com/apache/grails-core/pull/16631#discussion_r4381632996
##########
grails-data-hibernate7/core/src/main/groovy/org/apache/grails/data/hibernate7/core/querytail/QueryTailStatementInspector.groovy:
##########
@@ -0,0 +1,354 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * https://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.grails.data.hibernate7.core.querytail
+
+import groovy.transform.CompileStatic
+import groovy.transform.PackageScope
+import org.hibernate.SessionFactory
+import org.hibernate.SessionFactoryObserver
+import org.hibernate.boot.registry.StandardServiceRegistryBuilder
+import org.hibernate.boot.registry.selector.spi.StrategySelectionException
+import org.hibernate.boot.registry.selector.spi.StrategySelector
+import org.hibernate.cfg.AvailableSettings
+import org.hibernate.cfg.Configuration
+import org.hibernate.dialect.AbstractTransactSQLDialect
+import org.hibernate.dialect.CockroachDialect
+import org.hibernate.dialect.DB2Dialect
+import org.hibernate.dialect.Dialect
+import org.hibernate.dialect.H2Dialect
+import org.hibernate.dialect.HANADialect
+import org.hibernate.dialect.HSQLDialect
+import org.hibernate.dialect.MySQLDialect
+import org.hibernate.dialect.OracleDialect
+import org.hibernate.dialect.PostgreSQLDialect
+import org.hibernate.dialect.SpannerDialect
+import org.hibernate.dialect.SpannerPostgreSQLDialect
+import org.hibernate.engine.spi.SessionFactoryImplementor
+import org.hibernate.resource.jdbc.spi.StatementInspector
+
+import org.apache.grails.data.core.querytail.QueryTailCapture
+import org.apache.grails.data.core.querytail.QueryTailSettings
+import org.apache.grails.data.core.querytail.QueryTailSyntax
+import org.grails.datastore.mapping.core.exceptions.ConfigurationException
+import org.grails.orm.hibernate.cfg.HibernateMappingContextConfiguration
+
+/**
+ * The statement inspector GORM installs on a session factory when query-tail capture is enabled. It
+ * records every statement Hibernate is about to run to the {@link QueryTailCapture} of the
+ * application, without its values, and returns the statement as it found it: the database runs
+ * it unchanged. The inspector the session factory would run without the capture keeps running,
+ * wherever the application configured it: with {@code hibernate.session_factory.statement_inspector},
+ * on the configuration with {@link Configuration#setStatementInspector}, or in the settings of the
+ * service registry, as a {@code hibernate.configClass} may. It sees the statement first, and the
+ * statement it returns is the one run and recorded.
+ *
+ * <p>The session factory owns the capture. The inspector observes it: it opens the capture when
+ * the session factory is created, takes from its dialect how the database quotes values, and
+ * closes the capture when the session factory closes, so a session factory that fails to start
+ * leaves no capture open either. An observer the application set on the configuration keeps
+ * being called, even one it sets as the configuration builds the session factory.</p>
+ *
+ * <p>Nothing of this is installed unless {@link QueryTailSettings#SETTING_ENABLED} is set, so an
+ * application that does not capture pays nothing per statement.</p>
+ *
+ * @since 8.1
+ */
+@CompileStatic
+class QueryTailStatementInspector implements StatementInspector, SessionFactoryObserver {
+
+ @Serial
+ private static final long serialVersionUID = 1L
+
+ /** The community dialects, which are not a dependency, that the inspector tells apart by name. */
+ private static final String COMMUNITY_INFORMIX = 'org.hibernate.community.dialect.InformixDialect'
+ private static final String COMMUNITY_SQL_ANYWHERE = 'org.hibernate.community.dialect.SybaseAnywhereDialect'
+
+ /** The connection source name statements are recorded under. */
+ final String dataSourceName
+
+ /** The file statements are recorded to. */
+ final File file
+
+ private SessionFactoryObserver observer
+ private StatementInspector delegate
+ private boolean wrapped
+ private transient volatile QueryTailCapture capture
+ private transient boolean closed
+
+ /**
+ * @param dataSourceName the connection source the inspected session factory belongs to
+ * @param file the capture file statements are recorded to
+ */
+ QueryTailStatementInspector(String dataSourceName, File file) {
+ this.dataSourceName = dataSourceName
+ this.file = file
+ }
+
+ /**
+ * Installs the inspector on the configuration of a session factory when capture is enabled, as
+ * the observer and the statement inspector the configuration puts in place of those of the
+ * session factory when it builds the session factory.
+ *
+ * @param dataSourceName the connection source the session factory is built for
+ * @param settings the query-tail settings of the connection source
+ * @param configuration the configuration the session factory is built from
+ * @return the installed inspector, or null when capture is not enabled
+ */
+ static QueryTailStatementInspector install(String dataSourceName, QueryTailSettings settings,
+ HibernateMappingContextConfiguration configuration) {
+ if (settings == null || !settings.enabled) {
+ return null
+ }
+ QueryTailStatementInspector inspector = new QueryTailStatementInspector(dataSourceName, settings.resolveFile())
+ configuration.queryTailStatementInspector = inspector
+ inspector
+ }
+
+ /**
+ * Takes the place of the statement inspector the session factory would run, and runs it: the
+ * one set on the configuration, which Hibernate prefers, else the one the settings of the
+ * service registry name, where the properties of the configuration are applied over the
+ * settings the registry builder came with. The configuration calls this as it builds the
+ * session factory, once every setting is in place.
+ *
+ * @param registry the builder of the service registry the session factory is built with
+ * @param configuration the configuration the session factory is built from
+ * @throws ConfigurationException when the inspector the application configured cannot be created
+ */
+ void wrap(StandardServiceRegistryBuilder registry, Configuration configuration) {
+ if (!wrapped) {
+ delegate = configuration.statementInspector ?: resolve(registry.bootstrapServiceRegistry.getService(StrategySelector),
+ registry.settings.get(AvailableSettings.STATEMENT_INSPECTOR))
+ wrapped = true
+ }
+ registry.applySetting(AvailableSettings.STATEMENT_INSPECTOR, this)
+ configuration.statementInspector = this
+ }
+
+ /**
+ * Takes the place of the session factory observer the configuration holds, and tells that one,
+ * set by the application, of every event first. The configuration calls this as it builds the
+ * session factory, after the application had every chance to set its own, even as the build
+ * began.
+ *
+ * @param configuration the configuration the session factory is built from
+ */
+ void observe(Configuration configuration) {
+ SessionFactoryObserver configured = configuration.sessionFactoryObserver
+ // a configuration built again holds this inspector already, in front of the observer it had
+ if (configured !== this) {
+ observer = configured
+ }
+ configuration.sessionFactoryObserver = this
+ }
+
+ /**
+ * @return the inspector the session factory would run without the capture, or null when it
+ * would run none; known once the session factory is built
+ */
+ StatementInspector getDelegate() {
+ delegate
+ }
+
+ /**
+ * @return the session factory observer the application configured, or null when it configured
+ * none; known once the session factory is built
+ */
+ SessionFactoryObserver getObserver() {
+ observer
+ }
+
+ /**
+ * Resolves the inspector an application configured as Hibernate does: an instance, a class, a
+ * class name or a name registered with the strategy selector of the service registry.
+ *
+ * @param strategies the strategy selector of the service registry the session factory is built with
+ * @param configured the configured value, or null
+ * @return the inspector, or null when nothing was configured
+ * @throws ConfigurationException when the configured inspector cannot be loaded or created
+ */
+ @PackageScope
+ static StatementInspector resolve(StrategySelector strategies, Object configured) {
+ if (configured == null) {
+ return null
+ }
+ String name = configured instanceof Class ? ((Class) configured).name : configured.toString()
+ try {
+ strategies.resolveStrategy(StatementInspector, configured)
+ }
+ catch (StrategySelectionException e) {
+ throw new ConfigurationException("Cannot load the statement inspector [${name}] configured with " +
+ AvailableSettings.STATEMENT_INSPECTOR, e)
+ }
+ catch (ClassCastException e) {
+ // the selector creates whatever class is named, and only the cast of what it creates tells it is no inspector
+ throw new ConfigurationException("The statement inspector [${name}] configured with " +
+ "${AvailableSettings.STATEMENT_INSPECTOR} does not implement ${StatementInspector.name}", e)
+ }
+ }
+
+ @Override
+ String inspect(String sql) {
+ String executed = sql
+ if (delegate != null) {
+ String inspected = delegate.inspect(sql)
+ if (inspected != null) {
+ executed = inspected
+ }
+ }
+ QueryTailCapture current = openCapture()
+ if (current != null) {
+ current.record(dataSourceName, executed)
+ }
+ executed
+ }
+
+ /**
+ * Opens the capture, and records the statements of the session factory in the syntax of its dialect.
+ */
+ @Override
+ void sessionFactoryCreated(SessionFactory factory) {
+ if (observer != null) {
+ observer.sessionFactoryCreated(factory)
+ }
+ QueryTailCapture current = openCapture()
+ if (current != null && factory instanceof SessionFactoryImplementor) {
+ current.syntax = syntaxOf(factory.jdbcServices.dialect)
+ }
+ }
+
+ /**
+ * The syntax the statements of a dialect are redacted in: that of its database, where the
+ * redactor knows the database's lexical rules. Cloud Spanner's PostgreSQL dialect is not known,
+ * as the parser of a Spanner PostgreSQL database has not been verified, and neither is SQL
+ * Anywhere's, a Transact-SQL dialect whose database reads {@code //} as a comment; the Informix
+ * dialect among the community dialects is known. Any other dialect, such as SQLite's, Firebird's
+ * or a dialect that is not Hibernate's, is redacted as for a database that is not known: a
+ * statement is then recorded only when every known syntax reads it the same way.
+ *
+ * @param dialect the dialect of a session factory
+ * @return the syntax its statements are redacted in
+ */
+ static QueryTailSyntax syntaxOf(Dialect dialect) {
+ if (dialect instanceof SpannerPostgreSQLDialect || extendsNamed(dialect, COMMUNITY_SQL_ANYWHERE)) {
+ return QueryTailSyntax.UNKNOWN
+ }
+ if (dialect instanceof AbstractTransactSQLDialect) {
+ return QueryTailSyntax.TRANSACT_SQL
+ }
+ if (dialect instanceof SpannerDialect) {
+ return QueryTailSyntax.GOOGLE_SQL
+ }
+ if (dialect instanceof MySQLDialect) {
+ return QueryTailSyntax.MYSQL
+ }
+ if (dialect instanceof CockroachDialect) {
+ return QueryTailSyntax.COCKROACH
+ }
+ if (dialect instanceof PostgreSQLDialect) {
+ return QueryTailSyntax.POSTGRESQL
+ }
+ if (dialect instanceof H2Dialect) {
+ return QueryTailSyntax.H2
+ }
+ if (dialect instanceof OracleDialect) {
+ return QueryTailSyntax.ORACLE
+ }
+ if (dialect instanceof DB2Dialect) {
+ return QueryTailSyntax.DB2
+ }
+ if (dialect instanceof HSQLDialect) {
+ return QueryTailSyntax.HSQLDB
+ }
+ if (dialect instanceof HANADialect) {
+ return QueryTailSyntax.HANA
+ }
+ if (extendsNamed(dialect, COMMUNITY_INFORMIX)) {
+ return QueryTailSyntax.INFORMIX
+ }
+ QueryTailSyntax.UNKNOWN
+ }
+
+ /** @return whether a dialect is, or extends, the named class, which may not be on the class path */
+ private static boolean extendsNamed(Dialect dialect, String className) {
+ for (Class<?> type = dialect.getClass(); type != null; type = type.superclass) {
+ if (type.name == className) {
+ return true
+ }
+ }
+ false
+ }
+
+ @Override
+ void sessionFactoryClosing(SessionFactory factory) {
+ if (observer != null) {
+ observer.sessionFactoryClosing(factory)
+ }
+ }
+
+ /**
+ * Closes the capture.
+ */
+ @Override
+ void sessionFactoryClosed(SessionFactory factory) {
+ try {
+ if (observer != null) {
+ observer.sessionFactoryClosed(factory)
+ }
+ }
+ finally {
+ close()
+ }
+ }
+
+ /**
+ * Stops recording and closes the capture; a capture that was never opened stays unopened.
+ */
+ synchronized void close() {
Review Comment:
Fixed in 52808b3641, on both Hibernate versions. Each session factory now gets its own inspector, and so its own capture handle:
- `install()` stores only the capture file on the configuration.
- Every `buildSessionFactory()` creates a new `QueryTailStatementInspector` for that factory, wraps the statement inspector and observer the application set, and builds the factory.
- After the build, the configuration's statement inspector and observer are restored, so the next build wraps them afresh instead of wrapping the previous inspector.
Closing one factory now closes only its own capture handle. The file stays open while any other factory still holds one, and a factory built after every handle has closed opens a new capture.
Hibernate 5 had a second gap in the same scenario. Its registry-closing observer destroyed the configuration's most recent service registry, so closing the first factory tore down the second's registry, and the second failed with `Provider is closed`. Each closing observer now holds the registry of its own build.
The new feature in both versions' `HibernateConnectionSourceFactorySpec` covers your scenario:
- One configuration builds two factories, each runs a statement, the first closes, and the second runs another. All three statements are recorded.
- Then the second closes, and a third factory built from the same configuration records into a new run.
Against the previous code it fails on both versions. On H7 the surviving factory's statement is missing from the capture, and on H5 the surviving factory fails with `Provider is closed`. The full `grails-data-hibernate5-core` and `grails-data-hibernate7-core` suites pass locally, along with their code style checks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@grails.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add the query-tail command to find query-heavy pages [grails-core]
Posted by "jdaugherty (via GitHub)" <gi...@apache.org>.
jdaugherty commented on code in PR #16631:
URL: https://github.com/apache/grails-core/pull/16631#discussion_r4371058328
##########
grails-data-mongodb/core/src/main/groovy/org/apache/grails/data/mongo/core/querytail/MongoCommandRedactor.groovy:
##########
@@ -0,0 +1,363 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * https://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.grails.data.mongo.core.querytail
+
+import groovy.transform.CompileStatic
+import org.bson.BsonDocument
+import org.bson.BsonValue
+
+import org.apache.grails.data.core.querytail.QueryTailRedactor
+
+/**
+ * Takes the values out of a MongoDB command before it is captured. A command is a document, so
+ * there is nothing to parse: every value becomes {@link QueryTailRedactor#PLACEHOLDER}, and the
+ * keys, which are the options of the command, its operators and the field names of the collection,
+ * are kept, except where a key may be data. The keys of a document stored in a map property, or in
+ * a property or a collection the {@linkplain MongoMappedFields mapping} does not know, are recorded
+ * as the placeholder too, and so is every segment of a dotted path past such a key. A numeric key
+ * is kept only where it indexes an array the mapping knows; in a map, in an element of an array,
+ * which is a map or a document unless the field is a list of lists, or at the top of a document,
+ * it is data. A {@code $} key is kept as an operator only where operators go, in a condition, a
+ * pipeline and the operators of an update; in stored data, the documents an insert writes, a
+ * replacement document and the values an update operator writes, it is a field name, which MongoDB
+ * allows to start with {@code $}, and is recorded as the placeholder unless the mapping knows it.
+ * The entries the driver adds for its session, the cluster time and the transaction, which say
+ * nothing about the command, are left out.
+ *
+ * <p>A command is recorded as its name, the database and collection it runs against, and the
+ * document of its remaining entries: {@code find shop.book {"filter": {"title": ?}, "limit": ?}}.</p>
+ *
+ * @since 8.1
+ */
+@CompileStatic
+final class MongoCommandRedactor {
+
+ private static final String PLACEHOLDER = QueryTailRedactor.PLACEHOLDER
+
+ private static final String DATABASE = '$db'
+
+ private static final String COLLECTION = 'collection'
+
+ /** The entries the driver adds, which describe its session and transaction rather than the command. */
+ private static final Set<String> LEFT_OUT = ['lsid', '$clusterTime', 'txnNumber', 'autocommit',
+ 'startTransaction', '$readPreference', 'apiVersion', 'apiStrict', 'apiDeprecationErrors'] as Set<String>
+
+ /**
+ * The options whose document, or documents, are keyed by the fields of the collection, wherever they appear,
+ * including those of the operations of a client-level {@code bulkWrite}.
+ */
+ private static final Set<String> FIELD_KEYED = ['filter', 'query', 'q', 'u', 'update', 'document', 'documents',
+ 'updateMods', 'pipeline', 'sort', 'projection', 'fields', 'key', 'hint', 'min', 'max', 'arrayFilters',
+ 'partialFilterExpression', 'wildcardProjection'] as Set<String>
+
+ /** The options whose documents are stored data: what an insert writes. */
+ private static final Set<String> DATA_KEYED = ['documents', 'document'] as Set<String>
+
+ /** The options that hold an update: update operators, a replacement document or a pipeline. */
+ private static final Set<String> UPDATE_KEYED = ['u', 'update', 'updateMods'] as Set<String>
+
+ /** The update operators whose fields take a condition rather than the value to write. */
+ private static final Set<String> CONDITION_OPERATORS = ['$pull', '$currentDate'] as Set<String>
+
+ /** The update operators that add to an array, whose value may hold modifiers. */
+ private static final Set<String> PUSH_OPERATORS = ['$push', '$addToSet'] as Set<String>
+
+ private static final String EACH = '$each'
+
+ /** The modifiers of a {@code $push} or an {@code $addToSet}. */
+ private static final Set<String> PUSH_MODIFIERS = [EACH, '$slice', '$sort', '$position'] as Set<String>
+
+ /** The operators of a pipeline whose operand is literal data rather than an expression. */
+ private static final Set<String> LITERAL_OPERATORS = ['$literal', '$documents'] as Set<String>
+
+ private MongoCommandRedactor() {
+ }
+
+ /**
+ * Redacts a command.
+ *
+ * @param database the database the command runs against
+ * @param commandName the name of the command, which is its first key
+ * @param command the command as the driver sends it
+ * @param fields the fields the mapping of the datastore knows, or null before the datastore is built
+ * @return the command with its values replaced
+ */
+ static String redact(String database, String commandName, BsonDocument command, MongoMappedFields fields) {
+ String collection = null
+ String collectionKey = null
+ BsonValue first = command.get(commandName)
+ if (first != null && first.isString()) {
+ collection = first.asString().value
+ collectionKey = commandName
+ }
+ else if (command.get(COLLECTION)?.isString()) {
+ collection = command.getString(COLLECTION).value
+ collectionKey = COLLECTION
+ }
+ StringBuilder out = new StringBuilder(commandName)
+ if (database != null || collection != null) {
+ out.append(' ')
+ if (database != null) {
+ out.append(database)
+ }
+ if (collection != null) {
+ out.append(database != null ? '.' : '').append(collection)
+ }
+ }
+ Scope options = Scope.options(collection, fields)
+ StringBuilder body = new StringBuilder()
+ for (Map.Entry<String, BsonValue> entry in command.entrySet()) {
+ String key = entry.key
+ if (key == DATABASE || key == collectionKey || LEFT_OUT.contains(key)) {
+ continue
+ }
+ body.append(body.length() == 0 ? '{' : ', ')
+ appendEntry(body, key, entry.value, options)
+ }
+ if (body.length() > 0) {
+ out.append(' ').append(body).append('}')
+ }
+ out.toString()
+ }
+
+ private static void appendEntry(StringBuilder out, String key, BsonValue value, Scope scope) {
+ Scope child = scope.of(key)
+ out.append(child.keyText).append(': ')
+ appendValue(out, value, child)
+ }
+
+ private static void appendValue(StringBuilder out, BsonValue value, Scope scope) {
+ if (value.isDocument()) {
+ out.append('{')
+ boolean first = true
+ for (Map.Entry<String, BsonValue> entry in value.asDocument().entrySet()) {
+ if (!first) {
+ out.append(', ')
+ }
+ first = false
+ appendEntry(out, entry.key, entry.value, scope)
+ }
+ out.append('}')
+ }
+ else if (value.isArray()) {
+ out.append('[')
+ boolean first = true
+ Scope element = scope.element()
+ for (BsonValue item in value.asArray()) {
+ if (!first) {
+ out.append(', ')
+ }
+ first = false
+ appendValue(out, item, element)
+ }
+ out.append(']')
+ }
+ else {
+ out.append(PLACEHOLDER)
+ }
+ }
+
+ private static String quote(String key) {
+ StringBuilder out = new StringBuilder(key.length() + 2).append('"')
+ for (int i = 0; i < key.length(); i++) {
+ char c = key.charAt(i)
+ if (c == '"' || c == '\\') {
+ out.append('\\').append(c)
+ }
+ else if (c < 0x20) {
+ out.append(String.format('\\u%04x', (int) c))
+ }
+ else {
+ out.append(c)
+ }
+ }
+ out.append('"').toString()
+ }
+
+ /**
+ * Where a document sits in the command, which decides what its keys are: the options of the
+ * command, kept; the fields and operators of a condition, a pipeline, a sort or a projection, where
+ * a {@code $} key is an operator; an update, whose {@code $} keys are update operators; or stored
+ * data, the documents an insert writes and the values an update writes, where a {@code $} key is a
+ * field name like any other and is kept only when the mapping knows it.
+ */
+ @CompileStatic
+ private static final class Scope {
+
+ final String collection
+ final MongoMappedFields fields
+ final Kind kind
+ /** The update operator whose fields a document of kind {@link Kind#UPDATE_FIELDS} holds. */
+ final String operator
+ /** The field path of the document; null in a document keyed by fields the mapping does not know. */
+ final List<String> path
+ /** How many indices into the field at the path have been taken: 0 at the field, 1 in one of its elements. */
+ final int depth
+ final String keyText
+
+ private Scope(String collection, MongoMappedFields fields, Kind kind, String operator, List<String> path,
+ int depth, String keyText) {
+ this.collection = collection
+ this.fields = fields
+ this.kind = kind
+ this.operator = operator
+ this.path = path
+ this.depth = depth
+ this.keyText = keyText
+ }
+
+ static Scope options(String collection, MongoMappedFields fields) {
+ new Scope(collection, fields, Kind.OPTIONS, null, Collections.<String>emptyList(), 0, null)
+ }
+
+ /**
+ * @return the scope of an element of an array in this scope: one index deeper into the field
+ * at the path, so that a numeric key inside the element is data unless the elements are arrays;
+ * the elements of an update that is an array are the stages of a pipeline
+ */
+ Scope element() {
+ if (kind == Kind.UPDATE) {
+ return child(Kind.QUERY, null, path, depth, keyText)
+ }
+ if (kind == Kind.PUSH) {
+ return child(Kind.DATA, null, path, depth + 2, keyText)
+ }
+ if ((kind == Kind.QUERY || kind == Kind.DATA) && path != null && !path.isEmpty()) {
+ return child(kind, null, path, depth + 1, keyText)
+ }
+ this
+ }
+
+ /**
+ * @return the scope of the value of a key in a document of this scope, with the key as it is recorded
+ */
+ Scope of(String key) {
+ if (kind == Kind.OPTIONS) {
+ Kind valueKind = DATA_KEYED.contains(key) ? Kind.DATA :
+ UPDATE_KEYED.contains(key) ? Kind.UPDATE :
+ FIELD_KEYED.contains(key) ? Kind.QUERY : Kind.OPTIONS
+ return child(valueKind, null, Collections.<String>emptyList(), 0, quote(key))
+ }
+ if (kind == Kind.QUERY && key.startsWith('$')) {
Review Comment:
Fixed in f69625477f. In a condition, the operand of a comparison (`$eq`, `$ne`, `$gt`, `$gte`, `$lt`, `$lte`, `$in`, `$nin`, `$all`) is now read as data, so `{"attrs":{"$eq":{"$example-secret":"value"}}}` records `{"attrs": {"$eq": {?: ?}}}`. The same holds under `$not`, `$elemMatch`, `$or`, `$pull`, `arrayFilters` and a `$match` stage. In an aggregation expression (`$expr`, the `let` of a command, and stages such as `$project`, `$group` and `$set`), a `$` key can only be an operator, so `{"$expr": {"$eq": [{"$toLower": "$title"}, "x"]}}` keeps `$toLower`; `$literal` stays data there.
`QueryTailCommandListenerSpec` now runs your filter against a real mongod, along with that `$expr` aggregation. It asserts that MongoDB returns the stored document both ways, that the capture keeps `$eq`, `$expr` and `$toLower`, and that `example-secret` does not reach the file.
While tracing the scopes I found more places where a key that may be data was kept, and fixed them in the same change. Each has a redactor row:
- the keys of a comment document (`comment`, `$comment`) and of the constants of an update (`c`)
- a `$jsonSchema`, whose `enum` lists values
- the `validator` and text-index `weights` documents, whose keys were kept as option names
- a `$literal` in the `let` of a command
Where it is not clear whether an operand is an expression, such as a find projection, the `let` of a `$lookup`, or `$all: [{"$elemMatch": ...}]`, a nested operator is now recorded as `?`. That withholds more, never less.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@grails.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add the query-tail command to find query-heavy pages [grails-core]
Posted by "jdaugherty (via GitHub)" <gi...@apache.org>.
jdaugherty commented on code in PR #16631:
URL: https://github.com/apache/grails-core/pull/16631#discussion_r4370696720
##########
grails-datamapping-core/src/main/groovy/org/apache/grails/data/core/querytail/QueryTailRequestAttribution.groovy:
##########
@@ -0,0 +1,410 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * https://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.grails.data.core.querytail
+
+import java.lang.reflect.Method
+import java.nio.charset.StandardCharsets
+import java.util.concurrent.atomic.AtomicLong
+import java.util.regex.Pattern
+
+import groovy.transform.CompileStatic
+
+import org.springframework.util.ClassUtils
+import org.springframework.web.context.request.RequestAttributes
+import org.springframework.web.context.request.RequestContextHolder
+import org.springframework.web.context.request.ServletRequestAttributes
+
+/**
+ * Attributes a captured statement to the web request on the current thread. Grails keeps the
+ * request, and the controller and action it resolved, on the thread that serves it, so a statement
+ * run by an interceptor, the action or the view all land on the same page. The first statement of
+ * a request gives it an id that every later statement of that request repeats, which is how the
+ * {@code query-tail} command counts statements per request. Work on any other thread is attributed to
+ * the thread by name.
+ *
+ * <p>The request is recorded as its method and path, with the values in the path replaced by
+ * {@code ?}: the values of the parameters the URL mapping took from it and those of its query, found
+ * whether or not the client percent-encoded them, and any segment that is a number, a UUID or a long
+ * hexadecimal token, so that an id, a token or a key carried in a path never reaches the capture
+ * file. A segment a value is glued into, as a mapping such as {@code /reset/${token}_confirm}
+ * produces, is replaced whole, whether the value shows as sent or decoded. Until the URL mapping has run, as for a statement of a
+ * filter that runs before the controller is chosen, or for a request that never reaches one, no
+ * segment's role is known and every segment is replaced; only the controller the framework set on
+ * the request says that it has run, never a parameter, which a client can send.</p>
+ *
+ * <p>Spring Web is optional: without it on the classpath every statement is attributed to its
+ * thread.</p>
+ *
+ * @since 8.1
+ */
+@CompileStatic
+class QueryTailRequestAttribution {
+
+ /** The request attribute holding the id this attribution gave a request. */
+ public static final String REQUEST_ID_ATTRIBUTE = 'org.apache.grails.data.core.querytail.REQUEST_ID'
+
+ /** The request attribute Grails sets the controller name in. */
+ public static final String CONTROLLER_NAME_ATTRIBUTE = 'org.grails.CONTROLLER_NAME_ATTRIBUTE'
+
+ /** The request attribute Grails sets the controller namespace in. */
+ public static final String CONTROLLER_NAMESPACE_ATTRIBUTE = 'org.grails.CONTROLLER_NAMESPACE_ATTRIBUTE'
+
+ /** The request attribute Grails sets the action name in. */
+ public static final String ACTION_NAME_ATTRIBUTE = 'org.grails.ACTION_NAME_ATTRIBUTE'
+
+ private static final boolean SPRING_WEB_PRESENT = ClassUtils.isPresent(
+ 'org.springframework.web.context.request.RequestContextHolder', QueryTailRequestAttribution.classLoader)
+
+ private final AtomicLong requestCounter = new AtomicLong()
+ private final String requestIdPrefix
+
+ /**
+ * An attribution whose request ids are {@code r1}, {@code r2} and so on.
+ */
+ QueryTailRequestAttribution() {
+ this('')
+ }
+
+ /**
+ * @param requestIdPrefix what every request id starts with, which tells the requests of one JVM
+ * from those of another writing the same capture
+ */
+ QueryTailRequestAttribution(String requestIdPrefix) {
+ this.requestIdPrefix = requestIdPrefix ?: ''
+ }
+
+ /**
+ * Fills in the request id, page and request of a record from the current thread.
+ *
+ * @param record the record of a statement that just ran on this thread
+ * @return the same record
+ */
+ QueryTailRecord attribute(QueryTailRecord record) {
+ if (SPRING_WEB_PRESENT && WebRequestLookup.attribute(record, requestCounter, requestIdPrefix)) {
+ return record
+ }
+ record.requestId = null
+ record.request = null
+ record.page = QueryTailRecord.THREAD_PAGE_PREFIX + Thread.currentThread().name
+ record
+ }
+
+ /**
+ * Keeps every reference to Spring Web, in a signature as well as in a body, in a class that is only
+ * loaded once Spring Web is known to be present. The outer class can then be loaded, and reflected
+ * on, without Spring Web on the classpath.
+ */
+ @CompileStatic
+ private static class WebRequestLookup {
+
+ private static final String REQUEST_ID_PREFIX = 'r'
+
+ private static final String PLACEHOLDER = '?'
+
+ /** The parameters a URL mapping sets that name code rather than carry a value. */
+ private static final Set<String> MAPPING_PARAMETERS = ['controller', 'action', 'namespace', 'plugin', 'format'] as Set<String>
+
+ /** A segment that is a number, a UUID or a long hexadecimal token is a value whatever the mapping says. */
+ private static final Pattern VALUE_SHAPED = Pattern.compile(
+ '(?i)[0-9]+|[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}|[0-9a-f]{32,}')
+
+ private static final boolean SERVLET_PRESENT = ClassUtils.isPresent(
+ 'jakarta.servlet.http.HttpServletRequest', WebRequestLookup.classLoader)
+
+ /** The {@code getParams()} of a Grails web request, which holds the parameters its URL mapping took, by class. */
+ private static final ClassValue<Method> PARAMETERS = new ClassValue<Method>() {
+ @Override
+ protected Method computeValue(Class<?> type) {
+ try {
+ return type.getMethod('getParams')
+ }
+ catch (NoSuchMethodException ignored) {
+ return null
+ }
+ }
+ }
+
+ /**
+ * Fills in a record from the request on the current thread.
+ *
+ * @param record the record to fill in
+ * @param requestCounter the counter the next request id is taken from
+ * @param requestIdPrefix what every request id starts with
+ * @return whether there is a request on the current thread
+ */
+ static boolean attribute(QueryTailRecord record, AtomicLong requestCounter, String requestIdPrefix) {
+ RequestAttributes attributes = RequestContextHolder.getRequestAttributes()
+ if (attributes == null) {
+ return false
+ }
+ record.requestId = requestId(attributes, requestCounter, requestIdPrefix)
+ record.page = page(attributes)
+ record.request = SERVLET_PRESENT ? ServletRequestLookup.describe(attributes) : null
+ true
+ }
+
+ /**
+ * Builds the request line: the method and the path, with the values in the path, the parameters
+ * the URL mapping took from it among them, replaced.
+ *
+ * @param method the HTTP method
+ * @param path the request path
+ * @param attributes the request, whose mapped parameters are read when it is a Grails web request
+ * @param parameters the parameters of the servlet request, by name, each a {@code String[]}
+ */
+ static String describe(String method, String path, RequestAttributes attributes, Map<?, ?> parameters) {
+ // only the framework sets the controller attribute; a client can send a controller parameter
+ if (attributes.getAttribute(CONTROLLER_NAME_ATTRIBUTE, RequestAttributes.SCOPE_REQUEST) == null) {
+ return "${method} ${withholdPath(path)}".toString()
+ }
+ Set<String> values = new HashSet<>()
+ addValues(values, mappedParameters(attributes))
+ addValues(values, parameters)
+ "${method} ${redactPath(path, values)}".toString()
+ }
+
+ /**
+ * Replaces every segment of a path: before the URL mapping has run, nothing tells a literal
+ * segment from a token, so only the shape of the path is recorded.
+ */
+ private static String withholdPath(String path) {
+ String[] segments = path.split('/', -1)
+ StringBuilder out = new StringBuilder(path.length())
+ for (int i = 0; i < segments.length; i++) {
+ if (i > 0) {
+ out.append('/')
+ }
+ out.append(segments[i].isEmpty() ? segments[i] : PLACEHOLDER)
+ }
+ out.toString()
+ }
+
+ private static String requestId(RequestAttributes attributes, AtomicLong requestCounter, String requestIdPrefix) {
+ synchronized (attributes) {
+ Object existing = attributes.getAttribute(REQUEST_ID_ATTRIBUTE, RequestAttributes.SCOPE_REQUEST)
+ if (existing != null) {
+ return existing.toString()
+ }
+ String id = requestIdPrefix + REQUEST_ID_PREFIX + requestCounter.incrementAndGet()
+ attributes.setAttribute(REQUEST_ID_ATTRIBUTE, id, RequestAttributes.SCOPE_REQUEST)
+ return id
+ }
+ }
+
+ private static String page(RequestAttributes attributes) {
+ Object controller = attributes.getAttribute(CONTROLLER_NAME_ATTRIBUTE, RequestAttributes.SCOPE_REQUEST)
+ if (controller == null) {
+ return null
+ }
+ Object namespace = attributes.getAttribute(CONTROLLER_NAMESPACE_ATTRIBUTE, RequestAttributes.SCOPE_REQUEST)
+ Object action = attributes.getAttribute(ACTION_NAME_ATTRIBUTE, RequestAttributes.SCOPE_REQUEST)
+ StringBuilder page = new StringBuilder()
+ if (namespace != null) {
+ page.append(namespace).append(':')
+ }
+ page.append(controller)
+ if (action != null) {
+ page.append('/').append(action)
+ }
+ page.toString()
+ }
+
+ /**
+ * @return the parameters of a Grails web request, which include those its URL mapping took from
+ * the path, or null for a request that is not one
+ */
+ private static Map<?, ?> mappedParameters(RequestAttributes attributes) {
+ Method parameters = PARAMETERS.get(attributes.getClass())
+ if (parameters == null) {
+ return null
+ }
+ try {
+ Object value = parameters.invoke(attributes)
+ return value instanceof Map ? (Map<?, ?>) value : null
+ }
+ catch (ReflectiveOperationException | RuntimeException ignored) {
+ return null
+ }
+ }
+
+ private static void addValues(Set<String> values, Map<?, ?> parameters) {
+ if (parameters == null) {
+ return
+ }
+ for (Map.Entry<?, ?> entry in parameters.entrySet()) {
+ if (MAPPING_PARAMETERS.contains(String.valueOf(entry.key))) {
+ continue
+ }
+ Object value = entry.value
+ if (value instanceof CharSequence) {
+ values.add(value.toString())
+ }
+ else if (value instanceof Object[]) {
+ for (Object element in (Object[]) value) {
+ if (element instanceof CharSequence) {
+ values.add(element.toString())
+ }
+ }
+ }
+ }
+ }
+
+ /**
+ * Replaces the values in a path: a segment that is one of the values, with or without an
+ * extension, a run of segments that spells a value holding slashes, any segment shaped like a
+ * value, and any segment a value is glued into. The segments are matched as the client sent
+ * them and percent-decoded, since the parameters are decoded while the request URI is not; the
+ * path is written as sent.
+ */
+ private static String redactPath(String path, Set<String> values) {
+ String[] segments = path.split('/', -1)
+ String[] decoded = new String[segments.length]
+ boolean[] redacted = new boolean[segments.length]
+ for (int i = 0; i < segments.length; i++) {
+ decoded[i] = decode(segments[i])
+ redacted[i] = isValue(segments[i], values) || isValue(decoded[i], values)
+ }
+ for (String value in values) {
+ if (value.contains('/')) {
+ String[] run = value.split('/', -1)
+ markRun(segments, redacted, run)
+ markDecodedRun(decoded, redacted, run)
+ }
+ }
+ StringBuilder out = new StringBuilder(path.length())
+ for (int i = 0; i < segments.length; i++) {
+ if (i > 0) {
+ out.append('/')
+ }
+ boolean replaced = redacted[i] || holdsValue(segments[i], values) || holdsValue(decoded[i], values)
+ out.append(replaced ? PLACEHOLDER : segments[i])
+ }
+ out.toString()
+ }
+
+ /**
+ * Whether a value is glued into a segment that is not one itself, as a mapping that glues static
+ * text to a variable produces. A value of three or more characters counts wherever it appears, a
+ * shorter one only where no letter or digit continues it on either side, so that a one-letter
+ * query value does not take out every word of the path.
+ */
+ private static boolean holdsValue(String text, Set<String> values) {
+ for (String value in values) {
+ if (value.isEmpty()) {
+ continue
+ }
+ for (int at = text.indexOf(value); at >= 0; at = text.indexOf(value, at + 1)) {
+ int end = at + value.length()
+ if (value.length() >= 3 ||
+ ((at == 0 || !Character.isLetterOrDigit(text.charAt(at - 1))) &&
+ (end == text.length() || !Character.isLetterOrDigit(text.charAt(end))))) {
+ return true
+ }
+ }
+ }
+ false
+ }
+
+ /**
+ * @return the segment with its percent-encoding decoded, or as it is when it has none or is not
+ * valid percent-encoding; a plus sign is kept, since it is literal in a path
+ */
+ private static String decode(String segment) {
+ if (segment.indexOf('%') < 0) {
+ return segment
+ }
+ try {
+ return URLDecoder.decode(segment.replace('+', '%2B'), StandardCharsets.UTF_8)
+ }
+ catch (IllegalArgumentException ignored) {
+ return segment
+ }
+ }
+
+ private static boolean isValue(String segment, Set<String> values) {
+ if (segment.isEmpty()) {
+ return false
+ }
+ int dot = segment.lastIndexOf('.')
+ String bare = dot > 0 ? segment.substring(0, dot) : segment
+ values.contains(segment) || values.contains(bare) || VALUE_SHAPED.matcher(bare).matches()
+ }
+
+ /**
+ * Marks the segments whose decoded forms spell the run; a decoded segment may itself hold
+ * slashes, which the client encoded, so the decoded forms are split again before matching
+ * and a match marks the segments the matched parts came from.
+ */
+ private static void markDecodedRun(String[] decoded, boolean[] redacted, String[] run) {
+ List<String> parts = []
+ List<Integer> owners = []
+ for (int i = 0; i < decoded.length; i++) {
+ for (String part in decoded[i].split('/', -1)) {
+ parts.add(part)
+ owners.add(i)
+ }
+ }
+ boolean[] partRedacted = new boolean[parts.size()]
+ markRun(parts as String[], partRedacted, run)
+ for (int i = 0; i < partRedacted.length; i++) {
+ if (partRedacted[i]) {
+ redacted[owners.get(i)] = true
+ }
+ }
+ }
+
+ private static void markRun(String[] segments, boolean[] redacted, String[] run) {
+ if (run.length == 0 || run.length > segments.length) {
+ return
+ }
+ for (int start = 0; start + run.length <= segments.length; start++) {
+ boolean matches = true
+ for (int i = 0; i < run.length && matches; i++) {
+ matches = segments[start + i] == run[i]
+ }
+ if (matches) {
+ for (int i = 0; i < run.length; i++) {
+ redacted[start + i] = true
+ }
+ }
+ }
+ }
+ }
+
+ /**
+ * Keeps every reference to the servlet API in a class that is only loaded once the servlet API is
+ * known to be present, as {@link WebRequestLookup} does for Spring Web.
+ */
+ @CompileStatic
+ private static class ServletRequestLookup {
+
+ /**
+ * @return the request line of the servlet request on the current thread, or null when the request
+ * is not a servlet request
+ */
+ static String describe(RequestAttributes attributes) {
+ if (!(attributes instanceof ServletRequestAttributes)) {
+ return null
+ }
+ def request = ((ServletRequestAttributes) attributes).request
Review Comment:
Fixed in 73c20a13c7 (folded into the feature commit). The path parameters of a segment, everything after its first semicolon, are no longer part of the request line: `redactPath` splits them off before matching, matches the rest of the segment as before, and writes each path parameter as `;?`. So `/book/index;jsessionid=abc123secretXYZ` is recorded as `GET /book/index;?`, and `/book/show/42;jsessionid=...;v=2` as `GET /book/show/?;?;?` (the `42` is still found, which the path parameter used to hide). Before the URL mapping has run the whole segment is withheld anyway. Covered in the table of `QueryTailRequestAttributionSpec` and through the capture API in `QueryTailCaptureSpec`, which asserts that neither the session id nor `jsessionid` reaches the file; the guide and javadoc say so.
##########
grails-data-mongodb/core/src/main/groovy/org/apache/grails/data/mongo/core/querytail/MongoCommandRedactor.groovy:
##########
@@ -0,0 +1,265 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * https://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.grails.data.mongo.core.querytail
+
+import groovy.transform.CompileStatic
+import org.bson.BsonDocument
+import org.bson.BsonValue
+
+import org.apache.grails.data.core.querytail.QueryTailRedactor
+
+/**
+ * Takes the values out of a MongoDB command before it is captured. A command is a document, so
+ * there is nothing to parse: every value becomes {@link QueryTailRedactor#PLACEHOLDER}, and the
+ * keys, which are the options of the command, its operators and the field names of the collection,
+ * are kept, except where a key may be data. The keys of a document stored in a map property, or in
+ * a property or a collection the {@linkplain MongoMappedFields mapping} does not know, are recorded
+ * as the placeholder too, and so is every segment of a dotted path past such a key. A numeric key
+ * is kept only where it indexes an array the mapping knows; in a map, in an element of an array,
+ * which is a map or a document unless the field is a list of lists, or at the top of a document,
+ * it is data. The entries the driver adds for its session, the cluster time and the transaction,
+ * which say nothing about the command, are left out.
+ *
+ * <p>A command is recorded as its name, the database and collection it runs against, and the
+ * document of its remaining entries: {@code find shop.book {"filter": {"title": ?}, "limit": ?}}.</p>
+ *
+ * @since 8.1
+ */
+@CompileStatic
+final class MongoCommandRedactor {
+
+ private static final String PLACEHOLDER = QueryTailRedactor.PLACEHOLDER
+
+ private static final String DATABASE = '$db'
+
+ private static final String COLLECTION = 'collection'
+
+ /** The entries the driver adds, which describe its session and transaction rather than the command. */
+ private static final Set<String> LEFT_OUT = ['lsid', '$clusterTime', 'txnNumber', 'autocommit',
+ 'startTransaction', '$readPreference', 'apiVersion', 'apiStrict', 'apiDeprecationErrors'] as Set<String>
+
+ /**
+ * The options whose document, or documents, are keyed by the fields of the collection, wherever they appear,
+ * including those of the operations of a client-level {@code bulkWrite}.
+ */
+ private static final Set<String> FIELD_KEYED = ['filter', 'query', 'q', 'u', 'update', 'document', 'documents',
+ 'updateMods', 'pipeline', 'sort', 'projection', 'fields', 'key', 'hint', 'min', 'max', 'arrayFilters',
+ 'partialFilterExpression', 'wildcardProjection'] as Set<String>
+
+ private MongoCommandRedactor() {
+ }
+
+ /**
+ * Redacts a command.
+ *
+ * @param database the database the command runs against
+ * @param commandName the name of the command, which is its first key
+ * @param command the command as the driver sends it
+ * @param fields the fields the mapping of the datastore knows, or null before the datastore is built
+ * @return the command with its values replaced
+ */
+ static String redact(String database, String commandName, BsonDocument command, MongoMappedFields fields) {
+ String collection = null
+ String collectionKey = null
+ BsonValue first = command.get(commandName)
+ if (first != null && first.isString()) {
+ collection = first.asString().value
+ collectionKey = commandName
+ }
+ else if (command.get(COLLECTION)?.isString()) {
+ collection = command.getString(COLLECTION).value
+ collectionKey = COLLECTION
+ }
+ StringBuilder out = new StringBuilder(commandName)
+ if (database != null || collection != null) {
+ out.append(' ')
+ if (database != null) {
+ out.append(database)
+ }
+ if (collection != null) {
+ out.append(database != null ? '.' : '').append(collection)
+ }
+ }
+ Scope options = Scope.options(collection, fields)
+ StringBuilder body = new StringBuilder()
+ for (Map.Entry<String, BsonValue> entry in command.entrySet()) {
+ String key = entry.key
+ if (key == DATABASE || key == collectionKey || LEFT_OUT.contains(key)) {
+ continue
+ }
+ body.append(body.length() == 0 ? '{' : ', ')
+ appendEntry(body, key, entry.value, options)
+ }
+ if (body.length() > 0) {
+ out.append(' ').append(body).append('}')
+ }
+ out.toString()
+ }
+
+ private static void appendEntry(StringBuilder out, String key, BsonValue value, Scope scope) {
+ Scope child = scope.of(key)
+ out.append(child.keyText).append(': ')
+ appendValue(out, value, child)
+ }
+
+ private static void appendValue(StringBuilder out, BsonValue value, Scope scope) {
+ if (value.isDocument()) {
+ out.append('{')
+ boolean first = true
+ for (Map.Entry<String, BsonValue> entry in value.asDocument().entrySet()) {
+ if (!first) {
+ out.append(', ')
+ }
+ first = false
+ appendEntry(out, entry.key, entry.value, scope)
+ }
+ out.append('}')
+ }
+ else if (value.isArray()) {
+ out.append('[')
+ boolean first = true
+ Scope element = scope.element()
+ for (BsonValue item in value.asArray()) {
+ if (!first) {
+ out.append(', ')
+ }
+ first = false
+ appendValue(out, item, element)
+ }
+ out.append(']')
+ }
+ else {
+ out.append(PLACEHOLDER)
+ }
+ }
+
+ private static String quote(String key) {
+ StringBuilder out = new StringBuilder(key.length() + 2).append('"')
+ for (int i = 0; i < key.length(); i++) {
+ char c = key.charAt(i)
+ if (c == '"' || c == '\\') {
+ out.append('\\').append(c)
+ }
+ else if (c < 0x20) {
+ out.append(String.format('\\u%04x', (int) c))
+ }
+ else {
+ out.append(c)
+ }
+ }
+ out.append('"').toString()
+ }
+
+ /**
+ * Where a document sits in the command: among the options of the command, whose keys are kept
+ * and name what their documents hold, or in a document keyed by fields, where a key is kept when
+ * it is an operator or a field the mapping knows at its path.
+ */
+ @CompileStatic
+ private static final class Scope {
+
+ final String collection
+ final MongoMappedFields fields
+ /** The field path of the document; null in a document keyed by fields the mapping does not know. */
+ final List<String> path
+ final boolean optionKeyed
+ final String keyText
+ /** How many indices into the field at the path have been taken: 0 at the field, 1 in one of its elements. */
+ final int depth
+
+ private Scope(String collection, MongoMappedFields fields, List<String> path, boolean optionKeyed,
+ String keyText, int depth) {
+ this.collection = collection
+ this.fields = fields
+ this.path = path
+ this.optionKeyed = optionKeyed
+ this.keyText = keyText
+ this.depth = depth
+ }
+
+ static Scope options(String collection, MongoMappedFields fields) {
+ new Scope(collection, fields, Collections.<String>emptyList(), true, null, 0)
+ }
+
+ /**
+ * @return the scope of an element of an array in this scope: one index deeper into the field
+ * at the path, so that a numeric key inside the element is data unless the elements are arrays
+ */
+ Scope element() {
+ optionKeyed || path == null || path.isEmpty() ? this : new Scope(collection, fields, path, false, keyText, depth + 1)
+ }
+
+ private boolean indexed(List<String> at, int atDepth) {
+ at != null && !at.isEmpty() && fields != null && fields.isIndexed(collection, at, atDepth)
+ }
+
+ /**
+ * @return the scope of the value of a key in a document of this scope, with the key as it is recorded
+ */
+ Scope of(String key) {
+ if (optionKeyed) {
+ boolean fieldKeyed = FIELD_KEYED.contains(key)
+ return new Scope(collection, fields, Collections.<String>emptyList(), !fieldKeyed, quote(key), 0)
+ }
+ if (key.startsWith('$')) {
Review Comment:
Fixed in 73c20a13c7. The redactor now knows what kind of document a key sits in, and keeps a `$` key as an operator only where operators go: a condition (`filter`, `q`, `query`, `$pull`, `$currentDate`), a pipeline, and the top of an update document. Stored data is a separate kind: the documents of `insert`/`bulkWrite` (`documents`, `document`), a replacement update document, the values an update operator writes (`$set`, `$setOnInsert`, `$inc`, ...), what `$push`/`$addToSet` add (their `$each`, `$slice`, `$sort`, `$position` modifiers stay operators), and the operand of `$literal`/`$documents`. There a `$` key is a field name like any other and is recorded as `?` unless the mapping knows it, so `attrs: {"$example-secret": "v"}` records `"attrs": {?: ?}` in an insert, a `$set`, a replacement and `$setOnInsert`, and a top-level `$` field of an inserted document is `?` too; an update that is a pipeline keeps its stage operators. Covered in `MongoCommandRedactorSpec` (eight new rows o
ver those shapes, plus `$` secrets in the never-survives command) and against a real mongod in `QueryTailCommandListenerSpec`, which sends the insert and a `$set` update with `runCommand`, checks that the database stored `$example-secret`, that the capture kept `$eq` and `$set`, and that the key is not in the file. The guide says where a `$` key counts as an operator.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@grails.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add the query-tail command to find query-heavy pages [grails-core]
Posted by "jamesfredley (via GitHub)" <gi...@apache.org>.
jamesfredley commented on code in PR #16631:
URL: https://github.com/apache/grails-core/pull/16631#discussion_r4372169633
##########
grails-datamapping-core/src/cli/groovy/org/apache/grails/data/core/cli/querytail/StatementCollapser.groovy:
##########
@@ -0,0 +1,131 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * https://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.grails.data.core.cli.querytail
+
+import java.util.regex.Pattern
+
+import groovy.transform.CompileStatic
+
+/**
+ * Shortens a statement to the part that identifies it: the table a select reads and its where
+ * clause, the table an update changes and its where clause, the table an insert writes to. The
+ * column lists Hibernate generates are long and say nothing about which code ran the statement,
+ * and dropping them lets the same query count as one statement wherever it is run from. A Cypher
+ * statement is shown whole: its pattern names what it reads or writes, and GORM generates nothing
+ * long in it. So is a MongoDB command, which is recorded as its name, its collection and the
+ * document that shapes it.
+ *
+ * @since 8.1
+ */
+@CompileStatic
+class StatementCollapser {
+
+ private static final Pattern COMMENT = Pattern.compile('/\\*.*?\\*/', Pattern.DOTALL)
+ private static final Pattern WHITESPACE = Pattern.compile('\\s+')
+ private static final String ELLIPSIS = '...'
+
+ /**
+ * @param sql a statement
+ * @return the shortened statement, or the statement with its comments and extra whitespace
+ * removed when it is not a SQL select, update or insert
+ */
+ static String collapse(String sql) {
+ String text = normalize(sql)
+ if (text.isEmpty() || StatementType.isCypher(text)) {
+ return text
+ }
+ switch (StatementType.of(text)) {
+ case StatementType.SELECT:
+ return collapseSelect(text)
+ case StatementType.UPDATE:
+ return collapseUpdate(text)
+ case StatementType.INSERT:
+ return collapseInsert(text)
+ default:
+ return text
+ }
+ }
+
+ /**
+ * @param sql a statement
+ * @return the statement without comments, its whitespace collapsed to single spaces
+ */
+ static String normalize(String sql) {
+ if (sql == null) {
+ return ''
+ }
+ WHITESPACE.matcher(COMMENT.matcher(sql).replaceAll(' ')).replaceAll(' ').trim()
+ }
+
+ private static String collapseSelect(String text) {
+ int from = topLevelIndexOf(text, ' from ')
+ from < 0 ? text : "select ${ELLIPSIS}${text.substring(from)}".toString()
+ }
+
+ private static String collapseUpdate(String text) {
+ int set = topLevelIndexOf(text, ' set ')
+ if (set < 0) {
+ return text
+ }
+ int where = topLevelIndexOf(text, ' where ')
+ String head = text.substring(0, set)
+ where > set ? "${head} ${ELLIPSIS}${text.substring(where)}".toString() : "${head} ${ELLIPSIS}".toString()
+ }
+
+ private static String collapseInsert(String text) {
+ int columns = topLevelIndexOf(text, '(')
+ int values = topLevelIndexOf(text, ' values ')
+ int cut = columns < 0 ? values : (values < 0 ? columns : Math.min(columns, values))
+ cut < 0 ? text : "${text.substring(0, cut).trim()} ${ELLIPSIS}".toString()
+ }
+
+ /**
+ * Finds a keyword outside parentheses and string literals, so that a subquery or a value
+ * never stands in for the outer statement's clause.
+ */
+ static int topLevelIndexOf(String text, String needle) {
+ String haystack = text.toLowerCase(Locale.ROOT)
+ if (haystack.length() != text.length()) {
+ haystack = text
+ }
+ int depth = 0
+ boolean quoted = false
+ for (int i = 0; i < haystack.length(); i++) {
+ char c = haystack.charAt(i)
+ if (c == ('\'' as char)) {
Review Comment:
`update "order set lines" set title=? where id=?` and `update "order set headers" set title=? where id=?` collapse into one statement. The scan skips single-quoted strings and still matches ` set ` inside a quoted identifier.
Skip quoted identifiers, including a doubled quote, before matching a clause.
##########
grails-data-hibernate7/core/src/main/groovy/org/apache/grails/data/hibernate7/core/querytail/QueryTailStatementInspector.groovy:
##########
@@ -0,0 +1,319 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * https://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.grails.data.hibernate7.core.querytail
+
+import groovy.transform.CompileStatic
+import groovy.transform.PackageScope
+import org.hibernate.SessionFactory
+import org.hibernate.SessionFactoryObserver
+import org.hibernate.cfg.AvailableSettings
+import org.hibernate.cfg.Configuration
+import org.hibernate.dialect.AbstractTransactSQLDialect
+import org.hibernate.dialect.CockroachDialect
+import org.hibernate.dialect.DB2Dialect
+import org.hibernate.dialect.Dialect
+import org.hibernate.dialect.H2Dialect
+import org.hibernate.dialect.HANADialect
+import org.hibernate.dialect.HSQLDialect
+import org.hibernate.dialect.MySQLDialect
+import org.hibernate.dialect.OracleDialect
+import org.hibernate.dialect.PostgreSQLDialect
+import org.hibernate.dialect.SpannerDialect
+import org.hibernate.dialect.SpannerPostgreSQLDialect
+import org.hibernate.engine.spi.SessionFactoryImplementor
+import org.hibernate.resource.jdbc.spi.StatementInspector
+import org.springframework.beans.BeanUtils
+import org.springframework.util.ClassUtils
+
+import org.apache.grails.data.core.querytail.QueryTailCapture
+import org.apache.grails.data.core.querytail.QueryTailSettings
+import org.apache.grails.data.core.querytail.QueryTailSyntax
+import org.grails.datastore.mapping.core.exceptions.ConfigurationException
+
+/**
+ * The statement inspector GORM installs on a session factory when query-tail capture is enabled. It
+ * records every statement Hibernate is about to run to the {@link QueryTailCapture} of the
+ * application, without its values, and returns the statement as it found it: the database runs
+ * it unchanged. An inspector the application configured itself through
+ * {@code hibernate.session_factory.statement_inspector} keeps running: it sees the statement
+ * first, and the statement it returns is the one run and recorded.
+ *
+ * <p>The session factory owns the capture. The inspector observes it: it opens the capture when
+ * the session factory is created, takes from its dialect how the database quotes values, and
+ * closes the capture when the session factory closes, so a session factory that fails to start
+ * leaves no capture open either. An observer the application set on the configuration keeps
+ * being called.</p>
+ *
+ * <p>Nothing of this is installed unless {@link QueryTailSettings#SETTING_ENABLED} is set, so an
+ * application that does not capture pays nothing per statement.</p>
+ *
+ * @since 8.1
+ */
+@CompileStatic
+class QueryTailStatementInspector implements StatementInspector, SessionFactoryObserver {
+
+ @Serial
+ private static final long serialVersionUID = 1L
+
+ /** The community dialects, which are not a dependency, that the inspector tells apart by name. */
+ private static final String COMMUNITY_INFORMIX = 'org.hibernate.community.dialect.InformixDialect'
+ private static final String COMMUNITY_SQL_ANYWHERE = 'org.hibernate.community.dialect.SybaseAnywhereDialect'
+
+ /** The connection source name statements are recorded under. */
+ final String dataSourceName
+
+ /** The file statements are recorded to. */
+ final File file
+
+ /** The inspector the application configured, or null. */
+ final StatementInspector delegate
+
+ /** The session factory observer the application configured, or null. */
+ final SessionFactoryObserver observer
+
+ private transient volatile QueryTailCapture capture
+ private transient boolean closed
+
+ /**
+ * @param dataSourceName the connection source the inspected session factory belongs to
+ * @param file the capture file statements are recorded to
+ * @param delegate the inspector the application configured, or null
+ * @param observer the session factory observer the application configured, or null
+ */
+ QueryTailStatementInspector(String dataSourceName, File file, StatementInspector delegate,
+ SessionFactoryObserver observer) {
+ this.dataSourceName = dataSourceName
+ this.file = file
+ this.delegate = delegate
+ this.observer = observer
+ }
+
+ /**
+ * Installs the inspector on the configuration of a session factory when capture is enabled, as
+ * its statement inspector and as the observer of the session factory it builds.
+ *
+ * @param dataSourceName the connection source the session factory is built for
+ * @param settings the query-tail settings of the connection source
+ * @param configuration the configuration the session factory is built from
+ * @return the installed inspector, or null when capture is not enabled
+ * @throws ConfigurationException when the inspector the application configured cannot be created
+ */
+ static QueryTailStatementInspector install(String dataSourceName, QueryTailSettings settings,
+ Configuration configuration) {
+ if (settings == null || !settings.enabled) {
+ return null
+ }
+ Properties properties = configuration.properties
+ StatementInspector configured = resolve(properties.get(AvailableSettings.STATEMENT_INSPECTOR))
Review Comment:
If `hibernate.configClass` calls `setStatementInspector`, Hibernate keeps that inspector apart from the property this code sets. The capture file opens and no statement is written.
Wrap the inspector already set, and install the wrapper with `setStatementInspector`. The Hibernate 5 installer has the same gap.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@grails.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add the query-tail command to find query-heavy pages [grails-core]
Posted by "jamesfredley (via GitHub)" <gi...@apache.org>.
jamesfredley commented on code in PR #16631:
URL: https://github.com/apache/grails-core/pull/16631#discussion_r4383278981
##########
grails-datamapping-core/src/main/groovy/org/apache/grails/data/core/querytail/QueryTailRequestAttribution.groovy:
##########
@@ -0,0 +1,729 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * https://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.grails.data.core.querytail
+
+import java.lang.reflect.Method
+import java.nio.charset.StandardCharsets
+import java.util.concurrent.atomic.AtomicLong
+import java.util.regex.Pattern
+
+import groovy.transform.CompileStatic
+import groovy.transform.PackageScope
+
+import jakarta.servlet.http.HttpServletRequest
+
+import org.springframework.util.ClassUtils
+import org.springframework.web.context.request.RequestAttributes
+import org.springframework.web.context.request.RequestContextHolder
+import org.springframework.web.context.request.ServletRequestAttributes
+
+/**
+ * Attributes a captured statement to the web request on the current thread. Grails keeps the
+ * request, and the controller and action it resolved, on the thread that serves it, so a statement
+ * run by an interceptor, the action or the view all land on the same page. The first statement of
+ * a request gives it an id that every later statement of that request repeats, which is how the
+ * {@code query-tail} command counts statements per request. Each capture file keeps its own request
+ * ids, so a request recorded to another file cannot consume or reuse one of its ids. Work on any other
+ * thread is attributed to the thread by name.
+ *
+ * <p>The request is recorded as its method and path, with the values in the path replaced by
+ * {@code ?}. Once Grails has chosen the controller, the path is read against the URL mapping it
+ * matched: a segment the mapping spells out is kept, as is a variable segment that names the
+ * controller, action or namespace, and every other variable segment is replaced. A request a Spring
+ * MVC handler serves is recorded as the pattern its handler mapping matched, whose variables are
+ * replaced. When the path cannot be read against the mapping, the values of the parameters the URL
+ * mapping took from it and those of its query are replaced, found
+ * whether or not the client percent-encoded them, and any segment that is a number, a UUID or a long
+ * hexadecimal token, so that an id, a token or a key carried in a path never reaches the capture
+ * file. A segment a value is glued into, as a mapping such as {@code /reset/${token}_confirm}
+ * produces, is replaced whole, whether the value shows as sent or decoded. That also holds for a value
+ * spanning several segments, with literal text before its first segment or after its last, and for every path
+ * parameter after a semicolon, such as a {@code jsessionid} the container rewrote the URL with.
+ * Until the URL mapping has run, as for a statement of a
+ * filter that runs before the controller is chosen, or for a request that never reaches one, no
+ * segment's role is known and every segment is replaced; only the controller, or the pattern, the
+ * framework set on the request says that it has run, never a parameter, which a client can send.
+ * A request is described once and its line reused for its later statements, until its path or the
+ * handler it reached changes, as a forward changes them.</p>
+ *
+ * <p>Work a request starts on another thread that carries the request with it, as Grails promises
+ * and {@code @Async} methods do, counts for the request. A statement that runs once the request
+ * has completed is attributed to its thread.</p>
+ *
+ * <p>Spring Web is optional: without it on the classpath every statement is attributed to its
+ * thread.</p>
+ *
+ * @since 9.0
+ */
+@CompileStatic
+class QueryTailRequestAttribution {
+
+ /** The prefix of the request attribute holding the id each attribution gave a request. */
+ public static final String REQUEST_ID_ATTRIBUTE = 'org.apache.grails.data.core.querytail.REQUEST_ID'
+
+ /** The request attribute Grails sets the controller name in. */
+ public static final String CONTROLLER_NAME_ATTRIBUTE = 'org.grails.CONTROLLER_NAME_ATTRIBUTE'
+
+ /** The request attribute Grails sets the controller namespace in. */
+ public static final String CONTROLLER_NAMESPACE_ATTRIBUTE = 'org.grails.CONTROLLER_NAMESPACE_ATTRIBUTE'
+
+ /** The request attribute Grails sets the action name in. */
+ public static final String ACTION_NAME_ATTRIBUTE = 'org.grails.ACTION_NAME_ATTRIBUTE'
+
+ private static final boolean SPRING_WEB_PRESENT = ClassUtils.isPresent(
+ 'org.springframework.web.context.request.RequestContextHolder', QueryTailRequestAttribution.classLoader)
+
+ private final AtomicLong requestCounter
+ private final String requestIdPrefix
+ private final String requestIdAttribute = REQUEST_ID_ATTRIBUTE + '.' + UUID.randomUUID()
+
+ /**
+ * An attribution whose request ids are {@code r1}, {@code r2} and so on.
+ */
+ QueryTailRequestAttribution() {
+ this('')
+ }
+
+ /**
+ * @param requestIdPrefix what every request id starts with, which tells the requests of one JVM
+ * from those of another writing the same capture
+ */
+ QueryTailRequestAttribution(String requestIdPrefix) {
+ this(requestIdPrefix, new AtomicLong())
+ }
+
+ /**
+ * @param requestIdPrefix what every request id starts with
+ * @param requestCounter the counter the requests are numbered by, which attributions that give
+ * their ids the same prefix share, so that none of them repeats an id another gave
+ */
+ @PackageScope
+ QueryTailRequestAttribution(String requestIdPrefix, AtomicLong requestCounter) {
+ this.requestIdPrefix = requestIdPrefix ?: ''
+ this.requestCounter = requestCounter
+ }
+
+ /**
+ * Fills in the request id, page and request of a record from the current thread.
+ *
+ * @param record the record of a statement that just ran on this thread
+ * @return the same record
+ */
+ QueryTailRecord attribute(QueryTailRecord record) {
+ if (SPRING_WEB_PRESENT) {
+ try {
+ if (WebRequestLookup.attribute(record, requestCounter, requestIdPrefix, requestIdAttribute)) {
+ return record
+ }
+ }
+ catch (IllegalStateException ignored) {
+ // the request has completed, as a task it started can outlive it, and is no longer to be read
+ }
+ }
+ record.requestId = null
+ record.request = null
+ record.page = QueryTailRecord.THREAD_PAGE_PREFIX + Thread.currentThread().name
+ record
+ }
+
+ /**
+ * Keeps every reference to Spring Web, in a signature as well as in a body, in a class that is only
+ * loaded once Spring Web is known to be present. The outer class can then be loaded, and reflected
+ * on, without Spring Web on the classpath.
+ */
+ @CompileStatic
+ private static class WebRequestLookup {
+
+ private static final String REQUEST_ID_PREFIX = 'r'
+
+ private static final String PLACEHOLDER = '?'
+
+ /** The parameters a URL mapping sets that name code rather than carry a value. */
+ private static final Set<String> MAPPING_PARAMETERS = ['controller', 'action', 'namespace', 'plugin', 'format'] as Set<String>
+
+ /** A segment that is a number, a UUID or a long hexadecimal token is a value whatever the mapping says. */
+ private static final Pattern VALUE_SHAPED = Pattern.compile(
+ '(?i)[0-9]+|[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}|[0-9a-f]{32,}')
+
+ private static final boolean SERVLET_PRESENT = ClassUtils.isPresent(
+ 'jakarta.servlet.http.HttpServletRequest', WebRequestLookup.classLoader)
+
+ /** The request attribute Grails keeps the URL mapping it matched a request with in. */
+ private static final String MATCHED_MAPPING_ATTRIBUTE = 'org.grails.url.match.info'
+
+ /** The request attribute a Spring MVC handler mapping keeps the pattern it matched a request with in. */
+ private static final String MATCHED_PATTERN_ATTRIBUTE = 'org.springframework.web.servlet.HandlerMapping.bestMatchingPattern'
+
+ /** The request attribute the line a request was described with is kept in, with what it was described from. */
+ private static final String REQUEST_LINE_ATTRIBUTE = 'org.apache.grails.data.core.querytail.REQUEST_LINE'
+
+ /** The {@code getParams()} of a Grails web request, which holds the parameters its URL mapping took, by class. */
+ private static final ClassValue<Method> PARAMETERS = publicMethod('getParams')
+
+ /** The {@code getUrlData()} of the URL mapping Grails matched a request with, by class. */
+ private static final ClassValue<Method> URL_DATA = publicMethod('getUrlData')
+
+ /** The {@code getTokens()} of the data of a URL mapping, by class. */
+ private static final ClassValue<Method> TOKENS = publicMethod('getTokens')
+
+ private static ClassValue<Method> publicMethod(String name) {
+ new ClassValue<Method>() {
+ @Override
+ protected Method computeValue(Class<?> type) {
+ try {
+ return type.getMethod(name)
+ }
+ catch (NoSuchMethodException ignored) {
+ return null
+ }
+ }
+ }
+ }
+
+ /**
+ * Fills in a record from the request on the current thread.
+ *
+ * @param record the record to fill in
+ * @param requestCounter the counter the next request id is taken from
+ * @param requestIdPrefix what every request id starts with
+ * @param requestIdAttribute the request attribute this capture keeps its ids in
+ * @return whether there is a request on the current thread
+ */
+ static boolean attribute(QueryTailRecord record, AtomicLong requestCounter, String requestIdPrefix, String requestIdAttribute) {
+ RequestAttributes attributes = RequestContextHolder.getRequestAttributes()
+ if (attributes == null) {
+ return false
+ }
+ record.requestId = requestId(attributes, requestCounter, requestIdPrefix, requestIdAttribute)
+ record.page = page(attributes)
+ record.request = SERVLET_PRESENT ? ServletRequestLookup.describe(attributes) : null
+ true
+ }
+
+ /**
+ * Gives the request line of a request: the one described for an earlier statement of the request,
+ * when the request has not moved on since, else one described now.
+ *
+ * @param method the HTTP method
+ * @param path the request path, as the client sent it, with the context path
+ * @param contextPath the context path of the application, which the path starts with
+ * @param queryString the query string of the request, or null
+ * @param attributes the request
+ */
+ static String describe(String method, String path, String contextPath, String queryString, RequestAttributes attributes) {
+ Object controller = attributes.getAttribute(CONTROLLER_NAME_ATTRIBUTE, RequestAttributes.SCOPE_REQUEST)
+ Object action = attributes.getAttribute(ACTION_NAME_ATTRIBUTE, RequestAttributes.SCOPE_REQUEST)
+ Object namespace = attributes.getAttribute(CONTROLLER_NAMESPACE_ATTRIBUTE, RequestAttributes.SCOPE_REQUEST)
+ Object mapping = attributes.getAttribute(MATCHED_MAPPING_ATTRIBUTE, RequestAttributes.SCOPE_REQUEST)
+ Object pattern = attributes.getAttribute(MATCHED_PATTERN_ATTRIBUTE, RequestAttributes.SCOPE_REQUEST)
+ RequestLine described = new RequestLine(method, path, queryString, controller, action, namespace, mapping, pattern)
+ Object earlier = attributes.getAttribute(REQUEST_LINE_ATTRIBUTE, RequestAttributes.SCOPE_REQUEST)
+ if (earlier instanceof RequestLine && ((RequestLine) earlier).sameAs(described)) {
+ return ((RequestLine) earlier).line
+ }
+ described.line = "${method} ${describePath(path, contextPath, queryString, attributes, described)}".toString()
+ attributes.setAttribute(REQUEST_LINE_ATTRIBUTE, described, RequestAttributes.SCOPE_REQUEST)
+ described.line
+ }
+
+ /**
+ * @return the path of a request with its values replaced: read against the mapping that matched it,
+ * or against the pattern a Spring MVC handler mapping matched, or by the values its parameters hold
+ */
+ private static String describePath(String path, String contextPath, String queryString, RequestAttributes attributes,
+ RequestLine request) {
+ // only the framework sets the controller attribute; a client can send a controller parameter
+ if (request.controller != null) {
+ Set<String> names = codeNames(request)
+ String mapped = readAgainstMapping(path, contextPath, request.mapping, names)
+ if (mapped != null) {
+ return mapped
+ }
+ Set<String> values = new HashSet<>()
+ addValues(values, mappedParameters(attributes))
+ addQueryValues(values, queryString)
+ return redactPath(path, values, names)
+ }
+ if (request.pattern != null) {
+ return contextPath + patternPath(request.pattern.toString())
+ }
+ withholdPath(path)
+ }
+
+ /** @return the names of the controller, action and namespace a request reached, which a path may spell out */
+ private static Set<String> codeNames(RequestLine request) {
+ Set<String> names = new HashSet<>()
+ for (Object name in [request.controller, request.action, request.namespace]) {
+ if (name != null && !name.toString().isEmpty()) {
+ names.add(name.toString())
+ }
+ }
+ names
+ }
+
+ /**
+ * Reads a path against the tokens of the URL mapping that matched it: a token the mapping spells
+ * out is kept where the path spells it out too, a variable segment that names the controller,
+ * action or namespace is kept, and every other variable is replaced, as is every segment a greedy
+ * token takes. The path parameters of a segment are replaced each.
+ *
+ * @return the path read so, or null when the mapping does not say how, or the path does not
+ * follow its tokens
+ */
+ private static String readAgainstMapping(String path, String contextPath, Object mapping, Set<String> names) {
+ String[] tokens = mappingTokens(mapping)
+ if (tokens == null) {
+ return null
+ }
+ String prefix = contextPath && path.startsWith(contextPath) ? contextPath : ''
+ String within = path.substring(prefix.length())
+ if (!within.startsWith('/')) {
+ return null
+ }
+ List<String> segments = new ArrayList<>(Arrays.asList(within.substring(1).split('/', -1)))
+ boolean trailingSlash = !segments.isEmpty() && segments.last().isEmpty()
+ if (trailingSlash) {
+ segments.removeLast()
+ }
+ List<String> spelled = tokens.findAll { String token -> !token.isEmpty() }
+ int greedy = spelled.findIndexOf { String token -> token.contains('**') }
+ List<String> out = []
+ int at = 0
+ for (int i = 0; i < spelled.size(); i++) {
+ String token = spelled[i]
+ boolean optional = token.endsWith('?')
+ String bare = optional ? token.substring(0, token.length() - 1) : token
+ if (i == greedy) {
+ int taken = segments.size() - (spelled.size() - 1)
+ if (taken < 0) {
+ return null
+ }
+ for (int k = 0; k < taken; k++) {
+ out.add(PLACEHOLDER + parametersOf(segments[at++]))
+ }
+ continue
+ }
+ if (at >= segments.size()) {
+ if (optional) {
+ continue
+ }
+ return null
+ }
+ String segment = segments[at]
+ int semicolon = segment.indexOf(';')
+ String bareSegment = semicolon < 0 ? segment : segment.substring(0, semicolon)
+ String decoded = decode(bareSegment)
+ boolean last = at == segments.size() - 1
+ if (!bare.contains('(') && !bare.contains('*')) {
+ if (decoded != bare && !(last && withoutExtension(decoded) == bare)) {
+ return null
+ }
+ out.add(bareSegment + parametersOf(segment))
+ }
+ else if (bare == '(*)' && (names.contains(decoded) || (last && names.contains(withoutExtension(decoded))))) {
+ out.add(bareSegment + parametersOf(segment))
+ }
+ else {
+ out.add(PLACEHOLDER + parametersOf(segment))
+ }
+ at++
+ }
+ if (at != segments.size()) {
+ return null
+ }
+ prefix + '/' + out.join('/') + (trailingSlash && !out.isEmpty() ? '/' : '')
+ }
+
+ private static String withoutExtension(String segment) {
+ int dot = segment.lastIndexOf('.')
+ dot > 0 ? segment.substring(0, dot) : segment
+ }
+
+ private static String parametersOf(String segment) {
+ int semicolon = segment.indexOf(';')
+ semicolon < 0 ? '' : withheldPathParameters(segment.substring(semicolon))
+ }
+
+ /**
+ * @return the tokens of the URL mapping Grails matched a request with, which spell out its
+ * literal segments and stand in for its variables, or null when there is none to read
+ */
+ private static String[] mappingTokens(Object mapping) {
+ if (mapping == null) {
+ return null
+ }
+ Object data = invoke(URL_DATA, mapping)
+ Object tokens = data == null ? null : invoke(TOKENS, data)
+ tokens instanceof String[] ? (String[]) tokens : null
+ }
+
+ private static Object invoke(ClassValue<Method> methods, Object target) {
+ Method method = methods.get(target.getClass())
+ if (method == null) {
+ return null
+ }
+ try {
+ return method.invoke(target)
+ }
+ catch (ReflectiveOperationException | RuntimeException ignored) {
+ return null
+ }
+ }
+
+ /**
+ * @return the pattern a Spring MVC handler mapping matched, with every segment that holds a
+ * variable or a wildcard replaced
+ */
+ private static String patternPath(String pattern) {
+ String[] segments = pattern.split('/', -1)
+ for (int i = 0; i < segments.length; i++) {
+ String segment = segments[i]
+ if (segment.contains('{') || segment.contains('*') || segment.contains('?')) {
+ segments[i] = PLACEHOLDER
+ }
+ }
+ segments.join('/')
+ }
+
+ /**
+ * Adds the values of a query string, decoded as the servlet container decodes them. The query
+ * string is read rather than the parameters of the request, as reading those makes the container
+ * parse a form posted in the body and use it up before the application reads it.
+ */
+ private static void addQueryValues(Set<String> values, String queryString) {
+ if (!queryString) {
+ return
+ }
+ for (String pair : queryString.split('&')) {
+ int equals = pair.indexOf('=')
+ if (equals >= 0 && !MAPPING_PARAMETERS.contains(decodeQuery(pair.substring(0, equals)))) {
+ values.add(decodeQuery(pair.substring(equals + 1)))
+ }
+ }
+ }
+
+ private static String decodeQuery(String text) {
+ try {
+ return URLDecoder.decode(text, StandardCharsets.UTF_8)
+ }
+ catch (IllegalArgumentException ignored) {
+ return text
+ }
+ }
+
+ /**
+ * Replaces every segment of a path: before the URL mapping has run, nothing tells a literal
+ * segment from a token, so only the shape of the path is recorded.
+ */
+ private static String withholdPath(String path) {
+ String[] segments = path.split('/', -1)
+ StringBuilder out = new StringBuilder(path.length())
+ for (int i = 0; i < segments.length; i++) {
+ if (i > 0) {
+ out.append('/')
+ }
+ out.append(segments[i].isEmpty() ? segments[i] : PLACEHOLDER)
+ }
+ out.toString()
+ }
+
+ private static String requestId(RequestAttributes attributes, AtomicLong requestCounter, String requestIdPrefix, String requestIdAttribute) {
+ // the request itself, which a task the request starts sees through attributes of its own
+ synchronized (SERVLET_PRESENT ? ServletRequestLookup.mutex(attributes) : attributes) {
+ Object existing = attributes.getAttribute(requestIdAttribute, RequestAttributes.SCOPE_REQUEST)
+ if (existing != null) {
+ return existing.toString()
+ }
+ String id = requestIdPrefix + REQUEST_ID_PREFIX + requestCounter.incrementAndGet()
+ attributes.setAttribute(requestIdAttribute, id, RequestAttributes.SCOPE_REQUEST)
+ return id
+ }
+ }
+
+ private static String page(RequestAttributes attributes) {
Review Comment:
For `/book`, the action is still null when a `before` interceptor runs. Those queries are recorded under `book`, and the action and view queries of the same request under `book/index`. `--page=book/index` then misses the interceptor queries.
Resolve the default action from the matched controller when the action attribute is absent. Please assert one page key for that request.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@grails.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add the query-tail command to find query-heavy pages [grails-core]
Posted by "jdaugherty (via GitHub)" <gi...@apache.org>.
jdaugherty commented on PR #16631:
URL: https://github.com/apache/grails-core/pull/16631#issuecomment-6428473079
@jamesfredley I've pushed another round of hardening on top of the last one. It's still a single commit on 9.0.x. Here's what changed:
**Capture never fails the statement**
- Recording is now guarded end to end. If the request can't be read or the values can't be redacted, that statement is left out: the capture warns once and logs at debug after that. The query itself runs as it would without the capture.
- MongoDB redaction now runs inside that guard, and it's skipped once the capture has stopped.
- If the file system can't lock the lock file, the capture falls back to writing uncoordinated instead of failing startup. Some network and shared folders behave this way, and so does a FIFO on macOS.
- A statement that runs after its request has completed is attributed to its thread.
**Request attribution**
- The request line is worked out once per request and cached on the request. It's only recomputed when the request moves on, as a forward moves it. Before this, it was rebuilt for every statement.
- The path is now read against the URL mapping Grails matched, so `"/book/show/$id"` records `/book/show/?` exactly. The value-matching heuristics are only a fallback for when the path doesn't follow the mapping. A segment that names the controller, action or namespace is always kept.
- Requests served by Spring MVC handlers are recorded as the pattern their handler mapping matched.
- Request ids are assigned under the request's mutex. Async task threads carry their own request attributes, so they now number a request once and count toward it.
**Settings and stores**
- `QueryTailSettings` now lives on `ConnectionSourceSettings` in `grails-datastore-core`. It's bound from `grails.gorm.queryTail` like any other setting, with the usual fallback, which replaces the per-store wiring in H5, H7, MongoDB and Neo4j.
- MongoDB clients that Spring Boot settings build are now captured. GORM adds its listener through a new client-factory constructor. Clients the application builds itself can't be captured, and GORM now warns when capture is enabled for one.
**Test runs**
- The Grails Gradle plugin names one run per build and passes it to test JVMs as `grails.gorm.queryTail.run`. Datastores that specs open and close one after another, and a project's `test` and `integrationTest` tasks, now append to one run instead of truncating each other. The next build starts its own.
**Command**
- `--page=book/show` also matches namespaced pages such as `admin:book/show`.
- An invalid `--top` value is reported with a clear message instead of a `NumberFormatException`.
- `EXPLAIN`/`PROFILE` of a Cypher statement is classified by the statement it plans. The plan of a SQL statement is no longer mistaken for Cypher.
I also changed the new API from `@since 8.1` to `@since 9.0`, and updated the guide and the command reference to match all of the above.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@grails.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add the query-tail command to find query-heavy pages [grails-core]
Posted by "jamesfredley (via GitHub)" <gi...@apache.org>.
jamesfredley commented on code in PR #16631:
URL: https://github.com/apache/grails-core/pull/16631#discussion_r4365443650
##########
grails-datamapping-core/src/main/groovy/org/apache/grails/data/core/querytail/H2SqlSyntax.groovy:
##########
@@ -0,0 +1,70 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * https://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.grails.data.core.querytail
+
+import groovy.transform.CompileStatic
+import groovy.transform.PackageScope
+
+/**
+ * H2: block comments nest, {@code //} starts a line comment, {@code $$} quotes a string without a
+ * tag, backticks delimit a name in every compatibility mode, and square brackets delimit a name in
+ * the SQL Server mode, where they are never a subscript, which is what a bracket after a name, a
+ * closing parenthesis or a closing bracket is in the other modes.
+ */
+@CompileStatic
+@PackageScope
+final class H2SqlSyntax extends StandardSqlSyntax {
+
+ H2SqlSyntax() {
+ super('H2')
+ }
+
+ @Override
+ int endOfComment(String sql, int index) {
+ sql.startsWith('//', index) ? endOfLine(sql, index) : super.endOfComment(sql, index)
+ }
+
+ @Override
+ int endOfValue(String sql, int index) {
+ int end = endOfDollarQuoted(sql, index, false)
+ end != NONE ? end : super.endOfValue(sql, index)
+ }
+
+ @Override
+ int endOfQuotedName(String sql, int index) {
+ char c = sql.charAt(index)
+ if (c == '`') {
+ return endOfQuoted(sql, index, '`' as char, false, true)
+ }
+ if (c == '[' && !subscript(sql, index)) {
Review Comment:
A space before `[` is still treated as a quoted identifier. `select ARRAY ['example-secret']` is valid H2, and the redactor copies that span unchanged. A comment between `ARRAY` and `[` does the same.
Do not treat brackets after whitespace as an identifier when they open an array, or withhold the statement. The existing array regression covers `array[7,8]` only, with no space. Please assert this literal does not reach the file.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@grails.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add the sql-tail command to find query-heavy pages [grails-core]
Posted by "jamesfredley (via GitHub)" <gi...@apache.org>.
jamesfredley commented on code in PR #16631:
URL: https://github.com/apache/grails-core/pull/16631#discussion_r4360807147
##########
grails-datamapping-core/src/main/groovy/org/grails/datastore/gorm/jdbc/sqltail/SqlTailRedactor.java:
##########
@@ -0,0 +1,452 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * https://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.grails.datastore.gorm.jdbc.sqltail;
+
+import java.util.ArrayList;
+import java.util.List;
+
+/**
+ * Takes the values out of a statement before it is captured. Every string, number and
+ * dollar-quoted literal becomes {@link #PLACEHOLDER} and every comment is dropped, so that a value
+ * written into the SQL text, a password in a native query or a key in a comment, never reaches the
+ * capture file. Only the recorded copy is redacted: the statement the database runs is not changed.
+ *
+ * <p>Databases disagree on parts of the lexical syntax: whether a backslash escapes a quote, whether
+ * comments nest, whether double quotes delimit an identifier or a string, whether a quote written
+ * twice escapes it or ends one string and starts another, whether a quote written three times
+ * opens a string that quotes can appear in, whether {@code #} starts a comment, and whether
+ * {@code #} and {@code @} are part of a name or operators. A statement is redacted under every
+ * reading its {@link Syntax} allows, and is recorded only when all of them give the same text. When
+ * they do not, or when a literal or a comment never ends, the statement is withheld:
+ * {@link #WITHHELD} is recorded in its place, never the statement.</p>
+ *
+ * @since 8.1
+ */
+public final class SqlTailRedactor {
+
+ /** What a literal is replaced with. */
+ public static final String PLACEHOLDER = "?";
+
+ /** What is recorded in place of a statement whose values cannot be told apart from its SQL. */
+ public static final String WITHHELD = "(statement withheld: its values could not be redacted)";
+
+ private static final boolean[] NO = {false};
+ private static final boolean[] YES = {true};
+ private static final boolean[] EITHER = {false, true};
+
+ /**
+ * The lexical syntax of the database a statement runs on, as far as finding its values goes.
+ */
+ public enum Syntax {
+
+ /**
+ * Not known, or a database that can read double-quoted text as a string although it quotes
+ * identifiers with double quotes, such as SQLite and Informix: a statement is recorded only when
+ * the readings of every syntax below agree.
+ */
+ UNKNOWN,
+
+ /**
+ * Standard SQL, where double quotes delimit identifiers: H2, PostgreSQL, Oracle, DB2, HSQLDB and HANA.
+ * They differ on {@code #} and {@code @}, which are operators in PostgreSQL and part of a name in Oracle
+ * and DB2, so a statement that has one next to a number is withheld.
+ */
+ STANDARD,
+
+ /** MySQL and MariaDB: double quotes delimit strings, and {@code #} and {@code -- } start comments. */
+ MYSQL,
+
+ /**
+ * Microsoft SQL Server and Sybase: square brackets delimit identifiers, comments nest, a dollar before
+ * digits is an amount, and double quotes delimit a string when {@code QUOTED_IDENTIFIER} is off, as it is
+ * by default on Sybase.
+ */
+ TRANSACT_SQL,
+
+ /**
+ * GoogleSQL, the dialect of Cloud Spanner: backticks delimit identifiers, single and double quotes
+ * delimit strings, a string that opens with its quote written three times ends only at the quote
+ * written three times again and holds any other quote, a backslash always escapes the character
+ * after it, a quote written twice is two strings, and {@code #} starts a comment.
+ */
+ GOOGLE_SQL
+ }
+
+ private SqlTailRedactor() {
+ }
+
+ /**
+ * Redacts a statement.
+ *
+ * @param sql the statement
+ * @param syntax the syntax of the database it runs on
+ * @return the statement with its literals replaced and its comments dropped, or {@link #WITHHELD}
+ */
+ public static String redact(String sql, Syntax syntax) {
+ if (sql == null) {
+ return null;
+ }
+ String redacted = null;
+ for (Reading reading : readings(sql, syntax == null ? Syntax.UNKNOWN : syntax)) {
+ String candidate = reading.redact(sql);
+ if (candidate == null || (redacted != null && !redacted.equals(candidate))) {
+ return WITHHELD;
+ }
+ redacted = candidate;
+ }
+ return redacted;
+ }
+
+ /**
+ * The readings a statement has to agree under. A reading that depends on a character the
+ * statement does not contain is the same as the one that does not, so it is left out.
+ */
+ private static List<Reading> readings(String sql, Syntax syntax) {
+ boolean[] backslash = sql.indexOf('\\') >= 0 ? EITHER : NO;
+ int comment = sql.indexOf("/*");
+ boolean[] nesting = comment >= 0 && sql.indexOf("/*", comment + 2) >= 0 ? EITHER : NO;
+ boolean[] symbols = sql.indexOf('#') >= 0 || sql.indexOf('@') >= 0 ? EITHER : YES;
+ List<Reading> readings = new ArrayList<>();
+ if (syntax == Syntax.STANDARD || syntax == Syntax.UNKNOWN) {
+ for (boolean backslashEscapes : backslash) {
+ for (boolean nestedComments : nesting) {
+ for (boolean symbolsInIdentifiers : symbols) {
+ readings.add(new Reading(Syntax.STANDARD, backslashEscapes, nestedComments,
+ symbolsInIdentifiers));
+ }
+ }
+ }
+ }
+ if (syntax == Syntax.MYSQL || syntax == Syntax.UNKNOWN) {
+ for (boolean backslashEscapes : backslash) {
+ readings.add(new Reading(Syntax.MYSQL, backslashEscapes, false, true));
+ }
+ }
+ if (syntax == Syntax.TRANSACT_SQL || syntax == Syntax.UNKNOWN) {
+ readings.add(new Reading(Syntax.TRANSACT_SQL, false, true, true));
+ }
+ if (syntax == Syntax.GOOGLE_SQL || syntax == Syntax.UNKNOWN) {
+ readings.add(new Reading(Syntax.GOOGLE_SQL, true, false, true));
+ }
+ return readings;
+ }
+
+ /**
+ * One way of reading a statement. MySQL and GoogleSQL delimit identifiers with backticks and start a
+ * comment with {@code #}; only MySQL needs whitespace after {@code --}; only Transact-SQL delimits
+ * identifiers with square brackets. Outside standard SQL, double quotes can delimit a string: in
+ * MySQL unless {@code ANSI_QUOTES} is set, in Transact-SQL when {@code QUOTED_IDENTIFIER} is off, and
+ * in GoogleSQL always. Either way the quoted run ends at the same place, so it is replaced as a
+ * value. A GoogleSQL string is read on its own terms: a quote written twice ends it, a backslash
+ * escapes, and a quote written three times opens a string that ends only at the next three. In
+ * Transact-SQL a dollar before digits prefixes an amount, which is a value; elsewhere it prefixes a
+ * parameter or a name.
+ *
+ * @param syntax the syntax read in, never {@link Syntax#UNKNOWN}
+ * @param backslashEscapes whether a backslash escapes the character after it in a string
+ * @param nestedComments whether a block comment can contain another
+ * @param symbolsInIdentifiers whether {@code #} and {@code @} are part of a name, as in Oracle, DB2 and
+ * Transact-SQL, rather than operators, as in PostgreSQL, where a number after one is a value
+ */
+ private record Reading(Syntax syntax, boolean backslashEscapes, boolean nestedComments,
+ boolean symbolsInIdentifiers) {
+
+ /**
+ * @return the redacted statement, or null when a literal, a quoted identifier or a comment never ends
+ */
+ String redact(String sql) {
+ boolean mysql = syntax == Syntax.MYSQL;
+ boolean googleSql = syntax == Syntax.GOOGLE_SQL;
+ boolean doubleQuotedValue = syntax != Syntax.STANDARD;
+ int length = sql.length();
+ StringBuilder out = new StringBuilder(length);
+ int i = 0;
+ while (i < length) {
+ char c = sql.charAt(i);
+ char next = i + 1 < length ? sql.charAt(i + 1) : 0;
+ char previous = i > 0 ? sql.charAt(i - 1) : 0;
+ int dollarTag = c == '$' && !isIdentifierPart(previous) ? dollarTagLength(sql, i) : 0;
+ int end;
+ if (googleSql && (c == '\'' || c == '"')) {
+ end = endOfGoogleSqlString(sql, i);
+ out.append(PLACEHOLDER);
+ }
+ else if (c == '\'') {
+ end = endOfQuotedLiteral(sql, i);
+ out.append(PLACEHOLDER);
+ }
+ else if (c == '"') {
+ end = endOfQuoted(sql, i, '"', mysql && backslashEscapes, true);
+ if (end >= 0) {
+ out.append(doubleQuotedValue ? PLACEHOLDER : sql.substring(i, end));
+ }
+ }
+ else if (c == '`' && (mysql || googleSql)) {
+ end = endOfQuoted(sql, i, '`', googleSql, mysql);
+ if (end >= 0) {
+ out.append(sql, i, end);
+ }
+ }
+ else if (c == '[' && syntax == Syntax.TRANSACT_SQL) {
+ end = endOfQuoted(sql, i, ']', false, true);
+ if (end >= 0) {
+ out.append(sql, i, end);
+ }
+ }
+ else if (c == '/' && next == '*') {
+ end = endOfBlockComment(sql, i);
+ out.append(' ');
+ }
+ else if ((c == '-' && next == '-' && (!mysql || i + 2 >= length || Character.isWhitespace(sql.charAt(i + 2)))) ||
Review Comment:
MySQL starts a `--` comment when the next character is whitespace or a control character. `Character.isWhitespace` misses U+0001. `select 1 --<U+0001> api-key=example-secret` keeps the secret in the file.
Treat a control character after `--` as a comment for MySQL, or withhold the statement.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@grails.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add the query-tail command to find query-heavy pages [grails-core]
Posted by "testlens-app[bot] (via GitHub)" <gi...@apache.org>.
testlens-app[bot] commented on PR #16631:
URL: https://github.com/apache/grails-core/pull/16631#issuecomment-6378388107
## 🚨 All tests passed but jobs failed 🚨
### Failed Jobs without Test Failures
❌ [End to End / End to End Tests \(end-to-end build only\)](https://github.com/apache/grails-core/actions/runs/40050679066/job/120742456599?pr=16631)
🏷️ Commit: b849a58ddcc3c09d73edee3316bf7213b4c10b44
▶️ Tests: 23202 executed
🟡 Checks: 37/90 completed
### Rerun Controls
Click the checkbox to trigger a rerun:
🔲 **Rerun jobs**
---
_Learn more about TestLens at [testlens.app/docs](https://testlens.app/docs/features/pr-comment/)._
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@grails.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add the query-tail command to find query-heavy pages [grails-core]
Posted by "jdaugherty (via GitHub)" <gi...@apache.org>.
jdaugherty commented on code in PR #16631:
URL: https://github.com/apache/grails-core/pull/16631#discussion_r4367377948
##########
grails-data-mongodb/core/src/main/groovy/org/apache/grails/data/mongo/core/querytail/MongoCommandRedactor.groovy:
##########
@@ -0,0 +1,237 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * https://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.grails.data.mongo.core.querytail
+
+import groovy.transform.CompileStatic
+import org.bson.BsonDocument
+import org.bson.BsonValue
+
+import org.apache.grails.data.core.querytail.QueryTailRedactor
+
+/**
+ * Takes the values out of a MongoDB command before it is captured. A command is a document, so
+ * there is nothing to parse: every value becomes {@link QueryTailRedactor#PLACEHOLDER}, and the
+ * keys, which are the options of the command, its operators and the field names of the collection,
+ * are kept, except where a key may be data. The keys of a document stored in a map property, or in
+ * a property or a collection the {@linkplain MongoMappedFields mapping} does not know, are recorded
+ * as the placeholder too, and so is every segment of a dotted path past such a key. The entries the
+ * driver adds for its session, the cluster time and the transaction, which say nothing about the
+ * command, are left out.
+ *
+ * <p>A command is recorded as its name, the database and collection it runs against, and the
+ * document of its remaining entries: {@code find shop.book {"filter": {"title": ?}, "limit": ?}}.</p>
+ *
+ * @since 8.1
+ */
+@CompileStatic
+final class MongoCommandRedactor {
+
+ private static final String PLACEHOLDER = QueryTailRedactor.PLACEHOLDER
+
+ private static final String DATABASE = '$db'
+
+ private static final String COLLECTION = 'collection'
+
+ /** The entries the driver adds, which describe its session and transaction rather than the command. */
+ private static final Set<String> LEFT_OUT = ['lsid', '$clusterTime', 'txnNumber', 'autocommit',
+ 'startTransaction', '$readPreference', 'apiVersion', 'apiStrict', 'apiDeprecationErrors'] as Set<String>
+
+ /**
+ * The options whose document, or documents, are keyed by the fields of the collection, wherever they appear,
+ * including those of the operations of a client-level {@code bulkWrite}.
+ */
+ private static final Set<String> FIELD_KEYED = ['filter', 'query', 'q', 'u', 'update', 'document', 'documents',
+ 'updateMods', 'pipeline', 'sort', 'projection', 'fields', 'key', 'hint', 'min', 'max', 'arrayFilters',
+ 'partialFilterExpression', 'wildcardProjection'] as Set<String>
+
+ private MongoCommandRedactor() {
+ }
+
+ /**
+ * Redacts a command.
+ *
+ * @param database the database the command runs against
+ * @param commandName the name of the command, which is its first key
+ * @param command the command as the driver sends it
+ * @param fields the fields the mapping of the datastore knows, or null before the datastore is built
+ * @return the command with its values replaced
+ */
+ static String redact(String database, String commandName, BsonDocument command, MongoMappedFields fields) {
+ String collection = null
+ String collectionKey = null
+ BsonValue first = command.get(commandName)
+ if (first != null && first.isString()) {
+ collection = first.asString().value
+ collectionKey = commandName
+ }
+ else if (command.get(COLLECTION)?.isString()) {
+ collection = command.getString(COLLECTION).value
+ collectionKey = COLLECTION
+ }
+ StringBuilder out = new StringBuilder(commandName)
+ if (database != null || collection != null) {
+ out.append(' ')
+ if (database != null) {
+ out.append(database)
+ }
+ if (collection != null) {
+ out.append(database != null ? '.' : '').append(collection)
+ }
+ }
+ Scope options = Scope.options(collection, fields)
+ StringBuilder body = new StringBuilder()
+ for (Map.Entry<String, BsonValue> entry in command.entrySet()) {
+ String key = entry.key
+ if (key == DATABASE || key == collectionKey || LEFT_OUT.contains(key)) {
+ continue
+ }
+ body.append(body.length() == 0 ? '{' : ', ')
+ appendEntry(body, key, entry.value, options)
+ }
+ if (body.length() > 0) {
+ out.append(' ').append(body).append('}')
+ }
+ out.toString()
+ }
+
+ private static void appendEntry(StringBuilder out, String key, BsonValue value, Scope scope) {
+ Scope child = scope.of(key)
+ out.append(child.keyText).append(': ')
+ appendValue(out, value, child)
+ }
+
+ private static void appendValue(StringBuilder out, BsonValue value, Scope scope) {
+ if (value.isDocument()) {
+ out.append('{')
+ boolean first = true
+ for (Map.Entry<String, BsonValue> entry in value.asDocument().entrySet()) {
+ if (!first) {
+ out.append(', ')
+ }
+ first = false
+ appendEntry(out, entry.key, entry.value, scope)
+ }
+ out.append('}')
+ }
+ else if (value.isArray()) {
+ out.append('[')
+ boolean first = true
+ for (BsonValue element in value.asArray()) {
+ if (!first) {
+ out.append(', ')
+ }
+ first = false
+ appendValue(out, element, scope)
+ }
+ out.append(']')
+ }
+ else {
+ out.append(PLACEHOLDER)
+ }
+ }
+
+ private static String quote(String key) {
+ StringBuilder out = new StringBuilder(key.length() + 2).append('"')
+ for (int i = 0; i < key.length(); i++) {
+ char c = key.charAt(i)
+ if (c == '"' || c == '\\') {
+ out.append('\\').append(c)
+ }
+ else if (c < 0x20) {
+ out.append(String.format('\\u%04x', (int) c))
+ }
+ else {
+ out.append(c)
+ }
+ }
+ out.append('"').toString()
+ }
+
+ /**
+ * Where a document sits in the command: among the options of the command, whose keys are kept
+ * and name what their documents hold, or in a document keyed by fields, where a key is kept when
+ * it is an operator or a field the mapping knows at its path.
+ */
+ @CompileStatic
+ private static final class Scope {
+
+ final String collection
+ final MongoMappedFields fields
+ /** The field path of the document; null in a document keyed by fields the mapping does not know. */
+ final List<String> path
+ final boolean optionKeyed
+ final String keyText
+
+ private Scope(String collection, MongoMappedFields fields, List<String> path, boolean optionKeyed,
+ String keyText) {
+ this.collection = collection
+ this.fields = fields
+ this.path = path
+ this.optionKeyed = optionKeyed
+ this.keyText = keyText
+ }
+
+ static Scope options(String collection, MongoMappedFields fields) {
+ new Scope(collection, fields, Collections.<String>emptyList(), true, null)
+ }
+
+ /**
+ * @return the scope of the value of a key in a document of this scope, with the key as it is recorded
+ */
+ Scope of(String key) {
+ if (optionKeyed) {
+ boolean fieldKeyed = FIELD_KEYED.contains(key)
+ return new Scope(collection, fields, Collections.<String>emptyList(), !fieldKeyed, quote(key))
+ }
+ if (key.startsWith('$')) {
+ return new Scope(collection, fields, path, false, quote(key))
+ }
+ List<String> current = path
+ String[] segments = key.split('\\.', -1)
+ StringBuilder recorded = new StringBuilder()
+ boolean anyKept = false
+ for (int i = 0; i < segments.length; i++) {
+ String segment = segments[i]
+ boolean kept
+ if (current == null) {
+ kept = false
+ }
+ else if (MongoMappedFields.isIndex(segment)) {
Review Comment:
Fixed in b3a97c2543 (folded into the feature commit). A numeric segment is now kept only when the path before it is a field the mapping knows as an array (`MongoMappedFields.isIndexed`: an array- or `Collection`-typed property, reached through embedded documents and collections as `isField` does); anywhere else, in a map property, under an unknown key, or at the top of a document, it is data and recorded as `?`, and the path past it is unknown. So `attrs: {"9876543210": "v"}` and `"attrs.9876543210"` both record `?`, while `stops.0.city` and `tags.0` keep the index. Covered in `MongoCommandRedactorSpec` (nested, dotted, top-level and `$set` forms, plus the never-survives case with a numeric secret) and through the capture API in `QueryTailCommandListenerSpec` against a real mongod, which saves a book whose `attrs` map has a numeric key, queries by that key, and asserts the key reaches neither the insert nor the find in the file. The guide says a numeric key is kept only where it i
ndexes a list the mapping knows.
##########
grails-datamapping-core/src/main/groovy/org/apache/grails/data/core/querytail/QueryTailCapture.groovy:
##########
@@ -0,0 +1,356 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * https://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.grails.data.core.querytail
+
+import java.nio.ByteBuffer
+import java.nio.channels.FileChannel
+import java.nio.channels.FileLock
+import java.nio.channels.OverlappingFileLockException
+import java.nio.charset.StandardCharsets
+import java.nio.file.Files
+import java.nio.file.StandardOpenOption
+import java.util.concurrent.atomic.AtomicBoolean
+
+import groovy.transform.CompileStatic
+import groovy.util.logging.Slf4j
+
+/**
+ * Writes the statements an application runs to the capture file the {@code query-tail} command
+ * tails. Each session factory, Neo4j driver or MongoDB client that captures opens its own capture
+ * and closes it when it closes; the captures of one file share it, so every data source of an
+ * application writes to the one file. The file is truncated when the first capture opens it, so it
+ * holds the statements of the current run only, and its first line names that run. It is closed
+ * when the last capture closes, and opened afresh by the next one. Each statement is written whole
+ * and at once, so that the command sees it as soon as it is recorded.
+ *
+ * <p>JVMs that run at the same time with capture enabled, such as the parallel forks of a test task,
+ * share the file: a lock file beside it tells the first to start, which truncates the file and names
+ * the run, from the others, which append to that run without truncating it and number their
+ * requests apart, so that the command reads one run from all of them.</p>
+ *
+ * <p>A statement is recorded without its values: {@link QueryTailRedactor} replaces its literals and
+ * drops its comments before it is written, in the {@link #syntax} of the database it runs on. A
+ * statement that is not text, such as a MongoDB command, has its values taken out by whoever records
+ * it, through {@link #recordRedacted}.</p>
+ *
+ * <p>A capture is only opened when {@link QueryTailSettings#SETTING_ENABLED} is set; an application
+ * started without it never reaches this class.</p>
+ *
+ * @since 8.1
+ */
+@CompileStatic
+@Slf4j
+class QueryTailCapture implements Closeable {
+
+ private static final Map<String, CaptureFile> OPEN = new HashMap<>()
+
+ /** The file the statements are written to. */
+ final File file
+
+ /** The syntax of the database the statements of this capture run on, which their values are found by. */
+ volatile QueryTailSyntax syntax = QueryTailSyntax.UNKNOWN
+
+ private final CaptureFile captureFile
+ private final AtomicBoolean closed = new AtomicBoolean()
+
+ private QueryTailCapture(File file, CaptureFile captureFile) {
+ this.file = file
+ this.captureFile = captureFile
+ }
+
+ /**
+ * Opens a capture of a file. The first capture of a file truncates it; a capture opened while
+ * another of the same file is open shares it. Every capture opened must be closed.
+ *
+ * @param file the capture file
+ * @return the capture
+ * @throws UncheckedIOException when the file cannot be created
+ */
+ static QueryTailCapture open(File file) {
+ String key
+ try {
+ key = file.canonicalPath
+ }
+ catch (IOException e) {
+ throw new UncheckedIOException("Cannot resolve the query-tail capture file ${file}: ${e.message}", e)
+ }
+ synchronized (OPEN) {
+ CaptureFile captureFile = OPEN.get(key)
+ if (captureFile == null) {
+ captureFile = new CaptureFile(file, key)
+ OPEN.put(key, captureFile)
+ if (captureFile.joined) {
+ log.info('Capturing statements to {} for query-tail, in the run of another JVM writing it', file)
+ }
+ else {
+ log.info('Capturing statements to {} for query-tail', file)
+ }
+ }
+ captureFile.captures++
+ return new QueryTailCapture(file, captureFile)
+ }
+ }
+
+ /**
+ * Opens the capture the settings name.
+ *
+ * @param settings the query-tail settings
+ * @return the capture
+ */
+ static QueryTailCapture open(QueryTailSettings settings) {
+ open(settings.resolveFile())
+ }
+
+ /**
+ * Records a statement that is about to run on the current thread, without its values.
+ *
+ * @param dataSource the connection source name the statement runs against
+ * @param sql the statement
+ */
+ void record(String dataSource, String sql) {
+ write(dataSource, QueryTailRedactor.redact(sql, syntax))
+ }
+
+ /**
+ * Records a statement that is about to run on the current thread and that holds no value any
+ * more: a MongoDB command, which is a document rather than text, is redacted field by field by
+ * the listener that records it. Nothing of the statement is changed here.
+ *
+ * @param dataSource the connection source name the statement runs against
+ * @param statement the statement, with its values already taken out
+ */
+ void recordRedacted(String dataSource, String statement) {
+ write(dataSource, statement)
+ }
+
+ private void write(String dataSource, String statement) {
+ if (closed.get()) {
+ return
+ }
+ QueryTailRecord record = new QueryTailRecord(timestamp: System.currentTimeMillis(), dataSource: dataSource,
+ sql: statement)
+ captureFile.write(captureFile.attribution.attribute(record))
+ }
+
+ /**
+ * @return whether this capture is still writing to its file
+ */
+ boolean isOpen() {
+ !closed.get() && captureFile.open
+ }
+
+ /**
+ * Closes this capture. The file is closed once every capture of it is.
+ */
+ @Override
+ void close() {
+ if (!closed.compareAndSet(false, true)) {
+ return
+ }
+ boolean last
+ synchronized (OPEN) {
+ last = --captureFile.captures == 0
+ if (last && OPEN.get(captureFile.key).is(captureFile)) {
+ OPEN.remove(captureFile.key)
+ }
+ }
+ if (last) {
+ captureFile.close()
+ }
+ }
+
+ /**
+ * A capture file and the captures writing to it. The file is written by one channel, a line at a
+ * time, and the requests of every data source are numbered by one attribution, so that two
+ * requests never share an id.
+ *
+ * <p>A lock file beside the capture coordinates the JVMs writing it. Every writer holds a shared
+ * lock on the {@link #WRITING} byte of the lock file while it writes. A starting writer first
+ * takes the {@link #STARTING} byte exclusively, so that starters take turns, then tries the
+ * {@link #WRITING} byte exclusively: when it gets it, no other JVM is writing, and it truncates the
+ * file and writes the header of a new run; when it does not, it joins the run being written. Either
+ * way it then holds the {@link #WRITING} byte shared until it closes. A lock file that cannot be
+ * created, as beside a device, leaves the writer uncoordinated, as the only one.</p>
+ */
+ @CompileStatic
+ private static class CaptureFile {
+
+ /** The byte of the lock file a starting writer holds, exclusively, while it decides whether to truncate. */
+ private static final long STARTING = 0L
+
+ /** The byte of the lock file every writer holds, shared, while it writes. */
+ private static final long WRITING = 1L
+
+ final File file
+ final String key
+ final QueryTailRequestAttribution attribution
+
+ /** Whether this JVM joined the run of another JVM writing the file, rather than starting one. */
+ final boolean joined
+
+ /** How many captures are open on the file; guarded by {@link QueryTailCapture#OPEN}. */
+ int captures
+
+ private FileChannel channel
+ private FileChannel lockChannel
+ private FileLock writing
+
+ CaptureFile(File file, String key) {
+ this.file = file
+ this.key = key
+ File parent = file.absoluteFile.parentFile
+ boolean first
+ try {
+ if (parent != null && !parent.isDirectory() && !parent.mkdirs() && !parent.isDirectory()) {
+ throw new IOException("Cannot create the query-tail capture directory ${parent}")
+ }
+ lockChannel = openLockFile(file)
+ FileLock starting = lockChannel == null ? null : lockChannel.lock(STARTING, 1L, false)
+ try {
+ FileLock alone = lockChannel == null ? null : lock(lockChannel, false, true)
+ first = lockChannel == null || alone != null
+ alone?.release()
+ if (first) {
+ Files.newByteChannel(file.toPath(), StandardOpenOption.CREATE, StandardOpenOption.WRITE,
+ StandardOpenOption.TRUNCATE_EXISTING).close()
+ }
+ channel = FileChannel.open(file.toPath(), StandardOpenOption.CREATE, StandardOpenOption.WRITE,
+ StandardOpenOption.APPEND)
+ writing = lockChannel == null ? null : lock(lockChannel, true, false)
+ }
+ finally {
Review Comment:
Fixed in b3a97c2543. The starter now writes the run header while it still holds the `STARTING` byte: the header write moved inside the locked section, after the shared `WRITING` lock is taken, so a second JVM that is blocked on `STARTING` cannot append anything before the header, and the javadoc of `CaptureFile` describes the order. I did not find a way to drive that window from a single JVM, since a second lock on the same byte in one process throws `OverlappingFileLockException` instead of blocking, so there is no new spec for the ordering itself; the existing joined-run and truncation specs still pass.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@grails.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add the sql-tail command to find query-heavy pages [grails-core]
Posted by "jdaugherty (via GitHub)" <gi...@apache.org>.
jdaugherty commented on PR #16631:
URL: https://github.com/apache/grails-core/pull/16631#issuecomment-6297188451
The JaCoCo change here (97e21ab3e9, with the lazy follow-up 153dc6da67) fixes a gap that is not specific to this branch: `GrailsJacocoPlugin` and its spec are byte-identical on 8.0.x, 8.1.x, 9.0.x and 10.0.x, and the cli source sets of grails-bootstrap, grails-core, grails-openapi, grails-scaffolding and grails-web-url-mappings have been missing from Codecov on all of them since the cli split.
Both commits are cherry-picked onto 8.0.x as #16689 so the fix reaches every branch through the merge-up. Once that lands on 9.0.x, the two commits here meet identical ones and this PR stays a sql-tail change only.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@grails.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add the query-tail command to find query-heavy pages [grails-core]
Posted by "jdaugherty (via GitHub)" <gi...@apache.org>.
jdaugherty commented on code in PR #16631:
URL: https://github.com/apache/grails-core/pull/16631#discussion_r4390361455
##########
grails-datamapping-core/src/main/groovy/org/apache/grails/datamapping/core/querytail/StandardSqlSyntax.groovy:
##########
@@ -0,0 +1,310 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * https://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.grails.datamapping.core.querytail
+
+import groovy.transform.CompileStatic
+import groovy.transform.PackageScope
+
+/**
+ * The rules of standard SQL, which every database shares: a single-quoted string in which the
+ * quote is written twice, a double-quoted name in which the quote is written twice, a number, a
+ * {@code --} comment to the end of the line, and a {@code /* ... *}{@code /} comment that does not
+ * nest. The syntax of a database extends it and overrides the hooks its database differs on.
+ */
+@CompileStatic
+@PackageScope
+abstract class StandardSqlSyntax implements QueryTailSyntax {
+
+ private static final char SPACE = ' '
+ private static final char ZERO = '0'
+ private static final char NINE = '9'
+ private static final char LOWER_A = 'a'
+ private static final char LOWER_Z = 'z'
+ private static final char UPPER_A = 'A'
+ private static final char UPPER_Z = 'Z'
+ private static final char UNDERSCORE = '_'
+ private static final char ASCII_LAST = 0x7F as char
+
+ private final String name
+
+ protected StandardSqlSyntax(String name) {
+ this.name = name
+ }
+
+ @Override
+ int endOfComment(String sql, int index) {
+ if (sql.startsWith('/*', index)) {
+ return endOfBlockComment(sql, index)
+ }
+ if (sql.startsWith('--', index)) {
+ return endOfLine(sql, index)
+ }
+ NONE
+ }
+
+ @Override
+ int endOfValue(String sql, int index) {
+ char c = sql.charAt(index)
+ if (c == '\'') {
+ return endOfString(sql, index)
+ }
+ if (c == '"' && doubleQuotedStrings()) {
+ return endOfQuoted(sql, index, '"' as char, backslashEscapes(), true)
+ }
+ if (startsNumber(sql, index) && standalone(sql, index)) {
+ return endOfNumber(sql, index)
+ }
+ NONE
+ }
+
+ @Override
+ int endOfQuotedName(String sql, int index) {
Review Comment:
Fixed in b8f737d030. A password is not always a literal: Oracle takes the one `IDENTIFIED BY` gives as a name, quoted or bare, and SAP HANA takes a user's `PASSWORD` the same way. So the redactor now withholds a statement that gives a password, whole and on every database:
- one with `IDENTIFIED BY`, including when a comment or line break splits the two words;
- one that creates or alters a user, role or login and gives it a `PASSWORD`;
- one that starts with `CONNECT`, since Db2's `CONNECT ... USING` takes a bare password too.
Your statement is asserted withheld, quoted and bare, with `ExampleSecret123` absent from the file:
- `QueryTailRedactorSpec`: withheld rows for Oracle (quoted, bare, `REPLACE`, a comment between `IDENTIFIED` and `BY`, role and database-link passwords, and before the database is known), SAP HANA, Db2, MySQL, PostgreSQL and Neo4j.
- `QueryTailCaptureSpec`: the capture file.
A `password` column in a query, Oracle's `CONNECT BY` and table DDL are still recorded as before.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@grails.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add the sql-tail command to find query-heavy pages [grails-core]
Posted by "jdaugherty (via GitHub)" <gi...@apache.org>.
jdaugherty commented on code in PR #16631:
URL: https://github.com/apache/grails-core/pull/16631#discussion_r4351167877
##########
grails-datamapping-core/src/main/groovy/org/grails/datastore/gorm/jdbc/sqltail/SqlTailRedactor.java:
##########
@@ -0,0 +1,363 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * https://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.grails.datastore.gorm.jdbc.sqltail;
+
+import java.util.ArrayList;
+import java.util.List;
+
+/**
+ * Takes the values out of a statement before it is captured. Every string, number and
+ * dollar-quoted literal becomes {@link #PLACEHOLDER} and every comment is dropped, so that a value
+ * written into the SQL text, a password in a native query or a key in a comment, never reaches the
+ * capture file. Only the recorded copy is redacted: the statement the database runs is not changed.
+ *
+ * <p>Databases disagree on parts of the lexical syntax: whether a backslash escapes a quote, whether
+ * comments nest, whether double quotes delimit an identifier or a string, and whether {@code #}
+ * starts a comment. A statement is redacted under every reading its {@link Syntax} allows, and is
+ * recorded only when all of them give the same text. When they do not, or when a literal or a
+ * comment never ends, the statement is withheld: {@link #WITHHELD} is recorded in its place, never
+ * the statement.</p>
+ *
+ * @since 8.1
+ */
+public final class SqlTailRedactor {
+
+ /** What a literal is replaced with. */
+ public static final String PLACEHOLDER = "?";
+
+ /** What is recorded in place of a statement whose values cannot be told apart from its SQL. */
+ public static final String WITHHELD = "(statement withheld: its values could not be redacted)";
+
+ private static final boolean[] NO = {false};
+ private static final boolean[] EITHER = {false, true};
+
+ /**
+ * The lexical syntax of the database a statement runs on, as far as finding its values goes.
+ */
+ public enum Syntax {
+
+ /** Not known: a statement is recorded only when the readings of every syntax below agree. */
+ UNKNOWN,
+
+ /** Standard SQL, where double quotes delimit identifiers: H2, PostgreSQL, Oracle, DB2 and most others. */
+ STANDARD,
+
+ /** MySQL and MariaDB: double quotes delimit strings, and {@code #} and {@code -- } start comments. */
+ MYSQL,
+
+ /** Microsoft SQL Server and Sybase: square brackets delimit identifiers, and comments nest. */
+ TRANSACT_SQL;
+
+ /**
+ * @param openQuote the character a dialect opens a quoted identifier with
+ * @return the syntax of the databases that quote identifiers with it
+ */
+ public static Syntax forIdentifierQuote(char openQuote) {
+ return switch (openQuote) {
+ case '"' -> STANDARD;
+ case '`' -> MYSQL;
+ case '[' -> TRANSACT_SQL;
+ default -> UNKNOWN;
+ };
+ }
+ }
+
+ private SqlTailRedactor() {
+ }
+
+ /**
+ * Redacts a statement.
+ *
+ * @param sql the statement
+ * @param syntax the syntax of the database it runs on
+ * @return the statement with its literals replaced and its comments dropped, or {@link #WITHHELD}
+ */
+ public static String redact(String sql, Syntax syntax) {
+ if (sql == null) {
+ return null;
+ }
+ String redacted = null;
+ for (Reading reading : readings(sql, syntax == null ? Syntax.UNKNOWN : syntax)) {
+ String candidate = reading.redact(sql);
+ if (candidate == null || (redacted != null && !redacted.equals(candidate))) {
+ return WITHHELD;
+ }
+ redacted = candidate;
+ }
+ return redacted;
+ }
+
+ /**
+ * The readings a statement has to agree under. A reading that depends on a character the
+ * statement does not contain is the same as the one that does not, so it is left out.
+ */
+ private static List<Reading> readings(String sql, Syntax syntax) {
+ boolean[] backslash = sql.indexOf('\\') >= 0 ? EITHER : NO;
+ int comment = sql.indexOf("/*");
+ boolean[] nesting = comment >= 0 && sql.indexOf("/*", comment + 2) >= 0 ? EITHER : NO;
+ List<Reading> readings = new ArrayList<>();
+ if (syntax == Syntax.STANDARD || syntax == Syntax.UNKNOWN) {
+ for (boolean backslashEscapes : backslash) {
+ for (boolean nestedComments : nesting) {
+ readings.add(new Reading(Syntax.STANDARD, backslashEscapes, nestedComments));
+ }
+ }
+ }
+ if (syntax == Syntax.MYSQL || syntax == Syntax.UNKNOWN) {
+ for (boolean backslashEscapes : backslash) {
+ readings.add(new Reading(Syntax.MYSQL, backslashEscapes, false));
+ }
+ }
+ if (syntax == Syntax.TRANSACT_SQL || syntax == Syntax.UNKNOWN) {
+ readings.add(new Reading(Syntax.TRANSACT_SQL, false, true));
+ }
+ return readings;
+ }
+
+ /**
+ * One way of reading a statement. Only MySQL delimits strings with double quotes and identifiers
+ * with backticks, starts a comment with {@code #}, and needs whitespace after {@code --}; only
+ * Transact-SQL delimits identifiers with square brackets.
+ *
+ * @param syntax the syntax read in, never {@link Syntax#UNKNOWN}
+ * @param backslashEscapes whether a backslash escapes the character after it in a string
+ * @param nestedComments whether a block comment can contain another
+ */
+ private record Reading(Syntax syntax, boolean backslashEscapes, boolean nestedComments) {
+
+ /**
+ * @return the redacted statement, or null when a literal, a quoted identifier or a comment never ends
+ */
+ String redact(String sql) {
+ boolean mysql = syntax == Syntax.MYSQL;
+ int length = sql.length();
+ StringBuilder out = new StringBuilder(length);
+ int i = 0;
+ while (i < length) {
+ char c = sql.charAt(i);
+ char next = i + 1 < length ? sql.charAt(i + 1) : 0;
+ char previous = i > 0 ? sql.charAt(i - 1) : 0;
+ int dollarTag = c == '$' && !isIdentifierPart(previous) ? dollarTagLength(sql, i) : 0;
+ int end;
+ if (c == '\'') {
+ end = endOfQuotedLiteral(sql, i);
+ out.append(PLACEHOLDER);
+ }
+ else if (c == '"') {
Review Comment:
Fixed in a562379a66. Double-quoted text is now recorded as `?` in every syntax except standard SQL. In Transact-SQL the quoted text ends at the same place whether `QUOTED_IDENTIFIER` is on or off, so it's replaced rather than withheld. The cost is that a double-quoted name in a hand-written SQL Server statement is recorded as `?` too. Your batch is now covered through the capture API in `SqlTailCaptureSpec` with the capture's syntax set to Transact-SQL, and in the Transact-SQL table of `SqlTailRedactorSpec`, along with a `""` escape inside the quotes.
Sybase had the same leak through a different path. Its dialect quotes identifiers with `"`, so the inspector classed it as standard SQL, but Sybase's `QUOTED_IDENTIFIER` is off by default. Both inspectors now pick the syntax from the dialect class instead of the quote character:
- SQL Server and every Sybase dialect (ASE, SQL Anywhere) are Transact-SQL.
- Dialects that quote identifiers with backticks (MySQL, MariaDB, Spanner) read as MySQL.
- Only H2, PostgreSQL, CockroachDB, Oracle, DB2, HSQLDB and HANA, plus Derby on Hibernate 5, are standard SQL.
- Any other dialect, including SQLite and Informix, which read double-quoted text as a string by default, is read in every syntax, so a double-quoted run there withholds the statement.
`SqlTailStatementInspector.syntaxOf` is covered with real dialects in both Hibernate specs, and the redaction section of the guide now describes this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@grails.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add the sql-tail command to find query-heavy pages [grails-core]
Posted by "jamesfredley (via GitHub)" <gi...@apache.org>.
jamesfredley commented on code in PR #16631:
URL: https://github.com/apache/grails-core/pull/16631#discussion_r4354197353
##########
grails-datamapping-core/src/main/groovy/org/grails/datastore/gorm/jdbc/sqltail/SqlTailRedactor.java:
##########
@@ -0,0 +1,414 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * https://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.grails.datastore.gorm.jdbc.sqltail;
+
+import java.util.ArrayList;
+import java.util.List;
+
+/**
+ * Takes the values out of a statement before it is captured. Every string, number and
+ * dollar-quoted literal becomes {@link #PLACEHOLDER} and every comment is dropped, so that a value
+ * written into the SQL text, a password in a native query or a key in a comment, never reaches the
+ * capture file. Only the recorded copy is redacted: the statement the database runs is not changed.
+ *
+ * <p>Databases disagree on parts of the lexical syntax: whether a backslash escapes a quote, whether
+ * comments nest, whether double quotes delimit an identifier or a string, whether a quote written
+ * twice escapes it or ends one string and starts another, whether a quote written three times
+ * opens a string that quotes can appear in, and whether {@code #} starts a comment. A statement is
+ * redacted under every reading its {@link Syntax} allows, and is recorded only when all of them give
+ * the same text. When they do not, or when a literal or a comment never ends, the statement is
+ * withheld: {@link #WITHHELD} is recorded in its place, never the statement.</p>
+ *
+ * @since 8.1
+ */
+public final class SqlTailRedactor {
+
+ /** What a literal is replaced with. */
+ public static final String PLACEHOLDER = "?";
+
+ /** What is recorded in place of a statement whose values cannot be told apart from its SQL. */
+ public static final String WITHHELD = "(statement withheld: its values could not be redacted)";
+
+ private static final boolean[] NO = {false};
+ private static final boolean[] EITHER = {false, true};
+
+ /**
+ * The lexical syntax of the database a statement runs on, as far as finding its values goes.
+ */
+ public enum Syntax {
+
+ /**
+ * Not known, or a database that can read double-quoted text as a string although it quotes
+ * identifiers with double quotes, such as SQLite and Informix: a statement is recorded only when
+ * the readings of every syntax below agree.
+ */
+ UNKNOWN,
+
+ /** Standard SQL, where double quotes delimit identifiers: H2, PostgreSQL, Oracle, DB2, HSQLDB and HANA. */
+ STANDARD,
+
+ /** MySQL and MariaDB: double quotes delimit strings, and {@code #} and {@code -- } start comments. */
+ MYSQL,
+
+ /**
+ * Microsoft SQL Server and Sybase: square brackets delimit identifiers, comments nest, and double
+ * quotes delimit a string when {@code QUOTED_IDENTIFIER} is off, as it is by default on Sybase.
+ */
+ TRANSACT_SQL,
+
+ /**
+ * GoogleSQL, the dialect of Cloud Spanner: backticks delimit identifiers, single and double quotes
+ * delimit strings, a string that opens with its quote written three times ends only at the quote
+ * written three times again and holds any other quote, a backslash always escapes the character
+ * after it, a quote written twice is two strings, and {@code #} starts a comment.
+ */
+ GOOGLE_SQL
+ }
+
+ private SqlTailRedactor() {
+ }
+
+ /**
+ * Redacts a statement.
+ *
+ * @param sql the statement
+ * @param syntax the syntax of the database it runs on
+ * @return the statement with its literals replaced and its comments dropped, or {@link #WITHHELD}
+ */
+ public static String redact(String sql, Syntax syntax) {
+ if (sql == null) {
+ return null;
+ }
+ String redacted = null;
+ for (Reading reading : readings(sql, syntax == null ? Syntax.UNKNOWN : syntax)) {
+ String candidate = reading.redact(sql);
+ if (candidate == null || (redacted != null && !redacted.equals(candidate))) {
+ return WITHHELD;
+ }
+ redacted = candidate;
+ }
+ return redacted;
+ }
+
+ /**
+ * The readings a statement has to agree under. A reading that depends on a character the
+ * statement does not contain is the same as the one that does not, so it is left out.
+ */
+ private static List<Reading> readings(String sql, Syntax syntax) {
+ boolean[] backslash = sql.indexOf('\\') >= 0 ? EITHER : NO;
+ int comment = sql.indexOf("/*");
+ boolean[] nesting = comment >= 0 && sql.indexOf("/*", comment + 2) >= 0 ? EITHER : NO;
+ List<Reading> readings = new ArrayList<>();
+ if (syntax == Syntax.STANDARD || syntax == Syntax.UNKNOWN) {
+ for (boolean backslashEscapes : backslash) {
+ for (boolean nestedComments : nesting) {
+ readings.add(new Reading(Syntax.STANDARD, backslashEscapes, nestedComments));
+ }
+ }
+ }
+ if (syntax == Syntax.MYSQL || syntax == Syntax.UNKNOWN) {
+ for (boolean backslashEscapes : backslash) {
+ readings.add(new Reading(Syntax.MYSQL, backslashEscapes, false));
+ }
+ }
+ if (syntax == Syntax.TRANSACT_SQL || syntax == Syntax.UNKNOWN) {
+ readings.add(new Reading(Syntax.TRANSACT_SQL, false, true));
+ }
+ if (syntax == Syntax.GOOGLE_SQL || syntax == Syntax.UNKNOWN) {
+ readings.add(new Reading(Syntax.GOOGLE_SQL, true, false));
+ }
+ return readings;
+ }
+
+ /**
+ * One way of reading a statement. MySQL and GoogleSQL delimit identifiers with backticks and start a
+ * comment with {@code #}; only MySQL needs whitespace after {@code --}; only Transact-SQL delimits
+ * identifiers with square brackets. Outside standard SQL, double quotes can delimit a string: in
+ * MySQL unless {@code ANSI_QUOTES} is set, in Transact-SQL when {@code QUOTED_IDENTIFIER} is off, and
+ * in GoogleSQL always. Either way the quoted run ends at the same place, so it is replaced as a
+ * value. A GoogleSQL string is read on its own terms: a quote written twice ends it, a backslash
+ * escapes, and a quote written three times opens a string that ends only at the next three.
+ *
+ * @param syntax the syntax read in, never {@link Syntax#UNKNOWN}
+ * @param backslashEscapes whether a backslash escapes the character after it in a string
+ * @param nestedComments whether a block comment can contain another
+ */
+ private record Reading(Syntax syntax, boolean backslashEscapes, boolean nestedComments) {
+
+ /**
+ * @return the redacted statement, or null when a literal, a quoted identifier or a comment never ends
+ */
+ String redact(String sql) {
+ boolean mysql = syntax == Syntax.MYSQL;
+ boolean googleSql = syntax == Syntax.GOOGLE_SQL;
+ boolean doubleQuotedValue = syntax != Syntax.STANDARD;
+ int length = sql.length();
+ StringBuilder out = new StringBuilder(length);
+ int i = 0;
+ while (i < length) {
+ char c = sql.charAt(i);
+ char next = i + 1 < length ? sql.charAt(i + 1) : 0;
+ char previous = i > 0 ? sql.charAt(i - 1) : 0;
+ int dollarTag = c == '$' && !isIdentifierPart(previous) ? dollarTagLength(sql, i) : 0;
+ int end;
+ if (googleSql && (c == '\'' || c == '"')) {
+ end = endOfGoogleSqlString(sql, i);
+ out.append(PLACEHOLDER);
+ }
+ else if (c == '\'') {
+ end = endOfQuotedLiteral(sql, i);
+ out.append(PLACEHOLDER);
+ }
+ else if (c == '"') {
+ end = endOfQuoted(sql, i, '"', mysql && backslashEscapes, true);
+ if (end >= 0) {
+ out.append(doubleQuotedValue ? PLACEHOLDER : sql.substring(i, end));
+ }
+ }
+ else if (c == '`' && (mysql || googleSql)) {
+ end = endOfQuoted(sql, i, '`', googleSql, mysql);
+ if (end >= 0) {
+ out.append(sql, i, end);
+ }
+ }
+ else if (c == '[' && syntax == Syntax.TRANSACT_SQL) {
+ end = endOfQuoted(sql, i, ']', false, true);
+ if (end >= 0) {
+ out.append(sql, i, end);
+ }
+ }
+ else if (c == '/' && next == '*') {
+ end = endOfBlockComment(sql, i);
+ out.append(' ');
+ }
+ else if ((c == '-' && next == '-' && (!mysql || i + 2 >= length || Character.isWhitespace(sql.charAt(i + 2)))) ||
+ (c == '#' && (mysql || googleSql))) {
+ end = endOfLine(sql, i);
+ out.append(' ');
+ }
+ else if (dollarTag > 0) {
+ String tag = sql.substring(i, i + dollarTag);
+ int close = sql.indexOf(tag, i + dollarTag);
+ end = close < 0 ? -1 : close + tag.length();
+ out.append(PLACEHOLDER);
+ }
+ else if ((isDigit(c) || (c == '.' && isDigit(next))) && !isIdentifierPart(previous)) {
+ end = endOfNumber(sql, i);
+ out.append(PLACEHOLDER);
+ }
+ else {
+ out.append(c);
+ end = i + 1;
+ }
+ if (end < 0) {
+ return null;
+ }
+ i = end;
+ }
+ return out.toString();
+ }
+
+ /**
+ * Ends a single-quoted literal, which may carry a prefix: {@code E'...'} always takes backslash
+ * escapes, and Oracle's {@code q'[...]'} ends at its own closing delimiter.
+ */
+ private int endOfQuotedLiteral(String sql, int start) {
+ char prefix = start > 0 ? sql.charAt(start - 1) : 0;
+ boolean standalonePrefix = start < 2 || !isIdentifierPart(sql.charAt(start - 2));
+ if ((prefix == 'q' || prefix == 'Q') && (standalonePrefix || isNationalPrefix(sql, start - 2)) &&
+ start + 1 < sql.length() && !Character.isWhitespace(sql.charAt(start + 1)) &&
+ sql.charAt(start + 1) != '\'') {
+ int close = sql.indexOf(closingDelimiter(sql.charAt(start + 1)) + "'", start + 2);
+ return close < 0 ? -1 : close + 2;
+ }
+ boolean escapeString = (prefix == 'e' || prefix == 'E') && standalonePrefix;
+ return endOfQuoted(sql, start, '\'', backslashEscapes || escapeString, true);
+ }
+
+ private int endOfBlockComment(String sql, int start) {
+ int depth = 1;
+ int i = start + 2;
+ while (i + 1 < sql.length()) {
+ char c = sql.charAt(i);
+ char next = sql.charAt(i + 1);
+ if (c == '*' && next == '/') {
+ depth--;
+ i += 2;
+ if (depth == 0) {
+ return i;
+ }
+ }
+ else if (nestedComments && c == '/' && next == '*') {
+ depth++;
+ i += 2;
+ }
+ else {
+ i++;
+ }
+ }
+ return -1;
+ }
+ }
+
+ /**
+ * Ends a quoted run.
+ *
+ * @param backslashEscapes whether a backslash escapes the character after it
+ * @param doubledClose whether the closing quote written twice stands for itself
+ * @return the index after the closing quote, or -1 when there is none
+ */
+ private static int endOfQuoted(String sql, int start, char close, boolean backslashEscapes, boolean doubledClose) {
+ int i = start + 1;
+ while (i < sql.length()) {
+ char c = sql.charAt(i);
+ if (backslashEscapes && c == '\\') {
+ i += 2;
+ }
+ else if (c == close) {
+ if (doubledClose && i + 1 < sql.length() && sql.charAt(i + 1) == close) {
+ i += 2;
+ }
+ else {
+ return i + 1;
+ }
+ }
+ else {
+ i++;
+ }
+ }
+ return -1;
+ }
+
+ /**
+ * Ends a GoogleSQL string, which a backslash escapes in and a quote written twice does not: that
+ * is an empty string, or the end of one string and the start of the next. A string that opens
+ * with its quote written three times is triple-quoted: it holds any quote, and ends at its quote
+ * written three times again.
+ *
+ * @return the index after the closing quote, or -1 when there is none
+ */
+ private static int endOfGoogleSqlString(String sql, int start) {
+ char quote = sql.charAt(start);
+ if (start + 2 < sql.length() && sql.charAt(start + 1) == quote && sql.charAt(start + 2) == quote) {
+ return endOfTripleQuoted(sql, start, quote);
+ }
+ return endOfQuoted(sql, start, quote, true, false);
+ }
+
+ private static int endOfTripleQuoted(String sql, int start, char quote) {
+ int i = start + 3;
+ while (i < sql.length()) {
+ char c = sql.charAt(i);
+ if (c == '\\') {
+ i += 2;
+ }
+ else if (c == quote && i + 2 < sql.length() && sql.charAt(i + 1) == quote && sql.charAt(i + 2) == quote) {
+ return i + 3;
+ }
+ else {
+ i++;
+ }
+ }
+ return -1;
+ }
+
+ private static int endOfLine(String sql, int start) {
+ int i = start;
+ while (i < sql.length() && sql.charAt(i) != '\n' && sql.charAt(i) != '\r') {
+ i++;
+ }
+ return i;
+ }
+
+ /**
+ * @return the length of the {@code $tag$} that opens a dollar-quoted string at the index, or 0
+ * when there is none: a dollar followed by a digit is a positional parameter
+ */
+ private static int dollarTagLength(String sql, int start) {
+ int i = start + 1;
+ while (i < sql.length()) {
+ char c = sql.charAt(i);
+ if (c == '$') {
+ return i + 1 - start;
+ }
+ boolean letter = Character.isLetter(c) || c == '_';
+ if (!letter && (i == start + 1 || !isDigit(c))) {
+ return 0;
+ }
+ i++;
+ }
+ return 0;
+ }
+
+ /**
+ * Ends a number: digits with a fraction and an exponent, or a hexadecimal number. Letters that
+ * follow it belong to it, so that nothing of a malformed number is left behind.
+ */
+ private static int endOfNumber(String sql, int start) {
+ int length = sql.length();
+ int i = start;
+ if (sql.charAt(i) == '0' && i + 1 < length && (sql.charAt(i + 1) == 'x' || sql.charAt(i + 1) == 'X')) {
+ i += 2;
+ }
+ else {
+ while (i < length && isDigit(sql.charAt(i))) {
+ i++;
+ }
+ if (i < length && sql.charAt(i) == '.') {
+ i++;
+ while (i < length && isDigit(sql.charAt(i))) {
+ i++;
+ }
+ }
+ if (i < length && (sql.charAt(i) == 'e' || sql.charAt(i) == 'E')) {
+ int exponent = i + 1 < length && (sql.charAt(i + 1) == '+' || sql.charAt(i + 1) == '-') ? i + 2 : i + 1;
+ if (exponent < length && isDigit(sql.charAt(exponent))) {
+ i = exponent;
+ }
+ }
+ }
+ while (i < length && isIdentifierPart(sql.charAt(i))) {
+ i++;
+ }
+ return i;
+ }
+
+ private static boolean isNationalPrefix(String sql, int index) {
+ return index >= 0 && (sql.charAt(index) == 'n' || sql.charAt(index) == 'N') &&
+ (index == 0 || !isIdentifierPart(sql.charAt(index - 1)));
+ }
+
+ private static String closingDelimiter(char open) {
+ return switch (open) {
+ case '[' -> "]";
+ case '(' -> ")";
+ case '{' -> "}";
+ case '<' -> ">";
+ default -> String.valueOf(open);
+ };
+ }
+
+ private static boolean isDigit(char c) {
+ return c >= '0' && c <= '9';
+ }
+
+ private static boolean isIdentifierPart(char c) {
Review Comment:
A `#` is treated as part of an identifier, so PostgreSQL `select account_id#987654321 from account` keeps the number. Transact-SQL `select $987654321` keeps the money literal. Both are values.
Treat `#` as an operator in PostgreSQL and `$` as a money prefix in Transact-SQL, or withhold the statement.
##########
grails-datamapping-core/src/cli/groovy/org/grails/datastore/gorm/jdbc/sqltail/cli/SqlTailFileReader.groovy:
##########
@@ -0,0 +1,188 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * https://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.grails.datastore.gorm.jdbc.sqltail.cli
+
+import java.nio.charset.StandardCharsets
+
+import groovy.transform.CompileStatic
+
+import org.grails.datastore.gorm.jdbc.sqltail.SqlTailRecord
+
+/**
+ * Follows a capture file the way {@code tail -f} does: each poll returns the records written
+ * since the last one. A line that has no terminator yet is left for the next poll, so a record
+ * the application is still writing is never read in half. An application that starts again
+ * truncates the file and names its new run in the first line; when that line names another run,
+ * or the file shrinks, reading starts over from the beginning. The run tells a restart apart from
+ * a file that grew past where the reader was before it was next polled. A file truncated between
+ * being measured and being read, so that the read comes up short, is a restart too: it is read again
+ * from the beginning in the same poll.
+ *
+ * @since 8.1
+ */
+@CompileStatic
+class SqlTailFileReader implements Closeable {
+
+ private static final byte NEWLINE = (byte) 10
+
+ /** How much of the file the first line is looked for in; a header is far shorter. */
+ private static final int HEADER_LIMIT = 256
+
+ final File file
+ private long position
+ private String run
+ private boolean restarted
+
+ /**
+ * @param file the capture file
+ * @param fromStart whether to read what the file already holds, or only what is written from now on
+ */
+ SqlTailFileReader(File file, boolean fromStart) {
+ this.file = file
+ this.position = fromStart ? 0 : file.length()
+ this.run = file.exists() ? readRun() : null
+ }
+
+ /**
+ * @return the records written since the last poll, or since the reader was opened
+ */
+ List<SqlTailRecord> poll() {
+ restarted = false
+ // a read that comes up short is retried once, from the start; the next poll takes it from there
+ for (int attempt = 0; attempt < 2; attempt++) {
+ long length = file.length()
+ String current = length > 0 ? readRun() : null
+ if (length < position || (current != null && run != null && current != run)) {
+ position = 0
+ restarted = true
+ }
+ if (current != null || restarted) {
+ // a file truncated before its header was written names its run on a later poll
+ run = current
+ }
+ if (length <= position) {
+ return []
+ }
+ byte[] bytes = readFrom(position, length)
+ if (bytes.length == length - position) {
+ return recordsIn(bytes)
+ }
+ // the application truncated the file after it was measured: it has started again
+ position = 0
+ restarted = true
+ run = null
+ }
+ []
+ }
+
+ /**
+ * @return whether the last poll found a new run in the file, which an application restart writes
+ */
+ boolean isRestarted() {
+ restarted
+ }
+
+ /**
+ * @return how far the file has been read
+ */
+ long getPosition() {
+ position
+ }
+
+ /**
+ * @return the run that wrote the file, as its first line names it; null before the line is complete
+ */
+ String getRun() {
+ run
+ }
+
+ @Override
+ void close() {
+ // the file is opened and closed per poll
+ }
+
+ /**
+ * Takes the complete lines out of what was read, and leaves a line still being written for the next poll.
+ */
+ private List<SqlTailRecord> recordsIn(byte[] bytes) {
+ int consumed = lastLineEnd(bytes)
+ if (consumed == 0) {
+ return []
+ }
+ position += consumed
+ String text = new String(bytes, 0, consumed, StandardCharsets.UTF_8)
+ List<SqlTailRecord> records = []
+ for (String line : text.split('\n')) {
+ SqlTailRecord record = SqlTailRecord.parse(line)
+ if (record != null) {
+ records.add(record)
+ }
+ }
+ records
+ }
+
+ private static int lastLineEnd(byte[] bytes) {
+ for (int i = bytes.length - 1; i >= 0; i--) {
+ if (bytes[i] == NEWLINE) {
+ return i + 1
+ }
+ }
+ 0
+ }
+
+ private String readRun() {
+ byte[] head = readFrom(0, Math.min(file.length(), (long) HEADER_LIMIT))
+ for (int i = 0; i < head.length; i++) {
+ if (head[i] == NEWLINE) {
+ return SqlTailRecord.parseHeader(new String(head, 0, i, StandardCharsets.UTF_8))
+ }
+ }
+ null
+ }
+
+ /**
+ * @return the bytes between the offsets, or fewer when the file ends before the second one or is
+ * gone: it was truncated, or removed, after it was measured
+ */
+ private byte[] readFrom(long from, long to) {
Review Comment:
`--all` and `--once` allocate one array for the whole unread range. A capture near 2 GiB makes that cast negative and throws `NegativeArraySizeException`. A smaller large file can exhaust the heap.
Read a bounded chunk and drain it before printing a one-shot report.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@grails.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add the sql-tail command to find query-heavy pages [grails-core]
Posted by "jdaugherty (via GitHub)" <gi...@apache.org>.
jdaugherty commented on PR #16631:
URL: https://github.com/apache/grails-core/pull/16631#issuecomment-6297093620
@borinquenkid You had introduced non-lazy gradle code and I've now fixed it. This needs back ported to 8.0.x though. I intend to cherry-pick those changes into 8.0.x in a separate PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@grails.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add the sql-tail command to find query-heavy pages [grails-core]
Posted by "testlens-app[bot] (via GitHub)" <gi...@apache.org>.
testlens-app[bot] commented on PR #16631:
URL: https://github.com/apache/grails-core/pull/16631#issuecomment-6370436124
## 🚨 All tests passed but jobs failed 🚨
### Failed Jobs without Test Failures
❌ [CI / Neo4j Functional Tests \(Java 21, indy\=true\)](https://github.com/apache/grails-core/actions/runs/39989858304/job/120541079525?pr=16631)
🏷️ Commit: 0dc78c283abfbbe7c7286b5a479098e47a75a02c
▶️ Tests: 6504 executed
🟡 Checks: 26/87 completed
### Rerun Controls
Click the checkbox to trigger a rerun:
🔲 **Rerun jobs**
---
_Learn more about TestLens at [testlens.app/docs](https://testlens.app/docs/features/pr-comment/)._
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@grails.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add the query-tail command to find query-heavy pages [grails-core]
Posted by "jdaugherty (via GitHub)" <gi...@apache.org>.
jdaugherty commented on PR #16631:
URL: https://github.com/apache/grails-core/pull/16631#issuecomment-6378267695
The failures on 83e2720c65 came from the branch's tests, not the feature, and are fixed in b849a58ddc, now rebased onto current 9.0.x:
- **Ten lanes died compiling the MongoDB core test sources out of heap.** One line in the new listener spec, a GORM `where { address.city == 'Paris' }` query on an embedded property of an entity declared in the same file, made that compile take 80 seconds longer and exhaust the forked compiler's 2 GB; without it the compile is back to its usual 7 seconds and fits in 600 MB on its own. The spec now runs the same query through the criteria API. The transformation's behaviour on that construct is worth its own issue; the reproducer is that one line.
- **The MongoDB listener spec assumed a server on port 27017.** It now extends `AutoStartedMongoSpec` like the other MongoDB specs here, so the test infrastructure's container serves it, which is why it failed in the Coverage and canary jobs and the MongoDB lanes that got past the compile.
- **The Windows shard failed the idle-poll reader spec** because `File.length()` on Windows asks the file for its path, which the spec's sealed file refused; it now refuses only an open.
- **Rebased onto 9.0.x**, which brings the branch its `ubuntu-22.04` pin for the MongoDB lanes whose kernel MongoDB 8.0 refuses, and the 9.0 matrix entry; the branch's own workflow still had the 24.04 image. Still one commit, no conflicts.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@grails.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add the query-tail command to find query-heavy pages [grails-core]
Posted by "testlens-app[bot] (via GitHub)" <gi...@apache.org>.
testlens-app[bot] commented on PR #16631:
URL: https://github.com/apache/grails-core/pull/16631#issuecomment-6382688720
## ✅ All tests passed ✅
🏷️ Commit: 54a6574fc5a2956f103afabd3b455190d8903b35
▶️ Tests: 138153 executed
⚪️ Checks: 97/97 completed
---
_Learn more about TestLens at [testlens.app/docs](https://testlens.app/docs/features/pr-comment/)._
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@grails.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add the query-tail command to find query-heavy pages [grails-core]
Posted by "testlens-app[bot] (via GitHub)" <gi...@apache.org>.
testlens-app[bot] commented on PR #16631:
URL: https://github.com/apache/grails-core/pull/16631#issuecomment-6375019045
## 🚨 TestLens detected 1 failed test 🚨
### Test Summary
> [!TIP]
> Inspect all details with the [TestLens CLI](https://testlens.app/docs/features/cli): `testlens pr 16631`
#### [CI / Build Grails-Core \(Windows JDK 25 shard 1\)](https://github.com/apache/grails-core/actions/runs/40023214024/job/120653485927?pr=16631) > :grails-datamapping-core:testCli
| Test | Runs | Flakiness |
|---|---|--:|
| QueryTailFileReaderSpec > a poll that finds the file as long as it left it does not open it | ❌ | 0% 🟢 |
🏷️ Commit: 83e2720c6533da2fc595b0113cd1438aa60f8897
▶️ Tests: 21364 executed
🟡 Checks: 36/88 completed
### Test Failures
<details><summary><strong>QueryTailFileReaderSpec > a poll that finds the file as long as it left it does not open it</strong> (:grails-datamapping-core:testCli in <a href="https://github.com/apache/grails-core/actions/runs/40023214024/job/120653485927?pr=16631">CI / Build Grails-Core (Windows JDK 25 shard 1)</a>)</summary>
```
Condition failed with Exception:
idle.isEmpty()
| |
null java.lang.NullPointerException: Cannot invoke method isEmpty() on null object
at org.apache.grails.data.core.cli.querytail.QueryTailFileReaderSpec.a poll that finds the file as long as it left it does not open it(QueryTailFileReaderSpec.groovy:295)
at org.apache.grails.data.core.cli.querytail.QueryTailFileReaderSpec.a poll that finds the file as long as it left it does not open it(QueryTailFileReaderSpec.groovy:295)
Caused by: java.lang.NullPointerException: Cannot invoke method isEmpty() on null object
... 1 more
```
</details>
### Rerun Controls
> [!NOTE]
> Checks are currently running using the configuration below.
Select tests to mute in this pull request:
🔲 QueryTailFileReaderSpec > a poll that finds the file as long as it left it does not open it <!-- uniqueId=[engine:spock]/[spec:org.apache.grails.data.core.cli.querytail.QueryTailFileReaderSpec]/[feature:$spock_feature_0_12] -->
Reuse successful test results:
🔲 ♻️ Only rerun the tests that failed or were muted before
Click the checkbox to trigger a rerun:
🔲 **Rerun jobs**
---
_Learn more about TestLens at [testlens.app/docs](https://testlens.app/docs/features/pr-comment/)._
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@grails.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add the sql-tail command to find query-heavy pages [grails-core]
Posted by "jdaugherty (via GitHub)" <gi...@apache.org>.
jdaugherty commented on PR #16631:
URL: https://github.com/apache/grails-core/pull/16631#issuecomment-6372943846
sql-tail now covers the other two stores GORM runs statements on, in 9f418b2417:
- **Neo4j.** A `SqlTailDriver` wraps the driver each connection source builds, so every session, transaction and transaction function records its Cypher before it runs; parameters are never recorded. Cypher gets a syntax of its own, `SqlTailSyntax.CYPHER`, verified against Neo4j 5 in a container and the embedded 3.5 the tests run on. The two versions differ on two points and the syntax takes neither side: whether a `//` comment ends at a carriage return is read both ways, and a backslash outside a string or comment, which Neo4j 5 decodes into a quote before parsing, withholds the statement.
- **MongoDB.** A `SqlTailCommandListener` on each client records a command as its name, namespace and entries with every value replaced, e.g. `find shop.book {"filter": {"title": ?}, "limit": ?}`. Since a command is a document, the only place data can hide is a key, so below the top level a key is kept only where the mapping knows it as a field, resolved through embedded entities; the keys of a map property, a dynamic attribute or an unmapped collection are recorded as `?` too. The connection source owns the capture, so a client stopped for a CRaC checkpoint keeps recording after the restore.
- **Command.** Cypher is typed by the clauses that write in it, Mongo commands by their name, and both are shown whole. Guide and reference updated.
Each store has an integration spec against a real server (embedded Neo4j, a mongod) covering values, multiple and runtime-added connection sources, and capture lifecycle, plus a table-driven spec for the Mongo redactor; the Cypher rows joined the redactor spec.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@grails.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add the query-tail command to find query-heavy pages [grails-core]
Posted by "testlens-app[bot] (via GitHub)" <gi...@apache.org>.
testlens-app[bot] commented on PR #16631:
URL: https://github.com/apache/grails-core/pull/16631#issuecomment-6386132637
## ✅ All tests passed ✅
🏷️ Commit: b3a97c25437e1590f5f898666365531a27f98e9c
▶️ Tests: 138157 executed
⚪️ Checks: 97/97 completed
---
_Learn more about TestLens at [testlens.app/docs](https://testlens.app/docs/features/pr-comment/)._
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@grails.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add the query-tail command to find query-heavy pages [grails-core]
Posted by "jdaugherty (via GitHub)" <gi...@apache.org>.
jdaugherty commented on code in PR #16631:
URL: https://github.com/apache/grails-core/pull/16631#discussion_r4375671891
##########
grails-datamapping-core/src/main/groovy/org/apache/grails/data/core/querytail/QueryTailCapture.groovy:
##########
@@ -0,0 +1,369 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * https://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.grails.data.core.querytail
+
+import java.nio.ByteBuffer
+import java.nio.channels.FileChannel
+import java.nio.channels.FileLock
+import java.nio.channels.OverlappingFileLockException
+import java.nio.charset.StandardCharsets
+import java.nio.file.Files
+import java.nio.file.StandardOpenOption
+import java.util.concurrent.atomic.AtomicBoolean
+import java.util.concurrent.atomic.AtomicLong
+
+import groovy.transform.CompileStatic
+import groovy.util.logging.Slf4j
+
+/**
+ * Writes the statements an application runs to the capture file the {@code query-tail} command
+ * tails. Each session factory, Neo4j driver or MongoDB client that captures opens its own capture
+ * and closes it when it closes; the captures of one file share it, so every data source of an
+ * application writes to the one file. The file is truncated when the first capture opens it, so it
+ * holds the statements of the current run only, and its first line names that run. It is closed
+ * when the last capture closes, and opened afresh by the next one. Each statement is written whole
+ * and at once, so that the command sees it as soon as it is recorded.
+ *
+ * <p>JVMs that run at the same time with capture enabled, such as the parallel forks of a test task,
+ * share the file: a lock file beside it tells the first to start, which truncates the file and names
+ * the run, from the others, which append to that run without truncating it and number their
+ * requests apart, so that the command reads one run from all of them.</p>
+ *
+ * <p>A statement is recorded without its values: {@link QueryTailRedactor} replaces its literals and
+ * drops its comments before it is written, in the {@link #syntax} of the database it runs on. A
+ * statement that is not text, such as a MongoDB command, has its values taken out by whoever records
+ * it, through {@link #recordRedacted}.</p>
+ *
+ * <p>A capture is only opened when {@link QueryTailSettings#SETTING_ENABLED} is set; an application
+ * started without it never reaches this class.</p>
+ *
+ * @since 8.1
+ */
+@CompileStatic
+@Slf4j
+class QueryTailCapture implements Closeable {
+
+ private static final Map<String, CaptureFile> OPEN = new HashMap<>()
+
+ /** The file the statements are written to. */
+ final File file
+
+ /** The syntax of the database the statements of this capture run on, which their values are found by. */
+ volatile QueryTailSyntax syntax = QueryTailSyntax.UNKNOWN
+
+ private final CaptureFile captureFile
+ private final AtomicBoolean closed = new AtomicBoolean()
+
+ private QueryTailCapture(File file, CaptureFile captureFile) {
+ this.file = file
+ this.captureFile = captureFile
+ }
+
+ /**
+ * Opens a capture of a file. The first capture of a file truncates it; a capture opened while
+ * another of the same file is open shares it. Every capture opened must be closed.
+ *
+ * @param file the capture file
+ * @return the capture
+ * @throws UncheckedIOException when the file cannot be created
+ */
+ static QueryTailCapture open(File file) {
+ String key
+ try {
+ key = file.canonicalPath
+ }
+ catch (IOException e) {
+ throw new UncheckedIOException("Cannot resolve the query-tail capture file ${file}: ${e.message}", e)
+ }
+ synchronized (OPEN) {
+ CaptureFile captureFile = OPEN.get(key)
+ if (captureFile == null) {
+ captureFile = new CaptureFile(file, key)
+ OPEN.put(key, captureFile)
+ if (captureFile.joined) {
+ log.info('Capturing statements to {} for query-tail, in the run of another JVM writing it', file)
+ }
+ else {
+ log.info('Capturing statements to {} for query-tail', file)
+ }
+ }
+ captureFile.captures++
+ return new QueryTailCapture(file, captureFile)
+ }
+ }
+
+ /**
+ * Opens the capture the settings name.
+ *
+ * @param settings the query-tail settings
+ * @return the capture
+ */
+ static QueryTailCapture open(QueryTailSettings settings) {
+ open(settings.resolveFile())
+ }
+
+ /**
+ * Records a statement that is about to run on the current thread, without its values.
+ *
+ * @param dataSource the connection source name the statement runs against
+ * @param sql the statement
+ */
+ void record(String dataSource, String sql) {
+ write(dataSource, QueryTailRedactor.redact(sql, syntax))
+ }
+
+ /**
+ * Records a statement that is about to run on the current thread and that holds no value any
+ * more: a MongoDB command, which is a document rather than text, is redacted field by field by
+ * the listener that records it. Nothing of the statement is changed here.
+ *
+ * @param dataSource the connection source name the statement runs against
+ * @param statement the statement, with its values already taken out
+ */
+ void recordRedacted(String dataSource, String statement) {
+ write(dataSource, statement)
+ }
+
+ private void write(String dataSource, String statement) {
+ if (closed.get()) {
+ return
+ }
+ QueryTailRecord record = new QueryTailRecord(timestamp: System.currentTimeMillis(), dataSource: dataSource,
+ sql: statement)
+ captureFile.write(captureFile.attribution.attribute(record))
+ }
+
+ /**
+ * @return whether this capture is still writing to its file
+ */
+ boolean isOpen() {
+ !closed.get() && captureFile.open
+ }
+
+ /**
+ * Closes this capture. The file is closed once every capture of it is.
+ */
+ @Override
+ void close() {
+ if (!closed.compareAndSet(false, true)) {
+ return
+ }
+ boolean last
+ synchronized (OPEN) {
+ last = --captureFile.captures == 0
+ if (last && OPEN.get(captureFile.key).is(captureFile)) {
Review Comment:
Fixed in 6306f94ef7. When the last capture of a file closes, it now releases the file and its lock inside the same `synchronized (OPEN)` block that removes the entry. A capture opened on the same path waits until the old lock is gone, then starts or joins a run with a writer lock of its own.
The new feature "reopening waits for the last owner to finish its write and release its coordination lock" in `QueryTailCaptureSpec` covers this:
- It holds the old writer in the middle of a write to a FIFO.
- It closes the last capture and opens the same path while that write is still pending.
- It checks that the open blocks rather than joining the closing writer.
- Once the write drains, it checks that the replacement holds the writer lock itself.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@grails.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add the sql-tail command to find query-heavy pages [grails-core]
Posted by "testlens-app[bot] (via GitHub)" <gi...@apache.org>.
testlens-app[bot] commented on PR #16631:
URL: https://github.com/apache/grails-core/pull/16631#issuecomment-6142448077
## 🚨 TestLens detected 1 failed test 🚨
### Test Summary
> [!TIP]
> Inspect all details with the [TestLens CLI](https://testlens.app/docs/features/cli): `testlens pr 16631`
#### [CI / Build Grails-Core \(Windows JDK 25 shard 1\)](https://github.com/apache/grails-core/actions/runs/38374015154/job/115217730995?pr=16631) > :grails-datamapping-core:test
| Test | Runs | Flakiness |
|---|---|--:|
| SqlTailSettingsSpec > a relative build directory is joined to the application directory when that is known | ❌ | 0% 🟢 |
🏷️ Commit: 918443372ec673d449e0cd9c99c969f559109fde
▶️ Tests: 6629 executed
🟡 Checks: 27/79 completed
### Test Failures
<details><summary><strong>SqlTailSettingsSpec > a relative build directory is joined to the application directory when that is known</strong> (:grails-datamapping-core:test in <a href="https://github.com/apache/grails-core/actions/runs/38374015154/job/115217730995?pr=16631">CI / Build Grails-Core (Windows JDK 25 shard 1)</a>)</summary>
```
Condition not satisfied:
SqlTailSettings.defaultFile() == new File('/var/build', 'sql-tail/capture.log')
| | | |
| | | \var\build\sql-tail\capture.log
| | false
| \apps\library\var\build\sql-tail\capture.log
class org.grails.datastore.gorm.jdbc.sqltail.SqlTailSettings
at org.grails.datastore.gorm.jdbc.sqltail.SqlTailSettingsSpec.a relative build directory is joined to the application directory when that is known(SqlTailSettingsSpec.groovy:92)
```
|expected|actual|
|---|---|
|Failed to render value due to:|Failed to render value due to:|
|||
|java.lang.reflect.InaccessibleObjectException: Unable to make field private static final java.io.FileSystem java.io.File.FS accessible: module java.base does not \"opens java.io\" to unnamed module <span>@</span>45ff54e6|java.lang.reflect.InaccessibleObjectException: Unable to make field private static final java.io.FileSystem java.io.File.FS accessible: module java.base does not \"opens java.io\" to unnamed module <span>@</span>45ff54e6|
| at java.base/java.lang.reflect.AccessibleObject.throwInaccessibleObjectException\(AccessibleObject.java:353\)| at java.base/java.lang.reflect.AccessibleObject.throwInaccessibleObjectException\(AccessibleObject.java:353\)|
| at java.base/java.lang.reflect.AccessibleObject.checkCanSetAccessible\(AccessibleObject.java:329\)| at java.base/java.lang.reflect.AccessibleObject.checkCanSetAccessible\(AccessibleObject.java:329\)|
| at java.base/java.lang.reflect.AccessibleObject.checkCanSetAccessible\(AccessibleObject.java:277\)| at java.base/java.lang.reflect.AccessibleObject.checkCanSetAccessible\(AccessibleObject.java:277\)|
| at java.base/java.lang.reflect.Field.checkCanSetAccessible\(Field.java:179\)| at java.base/java.lang.reflect.Field.checkCanSetAccessible\(Field.java:179\)|
| at java.base/java.lang.reflect.Field.setAccessible\(Field.java:173\)| at java.base/java.lang.reflect.Field.setAccessible\(Field.java:173\)|
| at org.spockframework.runtime.condition.DiffedObjectAsBeanRenderer.render\(DiffedObjectAsBeanRenderer.java:31\)| at org.spockframework.runtime.condition.DiffedObjectAsBeanRenderer.render\(DiffedObjectAsBeanRenderer.java:31\)|
| at org.spockframework.runtime.condition.ObjectRendererService.render\(ObjectRendererService.java:50\)| at org.spockframework.runtime.condition.ObjectRendererService.render\(ObjectRendererService.java:50\)|
| at org.spockframework.runtime.MasterRunSupervisor.renderValue\(MasterRunSupervisor.java:109\)| at org.spockframework.runtime.MasterRunSupervisor.renderValue\(MasterRunSupervisor.java:109\)|
| at org.spockframework.runtime.MasterRunSupervisor.convertToComparisonFailure\(MasterRunSupervisor.java:<s>95</s>\)| at org.spockframework.runtime.MasterRunSupervisor.convertToComparisonFailure\(MasterRunSupervisor.java:<b>92</b>\)|
| at org.spockframework.runtime.MasterRunSupervisor.transform\(MasterRunSupervisor.java:67\)| at org.spockframework.runtime.MasterRunSupervisor.transform\(MasterRunSupervisor.java:67\)|
| at org.spockframework.runtime.MasterRunSupervisor.error\(MasterRunSupervisor.java:46\)| at org.spockframework.runtime.MasterRunSupervisor.error\(MasterRunSupervisor.java:46\)|
| at org.spockframework.runtime.PlatformSpecRunner.invokeRaw\(PlatformSpecRunner.java:444\)| at org.spockframework.runtime.PlatformSpecRunner.invokeRaw\(PlatformSpecRunner.java:444\)|
| at org.spockframework.runtime.PlatformSpecRunner.invoke\(PlatformSpecRunner.java:425\)| at org.spockframework.runtime.PlatformSpecRunner.invoke\(PlatformSpecRunner.java:425\)|
| at org.spockframework.runtime.PlatformSpecRunner.runFeatureMethod\(PlatformSpecRunner.java:336\)| at org.spockframework.runtime.PlatformSpecRunner.runFeatureMethod\(PlatformSpecRunner.java:336\)|
| at org.spockframework.runtime.IterationNode.execute\(IterationNode.java:51\)| at org.spockframework.runtime.IterationNode.execute\(IterationNode.java:51\)|
| at org.spockframework.runtime.IterationNode.execute\(IterationNode.java:13\)| at org.spockframework.runtime.IterationNode.execute\(IterationNode.java:13\)|
| at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda\$executeRecursively\$2\(NodeTestTask.java:176\)| at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda\$executeRecursively\$2\(NodeTestTask.java:176\)|
| at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute\(ThrowableCollector.java:74\)| at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute\(ThrowableCollector.java:74\)|
| at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda\$executeRecursively\$1\(NodeTestTask.java:166\)| at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda\$executeRecursively\$1\(NodeTestTask.java:166\)|
| at org.spockframework.runtime.SpockNode.sneakyInvoke\(SpockNode.java:40\)| at org.spockframework.runtime.SpockNode.sneakyInvoke\(SpockNode.java:40\)|
| at org.spockframework.runtime.IterationNode.lambda\$around\$0\(IterationNode.java:68\)| at org.spockframework.runtime.IterationNode.lambda\$around\$0\(IterationNode.java:68\)|
| at org.spockframework.runtime.PlatformSpecRunner.lambda\$createMethodInfoForDoRunIteration\$5\(PlatformSpecRunner.java:244\)| at org.spockframework.runtime.PlatformSpecRunner.lambda\$createMethodInfoForDoRunIteration\$5\(PlatformSpecRunner.java:244\)|
| at org.spockframework.runtime.model.MethodInfo.invoke\(MethodInfo.java:157\)| at org.spockframework.runtime.model.MethodInfo.invoke\(MethodInfo.java:157\)|
| at org.spockframework.runtime.extension.MethodInvocation.proceed\(MethodInvocation.java:141\)| at org.spockframework.runtime.extension.MethodInvocation.proceed\(MethodInvocation.java:141\)|
| at org.spockframework.runtime.extension.builtin.RestoreSystemPropertiesInterceptor.intercept\(RestoreSystemPropertiesInterceptor.java:34\)| at org.spockframework.runtime.extension.builtin.RestoreSystemPropertiesInterceptor.intercept\(RestoreSystemPropertiesInterceptor.java:34\)|
| at org.spockframework.runtime.extension.MethodInvocation.proceed\(MethodInvocation.java:139\)| at org.spockframework.runtime.extension.MethodInvocation.proceed\(MethodInvocation.java:139\)|
| at org.spockframework.runtime.PlatformSpecRunner.invoke\(PlatformSpecRunner.java:433\)| at org.spockframework.runtime.PlatformSpecRunner.invoke\(PlatformSpecRunner.java:433\)|
| at org.spockframework.runtime.PlatformSpecRunner.runIteration\(PlatformSpecRunner.java:226\)| at org.spockframework.runtime.PlatformSpecRunner.runIteration\(PlatformSpecRunner.java:226\)|
| at org.spockframework.runtime.IterationNode.around\(IterationNode.java:68\)| at org.spockframework.runtime.IterationNode.around\(IterationNode.java:68\)|
| at org.spockframework.runtime.IterationNode.around\(IterationNode.java:13\)| at org.spockframework.runtime.IterationNode.around\(IterationNode.java:13\)|
| at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda\$executeRecursively\$0\(NodeTestTask.java:164\)| at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda\$executeRecursively\$0\(NodeTestTask.java:164\)|
| at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute\(ThrowableCollector.java:74\)| at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute\(ThrowableCollector.java:74\)|
| at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively\(NodeTestTask.java:163\)| at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively\(NodeTestTask.java:163\)|
| at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute\(NodeTestTask.java:116\)| at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute\(NodeTestTask.java:116\)|
| at org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.submit\(SameThreadHierarchicalTestExecutorService.java:36\)| at org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.submit\(SameThreadHierarchicalTestExecutorService.java:36\)|
| at org.junit.platform.engine.support.hierarchical.NodeTestTask\$DefaultDynamicTestExecutor.execute\(NodeTestTask.java:268\)| at org.junit.platform.engine.support.hierarchical.NodeTestTask\$DefaultDynamicTestExecutor.execute\(NodeTestTask.java:268\)|
| at org.spockframework.runtime.SimpleFeatureNode.execute\(SimpleFeatureNode.java:58\)| at org.spockframework.runtime.SimpleFeatureNode.execute\(SimpleFeatureNode.java:58\)|
| at org.spockframework.runtime.SimpleFeatureNode.execute\(SimpleFeatureNode.java:19\)| at org.spockframework.runtime.SimpleFeatureNode.execute\(SimpleFeatureNode.java:19\)|
| at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda\$executeRecursively\$2\(NodeTestTask.java:176\)| at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda\$executeRecursively\$2\(NodeTestTask.java:176\)|
| at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute\(ThrowableCollector.java:74\)| at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute\(ThrowableCollector.java:74\)|
| at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda\$executeRecursively\$1\(NodeTestTask.java:166\)| at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda\$executeRecursively\$1\(NodeTestTask.java:166\)|
| at org.spockframework.runtime.SpockNode.sneakyInvoke\(SpockNode.java:40\)| at org.spockframework.runtime.SpockNode.sneakyInvoke\(SpockNode.java:40\)|
| at org.spockframework.runtime.FeatureNode.lambda\$around\$0\(FeatureNode.java:73\)| at org.spockframework.runtime.FeatureNode.lambda\$around\$0\(FeatureNode.java:73\)|
| at org.spockframework.runtime.PlatformSpecRunner.lambda\$createMethodInfoForDoRunFeature\$4\(PlatformSpecRunner.java:208\)| at org.spockframework.runtime.PlatformSpecRunner.lambda\$createMethodInfoForDoRunFeature\$4\(PlatformSpecRunner.java:208\)|
| at org.spockframework.runtime.model.MethodInfo.invoke\(MethodInfo.java:157\)| at org.spockframework.runtime.model.MethodInfo.invoke\(MethodInfo.java:157\)|
| at org.spockframework.runtime.PlatformSpecRunner.invokeRaw\(PlatformSpecRunner.java:442\)| at org.spockframework.runtime.PlatformSpecRunner.invokeRaw\(PlatformSpecRunner.java:442\)|
| at org.spockframework.runtime.PlatformSpecRunner.invoke\(PlatformSpecRunner.java:425\)| at org.spockframework.runtime.PlatformSpecRunner.invoke\(PlatformSpecRunner.java:425\)|
| at org.spockframework.runtime.PlatformSpecRunner.runFeature\(PlatformSpecRunner.java:199\)| at org.spockframework.runtime.PlatformSpecRunner.runFeature\(PlatformSpecRunner.java:199\)|
| at org.spockframework.runtime.FeatureNode.around\(FeatureNode.java:73\)| at org.spockframework.runtime.FeatureNode.around\(FeatureNode.java:73\)|
| at org.spockframework.runtime.FeatureNode.around\(FeatureNode.java:30\)| at org.spockframework.runtime.FeatureNode.around\(FeatureNode.java:30\)|
| at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda\$executeRecursively\$0\(NodeTestTask.java:164\)| at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda\$executeRecursively\$0\(NodeTestTask.java:164\)|
| at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute\(ThrowableCollector.java:74\)| at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute\(ThrowableCollector.java:74\)|
| at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively\(NodeTestTask.java:163\)| at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively\(NodeTestTask.java:163\)|
| at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute\(NodeTestTask.java:116\)| at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute\(NodeTestTask.java:116\)|
| at java.base/java.util.ArrayList.forEach\(ArrayList.java:1604\)| at java.base/java.util.ArrayList.forEach\(ArrayList.java:1604\)|
| at org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.invokeAll\(SameThreadHierarchicalTestExecutorService.java:42\)| at org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.invokeAll\(SameThreadHierarchicalTestExecutorService.java:42\)|
| at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda\$executeRecursively\$2\(NodeTestTask.java:180\)| at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda\$executeRecursively\$2\(NodeTestTask.java:180\)|
| at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute\(ThrowableCollector.java:74\)| at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute\(ThrowableCollector.java:74\)|
| at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda\$executeRecursively\$1\(NodeTestTask.java:166\)| at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda\$executeRecursively\$1\(NodeTestTask.java:166\)|
| at org.spockframework.runtime.SpockNode.sneakyInvoke\(SpockNode.java:40\)| at org.spockframework.runtime.SpockNode.sneakyInvoke\(SpockNode.java:40\)|
| at org.spockframework.runtime.SpecNode.lambda\$around\$0\(SpecNode.java:72\)| at org.spockframework.runtime.SpecNode.lambda\$around\$0\(SpecNode.java:72\)|
| at org.spockframework.runtime.PlatformSpecRunner.lambda\$createMethodInfoForDoRunSpec\$0\(PlatformSpecRunner.java:66\)| at org.spockframework.runtime.PlatformSpecRunner.lambda\$createMethodInfoForDoRunSpec\$0\(PlatformSpecRunner.java:66\)|
| at org.spockframework.runtime.model.MethodInfo.invoke\(MethodInfo.java:157\)| at org.spockframework.runtime.model.MethodInfo.invoke\(MethodInfo.java:157\)|
| at org.spockframework.runtime.PlatformSpecRunner.invokeRaw\(PlatformSpecRunner.java:442\)| at org.spockframework.runtime.PlatformSpecRunner.invokeRaw\(PlatformSpecRunner.java:442\)|
| at org.spockframework.runtime.PlatformSpecRunner.invoke\(PlatformSpecRunner.java:425\)| at org.spockframework.runtime.PlatformSpecRunner.invoke\(PlatformSpecRunner.java:425\)|
| at org.spockframework.runtime.PlatformSpecRunner.runSpec\(PlatformSpecRunner.java:59\)| at org.spockframework.runtime.PlatformSpecRunner.runSpec\(PlatformSpecRunner.java:59\)|
| at org.spockframework.runtime.SpecNode.around\(SpecNode.java:72\)| at org.spockframework.runtime.SpecNode.around\(SpecNode.java:72\)|
| at org.spockframework.runtime.SpecNode.around\(SpecNode.java:12\)| at org.spockframework.runtime.SpecNode.around\(SpecNode.java:12\)|
| at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda\$executeRecursively\$0\(NodeTestTask.java:164\)| at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda\$executeRecursively\$0\(NodeTestTask.java:164\)|
| at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute\(ThrowableCollector.java:74\)| at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute\(ThrowableCollector.java:74\)|
| at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively\(NodeTestTask.java:163\)| at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively\(NodeTestTask.java:163\)|
| at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute\(NodeTestTask.java:116\)| at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute\(NodeTestTask.java:116\)|
| at java.base/java.util.ArrayList.forEach\(ArrayList.java:1604\)| at java.base/java.util.ArrayList.forEach\(ArrayList.java:1604\)|
| at org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.invokeAll\(SameThreadHierarchicalTestExecutorService.java:42\)| at org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.invokeAll\(SameThreadHierarchicalTestExecutorService.java:42\)|
| at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda\$executeRecursively\$2\(NodeTestTask.java:180\)| at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda\$executeRecursively\$2\(NodeTestTask.java:180\)|
| at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute\(ThrowableCollector.java:74\)| at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute\(ThrowableCollector.java:74\)|
| at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda\$executeRecursively\$1\(NodeTestTask.java:166\)| at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda\$executeRecursively\$1\(NodeTestTask.java:166\)|
| at org.junit.platform.engine.support.hierarchical.Node.around\(Node.java:138\)| at org.junit.platform.engine.support.hierarchical.Node.around\(Node.java:138\)|
| at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda\$executeRecursively\$0\(NodeTestTask.java:164\)| at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda\$executeRecursively\$0\(NodeTestTask.java:164\)|
| at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute\(ThrowableCollector.java:74\)| at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute\(ThrowableCollector.java:74\)|
| at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively\(NodeTestTask.java:163\)| at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively\(NodeTestTask.java:163\)|
| at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute\(NodeTestTask.java:116\)| at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute\(NodeTestTask.java:116\)|
| at org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.submit\(SameThreadHierarchicalTestExecutorService.java:36\)| at org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.submit\(SameThreadHierarchicalTestExecutorService.java:36\)|
| at org.junit.platform.engine.support.hierarchical.HierarchicalTestExecutor.execute\(HierarchicalTestExecutor.java:52\)| at org.junit.platform.engine.support.hierarchical.HierarchicalTestExecutor.execute\(HierarchicalTestExecutor.java:52\)|
| at org.junit.platform.engine.support.hierarchical.HierarchicalTestEngine.execute\(HierarchicalTestEngine.java:58\)| at org.junit.platform.engine.support.hierarchical.HierarchicalTestEngine.execute\(HierarchicalTestEngine.java:58\)|
| at org.junit.platform.launcher.core.EngineExecutionOrchestrator.executeEngine\(EngineExecutionOrchestrator.java:246\)| at org.junit.platform.launcher.core.EngineExecutionOrchestrator.executeEngine\(EngineExecutionOrchestrator.java:246\)|
| at org.junit.platform.launcher.core.EngineExecutionOrchestrator.failOrExecuteEngine\(EngineExecutionOrchestrator.java:218\)| at org.junit.platform.launcher.core.EngineExecutionOrchestrator.failOrExecuteEngine\(EngineExecutionOrchestrator.java:218\)|
| at org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute\(EngineExecutionOrchestrator.java:179\)| at org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute\(EngineExecutionOrchestrator.java:179\)|
| at org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute\(EngineExecutionOrchestrator.java:108\)| at org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute\(EngineExecutionOrchestrator.java:108\)|
| at org.junit.platform.launcher.core.EngineExecutionOrchestrator.lambda\$execute\$0\(EngineExecutionOrchestrator.java:66\)| at org.junit.platform.launcher.core.EngineExecutionOrchestrator.lambda\$execute\$0\(EngineExecutionOrchestrator.java:66\)|
| at org.junit.platform.launcher.core.EngineExecutionOrchestrator.withInterceptedStreams\(EngineExecutionOrchestrator.java:157\)| at org.junit.platform.launcher.core.EngineExecutionOrchestrator.withInterceptedStreams\(EngineExecutionOrchestrator.java:157\)|
| at org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute\(EngineExecutionOrchestrator.java:65\)| at org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute\(EngineExecutionOrchestrator.java:65\)|
| at org.junit.platform.launcher.core.DefaultLauncher.execute\(DefaultLauncher.java:125\)| at org.junit.platform.launcher.core.DefaultLauncher.execute\(DefaultLauncher.java:125\)|
| at org.junit.platform.launcher.core.DefaultLauncher.execute\(DefaultLauncher.java:114\)| at org.junit.platform.launcher.core.DefaultLauncher.execute\(DefaultLauncher.java:114\)|
| at org.junit.platform.launcher.core.DefaultLauncher.execute\(DefaultLauncher.java:93\)| at org.junit.platform.launcher.core.DefaultLauncher.execute\(DefaultLauncher.java:93\)|
| at org.junit.platform.launcher.core.DelegatingLauncher.execute\(DelegatingLauncher.java:48\)| at org.junit.platform.launcher.core.DelegatingLauncher.execute\(DelegatingLauncher.java:48\)|
| at org.junit.platform.launcher.core.InterceptingLauncher.lambda\$execute\$0\(InterceptingLauncher.java:41\)| at org.junit.platform.launcher.core.InterceptingLauncher.lambda\$execute\$0\(InterceptingLauncher.java:41\)|
| at org.junit.platform.launcher.core.ClasspathAlignmentCheckingLauncherInterceptor.intercept\(ClasspathAlignmentCheckingLauncherInterceptor.java:25\)| at org.junit.platform.launcher.core.ClasspathAlignmentCheckingLauncherInterceptor.intercept\(ClasspathAlignmentCheckingLauncherInterceptor.java:25\)|
| at org.junit.platform.launcher.core.InterceptingLauncher.execute\(InterceptingLauncher.java:40\)| at org.junit.platform.launcher.core.InterceptingLauncher.execute\(InterceptingLauncher.java:40\)|
| at org.junit.platform.launcher.core.DelegatingLauncher.execute\(DelegatingLauncher.java:48\)| at org.junit.platform.launcher.core.DelegatingLauncher.execute\(DelegatingLauncher.java:48\)|
| at org.gradle.api.internal.tasks.testing.junitplatform.JUnitPlatformTestDefinitionProcessor\$CollectThenExecuteTestDefinitionConsumer.processAllTestDefinitions\(JUnitPlatformTestDefinitionProcessor.java:206\)| at org.gradle.api.internal.tasks.testing.junitplatform.JUnitPlatformTestDefinitionProcessor\$CollectThenExecuteTestDefinitionConsumer.processAllTestDefinitions\(JUnitPlatformTestDefinitionProcessor.java:206\)|
| at org.gradle.api.internal.tasks.testing.junitplatform.JUnitPlatformTestDefinitionProcessor\$CollectThenExecuteTestDefinitionConsumer.access\$000\(JUnitPlatformTestDefinitionProcessor.java:123\)| at org.gradle.api.internal.tasks.testing.junitplatform.JUnitPlatformTestDefinitionProcessor\$CollectThenExecuteTestDefinitionConsumer.access\$000\(JUnitPlatformTestDefinitionProcessor.java:123\)|
| at org.gradle.api.internal.tasks.testing.junitplatform.JUnitPlatformTestDefinitionProcessor.stop\(JUnitPlatformTestDefinitionProcessor.java:117\)| at org.gradle.api.internal.tasks.testing.junitplatform.JUnitPlatformTestDefinitionProcessor.stop\(JUnitPlatformTestDefinitionProcessor.java:117\)|
| at org.gradle.api.internal.tasks.testing.SuiteTestDefinitionProcessor.stop\(SuiteTestDefinitionProcessor.java:63\)| at org.gradle.api.internal.tasks.testing.SuiteTestDefinitionProcessor.stop\(SuiteTestDefinitionProcessor.java:63\)|
| at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke\(DirectMethodHandleAccessor.java:104\)| at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke\(DirectMethodHandleAccessor.java:104\)|
| at java.base/java.lang.reflect.Method.invoke\(Method.java:565\)| at java.base/java.lang.reflect.Method.invoke\(Method.java:565\)|
| at org.gradle.internal.dispatch.MethodInvocation.invokeOn\(MethodInvocation.java:77\)| at org.gradle.internal.dispatch.MethodInvocation.invokeOn\(MethodInvocation.java:77\)|
| at org.gradle.internal.dispatch.ReflectionDispatch.dispatch\(ReflectionDispatch.java:28\)| at org.gradle.internal.dispatch.ReflectionDispatch.dispatch\(ReflectionDispatch.java:28\)|
| at org.gradle.internal.dispatch.ReflectionDispatch.dispatch\(ReflectionDispatch.java:19\)| at org.gradle.internal.dispatch.ReflectionDispatch.dispatch\(ReflectionDispatch.java:19\)|
| at org.gradle.internal.dispatch.ContextClassLoaderDispatch.dispatch\(ContextClassLoaderDispatch.java:33\)| at org.gradle.internal.dispatch.ContextClassLoaderDispatch.dispatch\(ContextClassLoaderDispatch.java:33\)|
| at org.gradle.internal.dispatch.ProxyDispatchAdapter\$DispatchingInvocationHandler.invoke\(ProxyDispatchAdapter.java:88\)| at org.gradle.internal.dispatch.ProxyDispatchAdapter\$DispatchingInvocationHandler.invoke\(ProxyDispatchAdapter.java:88\)|
| at jdk.proxy2/jdk.proxy2.\$Proxy6.stop\(Unknown Source\)| at jdk.proxy2/jdk.proxy2.\$Proxy6.stop\(Unknown Source\)|
| at org.gradle.api.internal.tasks.testing.worker.TestWorker\$3.run\(TestWorker.java:195\)| at org.gradle.api.internal.tasks.testing.worker.TestWorker\$3.run\(TestWorker.java:195\)|
| at org.gradle.api.internal.tasks.testing.worker.TestWorker.executeAndMaintainThreadName\(TestWorker.java:126\)| at org.gradle.api.internal.tasks.testing.worker.TestWorker.executeAndMaintainThreadName\(TestWorker.java:126\)|
| at org.gradle.api.internal.tasks.testing.worker.TestWorker.execute\(TestWorker.java:103\)| at org.gradle.api.internal.tasks.testing.worker.TestWorker.execute\(TestWorker.java:103\)|
| at org.gradle.api.internal.tasks.testing.worker.TestWorker.execute\(TestWorker.java:63\)| at org.gradle.api.internal.tasks.testing.worker.TestWorker.execute\(TestWorker.java:63\)|
| at org.gradle.process.internal.worker.child.ActionExecutionWorker.execute\(ActionExecutionWorker.java:56\)| at org.gradle.process.internal.worker.child.ActionExecutionWorker.execute\(ActionExecutionWorker.java:56\)|
| at org.gradle.process.internal.worker.child.SystemApplicationClassLoaderWorker.call\(SystemApplicationClassLoaderWorker.java:122\)| at org.gradle.process.internal.worker.child.SystemApplicationClassLoaderWorker.call\(SystemApplicationClassLoaderWorker.java:122\)|
| at org.gradle.process.internal.worker.child.SystemApplicationClassLoaderWorker.call\(SystemApplicationClassLoaderWorker.java:72\)| at org.gradle.process.internal.worker.child.SystemApplicationClassLoaderWorker.call\(SystemApplicationClassLoaderWorker.java:72\)|
| at worker.org.gradle.process.internal.worker.GradleWorkerMain.run\(GradleWorkerMain.java:69\)| at worker.org.gradle.process.internal.worker.GradleWorkerMain.run\(GradleWorkerMain.java:69\)|
| at worker.org.gradle.process.internal.worker.GradleWorkerMain.main\(GradleWorkerMain.java:74\)| at worker.org.gradle.process.internal.worker.GradleWorkerMain.main\(GradleWorkerMain.java:74\)|
</details>
### Rerun Controls
> [!NOTE]
> Checks are currently running using the configuration below.
Select tests to mute in this pull request:
🔲 SqlTailSettingsSpec > a relative build directory is joined to the application directory when that is known <!-- uniqueId=[engine:spock]/[spec:org.grails.datastore.gorm.jdbc.sqltail.SqlTailSettingsSpec]/[feature:$spock_feature_0_5] -->
Reuse successful test results:
🔲 ♻️ Only rerun the tests that failed or were muted before
Click the checkbox to trigger a rerun:
🔲 **Rerun jobs**
---
_Learn more about TestLens at [testlens.app/docs](https://testlens.app/docs/features/pr-comment/)._
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@grails.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add the query-tail command to find query-heavy pages [grails-core]
Posted by "jdaugherty (via GitHub)" <gi...@apache.org>.
jdaugherty commented on PR #16631:
URL: https://github.com/apache/grails-core/pull/16631#issuecomment-6387358129
@jamesfredley re review 5639132816: both items are fixed on head b0e9a8c9a4. The redactor scope now advances into array elements (and into the document of an operator on an array), and a numeric key is kept only where the mapping knows an array at that depth, so a numeric key in a `List<Map>` element, in `attributes.0.<key>` or under `$elemMatch` records `?`; asserted in `MongoCommandRedactorSpec` and through the capture API against a real mongod in `QueryTailCommandListenerSpec`. A request line written before the URL mapping has run records only the shape of the path (`GET /?/?/?`), since nothing tells a literal segment from a token yet; later statements of the same request get the mapped line. Details are in the replies on your two threads, left open for you.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@grails.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add the query-tail command to find query-heavy pages [grails-core]
Posted by "jdaugherty (via GitHub)" <gi...@apache.org>.
jdaugherty commented on code in PR #16631:
URL: https://github.com/apache/grails-core/pull/16631#discussion_r4368487120
##########
grails-datamapping-core/src/main/groovy/org/apache/grails/data/core/querytail/QueryTailRequestAttribution.groovy:
##########
@@ -0,0 +1,385 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * https://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.grails.data.core.querytail
+
+import java.lang.reflect.Method
+import java.nio.charset.StandardCharsets
+import java.util.concurrent.atomic.AtomicLong
+import java.util.regex.Pattern
+
+import groovy.transform.CompileStatic
+
+import org.springframework.util.ClassUtils
+import org.springframework.web.context.request.RequestAttributes
+import org.springframework.web.context.request.RequestContextHolder
+import org.springframework.web.context.request.ServletRequestAttributes
+
+/**
+ * Attributes a captured statement to the web request on the current thread. Grails keeps the
+ * request, and the controller and action it resolved, on the thread that serves it, so a statement
+ * run by an interceptor, the action or the view all land on the same page. The first statement of
+ * a request gives it an id that every later statement of that request repeats, which is how the
+ * {@code query-tail} command counts statements per request. Work on any other thread is attributed to
+ * the thread by name.
+ *
+ * <p>The request is recorded as its method and path, with the values in the path replaced by
+ * {@code ?}: the values of the parameters the URL mapping took from it and those of its query, found
+ * whether or not the client percent-encoded them, and any segment that is a number, a UUID or a long
+ * hexadecimal token, so that an id, a token or a key carried in a path never reaches the capture
+ * file. Until the URL mapping has run, as for a statement of a filter that runs before the controller
+ * is chosen, or for a request that never reaches one, no segment's role is known and every segment
+ * is replaced.</p>
+ *
+ * <p>Spring Web is optional: without it on the classpath every statement is attributed to its
+ * thread.</p>
+ *
+ * @since 8.1
+ */
+@CompileStatic
+class QueryTailRequestAttribution {
+
+ /** The request attribute holding the id this attribution gave a request. */
+ public static final String REQUEST_ID_ATTRIBUTE = 'org.apache.grails.data.core.querytail.REQUEST_ID'
+
+ /** The request attribute Grails sets the controller name in. */
+ public static final String CONTROLLER_NAME_ATTRIBUTE = 'org.grails.CONTROLLER_NAME_ATTRIBUTE'
+
+ /** The request attribute Grails sets the controller namespace in. */
+ public static final String CONTROLLER_NAMESPACE_ATTRIBUTE = 'org.grails.CONTROLLER_NAMESPACE_ATTRIBUTE'
+
+ /** The request attribute Grails sets the action name in. */
+ public static final String ACTION_NAME_ATTRIBUTE = 'org.grails.ACTION_NAME_ATTRIBUTE'
+
+ private static final boolean SPRING_WEB_PRESENT = ClassUtils.isPresent(
+ 'org.springframework.web.context.request.RequestContextHolder', QueryTailRequestAttribution.classLoader)
+
+ private final AtomicLong requestCounter = new AtomicLong()
+ private final String requestIdPrefix
+
+ /**
+ * An attribution whose request ids are {@code r1}, {@code r2} and so on.
+ */
+ QueryTailRequestAttribution() {
+ this('')
+ }
+
+ /**
+ * @param requestIdPrefix what every request id starts with, which tells the requests of one JVM
+ * from those of another writing the same capture
+ */
+ QueryTailRequestAttribution(String requestIdPrefix) {
+ this.requestIdPrefix = requestIdPrefix ?: ''
+ }
+
+ /**
+ * Fills in the request id, page and request of a record from the current thread.
+ *
+ * @param record the record of a statement that just ran on this thread
+ * @return the same record
+ */
+ QueryTailRecord attribute(QueryTailRecord record) {
+ if (SPRING_WEB_PRESENT && WebRequestLookup.attribute(record, requestCounter, requestIdPrefix)) {
+ return record
+ }
+ record.requestId = null
+ record.request = null
+ record.page = QueryTailRecord.THREAD_PAGE_PREFIX + Thread.currentThread().name
+ record
+ }
+
+ /**
+ * Keeps every reference to Spring Web, in a signature as well as in a body, in a class that is only
+ * loaded once Spring Web is known to be present. The outer class can then be loaded, and reflected
+ * on, without Spring Web on the classpath.
+ */
+ @CompileStatic
+ private static class WebRequestLookup {
+
+ private static final String REQUEST_ID_PREFIX = 'r'
+
+ private static final String PLACEHOLDER = '?'
+
+ /** The parameters a URL mapping sets that name code rather than carry a value. */
+ private static final Set<String> MAPPING_PARAMETERS = ['controller', 'action', 'namespace', 'plugin', 'format'] as Set<String>
+
+ /** A segment that is a number, a UUID or a long hexadecimal token is a value whatever the mapping says. */
+ private static final Pattern VALUE_SHAPED = Pattern.compile(
+ '(?i)[0-9]+|[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}|[0-9a-f]{32,}')
+
+ private static final boolean SERVLET_PRESENT = ClassUtils.isPresent(
+ 'jakarta.servlet.http.HttpServletRequest', WebRequestLookup.classLoader)
+
+ /** The {@code getParams()} of a Grails web request, which holds the parameters its URL mapping took, by class. */
+ private static final ClassValue<Method> PARAMETERS = new ClassValue<Method>() {
+ @Override
+ protected Method computeValue(Class<?> type) {
+ try {
+ return type.getMethod('getParams')
+ }
+ catch (NoSuchMethodException ignored) {
+ return null
+ }
+ }
+ }
+
+ /**
+ * Fills in a record from the request on the current thread.
+ *
+ * @param record the record to fill in
+ * @param requestCounter the counter the next request id is taken from
+ * @param requestIdPrefix what every request id starts with
+ * @return whether there is a request on the current thread
+ */
+ static boolean attribute(QueryTailRecord record, AtomicLong requestCounter, String requestIdPrefix) {
+ RequestAttributes attributes = RequestContextHolder.getRequestAttributes()
+ if (attributes == null) {
+ return false
+ }
+ record.requestId = requestId(attributes, requestCounter, requestIdPrefix)
+ record.page = page(attributes)
+ record.request = SERVLET_PRESENT ? ServletRequestLookup.describe(attributes) : null
+ true
+ }
+
+ /**
+ * Builds the request line: the method and the path, with the values in the path, the parameters
+ * the URL mapping took from it among them, replaced.
+ *
+ * @param method the HTTP method
+ * @param path the request path
+ * @param attributes the request, whose mapped parameters are read when it is a Grails web request
+ * @param parameters the parameters of the servlet request, by name, each a {@code String[]}
+ */
+ static String describe(String method, String path, RequestAttributes attributes, Map<?, ?> parameters) {
+ Map<?, ?> mapped = mappedParameters(attributes)
+ boolean mappingRan = attributes.getAttribute(CONTROLLER_NAME_ATTRIBUTE, RequestAttributes.SCOPE_REQUEST) != null ||
Review Comment:
Fixed in a50ddf396b (folded into the feature commit). Only the controller attribute the framework sets on the request now counts as the mapping having run; the parameter map, which holds the servlet parameters too, is no longer consulted for that, so `/password/reset/k9fQ2x?controller=x` before mapping is recorded as `GET /?/?/?`. `QueryTailRequestAttributionSpec` covers that query (a request whose parameters carry `controller=x` and no controller attribute), and its mapped-request helper now binds the controller and action as attributes, the way the URL mapping leaves a request, so the rows that exercise the value patterns still run mapped. The javadoc and the guide say that only the resolved controller counts, never a parameter a client sent.
##########
grails-datamapping-core/src/main/groovy/org/apache/grails/data/core/querytail/QueryTailRequestAttribution.groovy:
##########
@@ -0,0 +1,385 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * https://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.grails.data.core.querytail
+
+import java.lang.reflect.Method
+import java.nio.charset.StandardCharsets
+import java.util.concurrent.atomic.AtomicLong
+import java.util.regex.Pattern
+
+import groovy.transform.CompileStatic
+
+import org.springframework.util.ClassUtils
+import org.springframework.web.context.request.RequestAttributes
+import org.springframework.web.context.request.RequestContextHolder
+import org.springframework.web.context.request.ServletRequestAttributes
+
+/**
+ * Attributes a captured statement to the web request on the current thread. Grails keeps the
+ * request, and the controller and action it resolved, on the thread that serves it, so a statement
+ * run by an interceptor, the action or the view all land on the same page. The first statement of
+ * a request gives it an id that every later statement of that request repeats, which is how the
+ * {@code query-tail} command counts statements per request. Work on any other thread is attributed to
+ * the thread by name.
+ *
+ * <p>The request is recorded as its method and path, with the values in the path replaced by
+ * {@code ?}: the values of the parameters the URL mapping took from it and those of its query, found
+ * whether or not the client percent-encoded them, and any segment that is a number, a UUID or a long
+ * hexadecimal token, so that an id, a token or a key carried in a path never reaches the capture
+ * file. Until the URL mapping has run, as for a statement of a filter that runs before the controller
+ * is chosen, or for a request that never reaches one, no segment's role is known and every segment
+ * is replaced.</p>
+ *
+ * <p>Spring Web is optional: without it on the classpath every statement is attributed to its
+ * thread.</p>
+ *
+ * @since 8.1
+ */
+@CompileStatic
+class QueryTailRequestAttribution {
+
+ /** The request attribute holding the id this attribution gave a request. */
+ public static final String REQUEST_ID_ATTRIBUTE = 'org.apache.grails.data.core.querytail.REQUEST_ID'
+
+ /** The request attribute Grails sets the controller name in. */
+ public static final String CONTROLLER_NAME_ATTRIBUTE = 'org.grails.CONTROLLER_NAME_ATTRIBUTE'
+
+ /** The request attribute Grails sets the controller namespace in. */
+ public static final String CONTROLLER_NAMESPACE_ATTRIBUTE = 'org.grails.CONTROLLER_NAMESPACE_ATTRIBUTE'
+
+ /** The request attribute Grails sets the action name in. */
+ public static final String ACTION_NAME_ATTRIBUTE = 'org.grails.ACTION_NAME_ATTRIBUTE'
+
+ private static final boolean SPRING_WEB_PRESENT = ClassUtils.isPresent(
+ 'org.springframework.web.context.request.RequestContextHolder', QueryTailRequestAttribution.classLoader)
+
+ private final AtomicLong requestCounter = new AtomicLong()
+ private final String requestIdPrefix
+
+ /**
+ * An attribution whose request ids are {@code r1}, {@code r2} and so on.
+ */
+ QueryTailRequestAttribution() {
+ this('')
+ }
+
+ /**
+ * @param requestIdPrefix what every request id starts with, which tells the requests of one JVM
+ * from those of another writing the same capture
+ */
+ QueryTailRequestAttribution(String requestIdPrefix) {
+ this.requestIdPrefix = requestIdPrefix ?: ''
+ }
+
+ /**
+ * Fills in the request id, page and request of a record from the current thread.
+ *
+ * @param record the record of a statement that just ran on this thread
+ * @return the same record
+ */
+ QueryTailRecord attribute(QueryTailRecord record) {
+ if (SPRING_WEB_PRESENT && WebRequestLookup.attribute(record, requestCounter, requestIdPrefix)) {
+ return record
+ }
+ record.requestId = null
+ record.request = null
+ record.page = QueryTailRecord.THREAD_PAGE_PREFIX + Thread.currentThread().name
+ record
+ }
+
+ /**
+ * Keeps every reference to Spring Web, in a signature as well as in a body, in a class that is only
+ * loaded once Spring Web is known to be present. The outer class can then be loaded, and reflected
+ * on, without Spring Web on the classpath.
+ */
+ @CompileStatic
+ private static class WebRequestLookup {
+
+ private static final String REQUEST_ID_PREFIX = 'r'
+
+ private static final String PLACEHOLDER = '?'
+
+ /** The parameters a URL mapping sets that name code rather than carry a value. */
+ private static final Set<String> MAPPING_PARAMETERS = ['controller', 'action', 'namespace', 'plugin', 'format'] as Set<String>
+
+ /** A segment that is a number, a UUID or a long hexadecimal token is a value whatever the mapping says. */
+ private static final Pattern VALUE_SHAPED = Pattern.compile(
+ '(?i)[0-9]+|[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}|[0-9a-f]{32,}')
+
+ private static final boolean SERVLET_PRESENT = ClassUtils.isPresent(
+ 'jakarta.servlet.http.HttpServletRequest', WebRequestLookup.classLoader)
+
+ /** The {@code getParams()} of a Grails web request, which holds the parameters its URL mapping took, by class. */
+ private static final ClassValue<Method> PARAMETERS = new ClassValue<Method>() {
+ @Override
+ protected Method computeValue(Class<?> type) {
+ try {
+ return type.getMethod('getParams')
+ }
+ catch (NoSuchMethodException ignored) {
+ return null
+ }
+ }
+ }
+
+ /**
+ * Fills in a record from the request on the current thread.
+ *
+ * @param record the record to fill in
+ * @param requestCounter the counter the next request id is taken from
+ * @param requestIdPrefix what every request id starts with
+ * @return whether there is a request on the current thread
+ */
+ static boolean attribute(QueryTailRecord record, AtomicLong requestCounter, String requestIdPrefix) {
+ RequestAttributes attributes = RequestContextHolder.getRequestAttributes()
+ if (attributes == null) {
+ return false
+ }
+ record.requestId = requestId(attributes, requestCounter, requestIdPrefix)
+ record.page = page(attributes)
+ record.request = SERVLET_PRESENT ? ServletRequestLookup.describe(attributes) : null
+ true
+ }
+
+ /**
+ * Builds the request line: the method and the path, with the values in the path, the parameters
+ * the URL mapping took from it among them, replaced.
+ *
+ * @param method the HTTP method
+ * @param path the request path
+ * @param attributes the request, whose mapped parameters are read when it is a Grails web request
+ * @param parameters the parameters of the servlet request, by name, each a {@code String[]}
+ */
+ static String describe(String method, String path, RequestAttributes attributes, Map<?, ?> parameters) {
+ Map<?, ?> mapped = mappedParameters(attributes)
+ boolean mappingRan = attributes.getAttribute(CONTROLLER_NAME_ATTRIBUTE, RequestAttributes.SCOPE_REQUEST) != null ||
+ (mapped != null && mapped.containsKey('controller'))
+ if (!mappingRan) {
+ return "${method} ${withholdPath(path)}".toString()
+ }
+ Set<String> values = new HashSet<>()
+ addValues(values, mapped)
+ addValues(values, parameters)
+ "${method} ${redactPath(path, values)}".toString()
+ }
+
+ /**
+ * Replaces every segment of a path: before the URL mapping has run, nothing tells a literal
+ * segment from a token, so only the shape of the path is recorded.
+ */
+ private static String withholdPath(String path) {
+ String[] segments = path.split('/', -1)
+ StringBuilder out = new StringBuilder(path.length())
+ for (int i = 0; i < segments.length; i++) {
+ if (i > 0) {
+ out.append('/')
+ }
+ out.append(segments[i].isEmpty() ? segments[i] : PLACEHOLDER)
+ }
+ out.toString()
+ }
+
+ private static String requestId(RequestAttributes attributes, AtomicLong requestCounter, String requestIdPrefix) {
+ synchronized (attributes) {
+ Object existing = attributes.getAttribute(REQUEST_ID_ATTRIBUTE, RequestAttributes.SCOPE_REQUEST)
+ if (existing != null) {
+ return existing.toString()
+ }
+ String id = requestIdPrefix + REQUEST_ID_PREFIX + requestCounter.incrementAndGet()
+ attributes.setAttribute(REQUEST_ID_ATTRIBUTE, id, RequestAttributes.SCOPE_REQUEST)
+ return id
+ }
+ }
+
+ private static String page(RequestAttributes attributes) {
+ Object controller = attributes.getAttribute(CONTROLLER_NAME_ATTRIBUTE, RequestAttributes.SCOPE_REQUEST)
+ if (controller == null) {
+ return null
+ }
+ Object namespace = attributes.getAttribute(CONTROLLER_NAMESPACE_ATTRIBUTE, RequestAttributes.SCOPE_REQUEST)
+ Object action = attributes.getAttribute(ACTION_NAME_ATTRIBUTE, RequestAttributes.SCOPE_REQUEST)
+ StringBuilder page = new StringBuilder()
+ if (namespace != null) {
+ page.append(namespace).append(':')
+ }
+ page.append(controller)
+ if (action != null) {
+ page.append('/').append(action)
+ }
+ page.toString()
+ }
+
+ /**
+ * @return the parameters of a Grails web request, which include those its URL mapping took from
+ * the path, or null for a request that is not one
+ */
+ private static Map<?, ?> mappedParameters(RequestAttributes attributes) {
+ Method parameters = PARAMETERS.get(attributes.getClass())
+ if (parameters == null) {
+ return null
+ }
+ try {
+ Object value = parameters.invoke(attributes)
+ return value instanceof Map ? (Map<?, ?>) value : null
+ }
+ catch (ReflectiveOperationException | RuntimeException ignored) {
+ return null
+ }
+ }
+
+ private static void addValues(Set<String> values, Map<?, ?> parameters) {
+ if (parameters == null) {
+ return
+ }
+ for (Map.Entry<?, ?> entry in parameters.entrySet()) {
+ if (MAPPING_PARAMETERS.contains(String.valueOf(entry.key))) {
+ continue
+ }
+ Object value = entry.value
+ if (value instanceof CharSequence) {
+ values.add(value.toString())
+ }
+ else if (value instanceof Object[]) {
+ for (Object element in (Object[]) value) {
+ if (element instanceof CharSequence) {
+ values.add(element.toString())
+ }
+ }
+ }
+ }
+ }
+
+ /**
+ * Replaces the values in a path: a segment that is one of the values, with or without an
+ * extension, a run of segments that spells a value holding slashes, and any segment shaped
+ * like a value. The segments are matched as the client sent them and percent-decoded, since
+ * the parameters are decoded while the request URI is not; the path is written as sent.
+ */
+ private static String redactPath(String path, Set<String> values) {
+ String[] segments = path.split('/', -1)
+ String[] decoded = new String[segments.length]
+ boolean[] redacted = new boolean[segments.length]
+ for (int i = 0; i < segments.length; i++) {
+ decoded[i] = decode(segments[i])
+ redacted[i] = isValue(segments[i], values) || isValue(decoded[i], values)
+ }
+ for (String value in values) {
+ if (value.contains('/')) {
+ String[] run = value.split('/', -1)
+ markRun(segments, redacted, run)
+ markDecodedRun(decoded, redacted, run)
+ }
+ }
+ StringBuilder out = new StringBuilder(path.length())
+ for (int i = 0; i < segments.length; i++) {
+ if (i > 0) {
+ out.append('/')
+ }
+ out.append(redacted[i] ? PLACEHOLDER : segments[i])
+ }
+ out.toString()
+ }
+
+ /**
+ * @return the segment with its percent-encoding decoded, or as it is when it has none or is not
+ * valid percent-encoding; a plus sign is kept, since it is literal in a path
+ */
+ private static String decode(String segment) {
+ if (segment.indexOf('%') < 0) {
+ return segment
+ }
+ try {
+ return URLDecoder.decode(segment.replace('+', '%2B'), StandardCharsets.UTF_8)
+ }
+ catch (IllegalArgumentException ignored) {
+ return segment
+ }
+ }
+
+ private static boolean isValue(String segment, Set<String> values) {
+ if (segment.isEmpty()) {
+ return false
+ }
+ int dot = segment.lastIndexOf('.')
Review Comment:
Fixed in a50ddf396b. A mapped value is now cut out of a segment it is glued into: after the whole-segment checks, `withoutValues` replaces each parameter value within the segment, in the form (as sent or percent-decoded) it was found in, so `/reset/k9fQ2x_confirm` with `token=k9fQ2x` is written as `GET /reset/?_confirm` and `/order/42_summary.pdf` with `id=42` as `GET /order/?_summary.pdf`. A value of three or more characters is replaced wherever it appears; a shorter one only where no letter or digit continues it on either side, so that a one-letter query value does not pick holes in every word of the path (`q=a` leaves `/book/index` alone, while `q=ind` makes it `/book/?ex`). Both rules are in the table of `QueryTailRequestAttributionSpec`, and the guide and javadoc describe the glued case.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@grails.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add the query-tail command to find query-heavy pages [grails-core]
Posted by "jamesfredley (via GitHub)" <gi...@apache.org>.
jamesfredley commented on code in PR #16631:
URL: https://github.com/apache/grails-core/pull/16631#discussion_r4368470709
##########
grails-datamapping-core/src/main/groovy/org/apache/grails/data/core/querytail/QueryTailRequestAttribution.groovy:
##########
@@ -0,0 +1,385 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * https://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.grails.data.core.querytail
+
+import java.lang.reflect.Method
+import java.nio.charset.StandardCharsets
+import java.util.concurrent.atomic.AtomicLong
+import java.util.regex.Pattern
+
+import groovy.transform.CompileStatic
+
+import org.springframework.util.ClassUtils
+import org.springframework.web.context.request.RequestAttributes
+import org.springframework.web.context.request.RequestContextHolder
+import org.springframework.web.context.request.ServletRequestAttributes
+
+/**
+ * Attributes a captured statement to the web request on the current thread. Grails keeps the
+ * request, and the controller and action it resolved, on the thread that serves it, so a statement
+ * run by an interceptor, the action or the view all land on the same page. The first statement of
+ * a request gives it an id that every later statement of that request repeats, which is how the
+ * {@code query-tail} command counts statements per request. Work on any other thread is attributed to
+ * the thread by name.
+ *
+ * <p>The request is recorded as its method and path, with the values in the path replaced by
+ * {@code ?}: the values of the parameters the URL mapping took from it and those of its query, found
+ * whether or not the client percent-encoded them, and any segment that is a number, a UUID or a long
+ * hexadecimal token, so that an id, a token or a key carried in a path never reaches the capture
+ * file. Until the URL mapping has run, as for a statement of a filter that runs before the controller
+ * is chosen, or for a request that never reaches one, no segment's role is known and every segment
+ * is replaced.</p>
+ *
+ * <p>Spring Web is optional: without it on the classpath every statement is attributed to its
+ * thread.</p>
+ *
+ * @since 8.1
+ */
+@CompileStatic
+class QueryTailRequestAttribution {
+
+ /** The request attribute holding the id this attribution gave a request. */
+ public static final String REQUEST_ID_ATTRIBUTE = 'org.apache.grails.data.core.querytail.REQUEST_ID'
+
+ /** The request attribute Grails sets the controller name in. */
+ public static final String CONTROLLER_NAME_ATTRIBUTE = 'org.grails.CONTROLLER_NAME_ATTRIBUTE'
+
+ /** The request attribute Grails sets the controller namespace in. */
+ public static final String CONTROLLER_NAMESPACE_ATTRIBUTE = 'org.grails.CONTROLLER_NAMESPACE_ATTRIBUTE'
+
+ /** The request attribute Grails sets the action name in. */
+ public static final String ACTION_NAME_ATTRIBUTE = 'org.grails.ACTION_NAME_ATTRIBUTE'
+
+ private static final boolean SPRING_WEB_PRESENT = ClassUtils.isPresent(
+ 'org.springframework.web.context.request.RequestContextHolder', QueryTailRequestAttribution.classLoader)
+
+ private final AtomicLong requestCounter = new AtomicLong()
+ private final String requestIdPrefix
+
+ /**
+ * An attribution whose request ids are {@code r1}, {@code r2} and so on.
+ */
+ QueryTailRequestAttribution() {
+ this('')
+ }
+
+ /**
+ * @param requestIdPrefix what every request id starts with, which tells the requests of one JVM
+ * from those of another writing the same capture
+ */
+ QueryTailRequestAttribution(String requestIdPrefix) {
+ this.requestIdPrefix = requestIdPrefix ?: ''
+ }
+
+ /**
+ * Fills in the request id, page and request of a record from the current thread.
+ *
+ * @param record the record of a statement that just ran on this thread
+ * @return the same record
+ */
+ QueryTailRecord attribute(QueryTailRecord record) {
+ if (SPRING_WEB_PRESENT && WebRequestLookup.attribute(record, requestCounter, requestIdPrefix)) {
+ return record
+ }
+ record.requestId = null
+ record.request = null
+ record.page = QueryTailRecord.THREAD_PAGE_PREFIX + Thread.currentThread().name
+ record
+ }
+
+ /**
+ * Keeps every reference to Spring Web, in a signature as well as in a body, in a class that is only
+ * loaded once Spring Web is known to be present. The outer class can then be loaded, and reflected
+ * on, without Spring Web on the classpath.
+ */
+ @CompileStatic
+ private static class WebRequestLookup {
+
+ private static final String REQUEST_ID_PREFIX = 'r'
+
+ private static final String PLACEHOLDER = '?'
+
+ /** The parameters a URL mapping sets that name code rather than carry a value. */
+ private static final Set<String> MAPPING_PARAMETERS = ['controller', 'action', 'namespace', 'plugin', 'format'] as Set<String>
+
+ /** A segment that is a number, a UUID or a long hexadecimal token is a value whatever the mapping says. */
+ private static final Pattern VALUE_SHAPED = Pattern.compile(
+ '(?i)[0-9]+|[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}|[0-9a-f]{32,}')
+
+ private static final boolean SERVLET_PRESENT = ClassUtils.isPresent(
+ 'jakarta.servlet.http.HttpServletRequest', WebRequestLookup.classLoader)
+
+ /** The {@code getParams()} of a Grails web request, which holds the parameters its URL mapping took, by class. */
+ private static final ClassValue<Method> PARAMETERS = new ClassValue<Method>() {
+ @Override
+ protected Method computeValue(Class<?> type) {
+ try {
+ return type.getMethod('getParams')
+ }
+ catch (NoSuchMethodException ignored) {
+ return null
+ }
+ }
+ }
+
+ /**
+ * Fills in a record from the request on the current thread.
+ *
+ * @param record the record to fill in
+ * @param requestCounter the counter the next request id is taken from
+ * @param requestIdPrefix what every request id starts with
+ * @return whether there is a request on the current thread
+ */
+ static boolean attribute(QueryTailRecord record, AtomicLong requestCounter, String requestIdPrefix) {
+ RequestAttributes attributes = RequestContextHolder.getRequestAttributes()
+ if (attributes == null) {
+ return false
+ }
+ record.requestId = requestId(attributes, requestCounter, requestIdPrefix)
+ record.page = page(attributes)
+ record.request = SERVLET_PRESENT ? ServletRequestLookup.describe(attributes) : null
+ true
+ }
+
+ /**
+ * Builds the request line: the method and the path, with the values in the path, the parameters
+ * the URL mapping took from it among them, replaced.
+ *
+ * @param method the HTTP method
+ * @param path the request path
+ * @param attributes the request, whose mapped parameters are read when it is a Grails web request
+ * @param parameters the parameters of the servlet request, by name, each a {@code String[]}
+ */
+ static String describe(String method, String path, RequestAttributes attributes, Map<?, ?> parameters) {
+ Map<?, ?> mapped = mappedParameters(attributes)
+ boolean mappingRan = attributes.getAttribute(CONTROLLER_NAME_ATTRIBUTE, RequestAttributes.SCOPE_REQUEST) != null ||
+ (mapped != null && mapped.containsKey('controller'))
+ if (!mappingRan) {
+ return "${method} ${withholdPath(path)}".toString()
+ }
+ Set<String> values = new HashSet<>()
+ addValues(values, mapped)
+ addValues(values, parameters)
+ "${method} ${redactPath(path, values)}".toString()
+ }
+
+ /**
+ * Replaces every segment of a path: before the URL mapping has run, nothing tells a literal
+ * segment from a token, so only the shape of the path is recorded.
+ */
+ private static String withholdPath(String path) {
+ String[] segments = path.split('/', -1)
+ StringBuilder out = new StringBuilder(path.length())
+ for (int i = 0; i < segments.length; i++) {
+ if (i > 0) {
+ out.append('/')
+ }
+ out.append(segments[i].isEmpty() ? segments[i] : PLACEHOLDER)
+ }
+ out.toString()
+ }
+
+ private static String requestId(RequestAttributes attributes, AtomicLong requestCounter, String requestIdPrefix) {
+ synchronized (attributes) {
+ Object existing = attributes.getAttribute(REQUEST_ID_ATTRIBUTE, RequestAttributes.SCOPE_REQUEST)
+ if (existing != null) {
+ return existing.toString()
+ }
+ String id = requestIdPrefix + REQUEST_ID_PREFIX + requestCounter.incrementAndGet()
+ attributes.setAttribute(REQUEST_ID_ATTRIBUTE, id, RequestAttributes.SCOPE_REQUEST)
+ return id
+ }
+ }
+
+ private static String page(RequestAttributes attributes) {
+ Object controller = attributes.getAttribute(CONTROLLER_NAME_ATTRIBUTE, RequestAttributes.SCOPE_REQUEST)
+ if (controller == null) {
+ return null
+ }
+ Object namespace = attributes.getAttribute(CONTROLLER_NAMESPACE_ATTRIBUTE, RequestAttributes.SCOPE_REQUEST)
+ Object action = attributes.getAttribute(ACTION_NAME_ATTRIBUTE, RequestAttributes.SCOPE_REQUEST)
+ StringBuilder page = new StringBuilder()
+ if (namespace != null) {
+ page.append(namespace).append(':')
+ }
+ page.append(controller)
+ if (action != null) {
+ page.append('/').append(action)
+ }
+ page.toString()
+ }
+
+ /**
+ * @return the parameters of a Grails web request, which include those its URL mapping took from
+ * the path, or null for a request that is not one
+ */
+ private static Map<?, ?> mappedParameters(RequestAttributes attributes) {
+ Method parameters = PARAMETERS.get(attributes.getClass())
+ if (parameters == null) {
+ return null
+ }
+ try {
+ Object value = parameters.invoke(attributes)
+ return value instanceof Map ? (Map<?, ?>) value : null
+ }
+ catch (ReflectiveOperationException | RuntimeException ignored) {
+ return null
+ }
+ }
+
+ private static void addValues(Set<String> values, Map<?, ?> parameters) {
+ if (parameters == null) {
+ return
+ }
+ for (Map.Entry<?, ?> entry in parameters.entrySet()) {
+ if (MAPPING_PARAMETERS.contains(String.valueOf(entry.key))) {
+ continue
+ }
+ Object value = entry.value
+ if (value instanceof CharSequence) {
+ values.add(value.toString())
+ }
+ else if (value instanceof Object[]) {
+ for (Object element in (Object[]) value) {
+ if (element instanceof CharSequence) {
+ values.add(element.toString())
+ }
+ }
+ }
+ }
+ }
+
+ /**
+ * Replaces the values in a path: a segment that is one of the values, with or without an
+ * extension, a run of segments that spells a value holding slashes, and any segment shaped
+ * like a value. The segments are matched as the client sent them and percent-decoded, since
+ * the parameters are decoded while the request URI is not; the path is written as sent.
+ */
+ private static String redactPath(String path, Set<String> values) {
+ String[] segments = path.split('/', -1)
+ String[] decoded = new String[segments.length]
+ boolean[] redacted = new boolean[segments.length]
+ for (int i = 0; i < segments.length; i++) {
+ decoded[i] = decode(segments[i])
+ redacted[i] = isValue(segments[i], values) || isValue(decoded[i], values)
+ }
+ for (String value in values) {
+ if (value.contains('/')) {
+ String[] run = value.split('/', -1)
+ markRun(segments, redacted, run)
+ markDecodedRun(decoded, redacted, run)
+ }
+ }
+ StringBuilder out = new StringBuilder(path.length())
+ for (int i = 0; i < segments.length; i++) {
+ if (i > 0) {
+ out.append('/')
+ }
+ out.append(redacted[i] ? PLACEHOLDER : segments[i])
+ }
+ out.toString()
+ }
+
+ /**
+ * @return the segment with its percent-encoding decoded, or as it is when it has none or is not
+ * valid percent-encoding; a plus sign is kept, since it is literal in a path
+ */
+ private static String decode(String segment) {
+ if (segment.indexOf('%') < 0) {
+ return segment
+ }
+ try {
+ return URLDecoder.decode(segment.replace('+', '%2B'), StandardCharsets.UTF_8)
+ }
+ catch (IllegalArgumentException ignored) {
+ return segment
+ }
+ }
+
+ private static boolean isValue(String segment, Set<String> values) {
+ if (segment.isEmpty()) {
+ return false
+ }
+ int dot = segment.lastIndexOf('.')
Review Comment:
A mapping such as `/reset/${token}_confirm` with `token=k9fQ2x` is written as `GET /reset/k9fQ2x_confirm`. The check matches a whole segment, not a value with a static prefix or suffix.
Redact the value inside the segment, or withhold the segment.
##########
grails-datamapping-core/src/main/groovy/org/apache/grails/data/core/querytail/QueryTailRequestAttribution.groovy:
##########
@@ -0,0 +1,385 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * https://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.grails.data.core.querytail
+
+import java.lang.reflect.Method
+import java.nio.charset.StandardCharsets
+import java.util.concurrent.atomic.AtomicLong
+import java.util.regex.Pattern
+
+import groovy.transform.CompileStatic
+
+import org.springframework.util.ClassUtils
+import org.springframework.web.context.request.RequestAttributes
+import org.springframework.web.context.request.RequestContextHolder
+import org.springframework.web.context.request.ServletRequestAttributes
+
+/**
+ * Attributes a captured statement to the web request on the current thread. Grails keeps the
+ * request, and the controller and action it resolved, on the thread that serves it, so a statement
+ * run by an interceptor, the action or the view all land on the same page. The first statement of
+ * a request gives it an id that every later statement of that request repeats, which is how the
+ * {@code query-tail} command counts statements per request. Work on any other thread is attributed to
+ * the thread by name.
+ *
+ * <p>The request is recorded as its method and path, with the values in the path replaced by
+ * {@code ?}: the values of the parameters the URL mapping took from it and those of its query, found
+ * whether or not the client percent-encoded them, and any segment that is a number, a UUID or a long
+ * hexadecimal token, so that an id, a token or a key carried in a path never reaches the capture
+ * file. Until the URL mapping has run, as for a statement of a filter that runs before the controller
+ * is chosen, or for a request that never reaches one, no segment's role is known and every segment
+ * is replaced.</p>
+ *
+ * <p>Spring Web is optional: without it on the classpath every statement is attributed to its
+ * thread.</p>
+ *
+ * @since 8.1
+ */
+@CompileStatic
+class QueryTailRequestAttribution {
+
+ /** The request attribute holding the id this attribution gave a request. */
+ public static final String REQUEST_ID_ATTRIBUTE = 'org.apache.grails.data.core.querytail.REQUEST_ID'
+
+ /** The request attribute Grails sets the controller name in. */
+ public static final String CONTROLLER_NAME_ATTRIBUTE = 'org.grails.CONTROLLER_NAME_ATTRIBUTE'
+
+ /** The request attribute Grails sets the controller namespace in. */
+ public static final String CONTROLLER_NAMESPACE_ATTRIBUTE = 'org.grails.CONTROLLER_NAMESPACE_ATTRIBUTE'
+
+ /** The request attribute Grails sets the action name in. */
+ public static final String ACTION_NAME_ATTRIBUTE = 'org.grails.ACTION_NAME_ATTRIBUTE'
+
+ private static final boolean SPRING_WEB_PRESENT = ClassUtils.isPresent(
+ 'org.springframework.web.context.request.RequestContextHolder', QueryTailRequestAttribution.classLoader)
+
+ private final AtomicLong requestCounter = new AtomicLong()
+ private final String requestIdPrefix
+
+ /**
+ * An attribution whose request ids are {@code r1}, {@code r2} and so on.
+ */
+ QueryTailRequestAttribution() {
+ this('')
+ }
+
+ /**
+ * @param requestIdPrefix what every request id starts with, which tells the requests of one JVM
+ * from those of another writing the same capture
+ */
+ QueryTailRequestAttribution(String requestIdPrefix) {
+ this.requestIdPrefix = requestIdPrefix ?: ''
+ }
+
+ /**
+ * Fills in the request id, page and request of a record from the current thread.
+ *
+ * @param record the record of a statement that just ran on this thread
+ * @return the same record
+ */
+ QueryTailRecord attribute(QueryTailRecord record) {
+ if (SPRING_WEB_PRESENT && WebRequestLookup.attribute(record, requestCounter, requestIdPrefix)) {
+ return record
+ }
+ record.requestId = null
+ record.request = null
+ record.page = QueryTailRecord.THREAD_PAGE_PREFIX + Thread.currentThread().name
+ record
+ }
+
+ /**
+ * Keeps every reference to Spring Web, in a signature as well as in a body, in a class that is only
+ * loaded once Spring Web is known to be present. The outer class can then be loaded, and reflected
+ * on, without Spring Web on the classpath.
+ */
+ @CompileStatic
+ private static class WebRequestLookup {
+
+ private static final String REQUEST_ID_PREFIX = 'r'
+
+ private static final String PLACEHOLDER = '?'
+
+ /** The parameters a URL mapping sets that name code rather than carry a value. */
+ private static final Set<String> MAPPING_PARAMETERS = ['controller', 'action', 'namespace', 'plugin', 'format'] as Set<String>
+
+ /** A segment that is a number, a UUID or a long hexadecimal token is a value whatever the mapping says. */
+ private static final Pattern VALUE_SHAPED = Pattern.compile(
+ '(?i)[0-9]+|[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}|[0-9a-f]{32,}')
+
+ private static final boolean SERVLET_PRESENT = ClassUtils.isPresent(
+ 'jakarta.servlet.http.HttpServletRequest', WebRequestLookup.classLoader)
+
+ /** The {@code getParams()} of a Grails web request, which holds the parameters its URL mapping took, by class. */
+ private static final ClassValue<Method> PARAMETERS = new ClassValue<Method>() {
+ @Override
+ protected Method computeValue(Class<?> type) {
+ try {
+ return type.getMethod('getParams')
+ }
+ catch (NoSuchMethodException ignored) {
+ return null
+ }
+ }
+ }
+
+ /**
+ * Fills in a record from the request on the current thread.
+ *
+ * @param record the record to fill in
+ * @param requestCounter the counter the next request id is taken from
+ * @param requestIdPrefix what every request id starts with
+ * @return whether there is a request on the current thread
+ */
+ static boolean attribute(QueryTailRecord record, AtomicLong requestCounter, String requestIdPrefix) {
+ RequestAttributes attributes = RequestContextHolder.getRequestAttributes()
+ if (attributes == null) {
+ return false
+ }
+ record.requestId = requestId(attributes, requestCounter, requestIdPrefix)
+ record.page = page(attributes)
+ record.request = SERVLET_PRESENT ? ServletRequestLookup.describe(attributes) : null
+ true
+ }
+
+ /**
+ * Builds the request line: the method and the path, with the values in the path, the parameters
+ * the URL mapping took from it among them, replaced.
+ *
+ * @param method the HTTP method
+ * @param path the request path
+ * @param attributes the request, whose mapped parameters are read when it is a Grails web request
+ * @param parameters the parameters of the servlet request, by name, each a {@code String[]}
+ */
+ static String describe(String method, String path, RequestAttributes attributes, Map<?, ?> parameters) {
+ Map<?, ?> mapped = mappedParameters(attributes)
+ boolean mappingRan = attributes.getAttribute(CONTROLLER_NAME_ATTRIBUTE, RequestAttributes.SCOPE_REQUEST) != null ||
Review Comment:
`mapped.containsKey('controller')` is not proof that URL mapping ran. Servlet parameters are already in that map. `/password/reset/k9fQ2x?controller=x`, before mapping, is recorded as `GET /password/reset/k9fQ2x`.
Do not treat a client parameter as mapping evidence. Withhold every segment until the framework has mapped the request. Please cover that query.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@grails.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add the query-tail command to find query-heavy pages [grails-core]
Posted by "jdaugherty (via GitHub)" <gi...@apache.org>.
jdaugherty commented on PR #16631:
URL: https://github.com/apache/grails-core/pull/16631#issuecomment-6389417939
@jamesfredley re review 5639950802: both items are fixed on head a50ddf396b. Only the controller attribute the framework sets counts as the URL mapping having run, never a parameter, so `/password/reset/k9fQ2x?controller=x` before mapping records `GET /?/?/?` (covered in the attribution spec). A mapped value glued to static text is cut out of its segment, so `/reset/k9fQ2x_confirm` with `token=k9fQ2x` records `GET /reset/?_confirm`; values of three or more characters are cut wherever they appear, shorter ones only where no letter or digit continues them. Details are in the replies on your two threads, which I resolved since both are fixed; reopen either if you want to discuss it further.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@grails.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add the query-tail command to find query-heavy pages [grails-core]
Posted by "jdaugherty (via GitHub)" <gi...@apache.org>.
jdaugherty commented on code in PR #16631:
URL: https://github.com/apache/grails-core/pull/16631#discussion_r4371545960
##########
grails-datamapping-core/src/cli/groovy/org/apache/grails/data/core/cli/querytail/QueryTailFileReader.groovy:
##########
@@ -0,0 +1,259 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * https://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.grails.data.core.cli.querytail
+
+import java.nio.charset.StandardCharsets
+
+import groovy.transform.CompileStatic
+
+import org.apache.grails.data.core.querytail.QueryTailRecord
+
+/**
+ * Follows a capture file the way {@code tail -f} does: each poll returns the records written
+ * since the last one, up to a {@link #CHUNK chunk} of the file at a time, so that a capture of any
+ * size is read in pieces; a reader that is {@link #isBehind() behind} has more to read without
+ * waiting. A line that has no terminator yet is left for the next poll, so a record the
+ * application is still writing is never read in half, and a line longer than a chunk is dropped
+ * up to its end, as a line that is not a record is. An application that starts again
+ * truncates the file and names its new run in the first line; when that line names another run,
+ * or the file shrinks, reading starts over from the beginning. The run tells a restart apart from
+ * a file that grew past where the reader was before it was next polled. A file truncated between
+ * being measured and being read, so that the read comes up short, is a restart too: it is read again
+ * from the beginning in the same poll. A poll that finds the file as long as it left it reads nothing,
+ * not even its first line, so an idle command costs nothing; a restart that has written nothing yet
+ * is noticed with its first statement.
+ *
+ * @since 8.1
+ */
+@CompileStatic
+class QueryTailFileReader implements Closeable {
+
+ private static final byte NEWLINE = (byte) 10
+
+ /** How much of the file the first line is looked for in; a header is far shorter. */
+ private static final int HEADER_LIMIT = 256
+
+ /** How much of the file one poll reads at most, in bytes. */
+ public static final int CHUNK = 1024 * 1024
+
+ final File file
+ final int chunk
+ private long position
+ private long measured = -1L
+ private String run
+ private boolean restarted
+ private boolean behind
+ private boolean skipping
+
+ /**
+ * @param file the capture file
+ * @param fromStart whether to read what the file already holds, or only what is written from now on
+ */
+ QueryTailFileReader(File file, boolean fromStart) {
+ this(file, fromStart, CHUNK)
+ }
+
+ /**
+ * @param file the capture file
+ * @param fromStart whether to read what the file already holds, or only what is written from now on
+ * @param chunk how much of the file one poll reads at most, in bytes
+ */
+ QueryTailFileReader(File file, boolean fromStart, int chunk) {
+ this.file = file
+ this.chunk = chunk
+ this.position = fromStart ? 0 : file.length()
+ this.run = file.exists() ? readRun() : null
+ // opened in the middle of a line being written: the rest of that line is not a record
+ this.skipping = position > 0 && !endsLine(readFrom(position - 1, position))
+ }
+
+ /**
+ * @return the records written since the last poll, or since the reader was opened, up to a chunk of the file
+ */
+ List<QueryTailRecord> poll() {
+ restarted = false
+ behind = false
+ // a read that comes up short is retried once, from the start; the next poll takes it from there
+ for (int attempt = 0; attempt < 2; attempt++) {
+ long length = file.length()
+ if (length == position && length == measured) {
Review Comment:
Fixed in 6945601254. Every poll now reads the first line, which names the run (a random UUID per start), before it treats an unchanged length as idle. A poll is idle only when the run and the length are both unchanged. A restart that rewrites the capture to the same number of bytes is now reported as a restart, and its records are read from the start. That holds for a DevTools restart in the same JVM too, so the file's modified time would not have been a reliable marker. An idle poll now costs one read of at most 256 bytes every 250 ms. The old test that an idle poll never opens the file is replaced.
Tests:
- `QueryTailFileReaderSpec`: a new run exactly as long as what was read is reported as restarted, with run `run-2` and its own records; the next poll is idle; appended records are read without a restart.
- `QueryTailRunnerSpec`: the restart stream feature now also covers a new run as long as the last one, and checks that the new run's records are shown as a new request.
Against the previous reader, both same-length cases fail and the longer-run case passes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@grails.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add the query-tail command to find query-heavy pages [grails-core]
Posted by "jdaugherty (via GitHub)" <gi...@apache.org>.
jdaugherty commented on PR #16631:
URL: https://github.com/apache/grails-core/pull/16631#issuecomment-6425522114
@jamesfredley, review 5657578936 is addressed in 52808b3641, still a single commit on 9.0.x.
- **Two session factories from one configuration:** each factory now gets its own inspector and capture handle, created when it is built. Closing one leaves the other recording, and a factory built after both have closed records into a new run. The configuration's statement inspector and observer are restored after every build, so the next build wraps what the application set. On Hibernate 5, closing one factory also no longer destroys the other's service registry. Details are on the thread.
The new feature in both versions' `HibernateConnectionSourceFactorySpec` builds two factories, closes one, and checks the other still records. The full `grails-data-hibernate5-core` and `grails-data-hibernate7-core` suites pass locally.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@grails.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add the query-tail command to find query-heavy pages [grails-core]
Posted by "testlens-app[bot] (via GitHub)" <gi...@apache.org>.
testlens-app[bot] commented on PR #16631:
URL: https://github.com/apache/grails-core/pull/16631#issuecomment-6433324724
## ✅ All tests passed ✅
🏷️ Commit: d791a6f6f45b82e1386369df6fab4760656a1e70
▶️ Tests: 139988 executed
⚪️ Checks: 97/97 completed
---
_Learn more about TestLens at [testlens.app/docs](https://testlens.app/docs/features/pr-comment/)._
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@grails.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add the sql-tail command to find query-heavy pages [grails-core]
Posted by "jdaugherty (via GitHub)" <gi...@apache.org>.
jdaugherty commented on PR #16631:
URL: https://github.com/apache/grails-core/pull/16631#issuecomment-6370434748
The sql-tail code now lives in packages each jar owns, so no package is nested in another jar: `org.apache.grails.data.core.sqltail` in `grails-datamapping-core`, `org.apache.grails.data.core.cli.sqltail` in its `-cli` artifact, and `org.apache.grails.data.hibernate5.core.sqltail` / `org.apache.grails.data.hibernate7.core.sqltail` in the Hibernate cores, following the `org.apache.grails.data.hibernate*.core` precedent in those modules. Every file the branch adds is Groovy under `@CompileStatic`; the 16 Java files (redactor, syntaxes, inspectors) were converted. Behaviour and tests are unchanged apart from the moves.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@grails.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add the sql-tail command to find query-heavy pages [grails-core]
Posted by "jamesfredley (via GitHub)" <gi...@apache.org>.
jamesfredley commented on code in PR #16631:
URL: https://github.com/apache/grails-core/pull/16631#discussion_r4362919822
##########
grails-datamapping-core/src/main/groovy/org/grails/datastore/gorm/jdbc/sqltail/H2SqlSyntax.java:
##########
@@ -0,0 +1,65 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * https://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.grails.datastore.gorm.jdbc.sqltail;
+
+/**
+ * H2: block comments nest, {@code //} starts a line comment, {@code $$} quotes a string without a
+ * tag, backticks delimit a name in every compatibility mode, and square brackets delimit a name in
+ * the SQL Server mode, where they are never a subscript, which is what a bracket after a name, a
+ * closing parenthesis or a closing bracket is in the other modes.
+ */
+final class H2SqlSyntax extends StandardSqlSyntax {
+
+ H2SqlSyntax() {
+ super("H2");
+ }
+
+ @Override
+ public int endOfComment(String sql, int index) {
+ return sql.startsWith("//", index) ? endOfLine(sql, index) : super.endOfComment(sql, index);
+ }
+
+ @Override
+ public int endOfValue(String sql, int index) {
+ int end = endOfDollarQuoted(sql, index, false);
+ return end != NONE ? end : super.endOfValue(sql, index);
+ }
+
+ @Override
+ public int endOfQuotedName(String sql, int index) {
+ char c = sql.charAt(index);
+ if (c == '`') {
+ return endOfQuoted(sql, index, '`', false, true);
+ }
+ if (c == '[' && !subscript(sql, index)) {
+ return endOfQuoted(sql, index, ']', false, false);
+ }
+ return super.endOfQuotedName(sql, index);
+ }
+
+ @Override
+ protected boolean nestedComments() {
+ return true;
+ }
+
+ private boolean subscript(String sql, int index) {
Review Comment:
A space before `[` is treated as a quoted identifier. `select ARRAY ['example-secret']` is valid H2 and the secret is copied into the capture unchanged. A comment between `ARRAY` and `[` does the same.
Do not treat brackets after whitespace as an identifier when they open an array. Or withhold the statement. Please assert that literal does not reach the file.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@grails.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add the query-tail command to find query-heavy pages [grails-core]
Posted by "jdaugherty (via GitHub)" <gi...@apache.org>.
jdaugherty commented on PR #16631:
URL: https://github.com/apache/grails-core/pull/16631#issuecomment-6379742915
Root cause of the red lanes, and what changed on the branch.
**Cause.** Every lane that failed died in `:grails-data-mongodb-core:compileTestGroovy` with `OutOfMemoryError: Java heap space`, and the heap dumps the diagnostic commit uploaded were all the same size (1,257 MB) even though `CompilePlugin` asks for `-Xmx2G`. The MongoDB modules' shared test scripts (`gradle/mongodb-test-config.gradle`, `gradle/mongodb-forked-test-config.gradle`) set `groovyOptions.forkOptions.jvmArgs = ['-Xmx768m']` on `compileTestGroovy`. Gradle builds the Groovy compiler daemon's command line from the javac-side `options.forkOptions.jvmArgs` (where the shared `-Xmx2G` lives) followed by the groovy-side list, and the last `-Xmx` wins, so these test compiles ran with a 768 MB heap while every other module's got 2 GB. Reproduced locally with a GC log on the fork: `Heap Max Capacity: 768M`, out of memory at 754 MB live, same 1,257 MB dump. The module's test sources now need about 1 GB to compile (this PR adds two specs, and 8.1.x's merge-up added two more), so a f
resh compiler fork always dies; a lane passed only when Gradle handed the task an idle compiler daemon another task had started with the larger heap, which is why the failing lanes moved around between runs.
**Fix.** 428ae475ba removes the override so the shared `-Xmx2G` applies, as it does for the other modules (the test forks keep their own `-Xmx1028M`). Verified locally: `Heap Max Capacity: 2G`, no full GCs, peak about 1.05 GB, compile succeeds. The cap dates from the mono-repo integration (a58aa145cf) and is on every release branch, so the same commit is on #16747 against 7.0.x.
**Branch layout.** 67ba541f11 is the feature. 5ed78b9dee (the `isDomainClass` cache that pinned every compiled class node) and 428ae475ba are compiler/build fixes that ride here only so this PR can go green; both are in #16747 and will be dropped from this branch once that merges up to 9.0.x. The temporary diagnostic commit is gone.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@grails.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add the query-tail command to find query-heavy pages [grails-core]
Posted by "testlens-app[bot] (via GitHub)" <gi...@apache.org>.
testlens-app[bot] commented on PR #16631:
URL: https://github.com/apache/grails-core/pull/16631#issuecomment-6380512469
## ✅ All tests passed ✅
🏷️ Commit: 428ae475badd5fd12b40aac502bc7b598b1a10a5
▶️ Tests: 138047 executed
⚪️ Checks: 96/96 completed
---
_Learn more about TestLens at [testlens.app/docs](https://testlens.app/docs/features/pr-comment/)._
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@grails.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add the query-tail command to find query-heavy pages [grails-core]
Posted by "jamesfredley (via GitHub)" <gi...@apache.org>.
jamesfredley commented on code in PR #16631:
URL: https://github.com/apache/grails-core/pull/16631#discussion_r4366402139
##########
grails-datamapping-core/src/main/groovy/org/apache/grails/data/core/querytail/H2SqlSyntax.groovy:
##########
@@ -0,0 +1,102 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * https://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.grails.data.core.querytail
+
+import groovy.transform.CompileStatic
+import groovy.transform.PackageScope
+
+/**
+ * H2: block comments nest, {@code //} starts a line comment, {@code $$} quotes a string without a
+ * tag, backticks delimit a name in every compatibility mode, and square brackets delimit a name in
+ * the SQL Server mode, where they are never a subscript, which is what a bracket after a name, a
+ * closing parenthesis or a closing bracket is in the other modes. A bracket after whitespace, or
+ * after a comment, is a name in the SQL Server mode but opens an array literal after the
+ * {@code ARRAY} keyword, which may stand any distance before it, so a statement with one is read
+ * both ways: a name, which is recorded as written, agrees with itself, while an array literal,
+ * whose elements are values, does not and withholds the statement.
+ */
+@CompileStatic
+@PackageScope
+final class H2SqlSyntax extends StandardSqlSyntax {
+
+ private final boolean bracketAfterWhitespaceIsName
+
+ H2SqlSyntax() {
+ this(true)
+ }
+
+ private H2SqlSyntax(boolean bracketAfterWhitespaceIsName) {
+ super('H2')
+ this.bracketAfterWhitespaceIsName = bracketAfterWhitespaceIsName
+ }
+
+ @Override
+ List<QueryTailSyntax> readings(String sql) {
+ if (!hasBracketAfterWhitespace(sql)) {
+ return List.<QueryTailSyntax>of(this)
+ }
+ List.<QueryTailSyntax>of(new H2SqlSyntax(true), new H2SqlSyntax(false))
+ }
+
+ @Override
+ int endOfComment(String sql, int index) {
+ sql.startsWith('//', index) ? endOfLine(sql, index) : super.endOfComment(sql, index)
+ }
+
+ @Override
+ int endOfValue(String sql, int index) {
+ int end = endOfDollarQuoted(sql, index, false)
+ end != NONE ? end : super.endOfValue(sql, index)
+ }
+
+ @Override
+ int endOfQuotedName(String sql, int index) {
+ char c = sql.charAt(index)
+ if (c == '`') {
+ return endOfQuoted(sql, index, '`' as char, false, true)
+ }
+ if (c == '[' && !subscript(sql, index) && (bracketAfterWhitespaceIsName || !afterWhitespace(sql, index))) {
+ return endOfQuoted(sql, index, ']' as char, false, false)
+ }
+ super.endOfQuotedName(sql, index)
+ }
+
+ @Override
+ protected boolean nestedComments() {
+ true
+ }
+
+ private boolean subscript(String sql, int index) {
+ char previous = previous(sql, index)
+ isNamePart(previous) || previous == ')' || previous == ']'
+ }
+
+ private static boolean afterWhitespace(String sql, int index) {
+ index > 0 && Character.isWhitespace(sql.charAt(index - 1))
+ }
+
+ private static boolean hasBracketAfterWhitespace(String sql) {
Review Comment:
This still looks only at the character before `[`. `select ARRAY/**/['example-secret']` has `/` there, so the span is copied as an identifier and the secret reaches the file. This commit does not change that method.
Treat a comment immediately before `[` as ambiguous, or withhold the statement. Please cover `ARRAY/**/['example-secret']` through the capture API.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@grails.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add the query-tail command to find query-heavy pages [grails-core]
Posted by "jdaugherty (via GitHub)" <gi...@apache.org>.
jdaugherty commented on PR #16631:
URL: https://github.com/apache/grails-core/pull/16631#issuecomment-6382037533
@jamesfredley re your reviews 5636821537 and 5637351665: the comment-before-bracket leak is fixed on the current head, 54a6574fc5. `H2SqlSyntax` now treats the two characters that close a block comment like whitespace before a `[` (`afterGap` replaces `afterWhitespace` in both the readings decision and `endOfQuotedName`), so `ARRAY/**/['example-secret']` and `ARRAY /* c */['example-secret']` get the name reading and the array-literal reading and are withheld. Covered in `QueryTailRedactorSpec` (withheld table and never-survives list, including `ARRAY -- c\n[...]`) and through the capture API in `QueryTailCaptureSpec`, where those forms are withheld while `[order] from [user]` is still recorded; the guide names the comment case. Details are in the replies on your two threads (https://github.com/apache/grails-core/pull/16631#discussion_r4366432130 and https://github.com/apache/grails-core/pull/16631#discussion_r4366432303), which I resolved; happy to reopen them if you prefer to clo
se them yourself.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@grails.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add the sql-tail command to find query-heavy pages [grails-core]
Posted by "jdaugherty (via GitHub)" <gi...@apache.org>.
jdaugherty commented on code in PR #16631:
URL: https://github.com/apache/grails-core/pull/16631#discussion_r4360312343
##########
grails-datamapping-core/src/main/groovy/org/grails/datastore/gorm/jdbc/sqltail/SqlTailRedactor.java:
##########
@@ -0,0 +1,414 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * https://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.grails.datastore.gorm.jdbc.sqltail;
+
+import java.util.ArrayList;
+import java.util.List;
+
+/**
+ * Takes the values out of a statement before it is captured. Every string, number and
+ * dollar-quoted literal becomes {@link #PLACEHOLDER} and every comment is dropped, so that a value
+ * written into the SQL text, a password in a native query or a key in a comment, never reaches the
+ * capture file. Only the recorded copy is redacted: the statement the database runs is not changed.
+ *
+ * <p>Databases disagree on parts of the lexical syntax: whether a backslash escapes a quote, whether
+ * comments nest, whether double quotes delimit an identifier or a string, whether a quote written
+ * twice escapes it or ends one string and starts another, whether a quote written three times
+ * opens a string that quotes can appear in, and whether {@code #} starts a comment. A statement is
+ * redacted under every reading its {@link Syntax} allows, and is recorded only when all of them give
+ * the same text. When they do not, or when a literal or a comment never ends, the statement is
+ * withheld: {@link #WITHHELD} is recorded in its place, never the statement.</p>
+ *
+ * @since 8.1
+ */
+public final class SqlTailRedactor {
+
+ /** What a literal is replaced with. */
+ public static final String PLACEHOLDER = "?";
+
+ /** What is recorded in place of a statement whose values cannot be told apart from its SQL. */
+ public static final String WITHHELD = "(statement withheld: its values could not be redacted)";
+
+ private static final boolean[] NO = {false};
+ private static final boolean[] EITHER = {false, true};
+
+ /**
+ * The lexical syntax of the database a statement runs on, as far as finding its values goes.
+ */
+ public enum Syntax {
+
+ /**
+ * Not known, or a database that can read double-quoted text as a string although it quotes
+ * identifiers with double quotes, such as SQLite and Informix: a statement is recorded only when
+ * the readings of every syntax below agree.
+ */
+ UNKNOWN,
+
+ /** Standard SQL, where double quotes delimit identifiers: H2, PostgreSQL, Oracle, DB2, HSQLDB and HANA. */
+ STANDARD,
+
+ /** MySQL and MariaDB: double quotes delimit strings, and {@code #} and {@code -- } start comments. */
+ MYSQL,
+
+ /**
+ * Microsoft SQL Server and Sybase: square brackets delimit identifiers, comments nest, and double
+ * quotes delimit a string when {@code QUOTED_IDENTIFIER} is off, as it is by default on Sybase.
+ */
+ TRANSACT_SQL,
+
+ /**
+ * GoogleSQL, the dialect of Cloud Spanner: backticks delimit identifiers, single and double quotes
+ * delimit strings, a string that opens with its quote written three times ends only at the quote
+ * written three times again and holds any other quote, a backslash always escapes the character
+ * after it, a quote written twice is two strings, and {@code #} starts a comment.
+ */
+ GOOGLE_SQL
+ }
+
+ private SqlTailRedactor() {
+ }
+
+ /**
+ * Redacts a statement.
+ *
+ * @param sql the statement
+ * @param syntax the syntax of the database it runs on
+ * @return the statement with its literals replaced and its comments dropped, or {@link #WITHHELD}
+ */
+ public static String redact(String sql, Syntax syntax) {
+ if (sql == null) {
+ return null;
+ }
+ String redacted = null;
+ for (Reading reading : readings(sql, syntax == null ? Syntax.UNKNOWN : syntax)) {
+ String candidate = reading.redact(sql);
+ if (candidate == null || (redacted != null && !redacted.equals(candidate))) {
+ return WITHHELD;
+ }
+ redacted = candidate;
+ }
+ return redacted;
+ }
+
+ /**
+ * The readings a statement has to agree under. A reading that depends on a character the
+ * statement does not contain is the same as the one that does not, so it is left out.
+ */
+ private static List<Reading> readings(String sql, Syntax syntax) {
+ boolean[] backslash = sql.indexOf('\\') >= 0 ? EITHER : NO;
+ int comment = sql.indexOf("/*");
+ boolean[] nesting = comment >= 0 && sql.indexOf("/*", comment + 2) >= 0 ? EITHER : NO;
+ List<Reading> readings = new ArrayList<>();
+ if (syntax == Syntax.STANDARD || syntax == Syntax.UNKNOWN) {
+ for (boolean backslashEscapes : backslash) {
+ for (boolean nestedComments : nesting) {
+ readings.add(new Reading(Syntax.STANDARD, backslashEscapes, nestedComments));
+ }
+ }
+ }
+ if (syntax == Syntax.MYSQL || syntax == Syntax.UNKNOWN) {
+ for (boolean backslashEscapes : backslash) {
+ readings.add(new Reading(Syntax.MYSQL, backslashEscapes, false));
+ }
+ }
+ if (syntax == Syntax.TRANSACT_SQL || syntax == Syntax.UNKNOWN) {
+ readings.add(new Reading(Syntax.TRANSACT_SQL, false, true));
+ }
+ if (syntax == Syntax.GOOGLE_SQL || syntax == Syntax.UNKNOWN) {
+ readings.add(new Reading(Syntax.GOOGLE_SQL, true, false));
+ }
+ return readings;
+ }
+
+ /**
+ * One way of reading a statement. MySQL and GoogleSQL delimit identifiers with backticks and start a
+ * comment with {@code #}; only MySQL needs whitespace after {@code --}; only Transact-SQL delimits
+ * identifiers with square brackets. Outside standard SQL, double quotes can delimit a string: in
+ * MySQL unless {@code ANSI_QUOTES} is set, in Transact-SQL when {@code QUOTED_IDENTIFIER} is off, and
+ * in GoogleSQL always. Either way the quoted run ends at the same place, so it is replaced as a
+ * value. A GoogleSQL string is read on its own terms: a quote written twice ends it, a backslash
+ * escapes, and a quote written three times opens a string that ends only at the next three.
+ *
+ * @param syntax the syntax read in, never {@link Syntax#UNKNOWN}
+ * @param backslashEscapes whether a backslash escapes the character after it in a string
+ * @param nestedComments whether a block comment can contain another
+ */
+ private record Reading(Syntax syntax, boolean backslashEscapes, boolean nestedComments) {
+
+ /**
+ * @return the redacted statement, or null when a literal, a quoted identifier or a comment never ends
+ */
+ String redact(String sql) {
+ boolean mysql = syntax == Syntax.MYSQL;
+ boolean googleSql = syntax == Syntax.GOOGLE_SQL;
+ boolean doubleQuotedValue = syntax != Syntax.STANDARD;
+ int length = sql.length();
+ StringBuilder out = new StringBuilder(length);
+ int i = 0;
+ while (i < length) {
+ char c = sql.charAt(i);
+ char next = i + 1 < length ? sql.charAt(i + 1) : 0;
+ char previous = i > 0 ? sql.charAt(i - 1) : 0;
+ int dollarTag = c == '$' && !isIdentifierPart(previous) ? dollarTagLength(sql, i) : 0;
+ int end;
+ if (googleSql && (c == '\'' || c == '"')) {
+ end = endOfGoogleSqlString(sql, i);
+ out.append(PLACEHOLDER);
+ }
+ else if (c == '\'') {
+ end = endOfQuotedLiteral(sql, i);
+ out.append(PLACEHOLDER);
+ }
+ else if (c == '"') {
+ end = endOfQuoted(sql, i, '"', mysql && backslashEscapes, true);
+ if (end >= 0) {
+ out.append(doubleQuotedValue ? PLACEHOLDER : sql.substring(i, end));
+ }
+ }
+ else if (c == '`' && (mysql || googleSql)) {
+ end = endOfQuoted(sql, i, '`', googleSql, mysql);
+ if (end >= 0) {
+ out.append(sql, i, end);
+ }
+ }
+ else if (c == '[' && syntax == Syntax.TRANSACT_SQL) {
+ end = endOfQuoted(sql, i, ']', false, true);
+ if (end >= 0) {
+ out.append(sql, i, end);
+ }
+ }
+ else if (c == '/' && next == '*') {
+ end = endOfBlockComment(sql, i);
+ out.append(' ');
+ }
+ else if ((c == '-' && next == '-' && (!mysql || i + 2 >= length || Character.isWhitespace(sql.charAt(i + 2)))) ||
+ (c == '#' && (mysql || googleSql))) {
+ end = endOfLine(sql, i);
+ out.append(' ');
+ }
+ else if (dollarTag > 0) {
+ String tag = sql.substring(i, i + dollarTag);
+ int close = sql.indexOf(tag, i + dollarTag);
+ end = close < 0 ? -1 : close + tag.length();
+ out.append(PLACEHOLDER);
+ }
+ else if ((isDigit(c) || (c == '.' && isDigit(next))) && !isIdentifierPart(previous)) {
+ end = endOfNumber(sql, i);
+ out.append(PLACEHOLDER);
+ }
+ else {
+ out.append(c);
+ end = i + 1;
+ }
+ if (end < 0) {
+ return null;
+ }
+ i = end;
+ }
+ return out.toString();
+ }
+
+ /**
+ * Ends a single-quoted literal, which may carry a prefix: {@code E'...'} always takes backslash
+ * escapes, and Oracle's {@code q'[...]'} ends at its own closing delimiter.
+ */
+ private int endOfQuotedLiteral(String sql, int start) {
+ char prefix = start > 0 ? sql.charAt(start - 1) : 0;
+ boolean standalonePrefix = start < 2 || !isIdentifierPart(sql.charAt(start - 2));
+ if ((prefix == 'q' || prefix == 'Q') && (standalonePrefix || isNationalPrefix(sql, start - 2)) &&
+ start + 1 < sql.length() && !Character.isWhitespace(sql.charAt(start + 1)) &&
+ sql.charAt(start + 1) != '\'') {
+ int close = sql.indexOf(closingDelimiter(sql.charAt(start + 1)) + "'", start + 2);
+ return close < 0 ? -1 : close + 2;
+ }
+ boolean escapeString = (prefix == 'e' || prefix == 'E') && standalonePrefix;
+ return endOfQuoted(sql, start, '\'', backslashEscapes || escapeString, true);
+ }
+
+ private int endOfBlockComment(String sql, int start) {
+ int depth = 1;
+ int i = start + 2;
+ while (i + 1 < sql.length()) {
+ char c = sql.charAt(i);
+ char next = sql.charAt(i + 1);
+ if (c == '*' && next == '/') {
+ depth--;
+ i += 2;
+ if (depth == 0) {
+ return i;
+ }
+ }
+ else if (nestedComments && c == '/' && next == '*') {
+ depth++;
+ i += 2;
+ }
+ else {
+ i++;
+ }
+ }
+ return -1;
+ }
+ }
+
+ /**
+ * Ends a quoted run.
+ *
+ * @param backslashEscapes whether a backslash escapes the character after it
+ * @param doubledClose whether the closing quote written twice stands for itself
+ * @return the index after the closing quote, or -1 when there is none
+ */
+ private static int endOfQuoted(String sql, int start, char close, boolean backslashEscapes, boolean doubledClose) {
+ int i = start + 1;
+ while (i < sql.length()) {
+ char c = sql.charAt(i);
+ if (backslashEscapes && c == '\\') {
+ i += 2;
+ }
+ else if (c == close) {
+ if (doubledClose && i + 1 < sql.length() && sql.charAt(i + 1) == close) {
+ i += 2;
+ }
+ else {
+ return i + 1;
+ }
+ }
+ else {
+ i++;
+ }
+ }
+ return -1;
+ }
+
+ /**
+ * Ends a GoogleSQL string, which a backslash escapes in and a quote written twice does not: that
+ * is an empty string, or the end of one string and the start of the next. A string that opens
+ * with its quote written three times is triple-quoted: it holds any quote, and ends at its quote
+ * written three times again.
+ *
+ * @return the index after the closing quote, or -1 when there is none
+ */
+ private static int endOfGoogleSqlString(String sql, int start) {
+ char quote = sql.charAt(start);
+ if (start + 2 < sql.length() && sql.charAt(start + 1) == quote && sql.charAt(start + 2) == quote) {
+ return endOfTripleQuoted(sql, start, quote);
+ }
+ return endOfQuoted(sql, start, quote, true, false);
+ }
+
+ private static int endOfTripleQuoted(String sql, int start, char quote) {
+ int i = start + 3;
+ while (i < sql.length()) {
+ char c = sql.charAt(i);
+ if (c == '\\') {
+ i += 2;
+ }
+ else if (c == quote && i + 2 < sql.length() && sql.charAt(i + 1) == quote && sql.charAt(i + 2) == quote) {
+ return i + 3;
+ }
+ else {
+ i++;
+ }
+ }
+ return -1;
+ }
+
+ private static int endOfLine(String sql, int start) {
+ int i = start;
+ while (i < sql.length() && sql.charAt(i) != '\n' && sql.charAt(i) != '\r') {
+ i++;
+ }
+ return i;
+ }
+
+ /**
+ * @return the length of the {@code $tag$} that opens a dollar-quoted string at the index, or 0
+ * when there is none: a dollar followed by a digit is a positional parameter
+ */
+ private static int dollarTagLength(String sql, int start) {
+ int i = start + 1;
+ while (i < sql.length()) {
+ char c = sql.charAt(i);
+ if (c == '$') {
+ return i + 1 - start;
+ }
+ boolean letter = Character.isLetter(c) || c == '_';
+ if (!letter && (i == start + 1 || !isDigit(c))) {
+ return 0;
+ }
+ i++;
+ }
+ return 0;
+ }
+
+ /**
+ * Ends a number: digits with a fraction and an exponent, or a hexadecimal number. Letters that
+ * follow it belong to it, so that nothing of a malformed number is left behind.
+ */
+ private static int endOfNumber(String sql, int start) {
+ int length = sql.length();
+ int i = start;
+ if (sql.charAt(i) == '0' && i + 1 < length && (sql.charAt(i + 1) == 'x' || sql.charAt(i + 1) == 'X')) {
+ i += 2;
+ }
+ else {
+ while (i < length && isDigit(sql.charAt(i))) {
+ i++;
+ }
+ if (i < length && sql.charAt(i) == '.') {
+ i++;
+ while (i < length && isDigit(sql.charAt(i))) {
+ i++;
+ }
+ }
+ if (i < length && (sql.charAt(i) == 'e' || sql.charAt(i) == 'E')) {
+ int exponent = i + 1 < length && (sql.charAt(i + 1) == '+' || sql.charAt(i + 1) == '-') ? i + 2 : i + 1;
+ if (exponent < length && isDigit(sql.charAt(exponent))) {
+ i = exponent;
+ }
+ }
+ }
+ while (i < length && isIdentifierPart(sql.charAt(i))) {
+ i++;
+ }
+ return i;
+ }
+
+ private static boolean isNationalPrefix(String sql, int index) {
+ return index >= 0 && (sql.charAt(index) == 'n' || sql.charAt(index) == 'N') &&
+ (index == 0 || !isIdentifierPart(sql.charAt(index - 1)));
+ }
+
+ private static String closingDelimiter(char open) {
+ return switch (open) {
+ case '[' -> "]";
+ case '(' -> ")";
+ case '{' -> "}";
+ case '<' -> ">";
+ default -> String.valueOf(open);
+ };
+ }
+
+ private static boolean isDigit(char c) {
+ return c >= '0' && c <= '9';
+ }
+
+ private static boolean isIdentifierPart(char c) {
Review Comment:
Fixed in 0b6079cff3. Whether `#` and `@` are part of a name (Oracle, DB2, Transact-SQL) or operators (PostgreSQL) is now a reading of standard SQL, taken both ways when a statement contains one, so `select account_id#987654321 from account` and `select @987654321 from t` are withheld under `STANDARD`, while `t.col#` and `emp@remote`, which read the same either way, are still recorded. In `TRANSACT_SQL` a dollar before digits is an amount: `$987654321`, `$.5` and `-$12.34` record as `?`, `?`, `-?`; `a$1`, `@v` and `#temp` stay names, and `account_id#987654321` stays the name it is there. A dollar before digits stays a parameter in standard SQL and a name in MySQL, so `select $987654321` is withheld before the dialect is known. Rows for each in `SqlTailRedactorSpec`.
##########
grails-datamapping-core/src/cli/groovy/org/grails/datastore/gorm/jdbc/sqltail/cli/SqlTailFileReader.groovy:
##########
@@ -0,0 +1,188 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * https://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.grails.datastore.gorm.jdbc.sqltail.cli
+
+import java.nio.charset.StandardCharsets
+
+import groovy.transform.CompileStatic
+
+import org.grails.datastore.gorm.jdbc.sqltail.SqlTailRecord
+
+/**
+ * Follows a capture file the way {@code tail -f} does: each poll returns the records written
+ * since the last one. A line that has no terminator yet is left for the next poll, so a record
+ * the application is still writing is never read in half. An application that starts again
+ * truncates the file and names its new run in the first line; when that line names another run,
+ * or the file shrinks, reading starts over from the beginning. The run tells a restart apart from
+ * a file that grew past where the reader was before it was next polled. A file truncated between
+ * being measured and being read, so that the read comes up short, is a restart too: it is read again
+ * from the beginning in the same poll.
+ *
+ * @since 8.1
+ */
+@CompileStatic
+class SqlTailFileReader implements Closeable {
+
+ private static final byte NEWLINE = (byte) 10
+
+ /** How much of the file the first line is looked for in; a header is far shorter. */
+ private static final int HEADER_LIMIT = 256
+
+ final File file
+ private long position
+ private String run
+ private boolean restarted
+
+ /**
+ * @param file the capture file
+ * @param fromStart whether to read what the file already holds, or only what is written from now on
+ */
+ SqlTailFileReader(File file, boolean fromStart) {
+ this.file = file
+ this.position = fromStart ? 0 : file.length()
+ this.run = file.exists() ? readRun() : null
+ }
+
+ /**
+ * @return the records written since the last poll, or since the reader was opened
+ */
+ List<SqlTailRecord> poll() {
+ restarted = false
+ // a read that comes up short is retried once, from the start; the next poll takes it from there
+ for (int attempt = 0; attempt < 2; attempt++) {
+ long length = file.length()
+ String current = length > 0 ? readRun() : null
+ if (length < position || (current != null && run != null && current != run)) {
+ position = 0
+ restarted = true
+ }
+ if (current != null || restarted) {
+ // a file truncated before its header was written names its run on a later poll
+ run = current
+ }
+ if (length <= position) {
+ return []
+ }
+ byte[] bytes = readFrom(position, length)
+ if (bytes.length == length - position) {
+ return recordsIn(bytes)
+ }
+ // the application truncated the file after it was measured: it has started again
+ position = 0
+ restarted = true
+ run = null
+ }
+ []
+ }
+
+ /**
+ * @return whether the last poll found a new run in the file, which an application restart writes
+ */
+ boolean isRestarted() {
+ restarted
+ }
+
+ /**
+ * @return how far the file has been read
+ */
+ long getPosition() {
+ position
+ }
+
+ /**
+ * @return the run that wrote the file, as its first line names it; null before the line is complete
+ */
+ String getRun() {
+ run
+ }
+
+ @Override
+ void close() {
+ // the file is opened and closed per poll
+ }
+
+ /**
+ * Takes the complete lines out of what was read, and leaves a line still being written for the next poll.
+ */
+ private List<SqlTailRecord> recordsIn(byte[] bytes) {
+ int consumed = lastLineEnd(bytes)
+ if (consumed == 0) {
+ return []
+ }
+ position += consumed
+ String text = new String(bytes, 0, consumed, StandardCharsets.UTF_8)
+ List<SqlTailRecord> records = []
+ for (String line : text.split('\n')) {
+ SqlTailRecord record = SqlTailRecord.parse(line)
+ if (record != null) {
+ records.add(record)
+ }
+ }
+ records
+ }
+
+ private static int lastLineEnd(byte[] bytes) {
+ for (int i = bytes.length - 1; i >= 0; i--) {
+ if (bytes[i] == NEWLINE) {
+ return i + 1
+ }
+ }
+ 0
+ }
+
+ private String readRun() {
+ byte[] head = readFrom(0, Math.min(file.length(), (long) HEADER_LIMIT))
+ for (int i = 0; i < head.length; i++) {
+ if (head[i] == NEWLINE) {
+ return SqlTailRecord.parseHeader(new String(head, 0, i, StandardCharsets.UTF_8))
+ }
+ }
+ null
+ }
+
+ /**
+ * @return the bytes between the offsets, or fewer when the file ends before the second one or is
+ * gone: it was truncated, or removed, after it was measured
+ */
+ private byte[] readFrom(long from, long to) {
Review Comment:
Fixed in 0b6079cff3. A poll now reads at most a chunk (1 MiB, `SqlTailFileReader.CHUNK`) and reports `behind` while the file holds more; the runner polls on without pausing while it does, and `--once` prints its summary only once the reader has caught up. A chunk without a line end is a line longer than a chunk: it is dropped up to its end, as a line that is not a record is, so the reader never allocates more than a chunk. `SqlTailFileReaderSpec` covers reading in chunks and the over-long line with a small chunk, and `SqlTailRunnerSpec` has `--once` on a capture several chunks long reporting every record.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@grails.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add the query-tail command to find query-heavy pages [grails-core]
Posted by "jdaugherty (via GitHub)" <gi...@apache.org>.
jdaugherty commented on code in PR #16631:
URL: https://github.com/apache/grails-core/pull/16631#discussion_r4369051875
##########
grails-datamapping-core/src/main/groovy/org/apache/grails/data/core/querytail/QueryTailRequestAttribution.groovy:
##########
@@ -0,0 +1,425 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * https://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.grails.data.core.querytail
+
+import java.lang.reflect.Method
+import java.nio.charset.StandardCharsets
+import java.util.concurrent.atomic.AtomicLong
+import java.util.regex.Pattern
+
+import groovy.transform.CompileStatic
+
+import org.springframework.util.ClassUtils
+import org.springframework.web.context.request.RequestAttributes
+import org.springframework.web.context.request.RequestContextHolder
+import org.springframework.web.context.request.ServletRequestAttributes
+
+/**
+ * Attributes a captured statement to the web request on the current thread. Grails keeps the
+ * request, and the controller and action it resolved, on the thread that serves it, so a statement
+ * run by an interceptor, the action or the view all land on the same page. The first statement of
+ * a request gives it an id that every later statement of that request repeats, which is how the
+ * {@code query-tail} command counts statements per request. Work on any other thread is attributed to
+ * the thread by name.
+ *
+ * <p>The request is recorded as its method and path, with the values in the path replaced by
+ * {@code ?}: the values of the parameters the URL mapping took from it and those of its query, found
+ * whether or not the client percent-encoded them, and any segment that is a number, a UUID or a long
+ * hexadecimal token, so that an id, a token or a key carried in a path never reaches the capture
+ * file. A value glued to static text in a segment, as a mapping such as {@code /reset/${token}_confirm}
+ * produces, is replaced within the segment. Until the URL mapping has run, as for a statement of a
+ * filter that runs before the controller is chosen, or for a request that never reaches one, no
+ * segment's role is known and every segment is replaced; only the controller the framework set on
+ * the request says that it has run, never a parameter, which a client can send.</p>
+ *
+ * <p>Spring Web is optional: without it on the classpath every statement is attributed to its
+ * thread.</p>
+ *
+ * @since 8.1
+ */
+@CompileStatic
+class QueryTailRequestAttribution {
+
+ /** The request attribute holding the id this attribution gave a request. */
+ public static final String REQUEST_ID_ATTRIBUTE = 'org.apache.grails.data.core.querytail.REQUEST_ID'
+
+ /** The request attribute Grails sets the controller name in. */
+ public static final String CONTROLLER_NAME_ATTRIBUTE = 'org.grails.CONTROLLER_NAME_ATTRIBUTE'
+
+ /** The request attribute Grails sets the controller namespace in. */
+ public static final String CONTROLLER_NAMESPACE_ATTRIBUTE = 'org.grails.CONTROLLER_NAMESPACE_ATTRIBUTE'
+
+ /** The request attribute Grails sets the action name in. */
+ public static final String ACTION_NAME_ATTRIBUTE = 'org.grails.ACTION_NAME_ATTRIBUTE'
+
+ private static final boolean SPRING_WEB_PRESENT = ClassUtils.isPresent(
+ 'org.springframework.web.context.request.RequestContextHolder', QueryTailRequestAttribution.classLoader)
+
+ private final AtomicLong requestCounter = new AtomicLong()
+ private final String requestIdPrefix
+
+ /**
+ * An attribution whose request ids are {@code r1}, {@code r2} and so on.
+ */
+ QueryTailRequestAttribution() {
+ this('')
+ }
+
+ /**
+ * @param requestIdPrefix what every request id starts with, which tells the requests of one JVM
+ * from those of another writing the same capture
+ */
+ QueryTailRequestAttribution(String requestIdPrefix) {
+ this.requestIdPrefix = requestIdPrefix ?: ''
+ }
+
+ /**
+ * Fills in the request id, page and request of a record from the current thread.
+ *
+ * @param record the record of a statement that just ran on this thread
+ * @return the same record
+ */
+ QueryTailRecord attribute(QueryTailRecord record) {
+ if (SPRING_WEB_PRESENT && WebRequestLookup.attribute(record, requestCounter, requestIdPrefix)) {
+ return record
+ }
+ record.requestId = null
+ record.request = null
+ record.page = QueryTailRecord.THREAD_PAGE_PREFIX + Thread.currentThread().name
+ record
+ }
+
+ /**
+ * Keeps every reference to Spring Web, in a signature as well as in a body, in a class that is only
+ * loaded once Spring Web is known to be present. The outer class can then be loaded, and reflected
+ * on, without Spring Web on the classpath.
+ */
+ @CompileStatic
+ private static class WebRequestLookup {
+
+ private static final String REQUEST_ID_PREFIX = 'r'
+
+ private static final String PLACEHOLDER = '?'
+
+ /** The parameters a URL mapping sets that name code rather than carry a value. */
+ private static final Set<String> MAPPING_PARAMETERS = ['controller', 'action', 'namespace', 'plugin', 'format'] as Set<String>
+
+ /** A segment that is a number, a UUID or a long hexadecimal token is a value whatever the mapping says. */
+ private static final Pattern VALUE_SHAPED = Pattern.compile(
+ '(?i)[0-9]+|[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}|[0-9a-f]{32,}')
+
+ private static final boolean SERVLET_PRESENT = ClassUtils.isPresent(
+ 'jakarta.servlet.http.HttpServletRequest', WebRequestLookup.classLoader)
+
+ /** The {@code getParams()} of a Grails web request, which holds the parameters its URL mapping took, by class. */
+ private static final ClassValue<Method> PARAMETERS = new ClassValue<Method>() {
+ @Override
+ protected Method computeValue(Class<?> type) {
+ try {
+ return type.getMethod('getParams')
+ }
+ catch (NoSuchMethodException ignored) {
+ return null
+ }
+ }
+ }
+
+ /**
+ * Fills in a record from the request on the current thread.
+ *
+ * @param record the record to fill in
+ * @param requestCounter the counter the next request id is taken from
+ * @param requestIdPrefix what every request id starts with
+ * @return whether there is a request on the current thread
+ */
+ static boolean attribute(QueryTailRecord record, AtomicLong requestCounter, String requestIdPrefix) {
+ RequestAttributes attributes = RequestContextHolder.getRequestAttributes()
+ if (attributes == null) {
+ return false
+ }
+ record.requestId = requestId(attributes, requestCounter, requestIdPrefix)
+ record.page = page(attributes)
+ record.request = SERVLET_PRESENT ? ServletRequestLookup.describe(attributes) : null
+ true
+ }
+
+ /**
+ * Builds the request line: the method and the path, with the values in the path, the parameters
+ * the URL mapping took from it among them, replaced.
+ *
+ * @param method the HTTP method
+ * @param path the request path
+ * @param attributes the request, whose mapped parameters are read when it is a Grails web request
+ * @param parameters the parameters of the servlet request, by name, each a {@code String[]}
+ */
+ static String describe(String method, String path, RequestAttributes attributes, Map<?, ?> parameters) {
+ // only the framework sets the controller attribute; a client can send a controller parameter
+ if (attributes.getAttribute(CONTROLLER_NAME_ATTRIBUTE, RequestAttributes.SCOPE_REQUEST) == null) {
+ return "${method} ${withholdPath(path)}".toString()
+ }
+ Set<String> values = new HashSet<>()
+ addValues(values, mappedParameters(attributes))
+ addValues(values, parameters)
+ "${method} ${redactPath(path, values)}".toString()
+ }
+
+ /**
+ * Replaces every segment of a path: before the URL mapping has run, nothing tells a literal
+ * segment from a token, so only the shape of the path is recorded.
+ */
+ private static String withholdPath(String path) {
+ String[] segments = path.split('/', -1)
+ StringBuilder out = new StringBuilder(path.length())
+ for (int i = 0; i < segments.length; i++) {
+ if (i > 0) {
+ out.append('/')
+ }
+ out.append(segments[i].isEmpty() ? segments[i] : PLACEHOLDER)
+ }
+ out.toString()
+ }
+
+ private static String requestId(RequestAttributes attributes, AtomicLong requestCounter, String requestIdPrefix) {
+ synchronized (attributes) {
+ Object existing = attributes.getAttribute(REQUEST_ID_ATTRIBUTE, RequestAttributes.SCOPE_REQUEST)
+ if (existing != null) {
+ return existing.toString()
+ }
+ String id = requestIdPrefix + REQUEST_ID_PREFIX + requestCounter.incrementAndGet()
+ attributes.setAttribute(REQUEST_ID_ATTRIBUTE, id, RequestAttributes.SCOPE_REQUEST)
+ return id
+ }
+ }
+
+ private static String page(RequestAttributes attributes) {
+ Object controller = attributes.getAttribute(CONTROLLER_NAME_ATTRIBUTE, RequestAttributes.SCOPE_REQUEST)
+ if (controller == null) {
+ return null
+ }
+ Object namespace = attributes.getAttribute(CONTROLLER_NAMESPACE_ATTRIBUTE, RequestAttributes.SCOPE_REQUEST)
+ Object action = attributes.getAttribute(ACTION_NAME_ATTRIBUTE, RequestAttributes.SCOPE_REQUEST)
+ StringBuilder page = new StringBuilder()
+ if (namespace != null) {
+ page.append(namespace).append(':')
+ }
+ page.append(controller)
+ if (action != null) {
+ page.append('/').append(action)
+ }
+ page.toString()
+ }
+
+ /**
+ * @return the parameters of a Grails web request, which include those its URL mapping took from
+ * the path, or null for a request that is not one
+ */
+ private static Map<?, ?> mappedParameters(RequestAttributes attributes) {
+ Method parameters = PARAMETERS.get(attributes.getClass())
+ if (parameters == null) {
+ return null
+ }
+ try {
+ Object value = parameters.invoke(attributes)
+ return value instanceof Map ? (Map<?, ?>) value : null
+ }
+ catch (ReflectiveOperationException | RuntimeException ignored) {
+ return null
+ }
+ }
+
+ private static void addValues(Set<String> values, Map<?, ?> parameters) {
+ if (parameters == null) {
+ return
+ }
+ for (Map.Entry<?, ?> entry in parameters.entrySet()) {
+ if (MAPPING_PARAMETERS.contains(String.valueOf(entry.key))) {
+ continue
+ }
+ Object value = entry.value
+ if (value instanceof CharSequence) {
+ values.add(value.toString())
+ }
+ else if (value instanceof Object[]) {
+ for (Object element in (Object[]) value) {
+ if (element instanceof CharSequence) {
+ values.add(element.toString())
+ }
+ }
+ }
+ }
+ }
+
+ /**
+ * Replaces the values in a path: a segment that is one of the values, with or without an
+ * extension, a run of segments that spells a value holding slashes, any segment shaped like a
+ * value, and a value glued to static text within a segment. The segments are matched as the
+ * client sent them and percent-decoded, since the parameters are decoded while the request URI
+ * is not; the path is written as sent, except that a segment a value was cut out of is written
+ * in the form it was found in.
+ */
+ private static String redactPath(String path, Set<String> values) {
+ String[] segments = path.split('/', -1)
+ String[] decoded = new String[segments.length]
+ boolean[] redacted = new boolean[segments.length]
+ for (int i = 0; i < segments.length; i++) {
+ decoded[i] = decode(segments[i])
+ redacted[i] = isValue(segments[i], values) || isValue(decoded[i], values)
+ }
+ for (String value in values) {
+ if (value.contains('/')) {
+ String[] run = value.split('/', -1)
+ markRun(segments, redacted, run)
+ markDecodedRun(decoded, redacted, run)
+ }
+ }
+ StringBuilder out = new StringBuilder(path.length())
+ for (int i = 0; i < segments.length; i++) {
+ if (i > 0) {
+ out.append('/')
+ }
+ out.append(redacted[i] ? PLACEHOLDER : withoutValues(segments[i], decoded[i], values))
+ }
+ out.toString()
+ }
+
+ /**
+ * Cuts the values out of a segment that is not one itself, as a mapping that glues static text
+ * to a variable produces: a value of three or more characters wherever it appears, a shorter one
+ * only where no letter or digit continues it on either side, so that a one-letter query value
+ * does not pick holes in every word of the path.
+ *
+ * @return the segment with the values replaced, in the form, as sent or decoded, they were found in
+ */
+ private static String withoutValues(String segment, String decoded, Set<String> values) {
+ String cut = cutValues(segment, values)
+ if (cut != segment) {
Review Comment:
Fixed in 49a8b1d769 (folded into the feature commit). A segment a value is glued into is now replaced whole: `redactPath` writes `?` when either the segment as sent or its percent-decoded form holds a value (`holdsValue`, same bounding rule as before: a value of three or more characters anywhere, a shorter one only where no letter or digit continues it), instead of cutting the value out of whichever form matched first. So `/reset/%6B9fQ2x_confirm?q=confirm` with `token=k9fQ2x` is recorded as `GET /reset/?`, and the encoded token never reaches the file. Covered through `attribute()` in the table of `QueryTailRequestAttributionSpec`: that request, the same path without the `q` parameter, and the glued cases from the last round, which now all record the whole segment as `?`; the guide and javadoc say a segment a value is glued into is replaced whole.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@grails.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add the query-tail command to find query-heavy pages [grails-core]
Posted by "testlens-app[bot] (via GitHub)" <gi...@apache.org>.
testlens-app[bot] commented on PR #16631:
URL: https://github.com/apache/grails-core/pull/16631#issuecomment-6401104660
## ✅ All tests passed ✅
🏷️ Commit: 69456012547bb697a618f27b1db02f6093b1480d
▶️ Tests: 127006 executed
⚪️ Checks: 97/97 completed
---
_Learn more about TestLens at [testlens.app/docs](https://testlens.app/docs/features/pr-comment/)._
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@grails.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add the query-tail command to find query-heavy pages [grails-core]
Posted by "testlens-app[bot] (via GitHub)" <gi...@apache.org>.
testlens-app[bot] commented on PR #16631:
URL: https://github.com/apache/grails-core/pull/16631#issuecomment-6406139831
## ✅ All tests passed ✅
🏷️ Commit: 217fe2435cd2d934854d8a0c1a89f4307d3795c7
▶️ Tests: 139025 executed
⚪️ Checks: 97/97 completed
---
_Learn more about TestLens at [testlens.app/docs](https://testlens.app/docs/features/pr-comment/)._
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@grails.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add the query-tail command to find query-heavy pages [grails-core]
Posted by "testlens-app[bot] (via GitHub)" <gi...@apache.org>.
testlens-app[bot] commented on PR #16631:
URL: https://github.com/apache/grails-core/pull/16631#issuecomment-6454700026
## ✅ All tests passed ✅
🏷️ Commit: cf2b357fd38921c8a861ba7d1e35af46a1ab02a3
▶️ Tests: 140676 executed
⚪️ Checks: 97/97 completed
---
_Learn more about TestLens at [testlens.app/docs](https://testlens.app/docs/features/pr-comment/)._
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@grails.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add the query-tail command to find query-heavy pages [grails-core]
Posted by "jdaugherty (via GitHub)" <gi...@apache.org>.
jdaugherty commented on PR #16631:
URL: https://github.com/apache/grails-core/pull/16631#issuecomment-6412546359
@jamesfredley, review 5646564068 is addressed in 6306f94ef7. It is still a single commit on 9.0.x.
- **Closing the last writer:** releasing the file lock and removing the open entry now happen together, so a capture opened on the same path cannot join a writer that is closing. Details are on the thread.
The same commit also includes:
- **Request ids per capture file:** each capture file now numbers its own requests. Before, a request recorded to two files took its id from one file's counter, so the other file could give its next request the same id.
- **MongoDB implicit equality:** a document or array compared without an operator, as in `{"attrs": {"nested": {...}}}`, is now recorded as data at every level. This covers `filter`, `query`, `q`, `$match`, `restrictSearchWithMatch`, `arrayFilters`, `partialFilterExpression` and validators, so a key inside it is never kept as an operator, even one that starts with `$`.
- **GeoJSON geometries:** the operand of `$geometry` is recorded as data, so extra fields beside `type` and `coordinates` are not recorded.
- **Failed startup:** if the MongoDB client settings turn out to be invalid, the capture opened for that connection is closed. If a Neo4j capture cannot open, the new driver and any embedded server are closed before the failure is reported, so a corrected startup can reopen the same store.
Each item has tests, and the query-tail guide is updated. Locally, `:grails-datamapping-core:check` passes (2527 tests plus 128 CLI tests), as do the MongoDB and Neo4j query-tail specs, CodeNarc and Checkstyle.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@grails.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add the query-tail command to find query-heavy pages [grails-core]
Posted by "jamesfredley (via GitHub)" <gi...@apache.org>.
jamesfredley commented on code in PR #16631:
URL: https://github.com/apache/grails-core/pull/16631#discussion_r4376670157
##########
grails-datamapping-core/src/main/groovy/org/apache/grails/data/core/querytail/QueryTailCapture.groovy:
##########
@@ -0,0 +1,368 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * https://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.grails.data.core.querytail
+
+import java.nio.ByteBuffer
+import java.nio.channels.FileChannel
+import java.nio.channels.FileLock
+import java.nio.channels.OverlappingFileLockException
+import java.nio.charset.StandardCharsets
+import java.nio.file.Files
+import java.nio.file.StandardOpenOption
+import java.util.concurrent.atomic.AtomicBoolean
+import java.util.concurrent.atomic.AtomicLong
+
+import groovy.transform.CompileStatic
+import groovy.util.logging.Slf4j
+
+/**
+ * Writes the statements an application runs to the capture file the {@code query-tail} command
+ * tails. Each session factory, Neo4j driver or MongoDB client that captures opens its own capture
+ * and closes it when it closes; the captures of one file share it, so every data source of an
+ * application writes to the one file. The file is truncated when the first capture opens it, so it
+ * holds the statements of the current run only, and its first line names that run. It is closed
+ * when the last capture closes, and opened afresh by the next one. Each statement is written whole
+ * and at once, so that the command sees it as soon as it is recorded.
+ *
+ * <p>JVMs that run at the same time with capture enabled, such as the parallel forks of a test task,
+ * share the file: a lock file beside it tells the first to start, which truncates the file and names
+ * the run, from the others, which append to that run without truncating it and number their
+ * requests apart, so that the command reads one run from all of them.</p>
+ *
+ * <p>A statement is recorded without its values: {@link QueryTailRedactor} replaces its literals and
+ * drops its comments before it is written, in the {@link #syntax} of the database it runs on. A
+ * statement that is not text, such as a MongoDB command, has its values taken out by whoever records
+ * it, through {@link #recordRedacted}.</p>
+ *
+ * <p>A capture is only opened when {@link QueryTailSettings#SETTING_ENABLED} is set; an application
+ * started without it never reaches this class.</p>
+ *
+ * @since 8.1
+ */
+@CompileStatic
+@Slf4j
+class QueryTailCapture implements Closeable {
+
+ private static final Map<String, CaptureFile> OPEN = new HashMap<>()
+
+ /** The file the statements are written to. */
+ final File file
+
+ /** The syntax of the database the statements of this capture run on, which their values are found by. */
+ volatile QueryTailSyntax syntax = QueryTailSyntax.UNKNOWN
+
+ private final CaptureFile captureFile
+ private final AtomicBoolean closed = new AtomicBoolean()
+
+ private QueryTailCapture(File file, CaptureFile captureFile) {
+ this.file = file
+ this.captureFile = captureFile
+ }
+
+ /**
+ * Opens a capture of a file. The first capture of a file truncates it; a capture opened while
+ * another of the same file is open shares it. Every capture opened must be closed.
+ *
+ * @param file the capture file
+ * @return the capture
+ * @throws UncheckedIOException when the file cannot be created
+ */
+ static QueryTailCapture open(File file) {
+ String key
+ try {
+ key = file.canonicalPath
+ }
+ catch (IOException e) {
+ throw new UncheckedIOException("Cannot resolve the query-tail capture file ${file}: ${e.message}", e)
+ }
+ synchronized (OPEN) {
+ CaptureFile captureFile = OPEN.get(key)
+ if (captureFile == null) {
+ captureFile = new CaptureFile(file, key)
+ OPEN.put(key, captureFile)
+ if (captureFile.joined) {
+ log.info('Capturing statements to {} for query-tail, in the run of another JVM writing it', file)
+ }
+ else {
+ log.info('Capturing statements to {} for query-tail', file)
+ }
+ }
+ captureFile.captures++
+ return new QueryTailCapture(file, captureFile)
+ }
+ }
+
+ /**
+ * Opens the capture the settings name.
+ *
+ * @param settings the query-tail settings
+ * @return the capture
+ */
+ static QueryTailCapture open(QueryTailSettings settings) {
+ open(settings.resolveFile())
+ }
+
+ /**
+ * Records a statement that is about to run on the current thread, without its values.
+ *
+ * @param dataSource the connection source name the statement runs against
+ * @param sql the statement
+ */
+ void record(String dataSource, String sql) {
+ write(dataSource, QueryTailRedactor.redact(sql, syntax))
+ }
+
+ /**
+ * Records a statement that is about to run on the current thread and that holds no value any
+ * more: a MongoDB command, which is a document rather than text, is redacted field by field by
+ * the listener that records it. Nothing of the statement is changed here.
+ *
+ * @param dataSource the connection source name the statement runs against
+ * @param statement the statement, with its values already taken out
+ */
+ void recordRedacted(String dataSource, String statement) {
+ write(dataSource, statement)
+ }
+
+ private void write(String dataSource, String statement) {
+ if (closed.get()) {
+ return
+ }
+ QueryTailRecord record = new QueryTailRecord(timestamp: System.currentTimeMillis(), dataSource: dataSource,
+ sql: statement)
+ captureFile.write(captureFile.attribution.attribute(record))
+ }
+
+ /**
+ * @return whether this capture is still writing to its file
+ */
+ boolean isOpen() {
+ !closed.get() && captureFile.open
+ }
+
+ /**
+ * Closes this capture. The file is closed once every capture of it is.
+ */
+ @Override
+ void close() {
+ if (!closed.compareAndSet(false, true)) {
+ return
+ }
+ synchronized (OPEN) {
+ if (--captureFile.captures == 0) {
+ // A replacement must not join this writer's run before its coordination lock is released.
+ captureFile.close()
+ if (OPEN.get(captureFile.key).is(captureFile)) {
+ OPEN.remove(captureFile.key)
+ }
+ }
+ }
+ }
+
+ /**
+ * A capture file and the captures writing to it. The file is written by one channel, a line at a
+ * time, and the requests of every data source are numbered by one attribution, so that two
+ * requests never share an id. A JVM that joins the run of another numbers the requests of every run
+ * it joins by one counter, so that it repeats none of its ids when it closes its capture and joins
+ * the same run again.
+ *
+ * <p>A lock file beside the capture coordinates the JVMs writing it. Every writer holds a shared
+ * lock on the {@link #WRITING} byte of the lock file while it writes. A starting writer first
+ * takes the {@link #STARTING} byte exclusively, so that starters take turns, then tries the
+ * {@link #WRITING} byte exclusively: when it gets it, no other JVM is writing, and it truncates the
+ * file and writes the header of a new run before it lets the next starter in, so that the header
+ * is the first line of the run whatever a joiner writes after it; when it does not, it joins the
+ * run being written. Either way it takes the {@link #WRITING} byte shared before it lets go of
+ * {@link #STARTING} and holds it until it closes. A lock file that cannot be created, as beside a
+ * device, leaves the writer uncoordinated, as the only one.</p>
+ */
+ @CompileStatic
+ private static class CaptureFile {
+
+ /** The byte of the lock file a starting writer holds, exclusively, while it decides whether to truncate. */
+ private static final long STARTING = 0L
+
+ /** The byte of the lock file every writer holds, shared, while it writes. */
+ private static final long WRITING = 1L
+
+ /**
+ * Numbers the requests of every run this JVM joins, so that a JVM that closes its capture and
+ * joins the same run again never repeats an id it gave a request of that run before.
+ */
+ private static final AtomicLong JOINED_REQUESTS = new AtomicLong()
+
+ final File file
+ final String key
+ final QueryTailRequestAttribution attribution
+
+ /** Whether this JVM joined the run of another JVM writing the file, rather than starting one. */
+ final boolean joined
+
+ /** How many captures are open on the file; guarded by {@link QueryTailCapture#OPEN}. */
+ int captures
+
+ private FileChannel channel
+ private FileChannel lockChannel
+ private FileLock writing
+
+ CaptureFile(File file, String key) {
+ this.file = file
+ this.key = key
+ File parent = file.absoluteFile.parentFile
+ boolean first
+ try {
+ if (parent != null && !parent.isDirectory() && !parent.mkdirs() && !parent.isDirectory()) {
+ throw new IOException("Cannot create the query-tail capture directory ${parent}")
+ }
+ lockChannel = openLockFile(file)
+ FileLock starting = lockChannel == null ? null : lockChannel.lock(STARTING, 1L, false)
+ try {
+ FileLock alone = lockChannel == null ? null : lock(lockChannel, false, true)
+ first = lockChannel == null || alone != null
+ alone?.release()
+ if (first) {
+ Files.newByteChannel(file.toPath(), StandardOpenOption.CREATE, StandardOpenOption.WRITE,
+ StandardOpenOption.TRUNCATE_EXISTING).close()
+ }
+ channel = FileChannel.open(file.toPath(), StandardOpenOption.CREATE, StandardOpenOption.WRITE,
+ StandardOpenOption.APPEND)
+ writing = lockChannel == null ? null : lock(lockChannel, true, false)
+ if (first) {
+ writeLine(QueryTailRecord.header(UUID.randomUUID().toString()))
+ }
+ }
+ finally {
Review Comment:
If the first header write fails, `writeLine()` closes the lock channel and then this release throws `ClosedChannelException`. `open()` throws instead of returning a stopped capture. The new write-failure test hits this when the lock file can be created.
Release the startup lock before closing its channel, or skip a lock that is already invalid.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@grails.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add the query-tail command to find query-heavy pages [grails-core]
Posted by "testlens-app[bot] (via GitHub)" <gi...@apache.org>.
testlens-app[bot] commented on PR #16631:
URL: https://github.com/apache/grails-core/pull/16631#issuecomment-6420854023
## 🔎 No tests executed 🔎
🏷️ Commit: e4090dfa348b077154d68bbc9fd03f34bccc171b
▶️ Tests: 0 executed
⚪️ Checks: 10/10 completed
---
_Learn more about TestLens at [testlens.app/docs](https://testlens.app/docs/features/pr-comment/)._
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@grails.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add the query-tail command to find query-heavy pages [grails-core]
Posted by "jdaugherty (via GitHub)" <gi...@apache.org>.
jdaugherty commented on PR #16631:
URL: https://github.com/apache/grails-core/pull/16631#issuecomment-6420856186
@jamesfredley, I pushed e4090dfa34 over d0011c6b80, still a single commit on 9.0.x. It replaces how MongoDB commands are recognized.
d0011c6b80 recognized a command by its shape: a name, a namespace without `(`, and a document opening with `{"`. A collection name can contain almost any character, and a command with no options left after redaction, such as `find db.wide collection`, has no document at all. Commands like these were read as SQL, so for example `find db.book(archive)` with the field names `a" b` and `a" b` was recorded as one statement.
No pattern can tell such a command from SQL, so the capture no longer guesses. A statement recorded through `recordRedacted`, which is how the MongoDB listener records commands, is marked as a document. The record line carries that in a field before the statement: `document` or `-`. A document is written exactly as recorded, escaping only tabs, line breaks and backslashes, and read back as one. The CLI shows a document as recorded, through `StatementCollapser.statementOf(record)` in both the summary and the stream, and never normalizes or collapses it as SQL. The shape pattern is gone.
Tests:
- **`QueryTailRecordSpec`:** documents with escaped quotes, parentheses in a collection name, spaced collection names with no options, and tabs or line breaks round-trip through the new field. A statement that only looks like a command is still written as text.
- **`StatementCollapserSpec` and `QueryTailSummarySpec`:** cover the same shapes, including `insert db.book(archive) {...}`, which SQL collapsing would cut at the `(`. Each is counted once, as recorded.
- **`QueryTailCommandListenerSpec`:** against MongoDB, runs the field names in `book` and in a collection named with parentheses, and bare `find`s on two collections whose names differ only in their spaces. The database tells each pair apart, and so does the capture.
- With `recordRedacted` not marking documents, the new summary and MongoDB features fail.
`:grails-datamapping-core:check` passes locally, as do the MongoDB, Neo4j, H5 and H7 query-tail specs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@grails.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add the query-tail command to find query-heavy pages [grails-core]
Posted by "testlens-app[bot] (via GitHub)" <gi...@apache.org>.
testlens-app[bot] commented on PR #16631:
URL: https://github.com/apache/grails-core/pull/16631#issuecomment-6444852962
## ✅ All tests passed ✅
🏷️ Commit: 71a04557612f298fbe22480e7991e2462d66350a
▶️ Tests: 140198 executed
⚪️ Checks: 97/97 completed
---
_Learn more about TestLens at [testlens.app/docs](https://testlens.app/docs/features/pr-comment/)._
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@grails.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add the query-tail command to find query-heavy pages [grails-core]
Posted by "jdaugherty (via GitHub)" <gi...@apache.org>.
jdaugherty commented on PR #16631:
URL: https://github.com/apache/grails-core/pull/16631#issuecomment-6429631970
@jamesfredley, review 5660053162 is addressed, and the head is now 6b133f1c64. The feature commit was rebased onto 9.0.x (9549579958) unchanged as b7ff57e508, and two commits follow it:
**6b133f1c64: default action.** A request to `/book` keeps its `before` interceptor's statements on `book/index`; see the reply in the thread.
**bc9f326645: hardening.**
- **MongoDB restore**: a client that is not a `RestartableMongoClient` used to leak the capture of its replacement's listener after a CRaC restore, and recorded with no mapping bound. Now the replacement's listener takes the place of the closed client's, which is closed, and keeps its mapped fields. A connection source of another kind closes it at once. `createMongoClient` closes the capture nothing could own.
- **Unwritable capture file**: the file, or its directory, cannot be created. This used to stop Hibernate, MongoDB and embedded Neo4j from starting. Now it is logged as a warning, and the capture records nothing. The Neo4j driver and embedded-server cleanup that existed only for that failure is gone.
- **Relative `grails.gorm.queryTail.file`**: now joined to `base.dir` when it is set, as the default file and the command already were, so an application run from another working directory writes where `grails query-tail` reads.
- **Test runs**: the Gradle plugin tells every test JVM the build's run, not only when query-tail settings are Gradle properties, so capture enabled for the test environment in `application.yml` adds up to one run. A run given as `-Pgrails.gorm.queryTail.run` is passed as it is.
- **Extensions**: the extension of a last segment kept by its name is now recorded as `?`, e.g. `/book/show.json` → `/book/show.?`, since a client chooses it as it chooses a format.
- **MongoDB stored data**: unmapped keys at the top of stored data (inserted documents, replacement documents, the fields an update writes, and the operands of `$literal`, `$const` and `$documents`) are now `?`, as a dynamic attribute's name may be data. Condition, pipeline, sort and projection keys are still kept.
- **Mapped fields**: the entities of each collection are gathered again only when the mapping context reports a new entity, instead of a queue walk per key.
- **Neo4j**: a `Driver` handed to the datastore is captured. Its settings are now bound with `grails.gorm` fallbacks, as a driver GORM builds already was.
- **Locking**: `isOpen()` no longer takes the write lock. Opening a file waits on another JVM's starting writer under a per-file lock, not the one every open and close in the JVM takes.
- **Thread work**: `query-tail` lists threads after the pages of requests, with `-` for requests and averages, and no longer counts them as requests, so `BootStrap` or a test worker cannot top the ranking.
- **API cleanup**: removed the test-only `QueryTailSettings.from(...)`, `QueryTailCapture.open(QueryTailSettings)` and `recordRedacted(String, String)`. The Hibernate 5 and 7 inspectors now share `QueryTailRecorder` for the capture lifecycle.
The guide and the command reference are updated to match.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@grails.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add the query-tail command to find query-heavy pages [grails-core]
Posted by "testlens-app[bot] (via GitHub)" <gi...@apache.org>.
testlens-app[bot] commented on PR #16631:
URL: https://github.com/apache/grails-core/pull/16631#issuecomment-6434444549
## ✅ All tests passed ✅
🏷️ Commit: bd2ec2d99d236052335796b67f7bab77499b0281
▶️ Tests: 140020 executed
⚪️ Checks: 97/97 completed
---
_Learn more about TestLens at [testlens.app/docs](https://testlens.app/docs/features/pr-comment/)._
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@grails.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add the query-tail command to find query-heavy pages [grails-core]
Posted by "jamesfredley (via GitHub)" <gi...@apache.org>.
jamesfredley commented on code in PR #16631:
URL: https://github.com/apache/grails-core/pull/16631#discussion_r4397676365
##########
grails-data-mongodb/core/src/main/groovy/org/apache/grails/data/mongodb/core/querytail/MongoCommandRedactor.groovy:
##########
@@ -0,0 +1,574 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * https://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.grails.data.mongodb.core.querytail
+
+import groovy.transform.CompileStatic
+import org.bson.BsonDocument
+import org.bson.BsonValue
+
+import org.apache.grails.datamapping.core.querytail.QueryTailRedactor
+
+/**
+ * Takes the values out of a MongoDB command before it is captured. A command is a document, so
+ * there is nothing to parse: every value becomes {@link QueryTailRedactor#PLACEHOLDER}, and the
+ * keys, which are the options of the command, its operators and the field names of the collection,
+ * are kept, except where a key may be data. The keys of a document stored in a map property, or in
+ * a property or a collection the {@linkplain MongoMappedFields mapping} does not know, are recorded
+ * as the placeholder too, and so is every segment of a dotted path past such a key. So is a key at
+ * the top of stored data, the documents an insert writes, a replacement document, the fields an
+ * update writes and the operand of a literal, unless the mapping knows it, as the name of a dynamic
+ * attribute may be data; the keys at the top of a condition, a pipeline, a sort or a projection
+ * are the fields the application asks for, and are kept. An update that is a pipeline writes the
+ * documents its stages build: the keys at the top of what a {@code $set}, {@code $addFields} or
+ * {@code $project} stage holds, of the new root of a {@code $replaceWith} or {@code $replaceRoot}, and of a
+ * document {@code $mergeObjects} merges into it are fields written, kept only when the mapping knows
+ * them, as those of a classic update are. A numeric key
+ * is kept only where it indexes an array the mapping knows; in a map, in an element of an array,
+ * which is a map or a document unless the field is a list of lists, or at the top of a document,
+ * it is data. A {@code $} key is kept as an operator only where operators go, in a condition, a
+ * pipeline and the operators of an update, and in a condition only when it is one of MongoDB's query
+ * operators: any other, such as a key beside the geometry of {@code $near}, is recorded as the
+ * placeholder, and so is everything under it; in stored data, the documents an insert writes, a
+ * replacement document and the values an update operator writes, it is a field name, which MongoDB
+ * allows to start with {@code $}, and is recorded as the placeholder unless the mapping knows it.
+ * So is a {@code $} key in the operand of a comparison in a condition, such as {@code $eq} or
+ * {@code $in}, or an implicit document or array equality, which holds the value the field is compared
+ * with, and in a DBRef, whose {@code $ref}, {@code $id} and {@code $db} a condition reads as fields
+ * compared with their whole value, in any order; in an aggregation expression, which
+ * {@code $expr}, the {@code let} of a command and the stages that compute fields hold, the operands
+ * of a comparison are expressions, and their operators are kept, except in the operand of a
+ * {@code $literal} or {@code $const}, which is stored data. A comment, and the constants of an
+ * update, and the scope of a map-reduce command, may be any document, and none of their keys are
+ * kept. The entries the driver adds for its
+ * session, the cluster time and the transaction, which say nothing about the command, are left out.
+ *
+ * <p>An array is recorded with each of its different elements once: an element recorded as one before
+ * it in the array is left out, so that the documents of a batch and the values of an {@code $in} are
+ * recorded the same whatever their number, one or many, and a command is counted as the one statement
+ * it is: {@code {"$in": [?]}}. An empty array is recorded as {@code []}.</p>
+ *
+ * <p>A command is recorded as its name, the database and collection it runs against, and the
+ * document of its remaining entries: {@code find shop.book {"filter": {"title": ?}, "limit": ?}}.
+ * The collection is taken only from a command that runs against one; the first value of any other
+ * command is an argument, such as a user name, and is recorded as a value:
+ * {@code dropUser admin {"dropUser": ?}}.</p>
+ *
+ * @since 9.0
+ */
+@CompileStatic
+final class MongoCommandRedactor {
+
+ private static final String PLACEHOLDER = QueryTailRedactor.PLACEHOLDER
+
+ private static final String DATABASE = '$db'
+
+ private static final String COLLECTION = 'collection'
+
+ /**
+ * The commands whose first value is the collection they run against. The first value of any other
+ * command, such as the user {@code dropUser} drops, is an argument, and is recorded as a value.
+ */
+ private static final Set<String> COLLECTION_COMMANDS = ['find', 'insert', 'update', 'delete', 'aggregate',
+ 'count', 'distinct', 'findAndModify', 'findandmodify', 'mapReduce', 'mapreduce', 'killCursors',
+ 'create', 'drop', 'collMod', 'createIndexes', 'dropIndexes', 'deleteIndexes', 'listIndexes', 'reIndex',
+ 'compact', 'validate', 'collStats', 'convertToCapped', 'cloneCollectionAsCapped', 'analyze',
+ 'checkMetadataConsistency', 'createSearchIndexes', 'dropSearchIndex', 'updateSearchIndex',
+ 'planCacheClear', 'planCacheClearFilters', 'planCacheListFilters', 'planCacheSetFilter'] as Set<String>
+
+ /** The command that names its collection in its {@code collection} option, as it continues a cursor. */
+ private static final String GET_MORE = 'getMore'
+
+ /** The entries the driver adds, which describe its session and transaction rather than the command. */
+ private static final Set<String> LEFT_OUT = ['lsid', '$clusterTime', 'txnNumber', 'autocommit',
+ 'startTransaction', '$readPreference', 'apiVersion', 'apiStrict', 'apiDeprecationErrors'] as Set<String>
+
+ /**
+ * The options whose document, or documents, are keyed by the fields of the collection, wherever they appear,
+ * including those of the operations of a client-level {@code bulkWrite}.
+ */
+ private static final Set<String> FIELD_KEYED = ['filter', 'query', 'q', 'u', 'update', 'document', 'documents',
+ 'updateMods', 'pipeline', 'sort', 'projection', 'fields', 'key', 'hint', 'min', 'max', 'arrayFilters',
+ 'partialFilterExpression', 'wildcardProjection', 'validator', 'weights'] as Set<String>
+
+ /** Documents that match fields, whose field values may be implicit equality operands. */
+ private static final Set<String> CONDITION_KEYED = ['filter', 'query', 'q', 'arrayFilters',
+ 'partialFilterExpression', 'validator', 'restrictSearchWithMatch'] as Set<String>
+
+ /** The options whose documents are stored data: what an insert writes. */
+ private static final Set<String> DATA_KEYED = ['documents', 'document'] as Set<String>
+
+ /** The options and the operator that may hold any document, none of whose keys say what the command does. */
+ private static final Set<String> ANY_VALUE = ['comment', '$comment', 'c', 'scope'] as Set<String>
+
+ /** The option whose values are aggregation expressions. */
+ private static final String LET = 'let'
+
+ /** The options that hold an update: update operators, a replacement document or a pipeline. */
+ private static final Set<String> UPDATE_KEYED = ['u', 'update', 'updateMods'] as Set<String>
+
+ /** The update operators whose fields take a condition rather than the value to write. */
+ private static final Set<String> CONDITION_OPERATORS = ['$pull', '$currentDate'] as Set<String>
+
+ /** The update operators that add to an array, whose value may hold modifiers. */
+ private static final Set<String> PUSH_OPERATORS = ['$push', '$addToSet'] as Set<String>
+
+ private static final String EACH = '$each'
+
+ /** The modifiers of a {@code $push} or an {@code $addToSet}. */
+ private static final Set<String> PUSH_MODIFIERS = [EACH, '$slice', '$sort', '$position'] as Set<String>
+
+ /**
+ * The operators of a pipeline whose operand is literal data rather than an expression, including
+ * {@code $const}, which MongoDB accepts for {@code $literal}.
+ */
+ private static final Set<String> LITERAL_OPERATORS = ['$literal', '$const', '$documents'] as Set<String>
+
+ /**
+ * The fields of a DBRef, which a condition reads as fields compared with their whole value, never as
+ * operators, in any order and wherever a condition has fields.
+ */
+ private static final Set<String> DBREF_FIELDS = ['$ref', '$id', '$db'] as Set<String>
+
+ /**
+ * The query operators whose operand holds values rather than a condition: the comparisons, whose operand
+ * is the value a field is compared with, a JSON schema, whose {@code enum} lists values, and a
+ * GeoJSON geometry, which can carry additional data alongside its type and coordinates.
+ */
+ private static final Set<String> VALUE_OPERATORS = ['$eq', '$ne', '$gt', '$gte', '$lt', '$lte', '$in', '$nin',
+ '$all', '$jsonSchema', '$geometry'] as Set<String>
+
+ /**
+ * The operators of a condition, the only {@code $} keys a condition holds but the fields of a DBRef: the query
+ * operators of MongoDB, and the options of {@code $regex}, {@code $text} and the geospatial ones.
+ */
+ private static final Set<String> QUERY_OPERATORS = ['$eq', '$ne', '$gt', '$gte', '$lt', '$lte', '$in', '$nin',
+ '$and', '$or', '$not', '$nor', '$exists', '$type', '$expr', '$jsonSchema', '$mod', '$regex', '$options',
+ '$text', '$search', '$language', '$caseSensitive', '$diacriticSensitive', '$where', '$geoIntersects',
+ '$geoWithin', '$near', '$nearSphere', '$geometry', '$maxDistance', '$minDistance', '$box', '$center',
+ '$centerSphere', '$polygon', '$uniqueDocs', '$all', '$elemMatch', '$size', '$bitsAllClear', '$bitsAllSet',
+ '$bitsAnyClear', '$bitsAnySet', '$comment', '$rand', '$sampleRate'] as Set<String>
+
+ /** The operator of a condition, and the stages of a pipeline, whose operand is an aggregation expression. */
+ private static final Set<String> EXPRESSION_OPERATORS = ['$expr', '$addFields', '$set', '$project', '$group',
+ '$replaceRoot', '$replaceWith', '$redact', '$bucket', '$bucketAuto', '$sortByCount', '$setWindowFields',
+ '$fill'] as Set<String>
+
+ /** The stages of an update pipeline whose operand is the document they write, or the new root that replaces it. */
+ private static final Set<String> WRITING_STAGES = ['$set', '$addFields', '$project', '$replaceWith'] as Set<String>
+
+ /** The stage of an update pipeline whose options name the new root that replaces the document. */
+ private static final String REPLACE_ROOT = '$replaceRoot'
+
+ /** The option of {@code $replaceRoot} that holds the new root. */
+ private static final String NEW_ROOT = 'newRoot'
+
+ /** The operator whose operands are documents merged into one, which in an update pipeline is written. */
+ private static final String MERGE_OBJECTS = '$mergeObjects'
+
+ /** The kinds of document whose keys at the top are stored data: what an insert, a replacement or an update writes. */
+ private static final Set<Kind> STORED = EnumSet.of(Kind.DATA, Kind.UPDATE, Kind.UPDATE_FIELDS, Kind.WRITTEN)
+
+ private MongoCommandRedactor() {
+ }
+
+ /**
+ * Redacts a command.
+ *
+ * @param database the database the command runs against
+ * @param commandName the name of the command, which is its first key
+ * @param command the command as the driver sends it
+ * @param fields the fields the mapping of the datastore knows, or null before the datastore is built
+ * @return the command with its values replaced
+ */
+ static String redact(String database, String commandName, BsonDocument command, MongoMappedFields fields) {
+ String collection = null
+ String collectionKey = null
+ BsonValue first = command.get(commandName)
+ if (first != null && first.isString() && COLLECTION_COMMANDS.contains(commandName)) {
+ collection = first.asString().value
+ collectionKey = commandName
+ }
+ else if (commandName == GET_MORE && command.get(COLLECTION)?.isString()) {
+ collection = command.getString(COLLECTION).value
+ collectionKey = COLLECTION
+ }
+ StringBuilder out = new StringBuilder(commandName)
+ if (database != null || collection != null) {
+ out.append(' ')
+ if (database != null) {
+ out.append(database)
+ }
+ if (collection != null) {
+ out.append(database != null ? '.' : '').append(collection)
+ }
+ }
+ Scope options = Scope.options(collection, fields)
+ StringBuilder body = new StringBuilder()
+ for (Map.Entry<String, BsonValue> entry in command.entrySet()) {
+ String key = entry.key
+ if (key == DATABASE || key == collectionKey || LEFT_OUT.contains(key)) {
+ continue
+ }
+ body.append(body.length() == 0 ? '{' : ', ')
+ appendEntry(body, key, entry.value, options)
+ }
+ if (body.length() > 0) {
+ out.append(' ').append(body).append('}')
+ }
+ out.toString()
+ }
+
+ private static void appendEntry(StringBuilder out, String key, BsonValue value, Scope scope) {
+ Scope child = scope.of(key, value)
+ out.append(child.keyText).append(': ')
+ appendValue(out, value, child)
+ }
+
+ private static void appendValue(StringBuilder out, BsonValue value, Scope scope) {
+ if (value.isDocument()) {
+ out.append('{')
+ boolean first = true
+ for (Map.Entry<String, BsonValue> entry in value.asDocument().entrySet()) {
+ if (!first) {
+ out.append(', ')
+ }
+ first = false
+ appendEntry(out, entry.key, entry.value, scope)
+ }
+ out.append('}')
+ }
+ else if (value.isArray()) {
+ out.append('[')
+ Scope element = scope.element()
+ // a value is recorded as the placeholder whatever it is, so only a document or an array is rendered
+ // to be told apart from the elements before it, and a list of values costs no more than a look at each
+ boolean placeholder = false
+ Set<String> written = null
+ boolean first = true
+ for (BsonValue item in value.asArray()) {
+ String recorded
+ if (item.isDocument() || item.isArray()) {
+ StringBuilder text = new StringBuilder()
+ appendValue(text, item, element)
+ recorded = text.toString()
+ if (written == null) {
+ written = new HashSet<>()
+ }
+ if (!written.add(recorded)) {
+ continue
+ }
+ }
+ else if (placeholder) {
+ continue
+ }
+ else {
+ placeholder = true
+ recorded = PLACEHOLDER
+ }
+ if (!first) {
+ out.append(', ')
+ }
+ first = false
+ out.append(recorded)
+ }
+ out.append(']')
+ }
+ else {
+ out.append(PLACEHOLDER)
+ }
+ }
+
+ private static String quote(String key) {
+ StringBuilder out = new StringBuilder(key.length() + 2).append('"')
+ for (int i = 0; i < key.length(); i++) {
+ char c = key.charAt(i)
+ if (c == '"' || c == '\\') {
+ out.append('\\').append(c)
+ }
+ else if (c < 0x20) {
+ out.append(String.format('\\u%04x', (int) c))
+ }
+ else {
+ out.append(c)
+ }
+ }
+ out.append('"').toString()
+ }
+
+ /**
+ * Where a document sits in the command, which decides what its keys are: the options of the
+ * command, kept; the fields and operators of a condition, a pipeline, a sort or a projection, where
+ * a {@code $} key is an operator; an aggregation expression, whose {@code $} keys are operators
+ * too; an update, whose {@code $} keys are update operators; or stored data, the documents an
+ * insert writes, the values an update writes and the values a condition compares with, where a
+ * {@code $} key is a field name like any other and is kept only when the mapping knows it.
+ */
+ @CompileStatic
+ private static final class Scope {
+
+ final String collection
+ final MongoMappedFields fields
+ final Kind kind
+ /**
+ * The update operator whose fields a document of kind {@link Kind#UPDATE_FIELDS} holds, or the stage
+ * whose document one of kind {@link Kind#WRITTEN} is.
+ */
+ final String operator
+ /** The field path of the document; null in a document keyed by fields the mapping does not know. */
+ final List<String> path
+ /** How many indices into the field at the path have been taken: 0 at the field, 1 in one of its elements. */
+ final int depth
+ final String keyText
+
+ private Scope(String collection, MongoMappedFields fields, Kind kind, String operator, List<String> path,
+ int depth, String keyText) {
+ this.collection = collection
+ this.fields = fields
+ this.kind = kind
+ this.operator = operator
+ this.path = path
+ this.depth = depth
+ this.keyText = keyText
+ }
+
+ static Scope options(String collection, MongoMappedFields fields) {
+ new Scope(collection, fields, Kind.OPTIONS, null, Collections.<String>emptyList(), 0, null)
+ }
+
+ /**
+ * @return the scope of an element of an array in this scope: one index deeper into the field
+ * at the path, so that a numeric key inside the element is data unless the elements are arrays;
+ * the elements of an update that is an array are the stages of a pipeline, and an element of an
+ * expression at the top of a document, as the operands of an operator {@code $expr} holds are, is a
+ * value the expression builds, whose keys are kept only when the mapping knows them, as those of a
+ * document a stage of an update pipeline writes are
+ */
+ Scope element() {
+ if (kind == Kind.UPDATE) {
+ return child(Kind.UPDATE_PIPELINE, null, path, depth, keyText)
+ }
+ if (kind == Kind.PUSH) {
+ return child(Kind.DATA, null, path, depth + 2, keyText)
+ }
+ if ((kind == Kind.QUERY || kind == Kind.CONDITION || kind == Kind.EXPRESSION || kind == Kind.DATA) && path != null && !path.isEmpty()) {
+ return child(kind, null, path, depth + 1, keyText)
+ }
+ if (kind == Kind.EXPRESSION && path != null) {
+ return child(Kind.WRITTEN, null, path, depth, keyText)
+ }
+ this
+ }
+
+ /**
+ * @return the scope of the value of a key in a document of this scope, with the key as it is recorded
+ */
+ Scope of(String key, BsonValue value) {
+ if (kind == Kind.OPTIONS) {
+ if (ANY_VALUE.contains(key)) {
+ return child(Kind.DATA, null, null, 0, quote(key))
+ }
+ Kind valueKind = DATA_KEYED.contains(key) ? Kind.DATA :
+ UPDATE_KEYED.contains(key) ? Kind.UPDATE :
+ CONDITION_KEYED.contains(key) ? Kind.CONDITION :
+ FIELD_KEYED.contains(key) ? Kind.QUERY :
+ key == LET ? Kind.EXPRESSION : Kind.OPTIONS
+ return child(valueKind, null, Collections.<String>emptyList(), 0, quote(key))
+ }
+ if ((kind == Kind.QUERY || kind == Kind.CONDITION) && key.startsWith('$')) {
+ if (ANY_VALUE.contains(key)) {
+ return child(Kind.DATA, null, null, 0, quote(key))
+ }
+ if (kind == Kind.CONDITION && DBREF_FIELDS.contains(key)) {
+ return field(key, Kind.DATA)
+ }
+ if (kind == Kind.CONDITION && !QUERY_OPERATORS.contains(key)) {
Review Comment:
A filter location.$within.$box of [[0,0],[1,1]] is recorded as location: {?: {?: [[?]]}}. GORM emits $within for WithinBox, WithinPolygon, and WithinCircle. Keep $within and the shape operator. Keep redacting a key that is not an operator.
##########
grails-data-mongodb/core/src/main/groovy/org/apache/grails/data/mongodb/core/querytail/MongoCommandRedactor.groovy:
##########
@@ -0,0 +1,574 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * https://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.grails.data.mongodb.core.querytail
+
+import groovy.transform.CompileStatic
+import org.bson.BsonDocument
+import org.bson.BsonValue
+
+import org.apache.grails.datamapping.core.querytail.QueryTailRedactor
+
+/**
+ * Takes the values out of a MongoDB command before it is captured. A command is a document, so
+ * there is nothing to parse: every value becomes {@link QueryTailRedactor#PLACEHOLDER}, and the
+ * keys, which are the options of the command, its operators and the field names of the collection,
+ * are kept, except where a key may be data. The keys of a document stored in a map property, or in
+ * a property or a collection the {@linkplain MongoMappedFields mapping} does not know, are recorded
+ * as the placeholder too, and so is every segment of a dotted path past such a key. So is a key at
+ * the top of stored data, the documents an insert writes, a replacement document, the fields an
+ * update writes and the operand of a literal, unless the mapping knows it, as the name of a dynamic
+ * attribute may be data; the keys at the top of a condition, a pipeline, a sort or a projection
+ * are the fields the application asks for, and are kept. An update that is a pipeline writes the
+ * documents its stages build: the keys at the top of what a {@code $set}, {@code $addFields} or
+ * {@code $project} stage holds, of the new root of a {@code $replaceWith} or {@code $replaceRoot}, and of a
+ * document {@code $mergeObjects} merges into it are fields written, kept only when the mapping knows
+ * them, as those of a classic update are. A numeric key
+ * is kept only where it indexes an array the mapping knows; in a map, in an element of an array,
+ * which is a map or a document unless the field is a list of lists, or at the top of a document,
+ * it is data. A {@code $} key is kept as an operator only where operators go, in a condition, a
+ * pipeline and the operators of an update, and in a condition only when it is one of MongoDB's query
+ * operators: any other, such as a key beside the geometry of {@code $near}, is recorded as the
+ * placeholder, and so is everything under it; in stored data, the documents an insert writes, a
+ * replacement document and the values an update operator writes, it is a field name, which MongoDB
+ * allows to start with {@code $}, and is recorded as the placeholder unless the mapping knows it.
+ * So is a {@code $} key in the operand of a comparison in a condition, such as {@code $eq} or
+ * {@code $in}, or an implicit document or array equality, which holds the value the field is compared
+ * with, and in a DBRef, whose {@code $ref}, {@code $id} and {@code $db} a condition reads as fields
+ * compared with their whole value, in any order; in an aggregation expression, which
+ * {@code $expr}, the {@code let} of a command and the stages that compute fields hold, the operands
+ * of a comparison are expressions, and their operators are kept, except in the operand of a
+ * {@code $literal} or {@code $const}, which is stored data. A comment, and the constants of an
+ * update, and the scope of a map-reduce command, may be any document, and none of their keys are
+ * kept. The entries the driver adds for its
+ * session, the cluster time and the transaction, which say nothing about the command, are left out.
+ *
+ * <p>An array is recorded with each of its different elements once: an element recorded as one before
+ * it in the array is left out, so that the documents of a batch and the values of an {@code $in} are
+ * recorded the same whatever their number, one or many, and a command is counted as the one statement
+ * it is: {@code {"$in": [?]}}. An empty array is recorded as {@code []}.</p>
+ *
+ * <p>A command is recorded as its name, the database and collection it runs against, and the
+ * document of its remaining entries: {@code find shop.book {"filter": {"title": ?}, "limit": ?}}.
+ * The collection is taken only from a command that runs against one; the first value of any other
+ * command is an argument, such as a user name, and is recorded as a value:
+ * {@code dropUser admin {"dropUser": ?}}.</p>
+ *
+ * @since 9.0
+ */
+@CompileStatic
+final class MongoCommandRedactor {
+
+ private static final String PLACEHOLDER = QueryTailRedactor.PLACEHOLDER
+
+ private static final String DATABASE = '$db'
+
+ private static final String COLLECTION = 'collection'
+
+ /**
+ * The commands whose first value is the collection they run against. The first value of any other
+ * command, such as the user {@code dropUser} drops, is an argument, and is recorded as a value.
+ */
+ private static final Set<String> COLLECTION_COMMANDS = ['find', 'insert', 'update', 'delete', 'aggregate',
+ 'count', 'distinct', 'findAndModify', 'findandmodify', 'mapReduce', 'mapreduce', 'killCursors',
+ 'create', 'drop', 'collMod', 'createIndexes', 'dropIndexes', 'deleteIndexes', 'listIndexes', 'reIndex',
+ 'compact', 'validate', 'collStats', 'convertToCapped', 'cloneCollectionAsCapped', 'analyze',
+ 'checkMetadataConsistency', 'createSearchIndexes', 'dropSearchIndex', 'updateSearchIndex',
+ 'planCacheClear', 'planCacheClearFilters', 'planCacheListFilters', 'planCacheSetFilter'] as Set<String>
+
+ /** The command that names its collection in its {@code collection} option, as it continues a cursor. */
+ private static final String GET_MORE = 'getMore'
+
+ /** The entries the driver adds, which describe its session and transaction rather than the command. */
+ private static final Set<String> LEFT_OUT = ['lsid', '$clusterTime', 'txnNumber', 'autocommit',
+ 'startTransaction', '$readPreference', 'apiVersion', 'apiStrict', 'apiDeprecationErrors'] as Set<String>
+
+ /**
+ * The options whose document, or documents, are keyed by the fields of the collection, wherever they appear,
+ * including those of the operations of a client-level {@code bulkWrite}.
+ */
+ private static final Set<String> FIELD_KEYED = ['filter', 'query', 'q', 'u', 'update', 'document', 'documents',
+ 'updateMods', 'pipeline', 'sort', 'projection', 'fields', 'key', 'hint', 'min', 'max', 'arrayFilters',
+ 'partialFilterExpression', 'wildcardProjection', 'validator', 'weights'] as Set<String>
+
+ /** Documents that match fields, whose field values may be implicit equality operands. */
+ private static final Set<String> CONDITION_KEYED = ['filter', 'query', 'q', 'arrayFilters',
+ 'partialFilterExpression', 'validator', 'restrictSearchWithMatch'] as Set<String>
+
+ /** The options whose documents are stored data: what an insert writes. */
+ private static final Set<String> DATA_KEYED = ['documents', 'document'] as Set<String>
+
+ /** The options and the operator that may hold any document, none of whose keys say what the command does. */
+ private static final Set<String> ANY_VALUE = ['comment', '$comment', 'c', 'scope'] as Set<String>
+
+ /** The option whose values are aggregation expressions. */
+ private static final String LET = 'let'
+
+ /** The options that hold an update: update operators, a replacement document or a pipeline. */
+ private static final Set<String> UPDATE_KEYED = ['u', 'update', 'updateMods'] as Set<String>
+
+ /** The update operators whose fields take a condition rather than the value to write. */
+ private static final Set<String> CONDITION_OPERATORS = ['$pull', '$currentDate'] as Set<String>
+
+ /** The update operators that add to an array, whose value may hold modifiers. */
+ private static final Set<String> PUSH_OPERATORS = ['$push', '$addToSet'] as Set<String>
+
+ private static final String EACH = '$each'
+
+ /** The modifiers of a {@code $push} or an {@code $addToSet}. */
+ private static final Set<String> PUSH_MODIFIERS = [EACH, '$slice', '$sort', '$position'] as Set<String>
+
+ /**
+ * The operators of a pipeline whose operand is literal data rather than an expression, including
+ * {@code $const}, which MongoDB accepts for {@code $literal}.
+ */
+ private static final Set<String> LITERAL_OPERATORS = ['$literal', '$const', '$documents'] as Set<String>
+
+ /**
+ * The fields of a DBRef, which a condition reads as fields compared with their whole value, never as
+ * operators, in any order and wherever a condition has fields.
+ */
+ private static final Set<String> DBREF_FIELDS = ['$ref', '$id', '$db'] as Set<String>
+
+ /**
+ * The query operators whose operand holds values rather than a condition: the comparisons, whose operand
+ * is the value a field is compared with, a JSON schema, whose {@code enum} lists values, and a
+ * GeoJSON geometry, which can carry additional data alongside its type and coordinates.
+ */
+ private static final Set<String> VALUE_OPERATORS = ['$eq', '$ne', '$gt', '$gte', '$lt', '$lte', '$in', '$nin',
+ '$all', '$jsonSchema', '$geometry'] as Set<String>
+
+ /**
+ * The operators of a condition, the only {@code $} keys a condition holds but the fields of a DBRef: the query
+ * operators of MongoDB, and the options of {@code $regex}, {@code $text} and the geospatial ones.
+ */
+ private static final Set<String> QUERY_OPERATORS = ['$eq', '$ne', '$gt', '$gte', '$lt', '$lte', '$in', '$nin',
+ '$and', '$or', '$not', '$nor', '$exists', '$type', '$expr', '$jsonSchema', '$mod', '$regex', '$options',
+ '$text', '$search', '$language', '$caseSensitive', '$diacriticSensitive', '$where', '$geoIntersects',
+ '$geoWithin', '$near', '$nearSphere', '$geometry', '$maxDistance', '$minDistance', '$box', '$center',
+ '$centerSphere', '$polygon', '$uniqueDocs', '$all', '$elemMatch', '$size', '$bitsAllClear', '$bitsAllSet',
+ '$bitsAnyClear', '$bitsAnySet', '$comment', '$rand', '$sampleRate'] as Set<String>
+
+ /** The operator of a condition, and the stages of a pipeline, whose operand is an aggregation expression. */
+ private static final Set<String> EXPRESSION_OPERATORS = ['$expr', '$addFields', '$set', '$project', '$group',
+ '$replaceRoot', '$replaceWith', '$redact', '$bucket', '$bucketAuto', '$sortByCount', '$setWindowFields',
+ '$fill'] as Set<String>
+
+ /** The stages of an update pipeline whose operand is the document they write, or the new root that replaces it. */
+ private static final Set<String> WRITING_STAGES = ['$set', '$addFields', '$project', '$replaceWith'] as Set<String>
+
+ /** The stage of an update pipeline whose options name the new root that replaces the document. */
+ private static final String REPLACE_ROOT = '$replaceRoot'
+
+ /** The option of {@code $replaceRoot} that holds the new root. */
+ private static final String NEW_ROOT = 'newRoot'
+
+ /** The operator whose operands are documents merged into one, which in an update pipeline is written. */
+ private static final String MERGE_OBJECTS = '$mergeObjects'
+
+ /** The kinds of document whose keys at the top are stored data: what an insert, a replacement or an update writes. */
+ private static final Set<Kind> STORED = EnumSet.of(Kind.DATA, Kind.UPDATE, Kind.UPDATE_FIELDS, Kind.WRITTEN)
+
+ private MongoCommandRedactor() {
+ }
+
+ /**
+ * Redacts a command.
+ *
+ * @param database the database the command runs against
+ * @param commandName the name of the command, which is its first key
+ * @param command the command as the driver sends it
+ * @param fields the fields the mapping of the datastore knows, or null before the datastore is built
+ * @return the command with its values replaced
+ */
+ static String redact(String database, String commandName, BsonDocument command, MongoMappedFields fields) {
+ String collection = null
+ String collectionKey = null
+ BsonValue first = command.get(commandName)
+ if (first != null && first.isString() && COLLECTION_COMMANDS.contains(commandName)) {
+ collection = first.asString().value
+ collectionKey = commandName
+ }
+ else if (commandName == GET_MORE && command.get(COLLECTION)?.isString()) {
+ collection = command.getString(COLLECTION).value
+ collectionKey = COLLECTION
+ }
+ StringBuilder out = new StringBuilder(commandName)
+ if (database != null || collection != null) {
+ out.append(' ')
+ if (database != null) {
+ out.append(database)
+ }
+ if (collection != null) {
+ out.append(database != null ? '.' : '').append(collection)
+ }
+ }
+ Scope options = Scope.options(collection, fields)
+ StringBuilder body = new StringBuilder()
+ for (Map.Entry<String, BsonValue> entry in command.entrySet()) {
+ String key = entry.key
+ if (key == DATABASE || key == collectionKey || LEFT_OUT.contains(key)) {
+ continue
+ }
+ body.append(body.length() == 0 ? '{' : ', ')
+ appendEntry(body, key, entry.value, options)
+ }
+ if (body.length() > 0) {
+ out.append(' ').append(body).append('}')
+ }
+ out.toString()
+ }
+
+ private static void appendEntry(StringBuilder out, String key, BsonValue value, Scope scope) {
+ Scope child = scope.of(key, value)
+ out.append(child.keyText).append(': ')
+ appendValue(out, value, child)
+ }
+
+ private static void appendValue(StringBuilder out, BsonValue value, Scope scope) {
+ if (value.isDocument()) {
+ out.append('{')
+ boolean first = true
+ for (Map.Entry<String, BsonValue> entry in value.asDocument().entrySet()) {
+ if (!first) {
+ out.append(', ')
+ }
+ first = false
+ appendEntry(out, entry.key, entry.value, scope)
+ }
+ out.append('}')
+ }
+ else if (value.isArray()) {
+ out.append('[')
+ Scope element = scope.element()
+ // a value is recorded as the placeholder whatever it is, so only a document or an array is rendered
+ // to be told apart from the elements before it, and a list of values costs no more than a look at each
+ boolean placeholder = false
+ Set<String> written = null
+ boolean first = true
+ for (BsonValue item in value.asArray()) {
+ String recorded
+ if (item.isDocument() || item.isArray()) {
+ StringBuilder text = new StringBuilder()
+ appendValue(text, item, element)
+ recorded = text.toString()
+ if (written == null) {
+ written = new HashSet<>()
+ }
+ if (!written.add(recorded)) {
+ continue
+ }
+ }
+ else if (placeholder) {
+ continue
+ }
+ else {
+ placeholder = true
+ recorded = PLACEHOLDER
+ }
+ if (!first) {
+ out.append(', ')
+ }
+ first = false
+ out.append(recorded)
+ }
+ out.append(']')
+ }
+ else {
+ out.append(PLACEHOLDER)
+ }
+ }
+
+ private static String quote(String key) {
+ StringBuilder out = new StringBuilder(key.length() + 2).append('"')
+ for (int i = 0; i < key.length(); i++) {
+ char c = key.charAt(i)
+ if (c == '"' || c == '\\') {
+ out.append('\\').append(c)
+ }
+ else if (c < 0x20) {
+ out.append(String.format('\\u%04x', (int) c))
+ }
+ else {
+ out.append(c)
+ }
+ }
+ out.append('"').toString()
+ }
+
+ /**
+ * Where a document sits in the command, which decides what its keys are: the options of the
+ * command, kept; the fields and operators of a condition, a pipeline, a sort or a projection, where
+ * a {@code $} key is an operator; an aggregation expression, whose {@code $} keys are operators
+ * too; an update, whose {@code $} keys are update operators; or stored data, the documents an
+ * insert writes, the values an update writes and the values a condition compares with, where a
+ * {@code $} key is a field name like any other and is kept only when the mapping knows it.
+ */
+ @CompileStatic
+ private static final class Scope {
+
+ final String collection
+ final MongoMappedFields fields
+ final Kind kind
+ /**
+ * The update operator whose fields a document of kind {@link Kind#UPDATE_FIELDS} holds, or the stage
+ * whose document one of kind {@link Kind#WRITTEN} is.
+ */
+ final String operator
+ /** The field path of the document; null in a document keyed by fields the mapping does not know. */
+ final List<String> path
+ /** How many indices into the field at the path have been taken: 0 at the field, 1 in one of its elements. */
+ final int depth
+ final String keyText
+
+ private Scope(String collection, MongoMappedFields fields, Kind kind, String operator, List<String> path,
+ int depth, String keyText) {
+ this.collection = collection
+ this.fields = fields
+ this.kind = kind
+ this.operator = operator
+ this.path = path
+ this.depth = depth
+ this.keyText = keyText
+ }
+
+ static Scope options(String collection, MongoMappedFields fields) {
+ new Scope(collection, fields, Kind.OPTIONS, null, Collections.<String>emptyList(), 0, null)
+ }
+
+ /**
+ * @return the scope of an element of an array in this scope: one index deeper into the field
+ * at the path, so that a numeric key inside the element is data unless the elements are arrays;
+ * the elements of an update that is an array are the stages of a pipeline, and an element of an
+ * expression at the top of a document, as the operands of an operator {@code $expr} holds are, is a
+ * value the expression builds, whose keys are kept only when the mapping knows them, as those of a
+ * document a stage of an update pipeline writes are
+ */
+ Scope element() {
+ if (kind == Kind.UPDATE) {
+ return child(Kind.UPDATE_PIPELINE, null, path, depth, keyText)
+ }
+ if (kind == Kind.PUSH) {
+ return child(Kind.DATA, null, path, depth + 2, keyText)
+ }
+ if ((kind == Kind.QUERY || kind == Kind.CONDITION || kind == Kind.EXPRESSION || kind == Kind.DATA) && path != null && !path.isEmpty()) {
+ return child(kind, null, path, depth + 1, keyText)
+ }
+ if (kind == Kind.EXPRESSION && path != null) {
+ return child(Kind.WRITTEN, null, path, depth, keyText)
+ }
+ this
+ }
+
+ /**
+ * @return the scope of the value of a key in a document of this scope, with the key as it is recorded
+ */
+ Scope of(String key, BsonValue value) {
+ if (kind == Kind.OPTIONS) {
+ if (ANY_VALUE.contains(key)) {
+ return child(Kind.DATA, null, null, 0, quote(key))
+ }
+ Kind valueKind = DATA_KEYED.contains(key) ? Kind.DATA :
+ UPDATE_KEYED.contains(key) ? Kind.UPDATE :
+ CONDITION_KEYED.contains(key) ? Kind.CONDITION :
+ FIELD_KEYED.contains(key) ? Kind.QUERY :
+ key == LET ? Kind.EXPRESSION : Kind.OPTIONS
Review Comment:
applyOps with o: {attrs: {"example-secret": 1}} records example-secret. That payload stays in the options scope, so a stored key is kept. Treat the payload as data.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@grails.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add the query-tail command to find query-heavy pages [grails-core]
Posted by "jdaugherty (via GitHub)" <gi...@apache.org>.
jdaugherty merged PR #16631:
URL: https://github.com/apache/grails-core/pull/16631
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@grails.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add the query-tail command to find query-heavy pages [grails-core]
Posted by "jamesfredley (via GitHub)" <gi...@apache.org>.
jamesfredley commented on code in PR #16631:
URL: https://github.com/apache/grails-core/pull/16631#discussion_r4373361018
##########
grails-datamapping-core/src/main/groovy/org/apache/grails/data/core/querytail/QueryTailCapture.groovy:
##########
@@ -0,0 +1,369 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * https://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.grails.data.core.querytail
+
+import java.nio.ByteBuffer
+import java.nio.channels.FileChannel
+import java.nio.channels.FileLock
+import java.nio.channels.OverlappingFileLockException
+import java.nio.charset.StandardCharsets
+import java.nio.file.Files
+import java.nio.file.StandardOpenOption
+import java.util.concurrent.atomic.AtomicBoolean
+import java.util.concurrent.atomic.AtomicLong
+
+import groovy.transform.CompileStatic
+import groovy.util.logging.Slf4j
+
+/**
+ * Writes the statements an application runs to the capture file the {@code query-tail} command
+ * tails. Each session factory, Neo4j driver or MongoDB client that captures opens its own capture
+ * and closes it when it closes; the captures of one file share it, so every data source of an
+ * application writes to the one file. The file is truncated when the first capture opens it, so it
+ * holds the statements of the current run only, and its first line names that run. It is closed
+ * when the last capture closes, and opened afresh by the next one. Each statement is written whole
+ * and at once, so that the command sees it as soon as it is recorded.
+ *
+ * <p>JVMs that run at the same time with capture enabled, such as the parallel forks of a test task,
+ * share the file: a lock file beside it tells the first to start, which truncates the file and names
+ * the run, from the others, which append to that run without truncating it and number their
+ * requests apart, so that the command reads one run from all of them.</p>
+ *
+ * <p>A statement is recorded without its values: {@link QueryTailRedactor} replaces its literals and
+ * drops its comments before it is written, in the {@link #syntax} of the database it runs on. A
+ * statement that is not text, such as a MongoDB command, has its values taken out by whoever records
+ * it, through {@link #recordRedacted}.</p>
+ *
+ * <p>A capture is only opened when {@link QueryTailSettings#SETTING_ENABLED} is set; an application
+ * started without it never reaches this class.</p>
+ *
+ * @since 8.1
+ */
+@CompileStatic
+@Slf4j
+class QueryTailCapture implements Closeable {
+
+ private static final Map<String, CaptureFile> OPEN = new HashMap<>()
+
+ /** The file the statements are written to. */
+ final File file
+
+ /** The syntax of the database the statements of this capture run on, which their values are found by. */
+ volatile QueryTailSyntax syntax = QueryTailSyntax.UNKNOWN
+
+ private final CaptureFile captureFile
+ private final AtomicBoolean closed = new AtomicBoolean()
+
+ private QueryTailCapture(File file, CaptureFile captureFile) {
+ this.file = file
+ this.captureFile = captureFile
+ }
+
+ /**
+ * Opens a capture of a file. The first capture of a file truncates it; a capture opened while
+ * another of the same file is open shares it. Every capture opened must be closed.
+ *
+ * @param file the capture file
+ * @return the capture
+ * @throws UncheckedIOException when the file cannot be created
+ */
+ static QueryTailCapture open(File file) {
+ String key
+ try {
+ key = file.canonicalPath
+ }
+ catch (IOException e) {
+ throw new UncheckedIOException("Cannot resolve the query-tail capture file ${file}: ${e.message}", e)
+ }
+ synchronized (OPEN) {
+ CaptureFile captureFile = OPEN.get(key)
+ if (captureFile == null) {
+ captureFile = new CaptureFile(file, key)
+ OPEN.put(key, captureFile)
+ if (captureFile.joined) {
+ log.info('Capturing statements to {} for query-tail, in the run of another JVM writing it', file)
+ }
+ else {
+ log.info('Capturing statements to {} for query-tail', file)
+ }
+ }
+ captureFile.captures++
+ return new QueryTailCapture(file, captureFile)
+ }
+ }
+
+ /**
+ * Opens the capture the settings name.
+ *
+ * @param settings the query-tail settings
+ * @return the capture
+ */
+ static QueryTailCapture open(QueryTailSettings settings) {
+ open(settings.resolveFile())
+ }
+
+ /**
+ * Records a statement that is about to run on the current thread, without its values.
+ *
+ * @param dataSource the connection source name the statement runs against
+ * @param sql the statement
+ */
+ void record(String dataSource, String sql) {
+ write(dataSource, QueryTailRedactor.redact(sql, syntax))
+ }
+
+ /**
+ * Records a statement that is about to run on the current thread and that holds no value any
+ * more: a MongoDB command, which is a document rather than text, is redacted field by field by
+ * the listener that records it. Nothing of the statement is changed here.
+ *
+ * @param dataSource the connection source name the statement runs against
+ * @param statement the statement, with its values already taken out
+ */
+ void recordRedacted(String dataSource, String statement) {
+ write(dataSource, statement)
+ }
+
+ private void write(String dataSource, String statement) {
+ if (closed.get()) {
+ return
+ }
+ QueryTailRecord record = new QueryTailRecord(timestamp: System.currentTimeMillis(), dataSource: dataSource,
+ sql: statement)
+ captureFile.write(captureFile.attribution.attribute(record))
+ }
+
+ /**
+ * @return whether this capture is still writing to its file
+ */
+ boolean isOpen() {
+ !closed.get() && captureFile.open
+ }
+
+ /**
+ * Closes this capture. The file is closed once every capture of it is.
+ */
+ @Override
+ void close() {
+ if (!closed.compareAndSet(false, true)) {
+ return
+ }
+ boolean last
+ synchronized (OPEN) {
+ last = --captureFile.captures == 0
+ if (last && OPEN.get(captureFile.key).is(captureFile)) {
Review Comment:
The capture is removed from `OPEN` before `close()` releases the file lock. Another thread can open the same file in that gap, see the old lock, and join without a writer lock of its own. When the old close finishes, another JVM can truncate the file that is still being written.
Remove the entry and release the lock together, so a replacement cannot start until the old lock is gone.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@grails.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add the query-tail command to find query-heavy pages [grails-core]
Posted by "testlens-app[bot] (via GitHub)" <gi...@apache.org>.
testlens-app[bot] commented on PR #16631:
URL: https://github.com/apache/grails-core/pull/16631#issuecomment-6414741376
## ✅ All tests passed ✅
🏷️ Commit: 6306f94ef79064cf373c069b9ee8534784a91829
▶️ Tests: 139269 executed
⚪️ Checks: 97/97 completed
---
_Learn more about TestLens at [testlens.app/docs](https://testlens.app/docs/features/pr-comment/)._
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@grails.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add the query-tail command to find query-heavy pages [grails-core]
Posted by "jdaugherty (via GitHub)" <gi...@apache.org>.
jdaugherty commented on PR #16631:
URL: https://github.com/apache/grails-core/pull/16631#issuecomment-6420292135
@jamesfredley, I pushed d0011c6b80 over dcc4f0646e, still a single commit on 9.0.x. It fixes two more problems:
- **Form bodies consumed by attribution:** to describe a request, the capture read the servlet request's parameter map, even before checking whether a controller had been resolved. For an `application/x-www-form-urlencoded` POST, that makes the container parse the body. An application that read the raw body after a statement had run, for example a filter that queries first, then got an empty stream. Attribution now never touches the request parameters:
- Before the URL mapping has run, it records only the shape of the path, as before.
- After that, it uses the parameters the Grails mapping already holds, plus query-string values it parses itself and decodes as the container does.
- **JSON-escaped MongoDB field names:** the record writer and the CLI scanned quotes by SQL rules, where a quote is escaped by doubling it. MongoDB command keys escape a quote with a backslash, so `"a\" b"` and `"a\" b"` were written as the same statement, and `"a\" from b"` was collapsed as if it were a SQL select. A recorded MongoDB command is recognised by its shape: a name, an optional namespace, then a document opening with `{"`. It is now written exactly as recorded, escaping only tabs, line breaks and backslashes, and the CLI shows it as recorded instead of reading it as SQL.
Tests:
- **`QueryTailRequestAttributionSpec`:** posts a form to a request whose parameter getters use up the body, as the Servlet spec says a container does. It checks that the body is intact before and after the mapping has run, and that query-string values in the path are still redacted. The existing query-string features now set a real query string instead of parameters.
- **`QueryTailRecordSpec`, `StatementCollapserSpec`, `QueryTailSummarySpec`:** cover commands whose names differ only in spaces after an escaped quote, or hold a clause word. They check the write, read, normalization and counting paths, plus statements that only resemble a command.
- **`QueryTailCommandListenerSpec`:** runs the two field names against MongoDB. The database tells them apart, and so does the capture.
- Against the previous code, the new features fail; with the fix they pass. A real embedded-Tomcat POST also reads its body intact with capture on.
Locally, `:grails-datamapping-core:check` passes, as do the MongoDB, Neo4j, H5 and H7 query-tail specs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@grails.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org