You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@zeppelin.apache.org by "dev-donghwan (via GitHub)" <gi...@apache.org> on 2026/08/15 18:07:20 UTC
[PR] [ZEPPELIN-6414] Apply the authorization filter before the search cutoff [zeppelin]
dev-donghwan opened a new pull request, #5427:
URL: https://github.com/apache/zeppelin/pull/5427
### What is this PR for?
`/notebook/search` asked the search service for results and then dropped the ones the caller may not read. Both search services cut their results down to twenty before returning them, so that removal ran on an already shortened list:
* A caller with access to few notes is served fewer results than it is allowed to see. If the twenty highest scoring hits all belong to notes it cannot read, the search comes back empty while its own matching notes sit just below the cut.
* In `EmbeddingSearch` the entries the caller cannot read are also counted into the table boost of Phase 1, so notes that never reach the caller still move the ranking of the ones that do.
The read check now travels with the query, as a predicate over the note id, and every implementation applies it before it cuts anything.
| | before | after |
| --- | --- | --- |
| `EmbeddingSearch` | scores every entry, cuts to `MAX_RESULTS`, REST filters | skips unreadable entries while scoring, so neither the table weights nor the cutoff see them |
| `LuceneSearch` | `searcher.search(query, 20)`, REST filters | walks the hits in score order with `searchAfter` until a full page of readable hits is collected, or the hits run out |
| `NotebookRestApi` | four permission calls per result, list mutated while iterating | passes the predicate, no post-filtering |
`LuceneSearch` reads only the id field to decide on a hit and loads the whole document for the hits it keeps, so hits that are dropped cost one stored field read and no highlighting.
`query(String)` is removed rather than kept next to the new method: leaving it would leave a way to search without saying who is asking, which is the defect this issue is about. `SearchService` is bound in `ZeppelinServer` to the three implementations in the repository and is not reachable as an extension point, so nothing outside the tree implements it.
Note that this changes what the result limit means: it is now the top twenty results the caller may read, rather than what is left of the top twenty overall after filtering.
### What type of PR is it?
Bug Fix
### What is the Jira issue?
* https://issues.apache.org/jira/browse/ZEPPELIN-6414
### How should this be tested?
New tests in both search services, `keepsReadableResultsThatTheCutWouldHide`, put twenty-eight matching notes in the index and let the caller read three of them. Each test first checks its own fixture: it queries with an allow-all predicate and asserts that the readable notes really do fall outside the cut, so that the test cannot quietly stop testing anything if the limit or the scoring changes later. It then queries with the real predicate and expects all three readable notes back and nothing else.
`LuceneSearchTest.returnsNothingWhenTheCallerMayReadNothing` covers the walk ending on its own when no hit is readable.
Run locally:
* `LuceneSearchTest` - 12 tests, including the two new ones
* `EmbeddingSearchTest` - 12 tests, including the new one, run with `ZEPPELIN_EMBEDDING_TEST=true` after `bin/install-search-model.sh`
The existing call sites in both test classes now pass `id -> true` explicitly.
### Questions:
* Does the license files need to update? No
* Is there breaking changes for older versions? `SearchService.query(String)` is replaced by `query(String, Predicate<String>)`. Only the three in-tree implementations and the REST resource use it.
* Does this needs documentation? No
### Possible follow-up
Indexing the note id as its own field would let `LuceneSearch` hand the permission filter to Lucene instead of walking the hits, but it needs an index schema change and a rebuild, so it is left out of this issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@zeppelin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [ZEPPELIN-6414] Apply the authorization filter before the search cutoff [zeppelin]
Posted by "tbonelee (via GitHub)" <gi...@apache.org>.
tbonelee merged PR #5427:
URL: https://github.com/apache/zeppelin/pull/5427
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@zeppelin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [ZEPPELIN-6414] Apply the authorization filter before the search cutoff [zeppelin]
Posted by "tbonelee (via GitHub)" <gi...@apache.org>.
tbonelee commented on code in PR #5427:
URL: https://github.com/apache/zeppelin/pull/5427#discussion_r3888936375
##########
zeppelin-server/src/main/java/org/apache/zeppelin/rest/NotebookRestApi.java:
##########
@@ -1215,18 +1215,11 @@ public Response search(@QueryParam("q") String queryTerm) {
HashSet<String> userAndRoles = new HashSet<>();
userAndRoles.add(principal);
userAndRoles.addAll(roles);
- List<Map<String, String>> notesFound = noteSearchService.query(queryTerm);
- for (int i = 0; i < notesFound.size(); i++) {
- String[] ids = notesFound.get(i).get("id").split("/", 2);
- String noteId = ids[0];
- if (!authorizationService.isOwner(noteId, userAndRoles) &&
- !authorizationService.isReader(noteId, userAndRoles) &&
- !authorizationService.isWriter(noteId, userAndRoles) &&
- !authorizationService.isRunner(noteId, userAndRoles)) {
- notesFound.remove(i);
- i--;
- }
- }
+ // isReader() already covers owners, writers and runners. Handing the check to the
+ // search service keeps it in front of the cutoff, so a caller with access to few notes
+ // is still served a full result set of the notes it may read.
Review Comment:
```suggestion
```
##########
zeppelin-server/src/main/java/org/apache/zeppelin/search/LuceneSearch.java:
##########
@@ -144,64 +153,40 @@ public List<Map<String, String>> query(String queryStr) {
}
private List<Map<String, String>> doSearch(
- IndexSearcher searcher, Query query, Analyzer analyzer, Highlighter highlighter) {
+ IndexSearcher searcher, Query query, Analyzer analyzer, Highlighter highlighter,
+ Predicate<String> readable) {
List<Map<String, String>> matchingParagraphs = new ArrayList<>();
- ScoreDoc[] hits;
+ Map<String, Boolean> readableNotes = new HashMap<>();
try {
- hits = searcher.search(query, 20).scoreDocs;
- for (int i = 0; i < hits.length; i++) {
- LOGGER.debug("doc={} score={}", hits[i].doc, hits[i].score);
-
- int id = hits[i].doc;
- Document doc = searcher.doc(id);
- String path = doc.get(ID_FIELD);
- if (path != null) {
- LOGGER.debug( "{}. {}", (i + 1), path);
- String title = doc.get("title");
- if (title != null) {
- LOGGER.debug(" Title: {}", doc.get("title"));
+ // Walk the hits in score order and keep the ones the caller may read until the result
+ // set is full or the hits run out. Reading is checked here and not on the result set,
+ // because a cut that runs first would hide results the caller is allowed to see.
+ ScoreDoc lastHit = null;
+ while (matchingParagraphs.size() < MAX_RESULTS) {
+ ScoreDoc[] hits = (lastHit == null
+ ? searcher.search(query, HIT_BATCH_SIZE)
+ : searcher.searchAfter(lastHit, query, HIT_BATCH_SIZE)).scoreDocs;
+ if (hits.length == 0) {
+ break;
+ }
+ lastHit = hits[hits.length - 1];
Review Comment:
How about growing the batch size as the walk goes deeper?
`HIT_BATCH_SIZE` is fixed at 20, so walking all N hits calls `searchAfter` N/20 times. `searchAfter` re-runs the query from the start on each round (the score floor set by `after` does prune, but the posting lists are still traversed again), which comes to 500 re-executions for 10,000 hits.
Doubling the batch each round takes that from `O(N/20)` to `O(log N)`, about 9 rounds under the same conditions:
```java
int batchSize = HIT_BATCH_SIZE;
while (matchingParagraphs.size() < MAX_RESULTS) {
...
batchSize = Math.min(batchSize * 2, MAX_HIT_BATCH_SIZE);
}
```
And when `hits.length < batchSize` the index is exhausted, so could the loop stop right there? It looks like it currently runs one more search just to get an empty result back.
For what it is worth, the walk only goes deep on installations locked down with `zeppelin.notebook.public=false`. The default is `true`, and then the readers set is empty so `isMember()` returns true (`AuthorizationService.java:271`) and the first batch ends it. Worth noting anyway, since that setting is exactly the population this bug affects.
##########
zeppelin-server/src/main/java/org/apache/zeppelin/search/LuceneSearch.java:
##########
@@ -210,6 +195,55 @@ private List<Map<String, String>> doSearch(
return matchingParagraphs;
}
+ private Map<String, String> toMatch(IndexSearcher searcher, Query query, Analyzer analyzer,
+ Highlighter highlighter, int docId, Document doc)
+ throws IOException, InvalidTokenOffsetsException {
+ String path = doc.get(ID_FIELD);
+ String title = doc.get("title");
+ String text = doc.get(SEARCH_FIELD_TEXT);
+ String header = doc.get(SEARCH_FIELD_TITLE);
+ String fragment = "";
+
+ if (text != null) {
+ TokenStream tokenStream =
+ TokenSources.getTokenStream(
+ searcher.getIndexReader(), docId, SEARCH_FIELD_TEXT, analyzer);
+ TextFragment[] frags = highlighter.getBestTextFragments(tokenStream, text, true, 3);
+ LOGGER.debug(" {} fragments found for query '{}'", frags.length, query);
+ fragment = (frags != null && frags.length > 0) ? frags[0].toString() : "";
+ }
+
+ if (header != null) {
+ TokenStream tokenTitle =
+ TokenSources.getTokenStream(
+ searcher.getIndexReader(), docId, SEARCH_FIELD_TITLE, analyzer);
+ TextFragment[] frgTitle = highlighter.getBestTextFragments(tokenTitle, header, true, 3);
+ header = (frgTitle != null && frgTitle.length > 0) ? frgTitle[0].toString() : "";
+ } else {
+ header = "";
+ }
+ return ImmutableMap.<String, String>builder()
+ .put("id", path)
+ .put("name", title)
+ .put("snippet", fragment)
+ .put("text", text)
+ .put("header", header)
+ .put("title", header)
+ .put("tables", "")
+ .put("output", "")
+ .build();
+ }
+
+ /**
+ * The id of an indexed document is either a noteId or a noteId followed by the paragraph.
+ *
+ * @see #formatId(String, Paragraph)
+ */
+ private static String noteIdOf(String documentId) {
Review Comment:
Could `noteIdOf` move up to `SearchService`?
There are four copies of the same method right now: `LuceneSearch`, `EmbeddingSearch`, and both test classes.
The tests already pull `formatId`, which plays the same role, from the production class as a static import:
```java
// LuceneSearchTest.java:18
import static org.apache.zeppelin.search.LuceneSearch.formatId;
```
Putting `noteIdOf` on `SearchService` as a package-private static and importing it the same way would bring the four copies down to one.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@zeppelin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [ZEPPELIN-6414] Apply the authorization filter before the search cutoff [zeppelin]
Posted by "kkalyan (via GitHub)" <gi...@apache.org>.
kkalyan commented on PR #5427:
URL: https://github.com/apache/zeppelin/pull/5427#issuecomment-5653444892
thanks @dev-donghwan for fixing this!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@zeppelin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [ZEPPELIN-6414] Apply the authorization filter before the search cutoff [zeppelin]
Posted by "tbonelee (via GitHub)" <gi...@apache.org>.
tbonelee commented on PR #5427:
URL: https://github.com/apache/zeppelin/pull/5427#issuecomment-5651745676
Merged into master
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@zeppelin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org