You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@solr.apache.org by "dsmiley (via GitHub)" <gi...@apache.org> on 2026/10/13 00:37:42 UTC
[PR] SOLR-9957: /export supports document transformers in fl [solr]
dsmiley opened a new pull request, #5113:
URL: https://github.com/apache/solr/pull/5113
https://issues.apache.org/jira/browse/SOLR-9957
# Description
`/export` silently ignored document transformers in `fl` (e.g. `[docid]`, `[value]`, `[explain]`), and field aliases like `fl=x:myfield` came back under the original name. `fl` was also split naively on commas, so transformer local params containing commas broke.
# Solution
- Parse `fl` with `SolrReturnFields` (same as `/select`), so transformers, their local params, and aliases parse correctly. `score` is still rejected.
- When `fl` has a transformer, each doc's fields are collected into a `SolrDocument`, the transformer is applied with the global docid, and the wanted fields are written out. Without a transformer, the existing streaming path is unchanged (no per-doc allocation).
- Transformers get a `ResultContext` with the export's searcher, query and request. There is no `DocList`, so transformers needing one (or scores) aren't supported; this is documented.
- Fields a transformer needs (e.g. an alias's source) are fetched even if not listed in `fl`.
- Transformer output flows into tuples seen by the local streaming expression (`expr`).
Behavior change: `fl=x:myfield` now returns `x` rather than `myfield`. Noted in the changelog.
# Tests
- New `TestExportWriter.testDocTransformers`: `[docid]` matches `/select`'s, multi-valued fields survive, aliases and comma-containing local params work, transformer output reaches `expr`, and `score` is still rejected.
- Passing: `org.apache.solr.handler.export.*`, plus `StreamExpressionTest`, `StreamingTest`, `StreamDecoratorTest`; `./gradlew check -x test`.
🤖 This PR was written with [Claude Code](https://claude.com/claude-code) (AI-generated), directed and reviewed by the author.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-9957: /export supports document transformers in fl [solr]
Posted by "dsmiley (via GitHub)" <gi...@apache.org>.
dsmiley commented on code in PR #5113:
URL: https://github.com/apache/solr/pull/5113#discussion_r4253607919
##########
solr/core/src/java/org/apache/solr/handler/export/ExportWriter.java:
##########
@@ -483,33 +497,70 @@ void writeDoc(
throws IOException {
int ord = sortDoc.ord;
LeafReaderContext context = leaves.get(ord);
+ if (transformer == null) {
+ for (FieldWriter fieldWriter : writers) {
+ fieldWriter.write(sortDoc, context, ew);
+ }
+ return;
+ }
Review Comment:
fast-path for previous logic, unchanged
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-9957: /export supports document transformers in fl [solr]
Posted by "dsmiley (via GitHub)" <gi...@apache.org>.
dsmiley commented on PR #5113:
URL: https://github.com/apache/solr/pull/5113#issuecomment-6157049876
> **On line 283:** Unfortunately not: an absent `fl` and `fl=*` both become "wants all fields" in `SolrReturnFields`, and `fl=*` is valid here, so only the raw param distinguishes "not specified".
>
> **On the ref guide:** Good catch. The javabin path (which streaming uses) has no resolver for core types, so e.g. `[subquery]`'s result would come out as a `toString` string; `[child]`'s anonymous children and raw `[json]`/`[xml]` values also don't carry over. I'll call out that only transformers producing simple values, lists, or maps are supported.
done
> **On field order:** Yes. Though `/export` has never followed `fl` order (fields come out in the index's field order); transformer-added or renamed fields go last, same as in `/select`.
>
> 🤖 These replies were drafted with AI assistance (Claude).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-9957: /export supports document transformers in fl [solr]
Posted by "dsmiley (via GitHub)" <gi...@apache.org>.
dsmiley commented on PR #5113:
URL: https://github.com/apache/solr/pull/5113#issuecomment-6156939881
thanks for the review; yes motivated by #5126
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-9957: /export supports document transformers in fl [solr]
Posted by "kotman12 (via GitHub)" <gi...@apache.org>.
kotman12 commented on code in PR #5113:
URL: https://github.com/apache/solr/pull/5113#discussion_r4261864780
##########
solr/core/src/java/org/apache/solr/handler/export/ExportWriter.java:
##########
@@ -268,35 +279,38 @@ private void _write(OutputStream os) throws IOException {
}
}
SolrParams params = req.getParams();
- String fl = params.get("fl");
- String[] fields = null;
-
- if (fl == null) {
+ if (params.get(CommonParams.FL) == null) {
Review Comment:
can you check this on the `SolrReturnFields` object? Just for brevity...
##########
solr/core/src/java/org/apache/solr/handler/export/ExportWriter.java:
##########
@@ -483,49 +497,89 @@ void writeDoc(
throws IOException {
int ord = sortDoc.ord;
LeafReaderContext context = leaves.get(ord);
+ if (transformer == null) {
+ for (FieldWriter fieldWriter : writers) {
+ fieldWriter.write(sortDoc, context, ew);
+ }
+ return;
+ }
+
+ SolrDocument doc = new SolrDocument();
+ EntryWriter docWriter =
+ new EntryWriter() {
+ @Override
+ public EntryWriter put(CharSequence k, Object v) {
+ doc.setField(k.toString(), materialize(v));
+ return this;
+ }
+ };
for (FieldWriter fieldWriter : writers) {
- fieldWriter.write(sortDoc, context, ew);
+ fieldWriter.write(sortDoc, context, docWriter);
+ }
+ transformer.transform(doc, sortDoc.docBase + sortDoc.docId, DocIterationInfo.NONE);
Review Comment:
Will this will return the transformed fields out of `fl` order? Not a huge deal but just pointing it out.
##########
solr/solr-ref-guide/modules/query-guide/pages/exporting-result-sets.adoc:
##########
@@ -101,6 +101,9 @@ Note that sort fields still require docValues, regardless of this setting.
Wildcard patterns can be used for the field list (e.g. `fl=*_i`) and will be expanded to the list of fields that match the pattern and are able to be exported, see <<Field Requirements>>.
+Field aliases (e.g. `fl=id,price:price_d`) and xref:document-transformers.adoc[document transformers] (e.g. `fl=id,[docid]`) are supported, and their output is also visible to a local streaming expression.
+Transformers are applied one document at a time, and those that depend on scores or on a list of matching documents won't work.
Review Comment:
Generally transformers that don't return maps/lists/values won't work, right? I.e. `fl=id,myquery:[subquery]` won't work. The DocTransformer integration uncovers some gaps in the existing ExportWriter/javabin writer vs the text writer. It may not be a common use case here but maybe worth calling out.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-9957: /export supports document transformers in fl [solr]
Posted by "kotman12 (via GitHub)" <gi...@apache.org>.
kotman12 commented on PR #5113:
URL: https://github.com/apache/solr/pull/5113#issuecomment-6156462553
Looks cool, left some comments. I assume this is motivated by the highlighted stream work?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-9957: /export supports document transformers in fl [solr]
Posted by "dsmiley (via GitHub)" <gi...@apache.org>.
dsmiley merged PR #5113:
URL: https://github.com/apache/solr/pull/5113
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org