You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@solr.apache.org by "adityamparikh (via GitHub)" <gi...@apache.org> on 2026/09/16 01:53:23 UTC
[PR] feat(indexing): index-url — index a document set from an allow-listed http(s) URL in both transports (#208) [solr-mcp]
adityamparikh opened a new pull request, #210:
URL: https://github.com/apache/solr-mcp/pull/210
Implements #208: an `index-url` tool that indexes a JSON, CSV, XML or Markdown document set from an http(s) URL into a Solr collection without the payload passing through the model, registered in **both** the STDIO and HTTP transports.
Design record with every decision and its reason: `docs/superpowers/specs/2026-09-15-url-ingestion-design.md` (in this PR).
## Why
The four inline indexing tools take the payload as a tool-call argument, so the model emits every byte. Measured when #197 was closed: 61 records took over two minutes, of which Solr took under a second. The server's differentiator is the onboarding loop (look at data, design schema, index, verify, search) and it broke at the indexing step, with no workaround in chat-only clients such as Claude Desktop. A URL argument is ~20 tokens regardless of payload size.
## What
```
index-url(collection, url, format?)
```
- **Allow-listed hosts**, checked on the requested URL and on every redirect hop. `SOLR_INDEX_URL_ALLOWED_HOSTS` takes exact hosts, `*.suffix` patterns or `*`. Default: `raw.githubusercontent.com,*.githubusercontent.com,github.com`, so the tutorial URL works with no configuration and an HTTP deployment is not an SSRF primitive on day one.
- **Link-local and cloud-metadata addresses refused** on every hop even with `*`.
- **No credentials or caller headers, ever**; embedded `user:pass@` is rejected. Only `Accept` and `User-Agent` are sent.
- **Redirects:** five followed, the sixth is an error, `https`→`http` refused.
- **Non-2xx is an error before any body is read** (a GitHub 404 body would otherwise index as a CSV row).
- **Size cap** `SOLR_INDEX_URL_MAX_BYTES`, default 10 MB, checked from `Content-Length` and while reading. Over the cap, the error points at `bin/solr post` / the `/update` handler.
- **Format:** explicit `format` > requested URL extension (query string ignored) > final URL extension > `Content-Type`; `text/plain` and `text/html` never resolve.
- **Body read in memory** and handed to the existing document creators through a new shared `IndexingService.indexPayload`, which the four inline tools now also use (`IndexingServiceTest` passes unmodified). Nothing is indexed unless the whole document parses, so every fetch or parse error truthfully says so.
- **HTTP client:** Spring `RestClient` over `HttpURLConnection` (already on the classpath), whose read timeout covers the body; redirect following disabled at the connection so the policy sees each hop. `SOLR_INDEX_URL_CONNECT_TIMEOUT` 10s, `SOLR_INDEX_URL_READ_TIMEOUT` 30s.
- **Guidance:** the `index-data` prompt and the server instructions route URLs under the cap to `index-url`, larger datasets to `bin/solr post` or `curl` against `/update` (run where the file is, which also covers Claude Desktop local files), and pasted data to the inline tools.
## Relationship to earlier decisions
- **#194** closed a STDIO-only `index-file` for giving the transports different surfaces. This PR adds no transport-specific tool and reverses nothing recorded there; the local-file case is served by `bin/solr post` guidance.
- **#197** kept per-format inline tools for token shape. `index-url` carries no payload, so one tool with an optional `format` suffices.
- **#205** (forward CSV/XML to Solr's handlers): no conflict. This PR adds no parser code and touches the four inline tool methods only to extract `indexPayload`. After #205 merges, a follow-up can proxy the fetched body straight to Solr's `/update` for JSON/CSV/XML and drop the cap for those formats with the same tool contract.
- A first version that streamed through server-side parsers with a hand-written watchdog, kept `index-file`, and allowed any host by default is preserved at `wip/index-url-streaming` and is not proposed.
## Threat model
`THREAT_MODEL.md` §8.5 said the client "cannot inject a target URL" at critical severity; this tool exercises that deliberately behind an allow-list. The PR narrows §8.5 to the Solr backend and its credentials, adds the allow-listed fetch and the DNS-rebinding window as a §9 bounded property, adds the `*`-on-an-internal-network misuse to §11, the `index-url` non-finding to §11a, a dated entry to §12, qualifies §13, adds the three knobs to §5a, corrects the tool counts in §1/§5a, and updates `docs/security/stdio.md` and `http.md`.
**Two defaults are the author's choice and need maintainer confirmation:** the allow-list contents and the 10 MB cap. Both are single constants.
## Tests
- New, all running natively: `UrlTargetPolicyTest` (21), `IndexFormatsTest` (14), `UrlFetcherTest` (17, real JDK `HttpServer`), `UrlIndexingIntegrationTest` (5, Testcontainers Solr).
- `UrlIndexingServiceTest` (14, Mockito, the only class skipped natively).
- Both MCP client transports assert `index-url` and its hints through the shared base and add a URL round trip plus a refused metadata address.
- `IndexingServiceTest` unmodified.
## Verification
```
./gradlew build BUILD SUCCESSFUL — 478 tests, 0 failures, 7 skipped
(all 7 in OtlpExportIntegrationTest, @Disabled on main since 375a710)
./gradlew nativeTest -Pnative BUILD SUCCESSFUL — 317 successful, 0 failed, 158 skipped
(skipped = the pre-existing @DisabledInNativeImage Mockito classes and the
@Disabled OTLP class, plus UrlIndexingServiceTest, the one new Mockito class)
```
Manual STDIO run of the fat jar against a local Solr, indexing the raw GitHub `shows.json`:
```
[tools/list] index-url present
[index-url] isError=False :: Successfully indexed 61 of 61 documents into collection 'shows-url-manual'. Indexed field names ...
[search] isError=False :: {"numFound":61,...}
[index-url http://example.invalid/x.json] isError=True :: The URL's host is not on this server's allow-list. Allowed by default: raw.githubusercontent.com, *.githubusercontent.com, github.com. ...
```
🤖 Generated with [Claude Code](https://claude.com/claude-code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] feat(indexing): index-url — index a document set from an allow-listed http(s) URL in both transports (#208) [solr-mcp]
Posted by "adityamparikh (via GitHub)" <gi...@apache.org>.
adityamparikh commented on PR #210:
URL: https://github.com/apache/solr-mcp/pull/210#issuecomment-6299272748
Parked as a draft: `index-url` is blocked on MCP Tasks, tracked in #274 (see the comment on #208). A transfer can run for up to `solr.index-url.total-timeout` (5 min by default) with no progress feedback, so this waits for the MCP Java SDK and Spring AI to support the Tasks extension rather than merging as a blocking call. The streaming, allow-list and commit-after-complete work here carries over. I'll rebase it onto `main` from time to time so it does not go stale.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] feat(indexing): index-url — index a document set from an allow-listed http(s) URL in both transports (#208) [solr-mcp]
Posted by "github-advanced-security[bot] (via GitHub)" <gi...@apache.org>.
github-advanced-security[bot] commented on code in PR #210:
URL: https://github.com/apache/solr-mcp/pull/210#discussion_r4381647200
##########
src/main/java/org/apache/solr/mcp/server/indexing/UrlTargetPolicy.java:
##########
@@ -0,0 +1,127 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.mcp.server.indexing;
+
+import java.net.InetAddress;
+import java.net.URI;
+import java.net.UnknownHostException;
+import java.util.List;
+import java.util.Locale;
+import java.util.Set;
+
+/**
+ * Pure checks that decide whether {@code index-url} may fetch a URL: absolute http(s) with a host
+ * and no embedded credentials, a host on the operator's allow-list, and no link-local or
+ * cloud-metadata address even when the allow-list is {@code *}. The caller resolves the host, so
+ * this class never performs I/O and the same checks run on every redirect hop.
+ */
+final class UrlTargetPolicy {
+
+ static final String INVALID_URL = "Provide an absolute http or https URL.";
+ static final String EMBEDDED_CREDENTIALS =
+ "Remove the credentials from the URL; this server never sends credentials.";
+ static final String HOST_NOT_ALLOWED =
+ "The URL's host is not on this server's allow-list. Allowed by default: "
+ + "raw.githubusercontent.com, *.githubusercontent.com, github.com. The operator can change "
+ + "SOLR_INDEX_URL_ALLOWED_HOSTS (use * to allow any host). Nothing was indexed.";
+ static final String REFUSED_ADDRESS =
+ "This server does not fetch link-local or cloud-metadata addresses.";
+
+ /** The EC2 IPv6 instance-metadata address; not link-local, so listed explicitly. */
+ private static final Set<InetAddress> METADATA_ADDRESSES =
+ Set.of(
+ literal("fd00:ec2::254"), // AWS IPv6
Review Comment:
## SonarCloud / IP addresses should not be hardcoded
<!--SONAR_ISSUE_KEY:AaF5uESv2c1245phiFey-->Make sure using this hardcoded IP address is safe here. <p>See more on <a href="https://sonarcloud.io/project/issues?id=apache_solr-mcp&issues=AaF5uESv2c1245phiFey&open=AaF5uESv2c1245phiFey&pullRequest=210">SonarQube Cloud</a></p>
[Show more details](https://github.com/apache/solr-mcp/security/code-scanning/2)
##########
src/main/java/org/apache/solr/mcp/server/indexing/UrlTargetPolicy.java:
##########
@@ -0,0 +1,127 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.mcp.server.indexing;
+
+import java.net.InetAddress;
+import java.net.URI;
+import java.net.UnknownHostException;
+import java.util.List;
+import java.util.Locale;
+import java.util.Set;
+
+/**
+ * Pure checks that decide whether {@code index-url} may fetch a URL: absolute http(s) with a host
+ * and no embedded credentials, a host on the operator's allow-list, and no link-local or
+ * cloud-metadata address even when the allow-list is {@code *}. The caller resolves the host, so
+ * this class never performs I/O and the same checks run on every redirect hop.
+ */
+final class UrlTargetPolicy {
+
+ static final String INVALID_URL = "Provide an absolute http or https URL.";
+ static final String EMBEDDED_CREDENTIALS =
+ "Remove the credentials from the URL; this server never sends credentials.";
+ static final String HOST_NOT_ALLOWED =
+ "The URL's host is not on this server's allow-list. Allowed by default: "
+ + "raw.githubusercontent.com, *.githubusercontent.com, github.com. The operator can change "
+ + "SOLR_INDEX_URL_ALLOWED_HOSTS (use * to allow any host). Nothing was indexed.";
+ static final String REFUSED_ADDRESS =
+ "This server does not fetch link-local or cloud-metadata addresses.";
+
+ /** The EC2 IPv6 instance-metadata address; not link-local, so listed explicitly. */
+ private static final Set<InetAddress> METADATA_ADDRESSES =
+ Set.of(
+ literal("fd00:ec2::254"), // AWS IPv6
+ literal("100.100.100.200"), // Alibaba Cloud
+ literal("168.63.129.16")); // Azure WireServer
Review Comment:
## SonarCloud / IP addresses should not be hardcoded
<!--SONAR_ISSUE_KEY:AaF5uESv2c1245phiFe0-->Make sure using this hardcoded IP address is safe here. <p>See more on <a href="https://sonarcloud.io/project/issues?id=apache_solr-mcp&issues=AaF5uESv2c1245phiFe0&open=AaF5uESv2c1245phiFe0&pullRequest=210">SonarQube Cloud</a></p>
[Show more details](https://github.com/apache/solr-mcp/security/code-scanning/3)
##########
src/main/java/org/apache/solr/mcp/server/indexing/UrlTargetPolicy.java:
##########
@@ -0,0 +1,127 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.mcp.server.indexing;
+
+import java.net.InetAddress;
+import java.net.URI;
+import java.net.UnknownHostException;
+import java.util.List;
+import java.util.Locale;
+import java.util.Set;
+
+/**
+ * Pure checks that decide whether {@code index-url} may fetch a URL: absolute http(s) with a host
+ * and no embedded credentials, a host on the operator's allow-list, and no link-local or
+ * cloud-metadata address even when the allow-list is {@code *}. The caller resolves the host, so
+ * this class never performs I/O and the same checks run on every redirect hop.
+ */
+final class UrlTargetPolicy {
+
+ static final String INVALID_URL = "Provide an absolute http or https URL.";
+ static final String EMBEDDED_CREDENTIALS =
+ "Remove the credentials from the URL; this server never sends credentials.";
+ static final String HOST_NOT_ALLOWED =
+ "The URL's host is not on this server's allow-list. Allowed by default: "
+ + "raw.githubusercontent.com, *.githubusercontent.com, github.com. The operator can change "
+ + "SOLR_INDEX_URL_ALLOWED_HOSTS (use * to allow any host). Nothing was indexed.";
+ static final String REFUSED_ADDRESS =
+ "This server does not fetch link-local or cloud-metadata addresses.";
+
+ /** The EC2 IPv6 instance-metadata address; not link-local, so listed explicitly. */
+ private static final Set<InetAddress> METADATA_ADDRESSES =
+ Set.of(
+ literal("fd00:ec2::254"), // AWS IPv6
+ literal("100.100.100.200"), // Alibaba Cloud
Review Comment:
## SonarCloud / IP addresses should not be hardcoded
<!--SONAR_ISSUE_KEY:AaF5uESv2c1245phiFez-->Make sure using this hardcoded IP address is safe here. <p>See more on <a href="https://sonarcloud.io/project/issues?id=apache_solr-mcp&issues=AaF5uESv2c1245phiFez&open=AaF5uESv2c1245phiFez&pullRequest=210">SonarQube Cloud</a></p>
[Show more details](https://github.com/apache/solr-mcp/security/code-scanning/1)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] feat(indexing): index-url — index a document set from an allow-listed http(s) URL in both transports (#208) [solr-mcp]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #210:
URL: https://github.com/apache/solr-mcp/pull/210#issuecomment-6425552055
## [](https://sonarcloud.io/dashboard?id=apache_solr-mcp&pullRequest=210) **Quality Gate passed**
Issues
 [48 New issues](https://sonarcloud.io/project/issues?id=apache_solr-mcp&pullRequest=210&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_solr-mcp&pullRequest=210&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_solr-mcp&pullRequest=210&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_solr-mcp&pullRequest=210&metric=new_coverage&view=list)
 [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_solr-mcp&pullRequest=210&metric=new_duplicated_lines_density&view=list)
<!-- sqra-placement-anchor -->
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_solr-mcp&pullRequest=210)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org