You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@solr.apache.org by "prateeksinghalgit (via GitHub)" <gi...@apache.org> on 2025/10/09 20:09:36 UTC
[PR] SOLR-17949: Add Azure Blob Storage backup repository module [solr]
prateeksinghalgit opened a new pull request, #3750:
URL: https://github.com/apache/solr/pull/3750
<!--
_(If you are a project committer then you may remove some/all of the following template.)_
Before creating a pull request, please file an issue in the ASF Jira system for Solr:
* https://issues.apache.org/jira/projects/SOLR
For something minor (i.e. that wouldn't be worth putting in release notes), you can skip JIRA.
To create a Jira issue, you will need to create an account there first.
The title of the PR should reference the Jira issue number in the form:
* SOLR-####: <short description of problem or changes>
SOLR must be fully capitalized. A short description helps people scanning pull requests for items they can work on.
Properly referencing the issue in the title ensures that Jira is correctly updated with code review comments and commits. -->
Implements Azure Blob Storage backup repository as discussed in [SOLR-17949](https://issues.apache.org/jira/browse/SOLR-17949).
# Description
This PR adds a new backup repository implementation for Azure Blob Storage, enabling Solr collections to be backed up to and restored from Microsoft Azure.
**Key Features:**
- Full backup/restore functionality to Azure Blob Storage
- Support for 4 authentication methods (Connection String, Account Key, SAS Token, Azure Identity)
- Incremental backup support with versioning
- Data integrity verification (checksum validation)
- Compatible with Azurite emulator for local testing
- Comprehensive documentation and 76 passing unit tests
# Solution
The implementation follows Solr's `BackupRepository` interface pattern, similar to existing S3 and GCS repository modules:
- **BlobBackupRepository**: Main class implementing Solr's `BackupRepository` interface
- **BlobStorageClient**: Wrapper for Azure SDK, providing file operations
- **BlobIndexInput**: Custom Lucene `IndexInput` for reading from Azure blobs
- **BlobOutputStream**: Custom output stream for writing to Azure blobs
- **Authentication**: Supports 4 methods via flexible configuration in `solr.xml`
All streaming operations are compatible with Solr's `ResumableInputStream` for fault-tolerant transfers.
**Implementation stats:**
- 8 implementation files (1,606 LOC)
- 8 test files (2,180 LOC)
- All dependencies Apache 2.0 licensed
# Tests
**Unit Tests:** 76/76 passing (100%)
```bash
./gradlew :solr:modules:blob-repository:test
# Result: BUILD SUCCESSFUL - 76 test(s)
```
**Test Coverage:**
- Basic read/write operations
- Large file handling (1GB+)
- Binary data integrity
- Concurrent operations
- Stream lifecycle (close/resume behavior)
- Incremental backups
- All 4 authentication methods
- Integration with Azurite (local emulator)
- Integration with real Azure Blob Storage
**Testing Instructions:**
Can be tested locally with Azurite emulator (no Azure account needed) or with real Azure Blob Storage. See `solr/modules/blob-repository/README.md` for detailed setup instructions.
# Checklist
Please review the following and check all that apply:
- [x] I have reviewed the guidelines for [How to Contribute](https://github.com/apache/solr/blob/main/CONTRIBUTING.md) and my code conforms to the standards described there to the best of my ability.
- [x] I have created a Jira issue and added the issue ID to my pull request title.
- [ ] I have given Solr maintainers [access](https://help.github.com/en/articles/allowing-changes-to-a-pull-request-branch-created-from-a-fork) to contribute to my PR branch. (optional but recommended, not available for branches on forks living under an organisation)
- [x] I have developed this patch against the `main` branch.
- [x] I have run `./gradlew :solr:modules:blob-repository:check` (module-specific check passed).
- [x] I have added tests for my changes.
- [x] I have added documentation for the [Reference Guide](https://github.com/apache/solr/tree/main/solr/solr-ref-guide)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17949: Add Azure Blob Storage backup repository module [solr]
Posted by "janhoy (via GitHub)" <gi...@apache.org>.
janhoy commented on PR #3750:
URL: https://github.com/apache/solr/pull/3750#issuecomment-3556348621
Don’t have time to review but asked copilot for an opinion 😉
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17949: Add Azure Blob Storage backup repository module [solr]
Posted by "prateeksinghalgit (via GitHub)" <gi...@apache.org>.
prateeksinghalgit commented on code in PR #3750:
URL: https://github.com/apache/solr/pull/3750#discussion_r2563739297
##########
solr/solr-ref-guide/modules/deployment-guide/pages/backup-restore.adoc:
##########
@@ -383,7 +383,7 @@ If the status is anything other than "success", an error message will explain wh
Solr provides a repository abstraction to allow users to backup and restore their data to a variety of different storage systems.
For example, a Solr cluster running on a local filesystem (e.g., EXT3) can store backup data on the same disk, on a remote network-mounted drive, or in some popular "cloud storage" providers, depending on the 'repository' implementation chosen.
-Solr offers multiple different repository implementations out of the box (`LocalFileSystemRepository`, `GCSBackupRepository` and `S3BackupRepository`), and allows users to create plugins for their own storage systems as needed. It is also possible to create a `DelegatingBackupRepository` that delegates to another `BackupRepository` and adds or modifies some behavior on top of it.
+Solr offers multiple different repository implementations out of the box (`LocalFileSystemRepository`, `GCSBackupRepository`, `S3BackupRepository`, and `BlobBackupRepository`), and allows users to create plugins for their own storage systems as needed. It is also possible to create a `DelegatingBackupRepository` that delegates to another `BackupRepository` and adds or modifies some behavior on top of it.
Review Comment:
@janhoy Thank you for taking the time and effort to review the PR and providing valuable feedback.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17949: Add Azure Blob Storage backup repository module [solr]
Posted by "prateeksinghalgit (via GitHub)" <gi...@apache.org>.
prateeksinghalgit commented on code in PR #3750:
URL: https://github.com/apache/solr/pull/3750#discussion_r3232186613
##########
solr/modules/azure-blob-repository/src/java/org/apache/solr/azureblob/AzureBlobIndexInput.java:
##########
@@ -0,0 +1,200 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.azureblob;
+
+import java.io.EOFException;
+import java.io.IOException;
+import java.io.InputStream;
+import java.util.LinkedHashMap;
+import java.util.Map;
+import org.apache.lucene.store.IndexInput;
+
+class AzureBlobIndexInput extends IndexInput {
+
+ private static final int MIN_PAGE_SIZE = 4 * 1024;
+ private static final int DEFAULT_PAGE_SIZE = 512 * 1024;
+ private static final int MAX_CACHED_PAGES = 128;
+
+ private final String path;
+ private final AzureBlobStorageClient client;
+ private final long length;
+ private final int pageSize;
+ private final LruPageCache cache;
+
+ private long position = 0L;
+ private boolean closed = false;
+
+ AzureBlobIndexInput(String path, AzureBlobStorageClient client, long length) {
+ this(path, client, length, DEFAULT_PAGE_SIZE, MAX_CACHED_PAGES);
+ }
+
+ AzureBlobIndexInput(
+ String path, AzureBlobStorageClient client, long length, int pageSize, int maxCachedPages) {
+ super(path);
+ this.path = path;
+ this.client = client;
+ this.length = length;
+ this.pageSize = Math.max(MIN_PAGE_SIZE, pageSize);
+ this.cache = new LruPageCache(maxCachedPages);
+ }
+
+ @Override
+ public void close() throws IOException {
+ closed = true;
+ cache.clear();
+ }
+
+ @Override
+ public long getFilePointer() {
+ return position;
+ }
+
+ @Override
+ public void seek(long pos) throws IOException {
+ ensureOpen();
+ if (pos < 0 || pos > length) {
+ throw new IOException("Seek position out of bounds: " + pos);
+ }
+
+ position = pos;
+ }
+
+ @Override
+ public long length() {
+ return length;
+ }
+
+ @Override
+ public IndexInput slice(String sliceDescription, long offset, long length) throws IOException {
+ ensureOpen();
+ if (offset < 0 || length < 0 || offset + length > this.length) {
+ throw new IOException("Slice out of bounds: offset=" + offset + ", length=" + length);
+ }
+
+ AzureBlobIndexInput slice =
+ new AzureBlobIndexInput(
+ getFullSliceDescription(sliceDescription), client, length, pageSize, MAX_CACHED_PAGES);
+
+ slice.position = 0L;
Review Comment:
updated the logic for this as AzureBlobIndexInput uses BufferedIndexInput
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17949: Add Azure Blob Storage backup repository module [solr]
Posted by "prateeksinghalgit (via GitHub)" <gi...@apache.org>.
prateeksinghalgit commented on code in PR #3750:
URL: https://github.com/apache/solr/pull/3750#discussion_r3232162138
##########
solr/modules/azure-blob-repository/src/java/org/apache/solr/azureblob/AzureBlobStorageClient.java:
##########
@@ -0,0 +1,529 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.azureblob;
+
+import com.azure.core.credential.TokenCredential;
+import com.azure.identity.DefaultAzureCredentialBuilder;
+import com.azure.storage.blob.BlobClient;
+import com.azure.storage.blob.BlobContainerClient;
+import com.azure.storage.blob.BlobServiceClient;
+import com.azure.storage.blob.BlobServiceClientBuilder;
+import com.azure.storage.blob.models.BlobItem;
+import com.azure.storage.blob.models.BlobStorageException;
+import com.azure.storage.blob.models.ListBlobsOptions;
+import com.google.common.annotations.VisibleForTesting;
+import java.io.ByteArrayInputStream;
+import java.io.FilterInputStream;
+import java.io.InputStream;
+import java.io.OutputStream;
+import java.lang.invoke.MethodHandles;
+import java.util.Collection;
+import java.util.HashSet;
+import java.util.Locale;
+import java.util.Set;
+import java.util.stream.Collectors;
+import org.apache.solr.common.util.ResumableInputStream;
+import org.apache.solr.common.util.StrUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Creates a {@link BlobServiceClient} for communicating with Azure Blob Storage. Utilizes the
+ * default Azure credential provider chain.
+ */
+public class AzureBlobStorageClient {
+
+ private static final Logger log = LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+
+ static final String BLOB_FILE_PATH_DELIMITER = "/";
+ private static final int HTTP_NOT_FOUND = 404;
+ private static final int HTTP_CONFLICT = 409;
+ private static final int SKIP_BUFFER_SIZE = 8192;
+ private static final int DELETE_BATCH_SIZE = 1000;
+
+ private static final com.azure.core.http.HttpClient SHARED_HTTP_CLIENT =
Review Comment:
removed the static http client
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17949: Add Azure Blob Storage backup repository module [solr]
Posted by "psalagnac (via GitHub)" <gi...@apache.org>.
psalagnac commented on PR #3750:
URL: https://github.com/apache/solr/pull/3750#issuecomment-4326968039
Thanks for adding me as a reviewer.
I've been pretty busy recently and I haven't got the chance to do the review yet. I'll try to do a complete review before end of next week.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17949: Add Azure Blob Storage backup repository module [solr]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #3750:
URL: https://github.com/apache/solr/pull/3750#issuecomment-4064286390
This PR has had no activity for 60 days and is now labeled as stale. Any new activity will remove the stale label. To attract more reviewers, please tag people who might be familiar with the code area and/or notify the dev@solr.apache.org mailing list. To exempt this PR from being marked as stale, make it a draft PR or add the label "exempt-stale". If left unattended, this PR will be closed after another 60 days of inactivity. Thank you for your contribution!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17949: Add Azure Blob Storage backup repository module [solr]
Posted by "prateeksinghalgit (via GitHub)" <gi...@apache.org>.
prateeksinghalgit commented on code in PR #3750:
URL: https://github.com/apache/solr/pull/3750#discussion_r2563445786
##########
solr/modules/azure-blob-repository/src/test/org/apache/solr/azureblob/AbstractAzureBlobClientTest.java:
##########
@@ -0,0 +1,249 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.azureblob;
+
+import com.azure.core.http.HttpClient;
+import com.azure.core.http.okhttp.OkHttpAsyncHttpClientBuilder;
+import com.azure.storage.blob.BlobServiceClient;
+import com.azure.storage.blob.BlobServiceClientBuilder;
+import com.carrotsearch.randomizedtesting.ThreadFilter;
+import com.carrotsearch.randomizedtesting.annotations.ThreadLeakFilters;
+import java.io.IOException;
+import java.io.OutputStream;
+import java.net.Socket;
+import java.nio.charset.StandardCharsets;
+import okhttp3.OkHttpClient;
+import org.apache.solr.SolrTestCaseJ4;
+import org.junit.After;
+import org.junit.AfterClass;
+import org.junit.Before;
+import org.junit.BeforeClass;
+import org.junit.Rule;
+import org.junit.rules.TemporaryFolder;
+
+import static org.junit.Assume.assumeTrue;
+
+/**
+ * Abstract class for tests with Azure Blob Storage emulator.
+ *
+ * <p>Uses ThreadLeakFilters to ignore OkHttp's global singleton threads (TaskRunner and Okio
+ * Watchdog) which are JVM-wide and cannot be shut down per-client. This is a standard pattern in
+ * Solr tests for HTTP clients with global thread pools.
+ */
+@ThreadLeakFilters(
+ filters = {
+ AbstractAzureBlobClientTest.OkHttpThreadLeakFilter.class,
+ })
+public class AbstractAzureBlobClientTest extends SolrTestCaseJ4 {
+
+ protected String containerName;
+
+ @Rule public TemporaryFolder tempFolder = new TemporaryFolder();
+
+ AzureBlobStorageClient client;
+ private static String connectionString;
+ // Shared OkHttpClient across all tests to reuse the global thread pools
+ private static OkHttpClient sharedOkHttpClient;
+ protected org.apache.solr.client.solrj.cloud.SocketProxy proxy;
+
+ @BeforeClass
+ public static void setUpClass() {
+ // Create a single shared OkHttpClient for all tests
+ // This reuses OkHttp's global thread pools efficiently
+ sharedOkHttpClient = new OkHttpClient.Builder().build();
+ }
+
+ @Before
+ public void setUpClient() throws Exception {
+ // Check if Azurite is running before attempting to connect
+ assumeAzuriteIsRunning();
+
+ setAzureTestCredentials();
+
+ // Use Azurite connection string for local testing
+ connectionString =
+ "DefaultEndpointsProtocol=http;AccountName=devstoreaccount1;AccountKey=Eby8vdM02xNOcqFlqUwJPLlmEtlCDXJ1OUzFT50uSRZ6IFsuFq2UVErCz4I6tq/K1SZFPTOtr/KBHBeksoGMGw==;BlobEndpoint=http://localhost:10000/devstoreaccount1;";
+
+ // Put a proxy in front of Azurite to simulate connection loss like S3 tests
+ proxy = new org.apache.solr.client.solrj.cloud.SocketProxy();
+ proxy.open(new java.net.URI(getBlobServiceUrl()));
+
+ // Reuse the shared OkHttpClient
+ HttpClient httpClient = new OkHttpAsyncHttpClientBuilder(sharedOkHttpClient).build();
+
+ // Route Blob endpoint through the proxy by adjusting the connection string
+ String proxiedConn = connectionString.replace(":10000", ":" + proxy.getListenPort());
+ BlobServiceClient blobServiceClient =
+ new BlobServiceClientBuilder()
+ .connectionString(proxiedConn)
+ .httpClient(httpClient)
+ .buildClient();
+
+ containerName = "test-" + java.util.UUID.randomUUID();
+ client = new AzureBlobStorageClient(blobServiceClient, containerName);
+ }
+
+ /**
+ * Set up Azure test credentials to avoid using real Azure credentials during testing. Similar to
+ * how S3 tests use ProfileFileSystemSetting to avoid polluting the test environment.
+ */
+ public static void setAzureTestCredentials() {
+ // Set test Azure credentials to avoid using real credentials
+ System.setProperty("AZURE_CLIENT_ID", "test-client-id");
+ System.setProperty("AZURE_TENANT_ID", "test-tenant-id");
+ System.setProperty("AZURE_CLIENT_SECRET", "test-client-secret");
+
+ // Set Azurite-specific environment variables
+ System.setProperty(
+ "AZURE_STORAGE_CONNECTION_STRING",
+ "DefaultEndpointsProtocol=http;AccountName=devstoreaccount1;AccountKey=Eby8vdM02xNOcqFlqUwJPLlmEtlCDXJ1OUzFT50uSRZ6IFsuFq2UVErCz4I6tq/K1SZFPTOtr/KBHBeksoGMGw==;BlobEndpoint=http://localhost:10000/devstoreaccount1;");
+ }
+
+ @After
+ public void tearDownClient() {
+ if (client != null) {
+ try {
+ client.deleteContainerForTests();
+ } catch (Throwable ignored) {
+ }
+ client.close();
+ }
+ if (proxy != null) {
+ proxy.close();
+ proxy = null;
+ }
+ // Note: We don't clean up sharedOkHttpClient here as it's shared across all tests
+ // It will be cleaned up in afterAll()
+ }
+
+ /** Simulate a connection loss on the proxy similar to S3 tests. */
+ void initiateBlobConnectionLoss() throws AzureBlobException {
+ if (proxy != null) {
+ proxy.halfClose();
+ }
+ }
+
+ @AfterClass
+ public static void afterAll() {
+ // Clean up the shared OkHttpClient
+ if (sharedOkHttpClient != null) {
+ // Shutdown dispatcher's executor service
+ sharedOkHttpClient.dispatcher().executorService().shutdown();
+ // Cancel all queued calls
+ sharedOkHttpClient.dispatcher().cancelAll();
+ // Evict all connections
+ sharedOkHttpClient.connectionPool().evictAll();
+ // Close cache if present
+ try {
+ if (sharedOkHttpClient.cache() != null) {
+ sharedOkHttpClient.cache().close();
+ }
+ } catch (Throwable ignored) {
+ }
+ // Wait for threads to terminate
+ try {
+ sharedOkHttpClient
+ .dispatcher()
+ .executorService()
+ .awaitTermination(2, java.util.concurrent.TimeUnit.SECONDS);
+ } catch (Throwable ignored) {
+ }
+ sharedOkHttpClient = null;
+ }
+
+ // Shutdown Reactor schedulers to clean up background threads
+ try {
+ reactor.core.scheduler.Schedulers.shutdownNow();
+ // Give threads time to terminate
+ Thread.sleep(100);
+ } catch (Throwable ignored) {
+ }
+ }
+
+ /**
+ * Helper method to push a string to Azure Blob Storage.
+ *
+ * @param path Destination path in blob storage.
+ * @param content Arbitrary content for the test.
+ */
+ void pushContent(String path, String content) throws AzureBlobException {
+ pushContent(path, content.getBytes(StandardCharsets.UTF_8));
+ }
+
+ void pushContent(String path, byte[] content) throws AzureBlobException {
+ try (OutputStream output = client.pushStream(path)) {
+ output.write(content);
+ } catch (IOException e) {
+ throw new AzureBlobException("Failed to write content", e);
+ }
+ }
+
+ /** Get the connection string for tests that need direct access to the blob service. */
+ static String getConnectionString() {
+ return connectionString;
+ }
+
+ /** Get the blob service URL for tests that need direct access. */
+ String getBlobServiceUrl() {
+ return "http://localhost:10000";
+ }
+
+ /**
+ * Check if Azurite is running on localhost:10000. If not, skip the test.
+ *
+ * <p>Azurite must be started manually for these tests to run. This allows tests to pass in CI
+ * environments where Azurite may not be available.
+ */
+ private void assumeAzuriteIsRunning() {
+ boolean azuriteRunning = false;
+ try (Socket socket = new Socket("localhost", 10000)) {
Review Comment:
made the change to use testcontainers, it basically azurite in a docker container. Testcontainers automatically maps it to a random free host port, retrieving it via azuriteContainer.getMappedPort(BLOB_SERVICE_PORT) in getBlobServiceUrl().
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17949: Add Azure Blob Storage backup repository module [solr]
Posted by "prateeksinghalgit (via GitHub)" <gi...@apache.org>.
prateeksinghalgit commented on PR #3750:
URL: https://github.com/apache/solr/pull/3750#issuecomment-4772835763
@janhoy could you please approve workflow runs on this fork PR again. I had to make updates to the pr. Also I did run the ci jobs locally with all the flags to make sure .
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17949: Add Azure Blob Storage backup repository module [solr]
Posted by "prateeksinghalgit (via GitHub)" <gi...@apache.org>.
prateeksinghalgit commented on code in PR #3750:
URL: https://github.com/apache/solr/pull/3750#discussion_r3232168396
##########
solr/modules/azure-blob-repository/src/java/org/apache/solr/azureblob/AzureBlobOutputStream.java:
##########
@@ -0,0 +1,269 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.azureblob;
+
+import com.azure.storage.blob.BlobClient;
+import com.azure.storage.blob.models.BlobStorageException;
+import com.azure.storage.blob.specialized.BlockBlobClient;
+import java.io.ByteArrayInputStream;
+import java.io.IOException;
+import java.io.OutputStream;
+import java.lang.invoke.MethodHandles;
+import java.nio.ByteBuffer;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Base64;
+import java.util.List;
+import java.util.UUID;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * OutputStream implementation for Azure Blob Storage using block blobs. Supports chunked uploads
+ * for large files.
+ */
+public class AzureBlobOutputStream extends OutputStream {
+ private static final Logger log = LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+
+ static final int BLOCK_SIZE = 4 * 1024 * 1024;
+
+ private final BlobClient blobClient;
+ private final String blobPath;
+ private volatile boolean closed;
+ private final ByteBuffer buffer;
+ private BlockUpload blockUpload;
+ private boolean committed;
+
+ public AzureBlobOutputStream(BlobClient blobClient, String blobPath) {
+ this.blobClient = blobClient;
+ this.blobPath = blobPath;
+ this.closed = false;
+ this.buffer = ByteBuffer.allocate(BLOCK_SIZE);
+ this.blockUpload = null;
+ this.committed = false;
+
+ if (log.isDebugEnabled()) {
+ log.debug("Created BlobOutputStream for blobPath '{}'", blobPath);
+ }
+ }
+
+ @Override
+ public void write(int b) throws IOException {
+ if (closed) {
+ throw new IOException("Stream closed");
+ }
+
+ buffer.put((byte) b);
+
+ if (!buffer.hasRemaining()) {
+ uploadBlock();
+ }
+ }
+
+ @Override
+ public void write(byte[] b, int off, int len) throws IOException {
+ if (closed) {
+ throw new IOException("Stream closed");
+ }
+
+ if (outOfRange(off, b.length) || len < 0 || outOfRange(off + len, b.length)) {
+ throw new IndexOutOfBoundsException();
+ } else if (len == 0) {
+ return;
+ }
+
+ int currentOffset = off;
+ int lenRemaining = len;
+ while (buffer.remaining() < lenRemaining) {
+ int firstPart = buffer.remaining();
+ buffer.put(b, currentOffset, firstPart);
+ uploadBlock();
+
+ currentOffset += firstPart;
+ lenRemaining -= firstPart;
+ }
+ if (lenRemaining > 0) {
+ buffer.put(b, currentOffset, lenRemaining);
+ }
+ }
+
+ private static boolean outOfRange(int off, int len) {
+ return off < 0 || off > len;
+ }
+
+ private void uploadBlock() throws IOException {
+ int size = buffer.position() - buffer.arrayOffset();
+
+ if (size == 0) {
+ return;
+ }
+
+ if (blockUpload == null) {
+ if (log.isDebugEnabled()) {
+ log.debug("New block upload for blobPath '{}'", blobPath);
+ }
+
+ blockUpload = newBlockUpload();
+ }
+
+ try (ByteArrayInputStream inputStream =
+ new ByteArrayInputStream(buffer.array(), buffer.arrayOffset(), size)) {
+ blockUpload.uploadBlock(inputStream, size);
+ } catch (BlobStorageException e) {
+ if (blockUpload != null) {
+ blockUpload.abort();
+ if (log.isDebugEnabled()) {
+ log.debug("Block upload aborted for blobPath '{}'.", blobPath);
+ }
+ }
+
+ throw new IOException(
+ "Failed to upload block", AzureBlobStorageClient.handleBlobException(e));
+ }
+
+ buffer.clear();
+ }
+
+ @Override
+ public void flush() throws IOException {
+ if (closed) {
+ throw new IOException("Stream closed");
+ }
+
+ if (buffer.position() - buffer.arrayOffset() > 0) {
+ uploadBlock();
+ }
+
+ if (blockUpload != null) {
+ blockUpload.complete();
+ blockUpload = null;
+ committed = true;
+ }
+ }
+
+ @Override
+ public void close() throws IOException {
+ if (closed) {
+ return;
+ }
+
+ if (blockUpload != null && blockUpload.aborted) {
+ blockUpload = null;
+ closed = true;
+ return;
+ }
+
+ if (!committed) {
+ uploadBlock();
+ if (blockUpload != null) {
+ blockUpload.complete();
+ blockUpload = null;
+ committed = true;
+ } else {
+ try {
+ blobClient.upload(new ByteArrayInputStream(new byte[0]), 0, true);
+ } catch (BlobStorageException e) {
+ throw new IOException(
+ "Failed to create empty blob", AzureBlobStorageClient.handleBlobException(e));
+ }
+ }
+ } else {
+ if (blockUpload != null) {
+ blockUpload.complete();
+ blockUpload = null;
+ }
+ }
+
+ closed = true;
+ }
+
+ private BlockUpload newBlockUpload() throws IOException {
+ try {
+ return new BlockUpload();
+ } catch (BlobStorageException e) {
+ throw new IOException(
+ "Failed to create block upload", AzureBlobStorageClient.handleBlobException(e));
+ }
+ }
+
+ private class BlockUpload {
+ private final List<String> blockIds;
+ private boolean aborted = false;
+
+ public BlockUpload() {
+ this.blockIds = new ArrayList<>();
+ if (log.isDebugEnabled()) {
+ log.debug("Initiated block upload for blobPath '{}'", blobPath);
+ }
+
+ try {
+ BlockBlobClient blockBlobClient = blobClient.getBlockBlobClient();
+ blockBlobClient.deleteIfExists();
+ } catch (BlobStorageException e) {
+ // ignore; subsequent stage/commit will surface real issues
+ }
+ }
+
+ void uploadBlock(ByteArrayInputStream inputStream, long blockSize) {
+ if (aborted) {
+ throw new IllegalStateException(
+ "Can't upload new blocks on a BlockUpload that was aborted");
+ }
+
+ String blockId =
+ Base64.getEncoder()
+ .encodeToString(UUID.randomUUID().toString().getBytes(StandardCharsets.UTF_8));
+
+ if (log.isDebugEnabled()) {
+ log.debug("Uploading block {} for blobPath '{}'", blockId, blobPath);
+ }
+
+ try {
+ BlockBlobClient blockBlobClient = blobClient.getBlockBlobClient();
+ blockBlobClient.stageBlock(blockId, inputStream, blockSize);
+ blockIds.add(blockId);
+ } catch (BlobStorageException e) {
+ throw new RuntimeException("Failed to upload block", e);
+ }
+ }
+
+ void complete() {
+ if (aborted) {
+ throw new IllegalStateException("Can't complete a BlockUpload that was aborted");
+ }
+
+ if (log.isDebugEnabled()) {
+ log.debug("Completing block upload for blobPath '{}'", blobPath);
+ }
+
+ try {
+ BlockBlobClient blockBlobClient = blobClient.getBlockBlobClient();
+ blockBlobClient.commitBlockList(blockIds);
+ } catch (BlobStorageException e) {
+ throw new RuntimeException("Failed to commit block list", e);
+ }
+ }
+
+ public void abort() {
+ if (log.isWarnEnabled()) {
+ log.warn("Aborting block upload for blobPath '{}'", blobPath);
+ }
+
+ aborted = true;
Review Comment:
renamed it to failed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17949: Add Azure Blob Storage backup repository module [solr]
Posted by "prateeksinghalgit (via GitHub)" <gi...@apache.org>.
prateeksinghalgit commented on PR #3750:
URL: https://github.com/apache/solr/pull/3750#issuecomment-4775666316
@janhoy could you please trigger a CI re-run, seems like a unrelated flaky test in the CI run. I ran CI locally, it did run successfully.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17949: Add Azure Blob Storage backup repository module [solr]
Posted by "prateeksinghalgit (via GitHub)" <gi...@apache.org>.
prateeksinghalgit commented on code in PR #3750:
URL: https://github.com/apache/solr/pull/3750#discussion_r2555407396
##########
gradle/libs.versions.toml:
##########
@@ -51,6 +51,10 @@ aqute-bnd = "6.4.1"
asciidoctor-mathjax = "0.0.9"
# @keep Asciidoctor tabs version used in ref-guide
asciidoctor-tabs = "1.0.0-beta.6"
+azure-storage = "12.25.0"
+azure-identity = "1.12.0"
+azure-core = "1.52.0"
+azure-core-http-netty = "1.15.4"
Review Comment:
Yes, made the change to use azure-core-http-okhttp. I was having issues with using azure-core-http-jdk-httpclient.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17949: Add Azure Blob Storage backup repository module [solr]
Posted by "pratsgit (via GitHub)" <gi...@apache.org>.
pratsgit commented on PR #3750:
URL: https://github.com/apache/solr/pull/3750#issuecomment-3555962712
Quick update: following up on the dev@ thread.
To clarify scope — although the diff is large, the actual implementation is centered in 8 main files and 8 test files; the rest are license header updates.
Happy to split the PR into smaller logical chunks (core module / tests / docs) if that helps with review.
Thanks everyone!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17949: Add Azure Blob Storage backup repository module [solr]
Posted by "prateeksinghalgit (via GitHub)" <gi...@apache.org>.
prateeksinghalgit commented on code in PR #3750:
URL: https://github.com/apache/solr/pull/3750#discussion_r2553345594
##########
solr/licenses/azure-NOTICE.txt:
##########
@@ -0,0 +1,25 @@
+AWS SDK for Java 2.0
+Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved.
+
+This product includes software developed by
+Amazon Technologies, Inc (http://www.amazon.com/).
+
+**********************
+THIRD PARTY COMPONENTS
+**********************
+This software includes third party software subject to the following copyrights:
+- XML parsing and utility functions from JetS3t - Copyright 2006-2009 James Murty.
+- PKCS#1 PEM encoded private key parsing and utility functions from oauth.googlecode.com - Copyright 1998-2010 AOL Inc.
Review Comment:
updated it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17949: Add Azure Blob Storage backup repository module [solr]
Posted by "prateeksinghalgit (via GitHub)" <gi...@apache.org>.
prateeksinghalgit commented on PR #3750:
URL: https://github.com/apache/solr/pull/3750#issuecomment-4580360021
@janhoy could you please approve workflow runs on this fork PR again. I did run the ci jobs locally with all the flags used in CI to make sure .
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17949: Add Azure Blob Storage backup repository module [solr]
Posted by "prateeksinghalgit (via GitHub)" <gi...@apache.org>.
prateeksinghalgit commented on code in PR #3750:
URL: https://github.com/apache/solr/pull/3750#discussion_r3228749058
##########
solr/modules/azure-blob-repository/src/java/org/apache/solr/azureblob/AzureBlobStorageClient.java:
##########
@@ -0,0 +1,529 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.azureblob;
+
+import com.azure.core.credential.TokenCredential;
+import com.azure.identity.DefaultAzureCredentialBuilder;
+import com.azure.storage.blob.BlobClient;
+import com.azure.storage.blob.BlobContainerClient;
+import com.azure.storage.blob.BlobServiceClient;
+import com.azure.storage.blob.BlobServiceClientBuilder;
+import com.azure.storage.blob.models.BlobItem;
+import com.azure.storage.blob.models.BlobStorageException;
+import com.azure.storage.blob.models.ListBlobsOptions;
+import com.google.common.annotations.VisibleForTesting;
+import java.io.ByteArrayInputStream;
+import java.io.FilterInputStream;
+import java.io.InputStream;
+import java.io.OutputStream;
+import java.lang.invoke.MethodHandles;
+import java.util.Collection;
+import java.util.HashSet;
+import java.util.Locale;
+import java.util.Set;
+import java.util.stream.Collectors;
+import org.apache.solr.common.util.ResumableInputStream;
+import org.apache.solr.common.util.StrUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Creates a {@link BlobServiceClient} for communicating with Azure Blob Storage. Utilizes the
+ * default Azure credential provider chain.
+ */
+public class AzureBlobStorageClient {
+
+ private static final Logger log = LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+
+ static final String BLOB_FILE_PATH_DELIMITER = "/";
+ private static final int HTTP_NOT_FOUND = 404;
+ private static final int HTTP_CONFLICT = 409;
+ private static final int SKIP_BUFFER_SIZE = 8192;
+ private static final int DELETE_BATCH_SIZE = 1000;
+
+ private static final com.azure.core.http.HttpClient SHARED_HTTP_CLIENT =
+ new com.azure.core.http.okhttp.OkHttpAsyncHttpClientBuilder().build();
+
+ private final BlobContainerClient containerClient;
+
+ AzureBlobStorageClient(
+ String containerName,
+ String connectionString,
+ String endpoint,
+ String accountName,
+ String accountKey,
+ String sasToken,
+ String tenantId,
+ String clientId,
+ String clientSecret) {
+ this(
+ createInternalClient(
+ connectionString,
+ endpoint,
+ accountName,
+ accountKey,
+ sasToken,
+ tenantId,
+ clientId,
+ clientSecret),
+ containerName);
+ }
+
+ @VisibleForTesting
+ AzureBlobStorageClient(BlobServiceClient blobServiceClient, String containerName) {
+ this.containerClient = blobServiceClient.getBlobContainerClient(containerName);
+ try {
+ containerClient.create();
+ } catch (BlobStorageException e) {
+ if (e.getStatusCode() != HTTP_CONFLICT) {
+ throw e;
+ }
+ }
+ }
+
+ private static BlobServiceClient createInternalClient(
+ String connectionString,
+ String endpoint,
+ String accountName,
+ String accountKey,
+ String sasToken,
+ String tenantId,
+ String clientId,
Review Comment:
using them now to build, ClientSecretCredentialBuilder. Helps with users using service principal authentication method to do solr backup and restore to azure blob storage.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17949: Add Azure Blob Storage backup repository module [solr]
Posted by "janhoy (via GitHub)" <gi...@apache.org>.
janhoy commented on code in PR #3750:
URL: https://github.com/apache/solr/pull/3750#discussion_r3458518356
##########
solr/solr-ref-guide/modules/deployment-guide/pages/backup-restore.adoc:
##########
@@ -826,3 +826,124 @@ https://docs.aws.amazon.com/sdkref/latest/guide/settings-global.html[These optio
* Retries
** RetryMode (`LEGACY`, `STANDARD`, `ADAPTIVE`)
** Max Attempts
+
+=== AzureBlobBackupRepository
+
+Stores and retrieves backup files in a Microsoft Azure Blob Storage container.
+
+This is provided via the `azure-blob-repository` xref:configuration-guide:solr-modules.adoc[Solr Module] that needs to be enabled before use.
+
+This plugin supports multiple authentication methods: connection strings, account keys, SAS tokens, and Azure Identity (Managed Identity, Service Principal, Azure CLI).
+For Azure Identity, ensure the identity has the "Storage Blob Data Contributor" role on the storage account.
+
+[source,xml]
+----
+<backup>
+ <repository name="azure_blob" class="org.apache.solr.azureblob.AzureBlobBackupRepository" default="false">
+ <str name="azure.blob.container.name">solr-backup</str>
+ <str name="azure.blob.connection.string">DefaultEndpointsProtocol=https;AccountName=myaccount;AccountKey=mykey;EndpointSuffix=core.windows.net</str>
+ </repository>
+</backup>
+----
+
+AzureBlobBackupRepository accepts the following options for configuration:
+
+`azure.blob.container.name`::
++
+[%autowidth,frame=none]
+|===
+|Required |Default: none
+|===
++
+The name of the Azure Blob Storage container. The container must exist before performing backup operations.
+
+`azure.blob.connection.string`::
++
+[%autowidth,frame=none]
+|===
+|Optional |Default: none
+|===
++
+Complete Azure Storage connection string. Mutually exclusive with other authentication methods.
+
+`azure.blob.account.name`::
++
+[%autowidth,frame=none]
+|===
+|Optional |Default: none
+|===
++
+Azure Storage account name. Used with account key or SAS token authentication.
+
+`azure.blob.account.key`::
++
+[%autowidth,frame=none]
+|===
+|Optional |Default: none
+|===
++
+Azure Storage account access key. Mutually exclusive with SAS token and Azure Identity.
+
+`azure.blob.sas.token`::
++
+[%autowidth,frame=none]
+|===
+|Optional |Default: none
+|===
++
+SAS token for time-limited access. Must include `srt=sco` and `sp=rwdlac` permissions.
+The `&` characters must be XML-escaped as `&`.
+
+`azure.blob.endpoint`::
++
+[%autowidth,frame=none]
+|===
+|Optional |Default: none
+|===
++
+Azure Blob Storage endpoint URL (e.g., `https://myaccount.blob.core.windows.net`).
+Required for Azure Identity authentication.
+
+`azure.blob.tenant.id`::
++
+[%autowidth,frame=none]
+|===
+|Optional |Default: none
+|===
++
+Azure AD tenant ID for Service Principal authentication.
+
+`azure.blob.client.id`::
++
+[%autowidth,frame=none]
+|===
+|Optional |Default: none
+|===
++
+Azure AD application (client) ID for Service Principal authentication.
+
+`azure.blob.client.secret`::
++
+[%autowidth,frame=none]
+|===
+|Optional |Default: none
+|===
++
+Azure AD application secret for Service Principal authentication.
+
+`location`::
++
+[%autowidth,frame=none]
+|===
+|Optional |Default: none
+|===
++
+Default path prefix within the container for backup storage.
+
+The target container must already exist; it is not created automatically.
+
+==== Known Limitation: Azure Identity and the Security Manager
+
+Azure Identity authentication (Managed Identity, Service Principal, and `DefaultAzureCredential`) does not work when Solr is started with the Java Security Manager enabled, which is the default (`SOLR_SECURITY_MANAGER_ENABLED=true`).
+To use Azure Identity, set `SOLR_SECURITY_MANAGER_ENABLED=false` before starting Solr.
+The Connection String, Account Key, and SAS Token authentication methods are unaffected and work with the Security Manager enabled.
Review Comment:
Is it an option here to give an example of what to add to `security.policy` as an alternative to disabling?
##########
solr/solrj/gradle.lockfile:
##########
@@ -17,13 +17,19 @@ com.github.stephenc.jcip:jcip-annotations:1.0-1=compileClasspath,compileOnlyHelp
com.google.auto.service:auto-service-annotations:1.0.1=annotationProcessor,errorprone,testAnnotationProcessor
com.google.auto.value:auto-value-annotations:1.11.0=annotationProcessor,errorprone,testAnnotationProcessor
com.google.auto:auto-common:1.2.2=annotationProcessor,errorprone,testAnnotationProcessor
+com.google.code.findbugs:jsr305:3.0.2=spotless865458226
com.google.errorprone:error_prone_annotation:2.41.0=annotationProcessor,errorprone,testAnnotationProcessor
+com.google.errorprone:error_prone_annotations:2.18.0=spotless865458226
Review Comment:
Why do we suddenly have all these spotless dependencies added to lockfiles? I would not expect any chagnes to lockfiles outside of the new module itself?
##########
solr/core/gradle.lockfile:
##########
@@ -1,191 +1,191 @@
# This is a Gradle generated file for dependency locking.
# Manual edits can break the build and are not advised.
# This file is expected to be part of source control.
-biz.aQute.bnd:biz.aQute.bnd.annotation:7.1.0=compileClasspath,testCompileClasspath
-com.carrotsearch.randomizedtesting:randomizedtesting-runner:2.8.4=jarValidation,testCompileClasspath,testRuntimeClasspath
-com.carrotsearch:hppc:0.10.0=compileClasspath,jarValidation,runtimeClasspath,runtimeLibs,testCompileClasspath,testRuntimeClasspath
-com.fasterxml.jackson.core:jackson-annotations:2.21=apiHelper,compileClasspath,jarValidation,runtimeClasspath,runtimeLibs,testCompileClasspath,testRuntimeClasspath
-com.fasterxml.jackson.core:jackson-core:2.21.2=apiHelper,compileClasspath,jarValidation,runtimeClasspath,runtimeLibs,testCompileClasspath,testRuntimeClasspath
-com.fasterxml.jackson.core:jackson-databind:2.21.2=apiHelper,compileClasspath,jarValidation,runtimeClasspath,runtimeLibs,testCompileClasspath,testRuntimeClasspath
-com.fasterxml.jackson.dataformat:jackson-dataformat-cbor:2.21.2=compileClasspath,jarValidation,runtimeClasspath,runtimeLibs,testCompileClasspath,testRuntimeClasspath
-com.fasterxml.jackson.dataformat:jackson-dataformat-smile:2.21.2=compileClasspath,jarValidation,runtimeClasspath,runtimeLibs,testCompileClasspath,testRuntimeClasspath
-com.fasterxml.jackson.module:jackson-module-jakarta-xmlbind-annotations:2.21.2=compileClasspath,jarValidation,runtimeClasspath,runtimeLibs,testCompileClasspath,testRuntimeClasspath
-com.fasterxml.jackson:jackson-bom:2.21.2=apiHelper,compileClasspath,jarValidation,runtimeClasspath,runtimeLibs,testCompileClasspath,testRuntimeClasspath
-com.fasterxml.woodstox:woodstox-core:7.0.0=compileClasspath,jarValidation,runtimeClasspath,runtimeLibs,testCompileClasspath,testRuntimeClasspath
-com.github.ben-manes.caffeine:caffeine:3.2.3=annotationProcessor,compileClasspath,errorprone,jarValidation,runtimeClasspath,runtimeLibs,testAnnotationProcessor,testCompileClasspath,testRuntimeClasspath
+biz.aQute.bnd:biz.aQute.bnd.annotation:7.1.0=compileClasspath,compileClasspathCopy,testCompileClasspath,testCompileClasspathCopy
+com.carrotsearch.randomizedtesting:randomizedtesting-runner:2.8.4=jarValidation,testCompileClasspath,testCompileClasspathCopy,testRuntimeClasspath,testRuntimeClasspathCopy
+com.carrotsearch:hppc:0.10.0=compileClasspath,compileClasspathCopy,jarValidation,runtimeClasspath,runtimeClasspathCopy,runtimeLibs,testCompileClasspath,testCompileClasspathCopy,testRuntimeClasspath,testRuntimeClasspathCopy
Review Comment:
Why are `runtimeClasspathCopy`, `compileClasspathCopy`, `testCompileClasspathCopy` added to many of the gradle lock files?
##########
solr/modules/azure-blob-repository/gradle.lockfile:
##########
@@ -0,0 +1,195 @@
+# This is a Gradle generated file for dependency locking.
+# Manual edits can break the build and are not advised.
+# This file is expected to be part of source control.
+com.azure:azure-core-http-okhttp:1.13.3=jarValidation,runtimeClasspath,runtimeLibs,testCompileClasspath,testRuntimeClasspath
+com.azure:azure-core:1.57.1=compileClasspath,jarValidation,runtimeClasspath,runtimeLibs,testCompileClasspath,testRuntimeClasspath
+com.azure:azure-identity:1.18.2=compileClasspath,jarValidation,runtimeClasspath,runtimeLibs,testCompileClasspath,testRuntimeClasspath
+com.azure:azure-json:1.5.1=compileClasspath,jarValidation,runtimeClasspath,runtimeLibs,testCompileClasspath,testRuntimeClasspath
+com.azure:azure-sdk-bom:1.3.6=compileClasspath,jarValidation,runtimeClasspath,runtimeLibs,testCompileClasspath,testRuntimeClasspath
+com.azure:azure-storage-blob-batch:12.29.3=compileClasspath,jarValidation,runtimeClasspath,runtimeLibs,testCompileClasspath,testRuntimeClasspath
+com.azure:azure-storage-blob:12.33.3=compileClasspath,jarValidation,runtimeClasspath,runtimeLibs,testCompileClasspath,testRuntimeClasspath
+com.azure:azure-storage-common:12.32.2=compileClasspath,jarValidation,runtimeClasspath,runtimeLibs,testCompileClasspath,testRuntimeClasspath
+com.azure:azure-storage-internal-avro:12.18.2=compileClasspath,jarValidation,runtimeClasspath,runtimeLibs,testCompileClasspath,testRuntimeClasspath
+com.azure:azure-xml:1.2.1=compileClasspath,jarValidation,runtimeClasspath,runtimeLibs,testCompileClasspath,testRuntimeClasspath
+com.carrotsearch.randomizedtesting:randomizedtesting-runner:2.8.4=jarValidation,testCompileClasspath,testRuntimeClasspath
+com.carrotsearch:hppc:0.10.0=apiHelper,jarValidation,runtimeClasspath,runtimeLibs,solrPlatformLibs,testRuntimeClasspath
+com.fasterxml.jackson.core:jackson-annotations:2.21=apiHelper,compileClasspath,jarValidation,runtimeClasspath,runtimeLibs,solrPlatformLibs,testCompileClasspath,testRuntimeClasspath
+com.fasterxml.jackson.core:jackson-core:2.21.2=apiHelper,compileClasspath,jarValidation,runtimeClasspath,runtimeLibs,solrPlatformLibs,testCompileClasspath,testRuntimeClasspath
+com.fasterxml.jackson.core:jackson-databind:2.21.2=apiHelper,compileClasspath,jarValidation,runtimeClasspath,runtimeLibs,solrPlatformLibs,testCompileClasspath,testRuntimeClasspath
+com.fasterxml.jackson.dataformat:jackson-dataformat-cbor:2.21.2=apiHelper,jarValidation,runtimeClasspath,runtimeLibs,solrPlatformLibs,testRuntimeClasspath
+com.fasterxml.jackson.dataformat:jackson-dataformat-smile:2.21.2=apiHelper,jarValidation,runtimeClasspath,runtimeLibs,solrPlatformLibs,testRuntimeClasspath
+com.fasterxml.jackson.datatype:jackson-datatype-jsr310:2.21.2=compileClasspath,jarValidation,runtimeClasspath,runtimeLibs,testCompileClasspath,testRuntimeClasspath
+com.fasterxml.jackson.module:jackson-module-jakarta-xmlbind-annotations:2.21.2=apiHelper,jarValidation,runtimeClasspath,runtimeLibs,solrPlatformLibs,testRuntimeClasspath
+com.fasterxml.jackson:jackson-bom:2.21.2=apiHelper,compileClasspath,jarValidation,runtimeClasspath,runtimeLibs,solrPlatformLibs,testCompileClasspath,testRuntimeClasspath
+com.fasterxml.woodstox:woodstox-core:7.0.0=apiHelper,jarValidation,runtimeClasspath,runtimeLibs,solrPlatformLibs,testRuntimeClasspath
+com.github.ben-manes.caffeine:caffeine:3.2.3=annotationProcessor,apiHelper,errorprone,jarValidation,runtimeClasspath,runtimeLibs,solrPlatformLibs,testAnnotationProcessor,testRuntimeClasspath
+com.github.docker-java:docker-java-api:3.7.0=jarValidation,testCompileClasspath,testRuntimeClasspath
+com.github.docker-java:docker-java-transport-zerodep:3.7.0=jarValidation,testCompileClasspath,testRuntimeClasspath
+com.github.docker-java:docker-java-transport:3.7.0=jarValidation,testCompileClasspath,testRuntimeClasspath
+com.github.kevinstern:software-and-algorithms:1.0=annotationProcessor,errorprone,testAnnotationProcessor
+com.google.auto.service:auto-service-annotations:1.0.1=annotationProcessor,errorprone,testAnnotationProcessor
+com.google.auto.value:auto-value-annotations:1.11.0=annotationProcessor,errorprone,testAnnotationProcessor
+com.google.auto:auto-common:1.2.2=annotationProcessor,errorprone,testAnnotationProcessor
+com.google.errorprone:error_prone_annotation:2.41.0=annotationProcessor,errorprone,testAnnotationProcessor
+com.google.errorprone:error_prone_annotations:2.41.0=apiHelper,compileClasspath,jarValidation,runtimeClasspath,runtimeLibs,solrPlatformLibs,testCompileClasspath,testRuntimeClasspath
+com.google.errorprone:error_prone_annotations:2.43.0=annotationProcessor,errorprone,testAnnotationProcessor
+com.google.errorprone:error_prone_check_api:2.41.0=annotationProcessor,errorprone,testAnnotationProcessor
+com.google.errorprone:error_prone_core:2.41.0=annotationProcessor,errorprone,testAnnotationProcessor
+com.google.googlejavaformat:google-java-format:1.27.0=annotationProcessor,errorprone,testAnnotationProcessor
+com.google.guava:failureaccess:1.0.3=annotationProcessor,apiHelper,compileClasspath,errorprone,jarValidation,runtimeClasspath,runtimeLibs,solrPlatformLibs,testAnnotationProcessor,testCompileClasspath,testRuntimeClasspath
+com.google.guava:guava:33.5.0-jre=annotationProcessor,apiHelper,compileClasspath,errorprone,jarValidation,runtimeClasspath,runtimeLibs,solrPlatformLibs,testAnnotationProcessor,testCompileClasspath,testRuntimeClasspath
+com.google.guava:listenablefuture:9999.0-empty-to-avoid-conflict-with-guava=annotationProcessor,apiHelper,compileClasspath,errorprone,jarValidation,runtimeClasspath,runtimeLibs,solrPlatformLibs,testAnnotationProcessor,testCompileClasspath,testRuntimeClasspath
+com.google.j2objc:j2objc-annotations:3.1=annotationProcessor,apiHelper,compileClasspath,errorprone,jarValidation,runtimeClasspath,runtimeLibs,solrPlatformLibs,testAnnotationProcessor,testCompileClasspath,testRuntimeClasspath
+com.google.protobuf:protobuf-java:3.25.8=annotationProcessor,errorprone,testAnnotationProcessor
+com.j256.simplemagic:simplemagic:1.17=apiHelper,jarValidation,runtimeClasspath,runtimeLibs,solrPlatformLibs,testRuntimeClasspath
+com.jayway.jsonpath:json-path:2.9.0=apiHelper,jarValidation,runtimeClasspath,runtimeLibs,solrPlatformLibs,testRuntimeClasspath
+com.lmax:disruptor:4.0.0=solrPlatformLibs
+com.microsoft.azure:msal4j-persistence-extension:1.3.0=compileClasspath,jarValidation,runtimeClasspath,runtimeLibs,testCompileClasspath,testRuntimeClasspath
+com.microsoft.azure:msal4j:1.23.1=compileClasspath,jarValidation,runtimeClasspath,runtimeLibs,testCompileClasspath,testRuntimeClasspath
+com.squareup.okhttp3:okhttp:4.12.0=jarValidation,runtimeClasspath,runtimeLibs,testCompileClasspath,testRuntimeClasspath
+com.squareup.okio:okio-jvm:3.16.0=jarValidation,runtimeClasspath,runtimeLibs,testCompileClasspath,testRuntimeClasspath
+com.squareup.okio:okio:3.16.0=jarValidation,testCompileClasspath,testRuntimeClasspath
+com.tdunning:t-digest:3.3=apiHelper,jarValidation,runtimeClasspath,runtimeLibs,solrPlatformLibs,testRuntimeClasspath
+commons-cli:commons-cli:1.11.0=apiHelper,jarValidation,runtimeClasspath,runtimeLibs,solrPlatformLibs,testRuntimeClasspath
+commons-codec:commons-codec:1.21.0=apiHelper,jarValidation,runtimeClasspath,runtimeLibs,solrPlatformLibs,testCompileClasspath,testRuntimeClasspath
+commons-io:commons-io:2.21.0=apiHelper,compileClasspath,jarValidation,runtimeClasspath,runtimeLibs,solrPlatformLibs,testCompileClasspath,testRuntimeClasspath
+io.dropwizard.metrics:metrics-core:4.2.33=apiHelper,jarValidation,runtimeClasspath,runtimeLibs,solrPlatformLibs,testRuntimeClasspath
+io.github.eisop:dataflow-errorprone:3.41.0-eisop1=annotationProcessor,errorprone,testAnnotationProcessor
+io.github.java-diff-utils:java-diff-utils:4.12=annotationProcessor,errorprone,testAnnotationProcessor
+io.netty:netty-buffer:4.2.15.Final=apiHelper,compileClasspath,jarValidation,runtimeClasspath,runtimeLibs,solrPlatformLibs,testCompileClasspath,testRuntimeClasspath
+io.netty:netty-codec-base:4.2.15.Final=apiHelper,compileClasspath,jarValidation,runtimeClasspath,runtimeLibs,solrPlatformLibs,testCompileClasspath,testRuntimeClasspath
+io.netty:netty-common:4.2.15.Final=apiHelper,compileClasspath,jarValidation,runtimeClasspath,runtimeLibs,solrPlatformLibs,testCompileClasspath,testRuntimeClasspath
+io.netty:netty-handler:4.2.15.Final=apiHelper,compileClasspath,jarValidation,runtimeClasspath,runtimeLibs,solrPlatformLibs,testCompileClasspath,testRuntimeClasspath
+io.netty:netty-resolver:4.2.15.Final=apiHelper,compileClasspath,jarValidation,runtimeClasspath,runtimeLibs,solrPlatformLibs,testCompileClasspath,testRuntimeClasspath
+io.netty:netty-tcnative-boringssl-static:2.0.75.Final=apiHelper,compileClasspath,jarValidation,runtimeClasspath,runtimeLibs,solrPlatformLibs,testCompileClasspath,testRuntimeClasspath
+io.netty:netty-tcnative-classes:2.0.75.Final=apiHelper,compileClasspath,jarValidation,runtimeClasspath,runtimeLibs,solrPlatformLibs,testCompileClasspath,testRuntimeClasspath
+io.netty:netty-transport-classes-epoll:4.2.15.Final=apiHelper,compileClasspath,jarValidation,runtimeClasspath,runtimeLibs,solrPlatformLibs,testCompileClasspath,testRuntimeClasspath
+io.netty:netty-transport-native-epoll:4.2.15.Final=apiHelper,compileClasspath,jarValidation,runtimeClasspath,runtimeLibs,solrPlatformLibs,testCompileClasspath,testRuntimeClasspath
+io.netty:netty-transport-native-unix-common:4.2.15.Final=apiHelper,compileClasspath,jarValidation,runtimeClasspath,runtimeLibs,solrPlatformLibs,testCompileClasspath,testRuntimeClasspath
+io.netty:netty-transport:4.2.15.Final=apiHelper,compileClasspath,jarValidation,runtimeClasspath,runtimeLibs,solrPlatformLibs,testCompileClasspath,testRuntimeClasspath
+io.opentelemetry.instrumentation:opentelemetry-instrumentation-api-incubator:2.22.0-alpha=apiHelper,jarValidation,runtimeClasspath,runtimeLibs,solrPlatformLibs,testRuntimeClasspath
+io.opentelemetry.instrumentation:opentelemetry-instrumentation-api:2.22.0=apiHelper,jarValidation,runtimeClasspath,runtimeLibs,solrPlatformLibs,testRuntimeClasspath
+io.opentelemetry.instrumentation:opentelemetry-runtime-telemetry-java17:2.22.0-alpha=apiHelper,jarValidation,runtimeClasspath,runtimeLibs,solrPlatformLibs,testRuntimeClasspath
+io.opentelemetry.instrumentation:opentelemetry-runtime-telemetry-java8:2.22.0-alpha=apiHelper,jarValidation,runtimeClasspath,runtimeLibs,solrPlatformLibs,testRuntimeClasspath
+io.opentelemetry.semconv:opentelemetry-semconv:1.37.0=apiHelper,jarValidation,runtimeClasspath,runtimeLibs,solrPlatformLibs,testRuntimeClasspath
+io.opentelemetry:opentelemetry-api-incubator:1.56.0-alpha=apiHelper,jarValidation,runtimeClasspath,runtimeLibs,solrPlatformLibs,testRuntimeClasspath
+io.opentelemetry:opentelemetry-api:1.56.0=apiHelper,compileClasspath,jarValidation,runtimeClasspath,runtimeLibs,solrPlatformLibs,testCompileClasspath,testRuntimeClasspath
+io.opentelemetry:opentelemetry-common:1.56.0=apiHelper,compileClasspath,jarValidation,runtimeClasspath,runtimeLibs,solrPlatformLibs,testCompileClasspath,testRuntimeClasspath
+io.opentelemetry:opentelemetry-context:1.56.0=apiHelper,compileClasspath,jarValidation,runtimeClasspath,runtimeLibs,solrPlatformLibs,testCompileClasspath,testRuntimeClasspath
+io.opentelemetry:opentelemetry-exporter-prometheus:1.56.0-alpha=apiHelper,jarValidation,runtimeClasspath,runtimeLibs,solrPlatformLibs,testRuntimeClasspath
+io.opentelemetry:opentelemetry-sdk-common:1.56.0=apiHelper,jarValidation,runtimeClasspath,runtimeLibs,solrPlatformLibs,testRuntimeClasspath
+io.opentelemetry:opentelemetry-sdk-metrics:1.56.0=apiHelper,jarValidation,runtimeClasspath,runtimeLibs,solrPlatformLibs,testRuntimeClasspath
+io.opentelemetry:opentelemetry-sdk-trace:1.56.0=apiHelper,jarValidation,runtimeClasspath,runtimeLibs,solrPlatformLibs,testRuntimeClasspath
+io.opentelemetry:opentelemetry-sdk:1.56.0=apiHelper,jarValidation,runtimeClasspath,runtimeLibs,solrPlatformLibs,testRuntimeClasspath
+io.projectreactor:reactor-core:3.7.14=compileClasspath,jarValidation,runtimeClasspath,runtimeLibs,testCompileClasspath,testRuntimeClasspath
+io.prometheus:prometheus-metrics-exposition-formats:1.1.0=apiHelper,jarValidation,runtimeClasspath,runtimeLibs,solrPlatformLibs,testRuntimeClasspath
+io.prometheus:prometheus-metrics-model:1.1.0=apiHelper,jarValidation,runtimeClasspath,runtimeLibs,solrPlatformLibs,testRuntimeClasspath
+io.sgr:s2-geometry-library-java:1.0.0=apiHelper,jarValidation,runtimeClasspath,runtimeLibs,solrPlatformLibs,testRuntimeClasspath
+io.swagger.core.v3:swagger-annotations-jakarta:2.2.22=apiHelper,compileClasspath,jarValidation,runtimeClasspath,runtimeLibs,solrPlatformLibs,testCompileClasspath,testRuntimeClasspath
+jakarta.activation:jakarta.activation-api:2.1.3=apiHelper,jarValidation,runtimeClasspath,runtimeLibs,solrPlatformLibs,testRuntimeClasspath
+jakarta.annotation:jakarta.annotation-api:3.0.0=apiHelper,jarValidation,runtimeClasspath,runtimeLibs,solrPlatformLibs,testRuntimeClasspath
+jakarta.inject:jakarta.inject-api:2.0.1=apiHelper,jarValidation,runtimeClasspath,runtimeLibs,solrPlatformLibs,testRuntimeClasspath
+jakarta.servlet:jakarta.servlet-api:6.1.0=apiHelper,jarValidation,runtimeClasspath,runtimeLibs,solrPlatformLibs,testRuntimeClasspath
+jakarta.validation:jakarta.validation-api:3.1.0=apiHelper,jarValidation,runtimeClasspath,runtimeLibs,solrPlatformLibs,testRuntimeClasspath
+jakarta.ws.rs:jakarta.ws.rs-api:4.0.0=apiHelper,jarValidation,runtimeClasspath,runtimeLibs,solrPlatformLibs,testRuntimeClasspath
+jakarta.xml.bind:jakarta.xml.bind-api:4.0.2=apiHelper,jarValidation,runtimeClasspath,runtimeLibs,solrPlatformLibs,testRuntimeClasspath
+javax.inject:javax.inject:1=annotationProcessor,errorprone,testAnnotationProcessor
+junit:junit:4.13.2=jarValidation,testCompileClasspath,testRuntimeClasspath
+net.java.dev.jna:jna-platform:5.17.0=compileClasspath,jarValidation,runtimeClasspath,runtimeLibs,testCompileClasspath,testRuntimeClasspath
+net.java.dev.jna:jna:5.18.1=compileClasspath,jarValidation,runtimeClasspath,runtimeLibs,testCompileClasspath,testRuntimeClasspath
Review Comment:
Noticing that jna-platform introduced in this branch does not align with jna? Declare it to same ver in toml file?
##########
solr/solr-ref-guide/modules/deployment-guide/pages/backup-restore.adoc:
##########
@@ -826,3 +826,124 @@ https://docs.aws.amazon.com/sdkref/latest/guide/settings-global.html[These optio
* Retries
** RetryMode (`LEGACY`, `STANDARD`, `ADAPTIVE`)
** Max Attempts
+
+=== AzureBlobBackupRepository
+
+Stores and retrieves backup files in a Microsoft Azure Blob Storage container.
+
+This is provided via the `azure-blob-repository` xref:configuration-guide:solr-modules.adoc[Solr Module] that needs to be enabled before use.
+
+This plugin supports multiple authentication methods: connection strings, account keys, SAS tokens, and Azure Identity (Managed Identity, Service Principal, Azure CLI).
+For Azure Identity, ensure the identity has the "Storage Blob Data Contributor" role on the storage account.
+
+[source,xml]
+----
+<backup>
Review Comment:
Should perhaps re-iterate that this snippet goes in `solr.xml`, even if it is mentioned several hundred lines above?
##########
solr/modules/azure-blob-repository/README.md:
##########
@@ -0,0 +1,118 @@
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+
+# Apache Solr Azure Blob Storage Backup Repository
+
+A backup repository implementation for storing Solr backups in Azure Blob Storage.
+
+## Prerequisites
+
+- Azure Storage Account with a blob container (must already exist)
+- Network access to Azure Blob Storage (HTTPS port 443)
+
+Enable the module:
+```bash
+export SOLR_MODULES=azure-blob-repository
+```
+
+## Configuration
+
+Add to `solr.xml`:
+
+```xml
+<backup>
+ <repository name="azure_blob" class="org.apache.solr.azureblob.AzureBlobBackupRepository" default="false">
+ <str name="azure.blob.container.name">YOUR_CONTAINER_NAME</str>
+ <!-- Authentication options below -->
+ </repository>
+</backup>
+```
+
+## Authentication Methods
+
+### Connection String (Development)
+
+```xml
+<str name="azure.blob.connection.string">DefaultEndpointsProtocol=https;AccountName=...;AccountKey=...;EndpointSuffix=core.windows.net</str>
+```
+
+### Account Name + Account Key
+
+```xml
+<str name="azure.blob.endpoint">https://YOUR_ACCOUNT.blob.core.windows.net</str>
+<str name="azure.blob.account.name">YOUR_ACCOUNT</str>
+<str name="azure.blob.account.key">YOUR_ACCOUNT_KEY</str>
+```
+
+### SAS Token (Production)
+
+Generate a SAS token with permissions: Read, Write, Delete, List, Add, Create (`sp=rwdlac`) and resource types: Service, Container, Object (`srt=sco`).
+
+```xml
+<str name="azure.blob.endpoint">https://YOUR_ACCOUNT.blob.core.windows.net</str>
+<str name="azure.blob.sas.token">sv=2024-11-04&ss=b&srt=sco&sp=rwdlac&...</str>
+```
+
+Note: Escape `&` as `&` in XML.
+
+### Azure Identity (Production - Recommended)
+
+Uses Azure AD authentication. Requires "Storage Blob Data Contributor" role on the storage account.
+
+```xml
+<str name="azure.blob.endpoint">https://YOUR_ACCOUNT.blob.core.windows.net</str>
+<!-- No credentials needed when using Managed Identity or Azure CLI -->
+```
+
+For Service Principal, add:
+```xml
+<str name="azure.blob.tenant.id">YOUR_TENANT_ID</str>
+<str name="azure.blob.client.id">YOUR_CLIENT_ID</str>
+<str name="azure.blob.client.secret">YOUR_CLIENT_SECRET</str>
+```
+
+Or set environment variables: `AZURE_TENANT_ID`, `AZURE_CLIENT_ID`, `AZURE_CLIENT_SECRET`.
+
+## Known Limitations
+
+Azure Identity authentication (Service Principal, Managed Identity, `DefaultAzureCredential`) does not work when Solr is started with the Java `SecurityManager` enabled. The Azure Identity SDK relies on `doPrivileged` patterns that fail under Solr's default security policy; see the [upstream issue](https://github.com/Azure/azure-sdk-for-java/issues/37464) for details. Note that the `SecurityManager` is deprecated for removal in modern JDKs, but Solr still enables it by default via `SOLR_SECURITY_MANAGER_ENABLED=true`.
+
+Workaround: set `SOLR_SECURITY_MANAGER_ENABLED=false` (in `solr.in.sh` / `solr.in.cmd`, or as an environment variable) before starting Solr. The Connection String, Account Key, and SAS Token authentication methods are unaffected and work with the `SecurityManager` enabled.
+
+## Usage
+
+```bash
+# Backup
+curl "http://localhost:8983/solr/admin/collections?action=BACKUP&name=my-backup&collection=my-collection&repository=azure_blob&location=/"
Review Comment:
There are "Usage" and "Troubleshooting" sections in this README in the source folder. Do we expect end users to see / find this file? Should some of this instead be in the ref guide? What is the thinking here?
##########
solr/modules/azure-blob-repository/build.gradle:
##########
@@ -0,0 +1,74 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+apply plugin: 'java-library'
+
+description = 'Azure Blob Storage Repository'
+
+ext {
+ // Disable security manager for azure-blob-repository module tests
+ // Required because Testcontainers needs access to Docker socket and system properties
+ useSecurityManager = false
+}
+
+dependencies {
+ implementation platform(project(':platform'))
+ api(project(':solr:core'))
+ implementation project(':solr:solrj')
+
+ implementation libs.apache.lucene.core
+
+ implementation platform(libs.azure.sdk.bom)
+
+ // Azure Storage SDK dependencies
+ implementation(libs.azure.storage.blob) {
+ exclude group: 'com.azure', module: 'azure-core-http-netty'
+ }
+ implementation(libs.azure.storage.blob.batch) {
+ exclude group: 'com.azure', module: 'azure-core-http-netty'
+ }
+ implementation(libs.azure.identity) {
+ exclude group: 'com.azure', module: 'azure-core-http-netty'
+ }
Review Comment:
I don't think we need desktop-style credentials persistence here, can shave two jar dependencies.
```suggestion
exclude group: 'com.microsoft.azure', module: 'msal4j-persistence-extension'
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17949: Add Azure Blob Storage backup repository module [solr]
Posted by "prateeksinghalgit (via GitHub)" <gi...@apache.org>.
prateeksinghalgit commented on code in PR #3750:
URL: https://github.com/apache/solr/pull/3750#discussion_r2558060657
##########
solr/solr-ref-guide/modules/deployment-guide/pages/backup-restore.adoc:
##########
@@ -383,7 +383,7 @@ If the status is anything other than "success", an error message will explain wh
Solr provides a repository abstraction to allow users to backup and restore their data to a variety of different storage systems.
For example, a Solr cluster running on a local filesystem (e.g., EXT3) can store backup data on the same disk, on a remote network-mounted drive, or in some popular "cloud storage" providers, depending on the 'repository' implementation chosen.
-Solr offers multiple different repository implementations out of the box (`LocalFileSystemRepository`, `GCSBackupRepository` and `S3BackupRepository`), and allows users to create plugins for their own storage systems as needed. It is also possible to create a `DelegatingBackupRepository` that delegates to another `BackupRepository` and adds or modifies some behavior on top of it.
+Solr offers multiple different repository implementations out of the box (`LocalFileSystemRepository`, `GCSBackupRepository`, `S3BackupRepository`, and `BlobBackupRepository`), and allows users to create plugins for their own storage systems as needed. It is also possible to create a `DelegatingBackupRepository` that delegates to another `BackupRepository` and adds or modifies some behavior on top of it.
Review Comment:
made the change
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17949: Add Azure Blob Storage backup repository module [solr]
Posted by "pratsgit (via GitHub)" <gi...@apache.org>.
pratsgit commented on PR #3750:
URL: https://github.com/apache/solr/pull/3750#issuecomment-3594909885
Hi all,
I’ve pushed a set of updates based on the feedback so far:
1. Switched the integration tests to use Testcontainers with Azurite (no hard-coded ports / external prereqs) and disabled the Security Manager for this module’s tests, following the pattern used in the extraction module.
2. Addressed HTTP client concerns by using azure-core-http-okhttp and keeping those deps in test scope only, plus explicit test deps.
3. Cleaned up and significantly shortened the README and docs
4. Did additional cleanup to align naming, structure, and configuration patterns with those of the existing repository plugins.
Please let me know if anything else should be adjusted that would help move the review forward.
Happy to make further changes.
Thanks again for the thoughtful review and guidance so far!
— Prateek
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17949: Add Azure Blob Storage backup repository module [solr]
Posted by "janhoy (via GitHub)" <gi...@apache.org>.
janhoy commented on PR #3750:
URL: https://github.com/apache/solr/pull/3750#issuecomment-4794359744
Thanks for the latest iteration, to me this starts to look quite mature. But as said earlier I'll defer to someone who have an actual AZ environment at hand to test it and do the last blessing.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17949: Add Azure Blob Storage backup repository module [solr]
Posted by "prateeksinghalgit (via GitHub)" <gi...@apache.org>.
prateeksinghalgit commented on code in PR #3750:
URL: https://github.com/apache/solr/pull/3750#discussion_r3359089044
##########
solr/modules/azure-blob-repository/src/java/org/apache/solr/azureblob/AzureBlobStorageClient.java:
##########
@@ -0,0 +1,529 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.azureblob;
+
+import com.azure.core.credential.TokenCredential;
+import com.azure.identity.DefaultAzureCredentialBuilder;
+import com.azure.storage.blob.BlobClient;
+import com.azure.storage.blob.BlobContainerClient;
+import com.azure.storage.blob.BlobServiceClient;
+import com.azure.storage.blob.BlobServiceClientBuilder;
+import com.azure.storage.blob.models.BlobItem;
+import com.azure.storage.blob.models.BlobStorageException;
+import com.azure.storage.blob.models.ListBlobsOptions;
+import com.google.common.annotations.VisibleForTesting;
+import java.io.ByteArrayInputStream;
+import java.io.FilterInputStream;
+import java.io.InputStream;
+import java.io.OutputStream;
+import java.lang.invoke.MethodHandles;
+import java.util.Collection;
+import java.util.HashSet;
+import java.util.Locale;
+import java.util.Set;
+import java.util.stream.Collectors;
+import org.apache.solr.common.util.ResumableInputStream;
+import org.apache.solr.common.util.StrUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Creates a {@link BlobServiceClient} for communicating with Azure Blob Storage. Utilizes the
+ * default Azure credential provider chain.
+ */
+public class AzureBlobStorageClient {
+
+ private static final Logger log = LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+
+ static final String BLOB_FILE_PATH_DELIMITER = "/";
+ private static final int HTTP_NOT_FOUND = 404;
+ private static final int HTTP_CONFLICT = 409;
+ private static final int SKIP_BUFFER_SIZE = 8192;
+ private static final int DELETE_BATCH_SIZE = 1000;
+
+ private static final com.azure.core.http.HttpClient SHARED_HTTP_CLIENT =
+ new com.azure.core.http.okhttp.OkHttpAsyncHttpClientBuilder().build();
+
+ private final BlobContainerClient containerClient;
+
+ AzureBlobStorageClient(
+ String containerName,
+ String connectionString,
+ String endpoint,
+ String accountName,
+ String accountKey,
+ String sasToken,
+ String tenantId,
+ String clientId,
+ String clientSecret) {
+ this(
+ createInternalClient(
+ connectionString,
+ endpoint,
+ accountName,
+ accountKey,
+ sasToken,
+ tenantId,
+ clientId,
+ clientSecret),
+ containerName);
+ }
+
+ @VisibleForTesting
+ AzureBlobStorageClient(BlobServiceClient blobServiceClient, String containerName) {
+ this.containerClient = blobServiceClient.getBlobContainerClient(containerName);
+ try {
+ containerClient.create();
+ } catch (BlobStorageException e) {
+ if (e.getStatusCode() != HTTP_CONFLICT) {
+ throw e;
+ }
+ }
+ }
+
+ private static BlobServiceClient createInternalClient(
+ String connectionString,
+ String endpoint,
+ String accountName,
+ String accountKey,
+ String sasToken,
+ String tenantId,
+ String clientId,
+ String clientSecret) {
+
+ BlobServiceClientBuilder builder = new BlobServiceClientBuilder();
+ builder.httpClient(SHARED_HTTP_CLIENT);
+
+ if (StrUtils.isNotNullOrEmpty(connectionString)) {
+ builder.connectionString(connectionString);
+ } else if (StrUtils.isNotNullOrEmpty(endpoint)) {
+ builder.endpoint(endpoint);
+ if (StrUtils.isNotNullOrEmpty(accountName) && StrUtils.isNotNullOrEmpty(accountKey)) {
+ builder.credential(
+ new com.azure.storage.common.StorageSharedKeyCredential(accountName, accountKey));
+ } else if (StrUtils.isNotNullOrEmpty(sasToken)) {
+ builder.sasToken(sasToken);
+ } else {
+ TokenCredential credential = new DefaultAzureCredentialBuilder().tenantId(tenantId).build();
+ builder.credential(credential);
+ }
+ } else {
+ throw new IllegalArgumentException("Either connectionString or endpoint must be provided");
+ }
+
+ return builder.buildClient();
+ }
+
+ void createDirectory(String path) throws AzureBlobException {
+ String sanitizedDirPath = sanitizedDirPath(path);
+
+ if (!pathExists(sanitizedDirPath)) {
+ String parent = getParentDirectory(sanitizedDirPath);
+ if (!parent.isEmpty() && !parent.equals(BLOB_FILE_PATH_DELIMITER)) {
+ createDirectory(parent);
+ }
+
+ try {
+ BlobClient blobClient = containerClient.getBlobClient(sanitizedDirPath);
+ blobClient.upload(new ByteArrayInputStream(new byte[0]), 0, true);
+ java.util.Map<String, String> metadata = new java.util.HashMap<>();
+ metadata.put("hdi_isfolder", "true");
+ blobClient.setMetadata(metadata);
+ } catch (BlobStorageException e) {
+ throw handleBlobException(e);
+ }
+ }
+ }
+
+ void delete(Collection<String> paths) throws AzureBlobException {
+ Set<String> entries = new HashSet<>();
+ for (String path : paths) {
+ entries.add(sanitizedFilePath(path));
+ }
+ deleteBlobs(entries);
+ }
+
+ void deleteDirectory(String path) throws AzureBlobException {
+ path = sanitizedDirPath(path);
+
+ Set<String> entries = listAll(path);
+ if (pathExists(path)) {
+ entries.add(path);
+ }
+
+ deleteBlobs(entries);
+ }
+
+ String[] listDir(String path) throws AzureBlobException {
+ path = sanitizedDirPath(path);
+
+ try {
+ ListBlobsOptions options = new ListBlobsOptions().setPrefix(path).setMaxResultsPerPage(1000);
+
+ final String finalPath = path;
+ return containerClient.listBlobs(options, null).stream()
+ .map(BlobItem::getName)
+ .filter(s -> s.startsWith(finalPath))
+ .map(s -> s.substring(finalPath.length()))
+ .filter(s -> !s.isEmpty())
+ .filter(
+ s -> {
+ int slashIndex = s.indexOf(BLOB_FILE_PATH_DELIMITER);
+ return slashIndex == -1 || slashIndex == s.length() - 1;
+ })
+ .toArray(String[]::new);
+ } catch (BlobStorageException e) {
+ throw handleBlobException(e);
+ }
+ }
+
+ boolean pathExists(String path) throws AzureBlobException {
+ final String blobPath = sanitizedPath(path);
+
+ if (blobPath.isEmpty() || BLOB_FILE_PATH_DELIMITER.equals(blobPath)) {
+ return true;
+ }
+
+ try {
+ BlobClient blobClient = containerClient.getBlobClient(blobPath);
+ return blobClient.exists();
+ } catch (BlobStorageException e) {
+ throw handleBlobException(e);
+ }
+ }
+
+ boolean isDirectory(String path) throws AzureBlobException {
+ final String dirPrefix = sanitizedDirPath(path);
+
+ try {
+ ListBlobsOptions options =
+ new ListBlobsOptions().setPrefix(dirPrefix).setMaxResultsPerPage(1);
+ if (containerClient.listBlobs(options, null).iterator().hasNext()) {
+ return true;
+ }
+
+ BlobClient markerClient = containerClient.getBlobClient(dirPrefix);
+ if (markerClient.exists()) {
+ long size = markerClient.getProperties().getBlobSize();
+ if (size == 0) {
+ return true;
+ }
+ java.util.Map<String, String> md = markerClient.getProperties().getMetadata();
+ return md != null && md.containsKey("hdi_isfolder");
+ }
+
+ return false;
+ } catch (BlobStorageException e) {
+ throw handleBlobException(e);
+ }
+ }
+
+ long length(String path) throws AzureBlobException {
+ String blobPath = sanitizedFilePath(path);
+ try {
+ BlobClient blobClient = containerClient.getBlobClient(blobPath);
+ return blobClient.getProperties().getBlobSize();
+ } catch (BlobStorageException e) {
+ throw handleBlobException(e);
+ }
+ }
+
+ InputStream pullStream(String path) throws AzureBlobException {
+ final String blobPath = sanitizedFilePath(path);
+
+ try {
+ BlobClient blobClient = containerClient.getBlobClient(blobPath);
+ final long contentLength = blobClient.getProperties().getBlobSize();
+
+ if (contentLength == 0) {
+ return new ByteArrayInputStream(new byte[0]);
+ }
+
+ InputStream initial = new IdempotentCloseInputStream(blobClient.openInputStream());
+
+ return new ResumableInputStream(
+ initial,
+ bytesRead -> {
+ if (contentLength > 0 && bytesRead >= contentLength) {
+ return null;
+ }
+ try {
+ long remaining =
+ contentLength > 0 ? Math.max(0, contentLength - bytesRead) : Long.MAX_VALUE;
+ return pullRangeStream(path, bytesRead, remaining);
+ } catch (AzureBlobException e) {
+ throw new RuntimeException(e);
+ }
+ });
+ } catch (BlobStorageException e) {
+ throw handleBlobException(e);
+ }
+ }
+
+ InputStream pullRangeStream(String path, long offset, long length) throws AzureBlobException {
+ final String blobPath = sanitizedFilePath(path);
+ try {
+ BlobClient blobClient = containerClient.getBlobClient(blobPath);
+ com.azure.storage.blob.models.BlobRange range =
+ new com.azure.storage.blob.models.BlobRange(offset, length);
+ return new IdempotentCloseInputStream(blobClient.openInputStream(range, null));
+ } catch (BlobStorageException e) {
+ throw handleBlobException(e);
+ }
+ }
+
+ private static final class IdempotentCloseInputStream extends FilterInputStream {
+ private boolean closed;
+
+ IdempotentCloseInputStream(InputStream in) {
+ super(in);
+ this.closed = false;
+ }
+
+ @Override
+ public int read() throws java.io.IOException {
+ if (closed) {
+ throw new java.io.IOException("Stream is already closed");
+ }
+ try {
+ return super.read();
+ } catch (RuntimeException re) {
+ if (isAlreadyClosed(re)) {
+ throw new java.io.IOException("Stream is already closed", re);
+ }
+ throw re;
+ }
+ }
+
+ @Override
+ public int read(byte[] b, int off, int len) throws java.io.IOException {
+ if (closed) {
+ throw new java.io.IOException("Stream is already closed");
+ }
+ try {
+ return super.read(b, off, len);
+ } catch (RuntimeException re) {
+ if (isAlreadyClosed(re)) {
+ throw new java.io.IOException("Stream is already closed", re);
+ }
+ throw re;
+ }
+ }
+
+ @Override
+ public void close() throws java.io.IOException {
+ if (closed) {
+ return;
+ }
+ try {
+ super.close();
+ } catch (java.io.IOException e) {
+ String msg = e.getMessage();
+ if (msg == null || !msg.toLowerCase(java.util.Locale.ROOT).contains("already closed")) {
+ throw e;
+ }
+ // swallow "already closed" to make close idempotent
+ } finally {
+ closed = true;
+ }
+ }
+
+ @Override
+ public long skip(long n) throws java.io.IOException {
+ if (closed) {
+ throw new java.io.IOException("Stream is already closed");
+ }
+ if (n <= 0) {
+ return 0L;
+ }
+ long remaining = n;
+ byte[] discard = new byte[SKIP_BUFFER_SIZE];
+ try {
+ while (remaining > 0) {
+ int toRead = (int) Math.min(discard.length, remaining);
+ int read = super.read(discard, 0, toRead);
+ if (read < 0) {
+ break;
+ }
+ remaining -= read;
+ }
+ return n - remaining;
+ } catch (RuntimeException re) {
+ throw new java.io.IOException(re);
+ }
+ }
+
+ private static boolean isAlreadyClosed(Throwable t) {
+ String msg = t.getMessage();
+ return msg != null && msg.toLowerCase(java.util.Locale.ROOT).contains("already closed");
+ }
+ }
+
+ OutputStream pushStream(String path) throws AzureBlobException {
+ path = sanitizedFilePath(path);
+
+ if (!parentDirectoryExist(path)) {
+ String parentDirectory = getParentDirectory(path);
+ if (!parentDirectory.isEmpty() && !parentDirectory.equals(BLOB_FILE_PATH_DELIMITER)) {
+ createDirectory(parentDirectory);
+ }
+ }
+
+ try {
+ BlobClient blobClient = containerClient.getBlobClient(path);
+ return new AzureBlobOutputStream(blobClient, path);
+ } catch (BlobStorageException e) {
+ throw handleBlobException(e);
+ }
+ }
+
+ void close() {}
+
+ @VisibleForTesting
+ void deleteContainerForTests() {
+ try {
+ containerClient.delete();
+ } catch (BlobStorageException e) {
+ if (e.getStatusCode() != HTTP_NOT_FOUND) {
+ throw e;
+ }
+ }
+ }
+
+ private Collection<String> deleteBlobs(Collection<String> paths) throws AzureBlobException {
+ try {
+ return deleteBlobs(paths, DELETE_BATCH_SIZE);
+ } catch (BlobStorageException e) {
+ throw handleBlobException(e);
+ }
+ }
+
+ @VisibleForTesting
+ Collection<String> deleteBlobs(Collection<String> entries, int batchSize)
+ throws AzureBlobException {
+ Set<String> deletedPaths = new HashSet<>();
+
+ for (String path : entries) {
+ try {
+ BlobClient blobClient = containerClient.getBlobClient(path);
+ boolean existed = blobClient.deleteIfExists();
+ if (existed) {
+ deletedPaths.add(path);
+ }
+ } catch (BlobStorageException e) {
+ if (e.getStatusCode() == HTTP_NOT_FOUND) {
Review Comment:
made the change, removed deleteIfExists.Also using batched deletes now
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17949: Add Azure Blob Storage backup repository module [solr]
Posted by "prateeksinghalgit (via GitHub)" <gi...@apache.org>.
prateeksinghalgit commented on code in PR #3750:
URL: https://github.com/apache/solr/pull/3750#discussion_r3359082665
##########
solr/modules/azure-blob-repository/src/java/org/apache/solr/azureblob/AzureBlobOutputStream.java:
##########
@@ -0,0 +1,269 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.azureblob;
+
+import com.azure.storage.blob.BlobClient;
+import com.azure.storage.blob.models.BlobStorageException;
+import com.azure.storage.blob.specialized.BlockBlobClient;
+import java.io.ByteArrayInputStream;
+import java.io.IOException;
+import java.io.OutputStream;
+import java.lang.invoke.MethodHandles;
+import java.nio.ByteBuffer;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Base64;
+import java.util.List;
+import java.util.UUID;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * OutputStream implementation for Azure Blob Storage using block blobs. Supports chunked uploads
+ * for large files.
+ */
+public class AzureBlobOutputStream extends OutputStream {
+ private static final Logger log = LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+
+ static final int BLOCK_SIZE = 4 * 1024 * 1024;
+
+ private final BlobClient blobClient;
+ private final String blobPath;
+ private volatile boolean closed;
+ private final ByteBuffer buffer;
+ private BlockUpload blockUpload;
+ private boolean committed;
+
+ public AzureBlobOutputStream(BlobClient blobClient, String blobPath) {
+ this.blobClient = blobClient;
+ this.blobPath = blobPath;
+ this.closed = false;
+ this.buffer = ByteBuffer.allocate(BLOCK_SIZE);
+ this.blockUpload = null;
+ this.committed = false;
+
+ if (log.isDebugEnabled()) {
+ log.debug("Created BlobOutputStream for blobPath '{}'", blobPath);
+ }
+ }
+
+ @Override
+ public void write(int b) throws IOException {
+ if (closed) {
+ throw new IOException("Stream closed");
+ }
+
+ buffer.put((byte) b);
+
+ if (!buffer.hasRemaining()) {
+ uploadBlock();
+ }
+ }
+
+ @Override
+ public void write(byte[] b, int off, int len) throws IOException {
+ if (closed) {
+ throw new IOException("Stream closed");
+ }
+
+ if (outOfRange(off, b.length) || len < 0 || outOfRange(off + len, b.length)) {
+ throw new IndexOutOfBoundsException();
+ } else if (len == 0) {
+ return;
+ }
+
+ int currentOffset = off;
+ int lenRemaining = len;
+ while (buffer.remaining() < lenRemaining) {
+ int firstPart = buffer.remaining();
+ buffer.put(b, currentOffset, firstPart);
+ uploadBlock();
+
+ currentOffset += firstPart;
+ lenRemaining -= firstPart;
+ }
+ if (lenRemaining > 0) {
+ buffer.put(b, currentOffset, lenRemaining);
+ }
+ }
+
+ private static boolean outOfRange(int off, int len) {
+ return off < 0 || off > len;
+ }
+
+ private void uploadBlock() throws IOException {
+ int size = buffer.position() - buffer.arrayOffset();
+
+ if (size == 0) {
+ return;
+ }
+
+ if (blockUpload == null) {
+ if (log.isDebugEnabled()) {
+ log.debug("New block upload for blobPath '{}'", blobPath);
+ }
+
+ blockUpload = newBlockUpload();
+ }
+
+ try (ByteArrayInputStream inputStream =
+ new ByteArrayInputStream(buffer.array(), buffer.arrayOffset(), size)) {
+ blockUpload.uploadBlock(inputStream, size);
+ } catch (BlobStorageException e) {
+ if (blockUpload != null) {
+ blockUpload.abort();
+ if (log.isDebugEnabled()) {
+ log.debug("Block upload aborted for blobPath '{}'.", blobPath);
+ }
+ }
+
+ throw new IOException(
+ "Failed to upload block", AzureBlobStorageClient.handleBlobException(e));
+ }
+
+ buffer.clear();
+ }
+
+ @Override
+ public void flush() throws IOException {
+ if (closed) {
+ throw new IOException("Stream closed");
+ }
+
+ if (buffer.position() - buffer.arrayOffset() > 0) {
+ uploadBlock();
+ }
+
+ if (blockUpload != null) {
+ blockUpload.complete();
+ blockUpload = null;
+ committed = true;
+ }
+ }
+
+ @Override
+ public void close() throws IOException {
+ if (closed) {
+ return;
+ }
+
+ if (blockUpload != null && blockUpload.aborted) {
+ blockUpload = null;
+ closed = true;
+ return;
+ }
+
+ if (!committed) {
+ uploadBlock();
+ if (blockUpload != null) {
+ blockUpload.complete();
+ blockUpload = null;
+ committed = true;
+ } else {
+ try {
+ blobClient.upload(new ByteArrayInputStream(new byte[0]), 0, true);
+ } catch (BlobStorageException e) {
+ throw new IOException(
+ "Failed to create empty blob", AzureBlobStorageClient.handleBlobException(e));
+ }
+ }
+ } else {
+ if (blockUpload != null) {
+ blockUpload.complete();
+ blockUpload = null;
+ }
+ }
+
+ closed = true;
Review Comment:
made the change
##########
solr/modules/azure-blob-repository/src/java/org/apache/solr/azureblob/AzureBlobOutputStream.java:
##########
@@ -0,0 +1,269 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.azureblob;
+
+import com.azure.storage.blob.BlobClient;
+import com.azure.storage.blob.models.BlobStorageException;
+import com.azure.storage.blob.specialized.BlockBlobClient;
+import java.io.ByteArrayInputStream;
+import java.io.IOException;
+import java.io.OutputStream;
+import java.lang.invoke.MethodHandles;
+import java.nio.ByteBuffer;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Base64;
+import java.util.List;
+import java.util.UUID;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * OutputStream implementation for Azure Blob Storage using block blobs. Supports chunked uploads
+ * for large files.
+ */
+public class AzureBlobOutputStream extends OutputStream {
+ private static final Logger log = LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+
+ static final int BLOCK_SIZE = 4 * 1024 * 1024;
+
+ private final BlobClient blobClient;
+ private final String blobPath;
+ private volatile boolean closed;
Review Comment:
made the change
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17949: Add Azure Blob Storage backup repository module [solr]
Posted by "prateeksinghalgit (via GitHub)" <gi...@apache.org>.
prateeksinghalgit commented on code in PR #3750:
URL: https://github.com/apache/solr/pull/3750#discussion_r3232220445
##########
solr/modules/azure-blob-repository/src/java/org/apache/solr/azureblob/AzureBlobIndexInput.java:
##########
@@ -0,0 +1,200 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.azureblob;
+
+import java.io.EOFException;
+import java.io.IOException;
+import java.io.InputStream;
+import java.util.LinkedHashMap;
+import java.util.Map;
+import org.apache.lucene.store.IndexInput;
+
+class AzureBlobIndexInput extends IndexInput {
+
+ private static final int MIN_PAGE_SIZE = 4 * 1024;
+ private static final int DEFAULT_PAGE_SIZE = 512 * 1024;
+ private static final int MAX_CACHED_PAGES = 128;
+
+ private final String path;
+ private final AzureBlobStorageClient client;
+ private final long length;
+ private final int pageSize;
+ private final LruPageCache cache;
+
+ private long position = 0L;
+ private boolean closed = false;
+
+ AzureBlobIndexInput(String path, AzureBlobStorageClient client, long length) {
+ this(path, client, length, DEFAULT_PAGE_SIZE, MAX_CACHED_PAGES);
+ }
+
+ AzureBlobIndexInput(
+ String path, AzureBlobStorageClient client, long length, int pageSize, int maxCachedPages) {
+ super(path);
+ this.path = path;
+ this.client = client;
+ this.length = length;
+ this.pageSize = Math.max(MIN_PAGE_SIZE, pageSize);
+ this.cache = new LruPageCache(maxCachedPages);
+ }
+
+ @Override
+ public void close() throws IOException {
+ closed = true;
+ cache.clear();
+ }
+
+ @Override
+ public long getFilePointer() {
+ return position;
+ }
+
+ @Override
+ public void seek(long pos) throws IOException {
+ ensureOpen();
+ if (pos < 0 || pos > length) {
+ throw new IOException("Seek position out of bounds: " + pos);
Review Comment:
made the change to seekInternal function as extending BufferedIndexInput
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17949: Add Azure Blob Storage backup repository module [solr]
Posted by "prateeksinghalgit (via GitHub)" <gi...@apache.org>.
prateeksinghalgit commented on PR #3750:
URL: https://github.com/apache/solr/pull/3750#issuecomment-4428454450
> Only reviewed the production code (skiped tests and other stuff).
>
> Some general comments on the review:
>
> * can you replace all unnecessary qualified class names by imports? (replacing `java.lang.String` by `String`) This makes the code painful to read
> * there is a lot of shared logic between Azure backups and S3 backups. Lot of code is duplicated. I wonder if abstract classes should be introduced to factor out some logic
> * we always upload files with a _block list_. At least for small files, can we keep things simples with a simple pull? It think we may skip the commit call.
> * overall, I think it's pretty inefficient that `AzureBlobIndexInput` always reads files by small pages. Mostly for backup/restore, we will only fetch full files. You should not do a request by page, or use much bigger pages (without keeping them in memory).
>
> Please note I did a code review, but I have no way to test it with Azure.
@psalagnac , thank you for taking the time to review.
1) made the change by replacing all unnecessary qualified class names by imports
2) s3-repository is in active production use; a refactor would force shared abstractions and re-testing on a stable module to land a brand-new one. That's significant scope creep with no end-user benefit. Also the duplication is mostly in boilerplate (resolveDirectory shape, sanitization helpers); both repositories have different SDK contracts underneath.
Let me know what you think.
3) Given the comments on AzureBlobIndexInput, I have rewritten AzureBlobIndexInput to extend Lucene's standard BufferedIndexInput instead of IndexInput directly. Now only override readInternal(ByteBuffer) and seekInternal(long), and everything else (buffer management, slice, clone, EOF handling) comes from the parent class.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17949: Add Azure Blob Storage backup repository module [solr]
Posted by "prateeksinghalgit (via GitHub)" <gi...@apache.org>.
prateeksinghalgit commented on PR #3750:
URL: https://github.com/apache/solr/pull/3750#issuecomment-4290990310
Thanks @janhoy for adding more reviewers. @HoustonPutman and @psalagnac ,let me know if you have any questions related to the pr.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17949: Add Azure Blob Storage backup repository module [solr]
Posted by "prateeksinghalgit (via GitHub)" <gi...@apache.org>.
prateeksinghalgit commented on code in PR #3750:
URL: https://github.com/apache/solr/pull/3750#discussion_r3359080483
##########
solr/modules/azure-blob-repository/src/java/org/apache/solr/azureblob/AzureBlobOutputStream.java:
##########
@@ -0,0 +1,269 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.azureblob;
+
+import com.azure.storage.blob.BlobClient;
+import com.azure.storage.blob.models.BlobStorageException;
+import com.azure.storage.blob.specialized.BlockBlobClient;
+import java.io.ByteArrayInputStream;
+import java.io.IOException;
+import java.io.OutputStream;
+import java.lang.invoke.MethodHandles;
+import java.nio.ByteBuffer;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Base64;
+import java.util.List;
+import java.util.UUID;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * OutputStream implementation for Azure Blob Storage using block blobs. Supports chunked uploads
+ * for large files.
+ */
+public class AzureBlobOutputStream extends OutputStream {
+ private static final Logger log = LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+
+ static final int BLOCK_SIZE = 4 * 1024 * 1024;
+
+ private final BlobClient blobClient;
+ private final String blobPath;
+ private volatile boolean closed;
+ private final ByteBuffer buffer;
+ private BlockUpload blockUpload;
+ private boolean committed;
+
+ public AzureBlobOutputStream(BlobClient blobClient, String blobPath) {
+ this.blobClient = blobClient;
+ this.blobPath = blobPath;
+ this.closed = false;
+ this.buffer = ByteBuffer.allocate(BLOCK_SIZE);
+ this.blockUpload = null;
+ this.committed = false;
+
+ if (log.isDebugEnabled()) {
+ log.debug("Created BlobOutputStream for blobPath '{}'", blobPath);
+ }
+ }
+
+ @Override
+ public void write(int b) throws IOException {
+ if (closed) {
+ throw new IOException("Stream closed");
+ }
+
+ buffer.put((byte) b);
+
+ if (!buffer.hasRemaining()) {
+ uploadBlock();
+ }
+ }
+
+ @Override
+ public void write(byte[] b, int off, int len) throws IOException {
+ if (closed) {
+ throw new IOException("Stream closed");
+ }
+
+ if (outOfRange(off, b.length) || len < 0 || outOfRange(off + len, b.length)) {
+ throw new IndexOutOfBoundsException();
+ } else if (len == 0) {
+ return;
+ }
+
+ int currentOffset = off;
+ int lenRemaining = len;
+ while (buffer.remaining() < lenRemaining) {
+ int firstPart = buffer.remaining();
+ buffer.put(b, currentOffset, firstPart);
+ uploadBlock();
+
+ currentOffset += firstPart;
+ lenRemaining -= firstPart;
+ }
+ if (lenRemaining > 0) {
+ buffer.put(b, currentOffset, lenRemaining);
+ }
+ }
+
+ private static boolean outOfRange(int off, int len) {
+ return off < 0 || off > len;
+ }
+
+ private void uploadBlock() throws IOException {
+ int size = buffer.position() - buffer.arrayOffset();
+
+ if (size == 0) {
+ return;
+ }
+
+ if (blockUpload == null) {
+ if (log.isDebugEnabled()) {
+ log.debug("New block upload for blobPath '{}'", blobPath);
+ }
+
+ blockUpload = newBlockUpload();
+ }
+
+ try (ByteArrayInputStream inputStream =
+ new ByteArrayInputStream(buffer.array(), buffer.arrayOffset(), size)) {
+ blockUpload.uploadBlock(inputStream, size);
+ } catch (BlobStorageException e) {
+ if (blockUpload != null) {
+ blockUpload.abort();
+ if (log.isDebugEnabled()) {
+ log.debug("Block upload aborted for blobPath '{}'.", blobPath);
+ }
+ }
+
+ throw new IOException(
+ "Failed to upload block", AzureBlobStorageClient.handleBlobException(e));
+ }
+
+ buffer.clear();
+ }
+
+ @Override
+ public void flush() throws IOException {
+ if (closed) {
+ throw new IOException("Stream closed");
+ }
+
+ if (buffer.position() - buffer.arrayOffset() > 0) {
Review Comment:
made the change
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17949: Add Azure Blob Storage backup repository module [solr]
Posted by "prateeksinghalgit (via GitHub)" <gi...@apache.org>.
prateeksinghalgit commented on code in PR #3750:
URL: https://github.com/apache/solr/pull/3750#discussion_r3232137250
##########
solr/modules/azure-blob-repository/src/java/org/apache/solr/azureblob/AzureBlobStorageClient.java:
##########
@@ -0,0 +1,529 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.azureblob;
+
+import com.azure.core.credential.TokenCredential;
+import com.azure.identity.DefaultAzureCredentialBuilder;
+import com.azure.storage.blob.BlobClient;
+import com.azure.storage.blob.BlobContainerClient;
+import com.azure.storage.blob.BlobServiceClient;
+import com.azure.storage.blob.BlobServiceClientBuilder;
+import com.azure.storage.blob.models.BlobItem;
+import com.azure.storage.blob.models.BlobStorageException;
+import com.azure.storage.blob.models.ListBlobsOptions;
+import com.google.common.annotations.VisibleForTesting;
+import java.io.ByteArrayInputStream;
+import java.io.FilterInputStream;
+import java.io.InputStream;
+import java.io.OutputStream;
+import java.lang.invoke.MethodHandles;
+import java.util.Collection;
+import java.util.HashSet;
+import java.util.Locale;
+import java.util.Set;
+import java.util.stream.Collectors;
+import org.apache.solr.common.util.ResumableInputStream;
+import org.apache.solr.common.util.StrUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Creates a {@link BlobServiceClient} for communicating with Azure Blob Storage. Utilizes the
+ * default Azure credential provider chain.
+ */
+public class AzureBlobStorageClient {
+
+ private static final Logger log = LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+
+ static final String BLOB_FILE_PATH_DELIMITER = "/";
+ private static final int HTTP_NOT_FOUND = 404;
+ private static final int HTTP_CONFLICT = 409;
+ private static final int SKIP_BUFFER_SIZE = 8192;
+ private static final int DELETE_BATCH_SIZE = 1000;
+
+ private static final com.azure.core.http.HttpClient SHARED_HTTP_CLIENT =
+ new com.azure.core.http.okhttp.OkHttpAsyncHttpClientBuilder().build();
+
+ private final BlobContainerClient containerClient;
+
+ AzureBlobStorageClient(
+ String containerName,
+ String connectionString,
+ String endpoint,
+ String accountName,
+ String accountKey,
+ String sasToken,
+ String tenantId,
+ String clientId,
+ String clientSecret) {
+ this(
+ createInternalClient(
+ connectionString,
+ endpoint,
+ accountName,
+ accountKey,
+ sasToken,
+ tenantId,
+ clientId,
+ clientSecret),
+ containerName);
+ }
+
+ @VisibleForTesting
+ AzureBlobStorageClient(BlobServiceClient blobServiceClient, String containerName) {
+ this.containerClient = blobServiceClient.getBlobContainerClient(containerName);
+ try {
+ containerClient.create();
+ } catch (BlobStorageException e) {
+ if (e.getStatusCode() != HTTP_CONFLICT) {
+ throw e;
+ }
+ }
+ }
+
+ private static BlobServiceClient createInternalClient(
+ String connectionString,
+ String endpoint,
+ String accountName,
+ String accountKey,
+ String sasToken,
+ String tenantId,
+ String clientId,
+ String clientSecret) {
+
+ BlobServiceClientBuilder builder = new BlobServiceClientBuilder();
+ builder.httpClient(SHARED_HTTP_CLIENT);
+
+ if (StrUtils.isNotNullOrEmpty(connectionString)) {
+ builder.connectionString(connectionString);
+ } else if (StrUtils.isNotNullOrEmpty(endpoint)) {
+ builder.endpoint(endpoint);
+ if (StrUtils.isNotNullOrEmpty(accountName) && StrUtils.isNotNullOrEmpty(accountKey)) {
+ builder.credential(
+ new com.azure.storage.common.StorageSharedKeyCredential(accountName, accountKey));
+ } else if (StrUtils.isNotNullOrEmpty(sasToken)) {
+ builder.sasToken(sasToken);
+ } else {
+ TokenCredential credential = new DefaultAzureCredentialBuilder().tenantId(tenantId).build();
+ builder.credential(credential);
+ }
+ } else {
+ throw new IllegalArgumentException("Either connectionString or endpoint must be provided");
+ }
+
+ return builder.buildClient();
+ }
+
+ void createDirectory(String path) throws AzureBlobException {
+ String sanitizedDirPath = sanitizedDirPath(path);
+
+ if (!pathExists(sanitizedDirPath)) {
+ String parent = getParentDirectory(sanitizedDirPath);
+ if (!parent.isEmpty() && !parent.equals(BLOB_FILE_PATH_DELIMITER)) {
+ createDirectory(parent);
+ }
+
+ try {
+ BlobClient blobClient = containerClient.getBlobClient(sanitizedDirPath);
+ blobClient.upload(new ByteArrayInputStream(new byte[0]), 0, true);
+ java.util.Map<String, String> metadata = new java.util.HashMap<>();
+ metadata.put("hdi_isfolder", "true");
+ blobClient.setMetadata(metadata);
+ } catch (BlobStorageException e) {
+ throw handleBlobException(e);
+ }
+ }
+ }
+
+ void delete(Collection<String> paths) throws AzureBlobException {
+ Set<String> entries = new HashSet<>();
+ for (String path : paths) {
+ entries.add(sanitizedFilePath(path));
+ }
+ deleteBlobs(entries);
+ }
+
+ void deleteDirectory(String path) throws AzureBlobException {
+ path = sanitizedDirPath(path);
+
+ Set<String> entries = listAll(path);
+ if (pathExists(path)) {
+ entries.add(path);
+ }
+
+ deleteBlobs(entries);
+ }
+
+ String[] listDir(String path) throws AzureBlobException {
+ path = sanitizedDirPath(path);
+
+ try {
+ ListBlobsOptions options = new ListBlobsOptions().setPrefix(path).setMaxResultsPerPage(1000);
+
+ final String finalPath = path;
+ return containerClient.listBlobs(options, null).stream()
+ .map(BlobItem::getName)
+ .filter(s -> s.startsWith(finalPath))
+ .map(s -> s.substring(finalPath.length()))
+ .filter(s -> !s.isEmpty())
+ .filter(
+ s -> {
+ int slashIndex = s.indexOf(BLOB_FILE_PATH_DELIMITER);
+ return slashIndex == -1 || slashIndex == s.length() - 1;
+ })
+ .toArray(String[]::new);
+ } catch (BlobStorageException e) {
+ throw handleBlobException(e);
+ }
+ }
+
+ boolean pathExists(String path) throws AzureBlobException {
+ final String blobPath = sanitizedPath(path);
+
+ if (blobPath.isEmpty() || BLOB_FILE_PATH_DELIMITER.equals(blobPath)) {
+ return true;
+ }
+
+ try {
+ BlobClient blobClient = containerClient.getBlobClient(blobPath);
+ return blobClient.exists();
+ } catch (BlobStorageException e) {
+ throw handleBlobException(e);
+ }
+ }
+
+ boolean isDirectory(String path) throws AzureBlobException {
+ final String dirPrefix = sanitizedDirPath(path);
+
+ try {
+ ListBlobsOptions options =
+ new ListBlobsOptions().setPrefix(dirPrefix).setMaxResultsPerPage(1);
+ if (containerClient.listBlobs(options, null).iterator().hasNext()) {
+ return true;
+ }
+
+ BlobClient markerClient = containerClient.getBlobClient(dirPrefix);
+ if (markerClient.exists()) {
+ long size = markerClient.getProperties().getBlobSize();
+ if (size == 0) {
+ return true;
+ }
+ java.util.Map<String, String> md = markerClient.getProperties().getMetadata();
+ return md != null && md.containsKey("hdi_isfolder");
+ }
+
+ return false;
+ } catch (BlobStorageException e) {
+ throw handleBlobException(e);
+ }
+ }
+
+ long length(String path) throws AzureBlobException {
+ String blobPath = sanitizedFilePath(path);
+ try {
+ BlobClient blobClient = containerClient.getBlobClient(blobPath);
+ return blobClient.getProperties().getBlobSize();
+ } catch (BlobStorageException e) {
+ throw handleBlobException(e);
+ }
+ }
+
+ InputStream pullStream(String path) throws AzureBlobException {
+ final String blobPath = sanitizedFilePath(path);
+
+ try {
+ BlobClient blobClient = containerClient.getBlobClient(blobPath);
+ final long contentLength = blobClient.getProperties().getBlobSize();
Review Comment:
made the change
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17949: Add Azure Blob Storage backup repository module [solr]
Posted by "prateeksinghalgit (via GitHub)" <gi...@apache.org>.
prateeksinghalgit commented on code in PR #3750:
URL: https://github.com/apache/solr/pull/3750#discussion_r3232166756
##########
solr/modules/azure-blob-repository/src/java/org/apache/solr/azureblob/AzureBlobOutputStream.java:
##########
@@ -0,0 +1,269 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.azureblob;
+
+import com.azure.storage.blob.BlobClient;
+import com.azure.storage.blob.models.BlobStorageException;
+import com.azure.storage.blob.specialized.BlockBlobClient;
+import java.io.ByteArrayInputStream;
+import java.io.IOException;
+import java.io.OutputStream;
+import java.lang.invoke.MethodHandles;
+import java.nio.ByteBuffer;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Base64;
+import java.util.List;
+import java.util.UUID;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * OutputStream implementation for Azure Blob Storage using block blobs. Supports chunked uploads
+ * for large files.
+ */
+public class AzureBlobOutputStream extends OutputStream {
+ private static final Logger log = LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+
+ static final int BLOCK_SIZE = 4 * 1024 * 1024;
+
+ private final BlobClient blobClient;
+ private final String blobPath;
+ private volatile boolean closed;
+ private final ByteBuffer buffer;
+ private BlockUpload blockUpload;
+ private boolean committed;
+
+ public AzureBlobOutputStream(BlobClient blobClient, String blobPath) {
+ this.blobClient = blobClient;
+ this.blobPath = blobPath;
+ this.closed = false;
+ this.buffer = ByteBuffer.allocate(BLOCK_SIZE);
+ this.blockUpload = null;
+ this.committed = false;
+
+ if (log.isDebugEnabled()) {
+ log.debug("Created BlobOutputStream for blobPath '{}'", blobPath);
+ }
+ }
+
+ @Override
+ public void write(int b) throws IOException {
+ if (closed) {
+ throw new IOException("Stream closed");
+ }
+
+ buffer.put((byte) b);
+
+ if (!buffer.hasRemaining()) {
+ uploadBlock();
+ }
+ }
+
+ @Override
+ public void write(byte[] b, int off, int len) throws IOException {
+ if (closed) {
+ throw new IOException("Stream closed");
+ }
+
+ if (outOfRange(off, b.length) || len < 0 || outOfRange(off + len, b.length)) {
+ throw new IndexOutOfBoundsException();
+ } else if (len == 0) {
+ return;
+ }
+
+ int currentOffset = off;
+ int lenRemaining = len;
+ while (buffer.remaining() < lenRemaining) {
+ int firstPart = buffer.remaining();
+ buffer.put(b, currentOffset, firstPart);
+ uploadBlock();
+
+ currentOffset += firstPart;
+ lenRemaining -= firstPart;
+ }
+ if (lenRemaining > 0) {
+ buffer.put(b, currentOffset, lenRemaining);
+ }
+ }
+
+ private static boolean outOfRange(int off, int len) {
+ return off < 0 || off > len;
+ }
+
+ private void uploadBlock() throws IOException {
+ int size = buffer.position() - buffer.arrayOffset();
+
+ if (size == 0) {
+ return;
+ }
+
+ if (blockUpload == null) {
+ if (log.isDebugEnabled()) {
+ log.debug("New block upload for blobPath '{}'", blobPath);
+ }
+
+ blockUpload = newBlockUpload();
+ }
+
+ try (ByteArrayInputStream inputStream =
+ new ByteArrayInputStream(buffer.array(), buffer.arrayOffset(), size)) {
+ blockUpload.uploadBlock(inputStream, size);
+ } catch (BlobStorageException e) {
+ if (blockUpload != null) {
+ blockUpload.abort();
+ if (log.isDebugEnabled()) {
+ log.debug("Block upload aborted for blobPath '{}'.", blobPath);
+ }
+ }
+
+ throw new IOException(
+ "Failed to upload block", AzureBlobStorageClient.handleBlobException(e));
+ }
+
+ buffer.clear();
+ }
+
+ @Override
+ public void flush() throws IOException {
+ if (closed) {
+ throw new IOException("Stream closed");
+ }
+
+ if (buffer.position() - buffer.arrayOffset() > 0) {
+ uploadBlock();
+ }
+
+ if (blockUpload != null) {
+ blockUpload.complete();
+ blockUpload = null;
+ committed = true;
+ }
+ }
+
+ @Override
+ public void close() throws IOException {
+ if (closed) {
+ return;
+ }
+
+ if (blockUpload != null && blockUpload.aborted) {
+ blockUpload = null;
+ closed = true;
+ return;
+ }
+
+ if (!committed) {
+ uploadBlock();
+ if (blockUpload != null) {
+ blockUpload.complete();
+ blockUpload = null;
+ committed = true;
+ } else {
+ try {
+ blobClient.upload(new ByteArrayInputStream(new byte[0]), 0, true);
+ } catch (BlobStorageException e) {
+ throw new IOException(
+ "Failed to create empty blob", AzureBlobStorageClient.handleBlobException(e));
+ }
+ }
+ } else {
+ if (blockUpload != null) {
+ blockUpload.complete();
+ blockUpload = null;
+ }
+ }
+
+ closed = true;
+ }
+
+ private BlockUpload newBlockUpload() throws IOException {
+ try {
+ return new BlockUpload();
+ } catch (BlobStorageException e) {
+ throw new IOException(
+ "Failed to create block upload", AzureBlobStorageClient.handleBlobException(e));
+ }
+ }
+
+ private class BlockUpload {
+ private final List<String> blockIds;
+ private boolean aborted = false;
+
+ public BlockUpload() {
+ this.blockIds = new ArrayList<>();
+ if (log.isDebugEnabled()) {
+ log.debug("Initiated block upload for blobPath '{}'", blobPath);
+ }
+
+ try {
+ BlockBlobClient blockBlobClient = blobClient.getBlockBlobClient();
+ blockBlobClient.deleteIfExists();
Review Comment:
removed it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17949: Add Azure Blob Storage backup repository module [solr]
Posted by "prateeksinghalgit (via GitHub)" <gi...@apache.org>.
prateeksinghalgit commented on code in PR #3750:
URL: https://github.com/apache/solr/pull/3750#discussion_r3232147739
##########
solr/modules/azure-blob-repository/src/java/org/apache/solr/azureblob/AzureBlobStorageClient.java:
##########
@@ -0,0 +1,529 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.azureblob;
+
+import com.azure.core.credential.TokenCredential;
+import com.azure.identity.DefaultAzureCredentialBuilder;
+import com.azure.storage.blob.BlobClient;
+import com.azure.storage.blob.BlobContainerClient;
+import com.azure.storage.blob.BlobServiceClient;
+import com.azure.storage.blob.BlobServiceClientBuilder;
+import com.azure.storage.blob.models.BlobItem;
+import com.azure.storage.blob.models.BlobStorageException;
+import com.azure.storage.blob.models.ListBlobsOptions;
+import com.google.common.annotations.VisibleForTesting;
+import java.io.ByteArrayInputStream;
+import java.io.FilterInputStream;
+import java.io.InputStream;
+import java.io.OutputStream;
+import java.lang.invoke.MethodHandles;
+import java.util.Collection;
+import java.util.HashSet;
+import java.util.Locale;
+import java.util.Set;
+import java.util.stream.Collectors;
+import org.apache.solr.common.util.ResumableInputStream;
+import org.apache.solr.common.util.StrUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Creates a {@link BlobServiceClient} for communicating with Azure Blob Storage. Utilizes the
+ * default Azure credential provider chain.
+ */
+public class AzureBlobStorageClient {
+
+ private static final Logger log = LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+
+ static final String BLOB_FILE_PATH_DELIMITER = "/";
+ private static final int HTTP_NOT_FOUND = 404;
+ private static final int HTTP_CONFLICT = 409;
+ private static final int SKIP_BUFFER_SIZE = 8192;
+ private static final int DELETE_BATCH_SIZE = 1000;
+
+ private static final com.azure.core.http.HttpClient SHARED_HTTP_CLIENT =
+ new com.azure.core.http.okhttp.OkHttpAsyncHttpClientBuilder().build();
+
+ private final BlobContainerClient containerClient;
+
+ AzureBlobStorageClient(
+ String containerName,
+ String connectionString,
+ String endpoint,
+ String accountName,
+ String accountKey,
+ String sasToken,
+ String tenantId,
+ String clientId,
+ String clientSecret) {
+ this(
+ createInternalClient(
+ connectionString,
+ endpoint,
+ accountName,
+ accountKey,
+ sasToken,
+ tenantId,
+ clientId,
+ clientSecret),
+ containerName);
+ }
+
+ @VisibleForTesting
+ AzureBlobStorageClient(BlobServiceClient blobServiceClient, String containerName) {
+ this.containerClient = blobServiceClient.getBlobContainerClient(containerName);
+ try {
+ containerClient.create();
+ } catch (BlobStorageException e) {
+ if (e.getStatusCode() != HTTP_CONFLICT) {
+ throw e;
+ }
+ }
+ }
+
+ private static BlobServiceClient createInternalClient(
+ String connectionString,
+ String endpoint,
+ String accountName,
+ String accountKey,
+ String sasToken,
+ String tenantId,
+ String clientId,
+ String clientSecret) {
+
+ BlobServiceClientBuilder builder = new BlobServiceClientBuilder();
+ builder.httpClient(SHARED_HTTP_CLIENT);
+
+ if (StrUtils.isNotNullOrEmpty(connectionString)) {
+ builder.connectionString(connectionString);
+ } else if (StrUtils.isNotNullOrEmpty(endpoint)) {
+ builder.endpoint(endpoint);
+ if (StrUtils.isNotNullOrEmpty(accountName) && StrUtils.isNotNullOrEmpty(accountKey)) {
+ builder.credential(
+ new com.azure.storage.common.StorageSharedKeyCredential(accountName, accountKey));
+ } else if (StrUtils.isNotNullOrEmpty(sasToken)) {
+ builder.sasToken(sasToken);
+ } else {
+ TokenCredential credential = new DefaultAzureCredentialBuilder().tenantId(tenantId).build();
+ builder.credential(credential);
+ }
+ } else {
+ throw new IllegalArgumentException("Either connectionString or endpoint must be provided");
+ }
+
+ return builder.buildClient();
+ }
+
+ void createDirectory(String path) throws AzureBlobException {
+ String sanitizedDirPath = sanitizedDirPath(path);
+
+ if (!pathExists(sanitizedDirPath)) {
+ String parent = getParentDirectory(sanitizedDirPath);
+ if (!parent.isEmpty() && !parent.equals(BLOB_FILE_PATH_DELIMITER)) {
+ createDirectory(parent);
+ }
+
+ try {
+ BlobClient blobClient = containerClient.getBlobClient(sanitizedDirPath);
+ blobClient.upload(new ByteArrayInputStream(new byte[0]), 0, true);
+ java.util.Map<String, String> metadata = new java.util.HashMap<>();
+ metadata.put("hdi_isfolder", "true");
+ blobClient.setMetadata(metadata);
+ } catch (BlobStorageException e) {
+ throw handleBlobException(e);
+ }
+ }
+ }
+
+ void delete(Collection<String> paths) throws AzureBlobException {
+ Set<String> entries = new HashSet<>();
+ for (String path : paths) {
+ entries.add(sanitizedFilePath(path));
+ }
+ deleteBlobs(entries);
+ }
+
+ void deleteDirectory(String path) throws AzureBlobException {
+ path = sanitizedDirPath(path);
+
+ Set<String> entries = listAll(path);
+ if (pathExists(path)) {
+ entries.add(path);
+ }
+
+ deleteBlobs(entries);
+ }
+
+ String[] listDir(String path) throws AzureBlobException {
+ path = sanitizedDirPath(path);
+
+ try {
+ ListBlobsOptions options = new ListBlobsOptions().setPrefix(path).setMaxResultsPerPage(1000);
+
+ final String finalPath = path;
+ return containerClient.listBlobs(options, null).stream()
+ .map(BlobItem::getName)
+ .filter(s -> s.startsWith(finalPath))
+ .map(s -> s.substring(finalPath.length()))
+ .filter(s -> !s.isEmpty())
+ .filter(
+ s -> {
+ int slashIndex = s.indexOf(BLOB_FILE_PATH_DELIMITER);
+ return slashIndex == -1 || slashIndex == s.length() - 1;
+ })
+ .toArray(String[]::new);
+ } catch (BlobStorageException e) {
+ throw handleBlobException(e);
+ }
+ }
+
+ boolean pathExists(String path) throws AzureBlobException {
+ final String blobPath = sanitizedPath(path);
+
+ if (blobPath.isEmpty() || BLOB_FILE_PATH_DELIMITER.equals(blobPath)) {
+ return true;
+ }
+
+ try {
+ BlobClient blobClient = containerClient.getBlobClient(blobPath);
+ return blobClient.exists();
+ } catch (BlobStorageException e) {
+ throw handleBlobException(e);
+ }
+ }
+
+ boolean isDirectory(String path) throws AzureBlobException {
+ final String dirPrefix = sanitizedDirPath(path);
+
+ try {
+ ListBlobsOptions options =
+ new ListBlobsOptions().setPrefix(dirPrefix).setMaxResultsPerPage(1);
+ if (containerClient.listBlobs(options, null).iterator().hasNext()) {
+ return true;
+ }
+
+ BlobClient markerClient = containerClient.getBlobClient(dirPrefix);
+ if (markerClient.exists()) {
+ long size = markerClient.getProperties().getBlobSize();
+ if (size == 0) {
+ return true;
+ }
+ java.util.Map<String, String> md = markerClient.getProperties().getMetadata();
+ return md != null && md.containsKey("hdi_isfolder");
+ }
+
+ return false;
+ } catch (BlobStorageException e) {
+ throw handleBlobException(e);
+ }
+ }
+
+ long length(String path) throws AzureBlobException {
+ String blobPath = sanitizedFilePath(path);
+ try {
+ BlobClient blobClient = containerClient.getBlobClient(blobPath);
+ return blobClient.getProperties().getBlobSize();
+ } catch (BlobStorageException e) {
+ throw handleBlobException(e);
+ }
+ }
+
+ InputStream pullStream(String path) throws AzureBlobException {
+ final String blobPath = sanitizedFilePath(path);
+
+ try {
+ BlobClient blobClient = containerClient.getBlobClient(blobPath);
+ final long contentLength = blobClient.getProperties().getBlobSize();
+
+ if (contentLength == 0) {
+ return new ByteArrayInputStream(new byte[0]);
+ }
+
+ InputStream initial = new IdempotentCloseInputStream(blobClient.openInputStream());
+
+ return new ResumableInputStream(
+ initial,
+ bytesRead -> {
+ if (contentLength > 0 && bytesRead >= contentLength) {
+ return null;
+ }
+ try {
+ long remaining =
+ contentLength > 0 ? Math.max(0, contentLength - bytesRead) : Long.MAX_VALUE;
+ return pullRangeStream(path, bytesRead, remaining);
+ } catch (AzureBlobException e) {
+ throw new RuntimeException(e);
+ }
+ });
+ } catch (BlobStorageException e) {
+ throw handleBlobException(e);
+ }
+ }
+
+ InputStream pullRangeStream(String path, long offset, long length) throws AzureBlobException {
+ final String blobPath = sanitizedFilePath(path);
+ try {
+ BlobClient blobClient = containerClient.getBlobClient(blobPath);
+ com.azure.storage.blob.models.BlobRange range =
+ new com.azure.storage.blob.models.BlobRange(offset, length);
+ return new IdempotentCloseInputStream(blobClient.openInputStream(range, null));
+ } catch (BlobStorageException e) {
+ throw handleBlobException(e);
+ }
+ }
+
+ private static final class IdempotentCloseInputStream extends FilterInputStream {
+ private boolean closed;
+
+ IdempotentCloseInputStream(InputStream in) {
+ super(in);
+ this.closed = false;
+ }
+
+ @Override
+ public int read() throws java.io.IOException {
+ if (closed) {
+ throw new java.io.IOException("Stream is already closed");
+ }
+ try {
+ return super.read();
+ } catch (RuntimeException re) {
+ if (isAlreadyClosed(re)) {
+ throw new java.io.IOException("Stream is already closed", re);
+ }
+ throw re;
+ }
+ }
+
+ @Override
+ public int read(byte[] b, int off, int len) throws java.io.IOException {
+ if (closed) {
+ throw new java.io.IOException("Stream is already closed");
+ }
+ try {
+ return super.read(b, off, len);
+ } catch (RuntimeException re) {
+ if (isAlreadyClosed(re)) {
+ throw new java.io.IOException("Stream is already closed", re);
+ }
+ throw re;
+ }
+ }
+
+ @Override
+ public void close() throws java.io.IOException {
+ if (closed) {
+ return;
+ }
+ try {
+ super.close();
+ } catch (java.io.IOException e) {
+ String msg = e.getMessage();
+ if (msg == null || !msg.toLowerCase(java.util.Locale.ROOT).contains("already closed")) {
+ throw e;
+ }
+ // swallow "already closed" to make close idempotent
+ } finally {
+ closed = true;
+ }
+ }
+
+ @Override
+ public long skip(long n) throws java.io.IOException {
+ if (closed) {
+ throw new java.io.IOException("Stream is already closed");
+ }
+ if (n <= 0) {
+ return 0L;
+ }
+ long remaining = n;
+ byte[] discard = new byte[SKIP_BUFFER_SIZE];
+ try {
+ while (remaining > 0) {
+ int toRead = (int) Math.min(discard.length, remaining);
+ int read = super.read(discard, 0, toRead);
+ if (read < 0) {
+ break;
+ }
+ remaining -= read;
+ }
+ return n - remaining;
+ } catch (RuntimeException re) {
+ throw new java.io.IOException(re);
+ }
+ }
+
+ private static boolean isAlreadyClosed(Throwable t) {
+ String msg = t.getMessage();
+ return msg != null && msg.toLowerCase(java.util.Locale.ROOT).contains("already closed");
+ }
+ }
+
+ OutputStream pushStream(String path) throws AzureBlobException {
+ path = sanitizedFilePath(path);
+
+ if (!parentDirectoryExist(path)) {
+ String parentDirectory = getParentDirectory(path);
+ if (!parentDirectory.isEmpty() && !parentDirectory.equals(BLOB_FILE_PATH_DELIMITER)) {
+ createDirectory(parentDirectory);
+ }
+ }
+
+ try {
+ BlobClient blobClient = containerClient.getBlobClient(path);
+ return new AzureBlobOutputStream(blobClient, path);
+ } catch (BlobStorageException e) {
+ throw handleBlobException(e);
+ }
+ }
+
+ void close() {}
+
+ @VisibleForTesting
+ void deleteContainerForTests() {
+ try {
+ containerClient.delete();
+ } catch (BlobStorageException e) {
+ if (e.getStatusCode() != HTTP_NOT_FOUND) {
+ throw e;
+ }
+ }
+ }
+
+ private Collection<String> deleteBlobs(Collection<String> paths) throws AzureBlobException {
+ try {
+ return deleteBlobs(paths, DELETE_BATCH_SIZE);
Review Comment:
with the deletes updated to batch deletes, this parameter is getting used now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17949: Add Azure Blob Storage backup repository module [solr]
Posted by "prateeksinghalgit (via GitHub)" <gi...@apache.org>.
prateeksinghalgit commented on code in PR #3750:
URL: https://github.com/apache/solr/pull/3750#discussion_r3232260800
##########
solr/modules/azure-blob-repository/src/java/org/apache/solr/azureblob/AzureBlobOutputStream.java:
##########
@@ -0,0 +1,269 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.azureblob;
+
+import com.azure.storage.blob.BlobClient;
+import com.azure.storage.blob.models.BlobStorageException;
+import com.azure.storage.blob.specialized.BlockBlobClient;
+import java.io.ByteArrayInputStream;
+import java.io.IOException;
+import java.io.OutputStream;
+import java.lang.invoke.MethodHandles;
+import java.nio.ByteBuffer;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Base64;
+import java.util.List;
+import java.util.UUID;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * OutputStream implementation for Azure Blob Storage using block blobs. Supports chunked uploads
+ * for large files.
+ */
+public class AzureBlobOutputStream extends OutputStream {
+ private static final Logger log = LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+
+ static final int BLOCK_SIZE = 4 * 1024 * 1024;
+
+ private final BlobClient blobClient;
+ private final String blobPath;
+ private volatile boolean closed;
+ private final ByteBuffer buffer;
+ private BlockUpload blockUpload;
+ private boolean committed;
+
+ public AzureBlobOutputStream(BlobClient blobClient, String blobPath) {
+ this.blobClient = blobClient;
+ this.blobPath = blobPath;
+ this.closed = false;
+ this.buffer = ByteBuffer.allocate(BLOCK_SIZE);
+ this.blockUpload = null;
+ this.committed = false;
+
+ if (log.isDebugEnabled()) {
+ log.debug("Created BlobOutputStream for blobPath '{}'", blobPath);
+ }
+ }
+
+ @Override
+ public void write(int b) throws IOException {
+ if (closed) {
+ throw new IOException("Stream closed");
+ }
+
+ buffer.put((byte) b);
+
+ if (!buffer.hasRemaining()) {
+ uploadBlock();
+ }
+ }
+
+ @Override
+ public void write(byte[] b, int off, int len) throws IOException {
+ if (closed) {
+ throw new IOException("Stream closed");
+ }
+
+ if (outOfRange(off, b.length) || len < 0 || outOfRange(off + len, b.length)) {
+ throw new IndexOutOfBoundsException();
+ } else if (len == 0) {
+ return;
+ }
+
+ int currentOffset = off;
+ int lenRemaining = len;
+ while (buffer.remaining() < lenRemaining) {
+ int firstPart = buffer.remaining();
+ buffer.put(b, currentOffset, firstPart);
+ uploadBlock();
+
+ currentOffset += firstPart;
+ lenRemaining -= firstPart;
+ }
+ if (lenRemaining > 0) {
+ buffer.put(b, currentOffset, lenRemaining);
+ }
+ }
+
+ private static boolean outOfRange(int off, int len) {
+ return off < 0 || off > len;
+ }
+
+ private void uploadBlock() throws IOException {
+ int size = buffer.position() - buffer.arrayOffset();
+
+ if (size == 0) {
+ return;
+ }
+
+ if (blockUpload == null) {
+ if (log.isDebugEnabled()) {
+ log.debug("New block upload for blobPath '{}'", blobPath);
+ }
+
+ blockUpload = newBlockUpload();
+ }
+
+ try (ByteArrayInputStream inputStream =
+ new ByteArrayInputStream(buffer.array(), buffer.arrayOffset(), size)) {
+ blockUpload.uploadBlock(inputStream, size);
+ } catch (BlobStorageException e) {
+ if (blockUpload != null) {
+ blockUpload.abort();
+ if (log.isDebugEnabled()) {
+ log.debug("Block upload aborted for blobPath '{}'.", blobPath);
+ }
+ }
+
+ throw new IOException(
+ "Failed to upload block", AzureBlobStorageClient.handleBlobException(e));
+ }
+
+ buffer.clear();
+ }
+
+ @Override
+ public void flush() throws IOException {
+ if (closed) {
+ throw new IOException("Stream closed");
+ }
+
+ if (buffer.position() - buffer.arrayOffset() > 0) {
+ uploadBlock();
+ }
+
+ if (blockUpload != null) {
+ blockUpload.complete();
Review Comment:
fixed. flush() now only calls uploadBlock()
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17949: Add Azure Blob Storage backup repository module [solr]
Posted by "prateeksinghalgit (via GitHub)" <gi...@apache.org>.
prateeksinghalgit commented on code in PR #3750:
URL: https://github.com/apache/solr/pull/3750#discussion_r3469250665
##########
solr/solrj/gradle.lockfile:
##########
@@ -17,13 +17,19 @@ com.github.stephenc.jcip:jcip-annotations:1.0-1=compileClasspath,compileOnlyHelp
com.google.auto.service:auto-service-annotations:1.0.1=annotationProcessor,errorprone,testAnnotationProcessor
com.google.auto.value:auto-value-annotations:1.11.0=annotationProcessor,errorprone,testAnnotationProcessor
com.google.auto:auto-common:1.2.2=annotationProcessor,errorprone,testAnnotationProcessor
+com.google.code.findbugs:jsr305:3.0.2=spotless865458226
com.google.errorprone:error_prone_annotation:2.41.0=annotationProcessor,errorprone,testAnnotationProcessor
+com.google.errorprone:error_prone_annotations:2.18.0=spotless865458226
Review Comment:
Same as above, they came from running resolveAndLockAll --write-locks repo-wide while validating the CI build locally, it added *Copy/spotless config tags to existing modules. I've reverted those to match main.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17949: Add Azure Blob Storage backup repository module [solr]
Posted by "prateeksinghalgit (via GitHub)" <gi...@apache.org>.
prateeksinghalgit commented on code in PR #3750:
URL: https://github.com/apache/solr/pull/3750#discussion_r2553345413
##########
solr/licenses/reactor-NOTICE.txt:
##########
@@ -0,0 +1,25 @@
+AWS SDK for Java 2.0
+Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved.
+
+This product includes software developed by
+Amazon Technologies, Inc (http://www.amazon.com/).
Review Comment:
updated it.
##########
solr/licenses/msal4j-NOTICE.txt:
##########
@@ -0,0 +1,25 @@
+AWS SDK for Java 2.0
+Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved.
+
+This product includes software developed by
+Amazon Technologies, Inc (http://www.amazon.com/).
+
+**********************
+THIRD PARTY COMPONENTS
+**********************
+This software includes third party software subject to the following copyrights:
+- XML parsing and utility functions from JetS3t - Copyright 2006-2009 James Murty.
+- PKCS#1 PEM encoded private key parsing and utility functions from oauth.googlecode.com - Copyright 1998-2010 AOL Inc.
+- Apache Commons Lang - https://github.com/apache/commons-lang
+- Netty Reactive Streams - https://github.com/playframework/netty-reactive-streams
+- Jackson-core - https://github.com/FasterXML/jackson-core
+- Jackson-dataformat-cbor - https://github.com/FasterXML/jackson-dataformats-binary
+
+The licenses for these third party components are included in LICENSE.txt
+
Review Comment:
updated it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17949: Add Azure Blob Storage backup repository module [solr]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #3750:
URL: https://github.com/apache/solr/pull/3750#discussion_r2544701213
##########
solr/licenses/msal4j-NOTICE.txt:
##########
@@ -0,0 +1,25 @@
+AWS SDK for Java 2.0
+Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved.
+
+This product includes software developed by
+Amazon Technologies, Inc (http://www.amazon.com/).
+
+**********************
+THIRD PARTY COMPONENTS
+**********************
+This software includes third party software subject to the following copyrights:
+- XML parsing and utility functions from JetS3t - Copyright 2006-2009 James Murty.
+- PKCS#1 PEM encoded private key parsing and utility functions from oauth.googlecode.com - Copyright 1998-2010 AOL Inc.
+- Apache Commons Lang - https://github.com/apache/commons-lang
+- Netty Reactive Streams - https://github.com/playframework/netty-reactive-streams
+- Jackson-core - https://github.com/FasterXML/jackson-core
+- Jackson-dataformat-cbor - https://github.com/FasterXML/jackson-dataformats-binary
+
+The licenses for these third party components are included in LICENSE.txt
+
Review Comment:
Incorrect NOTICE file content: This file contains AWS/Amazon copyright information but should contain Microsoft copyright information for the MSAL4J library.
```suggestion
Microsoft Authentication Library for Java (MSAL4J)
Copyright (c) Microsoft Corporation. All rights reserved.
This product includes software developed by
Microsoft Corporation (https://github.com/AzureAD/microsoft-authentication-library-for-java).
Licensed under the MIT License.
**********************
THIRD PARTY COMPONENTS
**********************
This software may include third party software subject to the following copyrights:
```
##########
solr/licenses/azure-NOTICE.txt:
##########
@@ -0,0 +1,25 @@
+AWS SDK for Java 2.0
+Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved.
+
+This product includes software developed by
+Amazon Technologies, Inc (http://www.amazon.com/).
+
+**********************
+THIRD PARTY COMPONENTS
+**********************
+This software includes third party software subject to the following copyrights:
+- XML parsing and utility functions from JetS3t - Copyright 2006-2009 James Murty.
+- PKCS#1 PEM encoded private key parsing and utility functions from oauth.googlecode.com - Copyright 1998-2010 AOL Inc.
Review Comment:
Incorrect NOTICE file content: This file contains AWS/Amazon copyright information but should contain Azure/Microsoft copyright information for the Azure SDK libraries.
```suggestion
Azure SDK for Java
Copyright (c) Microsoft Corporation. All rights reserved.
This product includes software developed by
Microsoft Corporation (https://www.microsoft.com/).
**********************
THIRD PARTY COMPONENTS
**********************
This software includes third party software subject to the following copyrights:
```
##########
solr/licenses/reactor-NOTICE.txt:
##########
@@ -0,0 +1,25 @@
+AWS SDK for Java 2.0
+Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved.
+
+This product includes software developed by
+Amazon Technologies, Inc (http://www.amazon.com/).
Review Comment:
Incorrect NOTICE file content: This file contains AWS/Amazon copyright information but should contain Project Reactor copyright information.
##########
solr/modules/blob-repository/src/java/org/apache/solr/blob/BlobBackupRepository.java:
##########
@@ -0,0 +1,407 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.blob;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStream;
+import java.lang.invoke.MethodHandles;
+import java.net.URI;
+import java.net.URISyntaxException;
+import java.time.Duration;
+import java.time.Instant;
+import java.util.Collection;
+import java.util.Objects;
+import java.util.Set;
+import java.util.stream.Collectors;
+import org.apache.lucene.store.Directory;
+import org.apache.lucene.store.IOContext;
+import org.apache.lucene.store.IndexInput;
+import org.apache.lucene.store.IndexOutput;
+import org.apache.solr.common.SolrException;
+import org.apache.solr.common.util.NamedList;
+import org.apache.solr.common.util.StrUtils;
+import org.apache.solr.core.backup.repository.AbstractBackupRepository;
+import org.apache.solr.core.backup.repository.BackupRepository;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * A concrete implementation of {@link BackupRepository} interface supporting backup/restore of Solr
+ * indexes to Azure Blob Storage.
+ */
+public class BlobBackupRepository extends AbstractBackupRepository {
+
+ private static final Logger log = LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+
+ static final String BLOB_SCHEME = "blob";
+ private static final int CHUNK_SIZE = 16 * 1024 * 1024;
+
+ private BlobStorageClient client;
+
+ @Override
+ public void init(NamedList<?> args) {
+ super.init(args);
+ BlobBackupRepositoryConfig backupConfig = new BlobBackupRepositoryConfig(this.config);
+
+ // If a client was already created, close it to avoid any resource leak
+ if (client != null) {
+ client.close();
+ }
+
+ this.client = backupConfig.buildClient();
+ }
+
+ // Method to inject a mock client for testing
+ public void setClient(BlobStorageClient client) {
+ this.client = client;
+ }
+
+ @Override
+ @SuppressWarnings("unchecked")
+ public <T> T getConfigProperty(String name) {
+ return (T) this.config.get(name);
+ }
+
+ @Override
+ public URI createURI(String location) {
+ if (StrUtils.isNullOrEmpty(location)) {
+ throw new IllegalArgumentException("cannot create URI with an empty location");
+ }
+
+ URI result;
+ try {
+ if (location.startsWith(BLOB_SCHEME + ":")) {
+ result = new URI(location);
+ } else if (location.startsWith("/")) {
+ result = new URI(BLOB_SCHEME, "", location, null);
+ } else {
+ result = new URI(BLOB_SCHEME, "", "/" + location, null);
+ }
+ return result;
+ } catch (URISyntaxException ex) {
+ throw new SolrException(SolrException.ErrorCode.BAD_REQUEST, ex);
+ }
+ }
+
+ @Override
+ public URI createDirectoryURI(String location) {
+ if (StrUtils.isNullOrEmpty(location)) {
+ throw new IllegalArgumentException("cannot create URI with an empty location");
+ }
+
+ if (!location.endsWith("/")) {
+ location += "/";
+ }
+
+ return createURI(location);
+ }
+
+ @Override
+ public URI resolve(URI baseUri, String... pathComponents) {
+ if (!BLOB_SCHEME.equalsIgnoreCase(baseUri.getScheme())) {
+ throw new IllegalArgumentException("URI must begin with 'blob:' scheme");
+ }
+
+ String path = baseUri + "/" + String.join("/", pathComponents);
+ return URI.create(path).normalize();
+ }
+
+ @Override
+ public URI resolveDirectory(URI baseUri, String... pathComponents) {
+ if (pathComponents.length > 0) {
+ if (!pathComponents[pathComponents.length - 1].endsWith("/")) {
+ pathComponents[pathComponents.length - 1] = pathComponents[pathComponents.length - 1] + "/";
+ }
+ } else {
+ if (!baseUri.toString().endsWith("/")) {
+ baseUri = URI.create(baseUri + "/");
+ }
+ }
+ return resolve(baseUri, pathComponents);
+ }
+
+ @Override
+ public void createDirectory(URI path) throws IOException {
+ Objects.requireNonNull(path, "cannot create directory to a null URI");
+
+ String blobPath = getBlobPath(path);
+
+ if (log.isDebugEnabled()) {
+ log.debug("Create directory '{}'", blobPath);
+ }
+
+ try {
+ client.createDirectory(blobPath);
+ } catch (BlobException e) {
+ throw new IOException("Failed to create directory " + blobPath, e);
+ }
+ }
+
+ @Override
+ public void deleteDirectory(URI path) throws IOException {
+ Objects.requireNonNull(path, "cannot delete directory with a null URI");
+
+ String blobPath = getBlobPath(path);
+
+ if (log.isDebugEnabled()) {
+ log.debug("Delete directory '{}'", blobPath);
+ }
+
+ try {
+ client.deleteDirectory(blobPath);
+ } catch (BlobException e) {
+ throw new IOException("Failed to delete directory " + blobPath, e);
+ }
+ }
+
+ @Override
+ public void delete(URI path, Collection<String> files) throws IOException {
+ Objects.requireNonNull(path, "cannot delete with a null URI");
+ Objects.requireNonNull(files, "cannot delete with a null files collection");
+
+ String basePath = getBlobPath(path);
+ // If a file path was passed instead of a directory, use its parent directory as base
+ try {
+ if (!client.isDirectory(basePath)) {
+ int lastSlash = basePath.lastIndexOf('/');
+ basePath = lastSlash >= 0 ? basePath.substring(0, lastSlash) : "";
+ }
+ } catch (BlobException e) {
+ throw new IOException("Failed to check path type for " + basePath, e);
+ }
+
+ final String baseForPaths = basePath;
+ Set<String> fullPaths =
+ files.stream()
+ .map(file -> (baseForPaths.isEmpty() ? file : baseForPaths + "/" + file))
+ .collect(Collectors.toSet());
+
+ if (log.isDebugEnabled()) {
+ log.debug("Delete files '{}'", fullPaths);
+ }
+
+ try {
+ client.delete(fullPaths);
+ } catch (BlobException e) {
+ throw new IOException("Failed to delete files " + fullPaths, e);
+ }
+ }
+
+ @Override
+ public boolean exists(URI path) throws IOException {
+ Objects.requireNonNull(path, "cannot check existence with a null URI");
+
+ String blobPath = getBlobPath(path);
+
+ if (log.isDebugEnabled()) {
+ log.debug("Check existence '{}'", blobPath);
+ }
+
+ try {
+ return client.pathExists(blobPath);
+ } catch (BlobException e) {
+ throw new IOException("Failed to check existence of " + blobPath, e);
+ }
+ }
+
+ @Override
+ public PathType getPathType(URI path) throws IOException {
+ Objects.requireNonNull(path, "cannot get path type with a null URI");
+
+ String blobPath = getBlobPath(path);
+
+ if (log.isDebugEnabled()) {
+ log.debug("Get path type '{}'", blobPath);
+ }
+
+ try {
+ if (client.isDirectory(blobPath)) {
+ return BackupRepository.PathType.DIRECTORY;
+ } else {
+ return BackupRepository.PathType.FILE;
+ }
+ } catch (BlobException e) {
+ throw new IOException("Failed to get path type for " + blobPath, e);
+ }
+ }
+
+ @Override
+ public String[] listAll(URI path) throws IOException {
+ Objects.requireNonNull(path, "cannot list with a null URI");
+
+ String blobPath = getBlobPath(path);
+
+ if (log.isDebugEnabled()) {
+ log.debug("List all '{}'", blobPath);
+ }
+
+ try {
+ return client.listDir(blobPath);
+ } catch (BlobException e) {
+ throw new IOException("Failed to list directory " + blobPath, e);
+ }
+ }
+
+ @Override
+ public IndexInput openInput(URI dirPath, String fileName, IOContext ctx) throws IOException {
+ Objects.requireNonNull(dirPath, "cannot open input with a null URI");
+ Objects.requireNonNull(fileName, "cannot open input with a null fileName");
+
+ String base = getBlobPath(dirPath);
+ String blobPath = base.endsWith("/") ? base + fileName : base + "/" + fileName;
+
+ if (log.isDebugEnabled()) {
+ log.debug("Open input '{}'", blobPath);
+ }
+
+ try {
+ return new BlobIndexInput(blobPath, client, client.length(blobPath));
+ } catch (BlobException e) {
+ throw new IOException("Failed to open input stream for " + blobPath, e);
+ }
+ }
+
+ @Override
+ public OutputStream createOutput(URI path) throws IOException {
+ Objects.requireNonNull(path, "cannot create output with a null URI");
+
+ String blobPath = getBlobPath(path);
+
+ if (log.isDebugEnabled()) {
+ log.debug("Create output '{}'", blobPath);
+ }
+
+ try {
+ return client.pushStream(blobPath);
+ } catch (BlobException e) {
+ throw new IOException("Failed to create output stream for " + blobPath, e);
+ }
+ }
+
+ @Override
+ public void copyIndexFileFrom(
+ Directory sourceDir, String sourceFileName, URI dest, String destFileName)
+ throws IOException {
+ Objects.requireNonNull(sourceDir, "cannot copy with a null sourceDir");
+ Objects.requireNonNull(sourceFileName, "cannot copy with a null sourceFileName");
+ Objects.requireNonNull(dest, "cannot copy with a null dest");
+
+ String destPath = getBlobPath(dest);
+
+ String blobPath = destPath.endsWith("/") ? destPath + destFileName : destPath;
+
+ if (log.isDebugEnabled()) {
+ log.debug("Copy index file from '{}' to '{}'", sourceFileName, blobPath);
+ }
+
+ // Ensure destination parent directory exists
+ String parentDir =
+ blobPath.contains("/") ? blobPath.substring(0, blobPath.lastIndexOf('/') + 1) : "";
+ try {
+ if (!parentDir.isEmpty()) {
+ client.createDirectory(parentDir);
+ }
+ } catch (BlobException e) {
+ // ignore failures here; write will surface real issues
+ }
+
+ try (IndexInput input = sourceDir.openInput(sourceFileName, IOContext.DEFAULT);
+ OutputStream output = client.pushStream(blobPath)) {
+ // Copy bytes from IndexInput to OutputStream
+ byte[] buffer = new byte[8192];
+ long remaining = input.length();
+ while (remaining > 0) {
+ int toRead = (int) Math.min(buffer.length, remaining);
+ input.readBytes(buffer, 0, toRead);
+ output.write(buffer, 0, toRead);
+ remaining -= toRead;
+ }
+ } catch (BlobException e) {
+ throw new IOException("Failed to copy file from " + sourceFileName + " to " + blobPath, e);
+ }
+ }
+
+ /**
+ * Copy an index file from specified <code>sourceRepo</code> to the destination directory (i.e.
+ * restore).
+ *
+ * @param sourceDir The source URI hosting the file to be copied.
+ * @param dest The destination where the file should be copied.
+ * @throws IOException in case of errors.
+ */
+ @Override
+ public void copyIndexFileTo(
+ URI sourceDir, String sourceFileName, Directory dest, String destFileName)
+ throws IOException {
+ if (StrUtils.isNullOrEmpty(sourceFileName)) {
+ throw new IllegalArgumentException("must have a valid source file name to copy");
+ }
+ if (StrUtils.isNullOrEmpty(destFileName)) {
+ throw new IllegalArgumentException("must have a valid destination file name to copy");
+ }
+
+ String basePath = getBlobPath(sourceDir);
+ String blobPath;
+ // If sourceDir already points to the file, avoid duplicating the name
+ if (basePath.endsWith("/" + sourceFileName)
+ || basePath.equals(sourceFileName)
+ || basePath.equals("/" + sourceFileName)) {
+ blobPath = basePath;
+ } else {
+ URI filePath = resolve(sourceDir, sourceFileName);
+ blobPath = getBlobPath(filePath);
+ }
+
+ Instant start = Instant.now();
+ if (log.isDebugEnabled()) {
+ log.debug("Download started from blob '{}'", blobPath);
+ }
+
+ try (InputStream inputStream = client.pullStream(blobPath);
+ IndexOutput indexOutput = dest.createOutput(destFileName, IOContext.DEFAULT)) {
+ // Copy bytes from InputStream to IndexOutput
+ byte[] buffer = new byte[CHUNK_SIZE];
+ int len;
+ while ((len = inputStream.read(buffer)) != -1) {
+ indexOutput.writeBytes(buffer, 0, len);
+ }
+ } catch (BlobException e) {
+ throw new IOException("Failed to copy file from " + blobPath + " to " + destFileName, e);
+ }
+
+ long timeElapsed = Duration.between(start, Instant.now()).toMillis();
+
+ if (log.isInfoEnabled()) {
+ log.info("Download from S3 '{}' finished in {}ms", blobPath, timeElapsed);
Review Comment:
Copy-paste error: log message refers to 'S3' but should refer to 'Azure Blob Storage' or 'blob storage'.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17949: Add Azure Blob Storage backup repository module [solr]
Posted by "prateeksinghalgit (via GitHub)" <gi...@apache.org>.
prateeksinghalgit commented on code in PR #3750:
URL: https://github.com/apache/solr/pull/3750#discussion_r2555407396
##########
gradle/libs.versions.toml:
##########
@@ -51,6 +51,10 @@ aqute-bnd = "6.4.1"
asciidoctor-mathjax = "0.0.9"
# @keep Asciidoctor tabs version used in ref-guide
asciidoctor-tabs = "1.0.0-beta.6"
+azure-storage = "12.25.0"
+azure-identity = "1.12.0"
+azure-core = "1.52.0"
+azure-core-http-netty = "1.15.4"
Review Comment:
Yes, made the change to use azure-core-http-okhttp. I was having issue with using azure-core-http-jdk-httpclient.
Let me know if using azure-core-http-okhttp would be fine?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17949: Add Azure Blob Storage backup repository module [solr]
Posted by "prateeksinghalgit (via GitHub)" <gi...@apache.org>.
prateeksinghalgit commented on code in PR #3750:
URL: https://github.com/apache/solr/pull/3750#discussion_r3232147739
##########
solr/modules/azure-blob-repository/src/java/org/apache/solr/azureblob/AzureBlobStorageClient.java:
##########
@@ -0,0 +1,529 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.azureblob;
+
+import com.azure.core.credential.TokenCredential;
+import com.azure.identity.DefaultAzureCredentialBuilder;
+import com.azure.storage.blob.BlobClient;
+import com.azure.storage.blob.BlobContainerClient;
+import com.azure.storage.blob.BlobServiceClient;
+import com.azure.storage.blob.BlobServiceClientBuilder;
+import com.azure.storage.blob.models.BlobItem;
+import com.azure.storage.blob.models.BlobStorageException;
+import com.azure.storage.blob.models.ListBlobsOptions;
+import com.google.common.annotations.VisibleForTesting;
+import java.io.ByteArrayInputStream;
+import java.io.FilterInputStream;
+import java.io.InputStream;
+import java.io.OutputStream;
+import java.lang.invoke.MethodHandles;
+import java.util.Collection;
+import java.util.HashSet;
+import java.util.Locale;
+import java.util.Set;
+import java.util.stream.Collectors;
+import org.apache.solr.common.util.ResumableInputStream;
+import org.apache.solr.common.util.StrUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Creates a {@link BlobServiceClient} for communicating with Azure Blob Storage. Utilizes the
+ * default Azure credential provider chain.
+ */
+public class AzureBlobStorageClient {
+
+ private static final Logger log = LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+
+ static final String BLOB_FILE_PATH_DELIMITER = "/";
+ private static final int HTTP_NOT_FOUND = 404;
+ private static final int HTTP_CONFLICT = 409;
+ private static final int SKIP_BUFFER_SIZE = 8192;
+ private static final int DELETE_BATCH_SIZE = 1000;
+
+ private static final com.azure.core.http.HttpClient SHARED_HTTP_CLIENT =
+ new com.azure.core.http.okhttp.OkHttpAsyncHttpClientBuilder().build();
+
+ private final BlobContainerClient containerClient;
+
+ AzureBlobStorageClient(
+ String containerName,
+ String connectionString,
+ String endpoint,
+ String accountName,
+ String accountKey,
+ String sasToken,
+ String tenantId,
+ String clientId,
+ String clientSecret) {
+ this(
+ createInternalClient(
+ connectionString,
+ endpoint,
+ accountName,
+ accountKey,
+ sasToken,
+ tenantId,
+ clientId,
+ clientSecret),
+ containerName);
+ }
+
+ @VisibleForTesting
+ AzureBlobStorageClient(BlobServiceClient blobServiceClient, String containerName) {
+ this.containerClient = blobServiceClient.getBlobContainerClient(containerName);
+ try {
+ containerClient.create();
+ } catch (BlobStorageException e) {
+ if (e.getStatusCode() != HTTP_CONFLICT) {
+ throw e;
+ }
+ }
+ }
+
+ private static BlobServiceClient createInternalClient(
+ String connectionString,
+ String endpoint,
+ String accountName,
+ String accountKey,
+ String sasToken,
+ String tenantId,
+ String clientId,
+ String clientSecret) {
+
+ BlobServiceClientBuilder builder = new BlobServiceClientBuilder();
+ builder.httpClient(SHARED_HTTP_CLIENT);
+
+ if (StrUtils.isNotNullOrEmpty(connectionString)) {
+ builder.connectionString(connectionString);
+ } else if (StrUtils.isNotNullOrEmpty(endpoint)) {
+ builder.endpoint(endpoint);
+ if (StrUtils.isNotNullOrEmpty(accountName) && StrUtils.isNotNullOrEmpty(accountKey)) {
+ builder.credential(
+ new com.azure.storage.common.StorageSharedKeyCredential(accountName, accountKey));
+ } else if (StrUtils.isNotNullOrEmpty(sasToken)) {
+ builder.sasToken(sasToken);
+ } else {
+ TokenCredential credential = new DefaultAzureCredentialBuilder().tenantId(tenantId).build();
+ builder.credential(credential);
+ }
+ } else {
+ throw new IllegalArgumentException("Either connectionString or endpoint must be provided");
+ }
+
+ return builder.buildClient();
+ }
+
+ void createDirectory(String path) throws AzureBlobException {
+ String sanitizedDirPath = sanitizedDirPath(path);
+
+ if (!pathExists(sanitizedDirPath)) {
+ String parent = getParentDirectory(sanitizedDirPath);
+ if (!parent.isEmpty() && !parent.equals(BLOB_FILE_PATH_DELIMITER)) {
+ createDirectory(parent);
+ }
+
+ try {
+ BlobClient blobClient = containerClient.getBlobClient(sanitizedDirPath);
+ blobClient.upload(new ByteArrayInputStream(new byte[0]), 0, true);
+ java.util.Map<String, String> metadata = new java.util.HashMap<>();
+ metadata.put("hdi_isfolder", "true");
+ blobClient.setMetadata(metadata);
+ } catch (BlobStorageException e) {
+ throw handleBlobException(e);
+ }
+ }
+ }
+
+ void delete(Collection<String> paths) throws AzureBlobException {
+ Set<String> entries = new HashSet<>();
+ for (String path : paths) {
+ entries.add(sanitizedFilePath(path));
+ }
+ deleteBlobs(entries);
+ }
+
+ void deleteDirectory(String path) throws AzureBlobException {
+ path = sanitizedDirPath(path);
+
+ Set<String> entries = listAll(path);
+ if (pathExists(path)) {
+ entries.add(path);
+ }
+
+ deleteBlobs(entries);
+ }
+
+ String[] listDir(String path) throws AzureBlobException {
+ path = sanitizedDirPath(path);
+
+ try {
+ ListBlobsOptions options = new ListBlobsOptions().setPrefix(path).setMaxResultsPerPage(1000);
+
+ final String finalPath = path;
+ return containerClient.listBlobs(options, null).stream()
+ .map(BlobItem::getName)
+ .filter(s -> s.startsWith(finalPath))
+ .map(s -> s.substring(finalPath.length()))
+ .filter(s -> !s.isEmpty())
+ .filter(
+ s -> {
+ int slashIndex = s.indexOf(BLOB_FILE_PATH_DELIMITER);
+ return slashIndex == -1 || slashIndex == s.length() - 1;
+ })
+ .toArray(String[]::new);
+ } catch (BlobStorageException e) {
+ throw handleBlobException(e);
+ }
+ }
+
+ boolean pathExists(String path) throws AzureBlobException {
+ final String blobPath = sanitizedPath(path);
+
+ if (blobPath.isEmpty() || BLOB_FILE_PATH_DELIMITER.equals(blobPath)) {
+ return true;
+ }
+
+ try {
+ BlobClient blobClient = containerClient.getBlobClient(blobPath);
+ return blobClient.exists();
+ } catch (BlobStorageException e) {
+ throw handleBlobException(e);
+ }
+ }
+
+ boolean isDirectory(String path) throws AzureBlobException {
+ final String dirPrefix = sanitizedDirPath(path);
+
+ try {
+ ListBlobsOptions options =
+ new ListBlobsOptions().setPrefix(dirPrefix).setMaxResultsPerPage(1);
+ if (containerClient.listBlobs(options, null).iterator().hasNext()) {
+ return true;
+ }
+
+ BlobClient markerClient = containerClient.getBlobClient(dirPrefix);
+ if (markerClient.exists()) {
+ long size = markerClient.getProperties().getBlobSize();
+ if (size == 0) {
+ return true;
+ }
+ java.util.Map<String, String> md = markerClient.getProperties().getMetadata();
+ return md != null && md.containsKey("hdi_isfolder");
+ }
+
+ return false;
+ } catch (BlobStorageException e) {
+ throw handleBlobException(e);
+ }
+ }
+
+ long length(String path) throws AzureBlobException {
+ String blobPath = sanitizedFilePath(path);
+ try {
+ BlobClient blobClient = containerClient.getBlobClient(blobPath);
+ return blobClient.getProperties().getBlobSize();
+ } catch (BlobStorageException e) {
+ throw handleBlobException(e);
+ }
+ }
+
+ InputStream pullStream(String path) throws AzureBlobException {
+ final String blobPath = sanitizedFilePath(path);
+
+ try {
+ BlobClient blobClient = containerClient.getBlobClient(blobPath);
+ final long contentLength = blobClient.getProperties().getBlobSize();
+
+ if (contentLength == 0) {
+ return new ByteArrayInputStream(new byte[0]);
+ }
+
+ InputStream initial = new IdempotentCloseInputStream(blobClient.openInputStream());
+
+ return new ResumableInputStream(
+ initial,
+ bytesRead -> {
+ if (contentLength > 0 && bytesRead >= contentLength) {
+ return null;
+ }
+ try {
+ long remaining =
+ contentLength > 0 ? Math.max(0, contentLength - bytesRead) : Long.MAX_VALUE;
+ return pullRangeStream(path, bytesRead, remaining);
+ } catch (AzureBlobException e) {
+ throw new RuntimeException(e);
+ }
+ });
+ } catch (BlobStorageException e) {
+ throw handleBlobException(e);
+ }
+ }
+
+ InputStream pullRangeStream(String path, long offset, long length) throws AzureBlobException {
+ final String blobPath = sanitizedFilePath(path);
+ try {
+ BlobClient blobClient = containerClient.getBlobClient(blobPath);
+ com.azure.storage.blob.models.BlobRange range =
+ new com.azure.storage.blob.models.BlobRange(offset, length);
+ return new IdempotentCloseInputStream(blobClient.openInputStream(range, null));
+ } catch (BlobStorageException e) {
+ throw handleBlobException(e);
+ }
+ }
+
+ private static final class IdempotentCloseInputStream extends FilterInputStream {
+ private boolean closed;
+
+ IdempotentCloseInputStream(InputStream in) {
+ super(in);
+ this.closed = false;
+ }
+
+ @Override
+ public int read() throws java.io.IOException {
+ if (closed) {
+ throw new java.io.IOException("Stream is already closed");
+ }
+ try {
+ return super.read();
+ } catch (RuntimeException re) {
+ if (isAlreadyClosed(re)) {
+ throw new java.io.IOException("Stream is already closed", re);
+ }
+ throw re;
+ }
+ }
+
+ @Override
+ public int read(byte[] b, int off, int len) throws java.io.IOException {
+ if (closed) {
+ throw new java.io.IOException("Stream is already closed");
+ }
+ try {
+ return super.read(b, off, len);
+ } catch (RuntimeException re) {
+ if (isAlreadyClosed(re)) {
+ throw new java.io.IOException("Stream is already closed", re);
+ }
+ throw re;
+ }
+ }
+
+ @Override
+ public void close() throws java.io.IOException {
+ if (closed) {
+ return;
+ }
+ try {
+ super.close();
+ } catch (java.io.IOException e) {
+ String msg = e.getMessage();
+ if (msg == null || !msg.toLowerCase(java.util.Locale.ROOT).contains("already closed")) {
+ throw e;
+ }
+ // swallow "already closed" to make close idempotent
+ } finally {
+ closed = true;
+ }
+ }
+
+ @Override
+ public long skip(long n) throws java.io.IOException {
+ if (closed) {
+ throw new java.io.IOException("Stream is already closed");
+ }
+ if (n <= 0) {
+ return 0L;
+ }
+ long remaining = n;
+ byte[] discard = new byte[SKIP_BUFFER_SIZE];
+ try {
+ while (remaining > 0) {
+ int toRead = (int) Math.min(discard.length, remaining);
+ int read = super.read(discard, 0, toRead);
+ if (read < 0) {
+ break;
+ }
+ remaining -= read;
+ }
+ return n - remaining;
+ } catch (RuntimeException re) {
+ throw new java.io.IOException(re);
+ }
+ }
+
+ private static boolean isAlreadyClosed(Throwable t) {
+ String msg = t.getMessage();
+ return msg != null && msg.toLowerCase(java.util.Locale.ROOT).contains("already closed");
+ }
+ }
+
+ OutputStream pushStream(String path) throws AzureBlobException {
+ path = sanitizedFilePath(path);
+
+ if (!parentDirectoryExist(path)) {
+ String parentDirectory = getParentDirectory(path);
+ if (!parentDirectory.isEmpty() && !parentDirectory.equals(BLOB_FILE_PATH_DELIMITER)) {
+ createDirectory(parentDirectory);
+ }
+ }
+
+ try {
+ BlobClient blobClient = containerClient.getBlobClient(path);
+ return new AzureBlobOutputStream(blobClient, path);
+ } catch (BlobStorageException e) {
+ throw handleBlobException(e);
+ }
+ }
+
+ void close() {}
+
+ @VisibleForTesting
+ void deleteContainerForTests() {
+ try {
+ containerClient.delete();
+ } catch (BlobStorageException e) {
+ if (e.getStatusCode() != HTTP_NOT_FOUND) {
+ throw e;
+ }
+ }
+ }
+
+ private Collection<String> deleteBlobs(Collection<String> paths) throws AzureBlobException {
+ try {
+ return deleteBlobs(paths, DELETE_BATCH_SIZE);
Review Comment:
with the delete updated to batch deletes, this parameter is getting used now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17949: Add Azure Blob Storage backup repository module [solr]
Posted by "prateeksinghalgit (via GitHub)" <gi...@apache.org>.
prateeksinghalgit commented on code in PR #3750:
URL: https://github.com/apache/solr/pull/3750#discussion_r3232111208
##########
solr/modules/azure-blob-repository/src/java/org/apache/solr/azureblob/AzureBlobIndexInput.java:
##########
@@ -0,0 +1,200 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.azureblob;
+
+import java.io.EOFException;
+import java.io.IOException;
+import java.io.InputStream;
+import java.util.LinkedHashMap;
+import java.util.Map;
+import org.apache.lucene.store.IndexInput;
+
+class AzureBlobIndexInput extends IndexInput {
+
+ private static final int MIN_PAGE_SIZE = 4 * 1024;
+ private static final int DEFAULT_PAGE_SIZE = 512 * 1024;
+ private static final int MAX_CACHED_PAGES = 128;
+
+ private final String path;
+ private final AzureBlobStorageClient client;
+ private final long length;
+ private final int pageSize;
+ private final LruPageCache cache;
+
+ private long position = 0L;
+ private boolean closed = false;
+
+ AzureBlobIndexInput(String path, AzureBlobStorageClient client, long length) {
+ this(path, client, length, DEFAULT_PAGE_SIZE, MAX_CACHED_PAGES);
+ }
+
+ AzureBlobIndexInput(
+ String path, AzureBlobStorageClient client, long length, int pageSize, int maxCachedPages) {
+ super(path);
+ this.path = path;
+ this.client = client;
+ this.length = length;
+ this.pageSize = Math.max(MIN_PAGE_SIZE, pageSize);
+ this.cache = new LruPageCache(maxCachedPages);
+ }
+
+ @Override
+ public void close() throws IOException {
+ closed = true;
+ cache.clear();
+ }
+
+ @Override
+ public long getFilePointer() {
+ return position;
+ }
+
+ @Override
+ public void seek(long pos) throws IOException {
+ ensureOpen();
+ if (pos < 0 || pos > length) {
+ throw new IOException("Seek position out of bounds: " + pos);
+ }
+
+ position = pos;
+ }
+
+ @Override
+ public long length() {
+ return length;
+ }
+
+ @Override
+ public IndexInput slice(String sliceDescription, long offset, long length) throws IOException {
+ ensureOpen();
+ if (offset < 0 || length < 0 || offset + length > this.length) {
+ throw new IOException("Slice out of bounds: offset=" + offset + ", length=" + length);
+ }
+
+ AzureBlobIndexInput slice =
+ new AzureBlobIndexInput(
+ getFullSliceDescription(sliceDescription), client, length, pageSize, MAX_CACHED_PAGES);
+
+ slice.position = 0L;
+
+ // Wrap client in a view that remaps range requests by adding base offset
+ slice.clientViewBaseOffset = this.clientViewBaseOffset + offset;
+ return slice;
+ }
+
+ @Override
+ public byte readByte() throws IOException {
+ ensureOpen();
+ if (position >= length) {
+ throw new EOFException("End of stream reached");
+ }
+
+ byte[] page = getPage(pageIndex(position));
+ int inPageOffset = (int) (position % pageSize);
+ byte value = page[inPageOffset];
+ position += 1L;
+ return value;
+ }
+
+ @Override
+ public void readBytes(byte[] b, int offset, int len) throws IOException {
+ ensureOpen();
+ if (len < 0) {
+ throw new IOException("Length must be non-negative");
+ }
+
+ if (position + len > length) {
+ throw new EOFException("End of stream reached");
+ }
+
+ int remaining = len;
+ while (remaining > 0) {
+ long pageIdx = pageIndex(position);
+ byte[] page = getPage(pageIdx);
+ int inPageOffset = (int) (position % pageSize);
+ int toCopy = Math.min(remaining, pageSize - inPageOffset);
+ System.arraycopy(page, inPageOffset, b, offset + (len - remaining), toCopy);
+ position += toCopy;
+ remaining -= toCopy;
+ }
+ }
+
+ // Internal state for slices: base offset to add to all range requests
+ private long clientViewBaseOffset = 0L;
+
+ private byte[] getPage(long pageIdx) throws IOException {
+ byte[] page = cache.get(pageIdx);
+ if (page != null) {
+ return page;
+ }
+
+ long absoluteOffset = clientViewBaseOffset + pageIdx * (long) pageSize;
+ int bytesToRead = (int) Math.min(pageSize, length - pageIdx * (long) pageSize);
+ if (bytesToRead <= 0) {
+ throw new EOFException("End of stream reached");
+ }
+
+ page = new byte[bytesToRead];
+ try (InputStream in = client.pullRangeStream(path, absoluteOffset, bytesToRead)) {
+ int readTotal = 0;
+ while (readTotal < bytesToRead) {
+ int read = in.read(page, readTotal, bytesToRead - readTotal);
+ if (read == -1) break;
+ readTotal += read;
+ }
+
+ if (readTotal < bytesToRead) {
+ throw new EOFException(
+ "End of stream reached: expected " + bytesToRead + " bytes, got " + readTotal);
+ }
+ } catch (AzureBlobException e) {
+ throw new IOException("Failed to fetch range page", e);
+ }
+
+ cache.put(pageIdx, page);
+ return page;
+ }
+
+ private long pageIndex(long pos) {
+ return pos / pageSize;
+ }
+
+ private void ensureOpen() throws IOException {
+ if (closed) {
+ throw new IOException("IndexInput is closed");
+ }
+ }
+
+ private static final class LruPageCache extends LinkedHashMap<Long, byte[]> {
+ private final int maxEntries;
+
+ LruPageCache(int maxEntries) {
+ super(16, 0.75f, true);
+ this.maxEntries = maxEntries;
+ }
+
+ @Override
+ protected boolean removeEldestEntry(Map.Entry<Long, byte[]> eldest) {
+ return size() > maxEntries;
+ }
+
+ @Override
+ public void clear() {
Review Comment:
rewritten AzureBlobIndexInput to extend BufferedIndexInput. Not used anymore
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17949: Add Azure Blob Storage backup repository module [solr]
Posted by "prateeksinghalgit (via GitHub)" <gi...@apache.org>.
prateeksinghalgit commented on PR #3750:
URL: https://github.com/apache/solr/pull/3750#issuecomment-4792604405
@janhoy thank you for the review. I have addressed all review feedback.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17949: Add Azure Blob Storage backup repository module [solr]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #3750:
URL: https://github.com/apache/solr/pull/3750#issuecomment-5486656531
This PR has had no activity for 60 days and is now labeled as stale. Any new activity will remove the stale label. To attract more reviewers, please tag people who might be familiar with the code area and/or notify the dev@solr.apache.org mailing list. To exempt this PR from being marked as stale, make it a draft PR or add the label "exempt-stale". If left unattended, this PR will be closed after another 60 days of inactivity. Thank you for your contribution!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17949: Add Azure Blob Storage backup repository module [solr]
Posted by "elangelo (via GitHub)" <gi...@apache.org>.
elangelo commented on PR #3750:
URL: https://github.com/apache/solr/pull/3750#issuecomment-6165784494
Heads-up from the S3 side: this module copies the directory-marker design of the S3 repository, which turned out to break restores of copied backups. I'm fixing that for S3 in #5128.
The problem:
- `createDirectory()` (and `pushStream()` for missing parents) writes a zero-byte `hdi_isfolder=true` blob for every directory.
- `pathExists()` only checks for that exact blob, so a directory without its marker "doesn't exist". The javadoc already mentions this for azcopy.
- Core relies on `exists()` for directories during restore. Examples are the backup path in `BackupManager.forRestore()`, the configset dir in `BackupManager.uploadConfigDir()`, and the location check in `BackupAPIBase`. A backup that was copied to another container or account without its marker blobs therefore can't be restored. As far as I know, `azcopy` skips these directory stubs unless you pass `--include-directory-stub`.
`isDirectory()` already falls back to a prefix listing; `pathExists()` doesn't. In #5128 I went further and dropped the markers entirely:
- `createDirectory()` is a no-op.
- `exists()` on a directory path and `isDirectory()` are a single prefix listing (max 1 result). That also finds old marker blobs, so existing backups keep working, and `deleteDirectory()` cleans them up.
- Writes no longer check or create parent directories, which saves a request or more per uploaded file.
- To run the shared contract tests without empty directories, I added a `supportsEmptyDirectories()` hook to `AbstractBackupRepositoryTest`.
It may be worth doing the same here, or at least making `pathExists()` fall back to a prefix listing for directory paths. Hierarchical-namespace (ADLS Gen2) accounts have real directories and probably aren't affected.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17949: Add Azure Blob Storage backup repository module [solr]
Posted by "janhoy (via GitHub)" <gi...@apache.org>.
janhoy commented on PR #3750:
URL: https://github.com/apache/solr/pull/3750#issuecomment-3574263302
I'll leave the rest of the review to others more proficient in Azure Blob than me (have never used it). I'd love to test it with a read AzBlob but I'll leave that to someone who already have an active account.
I have a concern that the PR *feels* largely AI generated(?), lacking the care to details that we require for contributions. Have me excused @prateeksinghalgit if this is not correct, it was just a hunch I got while reviewing. I'd rather have a short well thought out README with useful advice for *users* than three pages of detailed step by step instructions for testing and developing the feature.
For other reviewers to pick up where I left, consider in partifular the HTTP client choice, correct licensing and avoiding hard coded ports in tests. I have *not* done a complete review, not read the ref-guide part at all, just commented on things me and Copilot saw immediately.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17949: Add Azure Blob Storage backup repository module [solr]
Posted by "prateeksinghalgit (via GitHub)" <gi...@apache.org>.
prateeksinghalgit commented on code in PR #3750:
URL: https://github.com/apache/solr/pull/3750#discussion_r3469346644
##########
solr/solr-ref-guide/modules/deployment-guide/pages/backup-restore.adoc:
##########
@@ -826,3 +826,124 @@ https://docs.aws.amazon.com/sdkref/latest/guide/settings-global.html[These optio
* Retries
** RetryMode (`LEGACY`, `STANDARD`, `ADAPTIVE`)
** Max Attempts
+
+=== AzureBlobBackupRepository
+
+Stores and retrieves backup files in a Microsoft Azure Blob Storage container.
+
+This is provided via the `azure-blob-repository` xref:configuration-guide:solr-modules.adoc[Solr Module] that needs to be enabled before use.
+
+This plugin supports multiple authentication methods: connection strings, account keys, SAS tokens, and Azure Identity (Managed Identity, Service Principal, Azure CLI).
+For Azure Identity, ensure the identity has the "Storage Blob Data Contributor" role on the storage account.
+
+[source,xml]
+----
+<backup>
+ <repository name="azure_blob" class="org.apache.solr.azureblob.AzureBlobBackupRepository" default="false">
+ <str name="azure.blob.container.name">solr-backup</str>
+ <str name="azure.blob.connection.string">DefaultEndpointsProtocol=https;AccountName=myaccount;AccountKey=mykey;EndpointSuffix=core.windows.net</str>
+ </repository>
+</backup>
+----
+
+AzureBlobBackupRepository accepts the following options for configuration:
+
+`azure.blob.container.name`::
++
+[%autowidth,frame=none]
+|===
+|Required |Default: none
+|===
++
+The name of the Azure Blob Storage container. The container must exist before performing backup operations.
+
+`azure.blob.connection.string`::
++
+[%autowidth,frame=none]
+|===
+|Optional |Default: none
+|===
++
+Complete Azure Storage connection string. Mutually exclusive with other authentication methods.
+
+`azure.blob.account.name`::
++
+[%autowidth,frame=none]
+|===
+|Optional |Default: none
+|===
++
+Azure Storage account name. Used with account key or SAS token authentication.
+
+`azure.blob.account.key`::
++
+[%autowidth,frame=none]
+|===
+|Optional |Default: none
+|===
++
+Azure Storage account access key. Mutually exclusive with SAS token and Azure Identity.
+
+`azure.blob.sas.token`::
++
+[%autowidth,frame=none]
+|===
+|Optional |Default: none
+|===
++
+SAS token for time-limited access. Must include `srt=sco` and `sp=rwdlac` permissions.
+The `&` characters must be XML-escaped as `&`.
+
+`azure.blob.endpoint`::
++
+[%autowidth,frame=none]
+|===
+|Optional |Default: none
+|===
++
+Azure Blob Storage endpoint URL (e.g., `https://myaccount.blob.core.windows.net`).
+Required for Azure Identity authentication.
+
+`azure.blob.tenant.id`::
++
+[%autowidth,frame=none]
+|===
+|Optional |Default: none
+|===
++
+Azure AD tenant ID for Service Principal authentication.
+
+`azure.blob.client.id`::
++
+[%autowidth,frame=none]
+|===
+|Optional |Default: none
+|===
++
+Azure AD application (client) ID for Service Principal authentication.
+
+`azure.blob.client.secret`::
++
+[%autowidth,frame=none]
+|===
+|Optional |Default: none
+|===
++
+Azure AD application secret for Service Principal authentication.
+
+`location`::
++
+[%autowidth,frame=none]
+|===
+|Optional |Default: none
+|===
++
+Default path prefix within the container for backup storage.
+
+The target container must already exist; it is not created automatically.
+
+==== Known Limitation: Azure Identity and the Security Manager
+
+Azure Identity authentication (Managed Identity, Service Principal, and `DefaultAzureCredential`) does not work when Solr is started with the Java Security Manager enabled, which is the default (`SOLR_SECURITY_MANAGER_ENABLED=true`).
+To use Azure Identity, set `SOLR_SECURITY_MANAGER_ENABLED=false` before starting Solr.
+The Connection String, Account Key, and SAS Token authentication methods are unaffected and work with the Security Manager enabled.
Review Comment:
Looked into it, there is.
The only SM denials come from the developer credentials that spawn a subprocess — Azure CLI and Azure PowerShell — e.g. access denied ("java.io.FilePermission" "/bin/sh" "execute").
For the Azure CLI credential specifically, I iterated to the minimal working grant and confirmed the backup then completed and the files landed in the container:
permission java.io.FilePermission "/bin/sh", "execute";
permission java.io.FilePermission "/dev/null", "read,write";
I've rewritten the ref-guide section accordingly: Managed Identity / Service Principal work with the default policy (no change needed); the CLI/PowerShell credentials need the grant above (paths are platform-specific) and are really only for local dev; otherwise prefer Managed Identity / Service Principal or set SOLR_SECURITY_MANAGER_ENABLED=false.
I deliberately did not add this grant to the shipped security.policy, to keep the default minimal.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17949: Add Azure Blob Storage backup repository module [solr]
Posted by "prateeksinghalgit (via GitHub)" <gi...@apache.org>.
prateeksinghalgit commented on code in PR #3750:
URL: https://github.com/apache/solr/pull/3750#discussion_r3359089044
##########
solr/modules/azure-blob-repository/src/java/org/apache/solr/azureblob/AzureBlobStorageClient.java:
##########
@@ -0,0 +1,529 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.azureblob;
+
+import com.azure.core.credential.TokenCredential;
+import com.azure.identity.DefaultAzureCredentialBuilder;
+import com.azure.storage.blob.BlobClient;
+import com.azure.storage.blob.BlobContainerClient;
+import com.azure.storage.blob.BlobServiceClient;
+import com.azure.storage.blob.BlobServiceClientBuilder;
+import com.azure.storage.blob.models.BlobItem;
+import com.azure.storage.blob.models.BlobStorageException;
+import com.azure.storage.blob.models.ListBlobsOptions;
+import com.google.common.annotations.VisibleForTesting;
+import java.io.ByteArrayInputStream;
+import java.io.FilterInputStream;
+import java.io.InputStream;
+import java.io.OutputStream;
+import java.lang.invoke.MethodHandles;
+import java.util.Collection;
+import java.util.HashSet;
+import java.util.Locale;
+import java.util.Set;
+import java.util.stream.Collectors;
+import org.apache.solr.common.util.ResumableInputStream;
+import org.apache.solr.common.util.StrUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Creates a {@link BlobServiceClient} for communicating with Azure Blob Storage. Utilizes the
+ * default Azure credential provider chain.
+ */
+public class AzureBlobStorageClient {
+
+ private static final Logger log = LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+
+ static final String BLOB_FILE_PATH_DELIMITER = "/";
+ private static final int HTTP_NOT_FOUND = 404;
+ private static final int HTTP_CONFLICT = 409;
+ private static final int SKIP_BUFFER_SIZE = 8192;
+ private static final int DELETE_BATCH_SIZE = 1000;
+
+ private static final com.azure.core.http.HttpClient SHARED_HTTP_CLIENT =
+ new com.azure.core.http.okhttp.OkHttpAsyncHttpClientBuilder().build();
+
+ private final BlobContainerClient containerClient;
+
+ AzureBlobStorageClient(
+ String containerName,
+ String connectionString,
+ String endpoint,
+ String accountName,
+ String accountKey,
+ String sasToken,
+ String tenantId,
+ String clientId,
+ String clientSecret) {
+ this(
+ createInternalClient(
+ connectionString,
+ endpoint,
+ accountName,
+ accountKey,
+ sasToken,
+ tenantId,
+ clientId,
+ clientSecret),
+ containerName);
+ }
+
+ @VisibleForTesting
+ AzureBlobStorageClient(BlobServiceClient blobServiceClient, String containerName) {
+ this.containerClient = blobServiceClient.getBlobContainerClient(containerName);
+ try {
+ containerClient.create();
+ } catch (BlobStorageException e) {
+ if (e.getStatusCode() != HTTP_CONFLICT) {
+ throw e;
+ }
+ }
+ }
+
+ private static BlobServiceClient createInternalClient(
+ String connectionString,
+ String endpoint,
+ String accountName,
+ String accountKey,
+ String sasToken,
+ String tenantId,
+ String clientId,
+ String clientSecret) {
+
+ BlobServiceClientBuilder builder = new BlobServiceClientBuilder();
+ builder.httpClient(SHARED_HTTP_CLIENT);
+
+ if (StrUtils.isNotNullOrEmpty(connectionString)) {
+ builder.connectionString(connectionString);
+ } else if (StrUtils.isNotNullOrEmpty(endpoint)) {
+ builder.endpoint(endpoint);
+ if (StrUtils.isNotNullOrEmpty(accountName) && StrUtils.isNotNullOrEmpty(accountKey)) {
+ builder.credential(
+ new com.azure.storage.common.StorageSharedKeyCredential(accountName, accountKey));
+ } else if (StrUtils.isNotNullOrEmpty(sasToken)) {
+ builder.sasToken(sasToken);
+ } else {
+ TokenCredential credential = new DefaultAzureCredentialBuilder().tenantId(tenantId).build();
+ builder.credential(credential);
+ }
+ } else {
+ throw new IllegalArgumentException("Either connectionString or endpoint must be provided");
+ }
+
+ return builder.buildClient();
+ }
+
+ void createDirectory(String path) throws AzureBlobException {
+ String sanitizedDirPath = sanitizedDirPath(path);
+
+ if (!pathExists(sanitizedDirPath)) {
+ String parent = getParentDirectory(sanitizedDirPath);
+ if (!parent.isEmpty() && !parent.equals(BLOB_FILE_PATH_DELIMITER)) {
+ createDirectory(parent);
+ }
+
+ try {
+ BlobClient blobClient = containerClient.getBlobClient(sanitizedDirPath);
+ blobClient.upload(new ByteArrayInputStream(new byte[0]), 0, true);
+ java.util.Map<String, String> metadata = new java.util.HashMap<>();
+ metadata.put("hdi_isfolder", "true");
+ blobClient.setMetadata(metadata);
+ } catch (BlobStorageException e) {
+ throw handleBlobException(e);
+ }
+ }
+ }
+
+ void delete(Collection<String> paths) throws AzureBlobException {
+ Set<String> entries = new HashSet<>();
+ for (String path : paths) {
+ entries.add(sanitizedFilePath(path));
+ }
+ deleteBlobs(entries);
+ }
+
+ void deleteDirectory(String path) throws AzureBlobException {
+ path = sanitizedDirPath(path);
+
+ Set<String> entries = listAll(path);
+ if (pathExists(path)) {
+ entries.add(path);
+ }
+
+ deleteBlobs(entries);
+ }
+
+ String[] listDir(String path) throws AzureBlobException {
+ path = sanitizedDirPath(path);
+
+ try {
+ ListBlobsOptions options = new ListBlobsOptions().setPrefix(path).setMaxResultsPerPage(1000);
+
+ final String finalPath = path;
+ return containerClient.listBlobs(options, null).stream()
+ .map(BlobItem::getName)
+ .filter(s -> s.startsWith(finalPath))
+ .map(s -> s.substring(finalPath.length()))
+ .filter(s -> !s.isEmpty())
+ .filter(
+ s -> {
+ int slashIndex = s.indexOf(BLOB_FILE_PATH_DELIMITER);
+ return slashIndex == -1 || slashIndex == s.length() - 1;
+ })
+ .toArray(String[]::new);
+ } catch (BlobStorageException e) {
+ throw handleBlobException(e);
+ }
+ }
+
+ boolean pathExists(String path) throws AzureBlobException {
+ final String blobPath = sanitizedPath(path);
+
+ if (blobPath.isEmpty() || BLOB_FILE_PATH_DELIMITER.equals(blobPath)) {
+ return true;
+ }
+
+ try {
+ BlobClient blobClient = containerClient.getBlobClient(blobPath);
+ return blobClient.exists();
+ } catch (BlobStorageException e) {
+ throw handleBlobException(e);
+ }
+ }
+
+ boolean isDirectory(String path) throws AzureBlobException {
+ final String dirPrefix = sanitizedDirPath(path);
+
+ try {
+ ListBlobsOptions options =
+ new ListBlobsOptions().setPrefix(dirPrefix).setMaxResultsPerPage(1);
+ if (containerClient.listBlobs(options, null).iterator().hasNext()) {
+ return true;
+ }
+
+ BlobClient markerClient = containerClient.getBlobClient(dirPrefix);
+ if (markerClient.exists()) {
+ long size = markerClient.getProperties().getBlobSize();
+ if (size == 0) {
+ return true;
+ }
+ java.util.Map<String, String> md = markerClient.getProperties().getMetadata();
+ return md != null && md.containsKey("hdi_isfolder");
+ }
+
+ return false;
+ } catch (BlobStorageException e) {
+ throw handleBlobException(e);
+ }
+ }
+
+ long length(String path) throws AzureBlobException {
+ String blobPath = sanitizedFilePath(path);
+ try {
+ BlobClient blobClient = containerClient.getBlobClient(blobPath);
+ return blobClient.getProperties().getBlobSize();
+ } catch (BlobStorageException e) {
+ throw handleBlobException(e);
+ }
+ }
+
+ InputStream pullStream(String path) throws AzureBlobException {
+ final String blobPath = sanitizedFilePath(path);
+
+ try {
+ BlobClient blobClient = containerClient.getBlobClient(blobPath);
+ final long contentLength = blobClient.getProperties().getBlobSize();
+
+ if (contentLength == 0) {
+ return new ByteArrayInputStream(new byte[0]);
+ }
+
+ InputStream initial = new IdempotentCloseInputStream(blobClient.openInputStream());
+
+ return new ResumableInputStream(
+ initial,
+ bytesRead -> {
+ if (contentLength > 0 && bytesRead >= contentLength) {
+ return null;
+ }
+ try {
+ long remaining =
+ contentLength > 0 ? Math.max(0, contentLength - bytesRead) : Long.MAX_VALUE;
+ return pullRangeStream(path, bytesRead, remaining);
+ } catch (AzureBlobException e) {
+ throw new RuntimeException(e);
+ }
+ });
+ } catch (BlobStorageException e) {
+ throw handleBlobException(e);
+ }
+ }
+
+ InputStream pullRangeStream(String path, long offset, long length) throws AzureBlobException {
+ final String blobPath = sanitizedFilePath(path);
+ try {
+ BlobClient blobClient = containerClient.getBlobClient(blobPath);
+ com.azure.storage.blob.models.BlobRange range =
+ new com.azure.storage.blob.models.BlobRange(offset, length);
+ return new IdempotentCloseInputStream(blobClient.openInputStream(range, null));
+ } catch (BlobStorageException e) {
+ throw handleBlobException(e);
+ }
+ }
+
+ private static final class IdempotentCloseInputStream extends FilterInputStream {
+ private boolean closed;
+
+ IdempotentCloseInputStream(InputStream in) {
+ super(in);
+ this.closed = false;
+ }
+
+ @Override
+ public int read() throws java.io.IOException {
+ if (closed) {
+ throw new java.io.IOException("Stream is already closed");
+ }
+ try {
+ return super.read();
+ } catch (RuntimeException re) {
+ if (isAlreadyClosed(re)) {
+ throw new java.io.IOException("Stream is already closed", re);
+ }
+ throw re;
+ }
+ }
+
+ @Override
+ public int read(byte[] b, int off, int len) throws java.io.IOException {
+ if (closed) {
+ throw new java.io.IOException("Stream is already closed");
+ }
+ try {
+ return super.read(b, off, len);
+ } catch (RuntimeException re) {
+ if (isAlreadyClosed(re)) {
+ throw new java.io.IOException("Stream is already closed", re);
+ }
+ throw re;
+ }
+ }
+
+ @Override
+ public void close() throws java.io.IOException {
+ if (closed) {
+ return;
+ }
+ try {
+ super.close();
+ } catch (java.io.IOException e) {
+ String msg = e.getMessage();
+ if (msg == null || !msg.toLowerCase(java.util.Locale.ROOT).contains("already closed")) {
+ throw e;
+ }
+ // swallow "already closed" to make close idempotent
+ } finally {
+ closed = true;
+ }
+ }
+
+ @Override
+ public long skip(long n) throws java.io.IOException {
+ if (closed) {
+ throw new java.io.IOException("Stream is already closed");
+ }
+ if (n <= 0) {
+ return 0L;
+ }
+ long remaining = n;
+ byte[] discard = new byte[SKIP_BUFFER_SIZE];
+ try {
+ while (remaining > 0) {
+ int toRead = (int) Math.min(discard.length, remaining);
+ int read = super.read(discard, 0, toRead);
+ if (read < 0) {
+ break;
+ }
+ remaining -= read;
+ }
+ return n - remaining;
+ } catch (RuntimeException re) {
+ throw new java.io.IOException(re);
+ }
+ }
+
+ private static boolean isAlreadyClosed(Throwable t) {
+ String msg = t.getMessage();
+ return msg != null && msg.toLowerCase(java.util.Locale.ROOT).contains("already closed");
+ }
+ }
+
+ OutputStream pushStream(String path) throws AzureBlobException {
+ path = sanitizedFilePath(path);
+
+ if (!parentDirectoryExist(path)) {
+ String parentDirectory = getParentDirectory(path);
+ if (!parentDirectory.isEmpty() && !parentDirectory.equals(BLOB_FILE_PATH_DELIMITER)) {
+ createDirectory(parentDirectory);
+ }
+ }
+
+ try {
+ BlobClient blobClient = containerClient.getBlobClient(path);
+ return new AzureBlobOutputStream(blobClient, path);
+ } catch (BlobStorageException e) {
+ throw handleBlobException(e);
+ }
+ }
+
+ void close() {}
+
+ @VisibleForTesting
+ void deleteContainerForTests() {
+ try {
+ containerClient.delete();
+ } catch (BlobStorageException e) {
+ if (e.getStatusCode() != HTTP_NOT_FOUND) {
+ throw e;
+ }
+ }
+ }
+
+ private Collection<String> deleteBlobs(Collection<String> paths) throws AzureBlobException {
+ try {
+ return deleteBlobs(paths, DELETE_BATCH_SIZE);
+ } catch (BlobStorageException e) {
+ throw handleBlobException(e);
+ }
+ }
+
+ @VisibleForTesting
+ Collection<String> deleteBlobs(Collection<String> entries, int batchSize)
+ throws AzureBlobException {
+ Set<String> deletedPaths = new HashSet<>();
+
+ for (String path : entries) {
+ try {
+ BlobClient blobClient = containerClient.getBlobClient(path);
+ boolean existed = blobClient.deleteIfExists();
+ if (existed) {
+ deletedPaths.add(path);
+ }
+ } catch (BlobStorageException e) {
+ if (e.getStatusCode() == HTTP_NOT_FOUND) {
Review Comment:
made the change, ie replaced deleteIfExists with batched deletes; HTTP_NOT_FOUND sub-errors are filtered
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17949: Add Azure Blob Storage backup repository module [solr]
Posted by "prateeksinghalgit (via GitHub)" <gi...@apache.org>.
prateeksinghalgit commented on PR #3750:
URL: https://github.com/apache/solr/pull/3750#issuecomment-4567142525
Could a committer please approve workflow runs on this fork PR? The checks are currently waiting for maintainer approval. cc @janhoy
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17949: Add Azure Blob Storage backup repository module [solr]
Posted by "prateeksinghalgit (via GitHub)" <gi...@apache.org>.
prateeksinghalgit commented on code in PR #3750:
URL: https://github.com/apache/solr/pull/3750#discussion_r3232112168
##########
solr/modules/azure-blob-repository/src/java/org/apache/solr/azureblob/AzureBlobIndexInput.java:
##########
@@ -0,0 +1,200 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.azureblob;
+
+import java.io.EOFException;
+import java.io.IOException;
+import java.io.InputStream;
+import java.util.LinkedHashMap;
+import java.util.Map;
+import org.apache.lucene.store.IndexInput;
+
+class AzureBlobIndexInput extends IndexInput {
+
+ private static final int MIN_PAGE_SIZE = 4 * 1024;
+ private static final int DEFAULT_PAGE_SIZE = 512 * 1024;
+ private static final int MAX_CACHED_PAGES = 128;
+
+ private final String path;
+ private final AzureBlobStorageClient client;
+ private final long length;
+ private final int pageSize;
+ private final LruPageCache cache;
+
+ private long position = 0L;
+ private boolean closed = false;
+
+ AzureBlobIndexInput(String path, AzureBlobStorageClient client, long length) {
+ this(path, client, length, DEFAULT_PAGE_SIZE, MAX_CACHED_PAGES);
+ }
+
+ AzureBlobIndexInput(
+ String path, AzureBlobStorageClient client, long length, int pageSize, int maxCachedPages) {
+ super(path);
+ this.path = path;
+ this.client = client;
+ this.length = length;
+ this.pageSize = Math.max(MIN_PAGE_SIZE, pageSize);
+ this.cache = new LruPageCache(maxCachedPages);
+ }
+
+ @Override
+ public void close() throws IOException {
+ closed = true;
+ cache.clear();
+ }
+
+ @Override
+ public long getFilePointer() {
+ return position;
+ }
+
+ @Override
+ public void seek(long pos) throws IOException {
+ ensureOpen();
+ if (pos < 0 || pos > length) {
+ throw new IOException("Seek position out of bounds: " + pos);
+ }
+
+ position = pos;
+ }
+
+ @Override
+ public long length() {
+ return length;
+ }
+
+ @Override
+ public IndexInput slice(String sliceDescription, long offset, long length) throws IOException {
+ ensureOpen();
+ if (offset < 0 || length < 0 || offset + length > this.length) {
+ throw new IOException("Slice out of bounds: offset=" + offset + ", length=" + length);
+ }
+
+ AzureBlobIndexInput slice =
+ new AzureBlobIndexInput(
Review Comment:
removed it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17949: Add Azure Blob Storage backup repository module [solr]
Posted by "prateeksinghalgit (via GitHub)" <gi...@apache.org>.
prateeksinghalgit commented on code in PR #3750:
URL: https://github.com/apache/solr/pull/3750#discussion_r3232277896
##########
solr/modules/azure-blob-repository/src/java/org/apache/solr/azureblob/AzureBlobOutputStream.java:
##########
@@ -0,0 +1,269 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.azureblob;
+
+import com.azure.storage.blob.BlobClient;
+import com.azure.storage.blob.models.BlobStorageException;
+import com.azure.storage.blob.specialized.BlockBlobClient;
+import java.io.ByteArrayInputStream;
+import java.io.IOException;
+import java.io.OutputStream;
+import java.lang.invoke.MethodHandles;
+import java.nio.ByteBuffer;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Base64;
+import java.util.List;
+import java.util.UUID;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * OutputStream implementation for Azure Blob Storage using block blobs. Supports chunked uploads
+ * for large files.
+ */
+public class AzureBlobOutputStream extends OutputStream {
+ private static final Logger log = LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+
+ static final int BLOCK_SIZE = 4 * 1024 * 1024;
+
+ private final BlobClient blobClient;
+ private final String blobPath;
+ private volatile boolean closed;
+ private final ByteBuffer buffer;
+ private BlockUpload blockUpload;
+ private boolean committed;
+
+ public AzureBlobOutputStream(BlobClient blobClient, String blobPath) {
+ this.blobClient = blobClient;
+ this.blobPath = blobPath;
+ this.closed = false;
+ this.buffer = ByteBuffer.allocate(BLOCK_SIZE);
+ this.blockUpload = null;
+ this.committed = false;
+
+ if (log.isDebugEnabled()) {
+ log.debug("Created BlobOutputStream for blobPath '{}'", blobPath);
+ }
+ }
+
+ @Override
+ public void write(int b) throws IOException {
+ if (closed) {
+ throw new IOException("Stream closed");
+ }
+
+ buffer.put((byte) b);
+
+ if (!buffer.hasRemaining()) {
+ uploadBlock();
+ }
+ }
+
+ @Override
+ public void write(byte[] b, int off, int len) throws IOException {
+ if (closed) {
+ throw new IOException("Stream closed");
+ }
+
+ if (outOfRange(off, b.length) || len < 0 || outOfRange(off + len, b.length)) {
+ throw new IndexOutOfBoundsException();
+ } else if (len == 0) {
+ return;
+ }
+
+ int currentOffset = off;
+ int lenRemaining = len;
+ while (buffer.remaining() < lenRemaining) {
+ int firstPart = buffer.remaining();
+ buffer.put(b, currentOffset, firstPart);
+ uploadBlock();
+
+ currentOffset += firstPart;
+ lenRemaining -= firstPart;
+ }
+ if (lenRemaining > 0) {
+ buffer.put(b, currentOffset, lenRemaining);
+ }
+ }
+
+ private static boolean outOfRange(int off, int len) {
+ return off < 0 || off > len;
+ }
+
+ private void uploadBlock() throws IOException {
+ int size = buffer.position() - buffer.arrayOffset();
+
+ if (size == 0) {
+ return;
+ }
+
+ if (blockUpload == null) {
+ if (log.isDebugEnabled()) {
+ log.debug("New block upload for blobPath '{}'", blobPath);
+ }
+
+ blockUpload = newBlockUpload();
+ }
+
+ try (ByteArrayInputStream inputStream =
Review Comment:
fixed. Now passing BinaryData.fromByteBuffer(ByteBuffer.wrap(buffer.array(), 0, size)) to the new stageBlock(String, BinaryData) overload, so no ByteArrayInputStream is involved.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17949: Add Azure Blob Storage backup repository module [solr]
Posted by "prateeksinghalgit (via GitHub)" <gi...@apache.org>.
prateeksinghalgit commented on code in PR #3750:
URL: https://github.com/apache/solr/pull/3750#discussion_r3232299832
##########
solr/modules/azure-blob-repository/src/java/org/apache/solr/azureblob/AzureBlobOutputStream.java:
##########
@@ -0,0 +1,269 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.azureblob;
+
+import com.azure.storage.blob.BlobClient;
+import com.azure.storage.blob.models.BlobStorageException;
+import com.azure.storage.blob.specialized.BlockBlobClient;
+import java.io.ByteArrayInputStream;
+import java.io.IOException;
+import java.io.OutputStream;
+import java.lang.invoke.MethodHandles;
+import java.nio.ByteBuffer;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Base64;
+import java.util.List;
+import java.util.UUID;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * OutputStream implementation for Azure Blob Storage using block blobs. Supports chunked uploads
+ * for large files.
+ */
+public class AzureBlobOutputStream extends OutputStream {
+ private static final Logger log = LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+
+ static final int BLOCK_SIZE = 4 * 1024 * 1024;
+
+ private final BlobClient blobClient;
+ private final String blobPath;
+ private volatile boolean closed;
+ private final ByteBuffer buffer;
+ private BlockUpload blockUpload;
+ private boolean committed;
+
+ public AzureBlobOutputStream(BlobClient blobClient, String blobPath) {
+ this.blobClient = blobClient;
+ this.blobPath = blobPath;
+ this.closed = false;
+ this.buffer = ByteBuffer.allocate(BLOCK_SIZE);
+ this.blockUpload = null;
+ this.committed = false;
+
+ if (log.isDebugEnabled()) {
+ log.debug("Created BlobOutputStream for blobPath '{}'", blobPath);
+ }
+ }
+
+ @Override
+ public void write(int b) throws IOException {
+ if (closed) {
+ throw new IOException("Stream closed");
+ }
+
+ buffer.put((byte) b);
+
+ if (!buffer.hasRemaining()) {
+ uploadBlock();
+ }
+ }
+
+ @Override
+ public void write(byte[] b, int off, int len) throws IOException {
+ if (closed) {
+ throw new IOException("Stream closed");
+ }
+
+ if (outOfRange(off, b.length) || len < 0 || outOfRange(off + len, b.length)) {
+ throw new IndexOutOfBoundsException();
+ } else if (len == 0) {
+ return;
+ }
+
+ int currentOffset = off;
+ int lenRemaining = len;
+ while (buffer.remaining() < lenRemaining) {
+ int firstPart = buffer.remaining();
+ buffer.put(b, currentOffset, firstPart);
+ uploadBlock();
+
+ currentOffset += firstPart;
+ lenRemaining -= firstPart;
+ }
+ if (lenRemaining > 0) {
+ buffer.put(b, currentOffset, lenRemaining);
+ }
+ }
+
+ private static boolean outOfRange(int off, int len) {
+ return off < 0 || off > len;
+ }
+
+ private void uploadBlock() throws IOException {
+ int size = buffer.position() - buffer.arrayOffset();
+
+ if (size == 0) {
+ return;
+ }
+
+ if (blockUpload == null) {
+ if (log.isDebugEnabled()) {
+ log.debug("New block upload for blobPath '{}'", blobPath);
+ }
+
+ blockUpload = newBlockUpload();
+ }
+
+ try (ByteArrayInputStream inputStream =
+ new ByteArrayInputStream(buffer.array(), buffer.arrayOffset(), size)) {
+ blockUpload.uploadBlock(inputStream, size);
+ } catch (BlobStorageException e) {
+ if (blockUpload != null) {
+ blockUpload.abort();
+ if (log.isDebugEnabled()) {
+ log.debug("Block upload aborted for blobPath '{}'.", blobPath);
+ }
+ }
+
+ throw new IOException(
+ "Failed to upload block", AzureBlobStorageClient.handleBlobException(e));
+ }
+
+ buffer.clear();
+ }
+
+ @Override
+ public void flush() throws IOException {
+ if (closed) {
+ throw new IOException("Stream closed");
+ }
+
+ if (buffer.position() - buffer.arrayOffset() > 0) {
+ uploadBlock();
+ }
+
+ if (blockUpload != null) {
+ blockUpload.complete();
+ blockUpload = null;
+ committed = true;
+ }
+ }
+
+ @Override
+ public void close() throws IOException {
+ if (closed) {
+ return;
+ }
+
+ if (blockUpload != null && blockUpload.aborted) {
+ blockUpload = null;
+ closed = true;
+ return;
+ }
+
+ if (!committed) {
+ uploadBlock();
+ if (blockUpload != null) {
+ blockUpload.complete();
+ blockUpload = null;
+ committed = true;
+ } else {
+ try {
+ blobClient.upload(new ByteArrayInputStream(new byte[0]), 0, true);
+ } catch (BlobStorageException e) {
+ throw new IOException(
+ "Failed to create empty blob", AzureBlobStorageClient.handleBlobException(e));
+ }
+ }
+ } else {
+ if (blockUpload != null) {
+ blockUpload.complete();
+ blockUpload = null;
+ }
+ }
+
+ closed = true;
+ }
+
+ private BlockUpload newBlockUpload() throws IOException {
+ try {
+ return new BlockUpload();
+ } catch (BlobStorageException e) {
+ throw new IOException(
+ "Failed to create block upload", AzureBlobStorageClient.handleBlobException(e));
+ }
+ }
+
+ private class BlockUpload {
+ private final List<String> blockIds;
+ private boolean aborted = false;
+
+ public BlockUpload() {
+ this.blockIds = new ArrayList<>();
+ if (log.isDebugEnabled()) {
+ log.debug("Initiated block upload for blobPath '{}'", blobPath);
+ }
+
+ try {
+ BlockBlobClient blockBlobClient = blobClient.getBlockBlobClient();
+ blockBlobClient.deleteIfExists();
+ } catch (BlobStorageException e) {
+ // ignore; subsequent stage/commit will surface real issues
+ }
+ }
+
+ void uploadBlock(ByteArrayInputStream inputStream, long blockSize) {
+ if (aborted) {
+ throw new IllegalStateException(
+ "Can't upload new blocks on a BlockUpload that was aborted");
+ }
+
+ String blockId =
+ Base64.getEncoder()
+ .encodeToString(UUID.randomUUID().toString().getBytes(StandardCharsets.UTF_8));
+
+ if (log.isDebugEnabled()) {
+ log.debug("Uploading block {} for blobPath '{}'", blockId, blobPath);
+ }
+
+ try {
+ BlockBlobClient blockBlobClient = blobClient.getBlockBlobClient();
+ blockBlobClient.stageBlock(blockId, inputStream, blockSize);
+ blockIds.add(blockId);
+ } catch (BlobStorageException e) {
+ throw new RuntimeException("Failed to upload block", e);
+ }
+ }
+
+ void complete() {
+ if (aborted) {
+ throw new IllegalStateException("Can't complete a BlockUpload that was aborted");
+ }
+
+ if (log.isDebugEnabled()) {
+ log.debug("Completing block upload for blobPath '{}'", blobPath);
+ }
+
+ try {
+ BlockBlobClient blockBlobClient = blobClient.getBlockBlobClient();
+ blockBlobClient.commitBlockList(blockIds);
+ } catch (BlobStorageException e) {
+ throw new RuntimeException("Failed to commit block list", e);
Review Comment:
made the change to IOException
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17949: Add Azure Blob Storage backup repository module [solr]
Posted by "psalagnac (via GitHub)" <gi...@apache.org>.
psalagnac commented on code in PR #3750:
URL: https://github.com/apache/solr/pull/3750#discussion_r3202835738
##########
solr/modules/azure-blob-repository/src/java/org/apache/solr/azureblob/AzureBlobStorageClient.java:
##########
@@ -0,0 +1,529 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.azureblob;
+
+import com.azure.core.credential.TokenCredential;
+import com.azure.identity.DefaultAzureCredentialBuilder;
+import com.azure.storage.blob.BlobClient;
+import com.azure.storage.blob.BlobContainerClient;
+import com.azure.storage.blob.BlobServiceClient;
+import com.azure.storage.blob.BlobServiceClientBuilder;
+import com.azure.storage.blob.models.BlobItem;
+import com.azure.storage.blob.models.BlobStorageException;
+import com.azure.storage.blob.models.ListBlobsOptions;
+import com.google.common.annotations.VisibleForTesting;
+import java.io.ByteArrayInputStream;
+import java.io.FilterInputStream;
+import java.io.InputStream;
+import java.io.OutputStream;
+import java.lang.invoke.MethodHandles;
+import java.util.Collection;
+import java.util.HashSet;
+import java.util.Locale;
+import java.util.Set;
+import java.util.stream.Collectors;
+import org.apache.solr.common.util.ResumableInputStream;
+import org.apache.solr.common.util.StrUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Creates a {@link BlobServiceClient} for communicating with Azure Blob Storage. Utilizes the
+ * default Azure credential provider chain.
+ */
+public class AzureBlobStorageClient {
+
+ private static final Logger log = LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+
+ static final String BLOB_FILE_PATH_DELIMITER = "/";
+ private static final int HTTP_NOT_FOUND = 404;
+ private static final int HTTP_CONFLICT = 409;
+ private static final int SKIP_BUFFER_SIZE = 8192;
+ private static final int DELETE_BATCH_SIZE = 1000;
+
+ private static final com.azure.core.http.HttpClient SHARED_HTTP_CLIENT =
+ new com.azure.core.http.okhttp.OkHttpAsyncHttpClientBuilder().build();
+
+ private final BlobContainerClient containerClient;
+
+ AzureBlobStorageClient(
+ String containerName,
+ String connectionString,
+ String endpoint,
+ String accountName,
+ String accountKey,
+ String sasToken,
+ String tenantId,
+ String clientId,
+ String clientSecret) {
+ this(
+ createInternalClient(
+ connectionString,
+ endpoint,
+ accountName,
+ accountKey,
+ sasToken,
+ tenantId,
+ clientId,
+ clientSecret),
+ containerName);
+ }
+
+ @VisibleForTesting
+ AzureBlobStorageClient(BlobServiceClient blobServiceClient, String containerName) {
+ this.containerClient = blobServiceClient.getBlobContainerClient(containerName);
+ try {
+ containerClient.create();
+ } catch (BlobStorageException e) {
+ if (e.getStatusCode() != HTTP_CONFLICT) {
+ throw e;
+ }
+ }
+ }
+
+ private static BlobServiceClient createInternalClient(
+ String connectionString,
+ String endpoint,
+ String accountName,
+ String accountKey,
+ String sasToken,
+ String tenantId,
+ String clientId,
Review Comment:
`clientId` and `clientSecret` are unused. These parameters could be removed from this class.
##########
solr/modules/azure-blob-repository/src/java/org/apache/solr/azureblob/AzureBlobStorageClient.java:
##########
@@ -0,0 +1,529 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.azureblob;
+
+import com.azure.core.credential.TokenCredential;
+import com.azure.identity.DefaultAzureCredentialBuilder;
+import com.azure.storage.blob.BlobClient;
+import com.azure.storage.blob.BlobContainerClient;
+import com.azure.storage.blob.BlobServiceClient;
+import com.azure.storage.blob.BlobServiceClientBuilder;
+import com.azure.storage.blob.models.BlobItem;
+import com.azure.storage.blob.models.BlobStorageException;
+import com.azure.storage.blob.models.ListBlobsOptions;
+import com.google.common.annotations.VisibleForTesting;
+import java.io.ByteArrayInputStream;
+import java.io.FilterInputStream;
+import java.io.InputStream;
+import java.io.OutputStream;
+import java.lang.invoke.MethodHandles;
+import java.util.Collection;
+import java.util.HashSet;
+import java.util.Locale;
+import java.util.Set;
+import java.util.stream.Collectors;
+import org.apache.solr.common.util.ResumableInputStream;
+import org.apache.solr.common.util.StrUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Creates a {@link BlobServiceClient} for communicating with Azure Blob Storage. Utilizes the
+ * default Azure credential provider chain.
+ */
+public class AzureBlobStorageClient {
+
+ private static final Logger log = LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+
+ static final String BLOB_FILE_PATH_DELIMITER = "/";
+ private static final int HTTP_NOT_FOUND = 404;
+ private static final int HTTP_CONFLICT = 409;
+ private static final int SKIP_BUFFER_SIZE = 8192;
+ private static final int DELETE_BATCH_SIZE = 1000;
+
+ private static final com.azure.core.http.HttpClient SHARED_HTTP_CLIENT =
Review Comment:
I'm not a fan of having a static HTTP client, mostly because it nis ever closed, including in tests and it might leak threads.
I'm not sure what is the expected lifecycle of this HTTP client. Is it closed when we close the blob client? I don't see any `close()` or similar method.
##########
solr/modules/azure-blob-repository/src/java/org/apache/solr/azureblob/AzureBlobOutputStream.java:
##########
@@ -0,0 +1,269 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.azureblob;
+
+import com.azure.storage.blob.BlobClient;
+import com.azure.storage.blob.models.BlobStorageException;
+import com.azure.storage.blob.specialized.BlockBlobClient;
+import java.io.ByteArrayInputStream;
+import java.io.IOException;
+import java.io.OutputStream;
+import java.lang.invoke.MethodHandles;
+import java.nio.ByteBuffer;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Base64;
+import java.util.List;
+import java.util.UUID;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * OutputStream implementation for Azure Blob Storage using block blobs. Supports chunked uploads
+ * for large files.
+ */
+public class AzureBlobOutputStream extends OutputStream {
+ private static final Logger log = LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+
+ static final int BLOCK_SIZE = 4 * 1024 * 1024;
+
+ private final BlobClient blobClient;
+ private final String blobPath;
+ private volatile boolean closed;
+ private final ByteBuffer buffer;
+ private BlockUpload blockUpload;
+ private boolean committed;
+
+ public AzureBlobOutputStream(BlobClient blobClient, String blobPath) {
+ this.blobClient = blobClient;
+ this.blobPath = blobPath;
+ this.closed = false;
+ this.buffer = ByteBuffer.allocate(BLOCK_SIZE);
+ this.blockUpload = null;
+ this.committed = false;
+
+ if (log.isDebugEnabled()) {
+ log.debug("Created BlobOutputStream for blobPath '{}'", blobPath);
+ }
+ }
+
+ @Override
+ public void write(int b) throws IOException {
+ if (closed) {
+ throw new IOException("Stream closed");
+ }
+
+ buffer.put((byte) b);
+
+ if (!buffer.hasRemaining()) {
+ uploadBlock();
+ }
+ }
+
+ @Override
+ public void write(byte[] b, int off, int len) throws IOException {
+ if (closed) {
+ throw new IOException("Stream closed");
+ }
+
+ if (outOfRange(off, b.length) || len < 0 || outOfRange(off + len, b.length)) {
+ throw new IndexOutOfBoundsException();
+ } else if (len == 0) {
+ return;
+ }
+
+ int currentOffset = off;
+ int lenRemaining = len;
+ while (buffer.remaining() < lenRemaining) {
+ int firstPart = buffer.remaining();
+ buffer.put(b, currentOffset, firstPart);
+ uploadBlock();
+
+ currentOffset += firstPart;
+ lenRemaining -= firstPart;
+ }
+ if (lenRemaining > 0) {
+ buffer.put(b, currentOffset, lenRemaining);
+ }
+ }
+
+ private static boolean outOfRange(int off, int len) {
+ return off < 0 || off > len;
+ }
+
+ private void uploadBlock() throws IOException {
+ int size = buffer.position() - buffer.arrayOffset();
Review Comment:
Logic here is wrong
```suggestion
int size = buffer.position();
```
Array offset is never set anyway
##########
solr/modules/azure-blob-repository/src/java/org/apache/solr/azureblob/AzureBlobStorageClient.java:
##########
@@ -0,0 +1,529 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.azureblob;
+
+import com.azure.core.credential.TokenCredential;
+import com.azure.identity.DefaultAzureCredentialBuilder;
+import com.azure.storage.blob.BlobClient;
+import com.azure.storage.blob.BlobContainerClient;
+import com.azure.storage.blob.BlobServiceClient;
+import com.azure.storage.blob.BlobServiceClientBuilder;
+import com.azure.storage.blob.models.BlobItem;
+import com.azure.storage.blob.models.BlobStorageException;
+import com.azure.storage.blob.models.ListBlobsOptions;
+import com.google.common.annotations.VisibleForTesting;
+import java.io.ByteArrayInputStream;
+import java.io.FilterInputStream;
+import java.io.InputStream;
+import java.io.OutputStream;
+import java.lang.invoke.MethodHandles;
+import java.util.Collection;
+import java.util.HashSet;
+import java.util.Locale;
+import java.util.Set;
+import java.util.stream.Collectors;
+import org.apache.solr.common.util.ResumableInputStream;
+import org.apache.solr.common.util.StrUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Creates a {@link BlobServiceClient} for communicating with Azure Blob Storage. Utilizes the
+ * default Azure credential provider chain.
+ */
+public class AzureBlobStorageClient {
+
+ private static final Logger log = LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+
+ static final String BLOB_FILE_PATH_DELIMITER = "/";
+ private static final int HTTP_NOT_FOUND = 404;
+ private static final int HTTP_CONFLICT = 409;
+ private static final int SKIP_BUFFER_SIZE = 8192;
+ private static final int DELETE_BATCH_SIZE = 1000;
+
+ private static final com.azure.core.http.HttpClient SHARED_HTTP_CLIENT =
+ new com.azure.core.http.okhttp.OkHttpAsyncHttpClientBuilder().build();
+
+ private final BlobContainerClient containerClient;
+
+ AzureBlobStorageClient(
+ String containerName,
+ String connectionString,
+ String endpoint,
+ String accountName,
+ String accountKey,
+ String sasToken,
+ String tenantId,
+ String clientId,
+ String clientSecret) {
+ this(
+ createInternalClient(
+ connectionString,
+ endpoint,
+ accountName,
+ accountKey,
+ sasToken,
+ tenantId,
+ clientId,
+ clientSecret),
+ containerName);
+ }
+
+ @VisibleForTesting
+ AzureBlobStorageClient(BlobServiceClient blobServiceClient, String containerName) {
+ this.containerClient = blobServiceClient.getBlobContainerClient(containerName);
+ try {
+ containerClient.create();
+ } catch (BlobStorageException e) {
+ if (e.getStatusCode() != HTTP_CONFLICT) {
+ throw e;
+ }
+ }
+ }
+
+ private static BlobServiceClient createInternalClient(
+ String connectionString,
+ String endpoint,
+ String accountName,
+ String accountKey,
+ String sasToken,
+ String tenantId,
+ String clientId,
+ String clientSecret) {
+
+ BlobServiceClientBuilder builder = new BlobServiceClientBuilder();
+ builder.httpClient(SHARED_HTTP_CLIENT);
+
+ if (StrUtils.isNotNullOrEmpty(connectionString)) {
+ builder.connectionString(connectionString);
+ } else if (StrUtils.isNotNullOrEmpty(endpoint)) {
+ builder.endpoint(endpoint);
+ if (StrUtils.isNotNullOrEmpty(accountName) && StrUtils.isNotNullOrEmpty(accountKey)) {
+ builder.credential(
+ new com.azure.storage.common.StorageSharedKeyCredential(accountName, accountKey));
+ } else if (StrUtils.isNotNullOrEmpty(sasToken)) {
+ builder.sasToken(sasToken);
+ } else {
+ TokenCredential credential = new DefaultAzureCredentialBuilder().tenantId(tenantId).build();
+ builder.credential(credential);
+ }
+ } else {
+ throw new IllegalArgumentException("Either connectionString or endpoint must be provided");
+ }
+
+ return builder.buildClient();
+ }
+
+ void createDirectory(String path) throws AzureBlobException {
+ String sanitizedDirPath = sanitizedDirPath(path);
+
+ if (!pathExists(sanitizedDirPath)) {
+ String parent = getParentDirectory(sanitizedDirPath);
+ if (!parent.isEmpty() && !parent.equals(BLOB_FILE_PATH_DELIMITER)) {
+ createDirectory(parent);
+ }
+
+ try {
+ BlobClient blobClient = containerClient.getBlobClient(sanitizedDirPath);
+ blobClient.upload(new ByteArrayInputStream(new byte[0]), 0, true);
Review Comment:
This is under optimized. We do two requests to create an empty object. Metadata should be set in the initial request.
Something like this should work:
```
BlobParallelUploadOptions options =
new BlobParallelUploadOptions(new ByteArrayInputStream(new byte[0]))
.setMetadata(metadata);
blobClient.uploadWithResponse(options, null, Context.NONE);
```
##########
solr/modules/azure-blob-repository/src/java/org/apache/solr/azureblob/AzureBlobIndexInput.java:
##########
@@ -0,0 +1,200 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.azureblob;
+
+import java.io.EOFException;
+import java.io.IOException;
+import java.io.InputStream;
+import java.util.LinkedHashMap;
+import java.util.Map;
+import org.apache.lucene.store.IndexInput;
+
+class AzureBlobIndexInput extends IndexInput {
+
+ private static final int MIN_PAGE_SIZE = 4 * 1024;
+ private static final int DEFAULT_PAGE_SIZE = 512 * 1024;
+ private static final int MAX_CACHED_PAGES = 128;
+
+ private final String path;
+ private final AzureBlobStorageClient client;
+ private final long length;
+ private final int pageSize;
+ private final LruPageCache cache;
+
+ private long position = 0L;
+ private boolean closed = false;
+
+ AzureBlobIndexInput(String path, AzureBlobStorageClient client, long length) {
+ this(path, client, length, DEFAULT_PAGE_SIZE, MAX_CACHED_PAGES);
+ }
+
+ AzureBlobIndexInput(
+ String path, AzureBlobStorageClient client, long length, int pageSize, int maxCachedPages) {
+ super(path);
+ this.path = path;
+ this.client = client;
+ this.length = length;
+ this.pageSize = Math.max(MIN_PAGE_SIZE, pageSize);
+ this.cache = new LruPageCache(maxCachedPages);
+ }
+
+ @Override
+ public void close() throws IOException {
+ closed = true;
+ cache.clear();
+ }
+
+ @Override
+ public long getFilePointer() {
+ return position;
+ }
+
+ @Override
+ public void seek(long pos) throws IOException {
+ ensureOpen();
+ if (pos < 0 || pos > length) {
+ throw new IOException("Seek position out of bounds: " + pos);
+ }
+
+ position = pos;
+ }
+
+ @Override
+ public long length() {
+ return length;
+ }
+
+ @Override
+ public IndexInput slice(String sliceDescription, long offset, long length) throws IOException {
+ ensureOpen();
+ if (offset < 0 || length < 0 || offset + length > this.length) {
+ throw new IOException("Slice out of bounds: offset=" + offset + ", length=" + length);
+ }
+
+ AzureBlobIndexInput slice =
+ new AzureBlobIndexInput(
+ getFullSliceDescription(sliceDescription), client, length, pageSize, MAX_CACHED_PAGES);
+
+ slice.position = 0L;
+
+ // Wrap client in a view that remaps range requests by adding base offset
+ slice.clientViewBaseOffset = this.clientViewBaseOffset + offset;
+ return slice;
+ }
+
+ @Override
+ public byte readByte() throws IOException {
+ ensureOpen();
+ if (position >= length) {
+ throw new EOFException("End of stream reached");
+ }
+
+ byte[] page = getPage(pageIndex(position));
+ int inPageOffset = (int) (position % pageSize);
+ byte value = page[inPageOffset];
+ position += 1L;
+ return value;
+ }
+
+ @Override
+ public void readBytes(byte[] b, int offset, int len) throws IOException {
+ ensureOpen();
+ if (len < 0) {
+ throw new IOException("Length must be non-negative");
+ }
+
+ if (position + len > length) {
+ throw new EOFException("End of stream reached");
+ }
+
+ int remaining = len;
+ while (remaining > 0) {
+ long pageIdx = pageIndex(position);
+ byte[] page = getPage(pageIdx);
+ int inPageOffset = (int) (position % pageSize);
+ int toCopy = Math.min(remaining, pageSize - inPageOffset);
+ System.arraycopy(page, inPageOffset, b, offset + (len - remaining), toCopy);
+ position += toCopy;
+ remaining -= toCopy;
+ }
+ }
+
+ // Internal state for slices: base offset to add to all range requests
+ private long clientViewBaseOffset = 0L;
Review Comment:
This should be immutable and passed to the constructor.
Also, I'm rather it is moved with other class members.
##########
solr/modules/azure-blob-repository/src/java/org/apache/solr/azureblob/AzureBlobBackupRepository.java:
##########
@@ -0,0 +1,396 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.azureblob;
+
+import com.google.common.annotations.VisibleForTesting;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStream;
+import java.lang.invoke.MethodHandles;
+import java.net.URI;
+import java.net.URISyntaxException;
+import java.time.Duration;
+import java.time.Instant;
+import java.util.Collection;
+import java.util.Objects;
+import java.util.Set;
+import java.util.stream.Collectors;
+import org.apache.lucene.store.Directory;
+import org.apache.lucene.store.IOContext;
+import org.apache.lucene.store.IndexInput;
+import org.apache.lucene.store.IndexOutput;
+import org.apache.solr.common.SolrException;
+import org.apache.solr.common.util.NamedList;
+import org.apache.solr.common.util.StrUtils;
+import org.apache.solr.core.backup.repository.AbstractBackupRepository;
+import org.apache.solr.core.backup.repository.BackupRepository;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * A concrete implementation of {@link BackupRepository} interface supporting backup/restore of Solr
+ * indexes to Azure Blob Storage.
+ */
+public class AzureBlobBackupRepository extends AbstractBackupRepository {
+
+ private static final Logger log = LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+
+ static final String BLOB_SCHEME = "blob";
+ private static final int CHUNK_SIZE = 16 * 1024 * 1024;
+ private static final int COPY_BUFFER_SIZE = 8192;
+
+ private AzureBlobStorageClient client;
+
+ @Override
+ public void init(NamedList<?> args) {
+ super.init(args);
+ AzureBlobBackupRepositoryConfig backupConfig = new AzureBlobBackupRepositoryConfig(this.config);
+
+ if (client != null) {
+ client.close();
+ }
+
+ this.client = backupConfig.buildClient();
+ }
+
+ @VisibleForTesting
+ public void setClient(AzureBlobStorageClient client) {
+ this.client = client;
+ }
+
+ @Override
+ @SuppressWarnings("unchecked")
+ public <T> T getConfigProperty(String name) {
+ return (T) this.config.get(name);
+ }
+
+ @Override
+ public URI createURI(String location) {
+ if (StrUtils.isNullOrEmpty(location)) {
+ throw new IllegalArgumentException("cannot create URI with an empty location");
+ }
+
+ URI result;
+ try {
+ if (location.startsWith(BLOB_SCHEME + ":")) {
+ result = new URI(location);
+ } else if (location.startsWith("/")) {
+ result = new URI(BLOB_SCHEME, "", location, null);
+ } else {
+ result = new URI(BLOB_SCHEME, "", "/" + location, null);
+ }
+ return result;
+ } catch (URISyntaxException ex) {
+ throw new SolrException(SolrException.ErrorCode.BAD_REQUEST, ex);
+ }
+ }
+
+ @Override
+ public URI createDirectoryURI(String location) {
+ if (StrUtils.isNullOrEmpty(location)) {
+ throw new IllegalArgumentException("cannot create URI with an empty location");
+ }
+
+ if (!location.endsWith("/")) {
+ location += "/";
+ }
+
+ return createURI(location);
+ }
+
+ @Override
+ public URI resolve(URI baseUri, String... pathComponents) {
+ if (!BLOB_SCHEME.equalsIgnoreCase(baseUri.getScheme())) {
+ throw new IllegalArgumentException("URI must begin with 'blob:' scheme");
+ }
+
+ String path = baseUri + "/" + String.join("/", pathComponents);
+ return URI.create(path).normalize();
+ }
+
+ @Override
+ public URI resolveDirectory(URI baseUri, String... pathComponents) {
+ if (pathComponents.length > 0) {
+ if (!pathComponents[pathComponents.length - 1].endsWith("/")) {
+ pathComponents[pathComponents.length - 1] = pathComponents[pathComponents.length - 1] + "/";
+ }
+ } else {
+ if (!baseUri.toString().endsWith("/")) {
+ baseUri = URI.create(baseUri + "/");
+ }
+ }
+ return resolve(baseUri, pathComponents);
+ }
+
+ @Override
+ public void createDirectory(URI path) throws IOException {
+ Objects.requireNonNull(path, "cannot create directory to a null URI");
+
+ String blobPath = getBlobPath(path);
+
+ if (log.isDebugEnabled()) {
+ log.debug("Create directory '{}'", blobPath);
+ }
+
+ try {
+ client.createDirectory(blobPath);
+ } catch (AzureBlobException e) {
+ throw new IOException("Failed to create directory " + blobPath, e);
+ }
+ }
+
+ @Override
+ public void deleteDirectory(URI path) throws IOException {
+ Objects.requireNonNull(path, "cannot delete directory with a null URI");
+
+ String blobPath = getBlobPath(path);
+
+ if (log.isDebugEnabled()) {
+ log.debug("Delete directory '{}'", blobPath);
+ }
+
+ try {
+ client.deleteDirectory(blobPath);
+ } catch (AzureBlobException e) {
+ throw new IOException("Failed to delete directory " + blobPath, e);
+ }
+ }
+
+ @Override
+ public void delete(URI path, Collection<String> files) throws IOException {
+ Objects.requireNonNull(path, "cannot delete with a null URI");
+ Objects.requireNonNull(files, "cannot delete with a null files collection");
+
+ String basePath = getBlobPath(path);
+
+ try {
+ if (!client.isDirectory(basePath)) {
+ int lastSlash = basePath.lastIndexOf('/');
+ basePath = lastSlash >= 0 ? basePath.substring(0, lastSlash) : "";
+ }
+ } catch (AzureBlobException e) {
+ throw new IOException("Failed to check path type for " + basePath, e);
+ }
+
+ final String baseForPaths = basePath;
+ Set<String> fullPaths =
+ files.stream()
+ .map(file -> (baseForPaths.isEmpty() ? file : baseForPaths + "/" + file))
+ .collect(Collectors.toSet());
+
+ if (log.isDebugEnabled()) {
+ log.debug("Delete files '{}'", fullPaths);
+ }
+
+ try {
+ client.delete(fullPaths);
+ } catch (AzureBlobException e) {
+ throw new IOException("Failed to delete files " + fullPaths, e);
+ }
+ }
+
+ @Override
+ public boolean exists(URI path) throws IOException {
+ Objects.requireNonNull(path, "cannot check existence with a null URI");
+
+ String blobPath = getBlobPath(path);
+
+ if (log.isDebugEnabled()) {
+ log.debug("Check existence '{}'", blobPath);
+ }
+
+ try {
+ return client.pathExists(blobPath);
+ } catch (AzureBlobException e) {
+ throw new IOException("Failed to check existence of " + blobPath, e);
+ }
+ }
+
+ @Override
+ public PathType getPathType(URI path) throws IOException {
+ Objects.requireNonNull(path, "cannot get path type with a null URI");
+
+ String blobPath = getBlobPath(path);
+
+ if (log.isDebugEnabled()) {
+ log.debug("Get path type '{}'", blobPath);
+ }
+
+ try {
+ if (client.isDirectory(blobPath)) {
+ return BackupRepository.PathType.DIRECTORY;
+ } else {
+ return BackupRepository.PathType.FILE;
+ }
+ } catch (AzureBlobException e) {
+ throw new IOException("Failed to get path type for " + blobPath, e);
+ }
+ }
+
+ @Override
+ public String[] listAll(URI path) throws IOException {
+ Objects.requireNonNull(path, "cannot list with a null URI");
+
+ String blobPath = getBlobPath(path);
+
+ if (log.isDebugEnabled()) {
+ log.debug("List all '{}'", blobPath);
+ }
+
+ try {
+ return client.listDir(blobPath);
+ } catch (AzureBlobException e) {
+ throw new IOException("Failed to list directory " + blobPath, e);
+ }
+ }
+
+ @Override
+ public IndexInput openInput(URI dirPath, String fileName, IOContext ctx) throws IOException {
+ Objects.requireNonNull(dirPath, "cannot open input with a null URI");
+ Objects.requireNonNull(fileName, "cannot open input with a null fileName");
+
+ String base = getBlobPath(dirPath);
+ String blobPath = base.endsWith("/") ? base + fileName : base + "/" + fileName;
+
+ if (log.isDebugEnabled()) {
+ log.debug("Open input '{}'", blobPath);
+ }
+
+ try {
+ return new AzureBlobIndexInput(blobPath, client, client.length(blobPath));
+ } catch (AzureBlobException e) {
+ throw new IOException("Failed to open input stream for " + blobPath, e);
+ }
+ }
+
+ @Override
+ public OutputStream createOutput(URI path) throws IOException {
+ Objects.requireNonNull(path, "cannot create output with a null URI");
+
+ String blobPath = getBlobPath(path);
+
+ if (log.isDebugEnabled()) {
+ log.debug("Create output '{}'", blobPath);
+ }
+
+ try {
+ return client.pushStream(blobPath);
+ } catch (AzureBlobException e) {
+ throw new IOException("Failed to create output stream for " + blobPath, e);
+ }
+ }
+
+ @Override
+ public void copyIndexFileFrom(
+ Directory sourceDir, String sourceFileName, URI dest, String destFileName)
+ throws IOException {
+ Objects.requireNonNull(sourceDir, "cannot copy with a null sourceDir");
+ Objects.requireNonNull(sourceFileName, "cannot copy with a null sourceFileName");
+ Objects.requireNonNull(dest, "cannot copy with a null dest");
+
+ String destPath = getBlobPath(dest);
+
+ String blobPath = destPath.endsWith("/") ? destPath + destFileName : destPath;
Review Comment:
I think something is wrong here. We fully drop `destFileName` when `destPath` does not end with a /
##########
solr/modules/azure-blob-repository/src/java/org/apache/solr/azureblob/AzureBlobStorageClient.java:
##########
@@ -0,0 +1,529 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.azureblob;
+
+import com.azure.core.credential.TokenCredential;
+import com.azure.identity.DefaultAzureCredentialBuilder;
+import com.azure.storage.blob.BlobClient;
+import com.azure.storage.blob.BlobContainerClient;
+import com.azure.storage.blob.BlobServiceClient;
+import com.azure.storage.blob.BlobServiceClientBuilder;
+import com.azure.storage.blob.models.BlobItem;
+import com.azure.storage.blob.models.BlobStorageException;
+import com.azure.storage.blob.models.ListBlobsOptions;
+import com.google.common.annotations.VisibleForTesting;
+import java.io.ByteArrayInputStream;
+import java.io.FilterInputStream;
+import java.io.InputStream;
+import java.io.OutputStream;
+import java.lang.invoke.MethodHandles;
+import java.util.Collection;
+import java.util.HashSet;
+import java.util.Locale;
+import java.util.Set;
+import java.util.stream.Collectors;
+import org.apache.solr.common.util.ResumableInputStream;
+import org.apache.solr.common.util.StrUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Creates a {@link BlobServiceClient} for communicating with Azure Blob Storage. Utilizes the
+ * default Azure credential provider chain.
+ */
+public class AzureBlobStorageClient {
+
+ private static final Logger log = LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+
+ static final String BLOB_FILE_PATH_DELIMITER = "/";
+ private static final int HTTP_NOT_FOUND = 404;
+ private static final int HTTP_CONFLICT = 409;
+ private static final int SKIP_BUFFER_SIZE = 8192;
+ private static final int DELETE_BATCH_SIZE = 1000;
+
+ private static final com.azure.core.http.HttpClient SHARED_HTTP_CLIENT =
+ new com.azure.core.http.okhttp.OkHttpAsyncHttpClientBuilder().build();
+
+ private final BlobContainerClient containerClient;
+
+ AzureBlobStorageClient(
+ String containerName,
+ String connectionString,
+ String endpoint,
+ String accountName,
+ String accountKey,
+ String sasToken,
+ String tenantId,
+ String clientId,
+ String clientSecret) {
+ this(
+ createInternalClient(
+ connectionString,
+ endpoint,
+ accountName,
+ accountKey,
+ sasToken,
+ tenantId,
+ clientId,
+ clientSecret),
+ containerName);
+ }
+
+ @VisibleForTesting
+ AzureBlobStorageClient(BlobServiceClient blobServiceClient, String containerName) {
+ this.containerClient = blobServiceClient.getBlobContainerClient(containerName);
+ try {
+ containerClient.create();
+ } catch (BlobStorageException e) {
+ if (e.getStatusCode() != HTTP_CONFLICT) {
+ throw e;
+ }
+ }
+ }
+
+ private static BlobServiceClient createInternalClient(
+ String connectionString,
+ String endpoint,
+ String accountName,
+ String accountKey,
+ String sasToken,
+ String tenantId,
+ String clientId,
+ String clientSecret) {
+
+ BlobServiceClientBuilder builder = new BlobServiceClientBuilder();
+ builder.httpClient(SHARED_HTTP_CLIENT);
+
+ if (StrUtils.isNotNullOrEmpty(connectionString)) {
+ builder.connectionString(connectionString);
+ } else if (StrUtils.isNotNullOrEmpty(endpoint)) {
+ builder.endpoint(endpoint);
+ if (StrUtils.isNotNullOrEmpty(accountName) && StrUtils.isNotNullOrEmpty(accountKey)) {
+ builder.credential(
+ new com.azure.storage.common.StorageSharedKeyCredential(accountName, accountKey));
+ } else if (StrUtils.isNotNullOrEmpty(sasToken)) {
+ builder.sasToken(sasToken);
+ } else {
+ TokenCredential credential = new DefaultAzureCredentialBuilder().tenantId(tenantId).build();
+ builder.credential(credential);
+ }
+ } else {
+ throw new IllegalArgumentException("Either connectionString or endpoint must be provided");
+ }
+
+ return builder.buildClient();
+ }
+
+ void createDirectory(String path) throws AzureBlobException {
+ String sanitizedDirPath = sanitizedDirPath(path);
+
+ if (!pathExists(sanitizedDirPath)) {
+ String parent = getParentDirectory(sanitizedDirPath);
+ if (!parent.isEmpty() && !parent.equals(BLOB_FILE_PATH_DELIMITER)) {
+ createDirectory(parent);
+ }
+
+ try {
+ BlobClient blobClient = containerClient.getBlobClient(sanitizedDirPath);
+ blobClient.upload(new ByteArrayInputStream(new byte[0]), 0, true);
+ java.util.Map<String, String> metadata = new java.util.HashMap<>();
+ metadata.put("hdi_isfolder", "true");
+ blobClient.setMetadata(metadata);
+ } catch (BlobStorageException e) {
+ throw handleBlobException(e);
+ }
+ }
+ }
+
+ void delete(Collection<String> paths) throws AzureBlobException {
+ Set<String> entries = new HashSet<>();
+ for (String path : paths) {
+ entries.add(sanitizedFilePath(path));
+ }
+ deleteBlobs(entries);
+ }
+
+ void deleteDirectory(String path) throws AzureBlobException {
+ path = sanitizedDirPath(path);
+
+ Set<String> entries = listAll(path);
+ if (pathExists(path)) {
+ entries.add(path);
+ }
+
+ deleteBlobs(entries);
+ }
+
+ String[] listDir(String path) throws AzureBlobException {
+ path = sanitizedDirPath(path);
+
+ try {
+ ListBlobsOptions options = new ListBlobsOptions().setPrefix(path).setMaxResultsPerPage(1000);
+
+ final String finalPath = path;
+ return containerClient.listBlobs(options, null).stream()
+ .map(BlobItem::getName)
+ .filter(s -> s.startsWith(finalPath))
+ .map(s -> s.substring(finalPath.length()))
+ .filter(s -> !s.isEmpty())
+ .filter(
+ s -> {
+ int slashIndex = s.indexOf(BLOB_FILE_PATH_DELIMITER);
+ return slashIndex == -1 || slashIndex == s.length() - 1;
+ })
+ .toArray(String[]::new);
+ } catch (BlobStorageException e) {
+ throw handleBlobException(e);
+ }
+ }
+
+ boolean pathExists(String path) throws AzureBlobException {
+ final String blobPath = sanitizedPath(path);
+
+ if (blobPath.isEmpty() || BLOB_FILE_PATH_DELIMITER.equals(blobPath)) {
+ return true;
+ }
+
+ try {
+ BlobClient blobClient = containerClient.getBlobClient(blobPath);
+ return blobClient.exists();
+ } catch (BlobStorageException e) {
+ throw handleBlobException(e);
+ }
+ }
+
+ boolean isDirectory(String path) throws AzureBlobException {
+ final String dirPrefix = sanitizedDirPath(path);
+
+ try {
+ ListBlobsOptions options =
+ new ListBlobsOptions().setPrefix(dirPrefix).setMaxResultsPerPage(1);
+ if (containerClient.listBlobs(options, null).iterator().hasNext()) {
+ return true;
+ }
+
+ BlobClient markerClient = containerClient.getBlobClient(dirPrefix);
+ if (markerClient.exists()) {
+ long size = markerClient.getProperties().getBlobSize();
+ if (size == 0) {
+ return true;
+ }
+ java.util.Map<String, String> md = markerClient.getProperties().getMetadata();
+ return md != null && md.containsKey("hdi_isfolder");
+ }
+
+ return false;
+ } catch (BlobStorageException e) {
+ throw handleBlobException(e);
+ }
+ }
+
+ long length(String path) throws AzureBlobException {
+ String blobPath = sanitizedFilePath(path);
+ try {
+ BlobClient blobClient = containerClient.getBlobClient(blobPath);
+ return blobClient.getProperties().getBlobSize();
+ } catch (BlobStorageException e) {
+ throw handleBlobException(e);
+ }
+ }
+
+ InputStream pullStream(String path) throws AzureBlobException {
+ final String blobPath = sanitizedFilePath(path);
+
+ try {
+ BlobClient blobClient = containerClient.getBlobClient(blobPath);
+ final long contentLength = blobClient.getProperties().getBlobSize();
Review Comment:
This is under optimized. We do two queries to open the stream. My understanding is invoking `blobClient.getProperties()` issues a HEAD query.
The result of `blobClient.openInputStream()` also contains the length. This may be retrieved with `blobInputStream.getProperties().getBlobSize()`
##########
solr/modules/azure-blob-repository/src/java/org/apache/solr/azureblob/AzureBlobStorageClient.java:
##########
@@ -0,0 +1,529 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.azureblob;
+
+import com.azure.core.credential.TokenCredential;
+import com.azure.identity.DefaultAzureCredentialBuilder;
+import com.azure.storage.blob.BlobClient;
+import com.azure.storage.blob.BlobContainerClient;
+import com.azure.storage.blob.BlobServiceClient;
+import com.azure.storage.blob.BlobServiceClientBuilder;
+import com.azure.storage.blob.models.BlobItem;
+import com.azure.storage.blob.models.BlobStorageException;
+import com.azure.storage.blob.models.ListBlobsOptions;
+import com.google.common.annotations.VisibleForTesting;
+import java.io.ByteArrayInputStream;
+import java.io.FilterInputStream;
+import java.io.InputStream;
+import java.io.OutputStream;
+import java.lang.invoke.MethodHandles;
+import java.util.Collection;
+import java.util.HashSet;
+import java.util.Locale;
+import java.util.Set;
+import java.util.stream.Collectors;
+import org.apache.solr.common.util.ResumableInputStream;
+import org.apache.solr.common.util.StrUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Creates a {@link BlobServiceClient} for communicating with Azure Blob Storage. Utilizes the
+ * default Azure credential provider chain.
+ */
+public class AzureBlobStorageClient {
+
+ private static final Logger log = LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+
+ static final String BLOB_FILE_PATH_DELIMITER = "/";
+ private static final int HTTP_NOT_FOUND = 404;
+ private static final int HTTP_CONFLICT = 409;
+ private static final int SKIP_BUFFER_SIZE = 8192;
+ private static final int DELETE_BATCH_SIZE = 1000;
+
+ private static final com.azure.core.http.HttpClient SHARED_HTTP_CLIENT =
+ new com.azure.core.http.okhttp.OkHttpAsyncHttpClientBuilder().build();
+
+ private final BlobContainerClient containerClient;
+
+ AzureBlobStorageClient(
+ String containerName,
+ String connectionString,
+ String endpoint,
+ String accountName,
+ String accountKey,
+ String sasToken,
+ String tenantId,
+ String clientId,
+ String clientSecret) {
+ this(
+ createInternalClient(
+ connectionString,
+ endpoint,
+ accountName,
+ accountKey,
+ sasToken,
+ tenantId,
+ clientId,
+ clientSecret),
+ containerName);
+ }
+
+ @VisibleForTesting
+ AzureBlobStorageClient(BlobServiceClient blobServiceClient, String containerName) {
+ this.containerClient = blobServiceClient.getBlobContainerClient(containerName);
+ try {
+ containerClient.create();
+ } catch (BlobStorageException e) {
+ if (e.getStatusCode() != HTTP_CONFLICT) {
+ throw e;
+ }
+ }
+ }
+
+ private static BlobServiceClient createInternalClient(
+ String connectionString,
+ String endpoint,
+ String accountName,
+ String accountKey,
+ String sasToken,
+ String tenantId,
+ String clientId,
+ String clientSecret) {
+
+ BlobServiceClientBuilder builder = new BlobServiceClientBuilder();
+ builder.httpClient(SHARED_HTTP_CLIENT);
+
+ if (StrUtils.isNotNullOrEmpty(connectionString)) {
+ builder.connectionString(connectionString);
+ } else if (StrUtils.isNotNullOrEmpty(endpoint)) {
+ builder.endpoint(endpoint);
+ if (StrUtils.isNotNullOrEmpty(accountName) && StrUtils.isNotNullOrEmpty(accountKey)) {
+ builder.credential(
+ new com.azure.storage.common.StorageSharedKeyCredential(accountName, accountKey));
+ } else if (StrUtils.isNotNullOrEmpty(sasToken)) {
+ builder.sasToken(sasToken);
+ } else {
+ TokenCredential credential = new DefaultAzureCredentialBuilder().tenantId(tenantId).build();
+ builder.credential(credential);
+ }
+ } else {
+ throw new IllegalArgumentException("Either connectionString or endpoint must be provided");
+ }
+
+ return builder.buildClient();
+ }
+
+ void createDirectory(String path) throws AzureBlobException {
+ String sanitizedDirPath = sanitizedDirPath(path);
+
+ if (!pathExists(sanitizedDirPath)) {
+ String parent = getParentDirectory(sanitizedDirPath);
+ if (!parent.isEmpty() && !parent.equals(BLOB_FILE_PATH_DELIMITER)) {
+ createDirectory(parent);
+ }
+
+ try {
+ BlobClient blobClient = containerClient.getBlobClient(sanitizedDirPath);
+ blobClient.upload(new ByteArrayInputStream(new byte[0]), 0, true);
+ java.util.Map<String, String> metadata = new java.util.HashMap<>();
+ metadata.put("hdi_isfolder", "true");
+ blobClient.setMetadata(metadata);
+ } catch (BlobStorageException e) {
+ throw handleBlobException(e);
+ }
+ }
+ }
+
+ void delete(Collection<String> paths) throws AzureBlobException {
+ Set<String> entries = new HashSet<>();
+ for (String path : paths) {
+ entries.add(sanitizedFilePath(path));
+ }
+ deleteBlobs(entries);
+ }
+
+ void deleteDirectory(String path) throws AzureBlobException {
+ path = sanitizedDirPath(path);
+
+ Set<String> entries = listAll(path);
+ if (pathExists(path)) {
+ entries.add(path);
+ }
+
+ deleteBlobs(entries);
+ }
+
+ String[] listDir(String path) throws AzureBlobException {
+ path = sanitizedDirPath(path);
+
+ try {
+ ListBlobsOptions options = new ListBlobsOptions().setPrefix(path).setMaxResultsPerPage(1000);
+
+ final String finalPath = path;
+ return containerClient.listBlobs(options, null).stream()
+ .map(BlobItem::getName)
+ .filter(s -> s.startsWith(finalPath))
+ .map(s -> s.substring(finalPath.length()))
+ .filter(s -> !s.isEmpty())
+ .filter(
+ s -> {
+ int slashIndex = s.indexOf(BLOB_FILE_PATH_DELIMITER);
+ return slashIndex == -1 || slashIndex == s.length() - 1;
+ })
+ .toArray(String[]::new);
+ } catch (BlobStorageException e) {
+ throw handleBlobException(e);
+ }
+ }
+
+ boolean pathExists(String path) throws AzureBlobException {
+ final String blobPath = sanitizedPath(path);
+
+ if (blobPath.isEmpty() || BLOB_FILE_PATH_DELIMITER.equals(blobPath)) {
+ return true;
+ }
+
+ try {
+ BlobClient blobClient = containerClient.getBlobClient(blobPath);
+ return blobClient.exists();
+ } catch (BlobStorageException e) {
+ throw handleBlobException(e);
+ }
+ }
+
+ boolean isDirectory(String path) throws AzureBlobException {
+ final String dirPrefix = sanitizedDirPath(path);
+
+ try {
+ ListBlobsOptions options =
+ new ListBlobsOptions().setPrefix(dirPrefix).setMaxResultsPerPage(1);
+ if (containerClient.listBlobs(options, null).iterator().hasNext()) {
+ return true;
+ }
+
+ BlobClient markerClient = containerClient.getBlobClient(dirPrefix);
+ if (markerClient.exists()) {
+ long size = markerClient.getProperties().getBlobSize();
+ if (size == 0) {
+ return true;
+ }
+ java.util.Map<String, String> md = markerClient.getProperties().getMetadata();
+ return md != null && md.containsKey("hdi_isfolder");
+ }
+
+ return false;
+ } catch (BlobStorageException e) {
+ throw handleBlobException(e);
+ }
+ }
+
+ long length(String path) throws AzureBlobException {
+ String blobPath = sanitizedFilePath(path);
+ try {
+ BlobClient blobClient = containerClient.getBlobClient(blobPath);
+ return blobClient.getProperties().getBlobSize();
+ } catch (BlobStorageException e) {
+ throw handleBlobException(e);
+ }
+ }
+
+ InputStream pullStream(String path) throws AzureBlobException {
+ final String blobPath = sanitizedFilePath(path);
+
+ try {
+ BlobClient blobClient = containerClient.getBlobClient(blobPath);
+ final long contentLength = blobClient.getProperties().getBlobSize();
+
+ if (contentLength == 0) {
+ return new ByteArrayInputStream(new byte[0]);
+ }
+
+ InputStream initial = new IdempotentCloseInputStream(blobClient.openInputStream());
+
+ return new ResumableInputStream(
+ initial,
+ bytesRead -> {
+ if (contentLength > 0 && bytesRead >= contentLength) {
+ return null;
+ }
+ try {
+ long remaining =
+ contentLength > 0 ? Math.max(0, contentLength - bytesRead) : Long.MAX_VALUE;
+ return pullRangeStream(path, bytesRead, remaining);
+ } catch (AzureBlobException e) {
+ throw new RuntimeException(e);
+ }
+ });
+ } catch (BlobStorageException e) {
+ throw handleBlobException(e);
+ }
+ }
+
+ InputStream pullRangeStream(String path, long offset, long length) throws AzureBlobException {
+ final String blobPath = sanitizedFilePath(path);
+ try {
+ BlobClient blobClient = containerClient.getBlobClient(blobPath);
+ com.azure.storage.blob.models.BlobRange range =
+ new com.azure.storage.blob.models.BlobRange(offset, length);
+ return new IdempotentCloseInputStream(blobClient.openInputStream(range, null));
+ } catch (BlobStorageException e) {
+ throw handleBlobException(e);
+ }
+ }
+
+ private static final class IdempotentCloseInputStream extends FilterInputStream {
+ private boolean closed;
+
+ IdempotentCloseInputStream(InputStream in) {
+ super(in);
+ this.closed = false;
+ }
+
+ @Override
+ public int read() throws java.io.IOException {
+ if (closed) {
+ throw new java.io.IOException("Stream is already closed");
+ }
+ try {
+ return super.read();
+ } catch (RuntimeException re) {
+ if (isAlreadyClosed(re)) {
+ throw new java.io.IOException("Stream is already closed", re);
+ }
+ throw re;
+ }
+ }
+
+ @Override
+ public int read(byte[] b, int off, int len) throws java.io.IOException {
+ if (closed) {
+ throw new java.io.IOException("Stream is already closed");
+ }
+ try {
+ return super.read(b, off, len);
+ } catch (RuntimeException re) {
+ if (isAlreadyClosed(re)) {
+ throw new java.io.IOException("Stream is already closed", re);
+ }
+ throw re;
+ }
+ }
+
+ @Override
+ public void close() throws java.io.IOException {
+ if (closed) {
+ return;
+ }
+ try {
+ super.close();
+ } catch (java.io.IOException e) {
+ String msg = e.getMessage();
+ if (msg == null || !msg.toLowerCase(java.util.Locale.ROOT).contains("already closed")) {
+ throw e;
+ }
+ // swallow "already closed" to make close idempotent
+ } finally {
+ closed = true;
+ }
+ }
+
+ @Override
+ public long skip(long n) throws java.io.IOException {
+ if (closed) {
+ throw new java.io.IOException("Stream is already closed");
+ }
+ if (n <= 0) {
+ return 0L;
+ }
+ long remaining = n;
+ byte[] discard = new byte[SKIP_BUFFER_SIZE];
+ try {
+ while (remaining > 0) {
+ int toRead = (int) Math.min(discard.length, remaining);
+ int read = super.read(discard, 0, toRead);
+ if (read < 0) {
+ break;
+ }
+ remaining -= read;
+ }
+ return n - remaining;
+ } catch (RuntimeException re) {
+ throw new java.io.IOException(re);
+ }
+ }
+
+ private static boolean isAlreadyClosed(Throwable t) {
+ String msg = t.getMessage();
+ return msg != null && msg.toLowerCase(java.util.Locale.ROOT).contains("already closed");
+ }
+ }
+
+ OutputStream pushStream(String path) throws AzureBlobException {
+ path = sanitizedFilePath(path);
+
+ if (!parentDirectoryExist(path)) {
+ String parentDirectory = getParentDirectory(path);
+ if (!parentDirectory.isEmpty() && !parentDirectory.equals(BLOB_FILE_PATH_DELIMITER)) {
+ createDirectory(parentDirectory);
+ }
+ }
+
+ try {
+ BlobClient blobClient = containerClient.getBlobClient(path);
+ return new AzureBlobOutputStream(blobClient, path);
+ } catch (BlobStorageException e) {
+ throw handleBlobException(e);
+ }
+ }
+
+ void close() {}
+
+ @VisibleForTesting
+ void deleteContainerForTests() {
+ try {
+ containerClient.delete();
+ } catch (BlobStorageException e) {
+ if (e.getStatusCode() != HTTP_NOT_FOUND) {
+ throw e;
+ }
+ }
+ }
+
+ private Collection<String> deleteBlobs(Collection<String> paths) throws AzureBlobException {
+ try {
+ return deleteBlobs(paths, DELETE_BATCH_SIZE);
Review Comment:
`DELETE_BATCH_SIZE` has no impact (unused parameter). Does it come from S3 code?
##########
solr/modules/azure-blob-repository/src/java/org/apache/solr/azureblob/AzureBlobOutputStream.java:
##########
@@ -0,0 +1,269 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.azureblob;
+
+import com.azure.storage.blob.BlobClient;
+import com.azure.storage.blob.models.BlobStorageException;
+import com.azure.storage.blob.specialized.BlockBlobClient;
+import java.io.ByteArrayInputStream;
+import java.io.IOException;
+import java.io.OutputStream;
+import java.lang.invoke.MethodHandles;
+import java.nio.ByteBuffer;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Base64;
+import java.util.List;
+import java.util.UUID;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * OutputStream implementation for Azure Blob Storage using block blobs. Supports chunked uploads
+ * for large files.
+ */
+public class AzureBlobOutputStream extends OutputStream {
+ private static final Logger log = LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+
+ static final int BLOCK_SIZE = 4 * 1024 * 1024;
+
+ private final BlobClient blobClient;
+ private final String blobPath;
+ private volatile boolean closed;
+ private final ByteBuffer buffer;
+ private BlockUpload blockUpload;
+ private boolean committed;
+
+ public AzureBlobOutputStream(BlobClient blobClient, String blobPath) {
+ this.blobClient = blobClient;
+ this.blobPath = blobPath;
+ this.closed = false;
+ this.buffer = ByteBuffer.allocate(BLOCK_SIZE);
+ this.blockUpload = null;
+ this.committed = false;
+
+ if (log.isDebugEnabled()) {
+ log.debug("Created BlobOutputStream for blobPath '{}'", blobPath);
+ }
+ }
+
+ @Override
+ public void write(int b) throws IOException {
+ if (closed) {
+ throw new IOException("Stream closed");
+ }
+
+ buffer.put((byte) b);
+
+ if (!buffer.hasRemaining()) {
+ uploadBlock();
+ }
+ }
+
+ @Override
+ public void write(byte[] b, int off, int len) throws IOException {
+ if (closed) {
+ throw new IOException("Stream closed");
+ }
+
+ if (outOfRange(off, b.length) || len < 0 || outOfRange(off + len, b.length)) {
+ throw new IndexOutOfBoundsException();
+ } else if (len == 0) {
+ return;
+ }
+
+ int currentOffset = off;
+ int lenRemaining = len;
+ while (buffer.remaining() < lenRemaining) {
+ int firstPart = buffer.remaining();
+ buffer.put(b, currentOffset, firstPart);
+ uploadBlock();
+
+ currentOffset += firstPart;
+ lenRemaining -= firstPart;
+ }
+ if (lenRemaining > 0) {
+ buffer.put(b, currentOffset, lenRemaining);
+ }
+ }
+
+ private static boolean outOfRange(int off, int len) {
+ return off < 0 || off > len;
+ }
+
+ private void uploadBlock() throws IOException {
+ int size = buffer.position() - buffer.arrayOffset();
+
+ if (size == 0) {
+ return;
+ }
+
+ if (blockUpload == null) {
+ if (log.isDebugEnabled()) {
+ log.debug("New block upload for blobPath '{}'", blobPath);
+ }
+
+ blockUpload = newBlockUpload();
+ }
+
+ try (ByteArrayInputStream inputStream =
+ new ByteArrayInputStream(buffer.array(), buffer.arrayOffset(), size)) {
+ blockUpload.uploadBlock(inputStream, size);
+ } catch (BlobStorageException e) {
+ if (blockUpload != null) {
+ blockUpload.abort();
+ if (log.isDebugEnabled()) {
+ log.debug("Block upload aborted for blobPath '{}'.", blobPath);
+ }
+ }
+
+ throw new IOException(
+ "Failed to upload block", AzureBlobStorageClient.handleBlobException(e));
+ }
+
+ buffer.clear();
+ }
+
+ @Override
+ public void flush() throws IOException {
+ if (closed) {
+ throw new IOException("Stream closed");
+ }
+
+ if (buffer.position() - buffer.arrayOffset() > 0) {
+ uploadBlock();
+ }
+
+ if (blockUpload != null) {
+ blockUpload.complete();
Review Comment:
This invokes `blockBlobClient.commitBlockList(blockIds)`. My understanding is the nothing can then be uploaded. This breaks the `flush()` contract.
##########
solr/modules/azure-blob-repository/src/java/org/apache/solr/azureblob/AzureBlobStorageClient.java:
##########
@@ -0,0 +1,529 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.azureblob;
+
+import com.azure.core.credential.TokenCredential;
+import com.azure.identity.DefaultAzureCredentialBuilder;
+import com.azure.storage.blob.BlobClient;
+import com.azure.storage.blob.BlobContainerClient;
+import com.azure.storage.blob.BlobServiceClient;
+import com.azure.storage.blob.BlobServiceClientBuilder;
+import com.azure.storage.blob.models.BlobItem;
+import com.azure.storage.blob.models.BlobStorageException;
+import com.azure.storage.blob.models.ListBlobsOptions;
+import com.google.common.annotations.VisibleForTesting;
+import java.io.ByteArrayInputStream;
+import java.io.FilterInputStream;
+import java.io.InputStream;
+import java.io.OutputStream;
+import java.lang.invoke.MethodHandles;
+import java.util.Collection;
+import java.util.HashSet;
+import java.util.Locale;
+import java.util.Set;
+import java.util.stream.Collectors;
+import org.apache.solr.common.util.ResumableInputStream;
+import org.apache.solr.common.util.StrUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Creates a {@link BlobServiceClient} for communicating with Azure Blob Storage. Utilizes the
+ * default Azure credential provider chain.
+ */
+public class AzureBlobStorageClient {
+
+ private static final Logger log = LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+
+ static final String BLOB_FILE_PATH_DELIMITER = "/";
+ private static final int HTTP_NOT_FOUND = 404;
+ private static final int HTTP_CONFLICT = 409;
+ private static final int SKIP_BUFFER_SIZE = 8192;
+ private static final int DELETE_BATCH_SIZE = 1000;
+
+ private static final com.azure.core.http.HttpClient SHARED_HTTP_CLIENT =
+ new com.azure.core.http.okhttp.OkHttpAsyncHttpClientBuilder().build();
+
+ private final BlobContainerClient containerClient;
+
+ AzureBlobStorageClient(
+ String containerName,
+ String connectionString,
+ String endpoint,
+ String accountName,
+ String accountKey,
+ String sasToken,
+ String tenantId,
+ String clientId,
+ String clientSecret) {
+ this(
+ createInternalClient(
+ connectionString,
+ endpoint,
+ accountName,
+ accountKey,
+ sasToken,
+ tenantId,
+ clientId,
+ clientSecret),
+ containerName);
+ }
+
+ @VisibleForTesting
+ AzureBlobStorageClient(BlobServiceClient blobServiceClient, String containerName) {
+ this.containerClient = blobServiceClient.getBlobContainerClient(containerName);
+ try {
+ containerClient.create();
+ } catch (BlobStorageException e) {
+ if (e.getStatusCode() != HTTP_CONFLICT) {
+ throw e;
+ }
+ }
+ }
+
+ private static BlobServiceClient createInternalClient(
+ String connectionString,
+ String endpoint,
+ String accountName,
+ String accountKey,
+ String sasToken,
+ String tenantId,
+ String clientId,
+ String clientSecret) {
+
+ BlobServiceClientBuilder builder = new BlobServiceClientBuilder();
+ builder.httpClient(SHARED_HTTP_CLIENT);
+
+ if (StrUtils.isNotNullOrEmpty(connectionString)) {
+ builder.connectionString(connectionString);
+ } else if (StrUtils.isNotNullOrEmpty(endpoint)) {
+ builder.endpoint(endpoint);
+ if (StrUtils.isNotNullOrEmpty(accountName) && StrUtils.isNotNullOrEmpty(accountKey)) {
+ builder.credential(
+ new com.azure.storage.common.StorageSharedKeyCredential(accountName, accountKey));
+ } else if (StrUtils.isNotNullOrEmpty(sasToken)) {
+ builder.sasToken(sasToken);
+ } else {
+ TokenCredential credential = new DefaultAzureCredentialBuilder().tenantId(tenantId).build();
+ builder.credential(credential);
+ }
+ } else {
+ throw new IllegalArgumentException("Either connectionString or endpoint must be provided");
+ }
+
+ return builder.buildClient();
+ }
+
+ void createDirectory(String path) throws AzureBlobException {
+ String sanitizedDirPath = sanitizedDirPath(path);
+
+ if (!pathExists(sanitizedDirPath)) {
+ String parent = getParentDirectory(sanitizedDirPath);
+ if (!parent.isEmpty() && !parent.equals(BLOB_FILE_PATH_DELIMITER)) {
+ createDirectory(parent);
+ }
+
+ try {
+ BlobClient blobClient = containerClient.getBlobClient(sanitizedDirPath);
+ blobClient.upload(new ByteArrayInputStream(new byte[0]), 0, true);
+ java.util.Map<String, String> metadata = new java.util.HashMap<>();
+ metadata.put("hdi_isfolder", "true");
+ blobClient.setMetadata(metadata);
+ } catch (BlobStorageException e) {
+ throw handleBlobException(e);
+ }
+ }
+ }
+
+ void delete(Collection<String> paths) throws AzureBlobException {
+ Set<String> entries = new HashSet<>();
+ for (String path : paths) {
+ entries.add(sanitizedFilePath(path));
+ }
+ deleteBlobs(entries);
+ }
+
+ void deleteDirectory(String path) throws AzureBlobException {
+ path = sanitizedDirPath(path);
+
+ Set<String> entries = listAll(path);
+ if (pathExists(path)) {
+ entries.add(path);
+ }
+
+ deleteBlobs(entries);
+ }
+
+ String[] listDir(String path) throws AzureBlobException {
+ path = sanitizedDirPath(path);
+
+ try {
+ ListBlobsOptions options = new ListBlobsOptions().setPrefix(path).setMaxResultsPerPage(1000);
+
+ final String finalPath = path;
+ return containerClient.listBlobs(options, null).stream()
+ .map(BlobItem::getName)
+ .filter(s -> s.startsWith(finalPath))
+ .map(s -> s.substring(finalPath.length()))
+ .filter(s -> !s.isEmpty())
+ .filter(
+ s -> {
+ int slashIndex = s.indexOf(BLOB_FILE_PATH_DELIMITER);
+ return slashIndex == -1 || slashIndex == s.length() - 1;
+ })
+ .toArray(String[]::new);
+ } catch (BlobStorageException e) {
+ throw handleBlobException(e);
+ }
+ }
+
+ boolean pathExists(String path) throws AzureBlobException {
+ final String blobPath = sanitizedPath(path);
+
+ if (blobPath.isEmpty() || BLOB_FILE_PATH_DELIMITER.equals(blobPath)) {
+ return true;
+ }
+
+ try {
+ BlobClient blobClient = containerClient.getBlobClient(blobPath);
+ return blobClient.exists();
+ } catch (BlobStorageException e) {
+ throw handleBlobException(e);
+ }
+ }
+
+ boolean isDirectory(String path) throws AzureBlobException {
+ final String dirPrefix = sanitizedDirPath(path);
+
+ try {
+ ListBlobsOptions options =
+ new ListBlobsOptions().setPrefix(dirPrefix).setMaxResultsPerPage(1);
+ if (containerClient.listBlobs(options, null).iterator().hasNext()) {
+ return true;
+ }
+
+ BlobClient markerClient = containerClient.getBlobClient(dirPrefix);
+ if (markerClient.exists()) {
+ long size = markerClient.getProperties().getBlobSize();
+ if (size == 0) {
+ return true;
+ }
+ java.util.Map<String, String> md = markerClient.getProperties().getMetadata();
+ return md != null && md.containsKey("hdi_isfolder");
+ }
+
+ return false;
+ } catch (BlobStorageException e) {
+ throw handleBlobException(e);
+ }
+ }
+
+ long length(String path) throws AzureBlobException {
+ String blobPath = sanitizedFilePath(path);
+ try {
+ BlobClient blobClient = containerClient.getBlobClient(blobPath);
+ return blobClient.getProperties().getBlobSize();
+ } catch (BlobStorageException e) {
+ throw handleBlobException(e);
+ }
+ }
+
+ InputStream pullStream(String path) throws AzureBlobException {
+ final String blobPath = sanitizedFilePath(path);
+
+ try {
+ BlobClient blobClient = containerClient.getBlobClient(blobPath);
+ final long contentLength = blobClient.getProperties().getBlobSize();
+
+ if (contentLength == 0) {
+ return new ByteArrayInputStream(new byte[0]);
+ }
+
+ InputStream initial = new IdempotentCloseInputStream(blobClient.openInputStream());
+
+ return new ResumableInputStream(
+ initial,
+ bytesRead -> {
+ if (contentLength > 0 && bytesRead >= contentLength) {
+ return null;
+ }
+ try {
+ long remaining =
+ contentLength > 0 ? Math.max(0, contentLength - bytesRead) : Long.MAX_VALUE;
+ return pullRangeStream(path, bytesRead, remaining);
+ } catch (AzureBlobException e) {
+ throw new RuntimeException(e);
+ }
+ });
+ } catch (BlobStorageException e) {
+ throw handleBlobException(e);
+ }
+ }
+
+ InputStream pullRangeStream(String path, long offset, long length) throws AzureBlobException {
+ final String blobPath = sanitizedFilePath(path);
+ try {
+ BlobClient blobClient = containerClient.getBlobClient(blobPath);
+ com.azure.storage.blob.models.BlobRange range =
+ new com.azure.storage.blob.models.BlobRange(offset, length);
+ return new IdempotentCloseInputStream(blobClient.openInputStream(range, null));
+ } catch (BlobStorageException e) {
+ throw handleBlobException(e);
+ }
+ }
+
+ private static final class IdempotentCloseInputStream extends FilterInputStream {
+ private boolean closed;
+
+ IdempotentCloseInputStream(InputStream in) {
+ super(in);
+ this.closed = false;
+ }
+
+ @Override
+ public int read() throws java.io.IOException {
+ if (closed) {
+ throw new java.io.IOException("Stream is already closed");
+ }
+ try {
+ return super.read();
+ } catch (RuntimeException re) {
+ if (isAlreadyClosed(re)) {
+ throw new java.io.IOException("Stream is already closed", re);
+ }
+ throw re;
+ }
+ }
+
+ @Override
+ public int read(byte[] b, int off, int len) throws java.io.IOException {
+ if (closed) {
+ throw new java.io.IOException("Stream is already closed");
+ }
+ try {
+ return super.read(b, off, len);
+ } catch (RuntimeException re) {
+ if (isAlreadyClosed(re)) {
+ throw new java.io.IOException("Stream is already closed", re);
+ }
+ throw re;
+ }
+ }
+
+ @Override
+ public void close() throws java.io.IOException {
+ if (closed) {
+ return;
+ }
+ try {
+ super.close();
+ } catch (java.io.IOException e) {
+ String msg = e.getMessage();
+ if (msg == null || !msg.toLowerCase(java.util.Locale.ROOT).contains("already closed")) {
+ throw e;
+ }
+ // swallow "already closed" to make close idempotent
+ } finally {
+ closed = true;
+ }
+ }
+
+ @Override
+ public long skip(long n) throws java.io.IOException {
+ if (closed) {
+ throw new java.io.IOException("Stream is already closed");
+ }
+ if (n <= 0) {
+ return 0L;
+ }
+ long remaining = n;
+ byte[] discard = new byte[SKIP_BUFFER_SIZE];
+ try {
+ while (remaining > 0) {
+ int toRead = (int) Math.min(discard.length, remaining);
+ int read = super.read(discard, 0, toRead);
+ if (read < 0) {
+ break;
+ }
+ remaining -= read;
+ }
+ return n - remaining;
+ } catch (RuntimeException re) {
+ throw new java.io.IOException(re);
+ }
+ }
+
+ private static boolean isAlreadyClosed(Throwable t) {
+ String msg = t.getMessage();
+ return msg != null && msg.toLowerCase(java.util.Locale.ROOT).contains("already closed");
+ }
+ }
+
+ OutputStream pushStream(String path) throws AzureBlobException {
+ path = sanitizedFilePath(path);
+
+ if (!parentDirectoryExist(path)) {
+ String parentDirectory = getParentDirectory(path);
+ if (!parentDirectory.isEmpty() && !parentDirectory.equals(BLOB_FILE_PATH_DELIMITER)) {
+ createDirectory(parentDirectory);
+ }
+ }
+
+ try {
+ BlobClient blobClient = containerClient.getBlobClient(path);
+ return new AzureBlobOutputStream(blobClient, path);
+ } catch (BlobStorageException e) {
+ throw handleBlobException(e);
+ }
+ }
+
+ void close() {}
+
+ @VisibleForTesting
+ void deleteContainerForTests() {
+ try {
+ containerClient.delete();
+ } catch (BlobStorageException e) {
+ if (e.getStatusCode() != HTTP_NOT_FOUND) {
+ throw e;
+ }
+ }
+ }
+
+ private Collection<String> deleteBlobs(Collection<String> paths) throws AzureBlobException {
+ try {
+ return deleteBlobs(paths, DELETE_BATCH_SIZE);
+ } catch (BlobStorageException e) {
+ throw handleBlobException(e);
+ }
+ }
+
+ @VisibleForTesting
+ Collection<String> deleteBlobs(Collection<String> entries, int batchSize)
+ throws AzureBlobException {
+ Set<String> deletedPaths = new HashSet<>();
+
+ for (String path : entries) {
Review Comment:
I think it's very ineffective to issue one request per file. We should batch the deletes (I think the API opens this).
##########
solr/modules/azure-blob-repository/src/java/org/apache/solr/azureblob/AzureBlobOutputStream.java:
##########
@@ -0,0 +1,269 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.azureblob;
+
+import com.azure.storage.blob.BlobClient;
+import com.azure.storage.blob.models.BlobStorageException;
+import com.azure.storage.blob.specialized.BlockBlobClient;
+import java.io.ByteArrayInputStream;
+import java.io.IOException;
+import java.io.OutputStream;
+import java.lang.invoke.MethodHandles;
+import java.nio.ByteBuffer;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Base64;
+import java.util.List;
+import java.util.UUID;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * OutputStream implementation for Azure Blob Storage using block blobs. Supports chunked uploads
+ * for large files.
+ */
+public class AzureBlobOutputStream extends OutputStream {
+ private static final Logger log = LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+
+ static final int BLOCK_SIZE = 4 * 1024 * 1024;
+
+ private final BlobClient blobClient;
+ private final String blobPath;
+ private volatile boolean closed;
+ private final ByteBuffer buffer;
+ private BlockUpload blockUpload;
+ private boolean committed;
+
+ public AzureBlobOutputStream(BlobClient blobClient, String blobPath) {
+ this.blobClient = blobClient;
+ this.blobPath = blobPath;
+ this.closed = false;
+ this.buffer = ByteBuffer.allocate(BLOCK_SIZE);
+ this.blockUpload = null;
+ this.committed = false;
+
+ if (log.isDebugEnabled()) {
+ log.debug("Created BlobOutputStream for blobPath '{}'", blobPath);
+ }
+ }
+
+ @Override
+ public void write(int b) throws IOException {
+ if (closed) {
+ throw new IOException("Stream closed");
+ }
+
+ buffer.put((byte) b);
+
+ if (!buffer.hasRemaining()) {
+ uploadBlock();
+ }
+ }
+
+ @Override
+ public void write(byte[] b, int off, int len) throws IOException {
+ if (closed) {
+ throw new IOException("Stream closed");
+ }
+
+ if (outOfRange(off, b.length) || len < 0 || outOfRange(off + len, b.length)) {
+ throw new IndexOutOfBoundsException();
+ } else if (len == 0) {
+ return;
+ }
+
+ int currentOffset = off;
+ int lenRemaining = len;
+ while (buffer.remaining() < lenRemaining) {
+ int firstPart = buffer.remaining();
+ buffer.put(b, currentOffset, firstPart);
+ uploadBlock();
+
+ currentOffset += firstPart;
+ lenRemaining -= firstPart;
+ }
+ if (lenRemaining > 0) {
+ buffer.put(b, currentOffset, lenRemaining);
+ }
+ }
+
+ private static boolean outOfRange(int off, int len) {
+ return off < 0 || off > len;
+ }
+
+ private void uploadBlock() throws IOException {
+ int size = buffer.position() - buffer.arrayOffset();
+
+ if (size == 0) {
+ return;
+ }
+
+ if (blockUpload == null) {
+ if (log.isDebugEnabled()) {
+ log.debug("New block upload for blobPath '{}'", blobPath);
+ }
+
+ blockUpload = newBlockUpload();
+ }
+
+ try (ByteArrayInputStream inputStream =
Review Comment:
I would directly use `ByteBufferContent` (from Azure SDK) instead of an input stream.
##########
solr/modules/azure-blob-repository/src/java/org/apache/solr/azureblob/AzureBlobOutputStream.java:
##########
@@ -0,0 +1,269 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.azureblob;
+
+import com.azure.storage.blob.BlobClient;
+import com.azure.storage.blob.models.BlobStorageException;
+import com.azure.storage.blob.specialized.BlockBlobClient;
+import java.io.ByteArrayInputStream;
+import java.io.IOException;
+import java.io.OutputStream;
+import java.lang.invoke.MethodHandles;
+import java.nio.ByteBuffer;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Base64;
+import java.util.List;
+import java.util.UUID;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * OutputStream implementation for Azure Blob Storage using block blobs. Supports chunked uploads
+ * for large files.
+ */
+public class AzureBlobOutputStream extends OutputStream {
+ private static final Logger log = LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+
+ static final int BLOCK_SIZE = 4 * 1024 * 1024;
+
+ private final BlobClient blobClient;
+ private final String blobPath;
+ private volatile boolean closed;
+ private final ByteBuffer buffer;
+ private BlockUpload blockUpload;
+ private boolean committed;
+
+ public AzureBlobOutputStream(BlobClient blobClient, String blobPath) {
+ this.blobClient = blobClient;
+ this.blobPath = blobPath;
+ this.closed = false;
+ this.buffer = ByteBuffer.allocate(BLOCK_SIZE);
+ this.blockUpload = null;
+ this.committed = false;
+
+ if (log.isDebugEnabled()) {
+ log.debug("Created BlobOutputStream for blobPath '{}'", blobPath);
+ }
+ }
+
+ @Override
+ public void write(int b) throws IOException {
+ if (closed) {
+ throw new IOException("Stream closed");
+ }
+
+ buffer.put((byte) b);
+
+ if (!buffer.hasRemaining()) {
+ uploadBlock();
+ }
+ }
+
+ @Override
+ public void write(byte[] b, int off, int len) throws IOException {
+ if (closed) {
+ throw new IOException("Stream closed");
+ }
+
+ if (outOfRange(off, b.length) || len < 0 || outOfRange(off + len, b.length)) {
+ throw new IndexOutOfBoundsException();
+ } else if (len == 0) {
+ return;
+ }
+
+ int currentOffset = off;
+ int lenRemaining = len;
+ while (buffer.remaining() < lenRemaining) {
+ int firstPart = buffer.remaining();
+ buffer.put(b, currentOffset, firstPart);
+ uploadBlock();
+
+ currentOffset += firstPart;
+ lenRemaining -= firstPart;
+ }
+ if (lenRemaining > 0) {
+ buffer.put(b, currentOffset, lenRemaining);
+ }
+ }
+
+ private static boolean outOfRange(int off, int len) {
+ return off < 0 || off > len;
+ }
+
+ private void uploadBlock() throws IOException {
+ int size = buffer.position() - buffer.arrayOffset();
+
+ if (size == 0) {
+ return;
+ }
+
+ if (blockUpload == null) {
+ if (log.isDebugEnabled()) {
+ log.debug("New block upload for blobPath '{}'", blobPath);
+ }
+
+ blockUpload = newBlockUpload();
+ }
+
+ try (ByteArrayInputStream inputStream =
+ new ByteArrayInputStream(buffer.array(), buffer.arrayOffset(), size)) {
+ blockUpload.uploadBlock(inputStream, size);
+ } catch (BlobStorageException e) {
+ if (blockUpload != null) {
+ blockUpload.abort();
+ if (log.isDebugEnabled()) {
+ log.debug("Block upload aborted for blobPath '{}'.", blobPath);
+ }
+ }
+
+ throw new IOException(
+ "Failed to upload block", AzureBlobStorageClient.handleBlobException(e));
+ }
+
+ buffer.clear();
+ }
+
+ @Override
+ public void flush() throws IOException {
+ if (closed) {
+ throw new IOException("Stream closed");
+ }
+
+ if (buffer.position() - buffer.arrayOffset() > 0) {
+ uploadBlock();
+ }
+
+ if (blockUpload != null) {
+ blockUpload.complete();
+ blockUpload = null;
+ committed = true;
+ }
+ }
+
+ @Override
+ public void close() throws IOException {
+ if (closed) {
+ return;
+ }
+
+ if (blockUpload != null && blockUpload.aborted) {
+ blockUpload = null;
+ closed = true;
+ return;
+ }
+
+ if (!committed) {
+ uploadBlock();
+ if (blockUpload != null) {
+ blockUpload.complete();
+ blockUpload = null;
+ committed = true;
+ } else {
+ try {
+ blobClient.upload(new ByteArrayInputStream(new byte[0]), 0, true);
+ } catch (BlobStorageException e) {
+ throw new IOException(
+ "Failed to create empty blob", AzureBlobStorageClient.handleBlobException(e));
+ }
+ }
+ } else {
+ if (blockUpload != null) {
+ blockUpload.complete();
+ blockUpload = null;
+ }
+ }
+
+ closed = true;
+ }
+
+ private BlockUpload newBlockUpload() throws IOException {
+ try {
+ return new BlockUpload();
+ } catch (BlobStorageException e) {
+ throw new IOException(
+ "Failed to create block upload", AzureBlobStorageClient.handleBlobException(e));
+ }
+ }
+
+ private class BlockUpload {
+ private final List<String> blockIds;
+ private boolean aborted = false;
+
+ public BlockUpload() {
+ this.blockIds = new ArrayList<>();
+ if (log.isDebugEnabled()) {
+ log.debug("Initiated block upload for blobPath '{}'", blobPath);
+ }
+
+ try {
+ BlockBlobClient blockBlobClient = blobClient.getBlockBlobClient();
+ blockBlobClient.deleteIfExists();
Review Comment:
What's the reason we always delete? Can't we just skip this step?
##########
solr/modules/azure-blob-repository/src/java/org/apache/solr/azureblob/AzureBlobOutputStream.java:
##########
@@ -0,0 +1,269 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.azureblob;
+
+import com.azure.storage.blob.BlobClient;
+import com.azure.storage.blob.models.BlobStorageException;
+import com.azure.storage.blob.specialized.BlockBlobClient;
+import java.io.ByteArrayInputStream;
+import java.io.IOException;
+import java.io.OutputStream;
+import java.lang.invoke.MethodHandles;
+import java.nio.ByteBuffer;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Base64;
+import java.util.List;
+import java.util.UUID;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * OutputStream implementation for Azure Blob Storage using block blobs. Supports chunked uploads
+ * for large files.
+ */
+public class AzureBlobOutputStream extends OutputStream {
+ private static final Logger log = LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+
+ static final int BLOCK_SIZE = 4 * 1024 * 1024;
+
+ private final BlobClient blobClient;
+ private final String blobPath;
+ private volatile boolean closed;
+ private final ByteBuffer buffer;
+ private BlockUpload blockUpload;
+ private boolean committed;
+
+ public AzureBlobOutputStream(BlobClient blobClient, String blobPath) {
+ this.blobClient = blobClient;
+ this.blobPath = blobPath;
+ this.closed = false;
+ this.buffer = ByteBuffer.allocate(BLOCK_SIZE);
+ this.blockUpload = null;
+ this.committed = false;
+
+ if (log.isDebugEnabled()) {
+ log.debug("Created BlobOutputStream for blobPath '{}'", blobPath);
+ }
+ }
+
+ @Override
+ public void write(int b) throws IOException {
+ if (closed) {
+ throw new IOException("Stream closed");
+ }
+
+ buffer.put((byte) b);
+
+ if (!buffer.hasRemaining()) {
+ uploadBlock();
+ }
+ }
+
+ @Override
+ public void write(byte[] b, int off, int len) throws IOException {
+ if (closed) {
+ throw new IOException("Stream closed");
+ }
+
+ if (outOfRange(off, b.length) || len < 0 || outOfRange(off + len, b.length)) {
+ throw new IndexOutOfBoundsException();
+ } else if (len == 0) {
+ return;
+ }
+
+ int currentOffset = off;
+ int lenRemaining = len;
+ while (buffer.remaining() < lenRemaining) {
+ int firstPart = buffer.remaining();
+ buffer.put(b, currentOffset, firstPart);
+ uploadBlock();
+
+ currentOffset += firstPart;
+ lenRemaining -= firstPart;
+ }
+ if (lenRemaining > 0) {
+ buffer.put(b, currentOffset, lenRemaining);
+ }
+ }
+
+ private static boolean outOfRange(int off, int len) {
+ return off < 0 || off > len;
+ }
+
+ private void uploadBlock() throws IOException {
+ int size = buffer.position() - buffer.arrayOffset();
+
+ if (size == 0) {
+ return;
+ }
+
+ if (blockUpload == null) {
+ if (log.isDebugEnabled()) {
+ log.debug("New block upload for blobPath '{}'", blobPath);
+ }
+
+ blockUpload = newBlockUpload();
+ }
+
+ try (ByteArrayInputStream inputStream =
+ new ByteArrayInputStream(buffer.array(), buffer.arrayOffset(), size)) {
+ blockUpload.uploadBlock(inputStream, size);
+ } catch (BlobStorageException e) {
+ if (blockUpload != null) {
+ blockUpload.abort();
+ if (log.isDebugEnabled()) {
+ log.debug("Block upload aborted for blobPath '{}'.", blobPath);
+ }
+ }
+
+ throw new IOException(
+ "Failed to upload block", AzureBlobStorageClient.handleBlobException(e));
+ }
+
+ buffer.clear();
+ }
+
+ @Override
+ public void flush() throws IOException {
+ if (closed) {
+ throw new IOException("Stream closed");
+ }
+
+ if (buffer.position() - buffer.arrayOffset() > 0) {
+ uploadBlock();
+ }
+
+ if (blockUpload != null) {
+ blockUpload.complete();
+ blockUpload = null;
+ committed = true;
+ }
+ }
+
+ @Override
+ public void close() throws IOException {
+ if (closed) {
+ return;
+ }
+
+ if (blockUpload != null && blockUpload.aborted) {
+ blockUpload = null;
+ closed = true;
+ return;
+ }
+
+ if (!committed) {
+ uploadBlock();
+ if (blockUpload != null) {
+ blockUpload.complete();
+ blockUpload = null;
+ committed = true;
+ } else {
+ try {
+ blobClient.upload(new ByteArrayInputStream(new byte[0]), 0, true);
+ } catch (BlobStorageException e) {
+ throw new IOException(
+ "Failed to create empty blob", AzureBlobStorageClient.handleBlobException(e));
+ }
+ }
+ } else {
+ if (blockUpload != null) {
+ blockUpload.complete();
+ blockUpload = null;
+ }
+ }
+
+ closed = true;
+ }
+
+ private BlockUpload newBlockUpload() throws IOException {
+ try {
+ return new BlockUpload();
+ } catch (BlobStorageException e) {
+ throw new IOException(
+ "Failed to create block upload", AzureBlobStorageClient.handleBlobException(e));
+ }
+ }
+
+ private class BlockUpload {
+ private final List<String> blockIds;
+ private boolean aborted = false;
+
+ public BlockUpload() {
+ this.blockIds = new ArrayList<>();
+ if (log.isDebugEnabled()) {
+ log.debug("Initiated block upload for blobPath '{}'", blobPath);
+ }
+
+ try {
+ BlockBlobClient blockBlobClient = blobClient.getBlockBlobClient();
+ blockBlobClient.deleteIfExists();
+ } catch (BlobStorageException e) {
+ // ignore; subsequent stage/commit will surface real issues
+ }
+ }
+
+ void uploadBlock(ByteArrayInputStream inputStream, long blockSize) {
+ if (aborted) {
+ throw new IllegalStateException(
+ "Can't upload new blocks on a BlockUpload that was aborted");
+ }
+
+ String blockId =
+ Base64.getEncoder()
+ .encodeToString(UUID.randomUUID().toString().getBytes(StandardCharsets.UTF_8));
+
+ if (log.isDebugEnabled()) {
+ log.debug("Uploading block {} for blobPath '{}'", blockId, blobPath);
+ }
+
+ try {
+ BlockBlobClient blockBlobClient = blobClient.getBlockBlobClient();
+ blockBlobClient.stageBlock(blockId, inputStream, blockSize);
+ blockIds.add(blockId);
+ } catch (BlobStorageException e) {
+ throw new RuntimeException("Failed to upload block", e);
Review Comment:
Should not be a `RuntimeException`. The caller should get an `IOException` or a subclass of it.
##########
solr/modules/azure-blob-repository/src/java/org/apache/solr/azureblob/AzureBlobStorageClient.java:
##########
@@ -0,0 +1,529 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.azureblob;
+
+import com.azure.core.credential.TokenCredential;
+import com.azure.identity.DefaultAzureCredentialBuilder;
+import com.azure.storage.blob.BlobClient;
+import com.azure.storage.blob.BlobContainerClient;
+import com.azure.storage.blob.BlobServiceClient;
+import com.azure.storage.blob.BlobServiceClientBuilder;
+import com.azure.storage.blob.models.BlobItem;
+import com.azure.storage.blob.models.BlobStorageException;
+import com.azure.storage.blob.models.ListBlobsOptions;
+import com.google.common.annotations.VisibleForTesting;
+import java.io.ByteArrayInputStream;
+import java.io.FilterInputStream;
+import java.io.InputStream;
+import java.io.OutputStream;
+import java.lang.invoke.MethodHandles;
+import java.util.Collection;
+import java.util.HashSet;
+import java.util.Locale;
+import java.util.Set;
+import java.util.stream.Collectors;
+import org.apache.solr.common.util.ResumableInputStream;
+import org.apache.solr.common.util.StrUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Creates a {@link BlobServiceClient} for communicating with Azure Blob Storage. Utilizes the
+ * default Azure credential provider chain.
+ */
+public class AzureBlobStorageClient {
+
+ private static final Logger log = LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+
+ static final String BLOB_FILE_PATH_DELIMITER = "/";
+ private static final int HTTP_NOT_FOUND = 404;
+ private static final int HTTP_CONFLICT = 409;
+ private static final int SKIP_BUFFER_SIZE = 8192;
+ private static final int DELETE_BATCH_SIZE = 1000;
+
+ private static final com.azure.core.http.HttpClient SHARED_HTTP_CLIENT =
+ new com.azure.core.http.okhttp.OkHttpAsyncHttpClientBuilder().build();
+
+ private final BlobContainerClient containerClient;
+
+ AzureBlobStorageClient(
+ String containerName,
+ String connectionString,
+ String endpoint,
+ String accountName,
+ String accountKey,
+ String sasToken,
+ String tenantId,
+ String clientId,
+ String clientSecret) {
+ this(
+ createInternalClient(
+ connectionString,
+ endpoint,
+ accountName,
+ accountKey,
+ sasToken,
+ tenantId,
+ clientId,
+ clientSecret),
+ containerName);
+ }
+
+ @VisibleForTesting
+ AzureBlobStorageClient(BlobServiceClient blobServiceClient, String containerName) {
+ this.containerClient = blobServiceClient.getBlobContainerClient(containerName);
+ try {
+ containerClient.create();
+ } catch (BlobStorageException e) {
+ if (e.getStatusCode() != HTTP_CONFLICT) {
+ throw e;
+ }
+ }
+ }
+
+ private static BlobServiceClient createInternalClient(
+ String connectionString,
+ String endpoint,
+ String accountName,
+ String accountKey,
+ String sasToken,
+ String tenantId,
+ String clientId,
+ String clientSecret) {
+
+ BlobServiceClientBuilder builder = new BlobServiceClientBuilder();
+ builder.httpClient(SHARED_HTTP_CLIENT);
+
+ if (StrUtils.isNotNullOrEmpty(connectionString)) {
+ builder.connectionString(connectionString);
+ } else if (StrUtils.isNotNullOrEmpty(endpoint)) {
+ builder.endpoint(endpoint);
+ if (StrUtils.isNotNullOrEmpty(accountName) && StrUtils.isNotNullOrEmpty(accountKey)) {
+ builder.credential(
+ new com.azure.storage.common.StorageSharedKeyCredential(accountName, accountKey));
+ } else if (StrUtils.isNotNullOrEmpty(sasToken)) {
+ builder.sasToken(sasToken);
+ } else {
+ TokenCredential credential = new DefaultAzureCredentialBuilder().tenantId(tenantId).build();
+ builder.credential(credential);
+ }
+ } else {
+ throw new IllegalArgumentException("Either connectionString or endpoint must be provided");
+ }
+
+ return builder.buildClient();
+ }
+
+ void createDirectory(String path) throws AzureBlobException {
+ String sanitizedDirPath = sanitizedDirPath(path);
+
+ if (!pathExists(sanitizedDirPath)) {
+ String parent = getParentDirectory(sanitizedDirPath);
+ if (!parent.isEmpty() && !parent.equals(BLOB_FILE_PATH_DELIMITER)) {
+ createDirectory(parent);
+ }
+
+ try {
+ BlobClient blobClient = containerClient.getBlobClient(sanitizedDirPath);
+ blobClient.upload(new ByteArrayInputStream(new byte[0]), 0, true);
+ java.util.Map<String, String> metadata = new java.util.HashMap<>();
+ metadata.put("hdi_isfolder", "true");
+ blobClient.setMetadata(metadata);
+ } catch (BlobStorageException e) {
+ throw handleBlobException(e);
+ }
+ }
+ }
+
+ void delete(Collection<String> paths) throws AzureBlobException {
+ Set<String> entries = new HashSet<>();
+ for (String path : paths) {
+ entries.add(sanitizedFilePath(path));
+ }
+ deleteBlobs(entries);
+ }
+
+ void deleteDirectory(String path) throws AzureBlobException {
+ path = sanitizedDirPath(path);
+
+ Set<String> entries = listAll(path);
+ if (pathExists(path)) {
+ entries.add(path);
+ }
+
+ deleteBlobs(entries);
+ }
+
+ String[] listDir(String path) throws AzureBlobException {
+ path = sanitizedDirPath(path);
+
+ try {
+ ListBlobsOptions options = new ListBlobsOptions().setPrefix(path).setMaxResultsPerPage(1000);
+
+ final String finalPath = path;
+ return containerClient.listBlobs(options, null).stream()
+ .map(BlobItem::getName)
+ .filter(s -> s.startsWith(finalPath))
+ .map(s -> s.substring(finalPath.length()))
+ .filter(s -> !s.isEmpty())
+ .filter(
+ s -> {
+ int slashIndex = s.indexOf(BLOB_FILE_PATH_DELIMITER);
+ return slashIndex == -1 || slashIndex == s.length() - 1;
+ })
+ .toArray(String[]::new);
+ } catch (BlobStorageException e) {
+ throw handleBlobException(e);
+ }
+ }
+
+ boolean pathExists(String path) throws AzureBlobException {
+ final String blobPath = sanitizedPath(path);
+
+ if (blobPath.isEmpty() || BLOB_FILE_PATH_DELIMITER.equals(blobPath)) {
+ return true;
+ }
+
+ try {
+ BlobClient blobClient = containerClient.getBlobClient(blobPath);
+ return blobClient.exists();
+ } catch (BlobStorageException e) {
+ throw handleBlobException(e);
+ }
+ }
+
+ boolean isDirectory(String path) throws AzureBlobException {
+ final String dirPrefix = sanitizedDirPath(path);
+
+ try {
+ ListBlobsOptions options =
+ new ListBlobsOptions().setPrefix(dirPrefix).setMaxResultsPerPage(1);
+ if (containerClient.listBlobs(options, null).iterator().hasNext()) {
+ return true;
+ }
+
+ BlobClient markerClient = containerClient.getBlobClient(dirPrefix);
+ if (markerClient.exists()) {
+ long size = markerClient.getProperties().getBlobSize();
+ if (size == 0) {
+ return true;
+ }
+ java.util.Map<String, String> md = markerClient.getProperties().getMetadata();
+ return md != null && md.containsKey("hdi_isfolder");
+ }
+
+ return false;
+ } catch (BlobStorageException e) {
+ throw handleBlobException(e);
+ }
+ }
+
+ long length(String path) throws AzureBlobException {
+ String blobPath = sanitizedFilePath(path);
+ try {
+ BlobClient blobClient = containerClient.getBlobClient(blobPath);
+ return blobClient.getProperties().getBlobSize();
+ } catch (BlobStorageException e) {
+ throw handleBlobException(e);
+ }
+ }
+
+ InputStream pullStream(String path) throws AzureBlobException {
+ final String blobPath = sanitizedFilePath(path);
+
+ try {
+ BlobClient blobClient = containerClient.getBlobClient(blobPath);
+ final long contentLength = blobClient.getProperties().getBlobSize();
+
+ if (contentLength == 0) {
+ return new ByteArrayInputStream(new byte[0]);
+ }
+
+ InputStream initial = new IdempotentCloseInputStream(blobClient.openInputStream());
+
+ return new ResumableInputStream(
+ initial,
+ bytesRead -> {
+ if (contentLength > 0 && bytesRead >= contentLength) {
+ return null;
+ }
+ try {
+ long remaining =
+ contentLength > 0 ? Math.max(0, contentLength - bytesRead) : Long.MAX_VALUE;
+ return pullRangeStream(path, bytesRead, remaining);
+ } catch (AzureBlobException e) {
+ throw new RuntimeException(e);
+ }
+ });
+ } catch (BlobStorageException e) {
+ throw handleBlobException(e);
+ }
+ }
+
+ InputStream pullRangeStream(String path, long offset, long length) throws AzureBlobException {
+ final String blobPath = sanitizedFilePath(path);
+ try {
+ BlobClient blobClient = containerClient.getBlobClient(blobPath);
+ com.azure.storage.blob.models.BlobRange range =
+ new com.azure.storage.blob.models.BlobRange(offset, length);
+ return new IdempotentCloseInputStream(blobClient.openInputStream(range, null));
+ } catch (BlobStorageException e) {
+ throw handleBlobException(e);
+ }
+ }
+
+ private static final class IdempotentCloseInputStream extends FilterInputStream {
+ private boolean closed;
+
+ IdempotentCloseInputStream(InputStream in) {
+ super(in);
+ this.closed = false;
+ }
+
+ @Override
+ public int read() throws java.io.IOException {
+ if (closed) {
+ throw new java.io.IOException("Stream is already closed");
+ }
+ try {
+ return super.read();
+ } catch (RuntimeException re) {
+ if (isAlreadyClosed(re)) {
+ throw new java.io.IOException("Stream is already closed", re);
+ }
+ throw re;
+ }
+ }
+
+ @Override
+ public int read(byte[] b, int off, int len) throws java.io.IOException {
+ if (closed) {
+ throw new java.io.IOException("Stream is already closed");
+ }
+ try {
+ return super.read(b, off, len);
+ } catch (RuntimeException re) {
+ if (isAlreadyClosed(re)) {
+ throw new java.io.IOException("Stream is already closed", re);
+ }
+ throw re;
+ }
+ }
+
+ @Override
+ public void close() throws java.io.IOException {
+ if (closed) {
+ return;
+ }
+ try {
+ super.close();
+ } catch (java.io.IOException e) {
+ String msg = e.getMessage();
+ if (msg == null || !msg.toLowerCase(java.util.Locale.ROOT).contains("already closed")) {
+ throw e;
+ }
+ // swallow "already closed" to make close idempotent
+ } finally {
+ closed = true;
+ }
+ }
+
+ @Override
+ public long skip(long n) throws java.io.IOException {
+ if (closed) {
+ throw new java.io.IOException("Stream is already closed");
+ }
+ if (n <= 0) {
+ return 0L;
+ }
+ long remaining = n;
+ byte[] discard = new byte[SKIP_BUFFER_SIZE];
+ try {
+ while (remaining > 0) {
+ int toRead = (int) Math.min(discard.length, remaining);
+ int read = super.read(discard, 0, toRead);
+ if (read < 0) {
+ break;
+ }
+ remaining -= read;
+ }
+ return n - remaining;
+ } catch (RuntimeException re) {
+ throw new java.io.IOException(re);
+ }
+ }
+
+ private static boolean isAlreadyClosed(Throwable t) {
+ String msg = t.getMessage();
+ return msg != null && msg.toLowerCase(java.util.Locale.ROOT).contains("already closed");
+ }
+ }
+
+ OutputStream pushStream(String path) throws AzureBlobException {
+ path = sanitizedFilePath(path);
+
+ if (!parentDirectoryExist(path)) {
+ String parentDirectory = getParentDirectory(path);
+ if (!parentDirectory.isEmpty() && !parentDirectory.equals(BLOB_FILE_PATH_DELIMITER)) {
+ createDirectory(parentDirectory);
+ }
+ }
+
+ try {
+ BlobClient blobClient = containerClient.getBlobClient(path);
+ return new AzureBlobOutputStream(blobClient, path);
+ } catch (BlobStorageException e) {
+ throw handleBlobException(e);
+ }
+ }
+
+ void close() {}
+
+ @VisibleForTesting
+ void deleteContainerForTests() {
+ try {
+ containerClient.delete();
+ } catch (BlobStorageException e) {
+ if (e.getStatusCode() != HTTP_NOT_FOUND) {
+ throw e;
+ }
+ }
+ }
+
+ private Collection<String> deleteBlobs(Collection<String> paths) throws AzureBlobException {
+ try {
+ return deleteBlobs(paths, DELETE_BATCH_SIZE);
+ } catch (BlobStorageException e) {
+ throw handleBlobException(e);
+ }
+ }
+
+ @VisibleForTesting
+ Collection<String> deleteBlobs(Collection<String> entries, int batchSize)
+ throws AzureBlobException {
+ Set<String> deletedPaths = new HashSet<>();
+
+ for (String path : entries) {
+ try {
+ BlobClient blobClient = containerClient.getBlobClient(path);
+ boolean existed = blobClient.deleteIfExists();
+ if (existed) {
+ deletedPaths.add(path);
+ }
+ } catch (BlobStorageException e) {
+ if (e.getStatusCode() == HTTP_NOT_FOUND) {
Review Comment:
I don't get how we can get an exception for a blob that does not exist since we call `deleteIfExists()`
##########
solr/modules/azure-blob-repository/src/java/org/apache/solr/azureblob/AzureBlobOutputStream.java:
##########
@@ -0,0 +1,269 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.azureblob;
+
+import com.azure.storage.blob.BlobClient;
+import com.azure.storage.blob.models.BlobStorageException;
+import com.azure.storage.blob.specialized.BlockBlobClient;
+import java.io.ByteArrayInputStream;
+import java.io.IOException;
+import java.io.OutputStream;
+import java.lang.invoke.MethodHandles;
+import java.nio.ByteBuffer;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Base64;
+import java.util.List;
+import java.util.UUID;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * OutputStream implementation for Azure Blob Storage using block blobs. Supports chunked uploads
+ * for large files.
+ */
+public class AzureBlobOutputStream extends OutputStream {
+ private static final Logger log = LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+
+ static final int BLOCK_SIZE = 4 * 1024 * 1024;
+
+ private final BlobClient blobClient;
+ private final String blobPath;
+ private volatile boolean closed;
+ private final ByteBuffer buffer;
+ private BlockUpload blockUpload;
+ private boolean committed;
+
+ public AzureBlobOutputStream(BlobClient blobClient, String blobPath) {
+ this.blobClient = blobClient;
+ this.blobPath = blobPath;
+ this.closed = false;
+ this.buffer = ByteBuffer.allocate(BLOCK_SIZE);
+ this.blockUpload = null;
+ this.committed = false;
+
+ if (log.isDebugEnabled()) {
+ log.debug("Created BlobOutputStream for blobPath '{}'", blobPath);
+ }
+ }
+
+ @Override
+ public void write(int b) throws IOException {
+ if (closed) {
+ throw new IOException("Stream closed");
+ }
+
+ buffer.put((byte) b);
+
+ if (!buffer.hasRemaining()) {
+ uploadBlock();
+ }
+ }
+
+ @Override
+ public void write(byte[] b, int off, int len) throws IOException {
+ if (closed) {
+ throw new IOException("Stream closed");
+ }
+
+ if (outOfRange(off, b.length) || len < 0 || outOfRange(off + len, b.length)) {
+ throw new IndexOutOfBoundsException();
+ } else if (len == 0) {
+ return;
+ }
+
+ int currentOffset = off;
+ int lenRemaining = len;
+ while (buffer.remaining() < lenRemaining) {
+ int firstPart = buffer.remaining();
+ buffer.put(b, currentOffset, firstPart);
+ uploadBlock();
+
+ currentOffset += firstPart;
+ lenRemaining -= firstPart;
+ }
+ if (lenRemaining > 0) {
+ buffer.put(b, currentOffset, lenRemaining);
+ }
+ }
+
+ private static boolean outOfRange(int off, int len) {
+ return off < 0 || off > len;
+ }
+
+ private void uploadBlock() throws IOException {
+ int size = buffer.position() - buffer.arrayOffset();
+
+ if (size == 0) {
+ return;
+ }
+
+ if (blockUpload == null) {
+ if (log.isDebugEnabled()) {
+ log.debug("New block upload for blobPath '{}'", blobPath);
+ }
+
+ blockUpload = newBlockUpload();
+ }
+
+ try (ByteArrayInputStream inputStream =
+ new ByteArrayInputStream(buffer.array(), buffer.arrayOffset(), size)) {
+ blockUpload.uploadBlock(inputStream, size);
+ } catch (BlobStorageException e) {
+ if (blockUpload != null) {
+ blockUpload.abort();
+ if (log.isDebugEnabled()) {
+ log.debug("Block upload aborted for blobPath '{}'.", blobPath);
+ }
+ }
+
+ throw new IOException(
+ "Failed to upload block", AzureBlobStorageClient.handleBlobException(e));
+ }
+
+ buffer.clear();
+ }
+
+ @Override
+ public void flush() throws IOException {
+ if (closed) {
+ throw new IOException("Stream closed");
+ }
+
+ if (buffer.position() - buffer.arrayOffset() > 0) {
+ uploadBlock();
+ }
+
+ if (blockUpload != null) {
+ blockUpload.complete();
+ blockUpload = null;
+ committed = true;
+ }
+ }
+
+ @Override
+ public void close() throws IOException {
+ if (closed) {
+ return;
+ }
+
+ if (blockUpload != null && blockUpload.aborted) {
+ blockUpload = null;
+ closed = true;
+ return;
+ }
+
+ if (!committed) {
+ uploadBlock();
+ if (blockUpload != null) {
+ blockUpload.complete();
+ blockUpload = null;
+ committed = true;
+ } else {
+ try {
+ blobClient.upload(new ByteArrayInputStream(new byte[0]), 0, true);
+ } catch (BlobStorageException e) {
+ throw new IOException(
+ "Failed to create empty blob", AzureBlobStorageClient.handleBlobException(e));
+ }
+ }
+ } else {
+ if (blockUpload != null) {
+ blockUpload.complete();
+ blockUpload = null;
+ }
+ }
+
+ closed = true;
+ }
+
+ private BlockUpload newBlockUpload() throws IOException {
+ try {
+ return new BlockUpload();
+ } catch (BlobStorageException e) {
+ throw new IOException(
+ "Failed to create block upload", AzureBlobStorageClient.handleBlobException(e));
+ }
+ }
+
+ private class BlockUpload {
+ private final List<String> blockIds;
+ private boolean aborted = false;
+
+ public BlockUpload() {
+ this.blockIds = new ArrayList<>();
+ if (log.isDebugEnabled()) {
+ log.debug("Initiated block upload for blobPath '{}'", blobPath);
+ }
+
+ try {
+ BlockBlobClient blockBlobClient = blobClient.getBlockBlobClient();
+ blockBlobClient.deleteIfExists();
+ } catch (BlobStorageException e) {
+ // ignore; subsequent stage/commit will surface real issues
+ }
+ }
+
+ void uploadBlock(ByteArrayInputStream inputStream, long blockSize) {
+ if (aborted) {
+ throw new IllegalStateException(
+ "Can't upload new blocks on a BlockUpload that was aborted");
+ }
+
+ String blockId =
+ Base64.getEncoder()
+ .encodeToString(UUID.randomUUID().toString().getBytes(StandardCharsets.UTF_8));
+
+ if (log.isDebugEnabled()) {
+ log.debug("Uploading block {} for blobPath '{}'", blockId, blobPath);
+ }
+
+ try {
+ BlockBlobClient blockBlobClient = blobClient.getBlockBlobClient();
+ blockBlobClient.stageBlock(blockId, inputStream, blockSize);
+ blockIds.add(blockId);
+ } catch (BlobStorageException e) {
+ throw new RuntimeException("Failed to upload block", e);
+ }
+ }
+
+ void complete() {
+ if (aborted) {
+ throw new IllegalStateException("Can't complete a BlockUpload that was aborted");
+ }
+
+ if (log.isDebugEnabled()) {
+ log.debug("Completing block upload for blobPath '{}'", blobPath);
+ }
+
+ try {
+ BlockBlobClient blockBlobClient = blobClient.getBlockBlobClient();
+ blockBlobClient.commitBlockList(blockIds);
+ } catch (BlobStorageException e) {
+ throw new RuntimeException("Failed to commit block list", e);
Review Comment:
Same for exception type
##########
solr/modules/azure-blob-repository/src/java/org/apache/solr/azureblob/AzureBlobOutputStream.java:
##########
@@ -0,0 +1,269 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.azureblob;
+
+import com.azure.storage.blob.BlobClient;
+import com.azure.storage.blob.models.BlobStorageException;
+import com.azure.storage.blob.specialized.BlockBlobClient;
+import java.io.ByteArrayInputStream;
+import java.io.IOException;
+import java.io.OutputStream;
+import java.lang.invoke.MethodHandles;
+import java.nio.ByteBuffer;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Base64;
+import java.util.List;
+import java.util.UUID;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * OutputStream implementation for Azure Blob Storage using block blobs. Supports chunked uploads
+ * for large files.
+ */
+public class AzureBlobOutputStream extends OutputStream {
+ private static final Logger log = LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+
+ static final int BLOCK_SIZE = 4 * 1024 * 1024;
+
+ private final BlobClient blobClient;
+ private final String blobPath;
+ private volatile boolean closed;
+ private final ByteBuffer buffer;
+ private BlockUpload blockUpload;
+ private boolean committed;
+
+ public AzureBlobOutputStream(BlobClient blobClient, String blobPath) {
+ this.blobClient = blobClient;
+ this.blobPath = blobPath;
+ this.closed = false;
+ this.buffer = ByteBuffer.allocate(BLOCK_SIZE);
+ this.blockUpload = null;
+ this.committed = false;
+
+ if (log.isDebugEnabled()) {
+ log.debug("Created BlobOutputStream for blobPath '{}'", blobPath);
+ }
+ }
+
+ @Override
+ public void write(int b) throws IOException {
+ if (closed) {
+ throw new IOException("Stream closed");
+ }
+
+ buffer.put((byte) b);
+
+ if (!buffer.hasRemaining()) {
+ uploadBlock();
+ }
+ }
+
+ @Override
+ public void write(byte[] b, int off, int len) throws IOException {
+ if (closed) {
+ throw new IOException("Stream closed");
+ }
+
+ if (outOfRange(off, b.length) || len < 0 || outOfRange(off + len, b.length)) {
+ throw new IndexOutOfBoundsException();
+ } else if (len == 0) {
+ return;
+ }
+
+ int currentOffset = off;
+ int lenRemaining = len;
+ while (buffer.remaining() < lenRemaining) {
+ int firstPart = buffer.remaining();
+ buffer.put(b, currentOffset, firstPart);
+ uploadBlock();
+
+ currentOffset += firstPart;
+ lenRemaining -= firstPart;
+ }
+ if (lenRemaining > 0) {
+ buffer.put(b, currentOffset, lenRemaining);
+ }
+ }
+
+ private static boolean outOfRange(int off, int len) {
+ return off < 0 || off > len;
+ }
+
+ private void uploadBlock() throws IOException {
+ int size = buffer.position() - buffer.arrayOffset();
+
+ if (size == 0) {
+ return;
+ }
+
+ if (blockUpload == null) {
+ if (log.isDebugEnabled()) {
+ log.debug("New block upload for blobPath '{}'", blobPath);
+ }
+
+ blockUpload = newBlockUpload();
+ }
+
+ try (ByteArrayInputStream inputStream =
+ new ByteArrayInputStream(buffer.array(), buffer.arrayOffset(), size)) {
+ blockUpload.uploadBlock(inputStream, size);
+ } catch (BlobStorageException e) {
+ if (blockUpload != null) {
+ blockUpload.abort();
+ if (log.isDebugEnabled()) {
+ log.debug("Block upload aborted for blobPath '{}'.", blobPath);
+ }
+ }
+
+ throw new IOException(
+ "Failed to upload block", AzureBlobStorageClient.handleBlobException(e));
+ }
+
+ buffer.clear();
+ }
+
+ @Override
+ public void flush() throws IOException {
+ if (closed) {
+ throw new IOException("Stream closed");
+ }
+
+ if (buffer.position() - buffer.arrayOffset() > 0) {
+ uploadBlock();
+ }
+
+ if (blockUpload != null) {
+ blockUpload.complete();
+ blockUpload = null;
+ committed = true;
+ }
+ }
+
+ @Override
+ public void close() throws IOException {
+ if (closed) {
+ return;
+ }
+
+ if (blockUpload != null && blockUpload.aborted) {
+ blockUpload = null;
+ closed = true;
+ return;
+ }
+
+ if (!committed) {
+ uploadBlock();
+ if (blockUpload != null) {
+ blockUpload.complete();
+ blockUpload = null;
+ committed = true;
+ } else {
+ try {
+ blobClient.upload(new ByteArrayInputStream(new byte[0]), 0, true);
+ } catch (BlobStorageException e) {
+ throw new IOException(
+ "Failed to create empty blob", AzureBlobStorageClient.handleBlobException(e));
+ }
+ }
+ } else {
+ if (blockUpload != null) {
+ blockUpload.complete();
+ blockUpload = null;
+ }
+ }
+
+ closed = true;
+ }
+
+ private BlockUpload newBlockUpload() throws IOException {
+ try {
+ return new BlockUpload();
+ } catch (BlobStorageException e) {
+ throw new IOException(
+ "Failed to create block upload", AzureBlobStorageClient.handleBlobException(e));
+ }
+ }
+
+ private class BlockUpload {
+ private final List<String> blockIds;
+ private boolean aborted = false;
+
+ public BlockUpload() {
+ this.blockIds = new ArrayList<>();
+ if (log.isDebugEnabled()) {
+ log.debug("Initiated block upload for blobPath '{}'", blobPath);
+ }
+
+ try {
+ BlockBlobClient blockBlobClient = blobClient.getBlockBlobClient();
+ blockBlobClient.deleteIfExists();
+ } catch (BlobStorageException e) {
+ // ignore; subsequent stage/commit will surface real issues
+ }
+ }
+
+ void uploadBlock(ByteArrayInputStream inputStream, long blockSize) {
+ if (aborted) {
+ throw new IllegalStateException(
+ "Can't upload new blocks on a BlockUpload that was aborted");
+ }
+
+ String blockId =
+ Base64.getEncoder()
+ .encodeToString(UUID.randomUUID().toString().getBytes(StandardCharsets.UTF_8));
+
+ if (log.isDebugEnabled()) {
+ log.debug("Uploading block {} for blobPath '{}'", blockId, blobPath);
+ }
+
+ try {
+ BlockBlobClient blockBlobClient = blobClient.getBlockBlobClient();
+ blockBlobClient.stageBlock(blockId, inputStream, blockSize);
+ blockIds.add(blockId);
+ } catch (BlobStorageException e) {
+ throw new RuntimeException("Failed to upload block", e);
+ }
+ }
+
+ void complete() {
+ if (aborted) {
+ throw new IllegalStateException("Can't complete a BlockUpload that was aborted");
+ }
+
+ if (log.isDebugEnabled()) {
+ log.debug("Completing block upload for blobPath '{}'", blobPath);
+ }
+
+ try {
+ BlockBlobClient blockBlobClient = blobClient.getBlockBlobClient();
+ blockBlobClient.commitBlockList(blockIds);
+ } catch (BlobStorageException e) {
+ throw new RuntimeException("Failed to commit block list", e);
+ }
+ }
+
+ public void abort() {
+ if (log.isWarnEnabled()) {
+ log.warn("Aborting block upload for blobPath '{}'", blobPath);
+ }
+
+ aborted = true;
Review Comment:
Aborted means failed, correct? That's not a choice of the caller.
If true, I would rename for clarity
##########
solr/modules/azure-blob-repository/src/java/org/apache/solr/azureblob/AzureBlobOutputStream.java:
##########
@@ -0,0 +1,269 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.azureblob;
+
+import com.azure.storage.blob.BlobClient;
+import com.azure.storage.blob.models.BlobStorageException;
+import com.azure.storage.blob.specialized.BlockBlobClient;
+import java.io.ByteArrayInputStream;
+import java.io.IOException;
+import java.io.OutputStream;
+import java.lang.invoke.MethodHandles;
+import java.nio.ByteBuffer;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Base64;
+import java.util.List;
+import java.util.UUID;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * OutputStream implementation for Azure Blob Storage using block blobs. Supports chunked uploads
+ * for large files.
+ */
+public class AzureBlobOutputStream extends OutputStream {
+ private static final Logger log = LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+
+ static final int BLOCK_SIZE = 4 * 1024 * 1024;
+
+ private final BlobClient blobClient;
+ private final String blobPath;
+ private volatile boolean closed;
+ private final ByteBuffer buffer;
+ private BlockUpload blockUpload;
+ private boolean committed;
+
+ public AzureBlobOutputStream(BlobClient blobClient, String blobPath) {
+ this.blobClient = blobClient;
+ this.blobPath = blobPath;
+ this.closed = false;
+ this.buffer = ByteBuffer.allocate(BLOCK_SIZE);
+ this.blockUpload = null;
+ this.committed = false;
+
+ if (log.isDebugEnabled()) {
+ log.debug("Created BlobOutputStream for blobPath '{}'", blobPath);
+ }
+ }
+
+ @Override
+ public void write(int b) throws IOException {
+ if (closed) {
+ throw new IOException("Stream closed");
+ }
+
+ buffer.put((byte) b);
+
+ if (!buffer.hasRemaining()) {
+ uploadBlock();
+ }
+ }
+
+ @Override
+ public void write(byte[] b, int off, int len) throws IOException {
+ if (closed) {
+ throw new IOException("Stream closed");
+ }
+
+ if (outOfRange(off, b.length) || len < 0 || outOfRange(off + len, b.length)) {
+ throw new IndexOutOfBoundsException();
+ } else if (len == 0) {
+ return;
+ }
+
+ int currentOffset = off;
+ int lenRemaining = len;
+ while (buffer.remaining() < lenRemaining) {
+ int firstPart = buffer.remaining();
+ buffer.put(b, currentOffset, firstPart);
+ uploadBlock();
+
+ currentOffset += firstPart;
+ lenRemaining -= firstPart;
+ }
+ if (lenRemaining > 0) {
+ buffer.put(b, currentOffset, lenRemaining);
+ }
+ }
+
+ private static boolean outOfRange(int off, int len) {
+ return off < 0 || off > len;
+ }
+
+ private void uploadBlock() throws IOException {
+ int size = buffer.position() - buffer.arrayOffset();
+
+ if (size == 0) {
+ return;
+ }
+
+ if (blockUpload == null) {
+ if (log.isDebugEnabled()) {
+ log.debug("New block upload for blobPath '{}'", blobPath);
+ }
+
+ blockUpload = newBlockUpload();
+ }
+
+ try (ByteArrayInputStream inputStream =
+ new ByteArrayInputStream(buffer.array(), buffer.arrayOffset(), size)) {
+ blockUpload.uploadBlock(inputStream, size);
+ } catch (BlobStorageException e) {
+ if (blockUpload != null) {
+ blockUpload.abort();
+ if (log.isDebugEnabled()) {
+ log.debug("Block upload aborted for blobPath '{}'.", blobPath);
+ }
+ }
+
+ throw new IOException(
+ "Failed to upload block", AzureBlobStorageClient.handleBlobException(e));
+ }
+
+ buffer.clear();
+ }
+
+ @Override
+ public void flush() throws IOException {
+ if (closed) {
+ throw new IOException("Stream closed");
+ }
+
+ if (buffer.position() - buffer.arrayOffset() > 0) {
Review Comment:
Same as previous comment
```suggestion
if (buffer.position() > 0) {
```
##########
solr/modules/azure-blob-repository/src/java/org/apache/solr/azureblob/AzureBlobOutputStream.java:
##########
@@ -0,0 +1,269 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.azureblob;
+
+import com.azure.storage.blob.BlobClient;
+import com.azure.storage.blob.models.BlobStorageException;
+import com.azure.storage.blob.specialized.BlockBlobClient;
+import java.io.ByteArrayInputStream;
+import java.io.IOException;
+import java.io.OutputStream;
+import java.lang.invoke.MethodHandles;
+import java.nio.ByteBuffer;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Base64;
+import java.util.List;
+import java.util.UUID;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * OutputStream implementation for Azure Blob Storage using block blobs. Supports chunked uploads
+ * for large files.
+ */
+public class AzureBlobOutputStream extends OutputStream {
+ private static final Logger log = LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+
+ static final int BLOCK_SIZE = 4 * 1024 * 1024;
+
+ private final BlobClient blobClient;
+ private final String blobPath;
+ private volatile boolean closed;
+ private final ByteBuffer buffer;
+ private BlockUpload blockUpload;
+ private boolean committed;
+
+ public AzureBlobOutputStream(BlobClient blobClient, String blobPath) {
+ this.blobClient = blobClient;
+ this.blobPath = blobPath;
+ this.closed = false;
+ this.buffer = ByteBuffer.allocate(BLOCK_SIZE);
+ this.blockUpload = null;
+ this.committed = false;
+
+ if (log.isDebugEnabled()) {
+ log.debug("Created BlobOutputStream for blobPath '{}'", blobPath);
+ }
+ }
+
+ @Override
+ public void write(int b) throws IOException {
+ if (closed) {
+ throw new IOException("Stream closed");
+ }
+
+ buffer.put((byte) b);
+
+ if (!buffer.hasRemaining()) {
+ uploadBlock();
+ }
+ }
+
+ @Override
+ public void write(byte[] b, int off, int len) throws IOException {
+ if (closed) {
+ throw new IOException("Stream closed");
+ }
+
+ if (outOfRange(off, b.length) || len < 0 || outOfRange(off + len, b.length)) {
+ throw new IndexOutOfBoundsException();
+ } else if (len == 0) {
+ return;
+ }
+
+ int currentOffset = off;
+ int lenRemaining = len;
+ while (buffer.remaining() < lenRemaining) {
+ int firstPart = buffer.remaining();
+ buffer.put(b, currentOffset, firstPart);
+ uploadBlock();
+
+ currentOffset += firstPart;
+ lenRemaining -= firstPart;
+ }
+ if (lenRemaining > 0) {
+ buffer.put(b, currentOffset, lenRemaining);
+ }
+ }
+
+ private static boolean outOfRange(int off, int len) {
+ return off < 0 || off > len;
+ }
+
+ private void uploadBlock() throws IOException {
+ int size = buffer.position() - buffer.arrayOffset();
+
+ if (size == 0) {
+ return;
+ }
+
+ if (blockUpload == null) {
+ if (log.isDebugEnabled()) {
+ log.debug("New block upload for blobPath '{}'", blobPath);
+ }
+
+ blockUpload = newBlockUpload();
+ }
+
+ try (ByteArrayInputStream inputStream =
+ new ByteArrayInputStream(buffer.array(), buffer.arrayOffset(), size)) {
+ blockUpload.uploadBlock(inputStream, size);
+ } catch (BlobStorageException e) {
+ if (blockUpload != null) {
+ blockUpload.abort();
+ if (log.isDebugEnabled()) {
+ log.debug("Block upload aborted for blobPath '{}'.", blobPath);
+ }
+ }
+
+ throw new IOException(
+ "Failed to upload block", AzureBlobStorageClient.handleBlobException(e));
+ }
+
+ buffer.clear();
+ }
+
+ @Override
+ public void flush() throws IOException {
+ if (closed) {
+ throw new IOException("Stream closed");
+ }
+
+ if (buffer.position() - buffer.arrayOffset() > 0) {
+ uploadBlock();
+ }
+
+ if (blockUpload != null) {
+ blockUpload.complete();
+ blockUpload = null;
+ committed = true;
+ }
+ }
+
+ @Override
+ public void close() throws IOException {
+ if (closed) {
+ return;
+ }
+
+ if (blockUpload != null && blockUpload.aborted) {
+ blockUpload = null;
+ closed = true;
+ return;
+ }
+
+ if (!committed) {
+ uploadBlock();
+ if (blockUpload != null) {
+ blockUpload.complete();
+ blockUpload = null;
+ committed = true;
+ } else {
+ try {
+ blobClient.upload(new ByteArrayInputStream(new byte[0]), 0, true);
+ } catch (BlobStorageException e) {
+ throw new IOException(
+ "Failed to create empty blob", AzureBlobStorageClient.handleBlobException(e));
+ }
+ }
+ } else {
+ if (blockUpload != null) {
+ blockUpload.complete();
+ blockUpload = null;
+ }
+ }
+
+ closed = true;
Review Comment:
Should be set to true even in case of exception
##########
solr/modules/azure-blob-repository/src/java/org/apache/solr/azureblob/AzureBlobIndexInput.java:
##########
@@ -0,0 +1,200 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.azureblob;
+
+import java.io.EOFException;
+import java.io.IOException;
+import java.io.InputStream;
+import java.util.LinkedHashMap;
+import java.util.Map;
+import org.apache.lucene.store.IndexInput;
+
+class AzureBlobIndexInput extends IndexInput {
+
+ private static final int MIN_PAGE_SIZE = 4 * 1024;
+ private static final int DEFAULT_PAGE_SIZE = 512 * 1024;
+ private static final int MAX_CACHED_PAGES = 128;
+
+ private final String path;
+ private final AzureBlobStorageClient client;
+ private final long length;
+ private final int pageSize;
+ private final LruPageCache cache;
+
+ private long position = 0L;
+ private boolean closed = false;
+
+ AzureBlobIndexInput(String path, AzureBlobStorageClient client, long length) {
+ this(path, client, length, DEFAULT_PAGE_SIZE, MAX_CACHED_PAGES);
+ }
+
+ AzureBlobIndexInput(
+ String path, AzureBlobStorageClient client, long length, int pageSize, int maxCachedPages) {
+ super(path);
+ this.path = path;
+ this.client = client;
+ this.length = length;
+ this.pageSize = Math.max(MIN_PAGE_SIZE, pageSize);
+ this.cache = new LruPageCache(maxCachedPages);
+ }
+
+ @Override
+ public void close() throws IOException {
+ closed = true;
+ cache.clear();
+ }
+
+ @Override
+ public long getFilePointer() {
+ return position;
+ }
+
+ @Override
+ public void seek(long pos) throws IOException {
+ ensureOpen();
+ if (pos < 0 || pos > length) {
+ throw new IOException("Seek position out of bounds: " + pos);
+ }
+
+ position = pos;
+ }
+
+ @Override
+ public long length() {
+ return length;
+ }
+
+ @Override
+ public IndexInput slice(String sliceDescription, long offset, long length) throws IOException {
+ ensureOpen();
+ if (offset < 0 || length < 0 || offset + length > this.length) {
+ throw new IOException("Slice out of bounds: offset=" + offset + ", length=" + length);
+ }
+
+ AzureBlobIndexInput slice =
+ new AzureBlobIndexInput(
+ getFullSliceDescription(sliceDescription), client, length, pageSize, MAX_CACHED_PAGES);
+
+ slice.position = 0L;
+
+ // Wrap client in a view that remaps range requests by adding base offset
+ slice.clientViewBaseOffset = this.clientViewBaseOffset + offset;
+ return slice;
+ }
+
+ @Override
+ public byte readByte() throws IOException {
+ ensureOpen();
+ if (position >= length) {
+ throw new EOFException("End of stream reached");
+ }
+
+ byte[] page = getPage(pageIndex(position));
+ int inPageOffset = (int) (position % pageSize);
+ byte value = page[inPageOffset];
+ position += 1L;
+ return value;
+ }
+
+ @Override
+ public void readBytes(byte[] b, int offset, int len) throws IOException {
+ ensureOpen();
+ if (len < 0) {
+ throw new IOException("Length must be non-negative");
+ }
+
+ if (position + len > length) {
+ throw new EOFException("End of stream reached");
+ }
+
+ int remaining = len;
+ while (remaining > 0) {
+ long pageIdx = pageIndex(position);
+ byte[] page = getPage(pageIdx);
+ int inPageOffset = (int) (position % pageSize);
+ int toCopy = Math.min(remaining, pageSize - inPageOffset);
+ System.arraycopy(page, inPageOffset, b, offset + (len - remaining), toCopy);
+ position += toCopy;
+ remaining -= toCopy;
+ }
+ }
+
+ // Internal state for slices: base offset to add to all range requests
+ private long clientViewBaseOffset = 0L;
+
+ private byte[] getPage(long pageIdx) throws IOException {
+ byte[] page = cache.get(pageIdx);
+ if (page != null) {
+ return page;
+ }
+
+ long absoluteOffset = clientViewBaseOffset + pageIdx * (long) pageSize;
+ int bytesToRead = (int) Math.min(pageSize, length - pageIdx * (long) pageSize);
+ if (bytesToRead <= 0) {
+ throw new EOFException("End of stream reached");
+ }
+
+ page = new byte[bytesToRead];
+ try (InputStream in = client.pullRangeStream(path, absoluteOffset, bytesToRead)) {
+ int readTotal = 0;
+ while (readTotal < bytesToRead) {
+ int read = in.read(page, readTotal, bytesToRead - readTotal);
+ if (read == -1) break;
+ readTotal += read;
+ }
+
+ if (readTotal < bytesToRead) {
+ throw new EOFException(
+ "End of stream reached: expected " + bytesToRead + " bytes, got " + readTotal);
+ }
+ } catch (AzureBlobException e) {
+ throw new IOException("Failed to fetch range page", e);
+ }
+
+ cache.put(pageIdx, page);
+ return page;
+ }
+
+ private long pageIndex(long pos) {
+ return pos / pageSize;
+ }
+
+ private void ensureOpen() throws IOException {
+ if (closed) {
+ throw new IOException("IndexInput is closed");
+ }
+ }
+
+ private static final class LruPageCache extends LinkedHashMap<Long, byte[]> {
+ private final int maxEntries;
+
+ LruPageCache(int maxEntries) {
+ super(16, 0.75f, true);
+ this.maxEntries = maxEntries;
+ }
+
+ @Override
+ protected boolean removeEldestEntry(Map.Entry<Long, byte[]> eldest) {
+ return size() > maxEntries;
+ }
+
+ @Override
+ public void clear() {
Review Comment:
Useless override
##########
solr/modules/azure-blob-repository/src/java/org/apache/solr/azureblob/AzureBlobOutputStream.java:
##########
@@ -0,0 +1,269 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.azureblob;
+
+import com.azure.storage.blob.BlobClient;
+import com.azure.storage.blob.models.BlobStorageException;
+import com.azure.storage.blob.specialized.BlockBlobClient;
+import java.io.ByteArrayInputStream;
+import java.io.IOException;
+import java.io.OutputStream;
+import java.lang.invoke.MethodHandles;
+import java.nio.ByteBuffer;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Base64;
+import java.util.List;
+import java.util.UUID;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * OutputStream implementation for Azure Blob Storage using block blobs. Supports chunked uploads
+ * for large files.
+ */
+public class AzureBlobOutputStream extends OutputStream {
+ private static final Logger log = LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+
+ static final int BLOCK_SIZE = 4 * 1024 * 1024;
+
+ private final BlobClient blobClient;
+ private final String blobPath;
+ private volatile boolean closed;
Review Comment:
This is the only volatile field. I don't think multi-thread should be considered here.
##########
solr/modules/azure-blob-repository/src/java/org/apache/solr/azureblob/AzureBlobIndexInput.java:
##########
@@ -0,0 +1,200 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.azureblob;
+
+import java.io.EOFException;
+import java.io.IOException;
+import java.io.InputStream;
+import java.util.LinkedHashMap;
+import java.util.Map;
+import org.apache.lucene.store.IndexInput;
+
+class AzureBlobIndexInput extends IndexInput {
+
+ private static final int MIN_PAGE_SIZE = 4 * 1024;
+ private static final int DEFAULT_PAGE_SIZE = 512 * 1024;
+ private static final int MAX_CACHED_PAGES = 128;
+
+ private final String path;
+ private final AzureBlobStorageClient client;
+ private final long length;
+ private final int pageSize;
+ private final LruPageCache cache;
+
+ private long position = 0L;
+ private boolean closed = false;
+
+ AzureBlobIndexInput(String path, AzureBlobStorageClient client, long length) {
+ this(path, client, length, DEFAULT_PAGE_SIZE, MAX_CACHED_PAGES);
+ }
+
+ AzureBlobIndexInput(
+ String path, AzureBlobStorageClient client, long length, int pageSize, int maxCachedPages) {
+ super(path);
+ this.path = path;
+ this.client = client;
+ this.length = length;
+ this.pageSize = Math.max(MIN_PAGE_SIZE, pageSize);
+ this.cache = new LruPageCache(maxCachedPages);
+ }
+
+ @Override
+ public void close() throws IOException {
+ closed = true;
+ cache.clear();
+ }
+
+ @Override
+ public long getFilePointer() {
+ return position;
+ }
+
+ @Override
+ public void seek(long pos) throws IOException {
+ ensureOpen();
+ if (pos < 0 || pos > length) {
+ throw new IOException("Seek position out of bounds: " + pos);
+ }
+
+ position = pos;
+ }
+
+ @Override
+ public long length() {
+ return length;
+ }
+
+ @Override
+ public IndexInput slice(String sliceDescription, long offset, long length) throws IOException {
+ ensureOpen();
+ if (offset < 0 || length < 0 || offset + length > this.length) {
+ throw new IOException("Slice out of bounds: offset=" + offset + ", length=" + length);
+ }
+
+ AzureBlobIndexInput slice =
+ new AzureBlobIndexInput(
Review Comment:
This seems broken to me. We pass the description as `path`. If we create a slice from a slice, there is a high change we end if fetching a bad object.
##########
solr/modules/azure-blob-repository/src/java/org/apache/solr/azureblob/AzureBlobIndexInput.java:
##########
@@ -0,0 +1,200 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.azureblob;
+
+import java.io.EOFException;
+import java.io.IOException;
+import java.io.InputStream;
+import java.util.LinkedHashMap;
+import java.util.Map;
+import org.apache.lucene.store.IndexInput;
+
+class AzureBlobIndexInput extends IndexInput {
+
+ private static final int MIN_PAGE_SIZE = 4 * 1024;
+ private static final int DEFAULT_PAGE_SIZE = 512 * 1024;
+ private static final int MAX_CACHED_PAGES = 128;
+
+ private final String path;
+ private final AzureBlobStorageClient client;
+ private final long length;
+ private final int pageSize;
+ private final LruPageCache cache;
+
+ private long position = 0L;
+ private boolean closed = false;
+
+ AzureBlobIndexInput(String path, AzureBlobStorageClient client, long length) {
+ this(path, client, length, DEFAULT_PAGE_SIZE, MAX_CACHED_PAGES);
+ }
+
+ AzureBlobIndexInput(
+ String path, AzureBlobStorageClient client, long length, int pageSize, int maxCachedPages) {
+ super(path);
+ this.path = path;
+ this.client = client;
+ this.length = length;
+ this.pageSize = Math.max(MIN_PAGE_SIZE, pageSize);
+ this.cache = new LruPageCache(maxCachedPages);
+ }
+
+ @Override
+ public void close() throws IOException {
+ closed = true;
+ cache.clear();
+ }
+
+ @Override
+ public long getFilePointer() {
+ return position;
+ }
+
+ @Override
+ public void seek(long pos) throws IOException {
+ ensureOpen();
+ if (pos < 0 || pos > length) {
+ throw new IOException("Seek position out of bounds: " + pos);
+ }
+
+ position = pos;
+ }
+
+ @Override
+ public long length() {
+ return length;
+ }
+
+ @Override
+ public IndexInput slice(String sliceDescription, long offset, long length) throws IOException {
+ ensureOpen();
+ if (offset < 0 || length < 0 || offset + length > this.length) {
+ throw new IOException("Slice out of bounds: offset=" + offset + ", length=" + length);
+ }
+
+ AzureBlobIndexInput slice =
+ new AzureBlobIndexInput(
+ getFullSliceDescription(sliceDescription), client, length, pageSize, MAX_CACHED_PAGES);
+
+ slice.position = 0L;
Review Comment:
Position is already set to 0 at construction
##########
solr/modules/azure-blob-repository/src/java/org/apache/solr/azureblob/AzureBlobIndexInput.java:
##########
@@ -0,0 +1,200 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.azureblob;
+
+import java.io.EOFException;
+import java.io.IOException;
+import java.io.InputStream;
+import java.util.LinkedHashMap;
+import java.util.Map;
+import org.apache.lucene.store.IndexInput;
+
+class AzureBlobIndexInput extends IndexInput {
+
+ private static final int MIN_PAGE_SIZE = 4 * 1024;
+ private static final int DEFAULT_PAGE_SIZE = 512 * 1024;
+ private static final int MAX_CACHED_PAGES = 128;
+
+ private final String path;
+ private final AzureBlobStorageClient client;
+ private final long length;
+ private final int pageSize;
+ private final LruPageCache cache;
+
+ private long position = 0L;
+ private boolean closed = false;
+
+ AzureBlobIndexInput(String path, AzureBlobStorageClient client, long length) {
+ this(path, client, length, DEFAULT_PAGE_SIZE, MAX_CACHED_PAGES);
+ }
+
+ AzureBlobIndexInput(
+ String path, AzureBlobStorageClient client, long length, int pageSize, int maxCachedPages) {
+ super(path);
+ this.path = path;
+ this.client = client;
+ this.length = length;
+ this.pageSize = Math.max(MIN_PAGE_SIZE, pageSize);
+ this.cache = new LruPageCache(maxCachedPages);
+ }
+
+ @Override
+ public void close() throws IOException {
+ closed = true;
+ cache.clear();
+ }
+
+ @Override
+ public long getFilePointer() {
+ return position;
+ }
+
+ @Override
+ public void seek(long pos) throws IOException {
+ ensureOpen();
+ if (pos < 0 || pos > length) {
+ throw new IOException("Seek position out of bounds: " + pos);
Review Comment:
Should be `EOFException`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17949: Add Azure Blob Storage backup repository module [solr]
Posted by "prateeksinghalgit (via GitHub)" <gi...@apache.org>.
prateeksinghalgit commented on code in PR #3750:
URL: https://github.com/apache/solr/pull/3750#discussion_r3232090766
##########
solr/modules/azure-blob-repository/src/java/org/apache/solr/azureblob/AzureBlobStorageClient.java:
##########
@@ -0,0 +1,529 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.azureblob;
+
+import com.azure.core.credential.TokenCredential;
+import com.azure.identity.DefaultAzureCredentialBuilder;
+import com.azure.storage.blob.BlobClient;
+import com.azure.storage.blob.BlobContainerClient;
+import com.azure.storage.blob.BlobServiceClient;
+import com.azure.storage.blob.BlobServiceClientBuilder;
+import com.azure.storage.blob.models.BlobItem;
+import com.azure.storage.blob.models.BlobStorageException;
+import com.azure.storage.blob.models.ListBlobsOptions;
+import com.google.common.annotations.VisibleForTesting;
+import java.io.ByteArrayInputStream;
+import java.io.FilterInputStream;
+import java.io.InputStream;
+import java.io.OutputStream;
+import java.lang.invoke.MethodHandles;
+import java.util.Collection;
+import java.util.HashSet;
+import java.util.Locale;
+import java.util.Set;
+import java.util.stream.Collectors;
+import org.apache.solr.common.util.ResumableInputStream;
+import org.apache.solr.common.util.StrUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Creates a {@link BlobServiceClient} for communicating with Azure Blob Storage. Utilizes the
+ * default Azure credential provider chain.
+ */
+public class AzureBlobStorageClient {
+
+ private static final Logger log = LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+
+ static final String BLOB_FILE_PATH_DELIMITER = "/";
+ private static final int HTTP_NOT_FOUND = 404;
+ private static final int HTTP_CONFLICT = 409;
+ private static final int SKIP_BUFFER_SIZE = 8192;
+ private static final int DELETE_BATCH_SIZE = 1000;
+
+ private static final com.azure.core.http.HttpClient SHARED_HTTP_CLIENT =
+ new com.azure.core.http.okhttp.OkHttpAsyncHttpClientBuilder().build();
+
+ private final BlobContainerClient containerClient;
+
+ AzureBlobStorageClient(
+ String containerName,
+ String connectionString,
+ String endpoint,
+ String accountName,
+ String accountKey,
+ String sasToken,
+ String tenantId,
+ String clientId,
+ String clientSecret) {
+ this(
+ createInternalClient(
+ connectionString,
+ endpoint,
+ accountName,
+ accountKey,
+ sasToken,
+ tenantId,
+ clientId,
+ clientSecret),
+ containerName);
+ }
+
+ @VisibleForTesting
+ AzureBlobStorageClient(BlobServiceClient blobServiceClient, String containerName) {
+ this.containerClient = blobServiceClient.getBlobContainerClient(containerName);
+ try {
+ containerClient.create();
+ } catch (BlobStorageException e) {
+ if (e.getStatusCode() != HTTP_CONFLICT) {
+ throw e;
+ }
+ }
+ }
+
+ private static BlobServiceClient createInternalClient(
+ String connectionString,
+ String endpoint,
+ String accountName,
+ String accountKey,
+ String sasToken,
+ String tenantId,
+ String clientId,
+ String clientSecret) {
+
+ BlobServiceClientBuilder builder = new BlobServiceClientBuilder();
+ builder.httpClient(SHARED_HTTP_CLIENT);
+
+ if (StrUtils.isNotNullOrEmpty(connectionString)) {
+ builder.connectionString(connectionString);
+ } else if (StrUtils.isNotNullOrEmpty(endpoint)) {
+ builder.endpoint(endpoint);
+ if (StrUtils.isNotNullOrEmpty(accountName) && StrUtils.isNotNullOrEmpty(accountKey)) {
+ builder.credential(
+ new com.azure.storage.common.StorageSharedKeyCredential(accountName, accountKey));
+ } else if (StrUtils.isNotNullOrEmpty(sasToken)) {
+ builder.sasToken(sasToken);
+ } else {
+ TokenCredential credential = new DefaultAzureCredentialBuilder().tenantId(tenantId).build();
+ builder.credential(credential);
+ }
+ } else {
+ throw new IllegalArgumentException("Either connectionString or endpoint must be provided");
+ }
+
+ return builder.buildClient();
+ }
+
+ void createDirectory(String path) throws AzureBlobException {
+ String sanitizedDirPath = sanitizedDirPath(path);
+
+ if (!pathExists(sanitizedDirPath)) {
+ String parent = getParentDirectory(sanitizedDirPath);
+ if (!parent.isEmpty() && !parent.equals(BLOB_FILE_PATH_DELIMITER)) {
+ createDirectory(parent);
+ }
+
+ try {
+ BlobClient blobClient = containerClient.getBlobClient(sanitizedDirPath);
+ blobClient.upload(new ByteArrayInputStream(new byte[0]), 0, true);
+ java.util.Map<String, String> metadata = new java.util.HashMap<>();
+ metadata.put("hdi_isfolder", "true");
+ blobClient.setMetadata(metadata);
+ } catch (BlobStorageException e) {
+ throw handleBlobException(e);
+ }
+ }
+ }
+
+ void delete(Collection<String> paths) throws AzureBlobException {
+ Set<String> entries = new HashSet<>();
+ for (String path : paths) {
+ entries.add(sanitizedFilePath(path));
+ }
+ deleteBlobs(entries);
+ }
+
+ void deleteDirectory(String path) throws AzureBlobException {
+ path = sanitizedDirPath(path);
+
+ Set<String> entries = listAll(path);
+ if (pathExists(path)) {
+ entries.add(path);
+ }
+
+ deleteBlobs(entries);
+ }
+
+ String[] listDir(String path) throws AzureBlobException {
+ path = sanitizedDirPath(path);
+
+ try {
+ ListBlobsOptions options = new ListBlobsOptions().setPrefix(path).setMaxResultsPerPage(1000);
+
+ final String finalPath = path;
+ return containerClient.listBlobs(options, null).stream()
+ .map(BlobItem::getName)
+ .filter(s -> s.startsWith(finalPath))
+ .map(s -> s.substring(finalPath.length()))
+ .filter(s -> !s.isEmpty())
+ .filter(
+ s -> {
+ int slashIndex = s.indexOf(BLOB_FILE_PATH_DELIMITER);
+ return slashIndex == -1 || slashIndex == s.length() - 1;
+ })
+ .toArray(String[]::new);
+ } catch (BlobStorageException e) {
+ throw handleBlobException(e);
+ }
+ }
+
+ boolean pathExists(String path) throws AzureBlobException {
+ final String blobPath = sanitizedPath(path);
+
+ if (blobPath.isEmpty() || BLOB_FILE_PATH_DELIMITER.equals(blobPath)) {
+ return true;
+ }
+
+ try {
+ BlobClient blobClient = containerClient.getBlobClient(blobPath);
+ return blobClient.exists();
+ } catch (BlobStorageException e) {
+ throw handleBlobException(e);
+ }
+ }
+
+ boolean isDirectory(String path) throws AzureBlobException {
+ final String dirPrefix = sanitizedDirPath(path);
+
+ try {
+ ListBlobsOptions options =
+ new ListBlobsOptions().setPrefix(dirPrefix).setMaxResultsPerPage(1);
+ if (containerClient.listBlobs(options, null).iterator().hasNext()) {
+ return true;
+ }
+
+ BlobClient markerClient = containerClient.getBlobClient(dirPrefix);
+ if (markerClient.exists()) {
+ long size = markerClient.getProperties().getBlobSize();
+ if (size == 0) {
+ return true;
+ }
+ java.util.Map<String, String> md = markerClient.getProperties().getMetadata();
+ return md != null && md.containsKey("hdi_isfolder");
+ }
+
+ return false;
+ } catch (BlobStorageException e) {
+ throw handleBlobException(e);
+ }
+ }
+
+ long length(String path) throws AzureBlobException {
+ String blobPath = sanitizedFilePath(path);
+ try {
+ BlobClient blobClient = containerClient.getBlobClient(blobPath);
+ return blobClient.getProperties().getBlobSize();
+ } catch (BlobStorageException e) {
+ throw handleBlobException(e);
+ }
+ }
+
+ InputStream pullStream(String path) throws AzureBlobException {
+ final String blobPath = sanitizedFilePath(path);
+
+ try {
+ BlobClient blobClient = containerClient.getBlobClient(blobPath);
+ final long contentLength = blobClient.getProperties().getBlobSize();
+
+ if (contentLength == 0) {
+ return new ByteArrayInputStream(new byte[0]);
+ }
+
+ InputStream initial = new IdempotentCloseInputStream(blobClient.openInputStream());
+
+ return new ResumableInputStream(
+ initial,
+ bytesRead -> {
+ if (contentLength > 0 && bytesRead >= contentLength) {
+ return null;
+ }
+ try {
+ long remaining =
+ contentLength > 0 ? Math.max(0, contentLength - bytesRead) : Long.MAX_VALUE;
+ return pullRangeStream(path, bytesRead, remaining);
+ } catch (AzureBlobException e) {
+ throw new RuntimeException(e);
+ }
+ });
+ } catch (BlobStorageException e) {
+ throw handleBlobException(e);
+ }
+ }
+
+ InputStream pullRangeStream(String path, long offset, long length) throws AzureBlobException {
+ final String blobPath = sanitizedFilePath(path);
+ try {
+ BlobClient blobClient = containerClient.getBlobClient(blobPath);
+ com.azure.storage.blob.models.BlobRange range =
+ new com.azure.storage.blob.models.BlobRange(offset, length);
+ return new IdempotentCloseInputStream(blobClient.openInputStream(range, null));
+ } catch (BlobStorageException e) {
+ throw handleBlobException(e);
+ }
+ }
+
+ private static final class IdempotentCloseInputStream extends FilterInputStream {
+ private boolean closed;
+
+ IdempotentCloseInputStream(InputStream in) {
+ super(in);
+ this.closed = false;
+ }
+
+ @Override
+ public int read() throws java.io.IOException {
+ if (closed) {
+ throw new java.io.IOException("Stream is already closed");
+ }
+ try {
+ return super.read();
+ } catch (RuntimeException re) {
+ if (isAlreadyClosed(re)) {
+ throw new java.io.IOException("Stream is already closed", re);
+ }
+ throw re;
+ }
+ }
+
+ @Override
+ public int read(byte[] b, int off, int len) throws java.io.IOException {
+ if (closed) {
+ throw new java.io.IOException("Stream is already closed");
+ }
+ try {
+ return super.read(b, off, len);
+ } catch (RuntimeException re) {
+ if (isAlreadyClosed(re)) {
+ throw new java.io.IOException("Stream is already closed", re);
+ }
+ throw re;
+ }
+ }
+
+ @Override
+ public void close() throws java.io.IOException {
+ if (closed) {
+ return;
+ }
+ try {
+ super.close();
+ } catch (java.io.IOException e) {
+ String msg = e.getMessage();
+ if (msg == null || !msg.toLowerCase(java.util.Locale.ROOT).contains("already closed")) {
+ throw e;
+ }
+ // swallow "already closed" to make close idempotent
+ } finally {
+ closed = true;
+ }
+ }
+
+ @Override
+ public long skip(long n) throws java.io.IOException {
+ if (closed) {
+ throw new java.io.IOException("Stream is already closed");
+ }
+ if (n <= 0) {
+ return 0L;
+ }
+ long remaining = n;
+ byte[] discard = new byte[SKIP_BUFFER_SIZE];
+ try {
+ while (remaining > 0) {
+ int toRead = (int) Math.min(discard.length, remaining);
+ int read = super.read(discard, 0, toRead);
+ if (read < 0) {
+ break;
+ }
+ remaining -= read;
+ }
+ return n - remaining;
+ } catch (RuntimeException re) {
+ throw new java.io.IOException(re);
+ }
+ }
+
+ private static boolean isAlreadyClosed(Throwable t) {
+ String msg = t.getMessage();
+ return msg != null && msg.toLowerCase(java.util.Locale.ROOT).contains("already closed");
+ }
+ }
+
+ OutputStream pushStream(String path) throws AzureBlobException {
+ path = sanitizedFilePath(path);
+
+ if (!parentDirectoryExist(path)) {
+ String parentDirectory = getParentDirectory(path);
+ if (!parentDirectory.isEmpty() && !parentDirectory.equals(BLOB_FILE_PATH_DELIMITER)) {
+ createDirectory(parentDirectory);
+ }
+ }
+
+ try {
+ BlobClient blobClient = containerClient.getBlobClient(path);
+ return new AzureBlobOutputStream(blobClient, path);
+ } catch (BlobStorageException e) {
+ throw handleBlobException(e);
+ }
+ }
+
+ void close() {}
+
+ @VisibleForTesting
+ void deleteContainerForTests() {
+ try {
+ containerClient.delete();
+ } catch (BlobStorageException e) {
+ if (e.getStatusCode() != HTTP_NOT_FOUND) {
+ throw e;
+ }
+ }
+ }
+
+ private Collection<String> deleteBlobs(Collection<String> paths) throws AzureBlobException {
+ try {
+ return deleteBlobs(paths, DELETE_BATCH_SIZE);
+ } catch (BlobStorageException e) {
+ throw handleBlobException(e);
+ }
+ }
+
+ @VisibleForTesting
+ Collection<String> deleteBlobs(Collection<String> entries, int batchSize)
+ throws AzureBlobException {
+ Set<String> deletedPaths = new HashSet<>();
+
+ for (String path : entries) {
Review Comment:
agreed. Updated it to do batch deletes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17949: Add Azure Blob Storage backup repository module [solr]
Posted by "prateeksinghalgit (via GitHub)" <gi...@apache.org>.
prateeksinghalgit commented on code in PR #3750:
URL: https://github.com/apache/solr/pull/3750#discussion_r3232194020
##########
solr/modules/azure-blob-repository/src/java/org/apache/solr/azureblob/AzureBlobIndexInput.java:
##########
@@ -0,0 +1,200 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.azureblob;
+
+import java.io.EOFException;
+import java.io.IOException;
+import java.io.InputStream;
+import java.util.LinkedHashMap;
+import java.util.Map;
+import org.apache.lucene.store.IndexInput;
+
+class AzureBlobIndexInput extends IndexInput {
+
+ private static final int MIN_PAGE_SIZE = 4 * 1024;
+ private static final int DEFAULT_PAGE_SIZE = 512 * 1024;
+ private static final int MAX_CACHED_PAGES = 128;
+
+ private final String path;
+ private final AzureBlobStorageClient client;
+ private final long length;
+ private final int pageSize;
+ private final LruPageCache cache;
+
+ private long position = 0L;
+ private boolean closed = false;
+
+ AzureBlobIndexInput(String path, AzureBlobStorageClient client, long length) {
+ this(path, client, length, DEFAULT_PAGE_SIZE, MAX_CACHED_PAGES);
+ }
+
+ AzureBlobIndexInput(
+ String path, AzureBlobStorageClient client, long length, int pageSize, int maxCachedPages) {
+ super(path);
+ this.path = path;
+ this.client = client;
+ this.length = length;
+ this.pageSize = Math.max(MIN_PAGE_SIZE, pageSize);
+ this.cache = new LruPageCache(maxCachedPages);
+ }
+
+ @Override
+ public void close() throws IOException {
+ closed = true;
+ cache.clear();
+ }
+
+ @Override
+ public long getFilePointer() {
+ return position;
+ }
+
+ @Override
+ public void seek(long pos) throws IOException {
+ ensureOpen();
+ if (pos < 0 || pos > length) {
+ throw new IOException("Seek position out of bounds: " + pos);
+ }
+
+ position = pos;
+ }
+
+ @Override
+ public long length() {
+ return length;
+ }
+
+ @Override
+ public IndexInput slice(String sliceDescription, long offset, long length) throws IOException {
+ ensureOpen();
+ if (offset < 0 || length < 0 || offset + length > this.length) {
+ throw new IOException("Slice out of bounds: offset=" + offset + ", length=" + length);
+ }
+
+ AzureBlobIndexInput slice =
+ new AzureBlobIndexInput(
+ getFullSliceDescription(sliceDescription), client, length, pageSize, MAX_CACHED_PAGES);
+
+ slice.position = 0L;
+
+ // Wrap client in a view that remaps range requests by adding base offset
+ slice.clientViewBaseOffset = this.clientViewBaseOffset + offset;
+ return slice;
+ }
+
+ @Override
+ public byte readByte() throws IOException {
+ ensureOpen();
+ if (position >= length) {
+ throw new EOFException("End of stream reached");
+ }
+
+ byte[] page = getPage(pageIndex(position));
+ int inPageOffset = (int) (position % pageSize);
+ byte value = page[inPageOffset];
+ position += 1L;
+ return value;
+ }
+
+ @Override
+ public void readBytes(byte[] b, int offset, int len) throws IOException {
+ ensureOpen();
+ if (len < 0) {
+ throw new IOException("Length must be non-negative");
+ }
+
+ if (position + len > length) {
+ throw new EOFException("End of stream reached");
+ }
+
+ int remaining = len;
+ while (remaining > 0) {
+ long pageIdx = pageIndex(position);
+ byte[] page = getPage(pageIdx);
+ int inPageOffset = (int) (position % pageSize);
+ int toCopy = Math.min(remaining, pageSize - inPageOffset);
+ System.arraycopy(page, inPageOffset, b, offset + (len - remaining), toCopy);
+ position += toCopy;
+ remaining -= toCopy;
+ }
+ }
+
+ // Internal state for slices: base offset to add to all range requests
+ private long clientViewBaseOffset = 0L;
Review Comment:
no longer being used.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17949: Add Azure Blob Storage backup repository module [solr]
Posted by "prateeksinghalgit (via GitHub)" <gi...@apache.org>.
prateeksinghalgit commented on code in PR #3750:
URL: https://github.com/apache/solr/pull/3750#discussion_r3232226265
##########
solr/modules/azure-blob-repository/src/java/org/apache/solr/azureblob/AzureBlobStorageClient.java:
##########
@@ -0,0 +1,529 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.azureblob;
+
+import com.azure.core.credential.TokenCredential;
+import com.azure.identity.DefaultAzureCredentialBuilder;
+import com.azure.storage.blob.BlobClient;
+import com.azure.storage.blob.BlobContainerClient;
+import com.azure.storage.blob.BlobServiceClient;
+import com.azure.storage.blob.BlobServiceClientBuilder;
+import com.azure.storage.blob.models.BlobItem;
+import com.azure.storage.blob.models.BlobStorageException;
+import com.azure.storage.blob.models.ListBlobsOptions;
+import com.google.common.annotations.VisibleForTesting;
+import java.io.ByteArrayInputStream;
+import java.io.FilterInputStream;
+import java.io.InputStream;
+import java.io.OutputStream;
+import java.lang.invoke.MethodHandles;
+import java.util.Collection;
+import java.util.HashSet;
+import java.util.Locale;
+import java.util.Set;
+import java.util.stream.Collectors;
+import org.apache.solr.common.util.ResumableInputStream;
+import org.apache.solr.common.util.StrUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Creates a {@link BlobServiceClient} for communicating with Azure Blob Storage. Utilizes the
+ * default Azure credential provider chain.
+ */
+public class AzureBlobStorageClient {
+
+ private static final Logger log = LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+
+ static final String BLOB_FILE_PATH_DELIMITER = "/";
+ private static final int HTTP_NOT_FOUND = 404;
+ private static final int HTTP_CONFLICT = 409;
+ private static final int SKIP_BUFFER_SIZE = 8192;
+ private static final int DELETE_BATCH_SIZE = 1000;
+
+ private static final com.azure.core.http.HttpClient SHARED_HTTP_CLIENT =
+ new com.azure.core.http.okhttp.OkHttpAsyncHttpClientBuilder().build();
+
+ private final BlobContainerClient containerClient;
+
+ AzureBlobStorageClient(
+ String containerName,
+ String connectionString,
+ String endpoint,
+ String accountName,
+ String accountKey,
+ String sasToken,
+ String tenantId,
+ String clientId,
+ String clientSecret) {
+ this(
+ createInternalClient(
+ connectionString,
+ endpoint,
+ accountName,
+ accountKey,
+ sasToken,
+ tenantId,
+ clientId,
+ clientSecret),
+ containerName);
+ }
+
+ @VisibleForTesting
+ AzureBlobStorageClient(BlobServiceClient blobServiceClient, String containerName) {
+ this.containerClient = blobServiceClient.getBlobContainerClient(containerName);
+ try {
+ containerClient.create();
+ } catch (BlobStorageException e) {
+ if (e.getStatusCode() != HTTP_CONFLICT) {
+ throw e;
+ }
+ }
+ }
+
+ private static BlobServiceClient createInternalClient(
+ String connectionString,
+ String endpoint,
+ String accountName,
+ String accountKey,
+ String sasToken,
+ String tenantId,
+ String clientId,
+ String clientSecret) {
+
+ BlobServiceClientBuilder builder = new BlobServiceClientBuilder();
+ builder.httpClient(SHARED_HTTP_CLIENT);
+
+ if (StrUtils.isNotNullOrEmpty(connectionString)) {
+ builder.connectionString(connectionString);
+ } else if (StrUtils.isNotNullOrEmpty(endpoint)) {
+ builder.endpoint(endpoint);
+ if (StrUtils.isNotNullOrEmpty(accountName) && StrUtils.isNotNullOrEmpty(accountKey)) {
+ builder.credential(
+ new com.azure.storage.common.StorageSharedKeyCredential(accountName, accountKey));
+ } else if (StrUtils.isNotNullOrEmpty(sasToken)) {
+ builder.sasToken(sasToken);
+ } else {
+ TokenCredential credential = new DefaultAzureCredentialBuilder().tenantId(tenantId).build();
+ builder.credential(credential);
+ }
+ } else {
+ throw new IllegalArgumentException("Either connectionString or endpoint must be provided");
+ }
+
+ return builder.buildClient();
+ }
+
+ void createDirectory(String path) throws AzureBlobException {
+ String sanitizedDirPath = sanitizedDirPath(path);
+
+ if (!pathExists(sanitizedDirPath)) {
+ String parent = getParentDirectory(sanitizedDirPath);
+ if (!parent.isEmpty() && !parent.equals(BLOB_FILE_PATH_DELIMITER)) {
+ createDirectory(parent);
+ }
+
+ try {
+ BlobClient blobClient = containerClient.getBlobClient(sanitizedDirPath);
+ blobClient.upload(new ByteArrayInputStream(new byte[0]), 0, true);
Review Comment:
thanks for the suggestion. Made the change
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17949: Add Azure Blob Storage backup repository module [solr]
Posted by "prateeksinghalgit (via GitHub)" <gi...@apache.org>.
prateeksinghalgit commented on code in PR #3750:
URL: https://github.com/apache/solr/pull/3750#discussion_r3232296536
##########
solr/modules/azure-blob-repository/src/java/org/apache/solr/azureblob/AzureBlobOutputStream.java:
##########
@@ -0,0 +1,269 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.azureblob;
+
+import com.azure.storage.blob.BlobClient;
+import com.azure.storage.blob.models.BlobStorageException;
+import com.azure.storage.blob.specialized.BlockBlobClient;
+import java.io.ByteArrayInputStream;
+import java.io.IOException;
+import java.io.OutputStream;
+import java.lang.invoke.MethodHandles;
+import java.nio.ByteBuffer;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Base64;
+import java.util.List;
+import java.util.UUID;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * OutputStream implementation for Azure Blob Storage using block blobs. Supports chunked uploads
+ * for large files.
+ */
+public class AzureBlobOutputStream extends OutputStream {
+ private static final Logger log = LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+
+ static final int BLOCK_SIZE = 4 * 1024 * 1024;
+
+ private final BlobClient blobClient;
+ private final String blobPath;
+ private volatile boolean closed;
+ private final ByteBuffer buffer;
+ private BlockUpload blockUpload;
+ private boolean committed;
+
+ public AzureBlobOutputStream(BlobClient blobClient, String blobPath) {
+ this.blobClient = blobClient;
+ this.blobPath = blobPath;
+ this.closed = false;
+ this.buffer = ByteBuffer.allocate(BLOCK_SIZE);
+ this.blockUpload = null;
+ this.committed = false;
+
+ if (log.isDebugEnabled()) {
+ log.debug("Created BlobOutputStream for blobPath '{}'", blobPath);
+ }
+ }
+
+ @Override
+ public void write(int b) throws IOException {
+ if (closed) {
+ throw new IOException("Stream closed");
+ }
+
+ buffer.put((byte) b);
+
+ if (!buffer.hasRemaining()) {
+ uploadBlock();
+ }
+ }
+
+ @Override
+ public void write(byte[] b, int off, int len) throws IOException {
+ if (closed) {
+ throw new IOException("Stream closed");
+ }
+
+ if (outOfRange(off, b.length) || len < 0 || outOfRange(off + len, b.length)) {
+ throw new IndexOutOfBoundsException();
+ } else if (len == 0) {
+ return;
+ }
+
+ int currentOffset = off;
+ int lenRemaining = len;
+ while (buffer.remaining() < lenRemaining) {
+ int firstPart = buffer.remaining();
+ buffer.put(b, currentOffset, firstPart);
+ uploadBlock();
+
+ currentOffset += firstPart;
+ lenRemaining -= firstPart;
+ }
+ if (lenRemaining > 0) {
+ buffer.put(b, currentOffset, lenRemaining);
+ }
+ }
+
+ private static boolean outOfRange(int off, int len) {
+ return off < 0 || off > len;
+ }
+
+ private void uploadBlock() throws IOException {
+ int size = buffer.position() - buffer.arrayOffset();
+
+ if (size == 0) {
+ return;
+ }
+
+ if (blockUpload == null) {
+ if (log.isDebugEnabled()) {
+ log.debug("New block upload for blobPath '{}'", blobPath);
+ }
+
+ blockUpload = newBlockUpload();
+ }
+
+ try (ByteArrayInputStream inputStream =
+ new ByteArrayInputStream(buffer.array(), buffer.arrayOffset(), size)) {
+ blockUpload.uploadBlock(inputStream, size);
+ } catch (BlobStorageException e) {
+ if (blockUpload != null) {
+ blockUpload.abort();
+ if (log.isDebugEnabled()) {
+ log.debug("Block upload aborted for blobPath '{}'.", blobPath);
+ }
+ }
+
+ throw new IOException(
+ "Failed to upload block", AzureBlobStorageClient.handleBlobException(e));
+ }
+
+ buffer.clear();
+ }
+
+ @Override
+ public void flush() throws IOException {
+ if (closed) {
+ throw new IOException("Stream closed");
+ }
+
+ if (buffer.position() - buffer.arrayOffset() > 0) {
+ uploadBlock();
+ }
+
+ if (blockUpload != null) {
+ blockUpload.complete();
+ blockUpload = null;
+ committed = true;
+ }
+ }
+
+ @Override
+ public void close() throws IOException {
+ if (closed) {
+ return;
+ }
+
+ if (blockUpload != null && blockUpload.aborted) {
+ blockUpload = null;
+ closed = true;
+ return;
+ }
+
+ if (!committed) {
+ uploadBlock();
+ if (blockUpload != null) {
+ blockUpload.complete();
+ blockUpload = null;
+ committed = true;
+ } else {
+ try {
+ blobClient.upload(new ByteArrayInputStream(new byte[0]), 0, true);
+ } catch (BlobStorageException e) {
+ throw new IOException(
+ "Failed to create empty blob", AzureBlobStorageClient.handleBlobException(e));
+ }
+ }
+ } else {
+ if (blockUpload != null) {
+ blockUpload.complete();
+ blockUpload = null;
+ }
+ }
+
+ closed = true;
+ }
+
+ private BlockUpload newBlockUpload() throws IOException {
+ try {
+ return new BlockUpload();
+ } catch (BlobStorageException e) {
+ throw new IOException(
+ "Failed to create block upload", AzureBlobStorageClient.handleBlobException(e));
+ }
+ }
+
+ private class BlockUpload {
+ private final List<String> blockIds;
+ private boolean aborted = false;
+
+ public BlockUpload() {
+ this.blockIds = new ArrayList<>();
+ if (log.isDebugEnabled()) {
+ log.debug("Initiated block upload for blobPath '{}'", blobPath);
+ }
+
+ try {
+ BlockBlobClient blockBlobClient = blobClient.getBlockBlobClient();
+ blockBlobClient.deleteIfExists();
+ } catch (BlobStorageException e) {
+ // ignore; subsequent stage/commit will surface real issues
+ }
+ }
+
+ void uploadBlock(ByteArrayInputStream inputStream, long blockSize) {
+ if (aborted) {
+ throw new IllegalStateException(
+ "Can't upload new blocks on a BlockUpload that was aborted");
+ }
+
+ String blockId =
+ Base64.getEncoder()
+ .encodeToString(UUID.randomUUID().toString().getBytes(StandardCharsets.UTF_8));
+
+ if (log.isDebugEnabled()) {
+ log.debug("Uploading block {} for blobPath '{}'", blockId, blobPath);
+ }
+
+ try {
+ BlockBlobClient blockBlobClient = blobClient.getBlockBlobClient();
+ blockBlobClient.stageBlock(blockId, inputStream, blockSize);
+ blockIds.add(blockId);
+ } catch (BlobStorageException e) {
+ throw new RuntimeException("Failed to upload block", e);
Review Comment:
Agreed, made the change to IOException
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17949: Add Azure Blob Storage backup repository module [solr]
Posted by "prateeksinghalgit (via GitHub)" <gi...@apache.org>.
prateeksinghalgit commented on code in PR #3750:
URL: https://github.com/apache/solr/pull/3750#discussion_r2554734264
##########
solr/licenses/msal4j-NOTICE.txt:
##########
@@ -0,0 +1,12 @@
+Microsoft Authentication Library for Java (MSAL4J)
+Copyright (c) Microsoft Corporation. All rights reserved.
+
+This product includes software developed by
+Microsoft Corporation (https://github.com/AzureAD/microsoft-authentication-library-for-java).
+
+Licensed under the MIT License.
Review Comment:
made the change.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17949: Add Azure Blob Storage backup repository module [solr]
Posted by "janhoy (via GitHub)" <gi...@apache.org>.
janhoy commented on code in PR #3750:
URL: https://github.com/apache/solr/pull/3750#discussion_r2554975363
##########
gradle/libs.versions.toml:
##########
@@ -51,6 +51,10 @@ aqute-bnd = "6.4.1"
asciidoctor-mathjax = "0.0.9"
# @keep Asciidoctor tabs version used in ref-guide
asciidoctor-tabs = "1.0.0-beta.6"
+azure-storage = "12.25.0"
+azure-identity = "1.12.0"
+azure-core = "1.52.0"
+azure-core-http-netty = "1.15.4"
Review Comment:
Did you have a chance to look at the http client?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17949: Add Azure Blob Storage backup repository module [solr]
Posted by "prateeksinghalgit (via GitHub)" <gi...@apache.org>.
prateeksinghalgit commented on PR #3750:
URL: https://github.com/apache/solr/pull/3750#issuecomment-3567034117
> Only immediate comment is naming. «Blob-repository» is too generic. Should contain word «azure»?
made the change to azure-blob-repository
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17949: Add Azure Blob Storage backup repository module [solr]
Posted by "prateeksinghalgit (via GitHub)" <gi...@apache.org>.
prateeksinghalgit commented on code in PR #3750:
URL: https://github.com/apache/solr/pull/3750#discussion_r2563445786
##########
solr/modules/azure-blob-repository/src/test/org/apache/solr/azureblob/AbstractAzureBlobClientTest.java:
##########
@@ -0,0 +1,249 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.azureblob;
+
+import com.azure.core.http.HttpClient;
+import com.azure.core.http.okhttp.OkHttpAsyncHttpClientBuilder;
+import com.azure.storage.blob.BlobServiceClient;
+import com.azure.storage.blob.BlobServiceClientBuilder;
+import com.carrotsearch.randomizedtesting.ThreadFilter;
+import com.carrotsearch.randomizedtesting.annotations.ThreadLeakFilters;
+import java.io.IOException;
+import java.io.OutputStream;
+import java.net.Socket;
+import java.nio.charset.StandardCharsets;
+import okhttp3.OkHttpClient;
+import org.apache.solr.SolrTestCaseJ4;
+import org.junit.After;
+import org.junit.AfterClass;
+import org.junit.Before;
+import org.junit.BeforeClass;
+import org.junit.Rule;
+import org.junit.rules.TemporaryFolder;
+
+import static org.junit.Assume.assumeTrue;
+
+/**
+ * Abstract class for tests with Azure Blob Storage emulator.
+ *
+ * <p>Uses ThreadLeakFilters to ignore OkHttp's global singleton threads (TaskRunner and Okio
+ * Watchdog) which are JVM-wide and cannot be shut down per-client. This is a standard pattern in
+ * Solr tests for HTTP clients with global thread pools.
+ */
+@ThreadLeakFilters(
+ filters = {
+ AbstractAzureBlobClientTest.OkHttpThreadLeakFilter.class,
+ })
+public class AbstractAzureBlobClientTest extends SolrTestCaseJ4 {
+
+ protected String containerName;
+
+ @Rule public TemporaryFolder tempFolder = new TemporaryFolder();
+
+ AzureBlobStorageClient client;
+ private static String connectionString;
+ // Shared OkHttpClient across all tests to reuse the global thread pools
+ private static OkHttpClient sharedOkHttpClient;
+ protected org.apache.solr.client.solrj.cloud.SocketProxy proxy;
+
+ @BeforeClass
+ public static void setUpClass() {
+ // Create a single shared OkHttpClient for all tests
+ // This reuses OkHttp's global thread pools efficiently
+ sharedOkHttpClient = new OkHttpClient.Builder().build();
+ }
+
+ @Before
+ public void setUpClient() throws Exception {
+ // Check if Azurite is running before attempting to connect
+ assumeAzuriteIsRunning();
+
+ setAzureTestCredentials();
+
+ // Use Azurite connection string for local testing
+ connectionString =
+ "DefaultEndpointsProtocol=http;AccountName=devstoreaccount1;AccountKey=Eby8vdM02xNOcqFlqUwJPLlmEtlCDXJ1OUzFT50uSRZ6IFsuFq2UVErCz4I6tq/K1SZFPTOtr/KBHBeksoGMGw==;BlobEndpoint=http://localhost:10000/devstoreaccount1;";
+
+ // Put a proxy in front of Azurite to simulate connection loss like S3 tests
+ proxy = new org.apache.solr.client.solrj.cloud.SocketProxy();
+ proxy.open(new java.net.URI(getBlobServiceUrl()));
+
+ // Reuse the shared OkHttpClient
+ HttpClient httpClient = new OkHttpAsyncHttpClientBuilder(sharedOkHttpClient).build();
+
+ // Route Blob endpoint through the proxy by adjusting the connection string
+ String proxiedConn = connectionString.replace(":10000", ":" + proxy.getListenPort());
+ BlobServiceClient blobServiceClient =
+ new BlobServiceClientBuilder()
+ .connectionString(proxiedConn)
+ .httpClient(httpClient)
+ .buildClient();
+
+ containerName = "test-" + java.util.UUID.randomUUID();
+ client = new AzureBlobStorageClient(blobServiceClient, containerName);
+ }
+
+ /**
+ * Set up Azure test credentials to avoid using real Azure credentials during testing. Similar to
+ * how S3 tests use ProfileFileSystemSetting to avoid polluting the test environment.
+ */
+ public static void setAzureTestCredentials() {
+ // Set test Azure credentials to avoid using real credentials
+ System.setProperty("AZURE_CLIENT_ID", "test-client-id");
+ System.setProperty("AZURE_TENANT_ID", "test-tenant-id");
+ System.setProperty("AZURE_CLIENT_SECRET", "test-client-secret");
+
+ // Set Azurite-specific environment variables
+ System.setProperty(
+ "AZURE_STORAGE_CONNECTION_STRING",
+ "DefaultEndpointsProtocol=http;AccountName=devstoreaccount1;AccountKey=Eby8vdM02xNOcqFlqUwJPLlmEtlCDXJ1OUzFT50uSRZ6IFsuFq2UVErCz4I6tq/K1SZFPTOtr/KBHBeksoGMGw==;BlobEndpoint=http://localhost:10000/devstoreaccount1;");
+ }
+
+ @After
+ public void tearDownClient() {
+ if (client != null) {
+ try {
+ client.deleteContainerForTests();
+ } catch (Throwable ignored) {
+ }
+ client.close();
+ }
+ if (proxy != null) {
+ proxy.close();
+ proxy = null;
+ }
+ // Note: We don't clean up sharedOkHttpClient here as it's shared across all tests
+ // It will be cleaned up in afterAll()
+ }
+
+ /** Simulate a connection loss on the proxy similar to S3 tests. */
+ void initiateBlobConnectionLoss() throws AzureBlobException {
+ if (proxy != null) {
+ proxy.halfClose();
+ }
+ }
+
+ @AfterClass
+ public static void afterAll() {
+ // Clean up the shared OkHttpClient
+ if (sharedOkHttpClient != null) {
+ // Shutdown dispatcher's executor service
+ sharedOkHttpClient.dispatcher().executorService().shutdown();
+ // Cancel all queued calls
+ sharedOkHttpClient.dispatcher().cancelAll();
+ // Evict all connections
+ sharedOkHttpClient.connectionPool().evictAll();
+ // Close cache if present
+ try {
+ if (sharedOkHttpClient.cache() != null) {
+ sharedOkHttpClient.cache().close();
+ }
+ } catch (Throwable ignored) {
+ }
+ // Wait for threads to terminate
+ try {
+ sharedOkHttpClient
+ .dispatcher()
+ .executorService()
+ .awaitTermination(2, java.util.concurrent.TimeUnit.SECONDS);
+ } catch (Throwable ignored) {
+ }
+ sharedOkHttpClient = null;
+ }
+
+ // Shutdown Reactor schedulers to clean up background threads
+ try {
+ reactor.core.scheduler.Schedulers.shutdownNow();
+ // Give threads time to terminate
+ Thread.sleep(100);
+ } catch (Throwable ignored) {
+ }
+ }
+
+ /**
+ * Helper method to push a string to Azure Blob Storage.
+ *
+ * @param path Destination path in blob storage.
+ * @param content Arbitrary content for the test.
+ */
+ void pushContent(String path, String content) throws AzureBlobException {
+ pushContent(path, content.getBytes(StandardCharsets.UTF_8));
+ }
+
+ void pushContent(String path, byte[] content) throws AzureBlobException {
+ try (OutputStream output = client.pushStream(path)) {
+ output.write(content);
+ } catch (IOException e) {
+ throw new AzureBlobException("Failed to write content", e);
+ }
+ }
+
+ /** Get the connection string for tests that need direct access to the blob service. */
+ static String getConnectionString() {
+ return connectionString;
+ }
+
+ /** Get the blob service URL for tests that need direct access. */
+ String getBlobServiceUrl() {
+ return "http://localhost:10000";
+ }
+
+ /**
+ * Check if Azurite is running on localhost:10000. If not, skip the test.
+ *
+ * <p>Azurite must be started manually for these tests to run. This allows tests to pass in CI
+ * environments where Azurite may not be available.
+ */
+ private void assumeAzuriteIsRunning() {
+ boolean azuriteRunning = false;
+ try (Socket socket = new Socket("localhost", 10000)) {
Review Comment:
made the change to use testcontainers, and using azurite in that test container. Testcontainers automatically maps it to a random free host port, retrieving it via azuriteContainer.getMappedPort(BLOB_SERVICE_PORT) in getBlobServiceUrl(). There shouldn't be any conflict risk.
##########
solr/modules/azure-blob-repository/src/test/org/apache/solr/azureblob/AbstractAzureBlobClientTest.java:
##########
@@ -0,0 +1,249 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.azureblob;
+
+import com.azure.core.http.HttpClient;
+import com.azure.core.http.okhttp.OkHttpAsyncHttpClientBuilder;
+import com.azure.storage.blob.BlobServiceClient;
+import com.azure.storage.blob.BlobServiceClientBuilder;
+import com.carrotsearch.randomizedtesting.ThreadFilter;
+import com.carrotsearch.randomizedtesting.annotations.ThreadLeakFilters;
+import java.io.IOException;
+import java.io.OutputStream;
+import java.net.Socket;
+import java.nio.charset.StandardCharsets;
+import okhttp3.OkHttpClient;
+import org.apache.solr.SolrTestCaseJ4;
+import org.junit.After;
+import org.junit.AfterClass;
+import org.junit.Before;
+import org.junit.BeforeClass;
+import org.junit.Rule;
+import org.junit.rules.TemporaryFolder;
+
+import static org.junit.Assume.assumeTrue;
+
+/**
+ * Abstract class for tests with Azure Blob Storage emulator.
+ *
+ * <p>Uses ThreadLeakFilters to ignore OkHttp's global singleton threads (TaskRunner and Okio
+ * Watchdog) which are JVM-wide and cannot be shut down per-client. This is a standard pattern in
+ * Solr tests for HTTP clients with global thread pools.
+ */
+@ThreadLeakFilters(
+ filters = {
+ AbstractAzureBlobClientTest.OkHttpThreadLeakFilter.class,
+ })
+public class AbstractAzureBlobClientTest extends SolrTestCaseJ4 {
+
+ protected String containerName;
+
+ @Rule public TemporaryFolder tempFolder = new TemporaryFolder();
+
+ AzureBlobStorageClient client;
+ private static String connectionString;
+ // Shared OkHttpClient across all tests to reuse the global thread pools
+ private static OkHttpClient sharedOkHttpClient;
+ protected org.apache.solr.client.solrj.cloud.SocketProxy proxy;
+
+ @BeforeClass
+ public static void setUpClass() {
+ // Create a single shared OkHttpClient for all tests
+ // This reuses OkHttp's global thread pools efficiently
+ sharedOkHttpClient = new OkHttpClient.Builder().build();
+ }
+
+ @Before
+ public void setUpClient() throws Exception {
+ // Check if Azurite is running before attempting to connect
+ assumeAzuriteIsRunning();
+
+ setAzureTestCredentials();
+
+ // Use Azurite connection string for local testing
+ connectionString =
+ "DefaultEndpointsProtocol=http;AccountName=devstoreaccount1;AccountKey=Eby8vdM02xNOcqFlqUwJPLlmEtlCDXJ1OUzFT50uSRZ6IFsuFq2UVErCz4I6tq/K1SZFPTOtr/KBHBeksoGMGw==;BlobEndpoint=http://localhost:10000/devstoreaccount1;";
+
+ // Put a proxy in front of Azurite to simulate connection loss like S3 tests
+ proxy = new org.apache.solr.client.solrj.cloud.SocketProxy();
+ proxy.open(new java.net.URI(getBlobServiceUrl()));
+
+ // Reuse the shared OkHttpClient
+ HttpClient httpClient = new OkHttpAsyncHttpClientBuilder(sharedOkHttpClient).build();
+
+ // Route Blob endpoint through the proxy by adjusting the connection string
+ String proxiedConn = connectionString.replace(":10000", ":" + proxy.getListenPort());
+ BlobServiceClient blobServiceClient =
+ new BlobServiceClientBuilder()
+ .connectionString(proxiedConn)
+ .httpClient(httpClient)
+ .buildClient();
+
+ containerName = "test-" + java.util.UUID.randomUUID();
+ client = new AzureBlobStorageClient(blobServiceClient, containerName);
+ }
+
+ /**
+ * Set up Azure test credentials to avoid using real Azure credentials during testing. Similar to
+ * how S3 tests use ProfileFileSystemSetting to avoid polluting the test environment.
+ */
+ public static void setAzureTestCredentials() {
+ // Set test Azure credentials to avoid using real credentials
+ System.setProperty("AZURE_CLIENT_ID", "test-client-id");
+ System.setProperty("AZURE_TENANT_ID", "test-tenant-id");
+ System.setProperty("AZURE_CLIENT_SECRET", "test-client-secret");
+
+ // Set Azurite-specific environment variables
+ System.setProperty(
+ "AZURE_STORAGE_CONNECTION_STRING",
+ "DefaultEndpointsProtocol=http;AccountName=devstoreaccount1;AccountKey=Eby8vdM02xNOcqFlqUwJPLlmEtlCDXJ1OUzFT50uSRZ6IFsuFq2UVErCz4I6tq/K1SZFPTOtr/KBHBeksoGMGw==;BlobEndpoint=http://localhost:10000/devstoreaccount1;");
+ }
+
+ @After
+ public void tearDownClient() {
+ if (client != null) {
+ try {
+ client.deleteContainerForTests();
+ } catch (Throwable ignored) {
+ }
+ client.close();
+ }
+ if (proxy != null) {
+ proxy.close();
+ proxy = null;
+ }
+ // Note: We don't clean up sharedOkHttpClient here as it's shared across all tests
+ // It will be cleaned up in afterAll()
+ }
+
+ /** Simulate a connection loss on the proxy similar to S3 tests. */
+ void initiateBlobConnectionLoss() throws AzureBlobException {
+ if (proxy != null) {
+ proxy.halfClose();
+ }
+ }
+
+ @AfterClass
+ public static void afterAll() {
+ // Clean up the shared OkHttpClient
+ if (sharedOkHttpClient != null) {
+ // Shutdown dispatcher's executor service
+ sharedOkHttpClient.dispatcher().executorService().shutdown();
+ // Cancel all queued calls
+ sharedOkHttpClient.dispatcher().cancelAll();
+ // Evict all connections
+ sharedOkHttpClient.connectionPool().evictAll();
+ // Close cache if present
+ try {
+ if (sharedOkHttpClient.cache() != null) {
+ sharedOkHttpClient.cache().close();
+ }
+ } catch (Throwable ignored) {
+ }
+ // Wait for threads to terminate
+ try {
+ sharedOkHttpClient
+ .dispatcher()
+ .executorService()
+ .awaitTermination(2, java.util.concurrent.TimeUnit.SECONDS);
+ } catch (Throwable ignored) {
+ }
+ sharedOkHttpClient = null;
+ }
+
+ // Shutdown Reactor schedulers to clean up background threads
+ try {
+ reactor.core.scheduler.Schedulers.shutdownNow();
+ // Give threads time to terminate
+ Thread.sleep(100);
+ } catch (Throwable ignored) {
+ }
+ }
+
+ /**
+ * Helper method to push a string to Azure Blob Storage.
+ *
+ * @param path Destination path in blob storage.
+ * @param content Arbitrary content for the test.
+ */
+ void pushContent(String path, String content) throws AzureBlobException {
+ pushContent(path, content.getBytes(StandardCharsets.UTF_8));
+ }
+
+ void pushContent(String path, byte[] content) throws AzureBlobException {
+ try (OutputStream output = client.pushStream(path)) {
+ output.write(content);
+ } catch (IOException e) {
+ throw new AzureBlobException("Failed to write content", e);
+ }
+ }
+
+ /** Get the connection string for tests that need direct access to the blob service. */
+ static String getConnectionString() {
+ return connectionString;
+ }
+
+ /** Get the blob service URL for tests that need direct access. */
+ String getBlobServiceUrl() {
+ return "http://localhost:10000";
+ }
+
+ /**
+ * Check if Azurite is running on localhost:10000. If not, skip the test.
+ *
+ * <p>Azurite must be started manually for these tests to run. This allows tests to pass in CI
+ * environments where Azurite may not be available.
+ */
+ private void assumeAzuriteIsRunning() {
+ boolean azuriteRunning = false;
+ try (Socket socket = new Socket("localhost", 10000)) {
Review Comment:
made the change to use testcontainers, and using azurite in that testcontainer. Testcontainers automatically maps it to a random free host port, retrieving it via azuriteContainer.getMappedPort(BLOB_SERVICE_PORT) in getBlobServiceUrl(). There shouldn't be any conflict risk.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17949: Add Azure Blob Storage backup repository module [solr]
Posted by "prateeksinghalgit (via GitHub)" <gi...@apache.org>.
prateeksinghalgit commented on code in PR #3750:
URL: https://github.com/apache/solr/pull/3750#discussion_r3469267027
##########
solr/solr-ref-guide/modules/deployment-guide/pages/backup-restore.adoc:
##########
@@ -826,3 +826,124 @@ https://docs.aws.amazon.com/sdkref/latest/guide/settings-global.html[These optio
* Retries
** RetryMode (`LEGACY`, `STANDARD`, `ADAPTIVE`)
** Max Attempts
+
+=== AzureBlobBackupRepository
+
+Stores and retrieves backup files in a Microsoft Azure Blob Storage container.
+
+This is provided via the `azure-blob-repository` xref:configuration-guide:solr-modules.adoc[Solr Module] that needs to be enabled before use.
+
+This plugin supports multiple authentication methods: connection strings, account keys, SAS tokens, and Azure Identity (Managed Identity, Service Principal, Azure CLI).
+For Azure Identity, ensure the identity has the "Storage Blob Data Contributor" role on the storage account.
+
+[source,xml]
+----
+<backup>
Review Comment:
Updated it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17949: Add Azure Blob Storage backup repository module [solr]
Posted by "prateeksinghalgit (via GitHub)" <gi...@apache.org>.
prateeksinghalgit commented on code in PR #3750:
URL: https://github.com/apache/solr/pull/3750#discussion_r3359089044
##########
solr/modules/azure-blob-repository/src/java/org/apache/solr/azureblob/AzureBlobStorageClient.java:
##########
@@ -0,0 +1,529 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.azureblob;
+
+import com.azure.core.credential.TokenCredential;
+import com.azure.identity.DefaultAzureCredentialBuilder;
+import com.azure.storage.blob.BlobClient;
+import com.azure.storage.blob.BlobContainerClient;
+import com.azure.storage.blob.BlobServiceClient;
+import com.azure.storage.blob.BlobServiceClientBuilder;
+import com.azure.storage.blob.models.BlobItem;
+import com.azure.storage.blob.models.BlobStorageException;
+import com.azure.storage.blob.models.ListBlobsOptions;
+import com.google.common.annotations.VisibleForTesting;
+import java.io.ByteArrayInputStream;
+import java.io.FilterInputStream;
+import java.io.InputStream;
+import java.io.OutputStream;
+import java.lang.invoke.MethodHandles;
+import java.util.Collection;
+import java.util.HashSet;
+import java.util.Locale;
+import java.util.Set;
+import java.util.stream.Collectors;
+import org.apache.solr.common.util.ResumableInputStream;
+import org.apache.solr.common.util.StrUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Creates a {@link BlobServiceClient} for communicating with Azure Blob Storage. Utilizes the
+ * default Azure credential provider chain.
+ */
+public class AzureBlobStorageClient {
+
+ private static final Logger log = LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+
+ static final String BLOB_FILE_PATH_DELIMITER = "/";
+ private static final int HTTP_NOT_FOUND = 404;
+ private static final int HTTP_CONFLICT = 409;
+ private static final int SKIP_BUFFER_SIZE = 8192;
+ private static final int DELETE_BATCH_SIZE = 1000;
+
+ private static final com.azure.core.http.HttpClient SHARED_HTTP_CLIENT =
+ new com.azure.core.http.okhttp.OkHttpAsyncHttpClientBuilder().build();
+
+ private final BlobContainerClient containerClient;
+
+ AzureBlobStorageClient(
+ String containerName,
+ String connectionString,
+ String endpoint,
+ String accountName,
+ String accountKey,
+ String sasToken,
+ String tenantId,
+ String clientId,
+ String clientSecret) {
+ this(
+ createInternalClient(
+ connectionString,
+ endpoint,
+ accountName,
+ accountKey,
+ sasToken,
+ tenantId,
+ clientId,
+ clientSecret),
+ containerName);
+ }
+
+ @VisibleForTesting
+ AzureBlobStorageClient(BlobServiceClient blobServiceClient, String containerName) {
+ this.containerClient = blobServiceClient.getBlobContainerClient(containerName);
+ try {
+ containerClient.create();
+ } catch (BlobStorageException e) {
+ if (e.getStatusCode() != HTTP_CONFLICT) {
+ throw e;
+ }
+ }
+ }
+
+ private static BlobServiceClient createInternalClient(
+ String connectionString,
+ String endpoint,
+ String accountName,
+ String accountKey,
+ String sasToken,
+ String tenantId,
+ String clientId,
+ String clientSecret) {
+
+ BlobServiceClientBuilder builder = new BlobServiceClientBuilder();
+ builder.httpClient(SHARED_HTTP_CLIENT);
+
+ if (StrUtils.isNotNullOrEmpty(connectionString)) {
+ builder.connectionString(connectionString);
+ } else if (StrUtils.isNotNullOrEmpty(endpoint)) {
+ builder.endpoint(endpoint);
+ if (StrUtils.isNotNullOrEmpty(accountName) && StrUtils.isNotNullOrEmpty(accountKey)) {
+ builder.credential(
+ new com.azure.storage.common.StorageSharedKeyCredential(accountName, accountKey));
+ } else if (StrUtils.isNotNullOrEmpty(sasToken)) {
+ builder.sasToken(sasToken);
+ } else {
+ TokenCredential credential = new DefaultAzureCredentialBuilder().tenantId(tenantId).build();
+ builder.credential(credential);
+ }
+ } else {
+ throw new IllegalArgumentException("Either connectionString or endpoint must be provided");
+ }
+
+ return builder.buildClient();
+ }
+
+ void createDirectory(String path) throws AzureBlobException {
+ String sanitizedDirPath = sanitizedDirPath(path);
+
+ if (!pathExists(sanitizedDirPath)) {
+ String parent = getParentDirectory(sanitizedDirPath);
+ if (!parent.isEmpty() && !parent.equals(BLOB_FILE_PATH_DELIMITER)) {
+ createDirectory(parent);
+ }
+
+ try {
+ BlobClient blobClient = containerClient.getBlobClient(sanitizedDirPath);
+ blobClient.upload(new ByteArrayInputStream(new byte[0]), 0, true);
+ java.util.Map<String, String> metadata = new java.util.HashMap<>();
+ metadata.put("hdi_isfolder", "true");
+ blobClient.setMetadata(metadata);
+ } catch (BlobStorageException e) {
+ throw handleBlobException(e);
+ }
+ }
+ }
+
+ void delete(Collection<String> paths) throws AzureBlobException {
+ Set<String> entries = new HashSet<>();
+ for (String path : paths) {
+ entries.add(sanitizedFilePath(path));
+ }
+ deleteBlobs(entries);
+ }
+
+ void deleteDirectory(String path) throws AzureBlobException {
+ path = sanitizedDirPath(path);
+
+ Set<String> entries = listAll(path);
+ if (pathExists(path)) {
+ entries.add(path);
+ }
+
+ deleteBlobs(entries);
+ }
+
+ String[] listDir(String path) throws AzureBlobException {
+ path = sanitizedDirPath(path);
+
+ try {
+ ListBlobsOptions options = new ListBlobsOptions().setPrefix(path).setMaxResultsPerPage(1000);
+
+ final String finalPath = path;
+ return containerClient.listBlobs(options, null).stream()
+ .map(BlobItem::getName)
+ .filter(s -> s.startsWith(finalPath))
+ .map(s -> s.substring(finalPath.length()))
+ .filter(s -> !s.isEmpty())
+ .filter(
+ s -> {
+ int slashIndex = s.indexOf(BLOB_FILE_PATH_DELIMITER);
+ return slashIndex == -1 || slashIndex == s.length() - 1;
+ })
+ .toArray(String[]::new);
+ } catch (BlobStorageException e) {
+ throw handleBlobException(e);
+ }
+ }
+
+ boolean pathExists(String path) throws AzureBlobException {
+ final String blobPath = sanitizedPath(path);
+
+ if (blobPath.isEmpty() || BLOB_FILE_PATH_DELIMITER.equals(blobPath)) {
+ return true;
+ }
+
+ try {
+ BlobClient blobClient = containerClient.getBlobClient(blobPath);
+ return blobClient.exists();
+ } catch (BlobStorageException e) {
+ throw handleBlobException(e);
+ }
+ }
+
+ boolean isDirectory(String path) throws AzureBlobException {
+ final String dirPrefix = sanitizedDirPath(path);
+
+ try {
+ ListBlobsOptions options =
+ new ListBlobsOptions().setPrefix(dirPrefix).setMaxResultsPerPage(1);
+ if (containerClient.listBlobs(options, null).iterator().hasNext()) {
+ return true;
+ }
+
+ BlobClient markerClient = containerClient.getBlobClient(dirPrefix);
+ if (markerClient.exists()) {
+ long size = markerClient.getProperties().getBlobSize();
+ if (size == 0) {
+ return true;
+ }
+ java.util.Map<String, String> md = markerClient.getProperties().getMetadata();
+ return md != null && md.containsKey("hdi_isfolder");
+ }
+
+ return false;
+ } catch (BlobStorageException e) {
+ throw handleBlobException(e);
+ }
+ }
+
+ long length(String path) throws AzureBlobException {
+ String blobPath = sanitizedFilePath(path);
+ try {
+ BlobClient blobClient = containerClient.getBlobClient(blobPath);
+ return blobClient.getProperties().getBlobSize();
+ } catch (BlobStorageException e) {
+ throw handleBlobException(e);
+ }
+ }
+
+ InputStream pullStream(String path) throws AzureBlobException {
+ final String blobPath = sanitizedFilePath(path);
+
+ try {
+ BlobClient blobClient = containerClient.getBlobClient(blobPath);
+ final long contentLength = blobClient.getProperties().getBlobSize();
+
+ if (contentLength == 0) {
+ return new ByteArrayInputStream(new byte[0]);
+ }
+
+ InputStream initial = new IdempotentCloseInputStream(blobClient.openInputStream());
+
+ return new ResumableInputStream(
+ initial,
+ bytesRead -> {
+ if (contentLength > 0 && bytesRead >= contentLength) {
+ return null;
+ }
+ try {
+ long remaining =
+ contentLength > 0 ? Math.max(0, contentLength - bytesRead) : Long.MAX_VALUE;
+ return pullRangeStream(path, bytesRead, remaining);
+ } catch (AzureBlobException e) {
+ throw new RuntimeException(e);
+ }
+ });
+ } catch (BlobStorageException e) {
+ throw handleBlobException(e);
+ }
+ }
+
+ InputStream pullRangeStream(String path, long offset, long length) throws AzureBlobException {
+ final String blobPath = sanitizedFilePath(path);
+ try {
+ BlobClient blobClient = containerClient.getBlobClient(blobPath);
+ com.azure.storage.blob.models.BlobRange range =
+ new com.azure.storage.blob.models.BlobRange(offset, length);
+ return new IdempotentCloseInputStream(blobClient.openInputStream(range, null));
+ } catch (BlobStorageException e) {
+ throw handleBlobException(e);
+ }
+ }
+
+ private static final class IdempotentCloseInputStream extends FilterInputStream {
+ private boolean closed;
+
+ IdempotentCloseInputStream(InputStream in) {
+ super(in);
+ this.closed = false;
+ }
+
+ @Override
+ public int read() throws java.io.IOException {
+ if (closed) {
+ throw new java.io.IOException("Stream is already closed");
+ }
+ try {
+ return super.read();
+ } catch (RuntimeException re) {
+ if (isAlreadyClosed(re)) {
+ throw new java.io.IOException("Stream is already closed", re);
+ }
+ throw re;
+ }
+ }
+
+ @Override
+ public int read(byte[] b, int off, int len) throws java.io.IOException {
+ if (closed) {
+ throw new java.io.IOException("Stream is already closed");
+ }
+ try {
+ return super.read(b, off, len);
+ } catch (RuntimeException re) {
+ if (isAlreadyClosed(re)) {
+ throw new java.io.IOException("Stream is already closed", re);
+ }
+ throw re;
+ }
+ }
+
+ @Override
+ public void close() throws java.io.IOException {
+ if (closed) {
+ return;
+ }
+ try {
+ super.close();
+ } catch (java.io.IOException e) {
+ String msg = e.getMessage();
+ if (msg == null || !msg.toLowerCase(java.util.Locale.ROOT).contains("already closed")) {
+ throw e;
+ }
+ // swallow "already closed" to make close idempotent
+ } finally {
+ closed = true;
+ }
+ }
+
+ @Override
+ public long skip(long n) throws java.io.IOException {
+ if (closed) {
+ throw new java.io.IOException("Stream is already closed");
+ }
+ if (n <= 0) {
+ return 0L;
+ }
+ long remaining = n;
+ byte[] discard = new byte[SKIP_BUFFER_SIZE];
+ try {
+ while (remaining > 0) {
+ int toRead = (int) Math.min(discard.length, remaining);
+ int read = super.read(discard, 0, toRead);
+ if (read < 0) {
+ break;
+ }
+ remaining -= read;
+ }
+ return n - remaining;
+ } catch (RuntimeException re) {
+ throw new java.io.IOException(re);
+ }
+ }
+
+ private static boolean isAlreadyClosed(Throwable t) {
+ String msg = t.getMessage();
+ return msg != null && msg.toLowerCase(java.util.Locale.ROOT).contains("already closed");
+ }
+ }
+
+ OutputStream pushStream(String path) throws AzureBlobException {
+ path = sanitizedFilePath(path);
+
+ if (!parentDirectoryExist(path)) {
+ String parentDirectory = getParentDirectory(path);
+ if (!parentDirectory.isEmpty() && !parentDirectory.equals(BLOB_FILE_PATH_DELIMITER)) {
+ createDirectory(parentDirectory);
+ }
+ }
+
+ try {
+ BlobClient blobClient = containerClient.getBlobClient(path);
+ return new AzureBlobOutputStream(blobClient, path);
+ } catch (BlobStorageException e) {
+ throw handleBlobException(e);
+ }
+ }
+
+ void close() {}
+
+ @VisibleForTesting
+ void deleteContainerForTests() {
+ try {
+ containerClient.delete();
+ } catch (BlobStorageException e) {
+ if (e.getStatusCode() != HTTP_NOT_FOUND) {
+ throw e;
+ }
+ }
+ }
+
+ private Collection<String> deleteBlobs(Collection<String> paths) throws AzureBlobException {
+ try {
+ return deleteBlobs(paths, DELETE_BATCH_SIZE);
+ } catch (BlobStorageException e) {
+ throw handleBlobException(e);
+ }
+ }
+
+ @VisibleForTesting
+ Collection<String> deleteBlobs(Collection<String> entries, int batchSize)
+ throws AzureBlobException {
+ Set<String> deletedPaths = new HashSet<>();
+
+ for (String path : entries) {
+ try {
+ BlobClient blobClient = containerClient.getBlobClient(path);
+ boolean existed = blobClient.deleteIfExists();
+ if (existed) {
+ deletedPaths.add(path);
+ }
+ } catch (BlobStorageException e) {
+ if (e.getStatusCode() == HTTP_NOT_FOUND) {
Review Comment:
Replaced deleteIfExists with batched deletes; HTTP_NOT_FOUND sub-errors are filtered
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17949: Add Azure Blob Storage backup repository module [solr]
Posted by "prateeksinghalgit (via GitHub)" <gi...@apache.org>.
prateeksinghalgit commented on code in PR #3750:
URL: https://github.com/apache/solr/pull/3750#discussion_r2553346877
##########
solr/modules/azure-blob-repository/src/java/org/apache/solr/azureblob/AzureBlobBackupRepository.java:
##########
@@ -0,0 +1,407 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.blob;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStream;
+import java.lang.invoke.MethodHandles;
+import java.net.URI;
+import java.net.URISyntaxException;
+import java.time.Duration;
+import java.time.Instant;
+import java.util.Collection;
+import java.util.Objects;
+import java.util.Set;
+import java.util.stream.Collectors;
+import org.apache.lucene.store.Directory;
+import org.apache.lucene.store.IOContext;
+import org.apache.lucene.store.IndexInput;
+import org.apache.lucene.store.IndexOutput;
+import org.apache.solr.common.SolrException;
+import org.apache.solr.common.util.NamedList;
+import org.apache.solr.common.util.StrUtils;
+import org.apache.solr.core.backup.repository.AbstractBackupRepository;
+import org.apache.solr.core.backup.repository.BackupRepository;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * A concrete implementation of {@link BackupRepository} interface supporting backup/restore of Solr
+ * indexes to Azure Blob Storage.
+ */
+public class BlobBackupRepository extends AbstractBackupRepository {
+
+ private static final Logger log = LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+
+ static final String BLOB_SCHEME = "blob";
+ private static final int CHUNK_SIZE = 16 * 1024 * 1024;
+
+ private BlobStorageClient client;
+
+ @Override
+ public void init(NamedList<?> args) {
+ super.init(args);
+ BlobBackupRepositoryConfig backupConfig = new BlobBackupRepositoryConfig(this.config);
+
+ // If a client was already created, close it to avoid any resource leak
+ if (client != null) {
+ client.close();
+ }
+
+ this.client = backupConfig.buildClient();
+ }
+
+ // Method to inject a mock client for testing
+ public void setClient(BlobStorageClient client) {
+ this.client = client;
+ }
+
+ @Override
+ @SuppressWarnings("unchecked")
+ public <T> T getConfigProperty(String name) {
+ return (T) this.config.get(name);
+ }
+
+ @Override
+ public URI createURI(String location) {
+ if (StrUtils.isNullOrEmpty(location)) {
+ throw new IllegalArgumentException("cannot create URI with an empty location");
+ }
+
+ URI result;
+ try {
+ if (location.startsWith(BLOB_SCHEME + ":")) {
+ result = new URI(location);
+ } else if (location.startsWith("/")) {
+ result = new URI(BLOB_SCHEME, "", location, null);
+ } else {
+ result = new URI(BLOB_SCHEME, "", "/" + location, null);
+ }
+ return result;
+ } catch (URISyntaxException ex) {
+ throw new SolrException(SolrException.ErrorCode.BAD_REQUEST, ex);
+ }
+ }
+
+ @Override
+ public URI createDirectoryURI(String location) {
+ if (StrUtils.isNullOrEmpty(location)) {
+ throw new IllegalArgumentException("cannot create URI with an empty location");
+ }
+
+ if (!location.endsWith("/")) {
+ location += "/";
+ }
+
+ return createURI(location);
+ }
+
+ @Override
+ public URI resolve(URI baseUri, String... pathComponents) {
+ if (!BLOB_SCHEME.equalsIgnoreCase(baseUri.getScheme())) {
+ throw new IllegalArgumentException("URI must begin with 'blob:' scheme");
+ }
+
+ String path = baseUri + "/" + String.join("/", pathComponents);
+ return URI.create(path).normalize();
+ }
+
+ @Override
+ public URI resolveDirectory(URI baseUri, String... pathComponents) {
+ if (pathComponents.length > 0) {
+ if (!pathComponents[pathComponents.length - 1].endsWith("/")) {
+ pathComponents[pathComponents.length - 1] = pathComponents[pathComponents.length - 1] + "/";
+ }
+ } else {
+ if (!baseUri.toString().endsWith("/")) {
+ baseUri = URI.create(baseUri + "/");
+ }
+ }
+ return resolve(baseUri, pathComponents);
+ }
+
+ @Override
+ public void createDirectory(URI path) throws IOException {
+ Objects.requireNonNull(path, "cannot create directory to a null URI");
+
+ String blobPath = getBlobPath(path);
+
+ if (log.isDebugEnabled()) {
+ log.debug("Create directory '{}'", blobPath);
+ }
+
+ try {
+ client.createDirectory(blobPath);
+ } catch (BlobException e) {
+ throw new IOException("Failed to create directory " + blobPath, e);
+ }
+ }
+
+ @Override
+ public void deleteDirectory(URI path) throws IOException {
+ Objects.requireNonNull(path, "cannot delete directory with a null URI");
+
+ String blobPath = getBlobPath(path);
+
+ if (log.isDebugEnabled()) {
+ log.debug("Delete directory '{}'", blobPath);
+ }
+
+ try {
+ client.deleteDirectory(blobPath);
+ } catch (BlobException e) {
+ throw new IOException("Failed to delete directory " + blobPath, e);
+ }
+ }
+
+ @Override
+ public void delete(URI path, Collection<String> files) throws IOException {
+ Objects.requireNonNull(path, "cannot delete with a null URI");
+ Objects.requireNonNull(files, "cannot delete with a null files collection");
+
+ String basePath = getBlobPath(path);
+ // If a file path was passed instead of a directory, use its parent directory as base
+ try {
+ if (!client.isDirectory(basePath)) {
+ int lastSlash = basePath.lastIndexOf('/');
+ basePath = lastSlash >= 0 ? basePath.substring(0, lastSlash) : "";
+ }
+ } catch (BlobException e) {
+ throw new IOException("Failed to check path type for " + basePath, e);
+ }
+
+ final String baseForPaths = basePath;
+ Set<String> fullPaths =
+ files.stream()
+ .map(file -> (baseForPaths.isEmpty() ? file : baseForPaths + "/" + file))
+ .collect(Collectors.toSet());
+
+ if (log.isDebugEnabled()) {
+ log.debug("Delete files '{}'", fullPaths);
+ }
+
+ try {
+ client.delete(fullPaths);
+ } catch (BlobException e) {
+ throw new IOException("Failed to delete files " + fullPaths, e);
+ }
+ }
+
+ @Override
+ public boolean exists(URI path) throws IOException {
+ Objects.requireNonNull(path, "cannot check existence with a null URI");
+
+ String blobPath = getBlobPath(path);
+
+ if (log.isDebugEnabled()) {
+ log.debug("Check existence '{}'", blobPath);
+ }
+
+ try {
+ return client.pathExists(blobPath);
+ } catch (BlobException e) {
+ throw new IOException("Failed to check existence of " + blobPath, e);
+ }
+ }
+
+ @Override
+ public PathType getPathType(URI path) throws IOException {
+ Objects.requireNonNull(path, "cannot get path type with a null URI");
+
+ String blobPath = getBlobPath(path);
+
+ if (log.isDebugEnabled()) {
+ log.debug("Get path type '{}'", blobPath);
+ }
+
+ try {
+ if (client.isDirectory(blobPath)) {
+ return BackupRepository.PathType.DIRECTORY;
+ } else {
+ return BackupRepository.PathType.FILE;
+ }
+ } catch (BlobException e) {
+ throw new IOException("Failed to get path type for " + blobPath, e);
+ }
+ }
+
+ @Override
+ public String[] listAll(URI path) throws IOException {
+ Objects.requireNonNull(path, "cannot list with a null URI");
+
+ String blobPath = getBlobPath(path);
+
+ if (log.isDebugEnabled()) {
+ log.debug("List all '{}'", blobPath);
+ }
+
+ try {
+ return client.listDir(blobPath);
+ } catch (BlobException e) {
+ throw new IOException("Failed to list directory " + blobPath, e);
+ }
+ }
+
+ @Override
+ public IndexInput openInput(URI dirPath, String fileName, IOContext ctx) throws IOException {
+ Objects.requireNonNull(dirPath, "cannot open input with a null URI");
+ Objects.requireNonNull(fileName, "cannot open input with a null fileName");
+
+ String base = getBlobPath(dirPath);
+ String blobPath = base.endsWith("/") ? base + fileName : base + "/" + fileName;
+
+ if (log.isDebugEnabled()) {
+ log.debug("Open input '{}'", blobPath);
+ }
+
+ try {
+ return new BlobIndexInput(blobPath, client, client.length(blobPath));
+ } catch (BlobException e) {
+ throw new IOException("Failed to open input stream for " + blobPath, e);
+ }
+ }
+
+ @Override
+ public OutputStream createOutput(URI path) throws IOException {
+ Objects.requireNonNull(path, "cannot create output with a null URI");
+
+ String blobPath = getBlobPath(path);
+
+ if (log.isDebugEnabled()) {
+ log.debug("Create output '{}'", blobPath);
+ }
+
+ try {
+ return client.pushStream(blobPath);
+ } catch (BlobException e) {
+ throw new IOException("Failed to create output stream for " + blobPath, e);
+ }
+ }
+
+ @Override
+ public void copyIndexFileFrom(
+ Directory sourceDir, String sourceFileName, URI dest, String destFileName)
+ throws IOException {
+ Objects.requireNonNull(sourceDir, "cannot copy with a null sourceDir");
+ Objects.requireNonNull(sourceFileName, "cannot copy with a null sourceFileName");
+ Objects.requireNonNull(dest, "cannot copy with a null dest");
+
+ String destPath = getBlobPath(dest);
+
+ String blobPath = destPath.endsWith("/") ? destPath + destFileName : destPath;
+
+ if (log.isDebugEnabled()) {
+ log.debug("Copy index file from '{}' to '{}'", sourceFileName, blobPath);
+ }
+
+ // Ensure destination parent directory exists
+ String parentDir =
+ blobPath.contains("/") ? blobPath.substring(0, blobPath.lastIndexOf('/') + 1) : "";
+ try {
+ if (!parentDir.isEmpty()) {
+ client.createDirectory(parentDir);
+ }
+ } catch (BlobException e) {
+ // ignore failures here; write will surface real issues
+ }
+
+ try (IndexInput input = sourceDir.openInput(sourceFileName, IOContext.DEFAULT);
+ OutputStream output = client.pushStream(blobPath)) {
+ // Copy bytes from IndexInput to OutputStream
+ byte[] buffer = new byte[8192];
+ long remaining = input.length();
+ while (remaining > 0) {
+ int toRead = (int) Math.min(buffer.length, remaining);
+ input.readBytes(buffer, 0, toRead);
+ output.write(buffer, 0, toRead);
+ remaining -= toRead;
+ }
+ } catch (BlobException e) {
+ throw new IOException("Failed to copy file from " + sourceFileName + " to " + blobPath, e);
+ }
+ }
+
+ /**
+ * Copy an index file from specified <code>sourceRepo</code> to the destination directory (i.e.
+ * restore).
+ *
+ * @param sourceDir The source URI hosting the file to be copied.
+ * @param dest The destination where the file should be copied.
+ * @throws IOException in case of errors.
+ */
+ @Override
+ public void copyIndexFileTo(
+ URI sourceDir, String sourceFileName, Directory dest, String destFileName)
+ throws IOException {
+ if (StrUtils.isNullOrEmpty(sourceFileName)) {
+ throw new IllegalArgumentException("must have a valid source file name to copy");
+ }
+ if (StrUtils.isNullOrEmpty(destFileName)) {
+ throw new IllegalArgumentException("must have a valid destination file name to copy");
+ }
+
+ String basePath = getBlobPath(sourceDir);
+ String blobPath;
+ // If sourceDir already points to the file, avoid duplicating the name
+ if (basePath.endsWith("/" + sourceFileName)
+ || basePath.equals(sourceFileName)
+ || basePath.equals("/" + sourceFileName)) {
+ blobPath = basePath;
+ } else {
+ URI filePath = resolve(sourceDir, sourceFileName);
+ blobPath = getBlobPath(filePath);
+ }
+
+ Instant start = Instant.now();
+ if (log.isDebugEnabled()) {
+ log.debug("Download started from blob '{}'", blobPath);
+ }
+
+ try (InputStream inputStream = client.pullStream(blobPath);
+ IndexOutput indexOutput = dest.createOutput(destFileName, IOContext.DEFAULT)) {
+ // Copy bytes from InputStream to IndexOutput
+ byte[] buffer = new byte[CHUNK_SIZE];
+ int len;
+ while ((len = inputStream.read(buffer)) != -1) {
+ indexOutput.writeBytes(buffer, 0, len);
+ }
+ } catch (BlobException e) {
+ throw new IOException("Failed to copy file from " + blobPath + " to " + destFileName, e);
+ }
+
+ long timeElapsed = Duration.between(start, Instant.now()).toMillis();
+
+ if (log.isInfoEnabled()) {
+ log.info("Download from S3 '{}' finished in {}ms", blobPath, timeElapsed);
Review Comment:
updated the logInfo message to refer Azure Blob Storage instead of S3.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17949: Add Azure Blob Storage backup repository module [solr]
Posted by "janhoy (via GitHub)" <gi...@apache.org>.
janhoy commented on code in PR #3750:
URL: https://github.com/apache/solr/pull/3750#discussion_r2553391024
##########
solr/licenses/msal4j-NOTICE.txt:
##########
@@ -0,0 +1,12 @@
+Microsoft Authentication Library for Java (MSAL4J)
+Copyright (c) Microsoft Corporation. All rights reserved.
+
+This product includes software developed by
+Microsoft Corporation (https://github.com/AzureAD/microsoft-authentication-library-for-java).
+
+Licensed under the MIT License.
Review Comment:
So the msal4j lib is licensed under MIT license. Then you need to change the LICENSE file to ba `-MIT` and the package probably does not require a NOTICE file at all.
Please check that correct license (and if ASL, also NOTICE file) is in place for each new dependency. Copy exactly the license and notice they provide with the lib. There may be issues with more than this one.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17949: Add Azure Blob Storage backup repository module [solr]
Posted by "prateeksinghalgit (via GitHub)" <gi...@apache.org>.
prateeksinghalgit commented on code in PR #3750:
URL: https://github.com/apache/solr/pull/3750#discussion_r2563445786
##########
solr/modules/azure-blob-repository/src/test/org/apache/solr/azureblob/AbstractAzureBlobClientTest.java:
##########
@@ -0,0 +1,249 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.azureblob;
+
+import com.azure.core.http.HttpClient;
+import com.azure.core.http.okhttp.OkHttpAsyncHttpClientBuilder;
+import com.azure.storage.blob.BlobServiceClient;
+import com.azure.storage.blob.BlobServiceClientBuilder;
+import com.carrotsearch.randomizedtesting.ThreadFilter;
+import com.carrotsearch.randomizedtesting.annotations.ThreadLeakFilters;
+import java.io.IOException;
+import java.io.OutputStream;
+import java.net.Socket;
+import java.nio.charset.StandardCharsets;
+import okhttp3.OkHttpClient;
+import org.apache.solr.SolrTestCaseJ4;
+import org.junit.After;
+import org.junit.AfterClass;
+import org.junit.Before;
+import org.junit.BeforeClass;
+import org.junit.Rule;
+import org.junit.rules.TemporaryFolder;
+
+import static org.junit.Assume.assumeTrue;
+
+/**
+ * Abstract class for tests with Azure Blob Storage emulator.
+ *
+ * <p>Uses ThreadLeakFilters to ignore OkHttp's global singleton threads (TaskRunner and Okio
+ * Watchdog) which are JVM-wide and cannot be shut down per-client. This is a standard pattern in
+ * Solr tests for HTTP clients with global thread pools.
+ */
+@ThreadLeakFilters(
+ filters = {
+ AbstractAzureBlobClientTest.OkHttpThreadLeakFilter.class,
+ })
+public class AbstractAzureBlobClientTest extends SolrTestCaseJ4 {
+
+ protected String containerName;
+
+ @Rule public TemporaryFolder tempFolder = new TemporaryFolder();
+
+ AzureBlobStorageClient client;
+ private static String connectionString;
+ // Shared OkHttpClient across all tests to reuse the global thread pools
+ private static OkHttpClient sharedOkHttpClient;
+ protected org.apache.solr.client.solrj.cloud.SocketProxy proxy;
+
+ @BeforeClass
+ public static void setUpClass() {
+ // Create a single shared OkHttpClient for all tests
+ // This reuses OkHttp's global thread pools efficiently
+ sharedOkHttpClient = new OkHttpClient.Builder().build();
+ }
+
+ @Before
+ public void setUpClient() throws Exception {
+ // Check if Azurite is running before attempting to connect
+ assumeAzuriteIsRunning();
+
+ setAzureTestCredentials();
+
+ // Use Azurite connection string for local testing
+ connectionString =
+ "DefaultEndpointsProtocol=http;AccountName=devstoreaccount1;AccountKey=Eby8vdM02xNOcqFlqUwJPLlmEtlCDXJ1OUzFT50uSRZ6IFsuFq2UVErCz4I6tq/K1SZFPTOtr/KBHBeksoGMGw==;BlobEndpoint=http://localhost:10000/devstoreaccount1;";
+
+ // Put a proxy in front of Azurite to simulate connection loss like S3 tests
+ proxy = new org.apache.solr.client.solrj.cloud.SocketProxy();
+ proxy.open(new java.net.URI(getBlobServiceUrl()));
+
+ // Reuse the shared OkHttpClient
+ HttpClient httpClient = new OkHttpAsyncHttpClientBuilder(sharedOkHttpClient).build();
+
+ // Route Blob endpoint through the proxy by adjusting the connection string
+ String proxiedConn = connectionString.replace(":10000", ":" + proxy.getListenPort());
+ BlobServiceClient blobServiceClient =
+ new BlobServiceClientBuilder()
+ .connectionString(proxiedConn)
+ .httpClient(httpClient)
+ .buildClient();
+
+ containerName = "test-" + java.util.UUID.randomUUID();
+ client = new AzureBlobStorageClient(blobServiceClient, containerName);
+ }
+
+ /**
+ * Set up Azure test credentials to avoid using real Azure credentials during testing. Similar to
+ * how S3 tests use ProfileFileSystemSetting to avoid polluting the test environment.
+ */
+ public static void setAzureTestCredentials() {
+ // Set test Azure credentials to avoid using real credentials
+ System.setProperty("AZURE_CLIENT_ID", "test-client-id");
+ System.setProperty("AZURE_TENANT_ID", "test-tenant-id");
+ System.setProperty("AZURE_CLIENT_SECRET", "test-client-secret");
+
+ // Set Azurite-specific environment variables
+ System.setProperty(
+ "AZURE_STORAGE_CONNECTION_STRING",
+ "DefaultEndpointsProtocol=http;AccountName=devstoreaccount1;AccountKey=Eby8vdM02xNOcqFlqUwJPLlmEtlCDXJ1OUzFT50uSRZ6IFsuFq2UVErCz4I6tq/K1SZFPTOtr/KBHBeksoGMGw==;BlobEndpoint=http://localhost:10000/devstoreaccount1;");
+ }
+
+ @After
+ public void tearDownClient() {
+ if (client != null) {
+ try {
+ client.deleteContainerForTests();
+ } catch (Throwable ignored) {
+ }
+ client.close();
+ }
+ if (proxy != null) {
+ proxy.close();
+ proxy = null;
+ }
+ // Note: We don't clean up sharedOkHttpClient here as it's shared across all tests
+ // It will be cleaned up in afterAll()
+ }
+
+ /** Simulate a connection loss on the proxy similar to S3 tests. */
+ void initiateBlobConnectionLoss() throws AzureBlobException {
+ if (proxy != null) {
+ proxy.halfClose();
+ }
+ }
+
+ @AfterClass
+ public static void afterAll() {
+ // Clean up the shared OkHttpClient
+ if (sharedOkHttpClient != null) {
+ // Shutdown dispatcher's executor service
+ sharedOkHttpClient.dispatcher().executorService().shutdown();
+ // Cancel all queued calls
+ sharedOkHttpClient.dispatcher().cancelAll();
+ // Evict all connections
+ sharedOkHttpClient.connectionPool().evictAll();
+ // Close cache if present
+ try {
+ if (sharedOkHttpClient.cache() != null) {
+ sharedOkHttpClient.cache().close();
+ }
+ } catch (Throwable ignored) {
+ }
+ // Wait for threads to terminate
+ try {
+ sharedOkHttpClient
+ .dispatcher()
+ .executorService()
+ .awaitTermination(2, java.util.concurrent.TimeUnit.SECONDS);
+ } catch (Throwable ignored) {
+ }
+ sharedOkHttpClient = null;
+ }
+
+ // Shutdown Reactor schedulers to clean up background threads
+ try {
+ reactor.core.scheduler.Schedulers.shutdownNow();
+ // Give threads time to terminate
+ Thread.sleep(100);
+ } catch (Throwable ignored) {
+ }
+ }
+
+ /**
+ * Helper method to push a string to Azure Blob Storage.
+ *
+ * @param path Destination path in blob storage.
+ * @param content Arbitrary content for the test.
+ */
+ void pushContent(String path, String content) throws AzureBlobException {
+ pushContent(path, content.getBytes(StandardCharsets.UTF_8));
+ }
+
+ void pushContent(String path, byte[] content) throws AzureBlobException {
+ try (OutputStream output = client.pushStream(path)) {
+ output.write(content);
+ } catch (IOException e) {
+ throw new AzureBlobException("Failed to write content", e);
+ }
+ }
+
+ /** Get the connection string for tests that need direct access to the blob service. */
+ static String getConnectionString() {
+ return connectionString;
+ }
+
+ /** Get the blob service URL for tests that need direct access. */
+ String getBlobServiceUrl() {
+ return "http://localhost:10000";
+ }
+
+ /**
+ * Check if Azurite is running on localhost:10000. If not, skip the test.
+ *
+ * <p>Azurite must be started manually for these tests to run. This allows tests to pass in CI
+ * environments where Azurite may not be available.
+ */
+ private void assumeAzuriteIsRunning() {
+ boolean azuriteRunning = false;
+ try (Socket socket = new Socket("localhost", 10000)) {
Review Comment:
made the change to use testcontainers, it basically azurite in a docker container. This change remove the need to specify port.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17949: Add Azure Blob Storage backup repository module [solr]
Posted by "janhoy (via GitHub)" <gi...@apache.org>.
janhoy commented on code in PR #3750:
URL: https://github.com/apache/solr/pull/3750#discussion_r2558918492
##########
solr/modules/azure-blob-repository/src/test/org/apache/solr/azureblob/AbstractAzureBlobClientTest.java:
##########
@@ -0,0 +1,249 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.azureblob;
+
+import com.azure.core.http.HttpClient;
+import com.azure.core.http.okhttp.OkHttpAsyncHttpClientBuilder;
+import com.azure.storage.blob.BlobServiceClient;
+import com.azure.storage.blob.BlobServiceClientBuilder;
+import com.carrotsearch.randomizedtesting.ThreadFilter;
+import com.carrotsearch.randomizedtesting.annotations.ThreadLeakFilters;
+import java.io.IOException;
+import java.io.OutputStream;
+import java.net.Socket;
+import java.nio.charset.StandardCharsets;
+import okhttp3.OkHttpClient;
+import org.apache.solr.SolrTestCaseJ4;
+import org.junit.After;
+import org.junit.AfterClass;
+import org.junit.Before;
+import org.junit.BeforeClass;
+import org.junit.Rule;
+import org.junit.rules.TemporaryFolder;
+
+import static org.junit.Assume.assumeTrue;
+
+/**
+ * Abstract class for tests with Azure Blob Storage emulator.
+ *
+ * <p>Uses ThreadLeakFilters to ignore OkHttp's global singleton threads (TaskRunner and Okio
+ * Watchdog) which are JVM-wide and cannot be shut down per-client. This is a standard pattern in
+ * Solr tests for HTTP clients with global thread pools.
+ */
+@ThreadLeakFilters(
+ filters = {
+ AbstractAzureBlobClientTest.OkHttpThreadLeakFilter.class,
+ })
+public class AbstractAzureBlobClientTest extends SolrTestCaseJ4 {
+
+ protected String containerName;
+
+ @Rule public TemporaryFolder tempFolder = new TemporaryFolder();
+
+ AzureBlobStorageClient client;
+ private static String connectionString;
+ // Shared OkHttpClient across all tests to reuse the global thread pools
+ private static OkHttpClient sharedOkHttpClient;
+ protected org.apache.solr.client.solrj.cloud.SocketProxy proxy;
+
+ @BeforeClass
+ public static void setUpClass() {
+ // Create a single shared OkHttpClient for all tests
+ // This reuses OkHttp's global thread pools efficiently
+ sharedOkHttpClient = new OkHttpClient.Builder().build();
+ }
+
+ @Before
+ public void setUpClient() throws Exception {
+ // Check if Azurite is running before attempting to connect
+ assumeAzuriteIsRunning();
+
+ setAzureTestCredentials();
+
+ // Use Azurite connection string for local testing
+ connectionString =
+ "DefaultEndpointsProtocol=http;AccountName=devstoreaccount1;AccountKey=Eby8vdM02xNOcqFlqUwJPLlmEtlCDXJ1OUzFT50uSRZ6IFsuFq2UVErCz4I6tq/K1SZFPTOtr/KBHBeksoGMGw==;BlobEndpoint=http://localhost:10000/devstoreaccount1;";
+
+ // Put a proxy in front of Azurite to simulate connection loss like S3 tests
+ proxy = new org.apache.solr.client.solrj.cloud.SocketProxy();
+ proxy.open(new java.net.URI(getBlobServiceUrl()));
+
+ // Reuse the shared OkHttpClient
+ HttpClient httpClient = new OkHttpAsyncHttpClientBuilder(sharedOkHttpClient).build();
+
+ // Route Blob endpoint through the proxy by adjusting the connection string
+ String proxiedConn = connectionString.replace(":10000", ":" + proxy.getListenPort());
+ BlobServiceClient blobServiceClient =
+ new BlobServiceClientBuilder()
+ .connectionString(proxiedConn)
+ .httpClient(httpClient)
+ .buildClient();
+
+ containerName = "test-" + java.util.UUID.randomUUID();
+ client = new AzureBlobStorageClient(blobServiceClient, containerName);
+ }
+
+ /**
+ * Set up Azure test credentials to avoid using real Azure credentials during testing. Similar to
+ * how S3 tests use ProfileFileSystemSetting to avoid polluting the test environment.
+ */
+ public static void setAzureTestCredentials() {
+ // Set test Azure credentials to avoid using real credentials
+ System.setProperty("AZURE_CLIENT_ID", "test-client-id");
+ System.setProperty("AZURE_TENANT_ID", "test-tenant-id");
+ System.setProperty("AZURE_CLIENT_SECRET", "test-client-secret");
+
+ // Set Azurite-specific environment variables
+ System.setProperty(
+ "AZURE_STORAGE_CONNECTION_STRING",
+ "DefaultEndpointsProtocol=http;AccountName=devstoreaccount1;AccountKey=Eby8vdM02xNOcqFlqUwJPLlmEtlCDXJ1OUzFT50uSRZ6IFsuFq2UVErCz4I6tq/K1SZFPTOtr/KBHBeksoGMGw==;BlobEndpoint=http://localhost:10000/devstoreaccount1;");
+ }
+
+ @After
+ public void tearDownClient() {
+ if (client != null) {
+ try {
+ client.deleteContainerForTests();
+ } catch (Throwable ignored) {
+ }
+ client.close();
+ }
+ if (proxy != null) {
+ proxy.close();
+ proxy = null;
+ }
+ // Note: We don't clean up sharedOkHttpClient here as it's shared across all tests
+ // It will be cleaned up in afterAll()
+ }
+
+ /** Simulate a connection loss on the proxy similar to S3 tests. */
+ void initiateBlobConnectionLoss() throws AzureBlobException {
+ if (proxy != null) {
+ proxy.halfClose();
+ }
+ }
+
+ @AfterClass
+ public static void afterAll() {
+ // Clean up the shared OkHttpClient
+ if (sharedOkHttpClient != null) {
+ // Shutdown dispatcher's executor service
+ sharedOkHttpClient.dispatcher().executorService().shutdown();
+ // Cancel all queued calls
+ sharedOkHttpClient.dispatcher().cancelAll();
+ // Evict all connections
+ sharedOkHttpClient.connectionPool().evictAll();
+ // Close cache if present
+ try {
+ if (sharedOkHttpClient.cache() != null) {
+ sharedOkHttpClient.cache().close();
+ }
+ } catch (Throwable ignored) {
+ }
+ // Wait for threads to terminate
+ try {
+ sharedOkHttpClient
+ .dispatcher()
+ .executorService()
+ .awaitTermination(2, java.util.concurrent.TimeUnit.SECONDS);
+ } catch (Throwable ignored) {
+ }
+ sharedOkHttpClient = null;
+ }
+
+ // Shutdown Reactor schedulers to clean up background threads
+ try {
+ reactor.core.scheduler.Schedulers.shutdownNow();
+ // Give threads time to terminate
+ Thread.sleep(100);
+ } catch (Throwable ignored) {
+ }
+ }
+
+ /**
+ * Helper method to push a string to Azure Blob Storage.
+ *
+ * @param path Destination path in blob storage.
+ * @param content Arbitrary content for the test.
+ */
+ void pushContent(String path, String content) throws AzureBlobException {
+ pushContent(path, content.getBytes(StandardCharsets.UTF_8));
+ }
+
+ void pushContent(String path, byte[] content) throws AzureBlobException {
+ try (OutputStream output = client.pushStream(path)) {
+ output.write(content);
+ } catch (IOException e) {
+ throw new AzureBlobException("Failed to write content", e);
+ }
+ }
+
+ /** Get the connection string for tests that need direct access to the blob service. */
+ static String getConnectionString() {
+ return connectionString;
+ }
+
+ /** Get the blob service URL for tests that need direct access. */
+ String getBlobServiceUrl() {
+ return "http://localhost:10000";
+ }
+
+ /**
+ * Check if Azurite is running on localhost:10000. If not, skip the test.
+ *
+ * <p>Azurite must be started manually for these tests to run. This allows tests to pass in CI
+ * environments where Azurite may not be available.
+ */
+ private void assumeAzuriteIsRunning() {
+ boolean azuriteRunning = false;
+ try (Socket socket = new Socket("localhost", 10000)) {
Review Comment:
We never hard code ports in tests, will cause weird port conflicts on CI hardware, tests running in parallel etc. Are you able to obtain a free port for this instead? Have not read all the test code, and I see some port replacements further down related to proxying..
Also I cannot find where this Azurite program is started in this test, it seems as it is a requirement of the test, in other locations it is described as an npm tool, but I cannot see initialization here? Please enlighten.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17949: Add Azure Blob Storage backup repository module [solr]
Posted by "prateeksinghalgit (via GitHub)" <gi...@apache.org>.
prateeksinghalgit commented on PR #3750:
URL: https://github.com/apache/solr/pull/3750#issuecomment-4773590864
@janhoy could you please approve the workflow runs on this PR again? The previous CI failure was because my branch hadn't merged the latest main
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17949: Add Azure Blob Storage backup repository module [solr]
Posted by "janhoy (via GitHub)" <gi...@apache.org>.
janhoy commented on code in PR #3750:
URL: https://github.com/apache/solr/pull/3750#discussion_r3458591665
##########
solr/solrj/gradle.lockfile:
##########
@@ -17,13 +17,19 @@ com.github.stephenc.jcip:jcip-annotations:1.0-1=compileClasspath,compileOnlyHelp
com.google.auto.service:auto-service-annotations:1.0.1=annotationProcessor,errorprone,testAnnotationProcessor
com.google.auto.value:auto-value-annotations:1.11.0=annotationProcessor,errorprone,testAnnotationProcessor
com.google.auto:auto-common:1.2.2=annotationProcessor,errorprone,testAnnotationProcessor
+com.google.code.findbugs:jsr305:3.0.2=spotless865458226
com.google.errorprone:error_prone_annotation:2.41.0=annotationProcessor,errorprone,testAnnotationProcessor
+com.google.errorprone:error_prone_annotations:2.18.0=spotless865458226
Review Comment:
Why do we suddenly have all these spotless dependencies added to lockfiles? I would not expect any changes to lockfiles outside of the new module itself?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17949: Add Azure Blob Storage backup repository module [solr]
Posted by "prateeksinghalgit (via GitHub)" <gi...@apache.org>.
prateeksinghalgit commented on code in PR #3750:
URL: https://github.com/apache/solr/pull/3750#discussion_r3469248764
##########
solr/core/gradle.lockfile:
##########
@@ -1,191 +1,191 @@
# This is a Gradle generated file for dependency locking.
# Manual edits can break the build and are not advised.
# This file is expected to be part of source control.
-biz.aQute.bnd:biz.aQute.bnd.annotation:7.1.0=compileClasspath,testCompileClasspath
-com.carrotsearch.randomizedtesting:randomizedtesting-runner:2.8.4=jarValidation,testCompileClasspath,testRuntimeClasspath
-com.carrotsearch:hppc:0.10.0=compileClasspath,jarValidation,runtimeClasspath,runtimeLibs,testCompileClasspath,testRuntimeClasspath
-com.fasterxml.jackson.core:jackson-annotations:2.21=apiHelper,compileClasspath,jarValidation,runtimeClasspath,runtimeLibs,testCompileClasspath,testRuntimeClasspath
-com.fasterxml.jackson.core:jackson-core:2.21.2=apiHelper,compileClasspath,jarValidation,runtimeClasspath,runtimeLibs,testCompileClasspath,testRuntimeClasspath
-com.fasterxml.jackson.core:jackson-databind:2.21.2=apiHelper,compileClasspath,jarValidation,runtimeClasspath,runtimeLibs,testCompileClasspath,testRuntimeClasspath
-com.fasterxml.jackson.dataformat:jackson-dataformat-cbor:2.21.2=compileClasspath,jarValidation,runtimeClasspath,runtimeLibs,testCompileClasspath,testRuntimeClasspath
-com.fasterxml.jackson.dataformat:jackson-dataformat-smile:2.21.2=compileClasspath,jarValidation,runtimeClasspath,runtimeLibs,testCompileClasspath,testRuntimeClasspath
-com.fasterxml.jackson.module:jackson-module-jakarta-xmlbind-annotations:2.21.2=compileClasspath,jarValidation,runtimeClasspath,runtimeLibs,testCompileClasspath,testRuntimeClasspath
-com.fasterxml.jackson:jackson-bom:2.21.2=apiHelper,compileClasspath,jarValidation,runtimeClasspath,runtimeLibs,testCompileClasspath,testRuntimeClasspath
-com.fasterxml.woodstox:woodstox-core:7.0.0=compileClasspath,jarValidation,runtimeClasspath,runtimeLibs,testCompileClasspath,testRuntimeClasspath
-com.github.ben-manes.caffeine:caffeine:3.2.3=annotationProcessor,compileClasspath,errorprone,jarValidation,runtimeClasspath,runtimeLibs,testAnnotationProcessor,testCompileClasspath,testRuntimeClasspath
+biz.aQute.bnd:biz.aQute.bnd.annotation:7.1.0=compileClasspath,compileClasspathCopy,testCompileClasspath,testCompileClasspathCopy
+com.carrotsearch.randomizedtesting:randomizedtesting-runner:2.8.4=jarValidation,testCompileClasspath,testCompileClasspathCopy,testRuntimeClasspath,testRuntimeClasspathCopy
+com.carrotsearch:hppc:0.10.0=compileClasspath,compileClasspathCopy,jarValidation,runtimeClasspath,runtimeClasspathCopy,runtimeLibs,testCompileClasspath,testCompileClasspathCopy,testRuntimeClasspath,testRuntimeClasspathCopy
Review Comment:
They came from running resolveAndLockAll --write-locks repo-wide while validating the CI build locally, it added *Copy/spotless config tags to existing modules. I've reverted those to match main.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17949: Add Azure Blob Storage backup repository module [solr]
Posted by "prateeksinghalgit (via GitHub)" <gi...@apache.org>.
prateeksinghalgit commented on code in PR #3750:
URL: https://github.com/apache/solr/pull/3750#discussion_r3469305946
##########
solr/modules/azure-blob-repository/README.md:
##########
@@ -0,0 +1,118 @@
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+
+# Apache Solr Azure Blob Storage Backup Repository
+
+A backup repository implementation for storing Solr backups in Azure Blob Storage.
+
+## Prerequisites
+
+- Azure Storage Account with a blob container (must already exist)
+- Network access to Azure Blob Storage (HTTPS port 443)
+
+Enable the module:
+```bash
+export SOLR_MODULES=azure-blob-repository
+```
+
+## Configuration
+
+Add to `solr.xml`:
+
+```xml
+<backup>
+ <repository name="azure_blob" class="org.apache.solr.azureblob.AzureBlobBackupRepository" default="false">
+ <str name="azure.blob.container.name">YOUR_CONTAINER_NAME</str>
+ <!-- Authentication options below -->
+ </repository>
+</backup>
+```
+
+## Authentication Methods
+
+### Connection String (Development)
+
+```xml
+<str name="azure.blob.connection.string">DefaultEndpointsProtocol=https;AccountName=...;AccountKey=...;EndpointSuffix=core.windows.net</str>
+```
+
+### Account Name + Account Key
+
+```xml
+<str name="azure.blob.endpoint">https://YOUR_ACCOUNT.blob.core.windows.net</str>
+<str name="azure.blob.account.name">YOUR_ACCOUNT</str>
+<str name="azure.blob.account.key">YOUR_ACCOUNT_KEY</str>
+```
+
+### SAS Token (Production)
+
+Generate a SAS token with permissions: Read, Write, Delete, List, Add, Create (`sp=rwdlac`) and resource types: Service, Container, Object (`srt=sco`).
+
+```xml
+<str name="azure.blob.endpoint">https://YOUR_ACCOUNT.blob.core.windows.net</str>
+<str name="azure.blob.sas.token">sv=2024-11-04&ss=b&srt=sco&sp=rwdlac&...</str>
+```
+
+Note: Escape `&` as `&` in XML.
+
+### Azure Identity (Production - Recommended)
+
+Uses Azure AD authentication. Requires "Storage Blob Data Contributor" role on the storage account.
+
+```xml
+<str name="azure.blob.endpoint">https://YOUR_ACCOUNT.blob.core.windows.net</str>
+<!-- No credentials needed when using Managed Identity or Azure CLI -->
+```
+
+For Service Principal, add:
+```xml
+<str name="azure.blob.tenant.id">YOUR_TENANT_ID</str>
+<str name="azure.blob.client.id">YOUR_CLIENT_ID</str>
+<str name="azure.blob.client.secret">YOUR_CLIENT_SECRET</str>
+```
+
+Or set environment variables: `AZURE_TENANT_ID`, `AZURE_CLIENT_ID`, `AZURE_CLIENT_SECRET`.
+
+## Known Limitations
+
+Azure Identity authentication (Service Principal, Managed Identity, `DefaultAzureCredential`) does not work when Solr is started with the Java `SecurityManager` enabled. The Azure Identity SDK relies on `doPrivileged` patterns that fail under Solr's default security policy; see the [upstream issue](https://github.com/Azure/azure-sdk-for-java/issues/37464) for details. Note that the `SecurityManager` is deprecated for removal in modern JDKs, but Solr still enables it by default via `SOLR_SECURITY_MANAGER_ENABLED=true`.
+
+Workaround: set `SOLR_SECURITY_MANAGER_ENABLED=false` (in `solr.in.sh` / `solr.in.cmd`, or as an environment variable) before starting Solr. The Connection String, Account Key, and SAS Token authentication methods are unaffected and work with the `SecurityManager` enabled.
+
+## Usage
+
+```bash
+# Backup
+curl "http://localhost:8983/solr/admin/collections?action=BACKUP&name=my-backup&collection=my-collection&repository=azure_blob&location=/"
Review Comment:
Agreed, I have trimmed README to a short developer note (how to enable the module). The curl examples were originally there to help a future developer debug the module locally; I've dropped them from the README.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17949: Add Azure Blob Storage backup repository module [solr]
Posted by "prateeksinghalgit (via GitHub)" <gi...@apache.org>.
prateeksinghalgit commented on PR #3750:
URL: https://github.com/apache/solr/pull/3750#issuecomment-4627101726
Status update: CI is now fully green (Gradle Precommit, Crave tests, and changelog validation all passing on the latest commit). All of @psalagnac's feedback is addressed.
@psalagnac, when you have a chance, could you please review the pr . @HoustonPutman, your feedback is welcome too. Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17949: Add Azure Blob Storage backup repository module [solr]
Posted by "prateeksinghalgit (via GitHub)" <gi...@apache.org>.
prateeksinghalgit commented on PR #3750:
URL: https://github.com/apache/solr/pull/3750#issuecomment-4428441333
@psalagnac , thank you for taking the time to review.
1. can you replace all unnecessary qualified class names by imports? made the change by replacing all unnecessary qualified class names by imports
2. there is a lot of shared logic between Azure backups and S3 backups. Lot of code is duplicated. I wonder if abstract classes should be introduced to factor out some logic? - s3-repository is in active production use; a refactor would force shared abstractions and re-testing on a stable module to land a brand-new one. That's significant scope creep with no end-user benefit. Also the duplication is mostly in boilerplate (resolveDirectory shape, sanitization helpers); both repositories have different SDK contracts underneath.
Let me know what you think.
3. overall, I think it's pretty inefficient that AzureBlobIndexInput always reads files by small pages. Mostly for backup/restore, we will only fetch full files. You should not do a request by page, or use much bigger pages (without keeping them in memory) ? Given the comments on AzureBlobIndexInput, I have rewritten AzureBlobIndexInput to extend Lucene's standard BufferedIndexInput instead of IndexInput directly. Now only override readInternal(ByteBuffer) and seekInternal(long), and everything else (buffer management, slice, clone, EOF handling) comes from the parent class.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17949: Add Azure Blob Storage backup repository module [solr]
Posted by "prateeksinghalgit (via GitHub)" <gi...@apache.org>.
prateeksinghalgit commented on PR #3750:
URL: https://github.com/apache/solr/pull/3750#issuecomment-5596982950
@janhoy, any pointers on who in the Solr community might have access to an Azure environment and could help with the final review/testing of this PR? Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17949: Add Azure Blob Storage backup repository module [solr]
Posted by "prateeksinghalgit (via GitHub)" <gi...@apache.org>.
prateeksinghalgit commented on PR #3750:
URL: https://github.com/apache/solr/pull/3750#issuecomment-3600575623
Hi All,
I’ve fixed the recent CI failures:
1. Added proper thread leak filters for Testcontainers / JNA so tests are cleanly skipped when Docker isn’t available in CI.
2. Fixed two error-prone findings (IntLongMath and MissingOverride) that only surfaced in CI because error-prone is disabled by default in local builds unless -Pvalidation.errorprone=true is used.
Please let me know if anything else should be adjusted to help move the review forward. I’m happy to iterate further as needed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17949: Add Azure Blob Storage backup repository module [solr]
Posted by "janhoy (via GitHub)" <gi...@apache.org>.
janhoy commented on code in PR #3750:
URL: https://github.com/apache/solr/pull/3750#discussion_r2544753855
##########
gradle/libs.versions.toml:
##########
@@ -51,6 +51,10 @@ aqute-bnd = "6.4.1"
asciidoctor-mathjax = "0.0.9"
# @keep Asciidoctor tabs version used in ref-guide
asciidoctor-tabs = "1.0.0-beta.6"
+azure-storage = "12.25.0"
+azure-identity = "1.12.0"
+azure-core = "1.52.0"
+azure-core-http-netty = "1.15.4"
Review Comment:
I see this PR adds a bunch of netty httpclient dependencies. We already have Netty as a dependency with version 4.2.6 but azure brings in v4.1.110 in addition.
Either force the use of 4.2.6 for azure-blob if possible, or select another http-client, switch to the `azure-core-http-jdk-httpclient` client or make a wrapper around the jetty client (probably a tall order).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17949: Add Azure Blob Storage backup repository module [solr]
Posted by "prateeksinghalgit (via GitHub)" <gi...@apache.org>.
prateeksinghalgit commented on code in PR #3750:
URL: https://github.com/apache/solr/pull/3750#discussion_r3228702224
##########
solr/modules/azure-blob-repository/src/java/org/apache/solr/azureblob/AzureBlobBackupRepository.java:
##########
@@ -0,0 +1,396 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.azureblob;
+
+import com.google.common.annotations.VisibleForTesting;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStream;
+import java.lang.invoke.MethodHandles;
+import java.net.URI;
+import java.net.URISyntaxException;
+import java.time.Duration;
+import java.time.Instant;
+import java.util.Collection;
+import java.util.Objects;
+import java.util.Set;
+import java.util.stream.Collectors;
+import org.apache.lucene.store.Directory;
+import org.apache.lucene.store.IOContext;
+import org.apache.lucene.store.IndexInput;
+import org.apache.lucene.store.IndexOutput;
+import org.apache.solr.common.SolrException;
+import org.apache.solr.common.util.NamedList;
+import org.apache.solr.common.util.StrUtils;
+import org.apache.solr.core.backup.repository.AbstractBackupRepository;
+import org.apache.solr.core.backup.repository.BackupRepository;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * A concrete implementation of {@link BackupRepository} interface supporting backup/restore of Solr
+ * indexes to Azure Blob Storage.
+ */
+public class AzureBlobBackupRepository extends AbstractBackupRepository {
+
+ private static final Logger log = LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+
+ static final String BLOB_SCHEME = "blob";
+ private static final int CHUNK_SIZE = 16 * 1024 * 1024;
+ private static final int COPY_BUFFER_SIZE = 8192;
+
+ private AzureBlobStorageClient client;
+
+ @Override
+ public void init(NamedList<?> args) {
+ super.init(args);
+ AzureBlobBackupRepositoryConfig backupConfig = new AzureBlobBackupRepositoryConfig(this.config);
+
+ if (client != null) {
+ client.close();
+ }
+
+ this.client = backupConfig.buildClient();
+ }
+
+ @VisibleForTesting
+ public void setClient(AzureBlobStorageClient client) {
+ this.client = client;
+ }
+
+ @Override
+ @SuppressWarnings("unchecked")
+ public <T> T getConfigProperty(String name) {
+ return (T) this.config.get(name);
+ }
+
+ @Override
+ public URI createURI(String location) {
+ if (StrUtils.isNullOrEmpty(location)) {
+ throw new IllegalArgumentException("cannot create URI with an empty location");
+ }
+
+ URI result;
+ try {
+ if (location.startsWith(BLOB_SCHEME + ":")) {
+ result = new URI(location);
+ } else if (location.startsWith("/")) {
+ result = new URI(BLOB_SCHEME, "", location, null);
+ } else {
+ result = new URI(BLOB_SCHEME, "", "/" + location, null);
+ }
+ return result;
+ } catch (URISyntaxException ex) {
+ throw new SolrException(SolrException.ErrorCode.BAD_REQUEST, ex);
+ }
+ }
+
+ @Override
+ public URI createDirectoryURI(String location) {
+ if (StrUtils.isNullOrEmpty(location)) {
+ throw new IllegalArgumentException("cannot create URI with an empty location");
+ }
+
+ if (!location.endsWith("/")) {
+ location += "/";
+ }
+
+ return createURI(location);
+ }
+
+ @Override
+ public URI resolve(URI baseUri, String... pathComponents) {
+ if (!BLOB_SCHEME.equalsIgnoreCase(baseUri.getScheme())) {
+ throw new IllegalArgumentException("URI must begin with 'blob:' scheme");
+ }
+
+ String path = baseUri + "/" + String.join("/", pathComponents);
+ return URI.create(path).normalize();
+ }
+
+ @Override
+ public URI resolveDirectory(URI baseUri, String... pathComponents) {
+ if (pathComponents.length > 0) {
+ if (!pathComponents[pathComponents.length - 1].endsWith("/")) {
+ pathComponents[pathComponents.length - 1] = pathComponents[pathComponents.length - 1] + "/";
+ }
+ } else {
+ if (!baseUri.toString().endsWith("/")) {
+ baseUri = URI.create(baseUri + "/");
+ }
+ }
+ return resolve(baseUri, pathComponents);
+ }
+
+ @Override
+ public void createDirectory(URI path) throws IOException {
+ Objects.requireNonNull(path, "cannot create directory to a null URI");
+
+ String blobPath = getBlobPath(path);
+
+ if (log.isDebugEnabled()) {
+ log.debug("Create directory '{}'", blobPath);
+ }
+
+ try {
+ client.createDirectory(blobPath);
+ } catch (AzureBlobException e) {
+ throw new IOException("Failed to create directory " + blobPath, e);
+ }
+ }
+
+ @Override
+ public void deleteDirectory(URI path) throws IOException {
+ Objects.requireNonNull(path, "cannot delete directory with a null URI");
+
+ String blobPath = getBlobPath(path);
+
+ if (log.isDebugEnabled()) {
+ log.debug("Delete directory '{}'", blobPath);
+ }
+
+ try {
+ client.deleteDirectory(blobPath);
+ } catch (AzureBlobException e) {
+ throw new IOException("Failed to delete directory " + blobPath, e);
+ }
+ }
+
+ @Override
+ public void delete(URI path, Collection<String> files) throws IOException {
+ Objects.requireNonNull(path, "cannot delete with a null URI");
+ Objects.requireNonNull(files, "cannot delete with a null files collection");
+
+ String basePath = getBlobPath(path);
+
+ try {
+ if (!client.isDirectory(basePath)) {
+ int lastSlash = basePath.lastIndexOf('/');
+ basePath = lastSlash >= 0 ? basePath.substring(0, lastSlash) : "";
+ }
+ } catch (AzureBlobException e) {
+ throw new IOException("Failed to check path type for " + basePath, e);
+ }
+
+ final String baseForPaths = basePath;
+ Set<String> fullPaths =
+ files.stream()
+ .map(file -> (baseForPaths.isEmpty() ? file : baseForPaths + "/" + file))
+ .collect(Collectors.toSet());
+
+ if (log.isDebugEnabled()) {
+ log.debug("Delete files '{}'", fullPaths);
+ }
+
+ try {
+ client.delete(fullPaths);
+ } catch (AzureBlobException e) {
+ throw new IOException("Failed to delete files " + fullPaths, e);
+ }
+ }
+
+ @Override
+ public boolean exists(URI path) throws IOException {
+ Objects.requireNonNull(path, "cannot check existence with a null URI");
+
+ String blobPath = getBlobPath(path);
+
+ if (log.isDebugEnabled()) {
+ log.debug("Check existence '{}'", blobPath);
+ }
+
+ try {
+ return client.pathExists(blobPath);
+ } catch (AzureBlobException e) {
+ throw new IOException("Failed to check existence of " + blobPath, e);
+ }
+ }
+
+ @Override
+ public PathType getPathType(URI path) throws IOException {
+ Objects.requireNonNull(path, "cannot get path type with a null URI");
+
+ String blobPath = getBlobPath(path);
+
+ if (log.isDebugEnabled()) {
+ log.debug("Get path type '{}'", blobPath);
+ }
+
+ try {
+ if (client.isDirectory(blobPath)) {
+ return BackupRepository.PathType.DIRECTORY;
+ } else {
+ return BackupRepository.PathType.FILE;
+ }
+ } catch (AzureBlobException e) {
+ throw new IOException("Failed to get path type for " + blobPath, e);
+ }
+ }
+
+ @Override
+ public String[] listAll(URI path) throws IOException {
+ Objects.requireNonNull(path, "cannot list with a null URI");
+
+ String blobPath = getBlobPath(path);
+
+ if (log.isDebugEnabled()) {
+ log.debug("List all '{}'", blobPath);
+ }
+
+ try {
+ return client.listDir(blobPath);
+ } catch (AzureBlobException e) {
+ throw new IOException("Failed to list directory " + blobPath, e);
+ }
+ }
+
+ @Override
+ public IndexInput openInput(URI dirPath, String fileName, IOContext ctx) throws IOException {
+ Objects.requireNonNull(dirPath, "cannot open input with a null URI");
+ Objects.requireNonNull(fileName, "cannot open input with a null fileName");
+
+ String base = getBlobPath(dirPath);
+ String blobPath = base.endsWith("/") ? base + fileName : base + "/" + fileName;
+
+ if (log.isDebugEnabled()) {
+ log.debug("Open input '{}'", blobPath);
+ }
+
+ try {
+ return new AzureBlobIndexInput(blobPath, client, client.length(blobPath));
+ } catch (AzureBlobException e) {
+ throw new IOException("Failed to open input stream for " + blobPath, e);
+ }
+ }
+
+ @Override
+ public OutputStream createOutput(URI path) throws IOException {
+ Objects.requireNonNull(path, "cannot create output with a null URI");
+
+ String blobPath = getBlobPath(path);
+
+ if (log.isDebugEnabled()) {
+ log.debug("Create output '{}'", blobPath);
+ }
+
+ try {
+ return client.pushStream(blobPath);
+ } catch (AzureBlobException e) {
+ throw new IOException("Failed to create output stream for " + blobPath, e);
+ }
+ }
+
+ @Override
+ public void copyIndexFileFrom(
+ Directory sourceDir, String sourceFileName, URI dest, String destFileName)
+ throws IOException {
+ Objects.requireNonNull(sourceDir, "cannot copy with a null sourceDir");
+ Objects.requireNonNull(sourceFileName, "cannot copy with a null sourceFileName");
+ Objects.requireNonNull(dest, "cannot copy with a null dest");
+
+ String destPath = getBlobPath(dest);
+
+ String blobPath = destPath.endsWith("/") ? destPath + destFileName : destPath;
Review Comment:
yes, updated it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17949: Add Azure Blob Storage backup repository module [solr]
Posted by "prateeksinghalgit (via GitHub)" <gi...@apache.org>.
prateeksinghalgit commented on code in PR #3750:
URL: https://github.com/apache/solr/pull/3750#discussion_r3232173682
##########
solr/modules/azure-blob-repository/src/java/org/apache/solr/azureblob/AzureBlobOutputStream.java:
##########
@@ -0,0 +1,269 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.azureblob;
+
+import com.azure.storage.blob.BlobClient;
+import com.azure.storage.blob.models.BlobStorageException;
+import com.azure.storage.blob.specialized.BlockBlobClient;
+import java.io.ByteArrayInputStream;
+import java.io.IOException;
+import java.io.OutputStream;
+import java.lang.invoke.MethodHandles;
+import java.nio.ByteBuffer;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Base64;
+import java.util.List;
+import java.util.UUID;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * OutputStream implementation for Azure Blob Storage using block blobs. Supports chunked uploads
+ * for large files.
+ */
+public class AzureBlobOutputStream extends OutputStream {
+ private static final Logger log = LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+
+ static final int BLOCK_SIZE = 4 * 1024 * 1024;
+
+ private final BlobClient blobClient;
+ private final String blobPath;
+ private volatile boolean closed;
+ private final ByteBuffer buffer;
+ private BlockUpload blockUpload;
+ private boolean committed;
+
+ public AzureBlobOutputStream(BlobClient blobClient, String blobPath) {
+ this.blobClient = blobClient;
+ this.blobPath = blobPath;
+ this.closed = false;
+ this.buffer = ByteBuffer.allocate(BLOCK_SIZE);
+ this.blockUpload = null;
+ this.committed = false;
+
+ if (log.isDebugEnabled()) {
+ log.debug("Created BlobOutputStream for blobPath '{}'", blobPath);
+ }
+ }
+
+ @Override
+ public void write(int b) throws IOException {
+ if (closed) {
+ throw new IOException("Stream closed");
+ }
+
+ buffer.put((byte) b);
+
+ if (!buffer.hasRemaining()) {
+ uploadBlock();
+ }
+ }
+
+ @Override
+ public void write(byte[] b, int off, int len) throws IOException {
+ if (closed) {
+ throw new IOException("Stream closed");
+ }
+
+ if (outOfRange(off, b.length) || len < 0 || outOfRange(off + len, b.length)) {
+ throw new IndexOutOfBoundsException();
+ } else if (len == 0) {
+ return;
+ }
+
+ int currentOffset = off;
+ int lenRemaining = len;
+ while (buffer.remaining() < lenRemaining) {
+ int firstPart = buffer.remaining();
+ buffer.put(b, currentOffset, firstPart);
+ uploadBlock();
+
+ currentOffset += firstPart;
+ lenRemaining -= firstPart;
+ }
+ if (lenRemaining > 0) {
+ buffer.put(b, currentOffset, lenRemaining);
+ }
+ }
+
+ private static boolean outOfRange(int off, int len) {
+ return off < 0 || off > len;
+ }
+
+ private void uploadBlock() throws IOException {
+ int size = buffer.position() - buffer.arrayOffset();
Review Comment:
updated it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17949: Add Azure Blob Storage backup repository module [solr]
Posted by "janhoy (via GitHub)" <gi...@apache.org>.
janhoy commented on code in PR #3750:
URL: https://github.com/apache/solr/pull/3750#discussion_r2556609430
##########
solr/solr-ref-guide/modules/deployment-guide/pages/backup-restore.adoc:
##########
@@ -383,7 +383,7 @@ If the status is anything other than "success", an error message will explain wh
Solr provides a repository abstraction to allow users to backup and restore their data to a variety of different storage systems.
For example, a Solr cluster running on a local filesystem (e.g., EXT3) can store backup data on the same disk, on a remote network-mounted drive, or in some popular "cloud storage" providers, depending on the 'repository' implementation chosen.
-Solr offers multiple different repository implementations out of the box (`LocalFileSystemRepository`, `GCSBackupRepository` and `S3BackupRepository`), and allows users to create plugins for their own storage systems as needed. It is also possible to create a `DelegatingBackupRepository` that delegates to another `BackupRepository` and adds or modifies some behavior on top of it.
+Solr offers multiple different repository implementations out of the box (`LocalFileSystemRepository`, `GCSBackupRepository`, `S3BackupRepository`, and `BlobBackupRepository`), and allows users to create plugins for their own storage systems as needed. It is also possible to create a `DelegatingBackupRepository` that delegates to another `BackupRepository` and adds or modifies some behavior on top of it.
Review Comment:
Update class name in docs
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17949: Add Azure Blob Storage backup repository module [solr]
Posted by "janhoy (via GitHub)" <gi...@apache.org>.
janhoy commented on code in PR #3750:
URL: https://github.com/apache/solr/pull/3750#discussion_r2558931280
##########
solr/solr-ref-guide/modules/deployment-guide/pages/backup-restore.adoc:
##########
@@ -383,7 +383,7 @@ If the status is anything other than "success", an error message will explain wh
Solr provides a repository abstraction to allow users to backup and restore their data to a variety of different storage systems.
For example, a Solr cluster running on a local filesystem (e.g., EXT3) can store backup data on the same disk, on a remote network-mounted drive, or in some popular "cloud storage" providers, depending on the 'repository' implementation chosen.
-Solr offers multiple different repository implementations out of the box (`LocalFileSystemRepository`, `GCSBackupRepository` and `S3BackupRepository`), and allows users to create plugins for their own storage systems as needed. It is also possible to create a `DelegatingBackupRepository` that delegates to another `BackupRepository` and adds or modifies some behavior on top of it.
+Solr offers multiple different repository implementations out of the box (`LocalFileSystemRepository`, `GCSBackupRepository`, `S3BackupRepository`, and `BlobBackupRepository`), and allows users to create plugins for their own storage systems as needed. It is also possible to create a `DelegatingBackupRepository` that delegates to another `BackupRepository` and adds or modifies some behavior on top of it.
Review Comment:
Thanks for quick and efficient implementation of review comments @prateeksinghalgit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17949: Add Azure Blob Storage backup repository module [solr]
Posted by "prateeksinghalgit (via GitHub)" <gi...@apache.org>.
prateeksinghalgit commented on code in PR #3750:
URL: https://github.com/apache/solr/pull/3750#discussion_r2563707779
##########
solr/modules/azure-blob-repository/src/test/org/apache/solr/azureblob/AbstractAzureBlobClientTest.java:
##########
@@ -0,0 +1,249 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.azureblob;
+
+import com.azure.core.http.HttpClient;
+import com.azure.core.http.okhttp.OkHttpAsyncHttpClientBuilder;
+import com.azure.storage.blob.BlobServiceClient;
+import com.azure.storage.blob.BlobServiceClientBuilder;
+import com.carrotsearch.randomizedtesting.ThreadFilter;
+import com.carrotsearch.randomizedtesting.annotations.ThreadLeakFilters;
+import java.io.IOException;
+import java.io.OutputStream;
+import java.net.Socket;
+import java.nio.charset.StandardCharsets;
+import okhttp3.OkHttpClient;
+import org.apache.solr.SolrTestCaseJ4;
+import org.junit.After;
+import org.junit.AfterClass;
+import org.junit.Before;
+import org.junit.BeforeClass;
+import org.junit.Rule;
+import org.junit.rules.TemporaryFolder;
+
+import static org.junit.Assume.assumeTrue;
+
+/**
+ * Abstract class for tests with Azure Blob Storage emulator.
+ *
+ * <p>Uses ThreadLeakFilters to ignore OkHttp's global singleton threads (TaskRunner and Okio
+ * Watchdog) which are JVM-wide and cannot be shut down per-client. This is a standard pattern in
+ * Solr tests for HTTP clients with global thread pools.
+ */
+@ThreadLeakFilters(
+ filters = {
+ AbstractAzureBlobClientTest.OkHttpThreadLeakFilter.class,
+ })
+public class AbstractAzureBlobClientTest extends SolrTestCaseJ4 {
+
+ protected String containerName;
+
+ @Rule public TemporaryFolder tempFolder = new TemporaryFolder();
+
+ AzureBlobStorageClient client;
+ private static String connectionString;
+ // Shared OkHttpClient across all tests to reuse the global thread pools
+ private static OkHttpClient sharedOkHttpClient;
+ protected org.apache.solr.client.solrj.cloud.SocketProxy proxy;
+
+ @BeforeClass
+ public static void setUpClass() {
+ // Create a single shared OkHttpClient for all tests
+ // This reuses OkHttp's global thread pools efficiently
+ sharedOkHttpClient = new OkHttpClient.Builder().build();
+ }
+
+ @Before
+ public void setUpClient() throws Exception {
+ // Check if Azurite is running before attempting to connect
+ assumeAzuriteIsRunning();
+
+ setAzureTestCredentials();
+
+ // Use Azurite connection string for local testing
+ connectionString =
+ "DefaultEndpointsProtocol=http;AccountName=devstoreaccount1;AccountKey=Eby8vdM02xNOcqFlqUwJPLlmEtlCDXJ1OUzFT50uSRZ6IFsuFq2UVErCz4I6tq/K1SZFPTOtr/KBHBeksoGMGw==;BlobEndpoint=http://localhost:10000/devstoreaccount1;";
+
+ // Put a proxy in front of Azurite to simulate connection loss like S3 tests
+ proxy = new org.apache.solr.client.solrj.cloud.SocketProxy();
+ proxy.open(new java.net.URI(getBlobServiceUrl()));
+
+ // Reuse the shared OkHttpClient
+ HttpClient httpClient = new OkHttpAsyncHttpClientBuilder(sharedOkHttpClient).build();
+
+ // Route Blob endpoint through the proxy by adjusting the connection string
+ String proxiedConn = connectionString.replace(":10000", ":" + proxy.getListenPort());
+ BlobServiceClient blobServiceClient =
+ new BlobServiceClientBuilder()
+ .connectionString(proxiedConn)
+ .httpClient(httpClient)
+ .buildClient();
+
+ containerName = "test-" + java.util.UUID.randomUUID();
+ client = new AzureBlobStorageClient(blobServiceClient, containerName);
+ }
+
+ /**
+ * Set up Azure test credentials to avoid using real Azure credentials during testing. Similar to
+ * how S3 tests use ProfileFileSystemSetting to avoid polluting the test environment.
+ */
+ public static void setAzureTestCredentials() {
+ // Set test Azure credentials to avoid using real credentials
+ System.setProperty("AZURE_CLIENT_ID", "test-client-id");
+ System.setProperty("AZURE_TENANT_ID", "test-tenant-id");
+ System.setProperty("AZURE_CLIENT_SECRET", "test-client-secret");
+
+ // Set Azurite-specific environment variables
+ System.setProperty(
+ "AZURE_STORAGE_CONNECTION_STRING",
+ "DefaultEndpointsProtocol=http;AccountName=devstoreaccount1;AccountKey=Eby8vdM02xNOcqFlqUwJPLlmEtlCDXJ1OUzFT50uSRZ6IFsuFq2UVErCz4I6tq/K1SZFPTOtr/KBHBeksoGMGw==;BlobEndpoint=http://localhost:10000/devstoreaccount1;");
+ }
+
+ @After
+ public void tearDownClient() {
+ if (client != null) {
+ try {
+ client.deleteContainerForTests();
+ } catch (Throwable ignored) {
+ }
+ client.close();
+ }
+ if (proxy != null) {
+ proxy.close();
+ proxy = null;
+ }
+ // Note: We don't clean up sharedOkHttpClient here as it's shared across all tests
+ // It will be cleaned up in afterAll()
+ }
+
+ /** Simulate a connection loss on the proxy similar to S3 tests. */
+ void initiateBlobConnectionLoss() throws AzureBlobException {
+ if (proxy != null) {
+ proxy.halfClose();
+ }
+ }
+
+ @AfterClass
+ public static void afterAll() {
+ // Clean up the shared OkHttpClient
+ if (sharedOkHttpClient != null) {
+ // Shutdown dispatcher's executor service
+ sharedOkHttpClient.dispatcher().executorService().shutdown();
+ // Cancel all queued calls
+ sharedOkHttpClient.dispatcher().cancelAll();
+ // Evict all connections
+ sharedOkHttpClient.connectionPool().evictAll();
+ // Close cache if present
+ try {
+ if (sharedOkHttpClient.cache() != null) {
+ sharedOkHttpClient.cache().close();
+ }
+ } catch (Throwable ignored) {
+ }
+ // Wait for threads to terminate
+ try {
+ sharedOkHttpClient
+ .dispatcher()
+ .executorService()
+ .awaitTermination(2, java.util.concurrent.TimeUnit.SECONDS);
+ } catch (Throwable ignored) {
+ }
+ sharedOkHttpClient = null;
+ }
+
+ // Shutdown Reactor schedulers to clean up background threads
+ try {
+ reactor.core.scheduler.Schedulers.shutdownNow();
+ // Give threads time to terminate
+ Thread.sleep(100);
+ } catch (Throwable ignored) {
+ }
+ }
+
+ /**
+ * Helper method to push a string to Azure Blob Storage.
+ *
+ * @param path Destination path in blob storage.
+ * @param content Arbitrary content for the test.
+ */
+ void pushContent(String path, String content) throws AzureBlobException {
+ pushContent(path, content.getBytes(StandardCharsets.UTF_8));
+ }
+
+ void pushContent(String path, byte[] content) throws AzureBlobException {
+ try (OutputStream output = client.pushStream(path)) {
+ output.write(content);
+ } catch (IOException e) {
+ throw new AzureBlobException("Failed to write content", e);
+ }
+ }
+
+ /** Get the connection string for tests that need direct access to the blob service. */
+ static String getConnectionString() {
+ return connectionString;
+ }
+
+ /** Get the blob service URL for tests that need direct access. */
+ String getBlobServiceUrl() {
+ return "http://localhost:10000";
+ }
+
+ /**
+ * Check if Azurite is running on localhost:10000. If not, skip the test.
+ *
+ * <p>Azurite must be started manually for these tests to run. This allows tests to pass in CI
+ * environments where Azurite may not be available.
+ */
+ private void assumeAzuriteIsRunning() {
+ boolean azuriteRunning = false;
+ try (Socket socket = new Socket("localhost", 10000)) {
Review Comment:
this is same approach as TikaServer tests are taking org/apache/solr/handler/extraction/TikaServerExtractionBackendTest.java
##########
solr/modules/azure-blob-repository/src/test/org/apache/solr/azureblob/AbstractAzureBlobClientTest.java:
##########
@@ -0,0 +1,249 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.azureblob;
+
+import com.azure.core.http.HttpClient;
+import com.azure.core.http.okhttp.OkHttpAsyncHttpClientBuilder;
+import com.azure.storage.blob.BlobServiceClient;
+import com.azure.storage.blob.BlobServiceClientBuilder;
+import com.carrotsearch.randomizedtesting.ThreadFilter;
+import com.carrotsearch.randomizedtesting.annotations.ThreadLeakFilters;
+import java.io.IOException;
+import java.io.OutputStream;
+import java.net.Socket;
+import java.nio.charset.StandardCharsets;
+import okhttp3.OkHttpClient;
+import org.apache.solr.SolrTestCaseJ4;
+import org.junit.After;
+import org.junit.AfterClass;
+import org.junit.Before;
+import org.junit.BeforeClass;
+import org.junit.Rule;
+import org.junit.rules.TemporaryFolder;
+
+import static org.junit.Assume.assumeTrue;
+
+/**
+ * Abstract class for tests with Azure Blob Storage emulator.
+ *
+ * <p>Uses ThreadLeakFilters to ignore OkHttp's global singleton threads (TaskRunner and Okio
+ * Watchdog) which are JVM-wide and cannot be shut down per-client. This is a standard pattern in
+ * Solr tests for HTTP clients with global thread pools.
+ */
+@ThreadLeakFilters(
+ filters = {
+ AbstractAzureBlobClientTest.OkHttpThreadLeakFilter.class,
+ })
+public class AbstractAzureBlobClientTest extends SolrTestCaseJ4 {
+
+ protected String containerName;
+
+ @Rule public TemporaryFolder tempFolder = new TemporaryFolder();
+
+ AzureBlobStorageClient client;
+ private static String connectionString;
+ // Shared OkHttpClient across all tests to reuse the global thread pools
+ private static OkHttpClient sharedOkHttpClient;
+ protected org.apache.solr.client.solrj.cloud.SocketProxy proxy;
+
+ @BeforeClass
+ public static void setUpClass() {
+ // Create a single shared OkHttpClient for all tests
+ // This reuses OkHttp's global thread pools efficiently
+ sharedOkHttpClient = new OkHttpClient.Builder().build();
+ }
+
+ @Before
+ public void setUpClient() throws Exception {
+ // Check if Azurite is running before attempting to connect
+ assumeAzuriteIsRunning();
+
+ setAzureTestCredentials();
+
+ // Use Azurite connection string for local testing
+ connectionString =
+ "DefaultEndpointsProtocol=http;AccountName=devstoreaccount1;AccountKey=Eby8vdM02xNOcqFlqUwJPLlmEtlCDXJ1OUzFT50uSRZ6IFsuFq2UVErCz4I6tq/K1SZFPTOtr/KBHBeksoGMGw==;BlobEndpoint=http://localhost:10000/devstoreaccount1;";
+
+ // Put a proxy in front of Azurite to simulate connection loss like S3 tests
+ proxy = new org.apache.solr.client.solrj.cloud.SocketProxy();
+ proxy.open(new java.net.URI(getBlobServiceUrl()));
+
+ // Reuse the shared OkHttpClient
+ HttpClient httpClient = new OkHttpAsyncHttpClientBuilder(sharedOkHttpClient).build();
+
+ // Route Blob endpoint through the proxy by adjusting the connection string
+ String proxiedConn = connectionString.replace(":10000", ":" + proxy.getListenPort());
+ BlobServiceClient blobServiceClient =
+ new BlobServiceClientBuilder()
+ .connectionString(proxiedConn)
+ .httpClient(httpClient)
+ .buildClient();
+
+ containerName = "test-" + java.util.UUID.randomUUID();
+ client = new AzureBlobStorageClient(blobServiceClient, containerName);
+ }
+
+ /**
+ * Set up Azure test credentials to avoid using real Azure credentials during testing. Similar to
+ * how S3 tests use ProfileFileSystemSetting to avoid polluting the test environment.
+ */
+ public static void setAzureTestCredentials() {
+ // Set test Azure credentials to avoid using real credentials
+ System.setProperty("AZURE_CLIENT_ID", "test-client-id");
+ System.setProperty("AZURE_TENANT_ID", "test-tenant-id");
+ System.setProperty("AZURE_CLIENT_SECRET", "test-client-secret");
+
+ // Set Azurite-specific environment variables
+ System.setProperty(
+ "AZURE_STORAGE_CONNECTION_STRING",
+ "DefaultEndpointsProtocol=http;AccountName=devstoreaccount1;AccountKey=Eby8vdM02xNOcqFlqUwJPLlmEtlCDXJ1OUzFT50uSRZ6IFsuFq2UVErCz4I6tq/K1SZFPTOtr/KBHBeksoGMGw==;BlobEndpoint=http://localhost:10000/devstoreaccount1;");
+ }
+
+ @After
+ public void tearDownClient() {
+ if (client != null) {
+ try {
+ client.deleteContainerForTests();
+ } catch (Throwable ignored) {
+ }
+ client.close();
+ }
+ if (proxy != null) {
+ proxy.close();
+ proxy = null;
+ }
+ // Note: We don't clean up sharedOkHttpClient here as it's shared across all tests
+ // It will be cleaned up in afterAll()
+ }
+
+ /** Simulate a connection loss on the proxy similar to S3 tests. */
+ void initiateBlobConnectionLoss() throws AzureBlobException {
+ if (proxy != null) {
+ proxy.halfClose();
+ }
+ }
+
+ @AfterClass
+ public static void afterAll() {
+ // Clean up the shared OkHttpClient
+ if (sharedOkHttpClient != null) {
+ // Shutdown dispatcher's executor service
+ sharedOkHttpClient.dispatcher().executorService().shutdown();
+ // Cancel all queued calls
+ sharedOkHttpClient.dispatcher().cancelAll();
+ // Evict all connections
+ sharedOkHttpClient.connectionPool().evictAll();
+ // Close cache if present
+ try {
+ if (sharedOkHttpClient.cache() != null) {
+ sharedOkHttpClient.cache().close();
+ }
+ } catch (Throwable ignored) {
+ }
+ // Wait for threads to terminate
+ try {
+ sharedOkHttpClient
+ .dispatcher()
+ .executorService()
+ .awaitTermination(2, java.util.concurrent.TimeUnit.SECONDS);
+ } catch (Throwable ignored) {
+ }
+ sharedOkHttpClient = null;
+ }
+
+ // Shutdown Reactor schedulers to clean up background threads
+ try {
+ reactor.core.scheduler.Schedulers.shutdownNow();
+ // Give threads time to terminate
+ Thread.sleep(100);
+ } catch (Throwable ignored) {
+ }
+ }
+
+ /**
+ * Helper method to push a string to Azure Blob Storage.
+ *
+ * @param path Destination path in blob storage.
+ * @param content Arbitrary content for the test.
+ */
+ void pushContent(String path, String content) throws AzureBlobException {
+ pushContent(path, content.getBytes(StandardCharsets.UTF_8));
+ }
+
+ void pushContent(String path, byte[] content) throws AzureBlobException {
+ try (OutputStream output = client.pushStream(path)) {
+ output.write(content);
+ } catch (IOException e) {
+ throw new AzureBlobException("Failed to write content", e);
+ }
+ }
+
+ /** Get the connection string for tests that need direct access to the blob service. */
+ static String getConnectionString() {
+ return connectionString;
+ }
+
+ /** Get the blob service URL for tests that need direct access. */
+ String getBlobServiceUrl() {
+ return "http://localhost:10000";
+ }
+
+ /**
+ * Check if Azurite is running on localhost:10000. If not, skip the test.
+ *
+ * <p>Azurite must be started manually for these tests to run. This allows tests to pass in CI
+ * environments where Azurite may not be available.
+ */
+ private void assumeAzuriteIsRunning() {
+ boolean azuriteRunning = false;
+ try (Socket socket = new Socket("localhost", 10000)) {
Review Comment:
this is same approach as TikaServer tests are taking `org/apache/solr/handler/extraction/TikaServerExtractionBackendTest.java`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17949: Add Azure Blob Storage backup repository module [solr]
Posted by "prateeksinghalgit (via GitHub)" <gi...@apache.org>.
prateeksinghalgit commented on PR #3750:
URL: https://github.com/apache/solr/pull/3750#issuecomment-4812107094
To make the final review easier: testing doesn't actually require an Azure account — the integration tests run against the Azurite emulator (Docker, free)- https://hub.docker.com/r/microsoft/azure-storage-azurite. I've separately verified backup/restore against a real Azure Storage account, including under the Java Security Manager.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17949: Add Azure Blob Storage backup repository module [solr]
Posted by "prateeksinghalgit (via GitHub)" <gi...@apache.org>.
prateeksinghalgit commented on PR #3750:
URL: https://github.com/apache/solr/pull/3750#issuecomment-4812106077
Thanks @janhoy. I've addressed all your points (dependency trim, lockfile cleanup, README, solr.xml, and the verified security.policy example). If you are ok with the updates could you please clear the "changes requested" status so it's not blocking the final reviewer.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17949: Add Azure Blob Storage backup repository module [solr]
Posted by "prateeksinghalgit (via GitHub)" <gi...@apache.org>.
prateeksinghalgit commented on code in PR #3750:
URL: https://github.com/apache/solr/pull/3750#discussion_r3469490063
##########
solr/modules/azure-blob-repository/gradle.lockfile:
##########
@@ -0,0 +1,195 @@
+# This is a Gradle generated file for dependency locking.
+# Manual edits can break the build and are not advised.
+# This file is expected to be part of source control.
+com.azure:azure-core-http-okhttp:1.13.3=jarValidation,runtimeClasspath,runtimeLibs,testCompileClasspath,testRuntimeClasspath
+com.azure:azure-core:1.57.1=compileClasspath,jarValidation,runtimeClasspath,runtimeLibs,testCompileClasspath,testRuntimeClasspath
+com.azure:azure-identity:1.18.2=compileClasspath,jarValidation,runtimeClasspath,runtimeLibs,testCompileClasspath,testRuntimeClasspath
+com.azure:azure-json:1.5.1=compileClasspath,jarValidation,runtimeClasspath,runtimeLibs,testCompileClasspath,testRuntimeClasspath
+com.azure:azure-sdk-bom:1.3.6=compileClasspath,jarValidation,runtimeClasspath,runtimeLibs,testCompileClasspath,testRuntimeClasspath
+com.azure:azure-storage-blob-batch:12.29.3=compileClasspath,jarValidation,runtimeClasspath,runtimeLibs,testCompileClasspath,testRuntimeClasspath
+com.azure:azure-storage-blob:12.33.3=compileClasspath,jarValidation,runtimeClasspath,runtimeLibs,testCompileClasspath,testRuntimeClasspath
+com.azure:azure-storage-common:12.32.2=compileClasspath,jarValidation,runtimeClasspath,runtimeLibs,testCompileClasspath,testRuntimeClasspath
+com.azure:azure-storage-internal-avro:12.18.2=compileClasspath,jarValidation,runtimeClasspath,runtimeLibs,testCompileClasspath,testRuntimeClasspath
+com.azure:azure-xml:1.2.1=compileClasspath,jarValidation,runtimeClasspath,runtimeLibs,testCompileClasspath,testRuntimeClasspath
+com.carrotsearch.randomizedtesting:randomizedtesting-runner:2.8.4=jarValidation,testCompileClasspath,testRuntimeClasspath
+com.carrotsearch:hppc:0.10.0=apiHelper,jarValidation,runtimeClasspath,runtimeLibs,solrPlatformLibs,testRuntimeClasspath
+com.fasterxml.jackson.core:jackson-annotations:2.21=apiHelper,compileClasspath,jarValidation,runtimeClasspath,runtimeLibs,solrPlatformLibs,testCompileClasspath,testRuntimeClasspath
+com.fasterxml.jackson.core:jackson-core:2.21.2=apiHelper,compileClasspath,jarValidation,runtimeClasspath,runtimeLibs,solrPlatformLibs,testCompileClasspath,testRuntimeClasspath
+com.fasterxml.jackson.core:jackson-databind:2.21.2=apiHelper,compileClasspath,jarValidation,runtimeClasspath,runtimeLibs,solrPlatformLibs,testCompileClasspath,testRuntimeClasspath
+com.fasterxml.jackson.dataformat:jackson-dataformat-cbor:2.21.2=apiHelper,jarValidation,runtimeClasspath,runtimeLibs,solrPlatformLibs,testRuntimeClasspath
+com.fasterxml.jackson.dataformat:jackson-dataformat-smile:2.21.2=apiHelper,jarValidation,runtimeClasspath,runtimeLibs,solrPlatformLibs,testRuntimeClasspath
+com.fasterxml.jackson.datatype:jackson-datatype-jsr310:2.21.2=compileClasspath,jarValidation,runtimeClasspath,runtimeLibs,testCompileClasspath,testRuntimeClasspath
+com.fasterxml.jackson.module:jackson-module-jakarta-xmlbind-annotations:2.21.2=apiHelper,jarValidation,runtimeClasspath,runtimeLibs,solrPlatformLibs,testRuntimeClasspath
+com.fasterxml.jackson:jackson-bom:2.21.2=apiHelper,compileClasspath,jarValidation,runtimeClasspath,runtimeLibs,solrPlatformLibs,testCompileClasspath,testRuntimeClasspath
+com.fasterxml.woodstox:woodstox-core:7.0.0=apiHelper,jarValidation,runtimeClasspath,runtimeLibs,solrPlatformLibs,testRuntimeClasspath
+com.github.ben-manes.caffeine:caffeine:3.2.3=annotationProcessor,apiHelper,errorprone,jarValidation,runtimeClasspath,runtimeLibs,solrPlatformLibs,testAnnotationProcessor,testRuntimeClasspath
+com.github.docker-java:docker-java-api:3.7.0=jarValidation,testCompileClasspath,testRuntimeClasspath
+com.github.docker-java:docker-java-transport-zerodep:3.7.0=jarValidation,testCompileClasspath,testRuntimeClasspath
+com.github.docker-java:docker-java-transport:3.7.0=jarValidation,testCompileClasspath,testRuntimeClasspath
+com.github.kevinstern:software-and-algorithms:1.0=annotationProcessor,errorprone,testAnnotationProcessor
+com.google.auto.service:auto-service-annotations:1.0.1=annotationProcessor,errorprone,testAnnotationProcessor
+com.google.auto.value:auto-value-annotations:1.11.0=annotationProcessor,errorprone,testAnnotationProcessor
+com.google.auto:auto-common:1.2.2=annotationProcessor,errorprone,testAnnotationProcessor
+com.google.errorprone:error_prone_annotation:2.41.0=annotationProcessor,errorprone,testAnnotationProcessor
+com.google.errorprone:error_prone_annotations:2.41.0=apiHelper,compileClasspath,jarValidation,runtimeClasspath,runtimeLibs,solrPlatformLibs,testCompileClasspath,testRuntimeClasspath
+com.google.errorprone:error_prone_annotations:2.43.0=annotationProcessor,errorprone,testAnnotationProcessor
+com.google.errorprone:error_prone_check_api:2.41.0=annotationProcessor,errorprone,testAnnotationProcessor
+com.google.errorprone:error_prone_core:2.41.0=annotationProcessor,errorprone,testAnnotationProcessor
+com.google.googlejavaformat:google-java-format:1.27.0=annotationProcessor,errorprone,testAnnotationProcessor
+com.google.guava:failureaccess:1.0.3=annotationProcessor,apiHelper,compileClasspath,errorprone,jarValidation,runtimeClasspath,runtimeLibs,solrPlatformLibs,testAnnotationProcessor,testCompileClasspath,testRuntimeClasspath
+com.google.guava:guava:33.5.0-jre=annotationProcessor,apiHelper,compileClasspath,errorprone,jarValidation,runtimeClasspath,runtimeLibs,solrPlatformLibs,testAnnotationProcessor,testCompileClasspath,testRuntimeClasspath
+com.google.guava:listenablefuture:9999.0-empty-to-avoid-conflict-with-guava=annotationProcessor,apiHelper,compileClasspath,errorprone,jarValidation,runtimeClasspath,runtimeLibs,solrPlatformLibs,testAnnotationProcessor,testCompileClasspath,testRuntimeClasspath
+com.google.j2objc:j2objc-annotations:3.1=annotationProcessor,apiHelper,compileClasspath,errorprone,jarValidation,runtimeClasspath,runtimeLibs,solrPlatformLibs,testAnnotationProcessor,testCompileClasspath,testRuntimeClasspath
+com.google.protobuf:protobuf-java:3.25.8=annotationProcessor,errorprone,testAnnotationProcessor
+com.j256.simplemagic:simplemagic:1.17=apiHelper,jarValidation,runtimeClasspath,runtimeLibs,solrPlatformLibs,testRuntimeClasspath
+com.jayway.jsonpath:json-path:2.9.0=apiHelper,jarValidation,runtimeClasspath,runtimeLibs,solrPlatformLibs,testRuntimeClasspath
+com.lmax:disruptor:4.0.0=solrPlatformLibs
+com.microsoft.azure:msal4j-persistence-extension:1.3.0=compileClasspath,jarValidation,runtimeClasspath,runtimeLibs,testCompileClasspath,testRuntimeClasspath
+com.microsoft.azure:msal4j:1.23.1=compileClasspath,jarValidation,runtimeClasspath,runtimeLibs,testCompileClasspath,testRuntimeClasspath
+com.squareup.okhttp3:okhttp:4.12.0=jarValidation,runtimeClasspath,runtimeLibs,testCompileClasspath,testRuntimeClasspath
+com.squareup.okio:okio-jvm:3.16.0=jarValidation,runtimeClasspath,runtimeLibs,testCompileClasspath,testRuntimeClasspath
+com.squareup.okio:okio:3.16.0=jarValidation,testCompileClasspath,testRuntimeClasspath
+com.tdunning:t-digest:3.3=apiHelper,jarValidation,runtimeClasspath,runtimeLibs,solrPlatformLibs,testRuntimeClasspath
+commons-cli:commons-cli:1.11.0=apiHelper,jarValidation,runtimeClasspath,runtimeLibs,solrPlatformLibs,testRuntimeClasspath
+commons-codec:commons-codec:1.21.0=apiHelper,jarValidation,runtimeClasspath,runtimeLibs,solrPlatformLibs,testCompileClasspath,testRuntimeClasspath
+commons-io:commons-io:2.21.0=apiHelper,compileClasspath,jarValidation,runtimeClasspath,runtimeLibs,solrPlatformLibs,testCompileClasspath,testRuntimeClasspath
+io.dropwizard.metrics:metrics-core:4.2.33=apiHelper,jarValidation,runtimeClasspath,runtimeLibs,solrPlatformLibs,testRuntimeClasspath
+io.github.eisop:dataflow-errorprone:3.41.0-eisop1=annotationProcessor,errorprone,testAnnotationProcessor
+io.github.java-diff-utils:java-diff-utils:4.12=annotationProcessor,errorprone,testAnnotationProcessor
+io.netty:netty-buffer:4.2.15.Final=apiHelper,compileClasspath,jarValidation,runtimeClasspath,runtimeLibs,solrPlatformLibs,testCompileClasspath,testRuntimeClasspath
+io.netty:netty-codec-base:4.2.15.Final=apiHelper,compileClasspath,jarValidation,runtimeClasspath,runtimeLibs,solrPlatformLibs,testCompileClasspath,testRuntimeClasspath
+io.netty:netty-common:4.2.15.Final=apiHelper,compileClasspath,jarValidation,runtimeClasspath,runtimeLibs,solrPlatformLibs,testCompileClasspath,testRuntimeClasspath
+io.netty:netty-handler:4.2.15.Final=apiHelper,compileClasspath,jarValidation,runtimeClasspath,runtimeLibs,solrPlatformLibs,testCompileClasspath,testRuntimeClasspath
+io.netty:netty-resolver:4.2.15.Final=apiHelper,compileClasspath,jarValidation,runtimeClasspath,runtimeLibs,solrPlatformLibs,testCompileClasspath,testRuntimeClasspath
+io.netty:netty-tcnative-boringssl-static:2.0.75.Final=apiHelper,compileClasspath,jarValidation,runtimeClasspath,runtimeLibs,solrPlatformLibs,testCompileClasspath,testRuntimeClasspath
+io.netty:netty-tcnative-classes:2.0.75.Final=apiHelper,compileClasspath,jarValidation,runtimeClasspath,runtimeLibs,solrPlatformLibs,testCompileClasspath,testRuntimeClasspath
+io.netty:netty-transport-classes-epoll:4.2.15.Final=apiHelper,compileClasspath,jarValidation,runtimeClasspath,runtimeLibs,solrPlatformLibs,testCompileClasspath,testRuntimeClasspath
+io.netty:netty-transport-native-epoll:4.2.15.Final=apiHelper,compileClasspath,jarValidation,runtimeClasspath,runtimeLibs,solrPlatformLibs,testCompileClasspath,testRuntimeClasspath
+io.netty:netty-transport-native-unix-common:4.2.15.Final=apiHelper,compileClasspath,jarValidation,runtimeClasspath,runtimeLibs,solrPlatformLibs,testCompileClasspath,testRuntimeClasspath
+io.netty:netty-transport:4.2.15.Final=apiHelper,compileClasspath,jarValidation,runtimeClasspath,runtimeLibs,solrPlatformLibs,testCompileClasspath,testRuntimeClasspath
+io.opentelemetry.instrumentation:opentelemetry-instrumentation-api-incubator:2.22.0-alpha=apiHelper,jarValidation,runtimeClasspath,runtimeLibs,solrPlatformLibs,testRuntimeClasspath
+io.opentelemetry.instrumentation:opentelemetry-instrumentation-api:2.22.0=apiHelper,jarValidation,runtimeClasspath,runtimeLibs,solrPlatformLibs,testRuntimeClasspath
+io.opentelemetry.instrumentation:opentelemetry-runtime-telemetry-java17:2.22.0-alpha=apiHelper,jarValidation,runtimeClasspath,runtimeLibs,solrPlatformLibs,testRuntimeClasspath
+io.opentelemetry.instrumentation:opentelemetry-runtime-telemetry-java8:2.22.0-alpha=apiHelper,jarValidation,runtimeClasspath,runtimeLibs,solrPlatformLibs,testRuntimeClasspath
+io.opentelemetry.semconv:opentelemetry-semconv:1.37.0=apiHelper,jarValidation,runtimeClasspath,runtimeLibs,solrPlatformLibs,testRuntimeClasspath
+io.opentelemetry:opentelemetry-api-incubator:1.56.0-alpha=apiHelper,jarValidation,runtimeClasspath,runtimeLibs,solrPlatformLibs,testRuntimeClasspath
+io.opentelemetry:opentelemetry-api:1.56.0=apiHelper,compileClasspath,jarValidation,runtimeClasspath,runtimeLibs,solrPlatformLibs,testCompileClasspath,testRuntimeClasspath
+io.opentelemetry:opentelemetry-common:1.56.0=apiHelper,compileClasspath,jarValidation,runtimeClasspath,runtimeLibs,solrPlatformLibs,testCompileClasspath,testRuntimeClasspath
+io.opentelemetry:opentelemetry-context:1.56.0=apiHelper,compileClasspath,jarValidation,runtimeClasspath,runtimeLibs,solrPlatformLibs,testCompileClasspath,testRuntimeClasspath
+io.opentelemetry:opentelemetry-exporter-prometheus:1.56.0-alpha=apiHelper,jarValidation,runtimeClasspath,runtimeLibs,solrPlatformLibs,testRuntimeClasspath
+io.opentelemetry:opentelemetry-sdk-common:1.56.0=apiHelper,jarValidation,runtimeClasspath,runtimeLibs,solrPlatformLibs,testRuntimeClasspath
+io.opentelemetry:opentelemetry-sdk-metrics:1.56.0=apiHelper,jarValidation,runtimeClasspath,runtimeLibs,solrPlatformLibs,testRuntimeClasspath
+io.opentelemetry:opentelemetry-sdk-trace:1.56.0=apiHelper,jarValidation,runtimeClasspath,runtimeLibs,solrPlatformLibs,testRuntimeClasspath
+io.opentelemetry:opentelemetry-sdk:1.56.0=apiHelper,jarValidation,runtimeClasspath,runtimeLibs,solrPlatformLibs,testRuntimeClasspath
+io.projectreactor:reactor-core:3.7.14=compileClasspath,jarValidation,runtimeClasspath,runtimeLibs,testCompileClasspath,testRuntimeClasspath
+io.prometheus:prometheus-metrics-exposition-formats:1.1.0=apiHelper,jarValidation,runtimeClasspath,runtimeLibs,solrPlatformLibs,testRuntimeClasspath
+io.prometheus:prometheus-metrics-model:1.1.0=apiHelper,jarValidation,runtimeClasspath,runtimeLibs,solrPlatformLibs,testRuntimeClasspath
+io.sgr:s2-geometry-library-java:1.0.0=apiHelper,jarValidation,runtimeClasspath,runtimeLibs,solrPlatformLibs,testRuntimeClasspath
+io.swagger.core.v3:swagger-annotations-jakarta:2.2.22=apiHelper,compileClasspath,jarValidation,runtimeClasspath,runtimeLibs,solrPlatformLibs,testCompileClasspath,testRuntimeClasspath
+jakarta.activation:jakarta.activation-api:2.1.3=apiHelper,jarValidation,runtimeClasspath,runtimeLibs,solrPlatformLibs,testRuntimeClasspath
+jakarta.annotation:jakarta.annotation-api:3.0.0=apiHelper,jarValidation,runtimeClasspath,runtimeLibs,solrPlatformLibs,testRuntimeClasspath
+jakarta.inject:jakarta.inject-api:2.0.1=apiHelper,jarValidation,runtimeClasspath,runtimeLibs,solrPlatformLibs,testRuntimeClasspath
+jakarta.servlet:jakarta.servlet-api:6.1.0=apiHelper,jarValidation,runtimeClasspath,runtimeLibs,solrPlatformLibs,testRuntimeClasspath
+jakarta.validation:jakarta.validation-api:3.1.0=apiHelper,jarValidation,runtimeClasspath,runtimeLibs,solrPlatformLibs,testRuntimeClasspath
+jakarta.ws.rs:jakarta.ws.rs-api:4.0.0=apiHelper,jarValidation,runtimeClasspath,runtimeLibs,solrPlatformLibs,testRuntimeClasspath
+jakarta.xml.bind:jakarta.xml.bind-api:4.0.2=apiHelper,jarValidation,runtimeClasspath,runtimeLibs,solrPlatformLibs,testRuntimeClasspath
+javax.inject:javax.inject:1=annotationProcessor,errorprone,testAnnotationProcessor
+junit:junit:4.13.2=jarValidation,testCompileClasspath,testRuntimeClasspath
+net.java.dev.jna:jna-platform:5.17.0=compileClasspath,jarValidation,runtimeClasspath,runtimeLibs,testCompileClasspath,testRuntimeClasspath
+net.java.dev.jna:jna:5.18.1=compileClasspath,jarValidation,runtimeClasspath,runtimeLibs,testCompileClasspath,testRuntimeClasspath
Review Comment:
jna-platform (5.17.0) was being pulled in transitively by azure-identity and didn't match the jna 5.18.1 already declared on main. Rather than pin a second version in the toml, I excluded jna-platform (and msal4j-persistence-extension) from azure-identity entirely.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-17949: Add Azure Blob Storage backup repository module [solr]
Posted by "prateeksinghalgit (via GitHub)" <gi...@apache.org>.
prateeksinghalgit commented on code in PR #3750:
URL: https://github.com/apache/solr/pull/3750#discussion_r3469426912
##########
solr/modules/azure-blob-repository/build.gradle:
##########
@@ -0,0 +1,74 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+apply plugin: 'java-library'
+
+description = 'Azure Blob Storage Repository'
+
+ext {
+ // Disable security manager for azure-blob-repository module tests
+ // Required because Testcontainers needs access to Docker socket and system properties
+ useSecurityManager = false
+}
+
+dependencies {
+ implementation platform(project(':platform'))
+ api(project(':solr:core'))
+ implementation project(':solr:solrj')
+
+ implementation libs.apache.lucene.core
+
+ implementation platform(libs.azure.sdk.bom)
+
+ // Azure Storage SDK dependencies
+ implementation(libs.azure.storage.blob) {
+ exclude group: 'com.azure', module: 'azure-core-http-netty'
+ }
+ implementation(libs.azure.storage.blob.batch) {
+ exclude group: 'com.azure', module: 'azure-core-http-netty'
+ }
+ implementation(libs.azure.identity) {
+ exclude group: 'com.azure', module: 'azure-core-http-netty'
+ }
Review Comment:
agreed. I've excluded msal4j-persistence-extension. Also excluded jna-platform from libs.azure.identity. Dropping them removes three jars now. I have deleted their license files as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org