You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@solr.apache.org by "risdenk (via GitHub)" <gi...@apache.org> on 2023/04/05 13:04:50 UTC
[GitHub] [solr] risdenk opened a new pull request, #1545: Update dependencies com.adobe.testing:s3mock-junit4 to v2.11.0, org.springframework.boot:spring-boot to 2.7.10, org.springframework:spring to 5.3.26, and software.amazon.awssdk to 2.19.1
risdenk opened a new pull request, #1545:
URL: https://github.com/apache/solr/pull/1545
This is take 3 after the first two attempts caused test failures:
* https://github.com/apache/solr/pull/1517 (permissions issues and directory paths trailing slash)
* https://github.com/apache/solr/pull/1535 (read timed out on slower Jenkins machines)
This reverts commit b6b1d18ebfbebaf017de40c8aff419078696553d.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] risdenk commented on a diff in pull request #1545: Update dependencies com.adobe.testing:s3mock-junit4 to v2.12.2 and org.springframework.boot:spring-boot to 2.7.12
Posted by "risdenk (via GitHub)" <gi...@apache.org>.
risdenk commented on code in PR #1545:
URL: https://github.com/apache/solr/pull/1545#discussion_r1223146361
##########
versions.props:
##########
@@ -71,7 +71,7 @@ org.osgi:osgi.annotation=8.1.0
org.quicktheories:quicktheories=0.26
org.semver4j:semver4j=4.3.0
org.slf4j:*=2.0.7
-org.springframework.boot:spring-boot*=2.5.14
-org.springframework:spring*=5.3.23
+org.springframework.boot:spring-boot*=2.7.10
+org.springframework:spring*=5.3.26
Review Comment:
Its not even worth getting the question about it. When s3mock upgrades spring-boot to whatever newer version then we can remove the constraint.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] HoustonPutman commented on a diff in pull request #1545: Update dependencies com.adobe.testing:s3mock-junit4 to v2.12.2 and org.springframework.boot:spring-boot to 2.7.12
Posted by "HoustonPutman (via GitHub)" <gi...@apache.org>.
HoustonPutman commented on code in PR #1545:
URL: https://github.com/apache/solr/pull/1545#discussion_r1223532737
##########
solr/core/src/java/org/apache/solr/core/backup/repository/BackupRepository.java:
##########
@@ -228,9 +228,20 @@ default void copyIndexFileFrom(
* Delete {@code files} at {@code path}
*
* @since 8.3.0
+ * @deprecated Use {@link #delete(URI, Collection)} instead
Review Comment:
Ok this looks good to me, if all tests pass. (and we can confirm nothing internally uses the deprecated method)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] HoustonPutman commented on pull request #1545: Update dependencies com.adobe.testing:s3mock-junit4 to v2.12.2 and org.springframework.boot:spring-boot to 2.7.12
Posted by "HoustonPutman (via GitHub)" <gi...@apache.org>.
HoustonPutman commented on PR #1545:
URL: https://github.com/apache/solr/pull/1545#issuecomment-1582877779
We do not care about that error, and don't even check for it. Should be easy enough just to change the test... Let me do that real quick
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] risdenk commented on pull request #1545: Update dependencies com.adobe.testing:s3mock-junit4 to v2.11.0, org.springframework.boot:spring-boot to 2.7.10, org.springframework:spring to 5.3.26, and software.amazon.awssdk to 2.19.1
Posted by "risdenk (via GitHub)" <gi...@apache.org>.
risdenk commented on PR #1545:
URL: https://github.com/apache/solr/pull/1545#issuecomment-1497490149
The default read timeout is 30s - https://github.com/aws/aws-sdk-java-v2/blob/master/http-client-spi/src/main/java/software/amazon/awssdk/http/SdkHttpConfigurationOption.java#L134 which seems like it should be plenty...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] risdenk commented on a diff in pull request #1545: Update dependencies com.adobe.testing:s3mock-junit4 to v2.12.2 and org.springframework.boot:spring-boot to 2.7.12
Posted by "risdenk (via GitHub)" <gi...@apache.org>.
risdenk commented on code in PR #1545:
URL: https://github.com/apache/solr/pull/1545#discussion_r1224247241
##########
solr/core/src/java/org/apache/solr/core/backup/repository/BackupRepository.java:
##########
@@ -228,9 +228,20 @@ default void copyIndexFileFrom(
* Delete {@code files} at {@code path}
*
* @since 8.3.0
+ * @deprecated Use {@link #delete(URI, Collection)} instead
Review Comment:
Yup reran all tests a few times and confirmed not using the deprecated method.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] risdenk commented on pull request #1545: Update dependencies com.adobe.testing:s3mock-junit4 to v2.11.0, org.springframework.boot:spring-boot to 2.7.10, org.springframework:spring to 5.3.26, and software.amazon.awssdk to 2.19.1
Posted by "risdenk (via GitHub)" <gi...@apache.org>.
risdenk commented on PR #1545:
URL: https://github.com/apache/solr/pull/1545#issuecomment-1497838599
So found the issue I think (I don't know how it manifests itself more during the upgrade, but this is a big improvement):
AwsChunkedDecodingInputStream (https://github.com/adobe/S3Mock/blame/main/server/src/main/java/com/adobe/testing/s3mock/util/AwsChunkedDecodingInputStream.java) is used during putObject and only implements read() which is a single byte read at a time.
A quick fix is to wrap the input source in new BufferedInputStream which then speeds everything up. It took the test on my laptop from 11s to <1s easily.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] risdenk commented on a diff in pull request #1545: Update dependencies com.adobe.testing:s3mock-junit4 to v2.11.0, org.springframework.boot:spring-boot to 2.7.10, org.springframework:spring to 5.3.26, and software.amazon.awssdk to 2.19.1
Posted by "risdenk (via GitHub)" <gi...@apache.org>.
risdenk commented on code in PR #1545:
URL: https://github.com/apache/solr/pull/1545#discussion_r1158815571
##########
solr/modules/s3-repository/src/test/com/adobe/testing/s3mock/util/AwsChunkedDecodingInputStream.java:
##########
@@ -0,0 +1,144 @@
+/*
+ * Copyright 2017-2022 Adobe.
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package com.adobe.testing.s3mock.util;
+
+import java.io.BufferedInputStream;
+import java.io.IOException;
+import java.io.InputStream;
+import java.nio.ByteBuffer;
+import java.nio.charset.StandardCharsets;
+
+/**
+ * Skips V4 style signing metadata from input streams.
+ * <p>The original stream looks like this (newlines are CRLF):</p>
+ *
+ * <pre>
+ * 5;chunk-signature=7ece820edcf094ce1ef6d643c8db60b67913e28831d9b0430efd2b56a9deec5e
+ * 12345
+ * 0;chunk-signature=ee2c094d7162170fcac17d2c76073cd834b0488bfe52e89e48599b8115c7ffa2
+ * </pre>
+ *
+ * <p>The format of each chunk of data is:</p>
+ *
+ * <pre>
+ * [hex-encoded-number-of-bytes-in-chunk];chunk-signature=[sha256-signature][crlf]
+ * [payload-bytes-of-this-chunk][crlf]
+ * </pre>
+ *
+ * @see
+ * <a href="http://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/auth/AwsChunkedEncodingInputStream.html">
+ * AwsChunkedEncodingInputStream</a>
+ */
+public class AwsChunkedDecodingInputStream extends InputStream {
+
+ /**
+ * That's the max chunk buffer size used in the AWS implementation.
+ */
+ private static final int MAX_CHUNK_SIZE = 256 * 1024;
+
+ private static final byte[] CRLF = "\r\n".getBytes(StandardCharsets.UTF_8);
+
+ private static final byte[] DELIMITER = ";".getBytes(StandardCharsets.UTF_8);
+
+ private final InputStream source;
+
+ private int remainingInChunk = 0;
+
+ private final ByteBuffer byteBuffer = ByteBuffer.allocate(MAX_CHUNK_SIZE);
+
+ /**
+ * Constructs a new {@link AwsChunkedDecodingInputStream}.
+ *
+ * @param source The {@link InputStream} to wrap.
+ */
+ public AwsChunkedDecodingInputStream(final InputStream source) {
+ // Remove this class after TODO open issue with s3mock
+ // Buffer the source InputStream since this class only implements read() so
+ // pass off the actual buffering to the BufferedInputStream to read bigger
+ // chunks at once. This avoids a lot of single byte reads.
+ this.source = new BufferedInputStream(source);
Review Comment:
Opened https://github.com/adobe/S3Mock/issues/1115 and added to the comment
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] risdenk commented on pull request #1545: Update dependencies com.adobe.testing:s3mock-junit4 to v2.11.0, org.springframework.boot:spring-boot to 2.7.10, org.springframework:spring to 5.3.26, and software.amazon.awssdk to 2.19.1
Posted by "risdenk (via GitHub)" <gi...@apache.org>.
risdenk commented on PR #1545:
URL: https://github.com/apache/solr/pull/1545#issuecomment-1497454035
FYI @gerlowskija and @HoustonPutman - no new changes yet just put this out there to make sure it doesn't get lost.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] janhoy commented on a diff in pull request #1545: Update dependencies com.adobe.testing:s3mock-junit4 to v2.12.2 and org.springframework.boot:spring-boot to 2.7.12
Posted by "janhoy (via GitHub)" <gi...@apache.org>.
janhoy commented on code in PR #1545:
URL: https://github.com/apache/solr/pull/1545#discussion_r1223143767
##########
versions.props:
##########
@@ -71,7 +71,7 @@ org.osgi:osgi.annotation=8.1.0
org.quicktheories:quicktheories=0.26
org.semver4j:semver4j=4.3.0
org.slf4j:*=2.0.7
-org.springframework.boot:spring-boot*=2.5.14
-org.springframework:spring*=5.3.23
+org.springframework.boot:spring-boot*=2.7.10
+org.springframework:spring*=5.3.26
Review Comment:
We don't ship spring-boot with Solr. Are the CVEs of such severity that you believe devs or CI running tests are at risk?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] risdenk commented on a diff in pull request #1545: Update dependencies com.adobe.testing:s3mock-junit4 to v2.11.0, org.springframework.boot:spring-boot to 2.7.10, org.springframework:spring to 5.3.26, and software.amazon.awssdk to 2.19.1
Posted by "risdenk (via GitHub)" <gi...@apache.org>.
risdenk commented on code in PR #1545:
URL: https://github.com/apache/solr/pull/1545#discussion_r1158796523
##########
solr/modules/s3-repository/src/test/com/adobe/testing/s3mock/util/AwsChunkedDecodingInputStream.java:
##########
@@ -0,0 +1,144 @@
+/*
+ * Copyright 2017-2022 Adobe.
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package com.adobe.testing.s3mock.util;
+
+import java.io.BufferedInputStream;
+import java.io.IOException;
+import java.io.InputStream;
+import java.nio.ByteBuffer;
+import java.nio.charset.StandardCharsets;
+
+/**
+ * Skips V4 style signing metadata from input streams.
+ * <p>The original stream looks like this (newlines are CRLF):</p>
+ *
+ * <pre>
+ * 5;chunk-signature=7ece820edcf094ce1ef6d643c8db60b67913e28831d9b0430efd2b56a9deec5e
+ * 12345
+ * 0;chunk-signature=ee2c094d7162170fcac17d2c76073cd834b0488bfe52e89e48599b8115c7ffa2
+ * </pre>
+ *
+ * <p>The format of each chunk of data is:</p>
+ *
+ * <pre>
+ * [hex-encoded-number-of-bytes-in-chunk];chunk-signature=[sha256-signature][crlf]
+ * [payload-bytes-of-this-chunk][crlf]
+ * </pre>
+ *
+ * @see
+ * <a href="http://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/auth/AwsChunkedEncodingInputStream.html">
+ * AwsChunkedEncodingInputStream</a>
+ */
+public class AwsChunkedDecodingInputStream extends InputStream {
+
+ /**
+ * That's the max chunk buffer size used in the AWS implementation.
+ */
+ private static final int MAX_CHUNK_SIZE = 256 * 1024;
+
+ private static final byte[] CRLF = "\r\n".getBytes(StandardCharsets.UTF_8);
+
+ private static final byte[] DELIMITER = ";".getBytes(StandardCharsets.UTF_8);
+
+ private final InputStream source;
+
+ private int remainingInChunk = 0;
+
+ private final ByteBuffer byteBuffer = ByteBuffer.allocate(MAX_CHUNK_SIZE);
+
+ /**
+ * Constructs a new {@link AwsChunkedDecodingInputStream}.
+ *
+ * @param source The {@link InputStream} to wrap.
+ */
+ public AwsChunkedDecodingInputStream(final InputStream source) {
+ // Remove this class after TODO open issue with s3mock
+ // Buffer the source InputStream since this class only implements read() so
+ // pass off the actual buffering to the BufferedInputStream to read bigger
+ // chunks at once. This avoids a lot of single byte reads.
+ this.source = new BufferedInputStream(source);
Review Comment:
This is the only change here - replace `this.source = source` with `this.source = new BufferedInputStream(source);` which improves the repeated reads.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] risdenk commented on pull request #1545: Update dependencies com.adobe.testing:s3mock-junit4 to v2.12.2 and org.springframework.boot:spring-boot to 2.7.12
Posted by "risdenk (via GitHub)" <gi...@apache.org>.
risdenk commented on PR #1545:
URL: https://github.com/apache/solr/pull/1545#issuecomment-1582740818
Currently have these two reproducible tests failing now:
```
ERROR: The following test(s) have failed:
- org.apache.solr.s3.S3BackupRepositoryTest.testCanDeleteIndividualFiles (:solr:modules:s3-repository)
Test output: /Users/risdenk/repos/apache/solr/solr/modules/s3-repository/build/test-results/test/outputs/OUTPUT-org.apache.solr.s3.S3BackupRepositoryTest.txt
Reproduce with: gradlew :solr:modules:s3-repository:test --tests "org.apache.solr.s3.S3BackupRepositoryTest.testCanDeleteIndividualFiles" -Ptests.jvms=5 "-Ptests.jvmargs=-XX:TieredStopAtLevel=1 -XX:+UseParallelGC -XX:ActiveProcessorCount=1 -XX:ReservedCodeCacheSize=120m" -Ptests.seed=5164E4ACC462FC43 -Ptests.file.encoding=ISO-8859-1
- org.apache.solr.s3.S3ReadWriteTest.testNotFound (:solr:modules:s3-repository)
Test output: /Users/risdenk/repos/apache/solr/solr/modules/s3-repository/build/test-results/test/outputs/OUTPUT-org.apache.solr.s3.S3ReadWriteTest.txt
Reproduce with: gradlew :solr:modules:s3-repository:test --tests "org.apache.solr.s3.S3ReadWriteTest.testNotFound" -Ptests.jvms=5 "-Ptests.jvmargs=-XX:TieredStopAtLevel=1 -XX:+UseParallelGC -XX:ActiveProcessorCount=1 -XX:ReservedCodeCacheSize=120m" -Ptests.seed=5164E4ACC462FC43 -Ptests.file.encoding=ISO-8859-1
```
haven't looked into it yet
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] risdenk commented on pull request #1545: Update dependencies com.adobe.testing:s3mock-junit4 to v2.12.2 and org.springframework.boot:spring-boot to 2.7.12
Posted by "risdenk (via GitHub)" <gi...@apache.org>.
risdenk commented on PR #1545:
URL: https://github.com/apache/solr/pull/1545#issuecomment-1582863918
Ah I found it in https://github.com/adobe/S3Mock/compare/2.11.0...2.12.1#diff-6d8eaed59cd5db175eb9be1dcd85a7c0481414eb19a5e12de2caf678c8a11314
s3mock is now matching what AWS actually does for deletes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] gerlowskija commented on a diff in pull request #1545: Update dependencies com.adobe.testing:s3mock-junit4 to v2.11.0, org.springframework.boot:spring-boot to 2.7.10, org.springframework:spring to 5.3.26, and software.amazon.awssdk to 2.19.1
Posted by "gerlowskija (via GitHub)" <gi...@apache.org>.
gerlowskija commented on code in PR #1545:
URL: https://github.com/apache/solr/pull/1545#discussion_r1187541020
##########
solr/modules/s3-repository/src/test/com/adobe/testing/s3mock/util/AwsChunkedDecodingInputStream.java:
##########
@@ -0,0 +1,144 @@
+/*
+ * Copyright 2017-2022 Adobe.
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package com.adobe.testing.s3mock.util;
+
+import java.io.BufferedInputStream;
+import java.io.IOException;
+import java.io.InputStream;
+import java.nio.ByteBuffer;
+import java.nio.charset.StandardCharsets;
+
+/**
+ * Skips V4 style signing metadata from input streams.
+ * <p>The original stream looks like this (newlines are CRLF):</p>
+ *
+ * <pre>
+ * 5;chunk-signature=7ece820edcf094ce1ef6d643c8db60b67913e28831d9b0430efd2b56a9deec5e
+ * 12345
+ * 0;chunk-signature=ee2c094d7162170fcac17d2c76073cd834b0488bfe52e89e48599b8115c7ffa2
+ * </pre>
+ *
+ * <p>The format of each chunk of data is:</p>
+ *
+ * <pre>
+ * [hex-encoded-number-of-bytes-in-chunk];chunk-signature=[sha256-signature][crlf]
+ * [payload-bytes-of-this-chunk][crlf]
+ * </pre>
+ *
+ * @see
+ * <a href="http://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/auth/AwsChunkedEncodingInputStream.html">
+ * AwsChunkedEncodingInputStream</a>
+ */
+public class AwsChunkedDecodingInputStream extends InputStream {
+
+ /**
+ * That's the max chunk buffer size used in the AWS implementation.
+ */
+ private static final int MAX_CHUNK_SIZE = 256 * 1024;
+
+ private static final byte[] CRLF = "\r\n".getBytes(StandardCharsets.UTF_8);
+
+ private static final byte[] DELIMITER = ";".getBytes(StandardCharsets.UTF_8);
+
+ private final InputStream source;
+
+ private int remainingInChunk = 0;
+
+ private final ByteBuffer byteBuffer = ByteBuffer.allocate(MAX_CHUNK_SIZE);
+
+ /**
+ * Constructs a new {@link AwsChunkedDecodingInputStream}.
+ *
+ * @param source The {@link InputStream} to wrap.
+ */
+ public AwsChunkedDecodingInputStream(final InputStream source) {
+ // Remove this class after TODO open issue with s3mock
+ // Buffer the source InputStream since this class only implements read() so
+ // pass off the actual buffering to the BufferedInputStream to read bigger
+ // chunks at once. This avoids a lot of single byte reads.
+ this.source = new BufferedInputStream(source);
Review Comment:
Wanted to check in on this a month on, see where we stand. Kevin's s3mock PR is still outstanding. OTOH though, there's still a few S3 flaky failures, but they haven't been too noticeable IMO. I'll set a reminder for another month out and we can check back in on s3mock's release progress and our flaky rate to see whether it might be worth copying the files in.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] risdenk commented on a diff in pull request #1545: Update dependencies com.adobe.testing:s3mock-junit4 to v2.11.0, org.springframework.boot:spring-boot to 2.7.10, org.springframework:spring to 5.3.26, and software.amazon.awssdk to 2.19.1
Posted by "risdenk (via GitHub)" <gi...@apache.org>.
risdenk commented on code in PR #1545:
URL: https://github.com/apache/solr/pull/1545#discussion_r1223100231
##########
solr/modules/s3-repository/src/test/com/adobe/testing/s3mock/util/AwsChunkedDecodingInputStream.java:
##########
@@ -0,0 +1,144 @@
+/*
+ * Copyright 2017-2022 Adobe.
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package com.adobe.testing.s3mock.util;
+
+import java.io.BufferedInputStream;
+import java.io.IOException;
+import java.io.InputStream;
+import java.nio.ByteBuffer;
+import java.nio.charset.StandardCharsets;
+
+/**
+ * Skips V4 style signing metadata from input streams.
+ * <p>The original stream looks like this (newlines are CRLF):</p>
+ *
+ * <pre>
+ * 5;chunk-signature=7ece820edcf094ce1ef6d643c8db60b67913e28831d9b0430efd2b56a9deec5e
+ * 12345
+ * 0;chunk-signature=ee2c094d7162170fcac17d2c76073cd834b0488bfe52e89e48599b8115c7ffa2
+ * </pre>
+ *
+ * <p>The format of each chunk of data is:</p>
+ *
+ * <pre>
+ * [hex-encoded-number-of-bytes-in-chunk];chunk-signature=[sha256-signature][crlf]
+ * [payload-bytes-of-this-chunk][crlf]
+ * </pre>
+ *
+ * @see
+ * <a href="http://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/auth/AwsChunkedEncodingInputStream.html">
+ * AwsChunkedEncodingInputStream</a>
+ */
+public class AwsChunkedDecodingInputStream extends InputStream {
+
+ /**
+ * That's the max chunk buffer size used in the AWS implementation.
+ */
+ private static final int MAX_CHUNK_SIZE = 256 * 1024;
+
+ private static final byte[] CRLF = "\r\n".getBytes(StandardCharsets.UTF_8);
+
+ private static final byte[] DELIMITER = ";".getBytes(StandardCharsets.UTF_8);
+
+ private final InputStream source;
+
+ private int remainingInChunk = 0;
+
+ private final ByteBuffer byteBuffer = ByteBuffer.allocate(MAX_CHUNK_SIZE);
+
+ /**
+ * Constructs a new {@link AwsChunkedDecodingInputStream}.
+ *
+ * @param source The {@link InputStream} to wrap.
+ */
+ public AwsChunkedDecodingInputStream(final InputStream source) {
+ // Remove this class after TODO open issue with s3mock
+ // Buffer the source InputStream since this class only implements read() so
+ // pass off the actual buffering to the BufferedInputStream to read bigger
+ // chunks at once. This avoids a lot of single byte reads.
+ this.source = new BufferedInputStream(source);
Review Comment:
https://github.com/adobe/S3Mock/issues/1115#issuecomment-1582037419
2.12.2 was just released with this fix - will update PR soon
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] risdenk commented on a diff in pull request #1545: Update dependencies com.adobe.testing:s3mock-junit4 to v2.11.0, org.springframework.boot:spring-boot to 2.7.10, org.springframework:spring to 5.3.26, and software.amazon.awssdk to 2.19.1
Posted by "risdenk (via GitHub)" <gi...@apache.org>.
risdenk commented on code in PR #1545:
URL: https://github.com/apache/solr/pull/1545#discussion_r1160710088
##########
solr/modules/s3-repository/src/test/com/adobe/testing/s3mock/util/AwsChunkedDecodingInputStream.java:
##########
@@ -0,0 +1,144 @@
+/*
+ * Copyright 2017-2022 Adobe.
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package com.adobe.testing.s3mock.util;
+
+import java.io.BufferedInputStream;
+import java.io.IOException;
+import java.io.InputStream;
+import java.nio.ByteBuffer;
+import java.nio.charset.StandardCharsets;
+
+/**
+ * Skips V4 style signing metadata from input streams.
+ * <p>The original stream looks like this (newlines are CRLF):</p>
+ *
+ * <pre>
+ * 5;chunk-signature=7ece820edcf094ce1ef6d643c8db60b67913e28831d9b0430efd2b56a9deec5e
+ * 12345
+ * 0;chunk-signature=ee2c094d7162170fcac17d2c76073cd834b0488bfe52e89e48599b8115c7ffa2
+ * </pre>
+ *
+ * <p>The format of each chunk of data is:</p>
+ *
+ * <pre>
+ * [hex-encoded-number-of-bytes-in-chunk];chunk-signature=[sha256-signature][crlf]
+ * [payload-bytes-of-this-chunk][crlf]
+ * </pre>
+ *
+ * @see
+ * <a href="http://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/auth/AwsChunkedEncodingInputStream.html">
+ * AwsChunkedEncodingInputStream</a>
+ */
+public class AwsChunkedDecodingInputStream extends InputStream {
+
+ /**
+ * That's the max chunk buffer size used in the AWS implementation.
+ */
+ private static final int MAX_CHUNK_SIZE = 256 * 1024;
+
+ private static final byte[] CRLF = "\r\n".getBytes(StandardCharsets.UTF_8);
+
+ private static final byte[] DELIMITER = ";".getBytes(StandardCharsets.UTF_8);
+
+ private final InputStream source;
+
+ private int remainingInChunk = 0;
+
+ private final ByteBuffer byteBuffer = ByteBuffer.allocate(MAX_CHUNK_SIZE);
+
+ /**
+ * Constructs a new {@link AwsChunkedDecodingInputStream}.
+ *
+ * @param source The {@link InputStream} to wrap.
+ */
+ public AwsChunkedDecodingInputStream(final InputStream source) {
+ // Remove this class after TODO open issue with s3mock
+ // Buffer the source InputStream since this class only implements read() so
+ // pass off the actual buffering to the BufferedInputStream to read bigger
+ // chunks at once. This avoids a lot of single byte reads.
+ this.source = new BufferedInputStream(source);
Review Comment:
There is a PR (https://github.com/adobe/S3Mock/pull/1120 ) and a comment on my issue from the maintainers about the fix being possible just they are having some issues w/ release so hopefully thats resolved soon.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] risdenk commented on pull request #1545: Update dependencies com.adobe.testing:s3mock-junit4 to v2.12.2 and org.springframework.boot:spring-boot to 2.7.12
Posted by "risdenk (via GitHub)" <gi...@apache.org>.
risdenk commented on PR #1545:
URL: https://github.com/apache/solr/pull/1545#issuecomment-1582937503
@HoustonPutman I pushed two commits:
* [4f25651](https://github.com/apache/solr/pull/1545/commits/4f25651dda72ce6f116b92052483a92f2ec89fa9) - don't test for the case we don't care about
* [4f475a4](https://github.com/apache/solr/pull/1545/commits/4f475a466305470dce39c6b2ff4126481794d295) - remove `ignoreNoSuchFileException` from API since it was always called w/ `true` in prod code
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] risdenk commented on a diff in pull request #1545: Update dependencies com.adobe.testing:s3mock-junit4 to v2.12.2 and org.springframework.boot:spring-boot to 2.7.12
Posted by "risdenk (via GitHub)" <gi...@apache.org>.
risdenk commented on code in PR #1545:
URL: https://github.com/apache/solr/pull/1545#discussion_r1223433923
##########
solr/core/src/java/org/apache/solr/core/backup/repository/BackupRepository.java:
##########
@@ -228,9 +228,20 @@ default void copyIndexFileFrom(
* Delete {@code files} at {@code path}
*
* @since 8.3.0
+ * @deprecated Use {@link #delete(URI, Collection)} instead
Review Comment:
New method is a default method on the interface:
```
default void delete(URI path, Collection<String> files) throws IOException
```
so there shouldn't be an issue with custom repositories.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] janhoy commented on a diff in pull request #1545: Update dependencies com.adobe.testing:s3mock-junit4 to v2.11.0, org.springframework.boot:spring-boot to 2.7.10, org.springframework:spring to 5.3.26, and software.amazon.awssdk to 2.19.1
Posted by "janhoy (via GitHub)" <gi...@apache.org>.
janhoy commented on code in PR #1545:
URL: https://github.com/apache/solr/pull/1545#discussion_r1213339840
##########
versions.props:
##########
@@ -71,7 +71,7 @@ org.osgi:osgi.annotation=8.1.0
org.quicktheories:quicktheories=0.26
org.semver4j:semver4j=4.3.0
org.slf4j:*=2.0.7
-org.springframework.boot:spring-boot*=2.5.14
-org.springframework:spring*=5.3.23
+org.springframework.boot:spring-boot*=2.7.10
+org.springframework:spring*=5.3.26
Review Comment:
We can now delete the two spring lines here and let s3mock pull them in transitively.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] risdenk merged pull request #1545: Update dependencies com.adobe.testing:s3mock-junit4 to v2.12.2 and org.springframework.boot:spring-boot to 2.7.12
Posted by "risdenk (via GitHub)" <gi...@apache.org>.
risdenk merged PR #1545:
URL: https://github.com/apache/solr/pull/1545
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] gerlowskija commented on a diff in pull request #1545: Update dependencies com.adobe.testing:s3mock-junit4 to v2.11.0, org.springframework.boot:spring-boot to 2.7.10, org.springframework:spring to 5.3.26, and software.amazon.awssdk to 2.19.1
Posted by "gerlowskija (via GitHub)" <gi...@apache.org>.
gerlowskija commented on code in PR #1545:
URL: https://github.com/apache/solr/pull/1545#discussion_r1187541020
##########
solr/modules/s3-repository/src/test/com/adobe/testing/s3mock/util/AwsChunkedDecodingInputStream.java:
##########
@@ -0,0 +1,144 @@
+/*
+ * Copyright 2017-2022 Adobe.
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package com.adobe.testing.s3mock.util;
+
+import java.io.BufferedInputStream;
+import java.io.IOException;
+import java.io.InputStream;
+import java.nio.ByteBuffer;
+import java.nio.charset.StandardCharsets;
+
+/**
+ * Skips V4 style signing metadata from input streams.
+ * <p>The original stream looks like this (newlines are CRLF):</p>
+ *
+ * <pre>
+ * 5;chunk-signature=7ece820edcf094ce1ef6d643c8db60b67913e28831d9b0430efd2b56a9deec5e
+ * 12345
+ * 0;chunk-signature=ee2c094d7162170fcac17d2c76073cd834b0488bfe52e89e48599b8115c7ffa2
+ * </pre>
+ *
+ * <p>The format of each chunk of data is:</p>
+ *
+ * <pre>
+ * [hex-encoded-number-of-bytes-in-chunk];chunk-signature=[sha256-signature][crlf]
+ * [payload-bytes-of-this-chunk][crlf]
+ * </pre>
+ *
+ * @see
+ * <a href="http://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/auth/AwsChunkedEncodingInputStream.html">
+ * AwsChunkedEncodingInputStream</a>
+ */
+public class AwsChunkedDecodingInputStream extends InputStream {
+
+ /**
+ * That's the max chunk buffer size used in the AWS implementation.
+ */
+ private static final int MAX_CHUNK_SIZE = 256 * 1024;
+
+ private static final byte[] CRLF = "\r\n".getBytes(StandardCharsets.UTF_8);
+
+ private static final byte[] DELIMITER = ";".getBytes(StandardCharsets.UTF_8);
+
+ private final InputStream source;
+
+ private int remainingInChunk = 0;
+
+ private final ByteBuffer byteBuffer = ByteBuffer.allocate(MAX_CHUNK_SIZE);
+
+ /**
+ * Constructs a new {@link AwsChunkedDecodingInputStream}.
+ *
+ * @param source The {@link InputStream} to wrap.
+ */
+ public AwsChunkedDecodingInputStream(final InputStream source) {
+ // Remove this class after TODO open issue with s3mock
+ // Buffer the source InputStream since this class only implements read() so
+ // pass off the actual buffering to the BufferedInputStream to read bigger
+ // chunks at once. This avoids a lot of single byte reads.
+ this.source = new BufferedInputStream(source);
Review Comment:
Wanted to check in on this a month on, see where we stand. Kevin's s3mock PR is still outstanding. It looks like there's also still a few S3 flaky failures, though they haven't been too noticeable IMO. I'll set a reminder for another month out and we can check back in on s3mock's release progress and our flaky rate to see whether it might be worth copying the files in.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] gerlowskija commented on a diff in pull request #1545: Update dependencies com.adobe.testing:s3mock-junit4 to v2.11.0, org.springframework.boot:spring-boot to 2.7.10, org.springframework:spring to 5.3.26, and software.amazon.awssdk to 2.19.1
Posted by "gerlowskija (via GitHub)" <gi...@apache.org>.
gerlowskija commented on code in PR #1545:
URL: https://github.com/apache/solr/pull/1545#discussion_r1160157029
##########
solr/modules/s3-repository/src/test/com/adobe/testing/s3mock/util/AwsChunkedDecodingInputStream.java:
##########
@@ -0,0 +1,144 @@
+/*
+ * Copyright 2017-2022 Adobe.
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package com.adobe.testing.s3mock.util;
+
+import java.io.BufferedInputStream;
+import java.io.IOException;
+import java.io.InputStream;
+import java.nio.ByteBuffer;
+import java.nio.charset.StandardCharsets;
+
+/**
+ * Skips V4 style signing metadata from input streams.
+ * <p>The original stream looks like this (newlines are CRLF):</p>
+ *
+ * <pre>
+ * 5;chunk-signature=7ece820edcf094ce1ef6d643c8db60b67913e28831d9b0430efd2b56a9deec5e
+ * 12345
+ * 0;chunk-signature=ee2c094d7162170fcac17d2c76073cd834b0488bfe52e89e48599b8115c7ffa2
+ * </pre>
+ *
+ * <p>The format of each chunk of data is:</p>
+ *
+ * <pre>
+ * [hex-encoded-number-of-bytes-in-chunk];chunk-signature=[sha256-signature][crlf]
+ * [payload-bytes-of-this-chunk][crlf]
+ * </pre>
+ *
+ * @see
+ * <a href="http://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/auth/AwsChunkedEncodingInputStream.html">
+ * AwsChunkedEncodingInputStream</a>
+ */
+public class AwsChunkedDecodingInputStream extends InputStream {
+
+ /**
+ * That's the max chunk buffer size used in the AWS implementation.
+ */
+ private static final int MAX_CHUNK_SIZE = 256 * 1024;
+
+ private static final byte[] CRLF = "\r\n".getBytes(StandardCharsets.UTF_8);
+
+ private static final byte[] DELIMITER = ";".getBytes(StandardCharsets.UTF_8);
+
+ private final InputStream source;
+
+ private int remainingInChunk = 0;
+
+ private final ByteBuffer byteBuffer = ByteBuffer.allocate(MAX_CHUNK_SIZE);
+
+ /**
+ * Constructs a new {@link AwsChunkedDecodingInputStream}.
+ *
+ * @param source The {@link InputStream} to wrap.
+ */
+ public AwsChunkedDecodingInputStream(final InputStream source) {
+ // Remove this class after TODO open issue with s3mock
+ // Buffer the source InputStream since this class only implements read() so
+ // pass off the actual buffering to the BufferedInputStream to read bigger
+ // chunks at once. This avoids a lot of single byte reads.
+ this.source = new BufferedInputStream(source);
Review Comment:
I could go either way. I get that copying the file just for tests is a pain, and we'd have to update our NOTICE.txt for it and all that. But OTOH, I just had to increase a timeout on the relatively new S3InstallShardTest that I'm 99% sure is failing due to general S3Mock slowness.
IMO it wouldn't hurt to merge this and then nuke the copied file when the next S3Mock release comes out. With the way the solrbot stuff works now we'd even get alerted about the S3Mock release by way of a new PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] risdenk commented on pull request #1545: Update dependencies com.adobe.testing:s3mock-junit4 to v2.11.0, org.springframework.boot:spring-boot to 2.7.10, org.springframework:spring to 5.3.26, and software.amazon.awssdk to 2.19.1
Posted by "risdenk (via GitHub)" <gi...@apache.org>.
risdenk commented on PR #1545:
URL: https://github.com/apache/solr/pull/1545#issuecomment-1497528664
As far as I can tell this fails always on the 10MB copy to s3mock:
```
// copy a 10Mb file
content += "1234567890".repeat(1024 * 1024);
doTestCopyFileFrom(content);
doTestCopyFileTo(content);
```
I don't know what changed but there seems to be some reason that there is a hang when writing the file.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] HoustonPutman commented on a diff in pull request #1545: Update dependencies com.adobe.testing:s3mock-junit4 to v2.11.0, org.springframework.boot:spring-boot to 2.7.10, org.springframework:spring to 5.3.26, and software.amazon.awssdk to 2.19.1
Posted by "HoustonPutman (via GitHub)" <gi...@apache.org>.
HoustonPutman commented on code in PR #1545:
URL: https://github.com/apache/solr/pull/1545#discussion_r1158841532
##########
solr/modules/s3-repository/src/test/com/adobe/testing/s3mock/util/AwsChunkedDecodingInputStream.java:
##########
@@ -0,0 +1,144 @@
+/*
+ * Copyright 2017-2022 Adobe.
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package com.adobe.testing.s3mock.util;
+
+import java.io.BufferedInputStream;
+import java.io.IOException;
+import java.io.InputStream;
+import java.nio.ByteBuffer;
+import java.nio.charset.StandardCharsets;
+
+/**
+ * Skips V4 style signing metadata from input streams.
+ * <p>The original stream looks like this (newlines are CRLF):</p>
+ *
+ * <pre>
+ * 5;chunk-signature=7ece820edcf094ce1ef6d643c8db60b67913e28831d9b0430efd2b56a9deec5e
+ * 12345
+ * 0;chunk-signature=ee2c094d7162170fcac17d2c76073cd834b0488bfe52e89e48599b8115c7ffa2
+ * </pre>
+ *
+ * <p>The format of each chunk of data is:</p>
+ *
+ * <pre>
+ * [hex-encoded-number-of-bytes-in-chunk];chunk-signature=[sha256-signature][crlf]
+ * [payload-bytes-of-this-chunk][crlf]
+ * </pre>
+ *
+ * @see
+ * <a href="http://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/auth/AwsChunkedEncodingInputStream.html">
+ * AwsChunkedEncodingInputStream</a>
+ */
+public class AwsChunkedDecodingInputStream extends InputStream {
+
+ /**
+ * That's the max chunk buffer size used in the AWS implementation.
+ */
+ private static final int MAX_CHUNK_SIZE = 256 * 1024;
+
+ private static final byte[] CRLF = "\r\n".getBytes(StandardCharsets.UTF_8);
+
+ private static final byte[] DELIMITER = ";".getBytes(StandardCharsets.UTF_8);
+
+ private final InputStream source;
+
+ private int remainingInChunk = 0;
+
+ private final ByteBuffer byteBuffer = ByteBuffer.allocate(MAX_CHUNK_SIZE);
+
+ /**
+ * Constructs a new {@link AwsChunkedDecodingInputStream}.
+ *
+ * @param source The {@link InputStream} to wrap.
+ */
+ public AwsChunkedDecodingInputStream(final InputStream source) {
+ // Remove this class after TODO open issue with s3mock
+ // Buffer the source InputStream since this class only implements read() so
+ // pass off the actual buffering to the BufferedInputStream to read bigger
+ // chunks at once. This avoids a lot of single byte reads.
+ this.source = new BufferedInputStream(source);
Review Comment:
yeah I say we just hold off on updating this until they make a release with the fix, especially since this is a test-dependency that we don't need to worry about in releases.
But awesome find!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] risdenk commented on a diff in pull request #1545: Update dependencies com.adobe.testing:s3mock-junit4 to v2.12.2, org.springframework.boot:spring-boot to 2.7.12, and org.springframework:spring to 5.3.27
Posted by "risdenk (via GitHub)" <gi...@apache.org>.
risdenk commented on code in PR #1545:
URL: https://github.com/apache/solr/pull/1545#discussion_r1223127205
##########
solr/modules/s3-repository/src/test/com/adobe/testing/s3mock/util/AwsChunkedDecodingInputStream.java:
##########
@@ -0,0 +1,144 @@
+/*
+ * Copyright 2017-2022 Adobe.
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package com.adobe.testing.s3mock.util;
+
+import java.io.BufferedInputStream;
+import java.io.IOException;
+import java.io.InputStream;
+import java.nio.ByteBuffer;
+import java.nio.charset.StandardCharsets;
+
+/**
+ * Skips V4 style signing metadata from input streams.
+ * <p>The original stream looks like this (newlines are CRLF):</p>
+ *
+ * <pre>
+ * 5;chunk-signature=7ece820edcf094ce1ef6d643c8db60b67913e28831d9b0430efd2b56a9deec5e
+ * 12345
+ * 0;chunk-signature=ee2c094d7162170fcac17d2c76073cd834b0488bfe52e89e48599b8115c7ffa2
+ * </pre>
+ *
+ * <p>The format of each chunk of data is:</p>
+ *
+ * <pre>
+ * [hex-encoded-number-of-bytes-in-chunk];chunk-signature=[sha256-signature][crlf]
+ * [payload-bytes-of-this-chunk][crlf]
+ * </pre>
+ *
+ * @see
+ * <a href="http://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/auth/AwsChunkedEncodingInputStream.html">
+ * AwsChunkedEncodingInputStream</a>
+ */
+public class AwsChunkedDecodingInputStream extends InputStream {
+
+ /**
+ * That's the max chunk buffer size used in the AWS implementation.
+ */
+ private static final int MAX_CHUNK_SIZE = 256 * 1024;
+
+ private static final byte[] CRLF = "\r\n".getBytes(StandardCharsets.UTF_8);
+
+ private static final byte[] DELIMITER = ";".getBytes(StandardCharsets.UTF_8);
+
+ private final InputStream source;
+
+ private int remainingInChunk = 0;
+
+ private final ByteBuffer byteBuffer = ByteBuffer.allocate(MAX_CHUNK_SIZE);
+
+ /**
+ * Constructs a new {@link AwsChunkedDecodingInputStream}.
+ *
+ * @param source The {@link InputStream} to wrap.
+ */
+ public AwsChunkedDecodingInputStream(final InputStream source) {
+ // Remove this class after TODO open issue with s3mock
+ // Buffer the source InputStream since this class only implements read() so
+ // pass off the actual buffering to the BufferedInputStream to read bigger
+ // chunks at once. This avoids a lot of single byte reads.
+ this.source = new BufferedInputStream(source);
Review Comment:
Reverted the commit that overrode this and pushed the changes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] HoustonPutman commented on a diff in pull request #1545: Update dependencies com.adobe.testing:s3mock-junit4 to v2.11.0, org.springframework.boot:spring-boot to 2.7.10, org.springframework:spring to 5.3.26, and software.amazon.awssdk to 2.19.1
Posted by "HoustonPutman (via GitHub)" <gi...@apache.org>.
HoustonPutman commented on code in PR #1545:
URL: https://github.com/apache/solr/pull/1545#discussion_r1223101735
##########
solr/modules/s3-repository/src/test/com/adobe/testing/s3mock/util/AwsChunkedDecodingInputStream.java:
##########
@@ -0,0 +1,144 @@
+/*
+ * Copyright 2017-2022 Adobe.
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package com.adobe.testing.s3mock.util;
+
+import java.io.BufferedInputStream;
+import java.io.IOException;
+import java.io.InputStream;
+import java.nio.ByteBuffer;
+import java.nio.charset.StandardCharsets;
+
+/**
+ * Skips V4 style signing metadata from input streams.
+ * <p>The original stream looks like this (newlines are CRLF):</p>
+ *
+ * <pre>
+ * 5;chunk-signature=7ece820edcf094ce1ef6d643c8db60b67913e28831d9b0430efd2b56a9deec5e
+ * 12345
+ * 0;chunk-signature=ee2c094d7162170fcac17d2c76073cd834b0488bfe52e89e48599b8115c7ffa2
+ * </pre>
+ *
+ * <p>The format of each chunk of data is:</p>
+ *
+ * <pre>
+ * [hex-encoded-number-of-bytes-in-chunk];chunk-signature=[sha256-signature][crlf]
+ * [payload-bytes-of-this-chunk][crlf]
+ * </pre>
+ *
+ * @see
+ * <a href="http://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/auth/AwsChunkedEncodingInputStream.html">
+ * AwsChunkedEncodingInputStream</a>
+ */
+public class AwsChunkedDecodingInputStream extends InputStream {
+
+ /**
+ * That's the max chunk buffer size used in the AWS implementation.
+ */
+ private static final int MAX_CHUNK_SIZE = 256 * 1024;
+
+ private static final byte[] CRLF = "\r\n".getBytes(StandardCharsets.UTF_8);
+
+ private static final byte[] DELIMITER = ";".getBytes(StandardCharsets.UTF_8);
+
+ private final InputStream source;
+
+ private int remainingInChunk = 0;
+
+ private final ByteBuffer byteBuffer = ByteBuffer.allocate(MAX_CHUNK_SIZE);
+
+ /**
+ * Constructs a new {@link AwsChunkedDecodingInputStream}.
+ *
+ * @param source The {@link InputStream} to wrap.
+ */
+ public AwsChunkedDecodingInputStream(final InputStream source) {
+ // Remove this class after TODO open issue with s3mock
+ // Buffer the source InputStream since this class only implements read() so
+ // pass off the actual buffering to the BufferedInputStream to read bigger
+ // chunks at once. This avoids a lot of single byte reads.
+ this.source = new BufferedInputStream(source);
Review Comment:
Thanks for keeping up with this yall 🙂
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] risdenk commented on a diff in pull request #1545: Update dependencies com.adobe.testing:s3mock-junit4 to v2.12.2, org.springframework.boot:spring-boot to 2.7.12, and org.springframework:spring to 5.3.27
Posted by "risdenk (via GitHub)" <gi...@apache.org>.
risdenk commented on code in PR #1545:
URL: https://github.com/apache/solr/pull/1545#discussion_r1223138790
##########
versions.props:
##########
@@ -71,7 +71,7 @@ org.osgi:osgi.annotation=8.1.0
org.quicktheories:quicktheories=0.26
org.semver4j:semver4j=4.3.0
org.slf4j:*=2.0.7
-org.springframework.boot:spring-boot*=2.5.14
-org.springframework:spring*=5.3.23
+org.springframework.boot:spring-boot*=2.7.10
+org.springframework:spring*=5.3.26
Review Comment:
We can do this for spring (and did that in this PR) but spring-boot is referenced as 2.7.7 in s3mock and has some CVEs to left spring-boot in here. https://central.sonatype.com/artifact/com.adobe.testing/s3mock/2.12.2/dependencies?smo=true (ie: https://spring.io/security/cve-2023-20883)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] HoustonPutman commented on a diff in pull request #1545: Update dependencies com.adobe.testing:s3mock-junit4 to v2.12.2 and org.springframework.boot:spring-boot to 2.7.12
Posted by "HoustonPutman (via GitHub)" <gi...@apache.org>.
HoustonPutman commented on code in PR #1545:
URL: https://github.com/apache/solr/pull/1545#discussion_r1223432350
##########
solr/core/src/java/org/apache/solr/core/backup/repository/BackupRepository.java:
##########
@@ -228,9 +228,20 @@ default void copyIndexFileFrom(
* Delete {@code files} at {@code path}
*
* @since 8.3.0
+ * @deprecated Use {@link #delete(URI, Collection)} instead
Review Comment:
So I think we kind of need it to be the opposite. For people who have custom BackupRepositories, this isn't back-compat. They will need to change their Repository to have this new method.
Maybe for 9.x we make the implementation details, but don't add the new method. And then `ignoreNoSuchFileException` will just be ignored.
For 10x, change it to the new method, and just remove the old one.
Thoughts?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] HoustonPutman commented on a diff in pull request #1545: Update dependencies com.adobe.testing:s3mock-junit4 to v2.12.2 and org.springframework.boot:spring-boot to 2.7.12
Posted by "HoustonPutman (via GitHub)" <gi...@apache.org>.
HoustonPutman commented on code in PR #1545:
URL: https://github.com/apache/solr/pull/1545#discussion_r1223440389
##########
solr/core/src/java/org/apache/solr/core/backup/repository/BackupRepository.java:
##########
@@ -228,9 +228,20 @@ default void copyIndexFileFrom(
* Delete {@code files} at {@code path}
*
* @since 8.3.0
+ * @deprecated Use {@link #delete(URI, Collection)} instead
Review Comment:
The default implementation will throw an error, so custom repositories will fail by default when they upgrade to 9.3 (without compilation errors, which are better than runtime errors).
How about we change all usages of `delete(path, files, bool)` internally to use `delete(path, files)`, and have `delete(path, files)` default to `delete(path, files, false)`. Then in `main`, we can remove `delete(path, files, bool)`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] risdenk commented on pull request #1545: Update dependencies com.adobe.testing:s3mock-junit4 to v2.12.2 and org.springframework.boot:spring-boot to 2.7.12
Posted by "risdenk (via GitHub)" <gi...@apache.org>.
risdenk commented on PR #1545:
URL: https://github.com/apache/solr/pull/1545#issuecomment-1582856921
So I don't know if this is a change in AWS SDK or s3mock is better about this now, but
> Note that if the object specified in the request is not found, Amazon S3 returns the result as deleted.
https://sdk.amazonaws.com/java/api/latest/software/amazon/awssdk/services/s3/S3Client.html#deleteObjects(software.amazon.awssdk.services.s3.model.DeleteObjectsRequest)
So `org.apache.solr.s3.S3ReadWriteTest.testNotFound` is testing an invalid thing and trying to check a delete response. @HoustonPutman do you have any opinions here? Do we actually care about the s3notfoundexception?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] risdenk commented on a diff in pull request #1545: Update dependencies com.adobe.testing:s3mock-junit4 to v2.12.2 and org.springframework.boot:spring-boot to 2.7.12
Posted by "risdenk (via GitHub)" <gi...@apache.org>.
risdenk commented on code in PR #1545:
URL: https://github.com/apache/solr/pull/1545#discussion_r1223467406
##########
solr/core/src/java/org/apache/solr/core/backup/repository/BackupRepository.java:
##########
@@ -228,9 +228,20 @@ default void copyIndexFileFrom(
* Delete {@code files} at {@code path}
*
* @since 8.3.0
+ * @deprecated Use {@link #delete(URI, Collection)} instead
Review Comment:
O right sorry missed that case. I think [cda786d](https://github.com/apache/solr/pull/1545/commits/cda786dd2ac95c4d70f133ef58df44eec7c73211) will work
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] gerlowskija commented on a diff in pull request #1545: Update dependencies com.adobe.testing:s3mock-junit4 to v2.11.0, org.springframework.boot:spring-boot to 2.7.10, org.springframework:spring to 5.3.26, and software.amazon.awssdk to 2.19.1
Posted by "gerlowskija (via GitHub)" <gi...@apache.org>.
gerlowskija commented on code in PR #1545:
URL: https://github.com/apache/solr/pull/1545#discussion_r1219631985
##########
solr/modules/s3-repository/src/test/com/adobe/testing/s3mock/util/AwsChunkedDecodingInputStream.java:
##########
@@ -0,0 +1,144 @@
+/*
+ * Copyright 2017-2022 Adobe.
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package com.adobe.testing.s3mock.util;
+
+import java.io.BufferedInputStream;
+import java.io.IOException;
+import java.io.InputStream;
+import java.nio.ByteBuffer;
+import java.nio.charset.StandardCharsets;
+
+/**
+ * Skips V4 style signing metadata from input streams.
+ * <p>The original stream looks like this (newlines are CRLF):</p>
+ *
+ * <pre>
+ * 5;chunk-signature=7ece820edcf094ce1ef6d643c8db60b67913e28831d9b0430efd2b56a9deec5e
+ * 12345
+ * 0;chunk-signature=ee2c094d7162170fcac17d2c76073cd834b0488bfe52e89e48599b8115c7ffa2
+ * </pre>
+ *
+ * <p>The format of each chunk of data is:</p>
+ *
+ * <pre>
+ * [hex-encoded-number-of-bytes-in-chunk];chunk-signature=[sha256-signature][crlf]
+ * [payload-bytes-of-this-chunk][crlf]
+ * </pre>
+ *
+ * @see
+ * <a href="http://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/auth/AwsChunkedEncodingInputStream.html">
+ * AwsChunkedEncodingInputStream</a>
+ */
+public class AwsChunkedDecodingInputStream extends InputStream {
+
+ /**
+ * That's the max chunk buffer size used in the AWS implementation.
+ */
+ private static final int MAX_CHUNK_SIZE = 256 * 1024;
+
+ private static final byte[] CRLF = "\r\n".getBytes(StandardCharsets.UTF_8);
+
+ private static final byte[] DELIMITER = ";".getBytes(StandardCharsets.UTF_8);
+
+ private final InputStream source;
+
+ private int remainingInChunk = 0;
+
+ private final ByteBuffer byteBuffer = ByteBuffer.allocate(MAX_CHUNK_SIZE);
+
+ /**
+ * Constructs a new {@link AwsChunkedDecodingInputStream}.
+ *
+ * @param source The {@link InputStream} to wrap.
+ */
+ public AwsChunkedDecodingInputStream(final InputStream source) {
+ // Remove this class after TODO open issue with s3mock
+ // Buffer the source InputStream since this class only implements read() so
+ // pass off the actual buffering to the BufferedInputStream to read bigger
+ // chunks at once. This avoids a lot of single byte reads.
+ this.source = new BufferedInputStream(source);
Review Comment:
No change from last month afaict. Kevin's s3mock PR is still outstanding. There are a few S3 mock related failures on fucit, but it's pretty far down the list in terms of our flaky tests.
Given that the fucit failures haven't flared up, it seems fine to keep waiting on s3mock if we don't want to duplicate the code locally. I'll set another reminder for a month from now to check back in...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org