You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@druid.apache.org by ka...@apache.org on 2023/04/14 12:19:26 UTC
[druid] branch 26.0.0 updated (0753a23848 -> 1b0c17c43a)
This is an automated email from the ASF dual-hosted git repository.
karan pushed a change to branch 26.0.0
in repository https://gitbox.apache.org/repos/asf/druid.git
from 0753a23848 Fix race condition in KubernetesTaskRunner between shutdown and getKnownTasks (#14030) (#14057)
add c4ba97d27e Fixing overlord unable to become a leader when syncing the lock from metadata store. (#14038)
add 5b85ff7e28 Frames: Ensure nulls are read as default values when appropriate. (#14020)
add 408d23526f MSQ: Support multiple result columns with the same name. (#14025)
add 0812606703 msq: add durable storage info (#14035)
new 1b0c17c43a Adding missed s3 retry handling in storage connector. (#14086)
The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
docs/multi-stage-query/reference.md | 48 +++++++-
docs/multi-stage-query/security.md | 15 +++
.../org/apache/druid/msq/exec/ControllerImpl.java | 100 +++++++++------
.../apache/druid/msq/indexing/ColumnMappings.java | 137 +++++++++++++++------
.../msq/indexing/report/MSQResultsReport.java | 69 ++++++++++-
.../apache/druid/msq/sql/MSQTaskQueryMaker.java | 3 -
.../org/apache/druid/msq/sql/MSQTaskSqlEngine.java | 7 +-
.../org/apache/druid/msq/exec/MSQFaultsTest.java | 56 +++++++--
.../org/apache/druid/msq/exec/MSQInsertTest.java | 24 ++++
.../org/apache/druid/msq/exec/MSQSelectTest.java | 68 ++++++++++
.../msq/indexing/report/MSQTaskReportTest.java | 3 +-
.../common/SortMergeJoinFrameProcessorTest.java | 74 +++++------
.../druid/msq/test/ExtractResultsFactory.java | 9 +-
.../org/apache/druid/msq/test/MSQTestBase.java | 59 +++++----
.../storage/s3/output/RetryableS3OutputStream.java | 12 +-
.../druid/indexing/overlord/TaskLockbox.java | 8 +-
.../druid/indexing/overlord/TaskLockboxTest.java | 6 +-
.../druid/testing/utils/MsqTestQueryHelper.java | 5 +-
.../data/input/impl/CombiningInputSource.java | 2 +-
.../druid/frame/field/DoubleFieldReader.java | 3 +-
.../apache/druid/frame/field/FloatFieldReader.java | 3 +-
.../apache/druid/frame/field/LongFieldReader.java | 5 +-
.../apache/druid/frame/field/LongFieldWriter.java | 23 +++-
.../druid/frame/field/StringFieldReader.java | 14 ++-
.../read/columnar/DoubleFrameColumnReader.java | 2 +-
.../read/columnar/FloatFrameColumnReader.java | 2 +-
.../frame/read/columnar/LongFrameColumnReader.java | 2 +-
.../org/apache/druid/utils/CollectionUtils.java | 23 ++++
.../apache/druid/frame/write/FrameWriterTest.java | 34 +++++
.../apache/druid/utils/CollectionUtilsTest.java | 64 +++++++---
.../apache/druid/sql/calcite/CalciteQueryTest.java | 4 +
31 files changed, 672 insertions(+), 212 deletions(-)
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[druid] 01/01: Adding missed s3 retry handling in storage connector. (#14086)
Posted by ka...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
karan pushed a commit to branch 26.0.0
in repository https://gitbox.apache.org/repos/asf/druid.git
commit 1b0c17c43aa1021db27013bbbe9cc4b997d8f728
Author: Karan Kumar <ka...@gmail.com>
AuthorDate: Fri Apr 14 17:21:39 2023 +0530
Adding missed s3 retry handling in storage connector. (#14086)
(cherry picked from commit bdc547709489adedb00263a7ca04a5cfc15eca90)
---
.../druid/storage/s3/output/RetryableS3OutputStream.java | 12 +++++++++---
1 file changed, 9 insertions(+), 3 deletions(-)
diff --git a/extensions-core/s3-extensions/src/main/java/org/apache/druid/storage/s3/output/RetryableS3OutputStream.java b/extensions-core/s3-extensions/src/main/java/org/apache/druid/storage/s3/output/RetryableS3OutputStream.java
index 4e53016b81..ba3d461bee 100644
--- a/extensions-core/s3-extensions/src/main/java/org/apache/druid/storage/s3/output/RetryableS3OutputStream.java
+++ b/extensions-core/s3-extensions/src/main/java/org/apache/druid/storage/s3/output/RetryableS3OutputStream.java
@@ -126,9 +126,15 @@ public class RetryableS3OutputStream extends OutputStream
this.s3 = s3;
this.s3Key = s3Key;
- final InitiateMultipartUploadResult result = s3.initiateMultipartUpload(
- new InitiateMultipartUploadRequest(config.getBucket(), s3Key)
- );
+ final InitiateMultipartUploadResult result;
+ try {
+ result = S3Utils.retryS3Operation(() -> s3.initiateMultipartUpload(
+ new InitiateMultipartUploadRequest(config.getBucket(), s3Key)
+ ), config.getMaxRetry());
+ }
+ catch (Exception e) {
+ throw new IOException("Unable to start multipart upload", e);
+ }
this.uploadId = result.getUploadId();
this.chunkStorePath = new File(config.getTempDir(), uploadId + UUID.randomUUID());
FileUtils.mkdirp(this.chunkStorePath);
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org