You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@druid.apache.org by GitBox <gi...@apache.org> on 2020/05/20 06:09:17 UTC
[GitHub] [druid] FrankChen021 opened a new pull request #9898: support Aliyun OSS service as deep storage
FrankChen021 opened a new pull request #9898:
URL: https://github.com/apache/druid/pull/9898
### Description
Aliyun is the 3rd largest cloud infrastructure provider in the world, and the 1st in China. It provides its own storage solution known as OSS (Object Storage Service), which is very similar to Amazon S3 from the client interface perspective view.
To support Aliyun OSS as a choice of deep storage is meaningful. It could enrich the ecosystem of druid, and could draw attentions of so many Aliyun customers.
Because the interfaces between Amazon S3 and Aliyun OSS are very similar, the code in this PR is based on that of s3 extensions.
I've tested this extensions in test environment based on 0.18.1 branch for several days, including ingestion/compact/kill segments, index task loges. All data are stored/retrieved/deleted successfully.
<hr>
This PR has:
- [X] been self-reviewed.
- [ ] using the [concurrency checklist](https://github.com/apache/druid/blob/master/dev/code-review/concurrency.md) (Remove this item if the PR doesn't have any relation to concurrency.)
- [X] added documentation for new or modified features or behaviors.
- [ ] added Javadocs for most classes and all non-trivial methods. Linked related entities via Javadoc links.
- [ ] added or updated version, license, or notice information in [licenses.yaml](https://github.com/apache/druid/blob/master/licenses.yaml)
- [ ] added comments explaining the "why" and the intent of the code wherever would not be obvious for an unfamiliar reader.
- [X] added unit tests or modified existing tests to cover new code paths.
- [ ] added integration tests.
- [X] been tested in a test Druid cluster.
<!-- Check the items by putting "x" in the brackets for the done things. Not all of these items apply to every PR. Remove the items which are not done or not relevant to the PR. None of the items from the checklist above are strictly necessary, but it would be very helpful if you at least self-review the PR. -->
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] jon-wei edited a comment on pull request #9898: support Aliyun OSS service as deep storage
Posted by GitBox <gi...@apache.org>.
jon-wei edited a comment on pull request #9898:
URL: https://github.com/apache/druid/pull/9898#issuecomment-650638936
Even if hypothetically OSS and S3 are 100% compatible right now, that feels like a fragile thing to rely on for the future (and do we know for sure there aren't bugs/small implementation differences that will cause us problems if we tried to share code or abstractions?)
> So, I cannot tell what the right way is until we figure out the compatibility level between OSS and S3
Since they're operated and maintained by different organizations, my view is that we should just treat them as if they were non-compatible and totally different things, I don't actually mind code duplication all that much for something like this.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] FrankChen021 commented on pull request #9898: support Aliyun OSS service as deep storage
Posted by GitBox <gi...@apache.org>.
FrankChen021 commented on pull request #9898:
URL: https://github.com/apache/druid/pull/9898#issuecomment-647895605
Hi @jon-wei @fjy Thank you for your opinions. I'm busy these days, and I will fix the code this week.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] jihoonson edited a comment on pull request #9898: support Aliyun OSS service as deep storage
Posted by GitBox <gi...@apache.org>.
jihoonson edited a comment on pull request #9898:
URL: https://github.com/apache/druid/pull/9898#issuecomment-650459835
@FrankChen021 can you explain more detailed motivation behind this PR? I have never used Aliyun OSS before, but it seems like it provides S3-compatible APIs (https://www.alibabacloud.com/help/doc-detail/64919.htm#title-37m-7gl-xy2). Can we use Aliyun OSS as deep storage with our s3 extension? Or do we need something special to do it but missing in the s3 extension?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] FrankChen021 commented on pull request #9898: support Aliyun OSS service as deep storage
Posted by GitBox <gi...@apache.org>.
FrankChen021 commented on pull request #9898:
URL: https://github.com/apache/druid/pull/9898#issuecomment-642413836
> > CI reports insufficient unit test line/function/branch coverage. Is it mandatory to meet that requirement ?
>
> Given the nature of the extension, I feel like adding an integration test that connects to Aliyun would be more useful than full unit test coverage, and I would consider that enough in terms of test coverage.
>
> We have some existing tests for S3/GCS/Azure at that you could use as a reference:
> ITS3ToS3ParallelIndexTest
> ITAzureToAzureParallelIndexTest
> ITGcsToGcsParallelIndexTest
>
> I'll keep you posted with more comments as I review, thanks!
I will add some IT test cases
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] FrankChen021 commented on a change in pull request #9898: support Aliyun OSS service as deep storage
Posted by GitBox <gi...@apache.org>.
FrankChen021 commented on a change in pull request #9898:
URL: https://github.com/apache/druid/pull/9898#discussion_r445612812
##########
File path: extensions-contrib/aliyun-oss-extensions/src/main/java/org/apache/druid/data/input/aliyun/OssInputSource.java
##########
@@ -0,0 +1,178 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.data.input.aliyun;
+
+import com.aliyun.oss.OSS;
+import com.aliyun.oss.model.OSSObjectSummary;
+import com.fasterxml.jackson.annotation.JacksonInject;
+import com.fasterxml.jackson.annotation.JsonCreator;
+import com.fasterxml.jackson.annotation.JsonProperty;
+import com.google.common.base.Preconditions;
+import com.google.common.base.Supplier;
+import com.google.common.base.Suppliers;
+import org.apache.druid.data.input.InputEntity;
+import org.apache.druid.data.input.InputFileAttribute;
+import org.apache.druid.data.input.InputSplit;
+import org.apache.druid.data.input.SplitHintSpec;
+import org.apache.druid.data.input.impl.CloudObjectInputSource;
+import org.apache.druid.data.input.impl.CloudObjectLocation;
+import org.apache.druid.data.input.impl.SplittableInputSource;
+import org.apache.druid.storage.aliyun.OssInputDataConfig;
+import org.apache.druid.storage.aliyun.OssStorageDruidModule;
+import org.apache.druid.storage.aliyun.OssUtils;
+import org.apache.druid.utils.Streams;
+
+import javax.annotation.Nonnull;
+import javax.annotation.Nullable;
+import java.net.URI;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Objects;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+public class OssInputSource extends CloudObjectInputSource
+{
+ private final Supplier<OSS> clientSupplier;
+ @JsonProperty("properties")
+ private final OssClientConfig inputSourceConfig;
+ private final OssInputDataConfig inputDataConfig;
+
+ /**
+ * Constructor for OssInputSource
+ *
+ * @param client The default client built with all default configs
+ * from Guice. This injected singleton client is used when {@param inputSourceConfig}
+ * is not provided and hence
+ * @param inputDataConfig Stores the configuration for options related to reading input data
+ * @param uris User provided uris to read input data
+ * @param prefixes User provided prefixes to read input data
+ * @param objects User provided cloud objects values to read input data
+ * @param inputSourceConfig User provided properties for overriding the default aliyun-oss configuration
+ */
+ @JsonCreator
+ public OssInputSource(
+ @JacksonInject OSS client,
+ @JacksonInject OssInputDataConfig inputDataConfig,
+ @JsonProperty("uris") @Nullable List<URI> uris,
+ @JsonProperty("prefixes") @Nullable List<URI> prefixes,
+ @JsonProperty("objects") @Nullable List<CloudObjectLocation> objects,
+ @JsonProperty("properties") @Nullable OssClientConfig inputSourceConfig
+ )
+ {
+ super(OssStorageDruidModule.SCHEME, uris, prefixes, objects);
+ this.inputDataConfig = Preconditions.checkNotNull(inputDataConfig, "inputDataConfig");
+ Preconditions.checkNotNull(client, "client");
+ this.inputSourceConfig = inputSourceConfig;
+ this.clientSupplier = Suppliers.memoize(
+ () -> {
+ if (inputSourceConfig != null) {
+ return inputSourceConfig.buildClient();
+ } else {
+ return client;
+ }
+ }
+ );
+ }
+
+
+ @Nullable
+ @JsonProperty("properties")
+ public OssClientConfig getOssInputSourceConfig()
+ {
+ return inputSourceConfig;
+ }
+
+ @Override
+ protected InputEntity createEntity(CloudObjectLocation location)
+ {
+ return new OssEntity(clientSupplier.get(), location);
+ }
+
+ @Override
+ protected Stream<InputSplit<List<CloudObjectLocation>>> getPrefixesSplitStream(@Nonnull SplitHintSpec splitHintSpec)
+ {
+ final Iterator<List<OSSObjectSummary>> splitIterator = splitHintSpec.split(
+ getIterableObjectsFromPrefixes().iterator(),
+ object -> new InputFileAttribute(object.getSize())
+ );
+
+ return Streams.sequentialStreamFrom(splitIterator)
+ .map(objects -> objects.stream()
+ .map(OssUtils::summaryToCloudObjectLocation)
+ .collect(Collectors.toList()))
+ .map(InputSplit::new);
+ }
+
+ @Override
+ public SplittableInputSource<List<CloudObjectLocation>> withSplit(InputSplit<List<CloudObjectLocation>> split)
+ {
+ return new OssInputSource(
+ clientSupplier.get(),
+ inputDataConfig,
+ null,
+ null,
+ split.get(),
+ getOssInputSourceConfig()
+ );
+ }
+
+ @Override
+ public int hashCode()
+ {
+ return Objects.hash(super.hashCode(), inputSourceConfig);
+ }
+
+ @Override
+ public boolean equals(Object o)
+ {
+ if (this == o) {
+ return true;
+ }
+ if (o == null || getClass() != o.getClass()) {
+ return false;
+ }
+ if (!super.equals(o)) {
+ return false;
+ }
+ OssInputSource that = (OssInputSource) o;
Review comment:
it's `super.equals` which checks uris and prefixes
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] FrankChen021 commented on a change in pull request #9898: support Aliyun OSS service as deep storage
Posted by GitBox <gi...@apache.org>.
FrankChen021 commented on a change in pull request #9898:
URL: https://github.com/apache/druid/pull/9898#discussion_r448148488
##########
File path: extensions-contrib/aliyun-oss-extensions/src/test/java/org/apache/druid/storage/aliyun/OssDataSegmentMoverTest.java
##########
@@ -0,0 +1,266 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.storage.aliyun;
+
+import com.aliyun.oss.OSSClient;
+import com.aliyun.oss.OSSException;
+import com.aliyun.oss.model.CopyObjectRequest;
+import com.aliyun.oss.model.CopyObjectResult;
+import com.aliyun.oss.model.ListObjectsRequest;
+import com.aliyun.oss.model.OSSObjectSummary;
+import com.aliyun.oss.model.ObjectListing;
+import com.aliyun.oss.model.PutObjectResult;
+import com.aliyun.oss.model.StorageClass;
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.ImmutableMap;
+import org.apache.druid.java.util.common.Intervals;
+import org.apache.druid.java.util.common.MapUtils;
+import org.apache.druid.segment.loading.SegmentLoadingException;
+import org.apache.druid.timeline.DataSegment;
+import org.apache.druid.timeline.partition.NoneShardSpec;
+import org.junit.Assert;
+import org.junit.Test;
+
+import java.io.File;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Map;
+import java.util.Set;
+
+public class OssDataSegmentMoverTest
+{
+ private static final DataSegment SOURCE_SEGMENT = new DataSegment(
+ "test",
+ Intervals.of("2013-01-01/2013-01-02"),
+ "1",
+ ImmutableMap.of(
+ "key",
+ "baseKey/test/2013-01-01T00:00:00.000Z_2013-01-02T00:00:00.000Z/1/0/index.zip",
+ "bucket",
+ "main"
+ ),
+ ImmutableList.of("dim1", "dim1"),
+ ImmutableList.of("metric1", "metric2"),
+ NoneShardSpec.instance(),
+ 0,
+ 1
+ );
+
+ @Test
+ public void testMove() throws Exception
+ {
+ MockClient mockClient = new MockClient();
+ OssDataSegmentMover mover = new OssDataSegmentMover(mockClient, new OssStorageConfig());
+
+ mockClient.putObject(
+ "main",
+ "baseKey/test/2013-01-01T00:00:00.000Z_2013-01-02T00:00:00.000Z/1/0/index.zip"
+ );
+
+ DataSegment movedSegment = mover.move(
+ SOURCE_SEGMENT,
+ ImmutableMap.of("baseKey", "targetBaseKey", "bucket", "archive")
+ );
+
+ Map<String, Object> targetLoadSpec = movedSegment.getLoadSpec();
+ Assert.assertEquals(
+ "targetBaseKey/test/2013-01-01T00:00:00.000Z_2013-01-02T00:00:00.000Z/1/0/index.zip",
+ MapUtils.getString(targetLoadSpec, "key")
+ );
+ Assert.assertEquals("archive", MapUtils.getString(targetLoadSpec, "bucket"));
+ Assert.assertTrue(mockClient.didMove());
+ }
+
+ @Test
+ public void testMoveNoop() throws Exception
+ {
+ MockClient mockS3Client = new MockClient();
Review comment:
all `s3` in this extension have been fixed
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] FrankChen021 commented on pull request #9898: support Aliyun OSS service as deep storage
Posted by GitBox <gi...@apache.org>.
FrankChen021 commented on pull request #9898:
URL: https://github.com/apache/druid/pull/9898#issuecomment-650562183
Hi @jihoonson , s3 extension does not work with aliyun OSS although OSS provides compatibility with s3. I tested to utilize OSS with s3 extension in the past, but the `putObject` returned HTTP 400 error, saying there were some unsupported HEADERS in the request generated by s3 SDK. I didn't explore that problem very deeply, I guess the latest s3 SDK adds some new HEADERS which are not compatible in OSS.
the APIs between OSS and s3 are very similar, even their names. but I don't think it's a good idea to use one extension to support both OSS and s3, or use some abstraction method to extract the same code between s3 and OSS. Actually, this kind of abstraction has been done by deep storage framework, so every implementation interacts with its underlying storage API. As s3 and OSS are provided by two companies, it's better to provide two extensions respectively so that they can evolve independently.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] jihoonson commented on pull request #9898: support Aliyun OSS service as deep storage
Posted by GitBox <gi...@apache.org>.
jihoonson commented on pull request #9898:
URL: https://github.com/apache/druid/pull/9898#issuecomment-650675024
@FrankChen021 thanks for the error. It does seem there is some incompatibility between them. We are using aws sdk 1.11.199 which was released in 2017. This version of sdk adds a couple of headers and I'm not sure what are not supported by OSS. Even though the OSS doc was updated last year, I guess the doc doesn't show the compatibility level correctly. This makes me think there could be some OSS APIs not compatible to s3 APIs.
Looking at the CI failure, some job failures seem legit.
```
../docs/development/extensions-contrib/aliyun-oss-extensions.md
44 | |`druid.storage.type`| | oss|Must be set.|
52 | |`druid.indexer.logs.type`| | oss|Must be set.|
```
The CI failed for doc, you need to add `oss` to the `./website/.spelling` file.
The integration test failure also looks legit. Would you please check it?
```
2020-06-27T00:11:30,200 ERROR [main] org.apache.druid.testing.utils.LoggerListener - Failed [org.apache.druid.tests.indexer.ITOssToOssParallelIndexTest.testAliyunOssIndexData] with parameters [Pair{lhs=uris, rhs=[oss://%%BUCKET%%/%%PATH%%wikipedia_index_data1.json, oss://%%BUCKET%%/%%PATH%%wikipedia_index_data2.json, oss://%%BUCKET%%/%%PATH%%wikipedia_index_data3.json]}]
java.lang.RuntimeException: java.lang.NullPointerException
at org.apache.druid.tests.indexer.AbstractOssInputSourceParallelIndexTest.lambda$doTest$0(AbstractOssInputSourceParallelIndexTest.java:117) ~[test-classes/:?]
at org.apache.druid.tests.indexer.AbstractITBatchIndexTest.doIndexTest(AbstractITBatchIndexTest.java:117) ~[test-classes/:?]
at org.apache.druid.tests.indexer.AbstractOssInputSourceParallelIndexTest.doTest(AbstractOssInputSourceParallelIndexTest.java:121) ~[test-classes/:?]
at org.apache.druid.tests.indexer.ITOssToOssParallelIndexTest.testAliyunOssIndexData(ITOssToOssParallelIndexTest.java:47) ~[test-classes/:?]
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] jon-wei commented on pull request #9898: support Aliyun OSS service as deep storage
Posted by GitBox <gi...@apache.org>.
jon-wei commented on pull request #9898:
URL: https://github.com/apache/druid/pull/9898#issuecomment-642357853
> CI reports insufficient unit test line/function/branch coverage. Is it mandatory to meet that requirement ?
Given the nature of the extension, I feel like adding an integration test that connects to Aliyun would be more useful than full unit test coverage, and I would consider that enough in terms of test coverage.
We have some existing tests for S3/GCS/Azure at that you could use as a reference:
ITS3ToS3ParallelIndexTest
ITAzureToAzureParallelIndexTest
ITGcsToGcsParallelIndexTest
I'll keep you posted with more comments as I review, thanks!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] FrankChen021 commented on pull request #9898: support Aliyun OSS service as deep storage
Posted by GitBox <gi...@apache.org>.
FrankChen021 commented on pull request #9898:
URL: https://github.com/apache/druid/pull/9898#issuecomment-650723821
@asdf2014 It's done. Thank you for your instruction.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] FrankChen021 commented on a change in pull request #9898: support Aliyun OSS service as deep storage
Posted by GitBox <gi...@apache.org>.
FrankChen021 commented on a change in pull request #9898:
URL: https://github.com/apache/druid/pull/9898#discussion_r446116676
##########
File path: extensions-contrib/aliyun-oss-extensions/src/main/java/org/apache/druid/storage/aliyun/ObjectSummaryIterator.java
##########
@@ -0,0 +1,163 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.storage.aliyun;
+
+import com.aliyun.oss.OSS;
+import com.aliyun.oss.OSSException;
+import com.aliyun.oss.model.ListObjectsRequest;
+import com.aliyun.oss.model.OSSObjectSummary;
+import com.aliyun.oss.model.ObjectListing;
+import org.apache.druid.java.util.common.RE;
+
+import java.net.URI;
+import java.util.Iterator;
+import java.util.NoSuchElementException;
+
+/**
+ * Iterator class used by {@link OssUtils#objectSummaryIterator}.
+ * <p>
+ * As required by the specification of that method, this iterator is computed incrementally in batches of
+ * {@code maxListLength}. The first call is made at the same time the iterator is constructed.
+ *
+ */
+public class ObjectSummaryIterator implements Iterator<OSSObjectSummary>
+{
+ private final OSS client;
+ private final Iterator<URI> prefixesIterator;
+ private final int maxListingLength;
+
+ private ListObjectsRequest request;
+ private ObjectListing result;
+ private Iterator<OSSObjectSummary> objectSummaryIterator;
+ private OSSObjectSummary currentObjectSummary;
+
+ ObjectSummaryIterator(
+ final OSS client,
+ final Iterable<URI> prefixes,
+ final int maxListingLength
+ )
+ {
+ this.client = client;
+ this.prefixesIterator = prefixes.iterator();
+ this.maxListingLength = maxListingLength;
+
+ prepareNextRequest();
+ fetchNextBatch();
+ advanceObjectSummary();
+ }
+
+ @Override
+ public boolean hasNext()
+ {
+ return currentObjectSummary != null;
+ }
+
+ @Override
+ public OSSObjectSummary next()
+ {
+ if (currentObjectSummary == null) {
+ throw new NoSuchElementException();
+ }
+
+ final OSSObjectSummary retVal = currentObjectSummary;
+ advanceObjectSummary();
+ return retVal;
+ }
+
+ private void prepareNextRequest()
+ {
+ final URI currentUri = prefixesIterator.next();
+ final String currentBucket = currentUri.getAuthority();
+ final String currentPrefix = OssUtils.extractKey(currentUri);
+
+ request = new ListObjectsRequest(currentBucket, currentPrefix, null, null, maxListingLength);
+ }
+
+ private void fetchNextBatch()
+ {
+ try {
+ result = OssUtils.retry(() -> client.listObjects(request));
+ request.setMarker(result.getNextMarker());
+ objectSummaryIterator = result.getObjectSummaries().iterator();
+ }
+ catch (OSSException e) {
+ throw new RE(
+ e,
+ "Failed to get object summaries from S3 bucket[%s], prefix[%s]; S3 error: %s",
+ request.getBucketName(),
+ request.getPrefix(),
+ e.getMessage()
+ );
+ }
+ catch (Exception e) {
+ throw new RE(
+ e,
+ "Failed to get object summaries from S3 bucket[%s], prefix[%s]",
+ request.getBucketName(),
+ request.getPrefix()
+ );
+ }
+ }
+
+ /**
+ * Advance objectSummaryIterator to the next non-placeholder, updating "currentObjectSummary".
+ */
+ private void advanceObjectSummary()
+ {
+ while (objectSummaryIterator.hasNext() || result.isTruncated() || prefixesIterator.hasNext()) {
+ while (objectSummaryIterator.hasNext()) {
+ currentObjectSummary = objectSummaryIterator.next();
+ // skips directories and empty objects
+ if (!isDirectoryPlaceholder(currentObjectSummary) && currentObjectSummary.getSize() > 0) {
+ return;
+ }
+ }
+
+ // Exhausted "objectSummaryIterator" without finding a non-placeholder.
+ if (result.isTruncated()) {
+ fetchNextBatch();
+ } else if (prefixesIterator.hasNext()) {
+ prepareNextRequest();
+ fetchNextBatch();
+ }
+ }
+
+ // Truly nothing left to read.
+ currentObjectSummary = null;
+ }
+
+ /**
+ * Checks if a given object is a directory placeholder and should be ignored.
Review comment:
@jon-wei LICENSE has been updated in the lastest commit, please check it again.
##########
File path: extensions-contrib/aliyun-oss-extensions/src/main/java/org/apache/druid/storage/aliyun/ObjectSummaryIterator.java
##########
@@ -0,0 +1,163 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.storage.aliyun;
+
+import com.aliyun.oss.OSS;
+import com.aliyun.oss.OSSException;
+import com.aliyun.oss.model.ListObjectsRequest;
+import com.aliyun.oss.model.OSSObjectSummary;
+import com.aliyun.oss.model.ObjectListing;
+import org.apache.druid.java.util.common.RE;
+
+import java.net.URI;
+import java.util.Iterator;
+import java.util.NoSuchElementException;
+
+/**
+ * Iterator class used by {@link OssUtils#objectSummaryIterator}.
+ * <p>
+ * As required by the specification of that method, this iterator is computed incrementally in batches of
+ * {@code maxListLength}. The first call is made at the same time the iterator is constructed.
+ *
+ */
+public class ObjectSummaryIterator implements Iterator<OSSObjectSummary>
+{
+ private final OSS client;
+ private final Iterator<URI> prefixesIterator;
+ private final int maxListingLength;
+
+ private ListObjectsRequest request;
+ private ObjectListing result;
+ private Iterator<OSSObjectSummary> objectSummaryIterator;
+ private OSSObjectSummary currentObjectSummary;
+
+ ObjectSummaryIterator(
+ final OSS client,
+ final Iterable<URI> prefixes,
+ final int maxListingLength
+ )
+ {
+ this.client = client;
+ this.prefixesIterator = prefixes.iterator();
+ this.maxListingLength = maxListingLength;
+
+ prepareNextRequest();
+ fetchNextBatch();
+ advanceObjectSummary();
+ }
+
+ @Override
+ public boolean hasNext()
+ {
+ return currentObjectSummary != null;
+ }
+
+ @Override
+ public OSSObjectSummary next()
+ {
+ if (currentObjectSummary == null) {
+ throw new NoSuchElementException();
+ }
+
+ final OSSObjectSummary retVal = currentObjectSummary;
+ advanceObjectSummary();
+ return retVal;
+ }
+
+ private void prepareNextRequest()
+ {
+ final URI currentUri = prefixesIterator.next();
+ final String currentBucket = currentUri.getAuthority();
+ final String currentPrefix = OssUtils.extractKey(currentUri);
+
+ request = new ListObjectsRequest(currentBucket, currentPrefix, null, null, maxListingLength);
+ }
+
+ private void fetchNextBatch()
+ {
+ try {
+ result = OssUtils.retry(() -> client.listObjects(request));
+ request.setMarker(result.getNextMarker());
+ objectSummaryIterator = result.getObjectSummaries().iterator();
+ }
+ catch (OSSException e) {
+ throw new RE(
+ e,
+ "Failed to get object summaries from S3 bucket[%s], prefix[%s]; S3 error: %s",
+ request.getBucketName(),
+ request.getPrefix(),
+ e.getMessage()
+ );
+ }
+ catch (Exception e) {
+ throw new RE(
+ e,
+ "Failed to get object summaries from S3 bucket[%s], prefix[%s]",
+ request.getBucketName(),
+ request.getPrefix()
+ );
+ }
+ }
+
+ /**
+ * Advance objectSummaryIterator to the next non-placeholder, updating "currentObjectSummary".
+ */
+ private void advanceObjectSummary()
+ {
+ while (objectSummaryIterator.hasNext() || result.isTruncated() || prefixesIterator.hasNext()) {
+ while (objectSummaryIterator.hasNext()) {
+ currentObjectSummary = objectSummaryIterator.next();
+ // skips directories and empty objects
+ if (!isDirectoryPlaceholder(currentObjectSummary) && currentObjectSummary.getSize() > 0) {
+ return;
+ }
+ }
+
+ // Exhausted "objectSummaryIterator" without finding a non-placeholder.
+ if (result.isTruncated()) {
+ fetchNextBatch();
+ } else if (prefixesIterator.hasNext()) {
+ prepareNextRequest();
+ fetchNextBatch();
+ }
+ }
+
+ // Truly nothing left to read.
+ currentObjectSummary = null;
+ }
+
+ /**
+ * Checks if a given object is a directory placeholder and should be ignored.
Review comment:
@jon-wei LICENSE has been updated in the latest commit, please check it again.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] fjy commented on pull request #9898: support Aliyun OSS service as deep storage
Posted by GitBox <gi...@apache.org>.
fjy commented on pull request #9898:
URL: https://github.com/apache/druid/pull/9898#issuecomment-642124194
@vogievetsky we should get a tile in the data loader in place once this is merged
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] jihoonson edited a comment on pull request #9898: support Aliyun OSS service as deep storage
Posted by GitBox <gi...@apache.org>.
jihoonson edited a comment on pull request #9898:
URL: https://github.com/apache/druid/pull/9898#issuecomment-650675024
@FrankChen021 thanks for the error. It does seem there is some incompatibility between them. We are using aws sdk 1.11.199 which was released in 2017. This version of sdk adds a couple of headers and I'm not sure what are not supported by OSS. Even though the OSS doc was updated last year, I guess the doc doesn't show the compatibility level correctly. This makes me think at least there could be some OSS APIs not compatible to s3 APIs.
Looking at the CI failure, some job failures seem legit.
```
../docs/development/extensions-contrib/aliyun-oss-extensions.md
44 | |`druid.storage.type`| | oss|Must be set.|
52 | |`druid.indexer.logs.type`| | oss|Must be set.|
```
The CI failed for doc, you need to add `oss` to the `./website/.spelling` file.
The integration test failure also looks legit. Would you please check it?
```
2020-06-27T00:11:30,200 ERROR [main] org.apache.druid.testing.utils.LoggerListener - Failed [org.apache.druid.tests.indexer.ITOssToOssParallelIndexTest.testAliyunOssIndexData] with parameters [Pair{lhs=uris, rhs=[oss://%%BUCKET%%/%%PATH%%wikipedia_index_data1.json, oss://%%BUCKET%%/%%PATH%%wikipedia_index_data2.json, oss://%%BUCKET%%/%%PATH%%wikipedia_index_data3.json]}]
java.lang.RuntimeException: java.lang.NullPointerException
at org.apache.druid.tests.indexer.AbstractOssInputSourceParallelIndexTest.lambda$doTest$0(AbstractOssInputSourceParallelIndexTest.java:117) ~[test-classes/:?]
at org.apache.druid.tests.indexer.AbstractITBatchIndexTest.doIndexTest(AbstractITBatchIndexTest.java:117) ~[test-classes/:?]
at org.apache.druid.tests.indexer.AbstractOssInputSourceParallelIndexTest.doTest(AbstractOssInputSourceParallelIndexTest.java:121) ~[test-classes/:?]
at org.apache.druid.tests.indexer.ITOssToOssParallelIndexTest.testAliyunOssIndexData(ITOssToOssParallelIndexTest.java:47) ~[test-classes/:?]
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] lgtm-com[bot] commented on pull request #9898: support Aliyun OSS service as deep storage
Posted by GitBox <gi...@apache.org>.
lgtm-com[bot] commented on pull request #9898:
URL: https://github.com/apache/druid/pull/9898#issuecomment-631284211
This pull request **introduces 1 alert** when merging 2860f94b6304fb349b94b2da24cea6f57466bb60 into 5b4b5d77a81817baf4359574d61e2dd2acabddba - [view on LGTM.com](https://lgtm.com/projects/g/apache/druid/rev/pr-741dd0d4ede5bde2202484e11d2f1e41832a516b)
**new alerts:**
* 1 for Spurious Javadoc @param tags
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] clintropolis commented on a change in pull request #9898: support Aliyun OSS service as deep storage
Posted by GitBox <gi...@apache.org>.
clintropolis commented on a change in pull request #9898:
URL: https://github.com/apache/druid/pull/9898#discussion_r447390473
##########
File path: integration-tests/src/test/java/org/apache/druid/tests/TestNGGroup.java
##########
@@ -76,6 +76,13 @@
*/
public static final String AZURE_DEEP_STORAGE = "azure-deep-storage";
+ /**
+ * This group is not part of CI. To run this group, azure configs/credentials for your azure must be provided in a file.
Review comment:
nit: azure -> aliyun
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] jon-wei commented on a change in pull request #9898: support Aliyun OSS service as deep storage
Posted by GitBox <gi...@apache.org>.
jon-wei commented on a change in pull request #9898:
URL: https://github.com/apache/druid/pull/9898#discussion_r441968129
##########
File path: extensions-contrib/aliyun-oss-extensions/src/main/java/org/apache/druid/data/input/aliyun/OssEntity.java
##########
@@ -0,0 +1,88 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.data.input.aliyun;
+
+import com.aliyun.oss.OSS;
+import com.aliyun.oss.OSSException;
+import com.aliyun.oss.model.GetObjectRequest;
+import com.aliyun.oss.model.OSSObject;
+import com.google.common.base.Predicate;
+import org.apache.druid.data.input.RetryingInputEntity;
+import org.apache.druid.data.input.impl.CloudObjectLocation;
+import org.apache.druid.java.util.common.ISE;
+import org.apache.druid.storage.aliyun.OssStorageDruidModule;
+import org.apache.druid.storage.aliyun.OssUtils;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.net.URI;
+
+public class OssEntity extends RetryingInputEntity
+{
+ private final OSS ossClient;
+ private final CloudObjectLocation object;
+
+ OssEntity(OSS ossClient, CloudObjectLocation coords)
+ {
+ this.ossClient = ossClient;
+ this.object = coords;
+ }
+
+ @Override
+ public URI getUri()
+ {
+ return object.toUri(OssStorageDruidModule.SCHEME);
+ }
+
+ @Override
+ protected InputStream readFrom(long offset) throws IOException
+ {
+ final GetObjectRequest request = new GetObjectRequest(object.getBucket(), object.getPath());
+ request.setRange(offset, -1 /*from offset to end*/);
+
+ try {
+ final OSSObject ossObject = ossClient.getObject(request);
+ if (ossObject == null) {
+ throw new ISE(
+ "Failed to get an s3 object for bucket[%s], key[%s], and start[%d]",
Review comment:
s3 -> Aliyun OSS
##########
File path: extensions-contrib/aliyun-oss-extensions/src/main/java/org/apache/druid/storage/aliyun/ObjectSummaryIterator.java
##########
@@ -0,0 +1,163 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.storage.aliyun;
+
+import com.aliyun.oss.OSS;
+import com.aliyun.oss.OSSException;
+import com.aliyun.oss.model.ListObjectsRequest;
+import com.aliyun.oss.model.OSSObjectSummary;
+import com.aliyun.oss.model.ObjectListing;
+import org.apache.druid.java.util.common.RE;
+
+import java.net.URI;
+import java.util.Iterator;
+import java.util.NoSuchElementException;
+
+/**
+ * Iterator class used by {@link OssUtils#objectSummaryIterator}.
+ * <p>
+ * As required by the specification of that method, this iterator is computed incrementally in batches of
+ * {@code maxListLength}. The first call is made at the same time the iterator is constructed.
+ *
+ */
+public class ObjectSummaryIterator implements Iterator<OSSObjectSummary>
Review comment:
Suggest renaming this to avoid having the same name as the S3 version
##########
File path: extensions-contrib/aliyun-oss-extensions/src/main/java/org/apache/druid/storage/aliyun/ObjectSummaryIterator.java
##########
@@ -0,0 +1,163 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.storage.aliyun;
+
+import com.aliyun.oss.OSS;
+import com.aliyun.oss.OSSException;
+import com.aliyun.oss.model.ListObjectsRequest;
+import com.aliyun.oss.model.OSSObjectSummary;
+import com.aliyun.oss.model.ObjectListing;
+import org.apache.druid.java.util.common.RE;
+
+import java.net.URI;
+import java.util.Iterator;
+import java.util.NoSuchElementException;
+
+/**
+ * Iterator class used by {@link OssUtils#objectSummaryIterator}.
+ * <p>
+ * As required by the specification of that method, this iterator is computed incrementally in batches of
+ * {@code maxListLength}. The first call is made at the same time the iterator is constructed.
+ *
+ */
+public class ObjectSummaryIterator implements Iterator<OSSObjectSummary>
+{
+ private final OSS client;
+ private final Iterator<URI> prefixesIterator;
+ private final int maxListingLength;
+
+ private ListObjectsRequest request;
+ private ObjectListing result;
+ private Iterator<OSSObjectSummary> objectSummaryIterator;
+ private OSSObjectSummary currentObjectSummary;
+
+ ObjectSummaryIterator(
+ final OSS client,
+ final Iterable<URI> prefixes,
+ final int maxListingLength
+ )
+ {
+ this.client = client;
+ this.prefixesIterator = prefixes.iterator();
+ this.maxListingLength = maxListingLength;
+
+ prepareNextRequest();
+ fetchNextBatch();
+ advanceObjectSummary();
+ }
+
+ @Override
+ public boolean hasNext()
+ {
+ return currentObjectSummary != null;
+ }
+
+ @Override
+ public OSSObjectSummary next()
+ {
+ if (currentObjectSummary == null) {
+ throw new NoSuchElementException();
+ }
+
+ final OSSObjectSummary retVal = currentObjectSummary;
+ advanceObjectSummary();
+ return retVal;
+ }
+
+ private void prepareNextRequest()
+ {
+ final URI currentUri = prefixesIterator.next();
+ final String currentBucket = currentUri.getAuthority();
+ final String currentPrefix = OssUtils.extractKey(currentUri);
+
+ request = new ListObjectsRequest(currentBucket, currentPrefix, null, null, maxListingLength);
+ }
+
+ private void fetchNextBatch()
+ {
+ try {
+ result = OssUtils.retry(() -> client.listObjects(request));
+ request.setMarker(result.getNextMarker());
+ objectSummaryIterator = result.getObjectSummaries().iterator();
+ }
+ catch (OSSException e) {
+ throw new RE(
+ e,
+ "Failed to get object summaries from S3 bucket[%s], prefix[%s]; S3 error: %s",
+ request.getBucketName(),
+ request.getPrefix(),
+ e.getMessage()
+ );
+ }
+ catch (Exception e) {
+ throw new RE(
+ e,
+ "Failed to get object summaries from S3 bucket[%s], prefix[%s]",
+ request.getBucketName(),
+ request.getPrefix()
+ );
+ }
+ }
+
+ /**
+ * Advance objectSummaryIterator to the next non-placeholder, updating "currentObjectSummary".
+ */
+ private void advanceObjectSummary()
+ {
+ while (objectSummaryIterator.hasNext() || result.isTruncated() || prefixesIterator.hasNext()) {
+ while (objectSummaryIterator.hasNext()) {
+ currentObjectSummary = objectSummaryIterator.next();
+ // skips directories and empty objects
+ if (!isDirectoryPlaceholder(currentObjectSummary) && currentObjectSummary.getSize() > 0) {
+ return;
+ }
+ }
+
+ // Exhausted "objectSummaryIterator" without finding a non-placeholder.
+ if (result.isTruncated()) {
+ fetchNextBatch();
+ } else if (prefixesIterator.hasNext()) {
+ prepareNextRequest();
+ fetchNextBatch();
+ }
+ }
+
+ // Truly nothing left to read.
+ currentObjectSummary = null;
+ }
+
+ /**
+ * Checks if a given object is a directory placeholder and should be ignored.
Review comment:
This should preserve the original javadoc about being adapted from jets3t code:
```
* Adapted from org.jets3t.service.model.StorageObject.isDirectoryPlaceholder(). Does not include the check for
* legacy JetS3t directory placeholder objects, since it is based on content-type, which isn't available in an
* S3ObjectSummary.
```
Can you also add an entry noting the usage here?
https://github.com/apache/druid/blob/master/LICENSE#L276
##########
File path: extensions-contrib/aliyun-oss-extensions/src/main/java/org/apache/druid/data/input/aliyun/OssInputSource.java
##########
@@ -0,0 +1,178 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.data.input.aliyun;
+
+import com.aliyun.oss.OSS;
+import com.aliyun.oss.model.OSSObjectSummary;
+import com.fasterxml.jackson.annotation.JacksonInject;
+import com.fasterxml.jackson.annotation.JsonCreator;
+import com.fasterxml.jackson.annotation.JsonProperty;
+import com.google.common.base.Preconditions;
+import com.google.common.base.Supplier;
+import com.google.common.base.Suppliers;
+import org.apache.druid.data.input.InputEntity;
+import org.apache.druid.data.input.InputFileAttribute;
+import org.apache.druid.data.input.InputSplit;
+import org.apache.druid.data.input.SplitHintSpec;
+import org.apache.druid.data.input.impl.CloudObjectInputSource;
+import org.apache.druid.data.input.impl.CloudObjectLocation;
+import org.apache.druid.data.input.impl.SplittableInputSource;
+import org.apache.druid.storage.aliyun.OssInputDataConfig;
+import org.apache.druid.storage.aliyun.OssStorageDruidModule;
+import org.apache.druid.storage.aliyun.OssUtils;
+import org.apache.druid.utils.Streams;
+
+import javax.annotation.Nonnull;
+import javax.annotation.Nullable;
+import java.net.URI;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Objects;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+public class OssInputSource extends CloudObjectInputSource
+{
+ private final Supplier<OSS> clientSupplier;
+ @JsonProperty("properties")
+ private final OssClientConfig inputSourceConfig;
+ private final OssInputDataConfig inputDataConfig;
+
+ /**
+ * Constructor for OssInputSource
+ *
+ * @param client The default client built with all default configs
+ * from Guice. This injected singleton client is used when {@param inputSourceConfig}
+ * is not provided and hence
+ * @param inputDataConfig Stores the configuration for options related to reading input data
+ * @param uris User provided uris to read input data
+ * @param prefixes User provided prefixes to read input data
+ * @param objects User provided cloud objects values to read input data
+ * @param inputSourceConfig User provided properties for overriding the default aliyun-oss configuration
+ */
+ @JsonCreator
+ public OssInputSource(
+ @JacksonInject OSS client,
+ @JacksonInject OssInputDataConfig inputDataConfig,
+ @JsonProperty("uris") @Nullable List<URI> uris,
+ @JsonProperty("prefixes") @Nullable List<URI> prefixes,
+ @JsonProperty("objects") @Nullable List<CloudObjectLocation> objects,
+ @JsonProperty("properties") @Nullable OssClientConfig inputSourceConfig
+ )
+ {
+ super(OssStorageDruidModule.SCHEME, uris, prefixes, objects);
+ this.inputDataConfig = Preconditions.checkNotNull(inputDataConfig, "inputDataConfig");
+ Preconditions.checkNotNull(client, "client");
+ this.inputSourceConfig = inputSourceConfig;
+ this.clientSupplier = Suppliers.memoize(
+ () -> {
+ if (inputSourceConfig != null) {
+ return inputSourceConfig.buildClient();
+ } else {
+ return client;
+ }
+ }
+ );
+ }
+
+
+ @Nullable
+ @JsonProperty("properties")
+ public OssClientConfig getOssInputSourceConfig()
+ {
+ return inputSourceConfig;
+ }
+
+ @Override
+ protected InputEntity createEntity(CloudObjectLocation location)
+ {
+ return new OssEntity(clientSupplier.get(), location);
+ }
+
+ @Override
+ protected Stream<InputSplit<List<CloudObjectLocation>>> getPrefixesSplitStream(@Nonnull SplitHintSpec splitHintSpec)
+ {
+ final Iterator<List<OSSObjectSummary>> splitIterator = splitHintSpec.split(
+ getIterableObjectsFromPrefixes().iterator(),
+ object -> new InputFileAttribute(object.getSize())
+ );
+
+ return Streams.sequentialStreamFrom(splitIterator)
+ .map(objects -> objects.stream()
+ .map(OssUtils::summaryToCloudObjectLocation)
+ .collect(Collectors.toList()))
+ .map(InputSplit::new);
+ }
+
+ @Override
+ public SplittableInputSource<List<CloudObjectLocation>> withSplit(InputSplit<List<CloudObjectLocation>> split)
+ {
+ return new OssInputSource(
+ clientSupplier.get(),
+ inputDataConfig,
+ null,
+ null,
+ split.get(),
+ getOssInputSourceConfig()
+ );
+ }
+
+ @Override
+ public int hashCode()
+ {
+ return Objects.hash(super.hashCode(), inputSourceConfig);
+ }
+
+ @Override
+ public boolean equals(Object o)
+ {
+ if (this == o) {
+ return true;
+ }
+ if (o == null || getClass() != o.getClass()) {
+ return false;
+ }
+ if (!super.equals(o)) {
+ return false;
+ }
+ OssInputSource that = (OssInputSource) o;
Review comment:
I think equals/hashcode should have uris, prefixes, etc.
##########
File path: extensions-contrib/aliyun-oss-extensions/src/main/java/org/apache/druid/storage/aliyun/OssStorageDruidModule.java
##########
@@ -0,0 +1,111 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.storage.aliyun;
+
+import com.aliyun.oss.OSS;
+import com.fasterxml.jackson.core.Version;
+import com.fasterxml.jackson.databind.Module;
+import com.google.common.collect.ImmutableList;
+import com.google.inject.Binder;
+import com.google.inject.Provides;
+import com.google.inject.multibindings.MapBinder;
+import org.apache.druid.data.SearchableVersionedDataFinder;
+import org.apache.druid.data.input.aliyun.OssClientConfig;
+import org.apache.druid.guice.Binders;
+import org.apache.druid.guice.JsonConfigProvider;
+import org.apache.druid.guice.LazySingleton;
+import org.apache.druid.initialization.DruidModule;
+
+import java.util.List;
+
+public class OssStorageDruidModule implements DruidModule
+{
+ public static final String SCHEME = "aliyun-oss";
+ public static final String SCHEME_S3N = "aliyun-oss_3n";
+ public static final String SCHEME_ZIP = "aliyun-oss_zip";
+
+ @Override
+ public List<? extends Module> getJacksonModules()
+ {
+ return ImmutableList.of(
+ new Module()
+ {
+ @Override
+ public String getModuleName()
+ {
+ return "DruidOss-" + System.identityHashCode(this);
+ }
+
+ @Override
+ public Version version()
+ {
+ return Version.unknownVersion();
+ }
+
+ @Override
+ public void setupModule(SetupContext context)
+ {
+ context.registerSubtypes(OssLoadSpec.class);
+ }
+ }
+ );
+ }
+
+ @Override
+ public void configure(Binder binder)
+ {
+ MapBinder.newMapBinder(binder, String.class, SearchableVersionedDataFinder.class)
+ .addBinding(SCHEME)
+ .to(OssTimestampVersionedDataFinder.class)
+ .in(LazySingleton.class);
+ MapBinder.newMapBinder(binder, String.class, SearchableVersionedDataFinder.class)
+ .addBinding(SCHEME_S3N)
Review comment:
If there's only one aliyun URI scheme, this binding isn't needed
##########
File path: extensions-contrib/aliyun-oss-extensions/src/main/java/org/apache/druid/storage/aliyun/OssUtils.java
##########
@@ -0,0 +1,279 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.storage.aliyun;
+
+import com.aliyun.oss.OSS;
+import com.aliyun.oss.OSSException;
+import com.aliyun.oss.model.AccessControlList;
+import com.aliyun.oss.model.CannedAccessControlList;
+import com.aliyun.oss.model.DeleteObjectsRequest;
+import com.aliyun.oss.model.ListObjectsRequest;
+import com.aliyun.oss.model.OSSObjectSummary;
+import com.aliyun.oss.model.ObjectListing;
+import com.aliyun.oss.model.PutObjectRequest;
+import com.google.common.base.Joiner;
+import com.google.common.base.Predicate;
+import com.google.common.collect.ImmutableList;
+import org.apache.druid.data.input.impl.CloudObjectLocation;
+import org.apache.druid.java.util.common.ISE;
+import org.apache.druid.java.util.common.RetryUtils;
+import org.apache.druid.java.util.common.RetryUtils.Task;
+import org.apache.druid.java.util.common.StringUtils;
+import org.apache.druid.java.util.common.logger.Logger;
+
+import java.io.File;
+import java.io.IOException;
+import java.net.URI;
+import java.util.ArrayList;
+import java.util.Iterator;
+import java.util.List;
+
+public class OssUtils
+{
+ private static final String SCHEME = OssStorageDruidModule.SCHEME;
+ private static final Joiner JOINER = Joiner.on("/").skipNulls();
+ private static final Logger log = new Logger(OssUtils.class);
+
+
+ static boolean isServiceExceptionRecoverable(OSSException ex)
+ {
+ final boolean isIOException = ex.getCause() instanceof IOException;
+ final boolean isTimeout = "RequestTimeout".equals(ex.getErrorCode());
+ final boolean badStatusCode = false; //ex. == 400 || ex.getStatusCode() == 403 || ex.getStatusCode() == 404;
+ return !badStatusCode && (isIOException || isTimeout);
+ }
+
+ public static final Predicate<Throwable> RETRYABLE = new Predicate<Throwable>()
+ {
+ @Override
+ public boolean apply(Throwable e)
+ {
+ if (e == null) {
+ return false;
+ } else if (e instanceof IOException) {
+ return true;
+ } else if (e instanceof OSSException) {
+ return isServiceExceptionRecoverable((OSSException) e);
+ } else {
+ return apply(e.getCause());
+ }
+ }
+ };
+
+ /**
+ * Retries aliyun-oss operations that fail due to io-related exceptions. Service-level exceptions (access denied, file not
+ * found, etc) are not retried.
+ */
+ static <T> T retry(Task<T> f) throws Exception
+ {
+ return RetryUtils.retry(f, RETRYABLE, RetryUtils.DEFAULT_MAX_TRIES);
+ }
+
+ static boolean isObjectInBucketIgnoringPermission(
+ OSS client,
+ String bucketName,
+ String objectKey
+ )
+ {
+ try {
+ return client.doesObjectExist(bucketName, objectKey);
+ }
+ catch (OSSException e) {
+ if (e.getErrorCode().equals("NoSuchKey")) {
+ // Object is inaccessible to current user, but does exist.
+ return true;
+ }
+ // Something else has gone wrong
+ throw e;
+ }
+ }
+
+ /**
+ * Create an iterator over a set of aliyun-oss objects specified by a set of prefixes.
+ * <p>
+ * For each provided prefix URI, the iterator will walk through all objects that are in the same bucket as the
+ * provided URI and whose keys start with that URI's path, except for directory placeholders (which will be
+ * ignored). The iterator is computed incrementally by calling {@link OSS#listObjects} for
+ * each prefix in batches of {@param maxListLength}. The first call is made at the same time the iterator is
+ * constructed.
+ */
+ public static Iterator<OSSObjectSummary> objectSummaryIterator(
+ final OSS client,
+ final Iterable<URI> prefixes,
+ final int maxListingLength
+ )
+ {
+ return new ObjectSummaryIterator(client, prefixes, maxListingLength);
+ }
+
+ /**
+ * Create an {@link URI} from the given {@link OSSObjectSummary}. The result URI is composed as below.
+ *
+ * <pre>
+ * {@code aliyun-oss://{BUCKET_NAME}/{OBJECT_KEY}}
+ * </pre>
+ */
+ public static URI summaryToUri(OSSObjectSummary object)
+ {
+ return summaryToCloudObjectLocation(object).toUri(SCHEME);
+ }
+
+ public static CloudObjectLocation summaryToCloudObjectLocation(OSSObjectSummary object)
+ {
+ return new CloudObjectLocation(object.getBucketName(), object.getKey());
+ }
+
+ static String constructSegmentPath(String baseKey, String storageDir)
+ {
+ return JOINER.join(
+ baseKey.isEmpty() ? null : baseKey,
+ storageDir
+ ) + "/index.zip";
+ }
+
+ static CannedAccessControlList grantFullControlToBucketOwner(OSS client, String bucket)
+ {
+ final AccessControlList acl = client.getBucketAcl(bucket);
+ return acl.getCannedACL();
+ //acl.grantAllPermissions(new Grant(new Grantee(acl.getOwner().getId()), Permission.FullControl));
Review comment:
Please remove the unused code
##########
File path: extensions-contrib/aliyun-oss-extensions/src/main/java/org/apache/druid/firehose/aliyun/StaticOssFirehoseFactory.java
##########
@@ -0,0 +1,237 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.firehose.aliyun;
+
+import com.aliyun.oss.OSS;
+import com.aliyun.oss.OSSException;
+import com.aliyun.oss.model.GetObjectRequest;
+import com.aliyun.oss.model.OSSObject;
+import com.aliyun.oss.model.OSSObjectSummary;
+import com.fasterxml.jackson.annotation.JacksonInject;
+import com.fasterxml.jackson.annotation.JsonCreator;
+import com.fasterxml.jackson.annotation.JsonProperty;
+import com.google.common.base.Preconditions;
+import com.google.common.base.Predicate;
+import org.apache.druid.data.input.FiniteFirehoseFactory;
+import org.apache.druid.data.input.InputSplit;
+import org.apache.druid.data.input.impl.StringInputRowParser;
+import org.apache.druid.data.input.impl.prefetch.PrefetchableTextFilesFirehoseFactory;
+import org.apache.druid.java.util.common.IAE;
+import org.apache.druid.java.util.common.ISE;
+import org.apache.druid.java.util.common.logger.Logger;
+import org.apache.druid.storage.aliyun.OssUtils;
+import org.apache.druid.utils.CompressionUtils;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.net.URI;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Objects;
+import java.util.stream.Collectors;
+
+/**
+ * Builds firehoses that read from a predefined list of aliyun-oss objects and then dry up.
+ */
+public class StaticOssFirehoseFactory extends PrefetchableTextFilesFirehoseFactory<URI>
+{
+ private static final Logger log = new Logger(StaticOssFirehoseFactory.class);
+ private static final int MAX_LISTING_LENGTH = 1024;
+
+ private final OSS client;
+ private final List<URI> uris;
+ private final List<URI> prefixes;
+
+ @JsonCreator
+ public StaticOssFirehoseFactory(
+ @JacksonInject OSS client,
+ @JsonProperty("uris") List<URI> uris,
+ @JsonProperty("prefixes") List<URI> prefixes,
+ @JsonProperty("maxCacheCapacityBytes") Long maxCacheCapacityBytes,
+ @JsonProperty("maxFetchCapacityBytes") Long maxFetchCapacityBytes,
+ @JsonProperty("prefetchTriggerBytes") Long prefetchTriggerBytes,
+ @JsonProperty("fetchTimeout") Long fetchTimeout,
+ @JsonProperty("maxFetchRetry") Integer maxFetchRetry
+ )
+ {
+ super(maxCacheCapacityBytes, maxFetchCapacityBytes, prefetchTriggerBytes, fetchTimeout, maxFetchRetry);
+ this.client = Preconditions.checkNotNull(client, "client");
+ this.uris = uris == null ? new ArrayList<>() : uris;
+ this.prefixes = prefixes == null ? new ArrayList<>() : prefixes;
+
+ if (!this.uris.isEmpty() && !this.prefixes.isEmpty()) {
+ throw new IAE("uris and prefixes cannot be used together");
+ }
+
+ if (this.uris.isEmpty() && this.prefixes.isEmpty()) {
+ throw new IAE("uris or prefixes must be specified");
+ }
+
+ for (final URI inputURI : this.uris) {
+ Preconditions.checkArgument("aliyun-oss".equals(inputURI.getScheme()), "input uri scheme == aliyun-oss (%s)", inputURI);
Review comment:
Suggest making `"aliyun-oss"` a constant
##########
File path: extensions-contrib/aliyun-oss-extensions/src/main/java/org/apache/druid/storage/aliyun/OssDataSegmentPusher.java
##########
@@ -0,0 +1,131 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.storage.aliyun;
+
+import com.aliyun.oss.OSS;
+import com.aliyun.oss.OSSException;
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.ImmutableMap;
+import com.google.inject.Inject;
+import org.apache.druid.java.util.common.StringUtils;
+import org.apache.druid.java.util.emitter.EmittingLogger;
+import org.apache.druid.segment.SegmentUtils;
+import org.apache.druid.segment.loading.DataSegmentPusher;
+import org.apache.druid.timeline.DataSegment;
+import org.apache.druid.utils.CompressionUtils;
+
+import java.io.File;
+import java.io.IOException;
+import java.net.URI;
+import java.util.List;
+import java.util.Map;
+
+public class OssDataSegmentPusher implements DataSegmentPusher
+{
+ private static final EmittingLogger log = new EmittingLogger(OssDataSegmentPusher.class);
+
+ private final OSS client;
+ private final OssStorageConfig config;
+
+ @Inject
+ public OssDataSegmentPusher(
+ OSS client,
+ OssStorageConfig config
+ )
+ {
+ this.client = client;
+ this.config = config;
+ }
+
+ @Override
+ public String getPathForHadoop()
+ {
+ return StringUtils.format("%s/%s", config.getBucket(), config.getPrefix());
Review comment:
Have you tested this with a Hadoop cluster before? If so, cool. If not, that's fine, we can just document that Hadoop is untested with it.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] jon-wei commented on pull request #9898: support Aliyun OSS service as deep storage
Posted by GitBox <gi...@apache.org>.
jon-wei commented on pull request #9898:
URL: https://github.com/apache/druid/pull/9898#issuecomment-650638936
Even if OSS and S3 are supposedly 100% compatible right now, that feels like a fragile thing to rely on (and do we know for sure there aren't bugs/small implementation differences that will cause us problems if we tried to share code or abstractions?)
> So, I cannot tell what the right way is until we figure out the compatibility level between OSS and S3
Since they're operated and maintained by different organizations, my view is that we should just treat them as if they were non-compatible and totally different things, I don't actually mind code duplication all that much for something like this.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] jon-wei commented on a change in pull request #9898: support Aliyun OSS service as deep storage
Posted by GitBox <gi...@apache.org>.
jon-wei commented on a change in pull request #9898:
URL: https://github.com/apache/druid/pull/9898#discussion_r445754298
##########
File path: extensions-contrib/aliyun-oss-extensions/src/main/java/org/apache/druid/storage/aliyun/ObjectSummaryIterator.java
##########
@@ -0,0 +1,163 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.storage.aliyun;
+
+import com.aliyun.oss.OSS;
+import com.aliyun.oss.OSSException;
+import com.aliyun.oss.model.ListObjectsRequest;
+import com.aliyun.oss.model.OSSObjectSummary;
+import com.aliyun.oss.model.ObjectListing;
+import org.apache.druid.java.util.common.RE;
+
+import java.net.URI;
+import java.util.Iterator;
+import java.util.NoSuchElementException;
+
+/**
+ * Iterator class used by {@link OssUtils#objectSummaryIterator}.
+ * <p>
+ * As required by the specification of that method, this iterator is computed incrementally in batches of
+ * {@code maxListLength}. The first call is made at the same time the iterator is constructed.
+ *
+ */
+public class ObjectSummaryIterator implements Iterator<OSSObjectSummary>
+{
+ private final OSS client;
+ private final Iterator<URI> prefixesIterator;
+ private final int maxListingLength;
+
+ private ListObjectsRequest request;
+ private ObjectListing result;
+ private Iterator<OSSObjectSummary> objectSummaryIterator;
+ private OSSObjectSummary currentObjectSummary;
+
+ ObjectSummaryIterator(
+ final OSS client,
+ final Iterable<URI> prefixes,
+ final int maxListingLength
+ )
+ {
+ this.client = client;
+ this.prefixesIterator = prefixes.iterator();
+ this.maxListingLength = maxListingLength;
+
+ prepareNextRequest();
+ fetchNextBatch();
+ advanceObjectSummary();
+ }
+
+ @Override
+ public boolean hasNext()
+ {
+ return currentObjectSummary != null;
+ }
+
+ @Override
+ public OSSObjectSummary next()
+ {
+ if (currentObjectSummary == null) {
+ throw new NoSuchElementException();
+ }
+
+ final OSSObjectSummary retVal = currentObjectSummary;
+ advanceObjectSummary();
+ return retVal;
+ }
+
+ private void prepareNextRequest()
+ {
+ final URI currentUri = prefixesIterator.next();
+ final String currentBucket = currentUri.getAuthority();
+ final String currentPrefix = OssUtils.extractKey(currentUri);
+
+ request = new ListObjectsRequest(currentBucket, currentPrefix, null, null, maxListingLength);
+ }
+
+ private void fetchNextBatch()
+ {
+ try {
+ result = OssUtils.retry(() -> client.listObjects(request));
+ request.setMarker(result.getNextMarker());
+ objectSummaryIterator = result.getObjectSummaries().iterator();
+ }
+ catch (OSSException e) {
+ throw new RE(
+ e,
+ "Failed to get object summaries from S3 bucket[%s], prefix[%s]; S3 error: %s",
+ request.getBucketName(),
+ request.getPrefix(),
+ e.getMessage()
+ );
+ }
+ catch (Exception e) {
+ throw new RE(
+ e,
+ "Failed to get object summaries from S3 bucket[%s], prefix[%s]",
+ request.getBucketName(),
+ request.getPrefix()
+ );
+ }
+ }
+
+ /**
+ * Advance objectSummaryIterator to the next non-placeholder, updating "currentObjectSummary".
+ */
+ private void advanceObjectSummary()
+ {
+ while (objectSummaryIterator.hasNext() || result.isTruncated() || prefixesIterator.hasNext()) {
+ while (objectSummaryIterator.hasNext()) {
+ currentObjectSummary = objectSummaryIterator.next();
+ // skips directories and empty objects
+ if (!isDirectoryPlaceholder(currentObjectSummary) && currentObjectSummary.getSize() > 0) {
+ return;
+ }
+ }
+
+ // Exhausted "objectSummaryIterator" without finding a non-placeholder.
+ if (result.isTruncated()) {
+ fetchNextBatch();
+ } else if (prefixesIterator.hasNext()) {
+ prepareNextRequest();
+ fetchNextBatch();
+ }
+ }
+
+ // Truly nothing left to read.
+ currentObjectSummary = null;
+ }
+
+ /**
+ * Checks if a given object is a directory placeholder and should be ignored.
Review comment:
@FrankChen021 The code is still derived from jets3t code, so it must be accounted for in LICENSE.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] jihoonson commented on a change in pull request #9898: support Aliyun OSS service as deep storage
Posted by GitBox <gi...@apache.org>.
jihoonson commented on a change in pull request #9898:
URL: https://github.com/apache/druid/pull/9898#discussion_r447407836
##########
File path: extensions-contrib/aliyun-oss-extensions/src/test/java/org/apache/druid/storage/aliyun/OssTaskLogsTest.java
##########
@@ -0,0 +1,336 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.storage.aliyun;
+
+import com.aliyun.oss.ClientException;
+import com.aliyun.oss.OSS;
+import com.aliyun.oss.model.AccessControlList;
+import com.aliyun.oss.model.DeleteObjectsRequest;
+import com.aliyun.oss.model.Grant;
+import com.aliyun.oss.model.OSSObjectSummary;
+import com.aliyun.oss.model.Owner;
+import com.aliyun.oss.model.PutObjectRequest;
+import com.aliyun.oss.model.PutObjectResult;
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.ImmutableMap;
+import org.apache.druid.common.utils.CurrentTimeMillisSupplier;
+import org.apache.druid.java.util.common.StringUtils;
+import org.easymock.EasyMock;
+import org.easymock.EasyMockRunner;
+import org.easymock.EasyMockSupport;
+import org.easymock.Mock;
+import org.junit.Assert;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.rules.TemporaryFolder;
+import org.junit.runner.RunWith;
+
+import java.io.File;
+import java.io.IOException;
+import java.net.URI;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
+
+@RunWith(EasyMockRunner.class)
+public class OssTaskLogsTest extends EasyMockSupport
+{
+
+ private static final String KEY_1 = "key1";
+ private static final String KEY_2 = "key2";
+ private static final String TEST_BUCKET = "test_bucket";
+ private static final String TEST_PREFIX = "test_prefix";
+ private static final URI PREFIX_URI = URI.create(StringUtils.format("s3://%s/%s", TEST_BUCKET, TEST_PREFIX));
Review comment:
Should be `oss` instead of `s3`.
##########
File path: extensions-contrib/aliyun-oss-extensions/src/main/java/org/apache/druid/storage/aliyun/OssStorageDruidModule.java
##########
@@ -0,0 +1,106 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.storage.aliyun;
+
+import com.aliyun.oss.OSS;
+import com.fasterxml.jackson.core.Version;
+import com.fasterxml.jackson.databind.Module;
+import com.google.common.collect.ImmutableList;
+import com.google.inject.Binder;
+import com.google.inject.Provides;
+import com.google.inject.multibindings.MapBinder;
+import org.apache.druid.data.SearchableVersionedDataFinder;
+import org.apache.druid.data.input.aliyun.OssClientConfig;
+import org.apache.druid.guice.Binders;
+import org.apache.druid.guice.JsonConfigProvider;
+import org.apache.druid.guice.LazySingleton;
+import org.apache.druid.initialization.DruidModule;
+
+import java.util.List;
+
+public class OssStorageDruidModule implements DruidModule
+{
+ public static final String SCHEME = "oss";
+ public static final String SCHEME_ZIP = "oss_zip";
+
+ @Override
+ public List<? extends Module> getJacksonModules()
+ {
+ return ImmutableList.of(
+ new Module()
+ {
+ @Override
+ public String getModuleName()
+ {
+ return "DruidOss-" + System.identityHashCode(this);
Review comment:
nit: hmm, `DruidOss` sounds like `Druid open source software`. Maybe `DruidAliyunOss` is better?
##########
File path: extensions-contrib/aliyun-oss-extensions/src/test/java/org/apache/druid/storage/aliyun/OssDataSegmentMoverTest.java
##########
@@ -0,0 +1,266 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.storage.aliyun;
+
+import com.aliyun.oss.OSSClient;
+import com.aliyun.oss.OSSException;
+import com.aliyun.oss.model.CopyObjectRequest;
+import com.aliyun.oss.model.CopyObjectResult;
+import com.aliyun.oss.model.ListObjectsRequest;
+import com.aliyun.oss.model.OSSObjectSummary;
+import com.aliyun.oss.model.ObjectListing;
+import com.aliyun.oss.model.PutObjectResult;
+import com.aliyun.oss.model.StorageClass;
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.ImmutableMap;
+import org.apache.druid.java.util.common.Intervals;
+import org.apache.druid.java.util.common.MapUtils;
+import org.apache.druid.segment.loading.SegmentLoadingException;
+import org.apache.druid.timeline.DataSegment;
+import org.apache.druid.timeline.partition.NoneShardSpec;
+import org.junit.Assert;
+import org.junit.Test;
+
+import java.io.File;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Map;
+import java.util.Set;
+
+public class OssDataSegmentMoverTest
+{
+ private static final DataSegment SOURCE_SEGMENT = new DataSegment(
+ "test",
+ Intervals.of("2013-01-01/2013-01-02"),
+ "1",
+ ImmutableMap.of(
+ "key",
+ "baseKey/test/2013-01-01T00:00:00.000Z_2013-01-02T00:00:00.000Z/1/0/index.zip",
+ "bucket",
+ "main"
+ ),
+ ImmutableList.of("dim1", "dim1"),
+ ImmutableList.of("metric1", "metric2"),
+ NoneShardSpec.instance(),
+ 0,
+ 1
+ );
+
+ @Test
+ public void testMove() throws Exception
+ {
+ MockClient mockClient = new MockClient();
+ OssDataSegmentMover mover = new OssDataSegmentMover(mockClient, new OssStorageConfig());
+
+ mockClient.putObject(
+ "main",
+ "baseKey/test/2013-01-01T00:00:00.000Z_2013-01-02T00:00:00.000Z/1/0/index.zip"
+ );
+
+ DataSegment movedSegment = mover.move(
+ SOURCE_SEGMENT,
+ ImmutableMap.of("baseKey", "targetBaseKey", "bucket", "archive")
+ );
+
+ Map<String, Object> targetLoadSpec = movedSegment.getLoadSpec();
+ Assert.assertEquals(
+ "targetBaseKey/test/2013-01-01T00:00:00.000Z_2013-01-02T00:00:00.000Z/1/0/index.zip",
+ MapUtils.getString(targetLoadSpec, "key")
+ );
+ Assert.assertEquals("archive", MapUtils.getString(targetLoadSpec, "bucket"));
+ Assert.assertTrue(mockClient.didMove());
+ }
+
+ @Test
+ public void testMoveNoop() throws Exception
+ {
+ MockClient mockS3Client = new MockClient();
Review comment:
nit: I see many variable names containing `s3`. They are all found in only unit tests and don't affect to production code, but would be nice to clean them up.
##########
File path: extensions-contrib/aliyun-oss-extensions/src/main/java/org/apache/druid/firehose/aliyun/StaticOssFirehoseFactory.java
##########
@@ -0,0 +1,243 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.firehose.aliyun;
+
+import com.aliyun.oss.OSS;
+import com.aliyun.oss.OSSException;
+import com.aliyun.oss.model.GetObjectRequest;
+import com.aliyun.oss.model.OSSObject;
+import com.aliyun.oss.model.OSSObjectSummary;
+import com.fasterxml.jackson.annotation.JacksonInject;
+import com.fasterxml.jackson.annotation.JsonCreator;
+import com.fasterxml.jackson.annotation.JsonProperty;
+import com.google.common.base.Preconditions;
+import com.google.common.base.Predicate;
+import org.apache.druid.data.input.FiniteFirehoseFactory;
+import org.apache.druid.data.input.InputSplit;
+import org.apache.druid.data.input.impl.StringInputRowParser;
+import org.apache.druid.data.input.impl.prefetch.PrefetchableTextFilesFirehoseFactory;
+import org.apache.druid.java.util.common.IAE;
+import org.apache.druid.java.util.common.ISE;
+import org.apache.druid.java.util.common.logger.Logger;
+import org.apache.druid.storage.aliyun.OssStorageDruidModule;
+import org.apache.druid.storage.aliyun.OssUtils;
+import org.apache.druid.utils.CompressionUtils;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.net.URI;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Objects;
+import java.util.stream.Collectors;
+
+/**
+ * Builds firehoses that read from a predefined list of aliyun OSS objects and then dry up.
+ */
+public class StaticOssFirehoseFactory extends PrefetchableTextFilesFirehoseFactory<URI>
Review comment:
nit: `FiniteFirehoseFactory` is deprecated and it's not recommend to implement it. You can keep this implementation if you want, but it can be deleted sooner or later.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] clintropolis merged pull request #9898: support Aliyun OSS service as deep storage
Posted by GitBox <gi...@apache.org>.
clintropolis merged pull request #9898:
URL: https://github.com/apache/druid/pull/9898
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] jihoonson commented on pull request #9898: support Aliyun OSS service as deep storage
Posted by GitBox <gi...@apache.org>.
jihoonson commented on pull request #9898:
URL: https://github.com/apache/druid/pull/9898#issuecomment-650459835
@FrankChen021 can you explain more detailed motivation behind this PR? I haven't used Aliyun OSS, but it seems like it provides S3-compatible APIs (https://www.alibabacloud.com/help/doc-detail/64919.htm#title-37m-7gl-xy2). Can we use Aliyun OSS as deep storage with our s3 extension? Or do we need something special to do it but missing in the s3 extension?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] FrankChen021 edited a comment on pull request #9898: support Aliyun OSS service as deep storage
Posted by GitBox <gi...@apache.org>.
FrankChen021 edited a comment on pull request #9898:
URL: https://github.com/apache/druid/pull/9898#issuecomment-649602917
Hi @jon-wei , I've updated the code, and re-tested all core functions. IT testcases runs OK, and real time ingestion task, compact task, kill task, index task log persistence runs well in our test cluster. Please check it again.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] FrankChen021 commented on pull request #9898: support Aliyun OSS service as deep storage
Posted by GitBox <gi...@apache.org>.
FrankChen021 commented on pull request #9898:
URL: https://github.com/apache/druid/pull/9898#issuecomment-650705280
Hi @jihoonson , CI excludes all other deep storage IT testcases through following property. I didn't found where this property is defined in project. Is it configured in CI ? The test group of aliyun OSS extension is `aliyun-oss-deep-storage`, which needs to be excluded from CI too.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] jihoonson commented on pull request #9898: support Aliyun OSS service as deep storage
Posted by GitBox <gi...@apache.org>.
jihoonson commented on pull request #9898:
URL: https://github.com/apache/druid/pull/9898#issuecomment-650608153
@FrankChen021 thank you for the explanation.
> Hi @jihoonson , s3 extension does not work with aliyun OSS although OSS provides compatibility with s3. I tested to utilize OSS with s3 extension in the past, but the `putObject` returned HTTP 400 error, saying there were some unsupported HEADERS in the request generated by s3 SDK. I didn't explore that problem very deeply, I guess the latest s3 SDK adds some new HEADERS which are not compatible in OSS.
Do you mean the OSS APIs are not actually compatible to S3 APIs? I think this is the most important question to decide what we should do here, and so we should figure it out before we proceed. Do you remember what the exact error was? Probably it would be one of the errors listed in https://www.alibabacloud.com/help/doc-detail/32005.htm. If you don't remember, since you seem to already have a testing environment, would you please test it again if it's not that hard for you?
> the APIs between OSS and s3 are very similar, even their names. but I don't think it's a good idea to use one extension to support both OSS and s3, or use some abstraction method to extract the same code between s3 and OSS. Actually, this kind of abstraction has been done by deep storage framework, so every implementation interacts with its underlying storage API. As s3 and OSS are provided by two companies, it's better to provide two extensions respectively so that they can evolve independently.
I agree abstraction layer is not a good idea. Since an extension cannot depend on another, an abstraction layer beween s3 and OSS requires to be in the core which I don't think is an option. I haven't looked at the PR very closely, but it looks almost identical to the s3 extension. An obvious downside in developing a new extension is the increasing maintenance burden; when we modify the s3 extension, we have to check the OSS extension if it's reasonable to apply the same change, or vice versa. However, there is a problem in using s3 extension for OSS as well. Whenever we modify the s3 extension, we have to check the change is causing any issue in using OSS. So, I cannot tell what the right way is until we figure out the compatibility level between OSS and S3. It makes sense to me to develop a new extension if one of the following statements is true.
- OSS APIs are not actually compatible to S3 APIs. They look compatible based on their docs (with small differences in permission system), but it might not be true based on your comment above.
- The APIs are compatible in most cases, but there are some exceptions in particular APIs which we are using.
- The APIs are completely compatible now, but OSS is planning to stop providing s3-compatibility. Is there anything about this you are aware of?
The first 2 statements can be verified through some testings. For the last statement, I couldn't find anything in 1 min with my googling skill.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] FrankChen021 commented on a change in pull request #9898: support Aliyun OSS service as deep storage
Posted by GitBox <gi...@apache.org>.
FrankChen021 commented on a change in pull request #9898:
URL: https://github.com/apache/druid/pull/9898#discussion_r448146198
##########
File path: integration-tests/src/test/java/org/apache/druid/tests/TestNGGroup.java
##########
@@ -76,6 +76,13 @@
*/
public static final String AZURE_DEEP_STORAGE = "azure-deep-storage";
+ /**
+ * This group is not part of CI. To run this group, azure configs/credentials for your azure must be provided in a file.
Review comment:
it's been fixed
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] asdf2014 commented on pull request #9898: support Aliyun OSS service as deep storage
Posted by GitBox <gi...@apache.org>.
asdf2014 commented on pull request #9898:
URL: https://github.com/apache/druid/pull/9898#issuecomment-650711249
@FrankChen021 You can exclude it by modifying the configuration file of Travis. FYI, https://github.com/apache/druid/blob/master/.travis.yml#L392
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] FrankChen021 commented on a change in pull request #9898: support Aliyun OSS service as deep storage
Posted by GitBox <gi...@apache.org>.
FrankChen021 commented on a change in pull request #9898:
URL: https://github.com/apache/druid/pull/9898#discussion_r445616439
##########
File path: extensions-contrib/aliyun-oss-extensions/src/main/java/org/apache/druid/storage/aliyun/OssDataSegmentPusher.java
##########
@@ -0,0 +1,131 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.storage.aliyun;
+
+import com.aliyun.oss.OSS;
+import com.aliyun.oss.OSSException;
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.ImmutableMap;
+import com.google.inject.Inject;
+import org.apache.druid.java.util.common.StringUtils;
+import org.apache.druid.java.util.emitter.EmittingLogger;
+import org.apache.druid.segment.SegmentUtils;
+import org.apache.druid.segment.loading.DataSegmentPusher;
+import org.apache.druid.timeline.DataSegment;
+import org.apache.druid.utils.CompressionUtils;
+
+import java.io.File;
+import java.io.IOException;
+import java.net.URI;
+import java.util.List;
+import java.util.Map;
+
+public class OssDataSegmentPusher implements DataSegmentPusher
+{
+ private static final EmittingLogger log = new EmittingLogger(OssDataSegmentPusher.class);
+
+ private final OSS client;
+ private final OssStorageConfig config;
+
+ @Inject
+ public OssDataSegmentPusher(
+ OSS client,
+ OssStorageConfig config
+ )
+ {
+ this.client = client;
+ this.config = config;
+ }
+
+ @Override
+ public String getPathForHadoop()
+ {
+ return StringUtils.format("%s/%s", config.getBucket(), config.getPrefix());
Review comment:
It has not been tested in hadoop cluster yet, because in our test environment, all data are ingested from kafka.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] FrankChen021 commented on pull request #9898: support Aliyun OSS service as deep storage
Posted by GitBox <gi...@apache.org>.
FrankChen021 commented on pull request #9898:
URL: https://github.com/apache/druid/pull/9898#issuecomment-650678797
@jihoonson OK, I'll fix the doc.
And for the IT test case, it seems like `BUCKET` and `PATH` are not configured. In my environment, I put these two parameters and other OSS related configuration parameters in a config file and use `-Ddruid.test.config.configFile` to reference that file. I don't know how to configure the IT test cases in CI environment to use OSS. Do IT test cases need real s3 or google cloud storage or Azure or OSS service ?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] FrankChen021 commented on a change in pull request #9898: support Aliyun OSS service as deep storage
Posted by GitBox <gi...@apache.org>.
FrankChen021 commented on a change in pull request #9898:
URL: https://github.com/apache/druid/pull/9898#discussion_r448150182
##########
File path: extensions-contrib/aliyun-oss-extensions/src/main/java/org/apache/druid/storage/aliyun/OssStorageDruidModule.java
##########
@@ -0,0 +1,106 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.storage.aliyun;
+
+import com.aliyun.oss.OSS;
+import com.fasterxml.jackson.core.Version;
+import com.fasterxml.jackson.databind.Module;
+import com.google.common.collect.ImmutableList;
+import com.google.inject.Binder;
+import com.google.inject.Provides;
+import com.google.inject.multibindings.MapBinder;
+import org.apache.druid.data.SearchableVersionedDataFinder;
+import org.apache.druid.data.input.aliyun.OssClientConfig;
+import org.apache.druid.guice.Binders;
+import org.apache.druid.guice.JsonConfigProvider;
+import org.apache.druid.guice.LazySingleton;
+import org.apache.druid.initialization.DruidModule;
+
+import java.util.List;
+
+public class OssStorageDruidModule implements DruidModule
+{
+ public static final String SCHEME = "oss";
+ public static final String SCHEME_ZIP = "oss_zip";
+
+ @Override
+ public List<? extends Module> getJacksonModules()
+ {
+ return ImmutableList.of(
+ new Module()
+ {
+ @Override
+ public String getModuleName()
+ {
+ return "DruidOss-" + System.identityHashCode(this);
Review comment:
it's been fixed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] fjy commented on pull request #9898: support Aliyun OSS service as deep storage
Posted by GitBox <gi...@apache.org>.
fjy commented on pull request #9898:
URL: https://github.com/apache/druid/pull/9898#issuecomment-647609774
@FrankChen021 can you check out the feedback?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] FrankChen021 commented on pull request #9898: support Aliyun OSS service as deep storage
Posted by GitBox <gi...@apache.org>.
FrankChen021 commented on pull request #9898:
URL: https://github.com/apache/druid/pull/9898#issuecomment-634001052
@clintropolis CI reports insufficient unit test line/function/branch coverage. Is it mandatory to meet that requirement ?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] FrankChen021 commented on pull request #9898: support Aliyun OSS service as deep storage
Posted by GitBox <gi...@apache.org>.
FrankChen021 commented on pull request #9898:
URL: https://github.com/apache/druid/pull/9898#issuecomment-650667153
@jon-wei I agree with you
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] FrankChen021 commented on pull request #9898: support Aliyun OSS service as deep storage
Posted by GitBox <gi...@apache.org>.
FrankChen021 commented on pull request #9898:
URL: https://github.com/apache/druid/pull/9898#issuecomment-649602917
Hi @jon-wei , I've updated the code, and re-tested all core functions. IT testcases runs OK, and real time ingestion task, compact task, kill task runs well in our test cluster. Please check it again.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] FrankChen021 commented on a change in pull request #9898: support Aliyun OSS service as deep storage
Posted by GitBox <gi...@apache.org>.
FrankChen021 commented on a change in pull request #9898:
URL: https://github.com/apache/druid/pull/9898#discussion_r445615267
##########
File path: extensions-contrib/aliyun-oss-extensions/src/main/java/org/apache/druid/storage/aliyun/ObjectSummaryIterator.java
##########
@@ -0,0 +1,163 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.storage.aliyun;
+
+import com.aliyun.oss.OSS;
+import com.aliyun.oss.OSSException;
+import com.aliyun.oss.model.ListObjectsRequest;
+import com.aliyun.oss.model.OSSObjectSummary;
+import com.aliyun.oss.model.ObjectListing;
+import org.apache.druid.java.util.common.RE;
+
+import java.net.URI;
+import java.util.Iterator;
+import java.util.NoSuchElementException;
+
+/**
+ * Iterator class used by {@link OssUtils#objectSummaryIterator}.
+ * <p>
+ * As required by the specification of that method, this iterator is computed incrementally in batches of
+ * {@code maxListLength}. The first call is made at the same time the iterator is constructed.
+ *
+ */
+public class ObjectSummaryIterator implements Iterator<OSSObjectSummary>
+{
+ private final OSS client;
+ private final Iterator<URI> prefixesIterator;
+ private final int maxListingLength;
+
+ private ListObjectsRequest request;
+ private ObjectListing result;
+ private Iterator<OSSObjectSummary> objectSummaryIterator;
+ private OSSObjectSummary currentObjectSummary;
+
+ ObjectSummaryIterator(
+ final OSS client,
+ final Iterable<URI> prefixes,
+ final int maxListingLength
+ )
+ {
+ this.client = client;
+ this.prefixesIterator = prefixes.iterator();
+ this.maxListingLength = maxListingLength;
+
+ prepareNextRequest();
+ fetchNextBatch();
+ advanceObjectSummary();
+ }
+
+ @Override
+ public boolean hasNext()
+ {
+ return currentObjectSummary != null;
+ }
+
+ @Override
+ public OSSObjectSummary next()
+ {
+ if (currentObjectSummary == null) {
+ throw new NoSuchElementException();
+ }
+
+ final OSSObjectSummary retVal = currentObjectSummary;
+ advanceObjectSummary();
+ return retVal;
+ }
+
+ private void prepareNextRequest()
+ {
+ final URI currentUri = prefixesIterator.next();
+ final String currentBucket = currentUri.getAuthority();
+ final String currentPrefix = OssUtils.extractKey(currentUri);
+
+ request = new ListObjectsRequest(currentBucket, currentPrefix, null, null, maxListingLength);
+ }
+
+ private void fetchNextBatch()
+ {
+ try {
+ result = OssUtils.retry(() -> client.listObjects(request));
+ request.setMarker(result.getNextMarker());
+ objectSummaryIterator = result.getObjectSummaries().iterator();
+ }
+ catch (OSSException e) {
+ throw new RE(
+ e,
+ "Failed to get object summaries from S3 bucket[%s], prefix[%s]; S3 error: %s",
+ request.getBucketName(),
+ request.getPrefix(),
+ e.getMessage()
+ );
+ }
+ catch (Exception e) {
+ throw new RE(
+ e,
+ "Failed to get object summaries from S3 bucket[%s], prefix[%s]",
+ request.getBucketName(),
+ request.getPrefix()
+ );
+ }
+ }
+
+ /**
+ * Advance objectSummaryIterator to the next non-placeholder, updating "currentObjectSummary".
+ */
+ private void advanceObjectSummary()
+ {
+ while (objectSummaryIterator.hasNext() || result.isTruncated() || prefixesIterator.hasNext()) {
+ while (objectSummaryIterator.hasNext()) {
+ currentObjectSummary = objectSummaryIterator.next();
+ // skips directories and empty objects
+ if (!isDirectoryPlaceholder(currentObjectSummary) && currentObjectSummary.getSize() > 0) {
+ return;
+ }
+ }
+
+ // Exhausted "objectSummaryIterator" without finding a non-placeholder.
+ if (result.isTruncated()) {
+ fetchNextBatch();
+ } else if (prefixesIterator.hasNext()) {
+ prepareNextRequest();
+ fetchNextBatch();
+ }
+ }
+
+ // Truly nothing left to read.
+ currentObjectSummary = null;
+ }
+
+ /**
+ * Checks if a given object is a directory placeholder and should be ignored.
Review comment:
There's no directory placeholder in aliyun OSS, so I made change to this function. As a result, there's no need to add a declaration to license
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] FrankChen021 commented on pull request #9898: support Aliyun OSS service as deep storage
Posted by GitBox <gi...@apache.org>.
FrankChen021 commented on pull request #9898:
URL: https://github.com/apache/druid/pull/9898#issuecomment-650663610
Hi @jihoonson , using s3 extension with OSS, the exception is as below
I think OSS was compatible at the time the doc was created, but as the s3 SDK evolves, OSS doesn't keep the pace with it.
````
2020-06-28T00:14:03,975 ERROR [task-runner-0-priority-0] org.apache.druid.indexing.seekablestream.SeekableStreamIndexTaskRunner - Encountered exception while running task.
java.util.concurrent.ExecutionException: java.lang.RuntimeException: java.io.IOException: com.amazonaws.services.s3.model.AmazonS3Exception: A header you provided implies functionality that is not implemented. (Service: Amazon S3; Status Code: 400; Error Code: NotImplemented; Request ID: 5EF7E0CBAF285D3538BCFD5C; S3 Extended Request ID: frankchen021-public.oss-cn-hangzhou-internal.aliyuncs.com), S3 Extended Request ID: frankchen021.oss-cn-hangzhou-internal.aliyuncs.com
at com.google.common.util.concurrent.AbstractFuture$Sync.getValue(AbstractFuture.java:299) ~[guava-16.0.1.jar:?]
at com.google.common.util.concurrent.AbstractFuture$Sync.get(AbstractFuture.java:286) ~[guava-16.0.1.jar:?]
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:116) ~[guava-16.0.1.jar:?]
at org.apache.druid.indexing.seekablestream.SeekableStreamIndexTaskRunner.runInternal(SeekableStreamIndexTaskRunner.java:825) ~[druid-indexing-service-0.18.1.jar:0.18.1]
at org.apache.druid.indexing.seekablestream.SeekableStreamIndexTaskRunner.run(SeekableStreamIndexTaskRunner.java:277) [druid-indexing-service-0.18.1.jar:0.18.1]
at org.apache.druid.indexing.seekablestream.SeekableStreamIndexTask.run(SeekableStreamIndexTask.java:164) [druid-indexing-service-0.18.1.jar:0.18.1]
at org.apache.druid.indexing.overlord.SingleTaskBackgroundRunner$SingleTaskBackgroundRunnerCallable.call(SingleTaskBackgroundRunner.java:421) [druid-indexing-service-0.18.1.jar:0.18.1]
at org.apache.druid.indexing.overlord.SingleTaskBackgroundRunner$SingleTaskBackgroundRunnerCallable.call(SingleTaskBackgroundRunner.java:393) [druid-indexing-service-0.18.1.jar:0.18.1]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_252]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_252]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_252]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_252]
Caused by: java.lang.RuntimeException: java.io.IOException: com.amazonaws.services.s3.model.AmazonS3Exception: A header you provided implies functionality that is not implemented. (Service: Amazon S3; Status Code: 400; Error Code: NotImplemented; Request ID: 5EF7E0CBAF285D3538BCFD5C; S3 Extended Request ID: frankchen021.oss-cn-hangzhou-internal.aliyuncs.com), S3 Extended Request ID: frankchen021-public.oss-cn-hangzhou-internal.aliyuncs.com
at org.apache.druid.segment.realtime.appenderator.AppenderatorImpl.mergeAndPush(AppenderatorImpl.java:822) ~[druid-server-0.18.1.jar:0.18.1]
at org.apache.druid.segment.realtime.appenderator.AppenderatorImpl.lambda$push$1(AppenderatorImpl.java:657) ~[druid-server-0.18.1.jar:0.18.1]
at com.google.common.util.concurrent.Futures$1.apply(Futures.java:713) ~[guava-16.0.1.jar:?]
at com.google.common.util.concurrent.Futures$ChainingListenableFuture.run(Futures.java:861) ~[guava-16.0.1.jar:?]
... 3 more
Caused by: java.io.IOException: com.amazonaws.services.s3.model.AmazonS3Exception: A header you provided implies functionality that is not implemented. (Service: Amazon S3; Status Code: 400; Error Code: NotImplemented; Request ID: 5EF7E0CBAF285D3538BCFD5C; S3 Extended Request ID: frankchen021-public.oss-cn-hangzhou-internal.aliyuncs.com), S3 Extended Request ID: frankchen021-public.oss-cn-hangzhou-internal.aliyuncs.com
at org.apache.druid.storage.s3.S3DataSegmentPusher.push(S3DataSegmentPusher.java:103) ~[?:?]
at org.apache.druid.segment.realtime.appenderator.AppenderatorImpl.lambda$mergeAndPush$4(AppenderatorImpl.java:791) ~[druid-server-0.18.1.jar:0.18.1]
at org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:87) ~[druid-core-0.18.1.jar:0.18.1]
at org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:115) ~[druid-core-0.18.1.jar:0.18.1]
at org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:105) ~[druid-core-0.18.1.jar:0.18.1]
at org.apache.druid.segment.realtime.appenderator.AppenderatorImpl.mergeAndPush(AppenderatorImpl.java:787) ~[druid-server-0.18.1.jar:0.18.1]
at org.apache.druid.segment.realtime.appenderator.AppenderatorImpl.lambda$push$1(AppenderatorImpl.java:657) ~[druid-server-0.18.1.jar:0.18.1]
at com.google.common.util.concurrent.Futures$1.apply(Futures.java:713) ~[guava-16.0.1.jar:?]
at com.google.common.util.concurrent.Futures$ChainingListenableFuture.run(Futures.java:861) ~[guava-16.0.1.jar:?]
... 3 more
Caused by: com.amazonaws.services.s3.model.AmazonS3Exception: A header you provided implies functionality that is not implemented. (Service: Amazon S3; Status Code: 400; Error Code: NotImplemented; Request ID: 5EF7E0CBAF285D3538BCFD5C; S3 Extended Request ID: frankchen021-public.oss-cn-hangzhou-internal.aliyuncs.com)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1638) ~[aws-java-sdk-core-1.11.199.jar:?]
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1303) ~[aws-java-sdk-core-1.11.199.jar:?]
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1055) ~[aws-java-sdk-core-1.11.199.jar:?]
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:743) ~[aws-java-sdk-core-1.11.199.jar:?]
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:717) ~[aws-java-sdk-core-1.11.199.jar:?]
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:699) ~[aws-java-sdk-core-1.11.199.jar:?]
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:667) ~[aws-java-sdk-core-1.11.199.jar:?]
at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:649) ~[aws-java-sdk-core-1.11.199.jar:?]
at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:513) ~[aws-java-sdk-core-1.11.199.jar:?]
at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4229) ~[aws-java-sdk-s3-1.11.199.jar:?]
at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4176) ~[aws-java-sdk-s3-1.11.199.jar:?]
at com.amazonaws.services.s3.AmazonS3Client.putObject(AmazonS3Client.java:1720) ~[aws-java-sdk-s3-1.11.199.jar:?]
at org.apache.druid.storage.s3.ServerSideEncryptingAmazonS3.putObject(ServerSideEncryptingAmazonS3.java:119) ~[?:?]
at org.apache.druid.storage.s3.S3Utils.uploadFileIfPossible(S3Utils.java:286) ~[?:?]
at org.apache.druid.storage.s3.S3DataSegmentPusher.lambda$push$0(S3DataSegmentPusher.java:96) ~[?:?]
at org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:87) ~[druid-core-0.18.1.jar:0.18.1]
at org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:115) ~[druid-core-0.18.1.jar:0.18.1]
at org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:105) ~[druid-core-0.18.1.jar:0.18.1]
at org.apache.druid.storage.s3.S3Utils.retryS3Operation(S3Utils.java:91) ~[?:?]
at org.apache.druid.storage.s3.S3DataSegmentPusher.push(S3DataSegmentPusher.java:94) ~[?:?]
at org.apache.druid.segment.realtime.appenderator.AppenderatorImpl.lambda$mergeAndPush$4(AppenderatorImpl.java:791) ~[druid-server-0.18.1.jar:0.18.1]
at org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:87) ~[druid-core-0.18.1.jar:0.18.1]
at org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:115) ~[druid-core-0.18.1.jar:0.18.1]
at org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:105) ~[druid-core-0.18.1.jar:0.18.1]
at org.apache.druid.segment.realtime.appenderator.AppenderatorImpl.mergeAndPush(AppenderatorImpl.java:787) ~[druid-server-0.18.1.jar:0.18.1]
at org.apache.druid.segment.realtime.appenderator.AppenderatorImpl.lambda$push$1(AppenderatorImpl.java:657) ~[druid-server-0.18.1.jar:0.18.1]
at com.google.common.util.concurrent.Futures$1.apply(Futures.java:713) ~[guava-16.0.1.jar:?]
at com.google.common.util.concurrent.Futures$ChainingListenableFuture.run(Futures.java:861) ~[guava-16.0.1.jar:?]
````
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] jon-wei commented on pull request #9898: support Aliyun OSS service as deep storage
Posted by GitBox <gi...@apache.org>.
jon-wei commented on pull request #9898:
URL: https://github.com/apache/druid/pull/9898#issuecomment-651498601
This LGTM, did you have any further comments @jihoonson
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org