You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@jackrabbit.apache.org by GitBox <gi...@apache.org> on 2022/03/03 16:16:39 UTC
[GitHub] [jackrabbit-oak] ArunOnCloud opened a new pull request #510: OAK-9713 sas uri support in oak-upgrade
ArunOnCloud opened a new pull request #510:
URL: https://github.com/apache/jackrabbit-oak/pull/510
sas uri support in oak-upgrade
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@jackrabbit.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [jackrabbit-oak] ArunOnCloud commented on a change in pull request #510: OAK-9713 sas uri support in oak-upgrade
Posted by GitBox <gi...@apache.org>.
ArunOnCloud commented on a change in pull request #510:
URL: https://github.com/apache/jackrabbit-oak/pull/510#discussion_r837541345
##########
File path: oak-segment-azure/src/main/java/org/apache/jackrabbit/oak/segment/azure/AzureUtilities.java
##########
@@ -116,6 +116,15 @@ public static CloudBlobDirectory cloudBlobDirectoryFrom(String connection, Strin
return container.getDirectoryReference(dir);
}
+ public static CloudBlobDirectory cloudBlobDirectoryFrom(String connection, String containerName,
Review comment:
@smiroslav Yes I missed re-implementation part. I will resolve it in next commit, if everything else is ok.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@jackrabbit.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [jackrabbit-oak] smiroslav commented on a change in pull request #510: OAK-9713 sas uri support in oak-upgrade
Posted by GitBox <gi...@apache.org>.
smiroslav commented on a change in pull request #510:
URL: https://github.com/apache/jackrabbit-oak/pull/510#discussion_r831986001
##########
File path: oak-upgrade/src/main/java/org/apache/jackrabbit/oak/upgrade/cli/node/SegmentAzureFactory.java
##########
@@ -87,6 +90,10 @@ public Builder containerName(String containerName) {
this.containerName = containerName;
return this;
}
+ public Builder sasToken(String sasToken) {
Review comment:
Ok, I see that this is set after parsing the connection in org.apache.jackrabbit.oak.upgrade.cli.parser.StoreType.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@jackrabbit.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [jackrabbit-oak] jelmini commented on a change in pull request #510: OAK-9713 sas uri support in oak-upgrade
Posted by GitBox <gi...@apache.org>.
jelmini commented on a change in pull request #510:
URL: https://github.com/apache/jackrabbit-oak/pull/510#discussion_r822543067
##########
File path: oak-upgrade/src/main/java/org/apache/jackrabbit/oak/upgrade/cli/parser/StoreType.java
##########
@@ -166,6 +168,7 @@ public StoreFactory createFactory(String[] paths, MigrationDirection direction,
direction == MigrationDirection.SRC)
.connectionString(config.get(KEY_CONNECTION_STRING))
.containerName(config.get(KEY_CONTAINER_NAME))
+ .sasToken(config.get(KEY_SHARED_ACCESS_SIGNATURE))
Review comment:
Setting the SAS token here is not strictly necessary because it's by definition already part of the connection string (given that it's extracted from the connection string by the call to `parseAzureConfigurationFromCustomConnection(path)` just above).
However, in the `else` branch below, if a SAS token is part of the URL parsed by `parseAzureConfigurationFromUri(path)` (where URL has form `https://path/container?sasToken`) then it should be set in the builder, otherwise the `AZURE_SECRET_KEY` env var will be used in that case.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@jackrabbit.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [jackrabbit-oak] jelmini commented on a change in pull request #510: OAK-9713 sas uri support in oak-upgrade
Posted by GitBox <gi...@apache.org>.
jelmini commented on a change in pull request #510:
URL: https://github.com/apache/jackrabbit-oak/pull/510#discussion_r822533815
##########
File path: oak-segment-azure/src/main/java/org/apache/jackrabbit/oak/segment/azure/AzureUtilities.java
##########
@@ -116,6 +116,12 @@ public static CloudBlobDirectory cloudBlobDirectoryFrom(String connection, Strin
return container.getDirectoryReference(dir);
}
+ public static CloudBlobDirectory cloudBlobDirectoryFromSasUri(String connection, String containerName,
Review comment:
@mattvryan This class seem to be exported and is accessed from the `oak-upgrade` module. However, I don't think it's creating a security issue. This class just wraps the Azure SDK. If someone has valid credentials, then it could just as well use the Azure SDK directly to successfully connect.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@jackrabbit.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [jackrabbit-oak] smiroslav commented on a change in pull request #510: OAK-9713 sas uri support in oak-upgrade
Posted by GitBox <gi...@apache.org>.
smiroslav commented on a change in pull request #510:
URL: https://github.com/apache/jackrabbit-oak/pull/510#discussion_r832037230
##########
File path: oak-segment-azure/src/main/java/org/apache/jackrabbit/oak/segment/azure/AzureUtilities.java
##########
@@ -116,6 +116,15 @@ public static CloudBlobDirectory cloudBlobDirectoryFrom(String connection, Strin
return container.getDirectoryReference(dir);
}
+ public static CloudBlobDirectory cloudBlobDirectoryFrom(String connection, String containerName,
Review comment:
@jelmini you are right. Util classes are also part of the public API and we should maintain compatibility.
Your suggested approach can be used to prevent code duplication.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@jackrabbit.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [jackrabbit-oak] smiroslav commented on a change in pull request #510: OAK-9713 sas uri support in oak-upgrade
Posted by GitBox <gi...@apache.org>.
smiroslav commented on a change in pull request #510:
URL: https://github.com/apache/jackrabbit-oak/pull/510#discussion_r819444297
##########
File path: oak-upgrade/src/main/java/org/apache/jackrabbit/oak/upgrade/cli/node/SegmentAzureFactory.java
##########
@@ -87,6 +90,10 @@ public Builder containerName(String containerName) {
this.containerName = containerName;
return this;
}
+ public Builder sasToken(String sasToken) {
Review comment:
Is it necessary to explicitly set ```sasToken```? I have impression that it can be the part of ```connectionString```.
https://docs.microsoft.com/en-us/azure/storage/common/storage-configure-connection-string#create-a-connection-string-using-a-shared-access-signature
##########
File path: oak-segment-azure/src/main/java/org/apache/jackrabbit/oak/segment/azure/AzureUtilities.java
##########
@@ -116,6 +116,12 @@ public static CloudBlobDirectory cloudBlobDirectoryFrom(String connection, Strin
return container.getDirectoryReference(dir);
}
+ public static CloudBlobDirectory cloudBlobDirectoryFromSasUri(String connection, String containerName,
Review comment:
Not sure this method is needed, since the only difference in this method compared to ```cloudBlobDirectoryFrom``` is that container creation is skipped. Otherwise in the method implementation I do not see anything specific to SAS URI.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@jackrabbit.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [jackrabbit-oak] jelmini commented on a change in pull request #510: OAK-9713 sas uri support in oak-upgrade
Posted by GitBox <gi...@apache.org>.
jelmini commented on a change in pull request #510:
URL: https://github.com/apache/jackrabbit-oak/pull/510#discussion_r822555766
##########
File path: oak-segment-azure/src/main/java/org/apache/jackrabbit/oak/segment/azure/AzureUtilities.java
##########
@@ -116,6 +116,12 @@ public static CloudBlobDirectory cloudBlobDirectoryFrom(String connection, Strin
return container.getDirectoryReference(dir);
}
+ public static CloudBlobDirectory cloudBlobDirectoryFromSasUri(String connection, String containerName,
Review comment:
I agree with @smiroslav that this new method is misleading. What do we need is the ability to skip creating the container. I suggest to add a boolean parameter `skipCreateContainer` to the existing `cloudBlobDirectoryFrom()` methods. The value could then come from `MigrationOptions`, so that when running the upgrade tool, the flag can be set at the command line.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@jackrabbit.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [jackrabbit-oak] smiroslav commented on a change in pull request #510: OAK-9713 sas uri support in oak-upgrade
Posted by GitBox <gi...@apache.org>.
smiroslav commented on a change in pull request #510:
URL: https://github.com/apache/jackrabbit-oak/pull/510#discussion_r837505922
##########
File path: oak-segment-azure/src/main/java/org/apache/jackrabbit/oak/segment/azure/AzureUtilities.java
##########
@@ -116,6 +116,15 @@ public static CloudBlobDirectory cloudBlobDirectoryFrom(String connection, Strin
return container.getDirectoryReference(dir);
}
+ public static CloudBlobDirectory cloudBlobDirectoryFrom(String connection, String containerName,
Review comment:
@ArunOnCloud I think this is still not resolved. to prevent code duplication, the suggestion from @jelmini should be applied.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@jackrabbit.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [jackrabbit-oak] jelmini commented on a change in pull request #510: OAK-9713 sas uri support in oak-upgrade
Posted by GitBox <gi...@apache.org>.
jelmini commented on a change in pull request #510:
URL: https://github.com/apache/jackrabbit-oak/pull/510#discussion_r832025657
##########
File path: oak-segment-azure/src/main/java/org/apache/jackrabbit/oak/segment/azure/AzureUtilities.java
##########
@@ -116,6 +116,15 @@ public static CloudBlobDirectory cloudBlobDirectoryFrom(String connection, Strin
return container.getDirectoryReference(dir);
}
+ public static CloudBlobDirectory cloudBlobDirectoryFrom(String connection, String containerName,
Review comment:
@smiroslav It's me that suggested to add an overload of `cloudBlobDirectoryFrom()`, in order to keep backward compatibility in case 3rd-party code is using it. Do you think it's not necessary?
If we keep them, however, one overload should call the second, instead of re-implementing it:
```java
public static CloudBlobDirectory cloudBlobDirectoryFrom(String connection, String containerName,
String dir) throws InvalidKeyException, URISyntaxException, StorageException {
return cloudBlobDirectoryFrom(connection, containerName, dir, false);
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@jackrabbit.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [jackrabbit-oak] jelmini commented on a change in pull request #510: OAK-9713 sas uri support in oak-upgrade
Posted by GitBox <gi...@apache.org>.
jelmini commented on a change in pull request #510:
URL: https://github.com/apache/jackrabbit-oak/pull/510#discussion_r832033675
##########
File path: oak-segment-azure/src/main/java/org/apache/jackrabbit/oak/segment/azure/AzureUtilities.java
##########
@@ -116,6 +116,15 @@ public static CloudBlobDirectory cloudBlobDirectoryFrom(String connection, Strin
return container.getDirectoryReference(dir);
}
+ public static CloudBlobDirectory cloudBlobDirectoryFrom(String connection, String containerName,
+ String dir,boolean skipCreateContainer) throws InvalidKeyException, URISyntaxException, StorageException {
Review comment:
Code formatting: for consistency with the rest of the code, please
* add spaces after commas and before `{` (in the `if` below)
* don't align parameter lists with the opening method `(`, a simple indent is sufficient:
```java
public static CloudBlobDirectory cloudBlobDirectoryFrom(String connection, String containerName,
String dir, boolean skipCreateContainer) throws InvalidKeyException, URISyntaxException, StorageException {
...
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@jackrabbit.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [jackrabbit-oak] mattvryan commented on a change in pull request #510: OAK-9713 sas uri support in oak-upgrade
Posted by GitBox <gi...@apache.org>.
mattvryan commented on a change in pull request #510:
URL: https://github.com/apache/jackrabbit-oak/pull/510#discussion_r821138015
##########
File path: oak-segment-azure/src/main/java/org/apache/jackrabbit/oak/segment/azure/AzureUtilities.java
##########
@@ -116,6 +116,12 @@ public static CloudBlobDirectory cloudBlobDirectoryFrom(String connection, Strin
return container.getDirectoryReference(dir);
}
+ public static CloudBlobDirectory cloudBlobDirectoryFromSasUri(String connection, String containerName,
Review comment:
I'm concerned about this. How accessible is this method? In particular, is it exported so that it can be called outside the bundle? I worry about security implications - it's giving access to cloud storage based on any valid connection string passed in to the method.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@jackrabbit.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [jackrabbit-oak] jelmini commented on a change in pull request #510: OAK-9713 sas uri support in oak-upgrade
Posted by GitBox <gi...@apache.org>.
jelmini commented on a change in pull request #510:
URL: https://github.com/apache/jackrabbit-oak/pull/510#discussion_r833576130
##########
File path: oak-upgrade/src/test/java/org/apache/jackrabbit/oak/upgrade/cli/SegmentTarToSegmentAzureWithSASUriTest.java
##########
@@ -0,0 +1,140 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.jackrabbit.oak.upgrade.cli;
+
+import java.io.IOException;
+
+import java.time.Duration;
+import java.time.Instant;
+import java.util.Date;
+import java.util.EnumSet;
+
+import com.google.common.base.Joiner;
+import com.microsoft.azure.storage.blob.CloudBlobContainer;
+import com.microsoft.azure.storage.blob.SharedAccessBlobPermissions;
+import com.microsoft.azure.storage.blob.SharedAccessBlobPolicy;
+
+import org.apache.jackrabbit.oak.segment.azure.AzuriteDockerRule;
+
+import org.apache.jackrabbit.oak.spi.state.NodeStore;
+import org.apache.jackrabbit.oak.upgrade.cli.container.NodeStoreContainer;
+import org.apache.jackrabbit.oak.upgrade.cli.container.SegmentAzureNodeStoreContainer;
+import org.apache.jackrabbit.oak.upgrade.cli.container.SegmentTarNodeStoreContainer;
+import org.apache.jackrabbit.oak.upgrade.cli.parser.OptionParserFactory;
+import org.junit.Before;
+import org.junit.ClassRule;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import static com.microsoft.azure.storage.blob.SharedAccessBlobPermissions.READ;
+import static com.microsoft.azure.storage.blob.SharedAccessBlobPermissions.WRITE;
+import static com.microsoft.azure.storage.blob.SharedAccessBlobPermissions.ADD;
+import static com.microsoft.azure.storage.blob.SharedAccessBlobPermissions.CREATE;
+import static com.microsoft.azure.storage.blob.SharedAccessBlobPermissions.LIST;
+
+
+public class SegmentTarToSegmentAzureWithSASUriTest extends AbstractOak2OakTest {
Review comment:
Please check formatting in this file. In several lines, spaces are missing. Applying the default IDE reformatter should be enough.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@jackrabbit.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [jackrabbit-oak] jelmini commented on pull request #510: OAK-9713 sas uri support in oak-upgrade
Posted by GitBox <gi...@apache.org>.
jelmini commented on pull request #510:
URL: https://github.com/apache/jackrabbit-oak/pull/510#issuecomment-1076715904
Please document the possibility to use SAS URIs in `upgrade_usage.txt`. The new `skip-create-azure-container` option should be documented there as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@jackrabbit.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [jackrabbit-oak] jelmini commented on a change in pull request #510: OAK-9713 sas uri support in oak-upgrade
Posted by GitBox <gi...@apache.org>.
jelmini commented on a change in pull request #510:
URL: https://github.com/apache/jackrabbit-oak/pull/510#discussion_r833545124
##########
File path: oak-upgrade/src/main/java/org/apache/jackrabbit/oak/upgrade/cli/parser/OptionParserFactory.java
##########
@@ -174,5 +176,6 @@ private static void addMiscOptions(OptionParser op) {
op.accepts(SKIP_CHECKPOINTS, "Don't copy checkpoints on the full segment->segment migration");
op.accepts(FORCE_CHECKPOINTS, "Copy checkpoints even if the --include,exclude,merge-paths option is specified");
op.accepts(ADD_SECONDARY_METADATA, "Adds the metadata required by secondary store");
+ op.accepts(SKIP_CREATE_AZURE_CONTAINER, "Create azure container if not exist");
Review comment:
Text should be the other way around: "Skip creating the Azure container if it does not exist".
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@jackrabbit.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [jackrabbit-oak] smiroslav commented on a change in pull request #510: OAK-9713 sas uri support in oak-upgrade
Posted by GitBox <gi...@apache.org>.
smiroslav commented on a change in pull request #510:
URL: https://github.com/apache/jackrabbit-oak/pull/510#discussion_r831984409
##########
File path: oak-segment-azure/src/main/java/org/apache/jackrabbit/oak/segment/azure/AzureUtilities.java
##########
@@ -116,6 +116,15 @@ public static CloudBlobDirectory cloudBlobDirectoryFrom(String connection, Strin
return container.getDirectoryReference(dir);
}
+ public static CloudBlobDirectory cloudBlobDirectoryFrom(String connection, String containerName,
Review comment:
I believe this method is not necessary. Rather previous signature could be changed, and add **skipCreateContainer** parameter. Seems it is only used in org.apache.jackrabbit.oak.run.DataStoreCommandTest.StoreFixture.AzureSegmentStoreFixture#init
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@jackrabbit.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [jackrabbit-oak] jelmini commented on a change in pull request #510: OAK-9713 sas uri support in oak-upgrade
Posted by GitBox <gi...@apache.org>.
jelmini commented on a change in pull request #510:
URL: https://github.com/apache/jackrabbit-oak/pull/510#discussion_r822545736
##########
File path: oak-upgrade/src/main/java/org/apache/jackrabbit/oak/upgrade/cli/node/SegmentAzureFactory.java
##########
@@ -87,6 +90,10 @@ public Builder containerName(String containerName) {
this.containerName = containerName;
return this;
}
+ public Builder sasToken(String sasToken) {
Review comment:
I think it can be useful in the case where an URL of form `https://path/container?sasToken` is used instead of a connection string.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@jackrabbit.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [jackrabbit-oak] jelmini commented on a change in pull request #510: OAK-9713 sas uri support in oak-upgrade
Posted by GitBox <gi...@apache.org>.
jelmini commented on a change in pull request #510:
URL: https://github.com/apache/jackrabbit-oak/pull/510#discussion_r832033675
##########
File path: oak-segment-azure/src/main/java/org/apache/jackrabbit/oak/segment/azure/AzureUtilities.java
##########
@@ -116,6 +116,15 @@ public static CloudBlobDirectory cloudBlobDirectoryFrom(String connection, Strin
return container.getDirectoryReference(dir);
}
+ public static CloudBlobDirectory cloudBlobDirectoryFrom(String connection, String containerName,
+ String dir,boolean skipCreateContainer) throws InvalidKeyException, URISyntaxException, StorageException {
Review comment:
Code formatting: for consistency with the rest of the code, please
* add spaces after commas (int this line) and before `{` (in the `if` below)
* don't align parameter lists with the opening method `(`, a simple indent is sufficient:
```java
public static CloudBlobDirectory cloudBlobDirectoryFrom(String connection, String containerName,
String dir, boolean skipCreateContainer) throws InvalidKeyException, URISyntaxException, StorageException {
...
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@jackrabbit.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [jackrabbit-oak] jelmini commented on a change in pull request #510: OAK-9713 sas uri support in oak-upgrade
Posted by GitBox <gi...@apache.org>.
jelmini commented on a change in pull request #510:
URL: https://github.com/apache/jackrabbit-oak/pull/510#discussion_r833583722
##########
File path: oak-upgrade/src/test/java/org/apache/jackrabbit/oak/upgrade/cli/SegmentTarToSegmentAzureWithSASUriTest.java
##########
@@ -0,0 +1,140 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.jackrabbit.oak.upgrade.cli;
+
+import java.io.IOException;
+
+import java.time.Duration;
+import java.time.Instant;
+import java.util.Date;
+import java.util.EnumSet;
+
+import com.google.common.base.Joiner;
+import com.microsoft.azure.storage.blob.CloudBlobContainer;
+import com.microsoft.azure.storage.blob.SharedAccessBlobPermissions;
+import com.microsoft.azure.storage.blob.SharedAccessBlobPolicy;
+
+import org.apache.jackrabbit.oak.segment.azure.AzuriteDockerRule;
+
+import org.apache.jackrabbit.oak.spi.state.NodeStore;
+import org.apache.jackrabbit.oak.upgrade.cli.container.NodeStoreContainer;
+import org.apache.jackrabbit.oak.upgrade.cli.container.SegmentAzureNodeStoreContainer;
+import org.apache.jackrabbit.oak.upgrade.cli.container.SegmentTarNodeStoreContainer;
+import org.apache.jackrabbit.oak.upgrade.cli.parser.OptionParserFactory;
+import org.junit.Before;
+import org.junit.ClassRule;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import static com.microsoft.azure.storage.blob.SharedAccessBlobPermissions.READ;
+import static com.microsoft.azure.storage.blob.SharedAccessBlobPermissions.WRITE;
+import static com.microsoft.azure.storage.blob.SharedAccessBlobPermissions.ADD;
+import static com.microsoft.azure.storage.blob.SharedAccessBlobPermissions.CREATE;
+import static com.microsoft.azure.storage.blob.SharedAccessBlobPermissions.LIST;
+
+
+public class SegmentTarToSegmentAzureWithSASUriTest extends AbstractOak2OakTest {
+
+ private static final Logger log = LoggerFactory.getLogger(SegmentTarToSegmentAzureWithSASUriTest.class);
+
+ public static final String DEFAULT_CONTAINER_NAME = "oak-test";
+
+ public static final String ACCOUNT_KEY = "AccountKey";
+
+ private final NodeStoreContainer source;
+
+ private final NodeStoreContainer destination;
+
+ public static final String KEY_SHARED_ACCESS_SIGNATURE = "SharedAccessSignature";
+
+ private String sasToken;
+
+ private static final EnumSet<SharedAccessBlobPermissions> READ_WRITE = EnumSet.of(READ, LIST, CREATE, WRITE, ADD);
+
+ @ClassRule
+ public static AzuriteDockerRule azurite = new AzuriteDockerRule();
+
+ public SegmentTarToSegmentAzureWithSASUriTest() throws IOException {
+ source = new SegmentTarNodeStoreContainer();
+ destination = new SegmentAzureNodeStoreContainer(azurite,"repository");
+ }
+
+ @Override
+ protected NodeStoreContainer getSourceContainer() {
+ return source;
+ }
+
+ @Override
+ protected NodeStoreContainer getDestinationContainer() {
+ return destination;
+ }
+
+ @Before
+ public void prepare() throws Exception {
Review comment:
This method overrides its superclass method and is duplicating a lot of code from there. I suggest to call the superclass method to avoid duplication:
```java
@Override
public void prepare() throws Exception {
CloudBlobContainer container = azurite.getContainer(DEFAULT_CONTAINER_NAME);
this.sasToken = container.generateSharedAccessSignature(policy(READ_WRITE), null);
super.prepare();
}
```
Adding the `@Override` annotation is also beneficial to make it clear that this method is actually an override.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@jackrabbit.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [jackrabbit-oak] jelmini commented on a change in pull request #510: OAK-9713 sas uri support in oak-upgrade
Posted by GitBox <gi...@apache.org>.
jelmini commented on a change in pull request #510:
URL: https://github.com/apache/jackrabbit-oak/pull/510#discussion_r833541080
##########
File path: oak-upgrade/src/main/java/org/apache/jackrabbit/oak/upgrade/cli/node/SegmentAzureFactory.java
##########
@@ -143,11 +159,16 @@ private AzurePersistence createAzurePersistence() throws StorageException, URISy
CloudBlobDirectory cloudBlobDirectory = null;
if (accountName != null && uri != null) {
- String key = System.getenv("AZURE_SECRET_KEY");
- StorageCredentials credentials = new StorageCredentialsAccountAndKey(accountName, key);
+ StorageCredentials credentials;
+ if (!StringUtils.isBlank(sasToken)) {
+ credentials = new StorageCredentialsSharedAccessSignature(sasToken);
+ } else {
+ String key = System.getenv("AZURE_SECRET_KEY");
+ credentials = new StorageCredentialsAccountAndKey(accountName, key);
+ }
cloudBlobDirectory = AzureUtilities.cloudBlobDirectoryFrom(credentials, uri, dir);
} else if (connectionString != null && containerName != null) {
- cloudBlobDirectory = AzureUtilities.cloudBlobDirectoryFrom(connectionString, containerName, dir);
+ cloudBlobDirectory = AzureUtilities.cloudBlobDirectoryFrom(connectionString, containerName, dir,skipCreateContainer);
Review comment:
Formatting: space after comma.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@jackrabbit.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [jackrabbit-oak] jelmini commented on a change in pull request #510: OAK-9713 sas uri support in oak-upgrade
Posted by GitBox <gi...@apache.org>.
jelmini commented on a change in pull request #510:
URL: https://github.com/apache/jackrabbit-oak/pull/510#discussion_r833622560
##########
File path: oak-upgrade/src/test/java/org/apache/jackrabbit/oak/upgrade/cli/SegmentTarToSegmentAzureWithSASUriTest.java
##########
@@ -0,0 +1,140 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.jackrabbit.oak.upgrade.cli;
+
+import java.io.IOException;
+
+import java.time.Duration;
+import java.time.Instant;
+import java.util.Date;
+import java.util.EnumSet;
+
+import com.google.common.base.Joiner;
+import com.microsoft.azure.storage.blob.CloudBlobContainer;
+import com.microsoft.azure.storage.blob.SharedAccessBlobPermissions;
+import com.microsoft.azure.storage.blob.SharedAccessBlobPolicy;
+
+import org.apache.jackrabbit.oak.segment.azure.AzuriteDockerRule;
+
+import org.apache.jackrabbit.oak.spi.state.NodeStore;
+import org.apache.jackrabbit.oak.upgrade.cli.container.NodeStoreContainer;
+import org.apache.jackrabbit.oak.upgrade.cli.container.SegmentAzureNodeStoreContainer;
+import org.apache.jackrabbit.oak.upgrade.cli.container.SegmentTarNodeStoreContainer;
+import org.apache.jackrabbit.oak.upgrade.cli.parser.OptionParserFactory;
+import org.junit.Before;
+import org.junit.ClassRule;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import static com.microsoft.azure.storage.blob.SharedAccessBlobPermissions.READ;
+import static com.microsoft.azure.storage.blob.SharedAccessBlobPermissions.WRITE;
+import static com.microsoft.azure.storage.blob.SharedAccessBlobPermissions.ADD;
+import static com.microsoft.azure.storage.blob.SharedAccessBlobPermissions.CREATE;
+import static com.microsoft.azure.storage.blob.SharedAccessBlobPermissions.LIST;
+
+
+public class SegmentTarToSegmentAzureWithSASUriTest extends AbstractOak2OakTest {
+
+ private static final Logger log = LoggerFactory.getLogger(SegmentTarToSegmentAzureWithSASUriTest.class);
+
+ public static final String DEFAULT_CONTAINER_NAME = "oak-test";
+
+ public static final String ACCOUNT_KEY = "AccountKey";
+
+ private final NodeStoreContainer source;
+
+ private final NodeStoreContainer destination;
+
+ public static final String KEY_SHARED_ACCESS_SIGNATURE = "SharedAccessSignature";
+
+ private String sasToken;
+
+ private static final EnumSet<SharedAccessBlobPermissions> READ_WRITE = EnumSet.of(READ, LIST, CREATE, WRITE, ADD);
+
+ @ClassRule
+ public static AzuriteDockerRule azurite = new AzuriteDockerRule();
+
+ public SegmentTarToSegmentAzureWithSASUriTest() throws IOException {
+ source = new SegmentTarNodeStoreContainer();
+ destination = new SegmentAzureNodeStoreContainer(azurite,"repository");
+ }
+
+ @Override
+ protected NodeStoreContainer getSourceContainer() {
+ return source;
+ }
+
+ @Override
+ protected NodeStoreContainer getDestinationContainer() {
+ return destination;
+ }
+
+ @Before
+ public void prepare() throws Exception {
+ NodeStore source = getSourceContainer().open();
+ try {
+ initContent(source);
+ } finally {
+ getSourceContainer().close();
+ }
+ CloudBlobContainer container= azurite.getContainer(DEFAULT_CONTAINER_NAME);
+ String sasToken = container.generateSharedAccessSignature(policy(READ_WRITE),null);
+ this.sasToken =sasToken;
+
+ String[] args = getArgs();
+ log.info("oak2oak {}", Joiner.on(' ').join(args));
+
+ OakUpgrade.main(args);
+ createSession();
+ }
+
+ @Override
+ protected String[] getArgs() {
+ String destinationString = destination.getDescription() + ";" + KEY_SHARED_ACCESS_SIGNATURE + "=" + sasToken;
Review comment:
This code could be simplified and made more clear. I suggest:
```java
StringBuilder builder = new StringBuilder();
String[] properties = destination.getDescription().split(";");
for (String str : properties) {
if (!str.startsWith(ACCOUNT_KEY)) {
builder.append(str).append(";");
}
}
String destinationString = builder.toString() + ";" + KEY_SHARED_ACCESS_SIGNATURE + "=" + sasToken;
return new String[] { source.getDescription(), destinationString, "--" + OptionParserFactory.SKIP_CREATE_AZURE_CONTAINER };
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@jackrabbit.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [jackrabbit-oak] jelmini commented on a change in pull request #510: OAK-9713 sas uri support in oak-upgrade
Posted by GitBox <gi...@apache.org>.
jelmini commented on a change in pull request #510:
URL: https://github.com/apache/jackrabbit-oak/pull/510#discussion_r833549402
##########
File path: oak-upgrade/src/main/java/org/apache/jackrabbit/oak/upgrade/cli/parser/MigrationOptions.java
##########
@@ -408,6 +415,10 @@ public void logOptions() {
log.info("Secondary metadata will be added");
}
+ if (!skipCreateAzureContainer) {
Review comment:
Log message and `skipCreateAzureContainer` semantics are mismatched. I suggest:
```java
if (skipCreateAzureContainer) {
log.info("Skipping creation of the Azure container, if it does not yet exists");
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@jackrabbit.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [jackrabbit-oak] jelmini commented on a change in pull request #510: OAK-9713 sas uri support in oak-upgrade
Posted by GitBox <gi...@apache.org>.
jelmini commented on a change in pull request #510:
URL: https://github.com/apache/jackrabbit-oak/pull/510#discussion_r833561695
##########
File path: oak-upgrade/src/main/java/org/apache/jackrabbit/oak/upgrade/cli/parser/StoreType.java
##########
@@ -176,6 +179,7 @@ public StoreFactory createFactory(String[] paths, MigrationDirection direction,
migrationOptions.getCacheSizeInMB(), direction == MigrationDirection.SRC)
.accountName(config.get(KEY_ACCOUNT_NAME))
.uri(config.get(KEY_STORAGE_URI))
+ .sasToken(config.get(KEY_SHARED_ACCESS_SIGNATURE))
Review comment:
Shouldn't `skipCreateContainer` be set here as well?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@jackrabbit.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [jackrabbit-oak] jelmini commented on a change in pull request #510: OAK-9713 sas uri support in oak-upgrade
Posted by GitBox <gi...@apache.org>.
jelmini commented on a change in pull request #510:
URL: https://github.com/apache/jackrabbit-oak/pull/510#discussion_r833558564
##########
File path: oak-segment-azure/src/main/java/org/apache/jackrabbit/oak/segment/azure/AzureUtilities.java
##########
@@ -116,6 +116,15 @@ public static CloudBlobDirectory cloudBlobDirectoryFrom(String connection, Strin
return container.getDirectoryReference(dir);
}
+ public static CloudBlobDirectory cloudBlobDirectoryFrom(String connection, String containerName,
Review comment:
Formatting: Please add an empty line between methods.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@jackrabbit.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org