You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@opendal.apache.org by "ArmandoZ (via GitHub)" <gi...@apache.org> on 2024/02/15 06:52:32 UTC
[PR] ci: Add hdfs test case for s3 and azure (#4168) [opendal]
ArmandoZ opened a new pull request, #4183:
URL: https://github.com/apache/opendal/pull/4183
Issue: #4168
Adds test case for hdfs over s3 and azure blob storage. Tested locally with own environment.
TODO: Need to double check formats of some of the secrets. Please find the comments below
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] ci: Add hdfs test case for s3 and azure (#4168) [opendal]
Posted by "Xuanwo (via GitHub)" <gi...@apache.org>.
Xuanwo closed pull request #4183: ci: Add hdfs test case for s3 and azure (#4168)
URL: https://github.com/apache/opendal/pull/4183
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] ci: Add hdfs test case for s3 and azure (#4168) [opendal]
Posted by "Xuanwo (via GitHub)" <gi...@apache.org>.
Xuanwo commented on code in PR #4183:
URL: https://github.com/apache/opendal/pull/4183#discussion_r1491234653
##########
.github/services/hdfs/hdfs_default_s3/action.yml:
##########
@@ -0,0 +1,71 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+name: hdfs_default_s3
+description: 'Behavior test for hdfs default over aws s3'
+
+runs:
+ using: "composite"
+ steps:
+ - name: Setup java env
+ uses: actions/setup-java@v4
+ with:
+ distribution: temurin
+ java-version: "11"
+ - name: Load secrets
+ uses: 1password/load-secrets-action@v1
+ with:
+ export-env: true
+ env:
+ # Need a secret with this name to securely encrypt them - otherwise there will be warning
+ # See https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/CredentialProviderAPI.html#Keystore_Passwords
+ HADOOP_CREDSTORE_PASSWORD: op://services/hdfs/credstore_password
+ # TODO: Does this start with / and ends with /
+ OPENDAL_S3_ROOT: op://services/s3/root
+ OPENDAL_S3_BUCKET: op://services/s3/bucket
Review Comment:
> * Does `op://services/s3/root` starts and ends with `/`?
No sure. opendal hdfs service should handle this correctly.
> * Does `op://services/s3/bucket ` starts and ends with `/`?
No.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] ci: Add hdfs test case for s3 and azure (#4168) [opendal]
Posted by "ArmandoZ (via GitHub)" <gi...@apache.org>.
ArmandoZ commented on PR #4183:
URL: https://github.com/apache/opendal/pull/4183#issuecomment-1948896356
The remaining PR for azurite is also ready - #4185 . Seems to me this PR can be closed if no longer needed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] ci: Add hdfs test case for s3 and azure (#4168) [opendal]
Posted by "ArmandoZ (via GitHub)" <gi...@apache.org>.
ArmandoZ commented on code in PR #4183:
URL: https://github.com/apache/opendal/pull/4183#discussion_r1490499475
##########
.github/services/hdfs/hdfs_default_azblob/action.yml:
##########
@@ -0,0 +1,71 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+name: hdfs_default_azblob
+description: 'Behavior test for hdfs default over azure blob storage'
+
+runs:
+ using: "composite"
+ steps:
+ - name: Setup java env
+ uses: actions/setup-java@v4
+ with:
+ distribution: temurin
+ java-version: "11"
+ - name: Load secrets
+ uses: 1password/load-secrets-action@v1
+ with:
+ export-env: true
+ env:
+ # Need a environment variable with this name to securely encrypt them - otherwise there will be warning
+ # See https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/CredentialProviderAPI.html#Keystore_Passwords
+ HADOOP_CREDSTORE_PASSWORD: op://services/hdfs/credstore_password
Review Comment:
Need a environment variable with name `HADOOP_CREDSTORE_PASSWORD` to securely encrypt the credentials. If this variable is not set, we can still encrypt the credentials with default password, but not so secure.
Reference: https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/CredentialProviderAPI.html#Keystore_Passwords
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] ci: Add hdfs test case for s3 and azure (#4168) [opendal]
Posted by "ArmandoZ (via GitHub)" <gi...@apache.org>.
ArmandoZ commented on code in PR #4183:
URL: https://github.com/apache/opendal/pull/4183#discussion_r1490500912
##########
.github/services/hdfs/hdfs_default_azblob/action.yml:
##########
@@ -0,0 +1,71 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+name: hdfs_default_azblob
+description: 'Behavior test for hdfs default over azure blob storage'
+
+runs:
+ using: "composite"
+ steps:
+ - name: Setup java env
+ uses: actions/setup-java@v4
+ with:
+ distribution: temurin
+ java-version: "11"
+ - name: Load secrets
+ uses: 1password/load-secrets-action@v1
+ with:
+ export-env: true
+ env:
+ # Need a environment variable with this name to securely encrypt them - otherwise there will be warning
+ # See https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/CredentialProviderAPI.html#Keystore_Passwords
+ HADOOP_CREDSTORE_PASSWORD: op://services/hdfs/credstore_password
+ # TODO1: Do we have this as a secret
+ # TODO2: Does this start with / and ends with /
+ OPENDAL_AZBLOB_ROOT: op://services/azblob/root
Review Comment:
Didn't find a secret named `op://services/azblob/root` for azure blob storage root. Should we just use a hardcoded value here for the required root directory, for example `/tmp/opendal` ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] ci: Add hdfs test case for s3 and azure (#4168) [opendal]
Posted by "ArmandoZ (via GitHub)" <gi...@apache.org>.
ArmandoZ commented on code in PR #4183:
URL: https://github.com/apache/opendal/pull/4183#discussion_r1491315572
##########
.github/services/hdfs/hdfs_default_azblob/action.yml:
##########
@@ -0,0 +1,71 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+name: hdfs_default_azblob
+description: 'Behavior test for hdfs default over azure blob storage'
+
+runs:
+ using: "composite"
+ steps:
+ - name: Setup java env
+ uses: actions/setup-java@v4
+ with:
+ distribution: temurin
+ java-version: "11"
+ - name: Load secrets
+ uses: 1password/load-secrets-action@v1
+ with:
+ export-env: true
+ env:
+ # Need a environment variable with this name to securely encrypt them - otherwise there will be warning
+ # See https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/CredentialProviderAPI.html#Keystore_Passwords
+ HADOOP_CREDSTORE_PASSWORD: op://services/hdfs/credstore_password
Review Comment:
The credentials such as `OPENDAL_AZBLOB_ACCOUNT_KEY` are encrypted and saved to `/tmp/hdfs/wasb.jceks` in the ci machine's local file system, so that hdfs can pick up the credentials. If we don't specify `HADOOP_CREDSTORE_PASSWORD`, the encrypting key will be a default value that is publicly known.
If no one has chance to read from `/tmp/hdfs/wasb.jceks` in ci machine, this issue will be fine and we don't need `HADOOP_CREDSTORE_PASSWORD`. But otherwise, they will be able to decrypt the real credentials
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] ci: Add hdfs test case for s3 and azure (#4168) [opendal]
Posted by "ArmandoZ (via GitHub)" <gi...@apache.org>.
ArmandoZ commented on code in PR #4183:
URL: https://github.com/apache/opendal/pull/4183#discussion_r1490504608
##########
.github/services/hdfs/hdfs_default_azblob/action.yml:
##########
@@ -0,0 +1,71 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+name: hdfs_default_azblob
+description: 'Behavior test for hdfs default over azure blob storage'
+
+runs:
+ using: "composite"
+ steps:
+ - name: Setup java env
+ uses: actions/setup-java@v4
+ with:
+ distribution: temurin
+ java-version: "11"
+ - name: Load secrets
+ uses: 1password/load-secrets-action@v1
+ with:
+ export-env: true
+ env:
+ # Need a environment variable with this name to securely encrypt them - otherwise there will be warning
+ # See https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/CredentialProviderAPI.html#Keystore_Passwords
+ HADOOP_CREDSTORE_PASSWORD: op://services/hdfs/credstore_password
+ # TODO1: Do we have this as a secret
+ # TODO2: Does this start with / and ends with /
+ OPENDAL_AZBLOB_ROOT: op://services/azblob/root
Review Comment:
looks like `OPENDAL_HDFS_ROOT` is optional
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] ci: Add hdfs test case for s3 and azure (#4168) [opendal]
Posted by "ArmandoZ (via GitHub)" <gi...@apache.org>.
ArmandoZ commented on code in PR #4183:
URL: https://github.com/apache/opendal/pull/4183#discussion_r1490504608
##########
.github/services/hdfs/hdfs_default_azblob/action.yml:
##########
@@ -0,0 +1,71 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+name: hdfs_default_azblob
+description: 'Behavior test for hdfs default over azure blob storage'
+
+runs:
+ using: "composite"
+ steps:
+ - name: Setup java env
+ uses: actions/setup-java@v4
+ with:
+ distribution: temurin
+ java-version: "11"
+ - name: Load secrets
+ uses: 1password/load-secrets-action@v1
+ with:
+ export-env: true
+ env:
+ # Need a environment variable with this name to securely encrypt them - otherwise there will be warning
+ # See https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/CredentialProviderAPI.html#Keystore_Passwords
+ HADOOP_CREDSTORE_PASSWORD: op://services/hdfs/credstore_password
+ # TODO1: Do we have this as a secret
+ # TODO2: Does this start with / and ends with /
+ OPENDAL_AZBLOB_ROOT: op://services/azblob/root
Review Comment:
OPENDAL_HDFS_ROOT
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] ci: Add hdfs test case for s3 and azure (#4168) [opendal]
Posted by "Xuanwo (via GitHub)" <gi...@apache.org>.
Xuanwo commented on code in PR #4183:
URL: https://github.com/apache/opendal/pull/4183#discussion_r1491237060
##########
.github/services/hdfs/hdfs_default_azblob/action.yml:
##########
@@ -0,0 +1,67 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+name: hdfs_default_azblob
Review Comment:
s3 has `minio` setup and `azblob` has `azurite` setup. They can be tested even in forked repo. Would you like to use the same setup?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] ci: Add hdfs test case for s3 and azure (#4168) [opendal]
Posted by "Xuanwo (via GitHub)" <gi...@apache.org>.
Xuanwo commented on PR #4183:
URL: https://github.com/apache/opendal/pull/4183#issuecomment-1948910927
> The remaining PR for azurite is also ready - #4185 . Seems to me this PR can be closed if no longer needed
Merged. Let's close this PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] ci: Add hdfs test case for s3 and azure (#4168) [opendal]
Posted by "ArmandoZ (via GitHub)" <gi...@apache.org>.
ArmandoZ commented on code in PR #4183:
URL: https://github.com/apache/opendal/pull/4183#discussion_r1491309237
##########
.github/services/hdfs/hdfs_default_azblob/action.yml:
##########
@@ -0,0 +1,67 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+name: hdfs_default_azblob
Review Comment:
sure! let me have them updated in new PRs
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] ci: Add hdfs test case for s3 and azure (#4168) [opendal]
Posted by "Xuanwo (via GitHub)" <gi...@apache.org>.
Xuanwo commented on code in PR #4183:
URL: https://github.com/apache/opendal/pull/4183#discussion_r1491882601
##########
.github/services/hdfs/hdfs_default_azblob/action.yml:
##########
@@ -0,0 +1,67 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+name: hdfs_default_azblob
Review Comment:
> Do you agree we should rename the test cases and file names accordingly?
Good idea, we can use `hdfs_default_on_minio_s3` and `minio-s3-core-site.xml`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] ci: Add hdfs test case for s3 and azure (#4168) [opendal]
Posted by "Xuanwo (via GitHub)" <gi...@apache.org>.
Xuanwo commented on PR #4183:
URL: https://github.com/apache/opendal/pull/4183#issuecomment-1946380401
Hi, @ArmandoZ, thanks for your efforts! Would you like to split them into two PRs for eaiser review and testing?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] ci: Add hdfs test case for s3 and azure (#4168) [opendal]
Posted by "ArmandoZ (via GitHub)" <gi...@apache.org>.
ArmandoZ commented on code in PR #4183:
URL: https://github.com/apache/opendal/pull/4183#discussion_r1491332615
##########
.github/services/hdfs/hdfs_default_azblob/action.yml:
##########
@@ -0,0 +1,67 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+name: hdfs_default_azblob
Review Comment:
Do you agree we should rename the test cases and file names accordingly? E.g.
- `hdfs_default_s3` -> `hdfs_default_minio_s3`
- `s3-core-site.xml` -> `minio-s3-core-site.xml`
would be more specific
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] ci: Add hdfs test case for s3 and azure (#4168) [opendal]
Posted by "ArmandoZ (via GitHub)" <gi...@apache.org>.
ArmandoZ commented on PR #4183:
URL: https://github.com/apache/opendal/pull/4183#issuecomment-1947351623
PR for hdfs test case over minio s3 is ready to review with test passed: #4184
For azurite environment looks like [`hadoop-azure`](https://github.com/apache/hadoop/tree/12498b35bbb754225b0b2ca90d5ad4f5cf628d56/hadoop-tools/hadoop-azure) may not be able to support custom domains (other than *.blob.core.windows.net). Still looking into it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] ci: Add hdfs test case for s3 and azure (#4168) [opendal]
Posted by "ArmandoZ (via GitHub)" <gi...@apache.org>.
ArmandoZ commented on code in PR #4183:
URL: https://github.com/apache/opendal/pull/4183#discussion_r1490504608
##########
.github/services/hdfs/hdfs_default_azblob/action.yml:
##########
@@ -0,0 +1,71 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+name: hdfs_default_azblob
+description: 'Behavior test for hdfs default over azure blob storage'
+
+runs:
+ using: "composite"
+ steps:
+ - name: Setup java env
+ uses: actions/setup-java@v4
+ with:
+ distribution: temurin
+ java-version: "11"
+ - name: Load secrets
+ uses: 1password/load-secrets-action@v1
+ with:
+ export-env: true
+ env:
+ # Need a environment variable with this name to securely encrypt them - otherwise there will be warning
+ # See https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/CredentialProviderAPI.html#Keystore_Passwords
+ HADOOP_CREDSTORE_PASSWORD: op://services/hdfs/credstore_password
+ # TODO1: Do we have this as a secret
+ # TODO2: Does this start with / and ends with /
+ OPENDAL_AZBLOB_ROOT: op://services/azblob/root
Review Comment:
looks like we don't need `OPENDAL_AZBLOB_ROOT: op://services/azblob/root`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] ci: Add hdfs test case for s3 and azure (#4168) [opendal]
Posted by "Xuanwo (via GitHub)" <gi...@apache.org>.
Xuanwo commented on code in PR #4183:
URL: https://github.com/apache/opendal/pull/4183#discussion_r1491232985
##########
.github/services/hdfs/hdfs_default_azblob/action.yml:
##########
@@ -0,0 +1,71 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+name: hdfs_default_azblob
+description: 'Behavior test for hdfs default over azure blob storage'
+
+runs:
+ using: "composite"
+ steps:
+ - name: Setup java env
+ uses: actions/setup-java@v4
+ with:
+ distribution: temurin
+ java-version: "11"
+ - name: Load secrets
+ uses: 1password/load-secrets-action@v1
+ with:
+ export-env: true
+ env:
+ # Need a environment variable with this name to securely encrypt them - otherwise there will be warning
+ # See https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/CredentialProviderAPI.html#Keystore_Passwords
+ HADOOP_CREDSTORE_PASSWORD: op://services/hdfs/credstore_password
Review Comment:
Can you explain more why we need to care about this in test CI?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] ci: Add hdfs test case for s3 and azure (#4168) [opendal]
Posted by "Xuanwo (via GitHub)" <gi...@apache.org>.
Xuanwo commented on code in PR #4183:
URL: https://github.com/apache/opendal/pull/4183#discussion_r1491882078
##########
.github/services/hdfs/hdfs_default_azblob/action.yml:
##########
@@ -0,0 +1,71 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+name: hdfs_default_azblob
+description: 'Behavior test for hdfs default over azure blob storage'
+
+runs:
+ using: "composite"
+ steps:
+ - name: Setup java env
+ uses: actions/setup-java@v4
+ with:
+ distribution: temurin
+ java-version: "11"
+ - name: Load secrets
+ uses: 1password/load-secrets-action@v1
+ with:
+ export-env: true
+ env:
+ # Need a environment variable with this name to securely encrypt them - otherwise there will be warning
+ # See https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/CredentialProviderAPI.html#Keystore_Passwords
+ HADOOP_CREDSTORE_PASSWORD: op://services/hdfs/credstore_password
Review Comment:
> If no one has chance to read from `/tmp/hdfs/wasb.jceks` in ci machine, this issue will be fine and we don't need `HADOOP_CREDSTORE_PASSWORD`. But otherwise, they will be able to decrypt the real credentials
Yes, no one can read. We trust our committers to not allow such thing happen.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] ci: Add hdfs test case for s3 and azure (#4168) [opendal]
Posted by "ArmandoZ (via GitHub)" <gi...@apache.org>.
ArmandoZ commented on code in PR #4183:
URL: https://github.com/apache/opendal/pull/4183#discussion_r1490501611
##########
.github/services/hdfs/hdfs_default_s3/action.yml:
##########
@@ -0,0 +1,71 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+name: hdfs_default_s3
+description: 'Behavior test for hdfs default over aws s3'
+
+runs:
+ using: "composite"
+ steps:
+ - name: Setup java env
+ uses: actions/setup-java@v4
+ with:
+ distribution: temurin
+ java-version: "11"
+ - name: Load secrets
+ uses: 1password/load-secrets-action@v1
+ with:
+ export-env: true
+ env:
+ # Need a secret with this name to securely encrypt them - otherwise there will be warning
+ # See https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/CredentialProviderAPI.html#Keystore_Passwords
+ HADOOP_CREDSTORE_PASSWORD: op://services/hdfs/credstore_password
+ # TODO: Does this start with / and ends with /
+ OPENDAL_S3_ROOT: op://services/s3/root
+ OPENDAL_S3_BUCKET: op://services/s3/bucket
Review Comment:
Double checking:
- Does `op://services/s3/root` starts and ends with `/`?
- Does `op://services/s3/bucket ` starts and ends with `/`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org