You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@opendal.apache.org by "Xuanwo (via GitHub)" <gi...@apache.org> on 2023/10/11 16:46:00 UTC
[PR] refactor: Introduce OpenDAL Workflow Planner [incubator-opendal]
Xuanwo opened a new pull request, #3258:
URL: https://github.com/apache/incubator-opendal/pull/3258
This PR aims to introduce the OpenDAL workflow planner, allowing us to reuse our test setup across different components.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] refactor: Introduce OpenDAL Workflow Planner [incubator-opendal]
Posted by "laipz8200 (via GitHub)" <gi...@apache.org>.
laipz8200 commented on PR #3258:
URL: https://github.com/apache/incubator-opendal/pull/3258#issuecomment-1760595876
> Is it possible to make them accept s3/memory in command directly so that we can concat them like cargo nextest run services_${{ inputs.service }}?
pytest has custom mark like `pytest.mark.s3`. By the way, -k is also worked for this scene, `pytest -vk memory` will select TestMemory too.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] refactor: Introduce OpenDAL Workflow Planner [incubator-opendal]
Posted by "Zheaoli (via GitHub)" <gi...@apache.org>.
Zheaoli commented on code in PR #3258:
URL: https://github.com/apache/incubator-opendal/pull/3258#discussion_r1357370267
##########
.github/actions/test-core/action.yaml:
##########
@@ -0,0 +1,51 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+name: Test Core
+description: 'Test Core with given setup and service'
+inputs:
+ setup:
+ description: "The setup action for test"
+ service:
+ description: "The service to test"
+ secrets:
+ description: "The secrets for test"
+ default: "{}"
+
+runs:
+ using: "composite"
+ steps:
+ - name: Setup
+ shell: bash
+ run: |
+ mkdir -p ./dynamic_test_core &&
+ cat <<EOF >./dynamic_test_core/action.yml
Review Comment:
It would be great to make it as a template instead of hard coding in the bash.
I think we can polish it later
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] refactor: Introduce OpenDAL Workflow Planner [incubator-opendal]
Posted by "tisonkun (via GitHub)" <gi...@apache.org>.
tisonkun commented on code in PR #3258:
URL: https://github.com/apache/incubator-opendal/pull/3258#discussion_r1357270315
##########
scripts/workflow_planner.py:
##########
@@ -0,0 +1,133 @@
+#!/usr/bin/env python3
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import sys
+import json
+import os
+from pathlib import Path
+
+
+# Check if we can access secrets based on github context.
+def check_secrets(github):
Review Comment:
Unused?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] refactor: Introduce OpenDAL Workflow Planner [incubator-opendal]
Posted by "Xuanwo (via GitHub)" <gi...@apache.org>.
Xuanwo merged PR #3258:
URL: https://github.com/apache/incubator-opendal/pull/3258
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] refactor: Introduce OpenDAL Workflow Planner [incubator-opendal]
Posted by "Xuanwo (via GitHub)" <gi...@apache.org>.
Xuanwo commented on PR #3258:
URL: https://github.com/apache/incubator-opendal/pull/3258#issuecomment-1760684763
> To match with `abc_def`, we can rename our class name or give a custom mark on it, but I think it cannot contain `-`. (`_` is ok)
Okay, let's handle snake case conversion on the actions side instead.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] refactor: Introduce OpenDAL Workflow Planner [incubator-opendal]
Posted by "tisonkun (via GitHub)" <gi...@apache.org>.
tisonkun commented on PR #3258:
URL: https://github.com/apache/incubator-opendal/pull/3258#issuecomment-1760210778
> make them accept s3/memory in command directly
Of course with JUnit's `@Tag`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] refactor: Introduce OpenDAL Workflow Planner [incubator-opendal]
Posted by "laipz8200 (via GitHub)" <gi...@apache.org>.
laipz8200 commented on PR #3258:
URL: https://github.com/apache/incubator-opendal/pull/3258#issuecomment-1760682179
> > By the way, -k is also worked for this scene, `pytest -vk memory` will select TestMemory too.
>
> Can we use `abc-def` with `TestAbcDef`?
Unfortunately, it cannot. pytest matches test cases by class name, function name, or custom mark (case-insensitive).
To match with `abc_def`, we can rename our class name or give a custom mark on it, but I think it cannot contain `-`. (`_` is ok)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] refactor: Introduce OpenDAL Workflow Planner [incubator-opendal]
Posted by "Xuanwo (via GitHub)" <gi...@apache.org>.
Xuanwo commented on PR #3258:
URL: https://github.com/apache/incubator-opendal/pull/3258#issuecomment-1760658523
> By the way, -k is also worked for this scene, `pytest -vk memory` will select TestMemory too.
Can we use `abc-def` with `TestAbcDef`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] refactor: Introduce OpenDAL Workflow Planner [incubator-opendal]
Posted by "tisonkun (via GitHub)" <gi...@apache.org>.
tisonkun commented on PR #3258:
URL: https://github.com/apache/incubator-opendal/pull/3258#issuecomment-1760208656
> How to implement the build/upload/download/test logic for java
Something like this -
https://github.com/korandoru/hawkeye/pull/99/commits/386d8158e7de9d991740109b98d31aa8f2068fbd
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] refactor: Introduce OpenDAL Workflow Planner [incubator-opendal]
Posted by "Zheaoli (via GitHub)" <gi...@apache.org>.
Zheaoli commented on code in PR #3258:
URL: https://github.com/apache/incubator-opendal/pull/3258#discussion_r1357368825
##########
scripts/workflow_planner.py:
##########
@@ -0,0 +1,133 @@
+#!/usr/bin/env python3
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import sys
+import json
+import os
+from pathlib import Path
+
+
+# Check if we can access secrets based on github context.
+def check_secrets(github):
+ if (
+ github["event_name"] == "push"
+ and github["repository"] == "apache/incubator-opendal"
+ ):
+ return True
+ if (
+ github["event_name"] == "pull_request"
+ and github["event"]["pull_request"]["head"]["full_name"]
+ == "apache/incubator-opendal"
+ and not github["event"]["pull_request"]["head"]["repo"]["fork"]
+ ):
+ return True
+ return False
+
+
+def get_provided_cases():
+ root_dir = ".github/services"
+
+ cases = [
+ {
+ "service": service,
+ "setup": setup,
+ "content": Path(
+ os.path.join(root_dir, service, setup, "action.yml")
+ ).read_text(),
+ }
+ for service in os.listdir(root_dir)
+ for setup in os.listdir(os.path.join(root_dir, service))
+ ]
+
+ # Check if this workflow needs to read secrets.
+ #
+ # We will check if pattern `secrets.XXX` exist in content.
+ if not bool(os.getenv("GITHUB_HAS_SECRETS")):
+ cases[:] = [v for v in cases if "secrets." not in v["content"]]
+
+ return cases
+
+
+def calculate_core_cases(cases, changed_files):
+ # If any of the core workflow changed, we will run all cases.
+ for p in [
+ ".github/workflows/core_test.yml",
+ ".github/workflows/test_planner.yml",
+ ".github/actions/test-core",
+ ]:
+ if p in changed_files:
+ return cases
+
+ # If any of the core files changed, we will run all cases.
+ if any(
+ p.startswith("core/src/") and not p.startswith("core/src/services")
+ for p in changed_files
+ ):
+ return cases
+ if any(p.startswith("core/tests/") for p in changed_files):
+ return cases
+
+ # Filter all cases that not shown un in changed files
+ cases = [v for v in cases if any(v["service"] in p for p in changed_files)]
Review Comment:
I think it would be great not to use complicate list comprehension here.
It will make people who don't familiar with python confuse.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org