You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "pvary (via GitHub)" <gi...@apache.org> on 2023/11/30 13:07:23 UTC
[PR] [FLINK-33556] Test infrastructure for externalized python code [flink]
pvary opened a new pull request, #23843:
URL: https://github.com/apache/flink/pull/23843
## What is the purpose of the change
A few changes are needed in the original python test code, so the externalized connectors could run their tests
## Brief change log
Marks the files with a comment which are used by the externalized connector tests:
- link-python/dev/build-wheels.sh
- flink-python/dev/install_command.sh
- flink-python/dev/lint-python.sh
Adds a possibility to modify the test's classpath in `flink-python/pyflink/pyflink_gateway_server.py`
## Verifying this change
Created the externalized tests for the kafka connector, and hacked this file to the downloaded infra and run the tests successfully.
## Does this pull request potentially affect one of the following parts:
- Dependencies (does it add or upgrade a dependency): no
- The public API, i.e., is any changed class annotated with `@Public(Evolving)`: no
- The serializers: no
- The runtime per-record code paths (performance sensitive): no
- Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: no
- The S3 file system connector: no
## Documentation
- Does this pull request introduce a new feature? no
- If yes, how is the feature documented? not applicable
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-33556] Test infrastructure for externalized python code [flink]
Posted by "gaborgsomogyi (via GitHub)" <gi...@apache.org>.
gaborgsomogyi merged PR #23843:
URL: https://github.com/apache/flink/pull/23843
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-33556] Test infrastructure for externalized python code [flink]
Posted by "gaborgsomogyi (via GitHub)" <gi...@apache.org>.
gaborgsomogyi commented on PR #23843:
URL: https://github.com/apache/flink/pull/23843#issuecomment-1842696709
If there are no further comments then I'm intended to merge this tomorrow.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-33556] Test infrastructure for externalized python code [flink]
Posted by "gaborgsomogyi (via GitHub)" <gi...@apache.org>.
gaborgsomogyi commented on code in PR #23843:
URL: https://github.com/apache/flink/pull/23843#discussion_r1411902788
##########
flink-python/dev/build-wheels.sh:
##########
@@ -13,6 +13,10 @@
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
+
+# The file is directly downloaded and used by the different connector implementations
Review Comment:
I think it would be better to make a deep copy in `flink-connector-shared-utils` repo because I can't really imagine that Flink and all the connectors require the same behaviour.
##########
flink-python/pyflink/pyflink_gateway_server.py:
##########
@@ -216,14 +216,21 @@ def construct_hadoop_classpath(env):
read_from_config(KEY_ENV_HBASE_CONF_DIR, hbase_conf_dir, flink_conf_file))])
-def construct_test_classpath():
+def construct_test_classpath(env):
test_jar_patterns = [
"flink-python/target/test-dependencies/*",
"flink-python/target/artifacts/testDataStream.jar",
"flink-python/target/flink-python*-tests.jar",
]
test_jars = []
- flink_source_root = _find_flink_source_root()
+
+ # Connector tests need to add specific jars to the gateway classpath
+ # flink_source_root = env['FLINK_TEST_LIB_DIR']
+ if 'FLINK_TEST_LIB_DIR' in env:
+ flink_source_root = env['FLINK_TEST_LIB_DIR']
Review Comment:
Maybe we can rename to `test_lib_root` or something similar.
##########
flink-python/dev/build-wheels.sh:
##########
@@ -13,6 +13,10 @@
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
+
+# The file is directly downloaded and used by the different connector implementations
Review Comment:
I think the following functionality is not covered but honestly no idea what is the general consequence:
```
export CFLAGS="-I. -include dev/glibc_version_fix.h"
```
##########
flink-python/dev/lint-python.sh:
##########
@@ -22,6 +22,8 @@
#
# You can refer to the README.MD in ${flink-python} to learn how easy to run the script.
#
+# The file is directly downloaded and used by the different connector implementations
Review Comment:
Same deep copy suggestion here. In mid/long term this script can be modularized and not having a single script.
##########
flink-python/dev/install_command.sh:
##########
@@ -16,6 +16,10 @@
# See the License for the specific language governing permissions and
# limitations under the License.
################################################################################
+
Review Comment:
Same deep copy suggestion here minus `apache-flink-libraries` part.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-33556] Test infrastructure for externalized python code [flink]
Posted by "flinkbot (via GitHub)" <gi...@apache.org>.
flinkbot commented on PR #23843:
URL: https://github.com/apache/flink/pull/23843#issuecomment-1833760576
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "a212a4e003b06ab493f0f782e365fd90bf6959ff",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "a212a4e003b06ab493f0f782e365fd90bf6959ff",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* a212a4e003b06ab493f0f782e365fd90bf6959ff UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org