[jira] [Created] (FLINK-34641) Possibility to add Python Files from http/https sources in PythonDriver

["Aleksei Perminov (Jira)" <ji...@apache.org>]

Aleksei Perminov created FLINK-34641:
----------------------------------------

             Summary: Possibility to add Python Files from http/https sources in PythonDriver
                 Key: FLINK-34641
                 URL: https://issues.apache.org/jira/browse/FLINK-34641
             Project: Flink
          Issue Type: New Feature
          Components: flink-contrib
    Affects Versions: 1.18.1, 1.18.0, 1.18.2
            Reporter: Aleksei Perminov


Problem: Now it is impossible to add python files from the external http sources, only fs sources are available. It is not very convenient to add Python packages to the Flink Docker image or external fs every time the new version of the Python job is built, especially when using the Session mode. 

Proposal: Add possibility to handle the http/https protocols in the PythonDriver so that it is possible to add packages from the external Python repositories (e.g. PyPi). 

Example of the solution in the PythonEnvUtils.java
{code:java}
private static void addToPythonPath(PythonEnvironment env, List<Path> pythonFiles) {
...
 if (isValidURL(pythonFile.toString()) && new URL(pythonFile.toString()).getProtocol().startsWith("http")) {
    getFileFromHTTP(pythonFile, targetPath);
} else if (!pythonFile.getFileSystem().isDistributedFS()) {
...
}{code}
 

 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)