You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "Samrat002 (via GitHub)" <gi...@apache.org> on 2023/01/28 05:55:32 UTC

[GitHub] [flink] Samrat002 opened a new pull request, #21770: [DRAFT][FLINK-30277][python]Allow PYTHONPATH of Python Worker configurable

Samrat002 opened a new pull request, #21770:
URL: https://github.com/apache/flink/pull/21770

   
   
   ## What is the purpose of the change
   
   Currently, below are the ways Python Worker gets the Python Flink Dependencies.
   
   1. Worker Node's System Python Path (/usr/local/lib64/python3.7/site-packages)
   2. Client passes the python Dependencies through `-pyfs` and `--pyarch` which is localised into `PYTHONPATH` of Python Worker.
   3. Client passes the requirements through `-requirement.txt` which gets installed on Worker Node and added into `PYTHONPATH` of Python Worker.
   
   This change allow `PYTHONPATH` of Python Worker configurable where Admin/Service provider can install the required python Flink dependencies on a custom path (`/usr/lib/pyflink/lib/python3.7/site-packages`) on all Worker Nodes and then set the path in the client machine configuration `flink-conf.yaml`. This way it works without any configurations from the Application Users and also without affecting any other components dependent on System Python Path.
   
   
   ## Brief change log
   
   
   ## Verifying this change
   
   Please make sure both new and modified tests in this PR follows the conventions defined in our code quality guide: https://flink.apache.org/contributing/code-style-and-quality-common.html#testing
   
   *(Please pick either of the following options)*
   
   This change is a trivial rework / code cleanup without any test coverage.
   
   *(or)*
   
   This change is already covered by existing tests, such as *(please describe tests)*.
   
   *(or)*
   
   This change added tests and can be verified as follows:
   
   *(example:)*
     - *Added integration tests for end-to-end deployment with large payloads (100MB)*
     - *Extended integration test for recovery after master (JobManager) failure*
     - *Added test that validates that TaskInfo is transferred only once across recoveries*
     - *Manually verified the change by running a 4 node cluster with 2 JobManagers and 4 TaskManagers, a stateful streaming program, and killing one JobManager and two TaskManagers during the execution, verifying that recovery happens correctly.*
   
   ## Does this pull request potentially affect one of the following parts:
   
     - Dependencies (does it add or upgrade a dependency): (yes / no)
     - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (yes / no)
     - The serializers: (yes / no / don't know)
     - The runtime per-record code paths (performance sensitive): (yes / no / don't know)
     - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (yes / no / don't know)
     - The S3 file system connector: (yes / no / don't know)
   
   ## Documentation
   
     - Does this pull request introduce a new feature? (yes / no)
     - If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [flink] Samrat002 commented on a diff in pull request #21770: [FLINK-30277][python]Allow PYTHONPATH of Python Worker configurable

Posted by "Samrat002 (via GitHub)" <gi...@apache.org>.
Samrat002 commented on code in PR #21770:
URL: https://github.com/apache/flink/pull/21770#discussion_r1101169464


##########
flink-python/src/main/java/org/apache/flink/python/env/AbstractPythonEnvironmentManager.java:
##########
@@ -166,6 +166,14 @@ public Map<String, String> constructEnvironmentVariables(String baseDirectory)
 
         constructFilesDirectory(env, baseDirectory);
 
+        if (this.dependencyInfo.getPythonPath() != null) {
+            List<String> configuredPythonPath = new ArrayList<>();
+            configuredPythonPath.add(this.dependencyInfo.getPythonPath());
+            appendToPythonPath(env, configuredPythonPath);

Review Comment:
   Ideally it should not affect path resolution on TM. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [flink] Samrat002 commented on a diff in pull request #21770: [FLINK-30277][python]Allow PYTHONPATH of Python Worker configurable

Posted by "Samrat002 (via GitHub)" <gi...@apache.org>.
Samrat002 commented on code in PR #21770:
URL: https://github.com/apache/flink/pull/21770#discussion_r1114841299


##########
docs/static/generated/rest_v1_dispatcher.yml:
##########
@@ -6,7 +6,7 @@ info:
   license:
     name: Apache 2.0
     url: https://www.apache.org/licenses/LICENSE-2.0.html
-  version: v1/1.17-SNAPSHOT
+  version: v1/1.18-SNAPSHOT

Review Comment:
   This is related to this change only which is created while generating docs.
   
   ```
   ./mvnw package -Dgenerate-rest-docs -pl flink-docs -am -nsu -DskipTests
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [flink] flinkbot commented on pull request #21770: [FLINK-30277][python]Allow PYTHONPATH of Python Worker configurable

Posted by "flinkbot (via GitHub)" <gi...@apache.org>.
flinkbot commented on PR #21770:
URL: https://github.com/apache/flink/pull/21770#issuecomment-1407303706

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "d10e4322f2fa44d666c447b94364e066009245e8",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "d10e4322f2fa44d666c447b94364e066009245e8",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * d10e4322f2fa44d666c447b94364e066009245e8 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [flink] Samrat002 commented on a diff in pull request #21770: [FLINK-30277][python]Allow PYTHONPATH of Python Worker configurable

Posted by "Samrat002 (via GitHub)" <gi...@apache.org>.
Samrat002 commented on code in PR #21770:
URL: https://github.com/apache/flink/pull/21770#discussion_r1117029594


##########
flink-python/src/main/java/org/apache/flink/python/PythonOptions.java:
##########
@@ -122,6 +122,19 @@ public class PythonOptions {
                                     + "optional parameter exists. The option is equivalent to the command line option "
                                     + "\"-pyreq\".");
 
+    /** The configuration allows user to define python path for client and workers. */
+    public static final ConfigOption<String> PYTHON_PATH =
+            ConfigOptions.key("env.PYTHONPATH")
+                    .stringType()
+                    .noDefaultValue()
+                    .withDescription(
+                            Description.builder()
+                                    .text(
+                                            "Specify the path on the Worker Node where the Flink Python Dependencies are installed, which "
+                                                    + "gets added into the PYTHONPATH of the Python Worker. "
+                                                    + "The option is equivalent to the command line option \"-penv.PYTHONPATH\".")

Review Comment:
   yes 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [flink] HuangXingBo closed pull request #21770: [FLINK-30277][python]Allow PYTHONPATH of Python Worker configurable

Posted by "HuangXingBo (via GitHub)" <gi...@apache.org>.
HuangXingBo closed pull request #21770: [FLINK-30277][python]Allow PYTHONPATH of Python Worker configurable
URL: https://github.com/apache/flink/pull/21770


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [flink] Samrat002 commented on a diff in pull request #21770: [FLINK-30277][python]Allow PYTHONPATH of Python Worker configurable

Posted by "Samrat002 (via GitHub)" <gi...@apache.org>.
Samrat002 commented on code in PR #21770:
URL: https://github.com/apache/flink/pull/21770#discussion_r1117031077


##########
flink-python/src/main/java/org/apache/flink/python/env/PythonDependencyInfo.java:
##########
@@ -100,13 +104,32 @@ public PythonDependencyInfo(
             @Nullable String requirementsCacheDir,
             @Nonnull Map<String, String> archives,
             @Nonnull String pythonExec,
-            @Nonnull String executionMode) {
+            @Nullable String pythonPath) {
+        this(
+                pythonFiles,
+                requirementsFilePath,
+                requirementsCacheDir,
+                archives,
+                pythonExec,
+                PYTHON_EXECUTION_MODE.defaultValue(),
+                pythonPath);
+    }

Review Comment:
   ack 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [flink] Samrat002 commented on pull request #21770: [FLINK-30277][python]Allow PYTHONPATH of Python Worker configurable

Posted by "Samrat002 (via GitHub)" <gi...@apache.org>.
Samrat002 commented on PR #21770:
URL: https://github.com/apache/flink/pull/21770#issuecomment-1441441999

   @flinkbot run azure


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [flink] Samrat002 commented on pull request #21770: [FLINK-30277][python]Allow PYTHONPATH of Python Worker configurable

Posted by "Samrat002 (via GitHub)" <gi...@apache.org>.
Samrat002 commented on PR #21770:
URL: https://github.com/apache/flink/pull/21770#issuecomment-1407309384

   @flinkbot run azure


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [flink] Samrat002 commented on pull request #21770: [FLINK-30277][python]Allow PYTHONPATH of Python Worker configurable

Posted by "Samrat002 (via GitHub)" <gi...@apache.org>.
Samrat002 commented on PR #21770:
URL: https://github.com/apache/flink/pull/21770#issuecomment-1423863102

   @HuangXingBo 
   Please review this small change in free time .


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [flink] Samrat002 commented on a diff in pull request #21770: [FLINK-30277][python]Allow PYTHONPATH of Python Worker configurable

Posted by "Samrat002 (via GitHub)" <gi...@apache.org>.
Samrat002 commented on code in PR #21770:
URL: https://github.com/apache/flink/pull/21770#discussion_r1114841299


##########
docs/static/generated/rest_v1_dispatcher.yml:
##########
@@ -6,7 +6,7 @@ info:
   license:
     name: Apache 2.0
     url: https://www.apache.org/licenses/LICENSE-2.0.html
-  version: v1/1.17-SNAPSHOT
+  version: v1/1.18-SNAPSHOT

Review Comment:
   No related to this change ! some unknown changes from rebase 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [flink] Samrat002 commented on pull request #21770: [FLINK-30277][python]Allow PYTHONPATH of Python Worker configurable

Posted by "Samrat002 (via GitHub)" <gi...@apache.org>.
Samrat002 commented on PR #21770:
URL: https://github.com/apache/flink/pull/21770#issuecomment-1437841702

   @flinkbot run azure


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [flink] Samrat002 commented on a diff in pull request #21770: [FLINK-30277][python]Allow PYTHONPATH of Python Worker configurable

Posted by "Samrat002 (via GitHub)" <gi...@apache.org>.
Samrat002 commented on code in PR #21770:
URL: https://github.com/apache/flink/pull/21770#discussion_r1117042392


##########
docs/layouts/shortcodes/generated/execution_config_configuration.html:
##########
@@ -15,7 +15,7 @@
             <td>The max number of async i/o operation that the async lookup join can trigger.</td>
         </tr>
         <tr>
-            <td><h5>table.exec.async-lookup.output-mode</h5><br> <span class="label label-primary">Batch</span> <span class="label label -primary">Streaming</span></td>
+            <td><h5>table.exec.async-lookup.output-mode</h5><br> <span class="label label-primary">Batch</span> <span class="label label-primary">Streaming</span></td>

Review Comment:
   reverted



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [flink] Samrat002 commented on a diff in pull request #21770: [FLINK-30277][python]Allow PYTHONPATH of Python Worker configurable

Posted by "Samrat002 (via GitHub)" <gi...@apache.org>.
Samrat002 commented on code in PR #21770:
URL: https://github.com/apache/flink/pull/21770#discussion_r1117029594


##########
flink-python/src/main/java/org/apache/flink/python/PythonOptions.java:
##########
@@ -122,6 +122,19 @@ public class PythonOptions {
                                     + "optional parameter exists. The option is equivalent to the command line option "
                                     + "\"-pyreq\".");
 
+    /** The configuration allows user to define python path for client and workers. */
+    public static final ConfigOption<String> PYTHON_PATH =
+            ConfigOptions.key("env.PYTHONPATH")
+                    .stringType()
+                    .noDefaultValue()
+                    .withDescription(
+                            Description.builder()
+                                    .text(
+                                            "Specify the path on the Worker Node where the Flink Python Dependencies are installed, which "
+                                                    + "gets added into the PYTHONPATH of the Python Worker. "
+                                                    + "The option is equivalent to the command line option \"-penv.PYTHONPATH\".")

Review Comment:
   created a separate sub tasks to provide command line support [FLINK-31214](https://issues.apache.org/jira/browse/FLINK-31214). 
   Will raise a different pr for the same . 
   
   Currently removing it from description



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [flink] Samrat002 commented on a diff in pull request #21770: [FLINK-30277][python]Allow PYTHONPATH of Python Worker configurable

Posted by "Samrat002 (via GitHub)" <gi...@apache.org>.
Samrat002 commented on code in PR #21770:
URL: https://github.com/apache/flink/pull/21770#discussion_r1114849025


##########
flink-python/src/main/java/org/apache/flink/client/python/PythonEnvUtils.java:
##########
@@ -324,18 +325,25 @@ private static void addToPythonPath(PythonEnvironment env, List<Path> pythonFile
     /**
      * Starts python process.
      *
+     * @param config configs.
      * @param pythonEnv the python Environment which will be in a process.
      * @param commands the commands that python process will execute.
      * @return the process represent the python process.
      * @throws IOException Thrown if an error occurred when python process start.
      */
     static Process startPythonProcess(
-            PythonEnvironment pythonEnv, List<String> commands, boolean redirectToPipe)
+            ReadableConfig config,
+            PythonEnvironment pythonEnv,
+            List<String> commands,
+            boolean redirectToPipe)
             throws IOException {
         ProcessBuilder pythonProcessBuilder = new ProcessBuilder();
         Map<String, String> env = pythonProcessBuilder.environment();
+
         if (pythonEnv.pythonPath != null) {
-            String defaultPythonPath = env.get("PYTHONPATH");
+            String defaultPythonPath =
+                    config.getOptional(PYTHON_PATH).orElse(env.get("PYTHON_PATH"));

Review Comment:
   > Will the PYTHON_PATH configuration work on both the client and the cluster side?
   
   Yes this is for both client side and cluster side.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [flink] Samrat002 commented on a diff in pull request #21770: [FLINK-30277][python]Allow PYTHONPATH of Python Worker configurable

Posted by "Samrat002 (via GitHub)" <gi...@apache.org>.
Samrat002 commented on code in PR #21770:
URL: https://github.com/apache/flink/pull/21770#discussion_r1114849025


##########
flink-python/src/main/java/org/apache/flink/client/python/PythonEnvUtils.java:
##########
@@ -324,18 +325,25 @@ private static void addToPythonPath(PythonEnvironment env, List<Path> pythonFile
     /**
      * Starts python process.
      *
+     * @param config configs.
      * @param pythonEnv the python Environment which will be in a process.
      * @param commands the commands that python process will execute.
      * @return the process represent the python process.
      * @throws IOException Thrown if an error occurred when python process start.
      */
     static Process startPythonProcess(
-            PythonEnvironment pythonEnv, List<String> commands, boolean redirectToPipe)
+            ReadableConfig config,
+            PythonEnvironment pythonEnv,
+            List<String> commands,
+            boolean redirectToPipe)
             throws IOException {
         ProcessBuilder pythonProcessBuilder = new ProcessBuilder();
         Map<String, String> env = pythonProcessBuilder.environment();
+
         if (pythonEnv.pythonPath != null) {
-            String defaultPythonPath = env.get("PYTHONPATH");
+            String defaultPythonPath =
+                    config.getOptional(PYTHON_PATH).orElse(env.get("PYTHON_PATH"));

Review Comment:
   > Will the PYTHON_PATH configuration work on both the client and the cluster side?
   
   I will make the change keeping `PYTHON_PATH` for both client and cluster side and test it.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [flink] HuangXingBo commented on a diff in pull request #21770: [FLINK-30277][python]Allow PYTHONPATH of Python Worker configurable

Posted by "HuangXingBo (via GitHub)" <gi...@apache.org>.
HuangXingBo commented on code in PR #21770:
URL: https://github.com/apache/flink/pull/21770#discussion_r1116691854


##########
docs/layouts/shortcodes/generated/execution_config_configuration.html:
##########
@@ -15,7 +15,7 @@
             <td>The max number of async i/o operation that the async lookup join can trigger.</td>
         </tr>
         <tr>
-            <td><h5>table.exec.async-lookup.output-mode</h5><br> <span class="label label-primary">Batch</span> <span class="label label -primary">Streaming</span></td>
+            <td><h5>table.exec.async-lookup.output-mode</h5><br> <span class="label label-primary">Batch</span> <span class="label label-primary">Streaming</span></td>

Review Comment:
   unrelated change?



##########
docs/static/generated/rest_v1_dispatcher.yml:
##########
@@ -6,7 +6,7 @@ info:
   license:
     name: Apache 2.0
     url: https://www.apache.org/licenses/LICENSE-2.0.html
-  version: v1/1.17-SNAPSHOT
+  version: v1/1.18-SNAPSHOT

Review Comment:
   I think we can split these unrelated change into different commit.



##########
flink-python/src/main/java/org/apache/flink/python/env/PythonDependencyInfo.java:
##########
@@ -100,13 +104,32 @@ public PythonDependencyInfo(
             @Nullable String requirementsCacheDir,
             @Nonnull Map<String, String> archives,
             @Nonnull String pythonExec,
-            @Nonnull String executionMode) {
+            @Nullable String pythonPath) {
+        this(
+                pythonFiles,
+                requirementsFilePath,
+                requirementsCacheDir,
+                archives,
+                pythonExec,
+                PYTHON_EXECUTION_MODE.defaultValue(),
+                pythonPath);
+    }

Review Comment:
   For compatibility reasons, we should not modify this constructor
   



##########
flink-python/src/main/java/org/apache/flink/python/env/PythonDependencyInfo.java:
##########
@@ -133,6 +156,10 @@ public String getExecutionMode() {
         return executionMode;
     }
 
+    public String getPythonPath() {

Review Comment:
   ```suggestion
       public Optional<String> getPythonPath() {
   ```



##########
flink-python/src/main/java/org/apache/flink/python/PythonOptions.java:
##########
@@ -122,6 +122,19 @@ public class PythonOptions {
                                     + "optional parameter exists. The option is equivalent to the command line option "
                                     + "\"-pyreq\".");
 
+    /** The configuration allows user to define python path for client and workers. */
+    public static final ConfigOption<String> PYTHON_PATH =
+            ConfigOptions.key("env.PYTHONPATH")
+                    .stringType()
+                    .noDefaultValue()
+                    .withDescription(
+                            Description.builder()
+                                    .text(
+                                            "Specify the path on the Worker Node where the Flink Python Dependencies are installed, which "
+                                                    + "gets added into the PYTHONPATH of the Python Worker. "
+                                                    + "The option is equivalent to the command line option \"-penv.PYTHONPATH\".")

Review Comment:
   Do you mean that you will add a new command line option `-penv.PYTHONPATH`? 



##########
flink-python/src/main/java/org/apache/flink/python/env/AbstractPythonEnvironmentManager.java:
##########
@@ -166,6 +166,14 @@ public Map<String, String> constructEnvironmentVariables(String baseDirectory)
 
         constructFilesDirectory(env, baseDirectory);
 
+        if (this.dependencyInfo.getPythonPath() != null) {
+            List<String> configuredPythonPath = new ArrayList<>();
+            configuredPythonPath.add(this.dependencyInfo.getPythonPath());
+            appendToPythonPath(env, configuredPythonPath);
+        }

Review Comment:
   ```suggestion
           if (dependencyInfo.getPythonPath().isPresent()) {
               appendToPythonPath(env, Collections.singletonList(dependencyInfo.getPythonPath().get()));
           }
   ```



##########
flink-python/src/main/java/org/apache/flink/python/PythonOptions.java:
##########
@@ -122,6 +122,19 @@ public class PythonOptions {
                                     + "optional parameter exists. The option is equivalent to the command line option "
                                     + "\"-pyreq\".");
 
+    /** The configuration allows user to define python path for client and workers. */
+    public static final ConfigOption<String> PYTHON_PATH =
+            ConfigOptions.key("env.PYTHONPATH")

Review Comment:
   `env.PYTHONPATH` is a bit odd as an option name. We might change the name to something like `python.pythonpath`, consistent with other python options



##########
flink-python/src/main/java/org/apache/flink/python/env/PythonDependencyInfo.java:
##########
@@ -133,6 +156,10 @@ public String getExecutionMode() {
         return executionMode;
     }
 
+    public String getPythonPath() {
+        return pythonPath;

Review Comment:
   ```suggestion
           return Optional.ofNullable(pythonPath);
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [flink] hlteoh37 commented on a diff in pull request #21770: [FLINK-30277][python]Allow PYTHONPATH of Python Worker configurable

Posted by "hlteoh37 (via GitHub)" <gi...@apache.org>.
hlteoh37 commented on code in PR #21770:
URL: https://github.com/apache/flink/pull/21770#discussion_r1101136689


##########
flink-python/src/main/java/org/apache/flink/python/env/AbstractPythonEnvironmentManager.java:
##########
@@ -166,6 +166,14 @@ public Map<String, String> constructEnvironmentVariables(String baseDirectory)
 
         constructFilesDirectory(env, baseDirectory);
 
+        if (this.dependencyInfo.getPythonPath() != null) {
+            List<String> configuredPythonPath = new ArrayList<>();
+            configuredPythonPath.add(this.dependencyInfo.getPythonPath());
+            appendToPythonPath(env, configuredPythonPath);

Review Comment:
   IS the PYTHONPATH used elsewhere? Will this affect any other path resolution on the TM?



##########
flink-python/src/main/java/org/apache/flink/python/PythonOptions.java:
##########
@@ -151,6 +151,14 @@ public class PythonOptions {
                                     + "Please ensure that the specified environment meets the above requirements. The "
                                     + "option is equivalent to the command line option \"-pyexec\".");
 
+    public static final ConfigOption<String> PYTHON_PATH =
+            ConfigOptions.key("env.PYTHONPATH")
+                    .stringType()
+                    .noDefaultValue()
+                    .withDescription(
+                            "Specify the path on the Worker Node where the Flink Python Dpenencies are installed which "

Review Comment:
   ```suggestion
                               "Specify the path on the Worker Node where the Flink Python Dependencies are installed which "
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [flink] HuangXingBo commented on a diff in pull request #21770: [FLINK-30277][python]Allow PYTHONPATH of Python Worker configurable

Posted by "HuangXingBo (via GitHub)" <gi...@apache.org>.
HuangXingBo commented on code in PR #21770:
URL: https://github.com/apache/flink/pull/21770#discussion_r1117925077


##########
flink-python/src/main/java/org/apache/flink/client/python/PythonEnvUtils.java:
##########
@@ -212,6 +213,15 @@ static PythonEnvironment preparePythonEnvironment(
                             });
         }
 
+        // 4. append configured env.PYTHONPATH to the PYTHONPATH.

Review Comment:
   ```suggestion
           // 4. append configured python.pythonpath to the PYTHONPATH.
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [flink] Samrat002 commented on a diff in pull request #21770: [FLINK-30277][python]Allow PYTHONPATH of Python Worker configurable

Posted by "Samrat002 (via GitHub)" <gi...@apache.org>.
Samrat002 commented on code in PR #21770:
URL: https://github.com/apache/flink/pull/21770#discussion_r1117029166


##########
docs/static/generated/rest_v1_dispatcher.yml:
##########
@@ -6,7 +6,7 @@ info:
   license:
     name: Apache 2.0
     url: https://www.apache.org/licenses/LICENSE-2.0.html
-  version: v1/1.17-SNAPSHOT
+  version: v1/1.18-SNAPSHOT

Review Comment:
   ack i will remove it 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [flink] Samrat002 commented on pull request #21770: [FLINK-30277][python]Allow PYTHONPATH of Python Worker configurable

Posted by "Samrat002 (via GitHub)" <gi...@apache.org>.
Samrat002 commented on PR #21770:
URL: https://github.com/apache/flink/pull/21770#issuecomment-1438005348

   Test failure associated ticket https://issues.apache.org/jira/browse/FLINK-31120


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [flink] Samrat002 commented on a diff in pull request #21770: [FLINK-30277][python]Allow PYTHONPATH of Python Worker configurable

Posted by "Samrat002 (via GitHub)" <gi...@apache.org>.
Samrat002 commented on code in PR #21770:
URL: https://github.com/apache/flink/pull/21770#discussion_r1114841299


##########
docs/static/generated/rest_v1_dispatcher.yml:
##########
@@ -6,7 +6,7 @@ info:
   license:
     name: Apache 2.0
     url: https://www.apache.org/licenses/LICENSE-2.0.html
-  version: v1/1.17-SNAPSHOT
+  version: v1/1.18-SNAPSHOT

Review Comment:
   This is related to this feature change only.
   
   This doc update is generated after running the below command
   
   ```
   ./mvnw package -Dgenerate-rest-docs -pl flink-docs -am -nsu -DskipTests
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [flink] Samrat002 commented on a diff in pull request #21770: [FLINK-30277][python]Allow PYTHONPATH of Python Worker configurable

Posted by "Samrat002 (via GitHub)" <gi...@apache.org>.
Samrat002 commented on code in PR #21770:
URL: https://github.com/apache/flink/pull/21770#discussion_r1117029594


##########
flink-python/src/main/java/org/apache/flink/python/PythonOptions.java:
##########
@@ -122,6 +122,19 @@ public class PythonOptions {
                                     + "optional parameter exists. The option is equivalent to the command line option "
                                     + "\"-pyreq\".");
 
+    /** The configuration allows user to define python path for client and workers. */
+    public static final ConfigOption<String> PYTHON_PATH =
+            ConfigOptions.key("env.PYTHONPATH")
+                    .stringType()
+                    .noDefaultValue()
+                    .withDescription(
+                            Description.builder()
+                                    .text(
+                                            "Specify the path on the Worker Node where the Flink Python Dependencies are installed, which "
+                                                    + "gets added into the PYTHONPATH of the Python Worker. "
+                                                    + "The option is equivalent to the command line option \"-penv.PYTHONPATH\".")

Review Comment:
   yes ! wanted to add commandline option for user. i will create a separate jira for that purpose . 
   i will remove the description for now 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [flink] Samrat002 commented on pull request #21770: [FLINK-30277][python]Allow PYTHONPATH of Python Worker configurable

Posted by "Samrat002 (via GitHub)" <gi...@apache.org>.
Samrat002 commented on PR #21770:
URL: https://github.com/apache/flink/pull/21770#issuecomment-1441397567

   Thank you for initially reviewing the changes. @HuangXingBo 
   I have made updates in the PR as per comments/suggestions. 
   Please review the changes whenever time .  


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [flink] Samrat002 closed pull request #21770: [FLINK-30277][python]Allow PYTHONPATH of Python Worker configurable

Posted by "Samrat002 (via GitHub)" <gi...@apache.org>.
Samrat002 closed pull request #21770: [FLINK-30277][python]Allow PYTHONPATH of Python Worker configurable
URL: https://github.com/apache/flink/pull/21770


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [flink] Samrat002 commented on a diff in pull request #21770: [FLINK-30277][python]Allow PYTHONPATH of Python Worker configurable

Posted by "Samrat002 (via GitHub)" <gi...@apache.org>.
Samrat002 commented on code in PR #21770:
URL: https://github.com/apache/flink/pull/21770#discussion_r1115352901


##########
flink-python/src/main/java/org/apache/flink/python/env/AbstractPythonEnvironmentManager.java:
##########
@@ -166,6 +166,14 @@ public Map<String, String> constructEnvironmentVariables(String baseDirectory)
 
         constructFilesDirectory(env, baseDirectory);
 
+        if (this.dependencyInfo.getPythonPath() != null) {
+            List<String> configuredPythonPath = new ArrayList<>();
+            configuredPythonPath.add(this.dependencyInfo.getPythonPath());
+            appendToPythonPath(env, configuredPythonPath);

Review Comment:
   This is already existing path. code executes append only operation, this doesn't modifies any pre configured python/path related configs and does only the append operation that user has configured 
   
   



##########
flink-python/src/main/java/org/apache/flink/python/env/AbstractPythonEnvironmentManager.java:
##########
@@ -166,6 +166,14 @@ public Map<String, String> constructEnvironmentVariables(String baseDirectory)
 
         constructFilesDirectory(env, baseDirectory);
 
+        if (this.dependencyInfo.getPythonPath() != null) {
+            List<String> configuredPythonPath = new ArrayList<>();
+            configuredPythonPath.add(this.dependencyInfo.getPythonPath());
+            appendToPythonPath(env, configuredPythonPath);

Review Comment:
   Ideally it should not affect path resolution on TM. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [flink] HuangXingBo commented on a diff in pull request #21770: [FLINK-30277][python]Allow PYTHONPATH of Python Worker configurable

Posted by "HuangXingBo (via GitHub)" <gi...@apache.org>.
HuangXingBo commented on code in PR #21770:
URL: https://github.com/apache/flink/pull/21770#discussion_r1113953192


##########
flink-python/src/main/java/org/apache/flink/client/python/PythonEnvUtils.java:
##########
@@ -324,18 +325,25 @@ private static void addToPythonPath(PythonEnvironment env, List<Path> pythonFile
     /**
      * Starts python process.
      *
+     * @param config configs.
      * @param pythonEnv the python Environment which will be in a process.
      * @param commands the commands that python process will execute.
      * @return the process represent the python process.
      * @throws IOException Thrown if an error occurred when python process start.
      */
     static Process startPythonProcess(
-            PythonEnvironment pythonEnv, List<String> commands, boolean redirectToPipe)
+            ReadableConfig config,
+            PythonEnvironment pythonEnv,
+            List<String> commands,
+            boolean redirectToPipe)
             throws IOException {
         ProcessBuilder pythonProcessBuilder = new ProcessBuilder();
         Map<String, String> env = pythonProcessBuilder.environment();
+
         if (pythonEnv.pythonPath != null) {
-            String defaultPythonPath = env.get("PYTHONPATH");
+            String defaultPythonPath =
+                    config.getOptional(PYTHON_PATH).orElse(env.get("PYTHON_PATH"));

Review Comment:
   ```suggestion
                       config.getOptional(PYTHON_PATH).orElse(env.get("PYTHONPATH"));
   ```



##########
docs/static/generated/rest_v1_dispatcher.yml:
##########
@@ -6,7 +6,7 @@ info:
   license:
     name: Apache 2.0
     url: https://www.apache.org/licenses/LICENSE-2.0.html
-  version: v1/1.17-SNAPSHOT
+  version: v1/1.18-SNAPSHOT

Review Comment:
   Modifications not related to this feature?



##########
flink-python/src/main/java/org/apache/flink/client/python/PythonEnvUtils.java:
##########
@@ -324,18 +325,25 @@ private static void addToPythonPath(PythonEnvironment env, List<Path> pythonFile
     /**
      * Starts python process.
      *
+     * @param config configs.
      * @param pythonEnv the python Environment which will be in a process.
      * @param commands the commands that python process will execute.
      * @return the process represent the python process.
      * @throws IOException Thrown if an error occurred when python process start.
      */
     static Process startPythonProcess(
-            PythonEnvironment pythonEnv, List<String> commands, boolean redirectToPipe)
+            ReadableConfig config,
+            PythonEnvironment pythonEnv,
+            List<String> commands,
+            boolean redirectToPipe)
             throws IOException {
         ProcessBuilder pythonProcessBuilder = new ProcessBuilder();
         Map<String, String> env = pythonProcessBuilder.environment();
+
         if (pythonEnv.pythonPath != null) {
-            String defaultPythonPath = env.get("PYTHONPATH");
+            String defaultPythonPath =
+                    config.getOptional(PYTHON_PATH).orElse(env.get("PYTHON_PATH"));

Review Comment:
   And I prefer these logic is in `preparePythonEnvironment` rather than adding another param `ReadableConfig` to `startPythonProcess`



##########
flink-python/src/main/java/org/apache/flink/client/python/PythonEnvUtils.java:
##########
@@ -324,18 +325,25 @@ private static void addToPythonPath(PythonEnvironment env, List<Path> pythonFile
     /**
      * Starts python process.
      *
+     * @param config configs.
      * @param pythonEnv the python Environment which will be in a process.
      * @param commands the commands that python process will execute.
      * @return the process represent the python process.
      * @throws IOException Thrown if an error occurred when python process start.
      */
     static Process startPythonProcess(
-            PythonEnvironment pythonEnv, List<String> commands, boolean redirectToPipe)
+            ReadableConfig config,
+            PythonEnvironment pythonEnv,
+            List<String> commands,
+            boolean redirectToPipe)
             throws IOException {
         ProcessBuilder pythonProcessBuilder = new ProcessBuilder();
         Map<String, String> env = pythonProcessBuilder.environment();
+
         if (pythonEnv.pythonPath != null) {
-            String defaultPythonPath = env.get("PYTHONPATH");
+            String defaultPythonPath =
+                    config.getOptional(PYTHON_PATH).orElse(env.get("PYTHON_PATH"));

Review Comment:
   Will the `PYTHON_PATH` configuration work on both the client and the cluster side?
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org