[I] [Bug] checkFlinkEnv do not refresh the hdfs flink lib dir while there are changes in local flink lib dir, [incubator-streampark]

["liuxinjie5211990 (via GitHub)" <gi...@apache.org>]

liuxinjie5211990 opened a new issue, #3524:
URL: https://github.com/apache/incubator-streampark/issues/3524

   ### Search before asking
   
   - [X] I had searched in the [issues](https://github.com/apache/incubator-streampark/issues?q=is%3Aissue+label%3A%22bug%22) and found no similar issues.
   
   
   ### Java Version
   
   java 8
   
   ### Scala Version
   
   2.11.x
   
   ### StreamPark Version
   
   2.1.1
   
   ### Flink Version
   
   1.16.2
   
   ### deploy mode
   
   yarn-application
   
   ### What happened
   
   while we have configured a flink home before and use it to start a flink job with our jar using yarn-application mode.At the same time, this JAR will use a new connector. Since we are building a source code JAR package, we are uploading the connector to the local lib directory of Flink.In this case, when executing the Flink JAR, an error occurs indicating that the class for the connector cannot be found.
   
   The issue arises because the EnvInitializer.checkFlinkEnv method uploads the Flink directory to the HDFS StreamPark workspace only on certain occasions, such as when starting or deploying the application. In the YARN application mode, the /streampark/flink/flink-1.16.2/lib directory is included in the classpath. However, since the checkFlinkEnv method checks whether the directory exists before uploading files (if (!fsOperator.exists(flinkHome))), any subsequent modifications to the local Flink lib directory will not be updated in HDFS.
   
   We can only manually upload any added JARs to HDFS, which is not very user-friendly. This is because, without inspecting the source code, users may not be aware that uploading to this directory（/streampark/flink/flink-1.16.2/lib） is required for working nomally.
   
   ### Error Exception
   
   ```log
   Caused by: java.lang.NoClassDefFoundError: com/ververica/cdc/connectors/postgres/PostgreSQLSource
   
   if (!fsOperator.exists(flinkHome)) {
         log.info("{} is not exists,upload beginning....", flinkHome);
         fsOperator.upload(flinkLocalHome, flinkHome, false, true);
    }
   ```
   
   
   ### Screenshots
   
   ![image](https://github.com/apache/incubator-streampark/assets/57188699/6573f97c-0abb-4d5c-94e2-7b3c56791a72)
   
   
   ### Are you willing to submit PR?
   
   - [X] Yes I am willing to submit a PR!(您是否要贡献这个PR?)
   
   ### Code of Conduct
   
   - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@streampark.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] [Bug] checkFlinkEnv do not refresh the hdfs flink lib dir while there are changes in local flink lib dir, [incubator-streampark]

["ZHI-XINHUA (via GitHub)" <gi...@apache.org>]

ZHI-XINHUA commented on issue #3524:
URL: https://github.com/apache/incubator-streampark/issues/3524#issuecomment-1916524891

   i have the same problem


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@streampark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org