You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@kyuubi.apache.org by "cutiechi (via GitHub)" <gi...@apache.org> on 2025/06/25 11:51:54 UTC
[PR] Skip Hadoop classpath check if flink-shaded-hadoop jar exists in Flink lib directory [kyuubi]
cutiechi opened a new pull request, #7113:
URL: https://github.com/apache/kyuubi/pull/7113
<!--
Thanks for sending a pull request!
Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/master/contributing/code/index.html
2. If the PR is related to an issue in https://github.com/apache/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->
### Why are the changes needed?
<!--
Please clarify why the changes are needed. For instance,
1. If you propose a new API, clarify the use case for a new API.
2. If you fix a bug, you can clarify why it is a bug, and what versions are affected.
-->
This change addresses an issue where the Flink engine in Kyuubi would perform a Hadoop classpath check even when a `flink-shaded-hadoop` jar is already present in the Flink `lib` directory. In such cases, the check is unnecessary and may cause confusion or warnings in environments where the shaded jar is used instead of a full Hadoop classpath. By skipping the check when a `flink-shaded-hadoop` jar exists, we improve compatibility and reduce unnecessary log output.
### How was this patch tested?
<!--
If tests were added, say they were added here. Please make sure to add some test cases that check the changes thoroughly including negative and positive cases if possible.
If it was tested in a way different from regular unit tests, please clarify how you tested step by step, ideally copy and paste-able, so that other reviewers can test and check, and descendants can verify in the future.
If tests were not added, please describe why they were not added and/or why it was difficult to add.
-->
The patch was tested by deploying Kyuubi with a Flink environment that includes a `flink-shaded-hadoop` jar in the `lib` directory and verifying that the classpath check is correctly skipped. Additional tests ensured that the check still occurs when neither the Hadoop classpath nor the shaded jar is present. Unit tests and manual verification steps were performed to confirm the fix.
### Was this patch authored or co-authored using generative AI tooling?
<!--
If a generative AI tooling has been used in the process of authoring this patch, please include
phrase 'Generated-by: ' followed by the name of the tool and its version.
If no, write 'No'.
Please refer to the [ASF Generative Tooling Guidance](https://www.apache.org/legal/generative-tooling.html) for details.
-->
No.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] Skip Hadoop classpath check if flink-shaded-hadoop jar exists in Flink lib directory [kyuubi]
Posted by "cutiechi (via GitHub)" <gi...@apache.org>.
cutiechi commented on code in PR #7113:
URL: https://github.com/apache/kyuubi/pull/7113#discussion_r2168166143
##########
docs/deployment/engine_on_yarn.md:
##########
@@ -219,7 +219,11 @@ $ echo "export HADOOP_CONF_DIR=/path/to/hadoop/conf" >> $KYUUBI_HOME/conf/kyuubi
#### Required Environment Variable
-The `FLINK_HADOOP_CLASSPATH` is required, too.
+The `FLINK_HADOOP_CLASSPATH` is required unless the necessary Hadoop client jars (such as `hadoop-client` or
+`flink-shaded-hadoop`) have already been placed in the Flink lib directory (`$FLINK_HOME/lib`).
+
+If the jars are not present in `$FLINK_HOME/lib`, you must set `FLINK_HADOOP_CLASSPATH` to include the appropriate
+Hadoop client jars.
Review Comment:
Thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] Skip Hadoop classpath check if flink-shaded-hadoop jar exists in Flink lib directory [kyuubi]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on code in PR #7113:
URL: https://github.com/apache/kyuubi/pull/7113#discussion_r2168172592
##########
docs/deployment/engine_on_yarn.md:
##########
@@ -219,7 +219,11 @@ $ echo "export HADOOP_CONF_DIR=/path/to/hadoop/conf" >> $KYUUBI_HOME/conf/kyuubi
#### Required Environment Variable
-The `FLINK_HADOOP_CLASSPATH` is required, too.
+The `FLINK_HADOOP_CLASSPATH` is required unless the necessary Hadoop client jars (such as `hadoop-client` or
+`flink-shaded-hadoop`) have already been placed in the Flink lib directory (`$FLINK_HOME/lib`).
+
+If the jars are not present in `$FLINK_HOME/lib`, you must set `FLINK_HADOOP_CLASSPATH` to include the appropriate
+Hadoop client jars.
Review Comment:
nvm, it's fine
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] Skip Hadoop classpath check if flink-shaded-hadoop jar exists in Flink lib directory [kyuubi]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on code in PR #7113:
URL: https://github.com/apache/kyuubi/pull/7113#discussion_r2166624423
##########
kyuubi-server/src/main/scala/org/apache/kyuubi/engine/flink/FlinkProcessBuilder.scala:
##########
@@ -228,7 +228,21 @@ class FlinkProcessBuilder(
hadoopCp.foreach(classpathEntries.add)
Review Comment:
when `hasHadoopJar` is true, we don't need to append FLINK_HADOOP_CLASSPATH, right?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] Skip Hadoop classpath check if flink-shaded-hadoop jar exists in Flink lib directory [kyuubi]
Posted by "cutiechi (via GitHub)" <gi...@apache.org>.
cutiechi commented on PR #7113:
URL: https://github.com/apache/kyuubi/pull/7113#issuecomment-3005005755
@pan3793 Please help me review again, thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] Skip Hadoop classpath check if flink-shaded-hadoop jar exists in Flink lib directory [kyuubi]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on PR #7113:
URL: https://github.com/apache/kyuubi/pull/7113#issuecomment-3027154165
Thanks, merged to master/1.10
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] Skip Hadoop classpath check if flink-shaded-hadoop jar exists in Flink lib directory [kyuubi]
Posted by "cutiechi (via GitHub)" <gi...@apache.org>.
cutiechi commented on PR #7113:
URL: https://github.com/apache/kyuubi/pull/7113#issuecomment-3026248981
@pan3793 Hi, I noticed that this pull request has not been merged yet. Is there anything else I should modify or improve? Please let me know if any further changes are needed. Thank you!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] Skip Hadoop classpath check if flink-shaded-hadoop jar exists in Flink lib directory [kyuubi]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on code in PR #7113:
URL: https://github.com/apache/kyuubi/pull/7113#discussion_r2168028406
##########
kyuubi-server/src/main/scala/org/apache/kyuubi/engine/flink/FlinkProcessBuilder.scala:
##########
@@ -228,9 +228,24 @@ class FlinkProcessBuilder(
hadoopCp.foreach(classpathEntries.add)
val extraCp = conf.get(ENGINE_FLINK_EXTRA_CLASSPATH)
extraCp.foreach(classpathEntries.add)
- if (hadoopCp.isEmpty && extraCp.isEmpty) {
- warn(s"The conf of ${FLINK_HADOOP_CLASSPATH_KEY} and " +
- s"${ENGINE_FLINK_EXTRA_CLASSPATH.key} is empty.")
+
+ val hasHadoopJar = Paths.get(flinkHome)
+ .resolve("lib")
+ .toFile
+ .listFiles(new FilenameFilter {
+ override def accept(dir: File, name: String): Boolean = {
+ name.startsWith("hadoop-client") ||
+ name.startsWith("flink-shaded-hadoop")
+ }
+ }).nonEmpty
+
+ if (!hasHadoopJar) {
+ hadoopCp.foreach(classpathEntries.add)
+ }
+
+ if (!hasHadoopJar && hadoopCp.isEmpty && extraCp.isEmpty) {
+ warn(s"No Hadoop jar detected, and the conf of ${FLINK_HADOOP_CLASSPATH_KEY} " +
Review Comment:
No hadoop client jars are found in ${flinkHome}/lib, and ...
update the following KyuubiException message too.
do we need to update docs too.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] Skip Hadoop classpath check if flink-shaded-hadoop jar exists in Flink lib directory [kyuubi]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on code in PR #7113:
URL: https://github.com/apache/kyuubi/pull/7113#discussion_r2168029432
##########
kyuubi-server/src/main/scala/org/apache/kyuubi/engine/flink/FlinkProcessBuilder.scala:
##########
@@ -228,7 +228,21 @@ class FlinkProcessBuilder(
hadoopCp.foreach(classpathEntries.add)
Review Comment:
do you forget to delete this line?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] Skip Hadoop classpath check if flink-shaded-hadoop jar exists in Flink lib directory [kyuubi]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 closed pull request #7113: Skip Hadoop classpath check if flink-shaded-hadoop jar exists in Flink lib directory
URL: https://github.com/apache/kyuubi/pull/7113
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] Skip Hadoop classpath check if flink-shaded-hadoop jar exists in Flink lib directory [kyuubi]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on code in PR #7113:
URL: https://github.com/apache/kyuubi/pull/7113#discussion_r2168161638
##########
docs/deployment/engine_on_yarn.md:
##########
@@ -219,7 +219,11 @@ $ echo "export HADOOP_CONF_DIR=/path/to/hadoop/conf" >> $KYUUBI_HOME/conf/kyuubi
#### Required Environment Variable
-The `FLINK_HADOOP_CLASSPATH` is required, too.
+The `FLINK_HADOOP_CLASSPATH` is required unless the necessary Hadoop client jars (such as `hadoop-client` or
+`flink-shaded-hadoop`) have already been placed in the Flink lib directory (`$FLINK_HOME/lib`).
+
+If the jars are not present in `$FLINK_HOME/lib`, you must set `FLINK_HADOOP_CLASSPATH` to include the appropriate
+Hadoop client jars.
Review Comment:
this line looks redundant, let's keep the words simple.
```suggestion
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] Skip Hadoop classpath check if flink-shaded-hadoop jar exists in Flink lib directory [kyuubi]
Posted by "cutiechi (via GitHub)" <gi...@apache.org>.
cutiechi commented on PR #7113:
URL: https://github.com/apache/kyuubi/pull/7113#issuecomment-3007135505
@pan3793 Thank you for your thorough review! I’ve updated the code and exception messages as suggested, and also added the relevant documentation. Please let me know if there are any further improvements needed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] Skip Hadoop classpath check if flink-shaded-hadoop jar exists in Flink lib directory [kyuubi]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on code in PR #7113:
URL: https://github.com/apache/kyuubi/pull/7113#discussion_r2168161638
##########
docs/deployment/engine_on_yarn.md:
##########
@@ -219,7 +219,11 @@ $ echo "export HADOOP_CONF_DIR=/path/to/hadoop/conf" >> $KYUUBI_HOME/conf/kyuubi
#### Required Environment Variable
-The `FLINK_HADOOP_CLASSPATH` is required, too.
+The `FLINK_HADOOP_CLASSPATH` is required unless the necessary Hadoop client jars (such as `hadoop-client` or
+`flink-shaded-hadoop`) have already been placed in the Flink lib directory (`$FLINK_HOME/lib`).
+
+If the jars are not present in `$FLINK_HOME/lib`, you must set `FLINK_HADOOP_CLASSPATH` to include the appropriate
+Hadoop client jars.
Review Comment:
~~this line looks redundant, let's keep the words simple.~~
```suggestion
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] Skip Hadoop classpath check if flink-shaded-hadoop jar exists in Flink lib directory [kyuubi]
Posted by "cutiechi (via GitHub)" <gi...@apache.org>.
cutiechi commented on PR #7113:
URL: https://github.com/apache/kyuubi/pull/7113#issuecomment-3026387365
> @cutiechi there are test failures in `FlinkProcessBuilderSuite`, can you fix it?
Thanks for pointing it out! I will look into the test failures in `FlinkProcessBuilderSuite` and provide a fix as soon as possible.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] Skip Hadoop classpath check if flink-shaded-hadoop jar exists in Flink lib directory [kyuubi]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on code in PR #7113:
URL: https://github.com/apache/kyuubi/pull/7113#discussion_r2168028406
##########
kyuubi-server/src/main/scala/org/apache/kyuubi/engine/flink/FlinkProcessBuilder.scala:
##########
@@ -228,9 +228,24 @@ class FlinkProcessBuilder(
hadoopCp.foreach(classpathEntries.add)
val extraCp = conf.get(ENGINE_FLINK_EXTRA_CLASSPATH)
extraCp.foreach(classpathEntries.add)
- if (hadoopCp.isEmpty && extraCp.isEmpty) {
- warn(s"The conf of ${FLINK_HADOOP_CLASSPATH_KEY} and " +
- s"${ENGINE_FLINK_EXTRA_CLASSPATH.key} is empty.")
+
+ val hasHadoopJar = Paths.get(flinkHome)
+ .resolve("lib")
+ .toFile
+ .listFiles(new FilenameFilter {
+ override def accept(dir: File, name: String): Boolean = {
+ name.startsWith("hadoop-client") ||
+ name.startsWith("flink-shaded-hadoop")
+ }
+ }).nonEmpty
+
+ if (!hasHadoopJar) {
+ hadoopCp.foreach(classpathEntries.add)
+ }
+
+ if (!hasHadoopJar && hadoopCp.isEmpty && extraCp.isEmpty) {
+ warn(s"No Hadoop jar detected, and the conf of ${FLINK_HADOOP_CLASSPATH_KEY} " +
Review Comment:
No hadoop client jars are found in ${flinkHome}/lib, and ...
update the following KyuubiException message too.
do we need to update docs too?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] Skip Hadoop classpath check if flink-shaded-hadoop jar exists in Flink lib directory [kyuubi]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on code in PR #7113:
URL: https://github.com/apache/kyuubi/pull/7113#discussion_r2166622488
##########
kyuubi-server/src/main/scala/org/apache/kyuubi/engine/flink/FlinkProcessBuilder.scala:
##########
@@ -228,7 +228,21 @@ class FlinkProcessBuilder(
hadoopCp.foreach(classpathEntries.add)
val extraCp = conf.get(ENGINE_FLINK_EXTRA_CLASSPATH)
extraCp.foreach(classpathEntries.add)
- if (hadoopCp.isEmpty && extraCp.isEmpty) {
+
+ val flinkLibDir = Paths.get(flinkHome)
+ .resolve("lib")
+ .toFile
+
+ val hadoopJars = Option(flinkLibDir.listFiles(new FilenameFilter {
+ override def accept(dir: File, name: String): Boolean = {
+ name.toLowerCase.contains("hadoop") ||
+ name.toLowerCase.contains("flink-shaded-hadoop")
+ }
+ })).getOrElse(Array.empty)
+
+ val hasHadoopJar = hadoopJars.nonEmpty
Review Comment:
simplify to
```suggestion
val hasHadoopJar = Paths.get(flinkHome)
.resolve("lib")
.toFile
.listFiles(new FilenameFilter {
override def accept(dir: File, name: String): Boolean = {
name.startsWith("hadoop-client") ||
name.startsWith("flink-shaded-hadoop")
}
}).nonEmpty
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] Skip Hadoop classpath check if flink-shaded-hadoop jar exists in Flink lib directory [kyuubi]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on PR #7113:
URL: https://github.com/apache/kyuubi/pull/7113#issuecomment-3026255423
@cutiechi there are test failures in `FlinkProcessBuilderSuite`, can you fix it?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] Skip Hadoop classpath check if flink-shaded-hadoop jar exists in Flink lib directory [kyuubi]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #7113:
URL: https://github.com/apache/kyuubi/pull/7113#issuecomment-3026676806
## [Codecov](https://app.codecov.io/gh/apache/kyuubi/pull/7113?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
Attention: Patch coverage is `0%` with `17 lines` in your changes missing coverage. Please review.
> Project coverage is 0.00%. Comparing base [(`aaac07f`)](https://app.codecov.io/gh/apache/kyuubi/commit/aaac07fa55cb73f9f5108d53110425af2ed610e3?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) to head [(`99a4bf8`)](https://app.codecov.io/gh/apache/kyuubi/commit/99a4bf834f9b1f54bc6b412ded58ac48d507aa16?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
> Report is 1 commits behind head on master.
| [Files with missing lines](https://app.codecov.io/gh/apache/kyuubi/pull/7113?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [...ache/kyuubi/engine/flink/FlinkProcessBuilder.scala](https://app.codecov.io/gh/apache/kyuubi/pull/7113?src=pr&el=tree&filepath=kyuubi-server%2Fsrc%2Fmain%2Fscala%2Forg%2Fapache%2Fkyuubi%2Fengine%2Fflink%2FFlinkProcessBuilder.scala&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-a3l1dWJpLXNlcnZlci9zcmMvbWFpbi9zY2FsYS9vcmcvYXBhY2hlL2t5dXViaS9lbmdpbmUvZmxpbmsvRmxpbmtQcm9jZXNzQnVpbGRlci5zY2FsYQ==) | 0.00% | [17 Missing :warning: ](https://app.codecov.io/gh/apache/kyuubi/pull/7113?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #7113 +/- ##
======================================
Coverage 0.00% 0.00%
======================================
Files 700 700
Lines 43364 43374 +10
Branches 5869 5873 +4
======================================
- Misses 43364 43374 +10
```
</details>
[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/apache/kyuubi/pull/7113?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
- :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] Skip Hadoop classpath check if flink-shaded-hadoop jar exists in Flink lib directory [kyuubi]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on code in PR #7113:
URL: https://github.com/apache/kyuubi/pull/7113#discussion_r2166627522
##########
kyuubi-server/src/main/scala/org/apache/kyuubi/engine/flink/FlinkProcessBuilder.scala:
##########
@@ -228,7 +228,21 @@ class FlinkProcessBuilder(
hadoopCp.foreach(classpathEntries.add)
val extraCp = conf.get(ENGINE_FLINK_EXTRA_CLASSPATH)
extraCp.foreach(classpathEntries.add)
- if (hadoopCp.isEmpty && extraCp.isEmpty) {
+
+ val flinkLibDir = Paths.get(flinkHome)
+ .resolve("lib")
+ .toFile
+
+ val hadoopJars = Option(flinkLibDir.listFiles(new FilenameFilter {
+ override def accept(dir: File, name: String): Boolean = {
+ name.toLowerCase.contains("hadoop") ||
+ name.toLowerCase.contains("flink-shaded-hadoop")
+ }
+ })).getOrElse(Array.empty)
+
+ val hasHadoopJar = hadoopJars.nonEmpty
+
+ if (!hasHadoopJar && hadoopCp.isEmpty && extraCp.isEmpty) {
warn(s"The conf of ${FLINK_HADOOP_CLASSPATH_KEY} and " +
s"${ENGINE_FLINK_EXTRA_CLASSPATH.key} is empty.")
Review Comment:
update warn log too.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] Skip Hadoop classpath check if flink-shaded-hadoop jar exists in Flink lib directory [kyuubi]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on code in PR #7113:
URL: https://github.com/apache/kyuubi/pull/7113#discussion_r2166622488
##########
kyuubi-server/src/main/scala/org/apache/kyuubi/engine/flink/FlinkProcessBuilder.scala:
##########
@@ -228,7 +228,21 @@ class FlinkProcessBuilder(
hadoopCp.foreach(classpathEntries.add)
val extraCp = conf.get(ENGINE_FLINK_EXTRA_CLASSPATH)
extraCp.foreach(classpathEntries.add)
- if (hadoopCp.isEmpty && extraCp.isEmpty) {
+
+ val flinkLibDir = Paths.get(flinkHome)
+ .resolve("lib")
+ .toFile
+
+ val hadoopJars = Option(flinkLibDir.listFiles(new FilenameFilter {
+ override def accept(dir: File, name: String): Boolean = {
+ name.toLowerCase.contains("hadoop") ||
+ name.toLowerCase.contains("flink-shaded-hadoop")
+ }
+ })).getOrElse(Array.empty)
+
+ val hasHadoopJar = hadoopJars.nonEmpty
Review Comment:
simplify to
```suggestion
val hasHadoopJar = Paths.get(flinkHome)
.resolve("lib")
.toFile
.listFiles(new FilenameFilter {
override def accept(dir: File, name: String): Boolean = {
name.toLowerCase.startsWith("hadoop-client") ||
name.toLowerCase.startsWith("flink-shaded-hadoop")
}
}).nonEmpty
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org