You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by GitBox <gi...@apache.org> on 2022/06/16 07:26:42 UTC
[GitHub] [flink] gaborgsomogyi opened a new pull request, #19986: [FLINK-7967][config] Remove deprecated Hadoop specific Flink configuration options
gaborgsomogyi opened a new pull request, #19986:
URL: https://github.com/apache/flink/pull/19986
## What is the purpose of the change
The following Flink configuration options are deprecated for quite some long time (10 minor releases) and it's time to remove them:
```
fs.hdfs.hdfsdefault
fs.hdfs.hdfssite
fs.hdfs.hadoopconf
```
## Brief change log
* Removed the mentioned Flink configuration options
* Minor bugfixes
## Verifying this change
Existing unit tests.
## Does this pull request potentially affect one of the following parts:
- Dependencies (does it add or upgrade a dependency): no
- The public API, i.e., is any changed class annotated with `@Public(Evolving)`: no
- The serializers: no
- The runtime per-record code paths (performance sensitive): no
- Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: yes
- The S3 file system connector: no
## Documentation
- Does this pull request introduce a new feature? no
- If yes, how is the feature documented? not applicable
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] gaborgsomogyi commented on a diff in pull request #19986: [FLINK-28081][config] Remove deprecated Hadoop specific Flink configuration options
Posted by GitBox <gi...@apache.org>.
gaborgsomogyi commented on code in PR #19986:
URL: https://github.com/apache/flink/pull/19986#discussion_r898783437
##########
flink-filesystems/flink-hadoop-fs/src/main/java/org/apache/flink/runtime/util/HadoopUtils.java:
##########
@@ -77,47 +75,18 @@ public static Configuration getHadoopConfiguration(
for (String possibleHadoopConfPath : possibleHadoopConfPaths) {
if (possibleHadoopConfPath != null) {
- foundHadoopConfiguration = addHadoopConfIfFound(result, possibleHadoopConfPath);
+ foundHadoopConfiguration |= addHadoopConfIfFound(result, possibleHadoopConfPath);
}
}
- // Approach 2: Flink configuration (deprecated)
- final String hdfsDefaultPath =
- flinkConfiguration.getString(ConfigConstants.HDFS_DEFAULT_CONFIG, null);
- if (hdfsDefaultPath != null) {
- result.addResource(new org.apache.hadoop.fs.Path(hdfsDefaultPath));
- LOG.debug(
- "Using hdfs-default configuration-file path from Flink config: {}",
- hdfsDefaultPath);
- foundHadoopConfiguration = true;
- }
-
- final String hdfsSitePath =
- flinkConfiguration.getString(ConfigConstants.HDFS_SITE_CONFIG, null);
- if (hdfsSitePath != null) {
- result.addResource(new org.apache.hadoop.fs.Path(hdfsSitePath));
- LOG.debug(
- "Using hdfs-site configuration-file path from Flink config: {}", hdfsSitePath);
- foundHadoopConfiguration = true;
- }
-
- final String hadoopConfigPath =
- flinkConfiguration.getString(ConfigConstants.PATH_HADOOP_CONFIG, null);
- if (hadoopConfigPath != null) {
- LOG.debug("Searching Hadoop configuration files in Flink config: {}", hadoopConfigPath);
- foundHadoopConfiguration =
- addHadoopConfIfFound(result, hadoopConfigPath) || foundHadoopConfiguration;
- }
-
- // Approach 3: HADOOP_CONF_DIR environment variable
+ // Approach 2: HADOOP_CONF_DIR environment variable
String hadoopConfDir = System.getenv("HADOOP_CONF_DIR");
if (hadoopConfDir != null) {
LOG.debug("Searching Hadoop configuration files in HADOOP_CONF_DIR: {}", hadoopConfDir);
- foundHadoopConfiguration =
- addHadoopConfIfFound(result, hadoopConfDir) || foundHadoopConfiguration;
+ foundHadoopConfiguration |= addHadoopConfIfFound(result, hadoopConfDir);
Review Comment:
Same here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] lmove commented on pull request #19986: DO_NOT_MERGE_UNTIL_2.X [FLINK-28081][config] Remove deprecated Hadoop specific Flink configuration options
Posted by GitBox <gi...@apache.org>.
lmove commented on PR #19986:
URL: https://github.com/apache/flink/pull/19986#issuecomment-1319805514
Hi! Our tool [BreakBot](https://github.com/alien-tools/breakbot) found that this pull request introduces **3 breaking changes** and they appear to impact **4 of your clients**. You can find the full BreakBot report in our fork repository: [report for PR#19986](https://github.com/breakbot-playground/flink/pull/6/checks).
We hope this information is valuable to you, and apologize otherwise. If you're willing to help, we would kindly ask for your help to fill in a [5-minutes survey](https://survey.breakbot.net/index.php/418865/newtest/Y?ID=apache-flink-19986) about the report. Your feedback will help us improve the tool and provide a better experience for users in the future!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] gaborgsomogyi commented on a diff in pull request #19986: [FLINK-28081][config] Remove deprecated Hadoop specific Flink configuration options
Posted by GitBox <gi...@apache.org>.
gaborgsomogyi commented on code in PR #19986:
URL: https://github.com/apache/flink/pull/19986#discussion_r898783294
##########
flink-filesystems/flink-hadoop-fs/src/main/java/org/apache/flink/runtime/util/HadoopUtils.java:
##########
@@ -77,47 +75,18 @@ public static Configuration getHadoopConfiguration(
for (String possibleHadoopConfPath : possibleHadoopConfPaths) {
if (possibleHadoopConfPath != null) {
- foundHadoopConfiguration = addHadoopConfIfFound(result, possibleHadoopConfPath);
+ foundHadoopConfiguration |= addHadoopConfIfFound(result, possibleHadoopConfPath);
Review Comment:
This is a simple boolean arithmetic in order not to overwrite the first round true with a second round false.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] gaborgsomogyi commented on pull request #19986: DO_NOT_MERGE_UNTIL_2.X [FLINK-28081][config] Remove deprecated Hadoop specific Flink configuration options
Posted by GitBox <gi...@apache.org>.
gaborgsomogyi commented on PR #19986:
URL: https://github.com/apache/flink/pull/19986#issuecomment-1352885289
I this this area needs a full refactor so closing it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] gaborgsomogyi commented on pull request #19986: [FLINK-28081][config] Remove deprecated Hadoop specific Flink configuration options
Posted by GitBox <gi...@apache.org>.
gaborgsomogyi commented on PR #19986:
URL: https://github.com/apache/flink/pull/19986#issuecomment-1157515540
cc @gyfora @mbalassi
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #19986: [FLINK-28081][config] Remove deprecated Hadoop specific Flink configuration options
Posted by GitBox <gi...@apache.org>.
flinkbot commented on PR #19986:
URL: https://github.com/apache/flink/pull/19986#issuecomment-1157332430
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f11dce31d9c555b754a8e12dc612ce32861e2f04",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f11dce31d9c555b754a8e12dc612ce32861e2f04",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f11dce31d9c555b754a8e12dc612ce32861e2f04 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] gaborgsomogyi commented on pull request #19986: [FLINK-28081][config] Remove deprecated Hadoop specific Flink configuration options
Posted by GitBox <gi...@apache.org>.
gaborgsomogyi commented on PR #19986:
URL: https://github.com/apache/flink/pull/19986#issuecomment-1157718866
@flinkbot run azure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] gaborgsomogyi closed pull request #19986: DO_NOT_MERGE_UNTIL_2.X [FLINK-28081][config] Remove deprecated Hadoop specific Flink configuration options
Posted by GitBox <gi...@apache.org>.
gaborgsomogyi closed pull request #19986: DO_NOT_MERGE_UNTIL_2.X [FLINK-28081][config] Remove deprecated Hadoop specific Flink configuration options
URL: https://github.com/apache/flink/pull/19986
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org