You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@flink.apache.org by aljoscha <gi...@git.apache.org> on 2017/10/30 11:06:31 UTC

[GitHub] flink pull request #4920: [FLINK-7944] Allow configuring Hadoop classpath

GitHub user aljoscha opened a pull request:

    https://github.com/apache/flink/pull/4920

    [FLINK-7944] Allow configuring Hadoop classpath

    R: @StephanEwen 

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/aljoscha/flink jira-7944-hadoop-config

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/flink/pull/4920.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #4920
    
----
commit c7fea5831148c986324d4dbc66535386a260e402
Author: Aljoscha Krettek <al...@gmail.com>
Date:   2017-10-30T11:05:23Z

    [FLINK-7944] Allow configuring Hadoop classpath

----


---

[GitHub] flink issue #4920: [FLINK-7944] Allow configuring Hadoop classpath

Posted by aljoscha <gi...@git.apache.org>.

Github user aljoscha commented on the issue:

https://github.com/apache/flink/pull/4920

You're right, I added a longer description in the commit message. This is the message:
```
Flink comes with bundled Hadoop dependencies in
lib/flink-shaded-hadoop2-uber-*.jar. Flink can also use the Hadoop jars
that are available on the system (from the distro, EMR, GCE, ...) and
Flink can run completely without Hadoop dependencies. Switching between
these comes down to either adding or removing the bundled jar from lib/
and/or telling Flink to use the system Hadoop classpaths or not.

This change introduces a new possible configuration file in
"bin/hadoop-classpath". If present, this should contain a classpath
definition. When the file is present Flink will append this to the
internal classpaths.

The new configuration script allows easily switching the different modes:

- use-system-hadoop: Try and determine the system Hadoop classpaths and
store them in bin/hadoop-classpath. The file can be tweaked manually
afterwards. Move the bundled Hadoop jar from lib/ to opt/ if present so
that Flink doesn't use it.

- use-bundled-hadoop: Remove the bin/hadoop-classpath configuration
file. Move the bundled Hadoop jar from opt/ to lib/ if it is not
already in lib/

- no-hadoop: Remove the bin/hadoop-classpath configuration and move the
bundled Hadoop jar from lib/ to opt/.

Flink will still add this to the classpath in config.sh:

"${HADOOP_CLASSPATH}:${HADOOP_CONF_DIR}:${YARN_CONF_DIR}:${HBASE_CONF_DIR}"

i.e. env variables on the system are still being picked up, no matter
the configuration in bin/hadoop-classpath.
```

That last part is interesting. We've always done it but now I'm wondering whether we should. (By default `HADOOP_CLASSPATH` is empty on AWS and GCE.

---

[GitHub] flink issue #4920: [FLINK-7944] Allow configuring Hadoop classpath

Posted by StephanEwen <gi...@git.apache.org>.

Github user StephanEwen commented on the issue:

    https://github.com/apache/flink/pull/4920
  
    My first thought is that moving libraries around is maybe not the best approach.
    Also, this seems like it implements a mix between explicitly configuring the Hadoop Classpath and implicitly picking up env variables.
    
    I would suggest to actually take a step back, make a proper design doc how we want this to work. If we add configuration scripts, why not make it an "all explicit, no magic" approach?
    
    Minor comments:
      - `bin/hadoop-classpath` seems like the wrong destination. Such files should be in `conf`.


---