You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Stephen Connolly <st...@gmail.com> on 2019/04/01 14:23:52 UTC

Re: Does flink configuration support configed by environment variables?

I don't think it does. I ended up writing a small CLI tool to enabling
templating the file from environment variables. There are loads of such
tools, but mine is https://github.com/stephenc/envsub

I have the dockerfile like so:

ARG FLINK_VERSION=1.7.2-alpine
FROM flink:${FLINK_VERSION}
ARG
ENVSUB=0.1.0::SHA::b10600c03236bbf0711476e11a1dff9ae285a50a48568bfd0bf6c6014fc69f0c
RUN apk add --no-cache tini curl \
    #
    # Get envsub
    #
\
    && curl -fsSL "
https://github.com/stephenc/envsub/releases/download/${ENVSUB%%::SHA::*}/envsub"
-o /usr/local/bin/envsub \
    && if [ "${ENVSUB##*::SHA::}" = "${ENVSUB}" ] ; then \
        echo "/usr/local/bin/envsub: Unverified" >&2 ; \
    else \
        echo "${ENVSUB##*::SHA::}  /usr/local/bin/envsub" | sha256sum -c -
; \
    fi \
    && chmod +x /usr/local/bin/envsub

COPY rootfs/ /

ENTRYPOINT ["/sbin/tini", "-g", "--", "/docker-entrypoint2.sh"]

CMD []


Then docker-entrypoint2.sh looks like

#!/usr/bin/env bash

if [[ "$#" -eq 0 ]]; then
    if [[ -z "${FLINK_ROLE}" ]]; then
        echo "Please set environment variable FLINK_ROLE to either
jobmanager or taskmanager"
        exit 1
    fi
    envsub < /opt/flink/conf/flink-conf.template.yaml >
/opt/flink/conf/flink-conf.yaml
    exec /docker-entrypoint.sh "${FLINK_ROLE}"
fi

exec /docker-entrypoint.sh "${@}"


and /opt/flink/conf/flink-conf.template.yaml has the environment variable
substitution like so:

fs.s3a.endpoint: ${FLINK_S3_ENDPOINT}
fs.s3a.path.style.access: true
fs.s3a.connection.ssl.enabled: false
fs.s3a.access.key: ${AWS_ACCESS_KEY_ID}
fs.s3a.secret.key: ${AWS_SECRET_KEY}


Hope that is sufficient for you to derive your own solution


On Fri, 29 Mar 2019 at 03:09, Lifei Chen <hu...@gmail.com> wrote:

> Hi guys,
>
> I am using flink 1.7.2 deployed by kubernetes,  and I want to change the
> configurations about flink,  for example customize
> `taskmanager.heap.size`.
>
> Does flink support using environment variables to override configurations
> in `conf/flink-conf.yaml` ?
>

Re: Does flink configuration support configed by environment variables?

Posted by Lifei Chen <hu...@gmail.com>.
Thanks,  maybe overwrite configuration in 'conf/flink-conf.yaml' by
docker-entrypoint.sh is a common way to do it, thanks for your advice.

Stephen Connolly <st...@gmail.com> 于2019年4月1日周一 下午10:33写道:

> I don't think it does. I ended up writing a small CLI tool to enabling
> templating the file from environment variables. There are loads of such
> tools, but mine is https://github.com/stephenc/envsub
>
> I have the dockerfile like so:
>
> ARG FLINK_VERSION=1.7.2-alpine
> FROM flink:${FLINK_VERSION}
> ARG
> ENVSUB=0.1.0::SHA::b10600c03236bbf0711476e11a1dff9ae285a50a48568bfd0bf6c6014fc69f0c
> RUN apk add --no-cache tini curl \
>     #
>     # Get envsub
>     #
> \
>     && curl -fsSL "
> https://github.com/stephenc/envsub/releases/download/${ENVSUB%%::SHA::*}/envsub"
> -o /usr/local/bin/envsub \
>     && if [ "${ENVSUB##*::SHA::}" = "${ENVSUB}" ] ; then \
>         echo "/usr/local/bin/envsub: Unverified" >&2 ; \
>     else \
>         echo "${ENVSUB##*::SHA::}  /usr/local/bin/envsub" | sha256sum -c -
> ; \
>     fi \
>     && chmod +x /usr/local/bin/envsub
>
> COPY rootfs/ /
>
> ENTRYPOINT ["/sbin/tini", "-g", "--", "/docker-entrypoint2.sh"]
>
> CMD []
>
>
> Then docker-entrypoint2.sh looks like
>
> #!/usr/bin/env bash
>
> if [[ "$#" -eq 0 ]]; then
>     if [[ -z "${FLINK_ROLE}" ]]; then
>         echo "Please set environment variable FLINK_ROLE to either
> jobmanager or taskmanager"
>         exit 1
>     fi
>     envsub < /opt/flink/conf/flink-conf.template.yaml >
> /opt/flink/conf/flink-conf.yaml
>     exec /docker-entrypoint.sh "${FLINK_ROLE}"
> fi
>
> exec /docker-entrypoint.sh "${@}"
>
>
> and /opt/flink/conf/flink-conf.template.yaml has the environment variable
> substitution like so:
>
> fs.s3a.endpoint: ${FLINK_S3_ENDPOINT}
> fs.s3a.path.style.access: true
> fs.s3a.connection.ssl.enabled: false
> fs.s3a.access.key: ${AWS_ACCESS_KEY_ID}
> fs.s3a.secret.key: ${AWS_SECRET_KEY}
>
>
> Hope that is sufficient for you to derive your own solution
>
>
> On Fri, 29 Mar 2019 at 03:09, Lifei Chen <hu...@gmail.com> wrote:
>
>> Hi guys,
>>
>> I am using flink 1.7.2 deployed by kubernetes,  and I want to change the
>> configurations about flink,  for example customize
>> `taskmanager.heap.size`.
>>
>> Does flink support using environment variables to override configurations
>> in `conf/flink-conf.yaml` ?
>>
>