You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Stephen Connolly <st...@gmail.com> on 2019/04/01 14:23:52 UTC
Re: Does flink configuration support configed by environment variables?
I don't think it does. I ended up writing a small CLI tool to enabling
templating the file from environment variables. There are loads of such
tools, but mine is https://github.com/stephenc/envsub
I have the dockerfile like so:
ARG FLINK_VERSION=1.7.2-alpine
FROM flink:${FLINK_VERSION}
ARG
ENVSUB=0.1.0::SHA::b10600c03236bbf0711476e11a1dff9ae285a50a48568bfd0bf6c6014fc69f0c
RUN apk add --no-cache tini curl \
#
# Get envsub
#
\
&& curl -fsSL "
https://github.com/stephenc/envsub/releases/download/${ENVSUB%%::SHA::*}/envsub"
-o /usr/local/bin/envsub \
&& if [ "${ENVSUB##*::SHA::}" = "${ENVSUB}" ] ; then \
echo "/usr/local/bin/envsub: Unverified" >&2 ; \
else \
echo "${ENVSUB##*::SHA::} /usr/local/bin/envsub" | sha256sum -c -
; \
fi \
&& chmod +x /usr/local/bin/envsub
COPY rootfs/ /
ENTRYPOINT ["/sbin/tini", "-g", "--", "/docker-entrypoint2.sh"]
CMD []
Then docker-entrypoint2.sh looks like
#!/usr/bin/env bash
if [[ "$#" -eq 0 ]]; then
if [[ -z "${FLINK_ROLE}" ]]; then
echo "Please set environment variable FLINK_ROLE to either
jobmanager or taskmanager"
exit 1
fi
envsub < /opt/flink/conf/flink-conf.template.yaml >
/opt/flink/conf/flink-conf.yaml
exec /docker-entrypoint.sh "${FLINK_ROLE}"
fi
exec /docker-entrypoint.sh "${@}"
and /opt/flink/conf/flink-conf.template.yaml has the environment variable
substitution like so:
fs.s3a.endpoint: ${FLINK_S3_ENDPOINT}
fs.s3a.path.style.access: true
fs.s3a.connection.ssl.enabled: false
fs.s3a.access.key: ${AWS_ACCESS_KEY_ID}
fs.s3a.secret.key: ${AWS_SECRET_KEY}
Hope that is sufficient for you to derive your own solution
On Fri, 29 Mar 2019 at 03:09, Lifei Chen <hu...@gmail.com> wrote:
> Hi guys,
>
> I am using flink 1.7.2 deployed by kubernetes, and I want to change the
> configurations about flink, for example customize
> `taskmanager.heap.size`.
>
> Does flink support using environment variables to override configurations
> in `conf/flink-conf.yaml` ?
>
Re: Does flink configuration support configed by environment variables?
Posted by Lifei Chen <hu...@gmail.com>.
Thanks, maybe overwrite configuration in 'conf/flink-conf.yaml' by
docker-entrypoint.sh is a common way to do it, thanks for your advice.
Stephen Connolly <st...@gmail.com> 于2019年4月1日周一 下午10:33写道:
> I don't think it does. I ended up writing a small CLI tool to enabling
> templating the file from environment variables. There are loads of such
> tools, but mine is https://github.com/stephenc/envsub
>
> I have the dockerfile like so:
>
> ARG FLINK_VERSION=1.7.2-alpine
> FROM flink:${FLINK_VERSION}
> ARG
> ENVSUB=0.1.0::SHA::b10600c03236bbf0711476e11a1dff9ae285a50a48568bfd0bf6c6014fc69f0c
> RUN apk add --no-cache tini curl \
> #
> # Get envsub
> #
> \
> && curl -fsSL "
> https://github.com/stephenc/envsub/releases/download/${ENVSUB%%::SHA::*}/envsub"
> -o /usr/local/bin/envsub \
> && if [ "${ENVSUB##*::SHA::}" = "${ENVSUB}" ] ; then \
> echo "/usr/local/bin/envsub: Unverified" >&2 ; \
> else \
> echo "${ENVSUB##*::SHA::} /usr/local/bin/envsub" | sha256sum -c -
> ; \
> fi \
> && chmod +x /usr/local/bin/envsub
>
> COPY rootfs/ /
>
> ENTRYPOINT ["/sbin/tini", "-g", "--", "/docker-entrypoint2.sh"]
>
> CMD []
>
>
> Then docker-entrypoint2.sh looks like
>
> #!/usr/bin/env bash
>
> if [[ "$#" -eq 0 ]]; then
> if [[ -z "${FLINK_ROLE}" ]]; then
> echo "Please set environment variable FLINK_ROLE to either
> jobmanager or taskmanager"
> exit 1
> fi
> envsub < /opt/flink/conf/flink-conf.template.yaml >
> /opt/flink/conf/flink-conf.yaml
> exec /docker-entrypoint.sh "${FLINK_ROLE}"
> fi
>
> exec /docker-entrypoint.sh "${@}"
>
>
> and /opt/flink/conf/flink-conf.template.yaml has the environment variable
> substitution like so:
>
> fs.s3a.endpoint: ${FLINK_S3_ENDPOINT}
> fs.s3a.path.style.access: true
> fs.s3a.connection.ssl.enabled: false
> fs.s3a.access.key: ${AWS_ACCESS_KEY_ID}
> fs.s3a.secret.key: ${AWS_SECRET_KEY}
>
>
> Hope that is sufficient for you to derive your own solution
>
>
> On Fri, 29 Mar 2019 at 03:09, Lifei Chen <hu...@gmail.com> wrote:
>
>> Hi guys,
>>
>> I am using flink 1.7.2 deployed by kubernetes, and I want to change the
>> configurations about flink, for example customize
>> `taskmanager.heap.size`.
>>
>> Does flink support using environment variables to override configurations
>> in `conf/flink-conf.yaml` ?
>>
>