You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@zeppelin.apache.org by Juan Rodríguez Hortalá <ju...@gmail.com> on 2020/04/04 16:05:23 UTC
Setup FLINK_HOME programmatically
Hi,
I'm setting up a Dockerfile to run Zeppelin using the Flink interpreter.
I'm starting from the Docker image apache/zeppelin:0.9.0 and following the
instructions on
https://medium.com/@zjffdu/flink-on-zeppelin-part-1-get-started-2591aaa6aa47
to download flink-1.10.0-bin-scala_2.11.tgz and
flink-shaded-hadoop-2-2.8.3-10.0.jar. This is mostly working, except for
the fact that I have to manually configure FLINK_HOME in the Zeppelin UI to
have it pointing to the path where I have installed Flink, in my case
/opt/flink/latest. When I do that I am able to use the Flink interpreters.
I have tried to setup that in the Dockerfile, but I'm not sure which files
I have to edit:
- I have tried editing /zeppelin/conf/interpreter.json, but it looks
like it is generated the first time Zeppelin runs, the file doesn't exist
while I build the Docker image
- I have tried editing
/zeppelin/interpreter/flink/interpreter-setting.json instead, but it has no
effect. I used the following statements in my Dockerfile:
WORKDIR /zeppelin
RUN apt-get install -y jq \
&& FLINK_CONF_FILE=interpreter/flink/interpreter-setting.json \
&& cp "${FLINK_CONF_FILE}" "${FLINK_CONF_FILE}.bak" \
&& jq '.[0].properties.FLINK_HOME.value = "/opt/flink/latest"'
"${FLINK_CONF_FILE}" > "${FLINK_CONF_FILE}.new" \
&& mv "${FLINK_CONF_FILE}.new" "${FLINK_CONF_FILE}"
- I have tried editing /zeppelin/conf/zeppelin-env.sh, but it also has
no effect. I used the following statements in my Dockerfile
WORKDIR /zeppelin
RUN cp conf/zeppelin-env.sh.template conf/zeppelin-env.sh \
&& echo 'export FLINK_HOME=/opt/flink/latest' >> conf/zeppelin-env.sh \
&& echo '' >> conf/zeppelin-env.sh
Which file should I edit to setup FLINK_HOME from a script? Any idea what
am I doing wrong? I guess this should be the same as setting up any
Zeppelin configuration key, so this is quite a basic question, but I have a
very rudimentary knowledge of Zeppelin.
Thanks a lot for your help
Juan
Re: Setup FLINK_HOME programmatically
Posted by Juan Rodríguez Hortalá <ju...@gmail.com>.
Hi Jeff,
I'll wait for the fix then, thanks for letting me know. In the meantime,
setting up FLINK_HOME in the UI is no big deal, just a couple of clicks,
I'm enjoying a lot the Flink integration and your blog posts about it,
thanks for working on that!
Greetings,
Juan
On Sun, Apr 5, 2020 at 9:14 AM Jeff Zhang <zj...@gmail.com> wrote:
> Hi Juan,
>
> I am afraid there's no workaround for now.
> But I think a better way is to allow set env FLINK_HOME, such as following:
> docker run -p 8080:8080 --rm -v /Users/jzhang/Java/lib/flink-1.10.0:/flink
> -e FLINK_HOME=/flink --name zeppelin apache/zeppelin:0.9.0
>
> Unfortunately, it doesn't work due to one bug, I will fix it as soon as I
> can.
>
> BTW, welcome to use flink on zeppelin and give feedback and comments.
>
>
>
> Juan Rodríguez Hortalá <ju...@gmail.com> 于2020年4月5日周日
> 上午12:05写道：
>
>> Hi,
>>
>> I'm setting up a Dockerfile to run Zeppelin using the Flink interpreter.
>> I'm starting from the Docker image apache/zeppelin:0.9.0 and following the
>> instructions on
>> https://medium.com/@zjffdu/flink-on-zeppelin-part-1-get-started-2591aaa6aa47
>> to download flink-1.10.0-bin-scala_2.11.tgz and
>> flink-shaded-hadoop-2-2.8.3-10.0.jar. This is mostly working, except for
>> the fact that I have to manually configure FLINK_HOME in the Zeppelin UI to
>> have it pointing to the path where I have installed Flink, in my case
>> /opt/flink/latest. When I do that I am able to use the Flink interpreters.
>> I have tried to setup that in the Dockerfile, but I'm not sure which
>> files I have to edit:
>>
>> - I have tried editing /zeppelin/conf/interpreter.json, but it looks
>> like it is generated the first time Zeppelin runs, the file doesn't exist
>> while I build the Docker image
>> - I have tried editing
>> /zeppelin/interpreter/flink/interpreter-setting.json instead, but it has no
>> effect. I used the following statements in my Dockerfile:
>>
>> WORKDIR /zeppelin
>> RUN apt-get install -y jq \
>> && FLINK_CONF_FILE=interpreter/flink/interpreter-setting.json \
>> && cp "${FLINK_CONF_FILE}" "${FLINK_CONF_FILE}.bak" \
>> && jq '.[0].properties.FLINK_HOME.value = "/opt/flink/latest"'
>> "${FLINK_CONF_FILE}" > "${FLINK_CONF_FILE}.new" \
>> && mv "${FLINK_CONF_FILE}.new" "${FLINK_CONF_FILE}"
>>
>> - I have tried editing /zeppelin/conf/zeppelin-env.sh, but it also
>> has no effect. I used the following statements in my Dockerfile
>>
>> WORKDIR /zeppelin
>> RUN cp conf/zeppelin-env.sh.template conf/zeppelin-env.sh \
>> && echo 'export FLINK_HOME=/opt/flink/latest' >> conf/zeppelin-env.sh \
>> && echo '' >> conf/zeppelin-env.sh
>>
>> Which file should I edit to setup FLINK_HOME from a script? Any idea what
>> am I doing wrong? I guess this should be the same as setting up any
>> Zeppelin configuration key, so this is quite a basic question, but I have a
>> very rudimentary knowledge of Zeppelin.
>>
>> Thanks a lot for your help
>>
>> Juan
>>
>>
>
> --
> Best Regards
>
> Jeff Zhang
>
Re: Setup FLINK_HOME programmatically
Posted by Jeff Zhang <zj...@gmail.com>.
Hi Juan,
I am afraid there's no workaround for now.
But I think a better way is to allow set env FLINK_HOME, such as following:
docker run -p 8080:8080 --rm -v /Users/jzhang/Java/lib/flink-1.10.0:/flink
-e FLINK_HOME=/flink --name zeppelin apache/zeppelin:0.9.0
Unfortunately, it doesn't work due to one bug, I will fix it as soon as I
can.
BTW, welcome to use flink on zeppelin and give feedback and comments.
Juan Rodríguez Hortalá <ju...@gmail.com> 于2020年4月5日周日
上午12:05写道：
> Hi,
>
> I'm setting up a Dockerfile to run Zeppelin using the Flink interpreter.
> I'm starting from the Docker image apache/zeppelin:0.9.0 and following the
> instructions on
> https://medium.com/@zjffdu/flink-on-zeppelin-part-1-get-started-2591aaa6aa47
> to download flink-1.10.0-bin-scala_2.11.tgz and
> flink-shaded-hadoop-2-2.8.3-10.0.jar. This is mostly working, except for
> the fact that I have to manually configure FLINK_HOME in the Zeppelin UI to
> have it pointing to the path where I have installed Flink, in my case
> /opt/flink/latest. When I do that I am able to use the Flink interpreters.
> I have tried to setup that in the Dockerfile, but I'm not sure which files
> I have to edit:
>
> - I have tried editing /zeppelin/conf/interpreter.json, but it looks
> like it is generated the first time Zeppelin runs, the file doesn't exist
> while I build the Docker image
> - I have tried editing
> /zeppelin/interpreter/flink/interpreter-setting.json instead, but it has no
> effect. I used the following statements in my Dockerfile:
>
> WORKDIR /zeppelin
> RUN apt-get install -y jq \
> && FLINK_CONF_FILE=interpreter/flink/interpreter-setting.json \
> && cp "${FLINK_CONF_FILE}" "${FLINK_CONF_FILE}.bak" \
> && jq '.[0].properties.FLINK_HOME.value = "/opt/flink/latest"'
> "${FLINK_CONF_FILE}" > "${FLINK_CONF_FILE}.new" \
> && mv "${FLINK_CONF_FILE}.new" "${FLINK_CONF_FILE}"
>
> - I have tried editing /zeppelin/conf/zeppelin-env.sh, but it also has
> no effect. I used the following statements in my Dockerfile
>
> WORKDIR /zeppelin
> RUN cp conf/zeppelin-env.sh.template conf/zeppelin-env.sh \
> && echo 'export FLINK_HOME=/opt/flink/latest' >> conf/zeppelin-env.sh \
> && echo '' >> conf/zeppelin-env.sh
>
> Which file should I edit to setup FLINK_HOME from a script? Any idea what
> am I doing wrong? I guess this should be the same as setting up any
> Zeppelin configuration key, so this is quite a basic question, but I have a
> very rudimentary knowledge of Zeppelin.
>
> Thanks a lot for your help
>
> Juan
>
>
--
Best Regards
Jeff Zhang