You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@zeppelin.apache.org by GitBox <gi...@apache.org> on 2020/08/19 11:51:22 UTC
[GitHub] [zeppelin] zhoulii opened a new pull request #3884: [ZEPPELIN-5004] support running flink interpreter on kubernetes
zhoulii opened a new pull request #3884:
URL: https://github.com/apache/zeppelin/pull/3884
### What is this PR for?
add support to run flink interpreter on kubernetes
### What type of PR is it?
[Improvement]
### Todos
* [ ] - Task
### What is the Jira issue?
* https://issues.apache.org/jira/browse/ZEPPELIN-5004
### How should this be tested?
* manually tested in kubernetes cluster
* https://travis-ci.org/github/zhoulii/zeppelin/builds/719203579
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update? **NO**
* Is there breaking changes for older versions? **NO**
* Does this needs documentation? **NO**
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [zeppelin] zjffdu commented on pull request #3884: [ZEPPELIN-5004] support running flink interpreter on kubernetes
Posted by GitBox <gi...@apache.org>.
zjffdu commented on pull request #3884:
URL: https://github.com/apache/zeppelin/pull/3884#issuecomment-678611118
> > @zhoulii What is the log4j conflict issue ？
>
> @zjffdu when starting flink interpreter with `flink 1.11.1`, I find this exception, it‘s resulted from log4j conflict：
>
> ```
> SLF4J: Class path contains multiple SLF4J bindings.
> SLF4J: Found binding in [jar:file:/flink/lib/log4j-slf4j-impl-2.12.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in [jar:file:/opt/zeppelin/interpreter/flink/zeppelin-flink-0.9.0-SNAPSHOT.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
> SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
> ERROR StatusLogger Unable to access file:///opt/zeppelin/conf/log4j2.properties
> java.io.FileNotFoundException: /opt/zeppelin/conf/log4j2.properties (No such file or directory)
> at java.io.FileInputStream.open0(Native Method)
> at java.io.FileInputStream.open(FileInputStream.java:195)
> at java.io.FileInputStream.<init>(FileInputStream.java:138)
> at java.io.FileInputStream.<init>(FileInputStream.java:93)
> ```
>
> btw, start flink interpreter with flink 1.10.0 is ok.
The reason is that flink 1.11 use log4j2 to replace log4j, and there's no log4j2.properties in zeppelin interpreter image.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [zeppelin] zhoulii commented on pull request #3884: [ZEPPELIN-5004] support running flink interpreter on kubernetes
Posted by GitBox <gi...@apache.org>.
zhoulii commented on pull request #3884:
URL: https://github.com/apache/zeppelin/pull/3884#issuecomment-678613170
> > > @zhoulii What is the log4j conflict issue ？
> >
> >
> > @zjffdu when starting flink interpreter with `flink 1.11.1`, I find this exception, it‘s resulted from log4j conflict：
> > ```
> > SLF4J: Class path contains multiple SLF4J bindings.
> > SLF4J: Found binding in [jar:file:/flink/lib/log4j-slf4j-impl-2.12.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> > SLF4J: Found binding in [jar:file:/opt/zeppelin/interpreter/flink/zeppelin-flink-0.9.0-SNAPSHOT.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> > SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
> > SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
> > ERROR StatusLogger Unable to access file:///opt/zeppelin/conf/log4j2.properties
> > java.io.FileNotFoundException: /opt/zeppelin/conf/log4j2.properties (No such file or directory)
> > at java.io.FileInputStream.open0(Native Method)
> > at java.io.FileInputStream.open(FileInputStream.java:195)
> > at java.io.FileInputStream.<init>(FileInputStream.java:138)
> > at java.io.FileInputStream.<init>(FileInputStream.java:93)
> > ```
> >
> >
> > btw, start flink interpreter with flink 1.10.0 is ok.
>
> The reason is that flink 1.11 use log4j2 to replace log4j, and there's no log4j2.properties in zeppelin interpreter image.
It's true, I think there are two ways to solve this issue: add log4j2.properties in conf or do not add `$FLINK_HOME\lib\log4j*.jar` to classpath when start flink interpreter.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [zeppelin] zjffdu commented on pull request #3884: [ZEPPELIN-5004] support running flink interpreter on kubernetes
Posted by GitBox <gi...@apache.org>.
zjffdu commented on pull request #3884:
URL: https://github.com/apache/zeppelin/pull/3884#issuecomment-678717579
> > > > @zhoulii What is the log4j conflict issue ？
> > >
> > >
> > > @zjffdu when starting flink interpreter with `flink 1.11.1`, I find this exception, it‘s resulted from log4j conflict：
> > > ```
> > > SLF4J: Class path contains multiple SLF4J bindings.
> > > SLF4J: Found binding in [jar:file:/flink/lib/log4j-slf4j-impl-2.12.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> > > SLF4J: Found binding in [jar:file:/opt/zeppelin/interpreter/flink/zeppelin-flink-0.9.0-SNAPSHOT.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> > > SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
> > > SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
> > > ERROR StatusLogger Unable to access file:///opt/zeppelin/conf/log4j2.properties
> > > java.io.FileNotFoundException: /opt/zeppelin/conf/log4j2.properties (No such file or directory)
> > > at java.io.FileInputStream.open0(Native Method)
> > > at java.io.FileInputStream.open(FileInputStream.java:195)
> > > at java.io.FileInputStream.<init>(FileInputStream.java:138)
> > > at java.io.FileInputStream.<init>(FileInputStream.java:93)
> > > ```
> > >
> > >
> > > btw, start flink interpreter with flink 1.10.0 is ok.
> >
> >
> > The reason is that flink 1.11 use log4j2 to replace log4j, and there's no log4j2.properties in zeppelin interpreter image.
>
> It's true, I think there are two ways to solve this issue: add log4j2.properties in conf or do not add `$FLINK_HOME\lib\log4j*.jar` to classpath when start flink interpreter.
Is there any behavior difference between these 2 approaches ? The purpose is to make logging correctly.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [zeppelin] zhoulii commented on pull request #3884: [ZEPPELIN-5004] support running flink interpreter on kubernetes
Posted by GitBox <gi...@apache.org>.
zhoulii commented on pull request #3884:
URL: https://github.com/apache/zeppelin/pull/3884#issuecomment-678862175
@zjffdu I have only checked the second approach that filter out `$FLINK_HOME\lib\log4j*.jar`, and it makes logging correctly.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [zeppelin] zhoulii commented on a change in pull request #3884: [ZEPPELIN-5004] support running flink interpreter on kubernetes
Posted by GitBox <gi...@apache.org>.
zhoulii commented on a change in pull request #3884:
URL: https://github.com/apache/zeppelin/pull/3884#discussion_r476093970
##########
File path: k8s/interpreter/100-interpreter-spec.yaml
##########
@@ -78,6 +78,21 @@ spec:
- name: spark-home
emptyDir: {}
{% endif %}
+ {% if zeppelin.k8s.interpreter.group.name == "flink" %}
+ volumeMounts:
+ - name: flink-home
+ mountPath: /flink
+ initContainers:
+ - name: flink-home-init
+ image: {{zeppelin.k8s.flink.container.image}}
+ command: ["sh", "-c", "cp -r /opt/flink/* /flink/"]
Review comment:
Maybe it's not a good idea to copy files from images, I will close this PR, Thanks for explanation.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [zeppelin] Reamer commented on a change in pull request #3884: [ZEPPELIN-5004] support running flink interpreter on kubernetes
Posted by GitBox <gi...@apache.org>.
Reamer commented on a change in pull request #3884:
URL: https://github.com/apache/zeppelin/pull/3884#discussion_r474026135
##########
File path: k8s/interpreter/100-interpreter-spec.yaml
##########
@@ -78,6 +78,21 @@ spec:
- name: spark-home
emptyDir: {}
{% endif %}
+ {% if zeppelin.k8s.interpreter.group.name == "flink" %}
+ volumeMounts:
+ - name: flink-home
+ mountPath: /flink
+ initContainers:
+ - name: flink-home-init
+ image: {{zeppelin.k8s.flink.container.image}}
+ command: ["sh", "-c", "cp -r /opt/flink/* /flink/"]
Review comment:
We can add Flink in PR #3859, just like we add Python and R packages.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [zeppelin] zjffdu commented on pull request #3884: [ZEPPELIN-5004] support running flink interpreter on kubernetes
Posted by GitBox <gi...@apache.org>.
zjffdu commented on pull request #3884:
URL: https://github.com/apache/zeppelin/pull/3884#issuecomment-676497373
Thanks for the contribution @zhoulii , I think this is only to run flink interpreter on k8s, what about the JobManager & TaskManager ? Do you have plan to run flink cluster in k8s in future ?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [zeppelin] zhoulii commented on a change in pull request #3884: [ZEPPELIN-5004] support running flink interpreter on kubernetes
Posted by GitBox <gi...@apache.org>.
zhoulii commented on a change in pull request #3884:
URL: https://github.com/apache/zeppelin/pull/3884#discussion_r474009816
##########
File path: k8s/interpreter/100-interpreter-spec.yaml
##########
@@ -78,6 +78,21 @@ spec:
- name: spark-home
emptyDir: {}
{% endif %}
+ {% if zeppelin.k8s.interpreter.group.name == "flink" %}
+ volumeMounts:
+ - name: flink-home
+ mountPath: /flink
+ initContainers:
+ - name: flink-home-init
+ image: {{zeppelin.k8s.flink.container.image}}
+ command: ["sh", "-c", "cp -r /opt/flink/* /flink/"]
Review comment:
that would need some manual work, including extend `ZEPPELIN_K8S_CONTAINER_IMAGE` and set flink home in interpreter, I want to do these work automatically.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [zeppelin] Reamer commented on a change in pull request #3884: [ZEPPELIN-5004] support running flink interpreter on kubernetes
Posted by GitBox <gi...@apache.org>.
Reamer commented on a change in pull request #3884:
URL: https://github.com/apache/zeppelin/pull/3884#discussion_r473114030
##########
File path: k8s/interpreter/100-interpreter-spec.yaml
##########
@@ -78,6 +78,21 @@ spec:
- name: spark-home
emptyDir: {}
{% endif %}
+ {% if zeppelin.k8s.interpreter.group.name == "flink" %}
+ volumeMounts:
+ - name: flink-home
+ mountPath: /flink
+ initContainers:
+ - name: flink-home-init
+ image: {{zeppelin.k8s.flink.container.image}}
+ command: ["sh", "-c", "cp -r /opt/flink/* /flink/"]
Review comment:
I think you should extend your `ZEPPELIN_K8S_CONTAINER_IMAGE` and add Flink in the `/opt/flink` directory.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [zeppelin] zhoulii commented on a change in pull request #3884: [ZEPPELIN-5004] support running flink interpreter on kubernetes
Posted by GitBox <gi...@apache.org>.
zhoulii commented on a change in pull request #3884:
URL: https://github.com/apache/zeppelin/pull/3884#discussion_r473964942
##########
File path: k8s/interpreter/100-interpreter-spec.yaml
##########
@@ -78,6 +78,21 @@ spec:
- name: spark-home
emptyDir: {}
{% endif %}
+ {% if zeppelin.k8s.interpreter.group.name == "flink" %}
+ volumeMounts:
+ - name: flink-home
+ mountPath: /flink
+ initContainers:
+ - name: flink-home-init
+ image: {{zeppelin.k8s.flink.container.image}}
+ command: ["sh", "-c", "cp -r /opt/flink/* /flink/"]
Review comment:
sorry for wrong description. Actually , I mean flink is in '/opt/flink' directory in image `ZEPPELIN_K8S_FLINK_CONTAINER_IMAGE`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [zeppelin] zhoulii commented on a change in pull request #3884: [ZEPPELIN-5004] support running flink interpreter on kubernetes
Posted by GitBox <gi...@apache.org>.
zhoulii commented on a change in pull request #3884:
URL: https://github.com/apache/zeppelin/pull/3884#discussion_r473964942
##########
File path: k8s/interpreter/100-interpreter-spec.yaml
##########
@@ -78,6 +78,21 @@ spec:
- name: spark-home
emptyDir: {}
{% endif %}
+ {% if zeppelin.k8s.interpreter.group.name == "flink" %}
+ volumeMounts:
+ - name: flink-home
+ mountPath: /flink
+ initContainers:
+ - name: flink-home-init
+ image: {{zeppelin.k8s.flink.container.image}}
+ command: ["sh", "-c", "cp -r /opt/flink/* /flink/"]
Review comment:
sorry for wrong description. Actually , I mean flink is in '/opt/flink' directory in image `ZEPPELIN_K8S_FLINK_CONTAINER_IMAGE`. This paragraph mainly aims to copy flink dist from `ZEPPELIN_K8S_FLINK_CONTAINER_IMAGE` to interpreter container.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [zeppelin] Reamer commented on a change in pull request #3884: [ZEPPELIN-5004] support running flink interpreter on kubernetes
Posted by GitBox <gi...@apache.org>.
Reamer commented on a change in pull request #3884:
URL: https://github.com/apache/zeppelin/pull/3884#discussion_r475727402
##########
File path: k8s/interpreter/100-interpreter-spec.yaml
##########
@@ -78,6 +78,21 @@ spec:
- name: spark-home
emptyDir: {}
{% endif %}
+ {% if zeppelin.k8s.interpreter.group.name == "flink" %}
+ volumeMounts:
+ - name: flink-home
+ mountPath: /flink
+ initContainers:
+ - name: flink-home-init
+ image: {{zeppelin.k8s.flink.container.image}}
+ command: ["sh", "-c", "cp -r /opt/flink/* /flink/"]
Review comment:
All in all you are right.
> add different kinds of interpreter related files together may make interpreter image too large.
Therefore we want to give the users the [responsibility](https://github.com/apache/zeppelin/pull/3859/files#diff-f55297559468e8c543ae5ab6ee356ec4R66) to build the best image for their use case.
I think the copy process reduces the start time of the container and it makes the `interpreter-spec` more complicated. Today we have a special flink routine, tomorrow another one.
> it’s not flexible to change the interpreter related files，for example, if we need to change the flink version, we have to rebuild interpreter image
I think this is the biggest advantage and disadvantage of images. If you need to update any software version, including Flink, OpenJDK and operating system libraries, you have to rebuild your image. The advantage, you can make a nice rollback if your image contains all the application software. The disadvantage, of course, it that you have to rebuild your image, which takes quite a long time.
If you want to start certain interpreter images in your environment to reduce the image size, I recommend to create two or more interpreter images and inset a small if-then-else change [here](https://github.com/apache/zeppelin/blob/8ab7e9c473c5ce5b487bbab2a7b682f3fb165cab/k8s/interpreter/100-interpreter-spec.yaml#L45).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [zeppelin] Reamer commented on a change in pull request #3884: [ZEPPELIN-5004] support running flink interpreter on kubernetes
Posted by GitBox <gi...@apache.org>.
Reamer commented on a change in pull request #3884:
URL: https://github.com/apache/zeppelin/pull/3884#discussion_r473114030
##########
File path: k8s/interpreter/100-interpreter-spec.yaml
##########
@@ -78,6 +78,21 @@ spec:
- name: spark-home
emptyDir: {}
{% endif %}
+ {% if zeppelin.k8s.interpreter.group.name == "flink" %}
+ volumeMounts:
+ - name: flink-home
+ mountPath: /flink
+ initContainers:
+ - name: flink-home-init
+ image: {{zeppelin.k8s.flink.container.image}}
+ command: ["sh", "-c", "cp -r /opt/flink/* /flink/"]
Review comment:
I think you should change your `ZEPPELIN_K8S_CONTAINER_IMAGE` and add flink to /opt/flink.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [zeppelin] zhoulii commented on a change in pull request #3884: [ZEPPELIN-5004] support running flink interpreter on kubernetes
Posted by GitBox <gi...@apache.org>.
zhoulii commented on a change in pull request #3884:
URL: https://github.com/apache/zeppelin/pull/3884#discussion_r473547453
##########
File path: k8s/interpreter/100-interpreter-spec.yaml
##########
@@ -78,6 +78,21 @@ spec:
- name: spark-home
emptyDir: {}
{% endif %}
+ {% if zeppelin.k8s.interpreter.group.name == "flink" %}
+ volumeMounts:
+ - name: flink-home
+ mountPath: /flink
+ initContainers:
+ - name: flink-home-init
+ image: {{zeppelin.k8s.flink.container.image}}
+ command: ["sh", "-c", "cp -r /opt/flink/* /flink/"]
Review comment:
Actually, Flink is alreadly in `/opt/flink` directory in `ZEPPELIN_K8S_CONTAINER_IMAGE `. command here aims to mv flink dist file from `ZEPPELIN_K8S_CONTAINER_IMAGE` to volume `flink-home` which is mounted to '/flink' directory in initContainers.
In interpreter container, I also mounted `flink-home` to '/flink' directory, maybe '/opt/flink' directory would be better. So I changed flink home to ‘/opt/flink’ in the new commit. [100-interpreter-spec.yaml Lin84](https://github.com/apache/zeppelin/pull/3884/files#diff-1b67202451861591b920d5d4a9e31ce8R84) , [zeppelin-server.yaml Line41](https://github.com/apache/zeppelin/pull/3884/files#diff-56ccb2e2c2617b27dbaae866d9431e51R41)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [zeppelin] zhoulii commented on pull request #3884: [ZEPPELIN-5004] support running flink interpreter on kubernetes
Posted by GitBox <gi...@apache.org>.
zhoulii commented on pull request #3884:
URL: https://github.com/apache/zeppelin/pull/3884#issuecomment-678607595
> @zhoulii What is the log4j conflict issue ？
@zjffdu when starting flink interpreter with `flink 1.11.1`, I find this exception, it‘s resulted from log4j conflict：
```
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/flink/lib/log4j-slf4j-impl-2.12.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/zeppelin/interpreter/flink/zeppelin-flink-0.9.0-SNAPSHOT.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
ERROR StatusLogger Unable to access file:///opt/zeppelin/conf/log4j2.properties
java.io.FileNotFoundException: /opt/zeppelin/conf/log4j2.properties (No such file or directory)
at java.io.FileInputStream.open0(Native Method)
at java.io.FileInputStream.open(FileInputStream.java:195)
at java.io.FileInputStream.<init>(FileInputStream.java:138)
at java.io.FileInputStream.<init>(FileInputStream.java:93)
```
btw, start flink interpreter with flink 1.10.0 is ok.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [zeppelin] zhoulii commented on pull request #3884: [ZEPPELIN-5004] support running flink interpreter on kubernetes
Posted by GitBox <gi...@apache.org>.
zhoulii commented on pull request #3884:
URL: https://github.com/apache/zeppelin/pull/3884#issuecomment-676882177
> Thanks for the contribution @zhoulii , I think this is only to run flink interpreter on k8s, what about the JobManager & TaskManager ? Do you have plan to run flink cluster in k8s in future ?
Hi @zjffdu , thanks for reviewing this pr. you are right, this pr only aims to run flink interpreter on k8s, and execution modes of flink job are still `local|remote|yarn`.
Indeed it would be more reasonable to run flink job in [flink-kubernetes-session](https://ci.apache.org/projects/flink/flink-docs-release-1.11/ops/deployment/native_kubernetes.html#flink-kubernetes-session) mode when flink interpreter is running on k8s. I will find out how to add this new execution mode to flink interpreter in next pr.
btw, is there any problem with the change in `common.sh` and `interpreter.sh`? I am not sure whether it is a good way to solve jar conflict of log4j.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [zeppelin] zhoulii commented on a change in pull request #3884: [ZEPPELIN-5004] support running flink interpreter on kubernetes
Posted by GitBox <gi...@apache.org>.
zhoulii commented on a change in pull request #3884:
URL: https://github.com/apache/zeppelin/pull/3884#discussion_r475059807
##########
File path: k8s/interpreter/100-interpreter-spec.yaml
##########
@@ -78,6 +78,21 @@ spec:
- name: spark-home
emptyDir: {}
{% endif %}
+ {% if zeppelin.k8s.interpreter.group.name == "flink" %}
+ volumeMounts:
+ - name: flink-home
+ mountPath: /flink
+ initContainers:
+ - name: flink-home-init
+ image: {{zeppelin.k8s.flink.container.image}}
+ command: ["sh", "-c", "cp -r /opt/flink/* /flink/"]
Review comment:
Hi @Reamer , the way that add Python and R packages is ok，but it may have some limitations, for example:
1. add different kinds of related packages together may make interpreter image too large.
2. it’s not flexible to change the package，for example, if we need to change the package version, we have to rebuild interpreter image.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [zeppelin] Reamer commented on a change in pull request #3884: [ZEPPELIN-5004] support running flink interpreter on kubernetes
Posted by GitBox <gi...@apache.org>.
Reamer commented on a change in pull request #3884:
URL: https://github.com/apache/zeppelin/pull/3884#discussion_r473992065
##########
File path: k8s/interpreter/100-interpreter-spec.yaml
##########
@@ -78,6 +78,21 @@ spec:
- name: spark-home
emptyDir: {}
{% endif %}
+ {% if zeppelin.k8s.interpreter.group.name == "flink" %}
+ volumeMounts:
+ - name: flink-home
+ mountPath: /flink
+ initContainers:
+ - name: flink-home-init
+ image: {{zeppelin.k8s.flink.container.image}}
+ command: ["sh", "-c", "cp -r /opt/flink/* /flink/"]
Review comment:
Okay, thanks for clarifying. Why don't you extend your `ZEPPELIN_K8S_CONTAINER_IMAGE` with Flink under `/opt/flink`? If you extend your image, copying should not be necessary.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [zeppelin] Reamer commented on a change in pull request #3884: [ZEPPELIN-5004] support running flink interpreter on kubernetes
Posted by GitBox <gi...@apache.org>.
Reamer commented on a change in pull request #3884:
URL: https://github.com/apache/zeppelin/pull/3884#discussion_r473952093
##########
File path: k8s/interpreter/100-interpreter-spec.yaml
##########
@@ -78,6 +78,21 @@ spec:
- name: spark-home
emptyDir: {}
{% endif %}
+ {% if zeppelin.k8s.interpreter.group.name == "flink" %}
+ volumeMounts:
+ - name: flink-home
+ mountPath: /flink
+ initContainers:
+ - name: flink-home-init
+ image: {{zeppelin.k8s.flink.container.image}}
+ command: ["sh", "-c", "cp -r /opt/flink/* /flink/"]
Review comment:
> Actually, Flink is alreadly in /opt/flink directory in ZEPPELIN_K8S_CONTAINER_IMAGE
If Flink is already in your image (`ZEPPELIN_K8S_CONTAINER_IMAGE'), why do you need to copy Flink files that are in the initContainer into the interpreter container?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [zeppelin] zjffdu commented on pull request #3884: [ZEPPELIN-5004] support running flink interpreter on kubernetes
Posted by GitBox <gi...@apache.org>.
zjffdu commented on pull request #3884:
URL: https://github.com/apache/zeppelin/pull/3884#issuecomment-677707443
@zhoulii What is the log4j conflict issue ？
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [zeppelin] Reamer commented on a change in pull request #3884: [ZEPPELIN-5004] support running flink interpreter on kubernetes
Posted by GitBox <gi...@apache.org>.
Reamer commented on a change in pull request #3884:
URL: https://github.com/apache/zeppelin/pull/3884#discussion_r473952093
##########
File path: k8s/interpreter/100-interpreter-spec.yaml
##########
@@ -78,6 +78,21 @@ spec:
- name: spark-home
emptyDir: {}
{% endif %}
+ {% if zeppelin.k8s.interpreter.group.name == "flink" %}
+ volumeMounts:
+ - name: flink-home
+ mountPath: /flink
+ initContainers:
+ - name: flink-home-init
+ image: {{zeppelin.k8s.flink.container.image}}
+ command: ["sh", "-c", "cp -r /opt/flink/* /flink/"]
Review comment:
> Actually, Flink is alreadly in /opt/flink directory in ZEPPELIN_K8S_CONTAINER_IMAGE
If Flink is already in your image (`ZEPPELIN_K8S_CONTAINER_IMAGE`), why do you need to copy Flink files that are in the initContainer into the interpreter container?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [zeppelin] zhoulii commented on a change in pull request #3884: [ZEPPELIN-5004] support running flink interpreter on kubernetes
Posted by GitBox <gi...@apache.org>.
zhoulii commented on a change in pull request #3884:
URL: https://github.com/apache/zeppelin/pull/3884#discussion_r475059807
##########
File path: k8s/interpreter/100-interpreter-spec.yaml
##########
@@ -78,6 +78,21 @@ spec:
- name: spark-home
emptyDir: {}
{% endif %}
+ {% if zeppelin.k8s.interpreter.group.name == "flink" %}
+ volumeMounts:
+ - name: flink-home
+ mountPath: /flink
+ initContainers:
+ - name: flink-home-init
+ image: {{zeppelin.k8s.flink.container.image}}
+ command: ["sh", "-c", "cp -r /opt/flink/* /flink/"]
Review comment:
Hi @Reamer , the way that add Python and R packages is ok，but it may have some limitations, for example:
1. add different kinds of interpreter related files together may make interpreter image too large.
2. it’s not flexible to change the interpreter related files，for example, if we need to change the flink version, we have to rebuild interpreter image.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [zeppelin] zhoulii closed pull request #3884: [ZEPPELIN-5004] support running flink interpreter on kubernetes
Posted by GitBox <gi...@apache.org>.
zhoulii closed pull request #3884:
URL: https://github.com/apache/zeppelin/pull/3884
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org