You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@seatunnel.apache.org by GitBox <gi...@apache.org> on 2022/07/28 12:23:01 UTC
[GitHub] [incubator-seatunnel] TyrantLucifer opened a new issue, #2291: [Bug] [Flink-connector-v2-e2e] Flink docker image not contains hadoop dependency
TyrantLucifer opened a new issue, #2291:
URL: https://github.com/apache/incubator-seatunnel/issues/2291
### Search before asking
- [X] I had searched in the [issues](https://github.com/apache/incubator-seatunnel/issues?q=is%3Aissue+label%3A%22bug%22) and found no similar issues.
### What happened
Because the flink official docker image didn't have hadoop dependency, so when we try to use flink run some connectors that need hadoop dependency such as `connector-file-hadoop` `connector-hive` and so on it will throw some exceptions like `NoClassFound`.
In order to solve this problem, I think we should build our own flink images. I have builded an example flink image (`tyrantlucifer/flink:1.13.6-scala_2.11_hadoop27`) and push it to docker hub, the docker file is very simple as the below shown:
```dockerfile
FROM flink:1.13.6-scala_2.11
ENV HADOOP_UBER_URL=https://repo.maven.apache.org/maven2/org/apache/flink/flink-shaded-hadoop-2-uber/2.7.5-10.0/flink-shaded-hadoop-2-uber-2.7.5-10.0.jar
RUN wget -nv -O /opt/flink/lib/flink-shaded-hadoop-2-uber.jar "${HADOOP_UBER_URL}";
```
If the internet speed is not good, we also can download the jar and use this dockerfile:
```dockerfile
FROM flink:1.13.6-scala_2.11
COPY ./flink-shaded-hadoop-2-uber-2.7.5-10.0.jar /opt/flink/lib/
```
What do you think about it? @CalvinKirs @ruanwenjun
Is to write my mirror or seatunnel official production of a special mirror?
### SeaTunnel Version
dev
### SeaTunnel Config
```conf
none
```
### Running Command
```shell
none
```
### Error Exception
```log
NoClassFound Exception
```
### Flink or Spark Version
_No response_
### Java or Scala Version
_No response_
### Screenshots
_No response_
### Are you willing to submit PR?
- [X] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] TyrantLucifer closed issue #2291: [Bug] [Flink-connector-v2-e2e] Flink docker image not contains hadoop dependency
Posted by GitBox <gi...@apache.org>.
TyrantLucifer closed issue #2291: [Bug] [Flink-connector-v2-e2e] Flink docker image not contains hadoop dependency
URL: https://github.com/apache/incubator-seatunnel/issues/2291
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] TyrantLucifer commented on issue #2291: [Bug] [Flink-connector-v2-e2e] Flink docker image not contains hadoop dependency
Posted by GitBox <gi...@apache.org>.
TyrantLucifer commented on issue #2291:
URL: https://github.com/apache/incubator-seatunnel/issues/2291#issuecomment-1199115619
> How about add it in our kubernetes guide? Add command to download jar in dockerfile.
It's a good idea, we can add the step in our own Dockfile and I can do this work. For e2e test case we can use the docker image that I had builded, what do you think about it?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] TyrantLucifer commented on issue #2291: [Bug] [Flink-connector-v2-e2e] Flink docker image not contains hadoop dependency
Posted by GitBox <gi...@apache.org>.
TyrantLucifer commented on issue #2291:
URL: https://github.com/apache/incubator-seatunnel/issues/2291#issuecomment-1239041038
pr #2297
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] Hisoka-X commented on issue #2291: [Bug] [Flink-connector-v2-e2e] Flink docker image not contains hadoop dependency
Posted by GitBox <gi...@apache.org>.
Hisoka-X commented on issue #2291:
URL: https://github.com/apache/incubator-seatunnel/issues/2291#issuecomment-1199146427
> > How about add it in our kubernetes guide? Add command to download jar in dockerfile.
>
> It's a good idea, we can add the step in our own Dockfile and I can do this work. For e2e test case we can use the docker image that I had builded, what do you think about it?
Ok for me.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] Hisoka-X commented on issue #2291: [Bug] [Flink-connector-v2-e2e] Flink docker image not contains hadoop dependency
Posted by GitBox <gi...@apache.org>.
Hisoka-X commented on issue #2291:
URL: https://github.com/apache/incubator-seatunnel/issues/2291#issuecomment-1199013057
How about add it in our kubernetes guide? Add command to download jar in dockerfile.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org