You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "pawel-big-lebowski (via GitHub)" <gi...@apache.org> on 2023/06/22 08:37:16 UTC
[GitHub] [iceberg] pawel-big-lebowski opened a new issue, #7879: [Flink] NullPointer when doing `FlinkEnvironmentContext.init` for Flink 1.17 and iceberg 1.3.0
pawel-big-lebowski opened a new issue, #7879:
URL: https://github.com/apache/iceberg/issues/7879
### Apache Iceberg version
1.3.0 (latest release)
### Query engine
Flink
### Please describe the bug 🐞
I am running Flink 1.17 and iceberg 1.3.0. I am trying to start simple example app with a code available here:
https://github.com/OpenLineage/OpenLineage/blob/main/integration/flink/examples/stateful/src/main/java/io/openlineage/flink/FlinkIcebergApplication.java#L19
Building a Flink source fails with NPE which originates in `FlinkEnvironmentContext` when doing
```
public static void init() {
EnvironmentContext.put("engine-name", "flink");
EnvironmentContext.put("engine-version", FlinkPackage.version());
}
```
The problem is that `FlinkPackage.version()` is null. The version is equal to:
```
private static final String VERSION = DataStream.class.getPackage().getImplementationVersion();
```
`DataStream.class` is on my classpath but `getImplementationVersion()` gives null.
The error is observer when running integration test in docker container `flink:1.17.1-java11`.
The same app used to work for previous flink & iceberg versions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] pawel-big-lebowski commented on issue #7879: [Flink] NullPointer when doing `FlinkEnvironmentContext.init` for Flink 1.17 and iceberg 1.3.0
Posted by "pawel-big-lebowski (via GitHub)" <gi...@apache.org>.
pawel-big-lebowski commented on issue #7879:
URL: https://github.com/apache/iceberg/issues/7879#issuecomment-1602558922
@nastra No particular reason behind having both. In one of my tests, I got rid of the top one and this did not change anything.
I am able to see `DataStream.class.getPackage().getName()` which suggests the jar is available on a classpath. And this is something I find confusing most.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] stevenzwu commented on issue #7879: [Flink] NullPointer when doing `FlinkEnvironmentContext.init` for Flink 1.17 and iceberg 1.3.0
Posted by "stevenzwu (via GitHub)" <gi...@apache.org>.
stevenzwu commented on issue #7879:
URL: https://github.com/apache/iceberg/issues/7879#issuecomment-1611467354
@pawel-big-lebowski it is good to improve this. feel free to submit a PR.
which uber jar you were referring to?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] Fokko commented on issue #7879: [Flink] NullPointer when doing `FlinkEnvironmentContext.init` for Flink 1.17 and iceberg 1.3.0
Posted by "Fokko (via GitHub)" <gi...@apache.org>.
Fokko commented on issue #7879:
URL: https://github.com/apache/iceberg/issues/7879#issuecomment-1602464370
cc @stevenzwu
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] nastra commented on issue #7879: [Flink] NullPointer when doing `FlinkEnvironmentContext.init` for Flink 1.17 and iceberg 1.3.0
Posted by "nastra (via GitHub)" <gi...@apache.org>.
nastra commented on issue #7879:
URL: https://github.com/apache/iceberg/issues/7879#issuecomment-1602258069
My first guess would be that there's maybe something wrong with the classpath. Any particular reason that you both have
```
compileOnly "org.apache.iceberg:iceberg-flink:$icebergVersion"
implementation "org.apache.iceberg:iceberg-flink-runtime-$flinkVersionShort:$icebergVersion"
```
on the classpath?
Do you know why `getImplementationVersion()` returns a `null` in your case?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] stevenzwu commented on issue #7879: [Flink] NullPointer when doing `FlinkEnvironmentContext.init` for Flink 1.17 and iceberg 1.3.0
Posted by "stevenzwu (via GitHub)" <gi...@apache.org>.
stevenzwu commented on issue #7879:
URL: https://github.com/apache/iceberg/issues/7879#issuecomment-1604527888
hmm. weird problem. I had the same suspicion as Eduard said earlier. seems like some classpath issue in local setup. I haven't seen this problem in our env.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] [Flink] NullPointer when doing `FlinkEnvironmentContext.init` for Flink 1.17 and iceberg 1.3.0 [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary closed issue #7879: [Flink] NullPointer when doing `FlinkEnvironmentContext.init` for Flink 1.17 and iceberg 1.3.0
URL: https://github.com/apache/iceberg/issues/7879
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] [Flink] NullPointer when doing `FlinkEnvironmentContext.init` for Flink 1.17 and iceberg 1.3.0 [iceberg]
Posted by "UlyanovskCN (via GitHub)" <gi...@apache.org>.
UlyanovskCN commented on issue #7879:
URL: https://github.com/apache/iceberg/issues/7879#issuecomment-1958492984
try set your flink-streaming dependency scope to provided,I solved same problem throught it
<groupId>org.apache.flink</groupId>
<artifactId>flink-streaming-java_${scala.binary.version}</artifactId>
<version>${flink.version}</version>
<scope>**provided**</scope>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] pawel-big-lebowski commented on issue #7879: [Flink] NullPointer when doing `FlinkEnvironmentContext.init` for Flink 1.17 and iceberg 1.3.0
Posted by "pawel-big-lebowski (via GitHub)" <gi...@apache.org>.
pawel-big-lebowski commented on issue #7879:
URL: https://github.com/apache/iceberg/issues/7879#issuecomment-1611303688
Hi @nastra @stevenzwu , I think I found the reason. So, for some reason we were creating Uber-jar with shadow plugin which contained Flink libraries including `org.apache.flink:flink-streaming-java`. This was not necessary and we can get rid of it and this solves the problem on our side.
So the scenario is:
* Someone tries to create Uber-jar containing `org.apache.flink:flink-streaming-java`,
* While using gradle's `shadowJar` plugin, manifests of dependencies get overwritten,
* The code
```
DataStream.class.getPackage().getImplementationVersion();
```
returns `null` and `NullPointerException` is thrown when doing `EnvironmentContext.put("engine-version", FlinkPackage.version());`
The bug is nasty to understand because one does have `DataStream.class` on the classpath but the manifest with `implementationVersion` is missing. And the only thing observed on the top is `NullPointerException`.
So, if you don't mind, I could prepare a PR which prevents putting `null` in `org.apache.iceberg.flink.FlinkEnvironmentContext#init` and throws more meaningful Exception. It would have been even better if you were able to find flink version some other way. Alternatively, we can close the issue if it so rare and uncommon to happen elsewhere.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] nastra commented on issue #7879: [Flink] NullPointer when doing `FlinkEnvironmentContext.init` for Flink 1.17 and iceberg 1.3.0
Posted by "nastra (via GitHub)" <gi...@apache.org>.
nastra commented on issue #7879:
URL: https://github.com/apache/iceberg/issues/7879#issuecomment-1611397779
Maybe we could use a fallback to a class which typically doesn't get shadowed? Something like `private static final String VERSION = Optional.ofNullable(DataStream.class.getPackage().getImplementationVersion()).orElse(...);`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] stevenzwu commented on issue #7879: [Flink] NullPointer when doing `FlinkEnvironmentContext.init` for Flink 1.17 and iceberg 1.3.0
Posted by "stevenzwu (via GitHub)" <gi...@apache.org>.
stevenzwu commented on issue #7879:
URL: https://github.com/apache/iceberg/issues/7879#issuecomment-1611467628
@pawel-big-lebowski it is good to improve this. feel free to submit a PR.
which uber jar you were referring to?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] [Flink] NullPointer when doing `FlinkEnvironmentContext.init` for Flink 1.17 and iceberg 1.3.0 [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary closed issue #7879: [Flink] NullPointer when doing `FlinkEnvironmentContext.init` for Flink 1.17 and iceberg 1.3.0
URL: https://github.com/apache/iceberg/issues/7879
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org