You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by GitBox <gi...@apache.org> on 2020/01/24 15:56:21 UTC
[GitHub] [flink] aljoscha opened a new pull request #10944: [FLINK-15739]
Fix TypeSerializerUpgradeTestBase on Java 12
aljoscha opened a new pull request #10944: [FLINK-15739] Fix TypeSerializerUpgradeTestBase on Java 12
URL: https://github.com/apache/flink/pull/10944
The problem was that some of the relocated classes in
PojoSerializerUpgradeTest have enums in them. Java 12 introduced the
EnumDesc class and enums will have an inner subclass of this defined
that does not have a ClassLoader. We now filter out classes that don't
have a classloader in the ClassRelocator because these classes don't
have bytecode that we could relocate.
cc @igalshilman Do you have an opinion on this?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] zentol commented on issue #10944: [FLINK-15739] Fix
TypeSerializerUpgradeTestBase on Java 12
Posted by GitBox <gi...@apache.org>.
zentol commented on issue #10944: [FLINK-15739] Fix TypeSerializerUpgradeTestBase on Java 12
URL: https://github.com/apache/flink/pull/10944#issuecomment-578698301
@aljoscha Could this also be achieved by having these classes in separate modules and loading them with different classloaders?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] igalshilman commented on issue #10944: [FLINK-15739] Fix
TypeSerializerUpgradeTestBase on Java 12
Posted by GitBox <gi...@apache.org>.
igalshilman commented on issue #10944: [FLINK-15739] Fix TypeSerializerUpgradeTestBase on Java 12
URL: https://github.com/apache/flink/pull/10944#issuecomment-579164401
@zentol The main problem that this approach solves is: we need two different definitions of the same class (same fully qualified class name, but different fields) to be present at runtime. Having the classes in different modules, means that these modules can not be in the class path simultaneously. (how would the pom.xml would look like with these constrains?)
So the relocating approach allows these two classes co-existing with different compile time names, but they are loaded into separate classloaders with the same (fully qualified name).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] zentol commented on issue #10944: [FLINK-15739] Fix
TypeSerializerUpgradeTestBase on Java 12
Posted by GitBox <gi...@apache.org>.
zentol commented on issue #10944: [FLINK-15739] Fix TypeSerializerUpgradeTestBase on Java 12
URL: https://github.com/apache/flink/pull/10944#issuecomment-579261556
@igalshilman According to Aljoscha they don't have to be on the test classpath at the same time; you need one version to be on _some_ classpath during the setup, the other on _some other_ classpath during the verification.
As for the classpath, the test module would exclude the old/new serializers modules from the test classpath via the surefire plugin, but load the jars through a ChildFirstClassloader.
@aljoscha That still doesn't quite explain why you'd need multiple modules; are you worried about dependency conflicts? Why would the test module not be able to depend on the uber-old-implementation module? (even then, you could exclude all the stuff you don't need...)
You would probably require an additional test module for each serializer (since the implementation module would depend on the main serializer module, but then the test can't be in the main module due to dependency recursion).
I'll happily admit that doing "clever" relocations make the test easier to write, but I'm wondering whether a "dumb" isn't more representative of what happens in reality.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] zentol commented on issue #10944: [FLINK-15739] Fix
TypeSerializerUpgradeTestBase on Java 12
Posted by GitBox <gi...@apache.org>.
zentol commented on issue #10944: [FLINK-15739] Fix TypeSerializerUpgradeTestBase on Java 12
URL: https://github.com/apache/flink/pull/10944#issuecomment-578531669
Does fix the issue.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] aljoscha commented on issue #10944: [FLINK-15739] Fix
TypeSerializerUpgradeTestBase on Java 12
Posted by GitBox <gi...@apache.org>.
aljoscha commented on issue #10944: [FLINK-15739] Fix TypeSerializerUpgradeTestBase on Java 12
URL: https://github.com/apache/flink/pull/10944#issuecomment-578687318
@zentol The tests do relocation because the fake a changing class. This is done by having two different classes in the code but relocating them to be one and the same class at runtime. At different points in time, though, so in the setup one class will be relocated to the target class name and in the verification step another class will be relocated to the target class name. You can see this in https://github.com/apache/flink/blob/15f175a7bcbe4cc83349fc13d022a5ef5ea0db71/flink-core/src/test/java/org/apache/flink/api/java/typeutils/runtime/PojoSerializerUpgradeTestSpecifications.java#L239 and the following verification class.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] igalshilman commented on issue #10944: [FLINK-15739] Fix
TypeSerializerUpgradeTestBase on Java 12
Posted by GitBox <gi...@apache.org>.
igalshilman commented on issue #10944: [FLINK-15739] Fix TypeSerializerUpgradeTestBase on Java 12
URL: https://github.com/apache/flink/pull/10944#issuecomment-578536689
@aljoscha I wasn't familiar with EnumDesc before. I think that your conclusion and approach make sense.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #10944: [FLINK-15739]
Fix TypeSerializerUpgradeTestBase on Java 12
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on issue #10944: [FLINK-15739] Fix TypeSerializerUpgradeTestBase on Java 12
URL: https://github.com/apache/flink/pull/10944#issuecomment-578198978
<!--
Meta data
Hash:de289e8371ce14475bba8bda0865aa8b9264da8c Status:SUCCESS URL:https://travis-ci.com/flink-ci/flink/builds/145963837 TriggerType:PUSH TriggerID:de289e8371ce14475bba8bda0865aa8b9264da8c
Hash:de289e8371ce14475bba8bda0865aa8b9264da8c Status:SUCCESS URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4612 TriggerType:PUSH TriggerID:de289e8371ce14475bba8bda0865aa8b9264da8c
-->
## CI report:
* de289e8371ce14475bba8bda0865aa8b9264da8c Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/145963837) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4612)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] zentol edited a comment on issue #10944: [FLINK-15739] Fix
TypeSerializerUpgradeTestBase on Java 12
Posted by GitBox <gi...@apache.org>.
zentol edited a comment on issue #10944: [FLINK-15739] Fix TypeSerializerUpgradeTestBase on Java 12
URL: https://github.com/apache/flink/pull/10944#issuecomment-578698301
@aljoscha Could this also be achieved by having these classes in separate modules(==jars) and loading them with different classloaders?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] zentol commented on issue #10944: [FLINK-15739] Fix
TypeSerializerUpgradeTestBase on Java 12
Posted by GitBox <gi...@apache.org>.
zentol commented on issue #10944: [FLINK-15739] Fix TypeSerializerUpgradeTestBase on Java 12
URL: https://github.com/apache/flink/pull/10944#issuecomment-578860300
@aljoscha Why would you need separate modules per test? Couldn't there be 1 module containing all the old version?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] aljoscha commented on issue #10944: [FLINK-15739] Fix
TypeSerializerUpgradeTestBase on Java 12
Posted by GitBox <gi...@apache.org>.
aljoscha commented on issue #10944: [FLINK-15739] Fix TypeSerializerUpgradeTestBase on Java 12
URL: https://github.com/apache/flink/pull/10944#issuecomment-578746519
@zentol Yes, that would theoretically also be possible, I think. But you would need basically three modules (one for old, one for new, and one for the actual test) per serializer upgrade test, of which there are roughly 40.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #10944: [FLINK-15739]
Fix TypeSerializerUpgradeTestBase on Java 12
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on issue #10944: [FLINK-15739] Fix TypeSerializerUpgradeTestBase on Java 12
URL: https://github.com/apache/flink/pull/10944#issuecomment-578198978
<!--
Meta data
Hash:de289e8371ce14475bba8bda0865aa8b9264da8c Status:PENDING URL:https://travis-ci.com/flink-ci/flink/builds/145963837 TriggerType:PUSH TriggerID:de289e8371ce14475bba8bda0865aa8b9264da8c
Hash:de289e8371ce14475bba8bda0865aa8b9264da8c Status:PENDING URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4612 TriggerType:PUSH TriggerID:de289e8371ce14475bba8bda0865aa8b9264da8c
-->
## CI report:
* de289e8371ce14475bba8bda0865aa8b9264da8c Travis: [PENDING](https://travis-ci.com/flink-ci/flink/builds/145963837) Azure: [PENDING](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4612)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] flinkbot commented on issue #10944: [FLINK-15739] Fix
TypeSerializerUpgradeTestBase on Java 12
Posted by GitBox <gi...@apache.org>.
flinkbot commented on issue #10944: [FLINK-15739] Fix TypeSerializerUpgradeTestBase on Java 12
URL: https://github.com/apache/flink/pull/10944#issuecomment-578198978
<!--
Meta data
Hash:de289e8371ce14475bba8bda0865aa8b9264da8c Status:UNKNOWN URL:TBD TriggerType:PUSH TriggerID:de289e8371ce14475bba8bda0865aa8b9264da8c
-->
## CI report:
* de289e8371ce14475bba8bda0865aa8b9264da8c UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] aljoscha commented on issue #10944: [FLINK-15739] Fix
TypeSerializerUpgradeTestBase on Java 12
Posted by GitBox <gi...@apache.org>.
aljoscha commented on issue #10944: [FLINK-15739] Fix TypeSerializerUpgradeTestBase on Java 12
URL: https://github.com/apache/flink/pull/10944#issuecomment-579148367
Because there are serializer upgrade tests in different modules, i.e. they are where the serializer is implemented.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] aljoscha commented on issue #10944: [FLINK-15739] Fix
TypeSerializerUpgradeTestBase on Java 12
Posted by GitBox <gi...@apache.org>.
aljoscha commented on issue #10944: [FLINK-15739] Fix TypeSerializerUpgradeTestBase on Java 12
URL: https://github.com/apache/flink/pull/10944#issuecomment-578687917
Merged!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] aljoscha commented on issue #10944: [FLINK-15739] Fix
TypeSerializerUpgradeTestBase on Java 12
Posted by GitBox <gi...@apache.org>.
aljoscha commented on issue #10944: [FLINK-15739] Fix TypeSerializerUpgradeTestBase on Java 12
URL: https://github.com/apache/flink/pull/10944#issuecomment-579304985
As I said, I think it's doable. But even if it requires only one additional test module per serializer that's quite a lot of modules.
But I wouldn't work on that since the test setup we have works well and is reasonably simple. Ignoring this current issue here. 😅
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] aljoscha closed pull request #10944: [FLINK-15739] Fix
TypeSerializerUpgradeTestBase on Java 12
Posted by GitBox <gi...@apache.org>.
aljoscha closed pull request #10944: [FLINK-15739] Fix TypeSerializerUpgradeTestBase on Java 12
URL: https://github.com/apache/flink/pull/10944
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] zentol commented on issue #10944: [FLINK-15739] Fix
TypeSerializerUpgradeTestBase on Java 12
Posted by GitBox <gi...@apache.org>.
zentol commented on issue #10944: [FLINK-15739] Fix TypeSerializerUpgradeTestBase on Java 12
URL: https://github.com/apache/flink/pull/10944#issuecomment-578227574
I'll try this fix out over the weekend.
Could someone explain to me why these tests need this class relocation logic in the first place?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #10944: [FLINK-15739]
Fix TypeSerializerUpgradeTestBase on Java 12
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on issue #10944: [FLINK-15739] Fix TypeSerializerUpgradeTestBase on Java 12
URL: https://github.com/apache/flink/pull/10944#issuecomment-578198978
<!--
Meta data
Hash:de289e8371ce14475bba8bda0865aa8b9264da8c Status:SUCCESS URL:https://travis-ci.com/flink-ci/flink/builds/145963837 TriggerType:PUSH TriggerID:de289e8371ce14475bba8bda0865aa8b9264da8c
Hash:de289e8371ce14475bba8bda0865aa8b9264da8c Status:PENDING URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4612 TriggerType:PUSH TriggerID:de289e8371ce14475bba8bda0865aa8b9264da8c
-->
## CI report:
* de289e8371ce14475bba8bda0865aa8b9264da8c Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/145963837) Azure: [PENDING](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4612)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] flinkbot commented on issue #10944: [FLINK-15739] Fix
TypeSerializerUpgradeTestBase on Java 12
Posted by GitBox <gi...@apache.org>.
flinkbot commented on issue #10944: [FLINK-15739] Fix TypeSerializerUpgradeTestBase on Java 12
URL: https://github.com/apache/flink/pull/10944#issuecomment-578190477
Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community
to review your pull request. We will use this comment to track the progress of the review.
## Automated Checks
Last check on commit de289e8371ce14475bba8bda0865aa8b9264da8c (Fri Jan 24 15:59:33 UTC 2020)
**Warnings:**
* No documentation files were touched! Remember to keep the Flink docs up to date!
* **This pull request references an unassigned [Jira ticket](https://issues.apache.org/jira/browse/FLINK-15739).** According to the [code contribution guide](https://flink.apache.org/contributing/contribute-code.html), tickets need to be assigned before starting with the implementation work.
<sub>Mention the bot in a comment to re-run the automated checks.</sub>
## Review Progress
* ❓ 1. The [description] looks good.
* ❓ 2. There is [consensus] that the contribution should go into to Flink.
* ❓ 3. Needs [attention] from.
* ❓ 4. The change fits into the overall [architecture].
* ❓ 5. Overall code [quality] is good.
Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process.<details>
The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required <summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`)
- `@flinkbot approve all` to approve all aspects
- `@flinkbot approve-until architecture` to approve everything until `architecture`
- `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention
- `@flinkbot disapprove architecture` to remove an approval you gave earlier
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services