You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@beam.apache.org by "Lukasz Gajowy (Jira)" <ji...@apache.org> on 2019/12/17 15:04:00 UTC
[jira] [Comment Edited] (BEAM-5495) PipelineResources algorithm is
not working in most environments
[ https://issues.apache.org/jira/browse/BEAM-5495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16998297#comment-16998297 ]
Lukasz Gajowy edited comment on BEAM-5495 at 12/17/19 3:03 PM:
---------------------------------------------------------------
[~romain.manni-bucau] well then, it seems that I was not careful enough - apologies. If you want to correct my error [?] always feel free to contribute your solution improving the one I proposed.
Regarding your concern - as I said I don't know much about xbean so it's hard for me to compare them in terms of quality, "giving guarantee of health" and specifically say which one is a better option for the job. All I can say is that all the input in the PR was talked about earlier on the [devlist|https://lists.apache.org/thread.html/61ae8750b4ed20413c6e93ba949ddd48dd0107a0a039ef518f9d6d21%40%3Cdev.beam.apache.org%3E] and classgraph seemed a sensible solution. This, of course, can be discussed further.
The bright side is that changing the implementation is as easy as implementing the PipelineResourcesDetector interface so it should be a relatively easy thing to do. The default algorithm can also be changed easily.
was (Author: łukaszg):
[~romain.manni-bucau] well then, it seems that I was not careful enough - apologies. If you want to correct my error(?) always feel free to contribute your solution improving the one I proposed.
Regarding your concern - as I said I don't know much about xbean so it's hard for me to compare them in terms of quality, "giving guarantee of health" and specifically say which one is a better option for the job. All I can say is that all the input in the PR was talked about earlier on the [devlist|https://lists.apache.org/thread.html/61ae8750b4ed20413c6e93ba949ddd48dd0107a0a039ef518f9d6d21%40%3Cdev.beam.apache.org%3E] and classgraph seemed a sensible solution. This, of course, can be discussed further.
The bright side is that changing the implementation is as easy as implementing the PipelineResourcesDetector interface so it should be a relatively easy thing to do. The default algorithm can also be changed easily.
> PipelineResources algorithm is not working in most environments
> ---------------------------------------------------------------
>
> Key: BEAM-5495
> URL: https://issues.apache.org/jira/browse/BEAM-5495
> Project: Beam
> Issue Type: Bug
> Components: runner-flink, runner-spark, sdk-java-core
> Reporter: Romain Manni-Bucau
> Assignee: Lukasz Gajowy
> Priority: Major
> Fix For: 2.19.0
>
> Time Spent: 14.5h
> Remaining Estimate: 0h
>
> Issue are:
> 1. it assumes the classloader is an URLClassLoader (not always true and java >= 9 breaks that as well for the app loader)
> 2. it uses loader.getURLs() which leads to including the JRE itself in the staged file
> Looks like this detect resource algorithm can't work and should be replaced by a SPI rather than a built-in and not extensible algorithm. Another valid alternative is to just drop that "guess" logic and force the user to set staged files.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)