You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "Sitan Pang (Jira)" <ji...@apache.org> on 2022/04/07 03:02:00 UTC
[jira] [Updated] (FLINK-27104) ClassNotFound when submit jobs to session cluster with external jars
[ https://issues.apache.org/jira/browse/FLINK-27104?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sitan Pang updated FLINK-27104:
-------------------------------
Description:
I'm trying to submit SQL jobs to session cluster on k8s, which need external local udfs jars.
I meet ClassNotFound Exception because of the reasons below:
* 'pipeline.jars' will be overwritten by '-j' option which only accepts one jar.
* 'pipeline.classpaths' will not be uploaded, so local files could not be found in TM.
* 'pipelines.cached-files' will not be added to classpaths. In Task#createUserCodeClassloader and ClientUtils#buildUserCodeClassLoader, only 'pipeline.jars' and 'pipeline.classpaths' are added to user class loader.
Only combining external jars into one could solve this, which means we need to re-create a new jar every time.
Is there a better way to submit with external jars?
Or could we make 'pipeline.jars' directly used by users?
was:
I'm trying to submit SQL jobs to session cluster on k8s, which need external local udfs jars.
I meet ClassNotFound Exception because of the reasons below:
* 'pipeline.jars' will be overwritten by '-j' option which only accepts one jar.
* 'pipeline.classpaths' will not be uploaded, so local files could not be found in TM.
* 'pipelines.cached-files' will not be added to classpaths. In Task#createUserCodeClassloader and ClientUtils#buildUserCodeClassLoader, only 'pipeline.jars' and 'pipeline.classpaths' are added to user class loader.
Only combining external jars into one could solve this, which means we need to re-create a new jar every time.
Is there a better way to submit with external jars?
Or could we make 'pipeline.jars' directly used by users?
> ClassNotFound when submit jobs to session cluster with external jars
> --------------------------------------------------------------------
>
> Key: FLINK-27104
> URL: https://issues.apache.org/jira/browse/FLINK-27104
> Project: Flink
> Issue Type: Improvement
> Components: Client / Job Submission
> Affects Versions: 1.11.0
> Reporter: Sitan Pang
> Priority: Minor
>
> I'm trying to submit SQL jobs to session cluster on k8s, which need external local udfs jars.
> I meet ClassNotFound Exception because of the reasons below:
> * 'pipeline.jars' will be overwritten by '-j' option which only accepts one jar.
> * 'pipeline.classpaths' will not be uploaded, so local files could not be found in TM.
> * 'pipelines.cached-files' will not be added to classpaths. In Task#createUserCodeClassloader and ClientUtils#buildUserCodeClassLoader, only 'pipeline.jars' and 'pipeline.classpaths' are added to user class loader.
> Only combining external jars into one could solve this, which means we need to re-create a new jar every time.
> Is there a better way to submit with external jars?
> Or could we make 'pipeline.jars' directly used by users?
>
--
This message was sent by Atlassian Jira
(v8.20.1#820001)