You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@flink.apache.org by "NEKRASSOV, ALEXEI" <an...@att.com> on 2018/07/03 17:43:19 UTC
Job jumbo jar can't be extracted on Windows
When I try to "flink run" a job jar that includes dependent jar's - it fails on Windows with the following Exception:
org.apache.flink.client.program.ProgramInvocationException: Unknown I/O error while extracting contained jar files.
at org.apache.flink.client.program.PackagedProgram.extractContainedLibraries(PackagedProgram.java:752)
at org.apache.flink.client.program.PackagedProgram.<init>(PackagedProgram.java:194)
at org.apache.flink.client.cli.CliFrontend.buildProgram(CliFrontend.java:833)
at org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:201)
at org.apache.flink.client.cli.CliFrontend.parseParameters(CliFrontend.java:1020)
at org.apache.flink.client.cli.CliFrontend.lambda$main$9(CliFrontend.java:1096)
at org.apache.flink.runtime.security.NoOpSecurityContext.runSecured(NoOpSecurityContext.java:30)
at org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:1096)
Caused by: org.apache.flink.client.program.ProgramInvocationException: An I/O error occurred while creating temporary file to extract nested library 'lib/flink-connector-kafka-0.10_2.11-1.4.1.jar'.
at org.apache.flink.client.program.PackagedProgram.extractContainedLibraries(PackagedProgram.java:708)
... 7 more
Caused by: java.io.IOException: Unable to create temporary file, C:\Users\XXX\AppData\Local\Temp\1863208689_4625824260650653022lib\flink-connector-kafka-0.10_2.11-1.4.1.jar
at java.io.File$TempDirectory.generateFile(Unknown Source)
at java.io.File.createTempFile(Unknown Source)
at java.io.File.createTempFile(Unknown Source)
at org.apache.flink.client.program.PackagedProgram.extractContainedLibraries(PackagedProgram.java:702)
... 7 more
I think we have a problem in PackagedProgram.java - it fails to extract contained jar's on Windows, because on Windows File.separatorChar is '\', but JarEntry.getName() returns '/' as file separator.
Line 699 of https://github.com/apache/flink/blob/master/flink-clients/src/main/java/org/apache/flink/client/program/PackagedProgram.java is no-op on Windows.
I think we need to replace line 699 with
name = name.replace('/', '_');
as I don't see a way to have JarEntry use platform-specific file separator, and ZIP spec dictates the use of forward-slashes anyway.
Line 680 already uses hard-coded '/'.
Alex Nekrassov
Re: Job jumbo jar can't be extracted on Windows
Posted by Chesnay Schepler <ch...@apache.org>.
https://issues.apache.org/jira/browse/FLINK-9743
On 04.07.2018 16:33, Chesnay Schepler wrote:
> could reproduce it, sounds about right that we should always replace "/".
>
> On 04.07.2018 16:14, NEKRASSOV, ALEXEI wrote:
>> To demonstrate this problem:
>>
>> 1. Get a jar that includes other jar's. For example, download
>> http://central.maven.org/maven2/org/eclipse/platform/org.eclipse.ant.ui/3.6.201/org.eclipse.ant.ui-3.6.201.jar
>> 2. Put downloaded jar in c:\usr\tmp\
>> 3. In cmd
>> (note that we attempt to create
>> 1751416743_6922010711856647205lib\antrunner.jar instead of
>> 1751416743_6922010711856647205lib_antrunner.jar)
>>
>> c:\Program Files\flink-1.5.0\bin>flink -v
>> Version: 1.5.0, Commit ID: c61b108
>>
>> c:\Program Files\flink-1.5.0\bin>flink run -c WhoCares
>> c:\usr\tmp\org.eclipse.ant.ui-3.6.201.jar
>>
>> ------------------------------------------------------------
>> The program finished with the following exception:
>>
>> org.apache.flink.client.program.ProgramInvocationException: Unknown
>> I/O error while extracting contained jar files.
>> at
>> org.apache.flink.client.program.PackagedProgram.extractContainedLibraries(PackagedProgram.java:752)
>> at
>> org.apache.flink.client.program.PackagedProgram.<init>(PackagedProgram.java:194)
>> at
>> org.apache.flink.client.cli.CliFrontend.buildProgram(CliFrontend.java:833)
>> at
>> org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:201)
>> at
>> org.apache.flink.client.cli.CliFrontend.parseParameters(CliFrontend.java:1020)
>> at
>> org.apache.flink.client.cli.CliFrontend.lambda$main$9(CliFrontend.java:1096)
>> at
>> org.apache.flink.runtime.security.NoOpSecurityContext.runSecured(NoOpSecurityContext.java:30)
>> at
>> org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:1096)
>> Caused by:
>> org.apache.flink.client.program.ProgramInvocationException: An I/O
>> error occurred while creating temporary file to extract nested
>> library 'lib/antrunner.jar'.
>> at
>> org.apache.flink.client.program.PackagedProgram.extractContainedLibraries(PackagedProgram.java:708)
>> ... 7 more
>> Caused by: java.io.IOException: Unable to create temporary file,
>> C:\Users\XXX\AppData\Local\Temp\1751416743_6922010711856647205lib\antrunner.jar
>>
>> at java.io.File$TempDirectory.generateFile(Unknown Source)
>> at java.io.File.createTempFile(Unknown Source)
>> at java.io.File.createTempFile(Unknown Source)
>> at
>> org.apache.flink.client.program.PackagedProgram.extractContainedLibraries(PackagedProgram.java:702)
>> ... 7 more
>>
>>
>>
>>
>>
>> -----Original Message-----
>> From: Chesnay Schepler [mailto:chesnay@apache.org]
>> Sent: Wednesday, July 04, 2018 5:01 AM
>> To: dev@flink.apache.org; Ted Yu <yu...@gmail.com>
>> Subject: Re: Job jumbo jar can't be extracted on Windows
>>
>> Can you provide us with instructions for reproducing this error?
>> Please also include the versions for the jdk/tools/build-system etc.
>>
>> I frequently submit jobs on windows and haven't ran into this issue.
>>
>> On 03.07.2018 20:05, Ted Yu wrote:
>>> Looking at javadoc of separatorChar :
>>>
>>> * The system-dependent default name-separator character. This
>>> field is
>>>
>>> * initialized to contain the first character of the value of the
>>> system
>>>
>>> * property <code>file.separator</code>. On UNIX systems the
>>> value of this
>>>
>>> * field is <code>'/'</code>; on Microsoft Windows systems it is
>>> <code> '\\'</code>.
>>>
>>> Looks like the code you quoted should get the correct separator.
>>>
>>> On Tue, Jul 3, 2018 at 10:43 AM, NEKRASSOV, ALEXEI <an...@att.com>
>>> wrote:
>>>
>>>> When I try to "flink run" a job jar that includes dependent jar's -
>>>> it fails on Windows with the following Exception:
>>>>
>>>> org.apache.flink.client.program.ProgramInvocationException: Unknown
>>>> I/O error while extracting contained jar files.
>>>> at org.apache.flink.client.program.PackagedProgram.
>>>> extractContainedLibraries(PackagedProgram.java:752)
>>>> at org.apache.flink.client.program.PackagedProgram.<init>
>>>> (PackagedProgram.java:194)
>>>> at org.apache.flink.client.cli.CliFrontend.buildProgram(
>>>> CliFrontend.java:833)
>>>> at org.apache.flink.client.cli.CliFrontend.run(CliFrontend.
>>>> java:201)
>>>> at org.apache.flink.client.cli.CliFrontend.parseParameters(
>>>> CliFrontend.java:1020)
>>>> at org.apache.flink.client.cli.CliFrontend.lambda$main$9(
>>>> CliFrontend.java:1096)
>>>> at org.apache.flink.runtime.security.NoOpSecurityContext.
>>>> runSecured(NoOpSecurityContext.java:30)
>>>> at org.apache.flink.client.cli.CliFrontend.main(CliFrontend.
>>>> java:1096)
>>>> Caused by:
>>>> org.apache.flink.client.program.ProgramInvocationException: An I/O
>>>> error occurred while creating temporary file to extract nested
>>>> library 'lib/flink-connector-kafka-0.10_2.11-1.4.1.jar'.
>>>> at org.apache.flink.client.program.PackagedProgram.
>>>> extractContainedLibraries(PackagedProgram.java:708)
>>>> ... 7 more
>>>> Caused by: java.io.IOException: Unable to create temporary file,
>>>> C:\Users\XXX\AppData\Local\Temp\1863208689_4625824260650653022lib\fli
>>>> nk-
>>>> connector-kafka-0.10_2.11-1.4.1.jar
>>>> at java.io.File$TempDirectory.generateFile(Unknown Source)
>>>> at java.io.File.createTempFile(Unknown Source)
>>>> at java.io.File.createTempFile(Unknown Source)
>>>> at org.apache.flink.client.program.PackagedProgram.
>>>> extractContainedLibraries(PackagedProgram.java:702)
>>>> ... 7 more
>>>>
>>>> I think we have a problem in PackagedProgram.java - it fails to
>>>> extract contained jar's on Windows, because on Windows
>>>> File.separatorChar is '\', but JarEntry.getName() returns '/' as
>>>> file separator.
>>>> Line 699 of
>>>> https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_apach
>>>> e_flink_blob_master_flink-2D&d=DwICaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=aQRK
>>>> g6d5fsz42zXkyiSdqg&m=Ir3LtsfzkU6HmnOt_cRkQiJjhlg1VdbQiEJdel2K3hQ&s=X4
>>>> KjnHsf9m-xy_zryMeMYSbrY-pXplleKkhHEJpO8VQ&e=
>>>> clients/src/main/java/org/apache/flink/client/program/PackagedProgram
>>>> .java
>>>> is no-op on Windows.
>>>>
>>>> I think we need to replace line 699 with
>>>> name = name.replace('/', '_'); as I don't see a way to
>>>> have JarEntry use platform-specific file separator, and ZIP spec
>>>> dictates the use of forward-slashes anyway.
>>>> Line 680 already uses hard-coded '/'.
>>>>
>>>>
>>>> Alex Nekrassov
>>>>
>
>
Re: Job jumbo jar can't be extracted on Windows
Posted by Chesnay Schepler <ch...@apache.org>.
could reproduce it, sounds about right that we should always replace "/".
On 04.07.2018 16:14, NEKRASSOV, ALEXEI wrote:
> To demonstrate this problem:
>
> 1. Get a jar that includes other jar's. For example, download http://central.maven.org/maven2/org/eclipse/platform/org.eclipse.ant.ui/3.6.201/org.eclipse.ant.ui-3.6.201.jar
> 2. Put downloaded jar in c:\usr\tmp\
> 3. In cmd
> (note that we attempt to create 1751416743_6922010711856647205lib\antrunner.jar instead of 1751416743_6922010711856647205lib_antrunner.jar)
>
> c:\Program Files\flink-1.5.0\bin>flink -v
> Version: 1.5.0, Commit ID: c61b108
>
> c:\Program Files\flink-1.5.0\bin>flink run -c WhoCares c:\usr\tmp\org.eclipse.ant.ui-3.6.201.jar
>
> ------------------------------------------------------------
> The program finished with the following exception:
>
> org.apache.flink.client.program.ProgramInvocationException: Unknown I/O error while extracting contained jar files.
> at org.apache.flink.client.program.PackagedProgram.extractContainedLibraries(PackagedProgram.java:752)
> at org.apache.flink.client.program.PackagedProgram.<init>(PackagedProgram.java:194)
> at org.apache.flink.client.cli.CliFrontend.buildProgram(CliFrontend.java:833)
> at org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:201)
> at org.apache.flink.client.cli.CliFrontend.parseParameters(CliFrontend.java:1020)
> at org.apache.flink.client.cli.CliFrontend.lambda$main$9(CliFrontend.java:1096)
> at org.apache.flink.runtime.security.NoOpSecurityContext.runSecured(NoOpSecurityContext.java:30)
> at org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:1096)
> Caused by: org.apache.flink.client.program.ProgramInvocationException: An I/O error occurred while creating temporary file to extract nested library 'lib/antrunner.jar'.
> at org.apache.flink.client.program.PackagedProgram.extractContainedLibraries(PackagedProgram.java:708)
> ... 7 more
> Caused by: java.io.IOException: Unable to create temporary file, C:\Users\XXX\AppData\Local\Temp\1751416743_6922010711856647205lib\antrunner.jar
> at java.io.File$TempDirectory.generateFile(Unknown Source)
> at java.io.File.createTempFile(Unknown Source)
> at java.io.File.createTempFile(Unknown Source)
> at org.apache.flink.client.program.PackagedProgram.extractContainedLibraries(PackagedProgram.java:702)
> ... 7 more
>
>
>
>
>
> -----Original Message-----
> From: Chesnay Schepler [mailto:chesnay@apache.org]
> Sent: Wednesday, July 04, 2018 5:01 AM
> To: dev@flink.apache.org; Ted Yu <yu...@gmail.com>
> Subject: Re: Job jumbo jar can't be extracted on Windows
>
> Can you provide us with instructions for reproducing this error? Please also include the versions for the jdk/tools/build-system etc.
>
> I frequently submit jobs on windows and haven't ran into this issue.
>
> On 03.07.2018 20:05, Ted Yu wrote:
>> Looking at javadoc of separatorChar :
>>
>> * The system-dependent default name-separator character. This
>> field is
>>
>> * initialized to contain the first character of the value of the
>> system
>>
>> * property <code>file.separator</code>. On UNIX systems the
>> value of this
>>
>> * field is <code>'/'</code>; on Microsoft Windows systems it is
>> <code> '\\'</code>.
>>
>> Looks like the code you quoted should get the correct separator.
>>
>> On Tue, Jul 3, 2018 at 10:43 AM, NEKRASSOV, ALEXEI <an...@att.com> wrote:
>>
>>> When I try to "flink run" a job jar that includes dependent jar's -
>>> it fails on Windows with the following Exception:
>>>
>>> org.apache.flink.client.program.ProgramInvocationException: Unknown
>>> I/O error while extracting contained jar files.
>>> at org.apache.flink.client.program.PackagedProgram.
>>> extractContainedLibraries(PackagedProgram.java:752)
>>> at org.apache.flink.client.program.PackagedProgram.<init>
>>> (PackagedProgram.java:194)
>>> at org.apache.flink.client.cli.CliFrontend.buildProgram(
>>> CliFrontend.java:833)
>>> at org.apache.flink.client.cli.CliFrontend.run(CliFrontend.
>>> java:201)
>>> at org.apache.flink.client.cli.CliFrontend.parseParameters(
>>> CliFrontend.java:1020)
>>> at org.apache.flink.client.cli.CliFrontend.lambda$main$9(
>>> CliFrontend.java:1096)
>>> at org.apache.flink.runtime.security.NoOpSecurityContext.
>>> runSecured(NoOpSecurityContext.java:30)
>>> at org.apache.flink.client.cli.CliFrontend.main(CliFrontend.
>>> java:1096)
>>> Caused by:
>>> org.apache.flink.client.program.ProgramInvocationException: An I/O
>>> error occurred while creating temporary file to extract nested library 'lib/flink-connector-kafka-0.10_2.11-1.4.1.jar'.
>>> at org.apache.flink.client.program.PackagedProgram.
>>> extractContainedLibraries(PackagedProgram.java:708)
>>> ... 7 more
>>> Caused by: java.io.IOException: Unable to create temporary file,
>>> C:\Users\XXX\AppData\Local\Temp\1863208689_4625824260650653022lib\fli
>>> nk-
>>> connector-kafka-0.10_2.11-1.4.1.jar
>>> at java.io.File$TempDirectory.generateFile(Unknown Source)
>>> at java.io.File.createTempFile(Unknown Source)
>>> at java.io.File.createTempFile(Unknown Source)
>>> at org.apache.flink.client.program.PackagedProgram.
>>> extractContainedLibraries(PackagedProgram.java:702)
>>> ... 7 more
>>>
>>> I think we have a problem in PackagedProgram.java - it fails to
>>> extract contained jar's on Windows, because on Windows
>>> File.separatorChar is '\', but JarEntry.getName() returns '/' as file separator.
>>> Line 699 of
>>> https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_apach
>>> e_flink_blob_master_flink-2D&d=DwICaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=aQRK
>>> g6d5fsz42zXkyiSdqg&m=Ir3LtsfzkU6HmnOt_cRkQiJjhlg1VdbQiEJdel2K3hQ&s=X4
>>> KjnHsf9m-xy_zryMeMYSbrY-pXplleKkhHEJpO8VQ&e=
>>> clients/src/main/java/org/apache/flink/client/program/PackagedProgram
>>> .java
>>> is no-op on Windows.
>>>
>>> I think we need to replace line 699 with
>>> name = name.replace('/', '_'); as I don't see a way to
>>> have JarEntry use platform-specific file separator, and ZIP spec
>>> dictates the use of forward-slashes anyway.
>>> Line 680 already uses hard-coded '/'.
>>>
>>>
>>> Alex Nekrassov
>>>
RE: Job jumbo jar can't be extracted on Windows
Posted by "NEKRASSOV, ALEXEI" <an...@att.com>.
To demonstrate this problem:
1. Get a jar that includes other jar's. For example, download http://central.maven.org/maven2/org/eclipse/platform/org.eclipse.ant.ui/3.6.201/org.eclipse.ant.ui-3.6.201.jar
2. Put downloaded jar in c:\usr\tmp\
3. In cmd
(note that we attempt to create 1751416743_6922010711856647205lib\antrunner.jar instead of 1751416743_6922010711856647205lib_antrunner.jar)
c:\Program Files\flink-1.5.0\bin>flink -v
Version: 1.5.0, Commit ID: c61b108
c:\Program Files\flink-1.5.0\bin>flink run -c WhoCares c:\usr\tmp\org.eclipse.ant.ui-3.6.201.jar
------------------------------------------------------------
The program finished with the following exception:
org.apache.flink.client.program.ProgramInvocationException: Unknown I/O error while extracting contained jar files.
at org.apache.flink.client.program.PackagedProgram.extractContainedLibraries(PackagedProgram.java:752)
at org.apache.flink.client.program.PackagedProgram.<init>(PackagedProgram.java:194)
at org.apache.flink.client.cli.CliFrontend.buildProgram(CliFrontend.java:833)
at org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:201)
at org.apache.flink.client.cli.CliFrontend.parseParameters(CliFrontend.java:1020)
at org.apache.flink.client.cli.CliFrontend.lambda$main$9(CliFrontend.java:1096)
at org.apache.flink.runtime.security.NoOpSecurityContext.runSecured(NoOpSecurityContext.java:30)
at org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:1096)
Caused by: org.apache.flink.client.program.ProgramInvocationException: An I/O error occurred while creating temporary file to extract nested library 'lib/antrunner.jar'.
at org.apache.flink.client.program.PackagedProgram.extractContainedLibraries(PackagedProgram.java:708)
... 7 more
Caused by: java.io.IOException: Unable to create temporary file, C:\Users\XXX\AppData\Local\Temp\1751416743_6922010711856647205lib\antrunner.jar
at java.io.File$TempDirectory.generateFile(Unknown Source)
at java.io.File.createTempFile(Unknown Source)
at java.io.File.createTempFile(Unknown Source)
at org.apache.flink.client.program.PackagedProgram.extractContainedLibraries(PackagedProgram.java:702)
... 7 more
-----Original Message-----
From: Chesnay Schepler [mailto:chesnay@apache.org]
Sent: Wednesday, July 04, 2018 5:01 AM
To: dev@flink.apache.org; Ted Yu <yu...@gmail.com>
Subject: Re: Job jumbo jar can't be extracted on Windows
Can you provide us with instructions for reproducing this error? Please also include the versions for the jdk/tools/build-system etc.
I frequently submit jobs on windows and haven't ran into this issue.
On 03.07.2018 20:05, Ted Yu wrote:
> Looking at javadoc of separatorChar :
>
> * The system-dependent default name-separator character. This
> field is
>
> * initialized to contain the first character of the value of the
> system
>
> * property <code>file.separator</code>. On UNIX systems the
> value of this
>
> * field is <code>'/'</code>; on Microsoft Windows systems it is
> <code> '\\'</code>.
>
> Looks like the code you quoted should get the correct separator.
>
> On Tue, Jul 3, 2018 at 10:43 AM, NEKRASSOV, ALEXEI <an...@att.com> wrote:
>
>> When I try to "flink run" a job jar that includes dependent jar's -
>> it fails on Windows with the following Exception:
>>
>> org.apache.flink.client.program.ProgramInvocationException: Unknown
>> I/O error while extracting contained jar files.
>> at org.apache.flink.client.program.PackagedProgram.
>> extractContainedLibraries(PackagedProgram.java:752)
>> at org.apache.flink.client.program.PackagedProgram.<init>
>> (PackagedProgram.java:194)
>> at org.apache.flink.client.cli.CliFrontend.buildProgram(
>> CliFrontend.java:833)
>> at org.apache.flink.client.cli.CliFrontend.run(CliFrontend.
>> java:201)
>> at org.apache.flink.client.cli.CliFrontend.parseParameters(
>> CliFrontend.java:1020)
>> at org.apache.flink.client.cli.CliFrontend.lambda$main$9(
>> CliFrontend.java:1096)
>> at org.apache.flink.runtime.security.NoOpSecurityContext.
>> runSecured(NoOpSecurityContext.java:30)
>> at org.apache.flink.client.cli.CliFrontend.main(CliFrontend.
>> java:1096)
>> Caused by:
>> org.apache.flink.client.program.ProgramInvocationException: An I/O
>> error occurred while creating temporary file to extract nested library 'lib/flink-connector-kafka-0.10_2.11-1.4.1.jar'.
>> at org.apache.flink.client.program.PackagedProgram.
>> extractContainedLibraries(PackagedProgram.java:708)
>> ... 7 more
>> Caused by: java.io.IOException: Unable to create temporary file,
>> C:\Users\XXX\AppData\Local\Temp\1863208689_4625824260650653022lib\fli
>> nk-
>> connector-kafka-0.10_2.11-1.4.1.jar
>> at java.io.File$TempDirectory.generateFile(Unknown Source)
>> at java.io.File.createTempFile(Unknown Source)
>> at java.io.File.createTempFile(Unknown Source)
>> at org.apache.flink.client.program.PackagedProgram.
>> extractContainedLibraries(PackagedProgram.java:702)
>> ... 7 more
>>
>> I think we have a problem in PackagedProgram.java - it fails to
>> extract contained jar's on Windows, because on Windows
>> File.separatorChar is '\', but JarEntry.getName() returns '/' as file separator.
>> Line 699 of
>> https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_apach
>> e_flink_blob_master_flink-2D&d=DwICaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=aQRK
>> g6d5fsz42zXkyiSdqg&m=Ir3LtsfzkU6HmnOt_cRkQiJjhlg1VdbQiEJdel2K3hQ&s=X4
>> KjnHsf9m-xy_zryMeMYSbrY-pXplleKkhHEJpO8VQ&e=
>> clients/src/main/java/org/apache/flink/client/program/PackagedProgram
>> .java
>> is no-op on Windows.
>>
>> I think we need to replace line 699 with
>> name = name.replace('/', '_'); as I don't see a way to
>> have JarEntry use platform-specific file separator, and ZIP spec
>> dictates the use of forward-slashes anyway.
>> Line 680 already uses hard-coded '/'.
>>
>>
>> Alex Nekrassov
>>
Re: Job jumbo jar can't be extracted on Windows
Posted by Chesnay Schepler <ch...@apache.org>.
Can you provide us with instructions for reproducing this error? Please
also include the versions for the jdk/tools/build-system etc.
I frequently submit jobs on windows and haven't ran into this issue.
On 03.07.2018 20:05, Ted Yu wrote:
> Looking at javadoc of separatorChar :
>
> * The system-dependent default name-separator character. This field is
>
> * initialized to contain the first character of the value of the system
>
> * property <code>file.separator</code>. On UNIX systems the value of
> this
>
> * field is <code>'/'</code>; on Microsoft Windows systems it is <code>
> '\\'</code>.
>
> Looks like the code you quoted should get the correct separator.
>
> On Tue, Jul 3, 2018 at 10:43 AM, NEKRASSOV, ALEXEI <an...@att.com> wrote:
>
>> When I try to "flink run" a job jar that includes dependent jar's - it
>> fails on Windows with the following Exception:
>>
>> org.apache.flink.client.program.ProgramInvocationException: Unknown I/O
>> error while extracting contained jar files.
>> at org.apache.flink.client.program.PackagedProgram.
>> extractContainedLibraries(PackagedProgram.java:752)
>> at org.apache.flink.client.program.PackagedProgram.<init>
>> (PackagedProgram.java:194)
>> at org.apache.flink.client.cli.CliFrontend.buildProgram(
>> CliFrontend.java:833)
>> at org.apache.flink.client.cli.CliFrontend.run(CliFrontend.
>> java:201)
>> at org.apache.flink.client.cli.CliFrontend.parseParameters(
>> CliFrontend.java:1020)
>> at org.apache.flink.client.cli.CliFrontend.lambda$main$9(
>> CliFrontend.java:1096)
>> at org.apache.flink.runtime.security.NoOpSecurityContext.
>> runSecured(NoOpSecurityContext.java:30)
>> at org.apache.flink.client.cli.CliFrontend.main(CliFrontend.
>> java:1096)
>> Caused by: org.apache.flink.client.program.ProgramInvocationException: An
>> I/O error occurred while creating temporary file to extract nested library
>> 'lib/flink-connector-kafka-0.10_2.11-1.4.1.jar'.
>> at org.apache.flink.client.program.PackagedProgram.
>> extractContainedLibraries(PackagedProgram.java:708)
>> ... 7 more
>> Caused by: java.io.IOException: Unable to create temporary file,
>> C:\Users\XXX\AppData\Local\Temp\1863208689_4625824260650653022lib\flink-
>> connector-kafka-0.10_2.11-1.4.1.jar
>> at java.io.File$TempDirectory.generateFile(Unknown Source)
>> at java.io.File.createTempFile(Unknown Source)
>> at java.io.File.createTempFile(Unknown Source)
>> at org.apache.flink.client.program.PackagedProgram.
>> extractContainedLibraries(PackagedProgram.java:702)
>> ... 7 more
>>
>> I think we have a problem in PackagedProgram.java - it fails to extract
>> contained jar's on Windows, because on Windows File.separatorChar is '\',
>> but JarEntry.getName() returns '/' as file separator.
>> Line 699 of https://github.com/apache/flink/blob/master/flink-
>> clients/src/main/java/org/apache/flink/client/program/PackagedProgram.java
>> is no-op on Windows.
>>
>> I think we need to replace line 699 with
>> name = name.replace('/', '_');
>> as I don't see a way to have JarEntry use platform-specific file
>> separator, and ZIP spec dictates the use of forward-slashes anyway.
>> Line 680 already uses hard-coded '/'.
>>
>>
>> Alex Nekrassov
>>
Re: Job jumbo jar can't be extracted on Windows
Posted by Ted Yu <yu...@gmail.com>.
Looking at javadoc of separatorChar :
* The system-dependent default name-separator character. This field is
* initialized to contain the first character of the value of the system
* property <code>file.separator</code>. On UNIX systems the value of
this
* field is <code>'/'</code>; on Microsoft Windows systems it is <code>
'\\'</code>.
Looks like the code you quoted should get the correct separator.
On Tue, Jul 3, 2018 at 10:43 AM, NEKRASSOV, ALEXEI <an...@att.com> wrote:
> When I try to "flink run" a job jar that includes dependent jar's - it
> fails on Windows with the following Exception:
>
> org.apache.flink.client.program.ProgramInvocationException: Unknown I/O
> error while extracting contained jar files.
> at org.apache.flink.client.program.PackagedProgram.
> extractContainedLibraries(PackagedProgram.java:752)
> at org.apache.flink.client.program.PackagedProgram.<init>
> (PackagedProgram.java:194)
> at org.apache.flink.client.cli.CliFrontend.buildProgram(
> CliFrontend.java:833)
> at org.apache.flink.client.cli.CliFrontend.run(CliFrontend.
> java:201)
> at org.apache.flink.client.cli.CliFrontend.parseParameters(
> CliFrontend.java:1020)
> at org.apache.flink.client.cli.CliFrontend.lambda$main$9(
> CliFrontend.java:1096)
> at org.apache.flink.runtime.security.NoOpSecurityContext.
> runSecured(NoOpSecurityContext.java:30)
> at org.apache.flink.client.cli.CliFrontend.main(CliFrontend.
> java:1096)
> Caused by: org.apache.flink.client.program.ProgramInvocationException: An
> I/O error occurred while creating temporary file to extract nested library
> 'lib/flink-connector-kafka-0.10_2.11-1.4.1.jar'.
> at org.apache.flink.client.program.PackagedProgram.
> extractContainedLibraries(PackagedProgram.java:708)
> ... 7 more
> Caused by: java.io.IOException: Unable to create temporary file,
> C:\Users\XXX\AppData\Local\Temp\1863208689_4625824260650653022lib\flink-
> connector-kafka-0.10_2.11-1.4.1.jar
> at java.io.File$TempDirectory.generateFile(Unknown Source)
> at java.io.File.createTempFile(Unknown Source)
> at java.io.File.createTempFile(Unknown Source)
> at org.apache.flink.client.program.PackagedProgram.
> extractContainedLibraries(PackagedProgram.java:702)
> ... 7 more
>
> I think we have a problem in PackagedProgram.java - it fails to extract
> contained jar's on Windows, because on Windows File.separatorChar is '\',
> but JarEntry.getName() returns '/' as file separator.
> Line 699 of https://github.com/apache/flink/blob/master/flink-
> clients/src/main/java/org/apache/flink/client/program/PackagedProgram.java
> is no-op on Windows.
>
> I think we need to replace line 699 with
> name = name.replace('/', '_');
> as I don't see a way to have JarEntry use platform-specific file
> separator, and ZIP spec dictates the use of forward-slashes anyway.
> Line 680 already uses hard-coded '/'.
>
>
> Alex Nekrassov
>