You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "junzhong qin (Jira)" <ji...@apache.org> on 2023/08/08 11:15:00 UTC
[jira] [Updated] (FLINK-32809) YarnClusterDescriptor.isarchiveonlyincludedinShipArchiveFiles dose not work as expected
[ https://issues.apache.org/jira/browse/FLINK-32809?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
junzhong qin updated FLINK-32809:
---------------------------------
Description:
YarnClusterDescriptor.isarchiveonlyincludedinShipArchiveFiles(List<File> shipFiles) check wether the shipFiles are all archive files, but it dose not work as expected.
{code:java}
public static boolean isArchiveOnlyIncludedInShipArchiveFiles(List<File> shipFiles) {
return shipFiles.stream()
.filter(File::isFile)
.map(File::getName)
.map(String::toLowerCase)
.allMatch(
name ->
name.endsWith(".tar.gz")
|| name.endsWith(".tar")
|| name.endsWith(".tgz")
|| name.endsWith(".dst")
|| name.endsWith(".jar")
|| name.endsWith(".zip"));
} {code}
When we pass a directory and an archive file it should return false but it returns true.
{code:java}
// dir1 is a directory and archive.zip is an archive file
List<File> files = Arrays.asList(new File("/tmp/dir1"), new File("/tmp/archive.zip"));
boolean result = isArchiveOnlyIncludedInShipArchiveFiles(files);
System.out.println(result); // Print true but is should print false{code}
If flink user want to add directory as ship file, they should use configuration: yarn.ship-files .
was:
YarnClusterDescriptor.isarchiveonlyincludedinShipArchiveFiles(List<File> shipFiles) check wether the shipFiles are all archive files, but it dose not work as expected.
{code:java}
public static boolean isArchiveOnlyIncludedInShipArchiveFiles(List<File> shipFiles) {
return shipFiles.stream()
.filter(File::isFile)
.map(File::getName)
.map(String::toLowerCase)
.allMatch(
name ->
name.endsWith(".tar.gz")
|| name.endsWith(".tar")
|| name.endsWith(".tgz")
|| name.endsWith(".dst")
|| name.endsWith(".jar")
|| name.endsWith(".zip"));
} {code}
When we pass a directory and an archive file it should return false but it returns true.
{code:java}
// dir1 is a directory and archive.zip is an archive file
List<File> files = Arrays.asList(new File("/tmp/dir1"), new File("/tmp/archive.zip"));
boolean result = isArchiveOnlyIncludedInShipArchiveFiles(files);
System.out.println(result); // Print true but is should print false{code}
If flink user want to add directory as ship file, they should use configuration: yarn.ship-files .
> YarnClusterDescriptor.isarchiveonlyincludedinShipArchiveFiles dose not work as expected
> ---------------------------------------------------------------------------------------
>
> Key: FLINK-32809
> URL: https://issues.apache.org/jira/browse/FLINK-32809
> Project: Flink
> Issue Type: Bug
> Components: Deployment / YARN
> Affects Versions: 1.16.2, 1.18.0, 1.17.1
> Reporter: junzhong qin
> Priority: Major
>
> YarnClusterDescriptor.isarchiveonlyincludedinShipArchiveFiles(List<File> shipFiles) check wether the shipFiles are all archive files, but it dose not work as expected.
> {code:java}
> public static boolean isArchiveOnlyIncludedInShipArchiveFiles(List<File> shipFiles) {
> return shipFiles.stream()
> .filter(File::isFile)
> .map(File::getName)
> .map(String::toLowerCase)
> .allMatch(
> name ->
> name.endsWith(".tar.gz")
> || name.endsWith(".tar")
> || name.endsWith(".tgz")
> || name.endsWith(".dst")
> || name.endsWith(".jar")
> || name.endsWith(".zip"));
> } {code}
> When we pass a directory and an archive file it should return false but it returns true.
> {code:java}
> // dir1 is a directory and archive.zip is an archive file
> List<File> files = Arrays.asList(new File("/tmp/dir1"), new File("/tmp/archive.zip"));
> boolean result = isArchiveOnlyIncludedInShipArchiveFiles(files);
> System.out.println(result); // Print true but is should print false{code}
> If flink user want to add directory as ship file, they should use configuration: yarn.ship-files .
--
This message was sent by Atlassian Jira
(v8.20.10#820010)