[GitHub] [flink] eogramow commented on a change in pull request #12791: [FLINK-18362][FLINK-13838][yarn] Add yarn.ship-archives to support LocalResourceType.ARCHIVE

[GitBox <gi...@apache.org>]

eogramow commented on a change in pull request #12791:
URL: https://github.com/apache/flink/pull/12791#discussion_r451402452



##########
File path: flink-yarn/src/main/java/org/apache/flink/yarn/YarnClusterDescriptor.java
##########
@@ -267,6 +278,29 @@ public void addShipFiles(List<File> shipFiles) {
 		this.shipFiles.addAll(shipFiles);
 	}
 
+	/**
+	 * Adds the given files to the list of archives to ship.
+	 **
+	 * @param shipArchives archives to ship
+	 */
+	public void addShipArchives(List<File> shipArchives) {
+		checkArgument(isArchiveOnlyIncludedInShipArchiveFiles(shipArchives), "Non-archive files are included.");
+		this.shipArchives.addAll(shipArchives);
+	}
+
+	private static boolean isArchiveOnlyIncludedInShipArchiveFiles(List<File> shipFiles) {
+		return shipFiles.stream()
+			.filter(File::isFile)
+			.map(File::getName)
+			.map(String::toLowerCase)
+			.allMatch(name -> name.endsWith(".tar.gz") ||
+				name.endsWith(".tar") ||
+				name.endsWith(".tgz") ||
+				name.endsWith(".dst") ||
+				name.endsWith(".jar") ||

Review comment:
       @kl0u I just checked all file types that yarn can support.
   https://github.com/apache/hadoop/blob/8e4517f251e9f12917f2f457831664905015612f/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/FSDownload.java#L336-L343
   If jars were disallowed, the reason should be documented. but, I am not sure why it should be removed.  😅 




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org