You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Hyukjin Kwon (Jira)" <ji...@apache.org> on 2021/02/05 00:39:00 UTC
[jira] [Resolved] (SPARK-34339) Expose the number of truncated
paths in Utils.buildLocationMetadata()
[ https://issues.apache.org/jira/browse/SPARK-34339?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hyukjin Kwon resolved SPARK-34339.
----------------------------------
Fix Version/s: 3.2.0
Resolution: Fixed
Issue resolved by pull request 31464
[https://github.com/apache/spark/pull/31464]
> Expose the number of truncated paths in Utils.buildLocationMetadata()
> ---------------------------------------------------------------------
>
> Key: SPARK-34339
> URL: https://issues.apache.org/jira/browse/SPARK-34339
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 3.2.0
> Reporter: Jungtaek Lim
> Assignee: Jungtaek Lim
> Priority: Major
> Fix For: 3.2.0
>
>
> SPARK-31793 introduces Utils.buildLocationMetadata() to reduce the length of location metadata. It effectively reduced the memory usage as only a few paths will be included (and it's controlled by threshold value), but there's no indication of the fact for truncation.
> If the first 2 of 5 paths are only fit to the threshold, Utils.buildLocationMetadata() shows the first 2 paths, but there's no mention that 3 paths are truncated. Even no mention that some paths have been truncated; it shows the same with the output just first 2 paths were presented. This could bring confusion.
> We should allow more space (like 10+ chars) to represent the fact N paths are truncated.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org