You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@spark.apache.org by "Jungtaek Lim (Jira)" <ji...@apache.org> on 2021/02/03 05:02:00 UTC

[jira] [Created] (SPARK-34339) Expose the number of truncated paths in Utils.buildLocationMetadata()

Jungtaek Lim created SPARK-34339:
------------------------------------

             Summary: Expose the number of truncated paths in Utils.buildLocationMetadata()
                 Key: SPARK-34339
                 URL: https://issues.apache.org/jira/browse/SPARK-34339
             Project: Spark
          Issue Type: Improvement
          Components: SQL
    Affects Versions: 3.2.0
            Reporter: Jungtaek Lim


SPARK-31793 introduces Utils.buildLocationMetadata() to reduce the length of location metadata. It effectively reduced the memory usage as only a few paths will be included (and it's controlled by threshold value), but there's no indication of the fact for truncation.

If the first 2 of 5 paths are only fit to the threshold, Utils.buildLocationMetadata() shows the first 2 paths, but there's no mention that 3 paths are truncated. Even no mention that some paths have been truncated; it shows the same with the output just first 2 paths were presented. This could bring confusion.

We should allow more space (like 10+ chars) to represent the fact N paths are truncated.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org