You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Nicholas Chammas (Jira)" <ji...@apache.org> on 2019/09/16 20:16:00 UTC
[jira] [Commented] (SPARK-29102) Read gzipped file into multiple
partitions without full gzip expansion on a single-node
[ https://issues.apache.org/jira/browse/SPARK-29102?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16930835#comment-16930835 ]
Nicholas Chammas commented on SPARK-29102:
------------------------------------------
cc [~cloud_fan] and [~hyukjin.kwon]: I noticed your work and comments on the PRs for SPARK-28366, so you may be interested in this issue.
Does this idea make sense? Does it seem feasible in theory at least?
> Read gzipped file into multiple partitions without full gzip expansion on a single-node
> ---------------------------------------------------------------------------------------
>
> Key: SPARK-29102
> URL: https://issues.apache.org/jira/browse/SPARK-29102
> Project: Spark
> Issue Type: Improvement
> Components: Input/Output
> Affects Versions: 2.4.4
> Reporter: Nicholas Chammas
> Priority: Minor
>
> Large gzipped files are a common stumbling block for new users (SPARK-5685, SPARK-28366) and an ongoing pain point for users who must process such files delivered from external parties who can't or won't break them up into smaller files or compress them using a splittable compression format like bzip2.
> To deal with large gzipped files today, users must either load them via a single task and then repartition the resulting RDD or DataFrame, or they must launch a preprocessing step outside of Spark to split up the file or recompress it using a splittable format. In either case, the user needs a single host capable of holding the entire decompressed file.
> Spark can potentially a) spare new users the confusion over why only one task is processing their gzipped data, and b) relieve new and experienced users alike from needing to maintain infrastructure capable of decompressing a large gzipped file on a single node, by directly loading gzipped files into multiple partitions across the cluster.
> The rough idea is to have tasks divide a given gzipped file into ranges and then have them all concurrently decompress the file, with each task throwing away the data leading up to the target range. (This kind of partial decompression is apparently [doable using standard Unix utilities|https://unix.stackexchange.com/a/415831/70630], so it should be doable in Spark too.)
> In this way multiple tasks can concurrently load a single gzipped file into multiple partitions. Even though every task will need to unpack the file from the beginning to the task's target range, and the stage will run no faster than what it would take with Spark's current gzip loading behavior, this nonetheless addresses the two problems called out above. Users no longer need to load and then repartition gzipped files, and their infrastructure does not need to decompress any large gzipped file on a single node.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org