You are viewing a plain text version of this content. The canonical link for it is here.

Posted to reviews@spark.apache.org by HyukjinKwon <gi...@git.apache.org> on 2016/03/18 01:21:26 UTC

[GitHub] spark pull request: [MINOR][SQL] Use Hadoop 2.0 default value for ...

GitHub user HyukjinKwon opened a pull request:

    https://github.com/apache/spark/pull/11806

    [MINOR][SQL] Use Hadoop 2.0 default value for compression in data sources.

    ## What changes were proposed in this pull request?
    
    Currently, JSON, TEXT and CSV data sources use `CompressionCodecs` class to set compression configurations via `option("compress", "codec")`.
    
    I made this uses Hadoop 1.x default value (block level compression). However, the default value in Hadoop 2.x is record level compression as described in [mapred-site.xml](https://hadoop.apache.org/docs/r2.7.1/hadoop-mapreduce-client/hadoop-mapreduce-client-core/mapred-default.xml).
    
    Since it drops Hadoop 1.x, it will make sense to use Hadoop 2.x default values.
    
    ## How was this patch tested?
    
    Via `./dev/run_tests` and unit tests.
    


You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/HyukjinKwon/spark minor-compression

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/spark/pull/11806.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #11806
    
----
commit 80749a84d60ce7c3c9bbcd6af72a40d7f90c80c3
Author: hyukjinkwon <gu...@gmail.com>
Date:   2016-03-18T00:13:32Z

    Use Hadoop 2.0 default value for compression

----


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark pull request: [SPARK-13997][SQL] Use Hadoop 2.0 default valu...

Posted by HyukjinKwon <gi...@git.apache.org>.

Github user HyukjinKwon commented on the pull request:

    https://github.com/apache/spark/pull/11806#issuecomment-198201760
  
    I see.. Should I maybe close this?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark pull request: [SPARK-13997][SQL] Use Hadoop 2.0 default valu...

Posted by HyukjinKwon <gi...@git.apache.org>.

Github user HyukjinKwon commented on the pull request:

    https://github.com/apache/spark/pull/11806#issuecomment-198206078
  
    Actually, I did not understand why the overhead of compression at record (I mean a row in Spark, a key-value in Hadoop output format) level would be very high. I think it is slightly high and the compression ratio is a bit lower but allows random accessing at record level.
    
    Maybe I have the lack of knowledge and experience. I would really appreciate if you help me.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark pull request: [SPARK-13997][SQL] Use Hadoop 2.0 default valu...

Posted by HyukjinKwon <gi...@git.apache.org>.

Github user HyukjinKwon commented on the pull request:

    https://github.com/apache/spark/pull/11806#issuecomment-198200770
  
    @rxin According to [Hadoop Definitive Guide 3th edition](https://www.safaribooksonline.com/library/view/hadoop-the-definitive/9781449328917/ch04.html), it looks it is right configurations to the unit of compression (record or block).
    



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark pull request: [SPARK-13997][SQL] Use Hadoop 2.0 default valu...

Posted by steveloughran <gi...@git.apache.org>.

Github user steveloughran commented on the pull request:

https://github.com/apache/spark/pull/11806#issuecomment-198322447

Summary: use an optimised storage format and dataframes, worry about compression afterwards

1. you need to use a compression code that lets you seek into a bit of the file before you read; this is what's needed for parallel access to data in different blocks. I don't remember which codecs are best for this or the specific switch to turn it on. LZO? Snappy?
1. Compression performance? With native libs it decompression can be fast; snappy has a good reputation there. Without the native libs things will be way slow. With the right settings, the performance costs of decompression are less than the wait times for the uncompressed data (that's on HDD; SSD changes the rules again & someone needs to do the experiments.)
1. Except for the specific case of the ingest phase, start by converting your data into an optimised format: Parquet, ORC, which does things like columnar storage and compression, This massively minimises IO when working with a few columns (provided the format/API supports predicate pushdowns...use the dataframes API & not RDDs directly). I *assume* you can compress these too, though with compression already happening on columns, gains would be less.
1. Ingress is a good streaming usecase, if you haven't played with it already ...

---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark pull request: [SPARK-13997][SQL] Use Hadoop 2.0 default valu...

Posted by HyukjinKwon <gi...@git.apache.org>.

Github user HyukjinKwon commented on the pull request:

    https://github.com/apache/spark/pull/11806#issuecomment-198326625
  
    Closing this.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org