You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "Owen O'Malley (JIRA)" <ji...@apache.org> on 2007/05/17 00:49:16 UTC
[jira] Created: (HADOOP-1381) The distance between sync blocks in
SequenceFiles should be configurable rather than hard coded to 2000 bytes
The distance between sync blocks in SequenceFiles should be configurable rather than hard coded to 2000 bytes
-------------------------------------------------------------------------------------------------------------
Key: HADOOP-1381
URL: https://issues.apache.org/jira/browse/HADOOP-1381
Project: Hadoop
Issue Type: Improvement
Components: io
Reporter: Owen O'Malley
Fix For: 0.14.0
Currently SequenceFiles put in sync blocks every 2000 bytes. It would be much better if it was configurable with a much higher default (1mb or so?).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-1381) The distance between sync blocks in
SequenceFiles should be configurable rather than hard coded to 2000 bytes
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1381?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12496973 ]
Doug Cutting commented on HADOOP-1381:
--------------------------------------
This sounds like a time/space tradeoff. We currently have a 1% space overhead. You've proposed a 1% time overhead (with 1MB syncs and 100MB block size, splits would overlap the end of the block by 1%, forcing 1% non-local reads). Perhaps there's a sweet spot somewhere between? 100k?
> The distance between sync blocks in SequenceFiles should be configurable rather than hard coded to 2000 bytes
> -------------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-1381
> URL: https://issues.apache.org/jira/browse/HADOOP-1381
> Project: Hadoop
> Issue Type: Improvement
> Components: io
> Reporter: Owen O'Malley
> Fix For: 0.14.0
>
>
> Currently SequenceFiles put in sync blocks every 2000 bytes. It would be much better if it was configurable with a much higher default (1mb or so?).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-1381) The distance between sync blocks in
SequenceFiles should be configurable rather than hard coded to 2000 bytes
Posted by "Owen O'Malley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1381?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12496977 ]
Owen O'Malley commented on HADOOP-1381:
---------------------------------------
As long as it is configurable, 100k as the default would be fine.
> The distance between sync blocks in SequenceFiles should be configurable rather than hard coded to 2000 bytes
> -------------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-1381
> URL: https://issues.apache.org/jira/browse/HADOOP-1381
> Project: Hadoop
> Issue Type: Improvement
> Components: io
> Reporter: Owen O'Malley
> Fix For: 0.14.0
>
>
> Currently SequenceFiles put in sync blocks every 2000 bytes. It would be much better if it was configurable with a much higher default (1mb or so?).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-1381) The distance between sync blocks in
SequenceFiles should be configurable rather than hard coded to 2000 bytes
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1381?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12496429 ]
Doug Cutting commented on HADOOP-1381:
--------------------------------------
Why would this be better? The current design is to add them as frequently as possible without significantly impacting file size. This minimizes the amount of data that must be scanned when sync'ing. What would making it larger help?
> The distance between sync blocks in SequenceFiles should be configurable rather than hard coded to 2000 bytes
> -------------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-1381
> URL: https://issues.apache.org/jira/browse/HADOOP-1381
> Project: Hadoop
> Issue Type: Improvement
> Components: io
> Reporter: Owen O'Malley
> Fix For: 0.14.0
>
>
> Currently SequenceFiles put in sync blocks every 2000 bytes. It would be much better if it was configurable with a much higher default (1mb or so?).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-1381) The distance between sync blocks in
SequenceFiles should be configurable rather than hard coded to 2000 bytes
Posted by "Owen O'Malley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1381?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12496432 ]
Owen O'Malley commented on HADOOP-1381:
---------------------------------------
If your input splits are roughly 128MB or so, putting in a sync block every 2k is too often. By moving it up to 1MB, you can get good granularity and reduce the overhead by a factor of 500. That seems pretty significant .
> The distance between sync blocks in SequenceFiles should be configurable rather than hard coded to 2000 bytes
> -------------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-1381
> URL: https://issues.apache.org/jira/browse/HADOOP-1381
> Project: Hadoop
> Issue Type: Improvement
> Components: io
> Reporter: Owen O'Malley
> Fix For: 0.14.0
>
>
> Currently SequenceFiles put in sync blocks every 2000 bytes. It would be much better if it was configurable with a much higher default (1mb or so?).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-1381) The distance between sync blocks in
SequenceFiles should be configurable rather than hard coded to 2000 bytes
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1381?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12496435 ]
Doug Cutting commented on HADOOP-1381:
--------------------------------------
> reduce the overhead by a factor of 500
But if the overhead is already considerably less than 1%, is this really important? We want to keep it substantially smaller than 1% of what we ever expect the FileSystem block size to be. 1MB seems to be pushing that boundary.
> The distance between sync blocks in SequenceFiles should be configurable rather than hard coded to 2000 bytes
> -------------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-1381
> URL: https://issues.apache.org/jira/browse/HADOOP-1381
> Project: Hadoop
> Issue Type: Improvement
> Components: io
> Reporter: Owen O'Malley
> Fix For: 0.14.0
>
>
> Currently SequenceFiles put in sync blocks every 2000 bytes. It would be much better if it was configurable with a much higher default (1mb or so?).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-1381) The distance between sync blocks in
SequenceFiles should be configurable rather than hard coded to 2000 bytes
Posted by "Owen O'Malley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1381?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12496970 ]
Owen O'Malley commented on HADOOP-1381:
---------------------------------------
I should also note that this would handle the current special case of the map outputs that turn sync blocks off...
> The distance between sync blocks in SequenceFiles should be configurable rather than hard coded to 2000 bytes
> -------------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-1381
> URL: https://issues.apache.org/jira/browse/HADOOP-1381
> Project: Hadoop
> Issue Type: Improvement
> Components: io
> Reporter: Owen O'Malley
> Fix For: 0.14.0
>
>
> Currently SequenceFiles put in sync blocks every 2000 bytes. It would be much better if it was configurable with a much higher default (1mb or so?).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-1381) The distance between sync blocks in
SequenceFiles should be configurable rather than hard coded to 2000 bytes
Posted by "Owen O'Malley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1381?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12496968 ]
Owen O'Malley commented on HADOOP-1381:
---------------------------------------
The current setup forces all non-block compressed sequence files to accept a 1% overhead for sync bytes. That is a fair amount of overhead on 100tb... Block compressed sequence files do just fine with 1mb between sync blocks.
> The distance between sync blocks in SequenceFiles should be configurable rather than hard coded to 2000 bytes
> -------------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-1381
> URL: https://issues.apache.org/jira/browse/HADOOP-1381
> Project: Hadoop
> Issue Type: Improvement
> Components: io
> Reporter: Owen O'Malley
> Fix For: 0.14.0
>
>
> Currently SequenceFiles put in sync blocks every 2000 bytes. It would be much better if it was configurable with a much higher default (1mb or so?).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.