You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "Andrzej Bialecki (JIRA)" <ji...@apache.org> on 2006/12/27 20:13:21 UTC

[jira] Created: (HADOOP-850) Add Writable implementations for variable-length integer types.

Add Writable implementations for variable-length integer types.
---------------------------------------------------------------

                 Key: HADOOP-850
                 URL: http://issues.apache.org/jira/browse/HADOOP-850
             Project: Hadoop
          Issue Type: Improvement
          Components: io
    Affects Versions: 0.10.0
            Reporter: Andrzej Bialecki 
         Assigned To: Andrzej Bialecki 
            Priority: Minor
             Fix For: 0.10.0


Currently Hadoop supports only three basic integer-like types: ByteWritable, IntWritable and LongWritable. They provide a fixed tradeoff between their value range and on-disk space consumption. But it is sometimes useful to be able to store integer values with broader allowed range, but less space consumption when possible.

This is especially useful when storing very long series of  values, combined with delta encoding.

Lucene already implements variable-length encoding for positive int and long. I propose to add similar Writable implementations, which use the same encoding methods.

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] Updated: (HADOOP-850) Add Writable implementations for variable-length integer types.

Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HADOOP-850?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Doug Cutting updated HADOOP-850:
--------------------------------

    Resolution: Fixed
        Status: Resolved  (was: Patch Available)

I just committed this.  Thanks, Andrzej!


> Add Writable implementations for variable-length integer types.
> ---------------------------------------------------------------
>
>                 Key: HADOOP-850
>                 URL: https://issues.apache.org/jira/browse/HADOOP-850
>             Project: Hadoop
>          Issue Type: Improvement
>          Components: io
>    Affects Versions: 0.10.0
>            Reporter: Andrzej Bialecki 
>         Assigned To: Andrzej Bialecki 
>            Priority: Minor
>             Fix For: 0.10.0
>
>         Attachments: variable-ints.patch
>
>
> Currently Hadoop supports only three basic integer-like types: ByteWritable, IntWritable and LongWritable. They provide a fixed tradeoff between their value range and on-disk space consumption. But it is sometimes useful to be able to store integer values with broader allowed range, but less space consumption when possible.
> This is especially useful when storing very long series of  values, combined with delta encoding.
> Lucene already implements variable-length encoding for positive int and long. I propose to add similar Writable implementations, which use the same encoding methods.

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] Updated: (HADOOP-850) Add Writable implementations for variable-length integer types.

Posted by "Andrzej Bialecki (JIRA)" <ji...@apache.org>.
     [ http://issues.apache.org/jira/browse/HADOOP-850?page=all ]

Andrzej Bialecki  updated HADOOP-850:
-------------------------------------

    Status: Open  (was: Patch Available)

> Add Writable implementations for variable-length integer types.
> ---------------------------------------------------------------
>
>                 Key: HADOOP-850
>                 URL: http://issues.apache.org/jira/browse/HADOOP-850
>             Project: Hadoop
>          Issue Type: Improvement
>          Components: io
>    Affects Versions: 0.10.0
>            Reporter: Andrzej Bialecki 
>         Assigned To: Andrzej Bialecki 
>            Priority: Minor
>             Fix For: 0.10.0
>
>         Attachments: variable-ints.patch, variable-ints.patch
>
>
> Currently Hadoop supports only three basic integer-like types: ByteWritable, IntWritable and LongWritable. They provide a fixed tradeoff between their value range and on-disk space consumption. But it is sometimes useful to be able to store integer values with broader allowed range, but less space consumption when possible.
> This is especially useful when storing very long series of  values, combined with delta encoding.
> Lucene already implements variable-length encoding for positive int and long. I propose to add similar Writable implementations, which use the same encoding methods.

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] Updated: (HADOOP-850) Add Writable implementations for variable-length integer types.

Posted by "Andrzej Bialecki (JIRA)" <ji...@apache.org>.
     [ http://issues.apache.org/jira/browse/HADOOP-850?page=all ]

Andrzej Bialecki  updated HADOOP-850:
-------------------------------------

    Attachment: variable-ints.patch

Indeed, I missed the implementation in WritableUtils, which is even better than Lucene's as it supports also negative numbers. Here's an updated patch.

> Add Writable implementations for variable-length integer types.
> ---------------------------------------------------------------
>
>                 Key: HADOOP-850
>                 URL: http://issues.apache.org/jira/browse/HADOOP-850
>             Project: Hadoop
>          Issue Type: Improvement
>          Components: io
>    Affects Versions: 0.10.0
>            Reporter: Andrzej Bialecki 
>         Assigned To: Andrzej Bialecki 
>            Priority: Minor
>             Fix For: 0.10.0
>
>         Attachments: variable-ints.patch
>
>
> Currently Hadoop supports only three basic integer-like types: ByteWritable, IntWritable and LongWritable. They provide a fixed tradeoff between their value range and on-disk space consumption. But it is sometimes useful to be able to store integer values with broader allowed range, but less space consumption when possible.
> This is especially useful when storing very long series of  values, combined with delta encoding.
> Lucene already implements variable-length encoding for positive int and long. I propose to add similar Writable implementations, which use the same encoding methods.

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] Updated: (HADOOP-850) Add Writable implementations for variable-length integer types.

Posted by "Andrzej Bialecki (JIRA)" <ji...@apache.org>.
     [ http://issues.apache.org/jira/browse/HADOOP-850?page=all ]

Andrzej Bialecki  updated HADOOP-850:
-------------------------------------

    Attachment: variable-ints.patch

Implementation of VIntWritable and VLongWritable. If there are no objections I'd like to commit them soon.

> Add Writable implementations for variable-length integer types.
> ---------------------------------------------------------------
>
>                 Key: HADOOP-850
>                 URL: http://issues.apache.org/jira/browse/HADOOP-850
>             Project: Hadoop
>          Issue Type: Improvement
>          Components: io
>    Affects Versions: 0.10.0
>            Reporter: Andrzej Bialecki 
>         Assigned To: Andrzej Bialecki 
>            Priority: Minor
>             Fix For: 0.10.0
>
>         Attachments: variable-ints.patch
>
>
> Currently Hadoop supports only three basic integer-like types: ByteWritable, IntWritable and LongWritable. They provide a fixed tradeoff between their value range and on-disk space consumption. But it is sometimes useful to be able to store integer values with broader allowed range, but less space consumption when possible.
> This is especially useful when storing very long series of  values, combined with delta encoding.
> Lucene already implements variable-length encoding for positive int and long. I propose to add similar Writable implementations, which use the same encoding methods.

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] Updated: (HADOOP-850) Add Writable implementations for variable-length integer types.

Posted by "Andrzej Bialecki (JIRA)" <ji...@apache.org>.
     [ http://issues.apache.org/jira/browse/HADOOP-850?page=all ]

Andrzej Bialecki  updated HADOOP-850:
-------------------------------------

    Status: Patch Available  (was: Open)

> Add Writable implementations for variable-length integer types.
> ---------------------------------------------------------------
>
>                 Key: HADOOP-850
>                 URL: http://issues.apache.org/jira/browse/HADOOP-850
>             Project: Hadoop
>          Issue Type: Improvement
>          Components: io
>    Affects Versions: 0.10.0
>            Reporter: Andrzej Bialecki 
>         Assigned To: Andrzej Bialecki 
>            Priority: Minor
>             Fix For: 0.10.0
>
>         Attachments: variable-ints.patch
>
>
> Currently Hadoop supports only three basic integer-like types: ByteWritable, IntWritable and LongWritable. They provide a fixed tradeoff between their value range and on-disk space consumption. But it is sometimes useful to be able to store integer values with broader allowed range, but less space consumption when possible.
> This is especially useful when storing very long series of  values, combined with delta encoding.
> Lucene already implements variable-length encoding for positive int and long. I propose to add similar Writable implementations, which use the same encoding methods.

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] Updated: (HADOOP-850) Add Writable implementations for variable-length integer types.

Posted by "Andrzej Bialecki (JIRA)" <ji...@apache.org>.
     [ http://issues.apache.org/jira/browse/HADOOP-850?page=all ]

Andrzej Bialecki  updated HADOOP-850:
-------------------------------------

    Attachment:     (was: variable-ints.patch)

> Add Writable implementations for variable-length integer types.
> ---------------------------------------------------------------
>
>                 Key: HADOOP-850
>                 URL: http://issues.apache.org/jira/browse/HADOOP-850
>             Project: Hadoop
>          Issue Type: Improvement
>          Components: io
>    Affects Versions: 0.10.0
>            Reporter: Andrzej Bialecki 
>         Assigned To: Andrzej Bialecki 
>            Priority: Minor
>             Fix For: 0.10.0
>
>         Attachments: variable-ints.patch
>
>
> Currently Hadoop supports only three basic integer-like types: ByteWritable, IntWritable and LongWritable. They provide a fixed tradeoff between their value range and on-disk space consumption. But it is sometimes useful to be able to store integer values with broader allowed range, but less space consumption when possible.
> This is especially useful when storing very long series of  values, combined with delta encoding.
> Lucene already implements variable-length encoding for positive int and long. I propose to add similar Writable implementations, which use the same encoding methods.

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] Commented: (HADOOP-850) Add Writable implementations for variable-length integer types.

Posted by "Owen O'Malley (JIRA)" <ji...@apache.org>.
    [ http://issues.apache.org/jira/browse/HADOOP-850?page=comments#action_12461326 ] 
            
Owen O'Malley commented on HADOOP-850:
--------------------------------------

Do we really need both VInt and VLong? They will serialize in exactly the same space, so all you are saving is in memory space. Other than that question/comment +1.

> Add Writable implementations for variable-length integer types.
> ---------------------------------------------------------------
>
>                 Key: HADOOP-850
>                 URL: http://issues.apache.org/jira/browse/HADOOP-850
>             Project: Hadoop
>          Issue Type: Improvement
>          Components: io
>    Affects Versions: 0.10.0
>            Reporter: Andrzej Bialecki 
>         Assigned To: Andrzej Bialecki 
>            Priority: Minor
>             Fix For: 0.10.0
>
>         Attachments: variable-ints.patch
>
>
> Currently Hadoop supports only three basic integer-like types: ByteWritable, IntWritable and LongWritable. They provide a fixed tradeoff between their value range and on-disk space consumption. But it is sometimes useful to be able to store integer values with broader allowed range, but less space consumption when possible.
> This is especially useful when storing very long series of  values, combined with delta encoding.
> Lucene already implements variable-length encoding for positive int and long. I propose to add similar Writable implementations, which use the same encoding methods.

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] Updated: (HADOOP-850) Add Writable implementations for variable-length integer types.

Posted by "Andrzej Bialecki (JIRA)" <ji...@apache.org>.
     [ http://issues.apache.org/jira/browse/HADOOP-850?page=all ]

Andrzej Bialecki  updated HADOOP-850:
-------------------------------------

    Attachment:     (was: variable-ints.patch)

> Add Writable implementations for variable-length integer types.
> ---------------------------------------------------------------
>
>                 Key: HADOOP-850
>                 URL: http://issues.apache.org/jira/browse/HADOOP-850
>             Project: Hadoop
>          Issue Type: Improvement
>          Components: io
>    Affects Versions: 0.10.0
>            Reporter: Andrzej Bialecki 
>         Assigned To: Andrzej Bialecki 
>            Priority: Minor
>             Fix For: 0.10.0
>
>
> Currently Hadoop supports only three basic integer-like types: ByteWritable, IntWritable and LongWritable. They provide a fixed tradeoff between their value range and on-disk space consumption. But it is sometimes useful to be able to store integer values with broader allowed range, but less space consumption when possible.
> This is especially useful when storing very long series of  values, combined with delta encoding.
> Lucene already implements variable-length encoding for positive int and long. I propose to add similar Writable implementations, which use the same encoding methods.

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] Commented: (HADOOP-850) Add Writable implementations for variable-length integer types.

Posted by "Owen O'Malley (JIRA)" <ji...@apache.org>.
    [ http://issues.apache.org/jira/browse/HADOOP-850?page=comments#action_12461139 ] 
            
Owen O'Malley commented on HADOOP-850:
--------------------------------------

Please use the VInt implementation that is already in org.apache.hadoop.io.WritableUtils. It makes sense to have a WritableComparable class for this, but I think it is confusing to have two incompatible vint implementations.

> Add Writable implementations for variable-length integer types.
> ---------------------------------------------------------------
>
>                 Key: HADOOP-850
>                 URL: http://issues.apache.org/jira/browse/HADOOP-850
>             Project: Hadoop
>          Issue Type: Improvement
>          Components: io
>    Affects Versions: 0.10.0
>            Reporter: Andrzej Bialecki 
>         Assigned To: Andrzej Bialecki 
>            Priority: Minor
>             Fix For: 0.10.0
>
>         Attachments: variable-ints.patch
>
>
> Currently Hadoop supports only three basic integer-like types: ByteWritable, IntWritable and LongWritable. They provide a fixed tradeoff between their value range and on-disk space consumption. But it is sometimes useful to be able to store integer values with broader allowed range, but less space consumption when possible.
> This is especially useful when storing very long series of  values, combined with delta encoding.
> Lucene already implements variable-length encoding for positive int and long. I propose to add similar Writable implementations, which use the same encoding methods.

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] Commented: (HADOOP-850) Add Writable implementations for variable-length integer types.

Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
    [ http://issues.apache.org/jira/browse/HADOOP-850?page=comments#action_12461099 ] 
            
Hadoop QA commented on HADOOP-850:
----------------------------------

+1, because http://issues.apache.org/jira/secure/attachment/12347954/variable-ints.patch applied and successfully tested against trunk revision r489707.

> Add Writable implementations for variable-length integer types.
> ---------------------------------------------------------------
>
>                 Key: HADOOP-850
>                 URL: http://issues.apache.org/jira/browse/HADOOP-850
>             Project: Hadoop
>          Issue Type: Improvement
>          Components: io
>    Affects Versions: 0.10.0
>            Reporter: Andrzej Bialecki 
>         Assigned To: Andrzej Bialecki 
>            Priority: Minor
>             Fix For: 0.10.0
>
>         Attachments: variable-ints.patch
>
>
> Currently Hadoop supports only three basic integer-like types: ByteWritable, IntWritable and LongWritable. They provide a fixed tradeoff between their value range and on-disk space consumption. But it is sometimes useful to be able to store integer values with broader allowed range, but less space consumption when possible.
> This is especially useful when storing very long series of  values, combined with delta encoding.
> Lucene already implements variable-length encoding for positive int and long. I propose to add similar Writable implementations, which use the same encoding methods.

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] Updated: (HADOOP-850) Add Writable implementations for variable-length integer types.

Posted by "Andrzej Bialecki (JIRA)" <ji...@apache.org>.
     [ http://issues.apache.org/jira/browse/HADOOP-850?page=all ]

Andrzej Bialecki  updated HADOOP-850:
-------------------------------------

    Attachment: variable-ints.patch

Fix javadoc warnings (references to external classes in Lucene).

> Add Writable implementations for variable-length integer types.
> ---------------------------------------------------------------
>
>                 Key: HADOOP-850
>                 URL: http://issues.apache.org/jira/browse/HADOOP-850
>             Project: Hadoop
>          Issue Type: Improvement
>          Components: io
>    Affects Versions: 0.10.0
>            Reporter: Andrzej Bialecki 
>         Assigned To: Andrzej Bialecki 
>            Priority: Minor
>             Fix For: 0.10.0
>
>         Attachments: variable-ints.patch, variable-ints.patch
>
>
> Currently Hadoop supports only three basic integer-like types: ByteWritable, IntWritable and LongWritable. They provide a fixed tradeoff between their value range and on-disk space consumption. But it is sometimes useful to be able to store integer values with broader allowed range, but less space consumption when possible.
> This is especially useful when storing very long series of  values, combined with delta encoding.
> Lucene already implements variable-length encoding for positive int and long. I propose to add similar Writable implementations, which use the same encoding methods.

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] Updated: (HADOOP-850) Add Writable implementations for variable-length integer types.

Posted by "Andrzej Bialecki (JIRA)" <ji...@apache.org>.
     [ http://issues.apache.org/jira/browse/HADOOP-850?page=all ]

Andrzej Bialecki  updated HADOOP-850:
-------------------------------------

    Status: Patch Available  (was: Open)

> Add Writable implementations for variable-length integer types.
> ---------------------------------------------------------------
>
>                 Key: HADOOP-850
>                 URL: http://issues.apache.org/jira/browse/HADOOP-850
>             Project: Hadoop
>          Issue Type: Improvement
>          Components: io
>    Affects Versions: 0.10.0
>            Reporter: Andrzej Bialecki 
>         Assigned To: Andrzej Bialecki 
>            Priority: Minor
>             Fix For: 0.10.0
>
>         Attachments: variable-ints.patch
>
>
> Currently Hadoop supports only three basic integer-like types: ByteWritable, IntWritable and LongWritable. They provide a fixed tradeoff between their value range and on-disk space consumption. But it is sometimes useful to be able to store integer values with broader allowed range, but less space consumption when possible.
> This is especially useful when storing very long series of  values, combined with delta encoding.
> Lucene already implements variable-length encoding for positive int and long. I propose to add similar Writable implementations, which use the same encoding methods.

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] Commented: (HADOOP-850) Add Writable implementations for variable-length integer types.

Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
    [ http://issues.apache.org/jira/browse/HADOOP-850?page=comments#action_12461082 ] 
            
Hadoop QA commented on HADOOP-850:
----------------------------------

-1, because the javadoc command appears to have generated warning messages when testing the latest attachment (http://issues.apache.org/jira/secure/attachment/12347952/variable-ints.patch) against trunk revision r489707. Please note that this message is automatically generated and may represent a problem with the automation system and not the patch.

> Add Writable implementations for variable-length integer types.
> ---------------------------------------------------------------
>
>                 Key: HADOOP-850
>                 URL: http://issues.apache.org/jira/browse/HADOOP-850
>             Project: Hadoop
>          Issue Type: Improvement
>          Components: io
>    Affects Versions: 0.10.0
>            Reporter: Andrzej Bialecki 
>         Assigned To: Andrzej Bialecki 
>            Priority: Minor
>             Fix For: 0.10.0
>
>         Attachments: variable-ints.patch
>
>
> Currently Hadoop supports only three basic integer-like types: ByteWritable, IntWritable and LongWritable. They provide a fixed tradeoff between their value range and on-disk space consumption. But it is sometimes useful to be able to store integer values with broader allowed range, but less space consumption when possible.
> This is especially useful when storing very long series of  values, combined with delta encoding.
> Lucene already implements variable-length encoding for positive int and long. I propose to add similar Writable implementations, which use the same encoding methods.

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira