You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hive.apache.org by "alex gemini (JIRA)" <ji...@apache.org> on 2012/10/16 10:39:02 UTC
[jira] [Created] (HIVE-3585) Integrate Trevni as another columnar
oriented file format
alex gemini created HIVE-3585:
---------------------------------
Summary: Integrate Trevni as another columnar oriented file format
Key: HIVE-3585
URL: https://issues.apache.org/jira/browse/HIVE-3585
Project: Hive
Issue Type: Improvement
Components: Serializers/Deserializers
Affects Versions: 0.10.0
Reporter: alex gemini
Priority: Minor
add new avro module trevni as another columnar format.New columnar format need a columnar SerDe,seems fastutil is a good choice.the shark project use fastutil library as columnar serde library but it seems too large (almost 15m) for just a few primitive array collection.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3585) Integrate Trevni as another columnar
oriented file format
Posted by "Russell Jurney (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-3585?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13496944#comment-13496944 ]
Russell Jurney commented on HIVE-3585:
--------------------------------------
It is worth noting that Hive already supports Avro via HIVE-895. Trevni is part of Avro. Therefore the possibility for confusion seems minimal.
Looking at the Serde document https://cwiki.apache.org/confluence/display/Hive/SerDe it seems the Serde mechanism was created specifically for adding other storage formats to Hive. Restricting Serdes based on their similarity or dissimilarity to RCFile is not covered in this document and seems unusual.
Pig and other tools are starting to generate Trevni data. There is demand to work with this data in Hive. RCFile seems completely unrelated to this feature JIRA, which is simply an extension of the existing Avro support in Hive.
> Integrate Trevni as another columnar oriented file format
> ---------------------------------------------------------
>
> Key: HIVE-3585
> URL: https://issues.apache.org/jira/browse/HIVE-3585
> Project: Hive
> Issue Type: Improvement
> Components: Serializers/Deserializers
> Affects Versions: 0.10.0
> Reporter: alex gemini
> Assignee: Jakob Homan
> Priority: Minor
>
> add new avro module trevni as another columnar format.New columnar format need a columnar SerDe,seems fastutil is a good choice.the shark project use fastutil library as columnar serde library but it seems too large (almost 15m) for just a few primitive array collection.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3585) Integrate Trevni as another columnar
oriented file format
Posted by "He Yongqiang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-3585?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13496389#comment-13496389 ]
He Yongqiang commented on HIVE-3585:
------------------------------------
vote for -1.
I did not see any benefit of adding one that is just a copycat of rcfile.
> Integrate Trevni as another columnar oriented file format
> ---------------------------------------------------------
>
> Key: HIVE-3585
> URL: https://issues.apache.org/jira/browse/HIVE-3585
> Project: Hive
> Issue Type: Improvement
> Components: Serializers/Deserializers
> Affects Versions: 0.10.0
> Reporter: alex gemini
> Assignee: Jakob Homan
> Priority: Minor
>
> add new avro module trevni as another columnar format.New columnar format need a columnar SerDe,seems fastutil is a good choice.the shark project use fastutil library as columnar serde library but it seems too large (almost 15m) for just a few primitive array collection.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (HIVE-3585) Integrate Trevni as another columnar
oriented file format
Posted by "Jakob Homan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-3585?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jakob Homan reassigned HIVE-3585:
---------------------------------
Assignee: Jakob Homan
> Integrate Trevni as another columnar oriented file format
> ---------------------------------------------------------
>
> Key: HIVE-3585
> URL: https://issues.apache.org/jira/browse/HIVE-3585
> Project: Hive
> Issue Type: Improvement
> Components: Serializers/Deserializers
> Affects Versions: 0.10.0
> Reporter: alex gemini
> Assignee: Jakob Homan
> Priority: Minor
>
> add new avro module trevni as another columnar format.New columnar format need a columnar SerDe,seems fastutil is a good choice.the shark project use fastutil library as columnar serde library but it seems too large (almost 15m) for just a few primitive array collection.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3585) Integrate Trevni as another columnar
oriented file format
Posted by "He Yongqiang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-3585?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13496546#comment-13496546 ]
He Yongqiang commented on HIVE-3585:
------------------------------------
Although it is so similar to RCFIle, i did not see any reference to RCFile in its doc. I assume that will help avoid confusion for its users. But as part of Hive, if we got two formats that are so similar to each other, the confusion will be thrown to all hive users.
> Integrate Trevni as another columnar oriented file format
> ---------------------------------------------------------
>
> Key: HIVE-3585
> URL: https://issues.apache.org/jira/browse/HIVE-3585
> Project: Hive
> Issue Type: Improvement
> Components: Serializers/Deserializers
> Affects Versions: 0.10.0
> Reporter: alex gemini
> Assignee: Jakob Homan
> Priority: Minor
>
> add new avro module trevni as another columnar format.New columnar format need a columnar SerDe,seems fastutil is a good choice.the shark project use fastutil library as columnar serde library but it seems too large (almost 15m) for just a few primitive array collection.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3585) Integrate Trevni as another columnar
oriented file format
Posted by "Carl Steinbach (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-3585?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13496530#comment-13496530 ]
Carl Steinbach commented on HIVE-3585:
--------------------------------------
@Yongqiang: Trevni aims to offer significant performance improvements over RCFile. Have you read the spec? http://avro.apache.org/docs/current/trevni/spec.html
> Integrate Trevni as another columnar oriented file format
> ---------------------------------------------------------
>
> Key: HIVE-3585
> URL: https://issues.apache.org/jira/browse/HIVE-3585
> Project: Hive
> Issue Type: Improvement
> Components: Serializers/Deserializers
> Affects Versions: 0.10.0
> Reporter: alex gemini
> Assignee: Jakob Homan
> Priority: Minor
>
> add new avro module trevni as another columnar format.New columnar format need a columnar SerDe,seems fastutil is a good choice.the shark project use fastutil library as columnar serde library but it seems too large (almost 15m) for just a few primitive array collection.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3585) Integrate Trevni as another columnar
oriented file format
Posted by "Russell Jurney (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-3585?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13496547#comment-13496547 ]
Russell Jurney commented on HIVE-3585:
--------------------------------------
Can you please explain how supporting additional formats will confuse anyone? This does not seem to follow.
> Integrate Trevni as another columnar oriented file format
> ---------------------------------------------------------
>
> Key: HIVE-3585
> URL: https://issues.apache.org/jira/browse/HIVE-3585
> Project: Hive
> Issue Type: Improvement
> Components: Serializers/Deserializers
> Affects Versions: 0.10.0
> Reporter: alex gemini
> Assignee: Jakob Homan
> Priority: Minor
>
> add new avro module trevni as another columnar format.New columnar format need a columnar SerDe,seems fastutil is a good choice.the shark project use fastutil library as columnar serde library but it seems too large (almost 15m) for just a few primitive array collection.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3585) Integrate Trevni as another columnar
oriented file format
Posted by "He Yongqiang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-3585?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13496537#comment-13496537 ]
He Yongqiang commented on HIVE-3585:
------------------------------------
Yeah i read some docs of it. But i really did not see a big difference. Some features can be added to RCFile easily. Please point out if you think there is a dramatic difference in some designs.
> Integrate Trevni as another columnar oriented file format
> ---------------------------------------------------------
>
> Key: HIVE-3585
> URL: https://issues.apache.org/jira/browse/HIVE-3585
> Project: Hive
> Issue Type: Improvement
> Components: Serializers/Deserializers
> Affects Versions: 0.10.0
> Reporter: alex gemini
> Assignee: Jakob Homan
> Priority: Minor
>
> add new avro module trevni as another columnar format.New columnar format need a columnar SerDe,seems fastutil is a good choice.the shark project use fastutil library as columnar serde library but it seems too large (almost 15m) for just a few primitive array collection.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3585) Integrate Trevni as another columnar
oriented file format
Posted by "Russell Jurney (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-3585?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13496543#comment-13496543 ]
Russell Jurney commented on HIVE-3585:
--------------------------------------
Trevni is an independent file format. It is not RCFile. It already exists. Why is RCFile relevant to this ticket?
> Integrate Trevni as another columnar oriented file format
> ---------------------------------------------------------
>
> Key: HIVE-3585
> URL: https://issues.apache.org/jira/browse/HIVE-3585
> Project: Hive
> Issue Type: Improvement
> Components: Serializers/Deserializers
> Affects Versions: 0.10.0
> Reporter: alex gemini
> Assignee: Jakob Homan
> Priority: Minor
>
> add new avro module trevni as another columnar format.New columnar format need a columnar SerDe,seems fastutil is a good choice.the shark project use fastutil library as columnar serde library but it seems too large (almost 15m) for just a few primitive array collection.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira