You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hive.apache.org by "Zheng Shao (JIRA)" <ji...@apache.org> on 2009/07/27 22:46:15 UTC
[jira] Created: (HIVE-693) Add a AWS S3 log format deserializer
Add a AWS S3 log format deserializer
------------------------------------
Key: HIVE-693
URL: https://issues.apache.org/jira/browse/HIVE-693
Project: Hadoop Hive
Issue Type: New Feature
Components: Serializers/Deserializers
Reporter: Zheng Shao
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-693) Add a AWS S3 log format deserializer
Posted by "Andraz Tori (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-693?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Andraz Tori updated HIVE-693:
-----------------------------
Attachment: inputs3.q
actually, the input.q was a bit old, sorry for that, here's the fixed one.
everything else seems ok
> Add a AWS S3 log format deserializer
> ------------------------------------
>
> Key: HIVE-693
> URL: https://issues.apache.org/jira/browse/HIVE-693
> Project: Hadoop Hive
> Issue Type: New Feature
> Components: Serializers/Deserializers
> Reporter: Zheng Shao
> Assignee: Andraz Tori
> Fix For: 0.3.0
>
> Attachments: HIVE-693.1.patch, inputs3.q, s3.log, s3deserializer.diff, S3LogDeserializer.java, S3LogStruct.java
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-693) Add a AWS S3 log format deserializer
Posted by "Andraz Tori (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-693?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Andraz Tori updated HIVE-693:
-----------------------------
Attachment: s3.log
... forgot to add a s3.log for previous patch
are there any chances of getting this into 0.4 ?
> Add a AWS S3 log format deserializer
> ------------------------------------
>
> Key: HIVE-693
> URL: https://issues.apache.org/jira/browse/HIVE-693
> Project: Hadoop Hive
> Issue Type: New Feature
> Components: Serializers/Deserializers
> Reporter: Zheng Shao
> Assignee: Andraz Tori
> Fix For: 0.3.0
>
> Attachments: s3.log, s3deserializer.diff, S3LogDeserializer.java, S3LogStruct.java
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-693) Add a AWS S3 log format deserializer
Posted by "Namit Jain (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-693?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Namit Jain updated HIVE-693:
----------------------------
Resolution: Fixed
Fix Version/s: (was: 0.3.0)
0.5.0
Hadoop Flags: [Reviewed]
Status: Resolved (was: Patch Available)
Committed. Thanks Zheng and Andraz
> Add a AWS S3 log format deserializer
> ------------------------------------
>
> Key: HIVE-693
> URL: https://issues.apache.org/jira/browse/HIVE-693
> Project: Hadoop Hive
> Issue Type: New Feature
> Components: Serializers/Deserializers
> Reporter: Zheng Shao
> Assignee: Andraz Tori
> Fix For: 0.5.0
>
> Attachments: HIVE-693.1.patch, HIVE-693.2.patch, inputs3.q, s3.log, s3deserializer.diff, S3LogDeserializer.java, S3LogStruct.java
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-693) Add a AWS S3 log format deserializer
Posted by "Zheng Shao (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-693?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Zheng Shao updated HIVE-693:
----------------------------
Attachment: HIVE-693.1.patch
HIVE-693.1.patch: Andraz, I've moved all data and code to contrib. Can you review and comment?
Please note that when you want to upgrade from hive 0.3 to hive 0.4 to use this new serde, you would need to manually go through the metastore tables and replace the name of the SerDe class (since it's changed to org.apache.hadoop.hive.contrib.serde2.s3.S3LogDeserializer.
> Add a AWS S3 log format deserializer
> ------------------------------------
>
> Key: HIVE-693
> URL: https://issues.apache.org/jira/browse/HIVE-693
> Project: Hadoop Hive
> Issue Type: New Feature
> Components: Serializers/Deserializers
> Reporter: Zheng Shao
> Assignee: Andraz Tori
> Fix For: 0.3.0
>
> Attachments: HIVE-693.1.patch, s3.log, s3deserializer.diff, S3LogDeserializer.java, S3LogStruct.java
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-693) Add a AWS S3 log format deserializer
Posted by "Zheng Shao (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12736337#action_12736337 ]
Zheng Shao commented on HIVE-693:
---------------------------------
Cool. This looks like a good example for deserializing into a user-predefined Java class.
Can you put the Hive commands to create table, load data, and query data into a comment here?
Can you clean up the code a bit and move the test into a junit class?
It will be great if you can move the deserializer into contrib package as well.
> Add a AWS S3 log format deserializer
> ------------------------------------
>
> Key: HIVE-693
> URL: https://issues.apache.org/jira/browse/HIVE-693
> Project: Hadoop Hive
> Issue Type: New Feature
> Components: Serializers/Deserializers
> Reporter: Zheng Shao
> Attachments: S3LogDeserializer.java, S3LogStruct.java
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-693) Add a AWS S3 log format deserializer
Posted by "Andraz Tori (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-693?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Andraz Tori updated HIVE-693:
-----------------------------
Fix Version/s: 0.3.0
Status: Patch Available (was: Open)
here's a patch with expected inputs and outputs so unittests can be created...
I am still new to Hive source tree, so someone else should take care of moving it to contrib.
> Add a AWS S3 log format deserializer
> ------------------------------------
>
> Key: HIVE-693
> URL: https://issues.apache.org/jira/browse/HIVE-693
> Project: Hadoop Hive
> Issue Type: New Feature
> Components: Serializers/Deserializers
> Reporter: Zheng Shao
> Assignee: Andraz Tori
> Fix For: 0.3.0
>
> Attachments: S3LogDeserializer.java, S3LogStruct.java
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Assigned: (HIVE-693) Add a AWS S3 log format deserializer
Posted by "Ashish Thusoo (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-693?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ashish Thusoo reassigned HIVE-693:
----------------------------------
Assignee: Andraz Tori
> Add a AWS S3 log format deserializer
> ------------------------------------
>
> Key: HIVE-693
> URL: https://issues.apache.org/jira/browse/HIVE-693
> Project: Hadoop Hive
> Issue Type: New Feature
> Components: Serializers/Deserializers
> Reporter: Zheng Shao
> Assignee: Andraz Tori
> Attachments: S3LogDeserializer.java, S3LogStruct.java
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-693) Add a AWS S3 log format deserializer
Posted by "Namit Jain (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12863397#action_12863397 ]
Namit Jain commented on HIVE-693:
---------------------------------
Can you file a new patch and a fix for the same ? We can take a look at it immediately.
> Add a AWS S3 log format deserializer
> ------------------------------------
>
> Key: HIVE-693
> URL: https://issues.apache.org/jira/browse/HIVE-693
> Project: Hadoop Hive
> Issue Type: New Feature
> Components: Serializers/Deserializers
> Reporter: Zheng Shao
> Assignee: Andraz Tori
> Fix For: 0.5.0
>
> Attachments: HIVE-693.1.patch, HIVE-693.2.patch, inputs3.q, s3.log, s3deserializer.diff, S3LogDeserializer.java, S3LogStruct.java
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-693) Add a AWS S3 log format deserializer
Posted by "Andraz Tori (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12863056#action_12863056 ]
Andraz Tori commented on HIVE-693:
----------------------------------
Anyone caring to apply this minor fix before next version?
> Add a AWS S3 log format deserializer
> ------------------------------------
>
> Key: HIVE-693
> URL: https://issues.apache.org/jira/browse/HIVE-693
> Project: Hadoop Hive
> Issue Type: New Feature
> Components: Serializers/Deserializers
> Reporter: Zheng Shao
> Assignee: Andraz Tori
> Fix For: 0.5.0
>
> Attachments: HIVE-693.1.patch, HIVE-693.2.patch, inputs3.q, s3.log, s3deserializer.diff, S3LogDeserializer.java, S3LogStruct.java
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-693) Add a AWS S3 log format deserializer
Posted by "Andraz Tori (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12739802#action_12739802 ]
Andraz Tori commented on HIVE-693:
----------------------------------
I am developing against 0.3 [which is what we have on our servers] which doesn't have contrib directory. So it would be much easier for someone actually familiar with the trunk to do it.
Moving it to contrib should be trivial for someone having a buildable trunk checkout.
The tests are attached in a separate patch - inputs and outputs, so main class can be easily deleted if it bothers anyone. I have found it extremely helpful to have everything in one place during development.
> Add a AWS S3 log format deserializer
> ------------------------------------
>
> Key: HIVE-693
> URL: https://issues.apache.org/jira/browse/HIVE-693
> Project: Hadoop Hive
> Issue Type: New Feature
> Components: Serializers/Deserializers
> Reporter: Zheng Shao
> Assignee: Andraz Tori
> Fix For: 0.3.0
>
> Attachments: s3.log, s3deserializer.diff, S3LogDeserializer.java, S3LogStruct.java
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-693) Add a AWS S3 log format deserializer
Posted by "Ashish Thusoo (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12740741#action_12740741 ]
Ashish Thusoo commented on HIVE-693:
------------------------------------
Zheng,
I noticed that in contrib/build.xml we have the same logFile name in the gen-test target as the one that appears in ql. This will overwrite the logs. We should fix that, perhaps as part of this patch?
> Add a AWS S3 log format deserializer
> ------------------------------------
>
> Key: HIVE-693
> URL: https://issues.apache.org/jira/browse/HIVE-693
> Project: Hadoop Hive
> Issue Type: New Feature
> Components: Serializers/Deserializers
> Reporter: Zheng Shao
> Assignee: Andraz Tori
> Fix For: 0.3.0
>
> Attachments: HIVE-693.1.patch, inputs3.q, s3.log, s3deserializer.diff, S3LogDeserializer.java, S3LogStruct.java
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Resolved: (HIVE-693) Add a AWS S3 log format deserializer
Posted by "Carl Steinbach (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-693?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Carl Steinbach resolved HIVE-693.
---------------------------------
Resolution: Fixed
Filed HIVE-1483 to cover the required update to the regex.
> Add a AWS S3 log format deserializer
> ------------------------------------
>
> Key: HIVE-693
> URL: https://issues.apache.org/jira/browse/HIVE-693
> Project: Hadoop Hive
> Issue Type: New Feature
> Components: Serializers/Deserializers
> Reporter: Zheng Shao
> Assignee: Andraz Tori
> Fix For: 0.5.0
>
> Attachments: HIVE-693.1.patch, HIVE-693.2.patch, inputs3.q, s3.log, s3deserializer.diff, S3LogDeserializer.java, S3LogStruct.java
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-693) Add a AWS S3 log format deserializer
Posted by "Namit Jain (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12742730#action_12742730 ]
Namit Jain commented on HIVE-693:
---------------------------------
Should create table be allowed where the serde specified does not implement the serializer ?
If one tries to insert into this table, it will fail.
> Add a AWS S3 log format deserializer
> ------------------------------------
>
> Key: HIVE-693
> URL: https://issues.apache.org/jira/browse/HIVE-693
> Project: Hadoop Hive
> Issue Type: New Feature
> Components: Serializers/Deserializers
> Reporter: Zheng Shao
> Assignee: Andraz Tori
> Fix For: 0.3.0
>
> Attachments: HIVE-693.1.patch, HIVE-693.2.patch, inputs3.q, s3.log, s3deserializer.diff, S3LogDeserializer.java, S3LogStruct.java
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-693) Add a AWS S3 log format deserializer
Posted by "Ashish Thusoo (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12739798#action_12739798 ]
Ashish Thusoo commented on HIVE-693:
------------------------------------
Hi Andraz,
Can you add the new serde to hive/contrib/src ... Ideally we would want to have the core serde directory just have the lazy serde which is native to hive and have all others in contrib. Also you would have to take the main part out from the serde class and put it in hive/serde/test directory. You can look at some of the examples there to see how it is being done for other serdes. If you can get those things we can bring it into hive contrib. We can also get it into 0.4 as well.
> Add a AWS S3 log format deserializer
> ------------------------------------
>
> Key: HIVE-693
> URL: https://issues.apache.org/jira/browse/HIVE-693
> Project: Hadoop Hive
> Issue Type: New Feature
> Components: Serializers/Deserializers
> Reporter: Zheng Shao
> Assignee: Andraz Tori
> Fix For: 0.3.0
>
> Attachments: s3.log, s3deserializer.diff, S3LogDeserializer.java, S3LogStruct.java
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-693) Add a AWS S3 log format deserializer
Posted by "Zheng Shao (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-693?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Zheng Shao updated HIVE-693:
----------------------------
Attachment: HIVE-693.2.patch
Incorporated Ashish's comments.
Also removed the column definition since they will come directly from serde.
@Andraz: For speed improvement: Instead of using regex, you can read in the data as org.apache.hadoop.io.Text, and do split by yourself. Each field can be stored in a Text as well, and the Text object can be reused across the rows. In this way, the processing will be much faster.
> Add a AWS S3 log format deserializer
> ------------------------------------
>
> Key: HIVE-693
> URL: https://issues.apache.org/jira/browse/HIVE-693
> Project: Hadoop Hive
> Issue Type: New Feature
> Components: Serializers/Deserializers
> Reporter: Zheng Shao
> Assignee: Andraz Tori
> Fix For: 0.3.0
>
> Attachments: HIVE-693.1.patch, HIVE-693.2.patch, inputs3.q, s3.log, s3deserializer.diff, S3LogDeserializer.java, S3LogStruct.java
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-693) Add a AWS S3 log format deserializer
Posted by "Andraz Tori (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-693?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Andraz Tori updated HIVE-693:
-----------------------------
Attachment: S3LogStruct.java
S3LogDeserializer.java
Deserializer implementation.
While it works, code is by no means release-ready, it has to be cleaned up first. But it is better than nothing as a starting point for someone looking to integrate S3 log deserializer.
I was quite amazed to find out that no one else needed this/published this.
> Add a AWS S3 log format deserializer
> ------------------------------------
>
> Key: HIVE-693
> URL: https://issues.apache.org/jira/browse/HIVE-693
> Project: Hadoop Hive
> Issue Type: New Feature
> Components: Serializers/Deserializers
> Reporter: Zheng Shao
> Attachments: S3LogDeserializer.java, S3LogStruct.java
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-693) Add a AWS S3 log format deserializer
Posted by "Andraz Tori (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-693?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Andraz Tori updated HIVE-693:
-----------------------------
Attachment: s3deserializer.diff
the patch...
> Add a AWS S3 log format deserializer
> ------------------------------------
>
> Key: HIVE-693
> URL: https://issues.apache.org/jira/browse/HIVE-693
> Project: Hadoop Hive
> Issue Type: New Feature
> Components: Serializers/Deserializers
> Reporter: Zheng Shao
> Assignee: Andraz Tori
> Fix For: 0.3.0
>
> Attachments: s3deserializer.diff, S3LogDeserializer.java, S3LogStruct.java
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Reopened: (HIVE-693) Add a AWS S3 log format deserializer
Posted by "Andraz Tori (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-693?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Andraz Tori reopened HIVE-693:
------------------------------
Amazon Changed the format of the logs at the beginning of February 2010, so now the new regex is:
static Pattern regexpat = Pattern.compile( "(\\S+) (\\S+) \\[(.*?)\\] (\\S+) (\\S+) (\\S+) (\\S+) (\\S+) \"(.+)\" (\\S+) (\\S+) (\\S+) (\\S+) (\\S+) (\\S+) \"(.*)\" \"(.*)\"(?: -)?");
(the only difference is addition of (?: -)? at the end.
Since Amazon hasn't yet documented the last field, I don't know if it is ok to do a catch-all regex for that field instead of the very specific one I've added.
> Add a AWS S3 log format deserializer
> ------------------------------------
>
> Key: HIVE-693
> URL: https://issues.apache.org/jira/browse/HIVE-693
> Project: Hadoop Hive
> Issue Type: New Feature
> Components: Serializers/Deserializers
> Reporter: Zheng Shao
> Assignee: Andraz Tori
> Fix For: 0.5.0
>
> Attachments: HIVE-693.1.patch, HIVE-693.2.patch, inputs3.q, s3.log, s3deserializer.diff, S3LogDeserializer.java, S3LogStruct.java
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-693) Add a AWS S3 log format deserializer
Posted by "Andraz Tori (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12740742#action_12740742 ]
Andraz Tori commented on HIVE-693:
----------------------------------
Zheng, I have two questions:
1) How does one "manually go through the metastore tables" ?
2) What should be done to optimize the code so it executes faster. Are there any optimizations that you can spot on the first sight? Are there any benchmarking tools to test different implementations of deserializers.
In the above, my doubts are about regex speed and creating so many new strings every time.
> Add a AWS S3 log format deserializer
> ------------------------------------
>
> Key: HIVE-693
> URL: https://issues.apache.org/jira/browse/HIVE-693
> Project: Hadoop Hive
> Issue Type: New Feature
> Components: Serializers/Deserializers
> Reporter: Zheng Shao
> Assignee: Andraz Tori
> Fix For: 0.3.0
>
> Attachments: HIVE-693.1.patch, inputs3.q, s3.log, s3deserializer.diff, S3LogDeserializer.java, S3LogStruct.java
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.