[jira] Created: (HADOOP-4555) Hive: add a RegularExpressionDeserializer

["Zheng Shao (JIRA)" <ji...@apache.org>]

Hive: add a RegularExpressionDeserializer
-----------------------------------------

                 Key: HADOOP-4555
                 URL: https://issues.apache.org/jira/browse/HADOOP-4555
             Project: Hadoop Core
          Issue Type: New Feature
            Reporter: Zheng Shao


We need a RegularExpressionDeserializer to read data based on a regex.  This will be very useful for reading files like apache log.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Commented: (HADOOP-4555) Hive: add a RegularExpressionDeserializer

["Zheng Shao (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HADOOP-4555?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12644184#action_12644184 ] 

Zheng Shao commented on HADOOP-4555:
------------------------------------

Here is some information from src/contrib/hive/serde/README (not committed yet)

What is SerDe
-----------
SerDe is a short name for Serializer and Deserializer.
Hive uses SerDe (and FileFormat) to read from/write to tables.

* HDFS files --(InputFileFormat)--> <key, value> --(Deserializer)--> Row object
* Row object --(Serializer)--> <key, value> --(OutputFileFormat)--> HDFS files

Note that the "key" part is ignored when reading, and is always a constant when
writing. Basically the row object is only stored into the "value".

One principle of Hive is that Hive does not own the HDFS file format - Users
should be able to directly read the HDFS files in the Hive tables using other
tools, or use other tools to directly write to HDFS files that can be read by
Hive through "CREATE EXTERNAL TABLE", or can be loaded into Hive through "LOAD
DATA INPATH" which just move the file into Hive table directory.


Note that org.apache.hadoop.hive.serde is the deprecated old serde library.
Please look at org.apache.hadoop.hive.serde2 for the latest version.


Existing FileFormats and SerDe classes
------------------------
Hive currently use these FileFormats to read/write to files:

* TextInputFormat/NoKeyTextOutputFormat
  These 2 classes read/write data in plain text file format.

* SequenceFileInputFormat/SequenceFileOutputFormat
  These 2 classes read/write data in hadoop SequenceFile format.

Hive currently use these SerDe classes to serialize and deserialize data:

* MetadataTypedColumnsetSerDe
  This serde is used to read/write delimited records like CSV, tab-separated
  control-A separated records.

* ThriftSerDe
  This serde is used to read/write thrift serialized objects.  The class file
  for the Thrift object must be loaded first.

* DynamicSerDe
  This serde also read/write thrift serialized objects, but it understands thrift
  DDL so the schema of the object can be provided at runtime.  Also it supports
  a lot of different protocols, including TBinaryProtocol, TJSONProtocol,
  TCTLSeparatedProtocol (which writes data in delimited records).



How to load data into Hive
------------------------
In order to load data into Hive, we need to tell Hive the format of the data
through "CREATE TABLE" statement:

* FileFormat: the data has to be in Text or SequenceFile.
* Format of the row:
  * If the data is in delimited format, use MetadataTypedColumnsetSerDe
  * If the data is in delimited format and has more than 1 levels of delimitor,
    use DynamicSerDe with TCTLSeparatedProtocol
  * If the data is a serialized thrift object, use ThriftSerDe

The steps to load the data:
1 Create a table:

  CREATE TABLE t (foo STRING, bar STRING)
  ROW FORMAT DELIMITED
  FIELDS TERMINATED BY '\t'
  STORED AS TEXTFILE;

  CREATE TABLE t2 (foo STRING, bar ARRAY<STRING>)
  ROW FORMAT DELIMITED
  FIELDS TERMINATED BY '\t'
  COLLECTION ITEMS TERMINATED BY ','
  STORED AS TEXTFILE;

  CREATE TABLE t3 (foo STRING, bar MAP<STRING,STRING>)
  ROW FORMAT DELIMITED
  FIELDS TERMINATED BY '\t'
  COLLECTION ITEMS TERMINATED BY ','
  MAP KEYS TERMINATED BY ':'
  STORED AS TEXTFILE;

  CREATE TABLE t4 (foo STRING, bar MAP<STRING,STRING>)
  ROW FORMAT SERIALIZER 'org.apache.hadoop.hive.serde2.MetadataTypedColumnsetSerDe'
  WITH SERDEPROPERTIES ('columns'='foo,bar','SERIALIZATION.FORMAT'='9');

  (RegexDeserializer is not done yet)
  CREATE TABLE t5 (foo STRING, bar STRING)
  ROW FORMAT SERIALIZER 'org.apache.hadoop.hive.serde2.RegexDeserializer'
  WITH SERDEPROPERTIES ('regex'='([a-z]*) *([a-z]*)');

2 Load the data:
  LOAD DATA LOCAL INPATH '../examples/files/kv1.txt' OVERWRITE INTO TABLE t;



How to read data from Hive tables
------------------------
In order to read data from Hive tables, we need to know the same 2 things:
* File Format
* Row Format

Then we just need to directly open the HDFS file and read the data.


How to write your own SerDe
------------------------

In most cases, users want to write a Deserializer instead of a SerDe.
For example, the RegexDeserializer will deserialize the data using the
configuration parameter 'regex', and possibly a list of column names (see
serde2.MetadataTypedColumnsetSerDe).

Please see serde2/Deserializer.java for details.


> Hive: add a RegularExpressionDeserializer
> -----------------------------------------
>
>                 Key: HADOOP-4555
>                 URL: https://issues.apache.org/jira/browse/HADOOP-4555
>             Project: Hadoop Core
>          Issue Type: New Feature
>            Reporter: Zheng Shao
>
> We need a RegularExpressionDeserializer to read data based on a regex.  This will be very useful for reading files like apache log.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.