[jira] [Commented] (TAJO-1614) Configuration format proposal for generic storage support

["Jihoon Son (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/TAJO-1614?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14549748#comment-14549748 ] 

Jihoon Son commented on TAJO-1614:
----------------------------------

Hi [~hyunsik], thanks for the nice proposal. 
It looks sufficiently general to represent wide range of storage and file types. 
I have two questions. According to your proposal, properties cab be defined for each specific table space types. Is this able to be difined for storage and file types? 
Next, do you have any plans to provide customizable properties or only predefined ones? 

> Configuration format proposal for generic storage support
> ---------------------------------------------------------
>
>                 Key: TAJO-1614
>                 URL: https://issues.apache.org/jira/browse/TAJO-1614
>             Project: Tajo
>          Issue Type: Sub-task
>          Components: storage
>            Reporter: Hyunsik Choi
>            Assignee: Hyunsik Choi
>             Fix For: 0.11.0
>
>
> This is an example of the first draft of new storage configuration file for generic storage. It allows users to specify:
>  * table spaces
>  * space handlers
>  * data formats
>  * relationships among them
> More detailed explanation is as follows:
>  * {{/storages}} specifies all storage types.
>    * {{/storages/*}} specifies which handler deals with which uri scheme.
>  * {{/formats}} can specify each data format or file format
>    * {{/formats/*}} specifies which format is available in which storage types.
>    * {{/formats/*/handler}} specifies a handler class to allows clients to get scanner or appender. 
> {code}
> {
>   "spaces": {
>     "default": {
>       "uri": "hdfs://localhost:8020/tajo/warehouse",
>       "configs": [
>         {"x": "y"},
>         {"x": "y"}
>       ]
>     },
>     "ssd1": {
>       "uri": "hdfs://localhost:8020/....",
>       "configs": [
>         {"x": "y"},
>         {"x": "y"}
>       ]
>     },
>     "mysql": {
>       "uri": "jdbc://....",
>       "configs": [
>         {"x": "y"},
>         {"x": "y"}
>       ]
>     }
>   },
>   
>   "storages": {
>     "hdfs": {
>       "handler": "org.apache.tajo.storage.HdfsTablespace",
>       "default-format": "text"
>     },
>     "file": {
>       "handler": "org.apache.tajo.storage.FileTablespace",
>       "default-format": "text"
>     },
>     "hbase": {
>       "handler": "org.apache.tajo.storage.hbase.HBaseTablespace",
>       "default-format": "hbase"
>     },
>     "s3": {
>       "handler": "org.apache.tajo.storage.S3Tablespace",
>       "default-format": "text"
>     },
>     "jdbc": {
>       "handler": "org.apache.tajo.storage.JdbcTablespace",
>       "default-format": "rowstore"
>     }
>   },
>   
>   "formats": {
>     
>     "avro": {
>       "storage-support": ["hdfs", "file", "s3"],
>       "handler": "org.apache.tajo.storage.AvroHandler"
>     },
>     "csv": {
>       "storage-support": ["hdfs", "file", "s3"],
>       "handler": "org.apache.tajo.storage.CSVHandler"
>     },
>     "text": {
>       "storage-support": ["hdfs", "file", "s3"],
>       "handler": "org.apache.tajo.storage.TextHandler"
>     },
>     "parquet": {
>       "storage-support": ["hdfs", "file", "s3"],
>       "handler": "org.apache.tajo.storage.ParquetHandler"
>     },
>     "raw": {
>       "storage-support": ["hdfs", "file", "s3"],
>       "handler": "org.apache.tajo.storage.RawHandler"
>     },
>     "rcfile": {
>       "storage-support": ["hdfs", "file", "s3"],
>       "handler": "org.apache.tajo.storage.RCFileHandler"
>     },
>     "row": {
>       "storage-support": ["hdfs", "file", "s3"],
>       "handler": "org.apache.tajo.storage.RowHandler"
>     },
>     "sequencefile": {
>       "storage-support": ["hdfs", "file", "s3"],
>       "handler": "org.apache.tajo.storage.SeqFileHandler"
>     },
>     
>     
>     "hbase": {
>       "storage-support": ["hbase"],
>       "handler": "org.apache.tajo.storage.HbaseHandler"
>     },
>     "kyro": {
>       "storage-support": ["hbase"],
>       "handler": "org.apache.tajo.storage.KyroHandler"
>     },
>     
>     "rowstore": {
>       "storage-support": ["jdbc"],
>       "handler": "org.apache.tajo.storage.RowStoreHandler"
>     }
>   }
> }
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)