You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@tajo.apache.org by babokim <gi...@git.apache.org> on 2014/11/10 09:12:08 UTC

[GitHub] tajo pull request: TAJO-1131: Supports Inserting or Creating table...

GitHub user babokim opened a pull request:

    https://github.com/apache/tajo/pull/232

    TAJO-1131: Supports Inserting or Creating table into the HBase mapped table.

    First, I apologize. The TAJO-1127 Implements HBaseStorageManager issue is also implemented in this issue together. 
    This issue has been implemented the following features:
    
    Mapping HBase table to Tajo table.
    Supports a composite rowkey
    Supports creation multiple Region.
    Scanning HBase table using Tajo selection query.
    Inserting into HBase table using Tajo insertion or creation query.
    Supports both of PUT and BULK.

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/babokim/tajo TAJO-1131

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/tajo/pull/232.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #232
    
----
commit b6e372108d293d8af76185c475b592b10dac012e
Author: HyoungJun Kim <ba...@babokim-macbook-pro.local>
Date:   2014-10-23T07:31:57Z

    TAJO-1123: Use Fragment instead of FileFragment.

commit ec11bef177e5b95ac8842df465ff111759cec501
Author: HyoungJun Kim <ba...@babokim-macbook-pro.local>
Date:   2014-10-24T10:26:52Z

    TAJO-1127: Implements HBaseStorageManager
    
    Implements HBaseStorageManager and HBaseScanner.

commit 6b9e57c761dd613b0455824452de27e2bcee930f
Author: HyoungJun Kim <ba...@babokim-macbook-pro.local>
Date:   2014-10-24T16:42:39Z

    TAJO-1127: Implements HBaseStorageManager
    
    Implements non-forward query.

commit eb2e8ecef5ac344d3286aa8d227b25489c6b4273
Author: HyoungJun Kim <ba...@babokim-macbook-pro.local>
Date:   2014-10-25T10:12:59Z

    TAJO-1127: Implements HBaseStorageManager.

commit d1ccd12ceefceef5c5505fb64e5271d475d6927e
Author: HyoungJun Kim <ba...@babokim-macbook-pro.local>
Date:   2014-10-26T20:40:04Z

    Merge branch 'hbase_storage' of https://git-wip-us.apache.org/repos/asf/tajo into TAJO-1123
    
    Conflicts:
    	tajo-core/src/main/java/org/apache/tajo/engine/planner/PhysicalPlannerImpl.java
    	tajo-core/src/main/java/org/apache/tajo/master/GlobalEngine.java
    	tajo-core/src/main/java/org/apache/tajo/master/querymaster/Repartitioner.java
    	tajo-core/src/main/java/org/apache/tajo/master/querymaster/SubQuery.java
    	tajo-plan/src/main/java/org/apache/tajo/plan/util/PlannerUtil.java

commit d7198b5a028283a012fbcffe0bc6fde7ee93360c
Author: HyoungJun Kim <ba...@babokim-macbook-pro.local>
Date:   2014-10-26T21:26:11Z

    Merge branch 'TAJO-1123' of https://github.com/babokim/tajo into TAJO-1127
    
    Conflicts:
    	tajo-core/src/main/java/org/apache/tajo/master/NonForwardQueryResultScanner.java
    	tajo-core/src/test/java/org/apache/tajo/engine/planner/TestPlannerUtil.java
    	tajo-plan/src/main/java/org/apache/tajo/plan/util/PlannerUtil.java

commit 77941b843c99786e4eafd26b8d4d32f5349c38aa
Author: HyoungJun Kim <ba...@babokim-mbp.server.gruter.com>
Date:   2014-10-27T06:20:34Z

    TAJO-1127: Implements HBaseStorageManager.

commit cdfe6de1dac146d9a4f8731e65558b21b21a5b24
Author: HyoungJun Kim <ba...@babokim-mbp.server.gruter.com>
Date:   2014-10-27T06:55:12Z

    Merge branch 'hbase_storage' of https://git-wip-us.apache.org/repos/asf/tajo into TAJO-1127
    
    Conflicts:
    	tajo-common/src/main/java/org/apache/tajo/TajoConstants.java
    	tajo-core/src/main/java/org/apache/tajo/engine/planner/PhysicalPlannerImpl.java
    	tajo-core/src/main/java/org/apache/tajo/engine/planner/physical/SeqScanExec.java
    	tajo-core/src/main/java/org/apache/tajo/master/GlobalEngine.java
    	tajo-core/src/main/java/org/apache/tajo/master/LazyTaskScheduler.java
    	tajo-core/src/main/java/org/apache/tajo/master/NonForwardQueryResultScanner.java
    	tajo-core/src/main/java/org/apache/tajo/master/querymaster/Repartitioner.java
    	tajo-core/src/main/java/org/apache/tajo/master/querymaster/SubQuery.java
    	tajo-storage/src/main/java/org/apache/tajo/storage/FileStorageManager.java
    	tajo-storage/src/main/java/org/apache/tajo/storage/MergeScanner.java
    	tajo-storage/src/main/java/org/apache/tajo/storage/StorageManager.java

commit 40aa361cd480c8e29798d11e4e894fc491be661d
Author: HyoungJun Kim <ba...@babokim-macbook-pro.local>
Date:   2014-10-30T11:08:19Z

    TAJO-1131: Supports Inserting or Creating table into the HBase mapped table.

commit 8f09273e73cc47a31ad9d585cd3f7e6ad75e840f
Author: HyoungJun Kim <ba...@babokim-mbp.server.gruter.com>
Date:   2014-10-30T13:39:20Z

    TAJO-1131: Supports Inserting or Creating table into the HBase mapped table.
    
    All test cases pass.

commit 76fb0a06451ecf9e254d3b5d85dafa9a9d248da6
Author: HyoungJun Kim <ba...@babokim-macbook-pro.local>
Date:   2014-10-31T15:40:19Z

    TAJO-1131: Supports Inserting or Creating table into the HBase mapped table.
    
    Add data type verification

commit e8d50a05279517db8ee8e49eec5d5eb369d3a45b
Author: HyoungJun Kim <ba...@babokim-macbook-pro.local>
Date:   2014-11-01T06:54:56Z

    TAJO-1131: Supports Inserting or Creating table into the HBase mapped table.
    
    Supporting binary split key

commit 37942472653febd82f1a7f55c0cfc79b53d780f4
Author: HyoungJun Kim <ba...@babokim-macbook-pro.local>
Date:   2014-11-01T08:03:55Z

    TAJO-1131: Supports Inserting or Creating table into the HBase mapped table.
    
    Fix HBaseAppender unsorted bug.

commit b636faf5023ed6523077f46dd4583f9414209e39
Author: HyoungJun Kim <ba...@babokim-macbook-pro.local>
Date:   2014-11-02T15:46:55Z

    TAJO-1131: Supports Inserting or Creating table into the HBase mapped table.
    
    Support multiple rowkey field.

commit 79cea448833281ffb081d8f21aefa645b082db2f
Author: HyoungJun Kim <ba...@babokim-macbook-pro.local>
Date:   2014-11-02T16:17:38Z

    TAJO-1131: Supports Inserting or Creating table into the HBase mapped table.
    
    TupleComparator error

commit 529d48c112099cb8d85311f32a74986a7e2019b8
Author: HyoungJun Kim <ba...@babokim-macbook-pro.local>
Date:   2014-11-02T16:20:28Z

    TAJO-1131: Supports Inserting or Creating table into the HBase mapped table.
    
    TupleComparator error

commit 6ec23170f16d3a740714bc909dc8949ad00217e7
Author: HyoungJun Kim <ba...@babokim-macbook-pro.local>
Date:   2014-11-03T06:20:17Z

    TAJO-1131: Supports Inserting or Creating table into the HBase mapped table.
    
    Implements HBasePutAppender

commit 54974da8c1d0e0837035c9d92b87c4a30f13aa48
Author: HyoungJun Kim <ba...@babokim-macbook-pro.local>
Date:   2014-11-03T11:25:41Z

    TAJO-1131: Supports Inserting or Creating table into the HBase mapped table.
    
    Implements ColumnKey, Value mapping

commit bfe44fd7953cdeee9c51912aa9f20331cb0c002f
Author: HyoungJun Kim <ba...@babokim-macbook-pro.local>
Date:   2014-11-03T11:54:53Z

    TAJO-1131: Supports Inserting or Creating table into the HBase mapped table.
    
    Fix queryunit.jsp error

commit ce9de5231a09ad7013f7165258395c68adee8bc4
Author: HyoungJun Kim <ba...@babokim-macbook-pro.local>
Date:   2014-11-03T11:58:41Z

    remove ToCharInt function

commit 73126be997dce5ef77043de4a0aad99eb72c1031
Author: HyoungJun Kim <ba...@babokim-macbook-pro.local>
Date:   2014-11-03T12:53:52Z

    TAJO-1131: Supports Inserting or Creating table into the HBase mapped table.
    
    Fix index predication bug

commit f28586cfa88eaffab298b0eba38f7c3288dab098
Author: HyoungJun Kim <ba...@babokim-mbp.server.gruter.com>
Date:   2014-11-03T13:44:38Z

    TAJO-1131: Supports Inserting or Creating table into the HBase mapped table.
    
    Fix composite rowkey scanner

commit 97c32e41eb178af1138b7e6f822c3948d3063882
Author: HyoungJun Kim <ba...@babokim-macbook-pro.local>
Date:   2014-11-04T07:49:03Z

    TAJO-1131: Supports Inserting or Creating table into the HBase mapped table.
    
    Implements Insert into location using hbase

commit 402022f186192f0b1df504ec83b623f7a4f231b8
Author: HyoungJun Kim <ba...@babokim-macbook-pro.local>
Date:   2014-11-04T08:14:57Z

    TAJO-1131: Supports Inserting or Creating table into the HBase mapped table.
    
    Fix hbase region splitting

commit e156b1c642971e95f364c0668bfced96b567c600
Author: HyoungJun Kim <ba...@babokim-macbook-pro.local>
Date:   2014-11-04T13:45:11Z

    TAJO-1131: Supports Inserting or Creating table into the HBase mapped table.
    
    Optimize HBase Scanner

commit 12b30c15fb038b757fbf53c393d79336b7232514
Author: HyoungJun Kim <ba...@babokim-macbook-pro.local>
Date:   2014-11-05T09:07:00Z

    TAJO-1131: Supports Inserting or Creating table into the HBase mapped table.
    
    Add comments

commit 5a3ae8bfe650a07cf5f4f85ef5f3cae03f737a53
Author: HyoungJun Kim <ba...@babokim-macbook-pro.local>
Date:   2014-11-05T11:05:52Z

    Merge branch 'hbase_storage' of https://git-wip-us.apache.org/repos/asf/tajo into TAJO-1131
    
    Conflicts:
    	tajo-core/src/main/java/org/apache/tajo/engine/query/QueryContext.java
    	tajo-plan/src/main/java/org/apache/tajo/plan/util/PlannerUtil.java

commit 44e4d632df99c340ad520a1c39ba3efff94ebde7
Author: HyoungJun Kim <ba...@babokim-macbook-pro.local>
Date:   2014-11-05T11:27:21Z

    TAJO-1131: Supports Inserting or Creating table into the HBase mapped table.
    
    Change TupleComparator to BaseTupleComparator.

----


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] tajo pull request: TAJO-1131: Supports Inserting or Creating table...

Posted by hyunsik <gi...@git.apache.org>.
Github user hyunsik commented on a diff in the pull request:

    https://github.com/apache/tajo/pull/232#discussion_r20281223
  
    --- Diff: tajo-storage/src/main/java/org/apache/tajo/storage/hbase/ColumnMapping.java ---
    @@ -0,0 +1,239 @@
    +/**
    + * Licensed to the Apache Software Foundation (ASF) under one
    + * or more contributor license agreements.  See the NOTICE file
    + * distributed with this work for additional information
    + * regarding copyright ownership.  The ASF licenses this file
    + * to you under the Apache License, Version 2.0 (the
    + * "License"); you may not use this file except in compliance
    + * with the License.  You may obtain a copy of the License at
    + *
    + *     http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +
    +package org.apache.tajo.storage.hbase;
    +
    +import org.apache.hadoop.hbase.util.Bytes;
    +import org.apache.tajo.catalog.Schema;
    +import org.apache.tajo.catalog.TableMeta;
    +import org.apache.tajo.util.BytesUtils;
    +
    +import java.io.IOException;
    +import java.util.ArrayList;
    +import java.util.List;
    +
    +public class ColumnMapping {
    +  public static final String KEY_COLUMN_MAPPING = "key";
    +  public static final String VALUE_COLUMN_MAPPING = "value";
    +
    +  private TableMeta tableMeta;
    +  private Schema schema;
    +  private char rowKeyDelimiter;
    +
    +  private String hbaseTableName;
    +
    +  private int[] rowKeyFieldIndexes;
    +  private boolean[] isRowKeyMappings;
    +  private boolean[] isBinaryColumns;
    +  private boolean[] isColumnKeys;
    +  private boolean[] isColumnValues;
    +
    +  // schema order -> 0: cf name, 1: column name -> name bytes
    +  private byte[][][] mappingColumns;
    +
    +  private int numRowKeys;
    +
    +  public ColumnMapping(Schema schema, TableMeta tableMeta) throws IOException {
    +    this.schema = schema;
    +    this.tableMeta = tableMeta;
    +
    +    init();
    +  }
    +
    +  public void init() throws IOException {
    +    hbaseTableName = tableMeta.getOption(HBaseStorageManager.META_TABLE_KEY);
    +    String delim = tableMeta.getOption(HBaseStorageManager.META_ROWKEY_DELIMITER, "").trim();
    +    if (delim.length() > 0) {
    +      rowKeyDelimiter = delim.charAt(0);
    +    }
    +    isRowKeyMappings = new boolean[schema.size()];
    +    rowKeyFieldIndexes = new int[schema.size()];
    +    isBinaryColumns = new boolean[schema.size()];
    +    isColumnKeys = new boolean[schema.size()];
    +    isColumnValues = new boolean[schema.size()];
    +
    +    mappingColumns = new byte[schema.size()][][];
    +
    +    for (int i = 0; i < schema.size(); i++) {
    +      rowKeyFieldIndexes[i] = -1;
    +    }
    +
    +    String columnMapping = tableMeta.getOption(HBaseStorageManager.META_COLUMNS_KEY, "");
    +    if (columnMapping == null || columnMapping.isEmpty()) {
    +      throw new IOException("'columns' property is required.");
    +    }
    +
    +    String[] columnMappingTokens = columnMapping.split(",");
    +
    +    if (columnMappingTokens.length != schema.getColumns().size()) {
    +      throw new IOException("The number of mapped HBase columns is great than the number of Tajo table columns");
    +    }
    +
    +    int index = 0;
    +    for (String eachToken: columnMappingTokens) {
    +      mappingColumns[index] = new byte[2][];
    +
    +      byte[][] mappingTokens = BytesUtils.splitPreserveAllTokens(eachToken.trim().getBytes(), ':');
    +
    +      if (mappingTokens.length == 3) {
    --- End diff --
    
    They seem to be checked in the phase of logical plan verification. Otherwise, it will cause runtime exception during query processing.
    
    BTW, I know that in this time there is no system to insert this test into the phase of logical plan verification. We need to consider it later. Now, I added the requirement to https://issues.apache.org/jira/browse/TAJO-337.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] tajo pull request: TAJO-1131: Supports Inserting or Creating table...

Posted by hyunsik <gi...@git.apache.org>.
Github user hyunsik commented on the pull request:

    https://github.com/apache/tajo/pull/232#issuecomment-63140065
  
    Excellent! Here is my +1 for pushing the patch to ```hbase_storage``` branch. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] tajo pull request: TAJO-1131: Supports Inserting or Creating table...

Posted by hyunsik <gi...@git.apache.org>.
Github user hyunsik commented on a diff in the pull request:

    https://github.com/apache/tajo/pull/232#discussion_r20278771
  
    --- Diff: tajo-storage/src/test/java/org/apache/tajo/storage/hbase/TestColumnMapping.java ---
    @@ -0,0 +1,97 @@
    +/**
    + * Licensed to the Apache Software Foundation (ASF) under one
    + * or more contributor license agreements.  See the NOTICE file
    + * distributed with this work for additional information
    + * regarding copyright ownership.  The ASF licenses this file
    + * to you under the Apache License, Version 2.0 (the
    + * "License"); you may not use this file except in compliance
    + * with the License.  You may obtain a copy of the License at
    + *
    + *     http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +
    +package org.apache.tajo.storage.hbase;
    +
    +import org.apache.hadoop.hbase.protobuf.generated.HBaseProtos.RegionInfo;
    +import org.apache.tajo.catalog.Schema;
    +import org.apache.tajo.catalog.TableMeta;
    +import org.apache.tajo.catalog.proto.CatalogProtos.StoreType;
    +import org.apache.tajo.common.TajoDataTypes.Type;
    +import org.apache.tajo.util.Bytes;
    --- End diff --
    
    It's an unused import.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] tajo pull request: TAJO-1131: Supports Inserting or Creating table...

Posted by hyunsik <gi...@git.apache.org>.
Github user hyunsik commented on the pull request:

    https://github.com/apache/tajo/pull/232#issuecomment-62634590
  
    Could you rebase it against the latest patch?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] tajo pull request: TAJO-1131: Supports Inserting or Creating table...

Posted by hyunsik <gi...@git.apache.org>.
Github user hyunsik commented on a diff in the pull request:

    https://github.com/apache/tajo/pull/232#discussion_r20359538
  
    --- Diff: tajo-plan/src/main/java/org/apache/tajo/plan/logical/SortNode.java ---
    @@ -25,10 +25,16 @@
     import org.apache.tajo.util.TUtil;
     
     public final class SortNode extends UnaryNode implements Cloneable {
    +  public static enum SortRangeType {
    +    USING_SHUFFLE_DATA,
    --- End diff --
    
    Thank you :)


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] tajo pull request: TAJO-1131: Supports Inserting or Creating table...

Posted by babokim <gi...@git.apache.org>.
Github user babokim commented on the pull request:

    https://github.com/apache/tajo/pull/232#issuecomment-62682193
  
    I just rebase.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] tajo pull request: TAJO-1131: Supports Inserting or Creating table...

Posted by hyunsik <gi...@git.apache.org>.
Github user hyunsik commented on a diff in the pull request:

    https://github.com/apache/tajo/pull/232#discussion_r20280308
  
    --- Diff: tajo-plan/src/main/java/org/apache/tajo/plan/logical/SortNode.java ---
    @@ -25,10 +25,16 @@
     import org.apache.tajo.util.TUtil;
     
     public final class SortNode extends UnaryNode implements Cloneable {
    +  public static enum SortRangeType {
    +    USING_SHUFFLE_DATA,
    --- End diff --
    
    It's just my suggestion. If you agree, please reflect my suggestion.
    
    You seem to add SortRangeType in order to identify the purpose of SortNode. ```USING_STORAGE_MANAGER``` is for sorting rows for sorted HFiles of Hbase. Otherwise, ```USING_SHUFFLE_DATA``` will be used. BTW, ```USING_SHUFFLE_DATA``` is used for shuffle as well as just sort.  Even shuffle is also a part of distributed sort. So, I think that this names probably gives some confuse to code readers.
    
    So, I'd like to suggest as follows:
     * SortRangeType -> SORT_PURPOSE
     * USING_SHUFFLE_DATA -> NORMAL
     * USING_STORAGE_MANAGER -> STORAGE_SPECIFIED
    
    As I mentioned earlier, it's just my suggestion.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] tajo pull request: TAJO-1131: Supports Inserting or Creating table...

Posted by hyunsik <gi...@git.apache.org>.
Github user hyunsik commented on a diff in the pull request:

    https://github.com/apache/tajo/pull/232#discussion_r20282144
  
    --- Diff: tajo-storage/src/main/java/org/apache/tajo/storage/StorageManager.java ---
    @@ -76,20 +98,169 @@
       private static final Map<Class<?>, Constructor<?>> CONSTRUCTOR_CACHE =
           new ConcurrentHashMap<Class<?>, Constructor<?>>();
     
    -  protected abstract void storageInit() throws IOException ;
    -  public abstract void createTable(TableDesc tableDesc) throws IOException;
    +  public StorageManager(StoreType storeType) {
    +    this.storeType = storeType;
    +  }
    +
    +  /**
    +   * Initialize storage manager.
    +   * @throws IOException
    +   */
    +  protected abstract void storageInit() throws IOException;
    +
    +  /**
    +   * This method is called after executing "CREATE TABLE" statement.
    +   * If a storage is a file based storage, a storage manager may create directory.
    +   *
    +   * @param tableDesc Table description which is created.
    +   * @param ifNotExists Creates the table only when the table does not exist.
    +   * @throws IOException
    +   */
    +  public abstract void createTable(TableDesc tableDesc, boolean ifNotExists) throws IOException;
    +
    +  /**
    +   * This method is called after executing "DROP TABLE" statement with the 'PURGE' option
    +   * which is the option to delete all the data.
    +   *
    +   * @param tableDesc
    +   * @throws IOException
    +   */
       public abstract void purgeTable(TableDesc tableDesc) throws IOException;
    -  public abstract List<Fragment> getSplits(String fragmentId, TableDesc tableDesc) throws IOException;
     
    +  /**
    +   * Returns the splits that will serve as input for the scan tasks. The
    +   * number of splits matches the number of regions in a table.
    +   * @param fragmentId The table name or previous ExecutionBlockId
    +   * @param tableDesc The table description for the target data.
    +   * @param scanNode The logical node for scanning.
    +   * @return The list of input fragments.
    +   * @throws IOException
    +   */
    +  public abstract List<Fragment> getSplits(String fragmentId, TableDesc tableDesc,
    +                                           ScanNode scanNode) throws IOException;
    +
    +  /**
    +   * It returns the splits that will serve as input for the non-forward query scanner such as 'select * from table1'.
    +   * The result list should be small. If there is many fragments for scanning, TajoMaster uses the paging navigation.
    +   * @param tableDesc The table description for the target data.
    +   * @param currentPage The current page number within the entire list.
    +   * @param numFragments The number of fragments in the result.
    +   * @return The list of input fragments.
    +   * @throws IOException
    +   */
    +  public abstract List<Fragment> getNonForwardSplit(TableDesc tableDesc, int currentPage, int numFragments)
    +      throws IOException;
    +
    +  /**
    +   * It returns the storage property.
    +   * @return The storage property
    +   */
    +  public abstract StorageProperty getStorageProperty();
    +
    +  /**
    +   * Release storage manager resource
    +   */
    +  public abstract void closeStorageManager();
    +
    +  /**
    +   * It is called by a Repartitioner for range shuffling when the SortRangeType of SortNode is USING_STORAGE_MANAGER.
    +   * In general Repartitioner determines the partition range using previous output statistics data.
    +   * In the special cases, such as HBase Repartitioner uses the result of this method.
    +   *
    +   * @param queryContext The current query context which contains query properties.
    +   * @param tableDesc The table description for the target data.
    +   * @param inputSchema The input schema
    +   * @param sortSpecs The sort specification that contains the sort column and sort order.
    +   * @return The list of sort ranges.
    +   * @throws IOException
    +   */
    +  public abstract TupleRange[] getInsertSortRanges(OverridableConf queryContext, TableDesc tableDesc,
    +                                                   Schema inputSchema, SortSpec[] sortSpecs,
    +                                                   TupleRange dataRange) throws IOException;
    +
    +  /**
    +   * This method is called before executing 'INSERT' or 'CREATE TABLE as SELECT'.
    +   * In general Tajo creates the target table after finishing the final sub-query of CATS.
    +   * But In the special cases, such as HBase INSERT or CAST query uses the target table information.
    +   * That kind of the storage should implements the logic related to creating table in this method.
    +   *
    +   * @param node The child node of the root node.
    +   * @throws IOException
    +   */
    +  public abstract void beforeInsertOrCATS(LogicalNode node) throws IOException;
    +
    +  /**
    +   * It is called when the query failed.
    +   * Each storage manager should implement to be processed when the query fails in this method.
    +   *
    +   * @param node The child node of the root node.
    +   * @throws IOException
    +   */
    +  public abstract void queryFailed(LogicalNode node) throws IOException;
    --- End diff --
    
    I'd like to suggest ```rollbackOutputCommit```.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] tajo pull request: TAJO-1131: Supports Inserting or Creating table...

Posted by hyunsik <gi...@git.apache.org>.
Github user hyunsik commented on a diff in the pull request:

    https://github.com/apache/tajo/pull/232#discussion_r20359516
  
    --- Diff: tajo-storage/src/main/java/org/apache/tajo/storage/hbase/ColumnMapping.java ---
    @@ -0,0 +1,239 @@
    +/**
    + * Licensed to the Apache Software Foundation (ASF) under one
    + * or more contributor license agreements.  See the NOTICE file
    + * distributed with this work for additional information
    + * regarding copyright ownership.  The ASF licenses this file
    + * to you under the Apache License, Version 2.0 (the
    + * "License"); you may not use this file except in compliance
    + * with the License.  You may obtain a copy of the License at
    + *
    + *     http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +
    +package org.apache.tajo.storage.hbase;
    +
    +import org.apache.hadoop.hbase.util.Bytes;
    +import org.apache.tajo.catalog.Schema;
    +import org.apache.tajo.catalog.TableMeta;
    +import org.apache.tajo.util.BytesUtils;
    +
    +import java.io.IOException;
    +import java.util.ArrayList;
    +import java.util.List;
    +
    +public class ColumnMapping {
    +  public static final String KEY_COLUMN_MAPPING = "key";
    +  public static final String VALUE_COLUMN_MAPPING = "value";
    +
    +  private TableMeta tableMeta;
    +  private Schema schema;
    +  private char rowKeyDelimiter;
    +
    +  private String hbaseTableName;
    +
    +  private int[] rowKeyFieldIndexes;
    +  private boolean[] isRowKeyMappings;
    +  private boolean[] isBinaryColumns;
    +  private boolean[] isColumnKeys;
    +  private boolean[] isColumnValues;
    +
    +  // schema order -> 0: cf name, 1: column name -> name bytes
    +  private byte[][][] mappingColumns;
    +
    +  private int numRowKeys;
    +
    +  public ColumnMapping(Schema schema, TableMeta tableMeta) throws IOException {
    +    this.schema = schema;
    +    this.tableMeta = tableMeta;
    +
    +    init();
    +  }
    +
    +  public void init() throws IOException {
    +    hbaseTableName = tableMeta.getOption(HBaseStorageManager.META_TABLE_KEY);
    +    String delim = tableMeta.getOption(HBaseStorageManager.META_ROWKEY_DELIMITER, "").trim();
    +    if (delim.length() > 0) {
    +      rowKeyDelimiter = delim.charAt(0);
    +    }
    +    isRowKeyMappings = new boolean[schema.size()];
    +    rowKeyFieldIndexes = new int[schema.size()];
    +    isBinaryColumns = new boolean[schema.size()];
    +    isColumnKeys = new boolean[schema.size()];
    +    isColumnValues = new boolean[schema.size()];
    +
    +    mappingColumns = new byte[schema.size()][][];
    +
    +    for (int i = 0; i < schema.size(); i++) {
    +      rowKeyFieldIndexes[i] = -1;
    +    }
    +
    +    String columnMapping = tableMeta.getOption(HBaseStorageManager.META_COLUMNS_KEY, "");
    +    if (columnMapping == null || columnMapping.isEmpty()) {
    +      throw new IOException("'columns' property is required.");
    +    }
    +
    +    String[] columnMappingTokens = columnMapping.split(",");
    +
    +    if (columnMappingTokens.length != schema.getColumns().size()) {
    +      throw new IOException("The number of mapped HBase columns is great than the number of Tajo table columns");
    +    }
    +
    +    int index = 0;
    +    for (String eachToken: columnMappingTokens) {
    +      mappingColumns[index] = new byte[2][];
    +
    +      byte[][] mappingTokens = BytesUtils.splitPreserveAllTokens(eachToken.trim().getBytes(), ':');
    +
    +      if (mappingTokens.length == 3) {
    --- End diff --
    
    Sounds great.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] tajo pull request: TAJO-1131: Supports Inserting or Creating table...

Posted by babokim <gi...@git.apache.org>.
Github user babokim commented on a diff in the pull request:

    https://github.com/apache/tajo/pull/232#discussion_r20343876
  
    --- Diff: tajo-storage/src/test/java/org/apache/tajo/storage/hbase/TestColumnMapping.java ---
    @@ -0,0 +1,97 @@
    +/**
    + * Licensed to the Apache Software Foundation (ASF) under one
    + * or more contributor license agreements.  See the NOTICE file
    + * distributed with this work for additional information
    + * regarding copyright ownership.  The ASF licenses this file
    + * to you under the Apache License, Version 2.0 (the
    + * "License"); you may not use this file except in compliance
    + * with the License.  You may obtain a copy of the License at
    + *
    + *     http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +
    +package org.apache.tajo.storage.hbase;
    +
    +import org.apache.hadoop.hbase.protobuf.generated.HBaseProtos.RegionInfo;
    +import org.apache.tajo.catalog.Schema;
    +import org.apache.tajo.catalog.TableMeta;
    +import org.apache.tajo.catalog.proto.CatalogProtos.StoreType;
    +import org.apache.tajo.common.TajoDataTypes.Type;
    +import org.apache.tajo.util.Bytes;
    --- End diff --
    
    It is removed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] tajo pull request: TAJO-1131: Supports Inserting or Creating table...

Posted by babokim <gi...@git.apache.org>.
Github user babokim commented on a diff in the pull request:

    https://github.com/apache/tajo/pull/232#discussion_r20343179
  
    --- Diff: tajo-storage/src/main/java/org/apache/tajo/storage/hbase/ColumnMapping.java ---
    @@ -0,0 +1,239 @@
    +/**
    + * Licensed to the Apache Software Foundation (ASF) under one
    + * or more contributor license agreements.  See the NOTICE file
    + * distributed with this work for additional information
    + * regarding copyright ownership.  The ASF licenses this file
    + * to you under the Apache License, Version 2.0 (the
    + * "License"); you may not use this file except in compliance
    + * with the License.  You may obtain a copy of the License at
    + *
    + *     http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +
    +package org.apache.tajo.storage.hbase;
    +
    +import org.apache.hadoop.hbase.util.Bytes;
    +import org.apache.tajo.catalog.Schema;
    +import org.apache.tajo.catalog.TableMeta;
    +import org.apache.tajo.util.BytesUtils;
    +
    +import java.io.IOException;
    +import java.util.ArrayList;
    +import java.util.List;
    +
    +public class ColumnMapping {
    +  public static final String KEY_COLUMN_MAPPING = "key";
    +  public static final String VALUE_COLUMN_MAPPING = "value";
    +
    +  private TableMeta tableMeta;
    +  private Schema schema;
    +  private char rowKeyDelimiter;
    +
    +  private String hbaseTableName;
    +
    +  private int[] rowKeyFieldIndexes;
    +  private boolean[] isRowKeyMappings;
    +  private boolean[] isBinaryColumns;
    +  private boolean[] isColumnKeys;
    +  private boolean[] isColumnValues;
    +
    +  // schema order -> 0: cf name, 1: column name -> name bytes
    +  private byte[][][] mappingColumns;
    +
    +  private int numRowKeys;
    +
    +  public ColumnMapping(Schema schema, TableMeta tableMeta) throws IOException {
    +    this.schema = schema;
    +    this.tableMeta = tableMeta;
    +
    +    init();
    +  }
    +
    +  public void init() throws IOException {
    +    hbaseTableName = tableMeta.getOption(HBaseStorageManager.META_TABLE_KEY);
    +    String delim = tableMeta.getOption(HBaseStorageManager.META_ROWKEY_DELIMITER, "").trim();
    +    if (delim.length() > 0) {
    +      rowKeyDelimiter = delim.charAt(0);
    +    }
    +    isRowKeyMappings = new boolean[schema.size()];
    +    rowKeyFieldIndexes = new int[schema.size()];
    +    isBinaryColumns = new boolean[schema.size()];
    +    isColumnKeys = new boolean[schema.size()];
    +    isColumnValues = new boolean[schema.size()];
    +
    +    mappingColumns = new byte[schema.size()][][];
    +
    +    for (int i = 0; i < schema.size(); i++) {
    +      rowKeyFieldIndexes[i] = -1;
    +    }
    +
    +    String columnMapping = tableMeta.getOption(HBaseStorageManager.META_COLUMNS_KEY, "");
    +    if (columnMapping == null || columnMapping.isEmpty()) {
    +      throw new IOException("'columns' property is required.");
    +    }
    +
    +    String[] columnMappingTokens = columnMapping.split(",");
    +
    +    if (columnMappingTokens.length != schema.getColumns().size()) {
    +      throw new IOException("The number of mapped HBase columns is great than the number of Tajo table columns");
    +    }
    +
    +    int index = 0;
    +    for (String eachToken: columnMappingTokens) {
    +      mappingColumns[index] = new byte[2][];
    +
    +      byte[][] mappingTokens = BytesUtils.splitPreserveAllTokens(eachToken.trim().getBytes(), ':');
    +
    +      if (mappingTokens.length == 3) {
    --- End diff --
    
    GlobalEngine calls StorageManager.beforeInsertOrCATS() before running a insertion or creation query. In beforeInsertOrCATS() method ColumnMapping is checked.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] tajo pull request: TAJO-1131: Supports Inserting or Creating table...

Posted by hyunsik <gi...@git.apache.org>.
Github user hyunsik commented on a diff in the pull request:

    https://github.com/apache/tajo/pull/232#discussion_r20282564
  
    --- Diff: tajo-core/src/test/java/org/apache/tajo/engine/function/TestStringOperatorsAndFunctions.java ---
    @@ -613,4 +614,12 @@ public void testConcat_ws() throws IOException {
         testSimpleEval("select concat_ws(',', '22', '33', '33') ", new String[]{"22,33,33"});
         testSimpleEval("select concat_ws(',', null, '22') ", new String[]{"22"});
       }
    +
    +  @Test
    +  public void testToChar() throws IOException {
    +    DecimalFormat df = new DecimalFormat("0000000000");
    +    System.out.println(">>>>>>>>>>>" + df.format(1234));
    --- End diff --
    
    We may do not need ```System.out.println```.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] tajo pull request: TAJO-1131: Supports Inserting or Creating table...

Posted by hyunsik <gi...@git.apache.org>.
Github user hyunsik commented on a diff in the pull request:

    https://github.com/apache/tajo/pull/232#discussion_r20279190
  
    --- Diff: tajo-common/src/main/java/org/apache/tajo/conf/TajoConf.java ---
    @@ -302,6 +302,8 @@ public static int setDateOrder(int dateOrder) {
         HISTORY_EXPIRY_TIME_DAY("tajo.history.expiry-time-day", 7),
     
         // Misc -------------------------------------------------------------------
    +    // Fragment
    +    FRAGMENT_ALTERNATIVE_UNKNOWN_LENGTH("tajo.fragment.alternative.unknown.length", (long)(512 * 1024 * 1024)),
    --- End diff --
    
    Could you add some comments? It is hard to imagine its purpose from its config key name.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] tajo pull request: TAJO-1131: Supports Inserting or Creating table...

Posted by babokim <gi...@git.apache.org>.
Github user babokim closed the pull request at:

    https://github.com/apache/tajo/pull/232


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] tajo pull request: TAJO-1131: Supports Inserting or Creating table...

Posted by babokim <gi...@git.apache.org>.
Github user babokim commented on a diff in the pull request:

    https://github.com/apache/tajo/pull/232#discussion_r20343875
  
    --- Diff: tajo-storage/src/test/java/org/apache/tajo/storage/hbase/TestColumnMapping.java ---
    @@ -0,0 +1,97 @@
    +/**
    + * Licensed to the Apache Software Foundation (ASF) under one
    + * or more contributor license agreements.  See the NOTICE file
    + * distributed with this work for additional information
    + * regarding copyright ownership.  The ASF licenses this file
    + * to you under the Apache License, Version 2.0 (the
    + * "License"); you may not use this file except in compliance
    + * with the License.  You may obtain a copy of the License at
    + *
    + *     http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +
    +package org.apache.tajo.storage.hbase;
    +
    +import org.apache.hadoop.hbase.protobuf.generated.HBaseProtos.RegionInfo;
    --- End diff --
    
    It is removed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] tajo pull request: TAJO-1131: Supports Inserting or Creating table...

Posted by hyunsik <gi...@git.apache.org>.
Github user hyunsik commented on a diff in the pull request:

    https://github.com/apache/tajo/pull/232#discussion_r20278763
  
    --- Diff: tajo-storage/src/test/java/org/apache/tajo/storage/hbase/TestColumnMapping.java ---
    @@ -0,0 +1,97 @@
    +/**
    + * Licensed to the Apache Software Foundation (ASF) under one
    + * or more contributor license agreements.  See the NOTICE file
    + * distributed with this work for additional information
    + * regarding copyright ownership.  The ASF licenses this file
    + * to you under the Apache License, Version 2.0 (the
    + * "License"); you may not use this file except in compliance
    + * with the License.  You may obtain a copy of the License at
    + *
    + *     http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +
    +package org.apache.tajo.storage.hbase;
    +
    +import org.apache.hadoop.hbase.protobuf.generated.HBaseProtos.RegionInfo;
    --- End diff --
    
    It's an unused import.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] tajo pull request: TAJO-1131: Supports Inserting or Creating table...

Posted by hyunsik <gi...@git.apache.org>.
Github user hyunsik commented on a diff in the pull request:

    https://github.com/apache/tajo/pull/232#discussion_r20343154
  
    --- Diff: tajo-storage/src/main/java/org/apache/tajo/storage/hbase/HBaseScanner.java ---
    @@ -0,0 +1,445 @@
    +/**
    + * Licensed to the Apache Software Foundation (ASF) under one
    + * or more contributor license agreements.  See the NOTICE file
    + * distributed with this work for additional information
    + * regarding copyright ownership.  The ASF licenses this file
    + * to you under the Apache License, Version 2.0 (the
    + * "License"); you may not use this file except in compliance
    + * with the License.  You may obtain a copy of the License at
    + *
    + *     http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +
    +package org.apache.tajo.storage.hbase;
    +
    +import org.apache.commons.logging.Log;
    +import org.apache.commons.logging.LogFactory;
    +import org.apache.hadoop.conf.Configuration;
    +import org.apache.hadoop.hbase.client.*;
    +import org.apache.hadoop.hbase.filter.*;
    +import org.apache.tajo.catalog.Column;
    +import org.apache.tajo.catalog.Schema;
    +import org.apache.tajo.catalog.TableMeta;
    +import org.apache.tajo.catalog.proto.CatalogProtos.StoreType;
    +import org.apache.tajo.catalog.statistics.ColumnStats;
    +import org.apache.tajo.catalog.statistics.TableStats;
    +import org.apache.tajo.conf.TajoConf;
    +import org.apache.tajo.datum.Datum;
    +import org.apache.tajo.datum.NullDatum;
    +import org.apache.tajo.datum.TextDatum;
    +import org.apache.tajo.storage.Scanner;
    +import org.apache.tajo.storage.StorageManager;
    +import org.apache.tajo.storage.Tuple;
    +import org.apache.tajo.storage.VTuple;
    +import org.apache.tajo.storage.fragment.Fragment;
    +import org.apache.tajo.util.BytesUtils;
    +
    +import java.io.IOException;
    +import java.util.Collection;
    +import java.util.NavigableMap;
    +import java.util.Set;
    +import java.util.concurrent.atomic.AtomicBoolean;
    +
    +public class HBaseScanner implements Scanner {
    +  private static final Log LOG = LogFactory.getLog(HBaseScanner.class);
    +  private static final int DEFAULT_FETCH_SIZE = 1000;
    +  private static final int MAX_LIST_SIZE = 100;
    +
    +  protected boolean inited = false;
    +  private TajoConf conf;
    +  private Schema schema;
    +  private TableMeta meta;
    +  private HBaseFragment fragment;
    +  private Scan scan;
    +  private HTableInterface htable;
    +  private Configuration hbaseConf;
    +  private Column[] targets;
    +  private TableStats tableStats;
    +  private ResultScanner scanner;
    +  private AtomicBoolean finished = new AtomicBoolean(false);
    +  private float progress = 0.0f;
    +  private int scanFetchSize;
    +  private Result[] scanResults;
    +  private int scanResultIndex = -1;
    +  private Column[] schemaColumns;
    +
    +  private ColumnMapping columnMapping;
    +  private int[] targetIndexes;
    +
    +  private int numRows = 0;
    +
    +  private byte[][][] mappingColumnFamilies;
    +  private boolean[] isRowKeyMappings;
    +  private boolean[] isBinaryColumns;
    +  private boolean[] isColumnKeys;
    +  private boolean[] isColumnValues;
    +
    +  private int[] rowKeyFieldIndexes;
    +  private char rowKeyDelimiter;
    +
    +  public HBaseScanner (Configuration conf, Schema schema, TableMeta meta, Fragment fragment) throws IOException {
    +    this.conf = (TajoConf)conf;
    +    this.schema = schema;
    +    this.meta = meta;
    +    this.fragment = (HBaseFragment)fragment;
    +    this.tableStats = new TableStats();
    +  }
    +
    +  @Override
    +  public void init() throws IOException {
    +    inited = true;
    +    schemaColumns = schema.toArray();
    +    if (fragment != null) {
    +      tableStats.setNumBytes(0);
    +      tableStats.setNumBlocks(1);
    +    }
    +    if (schema != null) {
    +      for(Column eachColumn: schema.getColumns()) {
    +        ColumnStats columnStats = new ColumnStats(eachColumn);
    +        tableStats.addColumnStat(columnStats);
    +      }
    +    }
    +
    +    scanFetchSize = Integer.parseInt(meta.getOption("hbase.scanner.fetch,size", "" + DEFAULT_FETCH_SIZE));
    --- End diff --
    
    ```hbase.scanner.fetch,size``` includes comma (,). It seems to be a typo error. 
    
    Additionally, in my opinion, it is hard to imagine its exact meaning (e.g., the number of rows or the number of bytes) from its name. 
    
    If you don't mind, I'd like to suggest the table property name ```fetch.rownum```. I think that this table property will be also helpful for JDBC-based storages if my understanding is right.
    
    Thanks!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] tajo pull request: TAJO-1131: Supports Inserting or Creating table...

Posted by hyunsik <gi...@git.apache.org>.
Github user hyunsik commented on the pull request:

    https://github.com/apache/tajo/pull/232#issuecomment-63287704
  
    @babokim Could you close this issue? The 'ticket close' command in git commit log seems to work in only ```master``` branch.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] tajo pull request: TAJO-1131: Supports Inserting or Creating table...

Posted by babokim <gi...@git.apache.org>.
Github user babokim commented on a diff in the pull request:

    https://github.com/apache/tajo/pull/232#discussion_r20339040
  
    --- Diff: tajo-plan/src/main/java/org/apache/tajo/plan/logical/SortNode.java ---
    @@ -25,10 +25,16 @@
     import org.apache.tajo.util.TUtil;
     
     public final class SortNode extends UnaryNode implements Cloneable {
    +  public static enum SortRangeType {
    +    USING_SHUFFLE_DATA,
    --- End diff --
    
    I totally agree your opinion. 
    There is one question I think the naming rule of Enum type is the same as Class. Applying this rule "SORT_PURPOSE" should be "SortPurpose".


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] tajo pull request: TAJO-1131: Supports Inserting or Creating table...

Posted by babokim <gi...@git.apache.org>.
Github user babokim commented on a diff in the pull request:

    https://github.com/apache/tajo/pull/232#discussion_r20343893
  
    --- Diff: tajo-storage/src/main/java/org/apache/tajo/storage/StorageManager.java ---
    @@ -76,20 +98,169 @@
       private static final Map<Class<?>, Constructor<?>> CONSTRUCTOR_CACHE =
           new ConcurrentHashMap<Class<?>, Constructor<?>>();
     
    -  protected abstract void storageInit() throws IOException ;
    -  public abstract void createTable(TableDesc tableDesc) throws IOException;
    +  public StorageManager(StoreType storeType) {
    +    this.storeType = storeType;
    +  }
    +
    +  /**
    +   * Initialize storage manager.
    +   * @throws IOException
    +   */
    +  protected abstract void storageInit() throws IOException;
    +
    +  /**
    +   * This method is called after executing "CREATE TABLE" statement.
    +   * If a storage is a file based storage, a storage manager may create directory.
    +   *
    +   * @param tableDesc Table description which is created.
    +   * @param ifNotExists Creates the table only when the table does not exist.
    +   * @throws IOException
    +   */
    +  public abstract void createTable(TableDesc tableDesc, boolean ifNotExists) throws IOException;
    +
    +  /**
    +   * This method is called after executing "DROP TABLE" statement with the 'PURGE' option
    +   * which is the option to delete all the data.
    +   *
    +   * @param tableDesc
    +   * @throws IOException
    +   */
       public abstract void purgeTable(TableDesc tableDesc) throws IOException;
    -  public abstract List<Fragment> getSplits(String fragmentId, TableDesc tableDesc) throws IOException;
     
    +  /**
    +   * Returns the splits that will serve as input for the scan tasks. The
    +   * number of splits matches the number of regions in a table.
    +   * @param fragmentId The table name or previous ExecutionBlockId
    +   * @param tableDesc The table description for the target data.
    +   * @param scanNode The logical node for scanning.
    +   * @return The list of input fragments.
    +   * @throws IOException
    +   */
    +  public abstract List<Fragment> getSplits(String fragmentId, TableDesc tableDesc,
    +                                           ScanNode scanNode) throws IOException;
    +
    +  /**
    +   * It returns the splits that will serve as input for the non-forward query scanner such as 'select * from table1'.
    +   * The result list should be small. If there is many fragments for scanning, TajoMaster uses the paging navigation.
    +   * @param tableDesc The table description for the target data.
    +   * @param currentPage The current page number within the entire list.
    +   * @param numFragments The number of fragments in the result.
    +   * @return The list of input fragments.
    +   * @throws IOException
    +   */
    +  public abstract List<Fragment> getNonForwardSplit(TableDesc tableDesc, int currentPage, int numFragments)
    +      throws IOException;
    +
    +  /**
    +   * It returns the storage property.
    +   * @return The storage property
    +   */
    +  public abstract StorageProperty getStorageProperty();
    +
    +  /**
    +   * Release storage manager resource
    +   */
    +  public abstract void closeStorageManager();
    +
    +  /**
    +   * It is called by a Repartitioner for range shuffling when the SortRangeType of SortNode is USING_STORAGE_MANAGER.
    +   * In general Repartitioner determines the partition range using previous output statistics data.
    +   * In the special cases, such as HBase Repartitioner uses the result of this method.
    +   *
    +   * @param queryContext The current query context which contains query properties.
    +   * @param tableDesc The table description for the target data.
    +   * @param inputSchema The input schema
    +   * @param sortSpecs The sort specification that contains the sort column and sort order.
    +   * @return The list of sort ranges.
    +   * @throws IOException
    +   */
    +  public abstract TupleRange[] getInsertSortRanges(OverridableConf queryContext, TableDesc tableDesc,
    +                                                   Schema inputSchema, SortSpec[] sortSpecs,
    +                                                   TupleRange dataRange) throws IOException;
    +
    +  /**
    +   * This method is called before executing 'INSERT' or 'CREATE TABLE as SELECT'.
    +   * In general Tajo creates the target table after finishing the final sub-query of CATS.
    +   * But In the special cases, such as HBase INSERT or CAST query uses the target table information.
    +   * That kind of the storage should implements the logic related to creating table in this method.
    +   *
    +   * @param node The child node of the root node.
    +   * @throws IOException
    +   */
    +  public abstract void beforeInsertOrCATS(LogicalNode node) throws IOException;
    +
    +  /**
    +   * It is called when the query failed.
    +   * Each storage manager should implement to be processed when the query fails in this method.
    +   *
    +   * @param node The child node of the root node.
    +   * @throws IOException
    +   */
    +  public abstract void queryFailed(LogicalNode node) throws IOException;
    --- End diff --
    
    I will change to 'rollbackOutputCommit'


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] tajo pull request: TAJO-1131: Supports Inserting or Creating table...

Posted by hyunsik <gi...@git.apache.org>.
Github user hyunsik commented on the pull request:

    https://github.com/apache/tajo/pull/232#issuecomment-62871918
  
    This is awesome work! This feature will make Tajo much better in various workloads and use cases.
    
    I leave comments. In addition, the patch contains not few unused imports. Please remove out them.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] tajo pull request: TAJO-1131: Supports Inserting or Creating table...

Posted by babokim <gi...@git.apache.org>.
Github user babokim commented on a diff in the pull request:

    https://github.com/apache/tajo/pull/232#discussion_r20338252
  
    --- Diff: tajo-common/src/main/java/org/apache/tajo/conf/TajoConf.java ---
    @@ -302,6 +302,8 @@ public static int setDateOrder(int dateOrder) {
         HISTORY_EXPIRY_TIME_DAY("tajo.history.expiry-time-day", 7),
     
         // Misc -------------------------------------------------------------------
    +    // Fragment
    +    FRAGMENT_ALTERNATIVE_UNKNOWN_LENGTH("tajo.fragment.alternative.unknown.length", (long)(512 * 1024 * 1024)),
    --- End diff --
    
    When making physical plan, the length of fragment is used to determine the physical operation.
    But some storage does not know the size of the fragment. In this case PhysicalPlanner uses this value to determine.
    I will add comments.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] tajo pull request: TAJO-1131: Supports Inserting or Creating table...

Posted by hyunsik <gi...@git.apache.org>.
Github user hyunsik commented on a diff in the pull request:

    https://github.com/apache/tajo/pull/232#discussion_r20281319
  
    --- Diff: tajo-storage/src/main/java/org/apache/tajo/storage/hbase/HBaseScanner.java ---
    @@ -0,0 +1,445 @@
    +/**
    + * Licensed to the Apache Software Foundation (ASF) under one
    + * or more contributor license agreements.  See the NOTICE file
    + * distributed with this work for additional information
    + * regarding copyright ownership.  The ASF licenses this file
    + * to you under the Apache License, Version 2.0 (the
    + * "License"); you may not use this file except in compliance
    + * with the License.  You may obtain a copy of the License at
    + *
    + *     http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +
    +package org.apache.tajo.storage.hbase;
    +
    +import org.apache.commons.logging.Log;
    +import org.apache.commons.logging.LogFactory;
    +import org.apache.hadoop.conf.Configuration;
    +import org.apache.hadoop.hbase.client.*;
    +import org.apache.hadoop.hbase.filter.*;
    +import org.apache.tajo.catalog.Column;
    +import org.apache.tajo.catalog.Schema;
    +import org.apache.tajo.catalog.TableMeta;
    +import org.apache.tajo.catalog.proto.CatalogProtos.StoreType;
    +import org.apache.tajo.catalog.statistics.ColumnStats;
    +import org.apache.tajo.catalog.statistics.TableStats;
    +import org.apache.tajo.conf.TajoConf;
    +import org.apache.tajo.datum.Datum;
    +import org.apache.tajo.datum.NullDatum;
    +import org.apache.tajo.datum.TextDatum;
    +import org.apache.tajo.storage.Scanner;
    +import org.apache.tajo.storage.StorageManager;
    +import org.apache.tajo.storage.Tuple;
    +import org.apache.tajo.storage.VTuple;
    +import org.apache.tajo.storage.fragment.Fragment;
    +import org.apache.tajo.util.BytesUtils;
    +
    +import java.io.IOException;
    +import java.util.Collection;
    +import java.util.NavigableMap;
    +import java.util.Set;
    +import java.util.concurrent.atomic.AtomicBoolean;
    +
    +public class HBaseScanner implements Scanner {
    +  private static final Log LOG = LogFactory.getLog(HBaseScanner.class);
    +  private static final int DEFAULT_FETCH_SIZE = 1000;
    +  private static final int MAX_LIST_SIZE = 100;
    +
    +  protected boolean inited = false;
    +  private TajoConf conf;
    +  private Schema schema;
    +  private TableMeta meta;
    +  private HBaseFragment fragment;
    +  private Scan scan;
    +  private HTableInterface htable;
    +  private Configuration hbaseConf;
    +  private Column[] targets;
    +  private TableStats tableStats;
    +  private ResultScanner scanner;
    +  private AtomicBoolean finished = new AtomicBoolean(false);
    +  private float progress = 0.0f;
    +  private int scanFetchSize;
    +  private Result[] scanResults;
    +  private int scanResultIndex = -1;
    +  private Column[] schemaColumns;
    +
    +  private ColumnMapping columnMapping;
    +  private int[] targetIndexes;
    +
    +  private int numRows = 0;
    +
    +  private byte[][][] mappingColumnFamilies;
    +  private boolean[] isRowKeyMappings;
    +  private boolean[] isBinaryColumns;
    +  private boolean[] isColumnKeys;
    +  private boolean[] isColumnValues;
    +
    +  private int[] rowKeyFieldIndexes;
    +  private char rowKeyDelimiter;
    +
    +  public HBaseScanner (Configuration conf, Schema schema, TableMeta meta, Fragment fragment) throws IOException {
    +    this.conf = (TajoConf)conf;
    +    this.schema = schema;
    +    this.meta = meta;
    +    this.fragment = (HBaseFragment)fragment;
    +    this.tableStats = new TableStats();
    +  }
    +
    +  @Override
    +  public void init() throws IOException {
    +    inited = true;
    +    schemaColumns = schema.toArray();
    +    if (fragment != null) {
    +      tableStats.setNumBytes(0);
    +      tableStats.setNumBlocks(1);
    +    }
    +    if (schema != null) {
    +      for(Column eachColumn: schema.getColumns()) {
    +        ColumnStats columnStats = new ColumnStats(eachColumn);
    +        tableStats.addColumnStat(columnStats);
    +      }
    +    }
    +
    +    scanFetchSize = Integer.parseInt(meta.getOption("hbase.scanner.fetch,size", "" + DEFAULT_FETCH_SIZE));
    --- End diff --
    
    It would be better if ```hbase.scanner.fetch,size``` is used as a constant value instead of a string value.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] tajo pull request: TAJO-1131: Supports Inserting or Creating table...

Posted by babokim <gi...@git.apache.org>.
Github user babokim commented on the pull request:

    https://github.com/apache/tajo/pull/232#issuecomment-63016759
  
    @hyunsik Thanks for your review. I pushed the fetch which is applied your comment.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] tajo pull request: TAJO-1131: Supports Inserting or Creating table...

Posted by babokim <gi...@git.apache.org>.
Github user babokim commented on a diff in the pull request:

    https://github.com/apache/tajo/pull/232#discussion_r20343234
  
    --- Diff: tajo-storage/src/main/java/org/apache/tajo/storage/hbase/HBaseScanner.java ---
    @@ -0,0 +1,445 @@
    +/**
    + * Licensed to the Apache Software Foundation (ASF) under one
    + * or more contributor license agreements.  See the NOTICE file
    + * distributed with this work for additional information
    + * regarding copyright ownership.  The ASF licenses this file
    + * to you under the Apache License, Version 2.0 (the
    + * "License"); you may not use this file except in compliance
    + * with the License.  You may obtain a copy of the License at
    + *
    + *     http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +
    +package org.apache.tajo.storage.hbase;
    +
    +import org.apache.commons.logging.Log;
    +import org.apache.commons.logging.LogFactory;
    +import org.apache.hadoop.conf.Configuration;
    +import org.apache.hadoop.hbase.client.*;
    +import org.apache.hadoop.hbase.filter.*;
    +import org.apache.tajo.catalog.Column;
    +import org.apache.tajo.catalog.Schema;
    +import org.apache.tajo.catalog.TableMeta;
    +import org.apache.tajo.catalog.proto.CatalogProtos.StoreType;
    +import org.apache.tajo.catalog.statistics.ColumnStats;
    +import org.apache.tajo.catalog.statistics.TableStats;
    +import org.apache.tajo.conf.TajoConf;
    +import org.apache.tajo.datum.Datum;
    +import org.apache.tajo.datum.NullDatum;
    +import org.apache.tajo.datum.TextDatum;
    +import org.apache.tajo.storage.Scanner;
    +import org.apache.tajo.storage.StorageManager;
    +import org.apache.tajo.storage.Tuple;
    +import org.apache.tajo.storage.VTuple;
    +import org.apache.tajo.storage.fragment.Fragment;
    +import org.apache.tajo.util.BytesUtils;
    +
    +import java.io.IOException;
    +import java.util.Collection;
    +import java.util.NavigableMap;
    +import java.util.Set;
    +import java.util.concurrent.atomic.AtomicBoolean;
    +
    +public class HBaseScanner implements Scanner {
    +  private static final Log LOG = LogFactory.getLog(HBaseScanner.class);
    +  private static final int DEFAULT_FETCH_SIZE = 1000;
    +  private static final int MAX_LIST_SIZE = 100;
    +
    +  protected boolean inited = false;
    +  private TajoConf conf;
    +  private Schema schema;
    +  private TableMeta meta;
    +  private HBaseFragment fragment;
    +  private Scan scan;
    +  private HTableInterface htable;
    +  private Configuration hbaseConf;
    +  private Column[] targets;
    +  private TableStats tableStats;
    +  private ResultScanner scanner;
    +  private AtomicBoolean finished = new AtomicBoolean(false);
    +  private float progress = 0.0f;
    +  private int scanFetchSize;
    +  private Result[] scanResults;
    +  private int scanResultIndex = -1;
    +  private Column[] schemaColumns;
    +
    +  private ColumnMapping columnMapping;
    +  private int[] targetIndexes;
    +
    +  private int numRows = 0;
    +
    +  private byte[][][] mappingColumnFamilies;
    +  private boolean[] isRowKeyMappings;
    +  private boolean[] isBinaryColumns;
    +  private boolean[] isColumnKeys;
    +  private boolean[] isColumnValues;
    +
    +  private int[] rowKeyFieldIndexes;
    +  private char rowKeyDelimiter;
    +
    +  public HBaseScanner (Configuration conf, Schema schema, TableMeta meta, Fragment fragment) throws IOException {
    +    this.conf = (TajoConf)conf;
    +    this.schema = schema;
    +    this.meta = meta;
    +    this.fragment = (HBaseFragment)fragment;
    +    this.tableStats = new TableStats();
    +  }
    +
    +  @Override
    +  public void init() throws IOException {
    +    inited = true;
    +    schemaColumns = schema.toArray();
    +    if (fragment != null) {
    +      tableStats.setNumBytes(0);
    +      tableStats.setNumBlocks(1);
    +    }
    +    if (schema != null) {
    +      for(Column eachColumn: schema.getColumns()) {
    +        ColumnStats columnStats = new ColumnStats(eachColumn);
    +        tableStats.addColumnStat(columnStats);
    +      }
    +    }
    +
    +    scanFetchSize = Integer.parseInt(meta.getOption("hbase.scanner.fetch,size", "" + DEFAULT_FETCH_SIZE));
    --- End diff --
    
    I will change constant value. And all constant variables for HBase storage are placed in HBaseStorageConstants class.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] tajo pull request: TAJO-1131: Supports Inserting or Creating table...

Posted by hyunsik <gi...@git.apache.org>.
Github user hyunsik commented on a diff in the pull request:

    https://github.com/apache/tajo/pull/232#discussion_r20277970
  
    --- Diff: tajo-common/src/main/java/org/apache/tajo/util/TUtil.java ---
    @@ -119,6 +119,15 @@ public static boolean checkEquals(Object [] s1, Object [] s2) {
         return new ArrayList<T>();
       }
     
    +  public static List<Integer> newIntList(int...items) {
    --- End diff --
    
    It seems to be unused.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] tajo pull request: TAJO-1131: Supports Inserting or Creating table...

Posted by babokim <gi...@git.apache.org>.
Github user babokim commented on a diff in the pull request:

    https://github.com/apache/tajo/pull/232#discussion_r20343799
  
    --- Diff: tajo-core/src/test/java/org/apache/tajo/engine/function/TestStringOperatorsAndFunctions.java ---
    @@ -613,4 +614,12 @@ public void testConcat_ws() throws IOException {
         testSimpleEval("select concat_ws(',', '22', '33', '33') ", new String[]{"22,33,33"});
         testSimpleEval("select concat_ws(',', null, '22') ", new String[]{"22"});
       }
    +
    +  @Test
    +  public void testToChar() throws IOException {
    +    DecimalFormat df = new DecimalFormat("0000000000");
    +    System.out.println(">>>>>>>>>>>" + df.format(1234));
    --- End diff --
    
    This is a temporary code. I will remove it.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---