You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@tajo.apache.org by Jung JaeHwa <jh...@gruter.com> on 2014/03/19 16:45:02 UTC

Review Request 19406: TAJO-616: SequenceFile support

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/19406/
-----------------------------------------------------------

Review request for Tajo.


Bugs: TAJO-616
    https://issues.apache.org/jira/browse/TAJO-616


Repository: tajo


Description
-------

Hadoop users can create a sequence file format, and hive users also can create a table which was stored as SequenceFIle. So, we need to support sequence file format as follows:

* Update parser and catalog for SequenceFile
* Make serializer/deserializer configurable in SequenceFile
* Scanner for SequenceFile
* Support for compression/decompression of SequenceFile
* Compatible to apache hive


Diffs
-----

  tajo-catalog/tajo-catalog-common/src/main/java/org/apache/tajo/catalog/CatalogConstants.java fa23da1 
  tajo-catalog/tajo-catalog-common/src/main/java/org/apache/tajo/catalog/CatalogUtil.java 2a6727e 
  tajo-catalog/tajo-catalog-common/src/main/proto/CatalogProtos.proto 35171cc 
  tajo-catalog/tajo-catalog-drivers/tajo-hcatalog/src/main/java/org/apache/tajo/catalog/store/HCatalogStore.java 5387673 
  tajo-catalog/tajo-catalog-drivers/tajo-hcatalog/src/main/java/org/apache/tajo/catalog/store/HCatalogUtil.java fc2fdda 
  tajo-catalog/tajo-catalog-drivers/tajo-hcatalog/src/test/java/org/apache/tajo/catalog/store/TestHCatalogStore.java cb3973a 
  tajo-common/src/main/java/org/apache/tajo/util/Bytes.java f9ba923 
  tajo-core/tajo-core-backend/src/test/java/org/apache/tajo/engine/planner/physical/TestPhysicalPlanner.java 1975a57 
  tajo-storage/src/main/java/org/apache/tajo/storage/sequencefile/SequenceFileAppender.java PRE-CREATION 
  tajo-storage/src/main/java/org/apache/tajo/storage/sequencefile/SequenceFileScanner.java PRE-CREATION 
  tajo-storage/src/main/resources/storage-default.xml 5bf4453 
  tajo-storage/src/test/java/org/apache/tajo/storage/TestCompressionStorages.java a776eb6 
  tajo-storage/src/test/java/org/apache/tajo/storage/TestMergeScanner.java f2a66d9 
  tajo-storage/src/test/java/org/apache/tajo/storage/TestStorages.java 1b36b78 
  tajo-storage/src/test/java/org/apache/tajo/storage/v2/TestStorages.java 05be40a 
  tajo-storage/src/test/resources/storage-default.xml 304af10 

Diff: https://reviews.apache.org/r/19406/diff/


Testing
-------

mvn clean install -Phcatalog-0.12.0


Thanks,

Jung JaeHwa

Re: Review Request 19406: TAJO-616: SequenceFile support

Posted by Jung JaeHwa <jh...@gruter.com>.

> On March 23, 2014, 7:37 a.m., Jihoon Son wrote:
> > Hi, JaeHwa.
> > This patch looks great to me. I left just one question below. 
> > 
> > Also, I have a suggestion about the test.
> > Although the null data generally occur in real applications, I think that our tests are not enough to test the null handling of Tajo.
> > Would you add a test to evaluate the null handling of SequenceFileAppender and SequenceFileScanner?

Thanks Jihoon.

It's good idea. I'll update my patch to evaluate the null handling of Tajo. :)


> On March 23, 2014, 7:37 a.m., Jihoon Son wrote:
> > tajo-catalog/tajo-catalog-drivers/tajo-hcatalog/src/main/java/org/apache/tajo/catalog/store/HCatalogStore.java, line 421
> > <https://reviews.apache.org/r/19406/diff/1/?file=528051#file528051line421>
> >
> >     Do you have any reasons of using unmatched classes for input format class and output format class?

Because these are default configurations in hive.
If we create sequencefile format table, hive sets configurations as follows:
STORED AS INPUTFORMAT 
  'org.apache.hadoop.mapred.SequenceFileInputFormat' 
OUTPUTFORMAT 
  'org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat'


- Jung


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/19406/#review38250
-----------------------------------------------------------


On March 19, 2014, 3:44 p.m., Jung JaeHwa wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/19406/
> -----------------------------------------------------------
> 
> (Updated March 19, 2014, 3:44 p.m.)
> 
> 
> Review request for Tajo.
> 
> 
> Bugs: TAJO-616
>     https://issues.apache.org/jira/browse/TAJO-616
> 
> 
> Repository: tajo
> 
> 
> Description
> -------
> 
> Hadoop users can create a sequence file format, and hive users also can create a table which was stored as SequenceFIle. So, we need to support sequence file format as follows:
> 
> * Update parser and catalog for SequenceFile
> * Make serializer/deserializer configurable in SequenceFile
> * Scanner for SequenceFile
> * Support for compression/decompression of SequenceFile
> * Compatible to apache hive
> 
> 
> Diffs
> -----
> 
>   tajo-catalog/tajo-catalog-common/src/main/java/org/apache/tajo/catalog/CatalogConstants.java fa23da1 
>   tajo-catalog/tajo-catalog-common/src/main/java/org/apache/tajo/catalog/CatalogUtil.java 2a6727e 
>   tajo-catalog/tajo-catalog-common/src/main/proto/CatalogProtos.proto 35171cc 
>   tajo-catalog/tajo-catalog-drivers/tajo-hcatalog/src/main/java/org/apache/tajo/catalog/store/HCatalogStore.java 5387673 
>   tajo-catalog/tajo-catalog-drivers/tajo-hcatalog/src/main/java/org/apache/tajo/catalog/store/HCatalogUtil.java fc2fdda 
>   tajo-catalog/tajo-catalog-drivers/tajo-hcatalog/src/test/java/org/apache/tajo/catalog/store/TestHCatalogStore.java cb3973a 
>   tajo-common/src/main/java/org/apache/tajo/util/Bytes.java f9ba923 
>   tajo-core/tajo-core-backend/src/test/java/org/apache/tajo/engine/planner/physical/TestPhysicalPlanner.java 1975a57 
>   tajo-storage/src/main/java/org/apache/tajo/storage/sequencefile/SequenceFileAppender.java PRE-CREATION 
>   tajo-storage/src/main/java/org/apache/tajo/storage/sequencefile/SequenceFileScanner.java PRE-CREATION 
>   tajo-storage/src/main/resources/storage-default.xml 5bf4453 
>   tajo-storage/src/test/java/org/apache/tajo/storage/TestCompressionStorages.java a776eb6 
>   tajo-storage/src/test/java/org/apache/tajo/storage/TestMergeScanner.java f2a66d9 
>   tajo-storage/src/test/java/org/apache/tajo/storage/TestStorages.java 1b36b78 
>   tajo-storage/src/test/java/org/apache/tajo/storage/v2/TestStorages.java 05be40a 
>   tajo-storage/src/test/resources/storage-default.xml 304af10 
> 
> Diff: https://reviews.apache.org/r/19406/diff/
> 
> 
> Testing
> -------
> 
> mvn clean install -Phcatalog-0.12.0
> 
> 
> Thanks,
> 
> Jung JaeHwa
> 
>

Re: Review Request 19406: TAJO-616: SequenceFile support

Posted by Jihoon Son <ji...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/19406/#review38250
-----------------------------------------------------------


Hi, JaeHwa.
This patch looks great to me. I left just one question below. 

Also, I have a suggestion about the test.
Although the null data generally occur in real applications, I think that our tests are not enough to test the null handling of Tajo.
Would you add a test to evaluate the null handling of SequenceFileAppender and SequenceFileScanner?


tajo-catalog/tajo-catalog-drivers/tajo-hcatalog/src/main/java/org/apache/tajo/catalog/store/HCatalogStore.java
<https://reviews.apache.org/r/19406/#comment70303>

    Do you have any reasons of using unmatched classes for input format class and output format class?


- Jihoon Son


On March 20, 2014, 12:44 a.m., Jung JaeHwa wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/19406/
> -----------------------------------------------------------
> 
> (Updated March 20, 2014, 12:44 a.m.)
> 
> 
> Review request for Tajo.
> 
> 
> Bugs: TAJO-616
>     https://issues.apache.org/jira/browse/TAJO-616
> 
> 
> Repository: tajo
> 
> 
> Description
> -------
> 
> Hadoop users can create a sequence file format, and hive users also can create a table which was stored as SequenceFIle. So, we need to support sequence file format as follows:
> 
> * Update parser and catalog for SequenceFile
> * Make serializer/deserializer configurable in SequenceFile
> * Scanner for SequenceFile
> * Support for compression/decompression of SequenceFile
> * Compatible to apache hive
> 
> 
> Diffs
> -----
> 
>   tajo-catalog/tajo-catalog-common/src/main/java/org/apache/tajo/catalog/CatalogConstants.java fa23da1 
>   tajo-catalog/tajo-catalog-common/src/main/java/org/apache/tajo/catalog/CatalogUtil.java 2a6727e 
>   tajo-catalog/tajo-catalog-common/src/main/proto/CatalogProtos.proto 35171cc 
>   tajo-catalog/tajo-catalog-drivers/tajo-hcatalog/src/main/java/org/apache/tajo/catalog/store/HCatalogStore.java 5387673 
>   tajo-catalog/tajo-catalog-drivers/tajo-hcatalog/src/main/java/org/apache/tajo/catalog/store/HCatalogUtil.java fc2fdda 
>   tajo-catalog/tajo-catalog-drivers/tajo-hcatalog/src/test/java/org/apache/tajo/catalog/store/TestHCatalogStore.java cb3973a 
>   tajo-common/src/main/java/org/apache/tajo/util/Bytes.java f9ba923 
>   tajo-core/tajo-core-backend/src/test/java/org/apache/tajo/engine/planner/physical/TestPhysicalPlanner.java 1975a57 
>   tajo-storage/src/main/java/org/apache/tajo/storage/sequencefile/SequenceFileAppender.java PRE-CREATION 
>   tajo-storage/src/main/java/org/apache/tajo/storage/sequencefile/SequenceFileScanner.java PRE-CREATION 
>   tajo-storage/src/main/resources/storage-default.xml 5bf4453 
>   tajo-storage/src/test/java/org/apache/tajo/storage/TestCompressionStorages.java a776eb6 
>   tajo-storage/src/test/java/org/apache/tajo/storage/TestMergeScanner.java f2a66d9 
>   tajo-storage/src/test/java/org/apache/tajo/storage/TestStorages.java 1b36b78 
>   tajo-storage/src/test/java/org/apache/tajo/storage/v2/TestStorages.java 05be40a 
>   tajo-storage/src/test/resources/storage-default.xml 304af10 
> 
> Diff: https://reviews.apache.org/r/19406/diff/
> 
> 
> Testing
> -------
> 
> mvn clean install -Phcatalog-0.12.0
> 
> 
> Thanks,
> 
> Jung JaeHwa
> 
>

Re: Review Request 19406: TAJO-616: SequenceFile support

Posted by Jinho Kim <jh...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/19406/#review39129
-----------------------------------------------------------


Thank you for quick fix. but the patch need to rebase
Could you fix the findbugs warning message in associated hcatalogstore ? 

- Jinho Kim


On March 31, 2014, 3:08 p.m., Jung JaeHwa wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/19406/
> -----------------------------------------------------------
> 
> (Updated March 31, 2014, 3:08 p.m.)
> 
> 
> Review request for Tajo.
> 
> 
> Bugs: TAJO-616
>     https://issues.apache.org/jira/browse/TAJO-616
> 
> 
> Repository: tajo
> 
> 
> Description
> -------
> 
> Hadoop users can create a sequence file format, and hive users also can create a table which was stored as SequenceFIle. So, we need to support sequence file format as follows:
> 
> * Update parser and catalog for SequenceFile
> * Make serializer/deserializer configurable in SequenceFile
> * Scanner for SequenceFile
> * Support for compression/decompression of SequenceFile
> * Compatible to apache hive
> 
> 
> Diffs
> -----
> 
>   tajo-catalog/tajo-catalog-common/src/main/java/org/apache/tajo/catalog/CatalogConstants.java c7116f6 
>   tajo-catalog/tajo-catalog-common/src/main/java/org/apache/tajo/catalog/CatalogUtil.java 9078e60 
>   tajo-catalog/tajo-catalog-common/src/main/proto/CatalogProtos.proto 06ffcfd 
>   tajo-catalog/tajo-catalog-drivers/tajo-hcatalog/src/main/java/org/apache/tajo/catalog/store/HCatalogStore.java 6ef255f 
>   tajo-catalog/tajo-catalog-drivers/tajo-hcatalog/src/main/java/org/apache/tajo/catalog/store/HCatalogUtil.java d426369 
>   tajo-catalog/tajo-catalog-drivers/tajo-hcatalog/src/test/java/org/apache/tajo/catalog/store/TestHCatalogStore.java b9a734e 
>   tajo-common/src/main/java/org/apache/tajo/util/Bytes.java f9ba923 
>   tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/benchmark/TPCH.java 20015b5 
>   tajo-core/tajo-core-backend/src/test/java/org/apache/tajo/TpchTestBase.java 0b80231 
>   tajo-core/tajo-core-backend/src/test/java/org/apache/tajo/client/TestDDLBuilder.java 73f7d5e 
>   tajo-core/tajo-core-backend/src/test/java/org/apache/tajo/engine/query/TestCTASQuery.java e6840f6 
>   tajo-core/tajo-core-backend/src/test/java/org/apache/tajo/engine/query/TestNullValues.java 1f1d9e1 
>   tajo-docs/src/main/sphinx/table_management/csv.rst PRE-CREATION 
>   tajo-docs/src/main/sphinx/table_management/file_formats.rst 26edac4 
>   tajo-docs/src/main/sphinx/table_management/parquet.rst PRE-CREATION 
>   tajo-docs/src/main/sphinx/table_management/rcfile.rst PRE-CREATION 
>   tajo-docs/src/main/sphinx/table_management/sequencefile.rst PRE-CREATION 
>   tajo-storage/pom.xml 9f144bb 
>   tajo-storage/src/main/java/org/apache/tajo/storage/CSVFile.java 116e25c 
>   tajo-storage/src/main/java/org/apache/tajo/storage/parquet/TajoRecordMaterializer.java cce7ead 
>   tajo-storage/src/main/java/org/apache/tajo/storage/rcfile/RCFile.java 2fd3455 
>   tajo-storage/src/main/java/org/apache/tajo/storage/sequencefile/SequenceFileAppender.java PRE-CREATION 
>   tajo-storage/src/main/java/org/apache/tajo/storage/sequencefile/SequenceFileScanner.java PRE-CREATION 
>   tajo-storage/src/main/java/org/apache/tajo/storage/trevni/TrevniScanner.java ed71730 
>   tajo-storage/src/main/java/org/apache/tajo/storage/v2/CSVFileScanner.java 79ddfdd 
>   tajo-storage/src/main/resources/storage-default.xml e3a0eaf 
>   tajo-storage/src/test/java/org/apache/tajo/storage/TestCompressionStorages.java a776eb6 
>   tajo-storage/src/test/java/org/apache/tajo/storage/TestMergeScanner.java ad12c06 
>   tajo-storage/src/test/java/org/apache/tajo/storage/TestStorages.java b5338ff 
>   tajo-storage/src/test/resources/storage-default.xml 75274d0 
> 
> Diff: https://reviews.apache.org/r/19406/diff/
> 
> 
> Testing
> -------
> 
> mvn clean install -Phcatalog-0.12.0
> mvn clean install -Phcatalog-0.12.0 -Dtajo.catalog.store.class=org.apache.tajo.catalog.store.HCatalogStore
> 
> 
> Thanks,
> 
> Jung JaeHwa
> 
>

Re: Review Request 19406: TAJO-616: SequenceFile support

Posted by Jung JaeHwa <jh...@gruter.com>.

> On April 1, 2014, 7:36 a.m., Jinho Kim wrote:
> > +1 
> > Looks great for me.  I've verified 'MysqlCatalogStore, HcatalogStore, TPCH-1'.
> > Thank you!!

Jinho, thank you very much.
I'll push it.


- Jung


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/19406/#review39135
-----------------------------------------------------------


On April 1, 2014, 7:03 a.m., Jung JaeHwa wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/19406/
> -----------------------------------------------------------
> 
> (Updated April 1, 2014, 7:03 a.m.)
> 
> 
> Review request for Tajo.
> 
> 
> Bugs: TAJO-616
>     https://issues.apache.org/jira/browse/TAJO-616
> 
> 
> Repository: tajo
> 
> 
> Description
> -------
> 
> Hadoop users can create a sequence file format, and hive users also can create a table which was stored as SequenceFIle. So, we need to support sequence file format as follows:
> 
> * Update parser and catalog for SequenceFile
> * Make serializer/deserializer configurable in SequenceFile
> * Scanner for SequenceFile
> * Support for compression/decompression of SequenceFile
> * Compatible to apache hive
> 
> 
> Diffs
> -----
> 
>   tajo-catalog/tajo-catalog-common/src/main/java/org/apache/tajo/catalog/CatalogConstants.java c7116f6 
>   tajo-catalog/tajo-catalog-common/src/main/java/org/apache/tajo/catalog/CatalogUtil.java 22f54ca 
>   tajo-catalog/tajo-catalog-common/src/main/proto/CatalogProtos.proto d3abc70 
>   tajo-catalog/tajo-catalog-drivers/tajo-hcatalog/src/main/java/org/apache/tajo/catalog/store/HCatalogStore.java a4bdb28 
>   tajo-catalog/tajo-catalog-drivers/tajo-hcatalog/src/main/java/org/apache/tajo/catalog/store/HCatalogUtil.java d426369 
>   tajo-catalog/tajo-catalog-drivers/tajo-hcatalog/src/test/java/org/apache/tajo/catalog/store/TestHCatalogStore.java b9a734e 
>   tajo-common/src/main/java/org/apache/tajo/util/Bytes.java f9ba923 
>   tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/benchmark/TPCH.java 20015b5 
>   tajo-core/tajo-core-backend/src/test/java/org/apache/tajo/TpchTestBase.java 0b80231 
>   tajo-core/tajo-core-backend/src/test/java/org/apache/tajo/client/TestDDLBuilder.java 73f7d5e 
>   tajo-core/tajo-core-backend/src/test/java/org/apache/tajo/engine/query/TestCTASQuery.java 72e92d2 
>   tajo-core/tajo-core-backend/src/test/java/org/apache/tajo/engine/query/TestNullValues.java 1f1d9e1 
>   tajo-docs/src/main/sphinx/table_management/csv.rst PRE-CREATION 
>   tajo-docs/src/main/sphinx/table_management/file_formats.rst 26edac4 
>   tajo-docs/src/main/sphinx/table_management/parquet.rst PRE-CREATION 
>   tajo-docs/src/main/sphinx/table_management/rcfile.rst PRE-CREATION 
>   tajo-docs/src/main/sphinx/table_management/sequencefile.rst PRE-CREATION 
>   tajo-storage/pom.xml 9f144bb 
>   tajo-storage/src/main/java/org/apache/tajo/storage/CSVFile.java 116e25c 
>   tajo-storage/src/main/java/org/apache/tajo/storage/parquet/TajoRecordMaterializer.java cce7ead 
>   tajo-storage/src/main/java/org/apache/tajo/storage/rcfile/RCFile.java 2fd3455 
>   tajo-storage/src/main/java/org/apache/tajo/storage/sequencefile/SequenceFileAppender.java PRE-CREATION 
>   tajo-storage/src/main/java/org/apache/tajo/storage/sequencefile/SequenceFileScanner.java PRE-CREATION 
>   tajo-storage/src/main/java/org/apache/tajo/storage/trevni/TrevniScanner.java ed71730 
>   tajo-storage/src/main/java/org/apache/tajo/storage/v2/CSVFileScanner.java 79ddfdd 
>   tajo-storage/src/main/resources/storage-default.xml e3a0eaf 
>   tajo-storage/src/test/java/org/apache/tajo/storage/TestCompressionStorages.java a776eb6 
>   tajo-storage/src/test/java/org/apache/tajo/storage/TestMergeScanner.java ad12c06 
>   tajo-storage/src/test/java/org/apache/tajo/storage/TestStorages.java b5338ff 
>   tajo-storage/src/test/resources/storage-default.xml 75274d0 
> 
> Diff: https://reviews.apache.org/r/19406/diff/
> 
> 
> Testing
> -------
> 
> mvn clean install -Phcatalog-0.12.0
> mvn clean install -Phcatalog-0.12.0 -Dtajo.catalog.store.class=org.apache.tajo.catalog.store.HCatalogStore
> 
> 
> Thanks,
> 
> Jung JaeHwa
> 
>

Re: Review Request 19406: TAJO-616: SequenceFile support

Posted by Jinho Kim <jh...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/19406/#review39135
-----------------------------------------------------------

Ship it!


+1 
Looks great for me.  I've verified 'MysqlCatalogStore, HcatalogStore, TPCH-1'.
Thank you!!

- Jinho Kim


On April 1, 2014, 7:03 a.m., Jung JaeHwa wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/19406/
> -----------------------------------------------------------
> 
> (Updated April 1, 2014, 7:03 a.m.)
> 
> 
> Review request for Tajo.
> 
> 
> Bugs: TAJO-616
>     https://issues.apache.org/jira/browse/TAJO-616
> 
> 
> Repository: tajo
> 
> 
> Description
> -------
> 
> Hadoop users can create a sequence file format, and hive users also can create a table which was stored as SequenceFIle. So, we need to support sequence file format as follows:
> 
> * Update parser and catalog for SequenceFile
> * Make serializer/deserializer configurable in SequenceFile
> * Scanner for SequenceFile
> * Support for compression/decompression of SequenceFile
> * Compatible to apache hive
> 
> 
> Diffs
> -----
> 
>   tajo-catalog/tajo-catalog-common/src/main/java/org/apache/tajo/catalog/CatalogConstants.java c7116f6 
>   tajo-catalog/tajo-catalog-common/src/main/java/org/apache/tajo/catalog/CatalogUtil.java 22f54ca 
>   tajo-catalog/tajo-catalog-common/src/main/proto/CatalogProtos.proto d3abc70 
>   tajo-catalog/tajo-catalog-drivers/tajo-hcatalog/src/main/java/org/apache/tajo/catalog/store/HCatalogStore.java a4bdb28 
>   tajo-catalog/tajo-catalog-drivers/tajo-hcatalog/src/main/java/org/apache/tajo/catalog/store/HCatalogUtil.java d426369 
>   tajo-catalog/tajo-catalog-drivers/tajo-hcatalog/src/test/java/org/apache/tajo/catalog/store/TestHCatalogStore.java b9a734e 
>   tajo-common/src/main/java/org/apache/tajo/util/Bytes.java f9ba923 
>   tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/benchmark/TPCH.java 20015b5 
>   tajo-core/tajo-core-backend/src/test/java/org/apache/tajo/TpchTestBase.java 0b80231 
>   tajo-core/tajo-core-backend/src/test/java/org/apache/tajo/client/TestDDLBuilder.java 73f7d5e 
>   tajo-core/tajo-core-backend/src/test/java/org/apache/tajo/engine/query/TestCTASQuery.java 72e92d2 
>   tajo-core/tajo-core-backend/src/test/java/org/apache/tajo/engine/query/TestNullValues.java 1f1d9e1 
>   tajo-docs/src/main/sphinx/table_management/csv.rst PRE-CREATION 
>   tajo-docs/src/main/sphinx/table_management/file_formats.rst 26edac4 
>   tajo-docs/src/main/sphinx/table_management/parquet.rst PRE-CREATION 
>   tajo-docs/src/main/sphinx/table_management/rcfile.rst PRE-CREATION 
>   tajo-docs/src/main/sphinx/table_management/sequencefile.rst PRE-CREATION 
>   tajo-storage/pom.xml 9f144bb 
>   tajo-storage/src/main/java/org/apache/tajo/storage/CSVFile.java 116e25c 
>   tajo-storage/src/main/java/org/apache/tajo/storage/parquet/TajoRecordMaterializer.java cce7ead 
>   tajo-storage/src/main/java/org/apache/tajo/storage/rcfile/RCFile.java 2fd3455 
>   tajo-storage/src/main/java/org/apache/tajo/storage/sequencefile/SequenceFileAppender.java PRE-CREATION 
>   tajo-storage/src/main/java/org/apache/tajo/storage/sequencefile/SequenceFileScanner.java PRE-CREATION 
>   tajo-storage/src/main/java/org/apache/tajo/storage/trevni/TrevniScanner.java ed71730 
>   tajo-storage/src/main/java/org/apache/tajo/storage/v2/CSVFileScanner.java 79ddfdd 
>   tajo-storage/src/main/resources/storage-default.xml e3a0eaf 
>   tajo-storage/src/test/java/org/apache/tajo/storage/TestCompressionStorages.java a776eb6 
>   tajo-storage/src/test/java/org/apache/tajo/storage/TestMergeScanner.java ad12c06 
>   tajo-storage/src/test/java/org/apache/tajo/storage/TestStorages.java b5338ff 
>   tajo-storage/src/test/resources/storage-default.xml 75274d0 
> 
> Diff: https://reviews.apache.org/r/19406/diff/
> 
> 
> Testing
> -------
> 
> mvn clean install -Phcatalog-0.12.0
> mvn clean install -Phcatalog-0.12.0 -Dtajo.catalog.store.class=org.apache.tajo.catalog.store.HCatalogStore
> 
> 
> Thanks,
> 
> Jung JaeHwa
> 
>

Re: Review Request 19406: TAJO-616: SequenceFile support

Posted by Jung JaeHwa <jh...@gruter.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/19406/
-----------------------------------------------------------

(Updated April 1, 2014, 7:03 a.m.)


Review request for Tajo.


Changes
-------

Thanks Jinho.

I updated the patch to remove a warning message which reported by findbugs.


Bugs: TAJO-616
    https://issues.apache.org/jira/browse/TAJO-616


Repository: tajo


Description
-------

Hadoop users can create a sequence file format, and hive users also can create a table which was stored as SequenceFIle. So, we need to support sequence file format as follows:

* Update parser and catalog for SequenceFile
* Make serializer/deserializer configurable in SequenceFile
* Scanner for SequenceFile
* Support for compression/decompression of SequenceFile
* Compatible to apache hive


Diffs (updated)
-----

  tajo-catalog/tajo-catalog-common/src/main/java/org/apache/tajo/catalog/CatalogConstants.java c7116f6 
  tajo-catalog/tajo-catalog-common/src/main/java/org/apache/tajo/catalog/CatalogUtil.java 22f54ca 
  tajo-catalog/tajo-catalog-common/src/main/proto/CatalogProtos.proto d3abc70 
  tajo-catalog/tajo-catalog-drivers/tajo-hcatalog/src/main/java/org/apache/tajo/catalog/store/HCatalogStore.java a4bdb28 
  tajo-catalog/tajo-catalog-drivers/tajo-hcatalog/src/main/java/org/apache/tajo/catalog/store/HCatalogUtil.java d426369 
  tajo-catalog/tajo-catalog-drivers/tajo-hcatalog/src/test/java/org/apache/tajo/catalog/store/TestHCatalogStore.java b9a734e 
  tajo-common/src/main/java/org/apache/tajo/util/Bytes.java f9ba923 
  tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/benchmark/TPCH.java 20015b5 
  tajo-core/tajo-core-backend/src/test/java/org/apache/tajo/TpchTestBase.java 0b80231 
  tajo-core/tajo-core-backend/src/test/java/org/apache/tajo/client/TestDDLBuilder.java 73f7d5e 
  tajo-core/tajo-core-backend/src/test/java/org/apache/tajo/engine/query/TestCTASQuery.java 72e92d2 
  tajo-core/tajo-core-backend/src/test/java/org/apache/tajo/engine/query/TestNullValues.java 1f1d9e1 
  tajo-docs/src/main/sphinx/table_management/csv.rst PRE-CREATION 
  tajo-docs/src/main/sphinx/table_management/file_formats.rst 26edac4 
  tajo-docs/src/main/sphinx/table_management/parquet.rst PRE-CREATION 
  tajo-docs/src/main/sphinx/table_management/rcfile.rst PRE-CREATION 
  tajo-docs/src/main/sphinx/table_management/sequencefile.rst PRE-CREATION 
  tajo-storage/pom.xml 9f144bb 
  tajo-storage/src/main/java/org/apache/tajo/storage/CSVFile.java 116e25c 
  tajo-storage/src/main/java/org/apache/tajo/storage/parquet/TajoRecordMaterializer.java cce7ead 
  tajo-storage/src/main/java/org/apache/tajo/storage/rcfile/RCFile.java 2fd3455 
  tajo-storage/src/main/java/org/apache/tajo/storage/sequencefile/SequenceFileAppender.java PRE-CREATION 
  tajo-storage/src/main/java/org/apache/tajo/storage/sequencefile/SequenceFileScanner.java PRE-CREATION 
  tajo-storage/src/main/java/org/apache/tajo/storage/trevni/TrevniScanner.java ed71730 
  tajo-storage/src/main/java/org/apache/tajo/storage/v2/CSVFileScanner.java 79ddfdd 
  tajo-storage/src/main/resources/storage-default.xml e3a0eaf 
  tajo-storage/src/test/java/org/apache/tajo/storage/TestCompressionStorages.java a776eb6 
  tajo-storage/src/test/java/org/apache/tajo/storage/TestMergeScanner.java ad12c06 
  tajo-storage/src/test/java/org/apache/tajo/storage/TestStorages.java b5338ff 
  tajo-storage/src/test/resources/storage-default.xml 75274d0 

Diff: https://reviews.apache.org/r/19406/diff/


Testing
-------

mvn clean install -Phcatalog-0.12.0
mvn clean install -Phcatalog-0.12.0 -Dtajo.catalog.store.class=org.apache.tajo.catalog.store.HCatalogStore


Thanks,

Jung JaeHwa

Re: Review Request 19406: TAJO-616: SequenceFile support

Posted by Jung JaeHwa <jh...@gruter.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/19406/
-----------------------------------------------------------

(Updated March 31, 2014, 3:08 p.m.)


Review request for Tajo.


Changes
-------

I added a missing code.

Jinho, thank you for your review.
I updated all codes about getSimpleName keyword.


Bugs: TAJO-616
    https://issues.apache.org/jira/browse/TAJO-616


Repository: tajo


Description
-------

Hadoop users can create a sequence file format, and hive users also can create a table which was stored as SequenceFIle. So, we need to support sequence file format as follows:

* Update parser and catalog for SequenceFile
* Make serializer/deserializer configurable in SequenceFile
* Scanner for SequenceFile
* Support for compression/decompression of SequenceFile
* Compatible to apache hive


Diffs (updated)
-----

  tajo-catalog/tajo-catalog-common/src/main/java/org/apache/tajo/catalog/CatalogConstants.java c7116f6 
  tajo-catalog/tajo-catalog-common/src/main/java/org/apache/tajo/catalog/CatalogUtil.java 9078e60 
  tajo-catalog/tajo-catalog-common/src/main/proto/CatalogProtos.proto 06ffcfd 
  tajo-catalog/tajo-catalog-drivers/tajo-hcatalog/src/main/java/org/apache/tajo/catalog/store/HCatalogStore.java 6ef255f 
  tajo-catalog/tajo-catalog-drivers/tajo-hcatalog/src/main/java/org/apache/tajo/catalog/store/HCatalogUtil.java d426369 
  tajo-catalog/tajo-catalog-drivers/tajo-hcatalog/src/test/java/org/apache/tajo/catalog/store/TestHCatalogStore.java b9a734e 
  tajo-common/src/main/java/org/apache/tajo/util/Bytes.java f9ba923 
  tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/benchmark/TPCH.java 20015b5 
  tajo-core/tajo-core-backend/src/test/java/org/apache/tajo/TpchTestBase.java 0b80231 
  tajo-core/tajo-core-backend/src/test/java/org/apache/tajo/client/TestDDLBuilder.java 73f7d5e 
  tajo-core/tajo-core-backend/src/test/java/org/apache/tajo/engine/query/TestCTASQuery.java e6840f6 
  tajo-core/tajo-core-backend/src/test/java/org/apache/tajo/engine/query/TestNullValues.java 1f1d9e1 
  tajo-docs/src/main/sphinx/table_management/csv.rst PRE-CREATION 
  tajo-docs/src/main/sphinx/table_management/file_formats.rst 26edac4 
  tajo-docs/src/main/sphinx/table_management/parquet.rst PRE-CREATION 
  tajo-docs/src/main/sphinx/table_management/rcfile.rst PRE-CREATION 
  tajo-docs/src/main/sphinx/table_management/sequencefile.rst PRE-CREATION 
  tajo-storage/pom.xml 9f144bb 
  tajo-storage/src/main/java/org/apache/tajo/storage/CSVFile.java 116e25c 
  tajo-storage/src/main/java/org/apache/tajo/storage/parquet/TajoRecordMaterializer.java cce7ead 
  tajo-storage/src/main/java/org/apache/tajo/storage/rcfile/RCFile.java 2fd3455 
  tajo-storage/src/main/java/org/apache/tajo/storage/sequencefile/SequenceFileAppender.java PRE-CREATION 
  tajo-storage/src/main/java/org/apache/tajo/storage/sequencefile/SequenceFileScanner.java PRE-CREATION 
  tajo-storage/src/main/java/org/apache/tajo/storage/trevni/TrevniScanner.java ed71730 
  tajo-storage/src/main/java/org/apache/tajo/storage/v2/CSVFileScanner.java 79ddfdd 
  tajo-storage/src/main/resources/storage-default.xml e3a0eaf 
  tajo-storage/src/test/java/org/apache/tajo/storage/TestCompressionStorages.java a776eb6 
  tajo-storage/src/test/java/org/apache/tajo/storage/TestMergeScanner.java ad12c06 
  tajo-storage/src/test/java/org/apache/tajo/storage/TestStorages.java b5338ff 
  tajo-storage/src/test/resources/storage-default.xml 75274d0 

Diff: https://reviews.apache.org/r/19406/diff/


Testing
-------

mvn clean install -Phcatalog-0.12.0
mvn clean install -Phcatalog-0.12.0 -Dtajo.catalog.store.class=org.apache.tajo.catalog.store.HCatalogStore


Thanks,

Jung JaeHwa

Re: Review Request 19406: TAJO-616: SequenceFile support

Posted by Jung JaeHwa <jh...@gruter.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/19406/
-----------------------------------------------------------

(Updated March 31, 2014, 3:01 p.m.)


Review request for Tajo.


Changes
-------

I updated some codes which was written with getSimpleName keyword.


Bugs: TAJO-616
    https://issues.apache.org/jira/browse/TAJO-616


Repository: tajo


Description
-------

Hadoop users can create a sequence file format, and hive users also can create a table which was stored as SequenceFIle. So, we need to support sequence file format as follows:

* Update parser and catalog for SequenceFile
* Make serializer/deserializer configurable in SequenceFile
* Scanner for SequenceFile
* Support for compression/decompression of SequenceFile
* Compatible to apache hive


Diffs (updated)
-----

  tajo-catalog/tajo-catalog-common/src/main/java/org/apache/tajo/catalog/CatalogConstants.java c7116f6 
  tajo-catalog/tajo-catalog-common/src/main/java/org/apache/tajo/catalog/CatalogUtil.java 9078e60 
  tajo-catalog/tajo-catalog-common/src/main/proto/CatalogProtos.proto 06ffcfd 
  tajo-catalog/tajo-catalog-drivers/tajo-hcatalog/src/main/java/org/apache/tajo/catalog/store/HCatalogStore.java 6ef255f 
  tajo-catalog/tajo-catalog-drivers/tajo-hcatalog/src/main/java/org/apache/tajo/catalog/store/HCatalogUtil.java d426369 
  tajo-catalog/tajo-catalog-drivers/tajo-hcatalog/src/test/java/org/apache/tajo/catalog/store/TestHCatalogStore.java b9a734e 
  tajo-common/src/main/java/org/apache/tajo/util/Bytes.java f9ba923 
  tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/benchmark/TPCH.java 20015b5 
  tajo-core/tajo-core-backend/src/test/java/org/apache/tajo/TpchTestBase.java 0b80231 
  tajo-core/tajo-core-backend/src/test/java/org/apache/tajo/client/TestDDLBuilder.java 73f7d5e 
  tajo-core/tajo-core-backend/src/test/java/org/apache/tajo/engine/query/TestCTASQuery.java e6840f6 
  tajo-core/tajo-core-backend/src/test/java/org/apache/tajo/engine/query/TestNullValues.java 1f1d9e1 
  tajo-docs/src/main/sphinx/table_management/csv.rst PRE-CREATION 
  tajo-docs/src/main/sphinx/table_management/file_formats.rst 26edac4 
  tajo-docs/src/main/sphinx/table_management/parquet.rst PRE-CREATION 
  tajo-docs/src/main/sphinx/table_management/rcfile.rst PRE-CREATION 
  tajo-docs/src/main/sphinx/table_management/sequencefile.rst PRE-CREATION 
  tajo-storage/pom.xml 9f144bb 
  tajo-storage/src/main/java/org/apache/tajo/storage/CSVFile.java 116e25c 
  tajo-storage/src/main/java/org/apache/tajo/storage/parquet/TajoRecordMaterializer.java cce7ead 
  tajo-storage/src/main/java/org/apache/tajo/storage/rcfile/RCFile.java 2fd3455 
  tajo-storage/src/main/java/org/apache/tajo/storage/sequencefile/SequenceFileAppender.java PRE-CREATION 
  tajo-storage/src/main/java/org/apache/tajo/storage/sequencefile/SequenceFileScanner.java PRE-CREATION 
  tajo-storage/src/main/java/org/apache/tajo/storage/trevni/TrevniScanner.java ed71730 
  tajo-storage/src/main/java/org/apache/tajo/storage/v2/CSVFileScanner.java 79ddfdd 
  tajo-storage/src/main/resources/storage-default.xml e3a0eaf 
  tajo-storage/src/test/java/org/apache/tajo/storage/TestCompressionStorages.java a776eb6 
  tajo-storage/src/test/java/org/apache/tajo/storage/TestMergeScanner.java ad12c06 
  tajo-storage/src/test/java/org/apache/tajo/storage/TestStorages.java b5338ff 
  tajo-storage/src/test/resources/storage-default.xml 75274d0 

Diff: https://reviews.apache.org/r/19406/diff/


Testing
-------

mvn clean install -Phcatalog-0.12.0
mvn clean install -Phcatalog-0.12.0 -Dtajo.catalog.store.class=org.apache.tajo.catalog.store.HCatalogStore


Thanks,

Jung JaeHwa

Re: Review Request 19406: TAJO-616: SequenceFile support

Posted by Jinho Kim <jh...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/19406/#review39032
-----------------------------------------------------------



tajo-storage/src/main/java/org/apache/tajo/storage/sequencefile/SequenceFileScanner.java
<https://reviews.apache.org/r/19406/#comment71424>

    we should use getQualifiedName() instead of getSimpleName() in column projection;
    
    please update the rcfile,parquet,Trevni


- Jinho Kim


On March 31, 2014, 9:20 a.m., Jung JaeHwa wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/19406/
> -----------------------------------------------------------
> 
> (Updated March 31, 2014, 9:20 a.m.)
> 
> 
> Review request for Tajo.
> 
> 
> Bugs: TAJO-616
>     https://issues.apache.org/jira/browse/TAJO-616
> 
> 
> Repository: tajo
> 
> 
> Description
> -------
> 
> Hadoop users can create a sequence file format, and hive users also can create a table which was stored as SequenceFIle. So, we need to support sequence file format as follows:
> 
> * Update parser and catalog for SequenceFile
> * Make serializer/deserializer configurable in SequenceFile
> * Scanner for SequenceFile
> * Support for compression/decompression of SequenceFile
> * Compatible to apache hive
> 
> 
> Diffs
> -----
> 
>   tajo-catalog/tajo-catalog-common/src/main/java/org/apache/tajo/catalog/CatalogConstants.java c7116f6 
>   tajo-catalog/tajo-catalog-common/src/main/java/org/apache/tajo/catalog/CatalogUtil.java 9078e60 
>   tajo-catalog/tajo-catalog-common/src/main/proto/CatalogProtos.proto 06ffcfd 
>   tajo-catalog/tajo-catalog-drivers/tajo-hcatalog/src/main/java/org/apache/tajo/catalog/store/HCatalogStore.java 6ef255f 
>   tajo-catalog/tajo-catalog-drivers/tajo-hcatalog/src/main/java/org/apache/tajo/catalog/store/HCatalogUtil.java d426369 
>   tajo-catalog/tajo-catalog-drivers/tajo-hcatalog/src/test/java/org/apache/tajo/catalog/store/TestHCatalogStore.java b9a734e 
>   tajo-common/src/main/java/org/apache/tajo/util/Bytes.java f9ba923 
>   tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/benchmark/TPCH.java 20015b5 
>   tajo-core/tajo-core-backend/src/test/java/org/apache/tajo/TpchTestBase.java 0b80231 
>   tajo-core/tajo-core-backend/src/test/java/org/apache/tajo/client/TestDDLBuilder.java 73f7d5e 
>   tajo-core/tajo-core-backend/src/test/java/org/apache/tajo/engine/query/TestCTASQuery.java e6840f6 
>   tajo-core/tajo-core-backend/src/test/java/org/apache/tajo/engine/query/TestNullValues.java 1f1d9e1 
>   tajo-docs/src/main/sphinx/table_management/csv.rst PRE-CREATION 
>   tajo-docs/src/main/sphinx/table_management/file_formats.rst 26edac4 
>   tajo-docs/src/main/sphinx/table_management/parquet.rst PRE-CREATION 
>   tajo-docs/src/main/sphinx/table_management/rcfile.rst PRE-CREATION 
>   tajo-docs/src/main/sphinx/table_management/sequencefile.rst PRE-CREATION 
>   tajo-storage/pom.xml 9f144bb 
>   tajo-storage/src/main/java/org/apache/tajo/storage/CSVFile.java 116e25c 
>   tajo-storage/src/main/java/org/apache/tajo/storage/sequencefile/SequenceFileAppender.java PRE-CREATION 
>   tajo-storage/src/main/java/org/apache/tajo/storage/sequencefile/SequenceFileScanner.java PRE-CREATION 
>   tajo-storage/src/main/resources/storage-default.xml e3a0eaf 
>   tajo-storage/src/test/java/org/apache/tajo/storage/TestCompressionStorages.java a776eb6 
>   tajo-storage/src/test/java/org/apache/tajo/storage/TestMergeScanner.java ad12c06 
>   tajo-storage/src/test/java/org/apache/tajo/storage/TestStorages.java b5338ff 
>   tajo-storage/src/test/resources/storage-default.xml 75274d0 
> 
> Diff: https://reviews.apache.org/r/19406/diff/
> 
> 
> Testing
> -------
> 
> mvn clean install -Phcatalog-0.12.0
> mvn clean install -Phcatalog-0.12.0 -Dtajo.catalog.store.class=org.apache.tajo.catalog.store.HCatalogStore
> 
> 
> Thanks,
> 
> Jung JaeHwa
> 
>

Re: Review Request 19406: TAJO-616: SequenceFile support

Posted by Jung JaeHwa <jh...@gruter.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/19406/
-----------------------------------------------------------

(Updated March 31, 2014, 9:20 a.m.)


Review request for Tajo.


Changes
-------

I updated the patch as follows:
- add default delimiter at CatalogUtil
- remove dbcp classpath at tajo shell


Bugs: TAJO-616
    https://issues.apache.org/jira/browse/TAJO-616


Repository: tajo


Description
-------

Hadoop users can create a sequence file format, and hive users also can create a table which was stored as SequenceFIle. So, we need to support sequence file format as follows:

* Update parser and catalog for SequenceFile
* Make serializer/deserializer configurable in SequenceFile
* Scanner for SequenceFile
* Support for compression/decompression of SequenceFile
* Compatible to apache hive


Diffs (updated)
-----

  tajo-catalog/tajo-catalog-common/src/main/java/org/apache/tajo/catalog/CatalogConstants.java c7116f6 
  tajo-catalog/tajo-catalog-common/src/main/java/org/apache/tajo/catalog/CatalogUtil.java 9078e60 
  tajo-catalog/tajo-catalog-common/src/main/proto/CatalogProtos.proto 06ffcfd 
  tajo-catalog/tajo-catalog-drivers/tajo-hcatalog/src/main/java/org/apache/tajo/catalog/store/HCatalogStore.java 6ef255f 
  tajo-catalog/tajo-catalog-drivers/tajo-hcatalog/src/main/java/org/apache/tajo/catalog/store/HCatalogUtil.java d426369 
  tajo-catalog/tajo-catalog-drivers/tajo-hcatalog/src/test/java/org/apache/tajo/catalog/store/TestHCatalogStore.java b9a734e 
  tajo-common/src/main/java/org/apache/tajo/util/Bytes.java f9ba923 
  tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/benchmark/TPCH.java 20015b5 
  tajo-core/tajo-core-backend/src/test/java/org/apache/tajo/TpchTestBase.java 0b80231 
  tajo-core/tajo-core-backend/src/test/java/org/apache/tajo/client/TestDDLBuilder.java 73f7d5e 
  tajo-core/tajo-core-backend/src/test/java/org/apache/tajo/engine/query/TestCTASQuery.java e6840f6 
  tajo-core/tajo-core-backend/src/test/java/org/apache/tajo/engine/query/TestNullValues.java 1f1d9e1 
  tajo-docs/src/main/sphinx/table_management/csv.rst PRE-CREATION 
  tajo-docs/src/main/sphinx/table_management/file_formats.rst 26edac4 
  tajo-docs/src/main/sphinx/table_management/parquet.rst PRE-CREATION 
  tajo-docs/src/main/sphinx/table_management/rcfile.rst PRE-CREATION 
  tajo-docs/src/main/sphinx/table_management/sequencefile.rst PRE-CREATION 
  tajo-storage/pom.xml 9f144bb 
  tajo-storage/src/main/java/org/apache/tajo/storage/CSVFile.java 116e25c 
  tajo-storage/src/main/java/org/apache/tajo/storage/sequencefile/SequenceFileAppender.java PRE-CREATION 
  tajo-storage/src/main/java/org/apache/tajo/storage/sequencefile/SequenceFileScanner.java PRE-CREATION 
  tajo-storage/src/main/resources/storage-default.xml e3a0eaf 
  tajo-storage/src/test/java/org/apache/tajo/storage/TestCompressionStorages.java a776eb6 
  tajo-storage/src/test/java/org/apache/tajo/storage/TestMergeScanner.java ad12c06 
  tajo-storage/src/test/java/org/apache/tajo/storage/TestStorages.java b5338ff 
  tajo-storage/src/test/resources/storage-default.xml 75274d0 

Diff: https://reviews.apache.org/r/19406/diff/


Testing
-------

mvn clean install -Phcatalog-0.12.0
mvn clean install -Phcatalog-0.12.0 -Dtajo.catalog.store.class=org.apache.tajo.catalog.store.HCatalogStore


Thanks,

Jung JaeHwa

Re: Review Request 19406: TAJO-616: SequenceFile support

Posted by Jung JaeHwa <jh...@gruter.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/19406/
-----------------------------------------------------------

(Updated March 31, 2014, 9:12 a.m.)


Review request for Tajo.


Changes
-------

I fixed a bug about projection and added a document for SequenceFile.


Bugs: TAJO-616
    https://issues.apache.org/jira/browse/TAJO-616


Repository: tajo


Description
-------

Hadoop users can create a sequence file format, and hive users also can create a table which was stored as SequenceFIle. So, we need to support sequence file format as follows:

* Update parser and catalog for SequenceFile
* Make serializer/deserializer configurable in SequenceFile
* Scanner for SequenceFile
* Support for compression/decompression of SequenceFile
* Compatible to apache hive


Diffs (updated)
-----

  tajo-catalog/tajo-catalog-common/src/main/java/org/apache/tajo/catalog/CatalogConstants.java c7116f6 
  tajo-catalog/tajo-catalog-common/src/main/java/org/apache/tajo/catalog/CatalogUtil.java 9078e60 
  tajo-catalog/tajo-catalog-common/src/main/proto/CatalogProtos.proto 06ffcfd 
  tajo-catalog/tajo-catalog-drivers/tajo-hcatalog/src/main/java/org/apache/tajo/catalog/store/HCatalogStore.java 6ef255f 
  tajo-catalog/tajo-catalog-drivers/tajo-hcatalog/src/main/java/org/apache/tajo/catalog/store/HCatalogUtil.java d426369 
  tajo-catalog/tajo-catalog-drivers/tajo-hcatalog/src/test/java/org/apache/tajo/catalog/store/TestHCatalogStore.java b9a734e 
  tajo-common/src/main/java/org/apache/tajo/util/Bytes.java f9ba923 
  tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/benchmark/TPCH.java 20015b5 
  tajo-core/tajo-core-backend/src/test/java/org/apache/tajo/TpchTestBase.java 0b80231 
  tajo-core/tajo-core-backend/src/test/java/org/apache/tajo/client/TestDDLBuilder.java 73f7d5e 
  tajo-core/tajo-core-backend/src/test/java/org/apache/tajo/engine/query/TestCTASQuery.java e6840f6 
  tajo-core/tajo-core-backend/src/test/java/org/apache/tajo/engine/query/TestNullValues.java 1f1d9e1 
  tajo-dist/src/main/bin/tajo 77d4a02 
  tajo-docs/src/main/sphinx/table_management/csv.rst PRE-CREATION 
  tajo-docs/src/main/sphinx/table_management/file_formats.rst 26edac4 
  tajo-docs/src/main/sphinx/table_management/parquet.rst PRE-CREATION 
  tajo-docs/src/main/sphinx/table_management/rcfile.rst PRE-CREATION 
  tajo-docs/src/main/sphinx/table_management/sequencefile.rst PRE-CREATION 
  tajo-storage/pom.xml 9f144bb 
  tajo-storage/src/main/java/org/apache/tajo/storage/CSVFile.java 116e25c 
  tajo-storage/src/main/java/org/apache/tajo/storage/sequencefile/SequenceFileAppender.java PRE-CREATION 
  tajo-storage/src/main/java/org/apache/tajo/storage/sequencefile/SequenceFileScanner.java PRE-CREATION 
  tajo-storage/src/main/resources/storage-default.xml e3a0eaf 
  tajo-storage/src/test/java/org/apache/tajo/storage/TestCompressionStorages.java a776eb6 
  tajo-storage/src/test/java/org/apache/tajo/storage/TestMergeScanner.java ad12c06 
  tajo-storage/src/test/java/org/apache/tajo/storage/TestStorages.java b5338ff 
  tajo-storage/src/test/resources/storage-default.xml 75274d0 

Diff: https://reviews.apache.org/r/19406/diff/


Testing
-------

mvn clean install -Phcatalog-0.12.0
mvn clean install -Phcatalog-0.12.0 -Dtajo.catalog.store.class=org.apache.tajo.catalog.store.HCatalogStore


Thanks,

Jung JaeHwa

Re: Review Request 19406: TAJO-616: SequenceFile support

Posted by Jinho Kim <jh...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/19406/#review39028
-----------------------------------------------------------


Jung JaeHwa
Could you please take a look at this error?

Store : Hcatalog
Query : 
create table table2 (id integer, name text, year smallint) using sequencefile with ('compression.codec'='org.apache.hadoop.io.compress.SnappyCodec')
select name from table2;

2014-03-31 14:13:30,477 INFO  [Thread-95] worker.Task (Task.java:<init>(203)) - ==================================
2014-03-31 14:13:30,477 INFO  [Thread-95] worker.TaskAttemptContext (TaskAttemptContext.java:setState(110)) - Query status of ta_1396242125461_000
8_000001_000000_00 is changed to TA_RUNNING
2014-03-31 14:13:30,488 ERROR [Thread-95] worker.Task (Task.java:run(393)) - java.lang.ArrayIndexOutOfBoundsException: 1
        at org.apache.tajo.storage.sequencefile.SequenceFileScanner.makeTuple(SequenceFileScanner.java:209)
        at org.apache.tajo.storage.sequencefile.SequenceFileScanner.next(SequenceFileScanner.java:158)
        at org.apache.tajo.engine.planner.physical.SeqScanExec.next(SeqScanExec.java:168)
        at org.apache.tajo.engine.planner.physical.StoreTableExec.next(StoreTableExec.java:76)
        at org.apache.tajo.worker.Task.run(Task.java:384)
        at org.apache.tajo.worker.TaskRunner$1.run(TaskRunner.java:395)
        at java.lang.Thread.run(Thread.java:695)

2014-03-31 14:13:30,488 INFO  [Thread-95] worker.TaskAttemptContext (TaskAttemptContext.java:setState(110)) - Query status of ta_1396242125461_000
8_000001_000000_00 is changed to TA_FAILED
2014-03-31 14:13:30,488 INFO  [Thread-95] worker.Task (Task.java:run(447)) - Task Counter - total:12, succeeded: 5, killed: 0, failed: 7
2014-03-31 14:13:30,488 INFO  [Thread-95] worker.TaskRunner (TaskRunner.java:run(336)) - Request GetTask: eb_1396242125461_0008_000001,container_1
396242125461_0008_01_000016
2014-03-31 14:13:30,489 ERROR [AsyncDispatcher event handler] querymaster.QueryUnitAttempt (QueryUnitAttempt.java:transition(403)) - FROM kimjh.se
rver.gruter.com >> java.lang.ArrayIndexOutOfBoundsException: 1
        at org.apache.tajo.storage.sequencefile.SequenceFileScanner.makeTuple(SequenceFileScanner.java:209)
        at org.apache.tajo.storage.sequencefile.SequenceFileScanner.next(SequenceFileScanner.java:158)
        at org.apache.tajo.engine.planner.physical.SeqScanExec.next(SeqScanExec.java:168)
        at org.apache.tajo.engine.planner.physical.StoreTableExec.next(StoreTableExec.java:76)
        at org.apache.tajo.worker.Task.run(Task.java:384)
        at org.apache.tajo.worker.TaskRunner$1.run(TaskRunner.java:395)
        at java.lang.Thread.run(Thread.java:695)

2014-03-31 14:13:30,489 INFO  [AsyncDispatcher event handler] querymaster.QueryUnit (QueryUnit.java:transition(543)) - ===========================
==================================

- Jinho Kim


On March 28, 2014, 9:15 a.m., Jung JaeHwa wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/19406/
> -----------------------------------------------------------
> 
> (Updated March 28, 2014, 9:15 a.m.)
> 
> 
> Review request for Tajo.
> 
> 
> Bugs: TAJO-616
>     https://issues.apache.org/jira/browse/TAJO-616
> 
> 
> Repository: tajo
> 
> 
> Description
> -------
> 
> Hadoop users can create a sequence file format, and hive users also can create a table which was stored as SequenceFIle. So, we need to support sequence file format as follows:
> 
> * Update parser and catalog for SequenceFile
> * Make serializer/deserializer configurable in SequenceFile
> * Scanner for SequenceFile
> * Support for compression/decompression of SequenceFile
> * Compatible to apache hive
> 
> 
> Diffs
> -----
> 
>   tajo-catalog/tajo-catalog-common/src/main/java/org/apache/tajo/catalog/CatalogConstants.java c7116f6 
>   tajo-catalog/tajo-catalog-common/src/main/java/org/apache/tajo/catalog/CatalogUtil.java 9078e60 
>   tajo-catalog/tajo-catalog-common/src/main/proto/CatalogProtos.proto 06ffcfd 
>   tajo-catalog/tajo-catalog-drivers/tajo-hcatalog/src/main/java/org/apache/tajo/catalog/store/HCatalogStore.java 6ef255f 
>   tajo-catalog/tajo-catalog-drivers/tajo-hcatalog/src/main/java/org/apache/tajo/catalog/store/HCatalogUtil.java d426369 
>   tajo-catalog/tajo-catalog-drivers/tajo-hcatalog/src/test/java/org/apache/tajo/catalog/store/TestHCatalogStore.java b9a734e 
>   tajo-common/src/main/java/org/apache/tajo/util/Bytes.java f9ba923 
>   tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/benchmark/TPCH.java 20015b5 
>   tajo-core/tajo-core-backend/src/test/java/org/apache/tajo/TpchTestBase.java 0b80231 
>   tajo-core/tajo-core-backend/src/test/java/org/apache/tajo/client/TestDDLBuilder.java 73f7d5e 
>   tajo-core/tajo-core-backend/src/test/java/org/apache/tajo/engine/query/TestCTASQuery.java e6840f6 
>   tajo-core/tajo-core-backend/src/test/java/org/apache/tajo/engine/query/TestNullValues.java 1f1d9e1 
>   tajo-storage/pom.xml 9f144bb 
>   tajo-storage/src/main/java/org/apache/tajo/storage/CSVFile.java 116e25c 
>   tajo-storage/src/main/java/org/apache/tajo/storage/sequencefile/SequenceFileAppender.java PRE-CREATION 
>   tajo-storage/src/main/java/org/apache/tajo/storage/sequencefile/SequenceFileScanner.java PRE-CREATION 
>   tajo-storage/src/main/resources/storage-default.xml e3a0eaf 
>   tajo-storage/src/test/java/org/apache/tajo/storage/TestCompressionStorages.java a776eb6 
>   tajo-storage/src/test/java/org/apache/tajo/storage/TestMergeScanner.java ad12c06 
>   tajo-storage/src/test/java/org/apache/tajo/storage/TestStorages.java b5338ff 
>   tajo-storage/src/test/resources/storage-default.xml 75274d0 
> 
> Diff: https://reviews.apache.org/r/19406/diff/
> 
> 
> Testing
> -------
> 
> mvn clean install -Phcatalog-0.12.0
> mvn clean install -Phcatalog-0.12.0 -Dtajo.catalog.store.class=org.apache.tajo.catalog.store.HCatalogStore
> 
> 
> Thanks,
> 
> Jung JaeHwa
> 
>

Re: Review Request 19406: TAJO-616: SequenceFile support

Posted by Jung JaeHwa <jh...@gruter.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/19406/
-----------------------------------------------------------

(Updated March 28, 2014, 9:15 a.m.)


Review request for Tajo.


Changes
-------

I remove some codes for Datum because it seems to handle at another issue.


Bugs: TAJO-616
    https://issues.apache.org/jira/browse/TAJO-616


Repository: tajo


Description
-------

Hadoop users can create a sequence file format, and hive users also can create a table which was stored as SequenceFIle. So, we need to support sequence file format as follows:

* Update parser and catalog for SequenceFile
* Make serializer/deserializer configurable in SequenceFile
* Scanner for SequenceFile
* Support for compression/decompression of SequenceFile
* Compatible to apache hive


Diffs (updated)
-----

  tajo-catalog/tajo-catalog-common/src/main/java/org/apache/tajo/catalog/CatalogConstants.java c7116f6 
  tajo-catalog/tajo-catalog-common/src/main/java/org/apache/tajo/catalog/CatalogUtil.java 9078e60 
  tajo-catalog/tajo-catalog-common/src/main/proto/CatalogProtos.proto 06ffcfd 
  tajo-catalog/tajo-catalog-drivers/tajo-hcatalog/src/main/java/org/apache/tajo/catalog/store/HCatalogStore.java 6ef255f 
  tajo-catalog/tajo-catalog-drivers/tajo-hcatalog/src/main/java/org/apache/tajo/catalog/store/HCatalogUtil.java d426369 
  tajo-catalog/tajo-catalog-drivers/tajo-hcatalog/src/test/java/org/apache/tajo/catalog/store/TestHCatalogStore.java b9a734e 
  tajo-common/src/main/java/org/apache/tajo/util/Bytes.java f9ba923 
  tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/benchmark/TPCH.java 20015b5 
  tajo-core/tajo-core-backend/src/test/java/org/apache/tajo/TpchTestBase.java 0b80231 
  tajo-core/tajo-core-backend/src/test/java/org/apache/tajo/client/TestDDLBuilder.java 73f7d5e 
  tajo-core/tajo-core-backend/src/test/java/org/apache/tajo/engine/query/TestCTASQuery.java e6840f6 
  tajo-core/tajo-core-backend/src/test/java/org/apache/tajo/engine/query/TestNullValues.java 1f1d9e1 
  tajo-storage/pom.xml 9f144bb 
  tajo-storage/src/main/java/org/apache/tajo/storage/CSVFile.java 116e25c 
  tajo-storage/src/main/java/org/apache/tajo/storage/sequencefile/SequenceFileAppender.java PRE-CREATION 
  tajo-storage/src/main/java/org/apache/tajo/storage/sequencefile/SequenceFileScanner.java PRE-CREATION 
  tajo-storage/src/main/resources/storage-default.xml e3a0eaf 
  tajo-storage/src/test/java/org/apache/tajo/storage/TestCompressionStorages.java a776eb6 
  tajo-storage/src/test/java/org/apache/tajo/storage/TestMergeScanner.java ad12c06 
  tajo-storage/src/test/java/org/apache/tajo/storage/TestStorages.java b5338ff 
  tajo-storage/src/test/resources/storage-default.xml 75274d0 

Diff: https://reviews.apache.org/r/19406/diff/


Testing
-------

mvn clean install -Phcatalog-0.12.0
mvn clean install -Phcatalog-0.12.0 -Dtajo.catalog.store.class=org.apache.tajo.catalog.store.HCatalogStore


Thanks,

Jung JaeHwa

Re: Review Request 19406: TAJO-616: SequenceFile support

Posted by Jung JaeHwa <jh...@gruter.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/19406/
-----------------------------------------------------------

(Updated March 28, 2014, 8:39 a.m.)


Review request for Tajo.


Changes
-------

I attached the fifth patch after refactoring CatalogConstants.


Bugs: TAJO-616
    https://issues.apache.org/jira/browse/TAJO-616


Repository: tajo


Description
-------

Hadoop users can create a sequence file format, and hive users also can create a table which was stored as SequenceFIle. So, we need to support sequence file format as follows:

* Update parser and catalog for SequenceFile
* Make serializer/deserializer configurable in SequenceFile
* Scanner for SequenceFile
* Support for compression/decompression of SequenceFile
* Compatible to apache hive


Diffs (updated)
-----

  tajo-catalog/tajo-catalog-common/src/main/java/org/apache/tajo/catalog/CatalogConstants.java c7116f6 
  tajo-catalog/tajo-catalog-common/src/main/java/org/apache/tajo/catalog/CatalogUtil.java 9078e60 
  tajo-catalog/tajo-catalog-common/src/main/proto/CatalogProtos.proto 06ffcfd 
  tajo-catalog/tajo-catalog-drivers/tajo-hcatalog/src/main/java/org/apache/tajo/catalog/store/HCatalogStore.java 6ef255f 
  tajo-catalog/tajo-catalog-drivers/tajo-hcatalog/src/main/java/org/apache/tajo/catalog/store/HCatalogUtil.java d426369 
  tajo-catalog/tajo-catalog-drivers/tajo-hcatalog/src/test/java/org/apache/tajo/catalog/store/TestHCatalogStore.java b9a734e 
  tajo-common/src/main/java/org/apache/tajo/datum/DateDatum.java e489a09 
  tajo-common/src/main/java/org/apache/tajo/datum/TimeDatum.java 8b18f08 
  tajo-common/src/main/java/org/apache/tajo/datum/TimestampDatum.java e7a9861 
  tajo-common/src/main/java/org/apache/tajo/util/Bytes.java f9ba923 
  tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/benchmark/TPCH.java 20015b5 
  tajo-core/tajo-core-backend/src/test/java/org/apache/tajo/TpchTestBase.java 0b80231 
  tajo-core/tajo-core-backend/src/test/java/org/apache/tajo/client/TestDDLBuilder.java 73f7d5e 
  tajo-core/tajo-core-backend/src/test/java/org/apache/tajo/engine/query/TestCTASQuery.java e6840f6 
  tajo-core/tajo-core-backend/src/test/java/org/apache/tajo/engine/query/TestNullValues.java 1f1d9e1 
  tajo-storage/pom.xml 9f144bb 
  tajo-storage/src/main/java/org/apache/tajo/storage/CSVFile.java 116e25c 
  tajo-storage/src/main/java/org/apache/tajo/storage/sequencefile/SequenceFileAppender.java PRE-CREATION 
  tajo-storage/src/main/java/org/apache/tajo/storage/sequencefile/SequenceFileScanner.java PRE-CREATION 
  tajo-storage/src/main/resources/storage-default.xml e3a0eaf 
  tajo-storage/src/test/java/org/apache/tajo/storage/TestCompressionStorages.java a776eb6 
  tajo-storage/src/test/java/org/apache/tajo/storage/TestMergeScanner.java ad12c06 
  tajo-storage/src/test/java/org/apache/tajo/storage/TestStorages.java b5338ff 
  tajo-storage/src/test/resources/storage-default.xml 75274d0 

Diff: https://reviews.apache.org/r/19406/diff/


Testing
-------

mvn clean install -Phcatalog-0.12.0
mvn clean install -Phcatalog-0.12.0 -Dtajo.catalog.store.class=org.apache.tajo.catalog.store.HCatalogStore


Thanks,

Jung JaeHwa

Re: Review Request 19406: TAJO-616: SequenceFile support

Posted by Jung JaeHwa <jh...@gruter.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/19406/
-----------------------------------------------------------

(Updated March 28, 2014, 1:34 a.m.)


Review request for Tajo.


Changes
-------

I attached the third patch


Bugs: TAJO-616
    https://issues.apache.org/jira/browse/TAJO-616


Repository: tajo


Description
-------

Hadoop users can create a sequence file format, and hive users also can create a table which was stored as SequenceFIle. So, we need to support sequence file format as follows:

* Update parser and catalog for SequenceFile
* Make serializer/deserializer configurable in SequenceFile
* Scanner for SequenceFile
* Support for compression/decompression of SequenceFile
* Compatible to apache hive


Diffs (updated)
-----

  tajo-catalog/tajo-catalog-common/src/main/java/org/apache/tajo/catalog/CatalogConstants.java c7116f6 
  tajo-catalog/tajo-catalog-common/src/main/java/org/apache/tajo/catalog/CatalogUtil.java 9078e60 
  tajo-catalog/tajo-catalog-common/src/main/proto/CatalogProtos.proto 06ffcfd 
  tajo-catalog/tajo-catalog-drivers/tajo-hcatalog/src/main/java/org/apache/tajo/catalog/store/HCatalogStore.java 6ef255f 
  tajo-catalog/tajo-catalog-drivers/tajo-hcatalog/src/main/java/org/apache/tajo/catalog/store/HCatalogUtil.java d426369 
  tajo-catalog/tajo-catalog-drivers/tajo-hcatalog/src/test/java/org/apache/tajo/catalog/store/TestHCatalogStore.java b9a734e 
  tajo-common/src/main/java/org/apache/tajo/util/Bytes.java f9ba923 
  tajo-core/tajo-core-backend/src/test/java/org/apache/tajo/engine/query/TestCTASQuery.java e6840f6 
  tajo-storage/pom.xml 9f144bb 
  tajo-storage/src/main/java/org/apache/tajo/storage/sequencefile/SequenceFileAppender.java PRE-CREATION 
  tajo-storage/src/main/java/org/apache/tajo/storage/sequencefile/SequenceFileScanner.java PRE-CREATION 
  tajo-storage/src/main/resources/storage-default.xml e3a0eaf 
  tajo-storage/src/test/java/org/apache/tajo/storage/TestCompressionStorages.java a776eb6 
  tajo-storage/src/test/java/org/apache/tajo/storage/TestMergeScanner.java ad12c06 
  tajo-storage/src/test/java/org/apache/tajo/storage/TestStorages.java b5338ff 
  tajo-storage/src/test/resources/storage-default.xml 75274d0 

Diff: https://reviews.apache.org/r/19406/diff/


Testing
-------

mvn clean install -Phcatalog-0.12.0
mvn clean install -Phcatalog-0.12.0 -Dtajo.catalog.store.class=org.apache.tajo.catalog.store.HCatalogStore


Thanks,

Jung JaeHwa

Re: Review Request 19406: TAJO-616: SequenceFile support

Posted by Jung JaeHwa <jh...@gruter.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/19406/
-----------------------------------------------------------

(Updated March 27, 2014, 5:24 p.m.)


Review request for Tajo.


Bugs: TAJO-616
    https://issues.apache.org/jira/browse/TAJO-616


Repository: tajo


Description
-------

Hadoop users can create a sequence file format, and hive users also can create a table which was stored as SequenceFIle. So, we need to support sequence file format as follows:

* Update parser and catalog for SequenceFile
* Make serializer/deserializer configurable in SequenceFile
* Scanner for SequenceFile
* Support for compression/decompression of SequenceFile
* Compatible to apache hive


Diffs
-----

  tajo-catalog/tajo-catalog-common/src/main/java/org/apache/tajo/catalog/CatalogConstants.java c7116f6 
  tajo-catalog/tajo-catalog-common/src/main/java/org/apache/tajo/catalog/CatalogUtil.java 9078e60 
  tajo-catalog/tajo-catalog-common/src/main/proto/CatalogProtos.proto 06ffcfd 
  tajo-catalog/tajo-catalog-drivers/tajo-hcatalog/src/main/java/org/apache/tajo/catalog/store/HCatalogStore.java 6ef255f 
  tajo-catalog/tajo-catalog-drivers/tajo-hcatalog/src/main/java/org/apache/tajo/catalog/store/HCatalogUtil.java d426369 
  tajo-catalog/tajo-catalog-drivers/tajo-hcatalog/src/test/java/org/apache/tajo/catalog/store/TestHCatalogStore.java b9a734e 
  tajo-common/src/main/java/org/apache/tajo/util/Bytes.java f9ba923 
  tajo-core/tajo-core-backend/src/test/java/org/apache/tajo/engine/query/TestCTASQuery.java e6840f6 
  tajo-storage/pom.xml 9f144bb 
  tajo-storage/src/main/java/org/apache/tajo/storage/sequencefile/SequenceFileAppender.java PRE-CREATION 
  tajo-storage/src/main/java/org/apache/tajo/storage/sequencefile/SequenceFileScanner.java PRE-CREATION 
  tajo-storage/src/main/resources/storage-default.xml e3a0eaf 
  tajo-storage/src/test/java/org/apache/tajo/storage/TestCompressionStorages.java a776eb6 
  tajo-storage/src/test/java/org/apache/tajo/storage/TestMergeScanner.java ad12c06 
  tajo-storage/src/test/java/org/apache/tajo/storage/TestStorages.java b5338ff 
  tajo-storage/src/test/resources/storage-default.xml 75274d0 

Diff: https://reviews.apache.org/r/19406/diff/


Testing (updated)
-------

mvn clean install -Phcatalog-0.12.0
mvn clean install -Phcatalog-0.12.0 -Dtajo.catalog.store.class=org.apache.tajo.catalog.store.HCatalogStore


Thanks,

Jung JaeHwa

Re: Review Request 19406: TAJO-616: SequenceFile support

Posted by Jung JaeHwa <jh...@gruter.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/19406/
-----------------------------------------------------------

(Updated March 27, 2014, 5:05 p.m.)


Review request for Tajo.


Changes
-------

I attached the third patch after rebasing.

But this patch has detected weak performance.
To improve the performance, I need to optimize Hadoop SequenceFile.


So, I suggest to create an another issue for SequenceFile optimization after fixing this issue.


Bugs: TAJO-616
    https://issues.apache.org/jira/browse/TAJO-616


Repository: tajo


Description
-------

Hadoop users can create a sequence file format, and hive users also can create a table which was stored as SequenceFIle. So, we need to support sequence file format as follows:

* Update parser and catalog for SequenceFile
* Make serializer/deserializer configurable in SequenceFile
* Scanner for SequenceFile
* Support for compression/decompression of SequenceFile
* Compatible to apache hive


Diffs (updated)
-----

  tajo-catalog/tajo-catalog-common/src/main/java/org/apache/tajo/catalog/CatalogConstants.java c7116f6 
  tajo-catalog/tajo-catalog-common/src/main/java/org/apache/tajo/catalog/CatalogUtil.java 9078e60 
  tajo-catalog/tajo-catalog-common/src/main/proto/CatalogProtos.proto 06ffcfd 
  tajo-catalog/tajo-catalog-drivers/tajo-hcatalog/src/main/java/org/apache/tajo/catalog/store/HCatalogStore.java 6ef255f 
  tajo-catalog/tajo-catalog-drivers/tajo-hcatalog/src/main/java/org/apache/tajo/catalog/store/HCatalogUtil.java d426369 
  tajo-catalog/tajo-catalog-drivers/tajo-hcatalog/src/test/java/org/apache/tajo/catalog/store/TestHCatalogStore.java b9a734e 
  tajo-common/src/main/java/org/apache/tajo/util/Bytes.java f9ba923 
  tajo-core/tajo-core-backend/src/test/java/org/apache/tajo/engine/query/TestCTASQuery.java e6840f6 
  tajo-storage/pom.xml 9f144bb 
  tajo-storage/src/main/java/org/apache/tajo/storage/sequencefile/SequenceFileAppender.java PRE-CREATION 
  tajo-storage/src/main/java/org/apache/tajo/storage/sequencefile/SequenceFileScanner.java PRE-CREATION 
  tajo-storage/src/main/resources/storage-default.xml e3a0eaf 
  tajo-storage/src/test/java/org/apache/tajo/storage/TestCompressionStorages.java a776eb6 
  tajo-storage/src/test/java/org/apache/tajo/storage/TestMergeScanner.java ad12c06 
  tajo-storage/src/test/java/org/apache/tajo/storage/TestStorages.java b5338ff 
  tajo-storage/src/test/resources/storage-default.xml 75274d0 

Diff: https://reviews.apache.org/r/19406/diff/


Testing
-------

mvn clean install -Phcatalog-0.12.0


Thanks,

Jung JaeHwa

Re: Review Request 19406: TAJO-616: SequenceFile support

Posted by Jihoon Son <ji...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/19406/#review38441
-----------------------------------------------------------


Jaehwa, sorry for the late review.
The latest patch looks good, but it looks to be gone stale.
Would you rebase the patch, please?

- Jihoon Son


On March 24, 2014, 5:19 p.m., Jung JaeHwa wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/19406/
> -----------------------------------------------------------
> 
> (Updated March 24, 2014, 5:19 p.m.)
> 
> 
> Review request for Tajo.
> 
> 
> Bugs: TAJO-616
>     https://issues.apache.org/jira/browse/TAJO-616
> 
> 
> Repository: tajo
> 
> 
> Description
> -------
> 
> Hadoop users can create a sequence file format, and hive users also can create a table which was stored as SequenceFIle. So, we need to support sequence file format as follows:
> 
> * Update parser and catalog for SequenceFile
> * Make serializer/deserializer configurable in SequenceFile
> * Scanner for SequenceFile
> * Support for compression/decompression of SequenceFile
> * Compatible to apache hive
> 
> 
> Diffs
> -----
> 
>   tajo-catalog/tajo-catalog-common/src/main/java/org/apache/tajo/catalog/CatalogConstants.java fa23da1 
>   tajo-catalog/tajo-catalog-common/src/main/java/org/apache/tajo/catalog/CatalogUtil.java 2a6727e 
>   tajo-catalog/tajo-catalog-common/src/main/proto/CatalogProtos.proto 35171cc 
>   tajo-catalog/tajo-catalog-drivers/tajo-hcatalog/src/main/java/org/apache/tajo/catalog/store/HCatalogStore.java 5387673 
>   tajo-catalog/tajo-catalog-drivers/tajo-hcatalog/src/main/java/org/apache/tajo/catalog/store/HCatalogUtil.java fc2fdda 
>   tajo-catalog/tajo-catalog-drivers/tajo-hcatalog/src/test/java/org/apache/tajo/catalog/store/TestHCatalogStore.java cb3973a 
>   tajo-common/src/main/java/org/apache/tajo/util/Bytes.java f9ba923 
>   tajo-core/tajo-core-backend/src/test/java/org/apache/tajo/engine/planner/physical/TestPhysicalPlanner.java 1975a57 
>   tajo-storage/src/main/java/org/apache/tajo/storage/sequencefile/SequenceFileAppender.java PRE-CREATION 
>   tajo-storage/src/main/java/org/apache/tajo/storage/sequencefile/SequenceFileScanner.java PRE-CREATION 
>   tajo-storage/src/main/resources/storage-default.xml 5bf4453 
>   tajo-storage/src/test/java/org/apache/tajo/storage/TestCompressionStorages.java a776eb6 
>   tajo-storage/src/test/java/org/apache/tajo/storage/TestMergeScanner.java f2a66d9 
>   tajo-storage/src/test/java/org/apache/tajo/storage/TestStorages.java 1b36b78 
>   tajo-storage/src/test/java/org/apache/tajo/storage/v2/TestStorages.java 05be40a 
>   tajo-storage/src/test/resources/storage-default.xml 304af10 
> 
> Diff: https://reviews.apache.org/r/19406/diff/
> 
> 
> Testing
> -------
> 
> mvn clean install -Phcatalog-0.12.0
> 
> 
> Thanks,
> 
> Jung JaeHwa
> 
>

Re: Review Request 19406: TAJO-616: SequenceFile support

Posted by Jung JaeHwa <jh...@gruter.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/19406/
-----------------------------------------------------------

(Updated March 24, 2014, 8:19 a.m.)


Review request for Tajo.


Changes
-------

I added a method for handling null types.


Bugs: TAJO-616
    https://issues.apache.org/jira/browse/TAJO-616


Repository: tajo


Description
-------

Hadoop users can create a sequence file format, and hive users also can create a table which was stored as SequenceFIle. So, we need to support sequence file format as follows:

* Update parser and catalog for SequenceFile
* Make serializer/deserializer configurable in SequenceFile
* Scanner for SequenceFile
* Support for compression/decompression of SequenceFile
* Compatible to apache hive


Diffs (updated)
-----

  tajo-catalog/tajo-catalog-common/src/main/java/org/apache/tajo/catalog/CatalogConstants.java fa23da1 
  tajo-catalog/tajo-catalog-common/src/main/java/org/apache/tajo/catalog/CatalogUtil.java 2a6727e 
  tajo-catalog/tajo-catalog-common/src/main/proto/CatalogProtos.proto 35171cc 
  tajo-catalog/tajo-catalog-drivers/tajo-hcatalog/src/main/java/org/apache/tajo/catalog/store/HCatalogStore.java 5387673 
  tajo-catalog/tajo-catalog-drivers/tajo-hcatalog/src/main/java/org/apache/tajo/catalog/store/HCatalogUtil.java fc2fdda 
  tajo-catalog/tajo-catalog-drivers/tajo-hcatalog/src/test/java/org/apache/tajo/catalog/store/TestHCatalogStore.java cb3973a 
  tajo-common/src/main/java/org/apache/tajo/util/Bytes.java f9ba923 
  tajo-core/tajo-core-backend/src/test/java/org/apache/tajo/engine/planner/physical/TestPhysicalPlanner.java 1975a57 
  tajo-storage/src/main/java/org/apache/tajo/storage/sequencefile/SequenceFileAppender.java PRE-CREATION 
  tajo-storage/src/main/java/org/apache/tajo/storage/sequencefile/SequenceFileScanner.java PRE-CREATION 
  tajo-storage/src/main/resources/storage-default.xml 5bf4453 
  tajo-storage/src/test/java/org/apache/tajo/storage/TestCompressionStorages.java a776eb6 
  tajo-storage/src/test/java/org/apache/tajo/storage/TestMergeScanner.java f2a66d9 
  tajo-storage/src/test/java/org/apache/tajo/storage/TestStorages.java 1b36b78 
  tajo-storage/src/test/java/org/apache/tajo/storage/v2/TestStorages.java 05be40a 
  tajo-storage/src/test/resources/storage-default.xml 304af10 

Diff: https://reviews.apache.org/r/19406/diff/


Testing
-------

mvn clean install -Phcatalog-0.12.0


Thanks,

Jung JaeHwa