You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@tajo.apache.org by "Jihoon Son (JIRA)" <ji...@apache.org> on 2014/12/22 07:31:13 UTC
[jira] [Created] (TAJO-1266) Too many logs when writing a parquet
relation
Jihoon Son created TAJO-1266:
--------------------------------
Summary: Too many logs when writing a parquet relation
Key: TAJO-1266
URL: https://issues.apache.org/jira/browse/TAJO-1266
Project: Tajo
Issue Type: Improvement
Reporter: Jihoon Son
See the title.
The parquet writer emits too many logs by default.
Here is a part of the whole log.
{noformat}
Dec 22, 2014 3:28:59 PM parquet.Log info
INFO: written 337B for [p_retailprice] DOUBLE: 737,002 values, 79B raw, 79B comp, 6 pages, encodings: [PLAIN_DICTIONARY, BIT_PACKED, RLE], dic { 2 entries, 16B raw, 2B comp}
Dec 22, 2014 3:28:59 PM parquet.Log info
INFO: written 790B for [p_comment] BINARY: 737,002 values, 143B raw, 143B comp, 11 pages, encodings: [PLAIN_DICTIONARY, BIT_PACKED, RLE], dic { 2 entries, 34B raw, 2B comp}
Dec 22, 2014 3:28:59 PM parquet.Log info
INFO: written 337B for [s_suppkey] INT64: 737,002 values, 79B raw, 79B comp, 6 pages, encodings: [PLAIN_DICTIONARY, BIT_PACKED, RLE], dic { 2 entries, 16B raw, 2B comp}
{noformat}
As you can see, a new log is emitted even when only 337B is additionally written.
This will significantly degrade the write performance, so we need to turn off the parquet log by default.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)