You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hive.apache.org by j....@gmail.com on 2019/04/02 22:29:40 UTC
Review Request 70372: HIVE-21427: Syslog storage handler
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/70372/
-----------------------------------------------------------
Review request for hive, Ashutosh Chauhan and Jason Dere.
Bugs: HIVE-21427
https://issues.apache.org/jira/browse/HIVE-21427
Repository: hive-git
Description
-------
HIVE-21427: Syslog storage handler
Diffs
-----
common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 777f8b51215523fca8e396ddf77139420666311a
data/files/syslog-hs2-2.log PRE-CREATION
data/files/syslog-hs2.log PRE-CREATION
itests/src/test/resources/testconfiguration.properties 96dfbc4b56b6eb3dff6b8e1e42a2371d090426e7
llap-server/src/java/org/apache/hadoop/hive/llap/io/api/impl/LlapRecordReader.java 9ef7af4eb0c9787a33d2aa4c9a4528b8f356106b
ql/src/java/org/apache/hadoop/hive/ql/io/sarg/ConvertAstToSearchArg.java 27fe828b7531584138cd002956a9fcc20f238f71
ql/src/java/org/apache/hadoop/hive/ql/log/syslog/SyslogInputFormat.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/log/syslog/SyslogParser.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/log/syslog/SyslogSerDe.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/log/syslog/SyslogStorageHandler.java PRE-CREATION
ql/src/test/org/apache/hadoop/hive/ql/log/TestSyslogInputFormat.java PRE-CREATION
ql/src/test/queries/clientpositive/syslog_parser.q PRE-CREATION
ql/src/test/queries/clientpositive/syslog_parser_file_pruning.q PRE-CREATION
ql/src/test/results/clientpositive/llap/syslog_parser.q.out PRE-CREATION
ql/src/test/results/clientpositive/llap/syslog_parser_file_pruning.q.out PRE-CREATION
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/tools/schematool/MetastoreSchemaTool.java eafe0c6d46d448bce287e61fabac0384b12b9295
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/tools/schematool/SchemaToolCommandLine.java 6282078411c4c728beed8e957aa857ed3c02133c
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/tools/schematool/SchemaToolTaskCreateLogsTable.java PRE-CREATION
Diff: https://reviews.apache.org/r/70372/diff/1/
Testing
-------
Thanks,
Prasanth_J
Re: Review Request 70372: HIVE-21427: Syslog storage handler
Posted by j....@gmail.com.
> On April 3, 2019, 9:54 p.m., Jason Dere wrote:
> > llap-server/src/java/org/apache/hadoop/hive/llap/io/api/impl/LlapRecordReader.java
> > Line 615 (original), 615 (patched)
> > <https://reviews.apache.org/r/70372/diff/1/?file=2136896#file2136896line615>
> >
> > Just curious about this one, was there a difference between rbCtx.getRowColumnTypeInfos() and rbCtx.getDataColumnCount()? Or just the fact that rbCtx.getDataColumnCount() directly returns an int value?
Yes. There is place in encoded reader creation that counts just the non-null columns in vector batch. But in here we include all columns including the nulls (partition columns at the last) so the text llap reader was throwing exception.
> On April 3, 2019, 9:54 p.m., Jason Dere wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/log/syslog/SyslogSerDe.java
> > Lines 57 (patched)
> > <https://reviews.apache.org/r/70372/diff/1/?file=2136900#file2136900line57>
> >
> > Is the list of columns from SyslogSerDe fixed to (facility, severity, version, ts, hostname, app_name, proc_id, msg_id, structured_data, msg, unmatched)? If so then should the column list/types be hardcoded rather than set via LIST_COLUMNS/LIST_COLUMNS_TYPES properties?
Yes. It is fixed schema. make sense.. The columns and types doesn't change anyway. Will fix it in next patch.
- Prasanth_J
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/70372/#review214337
-----------------------------------------------------------
On April 2, 2019, 10:29 p.m., Prasanth_J wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/70372/
> -----------------------------------------------------------
>
> (Updated April 2, 2019, 10:29 p.m.)
>
>
> Review request for hive, Ashutosh Chauhan and Jason Dere.
>
>
> Bugs: HIVE-21427
> https://issues.apache.org/jira/browse/HIVE-21427
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> HIVE-21427: Syslog storage handler
>
>
> Diffs
> -----
>
> common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 777f8b51215523fca8e396ddf77139420666311a
> data/files/syslog-hs2-2.log PRE-CREATION
> data/files/syslog-hs2.log PRE-CREATION
> itests/src/test/resources/testconfiguration.properties 96dfbc4b56b6eb3dff6b8e1e42a2371d090426e7
> llap-server/src/java/org/apache/hadoop/hive/llap/io/api/impl/LlapRecordReader.java 9ef7af4eb0c9787a33d2aa4c9a4528b8f356106b
> ql/src/java/org/apache/hadoop/hive/ql/io/sarg/ConvertAstToSearchArg.java 27fe828b7531584138cd002956a9fcc20f238f71
> ql/src/java/org/apache/hadoop/hive/ql/log/syslog/SyslogInputFormat.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/log/syslog/SyslogParser.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/log/syslog/SyslogSerDe.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/log/syslog/SyslogStorageHandler.java PRE-CREATION
> ql/src/test/org/apache/hadoop/hive/ql/log/TestSyslogInputFormat.java PRE-CREATION
> ql/src/test/queries/clientpositive/syslog_parser.q PRE-CREATION
> ql/src/test/queries/clientpositive/syslog_parser_file_pruning.q PRE-CREATION
> ql/src/test/results/clientpositive/llap/syslog_parser.q.out PRE-CREATION
> ql/src/test/results/clientpositive/llap/syslog_parser_file_pruning.q.out PRE-CREATION
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/tools/schematool/MetastoreSchemaTool.java eafe0c6d46d448bce287e61fabac0384b12b9295
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/tools/schematool/SchemaToolCommandLine.java 6282078411c4c728beed8e957aa857ed3c02133c
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/tools/schematool/SchemaToolTaskCreateLogsTable.java PRE-CREATION
>
>
> Diff: https://reviews.apache.org/r/70372/diff/1/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Prasanth_J
>
>
Re: Review Request 70372: HIVE-21427: Syslog storage handler
Posted by Jason Dere <jd...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/70372/#review214337
-----------------------------------------------------------
llap-server/src/java/org/apache/hadoop/hive/llap/io/api/impl/LlapRecordReader.java
Line 615 (original), 615 (patched)
<https://reviews.apache.org/r/70372/#comment300536>
Just curious about this one, was there a difference between rbCtx.getRowColumnTypeInfos() and rbCtx.getDataColumnCount()? Or just the fact that rbCtx.getDataColumnCount() directly returns an int value?
ql/src/java/org/apache/hadoop/hive/ql/log/syslog/SyslogSerDe.java
Lines 57 (patched)
<https://reviews.apache.org/r/70372/#comment300537>
Is the list of columns from SyslogSerDe fixed to (facility, severity, version, ts, hostname, app_name, proc_id, msg_id, structured_data, msg, unmatched)? If so then should the column list/types be hardcoded rather than set via LIST_COLUMNS/LIST_COLUMNS_TYPES properties?
- Jason Dere
On April 2, 2019, 10:29 p.m., Prasanth_J wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/70372/
> -----------------------------------------------------------
>
> (Updated April 2, 2019, 10:29 p.m.)
>
>
> Review request for hive, Ashutosh Chauhan and Jason Dere.
>
>
> Bugs: HIVE-21427
> https://issues.apache.org/jira/browse/HIVE-21427
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> HIVE-21427: Syslog storage handler
>
>
> Diffs
> -----
>
> common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 777f8b51215523fca8e396ddf77139420666311a
> data/files/syslog-hs2-2.log PRE-CREATION
> data/files/syslog-hs2.log PRE-CREATION
> itests/src/test/resources/testconfiguration.properties 96dfbc4b56b6eb3dff6b8e1e42a2371d090426e7
> llap-server/src/java/org/apache/hadoop/hive/llap/io/api/impl/LlapRecordReader.java 9ef7af4eb0c9787a33d2aa4c9a4528b8f356106b
> ql/src/java/org/apache/hadoop/hive/ql/io/sarg/ConvertAstToSearchArg.java 27fe828b7531584138cd002956a9fcc20f238f71
> ql/src/java/org/apache/hadoop/hive/ql/log/syslog/SyslogInputFormat.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/log/syslog/SyslogParser.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/log/syslog/SyslogSerDe.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/log/syslog/SyslogStorageHandler.java PRE-CREATION
> ql/src/test/org/apache/hadoop/hive/ql/log/TestSyslogInputFormat.java PRE-CREATION
> ql/src/test/queries/clientpositive/syslog_parser.q PRE-CREATION
> ql/src/test/queries/clientpositive/syslog_parser_file_pruning.q PRE-CREATION
> ql/src/test/results/clientpositive/llap/syslog_parser.q.out PRE-CREATION
> ql/src/test/results/clientpositive/llap/syslog_parser_file_pruning.q.out PRE-CREATION
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/tools/schematool/MetastoreSchemaTool.java eafe0c6d46d448bce287e61fabac0384b12b9295
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/tools/schematool/SchemaToolCommandLine.java 6282078411c4c728beed8e957aa857ed3c02133c
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/tools/schematool/SchemaToolTaskCreateLogsTable.java PRE-CREATION
>
>
> Diff: https://reviews.apache.org/r/70372/diff/1/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Prasanth_J
>
>
Re: Review Request 70372: HIVE-21427: Syslog storage handler
Posted by j....@gmail.com.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/70372/
-----------------------------------------------------------
(Updated April 5, 2019, 11:05 p.m.)
Review request for hive, Ashutosh Chauhan and Jason Dere.
Changes
-------
The timestamp change to ConvertAstToSearchArg is incorrect. Orc does not handle TIMESTAMP and TIMESTAMPLOCALTZ types correctly. ORC-189 is required. The timestamps in syslog all assume UTC (TIMESTAMP type), but expression filters in syslog_parser_file_pruning.q gets converted to local TZ (TIMESTAMPLOCALTZ) giving incorrect results. Also there is no way to force using UTC for the qfile tests. If we pass "-Duser.timezone=UTC" while running qfile it gives correct results but then precommit servers cannot run qfile test with "-Duser.timezone=UTC" option. We probably need a separate CliDriver for qfile tests that runs tests in UTC but that is out of scope for this ticket. I will disable syslog_parser_file_pruning.q for now as enabling it will give incorrect results depending on which TZ ptest server runs. Also reverted changes to ConvertAstToSearchArg.java file as ORC-189 will take care it.
Bugs: HIVE-21427
https://issues.apache.org/jira/browse/HIVE-21427
Repository: hive-git
Description
-------
HIVE-21427: Syslog storage handler
Diffs (updated)
-----
common/src/java/org/apache/hadoop/hive/conf/HiveConf.java aa27a01174
data/files/syslog-hs2-2.log PRE-CREATION
data/files/syslog-hs2-exception.log PRE-CREATION
data/files/syslog-hs2.log PRE-CREATION
itests/src/test/resources/testconfiguration.properties 96dfbc4b56
itests/util/src/main/java/org/apache/hadoop/hive/cli/control/CliConfigs.java 6f04602acb
llap-server/src/java/org/apache/hadoop/hive/llap/io/api/impl/LlapRecordReader.java 9ef7af4eb0
ql/src/java/org/apache/hadoop/hive/ql/log/syslog/SyslogInputFormat.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/log/syslog/SyslogParser.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/log/syslog/SyslogSerDe.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/log/syslog/SyslogStorageHandler.java PRE-CREATION
ql/src/test/org/apache/hadoop/hive/ql/log/TestSyslogInputFormat.java PRE-CREATION
ql/src/test/queries/clientpositive/syslog_parser.q PRE-CREATION
ql/src/test/queries/clientpositive/syslog_parser_exception.q PRE-CREATION
ql/src/test/queries/clientpositive/syslog_parser_file_pruning.q PRE-CREATION
ql/src/test/results/clientpositive/llap/syslog_parser.q.out PRE-CREATION
ql/src/test/results/clientpositive/llap/syslog_parser_exception.q.out PRE-CREATION
ql/src/test/results/clientpositive/llap/syslog_parser_file_pruning.q.out PRE-CREATION
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/tools/schematool/MetastoreSchemaTool.java eafe0c6d46
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/tools/schematool/SchemaToolCommandLine.java 6282078411
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/tools/schematool/SchemaToolTaskCreateLogsTable.java PRE-CREATION
Diff: https://reviews.apache.org/r/70372/diff/5/
Changes: https://reviews.apache.org/r/70372/diff/4-5/
Testing
-------
Thanks,
Prasanth_J
Re: Review Request 70372: HIVE-21427: Syslog storage handler
Posted by j....@gmail.com.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/70372/
-----------------------------------------------------------
(Updated April 5, 2019, 7:39 p.m.)
Review request for hive, Ashutosh Chauhan and Jason Dere.
Changes
-------
SyslogInputFormat extends TextInputFormat, textIF considers CR as new line if line delimiter is unspecified as a result multi-line exceptions were not returned by query results. .4 patch fixes it and adds a test case.
Bugs: HIVE-21427
https://issues.apache.org/jira/browse/HIVE-21427
Repository: hive-git
Description
-------
HIVE-21427: Syslog storage handler
Diffs (updated)
-----
common/src/java/org/apache/hadoop/hive/conf/HiveConf.java aa27a01174
data/files/syslog-hs2-2.log PRE-CREATION
data/files/syslog-hs2-exception.log PRE-CREATION
data/files/syslog-hs2.log PRE-CREATION
itests/src/test/resources/testconfiguration.properties 96dfbc4b56
llap-server/src/java/org/apache/hadoop/hive/llap/io/api/impl/LlapRecordReader.java 9ef7af4eb0
ql/src/java/org/apache/hadoop/hive/ql/io/sarg/ConvertAstToSearchArg.java 27fe828b75
ql/src/java/org/apache/hadoop/hive/ql/log/syslog/SyslogInputFormat.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/log/syslog/SyslogParser.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/log/syslog/SyslogSerDe.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/log/syslog/SyslogStorageHandler.java PRE-CREATION
ql/src/test/org/apache/hadoop/hive/ql/log/TestSyslogInputFormat.java PRE-CREATION
ql/src/test/queries/clientpositive/syslog_parser.q PRE-CREATION
ql/src/test/queries/clientpositive/syslog_parser_exception.q PRE-CREATION
ql/src/test/queries/clientpositive/syslog_parser_file_pruning.q PRE-CREATION
ql/src/test/results/clientpositive/llap/syslog_parser.q.out PRE-CREATION
ql/src/test/results/clientpositive/llap/syslog_parser_exception.q.out PRE-CREATION
ql/src/test/results/clientpositive/llap/syslog_parser_file_pruning.q.out PRE-CREATION
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/tools/schematool/MetastoreSchemaTool.java eafe0c6d46
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/tools/schematool/SchemaToolCommandLine.java 6282078411
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/tools/schematool/SchemaToolTaskCreateLogsTable.java PRE-CREATION
Diff: https://reviews.apache.org/r/70372/diff/4/
Changes: https://reviews.apache.org/r/70372/diff/3-4/
Testing
-------
Thanks,
Prasanth_J
Re: Review Request 70372: HIVE-21427: Syslog storage handler
Posted by j....@gmail.com.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/70372/
-----------------------------------------------------------
(Updated April 4, 2019, 9:28 p.m.)
Review request for hive, Ashutosh Chauhan and Jason Dere.
Changes
-------
fixes a bug where input row can have data from previous row.
Bugs: HIVE-21427
https://issues.apache.org/jira/browse/HIVE-21427
Repository: hive-git
Description
-------
HIVE-21427: Syslog storage handler
Diffs (updated)
-----
common/src/java/org/apache/hadoop/hive/conf/HiveConf.java aa27a01174
data/files/syslog-hs2-2.log PRE-CREATION
data/files/syslog-hs2.log PRE-CREATION
itests/src/test/resources/testconfiguration.properties 96dfbc4b56
llap-server/src/java/org/apache/hadoop/hive/llap/io/api/impl/LlapRecordReader.java 9ef7af4eb0
ql/src/java/org/apache/hadoop/hive/ql/io/sarg/ConvertAstToSearchArg.java 27fe828b75
ql/src/java/org/apache/hadoop/hive/ql/log/syslog/SyslogInputFormat.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/log/syslog/SyslogParser.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/log/syslog/SyslogSerDe.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/log/syslog/SyslogStorageHandler.java PRE-CREATION
ql/src/test/org/apache/hadoop/hive/ql/log/TestSyslogInputFormat.java PRE-CREATION
ql/src/test/queries/clientpositive/syslog_parser.q PRE-CREATION
ql/src/test/queries/clientpositive/syslog_parser_file_pruning.q PRE-CREATION
ql/src/test/results/clientpositive/llap/syslog_parser.q.out PRE-CREATION
ql/src/test/results/clientpositive/llap/syslog_parser_file_pruning.q.out PRE-CREATION
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/tools/schematool/MetastoreSchemaTool.java eafe0c6d46
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/tools/schematool/SchemaToolCommandLine.java 6282078411
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/tools/schematool/SchemaToolTaskCreateLogsTable.java PRE-CREATION
Diff: https://reviews.apache.org/r/70372/diff/3/
Changes: https://reviews.apache.org/r/70372/diff/2-3/
Testing
-------
Thanks,
Prasanth_J
Re: Review Request 70372: HIVE-21427: Syslog storage handler
Posted by j....@gmail.com.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/70372/
-----------------------------------------------------------
(Updated April 4, 2019, 6:18 p.m.)
Review request for hive, Ashutosh Chauhan and Jason Dere.
Changes
-------
review comment update
Bugs: HIVE-21427
https://issues.apache.org/jira/browse/HIVE-21427
Repository: hive-git
Description
-------
HIVE-21427: Syslog storage handler
Diffs (updated)
-----
common/src/java/org/apache/hadoop/hive/conf/HiveConf.java aa27a01174
data/files/syslog-hs2-2.log PRE-CREATION
data/files/syslog-hs2.log PRE-CREATION
itests/src/test/resources/testconfiguration.properties 96dfbc4b56
llap-server/src/java/org/apache/hadoop/hive/llap/io/api/impl/LlapRecordReader.java 9ef7af4eb0
ql/src/java/org/apache/hadoop/hive/ql/io/sarg/ConvertAstToSearchArg.java 27fe828b75
ql/src/java/org/apache/hadoop/hive/ql/log/syslog/SyslogInputFormat.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/log/syslog/SyslogParser.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/log/syslog/SyslogSerDe.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/log/syslog/SyslogStorageHandler.java PRE-CREATION
ql/src/test/org/apache/hadoop/hive/ql/log/TestSyslogInputFormat.java PRE-CREATION
ql/src/test/queries/clientpositive/syslog_parser.q PRE-CREATION
ql/src/test/queries/clientpositive/syslog_parser_file_pruning.q PRE-CREATION
ql/src/test/results/clientpositive/llap/syslog_parser.q.out PRE-CREATION
ql/src/test/results/clientpositive/llap/syslog_parser_file_pruning.q.out PRE-CREATION
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/tools/schematool/MetastoreSchemaTool.java eafe0c6d46
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/tools/schematool/SchemaToolCommandLine.java 6282078411
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/tools/schematool/SchemaToolTaskCreateLogsTable.java PRE-CREATION
Diff: https://reviews.apache.org/r/70372/diff/2/
Changes: https://reviews.apache.org/r/70372/diff/1-2/
Testing
-------
Thanks,
Prasanth_J