You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@drill.apache.org by "Abhishek Girish (JIRA)" <ji...@apache.org> on 2014/11/14 01:45:33 UTC
[jira] [Commented] (DRILL-1562) Parquet Writer hangs when
converting TPCH text data (SF100)
[ https://issues.apache.org/jira/browse/DRILL-1562?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14211608#comment-14211608 ]
Abhishek Girish commented on DRILL-1562:
----------------------------------------
This issue is not reproducible on recent builds. Can be marked as resolved.
> Parquet Writer hangs when converting TPCH text data (SF100)
> -----------------------------------------------------------
>
> Key: DRILL-1562
> URL: https://issues.apache.org/jira/browse/DRILL-1562
> Project: Apache Drill
> Issue Type: Bug
> Components: Storage - Parquet
> Reporter: Abhishek Girish
> Assignee: Parth Chandra
> Fix For: 0.7.0
>
> Attachments: hang.log
>
>
> Converting TPCH text data into Parquet hangs.
> Table name: lineitem
> Table size: ~80GB
> Input format: psv ('|' separated)
> Number of drillbits: 4
> DRILL_MAX_DIRECT_MEMORY="64G"
> DRILL_MAX_HEAP="32G"
> Query:
> > create table lineitem as select
> . . . . . . . . . . . . . . . . . > cast(columns[0] as int) l_orderkey,
> . . . . . . . . . . . . . . . . . > cast(columns[1] as int) l_partkey,
> . . . . . . . . . . . . . . . . . > cast(columns[2] as int) l_suppkey,
> . . . . . . . . . . . . . . . . . > cast(columns[3] as int) l_linenumber,
> . . . . . . . . . . . . . . . . . > cast(columns[4] as double) l_quantity,
> . . . . . . . . . . . . . . . . . > cast(columns[5] as double) l_extendedprice,
> . . . . . . . . . . . . . . . . . > cast(columns[6] as double) l_discount,
> . . . . . . . . . . . . . . . . . > cast(columns[7] as double) l_tax,
> . . . . . . . . . . . . . . . . . > cast(columns[8] as char(1)) l_returnflag,
> . . . . . . . . . . . . . . . . . > cast(columns[9] as char(1)) l_linestatus,
> . . . . . . . . . . . . . . . . . > cast(columns[10] as date) l_shipdate,
> . . . . . . . . . . . . . . . . . > cast(columns[11] as date) l_commitdate,
> . . . . . . . . . . . . . . . . . > cast(columns[12] as date) l_receiptdate,
> . . . . . . . . . . . . . . . . . > cast(columns[13] as char(25)) l_shipinstruct,
> . . . . . . . . . . . . . . . . . > cast(columns[14] as char(10)) l_shipmode,
> . . . . . . . . . . . . . . . . . > cast(columns[15] as varchar(200)) l_comment
> . . . . . . . . . . . . . . . . . > from dfs.`/tpch-text/scale100/lineitem` lineitem;
> +------------+---------------------------+
> | Fragment | Number of records written |
> +------------+---------------------------+
> | 1_58 | 4072947 |
> | 1_90 | 4088667 |
> | 1_38 | 4072639 |
> ...
> ...
> | 1_14 | 6109440 |
> <hangs>
> ...
> The drill-bit endpoint gets set to null. And the point of hang varies on each run.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)