You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@drill.apache.org by Herman Tan <he...@redcubesg.com> on 2018/11/12 01:01:35 UTC
Re: ERROR is reading parquet data after create table
Hi,
I am running into this problem again.
The solution is to define a view that is a "union all" of individual select
queries of single csv files.
Can I request that the hardcoded timeout values be parametrized please?
Herman
On Tue, Oct 2, 2018 at 7:29 PM Herman Tan <he...@redcubesg.com> wrote:
> Hi,
>
> I found the hard coded parameter in the source code
>
>
> https://github.com/apache/drill/blob/8edeb49873d1a1710cfe28e0b49364d07eb1aef4/exec/java-exec/src/main/java/org/apache/drill/exec/store/TimedCallable.java
>
> LINE 52 : private static long TIMEOUT_PER_RUNNABLE_IN_MSECS = 15000;
> LINE 210 :timeout = TIMEOUT_PER_RUNNABLE_IN_MSECS * ((tasks.size() -
> 1)/parallelism + 1);
>
> parallelism param is also hardcoded
>
> https://github.com/apache/drill/blob/master/exec/java-exec/src/main/java/org/apache/drill/exec/store/parquet/metadata/Metadata.java
> LINE 343: 16
>
> In my case: task size is 29, 15000 * ((29-1)/(16+1)) = 15000 * 2 = 30000
> 16 runnables executes round robbin on 29 tasks. Each runnable given 30000
> ms to timeout.
>
> This is the error message:
> Waited for 30000 ms, but only 11 tasks for 'Fetch parquet metadata' are
> complete. Total number of tasks 29, parallelism 16.
>
> TimedCallable.java:
> LINE 248: final String errMsg = String.format("Waited for %d ms, but
> only %d tasks for '%s' are complete." +
> " Total number of tasks %d, parallelism %d.", timeout,
> futureMapper.count, activity, tasks.size(), parallelism);
>
> Shouldn't these be parameterized in "options" based on the infrastructure?
>
> Regards,
> Herman
>
>
>
>
> On Tue, Oct 2, 2018 at 6:47 PM Herman Tan <he...@redcubesg.com> wrote:
>
>> Hi,
>>
>> I have restarted drill and run the script again.
>>
>> select * from dfs.tmp.`load_pos_sales_detail_tbl`;
>> -- SQL Error: RESOURCE ERROR: Waited for 30000 ms, but only 11 tasks for
>> 'Fetch parquet metadata' are complete. Total number of tasks 29,
>> parallelism 16.
>>
>> The 29 tasks is related to the 29 parquet files in the folder.
>> To check if any of the parquet files has an error, I ran the following
>> SQL on each parquet file in the folder. ALL PASSED. (SQL Below).
>> select * from dfs.tmp.`load_pos_sales_detail_tbl/1_0_0.parquet`;
>>
>> So it seems that for this table drill can only get the metadata for 11
>> parquet files before it times out.
>> The time-out is a calculation and it varies from size of table.
>> I checked the source code but I cannot find the calculation of the
>> timeout of "30000 ms".
>> When I am lucky, drill can resolve the metadata for 29 files in 30000 ms
>> and it passes.
>>
>> I plan to use drill for production but this bothers me that there is a
>> limit on the number of parquet files and the timeout parameter cannot be
>> tuned.
>>
>> Does anyone have any ideas?
>>
>> Regards,
>> Herman
>> -------------- SQL BELOW -------------
>>
>> select * from dfs.tmp.`load_pos_sales_detail_tbl/1_0_0.parquet`;
>> select * from dfs.tmp.`load_pos_sales_detail_tbl/1_10_0.parquet`;
>> select * from dfs.tmp.`load_pos_sales_detail_tbl/1_10_1.parquet`;
>> select * from dfs.tmp.`load_pos_sales_detail_tbl/1_11_0.parquet`;
>> select * from dfs.tmp.`load_pos_sales_detail_tbl/1_11_1.parquet`;
>> select * from dfs.tmp.`load_pos_sales_detail_tbl/1_12_0.parquet`;
>> select * from dfs.tmp.`load_pos_sales_detail_tbl/1_12_1.parquet`;
>> select * from dfs.tmp.`load_pos_sales_detail_tbl/1_13_0.parquet`;
>> select * from dfs.tmp.`load_pos_sales_detail_tbl/1_13_1.parquet`;
>> select * from dfs.tmp.`load_pos_sales_detail_tbl/1_14_0.parquet`;
>> select * from dfs.tmp.`load_pos_sales_detail_tbl/1_15_0.parquet`;
>> select * from dfs.tmp.`load_pos_sales_detail_tbl/1_15_1.parquet`;
>> select * from dfs.tmp.`load_pos_sales_detail_tbl/1_16_0.parquet`;
>> select * from dfs.tmp.`load_pos_sales_detail_tbl/1_16_1.parquet`;
>> select * from dfs.tmp.`load_pos_sales_detail_tbl/1_1_0.parquet`;
>> select * from dfs.tmp.`load_pos_sales_detail_tbl/1_2_0.parquet`;
>> select * from dfs.tmp.`load_pos_sales_detail_tbl/1_3_0.parquet`;
>> select * from dfs.tmp.`load_pos_sales_detail_tbl/1_4_0.parquet`;
>> select * from dfs.tmp.`load_pos_sales_detail_tbl/1_4_1.parquet`;
>> select * from dfs.tmp.`load_pos_sales_detail_tbl/1_5_0.parquet`;
>> select * from dfs.tmp.`load_pos_sales_detail_tbl/1_5_1.parquet`;
>> select * from dfs.tmp.`load_pos_sales_detail_tbl/1_6_0.parquet`;
>> select * from dfs.tmp.`load_pos_sales_detail_tbl/1_6_1.parquet`;
>> select * from dfs.tmp.`load_pos_sales_detail_tbl/1_7_0.parquet`;
>> select * from dfs.tmp.`load_pos_sales_detail_tbl/1_7_1.parquet`;
>> select * from dfs.tmp.`load_pos_sales_detail_tbl/1_8_0.parquet`;
>> select * from dfs.tmp.`load_pos_sales_detail_tbl/1_8_1.parquet`;
>> select * from dfs.tmp.`load_pos_sales_detail_tbl/1_9_0.parquet`;
>> select * from dfs.tmp.`load_pos_sales_detail_tbl/1_9_1.parquet`;
>>
>>
>> On Tue, Oct 2, 2018 at 4:44 PM Herman Tan <he...@redcubesg.com> wrote:
>>
>>> Hi Divya and everyone,
>>>
>>> The problem has disappeared.
>>> Drill was not restarted.
>>> This appears to be intermittent.
>>> Before I submitted the error report, I ran the script several times and
>>> it failed all the time.
>>> Today I ran it again and it succeeded.
>>> I will restart and test again.
>>>
>>> Regards,
>>> Herman
>>>
>>>
>>>
>>> On Thu, Sep 27, 2018 at 11:50 AM Divya Gehlot <di...@gmail.com>
>>> wrote:
>>>
>>>> Hi Herman,
>>>> Just to ensure that your parquet file format is not corrupted , Can you
>>>> please query a folder like just 2001 or some of the files underneath
>>>> .Instead of querying the whole data set at once .
>>>>
>>>> Thanks,
>>>> Divya
>>>>
>>>> On Wed, 26 Sep 2018 at 15:35, Herman Tan <he...@redcubesg.com> wrote:
>>>>
>>>> > Hi Kunal,
>>>> >
>>>> > ----
>>>> > That said, could you provide some details about the parquet data
>>>> you've
>>>> > created, like the schema, parquet version and the tool used to
>>>> generate.
>>>> > Usually, the schema (and meta) provides most of these details for any
>>>> > parquet file.
>>>> > ----
>>>> >
>>>> > 1. The schema is under dfs.tmp, the queries to generate are all
>>>> documented
>>>> > below.
>>>> > 2. I don't know how to find the parquet version of the data file
>>>> > 3. The tool used to generate the parquest is apache drill. The CTAS
>>>> is
>>>> > detailed below.
>>>> >
>>>> > Regards,
>>>> > Herman
>>>> > ____________
>>>> >
>>>> > *This is the Text data*
>>>> >
>>>> > This is the folders of the files
>>>> > Total # of lines about 50 million rows
>>>> > ----------
>>>> > show files from
>>>> dfs.`D:\retail_sandbox\pos\sales_pos_detail\pos_details_
>>>> > 20180825`
>>>> > ;
>>>> > show files from
>>>> dfs.`D:\retail_sandbox\pos\sales_pos_detail\pos_details_
>>>> > 20180825\2011`
>>>> > ;
>>>> > -----
>>>> > sales_pos_detail
>>>> > \pos_details_20180825
>>>> > \2007
>>>> > \2008
>>>> > \2009
>>>> > \2010
>>>> > \2011
>>>> > \pos_details_0.csv
>>>> > \pos_details_1.csv
>>>> > \pos_details_2.csv
>>>> > \pos_details_3.csv
>>>> > \pos_details_4.csv
>>>> > \pos_details_5.csv
>>>> > \pos_details_6.csv
>>>> > \pos_details_7.csv
>>>> > \pos_details_8.csv
>>>> > \2012
>>>> > \2013
>>>> > \2014
>>>> > \2015
>>>> > \2016
>>>> > \2017
>>>> > \2018
>>>> > \others
>>>> > -----
>>>> >
>>>> > *This is the view with the metadata defined:*
>>>> >
>>>> > create or replace view dfs.tmp.load_pos_sales_detail as
>>>> > SELECT
>>>> > -- dimension keys
>>>> > cast(dim_date_key as int) dim_date_key
>>>> > ,cast(dim_site_key as int) dim_site_key
>>>> > ,cast(dim_pos_header_key as bigint) dim_pos_header_key
>>>> > ,cast(dim_pos_cashier_key as int) dim_pos_cashier_key
>>>> > ,cast(dim_card_number_key as int) dim_card_number_key
>>>> > ,cast(dim_hour_minute_key as int) dim_hour_minute_key
>>>> > ,cast(dim_pos_clerk_key as int) dim_pos_clerk_key
>>>> > ,cast(dim_product_key as int) dim_product_key
>>>> > ,cast(dim_pos_employee_purchase_key as int)
>>>> dim_pos_employee_purchase_key
>>>> > ,cast(dim_pos_terminal_key as int) dim_pos_terminal_key
>>>> > ,cast(dim_campaign_key as int) dim_campaign_key
>>>> > ,cast(dim_promo_key as int) dim_promo_key
>>>> > ,cast( case when dim_site_lfl_key = '' then 0 else dim_site_lfl_key
>>>> end as
>>>> > int) dim_site_lfl_key
>>>> > -- derived from keys
>>>> > ,dim_date_str
>>>> > ,`year` as `trx_year`
>>>> > -- Measures
>>>> > ,Product_Sales_Qty
>>>> > ,Product_Sales_Price
>>>> > ,Product_Cost_Price
>>>> > ,Product_Cost_Amt
>>>> > ,Product_Sales_Gross_Amt
>>>> > ,Product_Sales_Promo_Disc_Amt
>>>> > ,Product_Sales_Add_Promo_Disc_Amt
>>>> > ,Product_Sales_Total_Promo_Disc_Amt
>>>> > ,Product_Sales_Retail_Promo_Amt
>>>> > ,Product_Sales_Retail_Amt
>>>> > ,Product_Sales_VAT_Amt
>>>> > ,Product_Sales_Product_Margin_Amt
>>>> > ,Product_Sales_Initial_Margin_Amt
>>>> > from dfs.`D:\retail_sandbox\pos\sales_pos_detail\pos_details_20180825`
>>>> > ;
>>>> >
>>>> >
>>>> > *This is the CTAS that generates the parquet from the view above:*
>>>> >
>>>> > drop table if exists dfs.tmp.load_pos_sales_detail_tbl
>>>> > ;
>>>> >
>>>> > create table dfs.tmp.load_pos_sales_detail_tbl AS
>>>> > SELECT
>>>> > -- dimension keys
>>>> > dim_date_key
>>>> > ,dim_site_key
>>>> > ,dim_pos_header_key
>>>> > ,dim_pos_cashier_key
>>>> > ,dim_card_number_key
>>>> > ,dim_hour_minute_key
>>>> > ,dim_pos_clerk_key
>>>> > ,dim_product_key
>>>> > ,dim_pos_employee_purchase_key
>>>> > ,dim_pos_terminal_key
>>>> > ,dim_campaign_key
>>>> > ,dim_promo_key
>>>> > ,dim_site_lfl_key
>>>> > -- derived from keys
>>>> > ,dim_date_str
>>>> > ,`trx_year`
>>>> > -- Measures
>>>> > ,Product_Sales_Qty Sales_Qty
>>>> > ,Product_Sales_Price Sales_Price
>>>> > ,Product_Cost_Price Cost_Price
>>>> > ,Product_Cost_Amt Cost_Amt
>>>> > ,Product_Sales_Gross_Amt Sales_Gross_Amt
>>>> > ,Product_Sales_Promo_Disc_Amt Sales_Promo_Disc_Amt
>>>> > ,Product_Sales_Add_Promo_Disc_Amt Add_Promo_Disc_Amt
>>>> > ,Product_Sales_Total_Promo_Disc_Amt Total_Promo_Disc_Amt
>>>> > ,Product_Sales_Retail_Promo_Amt Retail_Promo_Amt
>>>> > ,Product_Sales_Retail_Amt Retail_Amt
>>>> > ,Product_Sales_VAT_Amt VAT_Amt
>>>> > ,Product_Sales_Product_Margin_Amt Product_Margin_Amt
>>>> > ,Product_Sales_Initial_Margin_Amt Initial_Margin_Amt
>>>> > from dfs.tmp.load_pos_sales_detail
>>>> > ;
>>>> >
>>>> >
>>>> > *This is the select query that generated the error:*
>>>> >
>>>> > select *
>>>> > from dfs.tmp.load_pos_sales_detail_tbl
>>>> > ;
>>>> >
>>>> > ----- ERROR ----------------------------
>>>> >
>>>> > SQL Error: RESOURCE ERROR: Waited for 30000 ms, but only 10 tasks for
>>>> > 'Fetch parquet metadata' are complete. Total number of tasks 29,
>>>> > parallelism 16.
>>>> >
>>>> >
>>>> > On Mon, Sep 24, 2018 at 9:08 AM, Kunal Khatua <ku...@apache.org>
>>>> wrote:
>>>> >
>>>> > > Hi Herman
>>>> > >
>>>> > > Assuming that you're doing analytics on your data. If that's the
>>>> case,
>>>> > > parquet format is the way to go.
>>>> > >
>>>> > > That said, could you provide some details about the parquet data
>>>> you've
>>>> > > created, like the schema, parquet version and the tool used to
>>>> generate.
>>>> > > Usually, the schema (and meta) provides most of these details for
>>>> any
>>>> > > parquet file.
>>>> > >
>>>> > > It'll be useful to know if there is a pattern in the failure
>>>> because of
>>>> > > which there might be corruption occurring.
>>>> > >
>>>> > > Kunal
>>>> > >
>>>> > >
>>>> > > On 9/22/2018 11:49:36 PM, Herman Tan <he...@redcubesg.com> wrote:
>>>> > > Hi Karthik,
>>>> > >
>>>> > > Thank you for pointing me to the mail archive in May 2018.
>>>> > > That is exactly the same problem I am facing.
>>>> > >
>>>> > > I thought of using Drill as an ETL where I load the warehouse
>>>> parquet
>>>> > > tables from text source files.
>>>> > > Then I query the parquet tables.
>>>> > > It works on some parquet tables but am having problems with large
>>>> ones
>>>> > that
>>>> > > consist of several files. (I think)
>>>> > > Still investigating.
>>>> > > Anyone in the community have other experience?
>>>> > > Should I work with all text files instead of parquet?
>>>> > >
>>>> > >
>>>> > > Herman
>>>> > >
>>>> > >
>>>> > > On Fri, Sep 21, 2018 at 2:15 AM, Karthikeyan Manivannan
>>>> > > kmanivannan@mapr.com> wrote:
>>>> > >
>>>> > > > Hi Herman,
>>>> > > >
>>>> > > > I am not sure what the exact problem here is but can you check to
>>>> see
>>>> > if
>>>> > > > you are not hitting the problem described here:
>>>> > > >
>>>> > > > http://mail-archives.apache.org/mod_mbox/drill-user/201805.mbox/%
>>>> > > >
>>>> 3CCACwRgneXLXoP2vCYuGsA4Gwd1jGS8F+rcpzQ8rHuatFW5fmRaQ@mail.gmail.com
>>>> > %3E
>>>> > > >
>>>> > > > Thanks
>>>> > > >
>>>> > > > Karthik
>>>> > > >
>>>> > > > On Wed, Sep 19, 2018 at 7:02 PM Herman Tan wrote:
>>>> > > >
>>>> > > > > Hi,
>>>> > > > >
>>>> > > > > I encountered the following error.
>>>> > > > > The Steps I did are as follows:
>>>> > > > > 1. Create a view to fix the data type of fields with cast
>>>> > > > > 2. Create table (parquet) using the view
>>>> > > > > 3. Query select * from table (query a field also does not work)
>>>> > > > >
>>>> > > > > The error:
>>>> > > > > SQL Error: RESOURCE ERROR: Waited for 30000 ms, but only 10
>>>> tasks for
>>>> > > > > 'Fetch parquet metadata' are complete. Total number of tasks 29,
>>>> > > > > parallelism 16.
>>>> > > > >
>>>> > > > > When I re-run this, the number of tasks will vary.
>>>> > > > >
>>>> > > > > What could be the problem?
>>>> > > > >
>>>> > > > > Regards,
>>>> > > > > Herman Tan
>>>> > > > >
>>>> > > > > More info below:
>>>> > > > >
>>>> > > > > This is the folders of the files
>>>> > > > > Total # of lines, 50 million
>>>> > > > > ----------
>>>> > > > > show files from
>>>> > > > >
>>>> dfs.`D:\retail_sandbox\pos\sales_pos_detail\pos_details_20180825`
>>>> > > > > ;
>>>> > > > > show files from
>>>> > > > >
>>>> > dfs.`D:\retail_sandbox\pos\sales_pos_detail\pos_details_20180825\2011`
>>>> > > > > ;
>>>> > > > > -----
>>>> > > > > sales_pos_detail
>>>> > > > > \pos_details_20180825
>>>> > > > > \2007
>>>> > > > > \2008
>>>> > > > > \2009
>>>> > > > > \2010
>>>> > > > > \2011
>>>> > > > > \pos_details_0.csv
>>>> > > > > \pos_details_1.csv
>>>> > > > > \pos_details_2.csv
>>>> > > > > \pos_details_3.csv
>>>> > > > > \pos_details_4.csv
>>>> > > > > \pos_details_5.csv
>>>> > > > > \pos_details_6.csv
>>>> > > > > \pos_details_7.csv
>>>> > > > > \pos_details_8.csv
>>>> > > > > \2012
>>>> > > > > \2013
>>>> > > > > \2014
>>>> > > > > \2015
>>>> > > > > \2016
>>>> > > > > \2017
>>>> > > > > \2018
>>>> > > > > \others
>>>> > > > > -----
>>>> > > > >
>>>> > > > > create or replace view dfs.tmp.load_pos_sales_detail as
>>>> > > > > SELECT
>>>> > > > > -- dimension keys
>>>> > > > > cast(dim_date_key as int) dim_date_key
>>>> > > > > ,cast(dim_site_key as int) dim_site_key
>>>> > > > > ,cast(dim_pos_header_key as bigint) dim_pos_header_key
>>>> > > > > ,cast(dim_pos_cashier_key as int) dim_pos_cashier_key
>>>> > > > > ,cast(dim_card_number_key as int) dim_card_number_key
>>>> > > > > ,cast(dim_hour_minute_key as int) dim_hour_minute_key
>>>> > > > > ,cast(dim_pos_clerk_key as int) dim_pos_clerk_key
>>>> > > > > ,cast(dim_product_key as int) dim_product_key
>>>> > > > > ,cast(dim_pos_employee_purchase_key as int)
>>>> > > > dim_pos_employee_purchase_key
>>>> > > > > ,cast(dim_pos_terminal_key as int) dim_pos_terminal_key
>>>> > > > > ,cast(dim_campaign_key as int) dim_campaign_key
>>>> > > > > ,cast(dim_promo_key as int) dim_promo_key
>>>> > > > > ,cast( case when dim_site_lfl_key = '' then 0 else
>>>> dim_site_lfl_key
>>>> > end
>>>> > > > as
>>>> > > > > int) dim_site_lfl_key
>>>> > > > > -- derived from keys
>>>> > > > > ,dim_date_str
>>>> > > > > ,`year` as `trx_year`
>>>> > > > > -- Measures
>>>> > > > > ,Product_Sales_Qty
>>>> > > > > ,Product_Sales_Price
>>>> > > > > ,Product_Cost_Price
>>>> > > > > ,Product_Cost_Amt
>>>> > > > > ,Product_Sales_Gross_Amt
>>>> > > > > ,Product_Sales_Promo_Disc_Amt
>>>> > > > > ,Product_Sales_Add_Promo_Disc_Amt
>>>> > > > > ,Product_Sales_Total_Promo_Disc_Amt
>>>> > > > > ,Product_Sales_Retail_Promo_Amt
>>>> > > > > ,Product_Sales_Retail_Amt
>>>> > > > > ,Product_Sales_VAT_Amt
>>>> > > > > ,Product_Sales_Product_Margin_Amt
>>>> > > > > ,Product_Sales_Initial_Margin_Amt
>>>> > > > > from
>>>> > dfs.`D:\retail_sandbox\pos\sales_pos_detail\pos_details_20180825`
>>>> > > > > ;
>>>> > > > >
>>>> > > > > drop table if exists dfs.tmp.load_pos_sales_detail_tbl
>>>> > > > > ;
>>>> > > > >
>>>> > > > > create table dfs.tmp.load_pos_sales_detail_tbl AS
>>>> > > > > SELECT
>>>> > > > > -- dimension keys
>>>> > > > > dim_date_key
>>>> > > > > ,dim_site_key
>>>> > > > > ,dim_pos_header_key
>>>> > > > > ,dim_pos_cashier_key
>>>> > > > > ,dim_card_number_key
>>>> > > > > ,dim_hour_minute_key
>>>> > > > > ,dim_pos_clerk_key
>>>> > > > > ,dim_product_key
>>>> > > > > ,dim_pos_employee_purchase_key
>>>> > > > > ,dim_pos_terminal_key
>>>> > > > > ,dim_campaign_key
>>>> > > > > ,dim_promo_key
>>>> > > > > ,dim_site_lfl_key
>>>> > > > > -- derived from keys
>>>> > > > > ,dim_date_str
>>>> > > > > ,`trx_year`
>>>> > > > > -- Measures
>>>> > > > > ,Product_Sales_Qty Sales_Qty
>>>> > > > > ,Product_Sales_Price Sales_Price
>>>> > > > > ,Product_Cost_Price Cost_Price
>>>> > > > > ,Product_Cost_Amt Cost_Amt
>>>> > > > > ,Product_Sales_Gross_Amt Sales_Gross_Amt
>>>> > > > > ,Product_Sales_Promo_Disc_Amt Sales_Promo_Disc_Amt
>>>> > > > > ,Product_Sales_Add_Promo_Disc_Amt Add_Promo_Disc_Amt
>>>> > > > > ,Product_Sales_Total_Promo_Disc_Amt Total_Promo_Disc_Amt
>>>> > > > > ,Product_Sales_Retail_Promo_Amt Retail_Promo_Amt
>>>> > > > > ,Product_Sales_Retail_Amt Retail_Amt
>>>> > > > > ,Product_Sales_VAT_Amt VAT_Amt
>>>> > > > > ,Product_Sales_Product_Margin_Amt Product_Margin_Amt
>>>> > > > > ,Product_Sales_Initial_Margin_Amt Initial_Margin_Amt
>>>> > > > > from dfs.tmp.load_pos_sales_detail
>>>> > > > > ;
>>>> > > > >
>>>> > > > > select *
>>>> > > > > from dfs.tmp.load_pos_sales_detail_tbl
>>>> > > > > ;
>>>> > > > >
>>>> > > > > ----- ERROR ----------------------------
>>>> > > > >
>>>> > > > > SQL Error: RESOURCE ERROR: Waited for 30000 ms, but only 10
>>>> tasks for
>>>> > > > > 'Fetch parquet metadata' are complete. Total number of tasks 29,
>>>> > > > > parallelism 16.
>>>> > > > >
>>>> > > > >
>>>> > > > > [Error Id: 3b079174-f5d0-4313-8097-25a0b3070854 on
>>>> > > > > IORA-G9KY9P2.stf.nus.edu.sg:31010]
>>>> > > > > RESOURCE ERROR: Waited for 30000 ms, but only 10 tasks for
>>>> 'Fetch
>>>> > > > parquet
>>>> > > > > metadata' are complete. Total number of tasks 29, parallelism
>>>> 16.
>>>> > > > >
>>>> > > > >
>>>> > > > > [Error Id: 3b079174-f5d0-4313-8097-25a0b3070854 on
>>>> > > > > IORA-G9KY9P2.stf.nus.edu.sg:31010]
>>>> > > > > RESOURCE ERROR: Waited for 30000 ms, but only 10 tasks for
>>>> 'Fetch
>>>> > > > > parquet metadata' are complete. Total number of tasks 29,
>>>> parallelism
>>>> > > 16.
>>>> > > > >
>>>> > > > >
>>>> > > > > [Error Id: 3b079174-f5d0-4313-8097-25a0b3070854 on
>>>> > > > > IORA-G9KY9P2.stf.nus.edu.sg:31010]
>>>> > > > > RESOURCE ERROR: Waited for 30000 ms, but only 10 tasks for
>>>> 'Fetch
>>>> > > > > parquet metadata' are complete. Total number of tasks 29,
>>>> parallelism
>>>> > > 16.
>>>> > > > >
>>>> > > > >
>>>> > > > > [Error Id: 3b079174-f5d0-4313-8097-25a0b3070854 on
>>>> > > > > IORA-G9KY9P2.stf.nus.edu.sg:31010]
>>>> > > > >
>>>> > > > > ----------------------------------------
>>>> > > > > From Drill log:
>>>> > > > >
>>>> > > > > 2018-09-20 08:58:12,035
>>>> > [245d0f5a-ae5f-bfa2-ff04-40f7bdd1c2bf:foreman]
>>>> > > > > INFO o.a.drill.exec.work.foreman.Foreman - Query text for query
>>>> id
>>>> > > > > 245d0f5a-ae5f-bfa2-ff04-40f7bdd1c2bf: select *
>>>> > > > > from dfs.tmp.load_pos_sales_detail_tbl
>>>> > > > >
>>>> > > > > 2018-09-20 08:58:53,068
>>>> > [245d0f5a-ae5f-bfa2-ff04-40f7bdd1c2bf:foreman]
>>>> > > > > ERROR o.a.d.e.s.parquet.metadata.Metadata - Waited for 30000
>>>> ms, but
>>>> > > > only
>>>> > > > > 10 tasks for 'Fetch parquet metadata' are complete. Total
>>>> number of
>>>> > > tasks
>>>> > > > > 29, parallelism 16.
>>>> > > > > java.util.concurrent.CancellationException: null
>>>> > > > > at
>>>> > > > >
>>>> > > > > org.apache.drill.exec.store.TimedCallable$FutureMapper.
>>>> > > > apply(TimedCallable.java:86)
>>>> > > > > ~[drill-java-exec-1.14.0.jar:1.14.0]
>>>> > > > > at
>>>> > > > >
>>>> > > > > org.apache.drill.exec.store.TimedCallable$FutureMapper.
>>>> > > > apply(TimedCallable.java:57)
>>>> > > > > ~[drill-java-exec-1.14.0.jar:1.14.0]
>>>> > > > > at
>>>> > > > >
>>>> > > > > org.apache.drill.common.collections.Collectors.lambda$
>>>> > > > toList$2(Collectors.java:97)
>>>> > > > > ~[drill-common-1.14.0.jar:1.14.0]
>>>> > > > > at java.util.ArrayList.forEach(ArrayList.java:1257)
>>>> ~[na:1.8.0_172]
>>>> > > > > at
>>>> > > > > org.apache.drill.common.collections.Collectors.toList(
>>>> > > > Collectors.java:97)
>>>> > > > > ~[drill-common-1.14.0.jar:1.14.0]
>>>> > > > > at org.apache.drill.exec.store.TimedCallable.run(
>>>> > > TimedCallable.java:214)
>>>> > > > > ~[drill-java-exec-1.14.0.jar:1.14.0]
>>>> > > > > at
>>>> > > > >
>>>> > > > > org.apache.drill.exec.store.parquet.metadata.Metadata.
>>>> > > > getParquetFileMetadata_v3(Metadata.java:340)
>>>> > > > > [drill-java-exec-1.14.0.jar:1.14.0]
>>>> > > > > at
>>>> > > > >
>>>> > > > > org.apache.drill.exec.store.parquet.metadata.Metadata.
>>>> > > > getParquetTableMetadata(Metadata.java:324)
>>>> > > > > [drill-java-exec-1.14.0.jar:1.14.0]
>>>> > > > > at
>>>> > > > >
>>>> > > > > org.apache.drill.exec.store.parquet.metadata.Metadata.
>>>> > > > getParquetTableMetadata(Metadata.java:305)
>>>> > > > > [drill-java-exec-1.14.0.jar:1.14.0]
>>>> > > > > at
>>>> > > > >
>>>> > > > > org.apache.drill.exec.store.parquet.metadata.Metadata.
>>>> > > > getParquetTableMetadata(Metadata.java:124)
>>>> > > > > [drill-java-exec-1.14.0.jar:1.14.0]
>>>> > > > > at
>>>> > > > >
>>>> > > > > org.apache.drill.exec.store.parquet.ParquetGroupScan.
>>>> > > > initInternal(ParquetGroupScan.java:254)
>>>> > > > > [drill-java-exec-1.14.0.jar:1.14.0]
>>>> > > > > at
>>>> > > > >
>>>> > > > >
>>>> org.apache.drill.exec.store.parquet.AbstractParquetGroupScan.init(
>>>> > > > AbstractParquetGroupScan.java:380)
>>>> > > > > [drill-java-exec-1.14.0.jar:1.14.0]
>>>> > > > > at
>>>> > > > >
>>>> > > > > org.apache.drill.exec.store.parquet.ParquetGroupScan.
>>>> > > > init>(ParquetGroupScan.java:132)
>>>> > > > > [drill-java-exec-1.14.0.jar:1.14.0]
>>>> > > > > at
>>>> > > > >
>>>> > > > > org.apache.drill.exec.store.parquet.ParquetGroupScan.
>>>> > > > init>(ParquetGroupScan.java:102)
>>>> > > > > [drill-java-exec-1.14.0.jar:1.14.0]
>>>> > > > > at
>>>> > > > >
>>>> > > > >
>>>> org.apache.drill.exec.store.parquet.ParquetFormatPlugin.getGroupScan(
>>>> > > > ParquetFormatPlugin.java:180)
>>>> > > > > [drill-java-exec-1.14.0.jar:1.14.0]
>>>> > > > > at
>>>> > > > >
>>>> > > > >
>>>> org.apache.drill.exec.store.parquet.ParquetFormatPlugin.getGroupScan(
>>>> > > > ParquetFormatPlugin.java:70)
>>>> > > > > [drill-java-exec-1.14.0.jar:1.14.0]
>>>> > > > > at
>>>> > > > >
>>>> > > > >
>>>> org.apache.drill.exec.store.dfs.FileSystemPlugin.getPhysicalScan(
>>>> > > > FileSystemPlugin.java:136)
>>>> > > > > [drill-java-exec-1.14.0.jar:1.14.0]
>>>> > > > > at
>>>> > > > >
>>>> > > > >
>>>> org.apache.drill.exec.store.AbstractStoragePlugin.getPhysicalScan(
>>>> > > > AbstractStoragePlugin.java:116)
>>>> > > > > [drill-java-exec-1.14.0.jar:1.14.0]
>>>> > > > > at
>>>> > > > >
>>>> > > > >
>>>> org.apache.drill.exec.store.AbstractStoragePlugin.getPhysicalScan(
>>>> > > > AbstractStoragePlugin.java:111)
>>>> > > > > [drill-java-exec-1.14.0.jar:1.14.0]
>>>> > > > > at
>>>> > > > >
>>>> > > > > org.apache.drill.exec.planner.logical.DrillTable.
>>>> > > > getGroupScan(DrillTable.java:99)
>>>> > > > > [drill-java-exec-1.14.0.jar:1.14.0]
>>>> > > > > at
>>>> > > > >
>>>> > > > > org.apache.drill.exec.planner.logical.DrillScanRel.(
>>>> > > > DrillScanRel.java:89)
>>>> > > > > [drill-java-exec-1.14.0.jar:1.14.0]
>>>> > > > > at
>>>> > > > >
>>>> > > > > org.apache.drill.exec.planner.logical.DrillScanRel.(
>>>> > > > DrillScanRel.java:69)
>>>> > > > > [drill-java-exec-1.14.0.jar:1.14.0]
>>>> > > > > at
>>>> > > > >
>>>> > > > > org.apache.drill.exec.planner.logical.DrillScanRel.(
>>>> > > > DrillScanRel.java:62)
>>>> > > > > [drill-java-exec-1.14.0.jar:1.14.0]
>>>> > > > > at
>>>> > > > >
>>>> > > > > org.apache.drill.exec.planner.logical.DrillScanRule.onMatch(
>>>> > > > DrillScanRule.java:38)
>>>> > > > > [drill-java-exec-1.14.0.jar:1.14.0]
>>>> > > > > at
>>>> > > > >
>>>> > > > > org.apache.calcite.plan.volcano.VolcanoRuleCall.
>>>> > > > onMatch(VolcanoRuleCall.java:212)
>>>> > > > > [calcite-core-1.16.0-drill-r6.jar:1.16.0-drill-r6]
>>>> > > > > at
>>>> > > > >
>>>> > > > > org.apache.calcite.plan.volcano.VolcanoPlanner.
>>>> > > > findBestExp(VolcanoPlanner.java:652)
>>>> > > > > [calcite-core-1.16.0-drill-r6.jar:1.16.0-drill-r6]
>>>> > > > > at org.apache.calcite.tools.Programs$RuleSetProgram.run(
>>>> > > > Programs.java:368)
>>>> > > > > [calcite-core-1.16.0-drill-r6.jar:1.16.0-drill-r6]
>>>> > > > > at
>>>> > > > >
>>>> > > > > org.apache.drill.exec.planner.sql.handlers.
>>>> > > DefaultSqlHandler.transform(
>>>> > > > DefaultSqlHandler.java:429)
>>>> > > > > [drill-java-exec-1.14.0.jar:1.14.0]
>>>> > > > > at
>>>> > > > >
>>>> > > > > org.apache.drill.exec.planner.sql.handlers.
>>>> > > DefaultSqlHandler.transform(
>>>> > > > DefaultSqlHandler.java:369)
>>>> > > > > [drill-java-exec-1.14.0.jar:1.14.0]
>>>> > > > > at
>>>> > > > >
>>>> > > > > org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.
>>>> > > > convertToRawDrel(DefaultSqlHandler.java:255)
>>>> > > > > [drill-java-exec-1.14.0.jar:1.14.0]
>>>> > > > > at
>>>> > > > >
>>>> > > > > org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.
>>>> > > > convertToDrel(DefaultSqlHandler.java:318)
>>>> > > > > [drill-java-exec-1.14.0.jar:1.14.0]
>>>> > > > > at
>>>> > > > >
>>>> > > > >
>>>> org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.getPlan(
>>>> > > > DefaultSqlHandler.java:180)
>>>> > > > > [drill-java-exec-1.14.0.jar:1.14.0]
>>>> > > > > at
>>>> > > > >
>>>> > > > > org.apache.drill.exec.planner.sql.DrillSqlWorker.
>>>> > > > getQueryPlan(DrillSqlWorker.java:145)
>>>> > > > > [drill-java-exec-1.14.0.jar:1.14.0]
>>>> > > > > at
>>>> > > > >
>>>> > > > > org.apache.drill.exec.planner.sql.DrillSqlWorker.getPlan(
>>>> > > > DrillSqlWorker.java:83)
>>>> > > > > [drill-java-exec-1.14.0.jar:1.14.0]
>>>> > > > > at org.apache.drill.exec.work
>>>> > .foreman.Foreman.runSQL(Foreman.java:567)
>>>> > > > > [drill-java-exec-1.14.0.jar:1.14.0]
>>>> > > > > at org.apache.drill.exec.work
>>>> .foreman.Foreman.run(Foreman.java:266)
>>>> > > > > [drill-java-exec-1.14.0.jar:1.14.0]
>>>> > > > > at
>>>> > > > >
>>>> > > > > java.util.concurrent.ThreadPoolExecutor.runWorker(
>>>> > > > ThreadPoolExecutor.java:1149)
>>>> > > > > [na:1.8.0_172]
>>>> > > > > at
>>>> > > > >
>>>> > > > > java.util.concurrent.ThreadPoolExecutor$Worker.run(
>>>> > > > ThreadPoolExecutor.java:624)
>>>> > > > > [na:1.8.0_172]
>>>> > > > > at java.lang.Thread.run(Thread.java:748) [na:1.8.0_172]
>>>> > > > > 2018-09-20 08:58:53,080
>>>> > [245d0f5a-ae5f-bfa2-ff04-40f7bdd1c2bf:foreman]
>>>> > > > > INFO o.a.d.e.s.parquet.metadata.Metadata - User Error Occurred:
>>>> > Waited
>>>> > > > for
>>>> > > > > 30000 ms, but only 10 tasks for 'Fetch parquet metadata' are
>>>> > complete.
>>>> > > > > Total number of tasks 29, parallelism 16. (null)
>>>> > > > > org.apache.drill.common.exceptions.UserException: RESOURCE
>>>> ERROR:
>>>> > > Waited
>>>> > > > > for 30000 ms, but only 10 tasks for 'Fetch parquet metadata' are
>>>> > > > complete.
>>>> > > > > Total number of tasks 29, parallelism 16.
>>>> > > > >
>>>> > > > >
>>>> > > > > [Error Id: f887dcae-9f55-469c-be52-b6ce2a37eeb0 ]
>>>> > > > > at
>>>> > > > >
>>>> > > > > org.apache.drill.common.exceptions.UserException$
>>>> > > > Builder.build(UserException.java:633)
>>>> > > > > ~[drill-common-1.14.0.jar:1.14.0]
>>>> > > > > at org.apache.drill.exec.store.TimedCallable.run(
>>>> > > TimedCallable.java:253)
>>>> > > > > [drill-java-exec-1.14.0.jar:1.14.0]
>>>> > > > > at
>>>> > > > >
>>>> > > > > org.apache.drill.exec.store.parquet.metadata.Metadata.
>>>> > > > getParquetFileMetadata_v3(Metadata.java:340)
>>>> > > > > [drill-java-exec-1.14.0.jar:1.14.0]
>>>> > > > > at
>>>> > > > >
>>>> > > > > org.apache.drill.exec.store.parquet.metadata.Metadata.
>>>> > > > getParquetTableMetadata(Metadata.java:324)
>>>> > > > > [drill-java-exec-1.14.0.jar:1.14.0]
>>>> > > > > at
>>>> > > > >
>>>> > > > > org.apache.drill.exec.store.parquet.metadata.Metadata.
>>>> > > > getParquetTableMetadata(Metadata.java:305)
>>>> > > > > [drill-java-exec-1.14.0.jar:1.14.0]
>>>> > > > > at
>>>> > > > >
>>>> > > > > org.apache.drill.exec.store.parquet.metadata.Metadata.
>>>> > > > getParquetTableMetadata(Metadata.java:124)
>>>> > > > > [drill-java-exec-1.14.0.jar:1.14.0]
>>>> > > > > at
>>>> > > > >
>>>> > > > > org.apache.drill.exec.store.parquet.ParquetGroupScan.
>>>> > > > initInternal(ParquetGroupScan.java:254)
>>>> > > > > [drill-java-exec-1.14.0.jar:1.14.0]
>>>> > > > > at
>>>> > > > >
>>>> > > > >
>>>> org.apache.drill.exec.store.parquet.AbstractParquetGroupScan.init(
>>>> > > > AbstractParquetGroupScan.java:380)
>>>> > > > > [drill-java-exec-1.14.0.jar:1.14.0]
>>>> > > > > at
>>>> > > > >
>>>> > > > > org.apache.drill.exec.store.parquet.ParquetGroupScan.
>>>> > > > init>(ParquetGroupScan.java:132)
>>>> > > > > [drill-java-exec-1.14.0.jar:1.14.0]
>>>> > > > > at
>>>> > > > >
>>>> > > > > org.apache.drill.exec.store.parquet.ParquetGroupScan.
>>>> > > > init>(ParquetGroupScan.java:102)
>>>> > > > > [drill-java-exec-1.14.0.jar:1.14.0]
>>>> > > > > at
>>>> > > > >
>>>> > > > >
>>>> org.apache.drill.exec.store.parquet.ParquetFormatPlugin.getGroupScan(
>>>> > > > ParquetFormatPlugin.java:180)
>>>> > > > > [drill-java-exec-1.14.0.jar:1.14.0]
>>>> > > > > at
>>>> > > > >
>>>> > > > >
>>>> org.apache.drill.exec.store.parquet.ParquetFormatPlugin.getGroupScan(
>>>> > > > ParquetFormatPlugin.java:70)
>>>> > > > > [drill-java-exec-1.14.0.jar:1.14.0]
>>>> > > > > at
>>>> > > > >
>>>> > > > >
>>>> org.apache.drill.exec.store.dfs.FileSystemPlugin.getPhysicalScan(
>>>> > > > FileSystemPlugin.java:136)
>>>> > > > > [drill-java-exec-1.14.0.jar:1.14.0]
>>>> > > > > at
>>>> > > > >
>>>> > > > >
>>>> org.apache.drill.exec.store.AbstractStoragePlugin.getPhysicalScan(
>>>> > > > AbstractStoragePlugin.java:116)
>>>> > > > > [drill-java-exec-1.14.0.jar:1.14.0]
>>>> > > > > at
>>>> > > > >
>>>> > > > >
>>>> org.apache.drill.exec.store.AbstractStoragePlugin.getPhysicalScan(
>>>> > > > AbstractStoragePlugin.java:111)
>>>> > > > > [drill-java-exec-1.14.0.jar:1.14.0]
>>>> > > > > at
>>>> > > > >
>>>> > > > > org.apache.drill.exec.planner.logical.DrillTable.
>>>> > > > getGroupScan(DrillTable.java:99)
>>>> > > > > [drill-java-exec-1.14.0.jar:1.14.0]
>>>> > > > > at
>>>> > > > >
>>>> > > > > org.apache.drill.exec.planner.logical.DrillScanRel.(
>>>> > > > DrillScanRel.java:89)
>>>> > > > > [drill-java-exec-1.14.0.jar:1.14.0]
>>>> > > > > at
>>>> > > > >
>>>> > > > > org.apache.drill.exec.planner.logical.DrillScanRel.(
>>>> > > > DrillScanRel.java:69)
>>>> > > > > [drill-java-exec-1.14.0.jar:1.14.0]
>>>> > > > > at
>>>> > > > >
>>>> > > > > org.apache.drill.exec.planner.logical.DrillScanRel.(
>>>> > > > DrillScanRel.java:62)
>>>> > > > > [drill-java-exec-1.14.0.jar:1.14.0]
>>>> > > > > at
>>>> > > > >
>>>> > > > > org.apache.drill.exec.planner.logical.DrillScanRule.onMatch(
>>>> > > > DrillScanRule.java:38)
>>>> > > > > [drill-java-exec-1.14.0.jar:1.14.0]
>>>> > > > > at
>>>> > > > >
>>>> > > > > org.apache.calcite.plan.volcano.VolcanoRuleCall.
>>>> > > > onMatch(VolcanoRuleCall.java:212)
>>>> > > > > [calcite-core-1.16.0-drill-r6.jar:1.16.0-drill-r6]
>>>> > > > > at
>>>> > > > >
>>>> > > > > org.apache.calcite.plan.volcano.VolcanoPlanner.
>>>> > > > findBestExp(VolcanoPlanner.java:652)
>>>> > > > > [calcite-core-1.16.0-drill-r6.jar:1.16.0-drill-r6]
>>>> > > > > at org.apache.calcite.tools.Programs$RuleSetProgram.run(
>>>> > > > Programs.java:368)
>>>> > > > > [calcite-core-1.16.0-drill-r6.jar:1.16.0-drill-r6]
>>>> > > > > at
>>>> > > > >
>>>> > > > > org.apache.drill.exec.planner.sql.handlers.
>>>> > > DefaultSqlHandler.transform(
>>>> > > > DefaultSqlHandler.java:429)
>>>> > > > > [drill-java-exec-1.14.0.jar:1.14.0]
>>>> > > > > at
>>>> > > > >
>>>> > > > > org.apache.drill.exec.planner.sql.handlers.
>>>> > > DefaultSqlHandler.transform(
>>>> > > > DefaultSqlHandler.java:369)
>>>> > > > > [drill-java-exec-1.14.0.jar:1.14.0]
>>>> > > > > at
>>>> > > > >
>>>> > > > > org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.
>>>> > > > convertToRawDrel(DefaultSqlHandler.java:255)
>>>> > > > > [drill-java-exec-1.14.0.jar:1.14.0]
>>>> > > > > at
>>>> > > > >
>>>> > > > > org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.
>>>> > > > convertToDrel(DefaultSqlHandler.java:318)
>>>> > > > > [drill-java-exec-1.14.0.jar:1.14.0]
>>>> > > > > at
>>>> > > > >
>>>> > > > >
>>>> org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.getPlan(
>>>> > > > DefaultSqlHandler.java:180)
>>>> > > > > [drill-java-exec-1.14.0.jar:1.14.0]
>>>> > > > > at
>>>> > > > >
>>>> > > > > org.apache.drill.exec.planner.sql.DrillSqlWorker.
>>>> > > > getQueryPlan(DrillSqlWorker.java:145)
>>>> > > > > [drill-java-exec-1.14.0.jar:1.14.0]
>>>> > > > > at
>>>> > > > >
>>>> > > > > org.apache.drill.exec.planner.sql.DrillSqlWorker.getPlan(
>>>> > > > DrillSqlWorker.java:83)
>>>> > > > > [drill-java-exec-1.14.0.jar:1.14.0]
>>>> > > > > at org.apache.drill.exec.work
>>>> > .foreman.Foreman.runSQL(Foreman.java:567)
>>>> > > > > [drill-java-exec-1.14.0.jar:1.14.0]
>>>> > > > > at org.apache.drill.exec.work
>>>> .foreman.Foreman.run(Foreman.java:266)
>>>> > > > > [drill-java-exec-1.14.0.jar:1.14.0]
>>>> > > > > at
>>>> > > > >
>>>> > > > > java.util.concurrent.ThreadPoolExecutor.runWorker(
>>>> > > > ThreadPoolExecutor.java:1149)
>>>> > > > > [na:1.8.0_172]
>>>> > > > > at
>>>> > > > >
>>>> > > > > java.util.concurrent.ThreadPoolExecutor$Worker.run(
>>>> > > > ThreadPoolExecutor.java:624)
>>>> > > > > [na:1.8.0_172]
>>>> > > > > at java.lang.Thread.run(Thread.java:748) [na:1.8.0_172]
>>>> > > > > Caused by: java.util.concurrent.CancellationException: null
>>>> > > > > at
>>>> > > > >
>>>> > > > > org.apache.drill.exec.store.TimedCallable$FutureMapper.
>>>> > > > apply(TimedCallable.java:86)
>>>> > > > > ~[drill-java-exec-1.14.0.jar:1.14.0]
>>>> > > > > at
>>>> > > > >
>>>> > > > > org.apache.drill.exec.store.TimedCallable$FutureMapper.
>>>> > > > apply(TimedCallable.java:57)
>>>> > > > > ~[drill-java-exec-1.14.0.jar:1.14.0]
>>>> > > > > at
>>>> > > > >
>>>> > > > > org.apache.drill.common.collections.Collectors.lambda$
>>>> > > > toList$2(Collectors.java:97)
>>>> > > > > ~[drill-common-1.14.0.jar:1.14.0]
>>>> > > > > at java.util.ArrayList.forEach(ArrayList.java:1257)
>>>> ~[na:1.8.0_172]
>>>> > > > > at
>>>> > > > > org.apache.drill.common.collections.Collectors.toList(
>>>> > > > Collectors.java:97)
>>>> > > > > ~[drill-common-1.14.0.jar:1.14.0]
>>>> > > > > at org.apache.drill.exec.store.TimedCallable.run(
>>>> > > TimedCallable.java:214)
>>>> > > > > [drill-java-exec-1.14.0.jar:1.14.0]
>>>> > > > > ... 33 common frames omitted
>>>> > > > > 2018-09-20 09:02:10,608 [UserServer-1] WARN
>>>> > > > > o.a.drill.exec.rpc.user.UserServer - Message of mode REQUEST of
>>>> rpc
>>>> > > > type 3
>>>> > > > > took longer than 500ms. Actual duration was 2042ms.
>>>> > > > > 2018-09-20 09:02:10,608
>>>> > [245d0e6f-0dc1-2a4b-12a4-b9aaad4182fc:foreman]
>>>> > > > > INFO o.a.drill.exec.work.foreman.Foreman - Query text for query
>>>> id
>>>> > > > > 245d0e6f-0dc1-2a4b-12a4-b9aaad4182fc: select *
>>>> > > > > from dfs.tmp.load_pos_sales_detail_tbl
>>>> > > > >
>>>> > > > > 2018-09-20 09:02:42,615
>>>> > [245d0e6f-0dc1-2a4b-12a4-b9aaad4182fc:foreman]
>>>> > > > > ERROR o.a.d.e.s.parquet.metadata.Metadata - Waited for 30000
>>>> ms, but
>>>> > > > only
>>>> > > > > 10 tasks for 'Fetch parquet metadata' are complete. Total
>>>> number of
>>>> > > tasks
>>>> > > > > 29, parallelism 16.
>>>> > > > > java.util.concurrent.CancellationException: null
>>>> > > > > at
>>>> > > > >
>>>> > > > > org.apache.drill.exec.store.TimedCallable$FutureMapper.
>>>> > > > apply(TimedCallable.java:86)
>>>> > > > > ~[drill-java-exec-1.14.0.jar:1.14.0]
>>>> > > > > at
>>>> > > > >
>>>> > > > > org.apache.drill.exec.store.TimedCallable$FutureMapper.
>>>> > > > apply(TimedCallable.java:57)
>>>> > > > > ~[drill-java-exec-1.14.0.jar:1.14.0]
>>>> > > > > at
>>>> > > > >
>>>> > > > > org.apache.drill.common.collections.Collectors.lambda$
>>>> > > > toList$2(Collectors.java:97)
>>>> > > > > ~[drill-common-1.14.0.jar:1.14.0]
>>>> > > > > at java.util.ArrayList.forEach(ArrayList.java:1257)
>>>> ~[na:1.8.0_172]
>>>> > > > > at
>>>> > > > > org.apache.drill.common.collections.Collectors.toList(
>>>> > > > Collectors.java:97)
>>>> > > > > ~[drill-common-1.14.0.jar:1.14.0]
>>>> > > > > at org.apache.drill.exec.store.TimedCallable.run(
>>>> > > TimedCallable.java:214)
>>>> > > > > ~[drill-java-exec-1.14.0.jar:1.14.0]
>>>> > > > > at
>>>> > > > >
>>>> > > > > org.apache.drill.exec.store.parquet.metadata.Metadata.
>>>> > > > getParquetFileMetadata_v3(Metadata.java:340)
>>>> > > > > [drill-java-exec-1.14.0.jar:1.14.0]
>>>> > > > > at
>>>> > > > >
>>>> > > > > org.apache.drill.exec.store.parquet.metadata.Metadata.
>>>> > > > getParquetTableMetadata(Metadata.java:324)
>>>> > > > > [drill-java-exec-1.14.0.jar:1.14.0]
>>>> > > > > at
>>>> > > > >
>>>> > > > > org.apache.drill.exec.store.parquet.metadata.Metadata.
>>>> > > > getParquetTableMetadata(Metadata.java:305)
>>>> > > > > [drill-java-exec-1.14.0.jar:1.14.0]
>>>> > > > > at
>>>> > > > >
>>>> > > > > org.apache.drill.exec.store.parquet.metadata.Metadata.
>>>> > > > getParquetTableMetadata(Metadata.java:124)
>>>> > > > > [drill-java-exec-1.14.0.jar:1.14.0]
>>>> > > > > at
>>>> > > > >
>>>> > > > > org.apache.drill.exec.store.parquet.ParquetGroupScan.
>>>> > > > initInternal(ParquetGroupScan.java:254)
>>>> > > > > [drill-java-exec-1.14.0.jar:1.14.0]
>>>> > > > > at
>>>> > > > >
>>>> > > > >
>>>> org.apache.drill.exec.store.parquet.AbstractParquetGroupScan.init(
>>>> > > > AbstractParquetGroupScan.java:380)
>>>> > > > > [drill-java-exec-1.14.0.jar:1.14.0]
>>>> > > > > at
>>>> > > > >
>>>> > > > > org.apache.drill.exec.store.parquet.ParquetGroupScan.
>>>> > > > init>(ParquetGroupScan.java:132)
>>>> > > > > [drill-java-exec-1.14.0.jar:1.14.0]
>>>> > > > > at
>>>> > > > >
>>>> > > > > org.apache.drill.exec.store.parquet.ParquetGroupScan.
>>>> > > > init>(ParquetGroupScan.java:102)
>>>> > > > > [drill-java-exec-1.14.0.jar:1.14.0]
>>>> > > > > at
>>>> > > > >
>>>> > > > >
>>>> org.apache.drill.exec.store.parquet.ParquetFormatPlugin.getGroupScan(
>>>> > > > ParquetFormatPlugin.java:180)
>>>> > > > > [drill-java-exec-1.14.0.jar:1.14.0]
>>>> > > > > at
>>>> > > > >
>>>> > > > >
>>>> org.apache.drill.exec.store.parquet.ParquetFormatPlugin.getGroupScan(
>>>> > > > ParquetFormatPlugin.java:70)
>>>> > > > > [drill-java-exec-1.14.0.jar:1.14.0]
>>>> > > > > at
>>>> > > > >
>>>> > > > >
>>>> org.apache.drill.exec.store.dfs.FileSystemPlugin.getPhysicalScan(
>>>> > > > FileSystemPlugin.java:136)
>>>> > > > > [drill-java-exec-1.14.0.jar:1.14.0]
>>>> > > > > at
>>>> > > > >
>>>> > > > >
>>>> org.apache.drill.exec.store.AbstractStoragePlugin.getPhysicalScan(
>>>> > > > AbstractStoragePlugin.java:116)
>>>> > > > > [drill-java-exec-1.14.0.jar:1.14.0]
>>>> > > > > at
>>>> > > > >
>>>> > > > >
>>>> org.apache.drill.exec.store.AbstractStoragePlugin.getPhysicalScan(
>>>> > > > AbstractStoragePlugin.java:111)
>>>> > > > > [drill-java-exec-1.14.0.jar:1.14.0]
>>>> > > > > at
>>>> > > > >
>>>> > > > > org.apache.drill.exec.planner.logical.DrillTable.
>>>> > > > getGroupScan(DrillTable.java:99)
>>>> > > > > [drill-java-exec-1.14.0.jar:1.14.0]
>>>> > > > > at
>>>> > > > >
>>>> > > > > org.apache.drill.exec.planner.logical.DrillScanRel.(
>>>> > > > DrillScanRel.java:89)
>>>> > > > > [drill-java-exec-1.14.0.jar:1.14.0]
>>>> > > > > at
>>>> > > > >
>>>> > > > > org.apache.drill.exec.planner.logical.DrillScanRel.(
>>>> > > > DrillScanRel.java:69)
>>>> > > > > [drill-java-exec-1.14.0.jar:1.14.0]
>>>> > > > > at
>>>> > > > >
>>>> > > > > org.apache.drill.exec.planner.logical.DrillScanRel.(
>>>> > > > DrillScanRel.java:62)
>>>> > > > > [drill-java-exec-1.14.0.jar:1.14.0]
>>>> > > > > at
>>>> > > > >
>>>> > > > > org.apache.drill.exec.planner.logical.DrillScanRule.onMatch(
>>>> > > > DrillScanRule.java:38)
>>>> > > > > [drill-java-exec-1.14.0.jar:1.14.0]
>>>> > > > > at
>>>> > > > >
>>>> > > > > org.apache.calcite.plan.volcano.VolcanoRuleCall.
>>>> > > > onMatch(VolcanoRuleCall.java:212)
>>>> > > > > [calcite-core-1.16.0-drill-r6.jar:1.16.0-drill-r6]
>>>> > > > > at
>>>> > > > >
>>>> > > > > org.apache.calcite.plan.volcano.VolcanoPlanner.
>>>> > > > findBestExp(VolcanoPlanner.java:652)
>>>> > > > > [calcite-core-1.16.0-drill-r6.jar:1.16.0-drill-r6]
>>>> > > > > at org.apache.calcite.tools.Programs$RuleSetProgram.run(
>>>> > > > Programs.java:368)
>>>> > > > > [calcite-core-1.16.0-drill-r6.jar:1.16.0-drill-r6]
>>>> > > > > at
>>>> > > > >
>>>> > > > > org.apache.drill.exec.planner.sql.handlers.
>>>> > > DefaultSqlHandler.transform(
>>>> > > > DefaultSqlHandler.java:429)
>>>> > > > > [drill-java-exec-1.14.0.jar:1.14.0]
>>>> > > > > at
>>>> > > > >
>>>> > > > > org.apache.drill.exec.planner.sql.handlers.
>>>> > > DefaultSqlHandler.transform(
>>>> > > > DefaultSqlHandler.java:369)
>>>> > > > > [drill-java-exec-1.14.0.jar:1.14.0]
>>>> > > > > at
>>>> > > > >
>>>> > > > > org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.
>>>> > > > convertToRawDrel(DefaultSqlHandler.java:255)
>>>> > > > > [drill-java-exec-1.14.0.jar:1.14.0]
>>>> > > > > at
>>>> > > > >
>>>> > > > > org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.
>>>> > > > convertToDrel(DefaultSqlHandler.java:318)
>>>> > > > > [drill-java-exec-1.14.0.jar:1.14.0]
>>>> > > > > at
>>>> > > > >
>>>> > > > >
>>>> org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.getPlan(
>>>> > > > DefaultSqlHandler.java:180)
>>>> > > > > [drill-java-exec-1.14.0.jar:1.14.0]
>>>> > > > > at
>>>> > > > >
>>>> > > > > org.apache.drill.exec.planner.sql.DrillSqlWorker.
>>>> > > > getQueryPlan(DrillSqlWorker.java:145)
>>>> > > > > [drill-java-exec-1.14.0.jar:1.14.0]
>>>> > > > > at
>>>> > > > >
>>>> > > > > org.apache.drill.exec.planner.sql.DrillSqlWorker.getPlan(
>>>> > > > DrillSqlWorker.java:83)
>>>> > > > > [drill-java-exec-1.14.0.jar:1.14.0]
>>>> > > > > at org.apache.drill.exec.work
>>>> > .foreman.Foreman.runSQL(Foreman.java:567)
>>>> > > > > [drill-java-exec-1.14.0.jar:1.14.0]
>>>> > > > > at org.apache.drill.exec.work
>>>> .foreman.Foreman.run(Foreman.java:266)
>>>> > > > > [drill-java-exec-1.14.0.jar:1.14.0]
>>>> > > > > at
>>>> > > > >
>>>> > > > > java.util.concurrent.ThreadPoolExecutor.runWorker(
>>>> > > > ThreadPoolExecutor.java:1149)
>>>> > > > > [na:1.8.0_172]
>>>> > > > > at
>>>> > > > >
>>>> > > > > java.util.concurrent.ThreadPoolExecutor$Worker.run(
>>>> > > > ThreadPoolExecutor.java:624)
>>>> > > > > [na:1.8.0_172]
>>>> > > > > at java.lang.Thread.run(Thread.java:748) [na:1.8.0_172]
>>>> > > > > 2018-09-20 09:02:42,625
>>>> > [245d0e6f-0dc1-2a4b-12a4-b9aaad4182fc:foreman]
>>>> > > > > INFO o.a.d.e.s.parquet.metadata.Metadata - User Error Occurred:
>>>> > Waited
>>>> > > > for
>>>> > > > > 30000 ms, but only 10 tasks for 'Fetch parquet metadata' are
>>>> > complete.
>>>> > > > > Total number of tasks 29, parallelism 16. (null)
>>>> > > > > org.apache.drill.common.exceptions.UserException: RESOURCE
>>>> ERROR:
>>>> > > Waited
>>>> > > > > for 30000 ms, but only 10 tasks for 'Fetch parquet metadata' are
>>>> > > > complete.
>>>> > > > > Total number of tasks 29, parallelism 16.
>>>> > > > >
>>>> > > > >
>>>> > > > > [Error Id: 3b079174-f5d0-4313-8097-25a0b3070854 ]
>>>> > > > > at
>>>> > > > >
>>>> > > > > org.apache.drill.common.exceptions.UserException$
>>>> > > > Builder.build(UserException.java:633)
>>>> > > > > ~[drill-common-1.14.0.jar:1.14.0]
>>>> > > > > at org.apache.drill.exec.store.TimedCallable.run(
>>>> > > TimedCallable.java:253)
>>>> > > > > [drill-java-exec-1.14.0.jar:1.14.0]
>>>> > > > > at
>>>> > > > >
>>>> > > > > org.apache.drill.exec.store.parquet.metadata.Metadata.
>>>> > > > getParquetFileMetadata_v3(Metadata.java:340)
>>>> > > > > [drill-java-exec-1.14.0.jar:1.14.0]
>>>> > > > > at
>>>> > > > >
>>>> > > > > org.apache.drill.exec.store.parquet.metadata.Metadata.
>>>> > > > getParquetTableMetadata(Metadata.java:324)
>>>> > > > > [drill-java-exec-1.14.0.jar:1.14.0]
>>>> > > > > at
>>>> > > > >
>>>> > > > > org.apache.drill.exec.store.parquet.metadata.Metadata.
>>>> > > > getParquetTableMetadata(Metadata.java:305)
>>>> > > > > [drill-java-exec-1.14.0.jar:1.14.0]
>>>> > > > > at
>>>> > > > >
>>>> > > > > org.apache.drill.exec.store.parquet.metadata.Metadata.
>>>> > > > getParquetTableMetadata(Metadata.java:124)
>>>> > > > > [drill-java-exec-1.14.0.jar:1.14.0]
>>>> > > > > at
>>>> > > > >
>>>> > > > > org.apache.drill.exec.store.parquet.ParquetGroupScan.
>>>> > > > initInternal(ParquetGroupScan.java:254)
>>>> > > > > [drill-java-exec-1.14.0.jar:1.14.0]
>>>> > > > > at
>>>> > > > >
>>>> > > > >
>>>> org.apache.drill.exec.store.parquet.AbstractParquetGroupScan.init(
>>>> > > > AbstractParquetGroupScan.java:380)
>>>> > > > > [drill-java-exec-1.14.0.jar:1.14.0]
>>>> > > > > at
>>>> > > > >
>>>> > > > > org.apache.drill.exec.store.parquet.ParquetGroupScan.
>>>> > > > init>(ParquetGroupScan.java:132)
>>>> > > > > [drill-java-exec-1.14.0.jar:1.14.0]
>>>> > > > > at
>>>> > > > >
>>>> > > > > org.apache.drill.exec.store.parquet.ParquetGroupScan.
>>>> > > > init>(ParquetGroupScan.java:102)
>>>> > > > > [drill-java-exec-1.14.0.jar:1.14.0]
>>>> > > > > at
>>>> > > > >
>>>> > > > >
>>>> org.apache.drill.exec.store.parquet.ParquetFormatPlugin.getGroupScan(
>>>> > > > ParquetFormatPlugin.java:180)
>>>> > > > > [drill-java-exec-1.14.0.jar:1.14.0]
>>>> > > > > at
>>>> > > > >
>>>> > > > >
>>>> org.apache.drill.exec.store.parquet.ParquetFormatPlugin.getGroupScan(
>>>> > > > ParquetFormatPlugin.java:70)
>>>> > > > > [drill-java-exec-1.14.0.jar:1.14.0]
>>>> > > > > at
>>>> > > > >
>>>> > > > >
>>>> org.apache.drill.exec.store.dfs.FileSystemPlugin.getPhysicalScan(
>>>> > > > FileSystemPlugin.java:136)
>>>> > > > > [drill-java-exec-1.14.0.jar:1.14.0]
>>>> > > > > at
>>>> > > > >
>>>> > > > >
>>>> org.apache.drill.exec.store.AbstractStoragePlugin.getPhysicalScan(
>>>> > > > AbstractStoragePlugin.java:116)
>>>> > > > > [drill-java-exec-1.14.0.jar:1.14.0]
>>>> > > > > at
>>>> > > > >
>>>> > > > >
>>>> org.apache.drill.exec.store.AbstractStoragePlugin.getPhysicalScan(
>>>> > > > AbstractStoragePlugin.java:111)
>>>> > > > > [drill-java-exec-1.14.0.jar:1.14.0]
>>>> > > > > at
>>>> > > > >
>>>> > > > > org.apache.drill.exec.planner.logical.DrillTable.
>>>> > > > getGroupScan(DrillTable.java:99)
>>>> > > > > [drill-java-exec-1.14.0.jar:1.14.0]
>>>> > > > > at
>>>> > > > >
>>>> > > > > org.apache.drill.exec.planner.logical.DrillScanRel.(
>>>> > > > DrillScanRel.java:89)
>>>> > > > > [drill-java-exec-1.14.0.jar:1.14.0]
>>>> > > > > at
>>>> > > > >
>>>> > > > > org.apache.drill.exec.planner.logical.DrillScanRel.(
>>>> > > > DrillScanRel.java:69)
>>>> > > > > [drill-java-exec-1.14.0.jar:1.14.0]
>>>> > > > > at
>>>> > > > >
>>>> > > > > org.apache.drill.exec.planner.logical.DrillScanRel.(
>>>> > > > DrillScanRel.java:62)
>>>> > > > > [drill-java-exec-1.14.0.jar:1.14.0]
>>>> > > > > at
>>>> > > > >
>>>> > > > > org.apache.drill.exec.planner.logical.DrillScanRule.onMatch(
>>>> > > > DrillScanRule.java:38)
>>>> > > > > [drill-java-exec-1.14.0.jar:1.14.0]
>>>> > > > > at
>>>> > > > >
>>>> > > > > org.apache.calcite.plan.volcano.VolcanoRuleCall.
>>>> > > > onMatch(VolcanoRuleCall.java:212)
>>>> > > > > [calcite-core-1.16.0-drill-r6.jar:1.16.0-drill-r6]
>>>> > > > > at
>>>> > > > >
>>>> > > > > org.apache.calcite.plan.volcano.VolcanoPlanner.
>>>> > > > findBestExp(VolcanoPlanner.java:652)
>>>> > > > > [calcite-core-1.16.0-drill-r6.jar:1.16.0-drill-r6]
>>>> > > > > at org.apache.calcite.tools.Programs$RuleSetProgram.run(
>>>> > > > Programs.java:368)
>>>> > > > > [calcite-core-1.16.0-drill-r6.jar:1.16.0-drill-r6]
>>>> > > > > at
>>>> > > > >
>>>> > > > > org.apache.drill.exec.planner.sql.handlers.
>>>> > > DefaultSqlHandler.transform(
>>>> > > > DefaultSqlHandler.java:429)
>>>> > > > > [drill-java-exec-1.14.0.jar:1.14.0]
>>>> > > > > at
>>>> > > > >
>>>> > > > > org.apache.drill.exec.planner.sql.handlers.
>>>> > > DefaultSqlHandler.transform(
>>>> > > > DefaultSqlHandler.java:369)
>>>> > > > > [drill-java-exec-1.14.0.jar:1.14.0]
>>>> > > > > at
>>>> > > > >
>>>> > > > > org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.
>>>> > > > convertToRawDrel(DefaultSqlHandler.java:255)
>>>> > > > > [drill-java-exec-1.14.0.jar:1.14.0]
>>>> > > > > at
>>>> > > > >
>>>> > > > > org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.
>>>> > > > convertToDrel(DefaultSqlHandler.java:318)
>>>> > > > > [drill-java-exec-1.14.0.jar:1.14.0]
>>>> > > > > at
>>>> > > > >
>>>> > > > >
>>>> org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.getPlan(
>>>> > > > DefaultSqlHandler.java:180)
>>>> > > > > [drill-java-exec-1.14.0.jar:1.14.0]
>>>> > > > > at
>>>> > > > >
>>>> > > > > org.apache.drill.exec.planner.sql.DrillSqlWorker.
>>>> > > > getQueryPlan(DrillSqlWorker.java:145)
>>>> > > > > [drill-java-exec-1.14.0.jar:1.14.0]
>>>> > > > > at
>>>> > > > >
>>>> > > > > org.apache.drill.exec.planner.sql.DrillSqlWorker.getPlan(
>>>> > > > DrillSqlWorker.java:83)
>>>> > > > > [drill-java-exec-1.14.0.jar:1.14.0]
>>>> > > > > at org.apache.drill.exec.work
>>>> > .foreman.Foreman.runSQL(Foreman.java:567)
>>>> > > > > [drill-java-exec-1.14.0.jar:1.14.0]
>>>> > > > > at org.apache.drill.exec.work
>>>> .foreman.Foreman.run(Foreman.java:266)
>>>> > > > > [drill-java-exec-1.14.0.jar:1.14.0]
>>>> > > > > at
>>>> > > > >
>>>> > > > > java.util.concurrent.ThreadPoolExecutor.runWorker(
>>>> > > > ThreadPoolExecutor.java:1149)
>>>> > > > > [na:1.8.0_172]
>>>> > > > > at
>>>> > > > >
>>>> > > > > java.util.concurrent.ThreadPoolExecutor$Worker.run(
>>>> > > > ThreadPoolExecutor.java:624)
>>>> > > > > [na:1.8.0_172]
>>>> > > > > at java.lang.Thread.run(Thread.java:748) [na:1.8.0_172]
>>>> > > > > Caused by: java.util.concurrent.CancellationException: null
>>>> > > > > at
>>>> > > > >
>>>> > > > > org.apache.drill.exec.store.TimedCallable$FutureMapper.
>>>> > > > apply(TimedCallable.java:86)
>>>> > > > > ~[drill-java-exec-1.14.0.jar:1.14.0]
>>>> > > > > at
>>>> > > > >
>>>> > > > > org.apache.drill.exec.store.TimedCallable$FutureMapper.
>>>> > > > apply(TimedCallable.java:57)
>>>> > > > > ~[drill-java-exec-1.14.0.jar:1.14.0]
>>>> > > > > at
>>>> > > > >
>>>> > > > > org.apache.drill.common.collections.Collectors.lambda$
>>>> > > > toList$2(Collectors.java:97)
>>>> > > > > ~[drill-common-1.14.0.jar:1.14.0]
>>>> > > > > at java.util.ArrayList.forEach(ArrayList.java:1257)
>>>> ~[na:1.8.0_172]
>>>> > > > > at
>>>> > > > > org.apache.drill.common.collections.Collectors.toList(
>>>> > > > Collectors.java:97)
>>>> > > > > ~[drill-common-1.14.0.jar:1.14.0]
>>>> > > > > at org.apache.drill.exec.store.TimedCallable.run(
>>>> > > TimedCallable.java:214)
>>>> > > > > [drill-java-exec-1.14.0.jar:1.14.0]
>>>> > > > > ... 33 common frames omitted
>>>> > > > >
>>>> > > > >
>>>> > > > > ----------------------------------------
>>>> > > > > ----------
>>>> > > > >
>>>> > > >
>>>> > >
>>>> >
>>>>
>>>