You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pig.apache.org by "Cheolsoo Park (JIRA)" <ji...@apache.org> on 2014/11/13 06:47:33 UTC
[jira] [Updated] (PIG-4329) Fetch optimization should be disabled
when limit is not pushed up
[ https://issues.apache.org/jira/browse/PIG-4329?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Cheolsoo Park updated PIG-4329:
-------------------------------
Description:
Although PIG-4135 disable fetch optimization when there is no limit in the plan, that doesn't solve the problem completely. In fact, fetch optimization should be still disabled if limit is not pushed up. Consider the following query-
{code}
random_lists = load 'prodhive.schakraborty.search_server_denorm_impressions' using DseStorage();
random_lists = filter random_lists by entity_section=='random';
random_lists = limit random_lists 10;
dump random_lists;
{code}
Because the {{filter by}} blocks limit from being pushed up, POLoad actually scans the full table. In this case, fetch optimization makes the job extremely slow.
was:
Although PIG-4135 disable fetch optimization when there is no limit in the plan, that doesn't solve the problem completely. In fact, fetch optimization should be still disabled if limit is not pushed up. Consider the following query-
{code}
random_lists = load 'prodhive.schakraborty.search_server_denorm_impressions' using DseStorage();
random_lists = filter random_lists by entity_section=='random');
random_lists = limit random_lists 10;
dump random_lists;
{code}
Because the {{filter by}} blocks limit from being pushed up, POLoad actually scans the full table. In this case, fetch optimization makes the job extremely slow.
> Fetch optimization should be disabled when limit is not pushed up
> -----------------------------------------------------------------
>
> Key: PIG-4329
> URL: https://issues.apache.org/jira/browse/PIG-4329
> Project: Pig
> Issue Type: Bug
> Reporter: Cheolsoo Park
> Assignee: Cheolsoo Park
> Fix For: 0.15.0
>
>
> Although PIG-4135 disable fetch optimization when there is no limit in the plan, that doesn't solve the problem completely. In fact, fetch optimization should be still disabled if limit is not pushed up. Consider the following query-
> {code}
> random_lists = load 'prodhive.schakraborty.search_server_denorm_impressions' using DseStorage();
> random_lists = filter random_lists by entity_section=='random';
> random_lists = limit random_lists 10;
> dump random_lists;
> {code}
> Because the {{filter by}} blocks limit from being pushed up, POLoad actually scans the full table. In this case, fetch optimization makes the job extremely slow.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)