You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@tajo.apache.org by "ASF GitHub Bot (JIRA)" <ji...@apache.org> on 2015/11/05 09:59:27 UTC
[jira] [Commented] (TAJO-1952) Implement PartitionedFileFragment
[ https://issues.apache.org/jira/browse/TAJO-1952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14991356#comment-14991356 ]
ASF GitHub Bot commented on TAJO-1952:
--------------------------------------
GitHub user blrunner opened a pull request:
https://github.com/apache/tajo/pull/846
TAJO-1952: Implement PartitionedFileFragment
This patch contains following modifications:
* Remove partition paths from PartitionedTableScanNode
* Implement PartitionedFileFragment
* Separate a method for pruning partition paths from PartitionedTableRewriter to PartitionedTableUtil
* Build a type by using partition name which contains partition keys and values
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/blrunner/tajo TAJO-1952
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/tajo/pull/846.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #846
----
commit 0d2a634d2353efdeecced4729be9f585789acdb1
Author: JaeHwa Jung <bl...@apache.org>
Date: 2015-10-28T08:22:40Z
Implement PartitionedFileFragment
commit 4d7e73b7b20d1e6721b0f6b2ee53c4d04eb278d4
Author: JaeHwa Jung <bl...@apache.org>
Date: 2015-10-28T09:10:53Z
Add unit test cases for PartitionedFileFragment
commit 6fab5adadb303e690f7377547f842f84eb1f9286
Author: JaeHwa Jung <bl...@apache.org>
Date: 2015-10-29T07:25:47Z
Add PartitionedTableUtil for finding filtered partition directories.
commit b3bbcd188b0afc3b977f85005c0dffa20a8312dc
Author: JaeHwa Jung <bl...@apache.org>
Date: 2015-11-02T06:57:39Z
Remove the array of partition directories of rerwrite rule and apply PartitionedFileFragment.
commit 25163d0cdade5f45e7e524db4ceac4250b7ea805
Author: JaeHwa Jung <bl...@apache.org>
Date: 2015-11-02T07:01:56Z
Merge branch 'master' of https://git-wip-us.apache.org/repos/asf/tajo into TAJO-1952
Conflicts:
tajo-core-tests/src/test/java/org/apache/tajo/engine/planner/physical/TestPhysicalPlanner.java
commit 4f711fa2ff7a18979198d80a70f283f73b91edf9
Author: JaeHwa Jung <bl...@apache.org>
Date: 2015-11-02T07:14:49Z
Remove unnecessary method
commit 33dc1407a3d1417a81895e5e36d528f64c88bbbe
Author: JaeHwa Jung <bl...@apache.org>
Date: 2015-11-02T07:22:24Z
Update comments
commit dede3e2957a2cee7bccd235a3f873aac0ab40377
Author: JaeHwa Jung <bl...@apache.org>
Date: 2015-11-02T07:40:15Z
Remove unnecessary constructor parameter of PhysicalPlannerImpl
commit ccc4f6cb2e12bd642d00be08f393f6754e74db7f
Author: JaeHwa Jung <bl...@apache.org>
Date: 2015-11-02T07:55:16Z
Remove unnecessary parameter of PartitionedTableUtil::buildTupleFromPartitionName
commit d5f563a1d6764f21f80e91a2540a9de5330a38cf
Author: JaeHwa Jung <bl...@apache.org>
Date: 2015-11-02T07:59:32Z
Update wrong indent
commit 086b02beb700e125a6ba37cbe275965150a89183
Author: JaeHwa Jung <bl...@apache.org>
Date: 2015-11-02T07:59:57Z
Remove unused package
commit 22731ec4a13f1ad0e75d7987966c17715afbeb52
Author: JaeHwa Jung <bl...@apache.org>
Date: 2015-11-02T08:20:01Z
Update wrong comparison operator
commit 437f5ecdc7fad8b056bb638ea0897cd6e455b9b8
Author: JaeHwa Jung <bl...@apache.org>
Date: 2015-11-02T08:24:05Z
Update log message
commit d76f41aac39e7536f4acac265559fb136aa05b71
Author: JaeHwa Jung <bl...@apache.org>
Date: 2015-11-03T00:33:16Z
When rewriting PartitionedTableScanNode, set partition paths and table volume.
commit 126f5e06de3aa88563281fd0c382d03f4afab5bf
Author: JaeHwa Jung <bl...@apache.org>
Date: 2015-11-03T00:36:27Z
Merge branch 'master' of https://git-wip-us.apache.org/repos/asf/tajo into TAJO-1952
commit 9112ceb61547667020423bf4fbe18f99c07c2539
Author: JaeHwa Jung <bl...@apache.org>
Date: 2015-11-03T01:47:32Z
Update the result message of partition pruning
commit 71d65a5dec1571852448f0b349e121a9a0268a5e
Author: JaeHwa Jung <bl...@apache.org>
Date: 2015-11-05T08:57:54Z
Merge branch 'master' of https://git-wip-us.apache.org/repos/asf/tajo into TAJO-1952
----
> Implement PartitionedFileFragment
> ---------------------------------
>
> Key: TAJO-1952
> URL: https://issues.apache.org/jira/browse/TAJO-1952
> Project: Tajo
> Issue Type: Improvement
> Components: Planner/Optimizer, Storage
> Reporter: Jaehwa Jung
> Assignee: Jaehwa Jung
>
> Currently, PartitionedTableScanNode contains the list of partitions and it seems to me that the list has some problems as following:
> 1. Duplicate Informs: Task contains Fragment which specify target directory or target file for scanning. A path of partition lists already would write to Fragment.
> 2. Network Resource: When scanning lost of partition, it will occupy network resource, for example, several hundred kilobytes or more. It looks like an unnecessary resource because Fragment already has the path of partitions.
> I want to improve above problems by implementing new Fragment called PartitionedFileFragment. Currently, I'm planning the implementation as following:
> * PartitionedFileFragment will borrow FileFragment and it contains the partition path and the partition key values.
> * Remove the path array of partitions from PartitionedTableScanNode.
> * Implement a method for getting filtered partition directories in FileTableSpace.
> * Implement a method for making PartitionedFileFragment array.
> * Before making splits, call above method and use it for making splits.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)