You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@tajo.apache.org by "ASF GitHub Bot (JIRA)" <ji...@apache.org> on 2016/04/04 18:49:25 UTC
[jira] [Commented] (TAJO-2111) Optimize Partition Table Split
Computation for Amazon S3
[ https://issues.apache.org/jira/browse/TAJO-2111?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15224491#comment-15224491 ]
ASF GitHub Bot commented on TAJO-2111:
--------------------------------------
GitHub user blrunner opened a pull request:
https://github.com/apache/tajo/pull/994
TAJO-2111: Optimize Partition Table Split Computation for Amazon S3
It depends on https://github.com/apache/tajo/pull/846 and https://github.com/apache/tajo/pull/953
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/blrunner/tajo s3-split-improvement
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/tajo/pull/994.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #994
----
commit 0d2a634d2353efdeecced4729be9f585789acdb1
Author: JaeHwa Jung <bl...@apache.org>
Date: 2015-10-28T08:22:40Z
Implement PartitionedFileFragment
commit 4d7e73b7b20d1e6721b0f6b2ee53c4d04eb278d4
Author: JaeHwa Jung <bl...@apache.org>
Date: 2015-10-28T09:10:53Z
Add unit test cases for PartitionedFileFragment
commit 6fab5adadb303e690f7377547f842f84eb1f9286
Author: JaeHwa Jung <bl...@apache.org>
Date: 2015-10-29T07:25:47Z
Add PartitionedTableUtil for finding filtered partition directories.
commit b3bbcd188b0afc3b977f85005c0dffa20a8312dc
Author: JaeHwa Jung <bl...@apache.org>
Date: 2015-11-02T06:57:39Z
Remove the array of partition directories of rerwrite rule and apply PartitionedFileFragment.
commit 25163d0cdade5f45e7e524db4ceac4250b7ea805
Author: JaeHwa Jung <bl...@apache.org>
Date: 2015-11-02T07:01:56Z
Merge branch 'master' of https://git-wip-us.apache.org/repos/asf/tajo into TAJO-1952
Conflicts:
tajo-core-tests/src/test/java/org/apache/tajo/engine/planner/physical/TestPhysicalPlanner.java
commit 4f711fa2ff7a18979198d80a70f283f73b91edf9
Author: JaeHwa Jung <bl...@apache.org>
Date: 2015-11-02T07:14:49Z
Remove unnecessary method
commit 33dc1407a3d1417a81895e5e36d528f64c88bbbe
Author: JaeHwa Jung <bl...@apache.org>
Date: 2015-11-02T07:22:24Z
Update comments
commit dede3e2957a2cee7bccd235a3f873aac0ab40377
Author: JaeHwa Jung <bl...@apache.org>
Date: 2015-11-02T07:40:15Z
Remove unnecessary constructor parameter of PhysicalPlannerImpl
commit ccc4f6cb2e12bd642d00be08f393f6754e74db7f
Author: JaeHwa Jung <bl...@apache.org>
Date: 2015-11-02T07:55:16Z
Remove unnecessary parameter of PartitionedTableUtil::buildTupleFromPartitionName
commit d5f563a1d6764f21f80e91a2540a9de5330a38cf
Author: JaeHwa Jung <bl...@apache.org>
Date: 2015-11-02T07:59:32Z
Update wrong indent
commit 086b02beb700e125a6ba37cbe275965150a89183
Author: JaeHwa Jung <bl...@apache.org>
Date: 2015-11-02T07:59:57Z
Remove unused package
commit 22731ec4a13f1ad0e75d7987966c17715afbeb52
Author: JaeHwa Jung <bl...@apache.org>
Date: 2015-11-02T08:20:01Z
Update wrong comparison operator
commit 437f5ecdc7fad8b056bb638ea0897cd6e455b9b8
Author: JaeHwa Jung <bl...@apache.org>
Date: 2015-11-02T08:24:05Z
Update log message
commit d76f41aac39e7536f4acac265559fb136aa05b71
Author: JaeHwa Jung <bl...@apache.org>
Date: 2015-11-03T00:33:16Z
When rewriting PartitionedTableScanNode, set partition paths and table volume.
commit 126f5e06de3aa88563281fd0c382d03f4afab5bf
Author: JaeHwa Jung <bl...@apache.org>
Date: 2015-11-03T00:36:27Z
Merge branch 'master' of https://git-wip-us.apache.org/repos/asf/tajo into TAJO-1952
commit 9112ceb61547667020423bf4fbe18f99c07c2539
Author: JaeHwa Jung <bl...@apache.org>
Date: 2015-11-03T01:47:32Z
Update the result message of partition pruning
commit 71d65a5dec1571852448f0b349e121a9a0268a5e
Author: JaeHwa Jung <bl...@apache.org>
Date: 2015-11-05T08:57:54Z
Merge branch 'master' of https://git-wip-us.apache.org/repos/asf/tajo into TAJO-1952
commit c53dfab62079707d1e106e81b1361bc5bc21d0ad
Author: JaeHwa Jung <bl...@apache.org>
Date: 2015-11-06T07:19:33Z
Merge branch 'master' of https://git-wip-us.apache.org/repos/asf/tajo into TAJO-1952
commit a3af9ad23f55f43e0af8e0f4f84159f8448c9795
Author: JaeHwa Jung <bl...@apache.org>
Date: 2015-11-10T00:43:47Z
Merge branch 'master' of https://git-wip-us.apache.org/repos/asf/tajo into TAJO-1952
commit 548bd4293fb3de0ab725bc2372d514ffd5a70a96
Author: JaeHwa Jung <bl...@apache.org>
Date: 2015-11-19T02:54:44Z
Merge branch 'master' of https://git-wip-us.apache.org/repos/asf/tajo into TAJO-1952
Conflicts:
tajo-plan/src/main/java/org/apache/tajo/plan/logical/PartitionedTableScanNode.java
commit 66c1c496b9fb55ee2d872e843ffe2c2481adbd60
Author: JaeHwa Jung <bl...@apache.org>
Date: 2015-11-20T02:01:54Z
Remove unused member variable.
commit 25e23666ce826c5e0f2a64726f8e4e73ab204c2e
Author: JaeHwa Jung <bl...@apache.org>
Date: 2015-11-20T02:06:54Z
Remove unused method
commit c7f89f7b90cc65b8bdd294ab40426cebb73c99d0
Author: JaeHwa Jung <bl...@apache.org>
Date: 2015-11-20T02:14:53Z
Separate partition processing logic from existing split method.
commit e670f25eab4965bd3d5bcfbaf0540194a7ed37d9
Author: JaeHwa Jung <bl...@apache.org>
Date: 2015-11-20T02:22:30Z
Rename partitionName to partitionKeys in PartitionedFileFragmentProto
commit 83065d626fb2cc58e64fa17c3280191a44ddd471
Author: JaeHwa Jung <bl...@apache.org>
Date: 2015-11-21T15:16:13Z
Merge branch 'master' of https://git-wip-us.apache.org/repos/asf/tajo into TAJO-1952
commit c21d06538a7b99afd4d43ec73a8832d99754c598
Author: JaeHwa Jung <bl...@apache.org>
Date: 2015-11-21T15:26:50Z
Rename PartitionedFileFragment to PartitionFileFragment
commit 9d92e540c1d65b073508e5f783dd6d8358663408
Author: JaeHwa Jung <bl...@apache.org>
Date: 2015-11-21T15:33:00Z
Recover partition paths in LogicalNode
commit f9fcd273abb8960ff1663ffd181410d26a2a6681
Author: JaeHwa Jung <bl...@apache.org>
Date: 2015-11-21T15:35:43Z
Add PartitionedTableWriter::buildTupleFromPartitionName
commit 344384b5e7708be4125ba7a3bf578f13030516b4
Author: JaeHwa Jung <bl...@apache.org>
Date: 2015-11-23T03:19:16Z
PartitionedTableRewriter should set PartitionContent
commit b327993ba1c4749621f41fcdd67339551b64ef4e
Author: JaeHwa Jung <bl...@apache.org>
Date: 2015-11-23T03:22:21Z
Remove unused packages
----
> Optimize Partition Table Split Computation for Amazon S3
> --------------------------------------------------------
>
> Key: TAJO-2111
> URL: https://issues.apache.org/jira/browse/TAJO-2111
> Project: Tajo
> Issue Type: Sub-task
> Components: S3, Storage
> Reporter: Jaehwa Jung
> Assignee: Jaehwa Jung
>
> Currently, Split computation of partitioned table proceed as follows.
> * Listing all partition directories of specified partitioned table
> * Listing all files of each partition directories
> For examples, assume a table with 1000 partitions and each partitions includes 10 files. In above case, AWS S3 API will be called 1000 times and it will become a huge bottleneck.
> To improve current computation, we have to use {{S3::listObjects}} and implement the following algorithm to efficiently list multiple input locations:
> * Given a list of S3 locations, apply prefix listing to a common prefix to get the metadata of 1000 objects at a time.
> * While applying prefix listing in the above step, skip those objects that do not fall under the input list of S3 locations to avoid ending up listing large number of irrelevant objects in pathogenic cases.
> Honestly, I'm inspired by Qubole's blog post as follows https://www.qubole.com/blog/product/optimizing-s3-bulk-listings-for-performant-hive-queries/.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)