You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@tajo.apache.org by "ASF GitHub Bot (JIRA)" <ji...@apache.org> on 2015/07/17 12:05:05 UTC
[jira] [Commented] (TAJO-1343) Improve the memory usage of physical executors

    [ https://issues.apache.org/jira/browse/TAJO-1343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14631125#comment-14631125 ] 

ASF GitHub Bot commented on TAJO-1343:
--------------------------------------

GitHub user jihoonson opened a pull request:

    https://github.com/apache/tajo/pull/634

    TAJO-1343: Improve the memory usage of physical executors

    Here are some highlights.
    * Cleanup duplicated tuple creation.
    * To maintain tuple creation easily, I've added three new data structures, called TupleMap, TupleList, and TupleSet. They automatically clone tuples whenever a new item is added.
    * Fix a bug in WindowAggregation and two wrong tests.
    * Fix a bug in timezone.
    
    I will conduct some tests and share the results.

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/jihoonson/tajo-2 TAJO-1343_2

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/tajo/pull/634.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #634
    
----
commit 6b8ccc2f36712bd6c8d4d5610cbfdac70d1afba0
Author: Jihoon Son <ji...@apache.org>
Date:   2015-07-13T06:56:49Z

    TAJO-1343_2

commit b9bec79372d010f8645067237c9656cc4c0e60d4
Author: Jihoon Son <ji...@apache.org>
Date:   2015-07-13T07:42:08Z

    TAJO-1343_2

commit e9a40da63ee7376cd60ba537d1f5ab06f636bca6
Author: Jihoon Son <ji...@apache.org>
Date:   2015-07-13T08:49:23Z

    TAJO-1343_2

commit 877c20250d196954edf49ccfb70fd20a520b2985
Author: Jihoon Son <ji...@apache.org>
Date:   2015-07-14T01:34:57Z

    Fix a bug in window function

commit f5fa81d82901304833ae7b65c7f3c82044c40acb
Author: Jihoon Son <ji...@apache.org>
Date:   2015-07-14T13:55:44Z

    Remaining OOM

commit f6193e6881eb15d1e2be8d7594206e2ea2071f9c
Author: Jihoon Son <ji...@apache.org>
Date:   2015-07-15T10:10:13Z

    Fix OOM

commit dd91fdd31ec0c7ca561f29f2debf402d8dcf2be3
Author: Jihoon Son <ji...@apache.org>
Date:   2015-07-15T13:18:14Z

    Fix test failures

commit 482072fd9eaa4eccc40c12c9f32c0c822d67d3a8
Author: Jihoon Son <ji...@apache.org>
Date:   2015-07-16T08:07:35Z

    Fixed RangeShuffleWriteExec bug

commit d3793e4c5bb8234bf31bd99c55590b7065d27386
Author: Jihoon Son <ji...@apache.org>
Date:   2015-07-16T16:46:54Z

    Passed all tests

commit 230d57033791dd3dd78e560f7db36a9bbc0451d5
Author: Jihoon Son <ji...@apache.org>
Date:   2015-07-17T09:53:58Z

    Fix every test failure and add comments

commit 5ebe05dc9f7464af040050e0ec17bf2ca0cf4fdc
Author: Jihoon Son <ji...@apache.org>
Date:   2015-07-17T09:55:11Z

    Merge branch 'master' of https://git-wip-us.apache.org/repos/asf/tajo into TAJO-1343_2
    
    Conflicts:
    	tajo-core/src/test/java/org/apache/tajo/engine/planner/TestLogicalPlanner.java

----


> Improve the memory usage of physical executors
> ----------------------------------------------
>
>                 Key: TAJO-1343
>                 URL: https://issues.apache.org/jira/browse/TAJO-1343
>             Project: Tajo
>          Issue Type: Improvement
>          Components: physical operator
>            Reporter: Jihoon Son
>            Assignee: Jihoon Son
>            Priority: Critical
>             Fix For: 0.11.0
>
>
> *Introduction*
> Basically, the tuple instance is maintained as a singleton in physical operators. However, there are some memory-based operator types which need to keep multiple tuples in the memory. In these operators, multiple instances must be created for each tuple.
> *Problem*
> Currently, there are some temporal routines to avoid unexpected problems due to the singleton instance of tuple. However, the methodology is inconsistent and complex, which causes unexpected bugs.
> *Solution*
> A consistent methodology is needed to handle this problem. Only the operators that keep multiple tuples in memory must maintain those tuples with separate instances.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)