You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@tajo.apache.org by blrunner <gi...@git.apache.org> on 2016/03/16 11:39:16 UTC
[GitHub] tajo pull request: TAJO-2087: Implement DirectOutputCommitter
GitHub user blrunner opened a pull request:
https://github.com/apache/tajo/pull/979
TAJO-2087: Implement DirectOutputCommitter
Here is prototype codes for ``DirectOutputCommitter``. This PR is not ready to review, it shows my approach to implement ``DirectOutputCommitter``. Current version works as following:
- Register commit history to catalog (TODO).
- Each tasks will write the output data directly to the final location.
- In a commit phase, delete existing files with query type as follows. First, backup existing files or directories to staging directory. And then delete backup files or directories.
- Update the status of commit history to catalog (TODO).
- If query fails, QueryMaster will delete committed files and update the status of query history to catalog (TODO).
- When ``TajoMaster`` starting, it will check the status of query histories to catalog. If it find running query, it will delete committed files and update the status of query history (TODO).
- Add unit test cases for failed query (TODO).
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/blrunner/tajo direct-output-committer
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/tajo/pull/979.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #979
----
commit 083bed51db1e68ed840961e2e169695dde60e116
Author: JaeHwa Jung <bl...@apache.org>
Date: 2016-02-24T02:08:08Z
Add the list of output files and backup files to TaskAttemptContext
commit b39c8d1bcb153d53aae028577935499034bd4b6f
Author: JaeHwa Jung <bl...@apache.org>
Date: 2016-02-24T05:31:55Z
Add outputFiles and backupFiles to Protocol Buffer
commit e3b26ea738ba33e1a6c8b8c856793f5a584eb861
Author: JaeHwa Jung <bl...@apache.org>
Date: 2016-02-24T05:48:02Z
Add property for setting Direct Output Committer to TajoConf and SessionVars
commit 9efb4662957ff39ff215a3c829ece5e69d9ebe36
Author: JaeHwa Jung <bl...@apache.org>
Date: 2016-02-25T01:59:26Z
Remove related property from SessionVars
commit 234f2829768f18fab7c7894aab2ccf7780ae3ffb
Author: JaeHwa Jung <bl...@apache.org>
Date: 2016-03-04T02:44:52Z
Add temporary codes for testing
commit 7effec1fc663d246ffd3e25bfd4a98c803b22607
Author: JaeHwa Jung <bl...@apache.org>
Date: 2016-03-15T09:01:43Z
Merge branch 'master' of https://git-wip-us.apache.org/repos/asf/tajo into direct-output-committer
commit cb762766848c2af5d25e20ab552a2041c67924cc
Author: JaeHwa Jung <bl...@apache.org>
Date: 2016-03-15T09:30:36Z
Prefix of output file name must be the id of query.
commit dce41c6be686916a346dc15a033bea39cc79550b
Author: JaeHwa Jung <bl...@apache.org>
Date: 2016-03-16T05:50:32Z
Implement direct Output Committer to FileTablespace
commit 908ccd2b6c2ebbd602892b979c1ff41d7ed4a820
Author: JaeHwa Jung <bl...@apache.org>
Date: 2016-03-16T06:30:45Z
Implement a method for renaming recursively directories
commit bd1e1b3f16e8b6263ef4e762b621a4ba2235aa34
Author: JaeHwa Jung <bl...@apache.org>
Date: 2016-03-16T06:43:56Z
Remove proto modifications
commit 95e513a04bcfee10643ebe17b0e21074057f0be2
Author: JaeHwa Jung <bl...@apache.org>
Date: 2016-03-16T10:21:06Z
Add session variable and add more unit test cases
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] tajo pull request: TAJO-2087: Support DirectOutputCommitter for AW...
Posted by blrunner <gi...@git.apache.org>.
Github user blrunner commented on the pull request:
https://github.com/apache/tajo/pull/979#issuecomment-212344461
Added a document for DirectOutputCommitter as follows.
https://github.com/blrunner/tajo/blob/direct-output-committer/tajo-docs/src/main/sphinx/table_management/directoutputcommitter.rst
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] tajo pull request: TAJO-2087: Support DirectOutputCommitter for AW...
Posted by blrunner <gi...@git.apache.org>.
Github user blrunner commented on the pull request:
https://github.com/apache/tajo/pull/979#issuecomment-212331340
I implemented unit test cases for verifying following cases.
* ``DirectOutputCommitter`` can recover existing files successfully in query failure case.
* ``DirectOutputCommitter`` can remove output files successfully in query failure case.
* When executing ``INSERT INTO`` query, ``DirectOutputCommitter`` can maintain existing files.
For the reference, I found that outputs of ``TestInsertQuery`` and ``TestTablePartitions`` with ``DirectOutputCommitter`` were equals to outputs of them without ``DirectOutputCommitter``.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] tajo pull request: TAJO-2087: Support DirectOutputCommitter for AW...
Posted by blrunner <gi...@git.apache.org>.
Github user blrunner commented on the pull request:
https://github.com/apache/tajo/pull/979#issuecomment-208744154
I designed the table of direct output commit history as following:
Column Name | Column Type | Null | Desc
-------------------|----------------------|--------------------------|-------------------
QUERY_ID | VARCHAR(128) | NOT NULL | the id of ``Query``, PRIMARY KEY
PATH | VARCHAR(4096) | NOT NULL | the output path of table
START_TIME | BIGINT | NOT NULL | query start time
END_TIME | BIGINT | | query finish time
QUERY_STATE | VARCHAR(50) | NOT NULL | the state of ``Query``, ``TajoProtos.QueryState`` will be used
Implemented necessary codes to ``Query`` and ``TajoMaster`` as following:
* When query starting, add the history to catalog.
* If query fails, ``QueryMaster`` will delete committed files and update the status of output commit history to catalog
* When ``TajoMaster`` starting, it will check the status of output commit histories to catalog. If it find running query, it will delete committed files and update the status of output commit history.
Not yet implemented unit test cases for failed queries.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] tajo pull request: TAJO-2087: Support DirectOutputCommitter for AW...
Posted by blrunner <gi...@git.apache.org>.
Github user blrunner commented on the pull request:
https://github.com/apache/tajo/pull/979#issuecomment-215298258
Testing with S3 finished successfully as following:
* Table type: partitioned table and non-partitioned table
* Insert overwrite and CTAS queries
* Insert into a table without DirectOutputCommitter, and then insert into the table with DirectOutputCommitter, and then check output files for above two query.
* While inserting data, kill the query, and then check two factors: delete temporary data and rollback previous data.
* While inserting data, restart tajo cluster, and then check two factors: delete temporary data and rollback previous data.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---