You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@tajo.apache.org by eminency <gi...@git.apache.org> on 2015/11/24 07:39:54 UTC
[GitHub] tajo pull request: TAJO-1855: Add document for tablespace configur...
GitHub user eminency opened a pull request:
https://github.com/apache/tajo/pull/873
TAJO-1855: Add document for tablespace configuration to the Configuration chapter.
@hyunsik , @jihoonson
I added a document describing tablespace feature -exactly speaking, for storage-site.json.
Please review that.
Thanks.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/eminency/tajo TAJO-1855
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/tajo/pull/873.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #873
----
commit 7dde36edfa5f25bba14ea5694e7d84da51bcf564
Author: Jongyoung Park <em...@gmail.com>
Date: 2015-11-24T05:41:34Z
Don't use dash
commit 559ab3e44ca6f5f71aef11c5128fbcbd33906ce8
Author: Jongyoung Park <em...@gmail.com>
Date: 2015-11-24T06:01:03Z
storage-site.json document is added
commit 715f19fb9f5bf918107f662001fe66045539be6a
Author: Jongyoung Park <em...@gmail.com>
Date: 2015-11-24T06:05:18Z
Refined
commit d50ec1e69b970ef00d71d1d0c77c73d4250f49ac
Author: Jongyoung Park <em...@gmail.com>
Date: 2015-11-24T06:06:08Z
refined more
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] tajo pull request: TAJO-1855: Add document for tablespace configur...
Posted by jihoonson <gi...@git.apache.org>.
Github user jihoonson commented on a diff in the pull request:
https://github.com/apache/tajo/pull/873#discussion_r46116760
--- Diff: tajo-docs/src/main/sphinx/configuration/storage-site-json.rst ---
@@ -0,0 +1,63 @@
+**************************
+The storage-site.json File
+**************************
+
+This file is for configuring :doc:`/table_management/tablespaces`.
+The syntax of ``storage-site.json`` is like this basically:
+
+.. code:: json
+
+ {
+ "spaces": {
+ <space config 1>, <space config 2>, ...
+ },
+ "storages": {
+ <storage config 1>, <storage config 2>, ...
+ }
+ }
+
+------
+Spaces
+------
+
+This is a section for registering tablespaces. Some space config example is here:
+
+.. code:: json
+
+ "spaces": {
+ "jdbc_pgsql": {
+ "uri": "jdbc:postgresql://127.0.0.1:5740/origin",
+ "configs": {
+ "mapped_database":"tajo_pgsql_origin",
+ "connection_properties": {
+ "user":"tajouser",
+ "password":"123456"
+ }
+ }
+ },
+ ...
+ }
+
+* **space name** : Your own tablespace name which indicates a specific tablespace. Alpha-numeric characters and underscore(_) are permitted.
+* **uri** : An URI address of a tablespace
+* **configs** : It is optional. You can specify it as JSON object to pass to each tablespace handler.
+
+After you specify a tablespace, you can use it in `create table statement <../sql_language/ddl.html#create-table>`_.
+
+--------
+Storages
+--------
+
+This is for registering storage format and custom storage handler class.
+Tajo already supports HDFS, HBase, PostgreSQL, Amazon S3, Openstack Swift, etc, thus in usual cases using mentioned storages, you don't have to use ``storages`` configuration.
+Anyway for your knowledge, here is one storage conf example(it's quoted from Tajo internal hdfs storage configuration which is already being used).
--- End diff --
I suggest ```However, if you want to use your custom storage as one of Tajo's data source, you need to add a configuration for your storage. Here is an example of HDFS storage. Please see `:doc:storage_plugins/overview` for more information```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] tajo pull request: TAJO-1855: Add document for tablespace configur...
Posted by hyunsik <gi...@git.apache.org>.
Github user hyunsik commented on a diff in the pull request:
https://github.com/apache/tajo/pull/873#discussion_r45934286
--- Diff: tajo-docs/src/main/sphinx/configuration/storage-site-json.rst ---
@@ -0,0 +1,57 @@
+**************************
+The storage-site.json File
+**************************
+
+This file is for configuring :doc:`table spaces</table_management/tablespaces>`.
+The syntax of ``storage-site.json`` is like this basically:
+
+.. code:: json
+
+ {
+ "spaces": {
+ <space config 1>, <space config 2>, ...
+ },
+ "storages": {
+ <storage config 1>, <storage config 2>, ...
+ }
+ }
+
+------
+Spaces
+------
+
+This is a section for registering table spaces. Some space config example is here:
+
+.. code:: json
+
+ "spaces": {
+ "hbase_cluster1": {
+ "uri": "hbase://quorum1:port,quorum2:port/",
+ "config": { ... }
+ },
+ ...
+ }
+
+* **space name** : Your own table space name which indicates a specific table space. Alpha-numeric characters and underscore(_) are permitted.
+* **uri** : An URI address of a table space
+* **config** : It is optional. You can specify it as JSON object to pass to each table space handler.
--- End diff --
It should be ``configs`` instead of ``config``.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] tajo pull request: TAJO-1855: Add document for tablespace configur...
Posted by jihoonson <gi...@git.apache.org>.
Github user jihoonson commented on the pull request:
https://github.com/apache/tajo/pull/873#issuecomment-161279842
+1 thanks for quick update.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] tajo pull request: TAJO-1855: Add document for tablespace configur...
Posted by hyunsik <gi...@git.apache.org>.
Github user hyunsik commented on the pull request:
https://github.com/apache/tajo/pull/873#issuecomment-159766366
Even though, we need more description for them, the current patch looks good to me. Later, we can improve it. I leaved some comments. Please correct them.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] tajo pull request: TAJO-1855: Add document for tablespace configur...
Posted by eminency <gi...@git.apache.org>.
Github user eminency commented on the pull request:
https://github.com/apache/tajo/pull/873#issuecomment-160833947
Good advices.
Comments are applied, @jihoonson
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] tajo pull request: TAJO-1855: Add document for tablespace configur...
Posted by eminency <gi...@git.apache.org>.
Github user eminency commented on the pull request:
https://github.com/apache/tajo/pull/873#issuecomment-159792025
Comments are applied.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] tajo pull request: TAJO-1855: Add document for tablespace configur...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/tajo/pull/873
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] tajo pull request: TAJO-1855: Add document for tablespace configur...
Posted by hyunsik <gi...@git.apache.org>.
Github user hyunsik commented on a diff in the pull request:
https://github.com/apache/tajo/pull/873#discussion_r45934293
--- Diff: tajo-docs/src/main/sphinx/configuration/storage-site-json.rst ---
@@ -0,0 +1,57 @@
+**************************
+The storage-site.json File
+**************************
+
+This file is for configuring :doc:`table spaces</table_management/tablespaces>`.
+The syntax of ``storage-site.json`` is like this basically:
+
+.. code:: json
+
+ {
+ "spaces": {
+ <space config 1>, <space config 2>, ...
+ },
+ "storages": {
+ <storage config 1>, <storage config 2>, ...
+ }
+ }
+
+------
+Spaces
+------
+
+This is a section for registering table spaces. Some space config example is here:
+
+.. code:: json
+
+ "spaces": {
+ "hbase_cluster1": {
+ "uri": "hbase://quorum1:port,quorum2:port/",
+ "config": { ... }
--- End diff --
It should be ``configs`` instead of ``config``.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] tajo pull request: TAJO-1855: Add document for tablespace configur...
Posted by jihoonson <gi...@git.apache.org>.
Github user jihoonson commented on a diff in the pull request:
https://github.com/apache/tajo/pull/873#discussion_r46116563
--- Diff: tajo-docs/src/main/sphinx/configuration/storage-site-json.rst ---
@@ -0,0 +1,63 @@
+**************************
+The storage-site.json File
+**************************
+
+This file is for configuring :doc:`/table_management/tablespaces`.
+The syntax of ``storage-site.json`` is like this basically:
+
+.. code:: json
+
+ {
+ "spaces": {
+ <space config 1>, <space config 2>, ...
+ },
+ "storages": {
+ <storage config 1>, <storage config 2>, ...
+ }
+ }
+
+------
+Spaces
+------
+
+This is a section for registering tablespaces. Some space config example is here:
+
+.. code:: json
+
+ "spaces": {
+ "jdbc_pgsql": {
+ "uri": "jdbc:postgresql://127.0.0.1:5740/origin",
+ "configs": {
+ "mapped_database":"tajo_pgsql_origin",
+ "connection_properties": {
+ "user":"tajouser",
+ "password":"123456"
+ }
+ }
+ },
+ ...
+ }
+
+* **space name** : Your own tablespace name which indicates a specific tablespace. Alpha-numeric characters and underscore(_) are permitted.
+* **uri** : An URI address of a tablespace
+* **configs** : It is optional. You can specify it as JSON object to pass to each tablespace handler.
+
+After you specify a tablespace, you can use it in `create table statement <../sql_language/ddl.html#create-table>`_.
+
+--------
+Storages
+--------
+
+This is for registering storage format and custom storage handler class.
+Tajo already supports HDFS, HBase, PostgreSQL, Amazon S3, Openstack Swift, etc, thus in usual cases using mentioned storages, you don't have to use ``storages`` configuration.
--- End diff --
I suggest ```you don't have to add any ``storages`` configuration.```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] tajo pull request: TAJO-1855: Add document for tablespace configur...
Posted by jihoonson <gi...@git.apache.org>.
Github user jihoonson commented on a diff in the pull request:
https://github.com/apache/tajo/pull/873#discussion_r45935196
--- Diff: tajo-docs/src/main/sphinx/table_management/tablespaces.rst ---
@@ -39,6 +42,8 @@ The following is an example for two tablespaces for hbase and hdfs:
}
}
+For more tails, see :doc:`conf/storage-site.json</configuration/storage-site-json>`.
--- End diff --
```tails``` -> ```details```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] tajo pull request: TAJO-1855: Add document for tablespace configur...
Posted by jihoonson <gi...@git.apache.org>.
Github user jihoonson commented on a diff in the pull request:
https://github.com/apache/tajo/pull/873#discussion_r46236537
--- Diff: tajo-docs/src/main/sphinx/configuration/storage-site-json.rst ---
@@ -0,0 +1,64 @@
+**************************
+The storage-site.json File
+**************************
+
+This file is for configuring :doc:`/table_management/tablespaces`.
+The syntax of ``storage-site.json`` is like this basically:
+
+.. code:: json
+
+ {
+ "spaces": {
+ <space config 1>, <space config 2>, ...
+ },
+ "storages": {
+ <storage config 1>, <storage config 2>, ...
+ }
+ }
+
+------
+Spaces
+------
+
+This is a section for registering tablespaces. Some space config example is here:
+
+.. code:: json
+
+ "spaces": {
+ "jdbc_pgsql": {
+ "uri": "jdbc:postgresql://127.0.0.1:5740/origin",
+ "configs": {
+ "mapped_database":"tajo_pgsql_origin",
+ "connection_properties": {
+ "user":"tajouser",
+ "password":"123456"
+ }
+ }
+ },
+ ...
+ }
+
+* **space name** : Your own tablespace name which indicates a specific tablespace. Alpha-numeric characters and underscore(_) are permitted.
+* **uri** : An URI address of a tablespace
+* **configs** : It is optional. You can specify it as JSON object to pass to each tablespace handler.
+
+After you specify a tablespace, you can use it in `create table statement <../sql_language/ddl.html#create-table>`_.
+
+--------
+Storages
+--------
+
+This is for registering storage format and custom storage handler class.
+Tajo already supports HDFS, HBase, PostgreSQL, Amazon S3, Openstack Swift, etc, thus in usual cases using mentioned storages, you don't have to add any ``storages`` configuration.
+However, if you want to use your custom storage as one of Tajo's data source, you need to add a configuration for your storage. Here is an example of HDFS storage.
+See `:doc:storage_plugins/overview` for more information.
--- End diff --
This is my mistake. Please fix to ```:doc:`storage_plugins/overview` ```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] tajo pull request: TAJO-1855: Add document for tablespace configur...
Posted by jihoonson <gi...@git.apache.org>.
Github user jihoonson commented on a diff in the pull request:
https://github.com/apache/tajo/pull/873#discussion_r45935367
--- Diff: tajo-docs/src/main/sphinx/configuration/storage-site-json.rst ---
@@ -0,0 +1,57 @@
+**************************
+The storage-site.json File
+**************************
+
+This file is for configuring :doc:`table spaces</table_management/tablespaces>`.
+The syntax of ``storage-site.json`` is like this basically:
+
+.. code:: json
+
+ {
+ "spaces": {
+ <space config 1>, <space config 2>, ...
+ },
+ "storages": {
+ <storage config 1>, <storage config 2>, ...
+ }
+ }
+
+------
+Spaces
+------
+
+This is a section for registering table spaces. Some space config example is here:
--- End diff --
tablespace is a single word.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] tajo pull request: TAJO-1855: Add document for tablespace configur...
Posted by hyunsik <gi...@git.apache.org>.
Github user hyunsik commented on a diff in the pull request:
https://github.com/apache/tajo/pull/873#discussion_r45934407
--- Diff: tajo-docs/src/main/sphinx/configuration/storage-site-json.rst ---
@@ -0,0 +1,57 @@
+**************************
+The storage-site.json File
+**************************
+
+This file is for configuring :doc:`table spaces</table_management/tablespaces>`.
+The syntax of ``storage-site.json`` is like this basically:
+
+.. code:: json
+
+ {
+ "spaces": {
+ <space config 1>, <space config 2>, ...
+ },
+ "storages": {
+ <storage config 1>, <storage config 2>, ...
+ }
+ }
+
+------
+Spaces
+------
+
+This is a section for registering table spaces. Some space config example is here:
+
+.. code:: json
+
+ "spaces": {
+ "hbase_cluster1": {
+ "uri": "hbase://quorum1:port,quorum2:port/",
+ "config": { ... }
--- End diff --
It would be great if we give some examples to set multiple config key-value pairs.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---