You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@tajo.apache.org by blrunner <gi...@git.apache.org> on 2015/08/05 10:23:40 UTC

[GitHub] tajo pull request: TAJO-1740: Update Partition Table document

GitHub user blrunner opened a pull request:

    https://github.com/apache/tajo/pull/677

    TAJO-1740: Update Partition Table document

    Currently, Tajo doesn't provide enough informations about partition table. Thus, we need to add more informations to following documentation.
    
    http://tajo.apache.org/docs/current/partitioning/column_partitioning.html

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/blrunner/tajo TAJO-1740

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/tajo/pull/677.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #677
    
----
commit 56f5e805ea3f1dba9924901081c932172c4a007f
Author: JaeHwa Jung <bl...@apache.org>
Date:   2015-08-05T08:22:37Z

    TAJO-1740: Update Partition Table document

----


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] tajo pull request: TAJO-1740: Update Partition Table document

Posted by hyunsik <gi...@git.apache.org>.
Github user hyunsik commented on the pull request:

    https://github.com/apache/tajo/pull/677#issuecomment-143170169
  
    I recommend you to keep a single page for column partition. Each content is too short. Later, it may be hard to see all contents at once after we add new partition type.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] tajo pull request: TAJO-1740: Update Partition Table document

Posted by blrunner <gi...@git.apache.org>.
Github user blrunner commented on the pull request:

    https://github.com/apache/tajo/pull/677#issuecomment-136245900
  
    Hi @eminency 
    
    Thank you for your detailed review.
    Honestly, I've changed existing contents too much. So, I'll make new PR to keep existing ones. When making new PR, I'll must reference your feedback.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] tajo pull request: TAJO-1740: Update Partition Table document

Posted by eminency <gi...@git.apache.org>.
Github user eminency commented on a diff in the pull request:

    https://github.com/apache/tajo/pull/677#discussion_r37380089
  
    --- Diff: tajo-docs/src/main/sphinx/partitioning/alter_partition.rst ---
    @@ -0,0 +1,66 @@
    +********************************
    +Alter partition
    +********************************
    +
    +You can ALTER TABLE to add or drop partitions for partition table.
    +
    +For example, assume there is a ``student`` table composed of the following schema.
    +
    +.. code-block:: sql
    +
    +  CREATE TABLE student (
    +    id     INT,
    +    name   TEXT,
    +    grade  TEXT
    +  ) PARTITION BY COLUMN (country TEXT, city TEXT);
    +
    +
    +========================
    +ADD PARTITION
    +========================
    +
    +*Synopsis*
    +
    +.. code-block:: sql
    +
    +  ALTER TABLE <table_name> [IF NOT EXISTS] ADD PARTITION (<partition column> = <partition value>, ...) [LOCATION = <partition's path>]
    +
    +*Description*
    +
    +You can use ``ALTER TABLE ADD PARTITION`` to ADD PARTITIONs to a table. The location must be a directory inside of which data files reside. If the location doesn't exist on the file system, Tajo will make the location by force. ``ADD PARTITION`` changes the table metadata, but does not load data. If the data does not exist in the partition's location, queries will not return any results. An error is thrown if the partition for the table already exists. You can use ``IF NOT EXISTS`` to skip the error.
    +
    +*Examples*
    +
    +.. code-block:: sql
    +
    +  -- Each ADD PARTITION clause creates a subdirectory in HDFS.
    +  ALTER TABLE student ADD PARTITION (country='KOREA', city='SEOUL');
    +  ALTER TABLE student ADD PARTITION (country='KOREA', city='PUSAN');
    +  ALTER TABLE student ADD PARTITION (country='USA', city='NEWYORK');
    +  ALTER TABLE student ADD PARTITION (country='USA', city='BOSTON');
    +  -- Redirect queries, INSERT, and LOAD DATA for one partition to a specific different directory.
    +  ALTER TABLE student ADD PARTITION (country='USA', city='BOSTON') LOCATION '/usr/external_data/new_years_day';
    +
    +
    +========================
    + DROP PARTITION
    +========================
    +
    +*Synopsis*
    +
    +.. code-block:: sql
    +
    +  ALTER TABLE <table_name> [IF EXISTS] DROP PARTITION (<partition column> = <partition value>, ...) [PURGE]
    +
    +*Description*
    +
    +You can use ``ALTER TABLE DROP PARTITION`` to drop a partition for a table. This doesn't remove the data for partition table. But if ``PURGE`` is specified, the partition data will be removed. The metadata is completely lost in all cases. An error is thrown if the partition for the table doesn't exists. You can use ``IF EXISTS`` to skip the error.
    +
    +*Examples*
    +
    +.. code-block:: sql
    +
    +  -- Delete just metadata
    +  ALTER TABLE table1 DROP PARTITION (country = 'KOREA' , city = 'SEOUL');
    +  -- Delete table data and metadata
    +  ALTER TABLE table1 DROP PARTITION (country = 'USA', city = 'NEWYORK' ) PURGE
    --- End diff --
    
    ';' is missed


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] tajo pull request: TAJO-1740: Update Partition Table document

Posted by eminency <gi...@git.apache.org>.
Github user eminency commented on a diff in the pull request:

    https://github.com/apache/tajo/pull/677#discussion_r37379980
  
    --- Diff: tajo-docs/src/main/sphinx/partitioning/add_data_to_partition_table.rst ---
    @@ -0,0 +1,62 @@
    +********************************
    +Add data to Partition Table
    +********************************
    +
    +Tajo provides a very useful feature of dynamic partitioning. You don't need to use any syntax with both ``INSERT INTO ... SELECT`` and ``Create Table As Select(CTAS)`` statments for dynamic partitioning. Tajo will automatically filter the data, create directories, move filtered data to appropriate directory and create partition over it.
    +
    +For example, assume there are both ``student_source`` and ``student`` tables composed of the following schema.
    +
    +.. code-block:: sql
    +
    +  CREATE TABLE student_source (
    +    id        INT,
    +    name      TEXT,
    +    gender    char(1),
    +    grade     TEXT,
    +    country   TEXT,
    +    city      TEXT,
    +    phone     TEXT
    +  );
    +
    +  CREATE TABLE student (
    +    id     INT,
    +    name   TEXT,
    +    grade  TEXT
    +  ) PARTITION BY COLUMN (country TEXT, city TEXT);
    +
    +
    +================================================
    +How to INSERT dynamically to partition table
    +================================================
    +
    +If you want to load an entire country or an entire city in one fell swoop:
    +
    +.. code-block:: sql
    +
    +  INSERT OVERWRITE INTO student
    +  SELECT id, name, grade, country, city
    +  FROM   student_source;
    +
    +
    +================================================
    +How to CTAS dynamically to partition table
    +================================================
    +
    +If you want to load an entire country or an entire city in one fell swoop:
    --- End diff --
    
    Explanation is same precisely as above one.
    I think it's better to add some words like 'including CREATE TABLE' or 'when a partition table is created'


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] tajo pull request: TAJO-1740: Update Partition Table document

Posted by eminency <gi...@git.apache.org>.
Github user eminency commented on a diff in the pull request:

    https://github.com/apache/tajo/pull/677#discussion_r37380422
  
    --- Diff: tajo-docs/src/main/sphinx/partitioning/hive_compatibility.rst ---
    @@ -0,0 +1,90 @@
    +********************************
    +Hive Compatibility
    +********************************
    +
    +Tajo provides HiveCatalogStore to process the Hive partitioned tables directly. If you wish to use HiveCatalogStore, you should specify hive configurations to both tajo-env.sh file and catalog-site.xml file. Please see the following page.
    +
    +.. toctree::
    +    :maxdepth: 1
    +
    +   /hive_integration
    +
    +================================================
    +How to create partition table
    +================================================
    +
    +If you want to create a partition table as follows in Tajo:
    +
    +.. code-block:: sql
    +
    +  default> CREATE TABLE student (
    +    id     INT,
    +    name   TEXT,
    +    grade  TEXT
    +  ) PARTITION BY COLUMN (country TEXT, city TEXT);
    +
    +
    +And then you can get table information in Hive:
    +
    +.. code-block:: sql
    +
    +  hive> desc student;
    +  OK
    +  id                  	int
    +  name                	string
    +  grade               	string
    +  country             	string
    +  city                	string
    +
    +  # Partition Information
    +  # col_name            	data_type           	comment
    +
    +  country             	string
    +  city                	string
    +
    +
    +Or as you create the table in Hive:
    +
    +.. code-block:: sql
    +
    +  hive > CREATE TABLE student (
    +    id int,
    +    name string,
    +    grade string
    +  ) PARTITIONED BY (country string, city string)
    +  ROW FORMAT DELIMITED
    +    FIELDS TERMINATED BY '|' ;
    +
    +You will see table information in Tajo:
    +
    +.. code-block:: sql
    +
    +  default> \d student;
    +  table name: default.student
    +  table uri: hdfs://your_hdfs_namespace/user/hive/warehouse/student
    +  store type: TEXT
    +  number of rows: 0
    +  volume: 0 B
    +  Options:
    +    'text.null'='\\N'
    +    'transient_lastDdlTime'='1438756422'
    +    'text.delimiter'='|'
    +
    +  schema:
    +  id	INT4
    +  name	TEXT
    +  grade	TEXT
    +
    +  Partitions:
    +  type:COLUMN
    +  columns::default.student.country (TEXT), default.student.city (TEXT)
    +
    +
    +
    +================================================
    +How to add data to partition table
    +================================================
    +
    +In Tajo, you can add data dynamically to partition table of Hive with both ``INSERT INTO ... SELECT`` and ``Create Table As Select (CTAS)`` statments. Tajo will automatically filter the data to HiveMetastore, create directories and move filtered data to appropriate directory on the distributed file system
    --- End diff --
    
    '.' is missed at the end of last statement.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] tajo pull request: TAJO-1740: Update Partition Table document

Posted by eminency <gi...@git.apache.org>.
Github user eminency commented on a diff in the pull request:

    https://github.com/apache/tajo/pull/677#discussion_r37380187
  
    --- Diff: tajo-docs/src/main/sphinx/partitioning/define_partition_table.rst ---
    @@ -0,0 +1,73 @@
    +*********************************
    +Define Partition Table
    +*********************************
    +
    +Tajo makes it easy to specify an automatic partition scheme when the table is created.
    +
    +================================================
    +How to Create Partitione Table
    +================================================
    +
    +You can create a partitioned table by using the ``PARTITION BY`` clause. For a column partitioned table, you should use
    +the ``PARTITION BY COLUMN`` clause with partition keys.
    +
    +For example, assume there is a table ``student`` composed of the following schema.
    +
    +.. code-block:: sql
    +
    +  id     INT,
    +  name   TEXT,
    +  grade  TEXT
    +
    +Now you want to partition on country. Your Tajo definition would be this:
    +
    +.. code-block:: sql
    +
    +  CREATE TABLE student (
    +    id     INT,
    +    name   TEXT,
    +    grade  TEXT
    +  ) PARTITION BY COLUMN (country TEXT);
    +
    +Now your users will still query on ``WHERE country = '...'`` but the 2nd column will be the original values.
    +Here's an example statement to create a table:
    +
    +.. code-block:: sql
    +
    +  CREATE TABLE student (
    +    id     INT,
    +    name   TEXT,
    +    grade  TEXT
    +  ) USING PARQUET
    +  PARTITION BY COLUMN (country TEXT, city TEXT);
    +
    +The statement above creates the student table with id, name, grade. The table is also partitioned and data is stored in parquet files.
    --- End diff --
    
    Is 'parquet files' right?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] tajo pull request: TAJO-1740: Update Partition Table document

Posted by eminency <gi...@git.apache.org>.
Github user eminency commented on the pull request:

    https://github.com/apache/tajo/pull/677#issuecomment-132450030
  
    Hi, @blrunner .
    
    I reviewed your writing documents and left some inline comments.
    
    Please check them out.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] tajo pull request: TAJO-1740: Update Partition Table document

Posted by blrunner <gi...@git.apache.org>.
Github user blrunner closed the pull request at:

    https://github.com/apache/tajo/pull/677


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] tajo pull request: TAJO-1740: Update Partition Table document

Posted by eminency <gi...@git.apache.org>.
Github user eminency commented on a diff in the pull request:

    https://github.com/apache/tajo/pull/677#discussion_r37380110
  
    --- Diff: tajo-docs/src/main/sphinx/partitioning/define_partition_table.rst ---
    @@ -0,0 +1,73 @@
    +*********************************
    +Define Partition Table
    +*********************************
    +
    +Tajo makes it easy to specify an automatic partition scheme when the table is created.
    +
    +================================================
    +How to Create Partitione Table
    +================================================
    --- End diff --
    
    'Partitione' -> 'Partition'


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] tajo pull request: TAJO-1740: Update Partition Table document

Posted by hyunsik <gi...@git.apache.org>.
Github user hyunsik commented on the pull request:

    https://github.com/apache/tajo/pull/677#issuecomment-143170294
  
    I saw new PR. Please ignore the last comment.
    https://github.com/apache/tajo/pull/728


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] tajo pull request: TAJO-1740: Update Partition Table document

Posted by eminency <gi...@git.apache.org>.
Github user eminency commented on a diff in the pull request:

    https://github.com/apache/tajo/pull/677#discussion_r37380325
  
    --- Diff: tajo-docs/src/main/sphinx/partitioning/define_partition_table.rst ---
    @@ -0,0 +1,73 @@
    +*********************************
    +Define Partition Table
    +*********************************
    +
    +Tajo makes it easy to specify an automatic partition scheme when the table is created.
    +
    +================================================
    +How to Create Partitione Table
    +================================================
    +
    +You can create a partitioned table by using the ``PARTITION BY`` clause. For a column partitioned table, you should use
    +the ``PARTITION BY COLUMN`` clause with partition keys.
    +
    +For example, assume there is a table ``student`` composed of the following schema.
    +
    +.. code-block:: sql
    +
    +  id     INT,
    +  name   TEXT,
    +  grade  TEXT
    +
    +Now you want to partition on country. Your Tajo definition would be this:
    +
    +.. code-block:: sql
    +
    +  CREATE TABLE student (
    +    id     INT,
    +    name   TEXT,
    +    grade  TEXT
    +  ) PARTITION BY COLUMN (country TEXT);
    +
    +Now your users will still query on ``WHERE country = '...'`` but the 2nd column will be the original values.
    +Here's an example statement to create a table:
    +
    +.. code-block:: sql
    +
    +  CREATE TABLE student (
    +    id     INT,
    +    name   TEXT,
    +    grade  TEXT
    +  ) USING PARQUET
    +  PARTITION BY COLUMN (country TEXT, city TEXT);
    +
    +The statement above creates the student table with id, name, grade. The table is also partitioned and data is stored in parquet files.
    +
    +You might have noticed that while the partitioning key columns are a part of the table DDL, they’re only listed in the ``PARTITION BY`` clause. In Tajo, as data is written to disk, each partition of data will be automatically split out into different folders, e.g. country=USA/city=NEWYORK. During a read operation, Tajo will use the folder structure to quickly locate the right partitions and also return the partitioning columns as columns in the result set.
    +
    +
    +==================================================
    +Partition Pruning on Partition Table
    +==================================================
    +
    +The following predicates in the ``WHERE`` clause can be used to prune unqualified column partitions without processing
    +during query planning phase.
    +
    +* ``=``
    +* ``<>``
    +* ``>``
    +* ``<``
    +* ``>=``
    +* ``<=``
    +* LIKE predicates with a leading wild-card character
    +* IN list predicates
    +
    +Now above example table data is partitioned by country and city, so when the query is applied on table it can easily access the required row by the help partitions
    --- End diff --
    
    'the help of partitions' looks better


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---