You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hawq.apache.org by yo...@apache.org on 2016/10/27 19:56:36 UTC
[1/9] incubator-hawq-docs git commit: Updates for register --repair,
partitioning
Repository: incubator-hawq-docs
Updated Branches:
refs/heads/develop 5673447e0 -> 01f3f8e9d
Updates for register --repair, partitioning
Project: http://git-wip-us.apache.org/repos/asf/incubator-hawq-docs/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-hawq-docs/commit/baaf05f1
Tree: http://git-wip-us.apache.org/repos/asf/incubator-hawq-docs/tree/baaf05f1
Diff: http://git-wip-us.apache.org/repos/asf/incubator-hawq-docs/diff/baaf05f1
Branch: refs/heads/develop
Commit: baaf05f1455342f19c09c9de411b0be19b864b65
Parents: e169704
Author: Jane Beckman <jb...@pivotal.io>
Authored: Wed Oct 5 15:44:07 2016 -0700
Committer: Jane Beckman <jb...@pivotal.io>
Committed: Wed Oct 5 15:44:07 2016 -0700
----------------------------------------------------------------------
datamgmt/load/g-register_files.html.md.erb | 5 +++--
1 file changed, 3 insertions(+), 2 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-hawq-docs/blob/baaf05f1/datamgmt/load/g-register_files.html.md.erb
----------------------------------------------------------------------
diff --git a/datamgmt/load/g-register_files.html.md.erb b/datamgmt/load/g-register_files.html.md.erb
index 93625f1..f9c407d 100644
--- a/datamgmt/load/g-register_files.html.md.erb
+++ b/datamgmt/load/g-register_files.html.md.erb
@@ -22,7 +22,7 @@ Requirements for running `hawq register` on the server are:
Files or folders in HDFS can be registered into an existing table, allowing them to be managed as a HAWQ internal table. When registering files, you can optionally specify the maximum amount of data to be loaded, in bytes, using the `--eof` option. If registering a folder, the actual file sizes are used.
-Only HAWQ or Hive-generated Parquet tables are supported. Partitioned tables are not supported. Attempting to register these tables will result in an error.
+Only HAWQ or Hive-generated Parquet tables are supported. Only single-level partitioned tables are supported; registering partitioned tables with more than one level will result in an error.
Metadata for the Parquet file(s) and the destination table must be consistent. Different data types are used by HAWQ tables and Parquet files, so data must be mapped. You must verify that the structure of the parquet files and the HAWQ table are compatible before running `hawq register`.
@@ -66,7 +66,7 @@ select relname from pg_class where oid = segrelid
## <a id="topic1__section3"></a>Registering Data Using Information from a YAML Configuration File
-The `hawq register` command can register HDFS files by using metadata loaded from a YAML configuration file by using the `--config <yaml_config\>` option. Both AO and Parquet tables can be registered. Tables need not exist in HAWQ before being registered. This function can be useful in disaster recovery, allowing information created by the `hawq extract` command to re-create HAWQ tables.
+The `hawq register` command can register HDFS files by using metadata loaded from a YAML configuration file by using the `--config <yaml_config\>` option. Both AO and Parquet tables can be registered. Tables need not exist in HAWQ before being registered. In disaster recovery, information in a YAML-format file created by the `hawq extract` command can re-create HAWQ tables by using metadata from a backup checkpoint.
You can also use a YAML confguration file to append HDFS files to an existing HAWQ table or create a table and register it into HAWQ.
@@ -77,6 +77,7 @@ Data is registered according to the following conditions:
- Existing tables have files appended to the existing HAWQ table.
- If a table does not exist, it is created and registered into HAWQ. The catalog table will be updated with the file size specified by the YAML file.
- If the -\\\-force option is used, the data in existing catalog tables is erased and re-registered. All HDFS-related catalog contents in `pg_aoseg.pg_paqseg_$relid ` are cleared. The original files on HDFS are retained.
+- The -\\\-repair option rolls data back to a specified checkpoint. If the table already exists, both the file folder and `pg_aoseg.pg_paqseg_$relid` catalog entry are folled back to the checkpoint configuration in the YAML file. Files generated after the timestamp of the checkpoint will be deleted. Hash table redistribution, table truncate, and table drop are not supported. Using the -\\\- repair option with redistributed table data will result in an error.
Tables using random distribution are preferred for registering into HAWQ. If hash tables are to be registered, the distribution policy in the YAML file must match that of the table being registered into.
[8/9] incubator-hawq-docs git commit: Fix xref
Posted by yo...@apache.org.
Fix xref
Project: http://git-wip-us.apache.org/repos/asf/incubator-hawq-docs/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-hawq-docs/commit/8b65086b
Tree: http://git-wip-us.apache.org/repos/asf/incubator-hawq-docs/tree/8b65086b
Diff: http://git-wip-us.apache.org/repos/asf/incubator-hawq-docs/diff/8b65086b
Branch: refs/heads/develop
Commit: 8b65086b910d89bc169405bfc339730db6bf59d5
Parents: 0895144
Author: Jane Beckman <jb...@pivotal.io>
Authored: Wed Oct 19 15:12:06 2016 -0700
Committer: Jane Beckman <jb...@pivotal.io>
Committed: Wed Oct 19 15:12:06 2016 -0700
----------------------------------------------------------------------
datamgmt/load/g-register_files.html.md.erb | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-hawq-docs/blob/8b65086b/datamgmt/load/g-register_files.html.md.erb
----------------------------------------------------------------------
diff --git a/datamgmt/load/g-register_files.html.md.erb b/datamgmt/load/g-register_files.html.md.erb
index 97cd637..beea0f6 100644
--- a/datamgmt/load/g-register_files.html.md.erb
+++ b/datamgmt/load/g-register_files.html.md.erb
@@ -208,7 +208,7 @@ group {
### Extracting Metadata
-For more information on extracting metadata to a YAML file and the output content of the YAML file, refer to the reference page for [hawq extract](hawqextract.html#topic1).
+For more information on extracting metadata to a YAML file and the output content of the YAML file, refer to the reference page for [hawq extract](../../reference/cli/admin_utilities/hawqextract.html#topic1).
[7/9] incubator-hawq-docs git commit: Remove --repair option
Posted by yo...@apache.org.
Remove --repair option
Project: http://git-wip-us.apache.org/repos/asf/incubator-hawq-docs/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-hawq-docs/commit/08951440
Tree: http://git-wip-us.apache.org/repos/asf/incubator-hawq-docs/tree/08951440
Diff: http://git-wip-us.apache.org/repos/asf/incubator-hawq-docs/diff/08951440
Branch: refs/heads/develop
Commit: 08951440ec17ad5c82a908e7679da6d2ea8cf648
Parents: 73b8d8d
Author: Jane Beckman <jb...@pivotal.io>
Authored: Wed Oct 19 14:05:20 2016 -0700
Committer: Jane Beckman <jb...@pivotal.io>
Committed: Wed Oct 19 14:05:20 2016 -0700
----------------------------------------------------------------------
datamgmt/load/g-register_files.html.md.erb | 1 -
reference/cli/admin_utilities/hawqregister.html.md.erb | 6 +-----
2 files changed, 1 insertion(+), 6 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-hawq-docs/blob/08951440/datamgmt/load/g-register_files.html.md.erb
----------------------------------------------------------------------
diff --git a/datamgmt/load/g-register_files.html.md.erb b/datamgmt/load/g-register_files.html.md.erb
index 79f87ad..97cd637 100644
--- a/datamgmt/load/g-register_files.html.md.erb
+++ b/datamgmt/load/g-register_files.html.md.erb
@@ -80,7 +80,6 @@ Data is registered according to the following conditions:
- Existing tables have files appended to the existing HAWQ table.
- If a table does not exist, it is created and registered into HAWQ. The catalog table will be updated with the file size specified by the YAML file.
- If the -\\\-force option is used, the data in existing catalog tables is erased and re-registered. All HDFS-related catalog contents in `pg_aoseg.pg_paqseg_$relid ` are cleared. The original files on HDFS are retained.
-- If the -\\\-repair option is used, data is rolled back to a specified checkpoint. If the table already exists, both the file folder and `pg_aoseg.pg_paqseg_$relid` catalog entry are rolled back to the checkpoint configuration in the YAML file. Files generated after the timestamp of the checkpoint will be deleted. you cannot redistribute, truncate, or drop hash tables. Using the -\\\- repair option with redistributed table data will result in an error.
Tables using random distribution are preferred for registering into HAWQ.
http://git-wip-us.apache.org/repos/asf/incubator-hawq-docs/blob/08951440/reference/cli/admin_utilities/hawqregister.html.md.erb
----------------------------------------------------------------------
diff --git a/reference/cli/admin_utilities/hawqregister.html.md.erb b/reference/cli/admin_utilities/hawqregister.html.md.erb
index 21a8dd8..1bca10e 100644
--- a/reference/cli/admin_utilities/hawqregister.html.md.erb
+++ b/reference/cli/admin_utilities/hawqregister.html.md.erb
@@ -23,7 +23,6 @@ Misc. Options:
[-f <filepath>]
[-e <eof>]
[--force]
- [--repair]
[-c <yml_config>]
hawq register help | -?
hawq register --version
@@ -70,7 +69,7 @@ The register process behaves differently, according to different conditions.
- Existing tables have files appended to the existing HAWQ table.
- If a table does not exist, it is created and registered into HAWQ.
- If the -\\\-force option is used, the data in existing catalog tables is erased and re-registered.
-- If the -\\\-repair option is used, data is rolled back to the checkpoint configuration in the YAML file. Files generated after the timestamp of the checkpoint will be deleted. Hash table redistribution, table truncate, and table drop are not supported.
+
###Limitations for Registering Hive Tables to HAWQ
The currently-supported data types for generating Hive tables into HAWQ tables are: boolean, int, smallint, tinyint, bigint, float, double, string, binary, char, and varchar.
@@ -210,9 +209,6 @@ The following options are used with specific use models.
<dt>-F , -\\\-force</dt>
<dd>Used for disaster recovery of a cluster. Clears all HDFS-related catalog contents in `pg_aoseg.pg_paqseg_$relid `and re-registers files to a specified table. The HDFS files are not removed or modified. To use this option for recovery, data is assumed to be periodically imported to the cluster to be recovered. Used with Usage Model 2.</dd>
-<dt>-R , -\\\-repair \<yml_config\> </dt>
-<dd>Rolls data back to a specified checkpoint. If the table already exists, both the file folder and `pg_aoseg.pg_paqseg_$relid` catalog entry are rolled back to the checkpoint configuration in the YAML file. Deletes any files generated after the timestamp of the checkpoint. Used with Usage Model 2.</dd>
-
<dt>-c , -\\\-config \<yml_config\> </dt>
<dd>Registers files specified by YAML-format configuration files into HAWQ. Used with Usage Model 2.</dd>
[6/9] incubator-hawq-docs git commit: Additional mention of hawq
extract
Posted by yo...@apache.org.
Additional mention of hawq extract
Project: http://git-wip-us.apache.org/repos/asf/incubator-hawq-docs/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-hawq-docs/commit/73b8d8db
Tree: http://git-wip-us.apache.org/repos/asf/incubator-hawq-docs/tree/73b8d8db
Diff: http://git-wip-us.apache.org/repos/asf/incubator-hawq-docs/diff/73b8d8db
Branch: refs/heads/develop
Commit: 73b8d8db0eabb4293efaae92810c912c9f68d6e9
Parents: 285c92b
Author: Jane Beckman <jb...@pivotal.io>
Authored: Tue Oct 18 14:13:08 2016 -0700
Committer: Jane Beckman <jb...@pivotal.io>
Committed: Tue Oct 18 14:13:08 2016 -0700
----------------------------------------------------------------------
datamgmt/load/g-register_files.html.md.erb | 11 +++++++----
reference/cli/admin_utilities/hawqregister.html.md.erb | 2 +-
2 files changed, 8 insertions(+), 5 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-hawq-docs/blob/73b8d8db/datamgmt/load/g-register_files.html.md.erb
----------------------------------------------------------------------
diff --git a/datamgmt/load/g-register_files.html.md.erb b/datamgmt/load/g-register_files.html.md.erb
index dc2c8e1..79f87ad 100644
--- a/datamgmt/load/g-register_files.html.md.erb
+++ b/datamgmt/load/g-register_files.html.md.erb
@@ -82,10 +82,9 @@ Data is registered according to the following conditions:
- If the -\\\-force option is used, the data in existing catalog tables is erased and re-registered. All HDFS-related catalog contents in `pg_aoseg.pg_paqseg_$relid ` are cleared. The original files on HDFS are retained.
- If the -\\\-repair option is used, data is rolled back to a specified checkpoint. If the table already exists, both the file folder and `pg_aoseg.pg_paqseg_$relid` catalog entry are rolled back to the checkpoint configuration in the YAML file. Files generated after the timestamp of the checkpoint will be deleted. you cannot redistribute, truncate, or drop hash tables. Using the -\\\- repair option with redistributed table data will result in an error.
-Tables using random distribution are preferred for registering into HAWQ. If hash tables are to be registered, the distribution policy in the YAML file must match that of the table being registered into.
-
-In registering hash tables, the size of the registered file should be identical to or a multiple of the hash table bucket number. When registering hash distributed tables using a YAML file, the order of the files in the YAML file should reflect the hash distribution.
+Tables using random distribution are preferred for registering into HAWQ.
+There are additional restrictions when registering hash tables. When registering hash-distributed tables using a YAML file, the distribution policy in the YAML file must match that of the table being registered into and the order of the files in the YAML file should reflect the hash distribution. The size of the registered file should be identical to or a multiple of the hash table bucket number.
###Example: Registration using a YAML Configuration File
@@ -118,7 +117,7 @@ Select the new table and check to verify that the content has been registered.
## <a id="topic1__section4"></a>Data Type Mapping<a id="topic1__section4"></a>
-HIVE and Parquet tables use different data types than HAWQ tables. Mapping must be used for metadata compatibility. You are responsible for making sure your implementation is mapped to the appropriate data type before running `hawq register`. The tables below show equivalent data types, if available.
+HIVE and Parquet tables use different data types than HAWQ tables and must be mapped for metadata compatibility. You are responsible for making sure your implementation is mapped to the appropriate data type before running `hawq register`. The tables below show equivalent data types, if available.
<span class="tablecap">Table 1. HAWQ to Parquet Mapping</span>
@@ -208,5 +207,9 @@ group {
| varchar | varchar |
+### Extracting Metadata
+
+For more information on extracting metadata to a YAML file and the output content of the YAML file, refer to the reference page for [hawq extract](hawqextract.html#topic1).
+
http://git-wip-us.apache.org/repos/asf/incubator-hawq-docs/blob/73b8d8db/reference/cli/admin_utilities/hawqregister.html.md.erb
----------------------------------------------------------------------
diff --git a/reference/cli/admin_utilities/hawqregister.html.md.erb b/reference/cli/admin_utilities/hawqregister.html.md.erb
index 2f1259b..21a8dd8 100644
--- a/reference/cli/admin_utilities/hawqregister.html.md.erb
+++ b/reference/cli/admin_utilities/hawqregister.html.md.erb
@@ -56,7 +56,7 @@ Two usage models are available.
Metadata for the Parquet file(s) and the destination table must be consistent. Different data types are used by HAWQ tables and Parquet files, so the data is mapped. Refer to the section [Data Type Mapping](hawqregister.html#topic1__section7) below. You must verify that the structure of the Parquet files and the HAWQ table are compatible before running `hawq register`.
####Limitations
-Only HAWQ or Hive-generated Parquet tables are supported. Partitioned tables are supported, but only single-level partitioned tables can be registered.
+Only HAWQ or Hive-generated Parquet tables are supported. Partitioned tables are supported, but only single-level partitioned tables can be registered into HAWQ.
Hash tables are not supported in this use model.
###Usage Model 2: Use information from a YAML configuration file to register data
[9/9] incubator-hawq-docs git commit: Merge branch
'feature/newregister' of https://github.com/janebeckman/incubator-hawq-docs
into develop
Posted by yo...@apache.org.
Merge branch 'feature/newregister' of https://github.com/janebeckman/incubator-hawq-docs into develop
Project: http://git-wip-us.apache.org/repos/asf/incubator-hawq-docs/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-hawq-docs/commit/01f3f8e9
Tree: http://git-wip-us.apache.org/repos/asf/incubator-hawq-docs/tree/01f3f8e9
Diff: http://git-wip-us.apache.org/repos/asf/incubator-hawq-docs/diff/01f3f8e9
Branch: refs/heads/develop
Commit: 01f3f8e9d9314edc8d26d23c51894aaf9ac77613
Parents: 5673447 8b65086
Author: David Yozie <yo...@apache.org>
Authored: Thu Oct 27 12:56:18 2016 -0700
Committer: David Yozie <yo...@apache.org>
Committed: Thu Oct 27 12:56:18 2016 -0700
----------------------------------------------------------------------
datamgmt/load/g-register_files.html.md.erb | 36 ++++++++++++--------
.../cli/admin_utilities/hawqextract.html.md.erb | 30 ++++++++++++++--
.../admin_utilities/hawqregister.html.md.erb | 7 ++--
3 files changed, 53 insertions(+), 20 deletions(-)
----------------------------------------------------------------------
[2/9] incubator-hawq-docs git commit: hawq register --repair update
Posted by yo...@apache.org.
hawq register --repair update
Project: http://git-wip-us.apache.org/repos/asf/incubator-hawq-docs/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-hawq-docs/commit/3fde706c
Tree: http://git-wip-us.apache.org/repos/asf/incubator-hawq-docs/tree/3fde706c
Diff: http://git-wip-us.apache.org/repos/asf/incubator-hawq-docs/diff/3fde706c
Branch: refs/heads/develop
Commit: 3fde706c4cf33351096ad86681b02e03073bbacf
Parents: baaf05f
Author: Jane Beckman <jb...@pivotal.io>
Authored: Thu Oct 6 16:38:23 2016 -0700
Committer: Jane Beckman <jb...@pivotal.io>
Committed: Thu Oct 6 16:38:23 2016 -0700
----------------------------------------------------------------------
datamgmt/load/g-register_files.html.md.erb | 2 +-
reference/cli/admin_utilities/hawqregister.html.md.erb | 11 ++++++++---
2 files changed, 9 insertions(+), 4 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-hawq-docs/blob/3fde706c/datamgmt/load/g-register_files.html.md.erb
----------------------------------------------------------------------
diff --git a/datamgmt/load/g-register_files.html.md.erb b/datamgmt/load/g-register_files.html.md.erb
index f9c407d..1d95492 100644
--- a/datamgmt/load/g-register_files.html.md.erb
+++ b/datamgmt/load/g-register_files.html.md.erb
@@ -77,7 +77,7 @@ Data is registered according to the following conditions:
- Existing tables have files appended to the existing HAWQ table.
- If a table does not exist, it is created and registered into HAWQ. The catalog table will be updated with the file size specified by the YAML file.
- If the -\\\-force option is used, the data in existing catalog tables is erased and re-registered. All HDFS-related catalog contents in `pg_aoseg.pg_paqseg_$relid ` are cleared. The original files on HDFS are retained.
-- The -\\\-repair option rolls data back to a specified checkpoint. If the table already exists, both the file folder and `pg_aoseg.pg_paqseg_$relid` catalog entry are folled back to the checkpoint configuration in the YAML file. Files generated after the timestamp of the checkpoint will be deleted. Hash table redistribution, table truncate, and table drop are not supported. Using the -\\\- repair option with redistributed table data will result in an error.
+- If the -\\\-repair option is used, data is rolled back to a specified checkpoint. If the table already exists, both the file folder and `pg_aoseg.pg_paqseg_$relid` catalog entry are rolled back to the checkpoint configuration in the YAML file. Files generated after the timestamp of the checkpoint will be deleted. you cannot redistribute, truncate, or drop hash tables. Using the -\\\- repair option with redistributed table data will result in an error.
Tables using random distribution are preferred for registering into HAWQ. If hash tables are to be registered, the distribution policy in the YAML file must match that of the table being registered into.
http://git-wip-us.apache.org/repos/asf/incubator-hawq-docs/blob/3fde706c/reference/cli/admin_utilities/hawqregister.html.md.erb
----------------------------------------------------------------------
diff --git a/reference/cli/admin_utilities/hawqregister.html.md.erb b/reference/cli/admin_utilities/hawqregister.html.md.erb
index bd2907e..2f1259b 100644
--- a/reference/cli/admin_utilities/hawqregister.html.md.erb
+++ b/reference/cli/admin_utilities/hawqregister.html.md.erb
@@ -22,7 +22,8 @@ Connection Options:
Misc. Options:
[-f <filepath>]
[-e <eof>]
- [--force]
+ [--force]
+ [--repair]
[-c <yml_config>]
hawq register help | -?
hawq register --version
@@ -55,8 +56,8 @@ Two usage models are available.
Metadata for the Parquet file(s) and the destination table must be consistent. Different data types are used by HAWQ tables and Parquet files, so the data is mapped. Refer to the section [Data Type Mapping](hawqregister.html#topic1__section7) below. You must verify that the structure of the Parquet files and the HAWQ table are compatible before running `hawq register`.
####Limitations
-Only HAWQ or Hive-generated Parquet tables are supported.
-Hash tables and artitioned tables are not supported in this use model.
+Only HAWQ or Hive-generated Parquet tables are supported. Partitioned tables are supported, but only single-level partitioned tables can be registered.
+Hash tables are not supported in this use model.
###Usage Model 2: Use information from a YAML configuration file to register data
@@ -69,6 +70,7 @@ The register process behaves differently, according to different conditions.
- Existing tables have files appended to the existing HAWQ table.
- If a table does not exist, it is created and registered into HAWQ.
- If the -\\\-force option is used, the data in existing catalog tables is erased and re-registered.
+- If the -\\\-repair option is used, data is rolled back to the checkpoint configuration in the YAML file. Files generated after the timestamp of the checkpoint will be deleted. Hash table redistribution, table truncate, and table drop are not supported.
###Limitations for Registering Hive Tables to HAWQ
The currently-supported data types for generating Hive tables into HAWQ tables are: boolean, int, smallint, tinyint, bigint, float, double, string, binary, char, and varchar.
@@ -208,6 +210,9 @@ The following options are used with specific use models.
<dt>-F , -\\\-force</dt>
<dd>Used for disaster recovery of a cluster. Clears all HDFS-related catalog contents in `pg_aoseg.pg_paqseg_$relid `and re-registers files to a specified table. The HDFS files are not removed or modified. To use this option for recovery, data is assumed to be periodically imported to the cluster to be recovered. Used with Usage Model 2.</dd>
+<dt>-R , -\\\-repair \<yml_config\> </dt>
+<dd>Rolls data back to a specified checkpoint. If the table already exists, both the file folder and `pg_aoseg.pg_paqseg_$relid` catalog entry are rolled back to the checkpoint configuration in the YAML file. Deletes any files generated after the timestamp of the checkpoint. Used with Usage Model 2.</dd>
+
<dt>-c , -\\\-config \<yml_config\> </dt>
<dd>Registers files specified by YAML-format configuration files into HAWQ. Used with Usage Model 2.</dd>
[4/9] incubator-hawq-docs git commit: Schema and example updates
Posted by yo...@apache.org.
Schema and example updates
Project: http://git-wip-us.apache.org/repos/asf/incubator-hawq-docs/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-hawq-docs/commit/dbd8ecbd
Tree: http://git-wip-us.apache.org/repos/asf/incubator-hawq-docs/tree/dbd8ecbd
Diff: http://git-wip-us.apache.org/repos/asf/incubator-hawq-docs/diff/dbd8ecbd
Branch: refs/heads/develop
Commit: dbd8ecbdb1f07b202102d5f947e33f36797a24aa
Parents: 64586f7
Author: Jane Beckman <jb...@pivotal.io>
Authored: Wed Oct 12 15:31:02 2016 -0700
Committer: Jane Beckman <jb...@pivotal.io>
Committed: Wed Oct 12 15:31:02 2016 -0700
----------------------------------------------------------------------
.../cli/admin_utilities/hawqextract.html.md.erb | 30 +++++++++++++++++++-
1 file changed, 29 insertions(+), 1 deletion(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-hawq-docs/blob/dbd8ecbd/reference/cli/admin_utilities/hawqextract.html.md.erb
----------------------------------------------------------------------
diff --git a/reference/cli/admin_utilities/hawqextract.html.md.erb b/reference/cli/admin_utilities/hawqextract.html.md.erb
index 5b10818..3e1b597 100644
--- a/reference/cli/admin_utilities/hawqextract.html.md.erb
+++ b/reference/cli/admin_utilities/hawqextract.html.md.erb
@@ -87,6 +87,8 @@ Encoding: UTF8
AO_Schema:
- name: string
type: string
+ Bucketnum: 6
+ Distribution_policy: DISTRIBUTED RANDOMLY
AO_FileLocations:
Blocksize: int
@@ -109,7 +111,10 @@ AO_FileLocations:
- path: string
size: long
-�
+Parquet_Schema:
+ - name: string
+ type: string
+ �
Parquet_FileLocations:
RowGroupSize: long
PageSize: long
@@ -202,7 +207,9 @@ AO_FileLocations:
type: bpchar
- name: count
type: int4
+ Bucketnum: 6
DFS_URL: hdfs://127.0.0.1:9000
+ Distribution_policy: DISTRIBUTED RANDOMLY
Encoding: UTF8
FileFormat: AO
TableName: public.rank
@@ -284,6 +291,27 @@ Parquet_FileLocations:
PageSize: 1048576
RowGroupSize: 8388608
RowGroupSize: 8388608
+ Parquet_Schema:
+ - name: o_orderkey
+ type: int8
+ - name: o_custkey
+ type: int4
+ - name: o_orderstatus
+ type: bpchar
+ - name: o_totalprice
+ type: numeric
+ - name: o_orderdate
+ type: date
+ - name: o_orderpriority
+ type: bpchar
+ - name: o_clerk
+ type: bpchar
+ - name: o_shippriority
+ type: int4
+ - name: o_comment
+ type: varchar
+ Bucketnum: 6
+ Distribution_policy: DISTRIBUTED RANDOMLY
```
## See Also
[3/9] incubator-hawq-docs git commit: Reorganize info
Posted by yo...@apache.org.
Reorganize info
Project: http://git-wip-us.apache.org/repos/asf/incubator-hawq-docs/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-hawq-docs/commit/64586f73
Tree: http://git-wip-us.apache.org/repos/asf/incubator-hawq-docs/tree/64586f73
Diff: http://git-wip-us.apache.org/repos/asf/incubator-hawq-docs/diff/64586f73
Branch: refs/heads/develop
Commit: 64586f7320f6161e9fd24e2b8c8daee369daae23
Parents: 3fde706
Author: Jane Beckman <jb...@pivotal.io>
Authored: Mon Oct 10 15:13:33 2016 -0700
Committer: Jane Beckman <jb...@pivotal.io>
Committed: Mon Oct 10 15:13:33 2016 -0700
----------------------------------------------------------------------
datamgmt/load/g-register_files.html.md.erb | 21 ++++++++++++---------
1 file changed, 12 insertions(+), 9 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-hawq-docs/blob/64586f73/datamgmt/load/g-register_files.html.md.erb
----------------------------------------------------------------------
diff --git a/datamgmt/load/g-register_files.html.md.erb b/datamgmt/load/g-register_files.html.md.erb
index 1d95492..dc2c8e1 100644
--- a/datamgmt/load/g-register_files.html.md.erb
+++ b/datamgmt/load/g-register_files.html.md.erb
@@ -24,13 +24,12 @@ Files or folders in HDFS can be registered into an existing table, allowing them
Only HAWQ or Hive-generated Parquet tables are supported. Only single-level partitioned tables are supported; registering partitioned tables with more than one level will result in an error.
-Metadata for the Parquet file(s) and the destination table must be consistent. Different data types are used by HAWQ tables and Parquet files, so data must be mapped. You must verify that the structure of the parquet files and the HAWQ table are compatible before running `hawq register`.
+Metadata for the Parquet file(s) and the destination table must be consistent. Different data types are used by HAWQ tables and Parquet files, so data must be mapped. You must verify that the structure of the Parquet files and the HAWQ table are compatible before running `hawq register`. Not all HIVE data types can be mapped to HAWQ equivalents. The currently-supported HIVE data types are: boolean, int, smallint, tinyint, bigint, float, double, string, binary, char, and varchar.
As a best practice, create a copy of the Parquet file to be registered before running ```hawq register```
You can then then run ```hawq register``` on the copy, leaving the original file available for additional Hive queries or if a data mapping error is encountered.
-###Limitations for Registering Hive Tables to HAWQ
-The currently-supported data types for generating Hive tables into HAWQ tables are: boolean, int, smallint, tinyint, bigint, float, double, string, binary, char, and varchar.
+###Limitations for Registering Hive Tables to HAWQ
The following HIVE data types cannot be converted to HAWQ equivalents: timestamp, decimal, array, struct, map, and union.
@@ -40,26 +39,30 @@ This example shows how to register a HIVE-generated parquet file in HDFS into th
In this example, the location of the database is `hdfs://localhost:8020/hawq_default`, the tablespace id is 16385, the database id is 16387, the table filenode id is 77160, and the last file under the filenode is numbered 7.
-Enter:
+Run the `hawq register` command for the file location `hdfs://localhost:8020/temp/hive.paq`:
``` pre
$ hawq register -d postgres -f hdfs://localhost:8020/temp/hive.paq parquet_table
```
-After running the `hawq register` command for the file location `hdfs://localhost:8020/temp/hive.paq`, the corresponding new location of the file in HDFS is: `hdfs://localhost:8020/hawq_default/16385/16387/77160/8`.
+After running the `hawq register` command, the corresponding new location of the file in HDFS is: `hdfs://localhost:8020/hawq_default/16385/16387/77160/8`.
-The command then updates the metadata of the table `parquet_table` in HAWQ, which is contained in the table `pg_aoseg.pg_paqseg_77160`. The pg\_aoseg table is a fixed schema for row-oriented and Parquet AO tables. For row-oriented tables, the table name prefix is pg\_aoseg. The table name prefix for parquet tables is pg\_paqseg. 77160 is the relation id of the table.
+The command updates the metadata of the table `parquet_table` in HAWQ, which is contained in the table `pg_aoseg.pg_paqseg_77160`. The pg\_aoseg table is a fixed schema for row-oriented and Parquet AO tables. For row-oriented tables, the table name prefix is pg\_aoseg. For Parquet tables, the table name prefix is pg\_paqseg. 77160 is the relation id of the table.
-To locate the table, either find the relation ID by looking up the catalog table pg\_class in SQL by running
+You can locate the table by one of two methods, either by relation ID or by table name.
+
+To find the relation ID, run the following command on the catalog table pg\_class:
```
select oid from pg_class where relname=$relname
```
-or find the table name by using the SQL command
+To find the table name, run the command:
+
```
select segrelid from pg_appendonly where relid = $relid
```
-then running
+then run:
+
```
select relname from pg_class where oid = segrelid
```
[5/9] incubator-hawq-docs git commit: Remove bucketnum from samples
Posted by yo...@apache.org.
Remove bucketnum from samples
Project: http://git-wip-us.apache.org/repos/asf/incubator-hawq-docs/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-hawq-docs/commit/285c92b3
Tree: http://git-wip-us.apache.org/repos/asf/incubator-hawq-docs/tree/285c92b3
Diff: http://git-wip-us.apache.org/repos/asf/incubator-hawq-docs/diff/285c92b3
Branch: refs/heads/develop
Commit: 285c92b313ff0788e130ac13ea26815b7d9f2ac2
Parents: dbd8ecb
Author: Jane Beckman <jb...@pivotal.io>
Authored: Thu Oct 13 17:22:41 2016 -0700
Committer: Jane Beckman <jb...@pivotal.io>
Committed: Thu Oct 13 17:22:41 2016 -0700
----------------------------------------------------------------------
reference/cli/admin_utilities/hawqextract.html.md.erb | 6 ++----
1 file changed, 2 insertions(+), 4 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-hawq-docs/blob/285c92b3/reference/cli/admin_utilities/hawqextract.html.md.erb
----------------------------------------------------------------------
diff --git a/reference/cli/admin_utilities/hawqextract.html.md.erb b/reference/cli/admin_utilities/hawqextract.html.md.erb
index 3e1b597..b338523 100644
--- a/reference/cli/admin_utilities/hawqextract.html.md.erb
+++ b/reference/cli/admin_utilities/hawqextract.html.md.erb
@@ -98,7 +98,7 @@ AO_FileLocations:
PartitionBy: string ('PARTITION BY ...')
Files:
- path: string (/gpseg0/16385/35469/35470.1)
- size: long
+ size: long
Partitions:
- Blocksize: int
@@ -207,7 +207,6 @@ AO_FileLocations:
type: bpchar
- name: count
type: int4
- Bucketnum: 6
DFS_URL: hdfs://127.0.0.1:9000
Distribution_policy: DISTRIBUTED RANDOMLY
Encoding: UTF8
@@ -310,8 +309,7 @@ Parquet_FileLocations:
type: int4
- name: o_comment
type: varchar
- Bucketnum: 6
- Distribution_policy: DISTRIBUTED RANDOMLY
+ Distribution_policy: DISTRIBUTED RANDOMLY
```
## See Also