You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hawq.apache.org by xunzhang <gi...@git.apache.org> on 2016/09/02 07:33:42 UTC
[GitHub] incubator-hawq pull request #885: HAWQ-1033. Add --force option for hawq reg...
GitHub user xunzhang opened a pull request:
https://github.com/apache/incubator-hawq/pull/885
HAWQ-1033. Add --force option for hawq register.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/xunzhang/incubator-hawq HAWQ-1033
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-hawq/pull/885.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #885
----
commit 5629418b4c171eb0a3acb88179a9a24a74c14d88
Author: xunzhang <xu...@gmail.com>
Date: 2016-09-02T07:33:03Z
HAWQ-1033. Add --force option for hawq register.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hawq issue #885: HAWQ-1033. Add --force option for hawq register.
Posted by xunzhang <gi...@git.apache.org>.
Github user xunzhang commented on the issue:
https://github.com/apache/incubator-hawq/pull/885
cc @ictmalili @radarwave
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hawq pull request #885: HAWQ-1033. Add --force option for hawq reg...
Posted by ictmalili <gi...@git.apache.org>.
Github user ictmalili commented on a diff in the pull request:
https://github.com/apache/incubator-hawq/pull/885#discussion_r77564199
--- Diff: tools/bin/hawqregister ---
@@ -336,8 +338,25 @@ def insert_metadata_into_database(dburl, databasename, tablename, seg_name, firs
conn.commit()
conn.close()
except DatabaseError, ex:
- logger.error('Failed to connect to database, this script can only be run when the database is up')
- move_files_in_hdfs(database, tablename, files, firstsegno, tabledir, False)
+ logger.error('Failed to execute query "%s"' % query)
+ move_files_in_hdfs(databasename, tablename, files, firstsegno, tabledir, False)
+ sys.exit(1)
+
+def update_metadata_into_database(dburl, seg_name, files, eofs):
+ '''Update the catalog table in --force case'''
+ try:
+ query = "set allow_system_table_mods='dml';"
+ query += "begin transaction;"
+ segno_lst = [f.split('/')[-1] for f in files]
+ for i, eof in enumerate(eofs):
+ query += "update pg_aoseg.%s set eof = '%s' where segno = '%s';" % (seg_name, eof, segno_lst[i])
--- End diff --
Why use "update" instead of "delete" and re"insert"?? What if the new files are more than existing files??
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hawq pull request #885: HAWQ-1033. Add --force option for hawq reg...
Posted by xunzhang <gi...@git.apache.org>.
Github user xunzhang commented on a diff in the pull request:
https://github.com/apache/incubator-hawq/pull/885#discussion_r77452500
--- Diff: src/test/feature/ManagementTool/test_hawq_register.cpp ---
@@ -369,3 +369,25 @@ TEST_F(TestHawqRegister, TestUsage2Behavior2) {
util.execute("drop table simple_register_table;");
util.execute("drop table new_simple_register_table;");
}
+
+TEST_F(TestHawqRegister, TestUsage2ForceOption) {
+ SQLUtility util;
+ util.execute("drop table if exists simple_register_table_force;");
+ util.execute("drop table if exists new_simple_register_table_force;");
+ util.execute("create table simple_register_table_force(i int) with (appendonly=true, orientation=row) distributed randomly;");
+ util.execute("insert into simple_register_table_force values(1), (2), (3);");
+ EXPECT_EQ(0, Command::getCommandStatus("hawq extract -d " + (string) HAWQ_DB + " -o tmp.yml testhawqregister_testusage2forceoption.simple_register_table_force"));
--- End diff --
The meaningful name is tmp because of it is just a temporary yaml file without other meaning...
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hawq pull request #885: HAWQ-1033. Add --force option for hawq reg...
Posted by xunzhang <gi...@git.apache.org>.
Github user xunzhang commented on a diff in the pull request:
https://github.com/apache/incubator-hawq/pull/885#discussion_r78490873
--- Diff: src/test/feature/ManagementTool/test_hawq_register_usage1.cpp ---
@@ -0,0 +1,128 @@
+#include <vector>
+#include <string>
+
+#include "gtest/gtest.h"
+#include "lib/command.h"
+#include "lib/sql_util.h"
+#include "lib/string_util.h"
+#include "lib/hdfs_config.h"
+#include "test_hawq_register.h"
+
+using std::vector;
+using std::string;
+using hawq::test::SQLUtility;
+using hawq::test::Command;
+using hawq::test::HdfsConfig;
+
+TEST_F(TestHawqRegister, TestUsage1ExpectSuccess) {
+ // Register file/folder into HAWQ by specific file/folder name
+
+ SQLUtility util;
+ string rootPath(util.getTestRootPath());
+ string filePath = rootPath + "/ManagementTool/data/parquet200/dat.paq";
+ string folderPath = rootPath + "/ManagementTool/data/parquet200sum/";
+
+ vector<string> ddl_orientation_matrix = {"parquet"};
+ vector<string> distribution_policy_matrix = {"", "DISTRIBUTED RANDOMLY"};
+
+ for(int i = 0; i < ddl_orientation_matrix.size() * distribution_policy_matrix.size() * 4; ++i) {
+ util.execute(hawq::test::stringFormat("drop table if exists t_%s;", std::to_string(i).c_str()));
+ }
+ auto register_lambda = [&] () {
+ int suffix = 0;
+ // hawq register -d hawq_feature_test -f hdfs://localhost:8020/usage1dat.paq t_#
+ for(auto & ddl : ddl_orientation_matrix) {
+ for(auto & policy : distribution_policy_matrix) {
+ auto cmd = hawq::test::stringFormat("hdfs dfs -put -f %s %s/usage1dat.paq", filePath.c_str(), getHdfsLocation().c_str());
+ EXPECT_EQ(0, Command::getCommandStatus(cmd));
+
+ auto sql = hawq::test::stringFormat("CREATE TABLE t_%s(i int) with (appendonly=true, orientation=%s) %s;", std::to_string(suffix).c_str(), ddl.c_str(), policy.c_str());
+ util.execute(sql); util.query(hawq::test::stringFormat("SELECT * from t_%s", std::to_string(suffix).c_str()), 0);
+
+ cmd = hawq::test::stringFormat("hawq register -d %s -f %s/usage1dat.paq t_%s", HAWQ_DB, getHdfsLocation().c_str(), std::to_string(suffix).c_str());
+ EXPECT_EQ(0, Command::getCommandStatus(cmd));
+
+ util.query(hawq::test::stringFormat("select * from t_%s;", std::to_string(suffix).c_str()), 200);
+ util.execute(hawq::test::stringFormat("insert into t_%s values(201);", std::to_string(suffix).c_str()));
+ util.query(hawq::test::stringFormat("select * from t_%s;", std::to_string(suffix).c_str()), 201);
+
+ suffix ++;
+ }
+ }
+
+ // hawq register -d hawq_feature_test -f hdfs://localhost:8020/usage1dat.paq -e eof t_#
+ for(auto & ddl : ddl_orientation_matrix) {
+ for(auto & policy : distribution_policy_matrix) {
+ auto cmd = hawq::test::stringFormat("hdfs dfs -put -f %s %s/usage1dat.paq", filePath.c_str(), getHdfsLocation().c_str());
+ EXPECT_EQ(0, Command::getCommandStatus(cmd));
+
+ auto sql = hawq::test::stringFormat("CREATE TABLE t_%s(i int) with (appendonly=true, orientation=%s) %s;", std::to_string(suffix).c_str(), ddl.c_str(), policy.c_str());
+ util.execute(sql); util.query(hawq::test::stringFormat("SELECT * from t_%s", std::to_string(suffix).c_str()), 0);
+
+ cmd = hawq::test::stringFormat("hawq register -d %s -f %s/usage1dat.paq -e 596 t_%s", HAWQ_DB, getHdfsLocation().c_str(), std::to_string(suffix).c_str());
+ EXPECT_EQ(0, Command::getCommandStatus(cmd));
+
+ util.query(hawq::test::stringFormat("select * from t_%s;", std::to_string(suffix).c_str()), 100);
+ util.execute(hawq::test::stringFormat("insert into t_%s values(101);", std::to_string(suffix).c_str()));
+ util.query(hawq::test::stringFormat("select * from t_%s;", std::to_string(suffix).c_str()), 101);
+
+ suffix ++;
+ }
+ }
+
+ // hawq register -d hawq_feature_test -f hdfs://localhost:8020/usage1tmp/ t_#
+ for(auto & ddl : ddl_orientation_matrix) {
+ for(auto & policy : distribution_policy_matrix) {
+ auto cmd = hawq::test::stringFormat("hdfs dfs -mkdir -p %s/usage1tmp/", getHdfsLocation().c_str());
+ EXPECT_EQ(0, Command::getCommandStatus(cmd));
+ cmd = hawq::test::stringFormat("hdfs dfs -put -f %s/*.paq %s/usage1tmp/", folderPath.c_str(), getHdfsLocation().c_str());
+ EXPECT_EQ(0, Command::getCommandStatus(cmd));
+
+ auto sql = hawq::test::stringFormat("CREATE TABLE t_%s(i int) with (appendonly=true, orientation=%s) %s;", std::to_string(suffix).c_str(), ddl.c_str(), policy.c_str());
+ util.execute(sql); util.query(hawq::test::stringFormat("SELECT * from t_%s", std::to_string(suffix).c_str()), 0);
+
+ cmd = hawq::test::stringFormat("hawq register -d %s -f %s/usage1tmp/ t_%s", HAWQ_DB, getHdfsLocation().c_str(), std::to_string(suffix).c_str());
+ EXPECT_EQ(0, Command::getCommandStatus(cmd));
+
+ util.query(hawq::test::stringFormat("select * from t_%s;", std::to_string(suffix).c_str()), 200);
+ util.execute(hawq::test::stringFormat("insert into t_%s values(201);", std::to_string(suffix).c_str()));
+ util.query(hawq::test::stringFormat("select * from t_%s;", std::to_string(suffix).c_str()), 201);
+
+ suffix ++;
+ }
+ }
+
+ // hawq register -d hawq_feature_test -f hdfs://localhost:8020/usage1tmp t_#
--- End diff --
sure, thanks.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hawq pull request #885: HAWQ-1033. Add --force option for hawq reg...
Posted by ictmalili <gi...@git.apache.org>.
Github user ictmalili commented on a diff in the pull request:
https://github.com/apache/incubator-hawq/pull/885#discussion_r77564020
--- Diff: tools/bin/hawqregister ---
@@ -388,11 +415,31 @@ if __name__ == '__main__':
logger.error('Files to be registered in this case should not be the same with table path.')
sys.exit(1)
+ do_not_move, files_update, sizes_update = False, [], []
+ if force_mode:
+ existed_files = get_files_in_hdfs(filepath)
+ if len(files) == len(exited_files):
--- End diff --
Is this a spelling error? Should be existed_files?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hawq pull request #885: HAWQ-1033. Add --force option for hawq reg...
Posted by xunzhang <gi...@git.apache.org>.
Github user xunzhang closed the pull request at:
https://github.com/apache/incubator-hawq/pull/885
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hawq pull request #885: HAWQ-1033. Add --force option for hawq reg...
Posted by paul-guo- <gi...@git.apache.org>.
Github user paul-guo- commented on a diff in the pull request:
https://github.com/apache/incubator-hawq/pull/885#discussion_r77451727
--- Diff: src/test/feature/ManagementTool/test_hawq_register.cpp ---
@@ -369,3 +369,25 @@ TEST_F(TestHawqRegister, TestUsage2Behavior2) {
util.execute("drop table simple_register_table;");
util.execute("drop table new_simple_register_table;");
}
+
+TEST_F(TestHawqRegister, TestUsage2ForceOption) {
+ SQLUtility util;
+ util.execute("drop table if exists simple_register_table_force;");
+ util.execute("drop table if exists new_simple_register_table_force;");
+ util.execute("create table simple_register_table_force(i int) with (appendonly=true, orientation=row) distributed randomly;");
+ util.execute("insert into simple_register_table_force values(1), (2), (3);");
+ EXPECT_EQ(0, Command::getCommandStatus("hawq extract -d " + (string) HAWQ_DB + " -o tmp.yml testhawqregister_testusage2forceoption.simple_register_table_force"));
+ EXPECT_EQ(0, Command::getCommandStatus("hawq register -d " + (string) HAWQ_DB + " -c tmp.yml testhawqregister_testusage2forceoption.new_simple_register_table_force"));
+ util.query("select * from new_simple_register_table_force;", 3);
+
+ util.execute("drop table if exists simple_register_table_force;");
+ util.execute("create table simple_register_table_force(i int) with (appendonly=true, orientation=row) distributed randomly;");
+ util.execute("insert into simple_register_table_force values(4), (5), (6), (7);");
+ EXPECT_EQ(0, Command::getCommandStatus("hawq extract -d " + (string) HAWQ_DB + " -o tmp.yml testhawqregister_testusage2forceoption.simple_register_table_force"));
+ EXPECT_EQ(0, Command::getCommandStatus("hawq register -d " + (string) HAWQ_DB + " -c tmp.yml --force testhawqregister_testusage2forceoption.new_simple_register_table_force"));
+ util.query("select * from new_simple_register_table_force;", 4);
+
+ EXPECT_EQ(0, Command::getCommandStatus("rm -rf tmp.yml"));
--- End diff --
This is just for cleanup, maybe EXPECT_EQ is not required?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hawq issue #885: HAWQ-1033. Add --force option for hawq register.
Posted by xunzhang <gi...@git.apache.org>.
Github user xunzhang commented on the issue:
https://github.com/apache/incubator-hawq/pull/885
Merged into master.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hawq issue #885: HAWQ-1033. Add --force option for hawq register.
Posted by wcl14 <gi...@git.apache.org>.
Github user wcl14 commented on the issue:
https://github.com/apache/incubator-hawq/pull/885
+1
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hawq pull request #885: HAWQ-1033. Add --force option for hawq reg...
Posted by paul-guo- <gi...@git.apache.org>.
Github user paul-guo- commented on a diff in the pull request:
https://github.com/apache/incubator-hawq/pull/885#discussion_r77451823
--- Diff: src/test/feature/ManagementTool/test_hawq_register.cpp ---
@@ -369,3 +369,25 @@ TEST_F(TestHawqRegister, TestUsage2Behavior2) {
util.execute("drop table simple_register_table;");
util.execute("drop table new_simple_register_table;");
}
+
+TEST_F(TestHawqRegister, TestUsage2ForceOption) {
+ SQLUtility util;
+ util.execute("drop table if exists simple_register_table_force;");
+ util.execute("drop table if exists new_simple_register_table_force;");
+ util.execute("create table simple_register_table_force(i int) with (appendonly=true, orientation=row) distributed randomly;");
+ util.execute("insert into simple_register_table_force values(1), (2), (3);");
+ EXPECT_EQ(0, Command::getCommandStatus("hawq extract -d " + (string) HAWQ_DB + " -o tmp.yml testhawqregister_testusage2forceoption.simple_register_table_force"));
--- End diff --
I'd suggest using a meaningful temp file name instead of tmp.yml.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hawq pull request #885: HAWQ-1033. Add --force option for hawq reg...
Posted by wcl14 <gi...@git.apache.org>.
Github user wcl14 commented on a diff in the pull request:
https://github.com/apache/incubator-hawq/pull/885#discussion_r78490699
--- Diff: src/test/feature/ManagementTool/test_hawq_register_usage1.cpp ---
@@ -0,0 +1,128 @@
+#include <vector>
+#include <string>
+
+#include "gtest/gtest.h"
+#include "lib/command.h"
+#include "lib/sql_util.h"
+#include "lib/string_util.h"
+#include "lib/hdfs_config.h"
+#include "test_hawq_register.h"
+
+using std::vector;
+using std::string;
+using hawq::test::SQLUtility;
+using hawq::test::Command;
+using hawq::test::HdfsConfig;
+
+TEST_F(TestHawqRegister, TestUsage1ExpectSuccess) {
+ // Register file/folder into HAWQ by specific file/folder name
+
+ SQLUtility util;
+ string rootPath(util.getTestRootPath());
+ string filePath = rootPath + "/ManagementTool/data/parquet200/dat.paq";
+ string folderPath = rootPath + "/ManagementTool/data/parquet200sum/";
+
+ vector<string> ddl_orientation_matrix = {"parquet"};
+ vector<string> distribution_policy_matrix = {"", "DISTRIBUTED RANDOMLY"};
+
+ for(int i = 0; i < ddl_orientation_matrix.size() * distribution_policy_matrix.size() * 4; ++i) {
+ util.execute(hawq::test::stringFormat("drop table if exists t_%s;", std::to_string(i).c_str()));
+ }
+ auto register_lambda = [&] () {
+ int suffix = 0;
+ // hawq register -d hawq_feature_test -f hdfs://localhost:8020/usage1dat.paq t_#
+ for(auto & ddl : ddl_orientation_matrix) {
+ for(auto & policy : distribution_policy_matrix) {
+ auto cmd = hawq::test::stringFormat("hdfs dfs -put -f %s %s/usage1dat.paq", filePath.c_str(), getHdfsLocation().c_str());
+ EXPECT_EQ(0, Command::getCommandStatus(cmd));
+
+ auto sql = hawq::test::stringFormat("CREATE TABLE t_%s(i int) with (appendonly=true, orientation=%s) %s;", std::to_string(suffix).c_str(), ddl.c_str(), policy.c_str());
+ util.execute(sql); util.query(hawq::test::stringFormat("SELECT * from t_%s", std::to_string(suffix).c_str()), 0);
+
+ cmd = hawq::test::stringFormat("hawq register -d %s -f %s/usage1dat.paq t_%s", HAWQ_DB, getHdfsLocation().c_str(), std::to_string(suffix).c_str());
+ EXPECT_EQ(0, Command::getCommandStatus(cmd));
+
+ util.query(hawq::test::stringFormat("select * from t_%s;", std::to_string(suffix).c_str()), 200);
+ util.execute(hawq::test::stringFormat("insert into t_%s values(201);", std::to_string(suffix).c_str()));
+ util.query(hawq::test::stringFormat("select * from t_%s;", std::to_string(suffix).c_str()), 201);
+
+ suffix ++;
+ }
+ }
+
+ // hawq register -d hawq_feature_test -f hdfs://localhost:8020/usage1dat.paq -e eof t_#
+ for(auto & ddl : ddl_orientation_matrix) {
+ for(auto & policy : distribution_policy_matrix) {
+ auto cmd = hawq::test::stringFormat("hdfs dfs -put -f %s %s/usage1dat.paq", filePath.c_str(), getHdfsLocation().c_str());
+ EXPECT_EQ(0, Command::getCommandStatus(cmd));
+
+ auto sql = hawq::test::stringFormat("CREATE TABLE t_%s(i int) with (appendonly=true, orientation=%s) %s;", std::to_string(suffix).c_str(), ddl.c_str(), policy.c_str());
+ util.execute(sql); util.query(hawq::test::stringFormat("SELECT * from t_%s", std::to_string(suffix).c_str()), 0);
+
+ cmd = hawq::test::stringFormat("hawq register -d %s -f %s/usage1dat.paq -e 596 t_%s", HAWQ_DB, getHdfsLocation().c_str(), std::to_string(suffix).c_str());
+ EXPECT_EQ(0, Command::getCommandStatus(cmd));
+
+ util.query(hawq::test::stringFormat("select * from t_%s;", std::to_string(suffix).c_str()), 100);
+ util.execute(hawq::test::stringFormat("insert into t_%s values(101);", std::to_string(suffix).c_str()));
+ util.query(hawq::test::stringFormat("select * from t_%s;", std::to_string(suffix).c_str()), 101);
+
+ suffix ++;
+ }
+ }
+
+ // hawq register -d hawq_feature_test -f hdfs://localhost:8020/usage1tmp/ t_#
+ for(auto & ddl : ddl_orientation_matrix) {
+ for(auto & policy : distribution_policy_matrix) {
+ auto cmd = hawq::test::stringFormat("hdfs dfs -mkdir -p %s/usage1tmp/", getHdfsLocation().c_str());
+ EXPECT_EQ(0, Command::getCommandStatus(cmd));
+ cmd = hawq::test::stringFormat("hdfs dfs -put -f %s/*.paq %s/usage1tmp/", folderPath.c_str(), getHdfsLocation().c_str());
+ EXPECT_EQ(0, Command::getCommandStatus(cmd));
+
+ auto sql = hawq::test::stringFormat("CREATE TABLE t_%s(i int) with (appendonly=true, orientation=%s) %s;", std::to_string(suffix).c_str(), ddl.c_str(), policy.c_str());
+ util.execute(sql); util.query(hawq::test::stringFormat("SELECT * from t_%s", std::to_string(suffix).c_str()), 0);
+
+ cmd = hawq::test::stringFormat("hawq register -d %s -f %s/usage1tmp/ t_%s", HAWQ_DB, getHdfsLocation().c_str(), std::to_string(suffix).c_str());
+ EXPECT_EQ(0, Command::getCommandStatus(cmd));
+
+ util.query(hawq::test::stringFormat("select * from t_%s;", std::to_string(suffix).c_str()), 200);
+ util.execute(hawq::test::stringFormat("insert into t_%s values(201);", std::to_string(suffix).c_str()));
+ util.query(hawq::test::stringFormat("select * from t_%s;", std::to_string(suffix).c_str()), 201);
+
+ suffix ++;
+ }
+ }
+
+ // hawq register -d hawq_feature_test -f hdfs://localhost:8020/usage1tmp t_#
--- End diff --
These four cases can be concluded to one.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hawq issue #885: HAWQ-1033. Add --force option for hawq register.
Posted by ictmalili <gi...@git.apache.org>.
Github user ictmalili commented on the issue:
https://github.com/apache/incubator-hawq/pull/885
LGTM. +1
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hawq pull request #885: HAWQ-1033. Add --force option for hawq reg...
Posted by ictmalili <gi...@git.apache.org>.
Github user ictmalili commented on a diff in the pull request:
https://github.com/apache/incubator-hawq/pull/885#discussion_r77564087
--- Diff: src/test/feature/ManagementTool/test_hawq_register.cpp ---
@@ -369,3 +369,25 @@ TEST_F(TestHawqRegister, TestUsage2Behavior2) {
util.execute("drop table simple_register_table;");
util.execute("drop table new_simple_register_table;");
}
+
+TEST_F(TestHawqRegister, TestUsage2ForceOption) {
+ SQLUtility util;
+ util.execute("drop table if exists simple_register_table_force;");
+ util.execute("drop table if exists new_simple_register_table_force;");
+ util.execute("create table simple_register_table_force(i int) with (appendonly=true, orientation=row) distributed randomly;");
+ util.execute("insert into simple_register_table_force values(1), (2), (3);");
+ EXPECT_EQ(0, Command::getCommandStatus("hawq extract -d " + (string) HAWQ_DB + " -o tmp.yml testhawqregister_testusage2forceoption.simple_register_table_force"));
+ EXPECT_EQ(0, Command::getCommandStatus("hawq register -d " + (string) HAWQ_DB + " -c tmp.yml testhawqregister_testusage2forceoption.new_simple_register_table_force"));
+ util.query("select * from new_simple_register_table_force;", 3);
+
+ util.execute("drop table if exists simple_register_table_force;");
+ util.execute("create table simple_register_table_force(i int) with (appendonly=true, orientation=row) distributed randomly;");
+ util.execute("insert into simple_register_table_force values(4), (5), (6), (7);");
+ EXPECT_EQ(0, Command::getCommandStatus("hawq extract -d " + (string) HAWQ_DB + " -o tmp.yml testhawqregister_testusage2forceoption.simple_register_table_force"));
+ EXPECT_EQ(0, Command::getCommandStatus("hawq register -d " + (string) HAWQ_DB + " -c tmp.yml --force testhawqregister_testusage2forceoption.new_simple_register_table_force"));
+ util.query("select * from new_simple_register_table_force;", 4);
--- End diff --
Should here throw error, indicating the existing files for table 'new_simple_register_table_force' are not included in .yml file???
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hawq pull request #885: HAWQ-1033. Add --force option for hawq reg...
Posted by paul-guo- <gi...@git.apache.org>.
Github user paul-guo- commented on a diff in the pull request:
https://github.com/apache/incubator-hawq/pull/885#discussion_r77468589
--- Diff: src/test/feature/ManagementTool/test_hawq_register.cpp ---
@@ -369,3 +369,25 @@ TEST_F(TestHawqRegister, TestUsage2Behavior2) {
util.execute("drop table simple_register_table;");
util.execute("drop table new_simple_register_table;");
}
+
+TEST_F(TestHawqRegister, TestUsage2ForceOption) {
+ SQLUtility util;
+ util.execute("drop table if exists simple_register_table_force;");
+ util.execute("drop table if exists new_simple_register_table_force;");
+ util.execute("create table simple_register_table_force(i int) with (appendonly=true, orientation=row) distributed randomly;");
+ util.execute("insert into simple_register_table_force values(1), (2), (3);");
+ EXPECT_EQ(0, Command::getCommandStatus("hawq extract -d " + (string) HAWQ_DB + " -o tmp.yml testhawqregister_testusage2forceoption.simple_register_table_force"));
--- End diff --
I mean tmp_$testname.yml or similar. This helps users/developers to quickly
know what the file is for when something is wrong. This also avoids temp
file name conflicts when running cases in parallel in the future.
2016-09-04 21:25 GMT+08:00 Hong Wu <no...@github.com>:
> In src/test/feature/ManagementTool/test_hawq_register.cpp
> <https://github.com/apache/incubator-hawq/pull/885#discussion_r77452500>:
>
> > @@ -369,3 +369,25 @@ TEST_F(TestHawqRegister, TestUsage2Behavior2) {
> > util.execute("drop table simple_register_table;");
> > util.execute("drop table new_simple_register_table;");
> > }
> > +
> > +TEST_F(TestHawqRegister, TestUsage2ForceOption) {
> > + SQLUtility util;
> > + util.execute("drop table if exists simple_register_table_force;");
> > + util.execute("drop table if exists new_simple_register_table_force;");
> > + util.execute("create table simple_register_table_force(i int) with (appendonly=true, orientation=row) distributed randomly;");
> > + util.execute("insert into simple_register_table_force values(1), (2), (3);");
> > + EXPECT_EQ(0, Command::getCommandStatus("hawq extract -d " + (string) HAWQ_DB + " -o tmp.yml testhawqregister_testusage2forceoption.simple_register_table_force"));
>
> The meaningful name is tmp because of it is just a temporary yaml file
> without other meaning...
>
> \u2014
> You are receiving this because you commented.
> Reply to this email directly, view it on GitHub
> <https://github.com/apache/incubator-hawq/pull/885/files/b16f10b2b59f7d7f5bc81f789101c95af8716071#r77452500>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/AHI5jLp4m24by1aBj1Po_7MbjvDEc60hks5qmsa6gaJpZM4JzcWK>
> .
>
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hawq pull request #885: HAWQ-1033. Add --force option for hawq reg...
Posted by xunzhang <gi...@git.apache.org>.
Github user xunzhang commented on a diff in the pull request:
https://github.com/apache/incubator-hawq/pull/885#discussion_r77471760
--- Diff: src/test/feature/ManagementTool/test_hawq_register.cpp ---
@@ -369,3 +369,25 @@ TEST_F(TestHawqRegister, TestUsage2Behavior2) {
util.execute("drop table simple_register_table;");
util.execute("drop table new_simple_register_table;");
}
+
+TEST_F(TestHawqRegister, TestUsage2ForceOption) {
+ SQLUtility util;
+ util.execute("drop table if exists simple_register_table_force;");
+ util.execute("drop table if exists new_simple_register_table_force;");
+ util.execute("create table simple_register_table_force(i int) with (appendonly=true, orientation=row) distributed randomly;");
+ util.execute("insert into simple_register_table_force values(1), (2), (3);");
+ EXPECT_EQ(0, Command::getCommandStatus("hawq extract -d " + (string) HAWQ_DB + " -o tmp.yml testhawqregister_testusage2forceoption.simple_register_table_force"));
--- End diff --
ok, I think that make sense.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hawq pull request #885: HAWQ-1033. Add --force option for hawq reg...
Posted by xunzhang <gi...@git.apache.org>.
Github user xunzhang commented on a diff in the pull request:
https://github.com/apache/incubator-hawq/pull/885#discussion_r77471877
--- Diff: src/test/feature/ManagementTool/test_hawq_register.cpp ---
@@ -369,3 +369,25 @@ TEST_F(TestHawqRegister, TestUsage2Behavior2) {
util.execute("drop table simple_register_table;");
util.execute("drop table new_simple_register_table;");
}
+
+TEST_F(TestHawqRegister, TestUsage2ForceOption) {
+ SQLUtility util;
+ util.execute("drop table if exists simple_register_table_force;");
+ util.execute("drop table if exists new_simple_register_table_force;");
+ util.execute("create table simple_register_table_force(i int) with (appendonly=true, orientation=row) distributed randomly;");
+ util.execute("insert into simple_register_table_force values(1), (2), (3);");
+ EXPECT_EQ(0, Command::getCommandStatus("hawq extract -d " + (string) HAWQ_DB + " -o tmp.yml testhawqregister_testusage2forceoption.simple_register_table_force"));
+ EXPECT_EQ(0, Command::getCommandStatus("hawq register -d " + (string) HAWQ_DB + " -c tmp.yml testhawqregister_testusage2forceoption.new_simple_register_table_force"));
+ util.query("select * from new_simple_register_table_force;", 3);
+
+ util.execute("drop table if exists simple_register_table_force;");
+ util.execute("create table simple_register_table_force(i int) with (appendonly=true, orientation=row) distributed randomly;");
+ util.execute("insert into simple_register_table_force values(4), (5), (6), (7);");
+ EXPECT_EQ(0, Command::getCommandStatus("hawq extract -d " + (string) HAWQ_DB + " -o tmp.yml testhawqregister_testusage2forceoption.simple_register_table_force"));
+ EXPECT_EQ(0, Command::getCommandStatus("hawq register -d " + (string) HAWQ_DB + " -c tmp.yml --force testhawqregister_testusage2forceoption.new_simple_register_table_force"));
+ util.query("select * from new_simple_register_table_force;", 4);
+
+ EXPECT_EQ(0, Command::getCommandStatus("rm -rf tmp.yml"));
--- End diff --
The interface us getCommandStatus, not CommandExec.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hawq pull request #885: HAWQ-1033. Add --force option for hawq reg...
Posted by xunzhang <gi...@git.apache.org>.
Github user xunzhang commented on a diff in the pull request:
https://github.com/apache/incubator-hawq/pull/885#discussion_r77575744
--- Diff: tools/bin/hawqregister ---
@@ -336,8 +338,25 @@ def insert_metadata_into_database(dburl, databasename, tablename, seg_name, firs
conn.commit()
conn.close()
except DatabaseError, ex:
- logger.error('Failed to connect to database, this script can only be run when the database is up')
- move_files_in_hdfs(database, tablename, files, firstsegno, tabledir, False)
+ logger.error('Failed to execute query "%s"' % query)
+ move_files_in_hdfs(databasename, tablename, files, firstsegno, tabledir, False)
+ sys.exit(1)
+
+def update_metadata_into_database(dburl, seg_name, files, eofs):
+ '''Update the catalog table in --force case'''
+ try:
+ query = "set allow_system_table_mods='dml';"
+ query += "begin transaction;"
+ segno_lst = [f.split('/')[-1] for f in files]
+ for i, eof in enumerate(eofs):
+ query += "update pg_aoseg.%s set eof = '%s' where segno = '%s';" % (seg_name, eof, segno_lst[i])
--- End diff --
To unify all use cases.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hawq issue #885: HAWQ-1033. Add --force option for hawq register.
Posted by xunzhang <gi...@git.apache.org>.
Github user xunzhang commented on the issue:
https://github.com/apache/incubator-hawq/pull/885
cc @wcl14 @ictmalili plz review again
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---