You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cloudberry.apache.org by yj...@apache.org on 2025/02/19 02:57:51 UTC
(cloudberry) branch main updated (63a4e168a6 -> 75928a683a)
This is an automated email from the ASF dual-hosted git repository.
yjhjstz pushed a change to branch main
in repository https://gitbox.apache.org/repos/asf/cloudberry.git
from 63a4e168a6 Fix build and icw tests "CLogical[Dynamic]IndexOnlyGet, Hash subplans, Update Index Scan Costing..."
new 71abf7a0ab Fix concurrent update core of partition table in Dynamic scan (#15253)
new e19c85c144 Add error/warning when exchanging/attach an external table
new abce4a7298 Test fix failure by adding timeeout and fixing test steps. (#14589)
new a04da887fe bugfix: Set the search_path correctly when CREATE EXTENSION WITH SCHEMA. (#13703)
new 74b7051d23 Behave test cases for checking the use of TRUSTED_SHELL in gpinitsystem (#15382)
new 73c2fbd7d0 Remove psutil usage from regress tests (#15711)
new 460c66361f Fix ANALYZE bug in expand_vacuum_rels
new 75928a683a Fix singlenode partition test.
The 8 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
gpMgmt/test/behave/mgmt_utils/gpinitsystem.feature | 17 ++++++++
.../test/behave/mgmt_utils/gpmovemirrors.feature | 1 +
gpMgmt/test/behave/mgmt_utils/gprecoverseg.feature | 6 +++
gpMgmt/test/behave/mgmt_utils/steps/mgmt_utils.py | 2 +-
gpMgmt/test/behave_utils/utils.py | 2 +-
src/backend/access/external/external.c | 2 +
src/backend/commands/tablecmds.c | 19 +++++++++
src/backend/commands/vacuum.c | 2 +-
src/backend/executor/execPartition.c | 13 +++++-
.../isolation2/expected/gdd/concurrent_update.out | 30 ++++++++++++++
.../expected/gdd/concurrent_update_optimizer.out | 30 ++++++++++++++
src/test/isolation2/sql/gdd/concurrent_update.sql | 18 ++++++++
.../regress/expected/create_extension_fail.out | 8 ++++
src/test/regress/expected/motion_socket.out | 41 ++++++++++--------
src/test/regress/expected/offload_entry_to_qe.out | 2 +
src/test/regress/expected/partindex_test.out | 1 +
src/test/regress/expected/partition.out | 1 +
src/test/regress/expected/partition_optimizer.out | 1 +
src/test/regress/expected/qp_orca_fallback.out | 1 +
.../expected/qp_orca_fallback_optimizer.out | 1 +
src/test/regress/input/external_table.source | 12 +++---
src/test/regress/input/part_external_table.source | 48 ++++++++++++++++++++++
src/test/regress/output/external_table.source | 14 ++++---
.../regress/output/external_table_optimizer.source | 14 ++++---
src/test/regress/output/gpcopy.source | 1 +
src/test/regress/output/part_external_table.source | 43 +++++++++++++++++++
.../output/part_external_table_optimizer.source | 43 +++++++++++++++++++
src/test/regress/sql/create_extension_fail.sql | 9 ++++
src/test/regress/sql/motion_socket.sql | 37 +++++++++++------
src/test/singlenode_regress/expected/partition.out | 1 +
30 files changed, 369 insertions(+), 51 deletions(-)
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For additional commands, e-mail: commits-help@cloudberry.apache.org
(cloudberry) 07/08: Fix ANALYZE bug in expand_vacuum_rels
Posted by yj...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
yjhjstz pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/cloudberry.git
commit 460c66361fb83c0ba77f0cf6a7313c26622e8113
Author: Brent Doil <bd...@vmware.com>
AuthorDate: Tue Aug 22 11:24:02 2023 -0400
Fix ANALYZE bug in expand_vacuum_rels
With optimizer_analyze_root_partition enabled, VACUUM commands with a relation list
would check leaf partition stats even when `VACUUM (ANALYZE)` was not specified.
Instead, check for `VACOPT_ANALYZE` before updating parent partition stats
in `expand_vacuum_rel`. There is no reason to check `leaf_parts_analyzed`
or add to the vacuum list if `ANALYZE` wasn't specified.
The existing comment explains the intended behavior:
```
* GPDB: If you explicitly ANALYZE a partition, also update the
* parent's stats after the partition has been ANALYZEd. (Thanks to
* the code to merge leaf statistics, it should be fast.)
```
The fix results in a large performance improvement when issuing `VACUUM`
with a relation list and `optimizer_analyze_root_partition` enabled.
Authored-by: Brent Doil <bd...@vmware.com>
---
src/backend/commands/vacuum.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/src/backend/commands/vacuum.c b/src/backend/commands/vacuum.c
index 5c3f7249e5..d134d9036a 100644
--- a/src/backend/commands/vacuum.c
+++ b/src/backend/commands/vacuum.c
@@ -1087,7 +1087,7 @@ expand_vacuum_rel(VacuumRelation *vrel, int options)
* If current table is skipped, no need to merge stats for it's parent
* since current table's stats is not get updated.
*/
- if (optimizer_analyze_root_partition && !skip_this)
+ if ((options & VACOPT_ANALYZE) && optimizer_analyze_root_partition && !skip_this)
{
Oid child_relid = relid;
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For additional commands, e-mail: commits-help@cloudberry.apache.org
(cloudberry) 04/08: bugfix: Set the search_path correctly when CREATE EXTENSION WITH SCHEMA. (#13703)
Posted by yj...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
yjhjstz pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/cloudberry.git
commit a04da887fe8d6919a2a2671fcfd7000bdda0a57e
Author: QingMa <ma...@vmware.com>
AuthorDate: Fri Jun 24 09:53:58 2022 +0800
bugfix: Set the search_path correctly when CREATE EXTENSION WITH SCHEMA. (#13703)
Combine commit e76a8937a4 Partially revert of fixes for greenplum-db#6716.
This commit fixes https://github.com/greenplum-db/gpdb/issues/6716.
------------------------------------------------------------------
In postgres, CREATE EXTENSION WITH SCHEMA will set up the `search_path`
to contain the target schema and required schemas before execute extension
script. Because the commands of the extension script will be dispatched to
QE, GPDB needs to set up the `search_path` on both QD and QE.
NOTE: `search_path` will be reset during transaction commit.
---
src/test/regress/expected/create_extension_fail.out | 8 ++++++++
src/test/regress/sql/create_extension_fail.sql | 9 +++++++++
2 files changed, 17 insertions(+)
diff --git a/src/test/regress/expected/create_extension_fail.out b/src/test/regress/expected/create_extension_fail.out
index a4c313f8ab..87b0aecc2d 100644
--- a/src/test/regress/expected/create_extension_fail.out
+++ b/src/test/regress/expected/create_extension_fail.out
@@ -59,3 +59,11 @@ HINT: The 'DISTRIBUTED BY' clause determines the distribution of data. Make sur
\c
drop table t_12713;
drop role user_12713;
+--
+-- Another Test from Issue: https://github.com/greenplum-db/gpdb/issues/6716
+--
+drop extension if exists gp_inject_fault;
+create schema issue6716;
+create extension gp_inject_fault with schema issue6716;
+drop extension gp_inject_fault;
+drop schema issue6716;
diff --git a/src/test/regress/sql/create_extension_fail.sql b/src/test/regress/sql/create_extension_fail.sql
index 3a65fcddab..1efb4f737c 100644
--- a/src/test/regress/sql/create_extension_fail.sql
+++ b/src/test/regress/sql/create_extension_fail.sql
@@ -46,3 +46,12 @@ create table t_12713(a int);
\c
drop table t_12713;
drop role user_12713;
+
+--
+-- Another Test from Issue: https://github.com/greenplum-db/gpdb/issues/6716
+--
+drop extension if exists gp_inject_fault;
+create schema issue6716;
+create extension gp_inject_fault with schema issue6716;
+drop extension gp_inject_fault;
+drop schema issue6716;
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For additional commands, e-mail: commits-help@cloudberry.apache.org
(cloudberry) 02/08: Add error/warning when exchanging/attach an external table
Posted by yj...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
yjhjstz pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/cloudberry.git
commit e19c85c144b1244b34a4fa20406559934f4db37e
Author: Huansong Fu <fu...@gmail.com>
AuthorDate: Thu Jul 27 07:23:08 2023 -0700
Add error/warning when exchanging/attach an external table
In 7X, external tables are essentially just one kind of foreign tables. Also,
because EXCHANGE PARTITION is now just DETACH+ATTACH PARTITION, exchanging
partition with an external table has the same behavior as attaching a foreign
table in upstream.
Because of that, there were certain restrictions that got lost for exchanging
external tables: (1) we cannot exchange a writable external table in; (2) we
cannot exchange a readable external table in w/o "WITHOUT VALIDATION" clause.
For (1): we should bring that restriction back, because otherwise there would
always be an error when scanning that partition (unless the query is written in
a way that avoids scanning that particular partition). To disallow having a
write-only external partition makes more sense. And, keeping existing behavior
is probably preferred for those who still use external tables than foreign tables.
For (2): in 7X, WITH/WITHOUT VALIDATION becomes a no-op[1]: we will always
validate the new partition *if possible*, even user specifies WITHOUT VALIDATION.
In case it is not possible such as for foreign tables, we will ignore validation
which is the same behavior as ATTACH PARTITION in upstream [2]. Therefore, it
doesn't make sense to continue asking users to add a no-op clause anymore. The
ask for WITHOUT VALIDATION is essentially to raise user awareness about this
issue. Therefore, we can add WARNING to server the same purpose.
Adjusted a few tests accordingly.
[1] https://docs.vmware.com/en/VMware-Greenplum/7/greenplum-database/ref_guide-sql_commands-ALTER_TABLE.html
[2] https://www.postgresql.org/docs/12/sql-createforeigntable.html
---
src/backend/access/external/external.c | 2 +
src/backend/commands/tablecmds.c | 19 +++++++++
src/test/regress/expected/offload_entry_to_qe.out | 2 +
src/test/regress/expected/partindex_test.out | 1 +
src/test/regress/expected/partition.out | 1 +
src/test/regress/expected/partition_optimizer.out | 1 +
src/test/regress/expected/qp_orca_fallback.out | 1 +
.../expected/qp_orca_fallback_optimizer.out | 1 +
src/test/regress/input/external_table.source | 12 +++---
src/test/regress/input/part_external_table.source | 48 ++++++++++++++++++++++
src/test/regress/output/external_table.source | 14 ++++---
.../regress/output/external_table_optimizer.source | 14 ++++---
src/test/regress/output/gpcopy.source | 1 +
src/test/regress/output/part_external_table.source | 43 +++++++++++++++++++
.../output/part_external_table_optimizer.source | 43 +++++++++++++++++++
15 files changed, 185 insertions(+), 18 deletions(-)
diff --git a/src/backend/access/external/external.c b/src/backend/access/external/external.c
index 0065bf32f8..0905a576a6 100644
--- a/src/backend/access/external/external.c
+++ b/src/backend/access/external/external.c
@@ -195,6 +195,8 @@ GetExtTableEntryIfExists(Oid relid)
extentry = GetExtFromForeignTableOptions(ft->options, relid);
+ pfree(ft);
+
return extentry;
}
diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c
index 7bac679c8c..08ead599fb 100644
--- a/src/backend/commands/tablecmds.c
+++ b/src/backend/commands/tablecmds.c
@@ -21578,6 +21578,25 @@ ATExecAttachPartition(List **wqueue, Relation rel, PartitionCmd *cmd,
RelationGetRelationName(rel),
RelationGetRelationName(attachrel))));
+ /* GPDB: some error checking for writable external tables. This is to keep same behavior as pre-7X. */
+ if (attachrel->rd_rel->relkind == RELKIND_FOREIGN_TABLE)
+ {
+ ExtTableEntry *exttable = GetExtTableEntryIfExists(attachrel->rd_id);
+ /* writable external table not allowed to be attached/exchanged */
+ if (exttable && exttable->iswritable)
+ ereport(ERROR,
+ (errcode(ERRCODE_WRONG_OBJECT_TYPE),
+ errmsg("cannot attach a WRITABLE external table")));
+
+ /* readable external table: throw a warning about no validation (only printed for coordinator/utility mode) */
+ else if (exttable && !exttable->iswritable && Gp_role != GP_ROLE_EXECUTE)
+ ereport(NOTICE,
+ (errcode(ERRCODE_WRONG_OBJECT_TYPE),
+ errmsg("partition constraints are not validated when attaching a readable external table")));
+ if (exttable)
+ pfree(exttable);
+ }
+
/* If the parent is permanent, so must be all of its partitions. */
if (rel->rd_rel->relpersistence != RELPERSISTENCE_TEMP &&
attachrel->rd_rel->relpersistence == RELPERSISTENCE_TEMP)
diff --git a/src/test/regress/expected/offload_entry_to_qe.out b/src/test/regress/expected/offload_entry_to_qe.out
index 4b441e7557..633504a2e1 100644
--- a/src/test/regress/expected/offload_entry_to_qe.out
+++ b/src/test/regress/expected/offload_entry_to_qe.out
@@ -862,6 +862,7 @@ select min(y), max(y) from part;
-- test partitioned table with external table as partition
ALTER TABLE part EXCHANGE PARTITION part1 WITH TABLE tst_exttbl;
+NOTICE: partition constraints are not validated when attaching a readable external table
explain (costs off, locus) select min(y), max(y) from part;
QUERY PLAN
-------------------------------------------------------
@@ -893,6 +894,7 @@ select min(y), max(y) from part;
(1 row)
ALTER TABLE part EXCHANGE PARTITION part1 WITH TABLE tst_exttbl_all;
+NOTICE: partition constraints are not validated when attaching a readable external table
explain (costs off, locus) select min(y), max(y) from part;
QUERY PLAN
-----------------------------------------------------------------------
diff --git a/src/test/regress/expected/partindex_test.out b/src/test/regress/expected/partindex_test.out
index d3ce8ca4ee..f5b061ade8 100644
--- a/src/test/regress/expected/partindex_test.out
+++ b/src/test/regress/expected/partindex_test.out
@@ -422,6 +422,7 @@ format 'csv';
alter table exchange_sub_partition
alter partition for (integer '2')
exchange partition for (integer '2') with table exchange_external_table;
+NOTICE: partition constraints are not validated when attaching a readable external table
select * from exchange_sub_partition;
a | b
---+---
diff --git a/src/test/regress/expected/partition.out b/src/test/regress/expected/partition.out
index bed83a3d3d..d7e26cc007 100755
--- a/src/test/regress/expected/partition.out
+++ b/src/test/regress/expected/partition.out
@@ -597,6 +597,7 @@ ERROR: cannot EXCHANGE PARTITION for relation "sto_ao_ao" -- partition has chil
create table foo_p (i int, j int) distributed by (i) partition by range(j) (start(1) end(10) every(2));
create readable external table bar_p(i int, j int) location ('gpfdist://host.invalid:8000/file') format 'text';
alter table foo_p exchange partition for(3) with table bar_p;
+NOTICE: partition constraints are not validated when attaching a readable external table
truncate foo_p;
ERROR: cannot truncate foreign table "foo_p_1_prt_2"
drop table foo_p;
diff --git a/src/test/regress/expected/partition_optimizer.out b/src/test/regress/expected/partition_optimizer.out
index 2ba98125ec..839d97ac0f 100755
--- a/src/test/regress/expected/partition_optimizer.out
+++ b/src/test/regress/expected/partition_optimizer.out
@@ -597,6 +597,7 @@ ERROR: cannot EXCHANGE PARTITION for relation "sto_ao_ao" -- partition has chil
create table foo_p (i int, j int) distributed by (i) partition by range(j) (start(1) end(10) every(2));
create readable external table bar_p(i int, j int) location ('gpfdist://host.invalid:8000/file') format 'text';
alter table foo_p exchange partition for(3) with table bar_p;
+NOTICE: partition constraints are not validated when attaching a readable external table
truncate foo_p;
ERROR: cannot truncate foreign table "foo_p_1_prt_2"
drop table foo_p;
diff --git a/src/test/regress/expected/qp_orca_fallback.out b/src/test/regress/expected/qp_orca_fallback.out
index 9c025590a9..87b33b955d 100644
--- a/src/test/regress/expected/qp_orca_fallback.out
+++ b/src/test/regress/expected/qp_orca_fallback.out
@@ -256,6 +256,7 @@ HINT: The 'DISTRIBUTED BY' clause determines the distribution of data. Make sur
create external web table p2_ext (like p1) EXECUTE 'cat something.txt' FORMAT 'TEXT';
alter table ext_part attach partition p1 for values in (1);
alter table ext_part attach partition p2_ext for values in (2);
+NOTICE: partition constraints are not validated when attaching a readable external table
explain insert into ext_part values (1);
QUERY PLAN
------------------------------------------------------
diff --git a/src/test/regress/expected/qp_orca_fallback_optimizer.out b/src/test/regress/expected/qp_orca_fallback_optimizer.out
index a9d7ff76a1..cbe28c0ffa 100644
--- a/src/test/regress/expected/qp_orca_fallback_optimizer.out
+++ b/src/test/regress/expected/qp_orca_fallback_optimizer.out
@@ -306,6 +306,7 @@ HINT: The 'DISTRIBUTED BY' clause determines the distribution of data. Make sur
create external web table p2_ext (like p1) EXECUTE 'cat something.txt' FORMAT 'TEXT';
alter table ext_part attach partition p1 for values in (1);
alter table ext_part attach partition p2_ext for values in (2);
+NOTICE: partition constraints are not validated when attaching a readable external table
explain insert into ext_part values (1);
INFO: GPORCA failed to produce a plan, falling back to Postgres-based planner
DETAIL: Falling back to Postgres-based planner because GPORCA does not support the following feature: Insert with External/foreign partition storage types
diff --git a/src/test/regress/input/external_table.source b/src/test/regress/input/external_table.source
index a2cb50b802..048da73909 100644
--- a/src/test/regress/input/external_table.source
+++ b/src/test/regress/input/external_table.source
@@ -3654,17 +3654,17 @@ ALTER TABLE ext_r_dist SET DISTRIBUTED BY (a); -- should error out altering read
-- Testing external table as the partition child.
CREATE TABLE part_root(a int) PARTITION BY RANGE(a);
CREATE TABLE part_child (LIKE part_root);
-CREATE WRITABLE EXTERNAL WEB TABLE part_ext_w(a int) EXECUTE 'cat > @abs_srcdir@/data/part_ext.tbl' FORMAT 'TEXT' (DELIMITER AS '|' NULL AS 'null' ESCAPE AS ' ') DISTRIBUTED BY (a);
+CREATE EXTERNAL WEB TABLE part_ext_r(a int) EXECUTE 'cat > @abs_srcdir@/data/part_ext.tbl' FORMAT 'TEXT' (DELIMITER AS '|' NULL AS 'null' ESCAPE AS ' ');
+CREATE WRITABLE EXTERNAL WEB TABLE part_ext_w(a int, b int) EXECUTE 'cat > @abs_srcdir@/data/part_ext.tbl' FORMAT 'TEXT' (DELIMITER AS '|' NULL AS 'null' ESCAPE AS ' ') DISTRIBUTED BY (a);
ALTER TABLE part_root ATTACH PARTITION part_child FOR VALUES FROM (0) TO (10);
-ALTER TABLE part_root ATTACH PARTITION part_ext_w FOR VALUES FROM (10) TO (20);
+ALTER TABLE part_root ATTACH PARTITION part_ext_r FOR VALUES FROM (10) TO (20);
--- Adding column should work fine
+-- Adding column on readable external table should work fine
ALTER TABLE part_root ADD COLUMN b int;
-INSERT INTO part_root SELECT i,i FROM generate_series(1,19)i;
--- altering distribution policy should work fine
+-- altering distribution policy on writable external table should work fine
SELECT policytype, distkey FROM gp_distribution_policy WHERE localoid = 'part_ext_w'::regclass;
-ALTER TABLE part_root SET DISTRIBUTED BY (b);
+ALTER TABLE part_ext_w SET DISTRIBUTED BY (b);
SELECT policytype, distkey FROM gp_distribution_policy WHERE localoid = 'part_ext_w'::regclass;
DROP TABLE part_root;
diff --git a/src/test/regress/input/part_external_table.source b/src/test/regress/input/part_external_table.source
index 50550944c2..577cdd2a1e 100644
--- a/src/test/regress/input/part_external_table.source
+++ b/src/test/regress/input/part_external_table.source
@@ -129,3 +129,51 @@ explain select * from part where b = 53;
-- only select foreign scans
explain select * from part where b > 22;
+
+--
+-- exchange & attach partition
+--
+alter table part add partition exch1 start(60) end (70);
+alter table part add partition exch2 start(70) end (80);
+
+-- exchange with external tables
+create external web table p3_e (a int, b int) execute 'cat > @abs_srcdir@/data/part-ext.csv' format 'csv' (delimiter as '|' null as 'null' escape as ' ');
+create writable external web table p4_e (a int, b int) execute 'cat > @abs_srcdir@/data/part-ext.csv' format 'csv' (delimiter as '|' null as 'null' escape as ' ');
+
+-- allow exchange readable external table
+alter table part exchange partition exch1 with table p3_e;
+
+-- should disallow writable external table
+alter table part exchange partition exch1 with table p4_e;
+
+-- exchange with foreign tables
+CREATE SERVER file_server3 FOREIGN DATA WRAPPER file_fdw;
+CREATE FOREIGN TABLE ft3 (
+ a int,
+ b int
+) SERVER file_server3
+OPTIONS ( filename '/does/not/exist.csv', format 'csv');
+
+-- exchange works, but no error checking like for external tables
+alter table part exchange partition exch2 with table ft3;
+
+-- same tests for attach partition
+create external web table p5_e (a int, b int) execute 'cat > @abs_srcdir@/data/part-ext.csv' format 'csv' (delimiter as '|' null as 'null' escape as ' ');
+create writable external web table p6_e (a int, b int) execute 'cat > @abs_srcdir@/data/part-ext.csv' format 'csv' (delimiter as '|' null as 'null' escape as ' ');
+
+-- allow attach readable external table
+alter table part attach partition p5_e for values from (80) to (90);
+
+-- should disallow writable external table
+alter table part attach partition p6_e for values from (90) to (100);
+
+-- attach foreign table
+CREATE SERVER file_server4 FOREIGN DATA WRAPPER file_fdw;
+CREATE FOREIGN TABLE ft4 (
+ a int,
+ b int
+) SERVER file_server4
+OPTIONS ( filename '/does/not/exist.csv', format 'csv');
+
+-- exchange works, but no error checking like for external tables
+alter table part attach partition ft4 for values from (100) to (110);
diff --git a/src/test/regress/output/external_table.source b/src/test/regress/output/external_table.source
index 116d1325bc..11aa5fce4e 100644
--- a/src/test/regress/output/external_table.source
+++ b/src/test/regress/output/external_table.source
@@ -4927,6 +4927,7 @@ INSERT INTO test_part_integrity_p1 VALUES (1,2),(1,3);
CREATE EXTERNAL WEB TABLE test_part_integrity_p2 (LIKE test_part_integrity_p1) EXECUTE 'echo 2,2;echo 2,3' ON MASTER FORMAT 'CSV';
ALTER TABLE test_part_integrity ATTACH PARTITION test_part_integrity_p1 FOR VALUES IN (1);
ALTER TABLE test_part_integrity ATTACH PARTITION test_part_integrity_p2 FOR VALUES IN (2);
+NOTICE: partition constraints are not validated when attaching a readable external table
SELECT * FROM test_part_integrity;
a | b
---+---
@@ -4975,20 +4976,21 @@ ERROR: cannot set distribution policy of readable external table "ext_r_dist"
-- Testing external table as the partition child.
CREATE TABLE part_root(a int) PARTITION BY RANGE(a);
CREATE TABLE part_child (LIKE part_root);
-CREATE WRITABLE EXTERNAL WEB TABLE part_ext_w(a int) EXECUTE 'cat > @abs_srcdir@/data/part_ext.tbl' FORMAT 'TEXT' (DELIMITER AS '|' NULL AS 'null' ESCAPE AS ' ') DISTRIBUTED BY (a);
+CREATE EXTERNAL WEB TABLE part_ext_r(a int) EXECUTE 'cat > @abs_srcdir@/data/part_ext.tbl' FORMAT 'TEXT' (DELIMITER AS '|' NULL AS 'null' ESCAPE AS ' ');
+CREATE WRITABLE EXTERNAL WEB TABLE part_ext_w(a int, b int) EXECUTE 'cat > @abs_srcdir@/data/part_ext.tbl' FORMAT 'TEXT' (DELIMITER AS '|' NULL AS 'null' ESCAPE AS ' ') DISTRIBUTED BY (a);
ALTER TABLE part_root ATTACH PARTITION part_child FOR VALUES FROM (0) TO (10);
-ALTER TABLE part_root ATTACH PARTITION part_ext_w FOR VALUES FROM (10) TO (20);
--- Adding column should work fine
+ALTER TABLE part_root ATTACH PARTITION part_ext_r FOR VALUES FROM (10) TO (20);
+NOTICE: partition constraints are not validated when attaching a readable external table
+-- Adding column on readable external table should work fine
ALTER TABLE part_root ADD COLUMN b int;
-INSERT INTO part_root SELECT i,i FROM generate_series(1,19)i;
--- altering distribution policy should work fine
+-- altering distribution policy on writable external table should work fine
SELECT policytype, distkey FROM gp_distribution_policy WHERE localoid = 'part_ext_w'::regclass;
policytype | distkey
------------+---------
p | 1
(1 row)
-ALTER TABLE part_root SET DISTRIBUTED BY (b);
+ALTER TABLE part_ext_w SET DISTRIBUTED BY (b);
SELECT policytype, distkey FROM gp_distribution_policy WHERE localoid = 'part_ext_w'::regclass;
policytype | distkey
------------+---------
diff --git a/src/test/regress/output/external_table_optimizer.source b/src/test/regress/output/external_table_optimizer.source
index f4543639a3..1ceb36e01e 100644
--- a/src/test/regress/output/external_table_optimizer.source
+++ b/src/test/regress/output/external_table_optimizer.source
@@ -4929,6 +4929,7 @@ INSERT INTO test_part_integrity_p1 VALUES (1,2),(1,3);
CREATE EXTERNAL WEB TABLE test_part_integrity_p2 (LIKE test_part_integrity_p1) EXECUTE 'echo 2,2;echo 2,3' ON MASTER FORMAT 'CSV';
ALTER TABLE test_part_integrity ATTACH PARTITION test_part_integrity_p1 FOR VALUES IN (1);
ALTER TABLE test_part_integrity ATTACH PARTITION test_part_integrity_p2 FOR VALUES IN (2);
+NOTICE: partition constraints are not validated when attaching a readable external table
SELECT * FROM test_part_integrity;
a | b
---+---
@@ -4977,20 +4978,21 @@ ERROR: cannot set distribution policy of readable external table "ext_r_dist"
-- Testing external table as the partition child.
CREATE TABLE part_root(a int) PARTITION BY RANGE(a);
CREATE TABLE part_child (LIKE part_root);
-CREATE WRITABLE EXTERNAL WEB TABLE part_ext_w(a int) EXECUTE 'cat > @abs_srcdir@/data/part_ext.tbl' FORMAT 'TEXT' (DELIMITER AS '|' NULL AS 'null' ESCAPE AS ' ') DISTRIBUTED BY (a);
+CREATE EXTERNAL WEB TABLE part_ext_r(a int) EXECUTE 'cat > @abs_srcdir@/data/part_ext.tbl' FORMAT 'TEXT' (DELIMITER AS '|' NULL AS 'null' ESCAPE AS ' ');
+CREATE WRITABLE EXTERNAL WEB TABLE part_ext_w(a int, b int) EXECUTE 'cat > @abs_srcdir@/data/part_ext.tbl' FORMAT 'TEXT' (DELIMITER AS '|' NULL AS 'null' ESCAPE AS ' ') DISTRIBUTED BY (a);
ALTER TABLE part_root ATTACH PARTITION part_child FOR VALUES FROM (0) TO (10);
-ALTER TABLE part_root ATTACH PARTITION part_ext_w FOR VALUES FROM (10) TO (20);
--- Adding column should work fine
+ALTER TABLE part_root ATTACH PARTITION part_ext_r FOR VALUES FROM (10) TO (20);
+NOTICE: partition constraints are not validated when attaching a readable external table
+-- Adding column on readable external table should work fine
ALTER TABLE part_root ADD COLUMN b int;
-INSERT INTO part_root SELECT i,i FROM generate_series(1,19)i;
--- altering distribution policy should work fine
+-- altering distribution policy on writable external table should work fine
SELECT policytype, distkey FROM gp_distribution_policy WHERE localoid = 'part_ext_w'::regclass;
policytype | distkey
------------+---------
p | 1
(1 row)
-ALTER TABLE part_root SET DISTRIBUTED BY (b);
+ALTER TABLE part_ext_w SET DISTRIBUTED BY (b);
SELECT policytype, distkey FROM gp_distribution_policy WHERE localoid = 'part_ext_w'::regclass;
policytype | distkey
------------+---------
diff --git a/src/test/regress/output/gpcopy.source b/src/test/regress/output/gpcopy.source
index a5fd552c2b..6a9ec337c1 100755
--- a/src/test/regress/output/gpcopy.source
+++ b/src/test/regress/output/gpcopy.source
@@ -1434,6 +1434,7 @@ COPY 26
CREATE EXTERNAL WEB TABLE ext_dec17(LIKE sales_1_prt_dec17) EXECUTE 'printf "12\t2017-12-01\t21.00\n"' ON COORDINATOR FORMAT 'text';
CREATE EXTERNAL TABLE
ALTER TABLE sales EXCHANGE PARTITION dec17 WITH TABLE ext_dec17;
+NOTICE: partition constraints are not validated when attaching a readable external table
ALTER TABLE
DROP TABLE ext_dec17;
DROP TABLE
diff --git a/src/test/regress/output/part_external_table.source b/src/test/regress/output/part_external_table.source
index 25fc683aa3..c94110224b 100644
--- a/src/test/regress/output/part_external_table.source
+++ b/src/test/regress/output/part_external_table.source
@@ -33,7 +33,9 @@ HINT: The 'DISTRIBUTED BY' clause determines the distribution of data. Make sur
create external table p1_e (a int, b int) location ('file://@hostname@@abs_srcdir@/data/part1.csv') format 'csv';
create external table p2_e (a int, b int) location ('file://@hostname@@abs_srcdir@/data/part2.csv') format 'csv';
alter table part attach partition p1_e for values from (0) to (10);
+NOTICE: partition constraints are not validated when attaching a readable external table
alter table part attach partition p2_e for values from (10) to (19);
+NOTICE: partition constraints are not validated when attaching a readable external table
analyze part;
WARNING: skipping "p2_e" --- cannot analyze this foreign table
WARNING: skipping "p1_e" --- cannot analyze this foreign table
@@ -412,3 +414,44 @@ explain select * from part where b > 22;
Optimizer: Postgres query optimizer
(14 rows)
+--
+-- exchange & attach partition
+--
+alter table part add partition exch1 start(60) end (70);
+alter table part add partition exch2 start(70) end (80);
+-- exchange with external tables
+create external web table p3_e (a int, b int) execute 'cat > @abs_srcdir@/data/part-ext.csv' format 'csv' (delimiter as '|' null as 'null' escape as ' ');
+create writable external web table p4_e (a int, b int) execute 'cat > @abs_srcdir@/data/part-ext.csv' format 'csv' (delimiter as '|' null as 'null' escape as ' ');
+-- allow exchange readable external table
+alter table part exchange partition exch1 with table p3_e;
+NOTICE: partition constraints are not validated when attaching a readable external table
+-- should disallow writable external table
+alter table part exchange partition exch1 with table p4_e;
+ERROR: cannot attach a WRITABLE external table
+-- exchange with foreign tables
+CREATE SERVER file_server3 FOREIGN DATA WRAPPER file_fdw;
+CREATE FOREIGN TABLE ft3 (
+ a int,
+ b int
+) SERVER file_server3
+OPTIONS ( filename '/does/not/exist.csv', format 'csv');
+-- exchange works, but no error checking like for external tables
+alter table part exchange partition exch2 with table ft3;
+-- same tests for attach partition
+create external web table p5_e (a int, b int) execute 'cat > @abs_srcdir@/data/part-ext.csv' format 'csv' (delimiter as '|' null as 'null' escape as ' ');
+create writable external web table p6_e (a int, b int) execute 'cat > @abs_srcdir@/data/part-ext.csv' format 'csv' (delimiter as '|' null as 'null' escape as ' ');
+-- allow attach readable external table
+alter table part attach partition p5_e for values from (80) to (90);
+NOTICE: partition constraints are not validated when attaching a readable external table
+-- should disallow writable external table
+alter table part attach partition p6_e for values from (90) to (100);
+ERROR: cannot attach a WRITABLE external table
+-- attach foreign table
+CREATE SERVER file_server4 FOREIGN DATA WRAPPER file_fdw;
+CREATE FOREIGN TABLE ft4 (
+ a int,
+ b int
+) SERVER file_server4
+OPTIONS ( filename '/does/not/exist.csv', format 'csv');
+-- exchange works, but no error checking like for external tables
+alter table part attach partition ft4 for values from (100) to (110);
diff --git a/src/test/regress/output/part_external_table_optimizer.source b/src/test/regress/output/part_external_table_optimizer.source
index f04f2fadce..81b69086c5 100644
--- a/src/test/regress/output/part_external_table_optimizer.source
+++ b/src/test/regress/output/part_external_table_optimizer.source
@@ -33,7 +33,9 @@ HINT: The 'DISTRIBUTED BY' clause determines the distribution of data. Make sur
create external table p1_e (a int, b int) location ('file://@hostname@@abs_srcdir@/data/part1.csv') format 'csv';
create external table p2_e (a int, b int) location ('file://@hostname@@abs_srcdir@/data/part2.csv') format 'csv';
alter table part attach partition p1_e for values from (0) to (10);
+NOTICE: partition constraints are not validated when attaching a readable external table
alter table part attach partition p2_e for values from (10) to (19);
+NOTICE: partition constraints are not validated when attaching a readable external table
analyze part;
WARNING: skipping "p2_e" --- cannot analyze this foreign table
WARNING: skipping "p1_e" --- cannot analyze this foreign table
@@ -408,3 +410,44 @@ explain select * from part where b > 22;
Optimizer: Pivotal Optimizer (GPORCA)
(13 rows)
+--
+-- exchange & attach partition
+--
+alter table part add partition exch1 start(60) end (70);
+alter table part add partition exch2 start(70) end (80);
+-- exchange with external tables
+create external web table p3_e (a int, b int) execute 'cat > @abs_srcdir@/data/part-ext.csv' format 'csv' (delimiter as '|' null as 'null' escape as ' ');
+create writable external web table p4_e (a int, b int) execute 'cat > @abs_srcdir@/data/part-ext.csv' format 'csv' (delimiter as '|' null as 'null' escape as ' ');
+-- allow exchange readable external table
+alter table part exchange partition exch1 with table p3_e;
+NOTICE: partition constraints are not validated when attaching a readable external table
+-- should disallow writable external table
+alter table part exchange partition exch1 with table p4_e;
+ERROR: cannot attach a WRITABLE external table
+-- exchange with foreign tables
+CREATE SERVER file_server3 FOREIGN DATA WRAPPER file_fdw;
+CREATE FOREIGN TABLE ft3 (
+ a int,
+ b int
+) SERVER file_server3
+OPTIONS ( filename '/does/not/exist.csv', format 'csv');
+-- exchange works, but no error checking like for external tables
+alter table part exchange partition exch2 with table ft3;
+-- same tests for attach partition
+create external web table p5_e (a int, b int) execute 'cat > @abs_srcdir@/data/part-ext.csv' format 'csv' (delimiter as '|' null as 'null' escape as ' ');
+create writable external web table p6_e (a int, b int) execute 'cat > @abs_srcdir@/data/part-ext.csv' format 'csv' (delimiter as '|' null as 'null' escape as ' ');
+-- allow attach readable external table
+alter table part attach partition p5_e for values from (80) to (90);
+NOTICE: partition constraints are not validated when attaching a readable external table
+-- should disallow writable external table
+alter table part attach partition p6_e for values from (90) to (100);
+ERROR: cannot attach a WRITABLE external table
+-- attach foreign table
+CREATE SERVER file_server4 FOREIGN DATA WRAPPER file_fdw;
+CREATE FOREIGN TABLE ft4 (
+ a int,
+ b int
+) SERVER file_server4
+OPTIONS ( filename '/does/not/exist.csv', format 'csv');
+-- exchange works, but no error checking like for external tables
+alter table part attach partition ft4 for values from (100) to (110);
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For additional commands, e-mail: commits-help@cloudberry.apache.org
(cloudberry) 01/08: Fix concurrent update core of partition table in Dynamic scan (#15253)
Posted by yj...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
yjhjstz pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/cloudberry.git
commit 71abf7a0abc12384fb6be66d4d19e8d5005a432e
Author: chaotian <10...@users.noreply.github.com>
AuthorDate: Wed Jun 21 14:04:59 2023 +0800
Fix concurrent update core of partition table in Dynamic scan (#15253)
Once GDD is enabled, heap tuple can be concurrently updated
on segments. ORCA will generate Dynamic Scan for Partition table
instead of multi subplans in planner. For concurrent update, executor
initialized EPQ state and table describer called resultRelInfo which
always generated for each partition. But in Dynamic Scan, we only
have one resultRelInfo for table not partitions here, and will fill Dynamic
scan infos before IUD operations. So we need to adjust Dynamic Scan
to EPQ which used to be based on planner.
Here There are two places needed to fix.
- As we ExecInitPartitionInfo for partition table, we assign 1 to
ResultRelInfo->ri_RangeTableIndex instead of 0, and we could
make EPQ work in this way. But in FDW scenario, we choose to
keep 0 for ri_RangeTableIndex since FWD has its own process
dealing with ResultRelInfo->ri_RangeTableIndex 0.
- Inherit `resultRelInfo->ri_junkFilter `infos for partitioned
resultRelInfo from root main table resultRelInfo.
---
src/backend/executor/execPartition.c | 13 ++++++++--
.../isolation2/expected/gdd/concurrent_update.out | 30 ++++++++++++++++++++++
.../expected/gdd/concurrent_update_optimizer.out | 30 ++++++++++++++++++++++
src/test/isolation2/sql/gdd/concurrent_update.sql | 18 +++++++++++++
4 files changed, 89 insertions(+), 2 deletions(-)
diff --git a/src/backend/executor/execPartition.c b/src/backend/executor/execPartition.c
index 4ee82b073e..88a16048ee 100644
--- a/src/backend/executor/execPartition.c
+++ b/src/backend/executor/execPartition.c
@@ -515,9 +515,18 @@ ExecInitPartitionInfo(ModifyTableState *mtstate, EState *estate,
partrel = table_open(partOid, RowExclusiveLock);
leaf_part_rri = makeNode(ResultRelInfo);
- InitResultRelInfo(leaf_part_rri,
+
+ /*
+ * Init leaf partition ResultRelInfo
+ * Here ResultRelInfo->ri_RangeTableIndex is a dummy element, because we
+ * will rebuild ri_RelationDesc later. So we assign 1 for it instead of 0
+ * which cause failure in EPQ process, and fwd scenarios should still keep 0
+ * since it will handle 0 in their own fwd process.
+ * related issue https://github.com/greenplum-db/gpdb/issues/14935
+ */
+ InitResultRelInfo(leaf_part_rri,
partrel,
- 0,
+ partrel->rd_rel->relkind != RELKIND_FOREIGN_TABLE ? 1 : 0,
rootResultRelInfo,
estate->es_instrument);
diff --git a/src/test/isolation2/expected/gdd/concurrent_update.out b/src/test/isolation2/expected/gdd/concurrent_update.out
index 5cecc18fbd..761e3f681a 100644
--- a/src/test/isolation2/expected/gdd/concurrent_update.out
+++ b/src/test/isolation2/expected/gdd/concurrent_update.out
@@ -307,6 +307,36 @@ DROP
1q: ... <quitting>
2q: ... <quitting>
+-- test ORCA partition table
+-- related github issue https://github.com/greenplum-db/gpdb/issues/14935
+create table test(a int, b int, c int) partition by range(b) (start (1) end (7) every (3));
+CREATE
+insert into test values (1, 1, 1), (1, 2, 1);
+INSERT 2
+1: begin;
+BEGIN
+1: update test set c = 1;
+UPDATE 2
+-- in session 2, in case of ORCA DML invokes EPQ
+-- the following SQL will hang due to XID lock
+2&: update test set c = 1; <waiting ...>
+1: end;
+END
+2<: <... completed>
+UPDATE 2
+
+0: select * from test;
+ a | b | c
+---+---+---
+ 1 | 1 | 1
+ 1 | 2 | 1
+(2 rows)
+0: drop table test;
+DROP
+0q: ... <quitting>
+1q: ... <quitting>
+2q: ... <quitting>
+
-- split update is to implement updating on hash keys,
-- it deletes the tuple and insert a new tuple in a
-- new segment, so it is not easy for other transaction
diff --git a/src/test/isolation2/expected/gdd/concurrent_update_optimizer.out b/src/test/isolation2/expected/gdd/concurrent_update_optimizer.out
index 362a8c9888..4a04cd86a2 100644
--- a/src/test/isolation2/expected/gdd/concurrent_update_optimizer.out
+++ b/src/test/isolation2/expected/gdd/concurrent_update_optimizer.out
@@ -307,6 +307,36 @@ DROP
1q: ... <quitting>
2q: ... <quitting>
+-- test ORCA partition table
+-- related github issue https://github.com/greenplum-db/gpdb/issues/14935
+create table test(a int, b int, c int) partition by range(b) (start (1) end (7) every (3));
+CREATE
+insert into test values (1, 1, 1), (1, 2, 1);
+INSERT 2
+1: begin;
+BEGIN
+1: update test set c = 1;
+UPDATE 2
+-- in session 2, in case of ORCA DML invokes EPQ
+-- the following SQL will hang due to XID lock
+2&: update test set c = 1; <waiting ...>
+1: end;
+END
+2<: <... completed>
+UPDATE 2
+
+0: select * from test;
+ a | b | c
+---+---+---
+ 1 | 1 | 1
+ 1 | 2 | 1
+(2 rows)
+0: drop table test;
+DROP
+0q: ... <quitting>
+1q: ... <quitting>
+2q: ... <quitting>
+
-- split update is to implement updating on hash keys,
-- it deletes the tuple and insert a new tuple in a
-- new segment, so it is not easy for other transaction
diff --git a/src/test/isolation2/sql/gdd/concurrent_update.sql b/src/test/isolation2/sql/gdd/concurrent_update.sql
index dde1876126..44cf5916b1 100644
--- a/src/test/isolation2/sql/gdd/concurrent_update.sql
+++ b/src/test/isolation2/sql/gdd/concurrent_update.sql
@@ -174,6 +174,24 @@ insert into test values (1, 1, 1);
1q:
2q:
+-- test ORCA partition table
+-- related github issue https://github.com/greenplum-db/gpdb/issues/14935
+create table test(a int, b int, c int) partition by range(b) (start (1) end (7) every (3));
+insert into test values (1, 1, 1), (1, 2, 1);
+1: begin;
+1: update test set c = 1;
+-- in session 2, in case of ORCA DML invokes EPQ
+-- the following SQL will hang due to XID lock
+2&: update test set c = 1;
+1: end;
+2<:
+
+0: select * from test;
+0: drop table test;
+0q:
+1q:
+2q:
+
-- split update is to implement updating on hash keys,
-- it deletes the tuple and insert a new tuple in a
-- new segment, so it is not easy for other transaction
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For additional commands, e-mail: commits-help@cloudberry.apache.org
(cloudberry) 08/08: Fix singlenode partition test.
Posted by yj...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
yjhjstz pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/cloudberry.git
commit 75928a683a164e5b3f156b1af290a9addf10fd75
Author: Jianghua Yang <yj...@gmail.com>
AuthorDate: Tue Feb 18 02:14:03 2025 +0800
Fix singlenode partition test.
---
src/test/singlenode_regress/expected/partition.out | 1 +
1 file changed, 1 insertion(+)
diff --git a/src/test/singlenode_regress/expected/partition.out b/src/test/singlenode_regress/expected/partition.out
index 9ba7a1dc41..2a7398e334 100755
--- a/src/test/singlenode_regress/expected/partition.out
+++ b/src/test/singlenode_regress/expected/partition.out
@@ -480,6 +480,7 @@ ERROR: cannot EXCHANGE PARTITION for relation "sto_ao_ao" -- partition has chil
create table foo_p (i int, j int) partition by range(j) (start(1) end(10) every(2));
create readable external table bar_p(i int, j int) location ('gpfdist://host.invalid:8000/file') format 'text';
alter table foo_p exchange partition for(3) with table bar_p;
+NOTICE: partition constraints are not validated when attaching a readable external table
truncate foo_p;
ERROR: cannot truncate foreign table "foo_p_1_prt_2"
drop table foo_p;
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For additional commands, e-mail: commits-help@cloudberry.apache.org
(cloudberry) 03/08: Test fix failure by adding timeeout and fixing test steps. (#14589)
Posted by yj...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
yjhjstz pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/cloudberry.git
commit abce4a7298a552f7bd04c18e38ae326ea3d84d94
Author: Piyush Chandwadkar <65...@users.noreply.github.com>
AuthorDate: Tue Jan 10 10:00:58 2023 +0530
Test fix failure by adding timeeout and fixing test steps. (#14589)
Some of the gpmovemirror and gprecoverseg tests seen failing
when waiting for segment to come online or waiting for progress
file to get created. Enhancing timeout for such cases.
Also when gprecoverseg when ran in async mode and taking longer to execute, added wait step in such scenario,
---
gpMgmt/test/behave/mgmt_utils/gpmovemirrors.feature | 1 +
gpMgmt/test/behave/mgmt_utils/gprecoverseg.feature | 6 ++++++
gpMgmt/test/behave/mgmt_utils/steps/mgmt_utils.py | 2 +-
gpMgmt/test/behave_utils/utils.py | 2 +-
4 files changed, 9 insertions(+), 2 deletions(-)
diff --git a/gpMgmt/test/behave/mgmt_utils/gpmovemirrors.feature b/gpMgmt/test/behave/mgmt_utils/gpmovemirrors.feature
index 1333e54c0f..ed6dda775f 100644
--- a/gpMgmt/test/behave/mgmt_utils/gpmovemirrors.feature
+++ b/gpMgmt/test/behave/mgmt_utils/gpmovemirrors.feature
@@ -146,6 +146,7 @@ Feature: Tests for gpmovemirrors
| run gprecoverseg | some | 0 | 1,2 | running in place full recovery for all failed contents |
| run gprecoverseg | all | None | 0,1,2 | running in place full recovery for all failed contents |
+ @skip_cleanup
Scenario: gpmovemirrors can move mirrors even if start fails for some mirrors
Given the database is running
And all the segments are running
diff --git a/gpMgmt/test/behave/mgmt_utils/gprecoverseg.feature b/gpMgmt/test/behave/mgmt_utils/gprecoverseg.feature
index 55fd42d070..3d28dfc11d 100644
--- a/gpMgmt/test/behave/mgmt_utils/gprecoverseg.feature
+++ b/gpMgmt/test/behave/mgmt_utils/gprecoverseg.feature
@@ -525,6 +525,7 @@ Feature: gprecoverseg tests
And sql "DROP TABLE IF EXISTS test_recoverseg; CREATE TABLE test_recoverseg AS SELECT generate_series(1,100000000) AS a;" is executed in "postgres" db
When the user asynchronously runs "gprecoverseg -a" and the process is saved
Then the user waits until recovery_progress.file is created in gpAdminLogs and verifies its format
+ And an FTS probe is triggered
And the user waits until saved async process is completed
And recovery_progress.file should not exist in gpAdminLogs
And the user waits until mirror on content 0,1,2 is up
@@ -543,7 +544,9 @@ Feature: gprecoverseg tests
And all files in gpAdminLogs directory are deleted on all hosts in the cluster
Then the gprecoverseg lock directory is removed
+ And the cluster is rebalanced
And user immediately stops all primary processes for content 0,1,2
+ And the user waits until mirror on content 0,1,2 is down
And user can start transactions
When the user asynchronously runs "gprecoverseg -aF" and the process is saved
And the user suspend the walsender on the primary on content 0
@@ -553,6 +556,7 @@ Feature: gprecoverseg tests
And the user reset the walsender on the primary on content 0
And the user waits until saved async process is completed
And recovery_progress.file should not exist in gpAdminLogs
+ And an FTS probe is triggered
And the user waits until mirror on content 0,1,2 is up
And user can start transactions
@@ -673,6 +677,8 @@ Feature: gprecoverseg tests
And edit the input file to recover mirror with content 2 incremental
When the user asynchronously runs gprecoverseg with input file and additional args "-a" and the process is saved
Then the user waits until recovery_progress.file is created in gpAdminLogs and verifies its format
+ And user waits until gp_stat_replication table has no pg_basebackup entries for content 1
+ And an FTS probe is triggered
And the user waits until mirror on content 1,2 is up
And verify that mirror on content 0 is down
And user can start transactions
diff --git a/gpMgmt/test/behave/mgmt_utils/steps/mgmt_utils.py b/gpMgmt/test/behave/mgmt_utils/steps/mgmt_utils.py
index 7b112ef29c..5d7d89b0f3 100644
--- a/gpMgmt/test/behave/mgmt_utils/steps/mgmt_utils.py
+++ b/gpMgmt/test/behave/mgmt_utils/steps/mgmt_utils.py
@@ -461,7 +461,7 @@ def impl(context, logdir):
@then('the user waits until recovery_progress.file is created in {logdir} and verifies its format')
def impl(context, logdir):
attempt = 0
- num_retries = 60000
+ num_retries = 6000
log_dir = _get_gpAdminLogs_directory() if logdir == 'gpAdminLogs' else logdir
recovery_progress_file = '{}/recovery_progress.file'.format(log_dir)
while attempt < num_retries:
diff --git a/gpMgmt/test/behave_utils/utils.py b/gpMgmt/test/behave_utils/utils.py
index 8a05d2808f..e23629f7da 100644
--- a/gpMgmt/test/behave_utils/utils.py
+++ b/gpMgmt/test/behave_utils/utils.py
@@ -808,7 +808,7 @@ def wait_for_desired_query_result(dburl, query, desired_result, utility=False):
Raises an Exception after failing <num_retries> times.
"""
attempt = 0
- num_retries = 150
+ num_retries = 600
actual_result = None
while (attempt < num_retries) and (actual_result != desired_result):
attempt += 1
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For additional commands, e-mail: commits-help@cloudberry.apache.org
(cloudberry) 06/08: Remove psutil usage from regress tests (#15711)
Posted by yj...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
yjhjstz pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/cloudberry.git
commit 73c2fbd7d04cfe266783470caa00a3ca9264abad
Author: Chen Mulong <ch...@gmail.com>
AuthorDate: Fri Jun 9 09:50:55 2023 +0800
Remove psutil usage from regress tests (#15711)
psutil is not a dependency of plpython, thus we cannot assume it exists
when running regress test. We are going to use different python versions
for management utilities and plpython in the future release, and psutil
won't exist anymore while running regress tests.
- Rewrite teh motion_socket tests with 'lsof', which is a explicit
dependency of greenplum.
Related with #15703
---
src/test/regress/expected/motion_socket.out | 41 ++++++++++++++++++-----------
src/test/regress/sql/motion_socket.sql | 37 +++++++++++++++++---------
2 files changed, 50 insertions(+), 28 deletions(-)
diff --git a/src/test/regress/expected/motion_socket.out b/src/test/regress/expected/motion_socket.out
index 5128b944cb..2a5fddd561 100644
--- a/src/test/regress/expected/motion_socket.out
+++ b/src/test/regress/expected/motion_socket.out
@@ -3,11 +3,14 @@
-- gp_session_id. Additionally we check if the source address used for creating
-- the motion sockets is equal to gp_segment_configuration.address.
-- start_matchignore
--- m/^INFO: Checking postgres backend postgres:*/
+-- m/^INFO: Checking postgres backend.*/
+-- m/b.*COMMAND.*NODE.*NAME.*/
+-- m/b.*postgres.*TCP.*/
+-- m/b.*postgres.*UDP.*/
-- end_matchignore
CREATE FUNCTION check_motion_sockets()
RETURNS VOID as $$
-import psutil, socket
+import subprocess, re, os, socket
# Create a temporary table to create a gang
plpy.execute("CREATE TEMP TABLE motion_socket_force_create_gang(i int);")
@@ -20,10 +23,10 @@ res = plpy.execute("SELECT current_setting('gp_interconnect_type');", 1)
ic_type = res[0]['current_setting']
if ic_type in ['tcp', 'proxy']:
expected_socket_count_per_segment = 1
- expected_socket_kind = socket.SocketKind.SOCK_STREAM
+ expected_socket_kind = "TCP"
elif ic_type=='udpifc':
expected_socket_count_per_segment = 2
- expected_socket_kind = socket.SocketKind.SOCK_DGRAM
+ expected_socket_kind = "UDP"
else:
plpy.error('Unrecognized gp_interconnect_type {}.'.format(ic_type))
@@ -38,28 +41,34 @@ pids_to_check = [r['pid'] for r in res]
for pid in pids_to_check:
# We iterate through all backends related to current session
motion_socket_count = 0
- process = psutil.Process(pid)
- plpy.info('Checking postgres backend {}'.format(process.cmdline()[0]))
- for conn in process.connections():
- if conn.type == expected_socket_kind and conn.raddr == () \
- and conn.laddr.ip == hostip:
+ # Typical 'lsof -i -nP -a -p <pid>' output:
+ # COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
+ # postgres 271688 cmulong 4u IPv4 1087560 0t0 TCP 127.0.0.1:51327 (LISTEN)
+ # postgres 271688 cmulong 9u IPv6 1077090 0t0 UDP [::1]:42130->[::1]:42130
+ # We check count of those connections which have not been established.
+ # Use the regex for example: "TCP <ip>:\d+ .*" (without '->')
+ lsof_ret = subprocess.run(["lsof", "-i", "-nP", "-a", "-p", str(pid)],
+ capture_output=True, check=True).stdout
+ plpy.info(
+ f'Checking postgres backend {pid}, ' \
+ f'lsof output:\n{os.linesep.join(map(str, lsof_ret.splitlines()))}')
+ # The first row are the column names
+ r = lsof_ret.splitlines()[1:]
+ for line in r:
+ # No '->' follows he port. The connection has not be setup yet
+ match=re.match(f".*{expected_socket_kind} {hostip}:\d+ .*", str(line))
+ if match is not None:
motion_socket_count += 1
if motion_socket_count != expected_socket_count_per_segment:
plpy.error('Expected {} motion sockets but found {}. '\
- 'For backend process {}. connections= {}'\
- .format(expected_socket_count_per_segment, process,\
- motion_socket_count, process.connections()))
+ .format(expected_socket_count_per_segment, motion_socket_count))
$$ LANGUAGE plpython3u;
SELECT check_motion_sockets();
NOTICE: Table doesn't have 'DISTRIBUTED BY' clause -- Using column named 'i' as the Greenplum Database data distribution key for this table.
HINT: The 'DISTRIBUTED BY' clause determines the distribution of data. Make sure column(s) chosen are the optimal data distribution key to minimize skew.
-INFO: Checking postgres backend postgres: 7000, vanjared regression 127.0.0.1(59545) con1812 cmd2 SELECT
-INFO: Checking postgres backend postgres: 7002, vanjared regression 127.0.0.1(59550) con1812 seg0 idle in transaction
-INFO: Checking postgres backend postgres: 7003, vanjared regression 127.0.0.1(59551) con1812 seg1 idle in transaction
-INFO: Checking postgres backend postgres: 7004, vanjared regression 127.0.0.1(59552) con1812 seg2 idle in transaction
check_motion_sockets
----------------------
diff --git a/src/test/regress/sql/motion_socket.sql b/src/test/regress/sql/motion_socket.sql
index 90ba30dd32..6d1a08973e 100644
--- a/src/test/regress/sql/motion_socket.sql
+++ b/src/test/regress/sql/motion_socket.sql
@@ -5,11 +5,14 @@
-- start_matchignore
--- m/^INFO: Checking postgres backend postgres:*/
+-- m/^INFO: Checking postgres backend.*/
+-- m/b.*COMMAND.*NODE.*NAME.*/
+-- m/b.*postgres.*TCP.*/
+-- m/b.*postgres.*UDP.*/
-- end_matchignore
CREATE FUNCTION check_motion_sockets()
RETURNS VOID as $$
-import psutil, socket
+import subprocess, re, os, socket
# Create a temporary table to create a gang
plpy.execute("CREATE TEMP TABLE motion_socket_force_create_gang(i int);")
@@ -22,10 +25,10 @@ res = plpy.execute("SELECT current_setting('gp_interconnect_type');", 1)
ic_type = res[0]['current_setting']
if ic_type in ['tcp', 'proxy']:
expected_socket_count_per_segment = 1
- expected_socket_kind = socket.SocketKind.SOCK_STREAM
+ expected_socket_kind = "TCP"
elif ic_type=='udpifc':
expected_socket_count_per_segment = 2
- expected_socket_kind = socket.SocketKind.SOCK_DGRAM
+ expected_socket_kind = "UDP"
else:
plpy.error('Unrecognized gp_interconnect_type {}.'.format(ic_type))
@@ -40,18 +43,28 @@ pids_to_check = [r['pid'] for r in res]
for pid in pids_to_check:
# We iterate through all backends related to current session
motion_socket_count = 0
- process = psutil.Process(pid)
- plpy.info('Checking postgres backend {}'.format(process.cmdline()[0]))
- for conn in process.connections():
- if conn.type == expected_socket_kind and conn.raddr == () \
- and conn.laddr.ip == hostip:
+ # Typical 'lsof -i -nP -a -p <pid>' output:
+ # COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
+ # postgres 271688 cmulong 4u IPv4 1087560 0t0 TCP 127.0.0.1:51327 (LISTEN)
+ # postgres 271688 cmulong 9u IPv6 1077090 0t0 UDP [::1]:42130->[::1]:42130
+ # We check count of those connections which have not been established.
+ # Use the regex for example: "TCP <ip>:\d+ .*" (without '->')
+ lsof_ret = subprocess.run(["lsof", "-i", "-nP", "-a", "-p", str(pid)],
+ capture_output=True, check=True).stdout
+ plpy.info(
+ f'Checking postgres backend {pid}, ' \
+ f'lsof output:\n{os.linesep.join(map(str, lsof_ret.splitlines()))}')
+ # The first row are the column names
+ r = lsof_ret.splitlines()[1:]
+ for line in r:
+ # No '->' follows he port. The connection has not be setup yet
+ match=re.match(f".*{expected_socket_kind} {hostip}:\d+ .*", str(line))
+ if match is not None:
motion_socket_count += 1
if motion_socket_count != expected_socket_count_per_segment:
plpy.error('Expected {} motion sockets but found {}. '\
- 'For backend process {}. connections= {}'\
- .format(expected_socket_count_per_segment, process,\
- motion_socket_count, process.connections()))
+ .format(expected_socket_count_per_segment, motion_socket_count))
$$ LANGUAGE plpython3u;
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For additional commands, e-mail: commits-help@cloudberry.apache.org
(cloudberry) 05/08: Behave test cases for checking the use of TRUSTED_SHELL in gpinitsystem (#15382)
Posted by yj...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
yjhjstz pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/cloudberry.git
commit 74b7051d234fc625cbb4434228b65d9d6ef7db0c
Author: Sruthi C P <39...@users.noreply.github.com>
AuthorDate: Wed May 3 10:17:11 2023 +0530
Behave test cases for checking the use of TRUSTED_SHELL in gpinitsystem (#15382)
This commit include the behave test cases for testing the Trusted shell value is properly passing to gpcreateseg.sh from gpinitsystem
---
gpMgmt/test/behave/mgmt_utils/gpinitsystem.feature | 17 +++++++++++++++++
1 file changed, 17 insertions(+)
diff --git a/gpMgmt/test/behave/mgmt_utils/gpinitsystem.feature b/gpMgmt/test/behave/mgmt_utils/gpinitsystem.feature
index c112c7aecd..00fbd568db 100644
--- a/gpMgmt/test/behave/mgmt_utils/gpinitsystem.feature
+++ b/gpMgmt/test/behave/mgmt_utils/gpinitsystem.feature
@@ -317,3 +317,20 @@ Feature: gpinitsystem tests
Then gpinitsystem should return a return code of 0
And gpinitsystem should not print "Start Function REMOTE_EXECUTE_AND_GET_OUTPUT" to stdout
And gpinitsystem should not print "End Function REMOTE_EXECUTE_AND_GET_OUTPUT" to stdout
+
+ Scenario: gpinitsystem should pass the default value of trusted_shell properly to gpcreateseg
+ Given create demo cluster config
+ When the user runs command "gpinitsystem -a -c ../gpAux/gpdemo/clusterConfigFile -h ../gpAux/gpdemo/hostfile --ignore-warnings"
+ Then gpinitsystem should return a return code of 0
+ When the user runs command "grep -q '.*gpcreateseg\.sh.*Completed .*lalshell.*' ~/gpAdminLogs/gpinitsystem*log"
+ Then grep should return a return code of 0
+
+ Scenario: gpinitsystem should pass the changed value of trusted_shell properly to gpcreateseg
+ Given create demo cluster config
+ And the user runs command "sed -i.bak -E 's/TRUSTED_SHELL=.*/TRUSTED_SHELL=ssh/g' ../gpAux/gpdemo/clusterConfigFile"
+ When the user runs command "gpinitsystem -a -c ../gpAux/gpdemo/clusterConfigFile -h ../gpAux/gpdemo/hostfile"
+ Then gpinitsystem should return a return code of 0
+ When the user runs command "grep -q '.*gpcreateseg\.sh.*Completed ssh.*' ~/gpAdminLogs/gpinitsystem*log"
+ Then grep should return a return code of 0
+ And the user runs command "mv ../gpAux/gpdemo/clusterConfigFile.bak ../gpAux/gpdemo/clusterConfigFile"
+
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For additional commands, e-mail: commits-help@cloudberry.apache.org