You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cloudberry.apache.org by ma...@apache.org on 2025/01/10 02:28:15 UTC

(cloudberry) branch main updated (0b987de603 -> 1533deccf6)

This is an automated email from the ASF dual-hosted git repository.

maxyang pushed a change to branch main
in repository https://gitbox.apache.org/repos/asf/cloudberry.git


    from 0b987de603 Fix colLargeRowIndexes array overflow
     new b5b17b9384 Add system PATH to  findCmdInPath (#15477)
     new 088cc7749e using 'env perl' instead of absolute path (#15493)
     new 14c8244695 Fix the legacy bug of the DatabaseFrozenIds lock (#15414)
     new ec848dee4a Improve refresh materialized view with "no data" option
     new 46b79fd1f0 mem intensive agg should contains mixed type.
     new 80d004563e resove fixme for lateral left join in partition_join.sql (#15492)
     new 218df8193f Fix erroneous Valgrind markings in AllocSetRealloc.
     new 97dfb96b57 Fix bypass catalog unittest (#15499)
     new ab220b7b08 modify partition_join.out to fix pipeline (#15502)
     new 189aaf7a79 Fix spelling errors identified by lintian (#15483)
     new a48ec0258f Add test case for over-eager constraint exclusion (7X) (#15476)
     new 8c1b1367de FIX BUG: unrecognized node type: 147 (7X) (#15495)
     new f4a9a6f6be fix GPDB_96_MERGE_FIXME, do not need to check forceDistRandom in set_cte_pathlist (#15527)
     new 1f7de781f4 Fix up stale gp_default_storage_options comment (#15518)
     new 623db83b8e Using __gp_aoseg/__gp_aocsseg in missing/orphaned file views
     new 29ee40bb38 Exclude views from missing file view check
     new 87f3d033f3 Escape database name for dbconn.
     new 1533deccf6 Fix cherry-pick.

The 18 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 gpMgmt/bin/gppylib/commands/unix.py                |  3 ++
 gpcontrib/gp_toolkit/gp_toolkit--1.0.sql           | 61 ++++++++++++++--------
 src/backend/commands/matview.c                     | 15 ++++++
 src/backend/commands/tablecmds.c                   |  2 +-
 src/backend/commands/vacuum.c                      |  9 ++++
 src/backend/optimizer/path/allpaths.c              |  1 -
 src/backend/parser/parse_expr.c                    | 13 +++--
 src/backend/parser/parse_relation.c                |  2 +-
 src/backend/utils/misc/guc_gp.c                    |  4 +-
 src/backend/utils/mmgr/aset.c                      | 59 +++++++++++++--------
 src/backend/utils/resgroup/test/resgroup_test.c    |  6 ++-
 src/backend/utils/resource_manager/memquota.c      |  4 +-
 src/include/catalog/duplicate_oids                 |  2 +-
 .../expected/segwalrep/recoverseg_from_file.out    |  2 +-
 .../sql/segwalrep/recoverseg_from_file.sql         |  2 +-
 src/test/regress/expected/alter_table.out          |  6 +++
 src/test/regress/expected/bfv_cte.out              | 18 +++++++
 src/test/regress/expected/constraints_check.out    | 22 ++++++++
 src/test/regress/expected/matview.out              | 29 ++++++++++
 src/test/regress/expected/matview_optimizer.out    | 29 ++++++++++
 src/test/regress/expected/partition_expand.out     |  6 +--
 src/test/regress/expected/partition_join.out       | 14 +++--
 src/test/regress/expected/tidscan.out              | 26 ++++++---
 src/test/regress/expected/tidscan_optimizer.out    | 26 ++++++---
 src/test/regress/input/dispatch.source             | 11 ++--
 src/test/regress/output/dispatch.source            | 10 ++--
 src/test/regress/parallel_schedule                 |  3 ++
 src/test/regress/sql/alter_table.sql               |  7 +++
 src/test/regress/sql/bfv_cte.sql                   | 15 ++++++
 src/test/regress/sql/constraints_check.sql         | 13 +++++
 src/test/regress/sql/matview.sql                   | 17 ++++++
 src/test/regress/sql/partition_join.sql            | 14 +++--
 src/test/regress/sql/tidscan.sql                   | 17 +++---
 33 files changed, 365 insertions(+), 103 deletions(-)
 create mode 100644 src/test/regress/expected/constraints_check.out
 create mode 100644 src/test/regress/sql/constraints_check.sql


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For additional commands, e-mail: commits-help@cloudberry.apache.org


(cloudberry) 02/18: using 'env perl' instead of absolute path (#15493)

Posted by ma...@apache.org.
This is an automated email from the ASF dual-hosted git repository.

maxyang pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/cloudberry.git

commit 088cc7749e9b7d97119c53f5f3e49eb66139bad1
Author: RMT <d....@outlook.com>
AuthorDate: Mon May 8 17:05:33 2023 +0800

    using 'env perl' instead of absolute path (#15493)
    
    Use "env perl" instead of absolute path in duplicate_oids.
    
    Some environments, such as nix, there is no "/usr/bin/perl",
    but can find perl in PATH using 'env' command. And the 'duplicate_oids'
    will be executed when compile, so i think that use 'env' is a good choice
    for portable dev environments.
---
 src/include/catalog/duplicate_oids | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/src/include/catalog/duplicate_oids b/src/include/catalog/duplicate_oids
index b4e4c064a2..c34c0378f0 100755
--- a/src/include/catalog/duplicate_oids
+++ b/src/include/catalog/duplicate_oids
@@ -1,4 +1,4 @@
-#!/usr/bin/perl
+#!/usr/bin/env perl
 #----------------------------------------------------------------------
 #
 # duplicate_oids


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For additional commands, e-mail: commits-help@cloudberry.apache.org


(cloudberry) 18/18: Fix cherry-pick.

Posted by ma...@apache.org.
This is an automated email from the ASF dual-hosted git repository.

maxyang pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/cloudberry.git

commit 1533deccf6a42e8cdfeaf4e270489cbcb8a047ca
Author: Zhang Mingli <av...@gmail.com>
AuthorDate: Thu Jan 9 11:26:16 2025 +0800

    Fix cherry-pick.
    
    Authored-by: Zhang Mingli avamingli@gmail.com
---
 src/backend/commands/vacuum.c | 9 +++++++--
 1 file changed, 7 insertions(+), 2 deletions(-)

diff --git a/src/backend/commands/vacuum.c b/src/backend/commands/vacuum.c
index 622d612fa4..47d203d12c 100644
--- a/src/backend/commands/vacuum.c
+++ b/src/backend/commands/vacuum.c
@@ -2003,8 +2003,13 @@ vac_update_datfrozenxid(void)
 	/* chicken out if bogus data found */
 	if (bogus)
 	{
-		/* Cherry-pick from GPDB FIXME: is it right for PG14? */
-		UnLockDatabaseFrozenIds(ExclusiveLock);
+		/*
+		 * Cherry-pick from GPDB FIXME: is it right for PG14?
+		 * Ignore this for error.
+		 */
+#if 0
+		 UnLockDatabaseFrozenIds(ExclusiveLock);
+#endif
 		return;
 	}
 


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For additional commands, e-mail: commits-help@cloudberry.apache.org


(cloudberry) 11/18: Add test case for over-eager constraint exclusion (7X) (#15476)

Posted by ma...@apache.org.
This is an automated email from the ASF dual-hosted git repository.

maxyang pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/cloudberry.git

commit a48ec0258f692a5272a690e02f7abaef5498dd5b
Author: Yongtao Huang <99...@users.noreply.github.com>
AuthorDate: Wed May 10 08:15:34 2023 +0800

    Add test case for over-eager constraint exclusion (7X) (#15476)
    
    Long log:
    As the title said, add test case for over-eager constraint exclusion. Please refer to [issue 10287 comment](https://github.com/greenplum-db/gpdb/issues/10287#issuecomment-1333659985)
    
    **Signed-off-by**: Yongtao Huang <yo...@vmware.com>
---
 src/test/regress/expected/constraints_check.out | 22 ++++++++++++++++++++++
 src/test/regress/parallel_schedule              |  3 +++
 src/test/regress/sql/constraints_check.sql      | 13 +++++++++++++
 3 files changed, 38 insertions(+)

diff --git a/src/test/regress/expected/constraints_check.out b/src/test/regress/expected/constraints_check.out
new file mode 100644
index 0000000000..e65691e286
--- /dev/null
+++ b/src/test/regress/expected/constraints_check.out
@@ -0,0 +1,22 @@
+-- loose over-eager constraint exclusion
+-- please refer to https://github.com/greenplum-db/gpdb/issues/10287
+CREATE TABLE t_issue_10287(a INT CHECK(a = 1));
+INSERT INTO t_issue_10287 VALUES (NULL);
+SELECT * FROM t_issue_10287 WHERE a IS NULL;
+ a 
+---
+  
+(1 row)
+
+DROP TABLE t_issue_10287;
+CREATE FUNCTION sum2_issue_10287(int8, int8) RETURNS int8 AS 'select $1 + $2' LANGUAGE SQL;
+CREATE TABLE t_issue_10287_func(a INT, b INT, c INT CHECK(sum2_issue_10287(a, b)=3));
+INSERT INTO t_issue_10287_func VALUES (1,2,3), (NULL,NULL,3);
+SELECT * FROM t_issue_10287_func WHERE a IS NULL;
+ a | b | c 
+---+---+---
+   |   | 3
+(1 row)
+
+DROP TABLE t_issue_10287_func;
+DROP FUNCTION sum2_issue_10287;
diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule
index 34f917041d..7fd9bea920 100644
--- a/src/test/regress/parallel_schedule
+++ b/src/test/regress/parallel_schedule
@@ -174,6 +174,9 @@ test: event_trigger oidjoins
 # this test also uses event triggers, so likewise run it by itself
 test: fast_default
 
+# loose over-eager constraint exclusion check, please refer to issue 10287
+test: constraints_check
+
 # run stats by itself because its delay may be insufficient under heavy load
 test: stats
 
diff --git a/src/test/regress/sql/constraints_check.sql b/src/test/regress/sql/constraints_check.sql
new file mode 100644
index 0000000000..2b2d481530
--- /dev/null
+++ b/src/test/regress/sql/constraints_check.sql
@@ -0,0 +1,13 @@
+-- loose over-eager constraint exclusion
+-- please refer to https://github.com/greenplum-db/gpdb/issues/10287
+CREATE TABLE t_issue_10287(a INT CHECK(a = 1));
+INSERT INTO t_issue_10287 VALUES (NULL);
+SELECT * FROM t_issue_10287 WHERE a IS NULL;
+DROP TABLE t_issue_10287;
+
+CREATE FUNCTION sum2_issue_10287(int8, int8) RETURNS int8 AS 'select $1 + $2' LANGUAGE SQL;
+CREATE TABLE t_issue_10287_func(a INT, b INT, c INT CHECK(sum2_issue_10287(a, b)=3));
+INSERT INTO t_issue_10287_func VALUES (1,2,3), (NULL,NULL,3);
+SELECT * FROM t_issue_10287_func WHERE a IS NULL;
+DROP TABLE t_issue_10287_func;
+DROP FUNCTION sum2_issue_10287;


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For additional commands, e-mail: commits-help@cloudberry.apache.org


(cloudberry) 04/18: Improve refresh materialized view with "no data" option

Posted by ma...@apache.org.
This is an automated email from the ASF dual-hosted git repository.

maxyang pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/cloudberry.git

commit ec848dee4ac35157a90f9e186ddcdcd4a72458be
Author: wenru yan <yw...@vmware.com>
AuthorDate: Mon May 8 04:10:39 2023 +0000

    Improve refresh materialized view with "no data" option
    
    In postgres, with option "WITH NO DATA", the materialized view will be
    truncated without run the query.
    
    In GPDB, the refresh clause is dispatched to segments when execute the
    query plan. But for WITH NO DATA option, it should be like a TRUNCATE
    like in postgres, so it doesn't need to take long time to run the query.
    
    By adding an constant-FALSE to the qual to generate a result plan with
    One-Time Filter. We have hold matview's distribution policy into the
    query->intoPolicy, planner will create a motion node on the top so that
    the refresh clause will still dispatch to segments with the query plan
    and the query can be execute quickly.
---
 src/backend/commands/matview.c                  | 15 +++++++++++++
 src/test/regress/expected/matview.out           | 29 +++++++++++++++++++++++++
 src/test/regress/expected/matview_optimizer.out | 29 +++++++++++++++++++++++++
 src/test/regress/sql/matview.sql                | 17 +++++++++++++++
 4 files changed, 90 insertions(+)

diff --git a/src/backend/commands/matview.c b/src/backend/commands/matview.c
index 8448d2335f..a0180ce041 100644
--- a/src/backend/commands/matview.c
+++ b/src/backend/commands/matview.c
@@ -796,6 +796,21 @@ refresh_matview_datafill(DestReceiver *dest, Query *query,
 	if (list_length(rewritten) != 1)
 		elog(ERROR, "unexpected rewrite result for REFRESH MATERIALIZED VIEW");
 	query = (Query *) linitial(rewritten);
+	/*
+	 * In GPDB, the refresh clause is dispatched to segments when execute the query plan.
+	 * But for WITH NO DATA option, it's effectively like a TRUNCATE, so it doesn't need
+	 * to take a long time to run the query.
+	 *
+	 * Add a constant-FALSE to the qual to simulate a plan like this, dispatch the refresh
+	 * clause without run the long query:
+	 * Motion
+	 * 	Result  (cost=0.00..0.01 rows=1 width=0)
+	 * 	  One-Time Filter: false
+	 * Planner create the motion node on the top according to the matview's distribution
+	 * policy in the query->intoPolicy.
+	 */
+	if (refreshClause->skipData)
+		query->jointree->quals = (Node *) makeBoolConst(false, false);
 
 	/* Check for user-requested abort. */
 	CHECK_FOR_INTERRUPTS();
diff --git a/src/test/regress/expected/matview.out b/src/test/regress/expected/matview.out
index f0b1ad9c3d..22db3edd46 100644
--- a/src/test/regress/expected/matview.out
+++ b/src/test/regress/expected/matview.out
@@ -780,3 +780,32 @@ select count(*) from mv_fast_relfilenode_1 natural join mv_fast_relfilenode_2;
 
 reset gp_enable_refresh_fast_path;
 drop materialized view mv_fast;
+-- test REFRESH MATERIALIZED VIEW with 'WITH NO DATA' option can be executed immediately.
+DROP TABLE IF EXISTS mvtest_twn;
+CREATE TABLE mvtest_twn(a int);
+CREATE MATERIALIZED VIEW mat_view_twn as SELECT a.a as p, b.a as q, c.a as x, d.a as y FROM mvtest_twn a, mvtest_twn b, mvtest_twn c, mvtest_twn d;
+INSERT INTO mvtest_twn SELECT i FROM generate_series(1,10000)i;
+-- t1 contains 10000 tuples, after cross join it four times, the output is much too huge
+-- refresh with 'no data' should not actually execute the sql
+set statement_timeout = 5000;
+REFRESH MATERIALIZED VIEW mat_view_twn WITH NO DATA;
+reset statement_timeout;
+SELECT relispopulated FROM pg_class WHERE oid = 'mat_view_twn'::regclass;
+ relispopulated 
+----------------
+ f
+(1 row)
+
+SELECT relispopulated FROM gp_dist_random('pg_class') WHERE oid = 'mat_view_twn'::regclass;
+ relispopulated 
+----------------
+ f
+ f
+ f
+(3 rows)
+
+SELECT * FROM mat_view_twn;
+ERROR:  materialized view "mat_view_twn" has not been populated
+HINT:  Use the REFRESH MATERIALIZED VIEW command.
+DROP MATERIALIZED VIEW mat_view_twn;
+DROP TABLE mvtest_twn;
diff --git a/src/test/regress/expected/matview_optimizer.out b/src/test/regress/expected/matview_optimizer.out
index a0c61de98d..e5fd0ba751 100644
--- a/src/test/regress/expected/matview_optimizer.out
+++ b/src/test/regress/expected/matview_optimizer.out
@@ -797,3 +797,32 @@ select count(*) from mv_fast_relfilenode_1 natural join mv_fast_relfilenode_2;
 
 reset gp_enable_refresh_fast_path;
 drop materialized view mv_fast;
+-- test REFRESH MATERIALIZED VIEW with 'WITH NO DATA' option can be executed immediately.
+DROP TABLE IF EXISTS mvtest_twn;
+CREATE TABLE mvtest_twn(a int);
+CREATE MATERIALIZED VIEW mat_view_twn as SELECT a.a as p, b.a as q, c.a as x, d.a as y FROM mvtest_twn a, mvtest_twn b, mvtest_twn c, mvtest_twn d;
+INSERT INTO mvtest_twn SELECT i FROM generate_series(1,10000)i;
+-- t1 contains 10000 tuples, after cross join it four times, the output is much too huge
+-- refresh with 'no data' should not actually execute the sql
+set statement_timeout = 5000;
+REFRESH MATERIALIZED VIEW mat_view_twn WITH NO DATA;
+reset statement_timeout;
+SELECT relispopulated FROM pg_class WHERE oid = 'mat_view_twn'::regclass;
+ relispopulated 
+----------------
+ f
+(1 row)
+
+SELECT relispopulated FROM gp_dist_random('pg_class') WHERE oid = 'mat_view_twn'::regclass;
+ relispopulated 
+----------------
+ f
+ f
+ f
+(3 rows)
+
+SELECT * FROM mat_view_twn;
+ERROR:  materialized view "mat_view_twn" has not been populated
+HINT:  Use the REFRESH MATERIALIZED VIEW command.
+DROP MATERIALIZED VIEW mat_view_twn;
+DROP TABLE mvtest_twn;
diff --git a/src/test/regress/sql/matview.sql b/src/test/regress/sql/matview.sql
index ead969f8e4..b95d34e96f 100644
--- a/src/test/regress/sql/matview.sql
+++ b/src/test/regress/sql/matview.sql
@@ -341,3 +341,20 @@ select count(*) from mv_fast_relfilenode_1 natural join mv_fast_relfilenode_2;
 
 reset gp_enable_refresh_fast_path;
 drop materialized view mv_fast;
+
+-- test REFRESH MATERIALIZED VIEW with 'WITH NO DATA' option can be executed immediately.
+DROP TABLE IF EXISTS mvtest_twn;
+CREATE TABLE mvtest_twn(a int);
+CREATE MATERIALIZED VIEW mat_view_twn as SELECT a.a as p, b.a as q, c.a as x, d.a as y FROM mvtest_twn a, mvtest_twn b, mvtest_twn c, mvtest_twn d;
+INSERT INTO mvtest_twn SELECT i FROM generate_series(1,10000)i;
+-- t1 contains 10000 tuples, after cross join it four times, the output is much too huge
+-- refresh with 'no data' should not actually execute the sql
+set statement_timeout = 5000;
+REFRESH MATERIALIZED VIEW mat_view_twn WITH NO DATA;
+reset statement_timeout;
+SELECT relispopulated FROM pg_class WHERE oid = 'mat_view_twn'::regclass;
+SELECT relispopulated FROM gp_dist_random('pg_class') WHERE oid = 'mat_view_twn'::regclass;
+SELECT * FROM mat_view_twn;
+
+DROP MATERIALIZED VIEW mat_view_twn;
+DROP TABLE mvtest_twn;


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For additional commands, e-mail: commits-help@cloudberry.apache.org


(cloudberry) 06/18: resove fixme for lateral left join in partition_join.sql (#15492)

Posted by ma...@apache.org.
This is an automated email from the ASF dual-hosted git repository.

maxyang pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/cloudberry.git

commit 80d004563e7d6771c9f325d280005f76f72c3aec
Author: xuejing zhao <80...@users.noreply.github.com>
AuthorDate: Tue May 9 13:55:14 2023 +0800

    resove fixme for lateral left join in partition_join.sql (#15492)
    
    GPDB does not support the Lateral feature officially.
    some of the lateral features do work well, such as:
    ```
    create table t1(Tc1 int, tc2 int);
    create table t2(Tc3 int, tc4 int);
    explain SELECT * FROM t1 LEFT JOIN LATERAL
                                     (SELECT a.tc3, least(t1.tc1,a.tc3) from t2 a) ss
                                     ON t1.tc1=ss.tc3;
    ```
    we specify the keyword LATERAL to indicate in subquery we can use the column of t1.
    Above join do not need motion between t1 and ss. So it can work well.
    
    However, some of the lateral features do not work well, such as the following sql:
    ```
    create table t1(Tc1 int, tc2 int);
    create table t2(Tc3 int, tc4 int);
    
    explain SELECT * FROM t1 LEFT JOIN LATERAL
                                     (SELECT a.tc3, a.tc4, least(t1.tc1, a.tc3) from t2 a) ss
                                     ON t1.tc2=ss.tc4;
    ERROR:  could not devise a query plan for the given query (pathnode.c:275)
    ```
    If the inner path uses column of outer path, We can not add motion above inner path.
    however, in left join, outer path can not be replicated. We can not generate path for left join,
    So hit the error.
    
    Change the GPDB_12_MERGE_FIXME to GPDB_12_MERGE_FEATURE_NOT_SUPPORTED for
    lateral test in partition_join.sql
---
 src/test/regress/sql/partition_join.sql | 14 ++++++++++----
 1 file changed, 10 insertions(+), 4 deletions(-)

diff --git a/src/test/regress/sql/partition_join.sql b/src/test/regress/sql/partition_join.sql
index 1d188d1476..24baa458b1 100644
--- a/src/test/regress/sql/partition_join.sql
+++ b/src/test/regress/sql/partition_join.sql
@@ -83,8 +83,11 @@ SELECT sum(t1.a), avg(t1.a), sum(t1.b), avg(t1.b) FROM prt1 t1 WHERE NOT EXISTS
 SELECT sum(t1.a), avg(t1.a), sum(t1.b), avg(t1.b) FROM prt1 t1 WHERE NOT EXISTS (SELECT 1 FROM prt2 t2 WHERE t1.a = t2.b);
 
 -- lateral reference
--- GPDB_12_MERGE_FIXME: this is failing in GPDB. Known issue, there were similar
--- failures in other LATERAL tests before the v12 merge.
+-- GPDB_12_MERGE_FEATURE_NOT_SUPPORTED
+-- If lateral_relids of RelOptInfo for a inner path is not null, the inner path needs
+-- params of outer path. So We can not add motion above inner path.
+-- however, in left join, outer path can not be replicated. We can not generate path for left join,
+-- So hit the error: could not devise a query plan for the given query.
 EXPLAIN (COSTS OFF)
 SELECT * FROM prt1 t1 LEFT JOIN LATERAL
 			  (SELECT t2.a AS t2a, t3.a AS t3a, least(t1.a,t2.a,t3.b) FROM prt1 t2 JOIN prt2 t3 ON (t2.a = t3.b)) ss
@@ -380,8 +383,11 @@ SELECT t1.a, t1.c, t2.b, t2.c FROM (SELECT * FROM prt1_l WHERE prt1_l.b = 0) t1
 SELECT t1.a, t1.c, t2.b, t2.c FROM (SELECT * FROM prt1_l WHERE prt1_l.b = 0) t1 FULL JOIN (SELECT * FROM prt2_l WHERE prt2_l.a = 0) t2 ON (t1.a = t2.b AND t1.c = t2.c) ORDER BY t1.a, t2.b;
 
 -- lateral partitionwise join
--- GPDB_12_MERGE_FIXME: this is failing in GPDB. Known issue, there were similar
--- failures in other LATERAL tests before the v12 merge.
+-- GPDB_12_MERGE_FEATURE_NOT_SUPPORTED
+-- If lateral_relids of RelOptInfo for a inner path is not null, the inner path needs
+-- params of outer path. So We can not add motion above inner path.
+-- however, in left join, outer path can not be replicated. We can not generate path for left join,
+-- So hit the error: could not devise a query plan for the given query.
 EXPLAIN (COSTS OFF)
 SELECT * FROM prt1_l t1 LEFT JOIN LATERAL
 			  (SELECT t2.a AS t2a, t2.c AS t2c, t2.b AS t2b, t3.b AS t3b, least(t1.a,t2.a,t3.b) FROM prt1_l t2 JOIN prt2_l t3 ON (t2.a = t3.b AND t2.c = t3.c)) ss


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For additional commands, e-mail: commits-help@cloudberry.apache.org


(cloudberry) 12/18: FIX BUG: unrecognized node type: 147 (7X) (#15495)

Posted by ma...@apache.org.
This is an automated email from the ASF dual-hosted git repository.

maxyang pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/cloudberry.git

commit 8c1b1367de161321a3613a7ef3688001d39114a0
Author: Yongtao Huang <99...@users.noreply.github.com>
AuthorDate: Wed May 10 20:04:39 2023 +0800

    FIX BUG: unrecognized node type: 147 (7X) (#15495)
    
    Long log:
    
    This commit can fix but reported in
    [issue 15494](https://github.com/greenplum-db/gpdb/issues/15494)
    The solution is similar with
    [pr 15184](https://github.com/greenplum-db/gpdb/pull/15184)
    
    We define another new var of NullTest to make sure
    that the original parameters NullTest don't change,
    just as transformCollateClause() does.
    
    Signed-off-by: Yongtao Huang <yo...@vmware.com>
---
 src/backend/parser/parse_expr.c           | 13 ++++++++++---
 src/test/regress/expected/alter_table.out |  6 ++++++
 src/test/regress/sql/alter_table.sql      |  7 +++++++
 3 files changed, 23 insertions(+), 3 deletions(-)

diff --git a/src/backend/parser/parse_expr.c b/src/backend/parser/parse_expr.c
index fcbbf8a850..d8704c7253 100644
--- a/src/backend/parser/parse_expr.c
+++ b/src/backend/parser/parse_expr.c
@@ -274,10 +274,17 @@ transformExprRecurse(ParseState *pstate, Node *expr)
 			{
 				NullTest   *n = (NullTest *) expr;
 
-				n->arg = (Expr *) transformExprRecurse(pstate, (Node *) n->arg);
+				/* please refer to https://github.com/greenplum-db/gpdb/issues/15494 */
+				NullTest *newn;
+				newn = makeNode(NullTest);
+
+				newn->arg = (Expr *) transformExprRecurse(pstate, (Node *) n->arg);
 				/* the argument can be any type, so don't coerce it */
-				n->argisrow = type_is_rowtype(exprType((Node *) n->arg));
-				result = expr;
+				newn->argisrow = type_is_rowtype(exprType((Node *) newn->arg));
+				newn->nulltesttype = n->nulltesttype;
+				newn->location = n->location;
+
+				result = (Node *) newn;
 				break;
 			}
 
diff --git a/src/test/regress/expected/alter_table.out b/src/test/regress/expected/alter_table.out
index c891e6f414..660309425d 100644
--- a/src/test/regress/expected/alter_table.out
+++ b/src/test/regress/expected/alter_table.out
@@ -4745,3 +4745,9 @@ DETAIL:  Failing row contains (2, 1).
 -- ...and doesn't when the partition is detached along with its own partition
 alter table target_parted detach partition attach_parted;
 insert into attach_parted_part1 values (2, 1);
+CREATE TABLE IF NOT EXISTS table_issue_15494(c0 boolean NULL);
+ALTER TABLE table_issue_15494 ALTER c0 SET DEFAULT (6>5) IS NULL;
+DROP TABLE table_issue_15494;
+CREATE TABLE IF NOT EXISTS table_issue_15494(c0 boolean);
+ALTER TABLE table_issue_15494 ALTER c0 SET DEFAULT ((1.5::FLOAT) NOTNULL);
+DROP TABLE table_issue_15494;
diff --git a/src/test/regress/sql/alter_table.sql b/src/test/regress/sql/alter_table.sql
index 8603172dd6..8cfd07946c 100644
--- a/src/test/regress/sql/alter_table.sql
+++ b/src/test/regress/sql/alter_table.sql
@@ -3113,3 +3113,10 @@ insert into attach_parted_part1 values (2, 1);
 -- ...and doesn't when the partition is detached along with its own partition
 alter table target_parted detach partition attach_parted;
 insert into attach_parted_part1 values (2, 1);
+
+CREATE TABLE IF NOT EXISTS table_issue_15494(c0 boolean NULL);
+ALTER TABLE table_issue_15494 ALTER c0 SET DEFAULT (6>5) IS NULL;
+DROP TABLE table_issue_15494;
+CREATE TABLE IF NOT EXISTS table_issue_15494(c0 boolean);
+ALTER TABLE table_issue_15494 ALTER c0 SET DEFAULT ((1.5::FLOAT) NOTNULL);
+DROP TABLE table_issue_15494;


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For additional commands, e-mail: commits-help@cloudberry.apache.org


(cloudberry) 07/18: Fix erroneous Valgrind markings in AllocSetRealloc.

Posted by ma...@apache.org.
This is an automated email from the ASF dual-hosted git repository.

maxyang pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/cloudberry.git

commit 218df8193f9607b9f6bfc21a870f2abc5eff4dbf
Author: Tom Lane <tg...@sss.pgh.pa.us>
AuthorDate: Tue Feb 21 18:47:47 2023 -0500

    Fix erroneous Valgrind markings in AllocSetRealloc.
    
    If asked to decrease the size of a large (>8K) palloc chunk,
    AllocSetRealloc could improperly change the Valgrind state of memory
    beyond the new end of the chunk: it would mark data UNDEFINED as far
    as the old end of the chunk after having done the realloc(3) call,
    thus tromping on the state of memory that no longer belongs to it.
    One would normally expect that memory to now be marked NOACCESS,
    so that this mislabeling might prevent detection of later errors.
    If realloc() had chosen to move the chunk someplace else (unlikely,
    but well within its rights) we could also mismark perfectly-valid
    DEFINED data as UNDEFINED, causing false-positive valgrind reports
    later.  Also, any malloc bookkeeping data placed within this area
    might now be wrongly marked, causing additional problems.
    
    Fix by replacing relevant uses of "oldsize" with "Min(size, oldsize)".
    It's sufficient to mark as far as "size" when that's smaller, because
    whatever remains in the new chunk size will be marked NOACCESS below,
    and we expect realloc() to have taken care of marking the memory
    beyond the new official end of the chunk.
    
    While we're here, also rename the function's "oldsize" variable
    to "oldchksize" to more clearly explain what it actually holds,
    namely the distance to the end of the chunk (that is, requested size
    plus trailing padding).  This is more consistent with the use of
    "size" and "chksize" to hold the new requested size and chunk size.
    Add a new variable "oldsize" in the one stanza where we're actually
    talking about the old requested size.
    
    Oversight in commit c477f3e44.  Back-patch to all supported branches,
    as that was, just in case anybody wants to do valgrind testing on back
    branches.
    
    Karina Litskevich
    
    Discussion: https://postgr.es/m/CACiT8iaAET-fmzjjZLjaJC4zwSJmrFyL7LAdHwaYyjjQOQ4hcg@mail.gmail.com
---
 src/backend/utils/mmgr/aset.c | 59 ++++++++++++++++++++++++++-----------------
 1 file changed, 36 insertions(+), 23 deletions(-)

diff --git a/src/backend/utils/mmgr/aset.c b/src/backend/utils/mmgr/aset.c
index 05177a3ede..71de3287f5 100644
--- a/src/backend/utils/mmgr/aset.c
+++ b/src/backend/utils/mmgr/aset.c
@@ -1267,12 +1267,12 @@ AllocSetRealloc(MemoryContext context, void *pointer, Size size)
 {
 	AllocSet	set = (AllocSet) context;
 	AllocChunk	chunk = AllocPointerGetChunk(pointer);
-	Size		oldsize;
+	Size		oldchksize;
 
 	/* Allow access to private part of chunk header. */
 	VALGRIND_MAKE_MEM_DEFINED(chunk, ALLOCCHUNK_PRIVATE_LEN);
 
-	oldsize = chunk->size;
+	oldchksize = chunk->size;
 
 #ifdef USE_ASSERT_CHECKING
 	if (IsUnderPostmaster  && context != ErrorContext && mainthread() != 0 && !pthread_equal(main_tid, pthread_self()))
@@ -1287,13 +1287,13 @@ AllocSetRealloc(MemoryContext context, void *pointer, Size size)
 
 #ifdef MEMORY_CONTEXT_CHECKING
 	/* Test for someone scribbling on unused space in chunk */
-	if (chunk->requested_size < oldsize)
+	if (chunk->requested_size < oldchksize)
 		if (!sentinel_ok(pointer, chunk->requested_size))
 			elog(WARNING, "detected write past chunk end in %s %p",
 				 set->header.name, chunk);
 #endif
 
-	if (oldsize > set->allocChunkLimit)
+	if (oldchksize > set->allocChunkLimit)
 	{
 		/*
 		 * The chunk must have been allocated as a single-chunk block.  Use
@@ -1313,7 +1313,7 @@ AllocSetRealloc(MemoryContext context, void *pointer, Size size)
 		if (block->aset != set ||
 			block->freeptr != block->endptr ||
 			block->freeptr != ((char *) block) +
-			(oldsize + ALLOC_BLOCKHDRSZ + ALLOC_CHUNKHDRSZ))
+			(oldchksize + ALLOC_BLOCKHDRSZ + ALLOC_CHUNKHDRSZ))
 			elog(ERROR, "could not find block containing chunk %p", chunk);
 
 		/*
@@ -1356,21 +1356,29 @@ AllocSetRealloc(MemoryContext context, void *pointer, Size size)
 
 #ifdef MEMORY_CONTEXT_CHECKING
 #ifdef RANDOMIZE_ALLOCATED_MEMORY
-		/* We can only fill the extra space if we know the prior request */
+
+		/*
+		 * We can only randomize the extra space if we know the prior request.
+		 * When using Valgrind, randomize_mem() also marks memory UNDEFINED.
+		 */
 		if (size > chunk->requested_size)
 			randomize_mem((char *) pointer + chunk->requested_size,
 						  size - chunk->requested_size);
-#endif
+#else
 
 		/*
-		 * realloc() (or randomize_mem()) will have left any newly-allocated
-		 * part UNDEFINED, but we may need to adjust trailing bytes from the
-		 * old allocation.
+		 * If this is an increase, realloc() will have marked any
+		 * newly-allocated part (from oldchksize to chksize) UNDEFINED, but we
+		 * also need to adjust trailing bytes from the old allocation (from
+		 * chunk->requested_size to oldchksize) as they are marked NOACCESS.
+		 * Make sure not to mark too many bytes in case chunk->requested_size
+		 * < size < oldchksize.
 		 */
 #ifdef USE_VALGRIND
-		if (oldsize > chunk->requested_size)
+		if (Min(size, oldchksize) > chunk->requested_size)
 			VALGRIND_MAKE_MEM_UNDEFINED((char *) pointer + chunk->requested_size,
-										oldsize - chunk->requested_size);
+										Min(size, oldchksize) - chunk->requested_size);
+#endif
 #endif
 
 		chunk->requested_size = size;
@@ -1381,11 +1389,14 @@ AllocSetRealloc(MemoryContext context, void *pointer, Size size)
 #else							/* !MEMORY_CONTEXT_CHECKING */
 
 		/*
-		 * We don't know how much of the old chunk size was the actual
-		 * allocation; it could have been as small as one byte.  We have to be
-		 * conservative and just mark the entire old portion DEFINED.
+		 * We may need to adjust marking of bytes from the old allocation as
+		 * some of them may be marked NOACCESS.  We don't know how much of the
+		 * old chunk size was the requested size; it could have been as small
+		 * as one byte.  We have to be conservative and just mark the entire
+		 * old portion DEFINED.  Make sure not to mark memory beyond the new
+		 * allocation in case it's smaller than the old one.
 		 */
-		VALGRIND_MAKE_MEM_DEFINED(pointer, oldsize);
+		VALGRIND_MAKE_MEM_DEFINED(pointer, Min(size, oldchksize));
 #endif
 
 		/* Ensure any padding bytes are marked NOACCESS. */
@@ -1394,10 +1405,10 @@ AllocSetRealloc(MemoryContext context, void *pointer, Size size)
 		/* Disallow external access to private part of chunk header. */
 		VALGRIND_MAKE_MEM_NOACCESS(chunk, ALLOCCHUNK_PRIVATE_LEN);
 
-		if (chksize > oldsize)
-			MEMORY_ACCOUNT_INC_ALLOCATED(set, chksize - oldsize);
+		if (chksize > oldchksize)
+			MEMORY_ACCOUNT_INC_ALLOCATED(set, chksize - oldchksize);
 		else
-			MEMORY_ACCOUNT_DEC_ALLOCATED(set, oldsize - chksize);
+			MEMORY_ACCOUNT_DEC_ALLOCATED(set, oldchksize - chksize);
 
 		return pointer;
 	}
@@ -1407,7 +1418,7 @@ AllocSetRealloc(MemoryContext context, void *pointer, Size size)
 	 * allocated area already is >= the new size.  (In particular, we will
 	 * fall out here if the requested size is a decrease.)
 	 */
-	else if (oldsize >= size)
+	else if (oldchksize >= size)
 	{
 #ifdef MEMORY_CONTEXT_CHECKING
 		Size		oldrequest = chunk->requested_size;
@@ -1430,10 +1441,10 @@ AllocSetRealloc(MemoryContext context, void *pointer, Size size)
 										size - oldrequest);
 		else
 			VALGRIND_MAKE_MEM_NOACCESS((char *) pointer + size,
-									   oldsize - size);
+									   oldchksize - size);
 
 		/* set mark to catch clobber of "unused" space */
-		if (size < oldsize)
+		if (size < oldchksize)
 			set_sentinel(pointer, size);
 #else							/* !MEMORY_CONTEXT_CHECKING */
 
@@ -1442,7 +1453,7 @@ AllocSetRealloc(MemoryContext context, void *pointer, Size size)
 		 * the old request or shrinking it, so we conservatively mark the
 		 * entire new allocation DEFINED.
 		 */
-		VALGRIND_MAKE_MEM_NOACCESS(pointer, oldsize);
+		VALGRIND_MAKE_MEM_NOACCESS(pointer, oldchksize);
 		VALGRIND_MAKE_MEM_DEFINED(pointer, size);
 #endif
 
@@ -1470,6 +1481,7 @@ AllocSetRealloc(MemoryContext context, void *pointer, Size size)
 		 * memory indefinitely.  See pgsql-hackers archives for 2007-08-11.)
 		 */
 		AllocPointer newPointer;
+		Size		oldsize;
 
 		/* allocate new chunk */
 		newPointer = AllocSetAlloc((MemoryContext) set, size);
@@ -1494,6 +1506,7 @@ AllocSetRealloc(MemoryContext context, void *pointer, Size size)
 #ifdef MEMORY_CONTEXT_CHECKING
 		oldsize = chunk->requested_size;
 #else
+		oldsize = oldchksize;
 		VALGRIND_MAKE_MEM_DEFINED(pointer, oldsize);
 #endif
 


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For additional commands, e-mail: commits-help@cloudberry.apache.org


(cloudberry) 01/18: Add system PATH to findCmdInPath (#15477)

Posted by ma...@apache.org.
This is an automated email from the ASF dual-hosted git repository.

maxyang pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/cloudberry.git

commit b5b17b93840b7fe59f6a11b73726752f3f57767f
Author: RMT <d....@outlook.com>
AuthorDate: Sun May 7 21:29:07 2023 +0800

    Add system PATH to  findCmdInPath (#15477)
    
    Find cmd using PATH env.
    
    Current findCmdInPath only search cmd in specific CMDPATH,
    if user has some commands in other locations(but add those locations to PATH)
    and not in the CMDPATH, gpdb will cannot find it.
    
    For example, the paths in CMDPATH are not existed in NixOS, the PATH in NixOS is like this:
    “/nix/store/vr631vl97c7gfsy1zdwpj60zzxnnz8v6-openssh-9.2p1/bin:
    /nix/store/zbr5i1jzfaj29v62g9r9xlibhv8295r6-patchelf-0.15.0/bin:
    /nix/store/0h7yqpk1gd6argxfv59x1vz0zklhjcnv-gcc-wrapper-12.2.0/bin“
    
    To solve this problem, the system PATH should be added into CMDPATH.
---
 gpMgmt/bin/gppylib/commands/unix.py | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/gpMgmt/bin/gppylib/commands/unix.py b/gpMgmt/bin/gppylib/commands/unix.py
index 3f777e8409..e069654909 100644
--- a/gpMgmt/bin/gppylib/commands/unix.py
+++ b/gpMgmt/bin/gppylib/commands/unix.py
@@ -35,6 +35,9 @@ GPHOME = os.environ.get('GPHOME', None)
 # ---------------command path--------------------
 CMDPATH = ['/usr/kerberos/bin', '/usr/sfw/bin', '/opt/sfw/bin', '/bin', '/usr/local/bin',
            '/usr/bin', '/sbin', '/usr/sbin', '/usr/ucb', '/sw/bin', '/opt/Navisphere/bin']
+CMDPATH = CMDPATH + os.environ['PATH'].split(os.pathsep)
+# remove duplicate paths
+CMDPATH = list(set(CMDPATH))
 
 if GPHOME:
     CMDPATH.append(GPHOME)


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For additional commands, e-mail: commits-help@cloudberry.apache.org


(cloudberry) 14/18: Fix up stale gp_default_storage_options comment (#15518)

Posted by ma...@apache.org.
This is an automated email from the ASF dual-hosted git repository.

maxyang pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/cloudberry.git

commit 1f7de781f434dd6a2a00c6ef90693d2664471d57
Author: David Kimura <dk...@vmware.com>
AuthorDate: Fri May 12 09:28:06 2023 -0700

    Fix up stale gp_default_storage_options comment (#15518)
    
    After Postgres 12 merge the GUC to set default table type was moved to
    default_table_access_method as hinted by the error message.
    
      SET gp_default_storage_options='appendonly=true';
      ERROR:  invalid storage option "appendonly"
      HINT:  For table access methods use "default_table_access_method" instead.
    
    Instead seems one should run:
    
      SET default_table_access_method='ao_row';
    
    Remove the stale comment that suggested otherwise.
---
 src/backend/utils/misc/guc_gp.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/src/backend/utils/misc/guc_gp.c b/src/backend/utils/misc/guc_gp.c
index ff8d828569..e881e05a01 100644
--- a/src/backend/utils/misc/guc_gp.c
+++ b/src/backend/utils/misc/guc_gp.c
@@ -431,7 +431,7 @@ bool		gp_reject_internal_tcp_conn = true;
 bool		gp_enable_segment_copy_checking = true;
 /*
  * Default storage options GUC.  Value is comma-separated name=value
- * pairs.  E.g. "appendonly=true,orientation=column"
+ * pairs.  E.g. "blocksize=32768,compresstype=none,checksum=true"
  */
 char	   *gp_default_storage_options = NULL;
 


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For additional commands, e-mail: commits-help@cloudberry.apache.org


(cloudberry) 16/18: Exclude views from missing file view check

Posted by ma...@apache.org.
This is an automated email from the ASF dual-hosted git repository.

maxyang pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/cloudberry.git

commit 29ee40bb3882544aa8b2db596c75e5b9dc10950b
Author: Huansong Fu <fu...@gmail.com>
AuthorDate: Thu May 11 14:04:13 2023 -0700

    Exclude views from missing file view check
    
    We could create a view out of a table via the ON SELECT rule on the
    table. That leaves a view with valid relfilenode, but its relation file
    might've been cleaned up. Now do not show it when checking missing files.
    
    P.S. latest upstream has removed this feature (https://github.com/postgres/postgres/commit/b23cd185fd5).
    We considered backporting the same but eventually decided that it is
    too early to remove this feature which might have bigger impact to GPDB.
---
 gpcontrib/gp_toolkit/gp_toolkit--1.0.sql | 14 +++++++++++---
 1 file changed, 11 insertions(+), 3 deletions(-)

diff --git a/gpcontrib/gp_toolkit/gp_toolkit--1.0.sql b/gpcontrib/gp_toolkit/gp_toolkit--1.0.sql
index d50652ce88..9eedc52e45 100644
--- a/gpcontrib/gp_toolkit/gp_toolkit--1.0.sql
+++ b/gpcontrib/gp_toolkit/gp_toolkit--1.0.sql
@@ -2296,7 +2296,8 @@ CREATE OR REPLACE VIEW gp_toolkit.__get_expect_files AS
 SELECT s.reltablespace AS tablespace, s.relname, a.amname AS AM,
        (CASE WHEN s.relfilenode != 0 THEN s.relfilenode ELSE pg_relation_filenode(s.oid) END)::text AS filename
 FROM pg_class s
-LEFT JOIN pg_am a ON s.relam = a.oid;
+LEFT JOIN pg_am a ON s.relam = a.oid
+WHERE s.relkind != 'v'; -- view could have valid relfilenode if created from a table, but its relfile is gone
 
 GRANT SELECT ON gp_toolkit.__get_expect_files TO public;
 
@@ -2310,12 +2311,19 @@ GRANT SELECT ON gp_toolkit.__get_expect_files TO public;
 --        the extended data files for AO/CO tables.
 --        But ignore those w/ eof=0. They might be created just for
 --        modcount whereas no data has ever been inserted to the seg.
+--        Or, they could be created when a seg has only aborted rows.
+--        In both cases, we can ignore these segs, because no matter
+--        whether the data files exist or not, the rest of the system
+--        can handle them gracefully.
+--        Also exclude views which could have valid relfilenode if 
+--        created from a table, but their relfiles are gone.
 --
 --------------------------------------------------------------------------------
 CREATE OR REPLACE VIEW gp_toolkit.__get_expect_files_ext AS
 SELECT s.reltablespace AS tablespace, s.relname, a.amname AS AM,
        (CASE WHEN s.relfilenode != 0 THEN s.relfilenode ELSE pg_relation_filenode(s.oid) END)::text AS filename
 FROM pg_class s LEFT JOIN pg_am a ON s.relam = a.oid
+WHERE s.relkind != 'v'
 UNION
 -- AO extended files
 SELECT c.reltablespace AS tablespace, c.relname, a.amname AS AM,
@@ -2323,7 +2331,7 @@ SELECT c.reltablespace AS tablespace, c.relname, a.amname AS AM,
 FROM gp_toolkit.__get_ao_segno_list() s
 JOIN pg_class c ON s.relid = c.oid
 LEFT JOIN pg_am a ON c.relam = a.oid
-WHERE s.eof > 0
+WHERE s.eof > 0 AND c.relkind != 'v'
 UNION
 -- CO extended files
 SELECT c.reltablespace AS tablespace, c.relname, a.amname AS AM,
@@ -2331,7 +2339,7 @@ SELECT c.reltablespace AS tablespace, c.relname, a.amname AS AM,
 FROM gp_toolkit.__get_aoco_segno_list() s
 JOIN pg_class c ON s.relid = c.oid
 LEFT JOIN pg_am a ON c.relam = a.oid
-WHERE s.eof > 0;
+WHERE s.eof > 0 AND c.relkind != 'v';
 
 GRANT SELECT ON gp_toolkit.__get_expect_files_ext TO public;
 


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For additional commands, e-mail: commits-help@cloudberry.apache.org


(cloudberry) 03/18: Fix the legacy bug of the DatabaseFrozenIds lock (#15414)

Posted by ma...@apache.org.
This is an automated email from the ASF dual-hosted git repository.

maxyang pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/cloudberry.git

commit 14c8244695cf696009bc74caf96f33629aacad76
Author: Hongxu Ma <in...@outlook.com>
AuthorDate: Mon May 8 17:29:59 2023 +0800

    Fix the legacy bug of the DatabaseFrozenIds lock (#15414)
    
    This commit fixes an obvious legacy bug (introduced in the process of PG12 merge).
    
    But current lock behavior of DatabaseFrozenIds is still different from upstream, more discussion please check in: https://github.com/greenplum-db/gpdb/issues/15458
---
 src/backend/commands/vacuum.c | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/src/backend/commands/vacuum.c b/src/backend/commands/vacuum.c
index 79337453e1..622d612fa4 100644
--- a/src/backend/commands/vacuum.c
+++ b/src/backend/commands/vacuum.c
@@ -2002,7 +2002,11 @@ vac_update_datfrozenxid(void)
 
 	/* chicken out if bogus data found */
 	if (bogus)
+	{
+		/* Cherry-pick from GPDB FIXME: is it right for PG14? */
+		UnLockDatabaseFrozenIds(ExclusiveLock);
 		return;
+	}
 
 	Assert(TransactionIdIsNormal(newFrozenXid));
 	Assert(MultiXactIdIsValid(newMinMulti));


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For additional commands, e-mail: commits-help@cloudberry.apache.org


(cloudberry) 17/18: Escape database name for dbconn.

Posted by ma...@apache.org.
This is an automated email from the ASF dual-hosted git repository.

maxyang pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/cloudberry.git

commit 87f3d033f3d6993e769e740858bf8d0da6904dca
Author: Zhenghua Lyu <ka...@gmail.com>
AuthorDate: Mon Aug 22 09:48:12 2022 +0800

    Escape database name for dbconn.
    
    This is really a bug of the Python Package PyGreSQL,
    see issue: https://github.com/PyGreSQL/PyGreSQL/issues/77
    
    As a workaround, let's modify GPDB's code to manually
    escape before passing args to pgdb.
---
 src/test/regress/input/dispatch.source  | 11 +++++------
 src/test/regress/output/dispatch.source | 10 ++++------
 2 files changed, 9 insertions(+), 12 deletions(-)

diff --git a/src/test/regress/input/dispatch.source b/src/test/regress/input/dispatch.source
index 4068dc944e..41ed0a9bb7 100644
--- a/src/test/regress/input/dispatch.source
+++ b/src/test/regress/input/dispatch.source
@@ -52,12 +52,11 @@ reset client_min_messages;
 CREATE DATABASE "funny""db'with\\quotes";
 ALTER DATABASE "funny""db'with\\quotes" SET search_path="funny""schema'with\\quotes";
 
--- GPDB_91_MERGE_FIXME: It would be good to leave the database in place, to
--- also test gpcheckcat and pg_upgrade after all the regression tests have
--- completed. As of this writing pg_upgrade does not in fact handle that well.
--- Remove this DROP DATABASE once it's fixed. That should happen when we
--- reach commit a2385cac13, which was backported to PostgreSQL 9.1.
-DROP DATABASE "funny""db'with\\quotes";
+-- test if we can connect using dbconn
+\! python3 test_dbconn.py 0
+
+-- Leave the database in place, to also test gpcheckcat and pg_upgrade after
+-- all the regression tests have completed.
 
 -- set_config() used to have quoting problems as well when dispatching to
 -- segments.
diff --git a/src/test/regress/output/dispatch.source b/src/test/regress/output/dispatch.source
index cd8d872d86..36a66dc749 100644
--- a/src/test/regress/output/dispatch.source
+++ b/src/test/regress/output/dispatch.source
@@ -67,12 +67,10 @@ DROP DATABASE IF EXISTS "funny""db'with\\quotes";
 reset client_min_messages;
 CREATE DATABASE "funny""db'with\\quotes";
 ALTER DATABASE "funny""db'with\\quotes" SET search_path="funny""schema'with\\quotes";
--- GPDB_91_MERGE_FIXME: It would be good to leave the database in place, to
--- also test gpcheckcat and pg_upgrade after all the regression tests have
--- completed. As of this writing pg_upgrade does not in fact handle that well.
--- Remove this DROP DATABASE once it's fixed. That should happen when we
--- reach commit a2385cac13, which was backported to PostgreSQL 9.1.
-DROP DATABASE "funny""db'with\\quotes";
+-- test if we can connect using dbconn
+\! python3 test_dbconn.py 0
+-- Leave the database in place, to also test gpcheckcat and pg_upgrade after
+-- all the regression tests have completed.
 -- set_config() used to have quoting problems as well when dispatching to
 -- segments.
 CREATE TABLE IF NOT EXISTS should_be_visible();


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For additional commands, e-mail: commits-help@cloudberry.apache.org


(cloudberry) 10/18: Fix spelling errors identified by lintian (#15483)

Posted by ma...@apache.org.
This is an automated email from the ASF dual-hosted git repository.

maxyang pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/cloudberry.git

commit 189aaf7a7950f529cfb0f2a23912ae70dabbdc55
Author: Bradford Boyle <br...@users.noreply.github.com>
AuthorDate: Tue May 9 09:52:24 2023 -0700

    Fix spelling errors identified by lintian (#15483)
---
 src/backend/commands/tablecmds.c                   |  2 +-
 src/backend/parser/parse_relation.c                |  2 +-
 src/backend/utils/misc/guc_gp.c                    |  2 +-
 .../expected/segwalrep/recoverseg_from_file.out    |  2 +-
 .../sql/segwalrep/recoverseg_from_file.sql         |  2 +-
 src/test/regress/expected/partition_expand.out     |  6 ++---
 src/test/regress/expected/tidscan.out              | 26 ++++++++++++++++------
 src/test/regress/expected/tidscan_optimizer.out    | 26 ++++++++++++++++------
 src/test/regress/sql/tidscan.sql                   | 17 ++++++++------
 9 files changed, 56 insertions(+), 29 deletions(-)

diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c
index f6ee3a0c30..a149a189f1 100644
--- a/src/backend/commands/tablecmds.c
+++ b/src/backend/commands/tablecmds.c
@@ -5672,7 +5672,7 @@ ATPrepCmd(List **wqueue, Relation rel, AlterTableCmd *cmd,
 				{
 					ereport(NOTICE,
 							(errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
-							 errmsg("skipped, table \"%s\" has already been expanded partiton prepare",
+							 errmsg("skipped, table \"%s\" has already been expanded partition prepare",
 									RelationGetRelationName(rel))));
 					pass = AT_PASS_MISC;	/* We do nothing here */
 					break;
diff --git a/src/backend/parser/parse_relation.c b/src/backend/parser/parse_relation.c
index 2c274eefe9..be22e80874 100644
--- a/src/backend/parser/parse_relation.c
+++ b/src/backend/parser/parse_relation.c
@@ -1496,7 +1496,7 @@ addRangeTableEntry(ParseState *pstate,
 		lockmode = pstate->p_canOptSelectLockingClause ? RowShareLock : ExclusiveLock;
 		if (lockmode == ExclusiveLock && locking->waitPolicy != LockWaitBlock)
 			ereport(WARNING,
-					(errmsg("Upgrade the lockmode to ExclusiveLock on table(%s) and ingore the wait policy.",
+					(errmsg("Upgrade the lockmode to ExclusiveLock on table(%s) and ignore the wait policy.",
 					 RelationGetRelationName(rel))));
 
 		heap_close(rel, NoLock);
diff --git a/src/backend/utils/misc/guc_gp.c b/src/backend/utils/misc/guc_gp.c
index 3c8d8e1881..ff8d828569 100644
--- a/src/backend/utils/misc/guc_gp.c
+++ b/src/backend/utils/misc/guc_gp.c
@@ -2906,7 +2906,7 @@ struct config_bool ConfigureNamesBool_gp[] =
 
 	{
 		{"gp_allow_date_field_width_5digits", PGC_USERSET, COMPAT_OPTIONS_PREVIOUS,
-			gettext_noop("Allow parsing input date field with exactly continous 5 digits in non-standard YYYMMDD timeformat (follow pg12+ behave)"),
+			gettext_noop("Allow parsing input date field with exactly continuous 5 digits in non-standard YYYMMDD timeformat (follow pg12+ behave)"),
 			NULL,
 			GUC_NO_SHOW_ALL | GUC_NOT_IN_SAMPLE
 		},
diff --git a/src/test/isolation2/expected/segwalrep/recoverseg_from_file.out b/src/test/isolation2/expected/segwalrep/recoverseg_from_file.out
index 13d8ee0afb..00921bb1b3 100644
--- a/src/test/isolation2/expected/segwalrep/recoverseg_from_file.out
+++ b/src/test/isolation2/expected/segwalrep/recoverseg_from_file.out
@@ -5,7 +5,7 @@
 -- This is caused by gprecoverseg fetch the smallest dbid in
 -- gp_segment_configuration which is not occupied by others when
 -- adding a new mirror. When dbid in gp_segment_configuration is not
--- continous, the inconsistent issue will happen
+-- continuous, the inconsistent issue will happen
 
 --
 -- generate_recover_config_file:
diff --git a/src/test/isolation2/sql/segwalrep/recoverseg_from_file.sql b/src/test/isolation2/sql/segwalrep/recoverseg_from_file.sql
index 5f756962e4..eb7b0194c5 100644
--- a/src/test/isolation2/sql/segwalrep/recoverseg_from_file.sql
+++ b/src/test/isolation2/sql/segwalrep/recoverseg_from_file.sql
@@ -5,7 +5,7 @@
 -- This is caused by gprecoverseg fetch the smallest dbid in
 -- gp_segment_configuration which is not occupied by others when
 -- adding a new mirror. When dbid in gp_segment_configuration is not
--- continous, the inconsistent issue will happen
+-- continuous, the inconsistent issue will happen
 
 --
 -- generate_recover_config_file:
diff --git a/src/test/regress/expected/partition_expand.out b/src/test/regress/expected/partition_expand.out
index 2acdcb07f7..05e3d180a3 100644
--- a/src/test/regress/expected/partition_expand.out
+++ b/src/test/regress/expected/partition_expand.out
@@ -99,7 +99,7 @@ select gp_segment_id, localoid::regclass, policytype, numsegments, distkey, dist
 (12 rows)
 
 alter table t_hash_partition expand partition prepare;
-NOTICE:  skipped, table "t_hash_partition" has already been expanded partiton prepare
+NOTICE:  skipped, table "t_hash_partition" has already been expanded partition prepare
 --dml of parent table
 select count(*) from t_hash_partition;
  count 
@@ -282,7 +282,7 @@ select gp_segment_id, localoid::regclass, policytype, numsegments, distkey, dist
 
 		
 alter table t_randomly_partition expand partition prepare;
-NOTICE:  skipped, table "t_randomly_partition" has already been expanded partiton prepare
+NOTICE:  skipped, table "t_randomly_partition" has already been expanded partition prepare
 --dml of parent table
 select count(*) from t_randomly_partition;
  count 
@@ -553,7 +553,7 @@ select gp_segment_id, localoid::regclass, policytype, numsegments, distkey, dist
 (30 rows)
 
 alter table t_hash_subpartition expand partition prepare;
-NOTICE:  skipped, table "t_hash_subpartition" has already been expanded partiton prepare
+NOTICE:  skipped, table "t_hash_subpartition" has already been expanded partition prepare
 --dml of parent table
 select count(*) from t_hash_subpartition;
  count 
diff --git a/src/test/regress/expected/tidscan.out b/src/test/regress/expected/tidscan.out
index fa1482aa81..c36ba953ae 100644
--- a/src/test/regress/expected/tidscan.out
+++ b/src/test/regress/expected/tidscan.out
@@ -331,11 +331,23 @@ SELECT count(*) FROM tenk1 t1 JOIN tenk1 t2 ON t1.ctid = t2.ctid and t1.gp_segme
 (1 row)
 
 RESET enable_hashjoin;
--- GPDB_13_MERGE_FIXME
--- --check predicate lock on CTID
--- BEGIN ISOLATION LEVEL SERIALIZABLE;
--- SELECT * FROM tidscan WHERE ctid = '(0,1)';
--- -- locktype should be 'tuple'
--- SELECT locktype, mode FROM pg_locks WHERE pid = pg_backend_pid() AND mode = 'SIReadLock';
--- ROLLBACK;
+-- check predicate lock on CTID
+-- GPDB_12_MERGE_FEATURE_NOT_SUPPORTED: Greenplum does not support serializable transactions,
+-- ignore the below test case.
+-- start_ignore
+BEGIN ISOLATION LEVEL SERIALIZABLE;
+SELECT * FROM tidscan WHERE ctid = '(0,1)';
+ id 
+----
+  1
+(1 row)
+
+-- locktype should be 'tuple'
+SELECT locktype, mode FROM pg_locks WHERE pid = pg_backend_pid() AND mode = 'SIReadLock';
+ locktype | mode 
+----------+------
+(0 rows)
+
+ROLLBACK;
+-- end_ignore
 DROP TABLE tidscan;
diff --git a/src/test/regress/expected/tidscan_optimizer.out b/src/test/regress/expected/tidscan_optimizer.out
index ed4ae3e528..cac7499414 100644
--- a/src/test/regress/expected/tidscan_optimizer.out
+++ b/src/test/regress/expected/tidscan_optimizer.out
@@ -331,11 +331,23 @@ SELECT count(*) FROM tenk1 t1 JOIN tenk1 t2 ON t1.ctid = t2.ctid and t1.gp_segme
 (1 row)
 
 RESET enable_hashjoin;
--- GPDB_13_MERGE_FIXME
--- --check predicate lock on CTID
--- BEGIN ISOLATION LEVEL SERIALIZABLE;
--- SELECT * FROM tidscan WHERE ctid = '(0,1)';
--- -- locktype should be 'tuple'
--- SELECT locktype, mode FROM pg_locks WHERE pid = pg_backend_pid() AND mode = 'SIReadLock';
--- ROLLBACK;
+-- check predicate lock on CTID
+-- GPDB_12_MERGE_FEATURE_NOT_SUPPORTED: Greenplum does not support serializable transactions,
+-- ignore the below test case.
+-- start_ignore
+BEGIN ISOLATION LEVEL SERIALIZABLE;
+SELECT * FROM tidscan WHERE ctid = '(0,1)';
+ id 
+----
+  1
+(1 row)
+
+-- locktype should be 'tuple'
+SELECT locktype, mode FROM pg_locks WHERE pid = pg_backend_pid() AND mode = 'SIReadLock';
+ locktype | mode 
+----------+------
+(0 rows)
+
+ROLLBACK;
+-- end_ignore
 DROP TABLE tidscan;
diff --git a/src/test/regress/sql/tidscan.sql b/src/test/regress/sql/tidscan.sql
index c87caea18d..00b3e74fc0 100644
--- a/src/test/regress/sql/tidscan.sql
+++ b/src/test/regress/sql/tidscan.sql
@@ -118,12 +118,15 @@ SELECT count(*) FROM tenk1 t1 JOIN tenk1 t2 ON t1.ctid = t2.ctid and t1.gp_segme
 SELECT count(*) FROM tenk1 t1 JOIN tenk1 t2 ON t1.ctid = t2.ctid and t1.gp_segment_id = t2.gp_segment_id;
 RESET enable_hashjoin;
 
--- GPDB_13_MERGE_FIXME
--- --check predicate lock on CTID
--- BEGIN ISOLATION LEVEL SERIALIZABLE;
--- SELECT * FROM tidscan WHERE ctid = '(0,1)';
--- -- locktype should be 'tuple'
--- SELECT locktype, mode FROM pg_locks WHERE pid = pg_backend_pid() AND mode = 'SIReadLock';
--- ROLLBACK;
+-- check predicate lock on CTID
+-- GPDB_12_MERGE_FEATURE_NOT_SUPPORTED: Greenplum does not support serializable transactions,
+-- ignore the below test case.
+-- start_ignore
+BEGIN ISOLATION LEVEL SERIALIZABLE;
+SELECT * FROM tidscan WHERE ctid = '(0,1)';
+-- locktype should be 'tuple'
+SELECT locktype, mode FROM pg_locks WHERE pid = pg_backend_pid() AND mode = 'SIReadLock';
+ROLLBACK;
+-- end_ignore
 
 DROP TABLE tidscan;


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For additional commands, e-mail: commits-help@cloudberry.apache.org


(cloudberry) 13/18: fix GPDB_96_MERGE_FIXME, do not need to check forceDistRandom in set_cte_pathlist (#15527)

Posted by ma...@apache.org.
This is an automated email from the ASF dual-hosted git repository.

maxyang pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/cloudberry.git

commit f4a9a6f6bea70365fcb259492493646d626edd0c
Author: xuejing zhao <80...@users.noreply.github.com>
AuthorDate: Fri May 12 19:00:37 2023 +0800

    fix GPDB_96_MERGE_FIXME, do not need to check forceDistRandom in set_cte_pathlist (#15527)
    
    fix GPDB_96_MERGE_FIXME: Should we check forceDistRandom here, like set_subquery_pathlist() does?
    ```
    create table ttt(tc1 int,tc2 int) ;
    insert into ttt values(1,1);
    insert into ttt  values(2,2);
    
    WITH cte AS (
      SELECT oid, relname
      FROM pg_class
      WHERE oid <2000
    )
    SELECT *
    FROM gp_dist_random('cte')
    JOIN ttt ON cte.oid = ttt.tc2;
    
    ERROR:  relation "cte1" does not exist
    ```
    CET can not be the param of gp_dist_random.
    So in set_cte_pathlist we do not need to check forceDistRandom.
    Remove GPDB_96_MERGE_FIXME in allpaths.c
---
 src/backend/optimizer/path/allpaths.c |  1 -
 src/test/regress/expected/bfv_cte.out | 18 ++++++++++++++++++
 src/test/regress/sql/bfv_cte.sql      | 15 +++++++++++++++
 3 files changed, 33 insertions(+), 1 deletion(-)

diff --git a/src/backend/optimizer/path/allpaths.c b/src/backend/optimizer/path/allpaths.c
index 325409589d..19231adde3 100644
--- a/src/backend/optimizer/path/allpaths.c
+++ b/src/backend/optimizer/path/allpaths.c
@@ -3132,7 +3132,6 @@ set_cte_pathlist(PlannerInfo *root, RelOptInfo *rel, RangeTblEntry *rte)
 		List	   *pathkeys;
 		CdbPathLocus locus;
 
-		/* GPDB_96_MERGE_FIXME: Should we check forceDistRandom here, like set_subquery_pathlist() does? */
 		locus = cdbpathlocus_from_subquery(root, rel, subpath);
 
 		/* Convert subquery pathkeys to outer representation */
diff --git a/src/test/regress/expected/bfv_cte.out b/src/test/regress/expected/bfv_cte.out
index cb1f6125e0..ee556a2b6d 100644
--- a/src/test/regress/expected/bfv_cte.out
+++ b/src/test/regress/expected/bfv_cte.out
@@ -338,3 +338,21 @@ select const_a, const_b, sum(n)
  foo_a   | foo_b   |   3
 (2 rows)
 
+-- test cte can not be the param for gp_dist_random
+-- so in set_cte_pathlist we do not neet to check forceDistRandom
+create table ttt(tc1 int,tc2 int) ;
+NOTICE:  Table doesn't have 'DISTRIBUTED BY' clause -- Using column named 'tc1' as the Greenplum Database data distribution key for this table.
+HINT:  The 'DISTRIBUTED BY' clause determines the distribution of data. Make sure column(s) chosen are the optimal data distribution key to minimize skew.
+insert into ttt values(1,1);
+insert into ttt  values(2,2);
+WITH cte AS (
+  SELECT oid, relname
+  FROM pg_class
+  WHERE oid <2000
+)
+SELECT *
+FROM gp_dist_random('cte')
+JOIN ttt ON cte.oid = ttt.tc2;
+ERROR:  relation "cte" does not exist
+LINE 7: FROM gp_dist_random('cte')
+                            ^
diff --git a/src/test/regress/sql/bfv_cte.sql b/src/test/regress/sql/bfv_cte.sql
index 12dfe4c557..dc7426a07c 100644
--- a/src/test/regress/sql/bfv_cte.sql
+++ b/src/test/regress/sql/bfv_cte.sql
@@ -228,3 +228,18 @@ select const_a, const_b, sum(n)
  ) x
  group by const_a, const_b
 ;
+
+-- test cte can not be the param for gp_dist_random
+-- so in set_cte_pathlist we do not neet to check forceDistRandom
+create table ttt(tc1 int,tc2 int) ;
+insert into ttt values(1,1);
+insert into ttt  values(2,2);
+
+WITH cte AS (
+  SELECT oid, relname
+  FROM pg_class
+  WHERE oid <2000
+)
+SELECT *
+FROM gp_dist_random('cte')
+JOIN ttt ON cte.oid = ttt.tc2;


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For additional commands, e-mail: commits-help@cloudberry.apache.org


(cloudberry) 09/18: modify partition_join.out to fix pipeline (#15502)

Posted by ma...@apache.org.
This is an automated email from the ASF dual-hosted git repository.

maxyang pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/cloudberry.git

commit ab220b7b08871d7b2f1ab58a634b0b404ad555b9
Author: xuejing zhao <80...@users.noreply.github.com>
AuthorDate: Tue May 9 17:11:23 2023 +0800

    modify partition_join.out to fix pipeline (#15502)
    
    fix pipeline for commit:75a35c7b289998919abedfd23e2c924888bd5396
---
 src/test/regress/expected/partition_join.out | 14 ++++++++++----
 1 file changed, 10 insertions(+), 4 deletions(-)

diff --git a/src/test/regress/expected/partition_join.out b/src/test/regress/expected/partition_join.out
index ca5a88e86b..d18d34ce0f 100644
--- a/src/test/regress/expected/partition_join.out
+++ b/src/test/regress/expected/partition_join.out
@@ -420,8 +420,11 @@ SELECT sum(t1.a), avg(t1.a), sum(t1.b), avg(t1.b) FROM prt1 t1 WHERE NOT EXISTS
 (1 row)
 
 -- lateral reference
--- GPDB_12_MERGE_FIXME: this is failing in GPDB. Known issue, there were similar
--- failures in other LATERAL tests before the v12 merge.
+-- GPDB_12_MERGE_FEATURE_NOT_SUPPORTED
+-- If lateral_relids of RelOptInfo for a inner path is not null, the inner path needs
+-- params of outer path. So We can not add motion above inner path.
+-- however, in left join, outer path can not be replicated. We can not generate path for left join,
+-- So hit the error: could not devise a query plan for the given query.
 EXPLAIN (COSTS OFF)
 SELECT * FROM prt1 t1 LEFT JOIN LATERAL
 			  (SELECT t2.a AS t2a, t3.a AS t3a, least(t1.a,t2.a,t3.b) FROM prt1 t2 JOIN prt2 t3 ON (t2.a = t3.b)) ss
@@ -2175,8 +2178,11 @@ SELECT t1.a, t1.c, t2.b, t2.c FROM (SELECT * FROM prt1_l WHERE prt1_l.b = 0) t1
 (16 rows)
 
 -- lateral partitionwise join
--- GPDB_12_MERGE_FIXME: this is failing in GPDB. Known issue, there were similar
--- failures in other LATERAL tests before the v12 merge.
+-- GPDB_12_MERGE_FEATURE_NOT_SUPPORTED
+-- If lateral_relids of RelOptInfo for a inner path is not null, the inner path needs
+-- params of outer path. So We can not add motion above inner path.
+-- however, in left join, outer path can not be replicated. We can not generate path for left join,
+-- So hit the error: could not devise a query plan for the given query.
 EXPLAIN (COSTS OFF)
 SELECT * FROM prt1_l t1 LEFT JOIN LATERAL
 			  (SELECT t2.a AS t2a, t2.c AS t2c, t2.b AS t2b, t3.b AS t3b, least(t1.a,t2.a,t3.b) FROM prt1_l t2 JOIN prt2_l t3 ON (t2.a = t3.b AND t2.c = t3.c)) ss


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For additional commands, e-mail: commits-help@cloudberry.apache.org


(cloudberry) 08/18: Fix bypass catalog unittest (#15499)

Posted by ma...@apache.org.
This is an automated email from the ASF dual-hosted git repository.

maxyang pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/cloudberry.git

commit 97dfb96b5764760bd5fd83219255a7fcf98d9dcb
Author: RMT <d....@outlook.com>
AuthorDate: Tue May 9 16:34:20 2023 +0800

    Fix bypass catalog unittest (#15499)
    
    Fix bypass catalog unittest.
    
    Guc value gp_resource_group_bypass_catalog_query doesn't
    get default value in unittest environment, so i set the value in unittest.
    
    BTW, the unittest should return non-0 value when cases failed.
---
 src/backend/utils/resgroup/test/resgroup_test.c | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)

diff --git a/src/backend/utils/resgroup/test/resgroup_test.c b/src/backend/utils/resgroup/test/resgroup_test.c
index 42568102a1..5fb2931f62 100644
--- a/src/backend/utils/resgroup/test/resgroup_test.c
+++ b/src/backend/utils/resgroup/test/resgroup_test.c
@@ -349,6 +349,10 @@ test__shouldBypassQuery__with_only_catalog(void **state)
 {
 	MessageContext = NULL;
 
+	gp_resource_group_bypass_catalog_query = false;
+	assert_false(shouldBypassQuery("select * from pg_catalog.pg_rules"));
+
+	gp_resource_group_bypass_catalog_query = true;
 	assert_true(shouldBypassQuery("select * from pg_catalog.pg_rules"));
 }
 
@@ -384,5 +388,5 @@ main(int argc, char *argv[])
 											   ALLOCSET_DEFAULT_MINSIZE,
 											   ALLOCSET_DEFAULT_INITSIZE,
 											   ALLOCSET_DEFAULT_MAXSIZE);
-	run_tests(tests);
+	return run_tests(tests);
 }


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For additional commands, e-mail: commits-help@cloudberry.apache.org


(cloudberry) 15/18: Using __gp_aoseg/__gp_aocsseg in missing/orphaned file views

Posted by ma...@apache.org.
This is an automated email from the ASF dual-hosted git repository.

maxyang pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/cloudberry.git

commit 623db83b8ea80cfe47627563d45e107b5c175b2e
Author: Huansong Fu <fu...@gmail.com>
AuthorDate: Thu May 4 13:13:33 2023 -0700

    Using __gp_aoseg/__gp_aocsseg in missing/orphaned file views
    
    The __gp_aoseg/__gp_aocsseg functions provide more details such as the eof
    of segments. Use them for the check missing/orphaned file views. So for
    the missing file view, ignore those w/ eof<=0. They could be present just
    for the bookkeeping of modcount. For example, at the end of DELETE, when
    we are increasing modcount, we choose any non-reserved segment to do that.
    So say if we've only inserted into the reserved segment 0 before, then we'll
    choose segment 1 to update modcount. That will create the entry for segno=1
    in aoseg table, but we do not really insert anything to the data file so it
    doesn't exist.
---
 gpcontrib/gp_toolkit/gp_toolkit--1.0.sql | 51 +++++++++++++++++++-------------
 1 file changed, 30 insertions(+), 21 deletions(-)

diff --git a/gpcontrib/gp_toolkit/gp_toolkit--1.0.sql b/gpcontrib/gp_toolkit/gp_toolkit--1.0.sql
index 34bdddd041..d50652ce88 100644
--- a/gpcontrib/gp_toolkit/gp_toolkit--1.0.sql
+++ b/gpcontrib/gp_toolkit/gp_toolkit--1.0.sql
@@ -2131,6 +2131,7 @@ GRANT SELECT ON gp_toolkit.gp_workfile_mgr_used_diskspace TO public;
 -- @out:
 --        oid - relation oid
 --        int - segment number
+--        eof - eof of the segment file
 --
 -- @doc:
 --        UDF to retrieve AO segment file numbers for each ao_row table
@@ -2138,7 +2139,7 @@ GRANT SELECT ON gp_toolkit.gp_workfile_mgr_used_diskspace TO public;
 --------------------------------------------------------------------------------
 
 CREATE OR REPLACE FUNCTION gp_toolkit.__get_ao_segno_list()
-RETURNS TABLE (relid oid, segno int) AS
+RETURNS TABLE (relid oid, segno int, eof bigint) AS
 $$
 DECLARE
   table_name text;
@@ -2147,22 +2148,25 @@ DECLARE
   row record;
 BEGIN
   -- iterate over the aoseg relations
-  FOR rec IN SELECT sc.relname segrel, tc.oid tableoid 
-             FROM pg_appendonly a 
-             JOIN pg_class tc ON a.relid = tc.oid 
-             JOIN pg_am am ON tc.relam = am.oid 
-             JOIN pg_class sc ON a.segrelid = sc.oid 
+  FOR rec IN SELECT tc.oid tableoid, tc.relname, ns.nspname 
+             FROM pg_appendonly a
+             JOIN pg_class tc ON a.relid = tc.oid
+             JOIN pg_am am ON tc.relam = am.oid
+             JOIN pg_namespace ns ON tc.relnamespace = ns.oid
              WHERE amname = 'ao_row' 
   LOOP
-    table_name := rec.segrel;
-    -- Fetch and return each row from the aoseg table
+    table_name := rec.relname;
+    -- Fetch and return each row from the aoseg table.
     BEGIN
-      OPEN cur FOR EXECUTE format('SELECT segno FROM pg_aoseg.%I', table_name);
+      OPEN cur FOR EXECUTE format('SELECT segno, eof '
+                                  'FROM gp_toolkit.__gp_aoseg(''%I.%I'') ',
+                                   rec.nspname, rec.relname);
       SELECT rec.tableoid INTO relid;
       LOOP
         FETCH cur INTO row;
         EXIT WHEN NOT FOUND;
         segno := row.segno;
+        eof := row.eof;
         IF segno <> 0 THEN -- there's no '.0' file, it means the file w/o extension
           RETURN NEXT;
         END IF;
@@ -2171,7 +2175,7 @@ BEGIN
     EXCEPTION
       -- If failed to open the aoseg table (e.g. the table itself is missing), continue
       WHEN OTHERS THEN
-      RAISE WARNING 'Failed to read %: %', table_name, SQLERRM;
+      RAISE WARNING 'Failed to get aoseg info for %: %', table_name, SQLERRM;
     END;
   END LOOP;
   RETURN;
@@ -2190,6 +2194,7 @@ GRANT EXECUTE ON FUNCTION gp_toolkit.__get_ao_segno_list() TO public;
 -- @out:
 --        oid - relation oid
 --        int - segment number
+--        eof - eof of the segment file
 --
 -- @doc:
 --        UDF to retrieve AOCO segment file numbers for each ao_column table
@@ -2197,7 +2202,7 @@ GRANT EXECUTE ON FUNCTION gp_toolkit.__get_ao_segno_list() TO public;
 --------------------------------------------------------------------------------
 
 CREATE OR REPLACE FUNCTION gp_toolkit.__get_aoco_segno_list()
-RETURNS TABLE (relid oid, segno int) AS
+RETURNS TABLE (relid oid, segno int, eof bigint) AS
 $$
 DECLARE
   table_name text;
@@ -2206,25 +2211,25 @@ DECLARE
   row record;
 BEGIN
   -- iterate over the aocoseg relations
-  FOR rec IN SELECT sc.relname segrel, tc.oid tableoid
+  FOR rec IN SELECT tc.oid tableoid, tc.relname, ns.nspname
              FROM pg_appendonly a
              JOIN pg_class tc ON a.relid = tc.oid
              JOIN pg_am am ON tc.relam = am.oid
-             JOIN pg_class sc ON a.segrelid = sc.oid
+             JOIN pg_namespace ns ON tc.relnamespace = ns.oid
              WHERE amname = 'ao_column'
   LOOP
-    table_name := rec.segrel;
-    -- Fetch and return each extended segno corresponding to filenum and segno in the aocoseg table
+    table_name := rec.relname;
+    -- Fetch and return each extended segno corresponding to filenum and segno in the aocoseg table.
     BEGIN
-      OPEN cur FOR EXECUTE format('SELECT ((a.filenum - 1) * 128 + s.segno) as segno '
-                                  'FROM (SELECT * FROM pg_attribute_encoding '
-                                  'WHERE attrelid = %s) a CROSS JOIN pg_aoseg.%I s', 
-                                   rec.tableoid, table_name);
+      OPEN cur FOR EXECUTE format('SELECT physical_segno as segno, eof '
+                                  'FROM gp_toolkit.__gp_aocsseg(''%I.%I'') ',
+                                   rec.nspname, rec.relname);
       SELECT rec.tableoid INTO relid;
       LOOP
         FETCH cur INTO row;
         EXIT WHEN NOT FOUND;
         segno := row.segno;
+        eof := row.eof;
         IF segno <> 0 THEN -- there's no '.0' file, it means the file w/o extension
           RETURN NEXT;
         END IF;
@@ -2233,7 +2238,7 @@ BEGIN
     EXCEPTION
       -- If failed to open the aocoseg table (e.g. the table itself is missing), continue
       WHEN OTHERS THEN
-      RAISE WARNING 'Failed to read %: %', table_name, SQLERRM;
+      RAISE WARNING 'Failed to get aocsseg info for %: %', table_name, SQLERRM;
     END;
   END LOOP;
   RETURN;
@@ -2303,6 +2308,8 @@ GRANT SELECT ON gp_toolkit.__get_expect_files TO public;
 --        Retrieve a list of expected data files in the database,
 --        using the knowledge from catalogs. This includes all
 --        the extended data files for AO/CO tables.
+--        But ignore those w/ eof=0. They might be created just for
+--        modcount whereas no data has ever been inserted to the seg.
 --
 --------------------------------------------------------------------------------
 CREATE OR REPLACE VIEW gp_toolkit.__get_expect_files_ext AS
@@ -2316,13 +2323,15 @@ SELECT c.reltablespace AS tablespace, c.relname, a.amname AS AM,
 FROM gp_toolkit.__get_ao_segno_list() s
 JOIN pg_class c ON s.relid = c.oid
 LEFT JOIN pg_am a ON c.relam = a.oid
+WHERE s.eof > 0
 UNION
 -- CO extended files
 SELECT c.reltablespace AS tablespace, c.relname, a.amname AS AM,
        format(c.relfilenode::text || '.' || s.segno::text) AS filename
 FROM gp_toolkit.__get_aoco_segno_list() s
 JOIN pg_class c ON s.relid = c.oid
-LEFT JOIN pg_am a ON c.relam = a.oid;
+LEFT JOIN pg_am a ON c.relam = a.oid
+WHERE s.eof > 0;
 
 GRANT SELECT ON gp_toolkit.__get_expect_files_ext TO public;
 


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For additional commands, e-mail: commits-help@cloudberry.apache.org


(cloudberry) 05/18: mem intensive agg should contains mixed type.

Posted by ma...@apache.org.
This is an automated email from the ASF dual-hosted git repository.

maxyang pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/cloudberry.git

commit 46b79fd1f0be0f96ac6864ccef5a17c1324020a4
Author: erchuan <12...@qq.com>
AuthorDate: Tue May 9 13:42:22 2023 +0800

    mem intensive agg should contains mixed type.
    
    Mixed-Agg may appear when rollup or cube used in aggregation. Assuming that MIXED
    must contains hash-agg, it should be memory intensive.
    
    Authored-by: erchuan.ty <er...@alibaba-inc.com>
---
 src/backend/utils/resource_manager/memquota.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/src/backend/utils/resource_manager/memquota.c b/src/backend/utils/resource_manager/memquota.c
index 1d9aa22137..f1e272e9f2 100644
--- a/src/backend/utils/resource_manager/memquota.c
+++ b/src/backend/utils/resource_manager/memquota.c
@@ -130,7 +130,7 @@ contain_ordered_aggs_walker(Node *node, void *context)
 
 /**
  * Is an agg operator memory intensive? The following cases mean it is:
- * 1. If agg strategy is hashed
+ * 1. If agg strategy is hashed or mixed (assuming that MIXED must contains hash agg)
  * 2. If targetlist or qual contains a DQA
  * 3. If there is an ordered aggregated.
  */
@@ -138,7 +138,7 @@ static bool
 IsAggMemoryIntensive(Agg *agg)
 {
 	/* Case 1 */
-	if (agg->aggstrategy == AGG_HASHED)
+	if (agg->aggstrategy == AGG_HASHED || agg->aggstrategy == AGG_MIXED)
 		return true;
 
 	/* Cases 2 & 3 */


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For additional commands, e-mail: commits-help@cloudberry.apache.org