You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cloudberry.apache.org by "adnanhamdussalam (via GitHub)" <gi...@apache.org> on 2026/04/29 10:42:39 UTC

[I] Issue : server closed the connection unexpectedly This probably means the server terminated abnormally [cloudberry]

adnanhamdussalam opened a new issue, #1708:
URL: https://github.com/apache/cloudberry/issues/1708

   Hi Team,
   
   I am facing against the below query  as per my logs: 
   
       This probably means the server terminated abnormally
           before or while processing the request.",,,,,,"/* {""app"": ""dbt"", ""dbt_version"": ""10.2"", ""profile_name"": ""rio-dev-profile"", ""target_name"": ""dev"", ""node_id"": ""model.riodbt.ca""} */
   
   
   
   
   
     create temporary table ""aca__dbt_tmp120645746261""
   
   
       as
   
     (
   
   
       WITH base_pre_aca_filtered AS (
           SELECT *
           FROM ""gpmyafi"".""my_28"".""pre_aca"" aca
   
   
                   WHERE aca.call_seg_start_time >= '2026-04-20 21:56:12'::DATE - interval '90 days'
   
   
       ),
       aca_join_agent_attribute AS (
           SELECT
               aca.*
               , aa.agent_skilling
           FROM my_pre_a_filtered_base aca
           LEFT JOIN ""gpmyafi"".""my_28"".""t_agent_attributes"" as aa
           ON aca.ans_agent_id = aa.agent_id
           AND (aca.call_seg_start_time >= aa.login_time AND (aca.call_seg_start_time < aa.logout_time OR aa.logout_time is NULL))
       ),
   
       aca_join_haglog_1 AS (
           SELECT
               aca.primary_telephony_id
               , array_to_string(array_agg( distinct concat( h.skill_name, ':', h.skill_level::int) order by concat( h.skill_name, ':', h.skill_level::int) ) , ',') as agent_skilling
           FROM my_pre_a_filtered_base aca
           LEFT JOIN ""gpmyafi"".""my_28"".""haglog"" as h
           ON aca.switch_agent_id = h.agent_skill_employee_id::text
           AND ( aca.call_seg_start_time >= h.agent_skill_start_ts AND aca.call_seg_start_time < h.agent_skill_end_ts )
           where h.skill_name is not null and h.skill_level is not null
           GROUP BY aca.primary_telephony_id
       ),
   
       aca_join_haglog_2 AS (
           SELECT
               aca.primary_telephony_id
               , array_to_string(array_agg(distinct concat(h.agent_group, ':', -1::int) order by concat(h.agent_group, ':', -1::int)) , ',') as agent_skilling
           FROM my_pre_a_filtered_base aca
           LEFT JOIN ""gpmyafi"".""my_28"".""haglog"" as h
           ON aca.switch_agent_id = h.agent_group_employee_id::text
           AND ( aca.call_seg_start_time >= h.agent_group_start_ts AND aca.call_seg_start_time < h.agent_group_end_ts )
           where h.agent_group is not null
           GROUP BY aca.primary_telephony_id
       ),
   
       aca_join_haglog_3 AS
       (
           SELECT
               coalesce(h1.primary_telephony_id,h2.primary_telephony_id) as primary_telephony_id
               , concat_my(',',h1.agent_skilling, h2.agent_skilling) as agent_skilling
           FROM aca_join_haglog_1 h1
           full outer JOIN aca_join_haglog_2 as h2
           ON h1.primary_telephony_id = h2.primary_telephony_id
       ),
   
       aca_join_haglog AS
       (
           SELECT
               aca.primary_telephony_id,
               aca.call_id,
               aca.call_date,
               aca.call_seg_start_time,
               aca.call_seg_start_time_utc,
               aca.call_answer_time,
               aca.call_hang_up_time,
               aca.call_seg_end_time,
               aca.next_call_answer_time,
               aca.first_call_answer_time,
               aca.skill_id,
               aca.vdn_id,
               aca.routing_scope_id,
               aca.switch_agent_id,
               aca.call_disp,
               aca.transferred,
               aca.acd_attached_data,
               aca.first_telephony_id,
               aca.customer_phone_number,
               aca.customer_id_1,
               aca.customer_id_2,
               aca.customer_id_3,
               aca.customer_id_4,
               aca.customer_id_5,
               aca.customer_id_6,
               aca.parsed_customer_id,
               aca.parsed_phone_number,
               aca.native_skill_data,
               aca.target_expression,
               aca.routing_status,
               aca.acd_routing_status,
               aca.handle_time,
               aca.on_off,
               aca.extended_benchmark,
               aca.ans_agent_wait_time,
               aca.abandon,
               aca.outcome_id,
               aca.outcome_key,
               aca.outcome_status,
               aca.outcome_type,
               aca.outcome_callid,
               aca.outcome_customer_id_1,
               aca.outcome_customer_id_2,
               aca.outcome_customer_id_3,
               aca.outcome_customer_id_4,
               aca.outcome_customer_id_5,
               aca.outcome_customer_id_6,
               aca.outcome_agent_id,
               aca.outcome_binary,
               aca.outcome_revenue,
               aca.outcome_categorical,
               aca.outcome_continuous,
               aca.outcome_create_time,
               aca.outcome_create_time_secondary,
               aca.outcome_confirmation,
               aca.outcome_confirmation_agent,
               aca.outcome_confirmation_time,
               aca.outcome_confirmation_days,
               aca.attribution_outcome_id,
               aca.attribution_outcome_key,
               aca.attribution_outcome_binary,
               aca.attribution_outcome_revenue,
               aca.attribution_outcome_continuous,
               aca.attribution_outcome_create_time,
               aca.attribution_outcome_create_time_secondary,
               aca.attribution_outcome_status,
               aca.attribution_outcome_mode,
   
               aca.outcome_details,
               aca.p_vdn,
               aca.o_vdn,
               aca.from_transfer,
               aca.dialed_num,
               aca.call_priority,
               aca.agent_priority,
               aca.agent_choice,
               aca.caller_choice,
               aca.agent_choice_used,
               aca.caller_choice_used,
               aca.agent_percentile_actual,
               aca.call_percentile_actual,
               aca.anslogin,
               aca.origlogin,
               aca.disppriority,
               aca.request_type,
               aca.target,
               aca.day0_target,
               aca.is_relevant,
               aca.is_relevant_switch,
               aca.is_relevant_final,
               guid,
               aca.acd_error_code,
               aca.agent_station,
                   default_route_reason,
               aca.agent_group,
               aca.agent_group_string,
               aca.interaction_reason,
               aca.routed_agent_group,
               aca.ans_agent_id,
               aca.call_direction,
               coalesce(aca.agent_skilling, h.agent_skilling) agent_skilling
   
   
   
   
   
   
   
                   ,
   
                   outcome_id_19,
                   outcome_binary_19,
                   outcome_revenue_19,
                   outcome_continuous_19
   
               , aca.ixn_id_outcomes
               , aca.agent_mapping_id
               , aca.queued_skill_list
               , aca.customer_benchmark
   
   
   
               , aca.call_guid
               , aca.original_leg
   
                   , aca.original_interaction_id
   
   
           FROM my_a_j_agent_atrib aca
           LEFT JOIN aca_join_haglog_3 as h
           ON aca.primary_telephony_id = h.primary_telephony_id
       )
   
   
       SELECT *
           , left(aca.customer_phone_number,7) as customer_phone_number_grf
       FROM aca_join_haglog aca
   
   
     );
   
     ",0,,"cdbdispatchresult.c",510,"Stack trace:
   1    0xd6a018 postgres errstart + 0x258
   2    0xd6fd76 postgres ThrowErrorData + 0x16
   3    0xa0e216 postgres mppExecutorFinishup + 0xd6
   4    0x9ff1fa postgres standard_ExecutorEnd + 0xca
   5    0x968fb8 postgres ExecCreateTableAs + 0x398
   6    0xc1ef12 postgres <symbol not found> + 0xc1ef12
   7    0xc1dd49 postgres standard_ProcessUtility + 0x369
   8    0xc1e893 postgres ProcessUtility + 0xb3
   9    0xc1c2d5 postgres <symbol not found> + 0xc1c2d5
   10   0xc1c3fb postgres <symbol not found> + 0xc1c3fb
   11   0xc1ca03 postgres PortalRun + 0x2a3
   12   0xc15ed3 postgres exec_simple_query + 0x1b3
   13   0xc1a824 postgres PostgresMain + 0x2534
   14   0xb78390 postgres <symbol not found> + 0xb78390
   15   0xb7939c postgres PostmasterMain + 0xe0c
   16   0x7c45eb postgres main + 0x56b
   17   0x7fd52e62a610 libc.so.6 <symbol not found> + 0x2e62a610
   18   0x7fd52e62a6c0 libc.so.6 __libc_start_main + 0x80
   19   0x7d5075 postgres _start + 0x25
   "
   
   Kindly can you tell is it a bug?
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For additional commands, e-mail: commits-help@cloudberry.apache.org


Re: [I] Issue : server closed the connection unexpectedly This probably means the server terminated abnormally [cloudberry]

Posted by "yjhjstz (via GitHub)" <gi...@apache.org>.
yjhjstz closed issue #1708: Issue : server closed the connection unexpectedly This probably means the server terminated abnormally
URL: https://github.com/apache/cloudberry/issues/1708


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For additional commands, e-mail: commits-help@cloudberry.apache.org


Re: [I] Issue : server closed the connection unexpectedly This probably means the server terminated abnormally [cloudberry]

Posted by "my-ship-it (via GitHub)" <gi...@apache.org>.
my-ship-it commented on issue #1708:
URL: https://github.com/apache/cloudberry/issues/1708#issuecomment-4385461621

   Thanks for the update @adnanhamdussalam — that confirms it's a real SIGSEGV (not OOM, not a cancel). The single frame you pasted is just the signal handler itself, so we still can't see where it actually crashed. Two things would let us pin this down:
   
   **1. The full segment stack (frames 2 through ~12).**
   
   The PANIC line you found is on one specific segment — please open that segment's log directly (not the coordinator's). The CSV log truncates long fields, so prefer the plain `.log` file:
   
   ```bash
   # on the segment host where the PANIC happened
   grep -A 25 'received signal SIGSEGV' /data/primary/gpseg*/pg_log/*.log | head -120
   ```
   
   Frame 1 (`StandardHandlerForSigillSigsegvSigbus_OnMainThread`) is the handler. Frames 2+ are what we need — that's the actual crash site.
   
   If you have core dumps enabled, even better:
   
   ```bash
   gdb \$GPHOME/bin/postgres <core-file> -batch -ex 'bt full'
   ```
   
   **2. One quick retry with the legacy planner.**
   
   ```sql
   SET optimizer = off;
   -- then run the same CREATE TEMP TABLE AS ... query
   ```
   
   If it stops crashing → the bug is in ORCA. If it still crashes → the bug is in the executor and ORCA is off the table. Either result narrows the search a lot.
   
   Once we have the stack frames + the optimizer-off result, we can match this against fixes that landed after the 2.1.0-incubating tag. Your build is at \`bdf90c5\` (the release commit itself); main is several hundred commits ahead and includes a few SIGSEGV / ORCA fixes that are candidates here, but I don't want to point at a specific one until the stack tells us where we actually are.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For additional commands, e-mail: commits-help@cloudberry.apache.org


Re: [I] Issue : server closed the connection unexpectedly This probably means the server terminated abnormally [cloudberry]

Posted by "adnanhamdussalam (via GitHub)" <gi...@apache.org>.
adnanhamdussalam commented on issue #1708:
URL: https://github.com/apache/cloudberry/issues/1708#issuecomment-4343396518

   Hi @my-ship-it,
   
   PFB the update:
   The Cloudberry version (`SELECT version();`).
     ---- SELECT version();
                                                                                                 version
   ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
    PostgreSQL 14.4 (Apache Cloudberry 2.1.0-incubating build bdf90c5518f) on x86_64-pc-linux-gnu, compiled by gcc (GCC) 11.5.0 20240719 (Red Hat 11.5.0-11), 64-bit compiled on Mar  4 2026 23:25:31
   
   The definition of `concat_my` (`\df+ concat_my`). If it's a custom C function, could you try swapping it for `concat_ws(',', h1.agent_skilling, h2.agent_skilling)` once and see if the crash still happens? That alone would tell us a lot.
   -- it is concat_ws being used as function 
   Does it fail every time, or only on certain data sizes? Does a smaller window (say 7 days instead of 90) still reproduce it?
   -- it is in progresss will update in a while
   Values for `statement_mem`, `max_statement_mem`, `gp_workfile_limit_per_segment`, and which resource group this is running under.
   
   gpconfig -s statement_mem
   Values on all segments are consistent
   GUC              : statement_mem
   Coordinator value: 256MB
   Segment     value: 256MB
   
   gpconfig -s max_statement_mem
   Values on all segments are consistent
   GUC              : max_statement_mem
   Coordinator value: 10GB
   Segment     value: 10GB
   
   gpconfig -s gp_workfile_limit_per_segment
   Values on all segments are consistent
   GUC              : gp_workfile_limit_per_segment
   Coordinator value: 0
   Segment     value: 0
   
   Segment logs around the time it failed:
   
   Found these errors from segment that is causing issue:
   ERROR","25M01","canceling MPP operation"
   "LOG","00000","Process interrupt for 'query cancel pending' (execScan.c:42)",
   "LOG","00000","An exception was encountered during the execution of statement:
   
   `dmesg -T | grep -iE 'killed process|out of memory'` :
   No output on coordinator and segment nodes
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For additional commands, e-mail: commits-help@cloudberry.apache.org


Re: [I] Issue : server closed the connection unexpectedly This probably means the server terminated abnormally [cloudberry]

Posted by "adnanhamdussalam (via GitHub)" <gi...@apache.org>.
adnanhamdussalam commented on issue #1708:
URL: https://github.com/apache/cloudberry/issues/1708#issuecomment-4346065033

   Hi @my-ship-it,
   
   The query that I have shared earlier now crashed the segment:
   
   PANIC","XX000","Unexpected internal error: Segment process received signal SIGSEGV",,,,,,,0,,,,"1    0xd704a5 postgres StandardHandlerForSigillSigsegvSigbus_OnMainThread + 0x155
   
   Kindly can you share a workaround about this issue.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For additional commands, e-mail: commits-help@cloudberry.apache.org


Re: [I] Issue : server closed the connection unexpectedly This probably means the server terminated abnormally [cloudberry]

Posted by "yjhjstz (via GitHub)" <gi...@apache.org>.
yjhjstz commented on issue #1708:
URL: https://github.com/apache/cloudberry/issues/1708#issuecomment-4398984951

   > 2- Can you share the command to enable core dumps?
   1. ulimit -c unlimited
   2. sudo sysctl -w kernel.core_pattern=/tmp/core.%e.%p.%t
   3. restart cluster


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For additional commands, e-mail: commits-help@cloudberry.apache.org


Re: [I] Issue : server closed the connection unexpectedly This probably means the server terminated abnormally [cloudberry]

Posted by "adnanhamdussalam (via GitHub)" <gi...@apache.org>.
adnanhamdussalam commented on issue #1708:
URL: https://github.com/apache/cloudberry/issues/1708#issuecomment-4387131516

   # on the segment host where the PANIC happened
   grep -A 25 'received signal SIGSEGV' /data/cbdtabase/primary/gpseg*/log/*.csv | head -120
   
   /data/cbdatabase/primary/gpseg0/log/gpdb-2026-04-28_000000.csv:2026-04-28 17:07:00.877342 CEST,,,p2884481,th123456,,,2026-04-28 17:06:56 CEST,0,con17033,cmd18,seg0,slice2,,,,"PANIC","XX000","Unexpected internal error: Segment process received signal SIGSEGV",,,,,,,0,,,,"1    0xd704a5 postgres StandardHandlerForSigillSigsegvSigbus_OnMainThread + 0x155
   /data/cbdatabase/primary/gpseg0/log/gpdb-2026-04-28_000000.csv-2    0x7fe368e3fc30 libc.so.6 <symbol not found> + 0x68e3fc30
   /data/cbdatabase/primary/gpseg0/log/gpdb-2026-04-28_000000.csv-3    0xa090c2 postgres <symbol not found> + 0xa090c2
   /data/cbdatabase/primary/gpseg0/log/gpdb-2026-04-28_000000.csv-4    0xa0b14e postgres slot_getsomeattrs_int + 0x1e
   /data/cbdatabase/primary/gpseg0/log/gpdb-2026-04-28_000000.csv-5    0x9f9139 postgres <symbol not found> + 0x9f9139
   /data/cbdatabase/primary/gpseg0/log/gpdb-2026-04-28_000000.csv-6    0xa371d7 postgres <symbol not found> + 0xa371d7
   /data/cbdatabase/primary/gpseg0/log/gpdb-2026-04-28_000000.csv-7    0xa0584d postgres <symbol not found> + 0xa0584d
   /data/cbdatabase/primary/gpseg0/log/gpdb-2026-04-28_000000.csv-8    0xa4b25a postgres <symbol not found> + 0xa4b25a
   /data/cbdatabase/primary/gpseg0/log/gpdb-2026-04-28_000000.csv-9    0xa0584d postgres <symbol not found> + 0xa0584d
   /data/cbdatabase/primary/gpseg0/log/gpdb-2026-04-28_000000.csv-10   0x9fc32a postgres <symbol not found> + 0x9fc32a
   /data/cbdatabase/primary/gpseg0/log/gpdb-2026-04-28_000000.csv-11   0x9fd001 postgres standard_ExecutorRun + 0x751
   /data/cbdatabase/primary/gpseg0/log/gpdb-2026-04-28_000000.csv-12   0x9fd1a5 postgres ExecutorRun + 0xc5
   /data/cbdatabase/primary/gpseg0/log/gpdb-2026-04-28_000000.csv-13   0xc1b571 postgres <symbol not found> + 0xc1b571
   /data/cbdatabase/primary/gpseg0/log/gpdb-2026-04-28_000000.csv-14   0xc1c541 postgres <symbol not found> + 0xc1c541
   /data/cbdatabase/primary/gpseg0/log/gpdb-2026-04-28_000000.csv-15   0xc1ca03 postgres PortalRun + 0x2a3
   /data/cbdatabase/primary/gpseg0/log/gpdb-2026-04-28_000000.csv-16   0xc16c62 postgres <symbol not found> + 0xc16c62
   /data/cbdatabase/primary/gpseg0/log/gpdb-2026-04-28_000000.csv-17   0xc1a648 postgres PostgresMain + 0x2358
   /data/cbdatabase/primary/gpseg0/log/gpdb-2026-04-28_000000.csv-18   0xb78390 postgres <symbol not found> + 0xb78390
   /data/cbdatabase/primary/gpseg0/log/gpdb-2026-04-28_000000.csv-19   0xb7939c postgres PostmasterMain + 0xe0c
   /data/cbdatabase/primary/gpseg0/log/gpdb-2026-04-28_000000.csv-20   0x7c45eb postgres main + 0x56b
   /data/cbdatabase/primary/gpseg0/log/gpdb-2026-04-28_000000.csv-21   0x7fe368e2a610 libc.so.6 <symbol not found> + 0x68e2a610
   /data/cbdatabase/primary/gpseg0/log/gpdb-2026-04-28_000000.csv-22   0x7fe368e2a6c0 libc.so.6 __libc_start_main + 0x80
   /data/cbdatabase/primary/gpseg0/log/gpdb-2026-04-28_000000.csv-23   0x7d5075 postgres _start + 0x25
   /data/cbdatabase/primary/gpseg0/log/gpdb-2026-04-28_000000.csv-"
   /data/cbdatabase/primary/gpseg0/log/gpdb-2026-04-28_000000.csv-2026-04-28 17:07:00.877444 CEST,,,p2081152,th1761700032,,,,0,,,seg0,,,,,"LOG","00000","server process (PID 2884481) was terminated by signal 11: Segmentation fault","Failed process was running: /* {""app"": ""dbt"", ""dbt_version"": ""1.7.17"", ""profile_name"": ""airo-dev-profile"", ""target_name"": ""dev"", ""node_id"": ""model.airodbt.aca""} */
   /data/cbdatabase/primary/gpseg0/log/gpdb-2026-04-28_000000.csv-
   --
   /data/cbdatabase/primary/gpseg0/log/gpdb-2026-04-28_000000.csv:2026-04-28 18:10:33.684545 CEST,,,p2933859,th123456,,,2026-04-28 18:10:30 CEST,0,con17611,cmd18,seg0,slice2,,,,"PANIC","XX000","Unexpected internal error: Segment process received signal SIGSEGV",,,,,,,0,,,,"1    0xd704a5 postgres StandardHandlerForSigillSigsegvSigbus_OnMainThread + 0x155
   /data/cbdatabase/primary/gpseg0/log/gpdb-2026-04-28_000000.csv-2    0x7fe368e3fc30 libc.so.6 <symbol not found> + 0x68e3fc30
   /data/cbdatabase/primary/gpseg0/log/gpdb-2026-04-28_000000.csv-3    0xa090c2 postgres <symbol not found> + 0xa090c2
   /data/cbdatabase/primary/gpseg0/log/gpdb-2026-04-28_000000.csv-4    0xa0b14e postgres slot_getsomeattrs_int + 0x1e
   /data/cbdatabase/primary/gpseg0/log/gpdb-2026-04-28_000000.csv-5    0x9f9139 postgres <symbol not found> + 0x9f9139
   /data/cbdatabase/primary/gpseg0/log/gpdb-2026-04-28_000000.csv-6    0xa371d7 postgres <symbol not found> + 0xa371d7
   /data/cbdatabase/primary/gpseg0/log/gpdb-2026-04-28_000000.csv-7    0xa0584d postgres <symbol not found> + 0xa0584d
   /data/cbdatabase/primary/gpseg0/log/gpdb-2026-04-28_000000.csv-8    0xa4b25a postgres <symbol not found> + 0xa4b25a
   /data/cbdatabase/primary/gpseg0/log/gpdb-2026-04-28_000000.csv-9    0xa0584d postgres <symbol not found> + 0xa0584d
   /data/cbdatabase/primary/gpseg0/log/gpdb-2026-04-28_000000.csv-10   0x9fc32a postgres <symbol not found> + 0x9fc32a
   /data/cbdatabase/primary/gpseg0/log/gpdb-2026-04-28_000000.csv-11   0x9fd001 postgres standard_ExecutorRun + 0x751
   /data/cbdatabase/primary/gpseg0/log/gpdb-2026-04-28_000000.csv-12   0x9fd1a5 postgres ExecutorRun + 0xc5
   /data/cbdatabase/primary/gpseg0/log/gpdb-2026-04-28_000000.csv-13   0xc1b571 postgres <symbol not found> + 0xc1b571
   /data/cbdatabase/primary/gpseg0/log/gpdb-2026-04-28_000000.csv-14   0xc1c541 postgres <symbol not found> + 0xc1c541
   /data/cbdatabase/primary/gpseg0/log/gpdb-2026-04-28_000000.csv-15   0xc1ca03 postgres PortalRun + 0x2a3
   /data/cbdatabase/primary/gpseg0/log/gpdb-2026-04-28_000000.csv-16   0xc16c62 postgres <symbol not found> + 0xc16c62
   /data/cbdatabase/primary/gpseg0/log/gpdb-2026-04-28_000000.csv-17   0xc1a648 postgres PostgresMain + 0x2358
   /data/cbdatabase/primary/gpseg0/log/gpdb-2026-04-28_000000.csv-18   0xb78390 postgres <symbol not found> + 0xb78390
   /data/cbdatabase/primary/gpseg0/log/gpdb-2026-04-28_000000.csv-19   0xb7939c postgres PostmasterMain + 0xe0c
   /data/cbdatabase/primary/gpseg0/log/gpdb-2026-04-28_000000.csv-20   0x7c45eb postgres main + 0x56b
   /data/cbdatabase/primary/gpseg0/log/gpdb-2026-04-28_000000.csv-21   0x7fe368e2a610 libc.so.6 <symbol not found> + 0x68e2a610
   /data/cbdatabase/primary/gpseg0/log/gpdb-2026-04-28_000000.csv-22   0x7fe368e2a6c0 libc.so.6 __libc_start_main + 0x80
   /data/cbdatabase/primary/gpseg0/log/gpdb-2026-04-28_000000.csv-23   0x7d5075 postgres _start + 0x25
   /data/cbdatabase/primary/gpseg0/log/gpdb-2026-04-28_000000.csv-"
   /data/cbdatabase/primary/gpseg0/log/gpdb-2026-04-28_000000.csv-2026-04-28 18:10:33.685773 CEST,,,p2081152,th1761700032,,,,0,,,seg0,,,,,"LOG","00000","server process (PID 2933859) was terminated by signal 11: Segmentation fault","Failed process was running: /* {""app"": ""dbt"", ""dbt_version"": ""1.7.17"", ""profile_name"": ""airo-dev-profile"", ""target_name"": ""dev"", ""node_id"": ""model.airodbt.aca""} */
   /data/cbdatabase/primary/gpseg0/log/gpdb-2026-04-28_000000.csv-
   --
   /data/cbdatabase/primary/gpseg0/log/gpdb-2026-04-29_000000.csv:2026-04-29 12:06:49.057129 CEST,,,p3775798,th123456,,,2026-04-29 12:06:45 CEST,0,con35066,cmd18,seg0,slice2,,,,"PANIC","XX000","Unexpected internal error: Segment process received signal SIGSEGV",,,,,,,0,,,,"1    0xd704a5 postgres StandardHandlerForSigillSigsegvSigbus_OnMainThread + 0x155
   /data/cbdatabase/primary/gpseg0/log/gpdb-2026-04-29_000000.csv-2    0x7fe368e3fc30 libc.so.6 <symbol not found> + 0x68e3fc30
   /data/cbdatabase/primary/gpseg0/log/gpdb-2026-04-29_000000.csv-3    0xa090c2 postgres <symbol not found> + 0xa090c2
   /data/cbdatabase/primary/gpseg0/log/gpdb-2026-04-29_000000.csv-4    0xa0b14e postgres slot_getsomeattrs_int + 0x1e
   /data/cbdatabase/primary/gpseg0/log/gpdb-2026-04-29_000000.csv-5    0x9f9139 postgres <symbol not found> + 0x9f9139
   /data/cbdatabase/primary/gpseg0/log/gpdb-2026-04-29_000000.csv-6    0xa371d7 postgres <symbol not found> + 0xa371d7
   /data/cbdatabase/primary/gpseg0/log/gpdb-2026-04-29_000000.csv-7    0xa0584d postgres <symbol not found> + 0xa0584d
   /data/cbdatabase/primary/gpseg0/log/gpdb-2026-04-29_000000.csv-8    0xa4b25a postgres <symbol not found> + 0xa4b25a
   /data/cbdatabase/primary/gpseg0/log/gpdb-2026-04-29_000000.csv-9    0xa0584d postgres <symbol not found> + 0xa0584d
   /data/cbdatabase/primary/gpseg0/log/gpdb-2026-04-29_000000.csv-10   0x9fc32a postgres <symbol not found> + 0x9fc32a
   /data/cbdatabase/primary/gpseg0/log/gpdb-2026-04-29_000000.csv-11   0x9fd001 postgres standard_ExecutorRun + 0x751
   /data/cbdatabase/primary/gpseg0/log/gpdb-2026-04-29_000000.csv-12   0x9fd1a5 postgres ExecutorRun + 0xc5
   /data/cbdatabase/primary/gpseg0/log/gpdb-2026-04-29_000000.csv-13   0xc1b571 postgres <symbol not found> + 0xc1b571
   /data/cbdatabase/primary/gpseg0/log/gpdb-2026-04-29_000000.csv-14   0xc1c541 postgres <symbol not found> + 0xc1c541
   /data/cbdatabase/primary/gpseg0/log/gpdb-2026-04-29_000000.csv-15   0xc1ca03 postgres PortalRun + 0x2a3
   /data/cbdatabase/primary/gpseg0/log/gpdb-2026-04-29_000000.csv-16   0xc16c62 postgres <symbol not found> + 0xc16c62
   /data/cbdatabase/primary/gpseg0/log/gpdb-2026-04-29_000000.csv-17   0xc1a648 postgres PostgresMain + 0x2358
   /data/cbdatabase/primary/gpseg0/log/gpdb-2026-04-29_000000.csv-18   0xb78390 postgres <symbol not found> + 0xb78390
   /data/cbdatabase/primary/gpseg0/log/gpdb-2026-04-29_000000.csv-19   0xb7939c postgres PostmasterMain + 0xe0c
   /data/cbdatabase/primary/gpseg0/log/gpdb-2026-04-29_000000.csv-20   0x7c45eb postgres main + 0x56b
   /data/cbdatabase/primary/gpseg0/log/gpdb-2026-04-29_000000.csv-21   0x7fe368e2a610 libc.so.6 <symbol not found> + 0x68e2a610
   /data/cbdatabase/primary/gpseg0/log/gpdb-2026-04-29_000000.csv-22   0x7fe368e2a6c0 libc.so.6 __libc_start_main + 0x80
   /data/cbdatabase/primary/gpseg0/log/gpdb-2026-04-29_000000.csv-23   0x7d5075 postgres _start + 0x25
   /data/cbdatabase/primary/gpseg0/log/gpdb-2026-04-29_000000.csv-"
   /data/cbdatabase/primary/gpseg0/log/gpdb-2026-04-29_000000.csv-2026-04-29 12:06:49.057295 CEST,,,p2081152,th1761700032,,,,0,,,seg0,,,,,"LOG","00000","server process (PID 3775798) was terminated by signal 11: Segmentation fault","Failed process was running: /* {""app"": ""dbt"", ""dbt_version"": ""1.7.17"", ""profile_name"": ""airo-dev-profile"", ""target_name"": ""dev"", ""node_id"": ""model.airodbt.aca""} */
   /data/cbdatabase/primary/gpseg0/log/gpdb-2026-04-29_000000.csv-
   
   2- Can you share the command to enable core dumps?
   
   3- will try with set optimizer off and will update you.
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For additional commands, e-mail: commits-help@cloudberry.apache.org


Re: [I] Issue : server closed the connection unexpectedly This probably means the server terminated abnormally [cloudberry]

Posted by "NadeemRajput-sys (via GitHub)" <gi...@apache.org>.
NadeemRajput-sys commented on issue #1708:
URL: https://github.com/apache/cloudberry/issues/1708#issuecomment-4345581109

   @my-ship-it it is observed that whenever CTE is executed that expect to process larger data set, such segment failure occures. If we convert CTE into using temp-tables instead, it mostly works fine.
   CTE with smaller dataset processing works okie mostly.
   
   Question is:
   
   Is it a Bug or support limitation to CTE in Apache Cloudberry? same CTEs works fine in GP6.x 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For additional commands, e-mail: commits-help@cloudberry.apache.org


Re: [I] Issue : server closed the connection unexpectedly This probably means the server terminated abnormally [cloudberry]

Posted by "my-ship-it (via GitHub)" <gi...@apache.org>.
my-ship-it commented on issue #1708:
URL: https://github.com/apache/cloudberry/issues/1708#issuecomment-4343029472

   Hey @adnanhamdussalam, thanks for the report — and for including the stack, that's already a useful starting point.
   
   I took a look, but I don't think we can call this a bug yet — at least not from what's here. The stack you pasted is actually from the coordinator side (`mppExecutorFinishup` → `cdbdispatchresult.c:510`). That's the path the QD goes down when it tries to collect results from a segment and finds the connection has already dropped. So the coordinator itself is fine — it's basically just telling us "one of my segments died on me." The real crash happened on a segment, and that part isn't in what you shared.
   
   The upside: the stack does rule out a lot. It's not a parser/planner issue, not a cancellation or deadlock, not a resource-group admission thing, and not a QD bug. What's left is roughly:
   
   - a segment getting OOM-killed (honestly my first guess given the 90-day window plus those `array_agg(distinct concat(...) order by ...)` aggregates),
   - a segfault inside a QE — often a custom UDF, and `concat_my` jumps out at me here,
   - or some PANIC path that fired after interconnect teardown (less likely).
   
   To tell which one it is, would you mind grabbing a few extra things when you get a chance?
   
   The two that would help the most:
   
   - **Segment logs around the time it failed.** Something like \`grep -E 'PANIC|SIGSEGV|signal|terminating' pg_log/cbdb-*.log\` on the segment hosts — the PANIC/signal line and the stack right after it is exactly what's missing.
   - **\`dmesg -T | grep -iE 'killed process|out of memory'\`** on the segment hosts. If the OOM-killer was involved, this is where it'll show up.
   
   And if it's easy:
   
   - The Cloudberry version (\`SELECT version();\`).
   - The definition of \`concat_my\` (\`\df+ concat_my\`). If it's a custom C function, could you try swapping it for \`concat_ws(',', h1.agent_skilling, h2.agent_skilling)\` once and see if the crash still happens? That alone would tell us a lot.
   - Does it fail every time, or only on certain data sizes? Does a smaller window (say 7 days instead of 90) still reproduce it?
   - Values for \`statement_mem\`, \`max_statement_mem\`, \`gp_workfile_limit_per_segment\`, and which resource group this is running under.
   
   A coredump from a segment would be ideal if you happen to have \`kernel.core_pattern\` set up, but no worries if not — even just the segment log line and \`dmesg\` output should let us classify this pretty quickly.
   
   Thanks again, and looking forward to whatever you can share!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For additional commands, e-mail: commits-help@cloudberry.apache.org


Re: [I] Issue : server closed the connection unexpectedly This probably means the server terminated abnormally [cloudberry]

Posted by "yjhjstz (via GitHub)" <gi...@apache.org>.
yjhjstz commented on issue #1708:
URL: https://github.com/apache/cloudberry/issues/1708#issuecomment-5342538644

   fixed https://github.com/apache/cloudberry/pull/1902


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For additional commands, e-mail: commits-help@cloudberry.apache.org