You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@flink.apache.org by "olivier-derom (via GitHub)" <gi...@apache.org> on 2024/03/06 08:55:05 UTC
[I] Error on Postgres-CDC using incremental snapshot with UUID column as PK [flink-cdc]
olivier-derom opened a new issue, #3108:
URL: https://github.com/apache/flink-cdc/issues/3108
A majority of our Postgres databases use UUIDs as primary keys.
When we enable 'scan.incremental.snapshot.enabled = true', Flink-CDC will try to split into chunks.
The splitTableIntoChunks function relies on the queryMinMax function, which fails when trying to calculate the MIN(UUID), as that is not supported in Postgres.
Is there a way around this?
When we convert our column to VARCHAR, rather than UUID, everything seems to work.
We did not find a way to cast our UUIDs to VARCHAR while splitting them into chunks without editing the source code or altering the source table.
Disabling incremental snapshots also fixes the issue, as we do not split into chunks anymore, but this would mean we get a global read lock on the data before snapshot reading, which we want to avoid.
Thanks in advance for the help!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@flink.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Error on Postgres-CDC using incremental snapshot with UUID column as PK [flink-cdc]
Posted by "PatrickRen (via GitHub)" <gi...@apache.org>.
PatrickRen closed issue #3108: Error on Postgres-CDC using incremental snapshot with UUID column as PK
URL: https://github.com/apache/flink-cdc/issues/3108
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Error on Postgres-CDC using incremental snapshot with UUID column as PK [flink-cdc]
Posted by "loserwang1024 (via GitHub)" <gi...@apache.org>.
loserwang1024 commented on issue #3108:
URL: https://github.com/apache/flink-cdc/issues/3108#issuecomment-2033865296
@olivier-derom It seems a bug, could you please create a [jira ](https://issues.apache.org/jira/browse/FLINK-34952)and fix it? If you don't have time, I'd like to help you todo it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Error on Postgres-CDC using incremental snapshot with UUID column as PK [flink-cdc]
Posted by "drorventura (via GitHub)" <gi...@apache.org>.
drorventura commented on issue #3108:
URL: https://github.com/apache/flink-cdc/issues/3108#issuecomment-2003458861
@olivier-derom were you able to make that work?
if not then this should solve your missing functions:
```
create or replace function min(uuid, uuid)
returns uuid
immutable parallel safe
language plpgsql as
$$
begin
return least($1, $2);
end
$$;
create aggregate min(uuid) (
sfunc = min,
stype = uuid,
combinefunc = min,
parallel = safe,
sortop = operator (<)
);
create or replace function max(uuid, uuid)
returns uuid
immutable parallel safe
language plpgsql as
$$
begin
return greatest($1, $2);
end
$$;
create aggregate max(uuid) (
sfunc = max,
stype = uuid,
combinefunc = max,
parallel = safe,
sortop = operator (>)
);
```
however it would still not work since uuid type is not supported or mapper in the
`com.ververica.cdc.connectors.postgres.source.utils.PostgresTypeUtils.convertFromColumn`
and requires a mapping to DataType.STRING()
I wasn't able to proceed with the contribution yet, however I do wonder how did you manage to get through?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Error on Postgres-CDC using incremental snapshot with UUID column as PK [flink-cdc]
Posted by "olivier-derom (via GitHub)" <gi...@apache.org>.
olivier-derom commented on issue #3108:
URL: https://github.com/apache/flink-cdc/issues/3108#issuecomment-2003696012
@drorventura
As per my last reply, I indeed created my own functions that are able to calculate the min/max of UUID's, similar to how you defined them.
While this allows us to select min/max(UUID) when executing a query on the database in an SQL client like DBeaver, Flink was not able to use these functions, and I haven't figured out how to make it work yet.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Error on Postgres-CDC using incremental snapshot with UUID column as PK [flink-cdc]
Posted by "olivier-derom (via GitHub)" <gi...@apache.org>.
olivier-derom commented on issue #3108:
URL: https://github.com/apache/flink-cdc/issues/3108#issuecomment-1980957733
I figured this is a limitation of Postgres and not Flink-CDC, so tried to create my own min(uuid, uuid) and max(uuid, uuid) as well as aggregate min(uuid) and max(uuid).
They work perfectly when I execute the SQL statement myself on the database, but Flink-CDC still says 'function min(uuid) does not exist'.
Does it somehow not have access to custom functions?
The database connection in Flink-CDC is made using the same credentials as I created the function with, so the user should have access.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Error on Postgres-CDC using incremental snapshot with UUID column as PK [flink-cdc]
Posted by "edwardzcn-decade (via GitHub)" <gi...@apache.org>.
edwardzcn-decade commented on issue #3108:
URL: https://github.com/apache/flink-cdc/issues/3108#issuecomment-3431161241
> Hi [@PatrickRen](https://github.com/PatrickRen), is there an official solution on this issue now?
The migrant issue [FLINK-34883 on Apache Jira](https://issues.apache.org/jira/browse/FLINK-34883) is marked as resolved via master: 94a0415475e7040cc6a61e9bd8aca9c3ccea80ad
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Error on Postgres-CDC using incremental snapshot with UUID column as PK [flink-cdc]
Posted by "PatrickRen (via GitHub)" <gi...@apache.org>.
PatrickRen commented on issue #3108:
URL: https://github.com/apache/flink-cdc/issues/3108#issuecomment-2044202366
Closing this issue as it has been migrated to [Apache Jira](https://issues.apache.org/jira).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Error on Postgres-CDC using incremental snapshot with UUID column as PK [flink-cdc]
Posted by "yiqinghuang (via GitHub)" <gi...@apache.org>.
yiqinghuang commented on issue #3108:
URL: https://github.com/apache/flink-cdc/issues/3108#issuecomment-3257205182
Hi @PatrickRen, is there a official solution on this now?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org