You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by GitBox <gi...@apache.org> on 2022/08/07 15:15:39 UTC
[GitHub] [pulsar] CC-Hsu opened a new issue, #16974: JDBC Connection pool support for Pulsar JDBC sink connectors
CC-Hsu opened a new issue, #16974:
URL: https://github.com/apache/pulsar/issues/16974
**Is your feature request related to a problem? Please describe.**
Hi, Team,
Recently I want to try Pulsar [CDC Source Connector](https://pulsar.apache.org/docs/next/io-cdc) and [JDBC Sink Connector for PGSQL](https://pulsar.apache.org/docs/next/io-quickstart#connect-pulsar-to-postgresql) to sink multiple table data between PGSQLs.
But I find that the [JDBC sink connector for PGSQL](https://pulsar.apache.org/docs/next/io-jdbc-sink/#example-for-postgresql) is configured per-target-table basis; it seems that I can not sink several different target tables in a config file.
Although I can set up multiple configuration files for each target tables (the same target PGSQL database), this might cause the target side PGSQL pool performance due to many occupied connections.
In PGSQL, it is always a good practice [to utilize connect pool when many client-side connections are needed.](https://wiki.postgresql.org/wiki/Number_Of_Database_Connections#The_Need_for_an_External_Pool); and this might also holds for other databases (e.g. Cassandra).
I think Pulsar can provide JDBC connection pool for all supported JDBC sink connectors and other sink connectors if possible.
https://jdbc.postgresql.org/documentation/head/datasource.html
https://docs.datastax.com/en/developer/java-driver/4.14/manual/core/pooling/
https://dev.mysql.com/doc/connector-j/8.0/en/connector-j-usagenotes-j2ee-concepts-connection-pooling.html
https://mariadb.com/docs/connect/programming-languages/java/connection-pools/
**Describe the solution you'd like**
If my understanding is correct, Pulsar IO connectors are controlled by [`pulsar-admin` tool](https://pulsar.apache.org/docs/next/io-use/), it would be great to create connection pool for specified connector with some option of `pulsar-admin sinks` command.
- Providing a subcommand `pulsar-admin sinks pool` to create a connection pool resource
- `pulsar-admin sinks pool` may provide `--name` and `--pool-config`, etc, just like [`pulsar-admin sinks create` suboption.](https://pulsar.apache.org/tools/pulsar-admin/2.11.0-SNAPSHOT/#-em-create-em--30)
- Providing a new [JDBC sink configuration property](https://pulsar.apache.org/docs/next/io-jdbc-sink/#configuration) `connpool` to link with pre-configured connection pool.
**Additional context**
As far as I know, in kafka connect there is no such feature available.
If Pulsar can provide such database performance-friendly feature, this feature can be attractive for new users.
Best Regards.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] JDBC Connection pool support for Pulsar JDBC sink connectors [pulsar]
Posted by "merlimat (via GitHub)" <gi...@apache.org>.
merlimat closed issue #16974: JDBC Connection pool support for Pulsar JDBC sink connectors
URL: https://github.com/apache/pulsar/issues/16974
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] github-actions[bot] commented on issue #16974: JDBC Connection pool support for Pulsar JDBC sink connectors
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on issue #16974:
URL: https://github.com/apache/pulsar/issues/16974#issuecomment-1238840644
The issue had no activity for 30 days, mark with Stale label.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] JDBC Connection pool support for Pulsar JDBC sink connectors [pulsar]
Posted by "merlimat (via GitHub)" <gi...@apache.org>.
merlimat commented on issue #16974:
URL: https://github.com/apache/pulsar/issues/16974#issuecomment-6377131729
Thanks @CC-Hsu for the suggestion. The connectors, including the JDBC sink, moved to apache/pulsar-connectors in PIP-465. Each sink instance uses a single connection and runs as its own instance, often in a separate process, so a connection pool shared across sinks does not fit how connectors run. I'm closing this here. If you still need it, please open an issue in apache/pulsar-connectors for what you are trying to do (for example, one JDBC sink that writes to several tables).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org