You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@flink.apache.org by ro...@apache.org on 2021/08/25 15:10:44 UTC
[flink] branch master updated (97650c0 -> ad104ee)
This is an automated email from the ASF dual-hosted git repository.
roman pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/flink.git.
from 97650c0 [FLINK-23911][table-planner] Apply metadata spec even if no metadata were used
new 221b5c3 [hotfix][docs] Remove list of supported JDBC drivers
new ace1ce7 [FLINK-23876][docs] Add JDBC XA DataSource examples
new ad104ee [FLINK-23776][docs] Clarify JDBC XA sink usage
The 3 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
docs/content/docs/connectors/datastream/jdbc.md | 58 ++++++++++++++++------
.../connector/jdbc/xa/JdbcXaSinkFunction.java | 4 --
2 files changed, 43 insertions(+), 19 deletions(-)
[flink] 02/03: [FLINK-23876][docs] Add JDBC XA DataSource examples
Posted by ro...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
roman pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/flink.git
commit ace1ce7119a7e5fb7b69a46b86c3249febd3bf46
Author: Roman Khachatryan <kh...@gmail.com>
AuthorDate: Thu Aug 19 21:17:56 2021 +0100
[FLINK-23876][docs] Add JDBC XA DataSource examples
---
docs/content/docs/connectors/datastream/jdbc.md | 28 ++++++++++++++++++++++---
1 file changed, 25 insertions(+), 3 deletions(-)
diff --git a/docs/content/docs/connectors/datastream/jdbc.md b/docs/content/docs/connectors/datastream/jdbc.md
index 171cdc6..db0a784 100644
--- a/docs/content/docs/connectors/datastream/jdbc.md
+++ b/docs/content/docs/connectors/datastream/jdbc.md
@@ -182,9 +182,31 @@ env.execute();
```
Postgres XADataSource Example:
```java
-PGXADataSource pgxaDataSource = new PGXADataSource();
-pgxaDataSource.setUrl(
-"jdbc:postgresql://localhost:5432/postgres");
+
+### `XADataSource` examples
+PostgreSQL `XADataSource` example:
+```java
+PGXADataSource xaDataSource = new org.postgresql.xa.PGXADataSource();
+xaDataSource.setUrl("jdbc:postgresql://localhost:5432/postgres");
+xaDataSource.setUser(username);
+xaDataSource.setPassword(password);
+```
+
+MySQL `XADataSource` example:
+```java
+MysqlXADataSource xaDataSource = new com.mysql.cj.jdbc.MysqlXADataSource();
+xaDataSource.setUrl("jdbc:mysql://localhost:3306/");
+xaDataSource.setUser(username);
+xaDataSource.setPassword(password);
+```
+
+Oracle `XADataSource` example:
+```java
+OracleXADataSource xaDataSource = new oracle.jdbc.xa.OracleXADataSource();
+xaDataSource.setURL("jdbc:oracle:oci8:@");
+xaDataSource.setUser("scott");
+xaDataSource.setPassword("tiger");
```
+Please also take Oracle connection pooling into account.
Please refer to the `JdbcXaSinkFunction` documentation for more details.
[flink] 01/03: [hotfix][docs] Remove list of supported JDBC drivers
Posted by ro...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
roman pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/flink.git
commit 221b5c3fd575fe6011527ce000a72529ab5de929
Author: Roman Khachatryan <kh...@gmail.com>
AuthorDate: Thu Aug 19 21:07:26 2021 +0100
[hotfix][docs] Remove list of supported JDBC drivers
JDBC DataStream sink only relies on JDBC standard
(as opposed to Table API which lists dialects
explicitly).
---
docs/content/docs/connectors/datastream/jdbc.md | 9 +--------
1 file changed, 1 insertion(+), 8 deletions(-)
diff --git a/docs/content/docs/connectors/datastream/jdbc.md b/docs/content/docs/connectors/datastream/jdbc.md
index 8a21340..171cdc6 100644
--- a/docs/content/docs/connectors/datastream/jdbc.md
+++ b/docs/content/docs/connectors/datastream/jdbc.md
@@ -32,15 +32,8 @@ To use it, add the following dependency to your project (along with your JDBC dr
{{< artifact flink-connector-jdbc withScalaVersion >}}
-A driver dependency is also required to connect to a specified database. Here are drivers currently supported:
-
-| Driver | Group Id | Artifact Id | JAR |
-| :-----------| :------------------| :----------------------| :----------------|
-| MySQL | `mysql` | `mysql-connector-java` | [Download](https://repo.maven.apache.org/maven2/mysql/mysql-connector-java/) |
-| PostgreSQL | `org.postgresql` | `postgresql` | [Download](https://jdbc.postgresql.org/download.html) |
-| Derby | `org.apache.derby` | `derby` | [Download](http://db.apache.org/derby/derby_downloads.html) |
-
Note that the streaming connectors are currently __NOT__ part of the binary distribution. See how to link with them for cluster execution [here]({{< ref "docs/dev/datastream/project-configuration" >}}).
+A driver dependency is also required to connect to a specified database. Please consult your database documentation on how to add the corresponding driver.
## `JdbcSink.sink`
[flink] 03/03: [FLINK-23776][docs] Clarify JDBC XA sink usage
Posted by ro...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
roman pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/flink.git
commit ad104eecaca9cb0189712a6a9044a92bbbddc931
Author: Roman Khachatryan <kh...@gmail.com>
AuthorDate: Thu Aug 19 21:20:20 2021 +0100
[FLINK-23776][docs] Clarify JDBC XA sink usage
---
docs/content/docs/connectors/datastream/jdbc.md | 21 +++++++++++++++++----
.../flink/connector/jdbc/xa/JdbcXaSinkFunction.java | 4 ----
2 files changed, 17 insertions(+), 8 deletions(-)
diff --git a/docs/content/docs/connectors/datastream/jdbc.md b/docs/content/docs/connectors/datastream/jdbc.md
index db0a784..76da068 100644
--- a/docs/content/docs/connectors/datastream/jdbc.md
+++ b/docs/content/docs/connectors/datastream/jdbc.md
@@ -145,15 +145,14 @@ public class JdbcSinkExample {
Since 1.13, Flink JDBC sink supports exactly-once mode.
The implementation relies on the JDBC driver support of XA
[standard](https://pubs.opengroup.org/onlinepubs/009680699/toc.pdf).
+Most drivers support XA if the database also supports XA (so the driver is usually the same).
To use it, create a sink using `exactlyOnceSink()` method as above and additionally provide:
- {{< javadoc name="exactly-once options" file="org/apache/flink/connector/jdbc/JdbcExactlyOnceOptions.html" >}}
- {{< javadoc name="execution options" file="org/apache/flink/connector/jdbc/JdbcExecutionOptions.html" >}}
- [XA DataSource](https://docs.oracle.com/javase/8/docs/api/javax/sql/XADataSource.html) Supplier
-**ATTENTION!** Currently `JdbcSink.exactlyOnceSink` can ensure exactly once semantics
-with `JdbcExecutionOptions.maxRetries == 0`; otherwise, duplicated results maybe produced.
-
+For example:
```java
StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
env
@@ -180,8 +179,22 @@ env
});
env.execute();
```
-Postgres XADataSource Example:
+**NOTE:** Some databases only allow a single XA transaction per connection (e.g. PostgreSQL, MySQL).
+In such cases, please use the following API to construct `JdbcExactlyOnceOptions`:
```java
+JdbcExactlyOnceOptions.builder()
+.withTransactionPerConnection(true)
+.build()
+```
+This will make Flink use a separate connection for every XA transaction. This may require adjusting connection limits.
+For PostgreSQL and MySQL, this can be done by increasing `max_connections`.
+
+Furthermore, XA needs to be enabled and/or configured in some databases.
+For PostgreSQL, you should set `max_prepared_transactions` to some value greater than zero.
+For MySQL v8+, you should grant `XA_RECOVER_ADMIN` to Flink DB user.
+
+**ATTENTION:** Currently, `JdbcSink.exactlyOnceSink` can ensure exactly once semantics
+with `JdbcExecutionOptions.maxRetries == 0`; otherwise, duplicated results maybe produced.
### `XADataSource` examples
PostgreSQL `XADataSource` example:
diff --git a/flink-connectors/flink-connector-jdbc/src/main/java/org/apache/flink/connector/jdbc/xa/JdbcXaSinkFunction.java b/flink-connectors/flink-connector-jdbc/src/main/java/org/apache/flink/connector/jdbc/xa/JdbcXaSinkFunction.java
index 8d38028..31ca668 100644
--- a/flink-connectors/flink-connector-jdbc/src/main/java/org/apache/flink/connector/jdbc/xa/JdbcXaSinkFunction.java
+++ b/flink-connectors/flink-connector-jdbc/src/main/java/org/apache/flink/connector/jdbc/xa/JdbcXaSinkFunction.java
@@ -124,10 +124,6 @@ import static org.apache.flink.connector.jdbc.xa.JdbcXaSinkFunctionState.of;
* </tbody>
* </table>
*
- * <p>Attention: JdbcXaSinkFunction does not support exactly-once mode with MySQL or other databases
- * that do not support multiple XA transaction per connection. We will improve the support in
- * FLINK-22239.
- *
* @since 1.13
*/
@Internal