You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by GitBox <gi...@apache.org> on 2022/10/11 16:03:46 UTC
[GitHub] [pulsar] nicoloboschi opened a new pull request, #18008: [improve][io] JDBC Sink: add flag to exclude non declared fields
nicoloboschi opened a new pull request, #18008:
URL: https://github.com/apache/pulsar/pull/18008
### Motivation
In the JDBC sink even if you don't add a field to `key` or `nonKey` all the table fields are read from the table metadata and included in the INSERT/UPSERT statement. This limits the possibility to leverage database fields default value.
### Modifications
* Added new flag `excludeNonDeclaredFields` (default false) to make the sink only include declared fields in the queries. The behavior is retro compatible with the current sinks since the default value is to still include non listed fields.
### Documentation
<!-- DO NOT REMOVE THIS SECTION. CHECK THE PROPER BOX ONLY. -->
- [x] `doc` <!-- Your PR contains doc changes. Please attach the local preview screenshots (run `sh start.sh` at `pulsar/site2/website`) to your PR description, or else your PR might not get merged. -->
- [ ] `doc-required` <!-- Your PR changes impact docs and you will update later -->
- [ ] `doc-not-needed` <!-- Your PR changes do not impact docs -->
- [ ] `doc-complete` <!-- Docs have been already added -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] Technoboy- closed pull request #18008: [improve][io] JDBC Sink: add flag to exclude non declared fields
Posted by GitBox <gi...@apache.org>.
Technoboy- closed pull request #18008: [improve][io] JDBC Sink: add flag to exclude non declared fields
URL: https://github.com/apache/pulsar/pull/18008
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] tisonkun commented on a diff in pull request #18008: [improve][io] JDBC Sink: add flag to exclude non declared fields
Posted by GitBox <gi...@apache.org>.
tisonkun commented on code in PR #18008:
URL: https://github.com/apache/pulsar/pull/18008#discussion_r996844061
##########
pulsar-io/jdbc/core/src/main/java/org/apache/pulsar/io/jdbc/JdbcSinkConfig.java:
##########
@@ -73,6 +73,17 @@ public class JdbcSinkConfig implements Serializable {
help = "Fields used in where condition of update and delete Events. A comma-separated list."
)
private String key;
+
+ @FieldDoc(
+ required = false,
+ defaultValue = "false",
+ help = "All the table fields are discovered automatically. 'excludeNonDeclaredFields' indicates if the "
+ + "table fields not explicitly listed in `nonKey` and `key` must be included in the query. "
+ + "By default all the table fields are included. To leverage of table fields defaults "
+ + "during insertion, it is suggested to set this value to `false`."
Review Comment:
```suggestion
+ "during insertion, it is suggested to set this value to `true`."
```
?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] nicoloboschi merged pull request #18008: [improve][io] JDBC Sink: add flag to exclude non declared fields
Posted by GitBox <gi...@apache.org>.
nicoloboschi merged PR #18008:
URL: https://github.com/apache/pulsar/pull/18008
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] nicoloboschi commented on a diff in pull request #18008: [improve][io] JDBC Sink: add flag to exclude non declared fields
Posted by GitBox <gi...@apache.org>.
nicoloboschi commented on code in PR #18008:
URL: https://github.com/apache/pulsar/pull/18008#discussion_r993525076
##########
site2/docs/io-jdbc-sink.md:
##########
@@ -15,19 +15,20 @@ The configuration of all JDBC sink connectors has the following properties.
### Property
-| Name | Type | Required | Default | Description |
-|-------------|--------|----------|--------------------|--------------------------------------------------------------------------------------------------------------------------|
-| `userName` | String | false | " " (empty string) | The username used to connect to the database specified by `jdbcUrl`.<br /><br />**Note: `userName` is case-sensitive.** |
-| `password` | String | false | " " (empty string) | The password used to connect to the database specified by `jdbcUrl`. <br /><br />**Note: `password` is case-sensitive.** |
-| `jdbcUrl` | String | true | " " (empty string) | The JDBC URL of the database that the connector connects to. |
-| `tableName` | String | true | " " (empty string) | The name of the table that the connector writes to. |
-| `nonKey` | String | false | " " (empty string) | A comma-separated list containing the fields used in updating events. |
-| `key` | String | false | " " (empty string) | A comma-separated list containing the fields used in `where` condition of updating and deleting events. |
-| `timeoutMs` | int | false | 500 | The JDBC operation timeout in milliseconds. |
-| `batchSize` | int | false | 200 | The batch size of updates made to the database. |
-| `insertMode` | enum( INSERT,UPSERT,UPDATE) | false | INSERT | If it is configured as UPSERT, the sink uses upsert semantics rather than plain INSERT/UPDATE statements. Upsert semantics refer to atomically adding a new row or updating the existing row if there is a primary key constraint violation, which provides idempotence. |
-| `nullValueAction` | enum(FAIL, DELETE) | false | FAIL | How to handle records with NULL values. Possible options are `DELETE` or `FAIL`. |
-| `useTransactions` | boolean | false | true | Enable transactions of the database.
+| Name | Type | Required | Default | Description |
+|-------------|--------|----------|--------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| `userName` | String | false | " " (empty string) | The username used to connect to the database specified by `jdbcUrl`.<br /><br />**Note: `userName` is case-sensitive.** |
+| `password` | String | false | " " (empty string) | The password used to connect to the database specified by `jdbcUrl`. <br /><br />**Note: `password` is case-sensitive.** |
+| `jdbcUrl` | String | true | " " (empty string) | The JDBC URL of the database that the connector connects to. |
+| `tableName` | String | true | " " (empty string) | The name of the table that the connector writes to. |
+| `nonKey` | String | false | " " (empty string) | A comma-separated list containing the fields used in updating events. |
+| `key` | String | false | " " (empty string) | A comma-separated list containing the fields used in `where` condition of updating and deleting events. |
+| `timeoutMs` | int | false | 500 | The JDBC operation timeout in milliseconds. |
+| `batchSize` | int | false | 200 | The batch size of updates made to the database. |
+| `insertMode` | enum( INSERT,UPSERT,UPDATE) | false | INSERT | If it is configured as UPSERT, the sink uses upsert semantics rather than plain INSERT/UPDATE statements. Upsert semantics refer to atomically adding a new row or updating the existing row if there is a primary key constraint violation, which provides idempotence. |
+| `nullValueAction` | enum(FAIL, DELETE) | false | FAIL | How to handle records with NULL values. Possible options are `DELETE` or `FAIL`. |
+| `useTransactions` | boolean | false | true | Enable transactions of the database.
+| `excludeNonDeclaredFields` | boolean | false | false | All the table fields are discovered automatically. This fields indicate if the fields not explicitly listed in `nonKey` and `key` must be included in the query. By default all the fields of the table are included. In order to leverage of fields defaults during insertion, it is suggested to set this value to `false`. |
Review Comment:
Nope but thanks, I updated the doc
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] nicoloboschi commented on a diff in pull request #18008: [improve][io] JDBC Sink: add flag to exclude non declared fields
Posted by GitBox <gi...@apache.org>.
nicoloboschi commented on code in PR #18008:
URL: https://github.com/apache/pulsar/pull/18008#discussion_r996862019
##########
pulsar-io/jdbc/core/src/main/java/org/apache/pulsar/io/jdbc/JdbcSinkConfig.java:
##########
@@ -73,6 +73,17 @@ public class JdbcSinkConfig implements Serializable {
help = "Fields used in where condition of update and delete Events. A comma-separated list."
)
private String key;
+
+ @FieldDoc(
+ required = false,
+ defaultValue = "false",
+ help = "All the table fields are discovered automatically. 'excludeNonDeclaredFields' indicates if the "
+ + "table fields not explicitly listed in `nonKey` and `key` must be included in the query. "
+ + "By default all the table fields are included. To leverage of table fields defaults "
+ + "during insertion, it is suggested to set this value to `false`."
Review Comment:
🤦 thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] Anonymitaet commented on a diff in pull request #18008: [improve][io] JDBC Sink: add flag to exclude non declared fields
Posted by GitBox <gi...@apache.org>.
Anonymitaet commented on code in PR #18008:
URL: https://github.com/apache/pulsar/pull/18008#discussion_r992925241
##########
site2/docs/io-jdbc-sink.md:
##########
@@ -15,19 +15,20 @@ The configuration of all JDBC sink connectors has the following properties.
### Property
-| Name | Type | Required | Default | Description |
-|-------------|--------|----------|--------------------|--------------------------------------------------------------------------------------------------------------------------|
-| `userName` | String | false | " " (empty string) | The username used to connect to the database specified by `jdbcUrl`.<br /><br />**Note: `userName` is case-sensitive.** |
-| `password` | String | false | " " (empty string) | The password used to connect to the database specified by `jdbcUrl`. <br /><br />**Note: `password` is case-sensitive.** |
-| `jdbcUrl` | String | true | " " (empty string) | The JDBC URL of the database that the connector connects to. |
-| `tableName` | String | true | " " (empty string) | The name of the table that the connector writes to. |
-| `nonKey` | String | false | " " (empty string) | A comma-separated list containing the fields used in updating events. |
-| `key` | String | false | " " (empty string) | A comma-separated list containing the fields used in `where` condition of updating and deleting events. |
-| `timeoutMs` | int | false | 500 | The JDBC operation timeout in milliseconds. |
-| `batchSize` | int | false | 200 | The batch size of updates made to the database. |
-| `insertMode` | enum( INSERT,UPSERT,UPDATE) | false | INSERT | If it is configured as UPSERT, the sink uses upsert semantics rather than plain INSERT/UPDATE statements. Upsert semantics refer to atomically adding a new row or updating the existing row if there is a primary key constraint violation, which provides idempotence. |
-| `nullValueAction` | enum(FAIL, DELETE) | false | FAIL | How to handle records with NULL values. Possible options are `DELETE` or `FAIL`. |
-| `useTransactions` | boolean | false | true | Enable transactions of the database.
+| Name | Type | Required | Default | Description |
+|-------------|--------|----------|--------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| `userName` | String | false | " " (empty string) | The username used to connect to the database specified by `jdbcUrl`.<br /><br />**Note: `userName` is case-sensitive.** |
+| `password` | String | false | " " (empty string) | The password used to connect to the database specified by `jdbcUrl`. <br /><br />**Note: `password` is case-sensitive.** |
+| `jdbcUrl` | String | true | " " (empty string) | The JDBC URL of the database that the connector connects to. |
+| `tableName` | String | true | " " (empty string) | The name of the table that the connector writes to. |
+| `nonKey` | String | false | " " (empty string) | A comma-separated list containing the fields used in updating events. |
+| `key` | String | false | " " (empty string) | A comma-separated list containing the fields used in `where` condition of updating and deleting events. |
+| `timeoutMs` | int | false | 500 | The JDBC operation timeout in milliseconds. |
+| `batchSize` | int | false | 200 | The batch size of updates made to the database. |
+| `insertMode` | enum( INSERT,UPSERT,UPDATE) | false | INSERT | If it is configured as UPSERT, the sink uses upsert semantics rather than plain INSERT/UPDATE statements. Upsert semantics refer to atomically adding a new row or updating the existing row if there is a primary key constraint violation, which provides idempotence. |
+| `nullValueAction` | enum(FAIL, DELETE) | false | FAIL | How to handle records with NULL values. Possible options are `DELETE` or `FAIL`. |
+| `useTransactions` | boolean | false | true | Enable transactions of the database.
+| `excludeNonDeclaredFields` | boolean | false | false | All the table fields are discovered automatically. This fields indicate if the fields not explicitly listed in `nonKey` and `key` must be included in the query. By default all the fields of the table are included. In order to leverage of fields defaults during insertion, it is suggested to set this value to `false`. |
Review Comment:
```suggestion
| `excludeNonDeclaredFields` | boolean | false | false | All the table fields are discovered automatically. These fields indicate if the fields are not explicitly listed in `nonKey` and `key`, they must be included in the query. By default, all the fields of the table are included. To leverage the default values of fields during insertion, it is suggested to set this value to `false`. |
```
Do you mean this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org