You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by GitBox <gi...@apache.org> on 2020/04/13 05:16:12 UTC
[GitHub] [flink] danny0405 opened a new pull request #11711: [FLINK-17103][table-planner-blink] Supports dynamic options for Blink…
danny0405 opened a new pull request #11711: [FLINK-17103][table-planner-blink] Supports dynamic options for Blink…
URL: https://github.com/apache/flink/pull/11711
… planner
## What is the purpose of the change
Supports syntax "table /*+ OPTIONS('k1'='v1', 'k2'='v2') */" to specify dynamic options within the scope of the appended table.
## Brief change log
* Implement dynamic options in Blink planner
* Implement CSV connector
* Add plan test OptionsHintTest
* Add a Csv ITCase in CatalogTableITCase
## Verifying this change
See ITCases and UT tests in the patch.
## Does this pull request potentially affect one of the following parts:
- Dependencies (does it add or upgrade a dependency): no
- The public API, i.e., is any changed class annotated with `@Public(Evolving)`: yes
- The serializers: no
- The runtime per-record code paths (performance sensitive): no
- Anything that affects deployment or recovery: no
- The S3 file system connector: no
## Documentation
- Does this pull request introduce a new feature? yes
- If yes, how is the feature documented? not documented
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] flinkbot commented on issue #11711: [FLINK-17103][table-planner-blink] Supports dynamic options for Blink…
Posted by GitBox <gi...@apache.org>.
flinkbot commented on issue #11711: [FLINK-17103][table-planner-blink] Supports dynamic options for Blink…
URL: https://github.com/apache/flink/pull/11711#issuecomment-612754969
Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community
to review your pull request. We will use this comment to track the progress of the review.
## Automated Checks
Last check on commit bffb4b07100999b88c67103a1c77d0609c524de9 (Mon Apr 13 05:19:59 UTC 2020)
**Warnings:**
* No documentation files were touched! Remember to keep the Flink docs up to date!
* **This pull request references an unassigned [Jira ticket](https://issues.apache.org/jira/browse/FLINK-17103).** According to the [code contribution guide](https://flink.apache.org/contributing/contribute-code.html), tickets need to be assigned before starting with the implementation work.
<sub>Mention the bot in a comment to re-run the automated checks.</sub>
## Review Progress
* ❓ 1. The [description] looks good.
* ❓ 2. There is [consensus] that the contribution should go into to Flink.
* ❓ 3. Needs [attention] from.
* ❓ 4. The change fits into the overall [architecture].
* ❓ 5. Overall code [quality] is good.
Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process.<details>
The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required <summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`)
- `@flinkbot approve all` to approve all aspects
- `@flinkbot approve-until architecture` to approve everything until `architecture`
- `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention
- `@flinkbot disapprove architecture` to remove an approval you gave earlier
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] flinkbot commented on issue #11711: [FLINK-17103][table-planner-blink] Supports dynamic options for Blink…
Posted by GitBox <gi...@apache.org>.
flinkbot commented on issue #11711: [FLINK-17103][table-planner-blink] Supports dynamic options for Blink…
URL: https://github.com/apache/flink/pull/11711#issuecomment-612758431
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "bffb4b07100999b88c67103a1c77d0609c524de9",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "bffb4b07100999b88c67103a1c77d0609c524de9",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* bffb4b07100999b88c67103a1c77d0609c524de9 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] KurtYoung commented on a change in pull request #11711: [FLINK-17103][table-planner-blink] Supports dynamic options for Blink…
Posted by GitBox <gi...@apache.org>.
KurtYoung commented on a change in pull request #11711: [FLINK-17103][table-planner-blink] Supports dynamic options for Blink…
URL: https://github.com/apache/flink/pull/11711#discussion_r408122192
##########
File path: flink-table/flink-table-planner-blink/src/main/scala/org/apache/flink/table/planner/plan/schema/TableSourceTable.scala
##########
@@ -61,11 +64,33 @@ class TableSourceTable[T](
tableIdentifier.getObjectName),
statistic) {
+ def this(relOptSchema: RelOptSchema,
+ tableIdentifier: ObjectIdentifier,
+ rowType: RelDataType,
+ statistic: FlinkStatistic,
+ tableSource: TableSource[T],
+ isStreamingMode: Boolean,
+ catalogTable: CatalogTable) = this(relOptSchema, tableIdentifier, rowType, statistic,
+ tableSource, isStreamingMode, catalogTable, Collections.emptyMap())
+
Preconditions.checkNotNull(tableSource)
Preconditions.checkNotNull(statistic)
Preconditions.checkNotNull(catalogTable)
- override def getQualifiedName: JList[String] = explainSourceAsString(tableSource)
+ override def getQualifiedName: JList[String] = {
+ val sourceExplain = explainSourceAsString(tableSource)
+ if (dynamicOptions.size() == 0) {
+ sourceExplain
+ } else {
+ // Add the dynamic options as part of the table digest,
+ // this is a temporary solution, we expect to avoid this
+ // before Calcite 1.23.0.
Review comment:
Create an issue to track this?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] danny0405 commented on a change in pull request #11711: [FLINK-17103][table-planner-blink] Supports dynamic options for Blink…
Posted by GitBox <gi...@apache.org>.
danny0405 commented on a change in pull request #11711: [FLINK-17103][table-planner-blink] Supports dynamic options for Blink…
URL: https://github.com/apache/flink/pull/11711#discussion_r408586396
##########
File path: flink-table/flink-table-planner-blink/src/main/scala/org/apache/flink/table/planner/plan/schema/TableSourceTable.scala
##########
@@ -61,11 +64,33 @@ class TableSourceTable[T](
tableIdentifier.getObjectName),
statistic) {
+ def this(relOptSchema: RelOptSchema,
+ tableIdentifier: ObjectIdentifier,
+ rowType: RelDataType,
+ statistic: FlinkStatistic,
+ tableSource: TableSource[T],
+ isStreamingMode: Boolean,
+ catalogTable: CatalogTable) = this(relOptSchema, tableIdentifier, rowType, statistic,
+ tableSource, isStreamingMode, catalogTable, Collections.emptyMap())
+
Preconditions.checkNotNull(tableSource)
Preconditions.checkNotNull(statistic)
Preconditions.checkNotNull(catalogTable)
- override def getQualifiedName: JList[String] = explainSourceAsString(tableSource)
+ override def getQualifiedName: JList[String] = {
+ val sourceExplain = explainSourceAsString(tableSource)
+ if (dynamicOptions.size() == 0) {
+ sourceExplain
+ } else {
+ // Add the dynamic options as part of the table digest,
+ // this is a temporary solution, we expect to avoid this
+ // before Calcite 1.23.0.
Review comment:
Thanks, i have fired a issue https://issues.apache.org/jira/browse/FLINK-17147
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] KurtYoung commented on a change in pull request #11711: [FLINK-17103][table-planner-blink] Supports dynamic options for Blink…
Posted by GitBox <gi...@apache.org>.
KurtYoung commented on a change in pull request #11711: [FLINK-17103][table-planner-blink] Supports dynamic options for Blink…
URL: https://github.com/apache/flink/pull/11711#discussion_r407362506
##########
File path: flink-table/flink-table-api-java/src/main/java/org/apache/flink/table/operations/CatalogSinkModifyOperation.java
##########
@@ -37,20 +37,23 @@
private final Map<String, String> staticPartitions;
private final QueryOperation child;
private final boolean overwrite;
+ private final List<?> tableHints;
Review comment:
why use `List<?>`?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #11711: [FLINK-17103][table-planner-blink] Supports dynamic options for Blink…
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on issue #11711: [FLINK-17103][table-planner-blink] Supports dynamic options for Blink…
URL: https://github.com/apache/flink/pull/11711#issuecomment-612758431
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "bffb4b07100999b88c67103a1c77d0609c524de9",
"status" : "FAILURE",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/160012254",
"triggerID" : "bffb4b07100999b88c67103a1c77d0609c524de9",
"triggerType" : "PUSH"
}, {
"hash" : "bffb4b07100999b88c67103a1c77d0609c524de9",
"status" : "FAILURE",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7381",
"triggerID" : "bffb4b07100999b88c67103a1c77d0609c524de9",
"triggerType" : "PUSH"
}, {
"hash" : "3ea01f3d73194e787d483bc463ddaa5b58a3dd9f",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "3ea01f3d73194e787d483bc463ddaa5b58a3dd9f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* bffb4b07100999b88c67103a1c77d0609c524de9 Travis: [FAILURE](https://travis-ci.com/github/flink-ci/flink/builds/160012254) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7381)
* 3ea01f3d73194e787d483bc463ddaa5b58a3dd9f UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] KurtYoung closed pull request #11711: [FLINK-17103][table-planner-blink] Supports dynamic options for Blink…
Posted by GitBox <gi...@apache.org>.
KurtYoung closed pull request #11711: [FLINK-17103][table-planner-blink] Supports dynamic options for Blink…
URL: https://github.com/apache/flink/pull/11711
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] KurtYoung commented on a change in pull request #11711: [FLINK-17103][table-planner-blink] Supports dynamic options for Blink…
Posted by GitBox <gi...@apache.org>.
KurtYoung commented on a change in pull request #11711: [FLINK-17103][table-planner-blink] Supports dynamic options for Blink…
URL: https://github.com/apache/flink/pull/11711#discussion_r408122192
##########
File path: flink-table/flink-table-planner-blink/src/main/scala/org/apache/flink/table/planner/plan/schema/TableSourceTable.scala
##########
@@ -61,11 +64,33 @@ class TableSourceTable[T](
tableIdentifier.getObjectName),
statistic) {
+ def this(relOptSchema: RelOptSchema,
+ tableIdentifier: ObjectIdentifier,
+ rowType: RelDataType,
+ statistic: FlinkStatistic,
+ tableSource: TableSource[T],
+ isStreamingMode: Boolean,
+ catalogTable: CatalogTable) = this(relOptSchema, tableIdentifier, rowType, statistic,
+ tableSource, isStreamingMode, catalogTable, Collections.emptyMap())
+
Preconditions.checkNotNull(tableSource)
Preconditions.checkNotNull(statistic)
Preconditions.checkNotNull(catalogTable)
- override def getQualifiedName: JList[String] = explainSourceAsString(tableSource)
+ override def getQualifiedName: JList[String] = {
+ val sourceExplain = explainSourceAsString(tableSource)
+ if (dynamicOptions.size() == 0) {
+ sourceExplain
+ } else {
+ // Add the dynamic options as part of the table digest,
+ // this is a temporary solution, we expect to avoid this
+ // before Calcite 1.23.0.
Review comment:
Create a issue to track this?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] danny0405 commented on a change in pull request #11711: [FLINK-17103][table-planner-blink] Supports dynamic options for Blink…
Posted by GitBox <gi...@apache.org>.
danny0405 commented on a change in pull request #11711: [FLINK-17103][table-planner-blink] Supports dynamic options for Blink…
URL: https://github.com/apache/flink/pull/11711#discussion_r407405012
##########
File path: flink-table/flink-table-api-java/src/main/java/org/apache/flink/table/catalog/ConnectorCatalogTable.java
##########
@@ -102,6 +102,11 @@ public boolean isBatch() {
throw new UnsupportedOperationException("ConnectorCatalogTable cannot be converted to properties");
}
+ @Override
+ public CatalogTable copy(Map<String, String> options) {
+ throw new UnsupportedOperationException("ConnectorCatalogTable cannot copy with new table options");
Review comment:
All the table translating through `CatalogTableImpl` can have valid dynamic table options, including the table from `TableEnvironment.connect`. There is no way for a `ConnectorCatalogTable` to override table options because the table source is already there.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #11711: [FLINK-17103][table-planner-blink] Supports dynamic options for Blink…
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on issue #11711: [FLINK-17103][table-planner-blink] Supports dynamic options for Blink…
URL: https://github.com/apache/flink/pull/11711#issuecomment-612758431
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "bffb4b07100999b88c67103a1c77d0609c524de9",
"status" : "DELETED",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/160012254",
"triggerID" : "bffb4b07100999b88c67103a1c77d0609c524de9",
"triggerType" : "PUSH"
}, {
"hash" : "bffb4b07100999b88c67103a1c77d0609c524de9",
"status" : "DELETED",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7381",
"triggerID" : "bffb4b07100999b88c67103a1c77d0609c524de9",
"triggerType" : "PUSH"
}, {
"hash" : "3ea01f3d73194e787d483bc463ddaa5b58a3dd9f",
"status" : "PENDING",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7456",
"triggerID" : "3ea01f3d73194e787d483bc463ddaa5b58a3dd9f",
"triggerType" : "PUSH"
}, {
"hash" : "3ea01f3d73194e787d483bc463ddaa5b58a3dd9f",
"status" : "SUCCESS",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/160195594",
"triggerID" : "3ea01f3d73194e787d483bc463ddaa5b58a3dd9f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 3ea01f3d73194e787d483bc463ddaa5b58a3dd9f Travis: [SUCCESS](https://travis-ci.com/github/flink-ci/flink/builds/160195594) Azure: [PENDING](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7456)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #11711: [FLINK-17103][table-planner-blink] Supports dynamic options for Blink…
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on issue #11711: [FLINK-17103][table-planner-blink] Supports dynamic options for Blink…
URL: https://github.com/apache/flink/pull/11711#issuecomment-612758431
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "bffb4b07100999b88c67103a1c77d0609c524de9",
"status" : "DELETED",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/160012254",
"triggerID" : "bffb4b07100999b88c67103a1c77d0609c524de9",
"triggerType" : "PUSH"
}, {
"hash" : "bffb4b07100999b88c67103a1c77d0609c524de9",
"status" : "DELETED",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7381",
"triggerID" : "bffb4b07100999b88c67103a1c77d0609c524de9",
"triggerType" : "PUSH"
}, {
"hash" : "3ea01f3d73194e787d483bc463ddaa5b58a3dd9f",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7456",
"triggerID" : "3ea01f3d73194e787d483bc463ddaa5b58a3dd9f",
"triggerType" : "PUSH"
}, {
"hash" : "3ea01f3d73194e787d483bc463ddaa5b58a3dd9f",
"status" : "SUCCESS",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/160195594",
"triggerID" : "3ea01f3d73194e787d483bc463ddaa5b58a3dd9f",
"triggerType" : "PUSH"
}, {
"hash" : "84098c147d17a469c44710ba87fbac41a9aa387d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "84098c147d17a469c44710ba87fbac41a9aa387d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 3ea01f3d73194e787d483bc463ddaa5b58a3dd9f Travis: [SUCCESS](https://travis-ci.com/github/flink-ci/flink/builds/160195594) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7456)
* 84098c147d17a469c44710ba87fbac41a9aa387d UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] KurtYoung commented on a change in pull request #11711: [FLINK-17103][table-planner-blink] Supports dynamic options for Blink…
Posted by GitBox <gi...@apache.org>.
KurtYoung commented on a change in pull request #11711: [FLINK-17103][table-planner-blink] Supports dynamic options for Blink…
URL: https://github.com/apache/flink/pull/11711#discussion_r407485717
##########
File path: flink-table/flink-table-api-java/src/main/java/org/apache/flink/table/operations/CatalogSinkModifyOperation.java
##########
@@ -37,20 +37,23 @@
private final Map<String, String> staticPartitions;
private final QueryOperation child;
private final boolean overwrite;
+ private final List<?> tableHints;
Review comment:
I think we need to extract useful information from List<RelHint> and place it here.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #11711: [FLINK-17103][table-planner-blink] Supports dynamic options for Blink…
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on issue #11711: [FLINK-17103][table-planner-blink] Supports dynamic options for Blink…
URL: https://github.com/apache/flink/pull/11711#issuecomment-612758431
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "bffb4b07100999b88c67103a1c77d0609c524de9",
"status" : "PENDING",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/160012254",
"triggerID" : "bffb4b07100999b88c67103a1c77d0609c524de9",
"triggerType" : "PUSH"
}, {
"hash" : "bffb4b07100999b88c67103a1c77d0609c524de9",
"status" : "PENDING",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7381",
"triggerID" : "bffb4b07100999b88c67103a1c77d0609c524de9",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* bffb4b07100999b88c67103a1c77d0609c524de9 Travis: [PENDING](https://travis-ci.com/github/flink-ci/flink/builds/160012254) Azure: [PENDING](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7381)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #11711: [FLINK-17103][table-planner-blink] Supports dynamic options for Blink…
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on issue #11711: [FLINK-17103][table-planner-blink] Supports dynamic options for Blink…
URL: https://github.com/apache/flink/pull/11711#issuecomment-612758431
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "bffb4b07100999b88c67103a1c77d0609c524de9",
"status" : "FAILURE",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/160012254",
"triggerID" : "bffb4b07100999b88c67103a1c77d0609c524de9",
"triggerType" : "PUSH"
}, {
"hash" : "bffb4b07100999b88c67103a1c77d0609c524de9",
"status" : "FAILURE",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7381",
"triggerID" : "bffb4b07100999b88c67103a1c77d0609c524de9",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* bffb4b07100999b88c67103a1c77d0609c524de9 Travis: [FAILURE](https://travis-ci.com/github/flink-ci/flink/builds/160012254) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7381)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #11711: [FLINK-17103][table-planner-blink] Supports dynamic options for Blink…
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on issue #11711: [FLINK-17103][table-planner-blink] Supports dynamic options for Blink…
URL: https://github.com/apache/flink/pull/11711#issuecomment-612758431
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "bffb4b07100999b88c67103a1c77d0609c524de9",
"status" : "DELETED",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/160012254",
"triggerID" : "bffb4b07100999b88c67103a1c77d0609c524de9",
"triggerType" : "PUSH"
}, {
"hash" : "bffb4b07100999b88c67103a1c77d0609c524de9",
"status" : "DELETED",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7381",
"triggerID" : "bffb4b07100999b88c67103a1c77d0609c524de9",
"triggerType" : "PUSH"
}, {
"hash" : "3ea01f3d73194e787d483bc463ddaa5b58a3dd9f",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7456",
"triggerID" : "3ea01f3d73194e787d483bc463ddaa5b58a3dd9f",
"triggerType" : "PUSH"
}, {
"hash" : "3ea01f3d73194e787d483bc463ddaa5b58a3dd9f",
"status" : "SUCCESS",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/160195594",
"triggerID" : "3ea01f3d73194e787d483bc463ddaa5b58a3dd9f",
"triggerType" : "PUSH"
}, {
"hash" : "84098c147d17a469c44710ba87fbac41a9aa387d",
"status" : "PENDING",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/160326996",
"triggerID" : "84098c147d17a469c44710ba87fbac41a9aa387d",
"triggerType" : "PUSH"
}, {
"hash" : "84098c147d17a469c44710ba87fbac41a9aa387d",
"status" : "PENDING",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7492",
"triggerID" : "84098c147d17a469c44710ba87fbac41a9aa387d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 3ea01f3d73194e787d483bc463ddaa5b58a3dd9f Travis: [SUCCESS](https://travis-ci.com/github/flink-ci/flink/builds/160195594) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7456)
* 84098c147d17a469c44710ba87fbac41a9aa387d Travis: [PENDING](https://travis-ci.com/github/flink-ci/flink/builds/160326996) Azure: [PENDING](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7492)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] danny0405 commented on a change in pull request #11711: [FLINK-17103][table-planner-blink] Supports dynamic options for Blink…
Posted by GitBox <gi...@apache.org>.
danny0405 commented on a change in pull request #11711: [FLINK-17103][table-planner-blink] Supports dynamic options for Blink…
URL: https://github.com/apache/flink/pull/11711#discussion_r407405962
##########
File path: flink-table/flink-table-api-java/src/main/java/org/apache/flink/table/operations/CatalogSinkModifyOperation.java
##########
@@ -37,20 +37,23 @@
private final Map<String, String> staticPartitions;
private final QueryOperation child;
private final boolean overwrite;
+ private final List<?> tableHints;
Review comment:
The module `flink-table-api-java` has no Calcite as dependency, use `List<?>` instead of `List` to fix the compiler warnings.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] danny0405 commented on a change in pull request #11711: [FLINK-17103][table-planner-blink] Supports dynamic options for Blink…
Posted by GitBox <gi...@apache.org>.
danny0405 commented on a change in pull request #11711: [FLINK-17103][table-planner-blink] Supports dynamic options for Blink…
URL: https://github.com/apache/flink/pull/11711#discussion_r407832593
##########
File path: flink-table/flink-table-api-java/src/main/java/org/apache/flink/table/operations/CatalogSinkModifyOperation.java
##########
@@ -37,20 +37,23 @@
private final Map<String, String> staticPartitions;
private final QueryOperation child;
private final boolean overwrite;
+ private final List<?> tableHints;
Review comment:
The `RelHint` itself is a POJO which has hint name and hint options as attributes, passing through these `RelHint`s and hand-over them to planner for specific manipulation can simplify the codes.
But i'm also fine if we pre-process these `RelHint`s and extract the contents we care about, such as the dynamic options or some other attributes (index in the future) directly in `CatalogSinkModifyOperation`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] danny0405 commented on a change in pull request #11711: [FLINK-17103][table-planner-blink] Supports dynamic options for Blink…
Posted by GitBox <gi...@apache.org>.
danny0405 commented on a change in pull request #11711: [FLINK-17103][table-planner-blink] Supports dynamic options for Blink…
URL: https://github.com/apache/flink/pull/11711#discussion_r408587966
##########
File path: flink-table/flink-table-planner-blink/src/test/scala/org/apache/flink/table/planner/catalog/CatalogTableITCase.scala
##########
@@ -250,6 +254,71 @@ class CatalogTableITCase(isStreamingMode: Boolean) extends AbstractTestBase {
assertEquals(expected, FileUtils.readFileUtf8(new File(new URI(sinkFilePath))))
}
+ @Test
+ def testReadWriteCsvWithDynamicTableOptions(): Unit = {
+ val csvRecords = Seq(
+ "2.02,Euro,2019-12-12 00:00:01.001001",
+ "1.11,US Dollar,2019-12-12 00:00:02.002001",
+ "50,Yen,2019-12-12 00:00:04.004001",
+ "3.1,Euro,2019-12-12 00:00:05.005001",
+ "5.33,US Dollar,2019-12-12 00:00:06.006001"
+ )
+ val tempFilePath = createTempFile(
+ "csv-order-test",
+ csvRecords.mkString("#"))
+ val sourceDDL =
+ s"""
+ |CREATE TABLE T1 (
+ | price DECIMAL(10, 2),
+ | currency STRING,
+ | ts6 TIMESTAMP(6),
+ | ts AS CAST(ts6 AS TIMESTAMP(3)),
+ | WATERMARK FOR ts AS ts
+ |) WITH (
+ | 'connector.type' = 'filesystem',
+ | 'connector.path' = '$tempFilePath',
+ | 'format.type' = 'csv',
+ | 'format.field-delimiter' = ','
+ |)
+ """.stripMargin
+ tableEnv.sqlUpdate(sourceDDL)
+
+ val sinkFilePath = getTempFilePath("csv-order-sink")
+ val sinkDDL =
+ s"""
+ |CREATE TABLE T2 (
+ | window_end TIMESTAMP(3),
+ | max_ts TIMESTAMP(6),
+ | counter BIGINT,
+ | total_price DECIMAL(10, 2)
+ |) with (
+ | 'connector.type' = 'filesystem',
+ | 'connector.path' = '$sinkFilePath',
+ | 'format.type' = 'csv'
+ |)
+ """.stripMargin
+ tableEnv.sqlUpdate(sinkDDL)
+
+ val query =
+ """
+ |INSERT INTO T2 /*+ OPTIONS('format.field-delimiter' = ',') */
Review comment:
Agree, thanks.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] KurtYoung commented on a change in pull request #11711: [FLINK-17103][table-planner-blink] Supports dynamic options for Blink…
Posted by GitBox <gi...@apache.org>.
KurtYoung commented on a change in pull request #11711: [FLINK-17103][table-planner-blink] Supports dynamic options for Blink…
URL: https://github.com/apache/flink/pull/11711#discussion_r407355643
##########
File path: flink-table/flink-table-common/src/main/java/org/apache/flink/table/catalog/CatalogTable.java
##########
@@ -46,4 +46,11 @@
* @return a map of properties
*/
Map<String, String> toProperties();
+
+ /**
+ * Returns a copy of this {@code CatalogTable} with given table options {@code options}.
+ *
+ * @return a new copy of this table with replaced table options
+ */
+ CatalogTable copy(Map<String, String> options);
Review comment:
use `properties` to be aligned with other places?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #11711: [FLINK-17103][table-planner-blink] Supports dynamic options for Blink…
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on issue #11711: [FLINK-17103][table-planner-blink] Supports dynamic options for Blink…
URL: https://github.com/apache/flink/pull/11711#issuecomment-612758431
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "bffb4b07100999b88c67103a1c77d0609c524de9",
"status" : "FAILURE",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/160012254",
"triggerID" : "bffb4b07100999b88c67103a1c77d0609c524de9",
"triggerType" : "PUSH"
}, {
"hash" : "bffb4b07100999b88c67103a1c77d0609c524de9",
"status" : "PENDING",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7381",
"triggerID" : "bffb4b07100999b88c67103a1c77d0609c524de9",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* bffb4b07100999b88c67103a1c77d0609c524de9 Travis: [FAILURE](https://travis-ci.com/github/flink-ci/flink/builds/160012254) Azure: [PENDING](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7381)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] KurtYoung commented on a change in pull request #11711: [FLINK-17103][table-planner-blink] Supports dynamic options for Blink…
Posted by GitBox <gi...@apache.org>.
KurtYoung commented on a change in pull request #11711: [FLINK-17103][table-planner-blink] Supports dynamic options for Blink…
URL: https://github.com/apache/flink/pull/11711#discussion_r407355268
##########
File path: flink-table/flink-table-api-java/src/main/java/org/apache/flink/table/api/config/TableConfigOptions.java
##########
@@ -0,0 +1,43 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.api.config;
+
+import org.apache.flink.annotation.docs.Documentation;
+import org.apache.flink.configuration.ConfigOption;
+
+import static org.apache.flink.configuration.ConfigOptions.key;
+
+/**
+ * This class holds {@link org.apache.flink.configuration.ConfigOption}s used by
+ * Flink's BLINK table planner.
Review comment:
Not necessary to emphasize this
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #11711: [FLINK-17103][table-planner-blink] Supports dynamic options for Blink…
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on issue #11711: [FLINK-17103][table-planner-blink] Supports dynamic options for Blink…
URL: https://github.com/apache/flink/pull/11711#issuecomment-612758431
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "bffb4b07100999b88c67103a1c77d0609c524de9",
"status" : "FAILURE",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/160012254",
"triggerID" : "bffb4b07100999b88c67103a1c77d0609c524de9",
"triggerType" : "PUSH"
}, {
"hash" : "bffb4b07100999b88c67103a1c77d0609c524de9",
"status" : "FAILURE",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7381",
"triggerID" : "bffb4b07100999b88c67103a1c77d0609c524de9",
"triggerType" : "PUSH"
}, {
"hash" : "3ea01f3d73194e787d483bc463ddaa5b58a3dd9f",
"status" : "PENDING",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7456",
"triggerID" : "3ea01f3d73194e787d483bc463ddaa5b58a3dd9f",
"triggerType" : "PUSH"
}, {
"hash" : "3ea01f3d73194e787d483bc463ddaa5b58a3dd9f",
"status" : "PENDING",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/160195594",
"triggerID" : "3ea01f3d73194e787d483bc463ddaa5b58a3dd9f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* bffb4b07100999b88c67103a1c77d0609c524de9 Travis: [FAILURE](https://travis-ci.com/github/flink-ci/flink/builds/160012254) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7381)
* 3ea01f3d73194e787d483bc463ddaa5b58a3dd9f Travis: [PENDING](https://travis-ci.com/github/flink-ci/flink/builds/160195594) Azure: [PENDING](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7456)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #11711: [FLINK-17103][table-planner-blink] Supports dynamic options for Blink…
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on issue #11711: [FLINK-17103][table-planner-blink] Supports dynamic options for Blink…
URL: https://github.com/apache/flink/pull/11711#issuecomment-612758431
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "bffb4b07100999b88c67103a1c77d0609c524de9",
"status" : "DELETED",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/160012254",
"triggerID" : "bffb4b07100999b88c67103a1c77d0609c524de9",
"triggerType" : "PUSH"
}, {
"hash" : "bffb4b07100999b88c67103a1c77d0609c524de9",
"status" : "DELETED",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7381",
"triggerID" : "bffb4b07100999b88c67103a1c77d0609c524de9",
"triggerType" : "PUSH"
}, {
"hash" : "3ea01f3d73194e787d483bc463ddaa5b58a3dd9f",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7456",
"triggerID" : "3ea01f3d73194e787d483bc463ddaa5b58a3dd9f",
"triggerType" : "PUSH"
}, {
"hash" : "3ea01f3d73194e787d483bc463ddaa5b58a3dd9f",
"status" : "SUCCESS",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/160195594",
"triggerID" : "3ea01f3d73194e787d483bc463ddaa5b58a3dd9f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 3ea01f3d73194e787d483bc463ddaa5b58a3dd9f Travis: [SUCCESS](https://travis-ci.com/github/flink-ci/flink/builds/160195594) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7456)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] danny0405 commented on a change in pull request #11711: [FLINK-17103][table-planner-blink] Supports dynamic options for Blink…
Posted by GitBox <gi...@apache.org>.
danny0405 commented on a change in pull request #11711: [FLINK-17103][table-planner-blink] Supports dynamic options for Blink…
URL: https://github.com/apache/flink/pull/11711#discussion_r407406550
##########
File path: flink-table/flink-table-common/src/main/java/org/apache/flink/table/catalog/CatalogTable.java
##########
@@ -46,4 +46,11 @@
* @return a map of properties
*/
Map<String, String> toProperties();
+
+ /**
+ * Returns a copy of this {@code CatalogTable} with given table options {@code options}.
+ *
+ * @return a new copy of this table with replaced table options
+ */
+ CatalogTable copy(Map<String, String> options);
Review comment:
As discussed in the mailing list, `options` is our new preference to describe the table properties.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] KurtYoung commented on a change in pull request #11711: [FLINK-17103][table-planner-blink] Supports dynamic options for Blink…
Posted by GitBox <gi...@apache.org>.
KurtYoung commented on a change in pull request #11711: [FLINK-17103][table-planner-blink] Supports dynamic options for Blink…
URL: https://github.com/apache/flink/pull/11711#discussion_r407360539
##########
File path: flink-table/flink-table-api-java/src/main/java/org/apache/flink/table/catalog/ConnectorCatalogTable.java
##########
@@ -102,6 +102,11 @@ public boolean isBatch() {
throw new UnsupportedOperationException("ConnectorCatalogTable cannot be converted to properties");
}
+ @Override
+ public CatalogTable copy(Map<String, String> options) {
+ throw new UnsupportedOperationException("ConnectorCatalogTable cannot copy with new table options");
Review comment:
Should we also support the path of `TableEnvironment.connect` to specify table option?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] KurtYoung commented on a change in pull request #11711: [FLINK-17103][table-planner-blink] Supports dynamic options for Blink…
Posted by GitBox <gi...@apache.org>.
KurtYoung commented on a change in pull request #11711: [FLINK-17103][table-planner-blink] Supports dynamic options for Blink…
URL: https://github.com/apache/flink/pull/11711#discussion_r408121554
##########
File path: flink-table/flink-table-planner-blink/src/test/scala/org/apache/flink/table/planner/catalog/CatalogTableITCase.scala
##########
@@ -250,6 +254,71 @@ class CatalogTableITCase(isStreamingMode: Boolean) extends AbstractTestBase {
assertEquals(expected, FileUtils.readFileUtf8(new File(new URI(sinkFilePath))))
}
+ @Test
+ def testReadWriteCsvWithDynamicTableOptions(): Unit = {
+ val csvRecords = Seq(
+ "2.02,Euro,2019-12-12 00:00:01.001001",
+ "1.11,US Dollar,2019-12-12 00:00:02.002001",
+ "50,Yen,2019-12-12 00:00:04.004001",
+ "3.1,Euro,2019-12-12 00:00:05.005001",
+ "5.33,US Dollar,2019-12-12 00:00:06.006001"
+ )
+ val tempFilePath = createTempFile(
+ "csv-order-test",
+ csvRecords.mkString("#"))
+ val sourceDDL =
+ s"""
+ |CREATE TABLE T1 (
+ | price DECIMAL(10, 2),
+ | currency STRING,
+ | ts6 TIMESTAMP(6),
+ | ts AS CAST(ts6 AS TIMESTAMP(3)),
+ | WATERMARK FOR ts AS ts
+ |) WITH (
+ | 'connector.type' = 'filesystem',
+ | 'connector.path' = '$tempFilePath',
+ | 'format.type' = 'csv',
+ | 'format.field-delimiter' = ','
+ |)
+ """.stripMargin
+ tableEnv.sqlUpdate(sourceDDL)
+
+ val sinkFilePath = getTempFilePath("csv-order-sink")
+ val sinkDDL =
+ s"""
+ |CREATE TABLE T2 (
+ | window_end TIMESTAMP(3),
+ | max_ts TIMESTAMP(6),
+ | counter BIGINT,
+ | total_price DECIMAL(10, 2)
+ |) with (
+ | 'connector.type' = 'filesystem',
+ | 'connector.path' = '$sinkFilePath',
+ | 'format.type' = 'csv'
+ |)
+ """.stripMargin
+ tableEnv.sqlUpdate(sinkDDL)
+
+ val query =
+ """
+ |INSERT INTO T2 /*+ OPTIONS('format.field-delimiter' = ',') */
Review comment:
`,` is actually the default delimiter even if you didn't configure it. Use another one to make sure this take affect.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services